2023-10-07T09:06:54.596 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-07T09:06:54.596 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --name yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314 -- /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/orig.config.yaml 2023-10-07T09:06:54.643 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-07T09:06:54.729 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314 branch: wip-yuri-testing-2023-10-06-0949-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: yweinste@redhat.com first_in_suite: false job_id: '7416314' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-10-06-0949-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 08cac0187919fc3a247efdf445f652e2fb30a647 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 08cac0187919fc3a247efdf445f652e2fb30a647 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2023-10-06-0949-quincy sha1: 08cac0187919fc3a247efdf445f652e2fb30a647 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4726 sha1: 08cac0187919fc3a247efdf445f652e2fb30a647 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri-testing-2023-10-06-0949-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_08cac0187919fc3a247efdf445f652e2fb30a647/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 08cac0187919fc3a247efdf445f652e2fb30a647 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJqOOCB35WNDUTDN3/gKRrTNq3QXQuxExXAesdmjvfolG4pVfUmnkky6PTTDSz8FqLkW8ZeCQoLktd/N/CGeKHE= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUGCvz9Yf0ykUhRBF9kRUjcEMHDC/y8DnTqecvi6cGh0BQfJacelvQWPLe7dcH3OM70m6zxsFLxJT+22gJWLQ0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-07_00:40:05 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-07T09:06:54.729 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_08cac0187919fc3a247efdf445f652e2fb30a647/qa; will attempt to use it 2023-10-07T09:06:54.730 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_08cac0187919fc3a247efdf445f652e2fb30a647/qa/tasks 2023-10-07T09:06:54.730 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-07T09:06:54.732 INFO:teuthology.task.internal:Checking packages... 2023-10-07T09:06:54.756 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '08cac0187919fc3a247efdf445f652e2fb30a647' 2023-10-07T09:06:54.756 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-07T09:06:54.756 INFO:teuthology.packaging:ref: None 2023-10-07T09:06:54.756 INFO:teuthology.packaging:tag: None 2023-10-07T09:06:54.756 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-06-0949-quincy 2023-10-07T09:06:54.756 INFO:teuthology.packaging:sha1: 08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-07T09:06:54.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-10-06-0949-quincy 2023-10-07T09:06:54.983 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1320.g08cac018 2023-10-07T09:06:54.984 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-07T09:06:54.989 INFO:teuthology.task.internal:no buildpackages task found 2023-10-07T09:06:54.990 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-07T09:06:55.004 INFO:teuthology.task.internal:Saving configuration 2023-10-07T09:06:55.019 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-07T09:06:55.025 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-07T09:06:55.051 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-07 09:01:25.049103', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJqOOCB35WNDUTDN3/gKRrTNq3QXQuxExXAesdmjvfolG4pVfUmnkky6PTTDSz8FqLkW8ZeCQoLktd/N/CGeKHE='} 2023-10-07T09:06:55.071 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-07 09:01:25.047323', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUGCvz9Yf0ykUhRBF9kRUjcEMHDC/y8DnTqecvi6cGh0BQfJacelvQWPLe7dcH3OM70m6zxsFLxJT+22gJWLQ0='} 2023-10-07T09:06:55.071 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-07T09:06:55.078 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-07T09:06:55.078 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-07T09:06:55.078 INFO:teuthology.run_tasks:Running task console_log... 2023-10-07T09:06:55.147 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe8fbd2bb80>, signals=[15]) 2023-10-07T09:06:55.148 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-07T09:06:55.154 INFO:teuthology.task.internal:Opening connections... 2023-10-07T09:06:55.154 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:06:55.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:06:55.223 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:06:55.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:06:55.292 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-07T09:06:55.297 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-10-07T09:06:55.320 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-10-07T09:06:55.320 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-07T09:06:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-10-07T09:06:55.383 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-07T09:06:55.383 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-07T09:06:55.383 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-07T09:06:55.383 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-10-07T09:06:55.410 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2023-10-07T09:06:55.433 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2023-10-07T09:06:55.433 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="8" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="8" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el8" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2023-10-07T09:06:55.495 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-07T09:06:55.496 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-07T09:06:55.496 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-07T09:06:55.496 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2023-10-07T09:06:55.518 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-07T09:06:55.526 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-07T09:06:55.531 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-07T09:06:55.532 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-10-07T09:06:55.534 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2023-10-07T09:06:55.556 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-07T09:06:55.561 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-07T09:06:55.562 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-10-07T09:06:55.595 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2023-10-07T09:06:55.628 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-07T09:06:55.678 INFO:teuthology.run_tasks:Running task kernel... 2023-10-07T09:06:55.762 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-07T09:06:55.762 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2023-10-07T09:06:55.762 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2023-10-07T09:06:55.763 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-07T09:06:55.763 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-07T09:06:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:06:55.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-10-07T09:06:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:06:55.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-10-07T09:06:55.955 DEBUG:teuthology.task.kernel:sha1 for client.1 is fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c 2023-10-07T09:06:55.955 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c"... 2023-10-07T09:06:55.955 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2023-10-07T09:06:55.978 INFO:teuthology.orchestra.run.smithi144.stdout:4.18.0-489.el8.x86_64 2023-10-07T09:06:55.978 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c 2023-10-07T09:06:55.979 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-07T09:06:55.979 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum upgrade -y linux-firmware 2023-10-07T09:06:56.023 DEBUG:teuthology.task.kernel:sha1 for client.0 is fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c 2023-10-07T09:06:56.023 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c"... 2023-10-07T09:06:56.023 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-10-07T09:06:56.046 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-489.el8.x86_64 2023-10-07T09:06:56.047 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c 2023-10-07T09:06:56.047 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-07T09:06:56.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum upgrade -y linux-firmware 2023-10-07T09:07:02.275 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - AppStream 9.1 MB/s | 33 MB 00:03 2023-10-07T09:07:04.336 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 9.2 MB/s | 33 MB 00:03 2023-10-07T09:07:09.708 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - BaseOS 38 MB/s | 48 MB 00:01 2023-10-07T09:07:12.231 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 33 MB/s | 48 MB 00:01 2023-10-07T09:09:36.606 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 133 B/s | 18 kB 02:17 2023-10-07T09:09:37.365 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-10-07T09:09:37.675 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-07T09:09:38.934 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-07T09:09:43.006 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 264 kB/s | 24 kB 00:00 2023-10-07T09:09:47.162 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout: linux-firmware noarch 20230824-118.git0e048b06.el8 baseos 286 M 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 1 Package 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:09:47.163 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 286 M 2023-10-07T09:09:47.164 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-10-07T09:09:55.691 INFO:teuthology.orchestra.run.smithi029.stdout:linux-firmware-20230824-118.git0e048b06.el8.noa 34 MB/s | 286 MB 00:08 2023-10-07T09:09:55.691 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-10-07T09:09:55.691 INFO:teuthology.orchestra.run.smithi029.stdout:Total 34 MB/s | 286 MB 00:08 2023-10-07T09:09:57.483 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-10-07T09:09:57.580 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-10-07T09:09:57.581 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-10-07T09:09:59.457 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-10-07T09:09:59.480 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-10-07T09:10:06.790 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - Extras 107 B/s | 18 kB 02:50 2023-10-07T09:10:07.382 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-10-07T09:10:07.693 INFO:teuthology.orchestra.run.smithi144.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-10-07T09:10:09.812 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 16 MB 00:01 2023-10-07T09:10:13.734 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 395 kB/s | 24 kB 00:00 2023-10-07T09:10:17.488 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout: linux-firmware noarch 20230824-118.git0e048b06.el8 baseos 286 M 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-10-07T09:10:17.489 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-10-07T09:10:17.490 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 1 Package 2023-10-07T09:10:17.490 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:10:17.490 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 286 M 2023-10-07T09:10:17.490 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2023-10-07T09:10:20.514 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-10-07T09:10:20.963 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : linux-firmware-20230824-118.git0e048b06.el8.noarch 1/2 2023-10-07T09:10:21.254 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2023-10-07T09:10:21.501 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2023-10-07T09:10:21.501 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : linux-firmware-20230824-118.git0e048b06.el8.noarch 1/2 2023-10-07T09:10:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2023-10-07T09:10:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:10:22.317 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-10-07T09:10:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: linux-firmware-20230824-118.git0e048b06.el8.noarch 2023-10-07T09:10:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:10:22.317 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-10-07T09:10:22.509 INFO:teuthology.task.kernel:Downloading kernel fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c on client.0... 2023-10-07T09:10:22.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c 2023-10-07T09:10:22.736 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c/centos/8/flavors/default/x86_64/ 2023-10-07T09:10:22.736 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc7_gfef64f93da2b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c/centos/8/flavors/default/x86_64/ --input-file=- 2023-10-07T09:10:22.738 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi029.front.sepia.ceph.com to complete... 2023-10-07T09:10:23.152 INFO:teuthology.orchestra.run.smithi144.stdout:linux-firmware-20230824-118.git0e048b06.el8.noa 51 MB/s | 286 MB 00:05 2023-10-07T09:10:23.152 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2023-10-07T09:10:23.152 INFO:teuthology.orchestra.run.smithi144.stdout:Total 51 MB/s | 286 MB 00:05 2023-10-07T09:10:24.851 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2023-10-07T09:10:24.914 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2023-10-07T09:10:24.915 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2023-10-07T09:10:26.705 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2023-10-07T09:10:26.734 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2023-10-07T09:10:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07 09:10:35 URL:https://3.chacra.ceph.com/r/kernel/testing/fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c/centos/8/flavors/default/x86_64/kernel-6.5.0_rc7_gfef64f93da2b-1.x86_64.rpm [380759012/380759012] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-07T09:10:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:FINISHED --2023-10-07 09:10:35-- 2023-10-07T09:10:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:Total wall clock time: 13s 2023-10-07T09:10:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:Downloaded: 1 files, 363M in 13s (28.9 MB/s) 2023-10-07T09:10:35.658 INFO:teuthology.task.kernel:Installing kernel fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c on client.0... 2023-10-07T09:10:35.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-07T09:10:37.408 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying... ######################################## 2023-10-07T09:10:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing... ######################################## 2023-10-07T09:10:37.571 INFO:teuthology.orchestra.run.smithi029.stdout:Updating / installing... 2023-10-07T09:10:45.720 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2023-10-07T09:10:46.062 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : linux-firmware-20230824-118.git0e048b06.el8.noarch 1/2 2023-10-07T09:10:46.291 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2023-10-07T09:10:46.518 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2023-10-07T09:10:46.518 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : linux-firmware-20230824-118.git0e048b06.el8.noarch 1/2 2023-10-07T09:10:47.261 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2023-10-07T09:10:47.261 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:10:47.261 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2023-10-07T09:10:47.261 INFO:teuthology.orchestra.run.smithi144.stdout: linux-firmware-20230824-118.git0e048b06.el8.noarch 2023-10-07T09:10:47.261 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:10:47.262 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-10-07T09:10:47.472 INFO:teuthology.task.kernel:Downloading kernel fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c on client.1... 2023-10-07T09:10:47.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c 2023-10-07T09:10:47.630 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c/centos/8/flavors/default/x86_64/ 2023-10-07T09:10:47.630 DEBUG:teuthology.orchestra.run.smithi144:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc7_gfef64f93da2b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c/centos/8/flavors/default/x86_64/ --input-file=- 2023-10-07T09:10:47.632 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi144.front.sepia.ceph.com to complete... 2023-10-07T09:11:00.690 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07 09:11:00 URL:https://3.chacra.ceph.com/r/kernel/testing/fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c/centos/8/flavors/default/x86_64/kernel-6.5.0_rc7_gfef64f93da2b-1.x86_64.rpm [380759012/380759012] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-07T09:11:00.690 INFO:teuthology.orchestra.run.smithi144.stderr:FINISHED --2023-10-07 09:11:00-- 2023-10-07T09:11:00.690 INFO:teuthology.orchestra.run.smithi144.stderr:Total wall clock time: 13s 2023-10-07T09:11:00.690 INFO:teuthology.orchestra.run.smithi144.stderr:Downloaded: 1 files, 363M in 13s (28.6 MB/s) 2023-10-07T09:11:00.691 INFO:teuthology.task.kernel:Installing kernel fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c on client.1... 2023-10-07T09:11:00.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-07T09:11:02.394 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying... ######################################## 2023-10-07T09:11:02.438 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing... ######################################## 2023-10-07T09:11:02.548 INFO:teuthology.orchestra.run.smithi144.stdout:Updating / installing... 2023-10-07T09:11:18.323 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-6.5.0_rc7_gfef64f93da2b-1 ######################################## 2023-10-07T09:11:40.548 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-6.5.0_rc7_gfef64f93da2b-1 ######################################## 2023-10-07T09:11:57.600 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-07T09:11:57.601 DEBUG:teuthology.orchestra.run.smithi029:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/System.map-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/config-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/vmlinuz-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86 2023-10-07T09:11:57.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-07T09:11:57.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel/cpu 2023-10-07T09:11:57.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel/cpu/mce 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm/kvm.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/ansi_cprng.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/anubis.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/arc4.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-07T09:11:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_pq.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_tx.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_xor.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/raid6test.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/authenc.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/authencesn.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/blake2b_generic.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/blowfish_common.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/blowfish_generic.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/camellia_generic.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cast5_generic.ko 2023-10-07T09:11:57.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cast6_generic.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cast_common.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/ccm.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cmac.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/crc32_generic.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/crypto_engine.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/crypto_user.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cts.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/des_generic.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/dh_generic.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/echainiv.ko 2023-10-07T09:11:57.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/essiv.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/fcrypt.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/khazad.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/lz4.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/lz4hc.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/md4.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/michael_mic.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/pcbc.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/pcrypt.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/rmd160.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/seed.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/serpent_generic.ko 2023-10-07T09:11:57.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/tcrypt.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/tea.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/twofish_common.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/twofish_generic.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/vmac.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/wp512.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/xcbc.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/xor.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/xxhash_generic.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-07T09:11:57.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/acpi_pad.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/custom_method.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/ec_sys.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/platform_profile.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/sbs.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/sbshc.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/video.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/acard-ahci.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/ahci_platform.ko 2023-10-07T09:11:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/ata_generic.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/libahci_platform.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_acpi.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ali.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_amd.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_artop.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_atiixp.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_atp867x.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_cmd640.ko 2023-10-07T09:11:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_cypress.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_efar.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt366.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_it8213.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_it821x.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_jmicron.ko 2023-10-07T09:11:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_marvell.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_mpiix.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_netcell.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ninja32.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ns87410.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ns87415.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_opti.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_optidma.ko 2023-10-07T09:11:57.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_piccolo.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_rdc.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sch.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_serverworks.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sil680.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sis.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_triflex.ko 2023-10-07T09:11:57.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_via.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pdc_adma.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_inic162x.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_mv.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_nv.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_promise.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_qstor.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sil.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sil24.ko 2023-10-07T09:11:57.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sis.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_svw.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sx4.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_uli.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_via.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_vsc.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/auxdisplay 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base/regmap 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-07T09:11:57.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/bcma 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/bcma/bcma.ko 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/aoe 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/aoe/aoe.ko 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/brd.ko 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/drbd 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/drbd/drbd.ko 2023-10-07T09:11:57.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/floppy.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/loop.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/mtip32xx 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/nbd.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/null_blk 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/pktcdvd.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/rbd.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/virtio_blk.ko 2023-10-07T09:11:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/xen-blkback 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/xen-blkfront.ko 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/zram 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/zram/zram.ko 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hangcheck-timer.ko 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-07T09:11:57.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-07T09:11:57.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/lp.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/mwave 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/mwave/mwave.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ppdev.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tlclk.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-07T09:11:57.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/uv_mmtimer.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/virtio_console.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto 2023-10-07T09:11:57.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/ccp 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-07T09:11:57.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/padlock-aes.ko 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/padlock-sha.ko 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/virtio 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dca 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dca/dca.ko 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/devfreq 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma 2023-10-07T09:11:57.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/dw 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/ioat 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/amd64_edac.ko 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/e752x_edac.ko 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-07T09:11:57.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i3000_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i3200_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i5100_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i5400_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i7300_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i7core_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i82975x_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/ie31200_edac.ko 2023-10-07T09:11:57.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/sb_edac.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/x38_edac.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/firmware 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/firmware/edd.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/gpio 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/gpio/gpio-ich.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-appleir.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-aureal.ko 2023-10-07T09:11:57.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-axff.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-dr.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-elecom.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-elo.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-emsff.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-gaff.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-gt683r.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-gyration.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-holtekff.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-hyperv.ko 2023-10-07T09:11:57.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-icade.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-keytouch.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-kye.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-lcpower.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-led.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-lenovo.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-multitouch.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-ortek.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-penmount.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-petalynx.ko 2023-10-07T09:11:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-picolcd.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-pl.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-plantronics.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-primax.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-rmi.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-07T09:11:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-saitek.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-samsung.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sjoy.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sony.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-speedlink.ko 2023-10-07T09:11:57.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-steelseries.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sunplus.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-tivo.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-tmff.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-topseed.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-twinhan.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-uclogic.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-waltop.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-wiimote.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-xinmo.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-zpff.ko 2023-10-07T09:11:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-zydacron.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/uhid.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/wacom.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv/hv_balloon.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv/hv_utils.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv/hv_vmbus.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/abituguru.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/abituguru3.ko 2023-10-07T09:11:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ad7414.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ad7418.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adc128d818.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1025.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1026.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1029.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1031.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm9240.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ads7828.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7410.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7411.ko 2023-10-07T09:11:57.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7462.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7470.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7475.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7x10.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/amc6821.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/applesmc.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/asb100.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/asc7621.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/atxp1.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/coretemp.ko 2023-10-07T09:11:57.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/dme1737.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ds1621.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ds620.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/emc1403.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/emc6w201.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/f71805f.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/f71882fg.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/f75375s.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/fschmd.ko 2023-10-07T09:11:57.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/g760a.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/g762.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/gl518sm.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/gl520sm.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ibmaem.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ibmpex.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ina209.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ina2xx.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/it87.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/k10temp.ko 2023-10-07T09:11:57.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/k8temp.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm63.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm73.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm75.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm77.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm78.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm80.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm83.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm85.ko 2023-10-07T09:11:57.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm87.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm90.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm92.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm93.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm95234.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm95241.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm95245.ko 2023-10-07T09:11:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc2945.ko 2023-10-07T09:11:57.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4151.ko 2023-10-07T09:11:57.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4215.ko 2023-10-07T09:11:57.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4222.ko 2023-10-07T09:11:57.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4245.ko 2023-10-07T09:11:57.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4260.ko 2023-10-07T09:11:57.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4261.ko 2023-10-07T09:11:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max16065.ko 2023-10-07T09:11:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max1619.ko 2023-10-07T09:11:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max1668.ko 2023-10-07T09:11:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max197.ko 2023-10-07T09:11:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max6639.ko 2023-10-07T09:11:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max6650.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max6697.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/mcp3021.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct6683.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct6775.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct7802.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pc87360.ko 2023-10-07T09:11:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pc87427.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pcf8591.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-07T09:11:57.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/powr1220.ko 2023-10-07T09:11:57.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sch5627.ko 2023-10-07T09:11:57.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sch5636.ko 2023-10-07T09:11:57.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-07T09:11:57.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sht15.ko 2023-10-07T09:11:57.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sht21.ko 2023-10-07T09:11:57.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/shtc1.ko 2023-10-07T09:11:57.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sis5595.ko 2023-10-07T09:11:57.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-07T09:11:57.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-07T09:11:57.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-07T09:11:57.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/thmc50.ko 2023-10-07T09:11:57.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp102.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp103.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp401.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp421.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/via686a.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/vt1211.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/vt8231.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83627hf.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83781d.ko 2023-10-07T09:11:57.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83791d.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83792d.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83793.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83795.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/algos 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-07T09:11:57.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-07T09:11:57.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-07T09:11:57.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-dev.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-mux.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-stub.ko 2023-10-07T09:11:57.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw 2023-10-07T09:11:57.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/cxgb4 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx4 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx5 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mthca 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp 2023-10-07T09:11:57.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/ipoib 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/iser 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/isert 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srp 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srpt 2023-10-07T09:11:57.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/gameport.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/lightning.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/ns558.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/joydev.ko 2023-10-07T09:11:57.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/keyboard 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/appletouch.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/sermouse.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/rmi4 2023-10-07T09:11:57.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/serio_raw.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/sparse-keymap.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/iommu 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/iommu/amd 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-blinkm.ko 2023-10-07T09:11:57.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-lm3530.ko 2023-10-07T09:11:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-lp3944.ko 2023-10-07T09:11:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-ss4200.ko 2023-10-07T09:11:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger 2023-10-07T09:11:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-07T09:11:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-07T09:11:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-07T09:11:57.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-07T09:11:57.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-07T09:11:57.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-07T09:11:57.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-07T09:11:57.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-07T09:11:57.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md 2023-10-07T09:11:57.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/bcache 2023-10-07T09:11:57.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/bcache/bcache.ko 2023-10-07T09:11:57.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-bio-prison.ko 2023-10-07T09:11:57.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-cache-smq.ko 2023-10-07T09:11:57.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-cache.ko 2023-10-07T09:11:57.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-crypt.ko 2023-10-07T09:11:57.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-delay.ko 2023-10-07T09:11:57.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-flakey.ko 2023-10-07T09:11:57.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-log-userspace.ko 2023-10-07T09:11:57.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-multipath.ko 2023-10-07T09:11:57.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-queue-length.ko 2023-10-07T09:11:57.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-raid.ko 2023-10-07T09:11:57.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-round-robin.ko 2023-10-07T09:11:57.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-service-time.ko 2023-10-07T09:11:57.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-switch.ko 2023-10-07T09:11:57.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-thin-pool.ko 2023-10-07T09:11:57.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-verity.ko 2023-10-07T09:11:57.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/faulty.ko 2023-10-07T09:11:57.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/linear.ko 2023-10-07T09:11:57.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/multipath.ko 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/persistent-data 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid0.ko 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid1.ko 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid10.ko 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid456.ko 2023-10-07T09:11:57.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptbase.ko 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptctl.ko 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptfc.ko 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptlan.ko 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptsas.ko 2023-10-07T09:11:57.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptspi.ko 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/lpc_ich.ko 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/lpc_sch.ko 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/mfd-core.ko 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/sm501.ko 2023-10-07T09:11:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/viperboard.ko 2023-10-07T09:11:57.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/vx855.ko 2023-10-07T09:11:57.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/wl1273-core.ko 2023-10-07T09:11:57.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc 2023-10-07T09:11:57.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/altera-stapl 2023-10-07T09:11:57.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-07T09:11:57.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/apds9802als.ko 2023-10-07T09:11:57.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/apds990x.ko 2023-10-07T09:11:57.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/bh1770glc.ko 2023-10-07T09:11:57.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/cb710 2023-10-07T09:11:57.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/cb710/cb710.ko 2023-10-07T09:11:57.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom 2023-10-07T09:11:57.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/at24.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/enclosure.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/hpilo.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/isl29003.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/isl29020.ko 2023-10-07T09:11:57.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/lis3lv02d 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei/mei-me.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei/mei.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-gru 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-07T09:11:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/tifm_core.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/tsl2550.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/vmw_balloon.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/vmw_vmci 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/bonding 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/bonding/bonding.ko 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa 2023-10-07T09:11:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/b53 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dummy.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/eql.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-07T09:11:57.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/adaptec 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/agere 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/alteon 2023-10-07T09:11:57.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/altera 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/alx 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-07T09:11:57.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-07T09:11:57.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/brocade 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/brocade/bna 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-07T09:11:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/cisco 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/cisco/enic 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-07T09:11:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dlink 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dnet.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/emulex 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/emulex/benet 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-07T09:11:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/i40e 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/iavf 2023-10-07T09:11:57.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igb 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/jme.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-07T09:11:57.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-07T09:11:57.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/micrel 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/myricom 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/natsemi 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/neterion 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/nvidia 2023-10-07T09:11:57.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/packetengines 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-07T09:11:57.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rdc 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rocker 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sfc 2023-10-07T09:11:57.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/silan 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sis 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-07T09:11:57.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-07T09:11:57.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-07T09:11:57.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/tehuti 2023-10-07T09:11:57.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-07T09:11:57.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/ti 2023-10-07T09:11:57.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-07T09:11:57.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/via 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/wiznet 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/xircom 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/geneve.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/hyperv 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ieee802154 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-07T09:11:57.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ifb.ko 2023-10-07T09:11:57.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ipvlan 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/macvlan.ko 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/macvtap.ko 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio.ko 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-07T09:11:57.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mii.ko 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/net_failover.ko 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/netconsole.ko 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/nlmon.ko 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/pcs 2023-10-07T09:11:57.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-07T09:11:57.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-07T09:11:57.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy 2023-10-07T09:11:57.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/amd.ko 2023-10-07T09:11:57.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-07T09:11:57.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-07T09:11:57.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/broadcom.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/cicada.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/davicom.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/dp83640.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/et1011c.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/icplus.ko 2023-10-07T09:11:57.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/lxt.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/marvell.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/micrel.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/national.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/phylink.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/qsemi.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/realtek.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/smsc.ko 2023-10-07T09:11:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/vitesse.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-07T09:11:57.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/pppoe.ko 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/pppox.ko 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/pptp.ko 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/slip 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/slip/slhc.ko 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/slip/slip.ko 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/sungem_phy.ko 2023-10-07T09:11:57.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/tap.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_random.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-07T09:11:57.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/tun.ko 2023-10-07T09:11:57.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/veth.ko 2023-10-07T09:11:57.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/virtio_net.ko 2023-10-07T09:11:57.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vmxnet3 2023-10-07T09:11:57.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-07T09:11:57.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vxlan 2023-10-07T09:11:57.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/xen-netback 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/xen-netfront.ko 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ntb 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ntb/ntb.ko 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme/host 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme/host/nvme.ko 2023-10-07T09:11:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport.ko 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport_cs.ko 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport_pc.ko 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport_serial.ko 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/hotplug 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-07T09:11:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/pcie 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/xen-pcifront.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia/i82092.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia/pd6729.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pinctrl 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pinctrl/intel 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-07T09:11:57.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-07T09:11:57.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/hdaps.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/intel 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-07T09:11:57.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/wmi.ko 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/powercap 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps 2023-10-07T09:11:57.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/pps_core.ko 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ptp 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ptp/ptp.ko 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-07T09:11:57.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-bq4802.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-07T09:11:57.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-07T09:11:57.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-07T09:11:57.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-07T09:11:57.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-07T09:11:57.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-07T09:11:57.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-07T09:11:57.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-07T09:11:57.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi 2023-10-07T09:11:57.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-07T09:11:57.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/3w-sas.ko 2023-10-07T09:11:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-07T09:11:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/BusLogic.ko 2023-10-07T09:11:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/a100u2w.ko 2023-10-07T09:11:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aacraid 2023-10-07T09:11:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-07T09:11:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/advansys.ko 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aic7xxx 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/am53c974.ko 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/arcmsr 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-07T09:11:57.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/atp870u.ko 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/be2iscsi 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bfa 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2fc 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-07T09:11:57.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2i 2023-10-07T09:11:57.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-07T09:11:57.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ch.ko 2023-10-07T09:11:57.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/csiostor 2023-10-07T09:11:57.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-07T09:11:57.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi 2023-10-07T09:11:57.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-07T09:11:57.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-07T09:11:57.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-07T09:11:57.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-07T09:11:57.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-07T09:11:57.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/dc395x.ko 2023-10-07T09:11:57.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler 2023-10-07T09:11:57.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-07T09:11:57.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-07T09:11:57.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-07T09:11:57.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-07T09:11:57.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/dmx3191d.ko 2023-10-07T09:11:57.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/esas2r 2023-10-07T09:11:57.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-07T09:11:57.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/esp_scsi.ko 2023-10-07T09:11:57.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fcoe 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fnic 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/hpsa.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/hptiop.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/initio.ko 2023-10-07T09:11:57.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ipr.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ips.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/isci 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/isci/isci.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libfc 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libiscsi.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-07T09:11:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libsas 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/lpfc 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mpt3sas 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-07T09:11:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mvsas 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mvumi.ko 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/pm8001 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/pmcraid.ko 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla1280.ko 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla2xxx 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-07T09:11:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla4xxx 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/raid_class.ko 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_debug.ko 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-07T09:11:57.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ses.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/st.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/stex.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/sym53c8xx_2 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-07T09:11:57.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/wd719x.ko 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ssb 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ssb/ssb.ko 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/iscsi 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/loopback 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_file.ko 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_iblock.ko 2023-10-07T09:11:57.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_mod.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_pscsi.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_user.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/tcm_fc 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-07T09:11:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-07T09:11:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/ipwireless 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/n_gsm.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/n_hdlc.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/nozomi.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/8250 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-07T09:11:57.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/jsm 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/synclink_gt.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/core 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/host 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_aec.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_cif.ko 2023-10-07T09:11:57.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_sercos3.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/usblp.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/usbtmc.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/host 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-07T09:11:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/image 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/image/mdc800.ko 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/image/microtek.ko 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/adutux.ko 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/emi26.ko 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/emi62.ko 2023-10-07T09:11:57.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/ezusb.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/idmouse.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/ldusb.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/sisusbvga 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/usb3503.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/usblcd.ko 2023-10-07T09:11:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/uss720.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/yurex.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/phy 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/aircable.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ark3116.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ch341.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/cp210x.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-07T09:11:57.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/empeg.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/io_ti.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ipaq.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ipw.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/keyspan.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-07T09:11:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/mos7720.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/mos7840.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/navman.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/omninet.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/opticon.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/option.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/oti6858.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/pl2303.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/qcaux.ko 2023-10-07T09:11:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/qcserial.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/quatech2.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/sierra.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ssu100.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/visor.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-07T09:11:57.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/uas.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-07T09:11:57.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-07T09:11:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-07T09:11:57.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio 2023-10-07T09:11:57.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/pci 2023-10-07T09:11:57.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-07T09:11:57.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-07T09:11:57.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/vfio.ko 2023-10-07T09:11:57.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-07T09:11:57.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost 2023-10-07T09:11:57.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost.ko 2023-10-07T09:11:57.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-07T09:11:57.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost_net.ko 2023-10-07T09:11:57.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-07T09:11:57.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight/lcd.ko 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio.ko 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-07T09:11:57.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_pci.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_ring.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/masters 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/masters/ds2482.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/masters/ds2490.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-07T09:11:57.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-07T09:11:57.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-07T09:11:57.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/wire.ko 2023-10-07T09:11:57.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog 2023-10-07T09:11:57.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-07T09:11:57.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-07T09:11:57.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-07T09:11:57.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/hpwdt.ko 2023-10-07T09:11:57.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/i6300esb.ko 2023-10-07T09:11:57.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-07T09:11:57.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-07T09:11:57.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-07T09:11:57.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/ibmasr.ko 2023-10-07T09:11:57.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/machzwd.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/nv_tco.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-07T09:11:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/softdog.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/via_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-07T09:11:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-evtchn.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-gntdev.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-pciback 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-privcmd.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xenfs 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/9p 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/9p/9p.ko 2023-10-07T09:11:57.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/affs 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/affs/affs.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/befs 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/befs/befs.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/binfmt_misc.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/btrfs 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/btrfs/btrfs.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cachefiles 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cachefiles/cachefiles.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ceph 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ceph/ceph.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/coda 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/coda/coda.ko 2023-10-07T09:11:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cramfs 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cramfs/cramfs.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/dlm 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/dlm/dlm.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ecryptfs 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/f2fs 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/f2fs/f2fs.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat/fat.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat/msdos.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat/vfat.ko 2023-10-07T09:11:57.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fscache 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fscache/fscache.ko 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fuse 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fuse/cuse.ko 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fuse/fuse.ko 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/gfs2 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/gfs2/gfs2.ko 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfs 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfs/hfs.ko 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfsplus 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfsplus/hfsplus.ko 2023-10-07T09:11:57.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/isofs 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/isofs/isofs.ko 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/jfs 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/jfs/jfs.ko 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/lockd 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/lockd/lockd.ko 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/minix 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/minix/minix.ko 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/netfs 2023-10-07T09:11:57.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/netfs/netfs.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/blocklayout 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/filelayout 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/flexfilelayout 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/nfs.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/nfsv3.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/nfsv4.ko 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs_common 2023-10-07T09:11:57.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs_common/grace.ko 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfsd 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfsd/nfsd.ko 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nilfs2 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nilfs2/nilfs2.ko 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-celtic.ko 2023-10-07T09:11:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-centeuro.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-croatian.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-cyrillic.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-gaelic.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-greek.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-iceland.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-inuit.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-roman.ko 2023-10-07T09:11:57.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-romanian.ko 2023-10-07T09:11:57.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-turkish.ko 2023-10-07T09:11:57.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp1250.ko 2023-10-07T09:11:57.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp1251.ko 2023-10-07T09:11:57.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp1255.ko 2023-10-07T09:11:57.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp737.ko 2023-10-07T09:11:57.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp775.ko 2023-10-07T09:11:57.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp850.ko 2023-10-07T09:11:57.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp852.ko 2023-10-07T09:11:57.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp855.ko 2023-10-07T09:11:57.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp857.ko 2023-10-07T09:11:57.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp860.ko 2023-10-07T09:11:57.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp861.ko 2023-10-07T09:11:57.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp862.ko 2023-10-07T09:11:57.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp863.ko 2023-10-07T09:11:57.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp864.ko 2023-10-07T09:11:57.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp865.ko 2023-10-07T09:11:57.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp866.ko 2023-10-07T09:11:57.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp869.ko 2023-10-07T09:11:57.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp874.ko 2023-10-07T09:11:57.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp932.ko 2023-10-07T09:11:57.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp936.ko 2023-10-07T09:11:57.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp949.ko 2023-10-07T09:11:57.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp950.ko 2023-10-07T09:11:57.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_euc-jp.ko 2023-10-07T09:11:57.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-07T09:11:57.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-07T09:11:57.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-07T09:11:57.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-07T09:11:57.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-07T09:11:57.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-07T09:11:57.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-07T09:11:57.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_koi8-r.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_koi8-u.ko 2023-10-07T09:11:57.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_utf8.ko 2023-10-07T09:11:57.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2 2023-10-07T09:11:57.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/cluster 2023-10-07T09:11:57.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-07T09:11:57.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlm 2023-10-07T09:11:57.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-07T09:11:57.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlmfs 2023-10-07T09:11:57.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-07T09:11:57.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2.ko 2023-10-07T09:11:57.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-07T09:11:57.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-07T09:11:57.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-07T09:11:57.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/overlayfs 2023-10-07T09:11:57.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/overlayfs/overlay.ko 2023-10-07T09:11:57.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/pstore 2023-10-07T09:11:57.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/pstore/ramoops.ko 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/reiserfs 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/reiserfs/reiserfs.ko 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/romfs 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/romfs/romfs.ko 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/client 2023-10-07T09:11:57.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/client/cifs.ko 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/common 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/common/cifs_md4.ko 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/squashfs 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/squashfs/squashfs.ko 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/sysv 2023-10-07T09:11:57.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/sysv/sysv.ko 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/udf 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/udf/udf.ko 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ufs 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ufs/ufs.ko 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/xfs 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/xfs/xfs.ko 2023-10-07T09:11:57.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel 2023-10-07T09:11:57.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/rcu 2023-10-07T09:11:57.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/rcu/rcutorture.ko 2023-10-07T09:11:57.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/torture.ko 2023-10-07T09:11:57.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/trace 2023-10-07T09:11:57.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-07T09:11:57.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib 2023-10-07T09:11:57.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/asn1_encoder.ko 2023-10-07T09:11:57.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crc-itu-t.ko 2023-10-07T09:11:57.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crc8.ko 2023-10-07T09:11:57.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crypto 2023-10-07T09:11:57.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crypto/libarc4.ko 2023-10-07T09:11:57.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crypto/libdes.ko 2023-10-07T09:11:57.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/libcrc32c.ko 2023-10-07T09:11:57.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lru_cache.ko 2023-10-07T09:11:57.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lz4 2023-10-07T09:11:57.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lz4/lz4_compress.ko 2023-10-07T09:11:57.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/math 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/math/cordic.ko 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/raid6 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/raid6/raid6_pq.ko 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/reed_solomon 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-07T09:11:57.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/ts_bm.ko 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/ts_fsm.ko 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/ts_kmp.ko 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/zstd 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/zstd/zstd_compress.ko 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/mm 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/mm/hwpoison-inject.ko 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net 2023-10-07T09:11:57.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/6lowpan.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_dest.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_hop.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_routing.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_udp.ko 2023-10-07T09:11:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/garp.ko 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/mrp.ko 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/p8022.ko 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/psnap.ko 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/stp.ko 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/8021q 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/8021q/8021q.ko 2023-10-07T09:11:57.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p 2023-10-07T09:11:57.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet.ko 2023-10-07T09:11:57.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet_fd.ko 2023-10-07T09:11:57.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet_rdma.ko 2023-10-07T09:11:57.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet_virtio.ko 2023-10-07T09:11:57.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/batman-adv 2023-10-07T09:11:57.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/batman-adv/batman-adv.ko 2023-10-07T09:11:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge 2023-10-07T09:11:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/br_netfilter.ko 2023-10-07T09:11:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/bridge.ko 2023-10-07T09:11:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter 2023-10-07T09:11:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-07T09:11:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-07T09:11:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-07T09:11:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-07T09:11:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-07T09:11:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-07T09:11:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-07T09:11:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-07T09:11:57.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-07T09:11:57.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-07T09:11:57.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-07T09:11:57.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-07T09:11:57.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-07T09:11:57.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-07T09:11:57.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ceph 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ceph/libceph.ko 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/core 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/core/failover.ko 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/core/pktgen.ko 2023-10-07T09:11:57.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp.ko 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp_diag.ko 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp_ipv4.ko 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp_ipv6.ko 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dns_resolver 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-07T09:11:57.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa 2023-10-07T09:11:57.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/dsa_core.ko 2023-10-07T09:11:57.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_brcm.ko 2023-10-07T09:11:57.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_dsa.ko 2023-10-07T09:11:57.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_none.ko 2023-10-07T09:11:57.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_trailer.ko 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/6lowpan 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/ieee802154.ko 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ah4.ko 2023-10-07T09:11:57.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/esp4.ko 2023-10-07T09:11:57.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/fou.ko 2023-10-07T09:11:57.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/gre.ko 2023-10-07T09:11:57.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/inet_diag.ko 2023-10-07T09:11:57.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ip_gre.ko 2023-10-07T09:11:57.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ip_tunnel.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ip_vti.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ipcomp.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ipip.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-07T09:11:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-07T09:11:57.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-07T09:11:57.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_bic.ko 2023-10-07T09:11:57.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-07T09:11:57.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_diag.ko 2023-10-07T09:11:57.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-07T09:11:57.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_htcp.ko 2023-10-07T09:11:57.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_hybla.ko 2023-10-07T09:11:57.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_illinois.ko 2023-10-07T09:11:57.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_lp.ko 2023-10-07T09:11:57.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_scalable.ko 2023-10-07T09:11:57.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_vegas.ko 2023-10-07T09:11:57.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_veno.ko 2023-10-07T09:11:57.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_westwood.ko 2023-10-07T09:11:57.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_yeah.ko 2023-10-07T09:11:57.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tunnel4.ko 2023-10-07T09:11:57.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/udp_diag.ko 2023-10-07T09:11:57.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/udp_tunnel.ko 2023-10-07T09:11:57.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-07T09:11:57.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6 2023-10-07T09:11:57.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ah6.ko 2023-10-07T09:11:57.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/esp6.ko 2023-10-07T09:11:57.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/fou6.ko 2023-10-07T09:11:57.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-07T09:11:57.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ip6_vti.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ipcomp6.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-07T09:11:57.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-07T09:11:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-07T09:11:57.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/sit.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/tunnel6.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/key 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/key/af_key.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_core.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_eth.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_ip.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-07T09:11:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/llc 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/llc/llc.ko 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mac802154 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mac802154/mac802154.ko 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mpls 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mpls/mpls_gso.ko 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-07T09:11:57.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-07T09:11:57.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-07T09:11:57.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conncount.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-07T09:11:57.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_flow_table.ko 2023-10-07T09:11:57.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_tables.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-07T09:11:57.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_compat.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_ct.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_fib.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_hash.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_limit.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_log.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_masq.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_nat.ko 2023-10-07T09:11:57.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_numgen.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_queue.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_quota.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_redir.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_reject.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_socket.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_tunnel.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CT.ko 2023-10-07T09:11:57.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_DSCP.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_HL.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_HMARK.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_LED.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_LOG.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-07T09:11:57.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TEE.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TRACE.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_addrtype.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_bpf.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_cgroup.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_cluster.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_comment.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connbytes.ko 2023-10-07T09:11:57.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connlabel.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connlimit.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connmark.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_conntrack.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_cpu.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_dccp.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_devgroup.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_dscp.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_ecn.ko 2023-10-07T09:11:57.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_esp.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_helper.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_hl.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_iprange.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_ipvs.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_l2tp.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_length.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_limit.ko 2023-10-07T09:11:57.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_mac.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_mark.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_multiport.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_nat.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_nfacct.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_osf.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_owner.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_physdev.ko 2023-10-07T09:11:57.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_pkttype.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_policy.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_quota.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_rateest.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_realm.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_recent.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_sctp.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_set.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_socket.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_state.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_statistic.ko 2023-10-07T09:11:57.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_string.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_time.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_u32.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netlink 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netlink/netlink_diag.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/nsh 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/nsh/nsh.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/openvswitch.ko 2023-10-07T09:11:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/vport-geneve.ko 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/vport-gre.ko 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/packet 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/packet/af_packet_diag.ko 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds/rds.ko 2023-10-07T09:11:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds/rds_rdma.ko 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds/rds_tcp.ko 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_csum.ko 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_gact.ko 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_ipt.ko 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_mirred.ko 2023-10-07T09:11:57.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_nat.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_pedit.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_police.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_simple.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_skbedit.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_vlan.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_basic.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_bpf.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_flow.ko 2023-10-07T09:11:57.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_fw.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_route.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_u32.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_cmp.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_ipset.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_meta.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_nbyte.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_text.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_u32.ko 2023-10-07T09:11:57.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_choke.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_codel.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_drr.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_fq.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_gred.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_hfsc.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_hhf.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_htb.ko 2023-10-07T09:11:57.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_ingress.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_mqprio.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_multiq.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_netem.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_pie.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_plug.ko 2023-10-07T09:11:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_prio.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_qfq.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_red.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_sfb.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_sfq.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_tbf.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_teql.ko 2023-10-07T09:11:57.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sctp 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sctp/sctp.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sctp/sctp_diag.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/auth_gss 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/sunrpc.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/xprtrdma 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/tipc 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/tipc/diag.ko 2023-10-07T09:11:57.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/tipc/tipc.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/unix 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/unix/unix_diag.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vsock.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/xfrm 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-07T09:11:57.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/encrypted-keys 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/trusted-keys 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/virt 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/virt/lib 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/virt/lib/irqbypass.ko 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.alias 2023-10-07T09:11:57.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.alias.bin 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.builtin 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.builtin.bin 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.builtin.modinfo 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.dep 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.dep.bin 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.devname 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.order 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.softdep 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.symbols 2023-10-07T09:11:57.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.symbols.bin 2023-10-07T09:11:57.892 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.892 DEBUG:teuthology.orchestra.run.smithi029:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-07T09:11:57.920 INFO:teuthology.orchestra.run.smithi029.stdout:postinstall scriptlet (using /bin/sh): 2023-10-07T09:11:57.920 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc7-gfef64f93da2b -a -r /boot/System.map-6.5.0-rc7-gfef64f93da2b ]; then 2023-10-07T09:11:57.920 INFO:teuthology.orchestra.run.smithi029.stdout:cp /boot/vmlinuz-6.5.0-rc7-gfef64f93da2b /boot/.vmlinuz-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:cp /boot/System.map-6.5.0-rc7-gfef64f93da2b /boot/.System.map-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:rm -f /boot/vmlinuz-6.5.0-rc7-gfef64f93da2b /boot/System.map-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/installkernel 6.5.0-rc7-gfef64f93da2b /boot/.vmlinuz-6.5.0-rc7-gfef64f93da2b-rpm /boot/.System.map-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:rm -f /boot/.vmlinuz-6.5.0-rc7-gfef64f93da2b-rpm /boot/.System.map-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-07T09:11:57.921 INFO:teuthology.orchestra.run.smithi029.stdout:new-kernel-pkg --remove 6.5.0-rc7-gfef64f93da2b --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc7-gfef64f93da2b.img 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-install remove 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/update-bootloader --remove 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:57.922 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-07T09:11:57.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-10-07T09:11:58.001 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-10-07T09:11:58.001 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.02 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 129.el8 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Group : System Environment/Base 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 9265586 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-10-07T09:11:58.002 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Relocations : (not relocatable) 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : CentOS BuildSystem 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-10-07T09:11:58.003 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:11:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-07T09:11:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-07T09:11:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-07T09:11:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-10-07T09:11:58.004 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:11:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-10-07T09:11:58.005 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-07T09:11:58.005 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:11:58.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-07T09:11:58.949 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-10-07T09:12:01.569 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-10-07T09:12:01.571 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-10-07T09:12:01.616 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.lvsQcwhXfv 2023-10-07T09:12:01.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lvsQcwhXfv 2023-10-07T09:12:01.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.lvsQcwhXfv 2023-10-07T09:12:01.827 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.lvsQcwhXfv is 6KB 2023-10-07T09:12:01.875 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.lvsQcwhXfv 2023-10-07T09:12:01.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-10-07T09:12:01.958 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-07T09:12:01.958 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-07T09:12:01.958 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-07T09:12:01.958 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-6.5.0-rc7-gfef64f93da2b.conf 2023-10-07T09:12:01.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:02.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-10-07T09:12:03.813 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-07T09:12:03.813 DEBUG:teuthology.orchestra.run.smithi144:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-07T09:12:03.869 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/System.map-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/config-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/vmlinuz-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-07T09:12:03.870 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-07T09:12:03.871 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel/cpu 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel/cpu/mce 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-07T09:12:03.872 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/arch/x86/kvm/kvm.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/ansi_cprng.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/anubis.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/arc4.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_pq.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_tx.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/async_xor.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/async_tx/raid6test.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/authenc.ko 2023-10-07T09:12:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/authencesn.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/blake2b_generic.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/blowfish_common.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/blowfish_generic.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/camellia_generic.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cast5_generic.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cast6_generic.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cast_common.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/ccm.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cmac.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/crc32_generic.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/crypto_engine.ko 2023-10-07T09:12:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/crypto_user.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/cts.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/des_generic.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/dh_generic.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/echainiv.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/essiv.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/fcrypt.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/khazad.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/lz4.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/lz4hc.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/md4.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/michael_mic.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/pcbc.ko 2023-10-07T09:12:03.875 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/pcrypt.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/rmd160.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/seed.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/serpent_generic.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/tcrypt.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/tea.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/twofish_common.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/twofish_generic.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/vmac.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/wp512.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/xcbc.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/xor.ko 2023-10-07T09:12:03.876 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/crypto/xxhash_generic.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/acpi_pad.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/custom_method.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/ec_sys.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/platform_profile.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/sbs.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/sbshc.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/acpi/video.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/acard-ahci.ko 2023-10-07T09:12:03.877 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/ahci_platform.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/ata_generic.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/libahci_platform.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_acpi.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ali.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_amd.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_artop.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_atiixp.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_atp867x.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_cmd640.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_cypress.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_efar.ko 2023-10-07T09:12:03.878 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt366.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_it8213.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_it821x.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_jmicron.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_marvell.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_mpiix.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_netcell.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ninja32.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ns87410.ko 2023-10-07T09:12:03.879 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_ns87415.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_opti.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_optidma.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_piccolo.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_rdc.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sch.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_serverworks.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sil680.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sis.ko 2023-10-07T09:12:03.880 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_triflex.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pata_via.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/pdc_adma.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_inic162x.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_mv.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_nv.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_promise.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_qstor.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sil.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sil24.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sis.ko 2023-10-07T09:12:03.881 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_svw.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_sx4.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_uli.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_via.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ata/sata_vsc.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/auxdisplay 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base/regmap 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/bcma 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/bcma/bcma.ko 2023-10-07T09:12:03.882 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/aoe 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/aoe/aoe.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/brd.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/drbd 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/drbd/drbd.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/floppy.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/loop.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/mtip32xx 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/nbd.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/null_blk 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-07T09:12:03.883 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/pktcdvd.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/rbd.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/virtio_blk.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/xen-blkback 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/xen-blkfront.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/zram 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/block/zram/zram.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hangcheck-timer.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-07T09:12:03.884 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/lp.ko 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/mwave 2023-10-07T09:12:03.885 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/mwave/mwave.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/ppdev.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tlclk.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/uv_mmtimer.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/char/virtio_console.ko 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq 2023-10-07T09:12:03.886 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/ccp 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-07T09:12:03.887 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/padlock-aes.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/padlock-sha.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/virtio 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dca 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dca/dca.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/devfreq 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma 2023-10-07T09:12:03.888 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/dw 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/ioat 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/amd64_edac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/e752x_edac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i3000_edac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i3200_edac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i5100_edac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i5400_edac.ko 2023-10-07T09:12:03.889 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i7300_edac.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i7core_edac.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/i82975x_edac.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/ie31200_edac.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/sb_edac.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/edac/x38_edac.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/firmware 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/firmware/edd.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/gpio 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/gpio/gpio-ich.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid 2023-10-07T09:12:03.890 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-appleir.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-aureal.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-axff.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-dr.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-elecom.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-elo.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-emsff.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-gaff.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-gt683r.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-gyration.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-07T09:12:03.891 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-holtekff.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-hyperv.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-icade.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-keytouch.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-kye.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-lcpower.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-led.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-lenovo.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-multitouch.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-ortek.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-penmount.ko 2023-10-07T09:12:03.892 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-petalynx.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-picolcd.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-pl.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-plantronics.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-primax.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-rmi.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-07T09:12:03.893 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-roccat.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-saitek.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-samsung.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sjoy.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sony.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-speedlink.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-steelseries.ko 2023-10-07T09:12:03.894 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-sunplus.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-tivo.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-tmff.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-topseed.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-twinhan.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-uclogic.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-waltop.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-wiimote.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-xinmo.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-zpff.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/hid-zydacron.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/uhid.ko 2023-10-07T09:12:03.895 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hid/wacom.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv/hv_balloon.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv/hv_utils.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hv/hv_vmbus.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/abituguru.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/abituguru3.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ad7414.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ad7418.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adc128d818.ko 2023-10-07T09:12:03.896 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1025.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1026.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1029.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm1031.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adm9240.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ads7828.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7410.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7411.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7462.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7470.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7475.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/adt7x10.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/amc6821.ko 2023-10-07T09:12:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/applesmc.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/asb100.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/asc7621.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/atxp1.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/coretemp.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/dme1737.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ds1621.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ds620.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/emc1403.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/emc6w201.ko 2023-10-07T09:12:03.898 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/f71805f.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/f71882fg.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/f75375s.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/fschmd.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/g760a.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/g762.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/gl518sm.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/gl520sm.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ibmaem.ko 2023-10-07T09:12:03.899 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ibmpex.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ina209.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ina2xx.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/it87.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/k10temp.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/k8temp.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm63.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm73.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm75.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm77.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm78.ko 2023-10-07T09:12:03.900 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm80.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm83.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm85.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm87.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm90.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm92.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm93.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm95234.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm95241.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/lm95245.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc2945.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4151.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4215.ko 2023-10-07T09:12:03.901 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4222.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4245.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4260.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/ltc4261.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max16065.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max1619.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max1668.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max197.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max6639.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max6650.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/max6697.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/mcp3021.ko 2023-10-07T09:12:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct6683.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct6775.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/nct7802.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pc87360.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pc87427.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pcf8591.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-07T09:12:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/powr1220.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sch5627.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sch5636.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sht15.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sht21.ko 2023-10-07T09:12:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/shtc1.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/sis5595.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/thmc50.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp102.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp103.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp401.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/tmp421.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/via686a.ko 2023-10-07T09:12:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/vt1211.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/vt8231.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83627hf.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83781d.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83791d.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83792d.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83793.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83795.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/algos 2023-10-07T09:12:03.906 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-07T09:12:03.907 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-07T09:12:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-dev.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-mux.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/i2c/i2c-stub.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-07T09:12:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/cxgb4 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx4 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx5 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mthca 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-07T09:12:03.910 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/ipoib 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/iser 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/isert 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srp 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srpt 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input 2023-10-07T09:12:03.911 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/gameport.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/lightning.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/gameport/ns558.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/joydev.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/keyboard 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/appletouch.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-07T09:12:03.912 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/sermouse.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/rmi4 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/serio/serio_raw.ko 2023-10-07T09:12:03.913 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/input/sparse-keymap.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/iommu 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/iommu/amd 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-blinkm.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-lm3530.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-lp3944.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/leds-ss4200.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-07T09:12:03.914 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/bcache 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/bcache/bcache.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-bio-prison.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-cache-smq.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-cache.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-crypt.ko 2023-10-07T09:12:03.915 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-delay.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-flakey.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-log-userspace.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-multipath.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-queue-length.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-raid.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-round-robin.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-service-time.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-switch.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-thin-pool.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/dm-verity.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/faulty.ko 2023-10-07T09:12:03.916 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/linear.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/multipath.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/persistent-data 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid0.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid1.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid10.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/md/raid456.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptbase.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptctl.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptfc.ko 2023-10-07T09:12:03.917 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptlan.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptsas.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/message/fusion/mptspi.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/lpc_ich.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/lpc_sch.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/mfd-core.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/sm501.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/viperboard.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/vx855.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/mfd/wl1273-core.ko 2023-10-07T09:12:03.918 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/altera-stapl 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/apds9802als.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/apds990x.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/bh1770glc.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/cb710 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/cb710/cb710.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/at24.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-07T09:12:03.919 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/enclosure.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/hpilo.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/isl29003.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/isl29020.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/lis3lv02d 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei/mei-me.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/mei/mei.ko 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-gru 2023-10-07T09:12:03.920 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/tifm_core.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/tsl2550.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/vmw_balloon.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/vmw_vmci 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/bonding 2023-10-07T09:12:03.921 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/bonding/bonding.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/b53 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/dummy.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/eql.ko 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com 2023-10-07T09:12:03.922 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/adaptec 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/agere 2023-10-07T09:12:03.923 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/alteon 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/altera 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/alx 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-07T09:12:03.924 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-07T09:12:03.925 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/brocade 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/brocade/bna 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-07T09:12:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/cisco 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/cisco/enic 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-07T09:12:03.927 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dlink 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/dnet.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/emulex 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/emulex/benet 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-07T09:12:03.928 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/i40e 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/iavf 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igb 2023-10-07T09:12:03.929 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/jme.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-07T09:12:03.930 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/micrel 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/myricom 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-07T09:12:03.931 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/natsemi 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/neterion 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/nvidia 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/packetengines 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-07T09:12:03.932 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rdc 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rocker 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-07T09:12:03.933 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sfc 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/silan 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sis 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro 2023-10-07T09:12:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/tehuti 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/ti 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-07T09:12:03.935 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/via 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/wiznet 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/xircom 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/geneve.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/hyperv 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ieee802154 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-07T09:12:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ifb.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ipvlan 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/macvlan.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/macvtap.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/mii.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/net_failover.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/netconsole.ko 2023-10-07T09:12:03.937 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/nlmon.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/pcs 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/amd.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/broadcom.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/cicada.ko 2023-10-07T09:12:03.938 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/davicom.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/dp83640.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/et1011c.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/icplus.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/lxt.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/marvell.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/micrel.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/national.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/phylink.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/qsemi.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/realtek.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/smsc.ko 2023-10-07T09:12:03.939 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/phy/vitesse.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/pppoe.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/pppox.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/ppp/pptp.ko 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/slip 2023-10-07T09:12:03.940 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/slip/slhc.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/slip/slip.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/sungem_phy.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/tap.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_random.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/tun.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/veth.ko 2023-10-07T09:12:03.941 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/virtio_net.ko 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vmxnet3 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vxlan 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/xen-netback 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/net/xen-netfront.ko 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ntb 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ntb/ntb.ko 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme/host 2023-10-07T09:12:03.942 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/nvme/host/nvme.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport_cs.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport_pc.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/parport/parport_serial.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/hotplug 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/pcie 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pci/xen-pcifront.ko 2023-10-07T09:12:03.943 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia/i82092.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia/pd6729.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pinctrl 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pinctrl/intel 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-07T09:12:03.944 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/hdaps.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/intel 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-07T09:12:03.945 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/platform/x86/wmi.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/powercap 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-07T09:12:03.946 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/pps/pps_core.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ptp 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ptp/ptp.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-bq4802.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-07T09:12:03.947 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-07T09:12:03.948 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-07T09:12:03.949 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/3w-sas.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/BusLogic.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/a100u2w.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aacraid 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-07T09:12:03.950 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/advansys.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aic7xxx 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/am53c974.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/arcmsr 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/atp870u.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/be2iscsi 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bfa 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-07T09:12:03.951 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2fc 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2i 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ch.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/csiostor 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-07T09:12:03.952 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/dc395x.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/dmx3191d.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/esas2r 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/esp_scsi.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fcoe 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-07T09:12:03.953 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fnic 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/hpsa.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/hptiop.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/initio.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ipr.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ips.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/isci 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/isci/isci.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-07T09:12:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libfc 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libiscsi.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libsas 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/lpfc 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-07T09:12:03.955 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mpt3sas 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mvsas 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/mvumi.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/pm8001 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/pmcraid.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla1280.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla2xxx 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-07T09:12:03.956 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla4xxx 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/raid_class.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_debug.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/ses.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/st.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/stex.ko 2023-10-07T09:12:03.957 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/sym53c8xx_2 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/scsi/wd719x.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ssb 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ssb/ssb.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/iscsi 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/loopback 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-07T09:12:03.958 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_file.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_iblock.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_mod.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_pscsi.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/target_core_user.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/tcm_fc 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-07T09:12:03.959 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-07T09:12:03.960 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/ipwireless 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/n_gsm.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/n_hdlc.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/nozomi.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/8250 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/jsm 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-07T09:12:03.961 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/tty/synclink_gt.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/core 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/host 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_aec.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_cif.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/uio/uio_sercos3.ko 2023-10-07T09:12:03.962 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/usblp.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/class/usbtmc.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/host 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/image 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/image/mdc800.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/image/microtek.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/adutux.ko 2023-10-07T09:12:03.963 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/emi26.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/emi62.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/ezusb.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/idmouse.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/ldusb.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/sisusbvga 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-07T09:12:03.964 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/usb3503.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/usblcd.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/uss720.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/misc/yurex.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/phy 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/aircable.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ark3116.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ch341.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/cp210x.ko 2023-10-07T09:12:03.965 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/empeg.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/io_ti.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ipaq.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ipw.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-07T09:12:03.966 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/keyspan.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/mos7720.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/mos7840.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/navman.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/omninet.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/opticon.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/option.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/oti6858.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/pl2303.ko 2023-10-07T09:12:03.967 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/qcaux.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/qcserial.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/quatech2.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/sierra.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ssu100.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-07T09:12:03.968 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/visor.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/uas.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-07T09:12:03.969 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/pci 2023-10-07T09:12:03.970 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/vfio.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost_net.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight/lcd.ko 2023-10-07T09:12:03.971 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_pci.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/virtio/virtio_ring.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/masters 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/masters/ds2482.ko 2023-10-07T09:12:03.972 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/masters/ds2490.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-07T09:12:03.973 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/w1/wire.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/hpwdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/i6300esb.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/ibmasr.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-07T09:12:03.974 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/machzwd.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/nv_tco.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/softdog.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/via_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-07T09:12:03.975 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-evtchn.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-gntdev.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-pciback 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xen-privcmd.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xenfs 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-07T09:12:03.976 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/9p 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/9p/9p.ko 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/affs 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/affs/affs.ko 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/befs 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/befs/befs.ko 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/binfmt_misc.ko 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/btrfs 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/btrfs/btrfs.ko 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cachefiles 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cachefiles/cachefiles.ko 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ceph 2023-10-07T09:12:03.977 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ceph/ceph.ko 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/coda 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/coda/coda.ko 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cramfs 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/cramfs/cramfs.ko 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/dlm 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/dlm/dlm.ko 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ecryptfs 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/f2fs 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/f2fs/f2fs.ko 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat 2023-10-07T09:12:03.978 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat/fat.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat/msdos.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fat/vfat.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fscache 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fscache/fscache.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fuse 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fuse/cuse.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/fuse/fuse.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/gfs2 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/gfs2/gfs2.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfs 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfs/hfs.ko 2023-10-07T09:12:03.979 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfsplus 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/hfsplus/hfsplus.ko 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/isofs 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/isofs/isofs.ko 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/jfs 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/jfs/jfs.ko 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/lockd 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/lockd/lockd.ko 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/minix 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/minix/minix.ko 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/netfs 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/netfs/netfs.ko 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs 2023-10-07T09:12:03.980 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/blocklayout 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/filelayout 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/flexfilelayout 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/nfs.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/nfsv3.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs/nfsv4.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs_common 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs_common/grace.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-07T09:12:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfsd 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nfsd/nfsd.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nilfs2 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nilfs2/nilfs2.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-celtic.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-centeuro.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-croatian.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-cyrillic.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-gaelic.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-greek.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-iceland.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-inuit.ko 2023-10-07T09:12:03.982 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-roman.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-romanian.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/mac-turkish.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp1250.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp1251.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp1255.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp737.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp775.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp850.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp852.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp855.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp857.ko 2023-10-07T09:12:03.983 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp860.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp861.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp862.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp863.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp864.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp865.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp866.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp869.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp874.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp932.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp936.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp949.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_cp950.ko 2023-10-07T09:12:03.984 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_euc-jp.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-07T09:12:03.985 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_koi8-r.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_koi8-u.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/nls/nls_utf8.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/cluster 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlm 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlmfs 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-07T09:12:03.986 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/overlayfs 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/overlayfs/overlay.ko 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/pstore 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/pstore/ramoops.ko 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/reiserfs 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/reiserfs/reiserfs.ko 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/romfs 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/romfs/romfs.ko 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/client 2023-10-07T09:12:03.987 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/client/cifs.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/common 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/smb/common/cifs_md4.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/squashfs 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/squashfs/squashfs.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/sysv 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/sysv/sysv.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/udf 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/udf/udf.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ufs 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/ufs/ufs.ko 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/xfs 2023-10-07T09:12:03.988 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/fs/xfs/xfs.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/rcu 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/rcu/rcutorture.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/torture.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/trace 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/asn1_encoder.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crc-itu-t.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crc8.ko 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crypto 2023-10-07T09:12:03.989 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crypto/libarc4.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/crypto/libdes.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/libcrc32c.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lru_cache.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lz4 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lz4/lz4_compress.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/math 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/math/cordic.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/raid6 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/raid6/raid6_pq.ko 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/reed_solomon 2023-10-07T09:12:03.990 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/ts_bm.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/ts_fsm.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/ts_kmp.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/zstd 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/lib/zstd/zstd_compress.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/mm 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/mm/hwpoison-inject.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/6lowpan.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_dest.ko 2023-10-07T09:12:03.991 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_hop.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_routing.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/6lowpan/nhc_udp.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/garp.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/mrp.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/p8022.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/psnap.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/802/stp.ko 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/8021q 2023-10-07T09:12:03.992 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/8021q/8021q.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet_fd.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet_rdma.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/9p/9pnet_virtio.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/batman-adv 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/batman-adv/batman-adv.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/br_netfilter.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/bridge.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-07T09:12:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-07T09:12:03.994 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ceph 2023-10-07T09:12:03.995 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ceph/libceph.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/core 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/core/failover.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/core/pktgen.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp_diag.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp_ipv4.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dccp/dccp_ipv6.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dns_resolver 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/dsa_core.ko 2023-10-07T09:12:03.996 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_brcm.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_dsa.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_none.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/dsa/tag_trailer.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/6lowpan 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/ieee802154.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ah4.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/esp4.ko 2023-10-07T09:12:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/fou.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/gre.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/inet_diag.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ip_gre.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ip_tunnel.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ip_vti.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ipcomp.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/ipip.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-07T09:12:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-07T09:12:03.999 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_bic.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_diag.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_htcp.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_hybla.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_illinois.ko 2023-10-07T09:12:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_lp.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_scalable.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_vegas.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_veno.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_westwood.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tcp_yeah.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/tunnel4.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/udp_diag.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/udp_tunnel.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ah6.ko 2023-10-07T09:12:04.001 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/esp6.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/fou6.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ip6_vti.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/ipcomp6.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-07T09:12:04.002 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-07T09:12:04.003 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/sit.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/tunnel6.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/key 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/key/af_key.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_core.ko 2023-10-07T09:12:04.004 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_eth.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_ip.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/llc 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/llc/llc.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mac802154 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mac802154/mac802154.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mpls 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/mpls/mpls_gso.ko 2023-10-07T09:12:04.005 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-07T09:12:04.006 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-07T09:12:04.007 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conncount.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-07T09:12:04.008 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_flow_table.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-07T09:12:04.009 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nf_tables.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-07T09:12:04.010 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_compat.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_ct.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_fib.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_hash.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_limit.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_log.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_masq.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_nat.ko 2023-10-07T09:12:04.011 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_numgen.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_queue.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_quota.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_redir.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_reject.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_socket.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/nft_tunnel.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-07T09:12:04.012 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_CT.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_DSCP.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_HL.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_HMARK.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_LED.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_LOG.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-07T09:12:04.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TEE.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_TRACE.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_addrtype.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_bpf.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_cgroup.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_cluster.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_comment.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connbytes.ko 2023-10-07T09:12:04.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connlabel.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connlimit.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_connmark.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_conntrack.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_cpu.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_dccp.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_devgroup.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_dscp.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_ecn.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_esp.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_helper.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_hl.ko 2023-10-07T09:12:04.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_iprange.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_ipvs.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_l2tp.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_length.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_limit.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_mac.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_mark.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_multiport.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_nat.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_nfacct.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_osf.ko 2023-10-07T09:12:04.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_owner.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_physdev.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_pkttype.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_policy.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_quota.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_rateest.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_realm.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_recent.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_sctp.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_set.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_socket.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_state.ko 2023-10-07T09:12:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_statistic.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_string.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_time.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netfilter/xt_u32.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netlink 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/netlink/netlink_diag.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/nsh 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/nsh/nsh.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/openvswitch.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/vport-geneve.ko 2023-10-07T09:12:04.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/vport-gre.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/packet 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/packet/af_packet_diag.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds/rds.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds/rds_rdma.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/rds/rds_tcp.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_csum.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_gact.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_ipt.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_mirred.ko 2023-10-07T09:12:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_nat.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_pedit.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_police.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_simple.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_skbedit.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/act_vlan.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_basic.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_bpf.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_flow.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_fw.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_route.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/cls_u32.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_cmp.ko 2023-10-07T09:12:04.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_ipset.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_meta.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_nbyte.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_text.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/em_u32.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_choke.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_codel.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_drr.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_fq.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_gred.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_hfsc.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_hhf.ko 2023-10-07T09:12:04.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_htb.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_ingress.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_mqprio.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_multiq.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_netem.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_pie.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_plug.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_prio.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_qfq.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_red.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_sfb.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_sfq.ko 2023-10-07T09:12:04.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_tbf.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sched/sch_teql.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sctp 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sctp/sctp.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sctp/sctp_diag.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/auth_gss 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/sunrpc.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/xprtrdma 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-07T09:12:04.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/tipc 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/tipc/diag.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/tipc/tipc.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/unix 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/unix/unix_diag.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vsock.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/xfrm 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-07T09:12:04.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/encrypted-keys 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/trusted-keys 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/virt 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/virt/lib 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/kernel/virt/lib/irqbypass.ko 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.alias 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.alias.bin 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.builtin 2023-10-07T09:12:04.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.builtin.bin 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.builtin.modinfo 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.dep 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.dep.bin 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.devname 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.order 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.softdep 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.symbols 2023-10-07T09:12:04.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.5.0-rc7-gfef64f93da2b/modules.symbols.bin 2023-10-07T09:12:04.027 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:04.027 DEBUG:teuthology.orchestra.run.smithi144:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-07T09:12:04.052 INFO:teuthology.orchestra.run.smithi144.stdout:postinstall scriptlet (using /bin/sh): 2023-10-07T09:12:04.052 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc7-gfef64f93da2b -a -r /boot/System.map-6.5.0-rc7-gfef64f93da2b ]; then 2023-10-07T09:12:04.052 INFO:teuthology.orchestra.run.smithi144.stdout:cp /boot/vmlinuz-6.5.0-rc7-gfef64f93da2b /boot/.vmlinuz-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:12:04.052 INFO:teuthology.orchestra.run.smithi144.stdout:cp /boot/System.map-6.5.0-rc7-gfef64f93da2b /boot/.System.map-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:12:04.052 INFO:teuthology.orchestra.run.smithi144.stdout:rm -f /boot/vmlinuz-6.5.0-rc7-gfef64f93da2b /boot/System.map-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/installkernel 6.5.0-rc7-gfef64f93da2b /boot/.vmlinuz-6.5.0-rc7-gfef64f93da2b-rpm /boot/.System.map-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:rm -f /boot/.vmlinuz-6.5.0-rc7-gfef64f93da2b-rpm /boot/.System.map-6.5.0-rc7-gfef64f93da2b-rpm 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:new-kernel-pkg --remove 6.5.0-rc7-gfef64f93da2b --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc7-gfef64f93da2b.img 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-install remove 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-07T09:12:04.053 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/update-bootloader --remove 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:04.054 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2023-10-07T09:12:04.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2023-10-07T09:12:04.128 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.02 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 129.el8 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Group : System Environment/Base 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 9265586 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-07T09:12:04.129 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:Relocations : (not relocatable) 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : CentOS BuildSystem 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2023-10-07T09:12:04.130 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:12:04.131 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2023-10-07T09:12:04.131 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-07T09:12:04.131 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:04.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-07T09:12:08.960 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2023-10-07T09:12:11.218 INFO:teuthology.orchestra.run.smithi144.stderr:done 2023-10-07T09:12:11.220 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2023-10-07T09:12:11.242 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.zvzlQUEtLV 2023-10-07T09:12:11.243 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zvzlQUEtLV 2023-10-07T09:12:11.324 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.zvzlQUEtLV 2023-10-07T09:12:11.467 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.zvzlQUEtLV is 6KB 2023-10-07T09:12:11.514 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.zvzlQUEtLV 2023-10-07T09:12:11.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2023-10-07T09:12:11.597 INFO:teuthology.orchestra.run.smithi144.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-07T09:12:11.597 INFO:teuthology.orchestra.run.smithi144.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-07T09:12:11.597 INFO:teuthology.orchestra.run.smithi144.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-07T09:12:11.597 INFO:teuthology.orchestra.run.smithi144.stdout:a1b29d6c289747bdb22930598d39874a-6.5.0-rc7-gfef64f93da2b.conf 2023-10-07T09:12:11.598 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.5.0-rc7-gfef64f93da2b 2023-10-07T09:12:11.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2023-10-07T09:12:32.214 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-07T09:12:32.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-07T09:12:32.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:12:41.802 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-07T09:12:41.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-07T09:12:41.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:12:50.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2023-10-07T09:12:59.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-07T09:12:59.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:13:00.282 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2023-10-07T09:13:09.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-07T09:13:09.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:13:12.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2023-10-07T09:13:24.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-07T09:13:24.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:13:56.218 DEBUG:teuthology.orchestra.run.smithi144:> true 2023-10-07T09:13:56.666 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-07T09:13:56.666 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.5.0-rc7-gfef64f93da2b"... 2023-10-07T09:13:56.667 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2023-10-07T09:13:56.711 INFO:teuthology.orchestra.run.smithi144.stdout:6.5.0-rc7-gfef64f93da2b 2023-10-07T09:13:56.712 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc7-gfef64f93da2b vs 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:13:56.712 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-07T09:13:56.712 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-07T09:13:57.713 DEBUG:teuthology.parallel:result is None 2023-10-07T09:13:59.565 DEBUG:teuthology.orchestra.remote:timed out 2023-10-07T09:14:11.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-07T09:14:11.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:14:11.950 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-10-07T09:14:12.455 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-07T09:14:12.455 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.5.0-rc7-gfef64f93da2b"... 2023-10-07T09:14:12.455 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-10-07T09:14:12.499 INFO:teuthology.orchestra.run.smithi029.stdout:6.5.0-rc7-gfef64f93da2b 2023-10-07T09:14:12.500 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc7-gfef64f93da2b vs 6.5.0-rc7-gfef64f93da2b 2023-10-07T09:14:12.500 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-07T09:14:12.500 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-07T09:14:13.500 DEBUG:teuthology.parallel:result is None 2023-10-07T09:14:13.501 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-07T09:14:13.513 INFO:teuthology.task.internal:Creating test directory... 2023-10-07T09:14:13.513 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-07T09:14:13.517 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-07T09:14:13.568 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-07T09:14:13.579 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-07T09:14:13.593 INFO:teuthology.task.internal:Creating archive directory... 2023-10-07T09:14:13.594 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-07T09:14:13.598 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-07T09:14:13.662 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-07T09:14:13.674 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-07T09:14:13.675 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-07T09:14:13.679 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-07T09:14:13.801 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-07T09:14:13.827 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-07T09:14:13.837 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-07T09:14:13.879 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-07T09:14:13.882 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-07T09:14:13.891 INFO:teuthology.task.internal:Configuring sudo... 2023-10-07T09:14:13.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-07T09:14:13.895 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-07T09:14:13.965 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-07T09:14:13.978 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-07T09:14:13.978 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-07T09:14:13.982 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-07T09:14:14.027 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-07T09:14:14.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-07T09:14:14.206 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-07T09:14:14.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-07T09:14:14.350 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:14:14.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-07T09:14:14.425 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-07T09:14:14.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-07T09:14:14.572 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-07T09:14:14.644 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-07T09:14:14.712 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:14:14.713 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-07T09:14:14.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-10-07T09:14:14.786 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2023-10-07T09:14:14.903 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-07T09:14:14.944 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-07T09:14:15.172 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-07T09:14:15.180 INFO:teuthology.task.internal:Starting timer... 2023-10-07T09:14:15.180 INFO:teuthology.run_tasks:Running task pcp... 2023-10-07T09:14:15.197 INFO:teuthology.run_tasks:Running task selinux... 2023-10-07T09:14:15.212 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-07T09:14:15.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-10-07T09:14:15.353 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-10-07T09:14:15.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2023-10-07T09:14:15.495 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2023-10-07T09:14:15.499 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-07T09:14:15.499 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-10-07T09:14:15.559 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-10-07T09:14:15.559 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2023-10-07T09:14:15.618 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2023-10-07T09:14:15.618 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2023-10-07T09:14:15.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-07T09:14:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:14:15.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-07T09:14:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:14:15.761 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-07T09:14:15.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-10-07T09:14:15.839 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2023-10-07T09:14:15.907 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-07T09:14:15.918 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-07T09:14:15.918 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-07T09:14:15.935 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-07T09:14:15.937 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-07T09:25:03.525 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2023-10-07T09:25:03.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-07T09:25:03.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:25:03.616 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-10-07T09:25:03.736 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-07T09:25:03.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-07T09:25:03.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-07T09:25:03.827 DEBUG:teuthology.orchestra.run.smithi144:> true 2023-10-07T09:25:03.944 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-07T09:25:03.944 INFO:teuthology.run_tasks:Running task clock... 2023-10-07T09:25:03.957 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-07T09:25:03.958 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-07T09:25:03.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-07T09:25:03.963 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-07T09:25:03.963 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-07T09:25:04.037 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-07T09:25:04.058 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-07T09:25:04.072 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-07T09:25:04.091 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-07T09:25:04.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-10-07T09:25:04.178 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2023-10-07T09:25:04.189 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-10-07T09:25:04.206 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2023-10-07T09:25:04.222 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-07T09:25:04.239 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-07T09:25:04.256 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-07T09:25:04.271 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-07T09:25:04.351 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2023-10-07T09:25:04.358 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-07T09:25:04.359 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-10-07T09:25:04.359 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.359 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.359 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.359 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.375 INFO:teuthology.orchestra.run.smithi144.stderr:bash: ntpq: command not found 2023-10-07T09:25:04.381 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-07T09:25:04.382 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2023-10-07T09:25:04.382 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.382 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.382 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.382 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T09:25:04.383 INFO:teuthology.run_tasks:Running task install... 2023-10-07T09:25:04.391 DEBUG:teuthology.task.install:project ceph 2023-10-07T09:25:04.391 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647'}} 2023-10-07T09:25:04.391 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647'} 2023-10-07T09:25:04.391 INFO:teuthology.task.install:Using flavor: default 2023-10-07T09:25:04.397 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-07T09:25:04.398 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-07T09:25:04.398 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647', 'tag': None, 'wait_for_package': False} 2023-10-07T09:25:04.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-07T09:25:04.400 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647', 'tag': None, 'wait_for_package': False} 2023-10-07T09:25:04.401 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-07T09:25:04.624 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/ 2023-10-07T09:25:04.624 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1320.g08cac018 2023-10-07T09:25:04.633 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/ 2023-10-07T09:25:04.634 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1320.g08cac018 2023-10-07T09:25:04.746 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-07T09:25:04.747 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:25:04.747 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-07T09:25:04.796 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-06-0949-quincy/08cac0187919fc3a247efdf445f652e2fb30a647/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-07T09:25:04.796 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:25:04.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-07T09:25:04.819 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-07T09:25:04.819 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/08cac0187919fc3a247efdf445f652e2fb30a647/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-07T09:25:04.869 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-07T09:25:04.869 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/08cac0187919fc3a247efdf445f652e2fb30a647/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-07T09:25:04.930 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-07T09:25:04.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-07T09:25:05.026 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-07T09:25:05.079 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-07T09:25:05.093 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2023-10-07T09:25:05.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2023-10-07T09:25:05.149 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-10-07T09:25:05.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-10-07T09:25:05.442 INFO:teuthology.orchestra.run.smithi144.stdout:76 files removed 2023-10-07T09:25:05.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-07T09:25:05.577 INFO:teuthology.orchestra.run.smithi029.stdout:76 files removed 2023-10-07T09:25:05.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-07T09:25:06.381 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 220 kB/s | 78 kB 00:00 2023-10-07T09:25:06.470 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 278 kB/s | 78 kB 00:00 2023-10-07T09:25:06.690 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 34 kB/s | 9.2 kB 00:00 2023-10-07T09:25:06.879 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 25 kB/s | 9.2 kB 00:00 2023-10-07T09:25:07.049 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 4.4 kB/s | 1.5 kB 00:00 2023-10-07T09:25:07.231 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2023-10-07T09:25:09.441 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - AppStream 14 MB/s | 33 MB 00:02 2023-10-07T09:25:10.009 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 12 MB/s | 33 MB 00:02 2023-10-07T09:25:17.308 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - BaseOS 34 MB/s | 48 MB 00:01 2023-10-07T09:25:18.700 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 48 MB 00:01 2023-10-07T09:25:44.006 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 1.0 kB/s | 18 kB 00:17 2023-10-07T09:25:44.239 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - Extras 926 B/s | 18 kB 00:19 2023-10-07T09:25:44.874 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-10-07T09:25:45.231 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - Extras common packages 8.6 kB/s | 6.9 kB 00:00 2023-10-07T09:25:45.676 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.5 MB 00:00 2023-10-07T09:25:45.982 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.5 MB 00:00 2023-10-07T09:26:02.478 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 231 B/s | 3.5 kB 00:15 2023-10-07T09:26:03.498 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-10-07T09:26:08.174 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 401 kB/s | 24 kB 00:00 2023-10-07T09:26:10.756 INFO:teuthology.orchestra.run.smithi144.stdout:Copr repo for python3-asyncssh owned by ceph 152 B/s | 3.5 kB 00:23 2023-10-07T09:26:11.873 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-07T09:26:12.862 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-07T09:26:12.862 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-07T09:26:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2023-10-07T09:26:12.877 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2023-10-07T09:26:12.877 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2023-10-07T09:26:12.879 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2023-10-07T09:26:12.880 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-07T09:26:12.882 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-07T09:26:12.885 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-07T09:26:12.886 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-07T09:26:13.124 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-10-07T09:26:13.138 INFO:teuthology.orchestra.run.smithi029.stdout:==================================================================================================== 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout:==================================================================================================== 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:17.2.6-1320.g08cac018.el8 ceph 6.4 k 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:17.2.6-1320.g08cac018.el8 ceph 5.2 M 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:17.2.6-1320.g08cac018.el8 ceph 876 k 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1320.g08cac018.el8 ceph 162 k 2023-10-07T09:26:13.139 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:17.2.6-1320.g08cac018.el8 ceph 1.6 M 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 121 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 1.7 M 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 7.4 M 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 47 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:17.2.6-1320.g08cac018.el8 ceph 17 M 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:17.2.6-1320.g08cac018.el8 ceph 52 M 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 267 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 80 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:17.2.6-1320.g08cac018.el8 ceph 255 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 18 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 baseos 62 k 2023-10-07T09:26:13.140 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:17.2.6-1320.g08cac018.el8 ceph 25 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 797 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:17.2.6-1320.g08cac018.el8 ceph 130 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.32.1-43.el8 baseos 100 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:1.1.1k-9.el8 baseos 3.2 M 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:17.2.6-1320.g08cac018.el8 ceph 190 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:17.2.6-1320.g08cac018.el8 ceph 371 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:17.2.6-1320.g08cac018.el8 ceph 358 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:17.2.6-1320.g08cac018.el8 ceph 107 k 2023-10-07T09:26:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:17.2.6-1320.g08cac018.el8 ceph 90 k 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.5 M 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:17.2.6-1320.g08cac018.el8 ceph 173 k 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-9.el8 baseos 143 k 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client x86_64 0.189-3.el8 baseos 76 k 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf x86_64 0.189-3.el8 baseos 232 k 2023-10-07T09:26:13.142 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs x86_64 0.189-3.el8 baseos 303 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid x86_64 2.32.1-43.el8 baseos 220 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk x86_64 2.32.1-43.el8 baseos 254 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: libmount x86_64 2.32.1-43.el8 baseos 236 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.5 M 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.6 M 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols x86_64 2.32.1-43.el8 baseos 179 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid x86_64 2.32.1-43.el8 baseos 99 k 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux x86_64 2.32.1-43.el8 baseos 2.5 M 2023-10-07T09:26:13.143 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user x86_64 2.32.1-43.el8 baseos 103 k 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump x86_64 3.1.8-6.el8 baseos 334 k 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:17.2.6-1320.g08cac018.el8 ceph 23 M 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 23 k 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:17.2.6-1320.g08cac018.el8 ceph 2.3 M 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 238 k 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.9 M 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:17.2.6-1320.g08cac018.el8 ceph 17 M 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 14 k 2023-10-07T09:26:13.144 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:17.2.6-1320.g08cac018.el8 ceph 24 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:17.2.6-1320.g08cac018.el8 ceph 174 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-07T09:26:13.145 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 507 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 5.6 M 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-07T09:26:13.146 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:17.2.6-1320.g08cac018.el8 ceph 45 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:17.2.6-1320.g08cac018.el8 ceph 115 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-07T09:26:13.147 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-07T09:26:13.148 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-07T09:26:13.149 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-07T09:26:13.150 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout:==================================================================================================== 2023-10-07T09:26:13.151 INFO:teuthology.orchestra.run.smithi029.stdout:Install 123 Packages 2023-10-07T09:26:13.152 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 16 Packages 2023-10-07T09:26:13.152 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:26:13.152 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 210 M 2023-10-07T09:26:13.152 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-10-07T09:26:13.802 INFO:teuthology.orchestra.run.smithi029.stdout:(1/139): ceph-17.2.6-1320.g08cac018.el8.x86_64. 31 kB/s | 6.4 kB 00:00 2023-10-07T09:26:14.228 INFO:teuthology.orchestra.run.smithi029.stdout:(2/139): ceph-base-17.2.6-1320.g08cac018.el8.x8 8.1 MB/s | 5.2 MB 00:00 2023-10-07T09:26:14.280 INFO:teuthology.orchestra.run.smithi029.stdout:(3/139): ceph-immutable-object-cache-17.2.6-132 3.1 MB/s | 162 kB 00:00 2023-10-07T09:26:14.572 INFO:teuthology.orchestra.run.smithi029.stdout:(4/139): ceph-fuse-17.2.6-1320.g08cac018.el8.x8 1.1 MB/s | 876 kB 00:00 2023-10-07T09:26:14.839 INFO:teuthology.orchestra.run.smithi029.stdout:(5/139): ceph-mds-17.2.6-1320.g08cac018.el8.x86 4.1 MB/s | 2.3 MB 00:00 2023-10-07T09:26:15.440 INFO:teuthology.orchestra.run.smithi029.stdout:(6/139): ceph-common-17.2.6-1320.g08cac018.el8. 13 MB/s | 23 MB 00:01 2023-10-07T09:26:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:(7/139): ceph-mgr-17.2.6-1320.g08cac018.el8.x86 1.7 MB/s | 1.6 MB 00:00 2023-10-07T09:26:15.616 INFO:teuthology.orchestra.run.smithi029.stdout:(8/139): ceph-mon-17.2.6-1320.g08cac018.el8.x86 5.1 MB/s | 3.9 MB 00:00 2023-10-07T09:26:15.675 INFO:teuthology.orchestra.run.smithi029.stdout:(9/139): ceph-selinux-17.2.6-1320.g08cac018.el8 418 kB/s | 24 kB 00:00 2023-10-07T09:26:16.340 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 378 kB/s | 24 kB 00:00 2023-10-07T09:26:16.609 INFO:teuthology.orchestra.run.smithi029.stdout:(10/139): ceph-osd-17.2.6-1320.g08cac018.el8.x8 14 MB/s | 17 MB 00:01 2023-10-07T09:26:16.668 INFO:teuthology.orchestra.run.smithi029.stdout:(11/139): cephfs-mirror-17.2.6-1320.g08cac018.e 4.3 MB/s | 255 kB 00:00 2023-10-07T09:26:16.718 INFO:teuthology.orchestra.run.smithi029.stdout:(12/139): libcephfs-devel-17.2.6-1320.g08cac018 499 kB/s | 25 kB 00:00 2023-10-07T09:26:16.836 INFO:teuthology.orchestra.run.smithi029.stdout:(13/139): libcephfs2-17.2.6-1320.g08cac018.el8. 6.7 MB/s | 797 kB 00:00 2023-10-07T09:26:16.903 INFO:teuthology.orchestra.run.smithi029.stdout:(14/139): libcephsqlite-17.2.6-1320.g08cac018.e 2.6 MB/s | 174 kB 00:00 2023-10-07T09:26:16.978 INFO:teuthology.orchestra.run.smithi029.stdout:(15/139): librados-devel-17.2.6-1320.g08cac018. 1.7 MB/s | 130 kB 00:00 2023-10-07T09:26:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:(16/139): libradosstriper1-17.2.6-1320.g08cac01 5.4 MB/s | 507 kB 00:00 2023-10-07T09:26:17.589 INFO:teuthology.orchestra.run.smithi029.stdout:(17/139): librgw2-17.2.6-1320.g08cac018.el8.x86 11 MB/s | 5.6 MB 00:00 2023-10-07T09:26:17.682 INFO:teuthology.orchestra.run.smithi029.stdout:(18/139): python3-ceph-argparse-17.2.6-1320.g08 486 kB/s | 45 kB 00:00 2023-10-07T09:26:17.757 INFO:teuthology.orchestra.run.smithi029.stdout:(19/139): python3-ceph-common-17.2.6-1320.g08ca 1.5 MB/s | 115 kB 00:00 2023-10-07T09:26:17.825 INFO:teuthology.orchestra.run.smithi029.stdout:(20/139): python3-cephfs-17.2.6-1320.g08cac018. 2.8 MB/s | 190 kB 00:00 2023-10-07T09:26:17.908 INFO:teuthology.orchestra.run.smithi029.stdout:(21/139): python3-rados-17.2.6-1320.g08cac018.e 4.4 MB/s | 371 kB 00:00 2023-10-07T09:26:17.992 INFO:teuthology.orchestra.run.smithi029.stdout:(22/139): python3-rbd-17.2.6-1320.g08cac018.el8 4.2 MB/s | 358 kB 00:00 2023-10-07T09:26:18.059 INFO:teuthology.orchestra.run.smithi029.stdout:(23/139): python3-rgw-17.2.6-1320.g08cac018.el8 1.6 MB/s | 107 kB 00:00 2023-10-07T09:26:18.143 INFO:teuthology.orchestra.run.smithi029.stdout:(24/139): rbd-fuse-17.2.6-1320.g08cac018.el8.x8 1.1 MB/s | 90 kB 00:00 2023-10-07T09:26:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:(25/139): ceph-radosgw-17.2.6-1320.g08cac018.el 5.3 MB/s | 17 MB 00:03 2023-10-07T09:26:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:(26/139): rbd-mirror-17.2.6-1320.g08cac018.el8. 5.2 MB/s | 3.5 MB 00:00 2023-10-07T09:26:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:(27/139): rbd-nbd-17.2.6-1320.g08cac018.el8.x86 989 kB/s | 173 kB 00:00 2023-10-07T09:26:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:(28/139): ceph-grafana-dashboards-17.2.6-1320.g 391 kB/s | 23 kB 00:00 2023-10-07T09:26:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:(29/139): ceph-mgr-cephadm-17.2.6-1320.g08cac01 1.8 MB/s | 121 kB 00:00 2023-10-07T09:26:19.047 INFO:teuthology.orchestra.run.smithi029.stdout:(30/139): ceph-mgr-dashboard-17.2.6-1320.g08cac 9.6 MB/s | 1.7 MB 00:00 2023-10-07T09:26:19.124 INFO:teuthology.orchestra.run.smithi029.stdout:(31/139): ceph-mgr-modules-core-17.2.6-1320.g08 3.0 MB/s | 238 kB 00:00 2023-10-07T09:26:19.175 INFO:teuthology.orchestra.run.smithi029.stdout:(32/139): ceph-mgr-rook-17.2.6-1320.g08cac018.e 947 kB/s | 47 kB 00:00 2023-10-07T09:26:19.217 INFO:teuthology.orchestra.run.smithi029.stdout:(33/139): ceph-prometheus-alerts-17.2.6-1320.g0 345 kB/s | 14 kB 00:00 2023-10-07T09:26:19.292 INFO:teuthology.orchestra.run.smithi029.stdout:(34/139): ceph-volume-17.2.6-1320.g08cac018.el8 3.5 MB/s | 267 kB 00:00 2023-10-07T09:26:19.360 INFO:teuthology.orchestra.run.smithi029.stdout:(35/139): cephadm-17.2.6-1320.g08cac018.el8.noa 1.2 MB/s | 80 kB 00:00 2023-10-07T09:26:19.411 INFO:teuthology.orchestra.run.smithi029.stdout:(36/139): cephfs-top-17.2.6-1320.g08cac018.el8. 360 kB/s | 18 kB 00:00 2023-10-07T09:26:19.745 INFO:teuthology.orchestra.run.smithi029.stdout:(37/139): ceph-mgr-diskprediction-local-17.2.6- 8.9 MB/s | 7.4 MB 00:00 2023-10-07T09:26:20.703 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-07T09:26:20.704 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-07T09:26:20.717 INFO:teuthology.orchestra.run.smithi144.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2023-10-07T09:26:20.717 INFO:teuthology.orchestra.run.smithi144.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2023-10-07T09:26:20.718 INFO:teuthology.orchestra.run.smithi144.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2023-10-07T09:26:20.719 INFO:teuthology.orchestra.run.smithi144.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2023-10-07T09:26:20.721 INFO:teuthology.orchestra.run.smithi144.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-07T09:26:20.723 INFO:teuthology.orchestra.run.smithi144.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-07T09:26:20.726 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-07T09:26:20.727 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-07T09:26:20.879 INFO:teuthology.orchestra.run.smithi029.stdout:(38/139): ceph-test-17.2.6-1320.g08cac018.el8.x 9.9 MB/s | 52 MB 00:05 2023-10-07T09:26:20.949 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-10-07T09:26:20.963 INFO:teuthology.orchestra.run.smithi029.stdout:(39/139): automake-1.16.1-8.el8.noarch.rpm 477 kB/s | 740 kB 00:01 2023-10-07T09:26:20.963 INFO:teuthology.orchestra.run.smithi144.stdout:==================================================================================================== 2023-10-07T09:26:20.964 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2023-10-07T09:26:20.964 INFO:teuthology.orchestra.run.smithi144.stdout:==================================================================================================== 2023-10-07T09:26:20.964 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2023-10-07T09:26:20.964 INFO:teuthology.orchestra.run.smithi144.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2023-10-07T09:26:20.964 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:17.2.6-1320.g08cac018.el8 ceph 6.4 k 2023-10-07T09:26:20.964 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:17.2.6-1320.g08cac018.el8 ceph 5.2 M 2023-10-07T09:26:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:17.2.6-1320.g08cac018.el8 ceph 876 k 2023-10-07T09:26:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1320.g08cac018.el8 ceph 162 k 2023-10-07T09:26:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:17.2.6-1320.g08cac018.el8 ceph 1.6 M 2023-10-07T09:26:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 121 k 2023-10-07T09:26:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 1.7 M 2023-10-07T09:26:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 7.4 M 2023-10-07T09:26:20.966 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 47 k 2023-10-07T09:26:20.966 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:17.2.6-1320.g08cac018.el8 ceph 17 M 2023-10-07T09:26:20.966 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:17.2.6-1320.g08cac018.el8 ceph 52 M 2023-10-07T09:26:20.966 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 267 k 2023-10-07T09:26:20.966 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 80 k 2023-10-07T09:26:20.966 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror x86_64 2:17.2.6-1320.g08cac018.el8 ceph 255 k 2023-10-07T09:26:20.967 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 18 k 2023-10-07T09:26:20.967 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 baseos 62 k 2023-10-07T09:26:20.967 INFO:teuthology.orchestra.run.smithi144.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2023-10-07T09:26:20.967 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2023-10-07T09:26:20.967 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:17.2.6-1320.g08cac018.el8 ceph 25 k 2023-10-07T09:26:20.967 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 797 k 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:17.2.6-1320.g08cac018.el8 ceph 130 k 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel x86_64 2.32.1-43.el8 baseos 100 k 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel x86_64 1:1.1.1k-9.el8 baseos 3.2 M 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:17.2.6-1320.g08cac018.el8 ceph 190 k 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:17.2.6-1320.g08cac018.el8 ceph 371 k 2023-10-07T09:26:20.968 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:17.2.6-1320.g08cac018.el8 ceph 358 k 2023-10-07T09:26:20.969 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:17.2.6-1320.g08cac018.el8 ceph 107 k 2023-10-07T09:26:20.969 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:17.2.6-1320.g08cac018.el8 ceph 90 k 2023-10-07T09:26:20.969 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.5 M 2023-10-07T09:26:20.969 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:17.2.6-1320.g08cac018.el8 ceph 173 k 2023-10-07T09:26:20.969 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2023-10-07T09:26:20.969 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-9.el8 baseos 143 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-debuginfod-client x86_64 0.189-3.el8 baseos 76 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf x86_64 0.189-3.el8 baseos 232 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libs x86_64 0.189-3.el8 baseos 303 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: libblkid x86_64 2.32.1-43.el8 baseos 220 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: libfdisk x86_64 2.32.1-43.el8 baseos 254 k 2023-10-07T09:26:20.970 INFO:teuthology.orchestra.run.smithi144.stdout: libmount x86_64 2.32.1-43.el8 baseos 236 k 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.5 M 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.6 M 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: libsmartcols x86_64 2.32.1-43.el8 baseos 179 k 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid x86_64 2.32.1-43.el8 baseos 99 k 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: util-linux x86_64 2.32.1-43.el8 baseos 2.5 M 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: util-linux-user x86_64 2.32.1-43.el8 baseos 103 k 2023-10-07T09:26:20.971 INFO:teuthology.orchestra.run.smithi144.stdout: xfsdump x86_64 3.1.8-6.el8 baseos 334 k 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:17.2.6-1320.g08cac018.el8 ceph 23 M 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 23 k 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:17.2.6-1320.g08cac018.el8 ceph 2.3 M 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 238 k 2023-10-07T09:26:20.972 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:17.2.6-1320.g08cac018.el8 ceph 3.9 M 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:17.2.6-1320.g08cac018.el8 ceph 17 M 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1320.g08cac018.el8 ceph-noarch 14 k 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:17.2.6-1320.g08cac018.el8 ceph 24 k 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-07T09:26:20.973 INFO:teuthology.orchestra.run.smithi144.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:17.2.6-1320.g08cac018.el8 ceph 174 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-07T09:26:20.974 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 507 k 2023-10-07T09:26:20.975 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-07T09:26:20.975 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:17.2.6-1320.g08cac018.el8 ceph 5.6 M 2023-10-07T09:26:20.975 INFO:teuthology.orchestra.run.smithi144.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2023-10-07T09:26:20.975 INFO:teuthology.orchestra.run.smithi144.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2023-10-07T09:26:20.975 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-07T09:26:20.975 INFO:teuthology.orchestra.run.smithi144.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2023-10-07T09:26:20.976 INFO:teuthology.orchestra.run.smithi144.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2023-10-07T09:26:20.976 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-07T09:26:20.976 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-07T09:26:20.977 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-07T09:26:20.977 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-07T09:26:20.977 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2023-10-07T09:26:20.977 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2023-10-07T09:26:20.977 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2023-10-07T09:26:20.977 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-07T09:26:20.978 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-07T09:26:20.978 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-07T09:26:20.978 INFO:teuthology.orchestra.run.smithi144.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-07T09:26:20.978 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-07T09:26:20.978 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse x86_64 2:17.2.6-1320.g08cac018.el8 ceph 45 k 2023-10-07T09:26:20.978 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:17.2.6-1320.g08cac018.el8 ceph 115 k 2023-10-07T09:26:20.979 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-07T09:26:20.979 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-07T09:26:20.979 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-07T09:26:20.979 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-07T09:26:20.979 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-07T09:26:20.979 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-07T09:26:20.980 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-07T09:26:20.980 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-07T09:26:20.980 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-07T09:26:20.980 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-07T09:26:20.980 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-07T09:26:20.980 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-07T09:26:20.981 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-07T09:26:20.981 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-07T09:26:20.981 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-07T09:26:20.981 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-07T09:26:20.981 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-07T09:26:20.981 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-07T09:26:20.982 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-07T09:26:20.982 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-07T09:26:20.982 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-07T09:26:20.982 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-07T09:26:20.982 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-07T09:26:20.983 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-07T09:26:20.984 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-07T09:26:20.985 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-10-07T09:26:20.986 INFO:teuthology.orchestra.run.smithi144.stdout:==================================================================================================== 2023-10-07T09:26:20.987 INFO:teuthology.orchestra.run.smithi144.stdout:Install 123 Packages 2023-10-07T09:26:20.987 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 16 Packages 2023-10-07T09:26:20.987 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:26:20.988 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 210 M 2023-10-07T09:26:20.988 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2023-10-07T09:26:21.391 INFO:teuthology.orchestra.run.smithi144.stdout:(1/139): ceph-17.2.6-1320.g08cac018.el8.x86_64. 50 kB/s | 6.4 kB 00:00 2023-10-07T09:26:21.446 INFO:teuthology.orchestra.run.smithi029.stdout:(40/139): flex-2.6.1-9.el8.x86_64.rpm 564 kB/s | 320 kB 00:00 2023-10-07T09:26:21.497 INFO:teuthology.orchestra.run.smithi029.stdout:(41/139): bison-3.0.4-10.el8.x86_64.rpm 393 kB/s | 688 kB 00:01 2023-10-07T09:26:21.523 INFO:teuthology.orchestra.run.smithi029.stdout:(42/139): jq-1.6-7.el8.x86_64.rpm 362 kB/s | 202 kB 00:00 2023-10-07T09:26:21.566 INFO:teuthology.orchestra.run.smithi144.stdout:(2/139): ceph-fuse-17.2.6-1320.g08cac018.el8.x8 4.9 MB/s | 876 kB 00:00 2023-10-07T09:26:21.617 INFO:teuthology.orchestra.run.smithi144.stdout:(3/139): ceph-immutable-object-cache-17.2.6-132 3.2 MB/s | 162 kB 00:00 2023-10-07T09:26:21.900 INFO:teuthology.orchestra.run.smithi144.stdout:(4/139): ceph-mds-17.2.6-1320.g08cac018.el8.x86 8.1 MB/s | 2.3 MB 00:00 2023-10-07T09:26:21.992 INFO:teuthology.orchestra.run.smithi144.stdout:(5/139): ceph-base-17.2.6-1320.g08cac018.el8.x8 7.1 MB/s | 5.2 MB 00:00 2023-10-07T09:26:22.048 INFO:teuthology.orchestra.run.smithi029.stdout:(43/139): lttng-ust-2.8.1-11.el8.x86_64.rpm 494 kB/s | 259 kB 00:00 2023-10-07T09:26:22.051 INFO:teuthology.orchestra.run.smithi144.stdout:(6/139): ceph-mgr-17.2.6-1320.g08cac018.el8.x86 10 MB/s | 1.6 MB 00:00 2023-10-07T09:26:22.075 INFO:teuthology.orchestra.run.smithi029.stdout:(44/139): librdkafka-0.11.4-3.el8.x86_64.rpm 563 kB/s | 354 kB 00:00 2023-10-07T09:26:22.293 INFO:teuthology.orchestra.run.smithi029.stdout:(45/139): libtool-2.4.6-25.el8.x86_64.rpm 891 kB/s | 709 kB 00:00 2023-10-07T09:26:22.335 INFO:teuthology.orchestra.run.smithi144.stdout:(7/139): ceph-mon-17.2.6-1320.g08cac018.el8.x86 11 MB/s | 3.9 MB 00:00 2023-10-07T09:26:22.551 INFO:teuthology.orchestra.run.smithi029.stdout:(46/139): oniguruma-6.8.2-2.el8.x86_64.rpm 373 kB/s | 187 kB 00:00 2023-10-07T09:26:23.318 INFO:teuthology.orchestra.run.smithi029.stdout:(47/139): openblas-0.3.15-4.el8.x86_64.rpm 3.8 MB/s | 4.7 MB 00:01 2023-10-07T09:26:23.511 INFO:teuthology.orchestra.run.smithi029.stdout:(48/139): openblas-threads-0.3.15-4.el8.x86_64. 4.0 MB/s | 4.9 MB 00:01 2023-10-07T09:26:23.760 INFO:teuthology.orchestra.run.smithi144.stdout:(8/139): ceph-osd-17.2.6-1320.g08cac018.el8.x86 9.8 MB/s | 17 MB 00:01 2023-10-07T09:26:23.811 INFO:teuthology.orchestra.run.smithi029.stdout:(49/139): python3-babel-2.5.1-7.el8.noarch.rpm 3.8 MB/s | 4.8 MB 00:01 2023-10-07T09:26:23.837 INFO:teuthology.orchestra.run.smithi029.stdout:(50/139): python3-jsonpatch-1.21-2.el8.noarch.r 82 kB/s | 27 kB 00:00 2023-10-07T09:26:23.879 INFO:teuthology.orchestra.run.smithi029.stdout:(51/139): python3-jinja2-2.10.1-3.el8.noarch.rp 961 kB/s | 538 kB 00:00 2023-10-07T09:26:23.960 INFO:teuthology.orchestra.run.smithi144.stdout:(9/139): ceph-radosgw-17.2.6-1320.g08cac018.el8 10 MB/s | 17 MB 00:01 2023-10-07T09:26:23.985 INFO:teuthology.orchestra.run.smithi144.stdout:(10/139): ceph-selinux-17.2.6-1320.g08cac018.el 108 kB/s | 24 kB 00:00 2023-10-07T09:26:24.038 INFO:teuthology.orchestra.run.smithi029.stdout:(52/139): python3-jsonpointer-1.10-11.el8.noarc 88 kB/s | 20 kB 00:00 2023-10-07T09:26:24.044 INFO:teuthology.orchestra.run.smithi144.stdout:(11/139): cephfs-mirror-17.2.6-1320.g08cac018.e 4.3 MB/s | 255 kB 00:00 2023-10-07T09:26:24.103 INFO:teuthology.orchestra.run.smithi144.stdout:(12/139): libcephfs-devel-17.2.6-1320.g08cac018 427 kB/s | 25 kB 00:00 2023-10-07T09:26:24.113 INFO:teuthology.orchestra.run.smithi029.stdout:(53/139): python3-markupsafe-0.23-19.el8.x86_64 165 kB/s | 39 kB 00:00 2023-10-07T09:26:24.236 INFO:teuthology.orchestra.run.smithi144.stdout:(13/139): libcephfs2-17.2.6-1320.g08cac018.el8. 5.8 MB/s | 797 kB 00:00 2023-10-07T09:26:24.282 INFO:teuthology.orchestra.run.smithi029.stdout:(54/139): python3-mako-1.0.6-14.el8.noarch.rpm 354 kB/s | 157 kB 00:00 2023-10-07T09:26:24.311 INFO:teuthology.orchestra.run.smithi144.stdout:(14/139): libcephsqlite-17.2.6-1320.g08cac018.e 2.3 MB/s | 174 kB 00:00 2023-10-07T09:26:24.379 INFO:teuthology.orchestra.run.smithi144.stdout:(15/139): librados-devel-17.2.6-1320.g08cac018. 1.9 MB/s | 130 kB 00:00 2023-10-07T09:26:24.470 INFO:teuthology.orchestra.run.smithi144.stdout:(16/139): libradosstriper1-17.2.6-1320.g08cac01 5.4 MB/s | 507 kB 00:00 2023-10-07T09:26:24.565 INFO:teuthology.orchestra.run.smithi029.stdout:(55/139): python3-numpy-f2py-1.14.3-10.el8.x86_ 499 kB/s | 225 kB 00:00 2023-10-07T09:26:24.590 INFO:teuthology.orchestra.run.smithi029.stdout:(56/139): python3-prettytable-0.7.2-14.el8.noar 142 kB/s | 44 kB 00:00 2023-10-07T09:26:24.925 INFO:teuthology.orchestra.run.smithi029.stdout:(57/139): python3-pyOpenSSL-19.0.0-1.el8.noarch 288 kB/s | 103 kB 00:00 2023-10-07T09:26:25.093 INFO:teuthology.orchestra.run.smithi029.stdout:(58/139): python3-numpy-1.14.3-10.el8.x86_64.rp 3.5 MB/s | 3.7 MB 00:01 2023-10-07T09:26:25.104 INFO:teuthology.orchestra.run.smithi144.stdout:(17/139): librgw2-17.2.6-1320.g08cac018.el8.x86 8.8 MB/s | 5.6 MB 00:00 2023-10-07T09:26:25.118 INFO:teuthology.orchestra.run.smithi029.stdout:(59/139): python3-pyasn1-0.3.7-6.el8.noarch.rpm 239 kB/s | 126 kB 00:00 2023-10-07T09:26:25.163 INFO:teuthology.orchestra.run.smithi144.stdout:(18/139): python3-ceph-argparse-17.2.6-1320.g08 770 kB/s | 45 kB 00:00 2023-10-07T09:26:25.221 INFO:teuthology.orchestra.run.smithi144.stdout:(19/139): python3-ceph-common-17.2.6-1320.g08ca 1.9 MB/s | 115 kB 00:00 2023-10-07T09:26:25.288 INFO:teuthology.orchestra.run.smithi144.stdout:(20/139): python3-cephfs-17.2.6-1320.g08cac018. 2.8 MB/s | 190 kB 00:00 2023-10-07T09:26:25.344 INFO:teuthology.orchestra.run.smithi029.stdout:(60/139): python3-pyasn1-modules-0.3.7-6.el8.no 263 kB/s | 110 kB 00:00 2023-10-07T09:26:25.363 INFO:teuthology.orchestra.run.smithi144.stdout:(21/139): python3-rados-17.2.6-1320.g08cac018.e 4.8 MB/s | 371 kB 00:00 2023-10-07T09:26:25.378 INFO:teuthology.orchestra.run.smithi029.stdout:(61/139): python3-pytz-2017.2-11.el8.noarch.rpm 200 kB/s | 57 kB 00:00 2023-10-07T09:26:25.439 INFO:teuthology.orchestra.run.smithi144.stdout:(22/139): python3-rbd-17.2.6-1320.g08cac018.el8 4.7 MB/s | 358 kB 00:00 2023-10-07T09:26:25.506 INFO:teuthology.orchestra.run.smithi144.stdout:(23/139): python3-rgw-17.2.6-1320.g08cac018.el8 1.6 MB/s | 107 kB 00:00 2023-10-07T09:26:25.798 INFO:teuthology.orchestra.run.smithi144.stdout:(24/139): ceph-common-17.2.6-1320.g08cac018.el8 5.2 MB/s | 23 MB 00:04 2023-10-07T09:26:25.815 INFO:teuthology.orchestra.run.smithi144.stdout:(25/139): rbd-fuse-17.2.6-1320.g08cac018.el8.x8 293 kB/s | 90 kB 00:00 2023-10-07T09:26:25.857 INFO:teuthology.orchestra.run.smithi144.stdout:(26/139): rbd-nbd-17.2.6-1320.g08cac018.el8.x86 4.1 MB/s | 173 kB 00:00 2023-10-07T09:26:25.915 INFO:teuthology.orchestra.run.smithi144.stdout:(27/139): ceph-grafana-dashboards-17.2.6-1320.g 396 kB/s | 23 kB 00:00 2023-10-07T09:26:25.921 INFO:teuthology.orchestra.run.smithi029.stdout:(62/139): socat-1.7.4.1-1.el8.x86_64.rpm 596 kB/s | 323 kB 00:00 2023-10-07T09:26:25.983 INFO:teuthology.orchestra.run.smithi144.stdout:(28/139): ceph-mgr-cephadm-17.2.6-1320.g08cac01 1.8 MB/s | 121 kB 00:00 2023-10-07T09:26:26.021 INFO:teuthology.orchestra.run.smithi029.stdout:(63/139): python3-werkzeug-0.12.2-4.el8.noarch. 676 kB/s | 457 kB 00:00 2023-10-07T09:26:26.105 INFO:teuthology.orchestra.run.smithi029.stdout:(64/139): elfutils-libelf-devel-0.189-3.el8.x86 740 kB/s | 62 kB 00:00 2023-10-07T09:26:26.163 INFO:teuthology.orchestra.run.smithi029.stdout:(65/139): fuse-2.9.7-17.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-10-07T09:26:26.206 INFO:teuthology.orchestra.run.smithi029.stdout:(66/139): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2023-10-07T09:26:26.233 INFO:teuthology.orchestra.run.smithi144.stdout:(29/139): ceph-mgr-dashboard-17.2.6-1320.g08cac 6.8 MB/s | 1.7 MB 00:00 2023-10-07T09:26:26.265 INFO:teuthology.orchestra.run.smithi029.stdout:(67/139): xmlstarlet-1.6.1-20.el8.x86_64.rpm 205 kB/s | 71 kB 00:00 2023-10-07T09:26:26.298 INFO:teuthology.orchestra.run.smithi029.stdout:(68/139): krb5-devel-1.18.2-22.el8.x86_64.rpm 5.9 MB/s | 561 kB 00:00 2023-10-07T09:26:26.375 INFO:teuthology.orchestra.run.smithi144.stdout:(30/139): rbd-mirror-17.2.6-1320.g08cac018.el8. 6.0 MB/s | 3.5 MB 00:00 2023-10-07T09:26:26.391 INFO:teuthology.orchestra.run.smithi029.stdout:(69/139): libcom_err-devel-1.45.6-5.el8.x86_64. 421 kB/s | 39 kB 00:00 2023-10-07T09:26:26.417 INFO:teuthology.orchestra.run.smithi029.stdout:(70/139): libaio-devel-0.3.112-1.el8.x86_64.rpm 123 kB/s | 19 kB 00:00 2023-10-07T09:26:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:(71/139): libgfortran-8.5.0-20.el8.x86_64.rpm 9.4 MB/s | 645 kB 00:00 2023-10-07T09:26:26.467 INFO:teuthology.orchestra.run.smithi144.stdout:(31/139): ceph-mgr-modules-core-17.2.6-1320.g08 2.5 MB/s | 238 kB 00:00 2023-10-07T09:26:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:(72/139): librabbitmq-0.9.0-4.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-10-07T09:26:26.528 INFO:teuthology.orchestra.run.smithi029.stdout:(73/139): libquadmath-8.5.0-20.el8.x86_64.rpm 1.5 MB/s | 172 kB 00:00 2023-10-07T09:26:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:(74/139): libselinux-devel-2.9-8.el8.x86_64.rpm 3.7 MB/s | 200 kB 00:00 2023-10-07T09:26:26.559 INFO:teuthology.orchestra.run.smithi144.stdout:(32/139): ceph-mgr-rook-17.2.6-1320.g08cac018.e 518 kB/s | 47 kB 00:00 2023-10-07T09:26:26.579 INFO:teuthology.orchestra.run.smithi029.stdout:(75/139): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-10-07T09:26:26.605 INFO:teuthology.orchestra.run.smithi029.stdout:(76/139): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 100 kB 00:00 2023-10-07T09:26:26.630 INFO:teuthology.orchestra.run.smithi029.stdout:(77/139): libverto-devel-0.3.2-2.el8.x86_64.rpm 358 kB/s | 18 kB 00:00 2023-10-07T09:26:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:(78/139): libzstd-devel-1.4.4-1.el8.x86_64.rpm 863 kB/s | 44 kB 00:00 2023-10-07T09:26:26.659 INFO:teuthology.orchestra.run.smithi144.stdout:(33/139): ceph-prometheus-alerts-17.2.6-1320.g0 144 kB/s | 14 kB 00:00 2023-10-07T09:26:26.715 INFO:teuthology.orchestra.run.smithi029.stdout:(79/139): pcre2-devel-10.32-3.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-10-07T09:26:26.760 INFO:teuthology.orchestra.run.smithi144.stdout:(34/139): ceph-volume-17.2.6-1320.g08cac018.el8 2.6 MB/s | 267 kB 00:00 2023-10-07T09:26:26.766 INFO:teuthology.orchestra.run.smithi029.stdout:(80/139): pcre2-utf16-10.32-3.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-10-07T09:26:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:(81/139): pcre2-utf32-10.32-3.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-10-07T09:26:26.852 INFO:teuthology.orchestra.run.smithi144.stdout:(35/139): cephadm-17.2.6-1320.g08cac018.el8.noa 877 kB/s | 80 kB 00:00 2023-10-07T09:26:26.900 INFO:teuthology.orchestra.run.smithi029.stdout:(82/139): openssl-devel-1.1.1k-9.el8.x86_64.rpm 12 MB/s | 3.2 MB 00:00 2023-10-07T09:26:26.934 INFO:teuthology.orchestra.run.smithi029.stdout:(83/139): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.1 MB/s | 249 kB 00:00 2023-10-07T09:26:26.976 INFO:teuthology.orchestra.run.smithi029.stdout:(84/139): python3-cryptography-3.2.1-6.el8.x86_ 8.3 MB/s | 641 kB 00:00 2023-10-07T09:26:26.977 INFO:teuthology.orchestra.run.smithi144.stdout:(36/139): ceph-mgr-diskprediction-local-17.2.6- 10 MB/s | 7.4 MB 00:00 2023-10-07T09:26:26.994 INFO:teuthology.orchestra.run.smithi144.stdout:(37/139): cephfs-top-17.2.6-1320.g08cac018.el8. 127 kB/s | 18 kB 00:00 2023-10-07T09:26:27.001 INFO:teuthology.orchestra.run.smithi029.stdout:(85/139): python3-jwt-1.6.1-2.el8.noarch.rpm 635 kB/s | 43 kB 00:00 2023-10-07T09:26:27.036 INFO:teuthology.orchestra.run.smithi029.stdout:(86/139): python3-oauthlib-2.1.0-1.el8.noarch.r 2.6 MB/s | 155 kB 00:00 2023-10-07T09:26:27.063 INFO:teuthology.orchestra.run.smithi029.stdout:(87/139): python3-pycparser-2.14-14.el8.noarch. 1.8 MB/s | 109 kB 00:00 2023-10-07T09:26:27.088 INFO:teuthology.orchestra.run.smithi029.stdout:(88/139): python3-requests-oauthlib-1.0.0-1.el8 817 kB/s | 43 kB 00:00 2023-10-07T09:26:27.122 INFO:teuthology.orchestra.run.smithi029.stdout:(89/139): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.2 MB/s | 74 kB 00:00 2023-10-07T09:26:27.156 INFO:teuthology.orchestra.run.smithi029.stdout:(90/139): zlib-devel-1.2.11-21.el8.x86_64.rpm 862 kB/s | 58 kB 00:00 2023-10-07T09:26:27.482 INFO:teuthology.orchestra.run.smithi029.stdout:(91/139): liboath-2.6.2-3.el8.x86_64.rpm 181 kB/s | 59 kB 00:00 2023-10-07T09:26:27.661 INFO:teuthology.orchestra.run.smithi144.stdout:(38/139): ceph-test-17.2.6-1320.g08cac018.el8.x 14 MB/s | 52 MB 00:03 2023-10-07T09:26:27.774 INFO:teuthology.orchestra.run.smithi029.stdout:(92/139): python3-scipy-1.0.0-21.module_el8.5.0 5.2 MB/s | 14 MB 00:02 2023-10-07T09:26:27.808 INFO:teuthology.orchestra.run.smithi029.stdout:(93/139): gperftools-libs-2.7-9.el8.x86_64.rpm 447 kB/s | 306 kB 00:00 2023-10-07T09:26:27.841 INFO:teuthology.orchestra.run.smithi029.stdout:(94/139): libunwind-1.3.1-3.el8.x86_64.rpm 209 kB/s | 75 kB 00:00 2023-10-07T09:26:27.925 INFO:teuthology.orchestra.run.smithi029.stdout:(95/139): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 560 kB/s | 44 kB 00:00 2023-10-07T09:26:28.001 INFO:teuthology.orchestra.run.smithi029.stdout:(96/139): python3-cachetools-3.1.1-4.el8.noarch 439 kB/s | 33 kB 00:00 2023-10-07T09:26:28.011 INFO:teuthology.orchestra.run.smithi144.stdout:(39/139): bison-3.0.4-10.el8.x86_64.rpm 677 kB/s | 688 kB 00:01 2023-10-07T09:26:28.035 INFO:teuthology.orchestra.run.smithi029.stdout:(97/139): python3-beautifulsoup4-4.6.3-2.el8.1. 989 kB/s | 185 kB 00:00 2023-10-07T09:26:28.036 INFO:teuthology.orchestra.run.smithi144.stdout:(40/139): automake-1.16.1-8.el8.noarch.rpm 698 kB/s | 740 kB 00:01 2023-10-07T09:26:28.077 INFO:teuthology.orchestra.run.smithi029.stdout:(98/139): python3-certifi-2018.10.15-7.el8.noar 215 kB/s | 16 kB 00:00 2023-10-07T09:26:28.144 INFO:teuthology.orchestra.run.smithi029.stdout:(99/139): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-10-07T09:26:28.219 INFO:teuthology.orchestra.run.smithi029.stdout:(100/139): python3-cherrypy-18.4.0-1.el8.noarch 2.6 MB/s | 384 kB 00:00 2023-10-07T09:26:28.228 INFO:teuthology.orchestra.run.smithi144.stdout:(41/139): flex-2.6.1-9.el8.x86_64.rpm 564 kB/s | 320 kB 00:00 2023-10-07T09:26:28.253 INFO:teuthology.orchestra.run.smithi029.stdout:(101/139): python3-google-auth-1.1.1-10.el8.noa 748 kB/s | 81 kB 00:00 2023-10-07T09:26:28.297 INFO:teuthology.orchestra.run.smithi029.stdout:(102/139): python3-asyncssh-2.7.0-2.el8.noarch. 910 kB/s | 476 kB 00:00 2023-10-07T09:26:28.323 INFO:teuthology.orchestra.run.smithi029.stdout:(103/139): python3-jaraco-6.2-6.el8.noarch.rpm 108 kB/s | 11 kB 00:00 2023-10-07T09:26:28.348 INFO:teuthology.orchestra.run.smithi029.stdout:(104/139): python3-jaraco-functools-2.0-4.el8.n 186 kB/s | 18 kB 00:00 2023-10-07T09:26:28.407 INFO:teuthology.orchestra.run.smithi029.stdout:(105/139): python3-logutils-0.3.5-11.el8.noarch 580 kB/s | 49 kB 00:00 2023-10-07T09:26:28.441 INFO:teuthology.orchestra.run.smithi029.stdout:(106/139): python3-more-itertools-7.2.0-3.el8.n 637 kB/s | 59 kB 00:00 2023-10-07T09:26:28.491 INFO:teuthology.orchestra.run.smithi029.stdout:(107/139): python3-kubernetes-11.0.0-6.el8.noar 6.0 MB/s | 1.2 MB 00:00 2023-10-07T09:26:28.512 INFO:teuthology.orchestra.run.smithi144.stdout:(42/139): jq-1.6-7.el8.x86_64.rpm 405 kB/s | 202 kB 00:00 2023-10-07T09:26:28.576 INFO:teuthology.orchestra.run.smithi029.stdout:(108/139): python3-natsort-7.1.1-2.el8.noarch.r 359 kB/s | 60 kB 00:00 2023-10-07T09:26:28.601 INFO:teuthology.orchestra.run.smithi029.stdout:(109/139): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-10-07T09:26:28.621 INFO:teuthology.orchestra.run.smithi144.stdout:(43/139): librdkafka-0.11.4-3.el8.x86_64.rpm 606 kB/s | 354 kB 00:00 2023-10-07T09:26:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:(110/139): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-10-07T09:26:28.668 INFO:teuthology.orchestra.run.smithi029.stdout:(111/139): python3-repoze-lru-0.7-6.el8.noarch. 364 kB/s | 34 kB 00:00 2023-10-07T09:26:28.702 INFO:teuthology.orchestra.run.smithi029.stdout:(112/139): python3-routes-2.4.1-12.el8.noarch.r 1.9 MB/s | 196 kB 00:00 2023-10-07T09:26:28.728 INFO:teuthology.orchestra.run.smithi029.stdout:(113/139): python3-rsa-4.9-2.el8.noarch.rpm 691 kB/s | 64 kB 00:00 2023-10-07T09:26:28.753 INFO:teuthology.orchestra.run.smithi029.stdout:(114/139): python3-simplegeneric-0.8.1-17.el8.n 221 kB/s | 19 kB 00:00 2023-10-07T09:26:28.779 INFO:teuthology.orchestra.run.smithi029.stdout:(115/139): python3-singledispatch-3.4.0.3-18.el 314 kB/s | 24 kB 00:00 2023-10-07T09:26:28.805 INFO:teuthology.orchestra.run.smithi029.stdout:(116/139): python3-tempora-1.14.1-5.el8.noarch. 384 kB/s | 29 kB 00:00 2023-10-07T09:26:28.830 INFO:teuthology.orchestra.run.smithi029.stdout:(117/139): python3-trustme-0.6.0-4.el8.noarch.r 361 kB/s | 27 kB 00:00 2023-10-07T09:26:28.890 INFO:teuthology.orchestra.run.smithi029.stdout:(118/139): python3-waitress-1.4.3-1.el8.noarch. 2.2 MB/s | 255 kB 00:00 2023-10-07T09:26:28.916 INFO:teuthology.orchestra.run.smithi029.stdout:(119/139): python3-websocket-client-0.56.0-5.el 717 kB/s | 61 kB 00:00 2023-10-07T09:26:28.950 INFO:teuthology.orchestra.run.smithi029.stdout:(120/139): python3-webob-1.8.5-1.el8.1.noarch.r 1.7 MB/s | 251 kB 00:00 2023-10-07T09:26:28.984 INFO:teuthology.orchestra.run.smithi029.stdout:(121/139): python3-webtest-2.0.33-1.el8.noarch. 921 kB/s | 85 kB 00:00 2023-10-07T09:26:29.009 INFO:teuthology.orchestra.run.smithi029.stdout:(122/139): python3-zc-lockfile-2.0-2.el8.noarch 247 kB/s | 23 kB 00:00 2023-10-07T09:26:29.029 INFO:teuthology.orchestra.run.smithi144.stdout:(44/139): libtool-2.4.6-25.el8.x86_64.rpm 886 kB/s | 709 kB 00:00 2023-10-07T09:26:29.054 INFO:teuthology.orchestra.run.smithi144.stdout:(45/139): lttng-ust-2.8.1-11.el8.x86_64.rpm 479 kB/s | 259 kB 00:00 2023-10-07T09:26:29.080 INFO:teuthology.orchestra.run.smithi144.stdout:(46/139): oniguruma-6.8.2-2.el8.x86_64.rpm 409 kB/s | 187 kB 00:00 2023-10-07T09:26:29.201 INFO:teuthology.orchestra.run.smithi029.stdout:(123/139): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2023-10-07T09:26:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:(124/139): NetworkManager-initscripts-updown-1. 1.1 MB/s | 143 kB 00:00 2023-10-07T09:26:29.419 INFO:teuthology.orchestra.run.smithi029.stdout:(125/139): elfutils-debuginfod-client-0.189-3.e 828 kB/s | 76 kB 00:00 2023-10-07T09:26:29.670 INFO:teuthology.orchestra.run.smithi029.stdout:(126/139): elfutils-libelf-0.189-3.el8.x86_64.r 927 kB/s | 232 kB 00:00 2023-10-07T09:26:29.787 INFO:teuthology.orchestra.run.smithi029.stdout:(127/139): elfutils-libs-0.189-3.el8.x86_64.rpm 2.5 MB/s | 303 kB 00:00 2023-10-07T09:26:29.988 INFO:teuthology.orchestra.run.smithi029.stdout:(128/139): librbd1-17.2.6-1320.g08cac018.el8.x8 3.7 MB/s | 3.6 MB 00:00 2023-10-07T09:26:30.088 INFO:teuthology.orchestra.run.smithi029.stdout:(129/139): librados2-17.2.6-1320.g08cac018.el8. 3.1 MB/s | 3.5 MB 00:01 2023-10-07T09:26:30.122 INFO:teuthology.orchestra.run.smithi029.stdout:(130/139): libblkid-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-10-07T09:26:30.198 INFO:teuthology.orchestra.run.smithi029.stdout:(131/139): iproute-6.2.0-5.el8.x86_64.rpm 2.0 MB/s | 854 kB 00:00 2023-10-07T09:26:30.248 INFO:teuthology.orchestra.run.smithi029.stdout:(132/139): libmount-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 236 kB 00:00 2023-10-07T09:26:30.282 INFO:teuthology.orchestra.run.smithi144.stdout:(47/139): python3-babel-2.5.1-7.el8.noarch.rpm 4.0 MB/s | 4.8 MB 00:01 2023-10-07T09:26:30.308 INFO:teuthology.orchestra.run.smithi029.stdout:(133/139): libfdisk-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 254 kB 00:00 2023-10-07T09:26:30.342 INFO:teuthology.orchestra.run.smithi029.stdout:(134/139): libsmartcols-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 179 kB 00:00 2023-10-07T09:26:30.355 INFO:teuthology.orchestra.run.smithi144.stdout:(48/139): openblas-0.3.15-4.el8.x86_64.rpm 3.6 MB/s | 4.7 MB 00:01 2023-10-07T09:26:30.376 INFO:teuthology.orchestra.run.smithi029.stdout:(135/139): libuuid-2.32.1-43.el8.x86_64.rpm 775 kB/s | 99 kB 00:00 2023-10-07T09:26:30.444 INFO:teuthology.orchestra.run.smithi029.stdout:(136/139): xfsdump-3.1.8-6.el8.x86_64.rpm 4.9 MB/s | 334 kB 00:00 2023-10-07T09:26:30.477 INFO:teuthology.orchestra.run.smithi029.stdout:(137/139): util-linux-user-2.32.1-43.el8.x86_64 763 kB/s | 103 kB 00:00 2023-10-07T09:26:30.514 INFO:teuthology.orchestra.run.smithi144.stdout:(49/139): openblas-threads-0.3.15-4.el8.x86_64. 3.4 MB/s | 4.9 MB 00:01 2023-10-07T09:26:30.561 INFO:teuthology.orchestra.run.smithi029.stdout:(138/139): util-linux-2.32.1-43.el8.x86_64.rpm 9.8 MB/s | 2.5 MB 00:00 2023-10-07T09:26:30.645 INFO:teuthology.orchestra.run.smithi029.stdout:(139/139): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2023-10-07T09:26:30.650 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-10-07T09:26:30.650 INFO:teuthology.orchestra.run.smithi029.stdout:Total 12 MB/s | 210 MB 00:17 2023-10-07T09:26:30.697 INFO:teuthology.orchestra.run.smithi144.stdout:(50/139): python3-jsonpatch-1.21-2.el8.noarch.r 78 kB/s | 27 kB 00:00 2023-10-07T09:26:30.748 INFO:teuthology.orchestra.run.smithi144.stdout:(51/139): python3-jsonpointer-1.10-11.el8.noarc 85 kB/s | 20 kB 00:00 2023-10-07T09:26:31.031 INFO:teuthology.orchestra.run.smithi144.stdout:(52/139): python3-markupsafe-0.23-19.el8.x86_64 137 kB/s | 39 kB 00:00 2023-10-07T09:26:31.057 INFO:teuthology.orchestra.run.smithi144.stdout:(53/139): python3-jinja2-2.10.1-3.el8.noarch.rp 695 kB/s | 538 kB 00:00 2023-10-07T09:26:31.123 INFO:teuthology.orchestra.run.smithi144.stdout:(54/139): python3-mako-1.0.6-14.el8.noarch.rpm 368 kB/s | 157 kB 00:00 2023-10-07T09:26:31.407 INFO:teuthology.orchestra.run.smithi144.stdout:(55/139): python3-prettytable-0.7.2-14.el8.noar 155 kB/s | 44 kB 00:00 2023-10-07T09:26:31.541 INFO:teuthology.orchestra.run.smithi144.stdout:(56/139): python3-numpy-f2py-1.14.3-10.el8.x86_ 465 kB/s | 225 kB 00:00 2023-10-07T09:26:31.816 INFO:teuthology.orchestra.run.smithi144.stdout:(57/139): python3-pyOpenSSL-19.0.0-1.el8.noarch 253 kB/s | 103 kB 00:00 2023-10-07T09:26:31.983 INFO:teuthology.orchestra.run.smithi144.stdout:(58/139): python3-pyasn1-0.3.7-6.el8.noarch.rpm 286 kB/s | 126 kB 00:00 2023-10-07T09:26:32.166 INFO:teuthology.orchestra.run.smithi144.stdout:(59/139): python3-pyasn1-modules-0.3.7-6.el8.no 314 kB/s | 110 kB 00:00 2023-10-07T09:26:32.267 INFO:teuthology.orchestra.run.smithi144.stdout:(60/139): python3-numpy-1.14.3-10.el8.x86_64.rp 3.0 MB/s | 3.7 MB 00:01 2023-10-07T09:26:32.293 INFO:teuthology.orchestra.run.smithi144.stdout:(61/139): python3-pytz-2017.2-11.el8.noarch.rpm 184 kB/s | 57 kB 00:00 2023-10-07T09:26:32.399 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-10-07T09:26:32.836 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-10-07T09:26:32.836 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-10-07T09:26:32.993 INFO:teuthology.orchestra.run.smithi144.stdout:(62/139): socat-1.7.4.1-1.el8.x86_64.rpm 462 kB/s | 323 kB 00:00 2023-10-07T09:26:33.019 INFO:teuthology.orchestra.run.smithi144.stdout:(63/139): python3-werkzeug-0.12.2-4.el8.noarch. 610 kB/s | 457 kB 00:00 2023-10-07T09:26:33.119 INFO:teuthology.orchestra.run.smithi144.stdout:(64/139): elfutils-libelf-devel-0.189-3.el8.x86 617 kB/s | 62 kB 00:00 2023-10-07T09:26:33.177 INFO:teuthology.orchestra.run.smithi144.stdout:(65/139): fuse-2.9.7-17.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-10-07T09:26:33.211 INFO:teuthology.orchestra.run.smithi144.stdout:(66/139): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-10-07T09:26:33.269 INFO:teuthology.orchestra.run.smithi144.stdout:(67/139): xmlstarlet-1.6.1-20.el8.x86_64.rpm 256 kB/s | 71 kB 00:00 2023-10-07T09:26:33.294 INFO:teuthology.orchestra.run.smithi144.stdout:(68/139): krb5-devel-1.18.2-22.el8.x86_64.rpm 6.6 MB/s | 561 kB 00:00 2023-10-07T09:26:33.336 INFO:teuthology.orchestra.run.smithi144.stdout:(69/139): libcom_err-devel-1.45.6-5.el8.x86_64. 931 kB/s | 39 kB 00:00 2023-10-07T09:26:33.353 INFO:teuthology.orchestra.run.smithi144.stdout:(70/139): libaio-devel-0.3.112-1.el8.x86_64.rpm 223 kB/s | 19 kB 00:00 2023-10-07T09:26:33.412 INFO:teuthology.orchestra.run.smithi144.stdout:(71/139): libgfortran-8.5.0-20.el8.x86_64.rpm 8.4 MB/s | 645 kB 00:00 2023-10-07T09:26:33.454 INFO:teuthology.orchestra.run.smithi144.stdout:(72/139): libquadmath-8.5.0-20.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2023-10-07T09:26:33.471 INFO:teuthology.orchestra.run.smithi144.stdout:(73/139): librabbitmq-0.9.0-4.el8.x86_64.rpm 805 kB/s | 47 kB 00:00 2023-10-07T09:26:33.513 INFO:teuthology.orchestra.run.smithi144.stdout:(74/139): libselinux-devel-2.9-8.el8.x86_64.rpm 3.3 MB/s | 200 kB 00:00 2023-10-07T09:26:33.530 INFO:teuthology.orchestra.run.smithi144.stdout:(75/139): libsepol-devel-2.9-3.el8.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2023-10-07T09:26:33.555 INFO:teuthology.orchestra.run.smithi144.stdout:(76/139): libuuid-devel-2.32.1-43.el8.x86_64.rp 2.3 MB/s | 100 kB 00:00 2023-10-07T09:26:33.581 INFO:teuthology.orchestra.run.smithi144.stdout:(77/139): libverto-devel-0.3.2-2.el8.x86_64.rpm 360 kB/s | 18 kB 00:00 2023-10-07T09:26:33.606 INFO:teuthology.orchestra.run.smithi144.stdout:(78/139): libzstd-devel-1.4.4-1.el8.x86_64.rpm 876 kB/s | 44 kB 00:00 2023-10-07T09:26:33.672 INFO:teuthology.orchestra.run.smithi144.stdout:(79/139): pcre2-devel-10.32-3.el8.x86_64.rpm 8.9 MB/s | 605 kB 00:00 2023-10-07T09:26:33.723 INFO:teuthology.orchestra.run.smithi144.stdout:(80/139): pcre2-utf16-10.32-3.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-10-07T09:26:33.765 INFO:teuthology.orchestra.run.smithi144.stdout:(81/139): pcre2-utf32-10.32-3.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-10-07T09:26:33.863 INFO:teuthology.orchestra.run.smithi144.stdout:(82/139): openssl-devel-1.1.1k-9.el8.x86_64.rpm 11 MB/s | 3.2 MB 00:00 2023-10-07T09:26:33.899 INFO:teuthology.orchestra.run.smithi144.stdout:(83/139): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.8 MB/s | 249 kB 00:00 2023-10-07T09:26:33.941 INFO:teuthology.orchestra.run.smithi144.stdout:(84/139): python3-cryptography-3.2.1-6.el8.x86_ 8.1 MB/s | 641 kB 00:00 2023-10-07T09:26:33.966 INFO:teuthology.orchestra.run.smithi144.stdout:(85/139): python3-jwt-1.6.1-2.el8.noarch.rpm 641 kB/s | 43 kB 00:00 2023-10-07T09:26:33.991 INFO:teuthology.orchestra.run.smithi144.stdout:(86/139): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-10-07T09:26:34.016 INFO:teuthology.orchestra.run.smithi144.stdout:(87/139): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-10-07T09:26:34.050 INFO:teuthology.orchestra.run.smithi144.stdout:(88/139): python3-requests-oauthlib-1.0.0-1.el8 723 kB/s | 43 kB 00:00 2023-10-07T09:26:34.076 INFO:teuthology.orchestra.run.smithi144.stdout:(89/139): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.2 MB/s | 74 kB 00:00 2023-10-07T09:26:34.102 INFO:teuthology.orchestra.run.smithi144.stdout:(90/139): zlib-devel-1.2.11-21.el8.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-10-07T09:26:34.385 INFO:teuthology.orchestra.run.smithi144.stdout:(91/139): python3-scipy-1.0.0-21.module_el8.5.0 6.2 MB/s | 14 MB 00:02 2023-10-07T09:26:34.460 INFO:teuthology.orchestra.run.smithi144.stdout:(92/139): liboath-2.6.2-3.el8.x86_64.rpm 164 kB/s | 59 kB 00:00 2023-10-07T09:26:34.471 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-10-07T09:26:34.474 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-10-07T09:26:34.660 INFO:teuthology.orchestra.run.smithi144.stdout:(93/139): gperftools-libs-2.7-9.el8.x86_64.rpm 524 kB/s | 306 kB 00:00 2023-10-07T09:26:34.777 INFO:teuthology.orchestra.run.smithi144.stdout:(94/139): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 374 kB/s | 44 kB 00:00 2023-10-07T09:26:34.802 INFO:teuthology.orchestra.run.smithi144.stdout:(95/139): libunwind-1.3.1-3.el8.x86_64.rpm 180 kB/s | 75 kB 00:00 2023-10-07T09:26:34.836 INFO:teuthology.orchestra.run.smithi144.stdout:(96/139): python3-asyncssh-2.7.0-2.el8.noarch.r 1.2 MB/s | 476 kB 00:00 2023-10-07T09:26:34.928 INFO:teuthology.orchestra.run.smithi144.stdout:(97/139): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-10-07T09:26:34.953 INFO:teuthology.orchestra.run.smithi144.stdout:(98/139): python3-cachetools-3.1.1-4.el8.noarch 219 kB/s | 33 kB 00:00 2023-10-07T09:26:34.970 INFO:teuthology.orchestra.run.smithi144.stdout:(99/139): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2023-10-07T09:26:35.062 INFO:teuthology.orchestra.run.smithi144.stdout:(100/139): python3-cheroot-8.5.2-1.el8.noarch.r 1.3 MB/s | 173 kB 00:00 2023-10-07T09:26:35.122 INFO:teuthology.orchestra.run.smithi144.stdout:(101/139): python3-cherrypy-18.4.0-1.el8.noarch 2.5 MB/s | 384 kB 00:00 2023-10-07T09:26:35.146 INFO:teuthology.orchestra.run.smithi144.stdout:(102/139): python3-google-auth-1.1.1-10.el8.noa 464 kB/s | 81 kB 00:00 2023-10-07T09:26:35.163 INFO:teuthology.orchestra.run.smithi144.stdout:(103/139): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2023-10-07T09:26:35.213 INFO:teuthology.orchestra.run.smithi144.stdout:(104/139): python3-jaraco-functools-2.0-4.el8.n 191 kB/s | 18 kB 00:00 2023-10-07T09:26:35.296 INFO:teuthology.orchestra.run.smithi144.stdout:(105/139): python3-logutils-0.3.5-11.el8.noarch 365 kB/s | 49 kB 00:00 2023-10-07T09:26:35.322 INFO:teuthology.orchestra.run.smithi144.stdout:(106/139): python3-more-itertools-7.2.0-3.el8.n 541 kB/s | 59 kB 00:00 2023-10-07T09:26:35.430 INFO:teuthology.orchestra.run.smithi144.stdout:(107/139): python3-kubernetes-11.0.0-6.el8.noar 4.1 MB/s | 1.2 MB 00:00 2023-10-07T09:26:35.448 INFO:teuthology.orchestra.run.smithi144.stdout:(108/139): python3-natsort-7.1.1-2.el8.noarch.r 400 kB/s | 60 kB 00:00 2023-10-07T09:26:35.481 INFO:teuthology.orchestra.run.smithi144.stdout:(109/139): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-10-07T09:26:35.523 INFO:teuthology.orchestra.run.smithi144.stdout:(110/139): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-10-07T09:26:35.565 INFO:teuthology.orchestra.run.smithi144.stdout:(111/139): python3-repoze-lru-0.7-6.el8.noarch. 288 kB/s | 34 kB 00:00 2023-10-07T09:26:35.607 INFO:teuthology.orchestra.run.smithi144.stdout:(112/139): python3-routes-2.4.1-12.el8.noarch.r 1.5 MB/s | 196 kB 00:00 2023-10-07T09:26:35.653 INFO:teuthology.orchestra.run.smithi144.stdout:(113/139): python3-rsa-4.9-2.el8.noarch.rpm 495 kB/s | 64 kB 00:00 2023-10-07T09:26:35.674 INFO:teuthology.orchestra.run.smithi144.stdout:(114/139): python3-simplegeneric-0.8.1-17.el8.n 172 kB/s | 19 kB 00:00 2023-10-07T09:26:35.734 INFO:teuthology.orchestra.run.smithi144.stdout:(115/139): python3-singledispatch-3.4.0.3-18.el 191 kB/s | 24 kB 00:00 2023-10-07T09:26:35.750 INFO:teuthology.orchestra.run.smithi144.stdout:(116/139): python3-tempora-1.14.1-5.el8.noarch. 303 kB/s | 29 kB 00:00 2023-10-07T09:26:35.792 INFO:teuthology.orchestra.run.smithi144.stdout:(117/139): python3-trustme-0.6.0-4.el8.noarch.r 234 kB/s | 27 kB 00:00 2023-10-07T09:26:35.859 INFO:teuthology.orchestra.run.smithi144.stdout:(118/139): python3-webob-1.8.5-1.el8.1.noarch.r 2.3 MB/s | 251 kB 00:00 2023-10-07T09:26:35.885 INFO:teuthology.orchestra.run.smithi144.stdout:(119/139): python3-waitress-1.4.3-1.el8.noarch. 1.7 MB/s | 255 kB 00:00 2023-10-07T09:26:35.935 INFO:teuthology.orchestra.run.smithi144.stdout:(120/139): python3-websocket-client-0.56.0-5.el 430 kB/s | 61 kB 00:00 2023-10-07T09:26:35.960 INFO:teuthology.orchestra.run.smithi144.stdout:(121/139): python3-webtest-2.0.33-1.el8.noarch. 850 kB/s | 85 kB 00:00 2023-10-07T09:26:35.985 INFO:teuthology.orchestra.run.smithi144.stdout:(122/139): python3-zc-lockfile-2.0-2.el8.noarch 228 kB/s | 23 kB 00:00 2023-10-07T09:26:36.419 INFO:teuthology.orchestra.run.smithi144.stdout:(123/139): thrift-0.13.0-2.el8.x86_64.rpm 3.6 MB/s | 1.7 MB 00:00 2023-10-07T09:26:36.494 INFO:teuthology.orchestra.run.smithi144.stdout:(124/139): NetworkManager-initscripts-updown-1. 1.9 MB/s | 143 kB 00:00 2023-10-07T09:26:36.555 INFO:teuthology.orchestra.run.smithi144.stdout:(125/139): elfutils-debuginfod-client-0.189-3.e 1.2 MB/s | 76 kB 00:00 2023-10-07T09:26:36.645 INFO:teuthology.orchestra.run.smithi144.stdout:(126/139): librados2-17.2.6-1320.g08cac018.el8. 5.1 MB/s | 3.5 MB 00:00 2023-10-07T09:26:36.670 INFO:teuthology.orchestra.run.smithi144.stdout:(127/139): elfutils-libelf-0.189-3.el8.x86_64.r 2.0 MB/s | 232 kB 00:00 2023-10-07T09:26:36.779 INFO:teuthology.orchestra.run.smithi144.stdout:(128/139): iproute-6.2.0-5.el8.x86_64.rpm 7.7 MB/s | 854 kB 00:00 2023-10-07T09:26:36.812 INFO:teuthology.orchestra.run.smithi144.stdout:(129/139): elfutils-libs-0.189-3.el8.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2023-10-07T09:26:36.854 INFO:teuthology.orchestra.run.smithi144.stdout:(130/139): libblkid-2.32.1-43.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2023-10-07T09:26:36.896 INFO:teuthology.orchestra.run.smithi144.stdout:(131/139): libfdisk-2.32.1-43.el8.x86_64.rpm 3.0 MB/s | 254 kB 00:00 2023-10-07T09:26:36.955 INFO:teuthology.orchestra.run.smithi144.stdout:(132/139): librbd1-17.2.6-1320.g08cac018.el8.x8 3.7 MB/s | 3.6 MB 00:00 2023-10-07T09:26:36.980 INFO:teuthology.orchestra.run.smithi144.stdout:(133/139): libmount-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 236 kB 00:00 2023-10-07T09:26:37.005 INFO:teuthology.orchestra.run.smithi144.stdout:(134/139): libsmartcols-2.32.1-43.el8.x86_64.rp 1.6 MB/s | 179 kB 00:00 2023-10-07T09:26:37.047 INFO:teuthology.orchestra.run.smithi144.stdout:(135/139): util-linux-user-2.32.1-43.el8.x86_64 2.4 MB/s | 103 kB 00:00 2023-10-07T09:26:37.073 INFO:teuthology.orchestra.run.smithi144.stdout:(136/139): libuuid-2.32.1-43.el8.x86_64.rpm 843 kB/s | 99 kB 00:00 2023-10-07T09:26:37.131 INFO:teuthology.orchestra.run.smithi144.stdout:(137/139): xfsdump-3.1.8-6.el8.x86_64.rpm 3.9 MB/s | 334 kB 00:00 2023-10-07T09:26:37.206 INFO:teuthology.orchestra.run.smithi144.stdout:(138/139): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.1 MB/s | 1.1 MB 00:00 2023-10-07T09:26:37.298 INFO:teuthology.orchestra.run.smithi144.stdout:(139/139): util-linux-2.32.1-43.el8.x86_64.rpm 7.8 MB/s | 2.5 MB 00:00 2023-10-07T09:26:37.301 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2023-10-07T09:26:37.302 INFO:teuthology.orchestra.run.smithi144.stdout:Total 13 MB/s | 210 MB 00:16 2023-10-07T09:26:38.992 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2023-10-07T09:26:39.403 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2023-10-07T09:26:39.403 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2023-10-07T09:26:39.909 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-10-07T09:26:39.971 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 1/1 2023-10-07T09:26:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 1/155 2023-10-07T09:26:40.237 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 1/155 2023-10-07T09:26:40.409 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 2/155 2023-10-07T09:26:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 2/155 2023-10-07T09:26:40.890 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2023-10-07T09:26:40.893 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2023-10-07T09:26:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/155 2023-10-07T09:26:41.524 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 4/155 2023-10-07T09:26:41.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.13.0-2.el8.x86_64 5/155 2023-10-07T09:26:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/155 2023-10-07T09:26:42.267 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/155 2023-10-07T09:26:42.728 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/155 2023-10-07T09:26:43.017 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:17.2.6-1320.g08cac018.el8.x86_64 8/155 2023-10-07T09:26:43.496 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:17.2.6-1320.g08cac018.el8.x86_64 8/155 2023-10-07T09:26:43.692 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 9/155 2023-10-07T09:26:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 9/155 2023-10-07T09:26:44.084 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 10/155 2023-10-07T09:26:44.200 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 10/155 2023-10-07T09:26:44.446 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:17.2.6-1320.g08cac018.el8.x86_64 11/155 2023-10-07T09:26:44.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 12/155 2023-10-07T09:26:45.304 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 12/155 2023-10-07T09:26:45.501 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 13/155 2023-10-07T09:26:45.752 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 13/155 2023-10-07T09:26:46.000 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/155 2023-10-07T09:26:46.153 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/155 2023-10-07T09:26:46.275 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2023-10-07T09:26:46.334 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 1/1 2023-10-07T09:26:46.435 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 15/155 2023-10-07T09:26:46.529 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 1/155 2023-10-07T09:26:46.610 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 1/155 2023-10-07T09:26:46.645 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/155 2023-10-07T09:26:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 17/155 2023-10-07T09:26:46.930 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 17/155 2023-10-07T09:26:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 2/155 2023-10-07T09:26:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 2/155 2023-10-07T09:26:47.092 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86 18/155 2023-10-07T09:26:47.283 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/155 2023-10-07T09:26:47.286 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86 18/155 2023-10-07T09:26:47.518 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/155 2023-10-07T09:26:47.704 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 4/155 2023-10-07T09:26:47.724 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 20/155 2023-10-07T09:26:47.884 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.13.0-2.el8.x86_64 5/155 2023-10-07T09:26:47.900 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/155 2023-10-07T09:26:48.059 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/155 2023-10-07T09:26:48.099 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/155 2023-10-07T09:26:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/155 2023-10-07T09:26:48.321 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/155 2023-10-07T09:26:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/155 2023-10-07T09:26:48.946 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:17.2.6-1320.g08cac018.el8.x86_64 8/155 2023-10-07T09:26:49.016 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/155 2023-10-07T09:26:49.242 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/155 2023-10-07T09:26:49.389 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:17.2.6-1320.g08cac018.el8.x86_64 8/155 2023-10-07T09:26:49.562 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 9/155 2023-10-07T09:26:49.711 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 9/155 2023-10-07T09:26:49.879 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 10/155 2023-10-07T09:26:49.993 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 10/155 2023-10-07T09:26:50.034 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/155 2023-10-07T09:26:50.225 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/155 2023-10-07T09:26:50.231 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:17.2.6-1320.g08cac018.el8.x86_64 11/155 2023-10-07T09:26:50.370 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 12/155 2023-10-07T09:26:50.989 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 12/155 2023-10-07T09:26:51.129 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/155 2023-10-07T09:26:51.146 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 13/155 2023-10-07T09:26:51.384 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 13/155 2023-10-07T09:26:51.508 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/155 2023-10-07T09:26:51.614 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/155 2023-10-07T09:26:51.708 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/155 2023-10-07T09:26:51.752 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/155 2023-10-07T09:26:51.944 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/155 2023-10-07T09:26:51.971 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 15/155 2023-10-07T09:26:52.146 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/155 2023-10-07T09:26:52.198 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/155 2023-10-07T09:26:52.262 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 17/155 2023-10-07T09:26:52.388 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 17/155 2023-10-07T09:26:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:17.2.6-1320.g08cac018.el8. 29/155 2023-10-07T09:26:52.528 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86 18/155 2023-10-07T09:26:52.638 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:17.2.6-1320.g08cac018.el 30/155 2023-10-07T09:26:52.675 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86 18/155 2023-10-07T09:26:52.925 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/155 2023-10-07T09:26:52.941 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:17.2.6-1320.g08cac018.el8.x86_6 31/155 2023-10-07T09:26:53.074 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 20/155 2023-10-07T09:26:53.203 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/155 2023-10-07T09:26:53.356 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/155 2023-10-07T09:26:53.495 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/155 2023-10-07T09:26:54.112 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/155 2023-10-07T09:26:54.174 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/155 2023-10-07T09:26:54.370 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/155 2023-10-07T09:26:54.497 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/155 2023-10-07T09:26:54.824 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/155 2023-10-07T09:26:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/155 2023-10-07T09:26:55.060 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/155 2023-10-07T09:26:55.245 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/155 2023-10-07T09:26:56.081 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/155 2023-10-07T09:26:56.440 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/155 2023-10-07T09:26:56.637 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/155 2023-10-07T09:26:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/155 2023-10-07T09:26:57.031 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/155 2023-10-07T09:26:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:17.2.6-1320.g08cac018.el8. 29/155 2023-10-07T09:26:57.308 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/155 2023-10-07T09:26:57.431 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:17.2.6-1320.g08cac018.el 30/155 2023-10-07T09:26:57.648 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/155 2023-10-07T09:26:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:17.2.6-1320.g08cac018.el8.x86_6 31/155 2023-10-07T09:26:57.834 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/155 2023-10-07T09:26:58.049 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el8.noarch 38/155 2023-10-07T09:26:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:17.2.6-1320.g08cac018.el8.x86_64 39/155 2023-10-07T09:26:58.582 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:17.2.6-1320.g08cac018.el8.x86_64 40/155 2023-10-07T09:26:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/155 2023-10-07T09:26:58.849 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:17.2.6-1320.g08cac018.el8.x86_6 41/155 2023-10-07T09:26:59.038 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 42/155 2023-10-07T09:26:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/155 2023-10-07T09:26:59.282 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-debuginfod-client-0.189-3.el8.x86_64 43/155 2023-10-07T09:26:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/155 2023-10-07T09:26:59.553 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/155 2023-10-07T09:26:59.590 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/155 2023-10-07T09:26:59.691 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/155 2023-10-07T09:26:59.906 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/155 2023-10-07T09:26:59.997 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/155 2023-10-07T09:27:00.181 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 46/155 2023-10-07T09:27:00.432 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/155 2023-10-07T09:27:01.120 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/155 2023-10-07T09:27:01.408 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/155 2023-10-07T09:27:01.438 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/155 2023-10-07T09:27:01.641 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/155 2023-10-07T09:27:01.709 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 49/155 2023-10-07T09:27:01.974 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/155 2023-10-07T09:27:02.023 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 50/155 2023-10-07T09:27:02.144 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/155 2023-10-07T09:27:02.249 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 51/155 2023-10-07T09:27:02.297 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.9-2.el8.noarch 38/155 2023-10-07T09:27:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/155 2023-10-07T09:27:02.484 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:17.2.6-1320.g08cac018.el8.x86_64 39/155 2023-10-07T09:27:02.664 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/155 2023-10-07T09:27:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:17.2.6-1320.g08cac018.el8.x86_64 40/155 2023-10-07T09:27:02.879 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/155 2023-10-07T09:27:02.885 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:17.2.6-1320.g08cac018.el8.x86_6 41/155 2023-10-07T09:27:03.054 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 42/155 2023-10-07T09:27:03.112 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/155 2023-10-07T09:27:03.218 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : elfutils-debuginfod-client-0.189-3.el8.x86_64 43/155 2023-10-07T09:27:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 56/155 2023-10-07T09:27:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/155 2023-10-07T09:27:03.559 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/155 2023-10-07T09:27:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 57/155 2023-10-07T09:27:03.737 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/155 2023-10-07T09:27:03.782 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 58/155 2023-10-07T09:27:03.809 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/155 2023-10-07T09:27:03.965 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 46/155 2023-10-07T09:27:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 59/155 2023-10-07T09:27:04.135 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/155 2023-10-07T09:27:04.215 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/155 2023-10-07T09:27:04.410 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/155 2023-10-07T09:27:04.644 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 62/155 2023-10-07T09:27:04.755 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/155 2023-10-07T09:27:04.931 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 63/155 2023-10-07T09:27:05.011 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/155 2023-10-07T09:27:05.040 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/155 2023-10-07T09:27:05.211 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 64/155 2023-10-07T09:27:05.218 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 49/155 2023-10-07T09:27:05.444 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 65/155 2023-10-07T09:27:05.511 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 50/155 2023-10-07T09:27:05.709 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 51/155 2023-10-07T09:27:05.818 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 66/155 2023-10-07T09:27:05.856 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/155 2023-10-07T09:27:06.005 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/155 2023-10-07T09:27:06.117 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 67/155 2023-10-07T09:27:06.182 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/155 2023-10-07T09:27:06.354 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 68/155 2023-10-07T09:27:06.384 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/155 2023-10-07T09:27:06.574 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 56/155 2023-10-07T09:27:06.620 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 69/155 2023-10-07T09:27:06.746 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 57/155 2023-10-07T09:27:06.870 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 70/155 2023-10-07T09:27:06.891 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 58/155 2023-10-07T09:27:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 59/155 2023-10-07T09:27:07.250 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/155 2023-10-07T09:27:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 71/155 2023-10-07T09:27:07.386 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/155 2023-10-07T09:27:07.583 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 62/155 2023-10-07T09:27:07.632 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 72/155 2023-10-07T09:27:07.768 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 63/155 2023-10-07T09:27:07.841 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 73/155 2023-10-07T09:27:08.031 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 64/155 2023-10-07T09:27:08.184 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 74/155 2023-10-07T09:27:08.207 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 65/155 2023-10-07T09:27:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 75/155 2023-10-07T09:27:08.528 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 66/155 2023-10-07T09:27:08.778 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 67/155 2023-10-07T09:27:09.006 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 68/155 2023-10-07T09:27:09.212 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 69/155 2023-10-07T09:27:09.437 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 70/155 2023-10-07T09:27:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/155 2023-10-07T09:27:09.837 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 71/155 2023-10-07T09:27:10.067 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 72/155 2023-10-07T09:27:10.212 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 77/155 2023-10-07T09:27:10.224 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 73/155 2023-10-07T09:27:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 78/155 2023-10-07T09:27:10.463 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 74/155 2023-10-07T09:27:10.696 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 79/155 2023-10-07T09:27:10.728 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 75/155 2023-10-07T09:27:10.996 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/155 2023-10-07T09:27:11.268 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/155 2023-10-07T09:27:11.567 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 82/155 2023-10-07T09:27:11.813 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 83/155 2023-10-07T09:27:12.002 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/155 2023-10-07T09:27:12.181 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 84/155 2023-10-07T09:27:12.363 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 77/155 2023-10-07T09:27:12.479 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 85/155 2023-10-07T09:27:12.602 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 78/155 2023-10-07T09:27:12.650 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 86/155 2023-10-07T09:27:12.813 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 79/155 2023-10-07T09:27:12.863 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/155 2023-10-07T09:27:13.085 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/155 2023-10-07T09:27:13.238 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/155 2023-10-07T09:27:13.279 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/155 2023-10-07T09:27:13.536 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 89/155 2023-10-07T09:27:13.548 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 82/155 2023-10-07T09:27:13.706 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/155 2023-10-07T09:27:13.738 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 83/155 2023-10-07T09:27:13.956 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/155 2023-10-07T09:27:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 84/155 2023-10-07T09:27:14.177 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/155 2023-10-07T09:27:14.314 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 85/155 2023-10-07T09:27:14.419 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.7-17.el8.x86_64 93/155 2023-10-07T09:27:14.453 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 86/155 2023-10-07T09:27:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/155 2023-10-07T09:27:14.757 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/155 2023-10-07T09:27:14.959 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 95/155 2023-10-07T09:27:14.961 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/155 2023-10-07T09:27:15.114 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 96/155 2023-10-07T09:27:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 89/155 2023-10-07T09:27:15.349 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/155 2023-10-07T09:27:15.543 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/155 2023-10-07T09:27:15.761 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/155 2023-10-07T09:27:15.970 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.7-17.el8.x86_64 93/155 2023-10-07T09:27:16.252 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/155 2023-10-07T09:27:16.435 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 95/155 2023-10-07T09:27:16.566 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 96/155 2023-10-07T09:27:18.341 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 97/155 2023-10-07T09:27:18.597 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 97/155 2023-10-07T09:27:19.291 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 97/155 2023-10-07T09:27:19.540 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 97/155 2023-10-07T09:27:19.585 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 98/155 2023-10-07T09:27:19.798 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 98/155 2023-10-07T09:27:19.816 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 97/155 2023-10-07T09:27:20.080 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 99/155 2023-10-07T09:27:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 97/155 2023-10-07T09:27:20.785 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 98/155 2023-10-07T09:27:20.992 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 98/155 2023-10-07T09:27:21.182 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 99/155 2023-10-07T09:27:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 99/155 2023-10-07T09:27:41.575 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2023-10-07T09:27:41.575 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2023-10-07T09:27:41.575 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /dev 2023-10-07T09:27:41.575 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /run 2023-10-07T09:27:41.576 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2023-10-07T09:27:41.576 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2023-10-07T09:27:41.576 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2023-10-07T09:27:41.576 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2023-10-07T09:27:41.576 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /dev 2023-10-07T09:27:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:27:41.962 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 100/155 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 99/155 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /run 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-10-07T09:27:42.653 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2023-10-07T09:27:42.654 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:27:43.077 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 100/155 2023-10-07T09:27:43.580 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 100/155 2023-10-07T09:27:43.823 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 101/155 2023-10-07T09:27:44.093 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 101/155 2023-10-07T09:27:44.262 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 102/155 2023-10-07T09:27:44.713 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 102/155 2023-10-07T09:27:44.798 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 100/155 2023-10-07T09:27:44.953 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 103/155 2023-10-07T09:27:45.003 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 103/155 2023-10-07T09:27:45.058 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 101/155 2023-10-07T09:27:45.203 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 104/155 2023-10-07T09:27:45.342 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 101/155 2023-10-07T09:27:45.382 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 105/155 2023-10-07T09:27:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 106/155 2023-10-07T09:27:45.594 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 102/155 2023-10-07T09:27:45.621 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 106/155 2023-10-07T09:27:45.965 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : jq-1.6-7.el8.x86_64 107/155 2023-10-07T09:27:46.075 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 102/155 2023-10-07T09:27:46.211 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : automake-1.16.1-8.el8.noarch 108/155 2023-10-07T09:27:46.269 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 103/155 2023-10-07T09:27:46.320 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 103/155 2023-10-07T09:27:46.595 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 104/155 2023-10-07T09:27:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 105/155 2023-10-07T09:27:46.839 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:17.2.6-1320.g08cac018.el8.noarch 109/155 2023-10-07T09:27:46.968 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 106/155 2023-10-07T09:27:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:17.2.6-1320.g08cac018.el8.noarch 109/155 2023-10-07T09:27:47.067 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 106/155 2023-10-07T09:27:47.135 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1320.g08cac018.e 110/155 2023-10-07T09:27:47.337 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1320.g08cac018. 111/155 2023-10-07T09:27:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-7.el8.x86_64 107/155 2023-10-07T09:27:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noa 112/155 2023-10-07T09:27:47.702 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.1-8.el8.noarch 108/155 2023-10-07T09:27:47.804 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noa 112/155 2023-10-07T09:27:48.047 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.n 113/155 2023-10-07T09:27:48.381 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:17.2.6-1320.g08cac018.el8.noarch 109/155 2023-10-07T09:27:48.561 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:17.2.6-1320.g08cac018.el8.noarch 109/155 2023-10-07T09:27:48.693 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1320.g08cac018.e 110/155 2023-10-07T09:27:48.963 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1320.g08cac018. 111/155 2023-10-07T09:27:49.153 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.n 113/155 2023-10-07T09:27:49.201 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noa 112/155 2023-10-07T09:27:49.393 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1320.g08c 114/155 2023-10-07T09:27:49.472 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noa 112/155 2023-10-07T09:27:49.525 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1320.g08c 114/155 2023-10-07T09:27:49.707 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.n 113/155 2023-10-07T09:27:49.906 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1320.g08cac018.el 115/155 2023-10-07T09:27:50.122 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 116/155 2023-10-07T09:27:50.188 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 116/155 2023-10-07T09:27:50.368 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 117/155 2023-10-07T09:27:50.396 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 117/155 2023-10-07T09:27:50.592 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:17.2.6-1320.g08cac018.el8.x86_64 118/155 2023-10-07T09:27:50.743 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/155 2023-10-07T09:27:50.887 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.n 113/155 2023-10-07T09:27:51.109 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1320.g08c 114/155 2023-10-07T09:27:51.245 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1320.g08c 114/155 2023-10-07T09:27:51.667 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1320.g08cac018.el 115/155 2023-10-07T09:27:51.882 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 116/155 2023-10-07T09:27:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 116/155 2023-10-07T09:27:52.159 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 117/155 2023-10-07T09:27:52.190 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 117/155 2023-10-07T09:27:52.394 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:17.2.6-1320.g08cac018.el8.x86_64 118/155 2023-10-07T09:27:52.576 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/155 2023-10-07T09:27:56.083 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/155 2023-10-07T09:27:57.920 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:17.2.6-1320.g08cac018.el8.x86_64 120/155 2023-10-07T09:27:58.154 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 121/155 2023-10-07T09:27:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 121/155 2023-10-07T09:27:58.279 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/155 2023-10-07T09:27:58.484 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1320.g08cac 122/155 2023-10-07T09:27:58.554 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1320.g08cac 122/155 2023-10-07T09:27:58.734 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 123/155 2023-10-07T09:27:59.139 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 123/155 2023-10-07T09:27:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 124/155 2023-10-07T09:27:59.470 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 124/155 2023-10-07T09:28:00.323 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:17.2.6-1320.g08cac018.el8.x86_64 120/155 2023-10-07T09:28:00.564 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 121/155 2023-10-07T09:28:00.669 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 125/155 2023-10-07T09:28:00.690 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 121/155 2023-10-07T09:28:00.900 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1320.g08cac 122/155 2023-10-07T09:28:00.973 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1320.g08cac 122/155 2023-10-07T09:28:00.989 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openssl-devel-1:1.1.1k-9.el8.x86_64 126/155 2023-10-07T09:28:01.175 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 123/155 2023-10-07T09:28:01.208 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 127/155 2023-10-07T09:28:01.422 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 128/155 2023-10-07T09:28:01.587 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 129/155 2023-10-07T09:28:01.607 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 123/155 2023-10-07T09:28:01.799 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:17.2.6-1320.g08cac018.el8.x86_ 130/155 2023-10-07T09:28:01.859 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 124/155 2023-10-07T09:28:01.992 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 124/155 2023-10-07T09:28:02.022 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : xfsdump-3.1.8-6.el8.x86_64 131/155 2023-10-07T09:28:02.227 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-top-2:17.2.6-1320.g08cac018.el8.noarch 132/155 2023-10-07T09:28:02.468 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 133/155 2023-10-07T09:28:02.830 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:17.2.6-1320.g08cac018.el8.x86_64 134/155 2023-10-07T09:28:03.129 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 135/155 2023-10-07T09:28:03.316 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-9.el 136/155 2023-10-07T09:28:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 125/155 2023-10-07T09:28:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-9.el 136/155 2023-10-07T09:28:03.543 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 137/155 2023-10-07T09:28:03.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:1.1.1k-9.el8.x86_64 126/155 2023-10-07T09:28:03.774 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flex-2.6.1-9.el8.x86_64 138/155 2023-10-07T09:28:03.884 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 138/155 2023-10-07T09:28:03.912 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 127/155 2023-10-07T09:28:04.116 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : bison-3.0.4-10.el8.x86_64 139/155 2023-10-07T09:28:04.143 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 139/155 2023-10-07T09:28:04.174 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 128/155 2023-10-07T09:28:04.302 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 140/155 2023-10-07T09:28:04.363 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 129/155 2023-10-07T09:28:04.454 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 141/155 2023-10-07T09:28:04.590 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 142/155 2023-10-07T09:28:04.590 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 143/155 2023-10-07T09:28:04.648 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:17.2.6-1320.g08cac018.el8.x86_ 130/155 2023-10-07T09:28:04.853 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 143/155 2023-10-07T09:28:04.854 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 144/155 2023-10-07T09:28:04.889 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsdump-3.1.8-6.el8.x86_64 131/155 2023-10-07T09:28:05.060 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 144/155 2023-10-07T09:28:05.077 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 145/155 2023-10-07T09:28:05.156 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:17.2.6-1320.g08cac018.el8.noarch 132/155 2023-10-07T09:28:05.189 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 145/155 2023-10-07T09:28:05.189 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 146/155 2023-10-07T09:28:05.423 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 146/155 2023-10-07T09:28:05.454 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 133/155 2023-10-07T09:28:05.626 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 147/155 2023-10-07T09:28:05.776 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 148/155 2023-10-07T09:28:05.856 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:17.2.6-1320.g08cac018.el8.x86_64 134/155 2023-10-07T09:28:06.001 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 149/155 2023-10-07T09:28:06.001 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 150/155 2023-10-07T09:28:06.169 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 135/155 2023-10-07T09:28:06.219 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 150/155 2023-10-07T09:28:06.219 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 151/155 2023-10-07T09:28:06.349 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-9.el 136/155 2023-10-07T09:28:06.380 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-9.el 136/155 2023-10-07T09:28:06.402 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 151/155 2023-10-07T09:28:06.403 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 152/155 2023-10-07T09:28:06.610 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 137/155 2023-10-07T09:28:06.635 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 152/155 2023-10-07T09:28:06.636 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 153/155 2023-10-07T09:28:06.840 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.1-9.el8.x86_64 138/155 2023-10-07T09:28:06.854 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 153/155 2023-10-07T09:28:06.855 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 154/155 2023-10-07T09:28:06.961 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 138/155 2023-10-07T09:28:07.044 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 154/155 2023-10-07T09:28:07.217 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 155/155 2023-10-07T09:28:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.0.4-10.el8.x86_64 139/155 2023-10-07T09:28:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 139/155 2023-10-07T09:28:07.444 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: elfutils-libelf-0.189-1.el8.x86_64 155/155 2023-10-07T09:28:07.444 INFO:teuthology.orchestra.run.smithi144.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-07T09:28:07.444 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:07.535 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 140/155 2023-10-07T09:28:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 141/155 2023-10-07T09:28:07.905 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 142/155 2023-10-07T09:28:07.906 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 143/155 2023-10-07T09:28:08.161 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:17.2.6-1320.g08cac018.el8.x86_64 1/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 2/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 3/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 4/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1320.g08cac 5/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 6/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 7/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 8/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 9/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 10/155 2023-10-07T09:28:08.162 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 11/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:17.2.6-1320.g08cac018.el8.x86_64 12/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 13/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:17.2.6-1320.g08cac018.el8.x86_ 14/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 15/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 16/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:17.2.6-1320.g08cac018.el8.x86_6 17/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86 18/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 19/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1320.g08cac018.el 20/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:17.2.6-1320.g08cac018.el8. 21/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:17.2.6-1320.g08cac018.el8.x86_6 22/155 2023-10-07T09:28:08.163 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:17.2.6-1320.g08cac018.el8.x86_64 23/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:17.2.6-1320.g08cac018.el8.x86_64 24/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:17.2.6-1320.g08cac018.el8.x86_64 25/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 26/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 27/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:17.2.6-1320.g08cac018.el8.x86_64 28/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1320.g08cac018. 29/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noa 30/155 2023-10-07T09:28:08.164 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.n 31/155 2023-10-07T09:28:08.176 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1320.g08c 32/155 2023-10-07T09:28:08.176 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1320.g08cac018.el 33/155 2023-10-07T09:28:08.176 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 34/155 2023-10-07T09:28:08.176 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1320.g08cac018.e 35/155 2023-10-07T09:28:08.176 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 36/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:17.2.6-1320.g08cac018.el8.noarch 37/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-top-2:17.2.6-1320.g08cac018.el8.noarch 38/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : jq-1.6-7.el8.x86_64 42/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 43/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 44/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 45/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 46/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 47/155 2023-10-07T09:28:08.177 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 48/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 49/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 50/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 51/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 52/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 53/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 54/155 2023-10-07T09:28:08.178 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 55/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 56/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 57/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 58/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 59/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 60/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 61/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 62/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 63/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 64/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 65/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 66/155 2023-10-07T09:28:08.179 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 67/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 68/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 69/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 70/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 71/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 72/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 73/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 74/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 75/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 76/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 77/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 78/155 2023-10-07T09:28:08.180 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-devel-1:1.1.1k-9.el8.x86_64 80/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 81/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 82/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 83/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 84/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 85/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 86/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 87/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 89/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 90/155 2023-10-07T09:28:08.181 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 91/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 92/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 93/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 94/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 95/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 96/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 98/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 99/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 100/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 101/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 102/155 2023-10-07T09:28:08.182 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 103/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 104/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 105/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 106/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 107/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 108/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 109/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-2.6-1.el8.noarch 110/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 111/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 112/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 113/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 114/155 2023-10-07T09:28:08.183 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 115/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 116/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 117/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 118/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 119/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 120/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 121/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 122/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 123/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:17.2.6-1320.g08cac018.el8.x86_64 124/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 125/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 126/155 2023-10-07T09:28:08.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 127/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-9.el 128/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 129/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el8.x86_64 130/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 131/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 132/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 133/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 134/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 135/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 136/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 137/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 138/155 2023-10-07T09:28:08.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 139/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 140/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 141/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 142/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 143/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 144/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 145/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 146/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 147/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 148/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 149/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 150/155 2023-10-07T09:28:08.186 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 151/155 2023-10-07T09:28:08.187 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsdump-3.1.8-6.el8.x86_64 152/155 2023-10-07T09:28:08.187 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 153/155 2023-10-07T09:28:08.187 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/155 2023-10-07T09:28:08.187 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 143/155 2023-10-07T09:28:08.187 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 144/155 2023-10-07T09:28:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 144/155 2023-10-07T09:28:08.512 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 145/155 2023-10-07T09:28:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 145/155 2023-10-07T09:28:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 146/155 2023-10-07T09:28:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 146/155 2023-10-07T09:28:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 147/155 2023-10-07T09:28:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 148/155 2023-10-07T09:28:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 149/155 2023-10-07T09:28:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 150/155 2023-10-07T09:28:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 150/155 2023-10-07T09:28:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 151/155 2023-10-07T09:28:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 151/155 2023-10-07T09:28:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 152/155 2023-10-07T09:28:10.278 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 152/155 2023-10-07T09:28:10.278 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 153/155 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/155 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-initscripts-updown-1:1.40.16-9.el8.noarch 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-debuginfod-client-0.189-3.el8.x86_64 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-0.189-3.el8.x86_64 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libs-0.189-3.el8.x86_64 2023-10-07T09:28:10.282 INFO:teuthology.orchestra.run.smithi144.stdout: iproute-6.2.0-5.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: libblkid-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: libfdisk-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: libmount-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: libsmartcols-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: util-linux-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: util-linux-user-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: xfsdump-3.1.8-6.el8.x86_64 2023-10-07T09:28:10.283 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-5.0.0-12.el8.x86_64 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: automake-1.16.1-8.el8.noarch 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: bison-3.0.4-10.el8.x86_64 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.284 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.285 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: flex-2.6.1-9.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: fuse-2.9.7-17.el8.x86_64 2023-10-07T09:28:10.286 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: jq-1.6-7.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-10-07T09:28:10.287 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libselinux-devel-2.9-8.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2023-10-07T09:28:10.288 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel-1:1.1.1k-9.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-07T09:28:10.289 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-07T09:28:10.290 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-07T09:28:10.291 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-2.6-1.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-07T09:28:10.292 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-07T09:28:10.293 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2023-10-07T09:28:10.294 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-07T09:28:10.295 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-10-07T09:28:10.295 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:10.295 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-10-07T09:28:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 153/155 2023-10-07T09:28:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 154/155 2023-10-07T09:28:10.605 DEBUG:teuthology.parallel:result is None 2023-10-07T09:28:10.786 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 154/155 2023-10-07T09:28:10.993 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 155/155 2023-10-07T09:28:11.211 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: elfutils-libelf-0.189-1.el8.x86_64 155/155 2023-10-07T09:28:11.211 INFO:teuthology.orchestra.run.smithi029.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-07T09:28:11.211 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:17.2.6-1320.g08cac018.el8.x86_64 1/155 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 2/155 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 3/155 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 4/155 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1320.g08cac 5/155 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 6/155 2023-10-07T09:28:11.953 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 7/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 8/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 9/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 10/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 11/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:17.2.6-1320.g08cac018.el8.x86_64 12/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 13/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:17.2.6-1320.g08cac018.el8.x86_ 14/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 15/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 16/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:17.2.6-1320.g08cac018.el8.x86_6 17/155 2023-10-07T09:28:11.954 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86 18/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 19/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1320.g08cac018.el 20/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:17.2.6-1320.g08cac018.el8. 21/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:17.2.6-1320.g08cac018.el8.x86_6 22/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:17.2.6-1320.g08cac018.el8.x86_64 23/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:17.2.6-1320.g08cac018.el8.x86_64 24/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:17.2.6-1320.g08cac018.el8.x86_64 25/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 26/155 2023-10-07T09:28:11.955 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 27/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:17.2.6-1320.g08cac018.el8.x86_64 28/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1320.g08cac018. 29/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noa 30/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.n 31/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1320.g08c 32/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1320.g08cac018.el 33/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 34/155 2023-10-07T09:28:11.971 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1320.g08cac018.e 35/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 36/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:17.2.6-1320.g08cac018.el8.noarch 37/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:17.2.6-1320.g08cac018.el8.noarch 38/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-7.el8.x86_64 42/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 43/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 44/155 2023-10-07T09:28:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 45/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 46/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 47/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 48/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 49/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 50/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 51/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 52/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 53/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 54/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 55/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 56/155 2023-10-07T09:28:11.973 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 57/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 58/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 59/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 60/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 61/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 62/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 63/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 64/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 65/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 66/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 67/155 2023-10-07T09:28:11.974 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 68/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 69/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 70/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 71/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 72/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 73/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 74/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 75/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 76/155 2023-10-07T09:28:11.975 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 77/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 78/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:1.1.1k-9.el8.x86_64 80/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 81/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 82/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 83/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 84/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 85/155 2023-10-07T09:28:11.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 86/155 2023-10-07T09:28:11.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 87/155 2023-10-07T09:28:11.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 89/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 90/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 91/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 92/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 93/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 94/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 95/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 96/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 98/155 2023-10-07T09:28:11.978 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 99/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 100/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 101/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 102/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 103/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 104/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 105/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 106/155 2023-10-07T09:28:11.979 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 107/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 108/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 109/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-2.6-1.el8.noarch 110/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 111/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 112/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 113/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 114/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 115/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 116/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 117/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 118/155 2023-10-07T09:28:11.980 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 119/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 120/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 121/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 122/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 123/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:17.2.6-1320.g08cac018.el8.x86_64 124/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 125/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 126/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 127/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-9.el 128/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 129/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el8.x86_64 130/155 2023-10-07T09:28:11.981 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 131/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 132/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 133/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 134/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 135/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 136/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 137/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 138/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 139/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 140/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 141/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 142/155 2023-10-07T09:28:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 143/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 144/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 145/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 146/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 147/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 148/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 149/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 150/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 151/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.8-6.el8.x86_64 152/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 153/155 2023-10-07T09:28:11.983 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/155 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/155 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-initscripts-updown-1:1.40.16-9.el8.noarch 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client-0.189-3.el8.x86_64 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-0.189-3.el8.x86_64 2023-10-07T09:28:14.086 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs-0.189-3.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: iproute-6.2.0-5.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: libmount-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.087 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump-3.1.8-6.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-5.0.0-12.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.1-8.el8.noarch 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.0.4-10.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.088 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:17.2.6-1320.g08cac018.el8.noarch 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2023-10-07T09:28:14.090 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.1-9.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.7-17.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-7.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.091 INFO:teuthology.orchestra.run.smithi029.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: libselinux-devel-2.9-8.el8.x86_64 2023-10-07T09:28:14.092 INFO:teuthology.orchestra.run.smithi029.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-07T09:28:14.093 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:1.1.1k-9.el8.x86_64 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-07T09:28:14.094 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-07T09:28:14.095 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-07T09:28:14.096 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-2.6-1.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.097 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-07T09:28:14.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:17.2.6-1320.g08cac018.el8.x86_64 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2023-10-07T09:28:14.099 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-07T09:28:14.100 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-10-07T09:28:14.100 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:14.100 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-10-07T09:28:14.394 DEBUG:teuthology.parallel:result is None 2023-10-07T09:28:14.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-07T09:28:14.632 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-07T09:28:14.690 INFO:teuthology.orchestra.run.smithi029.stdout:17.2.6-1320.g08cac018.el8 2023-10-07T09:28:14.690 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1320.g08cac018.el8 2023-10-07T09:28:14.690 INFO:teuthology.task.install:The correct ceph version 17.2.6-1320.g08cac018 is installed. 2023-10-07T09:28:14.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-07T09:28:14.834 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-07T09:28:14.891 INFO:teuthology.orchestra.run.smithi144.stdout:17.2.6-1320.g08cac018.el8 2023-10-07T09:28:14.891 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1320.g08cac018.el8 2023-10-07T09:28:14.891 INFO:teuthology.task.install:The correct ceph version 17.2.6-1320.g08cac018 is installed. 2023-10-07T09:28:14.892 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-07T09:28:14.892 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:14.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-07T09:28:14.966 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:14.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-07T09:28:15.043 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-07T09:28:15.043 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:15.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-10-07T09:28:15.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-07T09:28:15.235 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:15.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2023-10-07T09:28:15.336 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-07T09:28:15.450 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-07T09:28:15.451 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:15.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-07T09:28:15.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-07T09:28:15.638 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:15.638 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-07T09:28:15.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-07T09:28:15.818 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-07T09:28:15.818 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:15.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-10-07T09:28:15.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-07T09:28:16.005 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:16.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2023-10-07T09:28:16.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-07T09:28:16.188 INFO:teuthology.run_tasks:Running task ceph... 2023-10-07T09:28:16.254 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-07T09:28:16.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2023-10-07T09:28:16.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2023-10-07T09:28:16.326 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-07T09:28:16.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-07T09:28:16.331 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-07T09:28:16.399 INFO:tasks.ceph:Creating extra log directories... 2023-10-07T09:28:16.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-07T09:28:16.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-07T09:28:16.480 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-07T09:28:16.480 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-07T09:28:16.480 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-07T09:28:16.480 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:16.480 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-07T09:28:16.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-07T09:28:16.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-07T09:28:16.861 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:16.862 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-07T09:28:16.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-07T09:28:17.137 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-07T09:28:17.216 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-07T09:28:17.216 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-07T09:28:17.216 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314', 'branch': 'wip-yuri-testing-2023-10-06-0949-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7416314', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-06-0949-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-06-0949-quincy', 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4726, 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri-testing-2023-10-06-0949-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_08cac0187919fc3a247efdf445f652e2fb30a647/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '08cac0187919fc3a247efdf445f652e2fb30a647', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJqOOCB35WNDUTDN3/gKRrTNq3QXQuxExXAesdmjvfolG4pVfUmnkky6PTTDSz8FqLkW8ZeCQoLktd/N/CGeKHE=', 'smithi144.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUGCvz9Yf0ykUhRBF9kRUjcEMHDC/y8DnTqecvi6cGh0BQfJacelvQWPLe7dcH3OM70m6zxsFLxJT+22gJWLQ0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '08cac0187919fc3a247efdf445f652e2fb30a647', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-07_00:40:05', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-07T09:28:17.217 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-07T09:28:17.226 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-07T09:28:17.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-10-07T09:28:17.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2023-10-07T09:28:17.381 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:17.381 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2023-10-07T09:28:17.427 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-07T09:28:17.428 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 448 Links: 1 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-07 09:28:11.438544169 +0000 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-07 09:24:11.477901291 +0000 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-07 09:24:11.477901291 +0000 2023-10-07T09:28:17.516 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-07 09:24:11.454901663 +0000 2023-10-07T09:28:17.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-07T09:28:17.586 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-07T09:28:17.586 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-07T09:28:17.586 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000173382 s, 3.0 MB/s 2023-10-07T09:28:17.588 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-07T09:28:17.638 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 454 Links: 1 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-07 09:28:11.438544169 +0000 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-07 09:24:12.325887574 +0000 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-07 09:24:12.325887574 +0000 2023-10-07T09:28:17.686 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-07 09:24:12.298888010 +0000 2023-10-07T09:28:17.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-07T09:28:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-07T09:28:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-07T09:28:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000179187 s, 2.9 MB/s 2023-10-07T09:28:17.758 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-07T09:28:17.808 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 460 Links: 1 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-07 09:28:11.438544169 +0000 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-07 09:24:13.182873711 +0000 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-07 09:24:13.182873711 +0000 2023-10-07T09:28:17.857 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-07 09:24:13.158874099 +0000 2023-10-07T09:28:17.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-07T09:28:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-07T09:28:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-07T09:28:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000227515 s, 2.3 MB/s 2023-10-07T09:28:17.929 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-07T09:28:17.980 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2023-10-07T09:28:18.027 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-07T09:28:18.027 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:18.027 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 466 Links: 1 2023-10-07T09:28:18.028 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:18.028 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:18.028 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-07 09:28:11.439544153 +0000 2023-10-07T09:28:18.028 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-07 09:24:14.016860220 +0000 2023-10-07T09:28:18.028 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-07 09:24:14.016860220 +0000 2023-10-07T09:28:18.028 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-07 09:24:13.994860576 +0000 2023-10-07T09:28:18.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-07T09:28:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-07T09:28:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-07T09:28:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000178247 s, 2.9 MB/s 2023-10-07T09:28:18.106 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-07T09:28:18.157 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:18.157 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:18.157 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2023-10-07T09:28:18.206 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-07T09:28:18.206 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 448 Links: 1 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-07 09:28:07.660471748 +0000 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-07 09:23:52.374636703 +0000 2023-10-07T09:28:18.253 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-07 09:23:52.374636703 +0000 2023-10-07T09:28:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-07 09:23:52.348637115 +0000 2023-10-07T09:28:18.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-07T09:28:18.359 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-07T09:28:18.359 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-07T09:28:18.359 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000442404 s, 1.2 MB/s 2023-10-07T09:28:18.361 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-07T09:28:18.409 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2023-10-07T09:28:18.455 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-07T09:28:18.455 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:18.455 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 454 Links: 1 2023-10-07T09:28:18.455 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:18.455 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:18.455 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-07 09:28:07.660471748 +0000 2023-10-07T09:28:18.456 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-07 09:23:53.234623083 +0000 2023-10-07T09:28:18.456 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-07 09:23:53.234623083 +0000 2023-10-07T09:28:18.456 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-07 09:23:53.207623510 +0000 2023-10-07T09:28:18.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-07T09:28:18.561 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-07T09:28:18.561 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-07T09:28:18.562 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00028973 s, 1.8 MB/s 2023-10-07T09:28:18.564 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-07T09:28:18.611 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2023-10-07T09:28:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-07T09:28:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:18.657 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 460 Links: 1 2023-10-07T09:28:18.657 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:18.657 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:18.658 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-07 09:28:07.660471748 +0000 2023-10-07T09:28:18.658 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-07 09:23:54.122609019 +0000 2023-10-07T09:28:18.658 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-07 09:23:54.122609019 +0000 2023-10-07T09:28:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-07 09:23:54.099609383 +0000 2023-10-07T09:28:18.658 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-07T09:28:18.763 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-07T09:28:18.763 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-07T09:28:18.763 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000266758 s, 1.9 MB/s 2023-10-07T09:28:18.765 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-07T09:28:18.812 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2023-10-07T09:28:18.858 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-07T09:28:18.858 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-07T09:28:18.858 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 466 Links: 1 2023-10-07T09:28:18.858 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-07T09:28:18.858 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-07T09:28:18.858 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-07 09:28:07.660471748 +0000 2023-10-07T09:28:18.859 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-07 09:23:54.994595209 +0000 2023-10-07T09:28:18.859 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-07 09:23:54.994595209 +0000 2023-10-07T09:28:18.859 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-07 09:23:54.970595589 +0000 2023-10-07T09:28:18.860 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-07T09:28:18.967 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-07T09:28:18.967 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-07T09:28:18.967 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000480306 s, 1.1 MB/s 2023-10-07T09:28:18.970 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-07T09:28:19.018 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:19.018 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-07T09:28:19.018 INFO:tasks.ceph:Generating config... 2023-10-07T09:28:19.019 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-07T09:28:19.019 INFO:tasks.ceph:[client] debug client = 20 2023-10-07T09:28:19.019 INFO:tasks.ceph:[client] debug ms = 1 2023-10-07T09:28:19.020 INFO:tasks.ceph:[client] pid file = 2023-10-07T09:28:19.020 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-07T09:28:19.020 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-07T09:28:19.020 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-07T09:28:19.020 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-07T09:28:19.020 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-07T09:28:19.020 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-07T09:28:19.021 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-07T09:28:19.022 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-07T09:28:19.022 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-07T09:28:19.022 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-07T09:28:19.022 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-07T09:28:19.022 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-07T09:28:19.022 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-07T09:28:19.022 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-07T09:28:19.022 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-07T09:28:19.022 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-07T09:28:19.023 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-07T09:28:19.024 INFO:tasks.ceph:Setting up mon.a... 2023-10-07T09:28:19.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-07T09:28:19.125 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2023-10-07T09:28:19.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-07T09:28:19.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-07T09:28:19.304 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29'), ('mon.b', '172.21.15.144'), ('mon.c', '[v2:172.21.15.144:3301,v1:172.21.15.144:6790]')] 2023-10-07T09:28:19.304 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29,172.21.15.144,[v2:172.21.15.144:3301,v1:172.21.15.144:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-07T09:28:19.305 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:19.305 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-07T09:28:19.352 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --add b 172.21.15.144 --addv c '[v2:172.21.15.144:3301,v1:172.21.15.144:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:19.476 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 1e617e23-0c6c-40cc-8e09-10eaf7e38e24 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = octopus 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 1e617e23-0c6c-40cc-8e09-10eaf7e38e24 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2023-10-07T09:28:19.476928+0000 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:created 2023-10-07T09:28:19.476928+0000 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 15 (octopus) 2023-10-07T09:28:19.477 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2023-10-07T09:28:19.478 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-10-07T09:28:19.478 INFO:teuthology.orchestra.run.smithi029.stdout:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2023-10-07T09:28:19.478 INFO:teuthology.orchestra.run.smithi029.stdout:2: [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] mon.c 2023-10-07T09:28:19.478 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-07T09:28:19.479 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-07T09:28:19.526 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1e617e23-0c6c-40cc-8e09-10eaf7e38e24... 2023-10-07T09:28:19.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-07T09:28:19.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-07T09:28:19.650 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2023-10-07T09:28:19.650 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2023-10-07T09:28:19.650 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-07T09:28:19.650 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2023-10-07T09:28:19.651 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.652 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2023-10-07T09:28:19.653 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: mon health detail to clog = false 2023-10-07T09:28:19.654 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = "172.21.15.29,172.21.15.144,[v2:172.21.15.144:3301,v1:172.21.15.144:6790]" 2023-10-07T09:28:19.655 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on skipped message = False 2023-10-07T09:28:19.656 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = 1e617e23-0c6c-40cc-8e09-10eaf7e38e24 2023-10-07T09:28:19.656 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.656 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock profile = high_recovery_ops 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.657 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.658 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: bdev async discard = True 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: bdev enable discard = True 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore allocator = bitmap 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore block size = 96636764160 2023-10-07T09:28:19.659 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore fsck on mount = True 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluefs = 1/20 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluestore = 1/20 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 20 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: debug rocksdb = 4/10 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd backfillfull_ratio = 0.85 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd full ratio = 0.9 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd nearfull ratio = 0.8 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: osd failsafe full ratio = 0.95 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: osd objectstore = bluestore 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: osd op complaint time = 180 2023-10-07T09:28:19.660 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: debug client = 20 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: debug finisher = 20 2023-10-07T09:28:19.661 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2023-10-07T09:28:19.662 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: auth mon ticket ttl = 660 # 11m 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: auth service ticket ttl = 240 # 4m 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: # don't complain about insecure global_id in the test suite 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: # 1m isn't quite enough 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: mon_down_mkfs_grace = 2m 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_filestore_osds = false 2023-10-07T09:28:19.663 INFO:teuthology.orchestra.run.smithi144.stdout: mon op complaint time = 120 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: client mount timeout = 600 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: debug client = 20 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = "" 2023-10-07T09:28:19.664 INFO:teuthology.orchestra.run.smithi144.stdout: rados mon op timeout = 900 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout: rados osd op timeout = 900 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.b] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.c] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.a] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.c] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.b] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.d] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout:[mds] 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout: debug mds = 20 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout: mds debug frag = True 2023-10-07T09:28:19.665 INFO:teuthology.orchestra.run.smithi144.stdout: mds debug scatterstat = True 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi144.stdout: mds op complaint time = 180 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi144.stdout: mds verify scatter = True 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi144.stdout: osd op complaint time = 180 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi144.stdout: rados mon op timeout = 900 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi144.stdout: rados osd op timeout = 900 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-10-07T09:28:19.666 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-10-07T09:28:19.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-10-07T09:28:19.668 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-10-07T09:28:19.669 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "172.21.15.29,172.21.15.144,[v2:172.21.15.144:3301,v1:172.21.15.144:6790]" 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on skipped message = False 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 1e617e23-0c6c-40cc-8e09-10eaf7e38e24 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-10-07T09:28:19.670 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.671 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: bdev async discard = True 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: bdev enable discard = True 2023-10-07T09:28:19.672 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = bitmap 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1/20 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1/20 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 4/10 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-10-07T09:28:19.673 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: debug finisher = 20 2023-10-07T09:28:19.674 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.675 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-10-07T09:28:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: mon op complaint time = 120 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: client mount timeout = 600 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = "" 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2023-10-07T09:28:19.677 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout:[mds] 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds = 20 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug frag = True 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug scatterstat = True 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout: mds op complaint time = 180 2023-10-07T09:28:19.678 INFO:teuthology.orchestra.run.smithi029.stdout: mds verify scatter = True 2023-10-07T09:28:19.679 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2023-10-07T09:28:19.679 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2023-10-07T09:28:19.679 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2023-10-07T09:28:19.682 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-07T09:28:19.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-07T09:28:19.793 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-07T09:28:19.793 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:19.793 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-07T09:28:19.840 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:19.841 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-07T09:28:19.927 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:28:19.927 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:19.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-07T09:28:19.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-07T09:28:20.074 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:20.074 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:20.159 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:28:20.160 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:20.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-07T09:28:20.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-07T09:28:20.255 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:20.255 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:20.340 INFO:tasks.ceph:Setting up mon nodes... 2023-10-07T09:28:20.340 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-07T09:28:20.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-07T09:28:20.459 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-07T09:28:20.464 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-07T09:28:20.993 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-07T09:28:20.999 INFO:tasks.ceph:Setting up mds nodes... 2023-10-07T09:28:20.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-07T09:28:21.121 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-07T09:28:21.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-07T09:28:21.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-07T09:28:21.316 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-07T09:28:21.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-07T09:28:21.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-07T09:28:21.513 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-07T09:28:21.518 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-07T09:28:21.587 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-07T09:28:21.704 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-07T09:28:21.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-07T09:28:21.780 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-07T09:28:21.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-07T09:28:21.881 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-07T09:28:21.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-07T09:28:22.007 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-07T09:28:22.105 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-07T09:28:22.195 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-07T09:28:22.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-07T09:28:22.315 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-07T09:28:22.344 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-07T09:28:22.345 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-07T09:28:22.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-07T09:28:22.419 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:22.419 INFO:tasks.ceph:role: osd.0 2023-10-07T09:28:22.419 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:28:22.419 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-07T09:28:22.490 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:22.490 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:22.513 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-07T09:28:22.517 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-07T09:28:22.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-07T09:28:22.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-07T09:28:22.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-07T09:28:22.779 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:22.779 INFO:tasks.ceph:role: osd.1 2023-10-07T09:28:22.779 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:28:22.779 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-07T09:28:22.848 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:22.848 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:22.848 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:22.849 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:22.872 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-07T09:28:22.876 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-07T09:28:22.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-07T09:28:22.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-07T09:28:23.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-07T09:28:23.128 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:23.131 INFO:tasks.ceph:role: osd.2 2023-10-07T09:28:23.131 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:28:23.132 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-07T09:28:23.201 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:23.201 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:23.202 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:23.225 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-07T09:28:23.229 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-07T09:28:23.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-07T09:28:23.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-07T09:28:23.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-07T09:28:23.483 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:23.483 INFO:tasks.ceph:role: osd.3 2023-10-07T09:28:23.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:28:23.483 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-07T09:28:23.553 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:23.553 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:23.553 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:23.554 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:23.576 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-07T09:28:23.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-07T09:28:23.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-07T09:28:23.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-07T09:28:23.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:23.880+0000 7f4e429a5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-07T09:28:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:23.881+0000 7f4e429a5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-07T09:28:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:23.881+0000 7f4e429a5540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-07T09:28:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:23.881+0000 7f4e429a5540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-07T09:28:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:23.881+0000 7f4e429a5540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-07T09:28:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:23.882+0000 7f4e429a5540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-07T09:28:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:24.425+0000 7f4e429a5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:25.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:25.186+0000 7f4e429a5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:25.953+0000 7f4e429a5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:26.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-07T09:28:26.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:26.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:26.947+0000 7fac761c4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-07T09:28:26.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:26.947+0000 7fac761c4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-07T09:28:26.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:26.947+0000 7fac761c4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-07T09:28:26.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:26.947+0000 7fac761c4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-07T09:28:26.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:26.947+0000 7fac761c4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-07T09:28:26.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:26.949+0000 7fac761c4540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-07T09:28:27.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:27.981+0000 7fac761c4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:28.749+0000 7fac761c4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:29.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:29.513+0000 7fac761c4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:30.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-07T09:28:30.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:30.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:30.505+0000 7fd9bdf17540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-07T09:28:30.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:30.506+0000 7fd9bdf17540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-07T09:28:30.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:30.506+0000 7fd9bdf17540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-07T09:28:30.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:30.506+0000 7fd9bdf17540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-07T09:28:30.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:30.506+0000 7fd9bdf17540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-07T09:28:30.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:30.507+0000 7fd9bdf17540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-07T09:28:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:31.048+0000 7fd9bdf17540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:31.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:31.805+0000 7fd9bdf17540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:32.568+0000 7fd9bdf17540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:33.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-07T09:28:33.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:33.562+0000 7f2619542540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-07T09:28:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:33.563+0000 7f2619542540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-07T09:28:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:33.563+0000 7f2619542540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-07T09:28:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:33.563+0000 7f2619542540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-07T09:28:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:33.563+0000 7f2619542540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-07T09:28:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:33.564+0000 7f2619542540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-07T09:28:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:34.105+0000 7f2619542540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:34.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:34.867+0000 7f2619542540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:35.636+0000 7f2619542540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:36.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-07T09:28:36.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-07T09:28:36.611 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:36.611 INFO:tasks.ceph:role: osd.4 2023-10-07T09:28:36.611 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:28:36.611 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:36.691 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:37.128 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-07T09:28:37.132 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-07T09:28:37.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-07T09:28:37.327 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-07T09:28:37.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-07T09:28:37.476 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:37.477 INFO:tasks.ceph:role: osd.5 2023-10-07T09:28:37.477 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:28:37.477 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-07T09:28:37.553 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:37.553 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:37.553 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:37.553 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:37.553 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:37.554 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:37.554 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:37.554 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:37.554 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:37.554 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:37.966 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-07T09:28:37.970 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-07T09:28:37.970 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-07T09:28:38.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-07T09:28:38.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-07T09:28:38.255 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:38.255 INFO:tasks.ceph:role: osd.6 2023-10-07T09:28:38.255 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:28:38.256 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-07T09:28:38.332 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:38.332 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:38.332 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:38.332 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:38.333 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:38.333 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:38.333 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:38.333 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:38.333 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:38.333 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:38.781 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-07T09:28:38.785 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-07T09:28:38.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-07T09:28:38.970 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-07T09:28:39.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-07T09:28:39.117 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-07T09:28:39.117 INFO:tasks.ceph:role: osd.7 2023-10-07T09:28:39.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:28:39.117 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-07T09:28:39.193 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-07T09:28:39.630 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-07T09:28:39.772 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-07T09:28:39.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-07T09:28:39.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-07T09:28:39.965 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:40.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.087+0000 7fbaa40a7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-07T09:28:40.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.087+0000 7fbaa40a7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-07T09:28:40.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.088+0000 7fbaa40a7540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-07T09:28:40.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.088+0000 7fbaa40a7540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-07T09:28:40.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.088+0000 7fbaa40a7540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-07T09:28:40.090 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.089+0000 7fbaa40a7540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-07T09:28:40.635 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:40.634+0000 7fbaa40a7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:41.392 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:41.391+0000 7fbaa40a7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:42.154 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:42.153+0000 7fbaa40a7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:42.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-07T09:28:43.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:43.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.142+0000 7f2cdb6a6540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-07T09:28:43.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.143+0000 7f2cdb6a6540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-07T09:28:43.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.143+0000 7f2cdb6a6540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-07T09:28:43.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.143+0000 7f2cdb6a6540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-07T09:28:43.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.143+0000 7f2cdb6a6540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-07T09:28:43.145 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.144+0000 7f2cdb6a6540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-07T09:28:43.684 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:43.682+0000 7f2cdb6a6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:44.449 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:44.448+0000 7f2cdb6a6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:45.212 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:45.211+0000 7f2cdb6a6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:46.028 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-07T09:28:46.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:46.199 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.198+0000 7f0103eb4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-07T09:28:46.199 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.198+0000 7f0103eb4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-07T09:28:46.200 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.198+0000 7f0103eb4540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-07T09:28:46.200 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.198+0000 7f0103eb4540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-07T09:28:46.200 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.198+0000 7f0103eb4540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-07T09:28:46.201 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.200+0000 7f0103eb4540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-07T09:28:46.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:46.743+0000 7f0103eb4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:47.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:28:47.223 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:28:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:47.504+0000 7f0103eb4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:48.268 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:48.267+0000 7f0103eb4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:49.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-07T09:28:49.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:49.253 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.251+0000 7fb4f9c89540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-07T09:28:49.253 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.252+0000 7fb4f9c89540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-07T09:28:49.253 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.252+0000 7fb4f9c89540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-07T09:28:49.254 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.252+0000 7fb4f9c89540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-07T09:28:49.254 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.252+0000 7fb4f9c89540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-07T09:28:49.254 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.253+0000 7fb4f9c89540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-07T09:28:49.798 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:49.797+0000 7fb4f9c89540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:50.559 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:50.558+0000 7fb4f9c89540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:51.322 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:51.321+0000 7fb4f9c89540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-07T09:28:52.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-07T09:28:52.209 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-07T09:28:52.210 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-07T09:28:52.287 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-07T09:28:52.358 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-07T09:28:52.469 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-07T09:28:52.578 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-07T09:28:52.689 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-07T09:28:52.798 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:52.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-07T09:28:52.907 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:52.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-07T09:28:52.983 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:52.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-07T09:28:53.054 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-07T09:28:53.162 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.163 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-07T09:28:53.271 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-07T09:28:53.379 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-07T09:28:53.486 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.487 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-07T09:28:53.595 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:53.595 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-07T09:28:53.645 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:53.645 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-07T09:28:53.692 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.692 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-07T09:28:53.737 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:53.737 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-07T09:28:53.823 INFO:tasks.ceph:Adding keys to all mons... 2023-10-07T09:28:53.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-07T09:28:53.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ== 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg== 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww== 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw== 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2023-10-07T09:28:53.894 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng== 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A== 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw== 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg== 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ== 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ== 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2023-10-07T09:28:53.895 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA== 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ== 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg== 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ== 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g== 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout:[client.2] 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ== 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout:[client.1] 2023-10-07T09:28:53.896 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA== 2023-10-07T09:28:53.897 INFO:teuthology.orchestra.run.smithi029.stdout:[client.3] 2023-10-07T09:28:53.897 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA== 2023-10-07T09:28:53.930 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.y] 2023-10-07T09:28:53.930 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ== 2023-10-07T09:28:53.930 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.a] 2023-10-07T09:28:53.930 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg== 2023-10-07T09:28:53.930 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.c] 2023-10-07T09:28:53.930 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww== 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw== 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng== 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A== 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.3] 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw== 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg== 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.b] 2023-10-07T09:28:53.931 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.d] 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.4] 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.5] 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.6] 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.7] 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g== 2023-10-07T09:28:53.932 INFO:teuthology.orchestra.run.smithi144.stdout:[client.2] 2023-10-07T09:28:53.933 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ== 2023-10-07T09:28:53.933 INFO:teuthology.orchestra.run.smithi144.stdout:[client.1] 2023-10-07T09:28:53.933 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA== 2023-10-07T09:28:53.933 INFO:teuthology.orchestra.run.smithi144.stdout:[client.3] 2023-10-07T09:28:53.933 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA== 2023-10-07T09:28:53.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-07T09:28:53.942 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-07T09:28:54.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.298 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.620 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:54.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-07T09:28:54.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-07T09:28:54.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:54.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-07T09:28:55.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.052 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.159 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-07T09:28:55.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.588 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.809 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-07T09:28:55.914 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-07T09:28:55.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-07T09:28:55.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-07T09:28:56.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-07T09:28:56.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-07T09:28:56.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-07T09:28:56.358 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-07T09:28:56.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-07T09:28:56.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-07T09:28:56.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-07T09:28:56.680 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:56.685 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-07T09:28:56.730 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-07T09:28:56.730 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-07T09:28:56.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-07T09:28:56.735 INFO:tasks.ceph.mon.a:Started 2023-10-07T09:28:56.735 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-07T09:28:56.735 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-07T09:28:56.774 INFO:tasks.ceph.mon.b:Started 2023-10-07T09:28:56.774 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-07T09:28:56.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-07T09:28:56.779 INFO:tasks.ceph.mon.c:Started 2023-10-07T09:28:56.779 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-07T09:28:56.779 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-07T09:28:56.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-07T09:28:56.786 INFO:tasks.ceph.mgr.y:Started 2023-10-07T09:28:56.786 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-07T09:28:56.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-07T09:28:56.791 INFO:tasks.ceph.mgr.x:Started 2023-10-07T09:28:56.791 INFO:tasks.ceph:Setting crush tunables to default 2023-10-07T09:28:56.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2023-10-07T09:28:56.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.983+0000 7f8f88bff700 1 Processor -- start 2023-10-07T09:28:56.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.984+0000 7f8f88bff700 1 -- start start 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f88bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 0x7f8f840d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f88bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f840cbf20 con 0x7f8f84159750 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f88bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f840cc200 con 0x7f8f840ce8a0 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f88bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840cc4e0 con 0x7f8f84159b20 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f8259c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8f84159750 0x7f8f840ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41792/0 (socket says 172.21.15.29:41792) 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f8259c700 1 -- 172.21.15.29:0/1281919941 learned_addr learned my addr 172.21.15.29:0/1281919941 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f81d9b700 1 --2- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 0x7f8f840d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f81d9b700 1 -- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 msgr2=0x7f8f840d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-07T09:28:56.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f81d9b700 1 -- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 msgr2=0x7f8f840d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-07T09:28:56.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.985+0000 7f8f81d9b700 1 --2- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 0x7f8f840d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-07T09:28:56.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:56.986+0000 7f8f81d9b700 1 --2- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 0x7f8f840d10a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:28:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.133+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794631788 0 0) 0x7f8f840cbf20 con 0x7f8f84159750 2023-10-07T09:28:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.133+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c003570 con 0x7f8f84159750 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799407673 0 0) 0x7f8f840cc200 con 0x7f8f840ce8a0 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f840cbf20 con 0x7f8f840ce8a0 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283161261 0 0) 0x7f8f6c003570 con 0x7f8f84159750 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 msgr2=0x7f8f840d10a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 --2- 172.21.15.29:0/1281919941 >> v2:172.21.15.144:3300/0 conn(0x7f8f84159b20 0x7f8f840d10a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 >> v1:172.21.15.144:6789/0 conn(0x7f8f840ce8a0 legacy=0x7f8f840c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f840cc7c0 con 0x7f8f84159750 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f8f740015f0 con 0x7f8f84159750 2023-10-07T09:28:57.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 >> v1:172.21.15.29:6789/0 conn(0x7f8f84159750 legacy=0x7f8f840ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f6c003c40 0x7f8f6c004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f6c004810 0x7f8f6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f6c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840cc4e0 con 0x7f8f6c004810 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c00c1b0 con 0x7f8f6c003c40 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c00c370 con 0x7f8f6c0052e0 2023-10-07T09:28:57.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f82d9d700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f6c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:57.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.134+0000 7f8f8259c700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f6c003c40 0x7f8f6c004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:57.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f81d9b700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f6c004810 0x7f8f6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:57.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 -- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f6c003c40 msgr2=0x7f8f6c004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:57.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f6c003c40 0x7f8f6c004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 -- 172.21.15.29:0/1281919941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f6c004810 msgr2=0x7f8f6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:57.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f6c004810 0x7f8f6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 -- 172.21.15.29:0/1281919941 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c003570 con 0x7f8f6c0052e0 2023-10-07T09:28:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f81d9b700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f6c004810 0x7f8f6c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:28:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 -- 172.21.15.29:0/1281919941 reap_dead start 2023-10-07T09:28:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.135+0000 7f8f82d9d700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f6c00ba60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c002160 tx=0x7f8f7c0017e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9116712dec69a1aa server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.136+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c001af0 con 0x7f8f6c0052e0 2023-10-07T09:28:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.136+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c00c980 con 0x7f8f6c0052e0 2023-10-07T09:28:57.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.136+0000 7f8f8159a700 1 -- 172.21.15.29:0/1281919941 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00cd20 con 0x7f8f6c0052e0 2023-10-07T09:28:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 -- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 msgr2=0x7f8f6c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f6c00ba60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c002160 tx=0x7f8f7c0017e0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 -- 172.21.15.29:0/1281919941 shutdown_connections 2023-10-07T09:28:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 --2- 172.21.15.29:0/1281919941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f6c00ba60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 -- 172.21.15.29:0/1281919941 >> 172.21.15.29:0/1281919941 conn(0x7f8f84063a10 msgr2=0x7f8f840d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 -- 172.21.15.29:0/1281919941 shutdown_connections 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.137+0000 7f8f88bff700 1 -- 172.21.15.29:0/1281919941 wait complete. 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.139+0000 7f8f88bff700 1 Processor -- start 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.139+0000 7f8f88bff700 1 -- start start 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f88bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f84152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f88bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f840b4450 0x7f8f840ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f88bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841699c0 0x7f8f84169e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f88bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84153750 con 0x7f8f841699c0 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f88bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840b48c0 con 0x7f8f840b4450 2023-10-07T09:28:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f88bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f840b4ba0 con 0x7f8f6c0052e0 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f8259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f84152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f81d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f840b4450 0x7f8f840ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f8259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f84152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43130/0 (socket says 172.21.15.29:43130) 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.140+0000 7f8f8259c700 1 -- 172.21.15.29:0/1532150458 learned_addr learned my addr 172.21.15.29:0/1532150458 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f82d9d700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841699c0 0x7f8f84169e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f8259c700 1 -- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f840b4450 msgr2=0x7f8f840ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f8259c700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f840b4450 0x7f8f840ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f8259c700 1 -- 172.21.15.29:0/1532150458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841699c0 msgr2=0x7f8f84169e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f8259c700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841699c0 0x7f8f84169e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f8259c700 1 -- 172.21.15.29:0/1532150458 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f840b4e80 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f82d9d700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841699c0 0x7f8f84169e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.141+0000 7f8f8259c700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f84152750 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400f900 tx=0x7f8f74009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740072e0 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74007460 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f840b5160 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740057f0 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f84152c90 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8f74005b60 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.142+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8f7401dd80 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.143+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f840b5680 con 0x7f8f6c0052e0 2023-10-07T09:28:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.147+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8f840b5680 con 0x7f8f6c0052e0 2023-10-07T09:28:57.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.226+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8f8404c3c0 con 0x7f8f6c0052e0 2023-10-07T09:28:57.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.231+0000 7f8f727fc700 1 -- 172.21.15.29:0/1532150458 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8f7401d460 con 0x7f8f6c0052e0 2023-10-07T09:28:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 msgr2=0x7f8f84152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f84152750 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400f900 tx=0x7f8f74009040 comp rx=0 tx=0).stop 2023-10-07T09:28:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 shutdown_connections 2023-10-07T09:28:57.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f841699c0 0x7f8f84169e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8f6c0052e0 0x7f8f84152750 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 --2- 172.21.15.29:0/1532150458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f840b4450 0x7f8f840ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:57.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 >> 172.21.15.29:0/1532150458 conn(0x7f8f84063a10 msgr2=0x7f8f841550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 shutdown_connections 2023-10-07T09:28:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:28:57.232+0000 7f8f88bff700 1 -- 172.21.15.29:0/1532150458 wait complete. 2023-10-07T09:28:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-10-07T09:28:57.250 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-07T09:28:57.251 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:57.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-07T09:28:57.302 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:28:57.301+0000 7f31b186c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:28:57.317 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:28:57.316+0000 7f9b3986c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:28:57.328 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:57.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-07T09:28:57.409 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:57.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-07T09:28:57.490 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:28:57.489+0000 7f31b186c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:28:57.514 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-07T09:28:57.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-07T09:28:57.550 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:28:57.549+0000 7f9b3986c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:28:57.588 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:57.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-07T09:28:57.657 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:57.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-07T09:28:57.766 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:57.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-07T09:28:57.873 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-07T09:28:57.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-07T09:28:57.918 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:28:57.916+0000 7f31b186c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:28:57.974 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:28:57.972+0000 7f9b3986c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:28:57.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new a149e59b-23fb-41a2-90c8-0973d32e496a 0 2023-10-07T09:28:58.210 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.209+0000 7fd34b5ff700 1 Processor -- start 2023-10-07T09:28:58.210 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.209+0000 7fd34b5ff700 1 -- start start 2023-10-07T09:28:58.212 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd34b5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd344159750 0x7fd3440d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.212 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd34b5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd3440ce8a0 0x7fd3440c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd34b5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3440cbf20 con 0x7fd344159b20 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd34b5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3440cc200 con 0x7fd344159750 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd34b5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3440cc4e0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd349b9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd3440ce8a0 0x7fd3440c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd349b9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd3440ce8a0 0x7fd3440c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44742/0 (socket says 172.21.15.144:44742) 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd34939b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd344159750 0x7fd3440d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.213 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.211+0000 7fd349b9c700 1 -- 172.21.15.144:0/1775655590 learned_addr learned my addr 172.21.15.144:0/1775655590 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd349b9c700 1 -- 172.21.15.144:0/1775655590 >> v2:172.21.15.29:3300/0 conn(0x7fd344159750 msgr2=0x7fd3440d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd349b9c700 1 --2- 172.21.15.144:0/1775655590 >> v2:172.21.15.29:3300/0 conn(0x7fd344159750 0x7fd3440d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd349b9c700 1 -- 172.21.15.144:0/1775655590 >> v1:172.21.15.29:6789/0 conn(0x7fd344159b20 legacy=0x7fd3440ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd349b9c700 1 -- 172.21.15.144:0/1775655590 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3440cc7c0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd349b9c700 1 --2- 172.21.15.144:0/1775655590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3440ce8a0 0x7fd3440c5460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd34415e510 tx=0x7fd33c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c37b4a1c9898bd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd33bfff700 1 -- 172.21.15.144:0/1775655590 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00e070 con 0x7fd3440ce8a0 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd33bfff700 1 -- 172.21.15.144:0/1775655590 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00ba20 con 0x7fd3440ce8a0 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.212+0000 7fd33bfff700 1 -- 172.21.15.144:0/1775655590 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0092e0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1775655590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3440ce8a0 msgr2=0x7fd3440c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.215 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1775655590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3440ce8a0 0x7fd3440c5460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd34415e510 tx=0x7fd33c009840 comp rx=0 tx=0).stop 2023-10-07T09:28:58.216 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1775655590 shutdown_connections 2023-10-07T09:28:58.216 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1775655590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3440ce8a0 0x7fd3440c5460 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.216 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1775655590 >> v2:172.21.15.29:3300/0 conn(0x7fd344159750 0x7fd3440d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.216 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1775655590 >> 172.21.15.144:0/1775655590 conn(0x7fd344063a10 msgr2=0x7fd3440d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:58.216 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1775655590 shutdown_connections 2023-10-07T09:28:58.216 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.213+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1775655590 wait complete. 2023-10-07T09:28:58.218 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.215+0000 7fd34b5ff700 1 Processor -- start 2023-10-07T09:28:58.218 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.215+0000 7fd34b5ff700 1 -- start start 2023-10-07T09:28:58.218 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34b5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 0x7fd344150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34b5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd344159750 0x7fd344153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd344159b20 0x7fd3440b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34b5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440b41c0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34b5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3441552c0 con 0x7fd344159750 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3441546b0 con 0x7fd344159b20 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 0x7fd344150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 0x7fd344150f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.144:41952/0 (socket says 172.21.15.144:41952) 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd348b9a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd344159750 0x7fd344153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 -- 172.21.15.144:0/1836704084 learned_addr learned my addr 172.21.15.144:0/1836704084 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:58.219 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd349b9c700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd344159b20 0x7fd3440b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 -- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd344159b20 msgr2=0x7fd3440b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd344159b20 0x7fd3440b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 -- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd344159750 msgr2=0x7fd344153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd344159750 0x7fd344153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd34939b700 1 -- 172.21.15.144:0/1836704084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c009ae0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.216+0000 7fd349b9c700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd344159b20 0x7fd3440b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd34939b700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 0x7fd344150f10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd34000ec80 tx=0x7fd340009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34000c850 con 0x7fd3440ce8a0 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34000c9b0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.220 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34000cd30 con 0x7fd3440ce8a0 2023-10-07T09:28:58.221 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd344154990 con 0x7fd3440ce8a0 2023-10-07T09:28:58.221 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd344154f50 con 0x7fd3440ce8a0 2023-10-07T09:28:58.221 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.217+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd340013900 con 0x7fd3440ce8a0 2023-10-07T09:28:58.221 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.218+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd34001c970 con 0x7fd3440ce8a0 2023-10-07T09:28:58.221 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.218+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34404c3c0 con 0x7fd3440ce8a0 2023-10-07T09:28:58.222 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.221+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd340021060 con 0x7fd3440ce8a0 2023-10-07T09:28:58.307 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.306+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a149e59b-23fb-41a2-90c8-0973d32e496a", "id": 0} v 0) v1 -- 0x7fd344064040 con 0x7fd3440ce8a0 2023-10-07T09:28:58.310 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.309+0000 7fd3397fa700 1 -- 172.21.15.144:0/1836704084 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a149e59b-23fb-41a2-90c8-0973d32e496a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd340017070 con 0x7fd3440ce8a0 2023-10-07T09:28:58.311 INFO:teuthology.orchestra.run.smithi144.stdout:0 2023-10-07T09:28:58.312 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 msgr2=0x7fd344150f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.312 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 0x7fd344150f10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd34000ec80 tx=0x7fd340009120 comp rx=0 tx=0).stop 2023-10-07T09:28:58.312 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 shutdown_connections 2023-10-07T09:28:58.312 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd344159b20 0x7fd3440b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.312 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3440ce8a0 0x7fd344150f10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.313 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 --2- 172.21.15.144:0/1836704084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd344159750 0x7fd344153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.313 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 >> 172.21.15.144:0/1836704084 conn(0x7fd344063a10 msgr2=0x7fd3440c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:58.313 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 shutdown_connections 2023-10-07T09:28:58.313 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.311+0000 7fd34b5ff700 1 -- 172.21.15.144:0/1836704084 wait complete. 2023-10-07T09:28:58.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 492dce8b-159d-45a2-b902-e9797b66c967 1 2023-10-07T09:28:58.518 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.516+0000 7f75f37ff700 1 Processor -- start 2023-10-07T09:28:58.518 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.517+0000 7f75f37ff700 1 -- start start 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.518+0000 7f75f37ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75ec159b20 0x7f75ec0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.518+0000 7f75f37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.518+0000 7f75f37ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ec0cbf20 con 0x7f75ec159750 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f37ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ec0cc200 con 0x7f75ec159b20 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec0cc4e0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75ec159b20 0x7f75ec0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f1d9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75ec159b20 0x7f75ec0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.144:41960/0 (socket says 172.21.15.144:41960) 2023-10-07T09:28:58.521 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f1d9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:57806/0 (socket says 172.21.15.144:57806) 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 -- 172.21.15.144:0/2739402623 learned_addr learned my addr 172.21.15.144:0/2739402623 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f159b700 1 --1- 172.21.15.144:0/2739402623 >> v1:172.21.15.29:6789/0 conn(0x7f75ec159750 0x7f75ec0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.144:54538/0 (socket says 172.21.15.144:54538) 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 -- 172.21.15.144:0/2739402623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 msgr2=0x7f75ec0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 --2- 172.21.15.144:0/2739402623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 -- 172.21.15.144:0/2739402623 >> v1:172.21.15.29:6789/0 conn(0x7f75ec159750 legacy=0x7f75ec0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f0d9a700 1 -- 172.21.15.144:0/2739402623 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec0cc7c0 con 0x7f75ec159b20 2023-10-07T09:28:58.522 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.519+0000 7f75f1d9c700 1 --2- 172.21.15.144:0/2739402623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:58.523 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.520+0000 7f75f0d9a700 1 --2- 172.21.15.144:0/2739402623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159b20 0x7f75ec0d10a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0099b0 tx=0x7f75dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68acabd79ddaa668 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:58.523 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.520+0000 7f75e3fff700 1 -- 172.21.15.144:0/2739402623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc009ea0 con 0x7f75ec159b20 2023-10-07T09:28:58.523 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.520+0000 7f75e3fff700 1 -- 172.21.15.144:0/2739402623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc00b860 con 0x7f75ec159b20 2023-10-07T09:28:58.523 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.520+0000 7f75e3fff700 1 -- 172.21.15.144:0/2739402623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00bc00 con 0x7f75ec159b20 2023-10-07T09:28:58.526 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 -- 172.21.15.144:0/2739402623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159b20 msgr2=0x7f75ec0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.526 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2739402623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159b20 0x7f75ec0d10a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0099b0 tx=0x7f75dc009250 comp rx=0 tx=0).stop 2023-10-07T09:28:58.526 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 -- 172.21.15.144:0/2739402623 shutdown_connections 2023-10-07T09:28:58.526 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2739402623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.526 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2739402623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159b20 0x7f75ec0d10a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.526 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 -- 172.21.15.144:0/2739402623 >> 172.21.15.144:0/2739402623 conn(0x7f75ec063a10 msgr2=0x7f75ec0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 -- 172.21.15.144:0/2739402623 shutdown_connections 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.521+0000 7f75f37ff700 1 -- 172.21.15.144:0/2739402623 wait complete. 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.522+0000 7f75f37ff700 1 Processor -- start 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.522+0000 7f75f37ff700 1 -- start start 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f37ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159750 0x7f75ec144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f37ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75ec159b20 0x7f75ec0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f37ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec0b48e0 con 0x7f75ec159750 2023-10-07T09:28:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f37ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec0b3cd0 con 0x7f75ec159b20 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec0b3fb0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.523+0000 7f75f159b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:57814/0 (socket says 172.21.15.144:57814) 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 -- 172.21.15.144:0/2758650814 learned_addr learned my addr 172.21.15.144:0/2758650814 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f0d9a700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159750 0x7f75ec144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f1d9c700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75ec159b20 0x7f75ec0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 -- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75ec159b20 msgr2=0x7f75ec0695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75ec159b20 0x7f75ec0695c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 -- 172.21.15.144:0/2758650814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159750 msgr2=0x7f75ec144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159750 0x7f75ec144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 -- 172.21.15.144:0/2758650814 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4009a20 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75f159b700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0b3360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c790 tx=0x7f75e8007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8005ac0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.524+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e8005c40 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.529 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.525+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc0094f0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.530 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.525+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e80070d0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.530 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.525+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec0b4340 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.530 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.525+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f75e801c440 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.530 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.525+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f75e80127c0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.531 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.526+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ec04c3c0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.531 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.528+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f75e8017070 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.576 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:28:58.575+0000 7f31b186c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:28:58.615 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.614+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "492dce8b-159d-45a2-b902-e9797b66c967", "id": 1} v 0) v1 -- 0x7f75ec0ca740 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.619 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.618+0000 7f75e17fa700 1 -- 172.21.15.144:0/2758650814 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "492dce8b-159d-45a2-b902-e9797b66c967", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f75e801c8d0 con 0x7f75ec0ce8a0 2023-10-07T09:28:58.619 INFO:teuthology.orchestra.run.smithi144.stdout:1 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 msgr2=0x7f75ec0b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0b3360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c790 tx=0x7f75e8007f60 comp rx=0 tx=0).stop 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 shutdown_connections 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75ec159b20 0x7f75ec0695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75ec0ce8a0 0x7f75ec0b3360 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 --2- 172.21.15.144:0/2758650814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75ec159750 0x7f75ec144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 >> 172.21.15.144:0/2758650814 conn(0x7f75ec063a10 msgr2=0x7f75ec154530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:58.621 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 shutdown_connections 2023-10-07T09:28:58.622 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.620+0000 7f75f37ff700 1 -- 172.21.15.144:0/2758650814 wait complete. 2023-10-07T09:28:58.624 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:28:58.623+0000 7f9b3986c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:28:58.644 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new ba67af3d-2617-4e93-aa29-8b22dcc465af 2 2023-10-07T09:28:58.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:28:58.758+0000 7f31b186c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:28:58.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:28:58.798+0000 7f9b3986c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:28:58.832 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.831+0000 7fc808fff700 1 Processor -- start 2023-10-07T09:28:58.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.831+0000 7fc808fff700 1 -- start start 2023-10-07T09:28:58.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc808fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc804159790 0x7fc80406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc808fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc80406b3f0 0x7fc8040c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc808fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80406a430 con 0x7fc80406a8c0 2023-10-07T09:28:58.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc808fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80406b830 con 0x7fc80406b3f0 2023-10-07T09:28:58.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc808fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc804159b60 con 0x7fc804159790 2023-10-07T09:28:58.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc80259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc804159790 0x7fc80406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc802d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc80406b3f0 0x7fc8040c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc80259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc804159790 0x7fc80406aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:57828/0 (socket says 172.21.15.144:57828) 2023-10-07T09:28:58.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc802d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc80406b3f0 0x7fc8040c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44772/0 (socket says 172.21.15.144:44772) 2023-10-07T09:28:58.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc80259c700 1 -- 172.21.15.144:0/2941945920 learned_addr learned my addr 172.21.15.144:0/2941945920 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:58.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc802d9d700 1 -- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc804159790 msgr2=0x7fc80406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:58.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc802d9d700 1 --2- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc804159790 0x7fc80406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc802d9d700 1 -- 172.21.15.144:0/2941945920 >> v1:172.21.15.29:6789/0 conn(0x7fc80406a8c0 legacy=0x7fc8040c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.833+0000 7fc802d9d700 1 -- 172.21.15.144:0/2941945920 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804069820 con 0x7fc80406b3f0 2023-10-07T09:28:58.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.834+0000 7fc802d9d700 1 --2- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc80406b3f0 0x7fc8040c7650 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009e60 tx=0x7fc7fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12a52696e67cc16d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.834+0000 7fc80159a700 1 -- 172.21.15.144:0/2941945920 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00e070 con 0x7fc80406b3f0 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.834+0000 7fc80159a700 1 -- 172.21.15.144:0/2941945920 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc00bb60 con 0x7fc80406b3f0 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.834+0000 7fc80159a700 1 -- 172.21.15.144:0/2941945920 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc012430 con 0x7fc80406b3f0 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.834+0000 7fc808fff700 1 -- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc80406b3f0 msgr2=0x7fc8040c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.834+0000 7fc808fff700 1 --2- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc80406b3f0 0x7fc8040c7650 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009e60 tx=0x7fc7fc009780 comp rx=0 tx=0).stop 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.835+0000 7fc808fff700 1 -- 172.21.15.144:0/2941945920 shutdown_connections 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.835+0000 7fc808fff700 1 --2- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc80406b3f0 0x7fc8040c7650 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.835+0000 7fc808fff700 1 --2- 172.21.15.144:0/2941945920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc804159790 0x7fc80406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.835+0000 7fc808fff700 1 -- 172.21.15.144:0/2941945920 >> 172.21.15.144:0/2941945920 conn(0x7fc8040c9410 msgr2=0x7fc8040c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.835+0000 7fc808fff700 1 -- 172.21.15.144:0/2941945920 shutdown_connections 2023-10-07T09:28:58.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.835+0000 7fc808fff700 1 -- 172.21.15.144:0/2941945920 wait complete. 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.836+0000 7fc808fff700 1 Processor -- start 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.836+0000 7fc808fff700 1 -- start start 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc808fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc80406b3f0 0x7fc8040ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc808fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc804159790 0x7fc80415d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc808fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 0x7fc80406e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc808fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80414b5f0 con 0x7fc80406b3f0 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc808fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80416cc20 con 0x7fc804159790 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc808fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8040aef10 con 0x7fc8040ac100 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc802d9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 0x7fc80406e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.837+0000 7fc801d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc804159790 0x7fc80415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 0x7fc80406e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:57830/0 (socket says 172.21.15.144:57830) 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 -- 172.21.15.144:0/2637821847 learned_addr learned my addr 172.21.15.144:0/2637821847 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc80259c700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc80406b3f0 0x7fc8040ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 -- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc804159790 msgr2=0x7fc80415d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc804159790 0x7fc80415d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 -- 172.21.15.144:0/2637821847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc80406b3f0 msgr2=0x7fc8040ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc80406b3f0 0x7fc8040ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 -- 172.21.15.144:0/2637821847 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc009a20 con 0x7fc8040ac100 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc801d9b700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc804159790 0x7fc80415d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc802d9d700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 0x7fc80406e3b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc80406c760 tx=0x7fc7fc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc012ea0 con 0x7fc8040ac100 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc027470 con 0x7fc8040ac100 2023-10-07T09:28:58.841 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc027850 con 0x7fc8040ac100 2023-10-07T09:28:58.842 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.838+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8040ac570 con 0x7fc8040ac100 2023-10-07T09:28:58.842 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.839+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8040acb30 con 0x7fc8040ac100 2023-10-07T09:28:58.842 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.839+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc7fc024070 con 0x7fc8040ac100 2023-10-07T09:28:58.842 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.839+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fc7fc01dda0 con 0x7fc8040ac100 2023-10-07T09:28:58.842 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.840+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80404c3c0 con 0x7fc8040ac100 2023-10-07T09:28:58.843 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.842+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc7fc02c020 con 0x7fc8040ac100 2023-10-07T09:28:58.929 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.928+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ba67af3d-2617-4e93-aa29-8b22dcc465af", "id": 2} v 0) v1 -- 0x7fc8040abc40 con 0x7fc8040ac100 2023-10-07T09:28:58.933 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.932+0000 7fc7f27fc700 1 -- 172.21.15.144:0/2637821847 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ba67af3d-2617-4e93-aa29-8b22dcc465af", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc7fc01d420 con 0x7fc8040ac100 2023-10-07T09:28:58.933 INFO:teuthology.orchestra.run.smithi144.stdout:2 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 msgr2=0x7fc80406e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 0x7fc80406e3b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc80406c760 tx=0x7fc7fc019000 comp rx=0 tx=0).stop 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 shutdown_connections 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8040ac100 0x7fc80406e3b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc80406b3f0 0x7fc8040ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 --2- 172.21.15.144:0/2637821847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc804159790 0x7fc80415d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 >> 172.21.15.144:0/2637821847 conn(0x7fc8040c9410 msgr2=0x7fc80415bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 shutdown_connections 2023-10-07T09:28:58.935 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:58.933+0000 7fc808fff700 1 -- 172.21.15.144:0/2637821847 wait complete. 2023-10-07T09:28:58.951 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 7565c3dd-b998-49b7-888e-dcd298f58b12 3 2023-10-07T09:28:59.138 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.136+0000 7f938a9ff700 1 Processor -- start 2023-10-07T09:28:59.138 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.137+0000 7f938a9ff700 1 -- start start 2023-10-07T09:28:59.140 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f938a9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9384159750 0x7f93840d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.140 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f938a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9384159b20 0x7f93840ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f938a9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 0x7f93840ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f938a9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93840cbf20 con 0x7f9384159750 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f938a9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93840cc200 con 0x7f93840ce6d0 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f938a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93840cc4e0 con 0x7f9384159b20 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f9388f9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 0x7f93840ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f93837fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9384159b20 0x7f93840ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f9383fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9384159750 0x7f93840d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f9388f9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 0x7f93840ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44786/0 (socket says 172.21.15.144:44786) 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.138+0000 7f9388f9c700 1 -- 172.21.15.144:0/2208002525 learned_addr learned my addr 172.21.15.144:0/2208002525 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9383fff700 1 -- 172.21.15.144:0/2208002525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9384159b20 msgr2=0x7f93840ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:59.141 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9383fff700 1 --2- 172.21.15.144:0/2208002525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9384159b20 0x7f93840ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9383fff700 1 -- 172.21.15.144:0/2208002525 >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 msgr2=0x7f93840ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9383fff700 1 --2- 172.21.15.144:0/2208002525 >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 0x7f93840ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9383fff700 1 -- 172.21.15.144:0/2208002525 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93840cc7c0 con 0x7f9384159750 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9388f9c700 1 --2- 172.21.15.144:0/2208002525 >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 0x7f93840ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f93837fe700 1 --2- 172.21.15.144:0/2208002525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9384159b20 0x7f93840ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9383fff700 1 --2- 172.21.15.144:0/2208002525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840d10a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f93840d0d60 tx=0x7f9374009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132dfe8a148c8737 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9382ffd700 1 -- 172.21.15.144:0/2208002525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374005c10 con 0x7f9384159750 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9382ffd700 1 -- 172.21.15.144:0/2208002525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9374005d70 con 0x7f9384159750 2023-10-07T09:28:59.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.139+0000 7f9382ffd700 1 -- 172.21.15.144:0/2208002525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374006260 con 0x7f9384159750 2023-10-07T09:28:59.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 -- 172.21.15.144:0/2208002525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 msgr2=0x7f93840d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 --2- 172.21.15.144:0/2208002525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840d10a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f93840d0d60 tx=0x7f9374009120 comp rx=0 tx=0).stop 2023-10-07T09:28:59.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 -- 172.21.15.144:0/2208002525 shutdown_connections 2023-10-07T09:28:59.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 --2- 172.21.15.144:0/2208002525 >> v2:172.21.15.144:3300/0 conn(0x7f93840ce6d0 0x7f93840ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 --2- 172.21.15.144:0/2208002525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840d10a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 --2- 172.21.15.144:0/2208002525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9384159b20 0x7f93840ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 -- 172.21.15.144:0/2208002525 >> 172.21.15.144:0/2208002525 conn(0x7f9384063a10 msgr2=0x7f93840d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 -- 172.21.15.144:0/2208002525 shutdown_connections 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.140+0000 7f938a9ff700 1 -- 172.21.15.144:0/2208002525 wait complete. 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.142+0000 7f938a9ff700 1 Processor -- start 2023-10-07T09:28:59.144 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.142+0000 7f938a9ff700 1 -- start start 2023-10-07T09:28:59.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f938a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 0x7f93840c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f938a9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f938a9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9384159b20 0x7f93841534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f938a9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93840c8cd0 con 0x7f9384159750 2023-10-07T09:28:59.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f938a9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93840b4940 con 0x7f9384159b20 2023-10-07T09:28:59.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f938a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93840c8710 con 0x7f93840ce6d0 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 0x7f93840c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 0x7f93840c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:57848/0 (socket says 172.21.15.144:57848) 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 -- 172.21.15.144:0/1619242807 learned_addr learned my addr 172.21.15.144:0/1619242807 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f93837fe700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9388f9c700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9384159b20 0x7f93841534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 -- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9384159b20 msgr2=0x7f93841534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9384159b20 0x7f93841534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 -- 172.21.15.144:0/1619242807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 msgr2=0x7f93840c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.143+0000 7f9383fff700 1 -- 172.21.15.144:0/1619242807 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c009ae0 con 0x7f93840ce6d0 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.144+0000 7f9383fff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 0x7f93840c8050 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9374009160 tx=0x7f9374016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:59.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.144+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937401b760 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.144+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937401b8c0 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.144+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937401bc80 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.144+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9378009420 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.144+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93740093c0 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.145+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f93740268d0 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.145+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f937401bde0 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.145+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938404c3c0 con 0x7f93840ce6d0 2023-10-07T09:28:59.149 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.148+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9374026d10 con 0x7f93840ce6d0 2023-10-07T09:28:59.212 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:28:59.212+0000 7f31b186c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:28:59.236 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.235+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7565c3dd-b998-49b7-888e-dcd298f58b12", "id": 3} v 0) v1 -- 0x7f9384064040 con 0x7f93840ce6d0 2023-10-07T09:28:59.236 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:28:59.236+0000 7f9b3986c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:28:59.240 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.239+0000 7f936bfff700 1 -- 172.21.15.144:0/1619242807 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7565c3dd-b998-49b7-888e-dcd298f58b12", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9374011420 con 0x7f93840ce6d0 2023-10-07T09:28:59.240 INFO:teuthology.orchestra.run.smithi144.stdout:3 2023-10-07T09:28:59.241 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 msgr2=0x7f93840c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.241 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 0x7f93840c8050 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9374009160 tx=0x7f9374016000 comp rx=0 tx=0).stop 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 shutdown_connections 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9384159b20 0x7f93841534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93840ce6d0 0x7f93840c8050 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 --2- 172.21.15.144:0/1619242807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9384159750 0x7f93840c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 >> 172.21.15.144:0/1619242807 conn(0x7f9384063a10 msgr2=0x7f93841564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 shutdown_connections 2023-10-07T09:28:59.242 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.240+0000 7f938a9ff700 1 -- 172.21.15.144:0/1619242807 wait complete. 2023-10-07T09:28:59.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 4ae85342-f034-4c80-85fc-58bf598250cf 4 2023-10-07T09:28:59.446 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.445+0000 7fca235ff700 1 Processor -- start 2023-10-07T09:28:59.446 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.445+0000 7fca235ff700 1 -- start start 2023-10-07T09:28:59.449 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.446+0000 7fca235ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca1c159750 0x7fca1c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.449 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.446+0000 7fca235ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.449 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.446+0000 7fca235ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca1c0ce6d0 0x7fca1c0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.449 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.446+0000 7fca235ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c0cbf20 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.449 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca235ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c0cc200 con 0x7fca1c159750 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca235ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0cc4e0 con 0x7fca1c159b20 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca2139b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca1c159750 0x7fca1c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:57852/0 (socket says 172.21.15.144:57852) 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 -- 172.21.15.144:0/1888740454 learned_addr learned my addr 172.21.15.144:0/1888740454 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca2139b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca1c159750 0x7fca1c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44792/0 (socket says 172.21.15.144:44792) 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca21b9c700 1 --2- 172.21.15.144:0/1888740454 >> v2:172.21.15.29:3300/0 conn(0x7fca1c0ce6d0 0x7fca1c0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 -- 172.21.15.144:0/1888740454 >> v2:172.21.15.144:3300/0 conn(0x7fca1c159750 msgr2=0x7fca1c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 --2- 172.21.15.144:0/1888740454 >> v2:172.21.15.144:3300/0 conn(0x7fca1c159750 0x7fca1c0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 -- 172.21.15.144:0/1888740454 >> v2:172.21.15.29:3300/0 conn(0x7fca1c0ce6d0 msgr2=0x7fca1c0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 --2- 172.21.15.144:0/1888740454 >> v2:172.21.15.29:3300/0 conn(0x7fca1c0ce6d0 0x7fca1c0ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.447+0000 7fca20b9a700 1 -- 172.21.15.144:0/1888740454 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c0cc7c0 con 0x7fca1c159b20 2023-10-07T09:28:59.450 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.448+0000 7fca20b9a700 1 --2- 172.21.15.144:0/1888740454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0ce190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fca1c089840 tx=0x7fca0c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b522636874d1fcb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:59.451 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.448+0000 7fca13fff700 1 -- 172.21.15.144:0/1888740454 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00b640 con 0x7fca1c159b20 2023-10-07T09:28:59.451 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.448+0000 7fca13fff700 1 -- 172.21.15.144:0/1888740454 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c00b7a0 con 0x7fca1c159b20 2023-10-07T09:28:59.451 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.448+0000 7fca13fff700 1 -- 172.21.15.144:0/1888740454 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00bb60 con 0x7fca1c159b20 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.448+0000 7fca235ff700 1 -- 172.21.15.144:0/1888740454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 msgr2=0x7fca1c0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.448+0000 7fca235ff700 1 --2- 172.21.15.144:0/1888740454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0ce190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fca1c089840 tx=0x7fca0c009250 comp rx=0 tx=0).stop 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 -- 172.21.15.144:0/1888740454 shutdown_connections 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 --2- 172.21.15.144:0/1888740454 >> v2:172.21.15.144:3300/0 conn(0x7fca1c159750 0x7fca1c0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 --2- 172.21.15.144:0/1888740454 >> v2:172.21.15.29:3300/0 conn(0x7fca1c0ce6d0 0x7fca1c0ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 --2- 172.21.15.144:0/1888740454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0ce190 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 -- 172.21.15.144:0/1888740454 >> 172.21.15.144:0/1888740454 conn(0x7fca1c063a10 msgr2=0x7fca1c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 -- 172.21.15.144:0/1888740454 shutdown_connections 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.449+0000 7fca235ff700 1 -- 172.21.15.144:0/1888740454 wait complete. 2023-10-07T09:28:59.452 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.450+0000 7fca235ff700 1 Processor -- start 2023-10-07T09:28:59.453 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.450+0000 7fca235ff700 1 -- start start 2023-10-07T09:28:59.455 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca235ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 0x7fca1c14b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.455 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca235ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c159750 0x7fca1c0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.455 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca235ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.455 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca235ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0b53c0 con 0x7fca1c159750 2023-10-07T09:28:59.455 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca235ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c15ac00 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca235ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c15aee0 con 0x7fca1c159b20 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.451+0000 7fca2139b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 0x7fca1c14b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 0x7fca1c14b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44794/0 (socket says 172.21.15.144:44794) 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 -- 172.21.15.144:0/2095524836 learned_addr learned my addr 172.21.15.144:0/2095524836 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca21b9c700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca20b9a700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c159750 0x7fca1c0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 -- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 msgr2=0x7fca1c0b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 -- 172.21.15.144:0/2095524836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c159750 msgr2=0x7fca1c0b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c159750 0x7fca1c0b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.456 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 -- 172.21.15.144:0/2095524836 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0094f0 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.452+0000 7fca2139b700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 0x7fca1c14b2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fca1800ec80 tx=0x7fca18009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1800c8e0 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1800ca40 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c15b1c0 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1800cdc0 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c152760 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fca18013900 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.453+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fca1801cc30 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.457 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.454+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c04c3c0 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.458 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.457+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fca18021060 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.544 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.543+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4ae85342-f034-4c80-85fc-58bf598250cf", "id": 4} v 0) v1 -- 0x7fca1c064040 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.548 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.547+0000 7fca117fa700 1 -- 172.21.15.144:0/2095524836 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4ae85342-f034-4c80-85fc-58bf598250cf", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fca18017070 con 0x7fca1c0ce6d0 2023-10-07T09:28:59.548 INFO:teuthology.orchestra.run.smithi144.stdout:4 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.548+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 msgr2=0x7fca1c14b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.548+0000 7fca235ff700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 0x7fca1c14b2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fca1800ec80 tx=0x7fca18009120 comp rx=0 tx=0).stop 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 shutdown_connections 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca1c159b20 0x7fca1c0b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca1c0ce6d0 0x7fca1c14b2b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 --2- 172.21.15.144:0/2095524836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca1c159750 0x7fca1c0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 >> 172.21.15.144:0/2095524836 conn(0x7fca1c063a10 msgr2=0x7fca1c1561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 shutdown_connections 2023-10-07T09:28:59.550 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.549+0000 7fca235ff700 1 -- 172.21.15.144:0/2095524836 wait complete. 2023-10-07T09:28:59.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 84d41a87-cf09-485a-be54-e758964f42d9 5 2023-10-07T09:28:59.750 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.749+0000 7f502dfff700 1 Processor -- start 2023-10-07T09:28:59.751 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.750+0000 7f502dfff700 1 -- start start 2023-10-07T09:28:59.754 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f502dfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5028159b20 0x7f50280d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f502dfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280cbf20 con 0x7f5028159750 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f502dfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280cc200 con 0x7f50280ce8a0 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f502dfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280cc4e0 con 0x7f5028159b20 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f501ffff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5028159b20 0x7f50280d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f501ffff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5028159b20 0x7f50280d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44802/0 (socket says 172.21.15.144:44802) 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f501ffff700 1 -- 172.21.15.144:0/3113462513 learned_addr learned my addr 172.21.15.144:0/3113462513 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f5027fff700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f50280ce8a0 0x7f50280c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:43820/0 (socket says 172.21.15.144:43820) 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.751+0000 7f5026ffd700 1 -- 172.21.15.144:0/3113462513 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638915818 0 0) 0x7f50280cc200 con 0x7f50280ce8a0 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f5026ffd700 1 -- 172.21.15.144:0/3113462513 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5014003570 con 0x7f50280ce8a0 2023-10-07T09:28:59.755 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f5026ffd700 1 -- 172.21.15.144:0/3113462513 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192455820 0 0) 0x7f50280cbf20 con 0x7f5028159750 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f501ffff700 1 -- 172.21.15.144:0/3113462513 >> v1:172.21.15.144:6789/0 conn(0x7f50280ce8a0 legacy=0x7f50280c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f501ffff700 1 -- 172.21.15.144:0/3113462513 >> v1:172.21.15.29:6789/0 conn(0x7f5028159750 legacy=0x7f50280ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f501ffff700 1 -- 172.21.15.144:0/3113462513 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50280cc7c0 con 0x7f5028159b20 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f501ffff700 1 --2- 172.21.15.144:0/3113462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5028159b20 0x7f50280d10a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50100099b0 tx=0x7f5010009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29517bb023d97e8e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f5026ffd700 1 -- 172.21.15.144:0/3113462513 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010009ea0 con 0x7f5028159b20 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f5026ffd700 1 -- 172.21.15.144:0/3113462513 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501000b860 con 0x7f5028159b20 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.752+0000 7f5026ffd700 1 -- 172.21.15.144:0/3113462513 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000bbe0 con 0x7f5028159b20 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 -- 172.21.15.144:0/3113462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5028159b20 msgr2=0x7f50280d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 --2- 172.21.15.144:0/3113462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5028159b20 0x7f50280d10a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50100099b0 tx=0x7f5010009250 comp rx=0 tx=0).stop 2023-10-07T09:28:59.756 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 -- 172.21.15.144:0/3113462513 shutdown_connections 2023-10-07T09:28:59.757 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 --2- 172.21.15.144:0/3113462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5028159b20 0x7f50280d10a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.757 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 -- 172.21.15.144:0/3113462513 >> 172.21.15.144:0/3113462513 conn(0x7f5028063a10 msgr2=0x7f50280d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:59.757 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 -- 172.21.15.144:0/3113462513 shutdown_connections 2023-10-07T09:28:59.761 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.753+0000 7f502dfff700 1 -- 172.21.15.144:0/3113462513 wait complete. 2023-10-07T09:28:59.764 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.755+0000 7f502dfff700 1 Processor -- start 2023-10-07T09:28:59.764 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.755+0000 7f502dfff700 1 -- start start 2023-10-07T09:28:59.764 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f502dfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 0x7f50280b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.764 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f502dfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5028159750 0x7f502815bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f502dfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5028159b20 0x7f502813ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f502dfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502815c550 con 0x7f5028159b20 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f502dfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502815c830 con 0x7f50280ce8a0 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f502dfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502815cb10 con 0x7f5028159750 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f50277fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 0x7f50280b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f50277fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 0x7f50280b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44806/0 (socket says 172.21.15.144:44806) 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f501ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5028159750 0x7f502815bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f50277fe700 1 -- 172.21.15.144:0/2122327108 learned_addr learned my addr 172.21.15.144:0/2122327108 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.757+0000 7f5027fff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5028159b20 0x7f502813ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:28:59.765 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501ffff700 1 -- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 msgr2=0x7f50280b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501ffff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 0x7f50280b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501ffff700 1 -- 172.21.15.144:0/2122327108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5028159b20 msgr2=0x7f502813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501ffff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5028159b20 0x7f502813ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501ffff700 1 -- 172.21.15.144:0/2122327108 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50100094f0 con 0x7f5028159750 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f50277fe700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 0x7f50280b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501ffff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5028159750 0x7f502815bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5028089840 tx=0x7f5010017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000bbe0 con 0x7f5028159750 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5010011420 con 0x7f5028159750 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502815cdf0 con 0x7f5028159750 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010011800 con 0x7f5028159750 2023-10-07T09:28:59.766 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.758+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50280b38a0 con 0x7f5028159750 2023-10-07T09:28:59.767 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.759+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f501002e400 con 0x7f5028159750 2023-10-07T09:28:59.767 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.759+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f5010011e30 con 0x7f5028159750 2023-10-07T09:28:59.767 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.760+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50280b3cb0 con 0x7f5028159750 2023-10-07T09:28:59.767 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.764+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f50280b3cb0 con 0x7f5028159750 2023-10-07T09:28:59.849 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.848+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "84d41a87-cf09-485a-be54-e758964f42d9", "id": 5} v 0) v1 -- 0x7f5028156b10 con 0x7f5028159750 2023-10-07T09:28:59.853 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.852+0000 7f501f7fe700 1 -- 172.21.15.144:0/2122327108 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "84d41a87-cf09-485a-be54-e758964f42d9", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5010015020 con 0x7f5028159750 2023-10-07T09:28:59.853 INFO:teuthology.orchestra.run.smithi144.stdout:5 2023-10-07T09:28:59.855 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.853+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5028159750 msgr2=0x7f502815bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:28:59.855 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.853+0000 7f502dfff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5028159750 0x7f502815bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5028089840 tx=0x7f5010017000 comp rx=0 tx=0).stop 2023-10-07T09:28:59.855 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 shutdown_connections 2023-10-07T09:28:59.855 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5028159b20 0x7f502813ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.855 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50280ce8a0 0x7f50280b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.856 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 --2- 172.21.15.144:0/2122327108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5028159750 0x7f502815bb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:28:59.856 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 >> 172.21.15.144:0/2122327108 conn(0x7f5028063a10 msgr2=0x7f50280ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:28:59.856 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 shutdown_connections 2023-10-07T09:28:59.856 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:28:59.854+0000 7f502dfff700 1 -- 172.21.15.144:0/2122327108 wait complete. 2023-10-07T09:28:59.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new ce23a7b2-9566-436e-a3ba-12829f83c125 6 2023-10-07T09:29:00.061 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.060+0000 7fb9f6bff700 1 Processor -- start 2023-10-07T09:29:00.061 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.060+0000 7fb9f6bff700 1 -- start start 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9f6bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9f0159750 0x7fb9f00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9f6bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9f0159b20 0x7fb9f00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9f6bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f00cbf20 con 0x7fb9f00ce6d0 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9f6bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f00cc200 con 0x7fb9f0159750 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9f6bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f00cc4e0 con 0x7fb9f0159b20 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9effff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9f0159b20 0x7fb9f00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9effff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9f0159b20 0x7fb9f00ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44822/0 (socket says 172.21.15.144:44822) 2023-10-07T09:29:00.063 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.061+0000 7fb9effff700 1 -- 172.21.15.144:0/1248136989 learned_addr learned my addr 172.21.15.144:0/1248136989 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:29:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.062+0000 7fb9f499b700 1 --2- 172.21.15.144:0/1248136989 >> v2:172.21.15.29:3300/0 conn(0x7fb9f0159750 0x7fb9f00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.062+0000 7fb9effff700 1 -- 172.21.15.144:0/1248136989 >> v2:172.21.15.29:3300/0 conn(0x7fb9f0159750 msgr2=0x7fb9f00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.062+0000 7fb9effff700 1 --2- 172.21.15.144:0/1248136989 >> v2:172.21.15.29:3300/0 conn(0x7fb9f0159750 0x7fb9f00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.062+0000 7fb9effff700 1 -- 172.21.15.144:0/1248136989 >> v1:172.21.15.29:6789/0 conn(0x7fb9f00ce6d0 legacy=0x7fb9f015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.062+0000 7fb9effff700 1 -- 172.21.15.144:0/1248136989 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00cc7c0 con 0x7fb9f0159b20 2023-10-07T09:29:00.064 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.062+0000 7fb9effff700 1 --2- 172.21.15.144:0/1248136989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159b20 0x7fb9f00ce190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0088a70 tx=0x7fb9dc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc245f223f4c6c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9ef7fe700 1 -- 172.21.15.144:0/1248136989 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc009e70 con 0x7fb9f0159b20 2023-10-07T09:29:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9ef7fe700 1 -- 172.21.15.144:0/1248136989 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00b860 con 0x7fb9f0159b20 2023-10-07T09:29:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9ef7fe700 1 -- 172.21.15.144:0/1248136989 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00bc00 con 0x7fb9f0159b20 2023-10-07T09:29:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 -- 172.21.15.144:0/1248136989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159b20 msgr2=0x7fb9f00ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/1248136989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159b20 0x7fb9f00ce190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0088a70 tx=0x7fb9dc009250 comp rx=0 tx=0).stop 2023-10-07T09:29:00.065 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 -- 172.21.15.144:0/1248136989 shutdown_connections 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/1248136989 >> v2:172.21.15.29:3300/0 conn(0x7fb9f0159750 0x7fb9f00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/1248136989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159b20 0x7fb9f00ce190 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 -- 172.21.15.144:0/1248136989 >> 172.21.15.144:0/1248136989 conn(0x7fb9f0063a10 msgr2=0x7fb9f00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 -- 172.21.15.144:0/1248136989 shutdown_connections 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.063+0000 7fb9f6bff700 1 -- 172.21.15.144:0/1248136989 wait complete. 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.065+0000 7fb9f6bff700 1 Processor -- start 2023-10-07T09:29:00.066 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.065+0000 7fb9f6bff700 1 -- start start 2023-10-07T09:29:00.068 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f6bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f00ce6d0 0x7fb9f01531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.068 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f6bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 0x7fb9f016ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9f0159b20 0x7fb9f0169bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f6bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f016ac00 con 0x7fb9f00ce6d0 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f6bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f0153720 con 0x7fb9f0159750 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f015a0b0 con 0x7fb9f0159b20 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9effff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 0x7fb9f016ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9effff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 0x7fb9f016ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44838/0 (socket says 172.21.15.144:44838) 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9effff700 1 -- 172.21.15.144:0/2444715010 learned_addr learned my addr 172.21.15.144:0/2444715010 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f499b700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f00ce6d0 0x7fb9f01531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9f519c700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9f0159b20 0x7fb9f0169bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9effff700 1 -- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9f0159b20 msgr2=0x7fb9f0169bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9effff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9f0159b20 0x7fb9f0169bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.066+0000 7fb9effff700 1 -- 172.21.15.144:0/2444715010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f00ce6d0 msgr2=0x7fb9f01531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9effff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f00ce6d0 0x7fb9f01531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9effff700 1 -- 172.21.15.144:0/2444715010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0094f0 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9effff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 0x7fb9f016ada0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00cc620 tx=0x7fb9dc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00b9d0 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00bb30 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc025670 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f016a640 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.067+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f015a350 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.068+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb9dc023070 con 0x7fb9f0159750 2023-10-07T09:29:00.070 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.068+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fb9dc025ce0 con 0x7fb9f0159750 2023-10-07T09:29:00.071 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.069+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f0159ef0 con 0x7fb9f0159750 2023-10-07T09:29:00.072 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.071+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb9dc015020 con 0x7fb9f0159750 2023-10-07T09:29:00.145 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.144+0000 7f9b3986c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:29:00.158 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:00.157+0000 7f31b186c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:29:00.183 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.182+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ce23a7b2-9566-436e-a3ba-12829f83c125", "id": 6} v 0) v1 -- 0x7fb9f015a570 con 0x7fb9f0159750 2023-10-07T09:29:00.187 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.186+0000 7fb9ecff9700 1 -- 172.21.15.144:0/2444715010 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce23a7b2-9566-436e-a3ba-12829f83c125", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb9dc011470 con 0x7fb9f0159750 2023-10-07T09:29:00.187 INFO:teuthology.orchestra.run.smithi144.stdout:6 2023-10-07T09:29:00.188 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 msgr2=0x7fb9f016ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 0x7fb9f016ada0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00cc620 tx=0x7fb9dc017000 comp rx=0 tx=0).stop 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 shutdown_connections 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9f0159b20 0x7fb9f0169bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f00ce6d0 0x7fb9f01531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 --2- 172.21.15.144:0/2444715010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9f0159750 0x7fb9f016ada0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 >> 172.21.15.144:0/2444715010 conn(0x7fb9f0063a10 msgr2=0x7fb9f00ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 shutdown_connections 2023-10-07T09:29:00.189 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.187+0000 7fb9f6bff700 1 -- 172.21.15.144:0/2444715010 wait complete. 2023-10-07T09:29:00.205 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 41a28292-240f-45fc-b073-c2e8a14106bd 7 2023-10-07T09:29:00.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.237+0000 7f9b3986c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:29:00.256 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:00.255+0000 7f31b186c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:29:00.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.390+0000 7fa75f3ff700 1 Processor -- start 2023-10-07T09:29:00.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.390+0000 7fa75f3ff700 1 -- start start 2023-10-07T09:29:00.396 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.391+0000 7fa75f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159750 0x7fa7580d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.396 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.391+0000 7fa75f3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa758159b20 0x7fa7580ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.391+0000 7fa75f3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580cbf20 con 0x7fa7580ce6d0 2023-10-07T09:29:00.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75f3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7580cc200 con 0x7fa758159b20 2023-10-07T09:29:00.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580cc4e0 con 0x7fa758159750 2023-10-07T09:29:00.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d19b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159750 0x7fa7580d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d99c700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa7580ce6d0 0x7fa75815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:43828/0 (socket says 172.21.15.144:43828) 2023-10-07T09:29:00.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d99c700 1 -- 172.21.15.144:0/2165814034 learned_addr learned my addr 172.21.15.144:0/2165814034 (peer_addr_for_me v1:172.21.15.144:0/0) 2023-10-07T09:29:00.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75c99a700 1 --2- 172.21.15.144:0/2165814034 >> v2:172.21.15.29:3300/0 conn(0x7fa758159b20 0x7fa7580ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa74ffff700 1 -- 172.21.15.144:0/2165814034 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910909802 0 0) 0x7fa7580cbf20 con 0x7fa7580ce6d0 2023-10-07T09:29:00.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa74ffff700 1 -- 172.21.15.144:0/2165814034 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa744003570 con 0x7fa7580ce6d0 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d19b700 1 -- 172.21.15.144:0/2165814034 >> v2:172.21.15.29:3300/0 conn(0x7fa758159b20 msgr2=0x7fa7580ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d19b700 1 --2- 172.21.15.144:0/2165814034 >> v2:172.21.15.29:3300/0 conn(0x7fa758159b20 0x7fa7580ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d19b700 1 -- 172.21.15.144:0/2165814034 >> v1:172.21.15.144:6789/0 conn(0x7fa7580ce6d0 legacy=0x7fa75815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75d19b700 1 -- 172.21.15.144:0/2165814034 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580cc7c0 con 0x7fa758159750 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.392+0000 7fa75c99a700 1 --2- 172.21.15.144:0/2165814034 >> v2:172.21.15.29:3300/0 conn(0x7fa758159b20 0x7fa7580ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.393+0000 7fa75d19b700 1 --2- 172.21.15.144:0/2165814034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159750 0x7fa7580d10a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098d0 tx=0x7fa754009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7db21cfc7a90680 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:00.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.393+0000 7fa74ffff700 1 -- 172.21.15.144:0/2165814034 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754005c10 con 0x7fa758159750 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.393+0000 7fa74ffff700 1 -- 172.21.15.144:0/2165814034 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa754005d70 con 0x7fa758159750 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.393+0000 7fa74ffff700 1 -- 172.21.15.144:0/2165814034 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754006260 con 0x7fa758159750 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.393+0000 7fa75f3ff700 1 -- 172.21.15.144:0/2165814034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159750 msgr2=0x7fa7580d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.393+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/2165814034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159750 0x7fa7580d10a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098d0 tx=0x7fa754009120 comp rx=0 tx=0).stop 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.394+0000 7fa75f3ff700 1 -- 172.21.15.144:0/2165814034 shutdown_connections 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.394+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/2165814034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159750 0x7fa7580d10a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.394+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/2165814034 >> v2:172.21.15.29:3300/0 conn(0x7fa758159b20 0x7fa7580ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.394+0000 7fa75f3ff700 1 -- 172.21.15.144:0/2165814034 >> 172.21.15.144:0/2165814034 conn(0x7fa758063a10 msgr2=0x7fa7580d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.394+0000 7fa75f3ff700 1 -- 172.21.15.144:0/2165814034 shutdown_connections 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.394+0000 7fa75f3ff700 1 -- 172.21.15.144:0/2165814034 wait complete. 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.395+0000 7fa75f3ff700 1 Processor -- start 2023-10-07T09:29:00.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.395+0000 7fa75f3ff700 1 -- start start 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75f3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 0x7fa75814c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75f3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa758159750 0x7fa758158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159b20 0x7fa7580b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75f3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa758152b20 con 0x7fa758159750 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75f3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7581585b0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa758158f80 con 0x7fa758159b20 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75d19b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 0x7fa75814c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75d19b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 0x7fa75814c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44844/0 (socket says 172.21.15.144:44844) 2023-10-07T09:29:00.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.396+0000 7fa75d19b700 1 -- 172.21.15.144:0/3143666976 learned_addr learned my addr 172.21.15.144:0/3143666976 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d99c700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159b20 0x7fa7580b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75c99a700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa758159750 0x7fa758158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d19b700 1 -- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159b20 msgr2=0x7fa7580b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d19b700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159b20 0x7fa7580b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d19b700 1 -- 172.21.15.144:0/3143666976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa758159750 msgr2=0x7fa758158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d19b700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa758159750 0x7fa758158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d19b700 1 -- 172.21.15.144:0/3143666976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa748009420 con 0x7fa7580ce6d0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa75d19b700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 0x7fa75814c040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa754009ec0 tx=0x7fa754016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75401d6b0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75401d830 con 0x7fa7580ce6d0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.397+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75401dbf0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.398+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7540093c0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.398+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7580b5820 con 0x7fa7580ce6d0 2023-10-07T09:29:00.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.398+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa75401b700 con 0x7fa7580ce6d0 2023-10-07T09:29:00.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.398+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7fa754026ab0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.399+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75804c3c0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.401+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa75401bb40 con 0x7fa7580ce6d0 2023-10-07T09:29:00.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.408+0000 7f9b3986c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:29:00.440 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:00.439+0000 7f31b186c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:29:00.497 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.496+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "41a28292-240f-45fc-b073-c2e8a14106bd", "id": 7} v 0) v1 -- 0x7fa758064040 con 0x7fa7580ce6d0 2023-10-07T09:29:00.497 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.497+0000 7f9b3986c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:29:00.504 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.502+0000 7fa74d7fa700 1 -- 172.21.15.144:0/3143666976 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41a28292-240f-45fc-b073-c2e8a14106bd", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa75401bce0 con 0x7fa7580ce6d0 2023-10-07T09:29:00.504 INFO:teuthology.orchestra.run.smithi144.stdout:7 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 msgr2=0x7fa75814c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 0x7fa75814c040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa754009ec0 tx=0x7fa754016000 comp rx=0 tx=0).stop 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 shutdown_connections 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa758159b20 0x7fa7580b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ce6d0 0x7fa75814c040 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 --2- 172.21.15.144:0/3143666976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa758159750 0x7fa758158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 >> 172.21.15.144:0/3143666976 conn(0x7fa758063a10 msgr2=0x7fa7580ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 shutdown_connections 2023-10-07T09:29:00.505 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-07T09:29:00.504+0000 7fa75f3ff700 1 -- 172.21.15.144:0/3143666976 wait complete. 2023-10-07T09:29:00.521 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-07T09:29:00.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-07T09:29:00.528 INFO:tasks.ceph.osd.0:Started 2023-10-07T09:29:00.528 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-07T09:29:00.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-07T09:29:00.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:00.529+0000 7f31b186c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:29:00.535 INFO:tasks.ceph.osd.1:Started 2023-10-07T09:29:00.535 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-07T09:29:00.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-07T09:29:00.540 INFO:tasks.ceph.osd.2:Started 2023-10-07T09:29:00.540 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-07T09:29:00.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-07T09:29:00.547 INFO:tasks.ceph.osd.3:Started 2023-10-07T09:29:00.547 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-07T09:29:00.547 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-07T09:29:00.554 INFO:tasks.ceph.osd.4:Started 2023-10-07T09:29:00.554 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-07T09:29:00.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-07T09:29:00.560 INFO:tasks.ceph.osd.5:Started 2023-10-07T09:29:00.560 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-07T09:29:00.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-07T09:29:00.567 INFO:tasks.ceph.osd.6:Started 2023-10-07T09:29:00.567 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-07T09:29:00.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-07T09:29:00.573 INFO:tasks.ceph.osd.7:Started 2023-10-07T09:29:00.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:00.774 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.773+0000 7f9b3986c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:29:00.809 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:00.808+0000 7f31b186c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:29:00.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.842+0000 7f84f61ff700 1 Processor -- start 2023-10-07T09:29:00.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.842+0000 7f84f61ff700 1 -- start start 2023-10-07T09:29:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.843+0000 7f84f61ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f0069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.843+0000 7f84f61ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f0069400 con 0x7f84f00b0770 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84f61ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f0069dd0 con 0x7f84f0159300 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84f61ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f0068b10 con 0x7f84f015da10 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84eeffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84f00b0770 0x7f84f015d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41824/0 (socket says 172.21.15.29:41824) 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84eeffd700 1 -- 172.21.15.29:0/1301426696 learned_addr learned my addr 172.21.15.29:0/1301426696 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84effff700 1 --2- 172.21.15.29:0/1301426696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f0069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008389325 0 0) 0x7f84f0069400 con 0x7f84f00b0770 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d8003570 con 0x7f84f00b0770 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859259721 0 0) 0x7f84f0069dd0 con 0x7f84f0159300 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f0069400 con 0x7f84f0159300 2023-10-07T09:29:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84effff700 1 -- 172.21.15.29:0/1301426696 >> v1:172.21.15.144:6789/0 conn(0x7f84f0159300 legacy=0x7f84f00b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84effff700 1 -- 172.21.15.29:0/1301426696 >> v1:172.21.15.29:6789/0 conn(0x7f84f00b0770 legacy=0x7f84f015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.844+0000 7f84effff700 1 -- 172.21.15.29:0/1301426696 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f0068db0 con 0x7f84f015da10 2023-10-07T09:29:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.845+0000 7f84effff700 1 --2- 172.21.15.29:0/1301426696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f0069890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f84f0069f60 tx=0x7f84e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecafc9255ec21a42 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.845+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e8016070 con 0x7f84f015da10 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.845+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e800bab0 con 0x7f84f015da10 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.845+0000 7f84ee7fc700 1 -- 172.21.15.29:0/1301426696 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e800e070 con 0x7f84f015da10 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 -- 172.21.15.29:0/1301426696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 msgr2=0x7f84f0069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 --2- 172.21.15.29:0/1301426696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f0069890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f84f0069f60 tx=0x7f84e8009780 comp rx=0 tx=0).stop 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 -- 172.21.15.29:0/1301426696 shutdown_connections 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 --2- 172.21.15.29:0/1301426696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f0069890 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 -- 172.21.15.29:0/1301426696 >> 172.21.15.29:0/1301426696 conn(0x7f84f0067610 msgr2=0x7f84f0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 -- 172.21.15.29:0/1301426696 shutdown_connections 2023-10-07T09:29:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.846+0000 7f84f61ff700 1 -- 172.21.15.29:0/1301426696 wait complete. 2023-10-07T09:29:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.847+0000 7f84f61ff700 1 Processor -- start 2023-10-07T09:29:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.847+0000 7f84f61ff700 1 -- start start 2023-10-07T09:29:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.848+0000 7f84f61ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84f00b0770 0x7f84f00ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.848+0000 7f84f61ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84f0159300 0x7f84f0168bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.848+0000 7f84f61ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f00c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.848+0000 7f84f61ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00ae410 con 0x7f84f0159300 2023-10-07T09:29:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.848+0000 7f84f61ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00aede0 con 0x7f84f00b0770 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.848+0000 7f84f61ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c7f20 con 0x7f84f015da10 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84ef7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84f00b0770 0x7f84f00ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84eeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84f0159300 0x7f84f0168bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f00c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84ef7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84f00b0770 0x7f84f00ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:47052/0 (socket says 172.21.15.29:47052) 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84ef7fe700 1 -- 172.21.15.29:0/1454880647 learned_addr learned my addr 172.21.15.29:0/1454880647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 -- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84f00b0770 msgr2=0x7f84f00ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84f00b0770 0x7f84f00ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 -- 172.21.15.29:0/1454880647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84f0159300 msgr2=0x7f84f0168bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84f0159300 0x7f84f0168bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 -- 172.21.15.29:0/1454880647 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8009a20 con 0x7f84f015da10 2023-10-07T09:29:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.849+0000 7f84effff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f00c75b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f84e8015000 tx=0x7f84e8018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e8012d40 con 0x7f84f015da10 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e8027470 con 0x7f84f015da10 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e8027830 con 0x7f84f015da10 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f00c8200 con 0x7f84f015da10 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f00ae0a0 con 0x7f84f015da10 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f84e8024070 con 0x7f84f015da10 2023-10-07T09:29:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.850+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f84e8027ba0 con 0x7f84f015da10 2023-10-07T09:29:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.851+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f00c8060 con 0x7f84f015da10 2023-10-07T09:29:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.853+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f84f00c8060 con 0x7f84f015da10 2023-10-07T09:29:00.876 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.875+0000 7f9b3986c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:29:00.907 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:00.906+0000 7f31b186c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:29:00.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.927+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84f004c3c0 con 0x7f84f015da10 2023-10-07T09:29:00.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.928+0000 7f84d77fe700 1 -- 172.21.15.29:0/1454880647 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f84e802c020 con 0x7f84f015da10 2023-10-07T09:29:00.929 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:00.929 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:00.498643+0000","last_up_change":"0.000000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:00.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 msgr2=0x7f84f00c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:00.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f00c75b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f84e8015000 tx=0x7f84e8018000 comp rx=0 tx=0).stop 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 shutdown_connections 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84f015da10 0x7f84f00c75b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84f00b0770 0x7f84f00ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 --2- 172.21.15.29:0/1454880647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84f0159300 0x7f84f0168bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 >> 172.21.15.29:0/1454880647 conn(0x7f84f0067610 msgr2=0x7f84f0155bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.929+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 shutdown_connections 2023-10-07T09:29:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:00.930+0000 7f84f61ff700 1 -- 172.21.15.29:0/1454880647 wait complete. 2023-10-07T09:29:00.948 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-07T09:29:00.948 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-07T09:29:00.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:00.976+0000 7f9b3986c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:29:01.003 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:01.002+0000 7f31b186c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:29:01.088 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:01.087+0000 7f9b3986c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:29:01.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:01.129+0000 7f31b186c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:29:01.232 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-07T09:29:01.231+0000 7fa89d2fe540 -1 Falling back to public interface 2023-10-07T09:29:01.259 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-07T09:29:01.258+0000 7fe938589540 -1 Falling back to public interface 2023-10-07T09:29:01.260 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-07T09:29:01.259+0000 7f5a8d5d0540 -1 Falling back to public interface 2023-10-07T09:29:01.263 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-07T09:29:01.263+0000 7f1eb1f17540 -1 Falling back to public interface 2023-10-07T09:29:01.268 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-07T09:29:01.268+0000 7f6cd635e540 -1 Falling back to public interface 2023-10-07T09:29:01.272 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-07T09:29:01.271+0000 7fc9eaf9d540 -1 Falling back to public interface 2023-10-07T09:29:01.272 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-07T09:29:01.272+0000 7f837d86f540 -1 Falling back to public interface 2023-10-07T09:29:01.276 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-07T09:29:01.276+0000 7f6ccd5fd700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6823/0: (98) Address already in use 2023-10-07T09:29:01.283 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-07T09:29:01.282+0000 7f4cd8ee6540 -1 Falling back to public interface 2023-10-07T09:29:01.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:01.324+0000 7f9b3986c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:29:01.360 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:01.359+0000 7f31b186c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:29:01.427 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:01.426+0000 7f9b3986c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:29:01.457 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:01.456+0000 7f31b186c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:29:01.754 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:01.781 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:01.780+0000 7f9b3986c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:29:01.805 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:01.805+0000 7f31b186c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:29:01.874 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:01.873+0000 7f9b3986c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:29:01.894 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:01.893+0000 7f31b186c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:29:01.927 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.926+0000 7f6cedbff700 1 Processor -- start 2023-10-07T09:29:01.928 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.927+0000 7f6cedbff700 1 -- start start 2023-10-07T09:29:01.931 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6cedbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ce8159870 0x7f6ce814fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:01.931 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6cedbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ce8159c40 0x7f6ce8150040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:01.931 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6cedbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce8157290 con 0x7f6ce80afd80 2023-10-07T09:29:01.931 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6cedbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce8150580 con 0x7f6ce8159c40 2023-10-07T09:29:01.931 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6cedbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce8157570 con 0x7f6ce8159870 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6ce6ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ce8159c40 0x7f6ce8150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6ce7fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6ce80afd80 0x7f6ce80b0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41838/0 (socket says 172.21.15.29:41838) 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6ce7fff700 1 -- 172.21.15.29:0/1700425304 learned_addr learned my addr 172.21.15.29:0/1700425304 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.929+0000 7f6ce77fe700 1 --2- 172.21.15.29:0/1700425304 >> v2:172.21.15.144:3300/0 conn(0x7f6ce8159870 0x7f6ce814fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce6ffd700 1 -- 172.21.15.29:0/1700425304 >> v2:172.21.15.144:3300/0 conn(0x7f6ce8159870 msgr2=0x7f6ce814fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce6ffd700 1 --2- 172.21.15.29:0/1700425304 >> v2:172.21.15.144:3300/0 conn(0x7f6ce8159870 0x7f6ce814fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce6ffd700 1 -- 172.21.15.29:0/1700425304 >> v1:172.21.15.29:6789/0 conn(0x7f6ce80afd80 legacy=0x7f6ce80b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce6ffd700 1 -- 172.21.15.29:0/1700425304 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce8157850 con 0x7f6ce8159c40 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce67fc700 1 -- 172.21.15.29:0/1700425304 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936810808 0 0) 0x7f6ce8157290 con 0x7f6ce80afd80 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce6ffd700 1 --2- 172.21.15.29:0/1700425304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce8159c40 0x7f6ce8150040 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80aecc0 tx=0x7f6cdc0080e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d188f0a9a4c6a17 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:01.932 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce67fc700 1 -- 172.21.15.29:0/1700425304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00b5a0 con 0x7f6ce8159c40 2023-10-07T09:29:01.933 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce67fc700 1 -- 172.21.15.29:0/1700425304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc00b700 con 0x7f6ce8159c40 2023-10-07T09:29:01.933 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.930+0000 7f6ce67fc700 1 -- 172.21.15.29:0/1700425304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00baa0 con 0x7f6ce8159c40 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 -- 172.21.15.29:0/1700425304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce8159c40 msgr2=0x7f6ce8150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 --2- 172.21.15.29:0/1700425304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce8159c40 0x7f6ce8150040 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80aecc0 tx=0x7f6cdc0080e0 comp rx=0 tx=0).stop 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 -- 172.21.15.29:0/1700425304 shutdown_connections 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 --2- 172.21.15.29:0/1700425304 >> v2:172.21.15.144:3300/0 conn(0x7f6ce8159870 0x7f6ce814fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 --2- 172.21.15.29:0/1700425304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce8159c40 0x7f6ce8150040 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 -- 172.21.15.29:0/1700425304 >> 172.21.15.29:0/1700425304 conn(0x7f6ce8067610 msgr2=0x7f6ce8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 -- 172.21.15.29:0/1700425304 shutdown_connections 2023-10-07T09:29:01.934 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.931+0000 7f6cedbff700 1 -- 172.21.15.29:0/1700425304 wait complete. 2023-10-07T09:29:01.935 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.933+0000 7f6cedbff700 1 Processor -- start 2023-10-07T09:29:01.935 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.933+0000 7f6cedbff700 1 -- start start 2023-10-07T09:29:01.937 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6cedbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 0x7f6ce80c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:01.937 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6cedbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ce8159870 0x7f6ce816a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6cedbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ce8159c40 0x7f6ce80cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6cedbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80c79f0 con 0x7f6ce80afd80 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6cedbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce816aa50 con 0x7f6ce8159c40 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6cedbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80ca770 con 0x7f6ce8159870 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 0x7f6ce80c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 0x7f6ce80c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39116/0 (socket says 172.21.15.29:39116) 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 -- 172.21.15.29:0/2403104667 learned_addr learned my addr 172.21.15.29:0/2403104667 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce6ffd700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ce8159870 0x7f6ce816a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce7fff700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ce8159c40 0x7f6ce80cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 -- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ce8159870 msgr2=0x7f6ce816a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:01.938 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ce8159870 0x7f6ce816a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 -- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ce8159c40 msgr2=0x7f6ce80cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ce8159c40 0x7f6ce80cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.935+0000 7f6ce77fe700 1 -- 172.21.15.29:0/2403104667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc00a4d0 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6ce77fe700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 0x7f6ce80c6ea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8008ad0 tx=0x7f6cd800b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8010040 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8009ab0 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8009e30 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce80c9b60 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce80ca120 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.936+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6cd8016600 con 0x7f6ce80afd80 2023-10-07T09:29:01.939 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.937+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f6cd801fae0 con 0x7f6ce80afd80 2023-10-07T09:29:01.940 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.937+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce80caa50 con 0x7f6ce80afd80 2023-10-07T09:29:01.940 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:01.940+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6cd8016a40 con 0x7f6ce80afd80 2023-10-07T09:29:02.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.012+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ce804bb80 con 0x7f6ce80afd80 2023-10-07T09:29:02.014 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.013+0000 7f6ccf7fe700 1 -- 172.21.15.29:0/2403104667 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f6cd802e090 con 0x7f6ce80afd80 2023-10-07T09:29:02.014 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-07T09:29:02.014 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:00.498643+0000","last_up_change":"0.000000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:02.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 msgr2=0x7f6ce80c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:02.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 0x7f6ce80c6ea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8008ad0 tx=0x7f6cd800b780 comp rx=0 tx=0).stop 2023-10-07T09:29:02.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 shutdown_connections 2023-10-07T09:29:02.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ce8159c40 0x7f6ce80cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:02.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce80afd80 0x7f6ce80c6ea0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:02.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 --2- 172.21.15.29:0/2403104667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ce8159870 0x7f6ce816a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:02.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 >> 172.21.15.29:0/2403104667 conn(0x7f6ce8067610 msgr2=0x7f6ce8155290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:02.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 shutdown_connections 2023-10-07T09:29:02.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:02.014+0000 7f6cedbff700 1 -- 172.21.15.29:0/2403104667 wait complete. 2023-10-07T09:29:02.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-07T09:29:02.502 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:02.501+0000 7f31b186c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:29:02.514 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:02.513+0000 7f9b3986c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:29:02.878 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:02.877+0000 7f31b186c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:29:02.897 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:02.896+0000 7f9b3986c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:29:03.055 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:03.054+0000 7f31b186c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:29:03.079 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:03.078+0000 7f9b3986c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:29:03.490 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:03.488+0000 7f31b186c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:29:03.531 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:03.530+0000 7f9b3986c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:29:03.668 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:03.667+0000 7f31b186c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:29:03.714 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:03.713+0000 7f9b3986c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:29:03.875 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:03.874+0000 7f31b186c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:29:03.930 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:03.930+0000 7f9b3986c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:29:03.999 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:03.998+0000 7f31b186c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:29:04.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:29:04.094+0000 7f9b3986c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:29:04.800 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-07T09:29:04.800+0000 7fa89d2fe540 -1 osd.0 0 log_to_monitors true 2023-10-07T09:29:04.828 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-07T09:29:04.827+0000 7f5a8d5d0540 -1 osd.5 0 log_to_monitors true 2023-10-07T09:29:04.852 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-07T09:29:04.851+0000 7fe938589540 -1 osd.4 0 log_to_monitors true 2023-10-07T09:29:04.878 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-07T09:29:04.877+0000 7f6cd635e540 -1 osd.2 0 log_to_monitors true 2023-10-07T09:29:04.882 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-07T09:29:04.881+0000 7f837d86f540 -1 osd.3 0 log_to_monitors true 2023-10-07T09:29:05.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:05.020+0000 7f3170bfa700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-07T09:29:05.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:29:05.020+0000 7f3170bfa700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-07T09:29:05.050 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-07T09:29:05.049+0000 7f1eb1f17540 -1 osd.1 0 log_to_monitors true 2023-10-07T09:29:05.067 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-07T09:29:05.066+0000 7f4cd8ee6540 -1 osd.6 0 log_to_monitors true 2023-10-07T09:29:05.114 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-07T09:29:05.114+0000 7fc9eaf9d540 -1 osd.7 0 log_to_monitors true 2023-10-07T09:29:07.452 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-07T09:29:07.451+0000 7fa890df6700 -1 osd.0 0 waiting for initial osdmap 2023-10-07T09:29:07.463 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-07T09:29:07.462+0000 7fa889de8700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:07.777 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-07T09:29:07.776+0000 7f5a80ff6700 -1 osd.5 0 waiting for initial osdmap 2023-10-07T09:29:07.786 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-07T09:29:07.785+0000 7f5a79fe8700 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:08.219 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-07T09:29:08.218+0000 7f1ea59f6700 -1 osd.1 0 waiting for initial osdmap 2023-10-07T09:29:08.227 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-07T09:29:08.224+0000 7f1e9e9e8700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:08.259 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-07T09:29:08.258+0000 7f8372bf9700 -1 osd.3 0 waiting for initial osdmap 2023-10-07T09:29:08.264 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-07T09:29:08.263+0000 7f836a3e8700 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:08.314 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-07T09:29:08.314+0000 7f6cc9df6700 -1 osd.2 0 waiting for initial osdmap 2023-10-07T09:29:08.320 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-07T09:29:08.319+0000 7f6cc2de8700 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:08.835 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:09.011 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.010+0000 7faadcbff700 1 Processor -- start 2023-10-07T09:29:09.011 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.010+0000 7faadcbff700 1 -- start start 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.011+0000 7faadcbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 0x7faad815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.011+0000 7faadcbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faad80c5360 0x7faad80ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.011+0000 7faadcbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad80cba90 con 0x7faad815da30 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faadcbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad815dfb0 con 0x7faad80c5360 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faadcbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad815e290 con 0x7faad81598d0 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faad659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 0x7faad815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faad659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 0x7faad815d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:39622/0 (socket says 172.21.15.29:39622) 2023-10-07T09:29:09.013 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faad659c700 1 -- 172.21.15.29:0/2068406454 learned_addr learned my addr 172.21.15.29:0/2068406454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:09.014 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faad5d9b700 1 --2- 172.21.15.29:0/2068406454 >> v2:172.21.15.144:3300/0 conn(0x7faad80c5360 0x7faad80ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:09.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faad659c700 1 -- 172.21.15.29:0/2068406454 >> v2:172.21.15.144:3300/0 conn(0x7faad80c5360 msgr2=0x7faad80ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:09.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.012+0000 7faad659c700 1 --2- 172.21.15.29:0/2068406454 >> v2:172.21.15.144:3300/0 conn(0x7faad80c5360 0x7faad80ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad659c700 1 -- 172.21.15.29:0/2068406454 >> v1:172.21.15.144:6789/0 conn(0x7faad815da30 legacy=0x7faad80cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:09.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad659c700 1 -- 172.21.15.29:0/2068406454 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad80cb720 con 0x7faad81598d0 2023-10-07T09:29:09.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad559a700 1 -- 172.21.15.29:0/2068406454 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115961600 0 0) 0x7faad80cba90 con 0x7faad815da30 2023-10-07T09:29:09.015 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad5d9b700 1 --2- 172.21.15.29:0/2068406454 >> v2:172.21.15.144:3300/0 conn(0x7faad80c5360 0x7faad80ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad659c700 1 --2- 172.21.15.29:0/2068406454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 0x7faad815d4f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faac8009990 tx=0x7faac8009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=330602a6d318f0e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad559a700 1 -- 172.21.15.29:0/2068406454 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8009e80 con 0x7faad81598d0 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad559a700 1 -- 172.21.15.29:0/2068406454 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac8005c10 con 0x7faad81598d0 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.013+0000 7faad559a700 1 -- 172.21.15.29:0/2068406454 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac800b650 con 0x7faad81598d0 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 -- 172.21.15.29:0/2068406454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 msgr2=0x7faad815d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 --2- 172.21.15.29:0/2068406454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 0x7faad815d4f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faac8009990 tx=0x7faac8009220 comp rx=0 tx=0).stop 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 -- 172.21.15.29:0/2068406454 shutdown_connections 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 --2- 172.21.15.29:0/2068406454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad81598d0 0x7faad815d4f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 --2- 172.21.15.29:0/2068406454 >> v2:172.21.15.144:3300/0 conn(0x7faad80c5360 0x7faad80ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 -- 172.21.15.29:0/2068406454 >> 172.21.15.29:0/2068406454 conn(0x7faad8089ff0 msgr2=0x7faad8099910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:09.016 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 -- 172.21.15.29:0/2068406454 shutdown_connections 2023-10-07T09:29:09.017 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.014+0000 7faadcbff700 1 -- 172.21.15.29:0/2068406454 wait complete. 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.015+0000 7faadcbff700 1 Processor -- start 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.015+0000 7faadcbff700 1 -- start start 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.016+0000 7faadcbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 0x7faad80b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.016+0000 7faadcbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faad81598d0 0x7faad814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.016+0000 7faadcbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad815da30 0x7faad815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.016+0000 7faadcbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad80c9390 con 0x7faad80c5360 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.016+0000 7faadcbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad815b9e0 con 0x7faad81598d0 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.016+0000 7faadcbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad815bcc0 con 0x7faad815da30 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 0x7faad80b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:09.019 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 0x7faad80b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36192/0 (socket says 172.21.15.29:36192) 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 -- 172.21.15.29:0/1799892404 learned_addr learned my addr 172.21.15.29:0/1799892404 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad6d9d700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad815da30 0x7faad815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad5d9b700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faad81598d0 0x7faad814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 -- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad815da30 msgr2=0x7faad815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad815da30 0x7faad815b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 -- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faad81598d0 msgr2=0x7faad814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faad81598d0 0x7faad814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 -- 172.21.15.29:0/1799892404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc009420 con 0x7faad80c5360 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad5d9b700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faad81598d0 0x7faad814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faad659c700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 0x7faad80b3360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faac8007fd0 tx=0x7faac8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.017+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8009e80 con 0x7faad80c5360 2023-10-07T09:29:09.020 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.018+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac8009480 con 0x7faad80c5360 2023-10-07T09:29:09.021 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.018+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac8020070 con 0x7faad80c5360 2023-10-07T09:29:09.021 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.018+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8011d00 con 0x7faad80c5360 2023-10-07T09:29:09.021 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.018+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad815c280 con 0x7faad80c5360 2023-10-07T09:29:09.021 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.019+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad80c9160 con 0x7faad80c5360 2023-10-07T09:29:09.021 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.019+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7faac8022030 con 0x7faad80c5360 2023-10-07T09:29:09.022 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.021+0000 7faac67fc700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7faab8066ea0 0x7faab8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:09.022 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.021+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3627+0+0 (secure 0 0 0) 0x7faac8085980 con 0x7faad80c5360 2023-10-07T09:29:09.023 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.022+0000 7faad5d9b700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7faab8066ea0 0x7faab8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:09.023 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.022+0000 7faad5d9b700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7faab8066ea0 0x7faab8069350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faacc0093f0 tx=0x7faacc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:09.027 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.026+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faad80c9160 con 0x7faad80c5360 2023-10-07T09:29:09.230 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.229+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faad80c9160 con 0x7faad80c5360 2023-10-07T09:29:09.231 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.230+0000 7faac67fc700 1 -- 172.21.15.29:0/1799892404 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9178 (secure 0 0 0) 0x7faad80c9160 con 0x7faad80c5360 2023-10-07T09:29:09.231 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-07T09:29:09.231 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":14,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:08.458127+0000","last_up_change":"2023-10-07T09:29:08.458127+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7faab8066ea0 msgr2=0x7faab8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7faab8066ea0 0x7faab8069350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faacc0093f0 tx=0x7faacc008040 comp rx=0 tx=0).stop 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 msgr2=0x7faad80b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 0x7faad80b3360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faac8007fd0 tx=0x7faac8016000 comp rx=0 tx=0).stop 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 shutdown_connections 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7faab8066ea0 0x7faab8069350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faad815da30 0x7faad815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad80c5360 0x7faad80b3360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.234 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 --2- 172.21.15.29:0/1799892404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faad81598d0 0x7faad814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:09.235 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.232+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 >> 172.21.15.29:0/1799892404 conn(0x7faad8089ff0 msgr2=0x7faad80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:09.235 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.233+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 shutdown_connections 2023-10-07T09:29:09.235 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:09.233+0000 7faadcbff700 1 -- 172.21.15.29:0/1799892404 wait complete. 2023-10-07T09:29:09.248 DEBUG:teuthology.misc:5 of 8 OSDs are up 2023-10-07T09:29:09.264 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-07T09:29:09.263+0000 7fe92bff6700 -1 osd.4 0 waiting for initial osdmap 2023-10-07T09:29:09.271 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-07T09:29:09.270+0000 7f4ccc9f6700 -1 osd.6 0 waiting for initial osdmap 2023-10-07T09:29:09.272 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-07T09:29:09.271+0000 7fe924fe8700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:09.281 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-07T09:29:09.280+0000 7f4cc59e8700 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:09.515 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-07T09:29:09.514+0000 7fc9de9f6700 -1 osd.7 0 waiting for initial osdmap 2023-10-07T09:29:09.521 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-07T09:29:09.520+0000 7fc9d79e8700 -1 osd.7 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-07T09:29:16.056 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:16.238 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.237+0000 7fa6385ff700 1 Processor -- start 2023-10-07T09:29:16.238 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.237+0000 7fa6385ff700 1 -- start start 2023-10-07T09:29:16.239 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.238+0000 7fa6385ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa630159870 0x7fa63014fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.239 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.238+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa630159c40 0x7fa630150040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.240 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.238+0000 7fa6385ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa630157290 con 0x7fa6300afd80 2023-10-07T09:29:16.240 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.238+0000 7fa6385ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa630150580 con 0x7fa630159870 2023-10-07T09:29:16.240 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.238+0000 7fa6385ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300b08d0 con 0x7fa630159c40 2023-10-07T09:29:16.240 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.238+0000 7fa63639b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa630159870 0x7fa63014fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.240 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa630159870 0x7fa63014fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38504/0 (socket says 172.21.15.29:38504) 2023-10-07T09:29:16.240 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 -- 172.21.15.29:0/335995488 learned_addr learned my addr 172.21.15.29:0/335995488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:16.241 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 -- 172.21.15.29:0/335995488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa630159c40 msgr2=0x7fa630150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 --2- 172.21.15.29:0/335995488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa630159c40 0x7fa630150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 -- 172.21.15.29:0/335995488 >> v1:172.21.15.144:6789/0 conn(0x7fa6300afd80 legacy=0x7fa6300b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 -- 172.21.15.29:0/335995488 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630157570 con 0x7fa630159870 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa63639b700 1 --2- 172.21.15.29:0/335995488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa63014fb00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00ab50 tx=0x7fa62c00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163ddcee15efff43 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa635399700 1 -- 172.21.15.29:0/335995488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00ca20 con 0x7fa630159870 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa635399700 1 -- 172.21.15.29:0/335995488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c00cb80 con 0x7fa630159870 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.239+0000 7fa635399700 1 -- 172.21.15.29:0/335995488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c0183f0 con 0x7fa630159870 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 -- 172.21.15.29:0/335995488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 msgr2=0x7fa63014fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 --2- 172.21.15.29:0/335995488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa63014fb00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00ab50 tx=0x7fa62c00a3f0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 -- 172.21.15.29:0/335995488 shutdown_connections 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 --2- 172.21.15.29:0/335995488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa63014fb00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.242 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 --2- 172.21.15.29:0/335995488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa630159c40 0x7fa630150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.243 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 -- 172.21.15.29:0/335995488 >> 172.21.15.29:0/335995488 conn(0x7fa630067610 msgr2=0x7fa630067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:16.243 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 -- 172.21.15.29:0/335995488 shutdown_connections 2023-10-07T09:29:16.243 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.240+0000 7fa6385ff700 1 -- 172.21.15.29:0/335995488 wait complete. 2023-10-07T09:29:16.243 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.242+0000 7fa6385ff700 1 Processor -- start 2023-10-07T09:29:16.243 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.242+0000 7fa6385ff700 1 -- start start 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6300afd80 0x7fa63014da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa6300cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa630159c40 0x7fa6300cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa6385ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa630155290 con 0x7fa630159870 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa6385ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301546a0 con 0x7fa630159c40 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa6385ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa630154960 con 0x7fa6300afd80 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa635b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa6300cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.246 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa63639b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6300afd80 0x7fa63014da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa63639b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6300afd80 0x7fa63014da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:55000/0 (socket says 172.21.15.29:55000) 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa63639b700 1 -- 172.21.15.29:0/367604044 learned_addr learned my addr 172.21.15.29:0/367604044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa636b9c700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa630159c40 0x7fa6300cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa635b9a700 1 -- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6300afd80 msgr2=0x7fa63014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa635b9a700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6300afd80 0x7fa63014da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa635b9a700 1 -- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa630159c40 msgr2=0x7fa6300cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa635b9a700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa630159c40 0x7fa6300cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.244+0000 7fa635b9a700 1 -- 172.21.15.29:0/367604044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c00a690 con 0x7fa630159870 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.245+0000 7fa635b9a700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa6300cc1c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa62000b5e0 tx=0x7fa6200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.245+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62000f510 con 0x7fa630159870 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.245+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62000f670 con 0x7fa630159870 2023-10-07T09:29:16.247 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.245+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62000fa30 con 0x7fa630159870 2023-10-07T09:29:16.248 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.245+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa630154c40 con 0x7fa630159870 2023-10-07T09:29:16.248 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.245+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63014dfd0 con 0x7fa630159870 2023-10-07T09:29:16.248 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.246+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa630155400 con 0x7fa630159870 2023-10-07T09:29:16.248 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.247+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa62001a400 con 0x7fa630159870 2023-10-07T09:29:16.249 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.248+0000 7fa6267fc700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa614067000 0x7fa6140694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.249 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.248+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fa620083b70 con 0x7fa630159870 2023-10-07T09:29:16.249 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.248+0000 7fa63639b700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa614067000 0x7fa6140694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.250 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.249+0000 7fa63639b700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa614067000 0x7fa6140694b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa62c003080 tx=0x7fa62c019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:16.254 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.253+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa620051190 con 0x7fa630159870 2023-10-07T09:29:16.456 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.455+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa630157850 con 0x7fa630159870 2023-10-07T09:29:16.457 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.456+0000 7fa6267fc700 1 -- 172.21.15.29:0/367604044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12223 (secure 0 0 0) 0x7fa6200547b0 con 0x7fa630159870 2023-10-07T09:29:16.457 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-07T09:29:16.457 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":18,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:12.527002+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.458+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa614067000 msgr2=0x7fa6140694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.458+0000 7fa6385ff700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa614067000 0x7fa6140694b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa62c003080 tx=0x7fa62c019000 comp rx=0 tx=0).stop 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.458+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 msgr2=0x7fa6300cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.458+0000 7fa6385ff700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa6300cc1c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa62000b5e0 tx=0x7fa6200090f0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 shutdown_connections 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa614067000 0x7fa6140694b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa630159c40 0x7fa6300cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.460 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6300afd80 0x7fa63014da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.461 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 --2- 172.21.15.29:0/367604044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa630159870 0x7fa6300cc1c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.461 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 >> 172.21.15.29:0/367604044 conn(0x7fa630067610 msgr2=0x7fa6300c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:16.461 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 shutdown_connections 2023-10-07T09:29:16.461 INFO:teuthology.misc.health.smithi029.stderr:2023-10-07T09:29:16.459+0000 7fa6385ff700 1 -- 172.21.15.29:0/367604044 wait complete. 2023-10-07T09:29:16.475 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-07T09:29:16.475 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-07T09:29:16.475 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:29:16.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:29:16.479 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:29:16.480 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:29:16.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:29:16.483 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:29:16.484 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:29:16.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:29:16.489 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:29:16.489 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:29:16.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:29:16.494 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:29:16.494 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-07T09:29:16.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:16.631 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:29:16.631 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:29:16.637 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:29:16.640 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:29:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.703+0000 7f961b3ff700 1 Processor -- start 2023-10-07T09:29:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.703+0000 7f961b3ff700 1 -- start start 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961b3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9614159750 0x7f96140d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961b3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f96140ce8a0 0x7f96140c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961b3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96140cbf20 con 0x7f9614159b20 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961b3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96140cc200 con 0x7f96140ce8a0 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961b3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96140cc4e0 con 0x7f9614159750 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961999c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f96140ce8a0 0x7f96140c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961899a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9614159b20 0x7f96140ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38232/0 (socket says 172.21.15.29:38232) 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961999c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f96140ce8a0 0x7f96140c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38570/0 (socket says 172.21.15.29:38570) 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961899a700 1 -- 172.21.15.29:0/2960315912 learned_addr learned my addr 172.21.15.29:0/2960315912 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.705+0000 7f961919b700 1 --2- 172.21.15.29:0/2960315912 >> v2:172.21.15.144:3300/0 conn(0x7f9614159750 0x7f96140d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f961999c700 1 -- 172.21.15.29:0/2960315912 >> v2:172.21.15.144:3300/0 conn(0x7f9614159750 msgr2=0x7f96140d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f961999c700 1 --2- 172.21.15.29:0/2960315912 >> v2:172.21.15.144:3300/0 conn(0x7f9614159750 0x7f96140d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f961999c700 1 -- 172.21.15.29:0/2960315912 >> v1:172.21.15.29:6789/0 conn(0x7f9614159b20 legacy=0x7f96140ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f961999c700 1 -- 172.21.15.29:0/2960315912 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140cc7c0 con 0x7f96140ce8a0 2023-10-07T09:29:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f960bfff700 1 -- 172.21.15.29:0/2960315912 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038200042 0 0) 0x7f96140cbf20 con 0x7f9614159b20 2023-10-07T09:29:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f961999c700 1 --2- 172.21.15.29:0/2960315912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96140ce8a0 0x7f96140c5460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9614089840 tx=0x7f960c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560def220166e4b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f960bfff700 1 -- 172.21.15.29:0/2960315912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c00e070 con 0x7f96140ce8a0 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f960bfff700 1 -- 172.21.15.29:0/2960315912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c00bb60 con 0x7f96140ce8a0 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.706+0000 7f960bfff700 1 -- 172.21.15.29:0/2960315912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c012430 con 0x7f96140ce8a0 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 -- 172.21.15.29:0/2960315912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96140ce8a0 msgr2=0x7f96140c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2960315912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96140ce8a0 0x7f96140c5460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9614089840 tx=0x7f960c009780 comp rx=0 tx=0).stop 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 -- 172.21.15.29:0/2960315912 shutdown_connections 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2960315912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96140ce8a0 0x7f96140c5460 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2960315912 >> v2:172.21.15.144:3300/0 conn(0x7f9614159750 0x7f96140d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 -- 172.21.15.29:0/2960315912 >> 172.21.15.29:0/2960315912 conn(0x7f9614063a10 msgr2=0x7f96140d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 -- 172.21.15.29:0/2960315912 shutdown_connections 2023-10-07T09:29:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.707+0000 7f961b3ff700 1 -- 172.21.15.29:0/2960315912 wait complete. 2023-10-07T09:29:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.709+0000 7f961b3ff700 1 Processor -- start 2023-10-07T09:29:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.709+0000 7f961b3ff700 1 -- start start 2023-10-07T09:29:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.710+0000 7f961b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96140ce8a0 0x7f961415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.710+0000 7f961b3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 0x7f96140c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.710+0000 7f961b3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9614159b20 0x7f961413f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.710+0000 7f961b3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140c9570 con 0x7f9614159750 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.710+0000 7f961b3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140c9ff0 con 0x7f9614159b20 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.710+0000 7f961b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961416a7f0 con 0x7f96140ce8a0 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 0x7f96140c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 0x7f96140c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38582/0 (socket says 172.21.15.29:38582) 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 -- 172.21.15.29:0/2569692631 learned_addr learned my addr 172.21.15.29:0/2569692631 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961999c700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9614159b20 0x7f961413f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961919b700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96140ce8a0 0x7f961415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 -- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96140ce8a0 msgr2=0x7f961415d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96140ce8a0 0x7f961415d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 -- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9614159b20 msgr2=0x7f961413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9614159b20 0x7f961413f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 -- 172.21.15.29:0/2569692631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c009a20 con 0x7f9614159750 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961999c700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9614159b20 0x7f961413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.711+0000 7f961899a700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 0x7f96140c9030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f960400c5b0 tx=0x7f9604001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.712+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604005b30 con 0x7f9614159750 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.712+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604005c90 con 0x7f9614159750 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.712+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96140c9750 con 0x7f9614159750 2023-10-07T09:29:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.712+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604007300 con 0x7f9614159750 2023-10-07T09:29:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.712+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96140c9d10 con 0x7f9614159750 2023-10-07T09:29:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.713+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961404c3c0 con 0x7f9614159750 2023-10-07T09:29:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.714+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f960401b3b0 con 0x7f9614159750 2023-10-07T09:29:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.715+0000 7f96097fa700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95f8066f30 0x7f95f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.715+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f96040836e0 con 0x7f9614159750 2023-10-07T09:29:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.715+0000 7f961919b700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95f8066f30 0x7f95f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.716+0000 7f961919b700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95f8066f30 0x7f95f80693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9610009da0 tx=0x7f9610009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.719+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9604050c00 con 0x7f9614159750 2023-10-07T09:29:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.925+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9614064040 con 0x7f9614159750 2023-10-07T09:29:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.926+0000 7f96097fa700 1 -- 172.21.15.29:0/2569692631 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12223 (secure 0 0 0) 0x7f9604054220 con 0x7f9614159750 2023-10-07T09:29:16.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:16.927 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":18,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:12.527002+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95f8066f30 msgr2=0x7f95f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95f8066f30 0x7f95f80693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9610009da0 tx=0x7f9610009120 comp rx=0 tx=0).stop 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 msgr2=0x7f96140c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 0x7f96140c9030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f960400c5b0 tx=0x7f9604001070 comp rx=0 tx=0).stop 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 shutdown_connections 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95f8066f30 0x7f95f80693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9614159b20 0x7f961413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96140ce8a0 0x7f961415d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 --2- 172.21.15.29:0/2569692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9614159750 0x7f96140c9030 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 >> 172.21.15.29:0/2569692631 conn(0x7f9614063a10 msgr2=0x7f96141545b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 shutdown_connections 2023-10-07T09:29:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:16.928+0000 7f961b3ff700 1 -- 172.21.15.29:0/2569692631 wait complete. 2023-10-07T09:29:16.947 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:29:16.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:29:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.145+0000 7fd17b5ff700 1 Processor -- start 2023-10-07T09:29:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.145+0000 7fd17b5ff700 1 -- start start 2023-10-07T09:29:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17b5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd174153cc0 0x7fd1740b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd17415ade0 0x7fd1740b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17b5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd174154090 con 0x7fd1740b0000 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17b5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd174159f60 con 0x7fd174153cc0 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17415a240 con 0x7fd17415ade0 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd174153cc0 0x7fd1740b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd174153cc0 0x7fd1740b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38590/0 (socket says 172.21.15.29:38590) 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 -- 172.21.15.29:0/930630088 learned_addr learned my addr 172.21.15.29:0/930630088 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd179b9c700 1 --1- 172.21.15.29:0/930630088 >> v1:172.21.15.29:6789/0 conn(0x7fd1740b0000 0x7fd17415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38242/0 (socket says 172.21.15.29:38242) 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd178b9a700 1 --2- 172.21.15.29:0/930630088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd17415ade0 0x7fd1740b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 -- 172.21.15.29:0/930630088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd17415ade0 msgr2=0x7fd1740b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 --2- 172.21.15.29:0/930630088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd17415ade0 0x7fd1740b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 -- 172.21.15.29:0/930630088 >> v1:172.21.15.29:6789/0 conn(0x7fd1740b0000 legacy=0x7fd17415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.147+0000 7fd17939b700 1 -- 172.21.15.29:0/930630088 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17415a520 con 0x7fd174153cc0 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.148+0000 7fd17939b700 1 --2- 172.21.15.29:0/930630088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd1740b17c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd1700099a0 tx=0x7fd170009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2270860b8ac083 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.148+0000 7fd16bfff700 1 -- 172.21.15.29:0/930630088 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170009e90 con 0x7fd174153cc0 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.148+0000 7fd16bfff700 1 -- 172.21.15.29:0/930630088 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17000b860 con 0x7fd174153cc0 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.148+0000 7fd16bfff700 1 -- 172.21.15.29:0/930630088 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000bc20 con 0x7fd174153cc0 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.148+0000 7fd17b5ff700 1 -- 172.21.15.29:0/930630088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 msgr2=0x7fd1740b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.148+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/930630088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd1740b17c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd1700099a0 tx=0x7fd170009250 comp rx=0 tx=0).stop 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.149+0000 7fd17b5ff700 1 -- 172.21.15.29:0/930630088 shutdown_connections 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.149+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/930630088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd1740b17c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.149+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/930630088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd17415ade0 0x7fd1740b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.149+0000 7fd17b5ff700 1 -- 172.21.15.29:0/930630088 >> 172.21.15.29:0/930630088 conn(0x7fd174067610 msgr2=0x7fd174067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:17.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.149+0000 7fd17b5ff700 1 -- 172.21.15.29:0/930630088 shutdown_connections 2023-10-07T09:29:17.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.149+0000 7fd17b5ff700 1 -- 172.21.15.29:0/930630088 wait complete. 2023-10-07T09:29:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.150+0000 7fd17b5ff700 1 Processor -- start 2023-10-07T09:29:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.150+0000 7fd17b5ff700 1 -- start start 2023-10-07T09:29:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd17b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1740b0000 0x7fd174145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd17b5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd174158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd17b5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd17415ade0 0x7fd1741699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd17b5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd174156110 con 0x7fd174153cc0 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd17b5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741563f0 con 0x7fd17415ade0 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd17b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741566d0 con 0x7fd1740b0000 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.151+0000 7fd178b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd174158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd174158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38596/0 (socket says 172.21.15.29:38596) 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 -- 172.21.15.29:0/3719499244 learned_addr learned my addr 172.21.15.29:0/3719499244 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd17939b700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1740b0000 0x7fd174145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd179b9c700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd17415ade0 0x7fd1741699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 -- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1740b0000 msgr2=0x7fd174145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1740b0000 0x7fd174145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 -- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd17415ade0 msgr2=0x7fd1741699b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd17415ade0 0x7fd1741699b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 -- 172.21.15.29:0/3719499244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170009490 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd179b9c700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd17415ade0 0x7fd1741699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd178b9a700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd174158990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd16400e930 tx=0x7fd1640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164013680 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640137e0 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164013b80 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.152+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1741569b0 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.153+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174146370 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.154+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17404c3c0 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.155+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd16400c460 con 0x7fd174153cc0 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.156+0000 7fd1697fa700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd158066f30 0x7fd1580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.156+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fd1640843f0 con 0x7fd174153cc0 2023-10-07T09:29:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.156+0000 7fd17939b700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd158066f30 0x7fd1580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.156+0000 7fd17939b700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd158066f30 0x7fd1580693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1700099a0 tx=0x7fd170017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.160+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd164051910 con 0x7fd174153cc0 2023-10-07T09:29:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.363+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd17415a800 con 0x7fd174153cc0 2023-10-07T09:29:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.363+0000 7fd1697fa700 1 -- 172.21.15.29:0/3719499244 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd164054f30 con 0x7fd174153cc0 2023-10-07T09:29:17.365 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:29:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd158066f30 msgr2=0x7fd1580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd158066f30 0x7fd1580693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1700099a0 tx=0x7fd170017000 comp rx=0 tx=0).stop 2023-10-07T09:29:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 msgr2=0x7fd174158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd174158990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd16400e930 tx=0x7fd1640091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 shutdown_connections 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd158066f30 0x7fd1580693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd17415ade0 0x7fd1741699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1740b0000 0x7fd174145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 --2- 172.21.15.29:0/3719499244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174153cc0 0x7fd174158990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 >> 172.21.15.29:0/3719499244 conn(0x7fd174067610 msgr2=0x7fd1740b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 shutdown_connections 2023-10-07T09:29:17.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.366+0000 7fd17b5ff700 1 -- 172.21.15.29:0/3719499244 wait complete. 2023-10-07T09:29:17.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:29:17.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:29:17.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:29:17.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.585+0000 7fd4683ff700 1 Processor -- start 2023-10-07T09:29:17.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.585+0000 7fd4683ff700 1 -- start start 2023-10-07T09:29:17.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd4683ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159750 0x7fd4600d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd4683ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd4600ce8a0 0x7fd4600c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd4683ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600cbf20 con 0x7fd460159b20 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd4683ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4600cc200 con 0x7fd4600ce8a0 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd4683ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4600cc4e0 con 0x7fd460159750 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd46599a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd460159b20 0x7fd4600ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38254/0 (socket says 172.21.15.29:38254) 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd46599a700 1 -- 172.21.15.29:0/4198110696 learned_addr learned my addr 172.21.15.29:0/4198110696 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd46619b700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159750 0x7fd4600d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.586+0000 7fd46699c700 1 --2- 172.21.15.29:0/4198110696 >> v2:172.21.15.144:3300/0 conn(0x7fd4600ce8a0 0x7fd4600c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd465199700 1 -- 172.21.15.29:0/4198110696 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195791784 0 0) 0x7fd4600cbf20 con 0x7fd460159b20 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd465199700 1 -- 172.21.15.29:0/4198110696 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd44c003570 con 0x7fd460159b20 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd46699c700 1 -- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159750 msgr2=0x7fd4600d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd46699c700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159750 0x7fd4600d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd46699c700 1 -- 172.21.15.29:0/4198110696 >> v1:172.21.15.29:6789/0 conn(0x7fd460159b20 legacy=0x7fd4600ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd46699c700 1 -- 172.21.15.29:0/4198110696 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600cc7c0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd46619b700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159750 0x7fd4600d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.587+0000 7fd46699c700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4600ce8a0 0x7fd4600c5460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd458009e60 tx=0x7fd458009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3176c1e4918caf96 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd465199700 1 -- 172.21.15.29:0/4198110696 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45800e070 con 0x7fd4600ce8a0 2023-10-07T09:29:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd465199700 1 -- 172.21.15.29:0/4198110696 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45800bab0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd465199700 1 -- 172.21.15.29:0/4198110696 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45800be70 con 0x7fd4600ce8a0 2023-10-07T09:29:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 -- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4600ce8a0 msgr2=0x7fd4600c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4600ce8a0 0x7fd4600c5460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd458009e60 tx=0x7fd458009780 comp rx=0 tx=0).stop 2023-10-07T09:29:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 -- 172.21.15.29:0/4198110696 shutdown_connections 2023-10-07T09:29:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4600ce8a0 0x7fd4600c5460 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 --2- 172.21.15.29:0/4198110696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159750 0x7fd4600d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 -- 172.21.15.29:0/4198110696 >> 172.21.15.29:0/4198110696 conn(0x7fd460063a10 msgr2=0x7fd4600d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 -- 172.21.15.29:0/4198110696 shutdown_connections 2023-10-07T09:29:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.588+0000 7fd4683ff700 1 -- 172.21.15.29:0/4198110696 wait complete. 2023-10-07T09:29:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.590+0000 7fd4683ff700 1 Processor -- start 2023-10-07T09:29:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.590+0000 7fd4683ff700 1 -- start start 2023-10-07T09:29:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd4683ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 0x7fd4601535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd4683ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd460159750 0x7fd4601518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd4683ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159b20 0x7fd460169540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd4683ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd460153ca0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd4683ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd460153f80 con 0x7fd460159750 2023-10-07T09:29:17.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd4683ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd460169d90 con 0x7fd460159b20 2023-10-07T09:29:17.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 0x7fd4601535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 0x7fd4601535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38604/0 (socket says 172.21.15.29:38604) 2023-10-07T09:29:17.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 -- 172.21.15.29:0/2050232364 learned_addr learned my addr 172.21.15.29:0/2050232364 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46599a700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd460159750 0x7fd4601518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46699c700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159b20 0x7fd460169540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 -- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159b20 msgr2=0x7fd460169540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159b20 0x7fd460169540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 -- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd460159750 msgr2=0x7fd4601518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd460159750 0x7fd4601518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.591+0000 7fd46619b700 1 -- 172.21.15.29:0/2050232364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c0093c0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.592+0000 7fd46619b700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 0x7fd4601535f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00ec80 tx=0x7fd45c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.592+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c005a00 con 0x7fd4600ce8a0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.592+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c005b60 con 0x7fd4600ce8a0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.592+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00c850 con 0x7fd4600ce8a0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.592+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd458009a20 con 0x7fd4600ce8a0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.592+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46016a350 con 0x7fd4600ce8a0 2023-10-07T09:29:17.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.593+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd460154260 con 0x7fd4600ce8a0 2023-10-07T09:29:17.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.594+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd45c00cbc0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.595+0000 7fd4567fc700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd444066f30 0x7fd4440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:17.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.595+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fd45c0846d0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.595+0000 7fd46599a700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd444066f30 0x7fd4440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:17.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.596+0000 7fd46599a700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd444066f30 0x7fd4440693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd4600c7cf0 tx=0x7fd450009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:17.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.599+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd45c055680 con 0x7fd4600ce8a0 2023-10-07T09:29:17.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.848+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd46004c3c0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.849+0000 7fd4567fc700 1 -- 172.21.15.29:0/2050232364 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3527 (secure 0 0 0) 0x7fd45c020de0 con 0x7fd4600ce8a0 2023-10-07T09:29:17.850 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:17.850 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.851+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd444066f30 msgr2=0x7fd4440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.851+0000 7fd4683ff700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd444066f30 0x7fd4440693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd4600c7cf0 tx=0x7fd450009040 comp rx=0 tx=0).stop 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.851+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 msgr2=0x7fd4601535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.851+0000 7fd4683ff700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 0x7fd4601535f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00ec80 tx=0x7fd45c009120 comp rx=0 tx=0).stop 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 shutdown_connections 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd444066f30 0x7fd4440693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd460159b20 0x7fd460169540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4600ce8a0 0x7fd4601535f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 --2- 172.21.15.29:0/2050232364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd460159750 0x7fd4601518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 >> 172.21.15.29:0/2050232364 conn(0x7fd460063a10 msgr2=0x7fd4600c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 shutdown_connections 2023-10-07T09:29:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:17.852+0000 7fd4683ff700 1 -- 172.21.15.29:0/2050232364 wait complete. 2023-10-07T09:29:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2023-10-07T09:29:17.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.064+0000 7f1b153ff700 1 Processor -- start 2023-10-07T09:29:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.064+0000 7f1b153ff700 1 -- start start 2023-10-07T09:29:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b153ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b10157290 0x7f1b100cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b153ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100ae8b0 0x7f1b100b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b153ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1b100b1840 0x7f1b100ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b153ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b100b1ca0 con 0x7f1b10157290 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b153ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b10158ad0 con 0x7f1b100b1840 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b153ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1015b140 con 0x7f1b100ae8b0 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b0effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b10157290 0x7f1b100cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b0effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b10157290 0x7f1b100cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38618/0 (socket says 172.21.15.29:38618) 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 -- 172.21.15.29:0/950822186 learned_addr learned my addr 172.21.15.29:0/950822186 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.065+0000 7f1b0e7fc700 1 --2- 172.21.15.29:0/950822186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100ae8b0 0x7f1b100b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0f7fe700 1 --2- 172.21.15.29:0/950822186 >> v2:172.21.15.144:3300/0 conn(0x7f1b100b1840 0x7f1b100ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 -- 172.21.15.29:0/950822186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100ae8b0 msgr2=0x7f1b100b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 --2- 172.21.15.29:0/950822186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100ae8b0 0x7f1b100b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 -- 172.21.15.29:0/950822186 >> v2:172.21.15.144:3300/0 conn(0x7f1b100b1840 msgr2=0x7f1b100ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 --2- 172.21.15.29:0/950822186 >> v2:172.21.15.144:3300/0 conn(0x7f1b100b1840 0x7f1b100ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 -- 172.21.15.29:0/950822186 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1015b420 con 0x7f1b10157290 2023-10-07T09:29:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0effd700 1 --2- 172.21.15.29:0/950822186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b100cc870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b2220 tx=0x7f1b00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3931edfb156107e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.066+0000 7f1b0dffb700 1 -- 172.21.15.29:0/950822186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0000b640 con 0x7f1b10157290 2023-10-07T09:29:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b0dffb700 1 -- 172.21.15.29:0/950822186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0000b7a0 con 0x7f1b10157290 2023-10-07T09:29:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b0dffb700 1 -- 172.21.15.29:0/950822186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0000bb60 con 0x7f1b10157290 2023-10-07T09:29:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 -- 172.21.15.29:0/950822186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 msgr2=0x7f1b100cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 --2- 172.21.15.29:0/950822186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b100cc870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b2220 tx=0x7f1b00009250 comp rx=0 tx=0).stop 2023-10-07T09:29:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 -- 172.21.15.29:0/950822186 shutdown_connections 2023-10-07T09:29:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 --2- 172.21.15.29:0/950822186 >> v2:172.21.15.144:3300/0 conn(0x7f1b100b1840 0x7f1b100ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 --2- 172.21.15.29:0/950822186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b100cc870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 --2- 172.21.15.29:0/950822186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100ae8b0 0x7f1b100b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 -- 172.21.15.29:0/950822186 >> 172.21.15.29:0/950822186 conn(0x7f1b100680a0 msgr2=0x7f1b100cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 -- 172.21.15.29:0/950822186 shutdown_connections 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.067+0000 7f1b153ff700 1 -- 172.21.15.29:0/950822186 wait complete. 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.069+0000 7f1b153ff700 1 Processor -- start 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.069+0000 7f1b153ff700 1 -- start start 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.070+0000 7f1b153ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100b1840 0x7f1b10151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.070+0000 7f1b153ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b10145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.070+0000 7f1b153ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1015ed20 0x7f1b10155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.070+0000 7f1b153ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1015c0a0 con 0x7f1b10157290 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.070+0000 7f1b153ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b10146140 con 0x7f1b1015ed20 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.070+0000 7f1b153ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b10154740 con 0x7f1b100b1840 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b10145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100b1840 0x7f1b10151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b10145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38628/0 (socket says 172.21.15.29:38628) 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 -- 172.21.15.29:0/3107646122 learned_addr learned my addr 172.21.15.29:0/3107646122 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0f7fe700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1015ed20 0x7f1b10155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 -- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100b1840 msgr2=0x7f1b10151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100b1840 0x7f1b10151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 -- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1015ed20 msgr2=0x7f1b10155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1015ed20 0x7f1b10155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 -- 172.21.15.29:0/3107646122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000094f0 con 0x7f1b10157290 2023-10-07T09:29:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0f7fe700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1015ed20 0x7f1b10155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.071+0000 7f1b0e7fc700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b10145c00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400ec80 tx=0x7f1b040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.072+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400c850 con 0x7f1b10157290 2023-10-07T09:29:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.072+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b10154a20 con 0x7f1b10157290 2023-10-07T09:29:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.072+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0400c9b0 con 0x7f1b10157290 2023-10-07T09:29:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.072+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400cd50 con 0x7f1b10157290 2023-10-07T09:29:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.072+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b10154410 con 0x7f1b10157290 2023-10-07T09:29:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.073+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1015c250 con 0x7f1b10157290 2023-10-07T09:29:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.074+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1b04013680 con 0x7f1b10157290 2023-10-07T09:29:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.075+0000 7f1af6ffd700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1afc066f30 0x7f1afc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.075+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f1b04084570 con 0x7f1b10157290 2023-10-07T09:29:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.075+0000 7f1b0effd700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1afc066f30 0x7f1afc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.076+0000 7f1b0effd700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1afc066f30 0x7f1afc0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b2220 tx=0x7f1b00017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.079+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b1015c250 con 0x7f1b10157290 2023-10-07T09:29:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.281+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b1015c250 con 0x7f1b10157290 2023-10-07T09:29:18.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.282+0000 7f1af6ffd700 1 -- 172.21.15.29:0/3107646122 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12223 (secure 0 0 0) 0x7f1b1015c250 con 0x7f1b10157290 2023-10-07T09:29:18.283 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:18.284 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":18,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:12.527002+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1afc066f30 msgr2=0x7f1afc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1afc066f30 0x7f1afc0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b2220 tx=0x7f1b00017000 comp rx=0 tx=0).stop 2023-10-07T09:29:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 msgr2=0x7f1b10145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b10145c00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400ec80 tx=0x7f1b040091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 shutdown_connections 2023-10-07T09:29:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1afc066f30 0x7f1afc0693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1015ed20 0x7f1b10155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b100b1840 0x7f1b10151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 --2- 172.21.15.29:0/3107646122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b10157290 0x7f1b10145c00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 >> 172.21.15.29:0/3107646122 conn(0x7f1b100680a0 msgr2=0x7f1b1006d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 shutdown_connections 2023-10-07T09:29:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.284+0000 7f1b153ff700 1 -- 172.21.15.29:0/3107646122 wait complete. 2023-10-07T09:29:18.304 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:29:18.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:29:18.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.500+0000 7f860fbff700 1 Processor -- start 2023-10-07T09:29:18.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.501+0000 7f860fbff700 1 -- start start 2023-10-07T09:29:18.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860fbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8608153cc0 0x7f86080b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860fbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f86080b0710 0x7f86080b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860fbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8608154090 con 0x7f860815ade0 2023-10-07T09:29:18.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860fbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8608159f60 con 0x7f8608153cc0 2023-10-07T09:29:18.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860fbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860815a240 con 0x7f86080b0710 2023-10-07T09:29:18.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860d99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8608153cc0 0x7f86080b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860d99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8608153cc0 0x7f86080b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38640/0 (socket says 172.21.15.29:38640) 2023-10-07T09:29:18.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.502+0000 7f860d99b700 1 -- 172.21.15.29:0/3623854639 learned_addr learned my addr 172.21.15.29:0/3623854639 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:18.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860e19c700 1 --2- 172.21.15.29:0/3623854639 >> v2:172.21.15.144:3300/0 conn(0x7f86080b0710 0x7f86080b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860d99b700 1 -- 172.21.15.29:0/3623854639 >> v2:172.21.15.144:3300/0 conn(0x7f86080b0710 msgr2=0x7f86080b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:18.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860d99b700 1 --2- 172.21.15.29:0/3623854639 >> v2:172.21.15.144:3300/0 conn(0x7f86080b0710 0x7f86080b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860d99b700 1 -- 172.21.15.29:0/3623854639 >> v1:172.21.15.144:6789/0 conn(0x7f860815ade0 legacy=0x7f86080b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860d99b700 1 -- 172.21.15.29:0/3623854639 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860815a520 con 0x7f8608153cc0 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860d99b700 1 --2- 172.21.15.29:0/3623854639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8608153cc0 0x7f86080b17c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f860806c400 tx=0x7f8604009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ea07f3e1b98848 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860c999700 1 -- 172.21.15.29:0/3623854639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604005c10 con 0x7f8608153cc0 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860c999700 1 -- 172.21.15.29:0/3623854639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8604005d70 con 0x7f8608153cc0 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.503+0000 7f860c999700 1 -- 172.21.15.29:0/3623854639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860400b840 con 0x7f8608153cc0 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 -- 172.21.15.29:0/3623854639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8608153cc0 msgr2=0x7f86080b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 --2- 172.21.15.29:0/3623854639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8608153cc0 0x7f86080b17c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f860806c400 tx=0x7f8604009220 comp rx=0 tx=0).stop 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 -- 172.21.15.29:0/3623854639 shutdown_connections 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 --2- 172.21.15.29:0/3623854639 >> v2:172.21.15.144:3300/0 conn(0x7f86080b0710 0x7f86080b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 --2- 172.21.15.29:0/3623854639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8608153cc0 0x7f86080b17c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 -- 172.21.15.29:0/3623854639 >> 172.21.15.29:0/3623854639 conn(0x7f8608067610 msgr2=0x7f8608067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 -- 172.21.15.29:0/3623854639 shutdown_connections 2023-10-07T09:29:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.504+0000 7f860fbff700 1 -- 172.21.15.29:0/3623854639 wait complete. 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.505+0000 7f860fbff700 1 Processor -- start 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.505+0000 7f860fbff700 1 -- start start 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.506+0000 7f860fbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86080b0710 0x7f8608150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.506+0000 7f860fbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8608153cc0 0x7f8608157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.506+0000 7f860fbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 0x7f860815e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.506+0000 7f860fbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608169d30 con 0x7f860815ade0 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.506+0000 7f860fbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608151520 con 0x7f8608153cc0 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.506+0000 7f860fbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8608157fa0 con 0x7f86080b0710 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 0x7f860815e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 0x7f860815e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38648/0 (socket says 172.21.15.29:38648) 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 -- 172.21.15.29:0/96031714 learned_addr learned my addr 172.21.15.29:0/96031714 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860d99b700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86080b0710 0x7f8608150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860d19a700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8608153cc0 0x7f8608157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 -- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86080b0710 msgr2=0x7f8608150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86080b0710 0x7f8608150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 -- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8608153cc0 msgr2=0x7f8608157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8608153cc0 0x7f8608157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 -- 172.21.15.29:0/96031714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604009480 con 0x7f860815ade0 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.507+0000 7f860e19c700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 0x7f860815e840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f860000ce80 tx=0x7f8600007810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.508+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000ac80 con 0x7f860815ade0 2023-10-07T09:29:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.508+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860000ade0 con 0x7f860815ade0 2023-10-07T09:29:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.508+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600016680 con 0x7f860815ade0 2023-10-07T09:29:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.508+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8608158280 con 0x7f860815ade0 2023-10-07T09:29:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.508+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8608169950 con 0x7f860815ade0 2023-10-07T09:29:18.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.509+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860806a740 con 0x7f860815ade0 2023-10-07T09:29:18.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.510+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8600003020 con 0x7f860815ade0 2023-10-07T09:29:18.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.511+0000 7f85fdffb700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85ec066f30 0x7f85ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.511+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f8600012070 con 0x7f860815ade0 2023-10-07T09:29:18.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.511+0000 7f860d99b700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85ec066f30 0x7f85ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.512+0000 7f860d99b700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85ec066f30 0x7f85ec0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8604003080 tx=0x7f8604015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.515+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f86000524d0 con 0x7f860815ade0 2023-10-07T09:29:18.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.719+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8608169ee0 con 0x7f860815ade0 2023-10-07T09:29:18.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.720+0000 7f85fdffb700 1 -- 172.21.15.29:0/96031714 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8600055af0 con 0x7f860815ade0 2023-10-07T09:29:18.721 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85ec066f30 msgr2=0x7f85ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85ec066f30 0x7f85ec0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8604003080 tx=0x7f8604015000 comp rx=0 tx=0).stop 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 msgr2=0x7f860815e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 0x7f860815e840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f860000ce80 tx=0x7f8600007810 comp rx=0 tx=0).stop 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 shutdown_connections 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85ec066f30 0x7f85ec0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f860815ade0 0x7f860815e840 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86080b0710 0x7f8608150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 --2- 172.21.15.29:0/96031714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8608153cc0 0x7f8608157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 >> 172.21.15.29:0/96031714 conn(0x7f8608067610 msgr2=0x7f8608152db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.722+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 shutdown_connections 2023-10-07T09:29:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.723+0000 7f860fbff700 1 -- 172.21.15.29:0/96031714 wait complete. 2023-10-07T09:29:18.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:29:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.935+0000 7f79665ff700 1 Processor -- start 2023-10-07T09:29:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.936+0000 7f79665ff700 1 -- start start 2023-10-07T09:29:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f79665ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f79600d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f79665ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79600cbf20 con 0x7f7960159750 2023-10-07T09:29:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f79665ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79600cc200 con 0x7f79600ce8a0 2023-10-07T09:29:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f79665ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600cc4e0 con 0x7f7960159b20 2023-10-07T09:29:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f795ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7960159750 0x7f79600ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38260/0 (socket says 172.21.15.29:38260) 2023-10-07T09:29:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f795ffff700 1 -- 172.21.15.29:0/2967647562 learned_addr learned my addr 172.21.15.29:0/2967647562 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.937+0000 7f795f7fe700 1 --2- 172.21.15.29:0/2967647562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f79600d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154536545 0 0) 0x7f79600cbf20 con 0x7f7960159750 2023-10-07T09:29:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7948003570 con 0x7f7960159750 2023-10-07T09:29:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346393892 0 0) 0x7f79600cc200 con 0x7f79600ce8a0 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79600cbf20 con 0x7f79600ce8a0 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795f7fe700 1 -- 172.21.15.29:0/2967647562 >> v1:172.21.15.144:6789/0 conn(0x7f79600ce8a0 legacy=0x7f79600c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537751729 0 0) 0x7f7948003570 con 0x7f7960159750 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795f7fe700 1 -- 172.21.15.29:0/2967647562 >> v1:172.21.15.29:6789/0 conn(0x7f7960159750 legacy=0x7f79600ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795f7fe700 1 -- 172.21.15.29:0/2967647562 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79600cc7c0 con 0x7f7960159b20 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.938+0000 7f795f7fe700 1 --2- 172.21.15.29:0/2967647562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f79600d10a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7954009910 tx=0x7f79540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41543083f72a9a1e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79500015f0 con 0x7f7960159b20 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7954011420 con 0x7f7960159b20 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f795effd700 1 -- 172.21.15.29:0/2967647562 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79540117a0 con 0x7f7960159b20 2023-10-07T09:29:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 -- 172.21.15.29:0/2967647562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 msgr2=0x7f79600d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 --2- 172.21.15.29:0/2967647562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f79600d10a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7954009910 tx=0x7f79540091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 -- 172.21.15.29:0/2967647562 shutdown_connections 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 --2- 172.21.15.29:0/2967647562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f79600d10a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 -- 172.21.15.29:0/2967647562 >> 172.21.15.29:0/2967647562 conn(0x7f7960063a10 msgr2=0x7f79600d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 -- 172.21.15.29:0/2967647562 shutdown_connections 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.939+0000 7f79665ff700 1 -- 172.21.15.29:0/2967647562 wait complete. 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.941+0000 7f79665ff700 1 Processor -- start 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.941+0000 7f79665ff700 1 -- start start 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f79665ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 0x7f7960152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f79665ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960159750 0x7f79601538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f79665ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f7960153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f79665ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79600c85d0 con 0x7f79600ce8a0 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f79665ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79600c88b0 con 0x7f7960159750 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f79665ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600c8b90 con 0x7f7960159b20 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 0x7f7960152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 0x7f7960152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38652/0 (socket says 172.21.15.29:38652) 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 -- 172.21.15.29:0/2055308488 learned_addr learned my addr 172.21.15.29:0/2055308488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:18.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795f7fe700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960159750 0x7f79601538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f7964b9c700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f7960153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 -- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 msgr2=0x7f7960153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f7960153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 -- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960159750 msgr2=0x7f79601538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960159750 0x7f79601538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.942+0000 7f795ffff700 1 -- 172.21.15.29:0/2055308488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954009420 con 0x7f79600ce8a0 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.943+0000 7f795ffff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 0x7f7960152290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f795000cb70 tx=0x7f79500016e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.943+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79500070f0 con 0x7f79600ce8a0 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.943+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7950007270 con 0x7f79600ce8a0 2023-10-07T09:29:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.943+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950007630 con 0x7f79600ce8a0 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.943+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79600c8e70 con 0x7f79600ce8a0 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.943+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79601527d0 con 0x7f79600ce8a0 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.944+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796004c3c0 con 0x7f79600ce8a0 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.945+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7950005a00 con 0x7f79600ce8a0 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.946+0000 7f7947fff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f794c066f30 0x7f794c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.946+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f7950083e50 con 0x7f79600ce8a0 2023-10-07T09:29:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.946+0000 7f795f7fe700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f794c066f30 0x7f794c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.947+0000 7f795f7fe700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f794c066f30 0x7f794c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7954009860 tx=0x7f7954016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:18.950+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79500513f0 con 0x7f79600ce8a0 2023-10-07T09:29:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.156+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7960064040 con 0x7f79600ce8a0 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.157+0000 7f7947fff700 1 -- 172.21.15.29:0/2055308488 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7950054a10 con 0x7f79600ce8a0 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:29:19.158 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:29:19.159 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:29:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f794c066f30 msgr2=0x7f794c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f794c066f30 0x7f794c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7954009860 tx=0x7f7954016000 comp rx=0 tx=0).stop 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 msgr2=0x7f7960152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 0x7f7960152290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f795000cb70 tx=0x7f79500016e0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 shutdown_connections 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f794c066f30 0x7f794c0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7960159b20 0x7f7960153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ce8a0 0x7f7960152290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 --2- 172.21.15.29:0/2055308488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960159750 0x7f79601538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 >> 172.21.15.29:0/2055308488 conn(0x7f7960063a10 msgr2=0x7f79601530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 shutdown_connections 2023-10-07T09:29:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.159+0000 7f79665ff700 1 -- 172.21.15.29:0/2055308488 wait complete. 2023-10-07T09:29:19.177 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:29:19.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:29:19.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.375+0000 7fc19b5ff700 1 Processor -- start 2023-10-07T09:29:19.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.376+0000 7fc19b5ff700 1 -- start start 2023-10-07T09:29:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc19b5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1940b2a20 0x7fc1940cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc19b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cb6f0 0x7fc1940cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:19.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc19b5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1940cc940 con 0x7fc1940cf0f0 2023-10-07T09:29:19.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc19b5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1940cf530 con 0x7fc1940b2a20 2023-10-07T09:29:19.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc19b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940b2df0 con 0x7fc1940cb6f0 2023-10-07T09:29:19.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc199b9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc1940cf0f0 0x7fc194159ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38262/0 (socket says 172.21.15.29:38262) 2023-10-07T09:29:19.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc199b9c700 1 -- 172.21.15.29:0/4040755062 learned_addr learned my addr 172.21.15.29:0/4040755062 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:19.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc19939b700 1 --2- 172.21.15.29:0/4040755062 >> v2:172.21.15.144:3300/0 conn(0x7fc1940b2a20 0x7fc1940cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:19.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.377+0000 7fc198b9a700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cb6f0 0x7fc1940cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:19.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc18bfff700 1 -- 172.21.15.29:0/4040755062 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484594485 0 0) 0x7fc1940cc940 con 0x7fc1940cf0f0 2023-10-07T09:29:19.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc18bfff700 1 -- 172.21.15.29:0/4040755062 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc180003570 con 0x7fc1940cf0f0 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc19939b700 1 -- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cb6f0 msgr2=0x7fc1940cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc19939b700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cb6f0 0x7fc1940cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc19939b700 1 -- 172.21.15.29:0/4040755062 >> v1:172.21.15.29:6789/0 conn(0x7fc1940cf0f0 legacy=0x7fc194159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc19939b700 1 -- 172.21.15.29:0/4040755062 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1940cbd30 con 0x7fc1940b2a20 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc198b9a700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cb6f0 0x7fc1940cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.378+0000 7fc19939b700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940b2a20 0x7fc1940cb1b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc1900098d0 tx=0x7fc190009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9769334699ab609 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc18bfff700 1 -- 172.21.15.29:0/4040755062 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190005c10 con 0x7fc1940b2a20 2023-10-07T09:29:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc18bfff700 1 -- 172.21.15.29:0/4040755062 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc190005d70 con 0x7fc1940b2a20 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc18bfff700 1 -- 172.21.15.29:0/4040755062 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190006260 con 0x7fc1940b2a20 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 -- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940b2a20 msgr2=0x7fc1940cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940b2a20 0x7fc1940cb1b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc1900098d0 tx=0x7fc190009120 comp rx=0 tx=0).stop 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 -- 172.21.15.29:0/4040755062 shutdown_connections 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940b2a20 0x7fc1940cb1b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/4040755062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cb6f0 0x7fc1940cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 -- 172.21.15.29:0/4040755062 >> 172.21.15.29:0/4040755062 conn(0x7fc194099820 msgr2=0x7fc194063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 -- 172.21.15.29:0/4040755062 shutdown_connections 2023-10-07T09:29:19.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.379+0000 7fc19b5ff700 1 -- 172.21.15.29:0/4040755062 wait complete. 2023-10-07T09:29:19.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.381+0000 7fc19b5ff700 1 Processor -- start 2023-10-07T09:29:19.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.381+0000 7fc19b5ff700 1 -- start start 2023-10-07T09:29:19.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19b5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 0x7fc194143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:19.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19b5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940cb6f0 0x7fc19415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cf0f0 0x7fc19415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19b5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940b6210 con 0x7fc1940b2a20 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19b5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc194144290 con 0x7fc1940cb6f0 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940b5f30 con 0x7fc1940cf0f0 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 0x7fc194143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 0x7fc194143220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38664/0 (socket says 172.21.15.29:38664) 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 -- 172.21.15.29:0/3101044369 learned_addr learned my addr 172.21.15.29:0/3101044369 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc198b9a700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940cb6f0 0x7fc19415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc199b9c700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cf0f0 0x7fc19415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 -- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cf0f0 msgr2=0x7fc19415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:19.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cf0f0 0x7fc19415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 -- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940cb6f0 msgr2=0x7fc19415aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.382+0000 7fc19939b700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940cb6f0 0x7fc19415aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc19939b700 1 -- 172.21.15.29:0/3101044369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184009420 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc19939b700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 0x7fc194143220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc190006360 tx=0x7fc190016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19001d6b0 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19001d830 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19001dbf0 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1900093c0 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.383+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc194143f20 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.384+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1940718b0 con 0x7fc1940b2a20 2023-10-07T09:29:19.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.385+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc19001b480 con 0x7fc1940b2a20 2023-10-07T09:29:19.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.386+0000 7fc1897fa700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc178066f30 0x7fc1780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:19.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.386+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fc190085fe0 con 0x7fc1940b2a20 2023-10-07T09:29:19.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.387+0000 7fc198b9a700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc178066f30 0x7fc1780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:19.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.387+0000 7fc198b9a700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc178066f30 0x7fc1780693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ea0 tx=0x7fc1840091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:19.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.390+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc190053560 con 0x7fc1940b2a20 2023-10-07T09:29:19.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:19.597+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc1940cce70 con 0x7fc1940b2a20 2023-10-07T09:29:20.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.079+0000 7fc1897fa700 1 -- 172.21.15.29:0/3101044369 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc190056b80 con 0x7fc1940b2a20 2023-10-07T09:29:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.085+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc178066f30 msgr2=0x7fc1780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.085+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc178066f30 0x7fc1780693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ea0 tx=0x7fc1840091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.085+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 msgr2=0x7fc194143220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.085+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 0x7fc194143220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc190006360 tx=0x7fc190016000 comp rx=0 tx=0).stop 2023-10-07T09:29:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.086+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 shutdown_connections 2023-10-07T09:29:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.086+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc178066f30 0x7fc1780693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.086+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1940cf0f0 0x7fc19415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.086+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1940b2a20 0x7fc194143220 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.086+0000 7fc19b5ff700 1 --2- 172.21.15.29:0/3101044369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1940cb6f0 0x7fc19415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.086+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 >> 172.21.15.29:0/3101044369 conn(0x7fc194099820 msgr2=0x7fc1940c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:20.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.087+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 shutdown_connections 2023-10-07T09:29:20.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.087+0000 7fc19b5ff700 1 -- 172.21.15.29:0/3101044369 wait complete. 2023-10-07T09:29:20.092 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:29:20.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:29:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.316+0000 7f00095ff700 1 Processor -- start 2023-10-07T09:29:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.317+0000 7f00095ff700 1 -- start start 2023-10-07T09:29:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00095ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00040cb800 0x7f00040cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00095ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00040d0030 0x7f00040b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00095ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040c62d0 con 0x7f00040cfbb0 2023-10-07T09:29:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00095ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00040d0430 con 0x7f00040d0030 2023-10-07T09:29:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00095ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0004159e10 con 0x7f00040cb800 2023-10-07T09:29:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00037fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00040d0030 0x7f00040b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00037fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00040d0030 0x7f00040b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38670/0 (socket says 172.21.15.29:38670) 2023-10-07T09:29:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.318+0000 7f00037fe700 1 -- 172.21.15.29:0/376191269 learned_addr learned my addr 172.21.15.29:0/376191269 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f0002ffd700 1 --2- 172.21.15.29:0/376191269 >> v2:172.21.15.144:3300/0 conn(0x7f00040cb800 0x7f00040cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f00037fe700 1 -- 172.21.15.29:0/376191269 >> v2:172.21.15.144:3300/0 conn(0x7f00040cb800 msgr2=0x7f00040cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f00037fe700 1 --2- 172.21.15.29:0/376191269 >> v2:172.21.15.144:3300/0 conn(0x7f00040cb800 0x7f00040cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f00037fe700 1 -- 172.21.15.29:0/376191269 >> v1:172.21.15.144:6789/0 conn(0x7f00040cfbb0 legacy=0x7f00040c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f00037fe700 1 -- 172.21.15.29:0/376191269 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040cc110 con 0x7f00040d0030 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f00037fe700 1 --2- 172.21.15.29:0/376191269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040d0030 0x7f00040b3340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00040896f0 tx=0x7efffc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4ea800c4b687f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f0001ffb700 1 -- 172.21.15.29:0/376191269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00e070 con 0x7f00040d0030 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f0001ffb700 1 -- 172.21.15.29:0/376191269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc00baa0 con 0x7f00040d0030 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.319+0000 7f0001ffb700 1 -- 172.21.15.29:0/376191269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc0092e0 con 0x7f00040d0030 2023-10-07T09:29:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 -- 172.21.15.29:0/376191269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040d0030 msgr2=0x7f00040b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 --2- 172.21.15.29:0/376191269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040d0030 0x7f00040b3340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00040896f0 tx=0x7efffc009840 comp rx=0 tx=0).stop 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 -- 172.21.15.29:0/376191269 shutdown_connections 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 --2- 172.21.15.29:0/376191269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040d0030 0x7f00040b3340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 --2- 172.21.15.29:0/376191269 >> v2:172.21.15.144:3300/0 conn(0x7f00040cb800 0x7f00040cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 -- 172.21.15.29:0/376191269 >> 172.21.15.29:0/376191269 conn(0x7f0004063a10 msgr2=0x7f00040d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 -- 172.21.15.29:0/376191269 shutdown_connections 2023-10-07T09:29:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.320+0000 7f00095ff700 1 -- 172.21.15.29:0/376191269 wait complete. 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.322+0000 7f00095ff700 1 Processor -- start 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.322+0000 7f00095ff700 1 -- start start 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00095ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00040cb800 0x7f000409a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00095ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040cfbb0 0x7f0004155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00095ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00040d0030 0x7f000409db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00095ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000409b430 con 0x7f00040cfbb0 2023-10-07T09:29:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00095ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0004155b80 con 0x7f00040d0030 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00095ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000409e080 con 0x7f00040cb800 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00027fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040cfbb0 0x7f0004155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f0002ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00040cb800 0x7f000409a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f0002ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00040cb800 0x7f000409a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:55136/0 (socket says 172.21.15.29:55136) 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f0002ffd700 1 -- 172.21.15.29:0/422698296 learned_addr learned my addr 172.21.15.29:0/422698296 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00037fe700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00040d0030 0x7f000409db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00027fc700 1 -- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00040cb800 msgr2=0x7f000409a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00027fc700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00040cb800 0x7f000409a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00027fc700 1 -- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00040d0030 msgr2=0x7f000409db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00027fc700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00040d0030 0x7f000409db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00027fc700 1 -- 172.21.15.29:0/422698296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc009ae0 con 0x7f00040cfbb0 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.323+0000 7f00037fe700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00040d0030 0x7f000409db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.324+0000 7f00027fc700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040cfbb0 0x7f0004155640 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efff800c4e0 tx=0x7efff8007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.324+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff8011590 con 0x7f00040cfbb0 2023-10-07T09:29:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.324+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff8011710 con 0x7f00040cfbb0 2023-10-07T09:29:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.324+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff8011ad0 con 0x7f00040cfbb0 2023-10-07T09:29:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.324+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000409a820 con 0x7f00040cfbb0 2023-10-07T09:29:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.324+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000409ade0 con 0x7f00040cfbb0 2023-10-07T09:29:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.325+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00040718b0 con 0x7f00040cfbb0 2023-10-07T09:29:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.326+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efff801c460 con 0x7f00040cfbb0 2023-10-07T09:29:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.327+0000 7effeaffd700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efff0066f30 0x7efff00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.327+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5223+0+0 (secure 0 0 0) 0x7efff8084090 con 0x7f00040cfbb0 2023-10-07T09:29:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.328+0000 7f0002ffd700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efff0066f30 0x7efff00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.328+0000 7f0002ffd700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efff0066f30 0x7efff00693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efff4007ce0 tx=0x7efff4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:20.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.332+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efff8051440 con 0x7f00040cfbb0 2023-10-07T09:29:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:20.533+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f000409b570 con 0x7f00040cfbb0 2023-10-07T09:29:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.086+0000 7effeaffd700 1 -- 172.21.15.29:0/422698296 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7f000409b570 con 0x7f00040cfbb0 2023-10-07T09:29:21.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.095+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efff0066f30 msgr2=0x7efff00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:21.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.095+0000 7f00095ff700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efff0066f30 0x7efff00693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efff4007ce0 tx=0x7efff4008040 comp rx=0 tx=0).stop 2023-10-07T09:29:21.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.095+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040cfbb0 msgr2=0x7f0004155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:21.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.095+0000 7f00095ff700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040cfbb0 0x7f0004155640 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efff800c4e0 tx=0x7efff8007ea0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 shutdown_connections 2023-10-07T09:29:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efff0066f30 0x7efff00693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00040d0030 0x7f000409db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00040cb800 0x7f000409a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 --2- 172.21.15.29:0/422698296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00040cfbb0 0x7f0004155640 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 >> 172.21.15.29:0/422698296 conn(0x7f0004063a10 msgr2=0x7f0004158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.097+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 shutdown_connections 2023-10-07T09:29:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.098+0000 7f00095ff700 1 -- 172.21.15.29:0/422698296 wait complete. 2023-10-07T09:29:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:29:21.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:29:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.331+0000 7f2f129ff700 1 Processor -- start 2023-10-07T09:29:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.332+0000 7f2f129ff700 1 -- start start 2023-10-07T09:29:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f129ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f0c159750 0x7f2f0c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f129ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f129ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c0cbf20 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f129ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c0cc200 con 0x7f2f0c159750 2023-10-07T09:29:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f129ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0cc4e0 con 0x7f2f0c159b20 2023-10-07T09:29:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f0bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f0c159750 0x7f2f0c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f0bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f0c159750 0x7f2f0c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38698/0 (socket says 172.21.15.29:38698) 2023-10-07T09:29:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f0bfff700 1 -- 172.21.15.29:0/2452243347 learned_addr learned my addr 172.21.15.29:0/2452243347 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.333+0000 7f2f0b7fe700 1 --2- 172.21.15.29:0/2452243347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0bfff700 1 -- 172.21.15.29:0/2452243347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 msgr2=0x7f2f0c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0bfff700 1 --2- 172.21.15.29:0/2452243347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0bfff700 1 -- 172.21.15.29:0/2452243347 >> v1:172.21.15.29:6789/0 conn(0x7f2f0c0ce6d0 legacy=0x7f2f0c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0bfff700 1 -- 172.21.15.29:0/2452243347 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0cc7c0 con 0x7f2f0c159750 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0affd700 1 -- 172.21.15.29:0/2452243347 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206525708 0 0) 0x7f2f0c0cbf20 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0bfff700 1 --2- 172.21.15.29:0/2452243347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c159750 0x7f2f0c0d10a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009990 tx=0x7f2efc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9b2070fc72ea49 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0affd700 1 -- 172.21.15.29:0/2452243347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc009e80 con 0x7f2f0c159750 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0affd700 1 -- 172.21.15.29:0/2452243347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc005c10 con 0x7f2f0c159750 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.334+0000 7f2f0affd700 1 -- 172.21.15.29:0/2452243347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00b650 con 0x7f2f0c159750 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 -- 172.21.15.29:0/2452243347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c159750 msgr2=0x7f2f0c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 --2- 172.21.15.29:0/2452243347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c159750 0x7f2f0c0d10a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009990 tx=0x7f2efc009220 comp rx=0 tx=0).stop 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 -- 172.21.15.29:0/2452243347 shutdown_connections 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 --2- 172.21.15.29:0/2452243347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c159750 0x7f2f0c0d10a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 --2- 172.21.15.29:0/2452243347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 -- 172.21.15.29:0/2452243347 >> 172.21.15.29:0/2452243347 conn(0x7f2f0c063a10 msgr2=0x7f2f0c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 -- 172.21.15.29:0/2452243347 shutdown_connections 2023-10-07T09:29:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.335+0000 7f2f129ff700 1 -- 172.21.15.29:0/2452243347 wait complete. 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.336+0000 7f2f129ff700 1 Processor -- start 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 -- start start 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 0x7f2f0c15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c159750 0x7f2f0c14b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c15fe90 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c15f4c0 con 0x7f2f0c159750 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.337+0000 7f2f129ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c15e8b0 con 0x7f2f0c159b20 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 0x7f2f0c15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 0x7f2f0c15f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38706/0 (socket says 172.21.15.29:38706) 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 -- 172.21.15.29:0/248869947 learned_addr learned my addr 172.21.15.29:0/248869947 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0b7fe700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c159750 0x7f2f0c14b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f10f9c700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 -- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 msgr2=0x7f2f0c153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 -- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c159750 msgr2=0x7f2f0c14b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c159750 0x7f2f0c14b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 -- 172.21.15.29:0/248869947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc009480 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f10f9c700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.338+0000 7f2f0bfff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 0x7f2f0c15f950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00b3c0 tx=0x7f2efc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.339+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc01dc00 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.339+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc01dd80 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.339+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc01b6d0 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.339+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c15eb90 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.339+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c15f150 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.340+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c15f7a0 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.341+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2efc01ba40 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.342+0000 7f2ef3fff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2ef8066f30 0x7f2ef80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.342+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5605+0+0 (secure 0 0 0) 0x7f2efc0864d0 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.342+0000 7f2f0b7fe700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2ef8066f30 0x7f2ef80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.342+0000 7f2f0b7fe700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2ef8066f30 0x7f2ef80693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c160240 tx=0x7f2f000091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:21.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.346+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2efc053820 con 0x7f2f0c0ce6d0 2023-10-07T09:29:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:21.596+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2f0c04c3c0 con 0x7f2f0c0ce6d0 2023-10-07T09:29:22.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.088+0000 7f2ef3fff700 1 -- 172.21.15.29:0/248869947 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2efc02b0c0 con 0x7f2f0c0ce6d0 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2ef8066f30 msgr2=0x7f2ef80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2ef8066f30 0x7f2ef80693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c160240 tx=0x7f2f000091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 msgr2=0x7f2f0c15f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 0x7f2f0c15f950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00b3c0 tx=0x7f2efc016000 comp rx=0 tx=0).stop 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 shutdown_connections 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2ef8066f30 0x7f2ef80693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f0c159b20 0x7f2f0c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f0c0ce6d0 0x7f2f0c15f950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 --2- 172.21.15.29:0/248869947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c159750 0x7f2f0c14b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 >> 172.21.15.29:0/248869947 conn(0x7f2f0c063a10 msgr2=0x7f2f0c0c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:22.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 shutdown_connections 2023-10-07T09:29:22.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.092+0000 7f2f129ff700 1 -- 172.21.15.29:0/248869947 wait complete. 2023-10-07T09:29:22.100 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2023-10-07T09:29:22.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.315+0000 7fc3b4fff700 1 Processor -- start 2023-10-07T09:29:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.315+0000 7fc3b4fff700 1 -- start start 2023-10-07T09:29:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.316+0000 7fc3b4fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3b015ade0 0x7fc3b00b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.316+0000 7fc3b4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00b0540 0x7fc3b015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.316+0000 7fc3b4fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b00b09b0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.316+0000 7fc3b4fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0154090 con 0x7fc3b015ade0 2023-10-07T09:29:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.316+0000 7fc3b4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0159f60 con 0x7fc3b00b0540 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3b015ade0 0x7fc3b00b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3b015ade0 0x7fc3b00b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38722/0 (socket says 172.21.15.29:38722) 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 -- 172.21.15.29:0/995372068 learned_addr learned my addr 172.21.15.29:0/995372068 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3aed9d700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00b0540 0x7fc3b015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 -- 172.21.15.29:0/995372068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00b0540 msgr2=0x7fc3b015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00b0540 0x7fc3b015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 -- 172.21.15.29:0/995372068 >> v1:172.21.15.144:6789/0 conn(0x7fc3b0153cc0 legacy=0x7fc3b00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 -- 172.21.15.29:0/995372068 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b015a240 con 0x7fc3b015ade0 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3aed9d700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00b0540 0x7fc3b015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3add9b700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b015ade0 0x7fc3b00b0000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40099b0 tx=0x7fc3a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e495f3b74f12d481 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.317+0000 7fc3ad59a700 1 -- 172.21.15.29:0/995372068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a400b860 con 0x7fc3b015ade0 2023-10-07T09:29:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3ad59a700 1 -- 172.21.15.29:0/995372068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a400b9c0 con 0x7fc3b015ade0 2023-10-07T09:29:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3ad59a700 1 -- 172.21.15.29:0/995372068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a400bd60 con 0x7fc3b015ade0 2023-10-07T09:29:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 -- 172.21.15.29:0/995372068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b015ade0 msgr2=0x7fc3b00b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b015ade0 0x7fc3b00b0000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40099b0 tx=0x7fc3a4009250 comp rx=0 tx=0).stop 2023-10-07T09:29:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 -- 172.21.15.29:0/995372068 shutdown_connections 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00b0540 0x7fc3b015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/995372068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b015ade0 0x7fc3b00b0000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 -- 172.21.15.29:0/995372068 >> 172.21.15.29:0/995372068 conn(0x7fc3b0067610 msgr2=0x7fc3b0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 -- 172.21.15.29:0/995372068 shutdown_connections 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.318+0000 7fc3b4fff700 1 -- 172.21.15.29:0/995372068 wait complete. 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.320+0000 7fc3b4fff700 1 Processor -- start 2023-10-07T09:29:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.320+0000 7fc3b4fff700 1 -- start start 2023-10-07T09:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3b4fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00b0540 0x7fc3b0155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3b4fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 0x7fc3b015e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3b4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b015ade0 0x7fc3b015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3b4fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b01566d0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3b4fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b016a760 con 0x7fc3b00b0540 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3b4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0155b90 con 0x7fc3b015ade0 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3add9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 0x7fc3b015e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3add9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 0x7fc3b015e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38734/0 (socket says 172.21.15.29:38734) 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3add9b700 1 -- 172.21.15.29:0/3941899851 learned_addr learned my addr 172.21.15.29:0/3941899851 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3aed9d700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b015ade0 0x7fc3b015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.321+0000 7fc3ae59c700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00b0540 0x7fc3b0155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3add9b700 1 -- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b015ade0 msgr2=0x7fc3b015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3add9b700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b015ade0 0x7fc3b015e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3add9b700 1 -- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00b0540 msgr2=0x7fc3b0155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3add9b700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00b0540 0x7fc3b0155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3add9b700 1 -- 172.21.15.29:0/3941899851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40094f0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3add9b700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 0x7fc3b015e490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009290 tx=0x7fc3a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4011e00 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a401f070 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40266d0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b0155e30 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.322+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b01563f0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.323+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b016a8d0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.324+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc3a4026a40 con 0x7fc3b0153cc0 2023-10-07T09:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.325+0000 7fc39e7fc700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc390066f30 0x7fc3900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.325+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc3a4021070 con 0x7fc3b0153cc0 2023-10-07T09:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.326+0000 7fc3ae59c700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc390066f30 0x7fc3900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.327+0000 7fc3ae59c700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc390066f30 0x7fc3900693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3b015f090 tx=0x7fc3a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.329+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3a40b75a0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.531+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3b004c3c0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.532+0000 7fc39e7fc700 1 -- 172.21.15.29:0/3941899851 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15459 (secure 0 0 0) 0x7fc3a40533f0 con 0x7fc3b0153cc0 2023-10-07T09:29:22.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:22.533 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:21.598397+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc390066f30 msgr2=0x7fc3900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc390066f30 0x7fc3900693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3b015f090 tx=0x7fc3a0009040 comp rx=0 tx=0).stop 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 msgr2=0x7fc3b015e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 0x7fc3b015e490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009290 tx=0x7fc3a4017000 comp rx=0 tx=0).stop 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 shutdown_connections 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc390066f30 0x7fc3900693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b015ade0 0x7fc3b015e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00b0540 0x7fc3b0155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 --2- 172.21.15.29:0/3941899851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3b0153cc0 0x7fc3b015e490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 >> 172.21.15.29:0/3941899851 conn(0x7fc3b0067610 msgr2=0x7fc3b00af500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 shutdown_connections 2023-10-07T09:29:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.534+0000 7fc3b4fff700 1 -- 172.21.15.29:0/3941899851 wait complete. 2023-10-07T09:29:22.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.748+0000 7f5b01bff700 1 Processor -- start 2023-10-07T09:29:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.748+0000 7f5b01bff700 1 -- start start 2023-10-07T09:29:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5b01bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159750 0x7f5afc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5b01bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5afc159b20 0x7f5afc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5b01bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0cbf20 con 0x7f5afc0ce6d0 2023-10-07T09:29:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5b01bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc0cc200 con 0x7f5afc159b20 2023-10-07T09:29:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5b01bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0cc4e0 con 0x7f5afc159750 2023-10-07T09:29:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5afaffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5afc159b20 0x7f5afc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5afbfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5afc0ce6d0 0x7f5afc15d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38290/0 (socket says 172.21.15.29:38290) 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5afbfff700 1 -- 172.21.15.29:0/2041345909 learned_addr learned my addr 172.21.15.29:0/2041345909 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.749+0000 7f5afb7fe700 1 --2- 172.21.15.29:0/2041345909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159750 0x7f5afc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afaffd700 1 -- 172.21.15.29:0/2041345909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159750 msgr2=0x7f5afc0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afa7fc700 1 -- 172.21.15.29:0/2041345909 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705435791 0 0) 0x7f5afc0cbf20 con 0x7f5afc0ce6d0 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afaffd700 1 --2- 172.21.15.29:0/2041345909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159750 0x7f5afc0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afaffd700 1 -- 172.21.15.29:0/2041345909 >> v1:172.21.15.29:6789/0 conn(0x7f5afc0ce6d0 legacy=0x7f5afc15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afaffd700 1 -- 172.21.15.29:0/2041345909 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0cc7c0 con 0x7f5afc159b20 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afaffd700 1 --2- 172.21.15.29:0/2041345909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159b20 0x7f5afc0ce190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5af00098e0 tx=0x7f5af00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08ad7b5748627f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afa7fc700 1 -- 172.21.15.29:0/2041345909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000b640 con 0x7f5afc159b20 2023-10-07T09:29:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afa7fc700 1 -- 172.21.15.29:0/2041345909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af000b7a0 con 0x7f5afc159b20 2023-10-07T09:29:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.750+0000 7f5afa7fc700 1 -- 172.21.15.29:0/2041345909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000bb40 con 0x7f5afc159b20 2023-10-07T09:29:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 -- 172.21.15.29:0/2041345909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159b20 msgr2=0x7f5afc0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 --2- 172.21.15.29:0/2041345909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159b20 0x7f5afc0ce190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5af00098e0 tx=0x7f5af00091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 -- 172.21.15.29:0/2041345909 shutdown_connections 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 --2- 172.21.15.29:0/2041345909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159750 0x7f5afc0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 --2- 172.21.15.29:0/2041345909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159b20 0x7f5afc0ce190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 -- 172.21.15.29:0/2041345909 >> 172.21.15.29:0/2041345909 conn(0x7f5afc063a10 msgr2=0x7f5afc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 -- 172.21.15.29:0/2041345909 shutdown_connections 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.751+0000 7f5b01bff700 1 -- 172.21.15.29:0/2041345909 wait complete. 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.752+0000 7f5b01bff700 1 Processor -- start 2023-10-07T09:29:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.753+0000 7f5b01bff700 1 -- start start 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5b01bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5afc0ce6d0 0x7f5afc0c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5b01bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 0x7f5afc15e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5b01bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159b20 0x7f5afc0c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5b01bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0b4470 con 0x7f5afc159750 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5b01bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0c81d0 con 0x7f5afc0ce6d0 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5b01bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0c84b0 con 0x7f5afc159b20 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 0x7f5afc15e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 0x7f5afc15e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38752/0 (socket says 172.21.15.29:38752) 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 -- 172.21.15.29:0/3608038345 learned_addr learned my addr 172.21.15.29:0/3608038345 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afb7fe700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5afc0ce6d0 0x7f5afc0c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afbfff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159b20 0x7f5afc0c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 -- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159b20 msgr2=0x7f5afc0c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159b20 0x7f5afc0c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 -- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5afc0ce6d0 msgr2=0x7f5afc0c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5afc0ce6d0 0x7f5afc0c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 -- 172.21.15.29:0/3608038345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af0009420 con 0x7f5afc159750 2023-10-07T09:29:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.754+0000 7f5afaffd700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 0x7f5afc15e800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5afc088a40 tx=0x7f5af0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.755+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000bb20 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.755+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0000c90 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.755+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc0b4100 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.755+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0011630 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.755+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc153870 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.756+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc0b45e0 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.757+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5af00119a0 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.758+0000 7f5ae37fe700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5ae8066f30 0x7f5ae80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.758+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5af0085840 con 0x7f5afc159750 2023-10-07T09:29:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.758+0000 7f5afb7fe700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5ae8066f30 0x7f5ae80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.759+0000 7f5afb7fe700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5ae8066f30 0x7f5ae80693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009da0 tx=0x7f5aec009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.762+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5af0052ab0 con 0x7f5afc159750 2023-10-07T09:29:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.970+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5afc04c3c0 con 0x7f5afc159750 2023-10-07T09:29:22.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.971+0000 7f5ae37fe700 1 -- 172.21.15.29:0/3608038345 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f5af00560d0 con 0x7f5afc159750 2023-10-07T09:29:22.972 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:22.972 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.973+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5ae8066f30 msgr2=0x7f5ae80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.973+0000 7f5b01bff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5ae8066f30 0x7f5ae80693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009da0 tx=0x7f5aec009120 comp rx=0 tx=0).stop 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.973+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 msgr2=0x7f5afc15e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.973+0000 7f5b01bff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 0x7f5afc15e800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5afc088a40 tx=0x7f5af0017000 comp rx=0 tx=0).stop 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 shutdown_connections 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5ae8066f30 0x7f5ae80693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5afc159b20 0x7f5afc0c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5afc0ce6d0 0x7f5afc0c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 --2- 172.21.15.29:0/3608038345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5afc159750 0x7f5afc15e800 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 >> 172.21.15.29:0/3608038345 conn(0x7f5afc063a10 msgr2=0x7f5afc0cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 shutdown_connections 2023-10-07T09:29:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:22.974+0000 7f5b01bff700 1 -- 172.21.15.29:0/3608038345 wait complete. 2023-10-07T09:29:22.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:29:23.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.192+0000 7fed5a3ff700 1 Processor -- start 2023-10-07T09:29:23.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.192+0000 7fed5a3ff700 1 -- start start 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed5a3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed540b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed5a3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed540b0710 0x7fed540b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed5a3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed54154090 con 0x7fed5415ade0 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed5a3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed54159f60 con 0x7fed540b0710 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed5a3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5415a240 con 0x7fed54153cc0 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed537fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed5415ade0 0x7fed540b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36856/0 (socket says 172.21.15.29:36856) 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.193+0000 7fed537fe700 1 -- 172.21.15.29:0/2209037023 learned_addr learned my addr 172.21.15.29:0/2209037023 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed53fff700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed540b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed5899c700 1 --2- 172.21.15.29:0/2209037023 >> v2:172.21.15.144:3300/0 conn(0x7fed540b0710 0x7fed540b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810247348 0 0) 0x7fed54154090 con 0x7fed5415ade0 2023-10-07T09:29:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3c003570 con 0x7fed5415ade0 2023-10-07T09:29:23.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098496016 0 0) 0x7fed3c003570 con 0x7fed5415ade0 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 msgr2=0x7fed540b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed540b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 >> v2:172.21.15.144:3300/0 conn(0x7fed540b0710 msgr2=0x7fed540b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 --2- 172.21.15.29:0/2209037023 >> v2:172.21.15.144:3300/0 conn(0x7fed540b0710 0x7fed540b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5415a520 con 0x7fed5415ade0 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fed48001470 con 0x7fed5415ade0 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 >> v1:172.21.15.29:6789/0 conn(0x7fed5415ade0 legacy=0x7fed540b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed3c003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3c003fe0 0x7fed3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3c004d50 0x7fed3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54159f60 con 0x7fed3c004d50 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5415a240 con 0x7fed3c003fe0 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.194+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c00ba90 con 0x7fed54153cc0 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3c004d50 0x7fed3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed53fff700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed3c003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed537fe700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3c003fe0 0x7fed3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 -- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 msgr2=0x7fed3c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed3c003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 -- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3c003fe0 msgr2=0x7fed3c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3c003fe0 0x7fed3c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 -- 172.21.15.29:0/2209037023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c003570 con 0x7fed3c004d50 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed53fff700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed54153cc0 0x7fed3c003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed5899c700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3c004d50 0x7fed3c0051c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed540b0b10 tx=0x7fed4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1535254e570ef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.195+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00ea30 con 0x7fed3c004d50 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c00f040 con 0x7fed3c004d50 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed52ffd700 1 -- 172.21.15.29:0/2209037023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00ede0 con 0x7fed3c004d50 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2209037023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3c004d50 msgr2=0x7fed3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2209037023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3c004d50 0x7fed3c0051c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed540b0b10 tx=0x7fed4c00b040 comp rx=0 tx=0).stop 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5899c700 1 -- 172.21.15.29:0/2209037023 reap_dead start 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2209037023 shutdown_connections 2023-10-07T09:29:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2209037023 >> 172.21.15.29:0/2209037023 conn(0x7fed54067610 msgr2=0x7fed54067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2209037023 shutdown_connections 2023-10-07T09:29:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.196+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2209037023 wait complete. 2023-10-07T09:29:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.198+0000 7fed5a3ff700 1 Processor -- start 2023-10-07T09:29:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.198+0000 7fed5a3ff700 1 -- start start 2023-10-07T09:29:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5a3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3c004d50 0x7fed54155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5a3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed54159330 0x7fed54169790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5a3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 0x7fed5415cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5a3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5416a030 con 0x7fed5414ea50 2023-10-07T09:29:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5a3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5416a310 con 0x7fed54159330 2023-10-07T09:29:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5a3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5416a5f0 con 0x7fed3c004d50 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 0x7fed5415cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed53fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3c004d50 0x7fed54155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 0x7fed5415cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58376/0 (socket says 172.21.15.29:58376) 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 -- 172.21.15.29:0/2629535559 learned_addr learned my addr 172.21.15.29:0/2629535559 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed537fe700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed54159330 0x7fed54169790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 -- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3c004d50 msgr2=0x7fed54155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3c004d50 0x7fed54155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 -- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed54159330 msgr2=0x7fed54169790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed54159330 0x7fed54169790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed5899c700 1 -- 172.21.15.29:0/2629535559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5415ef90 con 0x7fed5414ea50 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.199+0000 7fed53fff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3c004d50 0x7fed54155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.200+0000 7fed5899c700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 0x7fed5415cbc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009750 tx=0x7fed4c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.200+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c01e640 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.200+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c022070 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.200+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c01e9d0 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.200+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed54169cd0 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.200+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5414eed0 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.201+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5414f0c0 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.202+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fed4c003040 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.203+0000 7fed3bfff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fed40066f30 0x7fed400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.203+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fed4c024070 con 0x7fed5414ea50 2023-10-07T09:29:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.203+0000 7fed53fff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fed40066f30 0x7fed400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:23.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.204+0000 7fed53fff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fed40066f30 0x7fed400693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fed44009710 tx=0x7fed44009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:23.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.207+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed4c053990 con 0x7fed5414ea50 2023-10-07T09:29:23.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:23.459+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fed5404c3c0 con 0x7fed5414ea50 2023-10-07T09:29:24.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.105+0000 7fed3bfff700 1 -- 172.21.15.29:0/2629535559 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fed4c056fb0 con 0x7fed5414ea50 2023-10-07T09:29:24.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.108+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fed40066f30 msgr2=0x7fed400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:24.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.108+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fed40066f30 0x7fed400693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fed44009710 tx=0x7fed44009120 comp rx=0 tx=0).stop 2023-10-07T09:29:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.108+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 msgr2=0x7fed5415cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.108+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 0x7fed5415cbc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009750 tx=0x7fed4c01a000 comp rx=0 tx=0).stop 2023-10-07T09:29:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 shutdown_connections 2023-10-07T09:29:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fed40066f30 0x7fed400693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed5414ea50 0x7fed5415cbc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3c004d50 0x7fed54155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 --2- 172.21.15.29:0/2629535559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed54159330 0x7fed54169790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 >> 172.21.15.29:0/2629535559 conn(0x7fed54067610 msgr2=0x7fed540af610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:24.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 shutdown_connections 2023-10-07T09:29:24.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.109+0000 7fed5a3ff700 1 -- 172.21.15.29:0/2629535559 wait complete. 2023-10-07T09:29:24.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:29:24.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.328+0000 7efeb13ff700 1 Processor -- start 2023-10-07T09:29:24.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.329+0000 7efeb13ff700 1 -- start start 2023-10-07T09:29:24.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeb13ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efeac06a8c0 0x7efeac0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:24.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeb13ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efeac06b5c0 0x7efeac0c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:24.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeb13ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac06a430 con 0x7efeac159790 2023-10-07T09:29:24.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeb13ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeac159b60 con 0x7efeac06b5c0 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeb13ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeac069820 con 0x7efeac06a8c0 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeab7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efeac06b5c0 0x7efeac0c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeab7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efeac06b5c0 0x7efeac0c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58390/0 (socket says 172.21.15.29:58390) 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeab7fe700 1 -- 172.21.15.29:0/54876577 learned_addr learned my addr 172.21.15.29:0/54876577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeaaffd700 1 --1- 172.21.15.29:0/54876577 >> v1:172.21.15.29:6789/0 conn(0x7efeac159790 0x7efeac06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36870/0 (socket says 172.21.15.29:36870) 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.330+0000 7efeaa7fc700 1 --2- 172.21.15.29:0/54876577 >> v2:172.21.15.144:3300/0 conn(0x7efeac06a8c0 0x7efeac0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efeab7fe700 1 -- 172.21.15.29:0/54876577 >> v2:172.21.15.144:3300/0 conn(0x7efeac06a8c0 msgr2=0x7efeac0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efeab7fe700 1 --2- 172.21.15.29:0/54876577 >> v2:172.21.15.144:3300/0 conn(0x7efeac06a8c0 0x7efeac0c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efeab7fe700 1 -- 172.21.15.29:0/54876577 >> v1:172.21.15.29:6789/0 conn(0x7efeac159790 legacy=0x7efeac06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:24.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efeab7fe700 1 -- 172.21.15.29:0/54876577 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac069b00 con 0x7efeac06b5c0 2023-10-07T09:29:24.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efea9ffb700 1 -- 172.21.15.29:0/54876577 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839602867 0 0) 0x7efeac06a430 con 0x7efeac159790 2023-10-07T09:29:24.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efeab7fe700 1 --2- 172.21.15.29:0/54876577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac06b5c0 0x7efeac0c7480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efeac06d820 tx=0x7efea4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6aff8ed5b781ff server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:24.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efea9ffb700 1 -- 172.21.15.29:0/54876577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400e070 con 0x7efeac06b5c0 2023-10-07T09:29:24.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efea9ffb700 1 -- 172.21.15.29:0/54876577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea400ba20 con 0x7efeac06b5c0 2023-10-07T09:29:24.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.331+0000 7efea9ffb700 1 -- 172.21.15.29:0/54876577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea40092e0 con 0x7efeac06b5c0 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 -- 172.21.15.29:0/54876577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac06b5c0 msgr2=0x7efeac0c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 --2- 172.21.15.29:0/54876577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac06b5c0 0x7efeac0c7480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efeac06d820 tx=0x7efea4009840 comp rx=0 tx=0).stop 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 -- 172.21.15.29:0/54876577 shutdown_connections 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 --2- 172.21.15.29:0/54876577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac06b5c0 0x7efeac0c7480 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 --2- 172.21.15.29:0/54876577 >> v2:172.21.15.144:3300/0 conn(0x7efeac06a8c0 0x7efeac0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 -- 172.21.15.29:0/54876577 >> 172.21.15.29:0/54876577 conn(0x7efeac0c9410 msgr2=0x7efeac0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 -- 172.21.15.29:0/54876577 shutdown_connections 2023-10-07T09:29:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.332+0000 7efeb13ff700 1 -- 172.21.15.29:0/54876577 wait complete. 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.333+0000 7efeb13ff700 1 Processor -- start 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.333+0000 7efeb13ff700 1 -- start start 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.334+0000 7efeb13ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efeac06b5c0 0x7efeac0c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.334+0000 7efeb13ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 0x7efeac0b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.334+0000 7efeb13ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac0b2390 0x7efeac16cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.334+0000 7efeb13ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0aed70 con 0x7efeac159790 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.334+0000 7efeb13ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0c37a0 con 0x7efeac0b2390 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.334+0000 7efeb13ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0b0e10 con 0x7efeac06b5c0 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 0x7efeac0b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 0x7efeac0b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58394/0 (socket says 172.21.15.29:58394) 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 -- 172.21.15.29:0/1861909007 learned_addr learned my addr 172.21.15.29:0/1861909007 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeab7fe700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac0b2390 0x7efeac16cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaaffd700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efeac06b5c0 0x7efeac0c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 -- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efeac06b5c0 msgr2=0x7efeac0c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efeac06b5c0 0x7efeac0c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 -- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac0b2390 msgr2=0x7efeac16cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac0b2390 0x7efeac16cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 -- 172.21.15.29:0/1861909007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea4009ae0 con 0x7efeac159790 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeab7fe700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac0b2390 0x7efeac16cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.335+0000 7efeaa7fc700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 0x7efeac0b1e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efea000e960 tx=0x7efea00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:24.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.336+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea000c850 con 0x7efeac159790 2023-10-07T09:29:24.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.336+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea000c9b0 con 0x7efeac159790 2023-10-07T09:29:24.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.336+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea000cd50 con 0x7efeac159790 2023-10-07T09:29:24.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.336+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac0b12a0 con 0x7efeac159790 2023-10-07T09:29:24.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.336+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac0b04e0 con 0x7efeac159790 2023-10-07T09:29:24.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.337+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac0b0f80 con 0x7efeac159790 2023-10-07T09:29:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.338+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efea0013680 con 0x7efeac159790 2023-10-07T09:29:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.339+0000 7efe92ffd700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efe98066f30 0x7efe980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.339+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7efea0084800 con 0x7efeac159790 2023-10-07T09:29:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.339+0000 7efeaaffd700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efe98066f30 0x7efe980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.340+0000 7efeaaffd700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efe98066f30 0x7efe980693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0011e0 tx=0x7efe9c007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.343+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efea00519d0 con 0x7efeac159790 2023-10-07T09:29:24.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:24.589+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7efeac04c3c0 con 0x7efeac159790 2023-10-07T09:29:25.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.107+0000 7efe92ffd700 1 -- 172.21.15.29:0/1861909007 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7efea0054ff0 con 0x7efeac159790 2023-10-07T09:29:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efe98066f30 msgr2=0x7efe980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efe98066f30 0x7efe980693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0011e0 tx=0x7efe9c007bf0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 msgr2=0x7efeac0b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 0x7efeac0b1e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efea000e960 tx=0x7efea00091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 shutdown_connections 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efe98066f30 0x7efe980693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac0b2390 0x7efeac16cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efeac06b5c0 0x7efeac0c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 --2- 172.21.15.29:0/1861909007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efeac159790 0x7efeac0b1e50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 >> 172.21.15.29:0/1861909007 conn(0x7efeac0c9410 msgr2=0x7efeac0cc120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 shutdown_connections 2023-10-07T09:29:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.110+0000 7efeb13ff700 1 -- 172.21.15.29:0/1861909007 wait complete. 2023-10-07T09:29:25.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:29:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.324+0000 7f5077bff700 1 Processor -- start 2023-10-07T09:29:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.324+0000 7f5077bff700 1 -- start start 2023-10-07T09:29:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f5077bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 0x7f50700cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f5077bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50700ae8b0 0x7f50700b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f5077bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5070158ad0 con 0x7f50700b1840 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f5077bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50700b1cc0 con 0x7f50700ae8b0 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f5077bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507015b140 con 0x7f5070157290 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f507619c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f50700b1840 0x7f50700ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36872/0 (socket says 172.21.15.29:36872) 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.325+0000 7f507619c700 1 -- 172.21.15.29:0/1876303356 learned_addr learned my addr 172.21.15.29:0/1876303356 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f507599b700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 0x7f50700cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f507519a700 1 --2- 172.21.15.29:0/1876303356 >> v2:172.21.15.144:3300/0 conn(0x7f50700ae8b0 0x7f50700b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524318786 0 0) 0x7f5070158ad0 con 0x7f50700b1840 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f505c003570 con 0x7f50700b1840 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2467922095 0 0) 0x7f505c003570 con 0x7f50700b1840 2023-10-07T09:29:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 msgr2=0x7f50700cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 0x7f50700cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 >> v2:172.21.15.144:3300/0 conn(0x7f50700ae8b0 msgr2=0x7f50700b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 --2- 172.21.15.29:0/1876303356 >> v2:172.21.15.144:3300/0 conn(0x7f50700ae8b0 0x7f50700b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507015b420 con 0x7f50700b1840 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f50680017f0 con 0x7f50700b1840 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 >> v1:172.21.15.29:6789/0 conn(0x7f50700b1840 legacy=0x7f50700ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f505c003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f505c004810 0x7f505c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 0x7f505c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700b1cc0 con 0x7f505c003ae0 2023-10-07T09:29:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507015b140 con 0x7f505c004810 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.326+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505c00bb10 con 0x7f5070157290 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f505c003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507519a700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f505c004810 0x7f505c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 -- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 msgr2=0x7f505c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507619c700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 0x7f505c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5070157290 0x7f505c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 -- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f505c004810 msgr2=0x7f505c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f505c004810 0x7f505c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 -- 172.21.15.29:0/1876303356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c003570 con 0x7f505c003ae0 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507519a700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f505c004810 0x7f505c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f507599b700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f505c003ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f506c009c00 tx=0x7f506c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf8b8c246747349 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.327+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00e710 con 0x7f505c003ae0 2023-10-07T09:29:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c00e870 con 0x7f505c003ae0 2023-10-07T09:29:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5074999700 1 -- 172.21.15.29:0/1876303356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00ec30 con 0x7f505c003ae0 2023-10-07T09:29:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5077bff700 1 -- 172.21.15.29:0/1876303356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 msgr2=0x7f505c003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5077bff700 1 --2- 172.21.15.29:0/1876303356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f505c003ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f506c009c00 tx=0x7f506c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f507619c700 1 -- 172.21.15.29:0/1876303356 reap_dead start 2023-10-07T09:29:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5077bff700 1 -- 172.21.15.29:0/1876303356 shutdown_connections 2023-10-07T09:29:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5077bff700 1 -- 172.21.15.29:0/1876303356 >> 172.21.15.29:0/1876303356 conn(0x7f50700680a0 msgr2=0x7f50700cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5077bff700 1 -- 172.21.15.29:0/1876303356 shutdown_connections 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.328+0000 7f5077bff700 1 -- 172.21.15.29:0/1876303356 wait complete. 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.330+0000 7f5077bff700 1 Processor -- start 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.330+0000 7f5077bff700 1 -- start start 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f5077bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f507006e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f5077bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50700b08c0 0x7f50700af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f5077bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f507014abe0 0x7f507006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f5077bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070155d00 con 0x7f505c003ae0 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f5077bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700af630 con 0x7f50700b08c0 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f5077bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50700b0430 con 0x7f507014abe0 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f507006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f507006e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58416/0 (socket says 172.21.15.29:58416) 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 -- 172.21.15.29:0/3852437924 learned_addr learned my addr 172.21.15.29:0/3852437924 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507519a700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50700b08c0 0x7f50700af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507619c700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f507014abe0 0x7f507006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 -- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f507014abe0 msgr2=0x7f507006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f507014abe0 0x7f507006ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 -- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50700b08c0 msgr2=0x7f50700af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50700b08c0 0x7f50700af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507599b700 1 -- 172.21.15.29:0/3852437924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50700af820 con 0x7f505c003ae0 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.331+0000 7f507619c700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f507014abe0 0x7f507006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.332+0000 7f507599b700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f507006e710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f506c00e4b0 tx=0x7f506c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.332+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c022070 con 0x7f505c003ae0 2023-10-07T09:29:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.332+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c00e9e0 con 0x7f505c003ae0 2023-10-07T09:29:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.332+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c027470 con 0x7f505c003ae0 2023-10-07T09:29:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.332+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50700afb00 con 0x7f505c003ae0 2023-10-07T09:29:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.332+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50700b00c0 con 0x7f505c003ae0 2023-10-07T09:29:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.333+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507004c3c0 con 0x7f505c003ae0 2023-10-07T09:29:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.334+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f506c027860 con 0x7f505c003ae0 2023-10-07T09:29:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.334+0000 7f5065ffb700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5054066f30 0x7f50540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.335+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f506c0860b0 con 0x7f505c003ae0 2023-10-07T09:29:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.335+0000 7f507519a700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5054066f30 0x7f50540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.336+0000 7f507519a700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5054066f30 0x7f50540693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50700b0cb0 tx=0x7f5060008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:25.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.339+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f506c053330 con 0x7f505c003ae0 2023-10-07T09:29:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.585+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f507015b700 con 0x7f505c003ae0 2023-10-07T09:29:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.585+0000 7f5065ffb700 1 -- 172.21.15.29:0/3852437924 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f506c056950 con 0x7f505c003ae0 2023-10-07T09:29:25.587 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:25.587 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:29:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5054066f30 msgr2=0x7f50540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5054066f30 0x7f50540693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50700b0cb0 tx=0x7f5060008040 comp rx=0 tx=0).stop 2023-10-07T09:29:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 msgr2=0x7f507006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f507006e710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f506c00e4b0 tx=0x7f506c01a000 comp rx=0 tx=0).stop 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 shutdown_connections 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5054066f30 0x7f50540693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f507014abe0 0x7f507006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f505c003ae0 0x7f507006e710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 --2- 172.21.15.29:0/3852437924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50700b08c0 0x7f50700af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 >> 172.21.15.29:0/3852437924 conn(0x7f50700680a0 msgr2=0x7f5070159cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 shutdown_connections 2023-10-07T09:29:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.588+0000 7f5077bff700 1 -- 172.21.15.29:0/3852437924 wait complete. 2023-10-07T09:29:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:29:25.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.800+0000 7fd8d7fff700 1 Processor -- start 2023-10-07T09:29:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.800+0000 7fd8d7fff700 1 -- start start 2023-10-07T09:29:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.801+0000 7fd8d7fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8d0159750 0x7fd8d00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.801+0000 7fd8d7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d00ce8a0 0x7fd8d00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.801+0000 7fd8d7fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d00cbf20 con 0x7fd8d0159b20 2023-10-07T09:29:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.801+0000 7fd8d7fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d00cc200 con 0x7fd8d0159750 2023-10-07T09:29:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.801+0000 7fd8d7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00cc4e0 con 0x7fd8d00ce8a0 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d5d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8d0159750 0x7fd8d00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d00ce8a0 0x7fd8d00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d559a700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd8d0159b20 0x7fd8d00ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:59722/0 (socket says 172.21.15.29:59722) 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d5d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8d0159750 0x7fd8d00d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:59552/0 (socket says 172.21.15.29:59552) 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d559a700 1 -- 172.21.15.29:0/1257700091 learned_addr learned my addr 172.21.15.29:0/1257700091 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d5d9b700 1 -- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d00ce8a0 msgr2=0x7fd8d00c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d5d9b700 1 --2- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d00ce8a0 0x7fd8d00c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d5d9b700 1 -- 172.21.15.29:0/1257700091 >> v1:172.21.15.144:6789/0 conn(0x7fd8d0159b20 legacy=0x7fd8d00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d5d9b700 1 -- 172.21.15.29:0/1257700091 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00cc7c0 con 0x7fd8d0159750 2023-10-07T09:29:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.802+0000 7fd8d4d99700 1 -- 172.21.15.29:0/1257700091 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219420614 0 0) 0x7fd8d00cbf20 con 0x7fd8d0159b20 2023-10-07T09:29:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.803+0000 7fd8d5d9b700 1 --2- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d0159750 0x7fd8d00d10a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0098d0 tx=0x7fd8cc009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=452f0d1e1351723c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.803+0000 7fd8d4d99700 1 -- 172.21.15.29:0/1257700091 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc005c10 con 0x7fd8d0159750 2023-10-07T09:29:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.803+0000 7fd8d4d99700 1 -- 172.21.15.29:0/1257700091 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc005d70 con 0x7fd8d0159750 2023-10-07T09:29:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.803+0000 7fd8d4d99700 1 -- 172.21.15.29:0/1257700091 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc006280 con 0x7fd8d0159750 2023-10-07T09:29:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.803+0000 7fd8d7fff700 1 -- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d0159750 msgr2=0x7fd8d00d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.803+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d0159750 0x7fd8d00d10a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0098d0 tx=0x7fd8cc009120 comp rx=0 tx=0).stop 2023-10-07T09:29:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.804+0000 7fd8d7fff700 1 -- 172.21.15.29:0/1257700091 shutdown_connections 2023-10-07T09:29:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.804+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d00ce8a0 0x7fd8d00c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.804+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/1257700091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d0159750 0x7fd8d00d10a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.804+0000 7fd8d7fff700 1 -- 172.21.15.29:0/1257700091 >> 172.21.15.29:0/1257700091 conn(0x7fd8d0063a10 msgr2=0x7fd8d00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.804+0000 7fd8d7fff700 1 -- 172.21.15.29:0/1257700091 shutdown_connections 2023-10-07T09:29:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.804+0000 7fd8d7fff700 1 -- 172.21.15.29:0/1257700091 wait complete. 2023-10-07T09:29:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.806+0000 7fd8d7fff700 1 Processor -- start 2023-10-07T09:29:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.806+0000 7fd8d7fff700 1 -- start start 2023-10-07T09:29:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d7fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d00ce8a0 0x7fd8d015abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d0159750 0x7fd8d0148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 0x7fd8d00c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d7fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00c8cd0 con 0x7fd8d0159b20 2023-10-07T09:29:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d7fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00b3ae0 con 0x7fd8d00ce8a0 2023-10-07T09:29:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00c89f0 con 0x7fd8d0159750 2023-10-07T09:29:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 0x7fd8d00c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 0x7fd8d00c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58422/0 (socket says 172.21.15.29:58422) 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d659c700 1 -- 172.21.15.29:0/3568921172 learned_addr learned my addr 172.21.15.29:0/3568921172 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.807+0000 7fd8d5d9b700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d00ce8a0 0x7fd8d015abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d559a700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d0159750 0x7fd8d0148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d659c700 1 -- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d0159750 msgr2=0x7fd8d0148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d659c700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d0159750 0x7fd8d0148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d659c700 1 -- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d00ce8a0 msgr2=0x7fd8d015abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d659c700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d00ce8a0 0x7fd8d015abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d659c700 1 -- 172.21.15.29:0/3568921172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc0093c0 con 0x7fd8d0159b20 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d5d9b700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d00ce8a0 0x7fd8d015abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d659c700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 0x7fd8d00c8030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009e90 tx=0x7fd8c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800cb70 con 0x7fd8d0159b20 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800f040 con 0x7fd8d0159b20 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d00b3490 con 0x7fd8d0159b20 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.808+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8014480 con 0x7fd8d0159b20 2023-10-07T09:29:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.809+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d0148670 con 0x7fd8d0159b20 2023-10-07T09:29:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.810+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d004c3c0 con 0x7fd8d0159b20 2023-10-07T09:29:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.811+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd8c8014830 con 0x7fd8d0159b20 2023-10-07T09:29:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.811+0000 7fd8c5ffb700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd8b4066f30 0x7fd8b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.811+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fd8c8084f60 con 0x7fd8d0159b20 2023-10-07T09:29:25.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.812+0000 7fd8d5d9b700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd8b4066f30 0x7fd8b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:25.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.812+0000 7fd8d5d9b700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd8b4066f30 0x7fd8b40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009160 tx=0x7fd8cc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:25.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:25.815+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8c80521d0 con 0x7fd8d0159b20 2023-10-07T09:29:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.017+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8d00ce680 con 0x7fd8d0159b20 2023-10-07T09:29:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.018+0000 7fd8c5ffb700 1 -- 172.21.15.29:0/3568921172 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7fd8c80557f0 con 0x7fd8d0159b20 2023-10-07T09:29:26.019 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:26.019 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.020+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd8b4066f30 msgr2=0x7fd8b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.020+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd8b4066f30 0x7fd8b40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009160 tx=0x7fd8cc016000 comp rx=0 tx=0).stop 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 msgr2=0x7fd8d00c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 0x7fd8d00c8030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009e90 tx=0x7fd8c800b040 comp rx=0 tx=0).stop 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 shutdown_connections 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd8b4066f30 0x7fd8b40693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8d0159b20 0x7fd8d00c8030 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8d00ce8a0 0x7fd8d015abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 --2- 172.21.15.29:0/3568921172 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8d0159750 0x7fd8d0148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 >> 172.21.15.29:0/3568921172 conn(0x7fd8d0063a10 msgr2=0x7fd8d00c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 shutdown_connections 2023-10-07T09:29:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.021+0000 7fd8d7fff700 1 -- 172.21.15.29:0/3568921172 wait complete. 2023-10-07T09:29:26.039 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:29:26.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.235+0000 7f74f9dff700 1 Processor -- start 2023-10-07T09:29:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.235+0000 7f74f9dff700 1 -- start start 2023-10-07T09:29:26.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.236+0000 7f74f9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.236+0000 7f74f9dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f74f40ce8a0 0x7f74f40c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f9dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40cbf20 con 0x7f74f4159750 2023-10-07T09:29:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f9dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f40cc200 con 0x7f74f40ce8a0 2023-10-07T09:29:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40cc4e0 con 0x7f74f4159b20 2023-10-07T09:29:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f37fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f74f4159750 0x7f74f40ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36888/0 (socket says 172.21.15.29:36888) 2023-10-07T09:29:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f37fe700 1 -- 172.21.15.29:0/1940899591 learned_addr learned my addr 172.21.15.29:0/1940899591 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:26.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f3fff700 1 --2- 172.21.15.29:0/1940899591 >> v2:172.21.15.144:3300/0 conn(0x7f74f40ce8a0 0x7f74f40c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f2ffd700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f27fc700 1 -- 172.21.15.29:0/1940899591 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577840720 0 0) 0x7f74f40cbf20 con 0x7f74f4159750 2023-10-07T09:29:26.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.237+0000 7f74f27fc700 1 -- 172.21.15.29:0/1940899591 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc003570 con 0x7f74f4159750 2023-10-07T09:29:26.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f3fff700 1 -- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 msgr2=0x7f74f40d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:26.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f3fff700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f40d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f3fff700 1 -- 172.21.15.29:0/1940899591 >> v1:172.21.15.29:6789/0 conn(0x7f74f4159750 legacy=0x7f74f40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f27fc700 1 -- 172.21.15.29:0/1940899591 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 940713161 0 0) 0x7f74dc003570 con 0x7f74f4159750 2023-10-07T09:29:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f3fff700 1 -- 172.21.15.29:0/1940899591 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f40cc7c0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f2ffd700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f40d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f3fff700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c5460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009e60 tx=0x7f74ec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0717d6ef4007f34 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f27fc700 1 -- 172.21.15.29:0/1940899591 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec00e070 con 0x7f74f40ce8a0 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f27fc700 1 -- 172.21.15.29:0/1940899591 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec00b7a0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.238+0000 7f74f27fc700 1 -- 172.21.15.29:0/1940899591 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e40015f0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 -- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f40ce8a0 msgr2=0x7f74f40c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c5460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009e60 tx=0x7f74ec009780 comp rx=0 tx=0).stop 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 -- 172.21.15.29:0/1940899591 shutdown_connections 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c5460 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1940899591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f40d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 -- 172.21.15.29:0/1940899591 >> 172.21.15.29:0/1940899591 conn(0x7f74f4063a10 msgr2=0x7f74f40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 -- 172.21.15.29:0/1940899591 shutdown_connections 2023-10-07T09:29:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.239+0000 7f74f9dff700 1 -- 172.21.15.29:0/1940899591 wait complete. 2023-10-07T09:29:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.240+0000 7f74f9dff700 1 Processor -- start 2023-10-07T09:29:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.240+0000 7f74f9dff700 1 -- start start 2023-10-07T09:29:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.241+0000 7f74f9dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.241+0000 7f74f9dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f4159750 0x7f74f4142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.241+0000 7f74f9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f4150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.241+0000 7f74f9dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c95c0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.241+0000 7f74f9dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f4152170 con 0x7f74f4159750 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.241+0000 7f74f9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f4153610 con 0x7f74f4159b20 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58428/0 (socket says 172.21.15.29:58428) 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 -- 172.21.15.29:0/1061081709 learned_addr learned my addr 172.21.15.29:0/1061081709 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 -- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 msgr2=0x7f74f4150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f4150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 -- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f4159750 msgr2=0x7f74f4142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f4159750 0x7f74f4142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 -- 172.21.15.29:0/1061081709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8009420 con 0x7f74f40ce8a0 2023-10-07T09:29:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f37fe700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c9080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74e400cb90 tx=0x7f74e4002ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4007300 con 0x7f74f40ce8a0 2023-10-07T09:29:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e4007460 con 0x7f74f40ce8a0 2023-10-07T09:29:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e40057f0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.242+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ec009a20 con 0x7f74f40ce8a0 2023-10-07T09:29:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.243+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f4151bb0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.244+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f404c3c0 con 0x7f74f40ce8a0 2023-10-07T09:29:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.244+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f74e4005c20 con 0x7f74f40ce8a0 2023-10-07T09:29:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.245+0000 7f74db7fe700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74e0066f30 0x7f74e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.245+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f74e4084410 con 0x7f74f40ce8a0 2023-10-07T09:29:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.246+0000 7f74f2ffd700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74e0066f30 0x7f74e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.246+0000 7f74f2ffd700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74e0066f30 0x7f74e00693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74f41531c0 tx=0x7f74e80091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:26.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.249+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74e4051660 con 0x7f74f40ce8a0 2023-10-07T09:29:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.448+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74f40ca7d0 con 0x7f74e0066f30 2023-10-07T09:29:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.458+0000 7f74db7fe700 1 -- 172.21.15.29:0/1061081709 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263941 (secure 0 0 0) 0x7f74f40ca7d0 con 0x7f74e0066f30 2023-10-07T09:29:26.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74e0066f30 msgr2=0x7f74e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74e0066f30 0x7f74e00693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74f41531c0 tx=0x7f74e80091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 msgr2=0x7f74f40c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c9080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74e400cb90 tx=0x7f74e4002ad0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 shutdown_connections 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74e0066f30 0x7f74e00693e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f4159b20 0x7f74f4150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f40ce8a0 0x7f74f40c9080 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 --2- 172.21.15.29:0/1061081709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f4159750 0x7f74f4142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 >> 172.21.15.29:0/1061081709 conn(0x7f74f4063a10 msgr2=0x7f74f40cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:26.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 shutdown_connections 2023-10-07T09:29:26.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.462+0000 7f74f9dff700 1 -- 172.21.15.29:0/1061081709 wait complete. 2023-10-07T09:29:26.466 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:26.480 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-07T09:29:26.017629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":196,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385312,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589408,"statfs":{"total":773094113280,"available":770651553792,"internally_reserved":0,"allocated":2981888,"data_stored":1785288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.937640"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.099417+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:16.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.097265+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:17:45.553586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.096393+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:22.795278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098781+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:35:34.023110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.099332+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:50:37.063563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:20.073275+0000","last_change":"2023-10-07T09:29:20.089824+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:20.073275+0000","last_undegraded":"2023-10-07T09:29:20.073275+0000","last_fullsized":"2023-10-07T09:29:20.073275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:16:03.391338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101453+0000","last_change":"2023-10-07T09:29:21.101590+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101453+0000","last_undegraded":"2023-10-07T09:29:21.101453+0000","last_fullsized":"2023-10-07T09:29:21.101453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098675+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:14:38.048638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.095975+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:38.123274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.095699+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:08.598857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.096497+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:07.659133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":161,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-07T09:29:20.090620+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:20.090620+0000","last_peered":"2023-10-07T09:29:20.090620+0000","last_clean":"2023-10-07T09:29:20.090620+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:20.090620+0000","last_undegraded":"2023-10-07T09:29:20.090620+0000","last_fullsized":"2023-10-07T09:29:20.090620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101516+0000","last_change":"2023-10-07T09:29:21.101650+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101516+0000","last_undegraded":"2023-10-07T09:29:21.101516+0000","last_fullsized":"2023-10-07T09:29:21.101516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.105215+0000","last_change":"2023-10-07T09:29:21.105367+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.105215+0000","last_undegraded":"2023-10-07T09:29:21.105215+0000","last_fullsized":"2023-10-07T09:29:21.105215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923235+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:21.923235+0000","last_peered":"2023-10-07T09:29:21.923235+0000","last_clean":"2023-10-07T09:29:21.923235+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:21.923235+0000","last_undegraded":"2023-10-07T09:29:21.923235+0000","last_fullsized":"2023-10-07T09:29:21.923235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101823+0000","last_change":"2023-10-07T09:29:21.101996+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101823+0000","last_undegraded":"2023-10-07T09:29:21.101823+0000","last_fullsized":"2023-10-07T09:29:21.101823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.102130+0000","last_change":"2023-10-07T09:29:21.102289+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.102130+0000","last_undegraded":"2023-10-07T09:29:21.102130+0000","last_fullsized":"2023-10-07T09:29:21.102130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098055+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:31:45.651295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.097549+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:35:30.084095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298048,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073792,"statfs":{"total":96636764160,"available":96331563008,"internally_reserved":0,"allocated":778240,"data_stored":648530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:26.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:26.648 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T09:29:26.647+0000 7f37eb7f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:29:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.714+0000 7f3bd9dff700 1 Processor -- start 2023-10-07T09:29:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.714+0000 7f3bd9dff700 1 -- start start 2023-10-07T09:29:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd9dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3bd4159c00 0x7f3bd40ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd9dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3bd4068620 0x7f3bd4068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd9dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd415af00 con 0x7f3bd40ad220 2023-10-07T09:29:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd9dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd415a050 con 0x7f3bd4068620 2023-10-07T09:29:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd9dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd4068f80 con 0x7f3bd4159c00 2023-10-07T09:29:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3bd4068620 0x7f3bd4068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3bd4068620 0x7f3bd4068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58434/0 (socket says 172.21.15.29:58434) 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 -- 172.21.15.29:0/1535411901 learned_addr learned my addr 172.21.15.29:0/1535411901 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd37fe700 1 --1- 172.21.15.29:0/1535411901 >> v1:172.21.15.29:6789/0 conn(0x7f3bd40ad220 0x7f3bd40ac490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36902/0 (socket says 172.21.15.29:36902) 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd2ffd700 1 --2- 172.21.15.29:0/1535411901 >> v2:172.21.15.144:3300/0 conn(0x7f3bd4159c00 0x7f3bd40ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 -- 172.21.15.29:0/1535411901 >> v2:172.21.15.144:3300/0 conn(0x7f3bd4159c00 msgr2=0x7f3bd40ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 --2- 172.21.15.29:0/1535411901 >> v2:172.21.15.144:3300/0 conn(0x7f3bd4159c00 0x7f3bd40ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 -- 172.21.15.29:0/1535411901 >> v1:172.21.15.29:6789/0 conn(0x7f3bd40ad220 legacy=0x7f3bd40ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd3fff700 1 -- 172.21.15.29:0/1535411901 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd415a250 con 0x7f3bd4068620 2023-10-07T09:29:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd2ffd700 1 --2- 172.21.15.29:0/1535411901 >> v2:172.21.15.144:3300/0 conn(0x7f3bd4159c00 0x7f3bd40ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.716+0000 7f3bd27fc700 1 -- 172.21.15.29:0/1535411901 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586113993 0 0) 0x7f3bd415af00 con 0x7f3bd40ad220 2023-10-07T09:29:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.717+0000 7f3bd3fff700 1 --2- 172.21.15.29:0/1535411901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd4068a40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406d000 tx=0x7f3bcc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d654b8b14f03a426 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.717+0000 7f3bd27fc700 1 -- 172.21.15.29:0/1535411901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc00e070 con 0x7f3bd4068620 2023-10-07T09:29:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.717+0000 7f3bd27fc700 1 -- 172.21.15.29:0/1535411901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc00ba50 con 0x7f3bd4068620 2023-10-07T09:29:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.717+0000 7f3bd27fc700 1 -- 172.21.15.29:0/1535411901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc0092e0 con 0x7f3bd4068620 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.717+0000 7f3bd9dff700 1 -- 172.21.15.29:0/1535411901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 msgr2=0x7f3bd4068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.717+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/1535411901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd4068a40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406d000 tx=0x7f3bcc009840 comp rx=0 tx=0).stop 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.718+0000 7f3bd9dff700 1 -- 172.21.15.29:0/1535411901 shutdown_connections 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.718+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/1535411901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd4068a40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.718+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/1535411901 >> v2:172.21.15.144:3300/0 conn(0x7f3bd4159c00 0x7f3bd40ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.718+0000 7f3bd9dff700 1 -- 172.21.15.29:0/1535411901 >> 172.21.15.29:0/1535411901 conn(0x7f3bd406dd00 msgr2=0x7f3bd406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.718+0000 7f3bd9dff700 1 -- 172.21.15.29:0/1535411901 shutdown_connections 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.718+0000 7f3bd9dff700 1 -- 172.21.15.29:0/1535411901 wait complete. 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.719+0000 7f3bd9dff700 1 Processor -- start 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.719+0000 7f3bd9dff700 1 -- start start 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd9dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd41455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3bd40ad220 0x7f3bd40ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd9dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bd4159c00 0x7f3bd4169a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd9dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd415b390 con 0x7f3bd4068620 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd9dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd415bed0 con 0x7f3bd4159c00 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd415b5a0 con 0x7f3bd40ad220 2023-10-07T09:29:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd41455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd41455a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58448/0 (socket says 172.21.15.29:58448) 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.720+0000 7f3bd37fe700 1 -- 172.21.15.29:0/3412209014 learned_addr learned my addr 172.21.15.29:0/3412209014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd37fe700 1 -- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3bd40ad220 msgr2=0x7f3bd40ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd37fe700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3bd40ad220 0x7f3bd40ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd37fe700 1 -- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bd4159c00 msgr2=0x7f3bd4169a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd37fe700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bd4159c00 0x7f3bd4169a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd37fe700 1 -- 172.21.15.29:0/3412209014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc009ae0 con 0x7f3bd4068620 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd37fe700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd41455a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400c5b0 tx=0x7f3bc4001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4005a70 con 0x7f3bd4068620 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc4005bd0 con 0x7f3bd4068620 2023-10-07T09:29:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc40070d0 con 0x7f3bd4068620 2023-10-07T09:29:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd415b880 con 0x7f3bd4068620 2023-10-07T09:29:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.721+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd40ca680 con 0x7f3bd4068620 2023-10-07T09:29:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.722+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd404c3c0 con 0x7f3bd4068620 2023-10-07T09:29:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.723+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3bc40074c0 con 0x7f3bd4068620 2023-10-07T09:29:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.724+0000 7f3bbb7fe700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3bc0066f30 0x7f3bc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.724+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3bc4083aa0 con 0x7f3bd4068620 2023-10-07T09:29:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.725+0000 7f3bd2ffd700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3bc0066f30 0x7f3bc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.725+0000 7f3bd2ffd700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3bc0066f30 0x7f3bc00693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8006fd0 tx=0x7f3bc8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.729+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bc4050c70 con 0x7f3bd4068620 2023-10-07T09:29:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.926+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3bd415c010 con 0x7f3bc0066f30 2023-10-07T09:29:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.935+0000 7f3bbb7fe700 1 -- 172.21.15.29:0/3412209014 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263941 (secure 0 0 0) 0x7f3bc8009e60 con 0x7f3bc0066f30 2023-10-07T09:29:26.940 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3bc0066f30 msgr2=0x7f3bc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3bc0066f30 0x7f3bc00693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8006fd0 tx=0x7f3bc8008040 comp rx=0 tx=0).stop 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 msgr2=0x7f3bd41455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd41455a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400c5b0 tx=0x7f3bc4001040 comp rx=0 tx=0).stop 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 shutdown_connections 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3bc0066f30 0x7f3bc00693e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bd4159c00 0x7f3bd4169a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd4068620 0x7f3bd41455a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 --2- 172.21.15.29:0/3412209014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3bd40ad220 0x7f3bd40ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 >> 172.21.15.29:0/3412209014 conn(0x7f3bd406dd00 msgr2=0x7f3bd40695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 shutdown_connections 2023-10-07T09:29:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:26.939+0000 7f3bd9dff700 1 -- 172.21.15.29:0/3412209014 wait complete. 2023-10-07T09:29:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:26.957 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-07T09:29:26.017629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":196,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385312,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589408,"statfs":{"total":773094113280,"available":770651553792,"internally_reserved":0,"allocated":2981888,"data_stored":1785288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.937640"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.099417+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:16.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.097265+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:17:45.553586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.096393+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:22.795278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098781+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:35:34.023110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.099332+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:50:37.063563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:20.073275+0000","last_change":"2023-10-07T09:29:20.089824+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:20.073275+0000","last_undegraded":"2023-10-07T09:29:20.073275+0000","last_fullsized":"2023-10-07T09:29:20.073275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:16:03.391338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101453+0000","last_change":"2023-10-07T09:29:21.101590+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101453+0000","last_undegraded":"2023-10-07T09:29:21.101453+0000","last_fullsized":"2023-10-07T09:29:21.101453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098675+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:14:38.048638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.095975+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:38.123274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.095699+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:08.598857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.096497+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:07.659133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":161,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-07T09:29:20.090620+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:20.090620+0000","last_peered":"2023-10-07T09:29:20.090620+0000","last_clean":"2023-10-07T09:29:20.090620+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:20.090620+0000","last_undegraded":"2023-10-07T09:29:20.090620+0000","last_fullsized":"2023-10-07T09:29:20.090620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101516+0000","last_change":"2023-10-07T09:29:21.101650+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101516+0000","last_undegraded":"2023-10-07T09:29:21.101516+0000","last_fullsized":"2023-10-07T09:29:21.101516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.105215+0000","last_change":"2023-10-07T09:29:21.105367+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.105215+0000","last_undegraded":"2023-10-07T09:29:21.105215+0000","last_fullsized":"2023-10-07T09:29:21.105215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923235+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:21.923235+0000","last_peered":"2023-10-07T09:29:21.923235+0000","last_clean":"2023-10-07T09:29:21.923235+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:21.923235+0000","last_undegraded":"2023-10-07T09:29:21.923235+0000","last_fullsized":"2023-10-07T09:29:21.923235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101823+0000","last_change":"2023-10-07T09:29:21.101996+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101823+0000","last_undegraded":"2023-10-07T09:29:21.101823+0000","last_fullsized":"2023-10-07T09:29:21.101823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.102130+0000","last_change":"2023-10-07T09:29:21.102289+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.102130+0000","last_undegraded":"2023-10-07T09:29:21.102130+0000","last_fullsized":"2023-10-07T09:29:21.102130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098055+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:31:45.651295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.097549+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:35:30.084095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298048,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073792,"statfs":{"total":96636764160,"available":96331563008,"internally_reserved":0,"allocated":778240,"data_stored":648530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:26.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.194+0000 7fda26dff700 1 Processor -- start 2023-10-07T09:29:27.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.194+0000 7fda26dff700 1 -- start start 2023-10-07T09:29:27.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda26dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda200d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda26dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda200ce8a0 0x7fda200c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda26dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200cbf20 con 0x7fda20159750 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda26dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200cc200 con 0x7fda200ce8a0 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda26dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200cc4e0 con 0x7fda20159b20 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda2539c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda200ce8a0 0x7fda200c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda24b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fda20159750 0x7fda200ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36906/0 (socket says 172.21.15.29:36906) 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda24b9b700 1 -- 172.21.15.29:0/361503998 learned_addr learned my addr 172.21.15.29:0/361503998 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda1ffff700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda200d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda1f7fe700 1 -- 172.21.15.29:0/361503998 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218322517 0 0) 0x7fda200cbf20 con 0x7fda20159750 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda1f7fe700 1 -- 172.21.15.29:0/361503998 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda10003570 con 0x7fda20159750 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda2539c700 1 -- 172.21.15.29:0/361503998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 msgr2=0x7fda200d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda2539c700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda200d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda2539c700 1 -- 172.21.15.29:0/361503998 >> v1:172.21.15.29:6789/0 conn(0x7fda20159750 legacy=0x7fda200ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda2539c700 1 -- 172.21.15.29:0/361503998 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200cc7c0 con 0x7fda200ce8a0 2023-10-07T09:29:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.196+0000 7fda1ffff700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda200d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:27.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.197+0000 7fda2539c700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda200ce8a0 0x7fda200c5460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e60 tx=0x7fda18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b543b4140abb10e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.197+0000 7fda1f7fe700 1 -- 172.21.15.29:0/361503998 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1800e070 con 0x7fda200ce8a0 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.197+0000 7fda1f7fe700 1 -- 172.21.15.29:0/361503998 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1800bb60 con 0x7fda200ce8a0 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.197+0000 7fda1f7fe700 1 -- 172.21.15.29:0/361503998 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18012430 con 0x7fda200ce8a0 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.197+0000 7fda26dff700 1 -- 172.21.15.29:0/361503998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda200ce8a0 msgr2=0x7fda200c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.197+0000 7fda26dff700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda200ce8a0 0x7fda200c5460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e60 tx=0x7fda18009780 comp rx=0 tx=0).stop 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.198+0000 7fda26dff700 1 -- 172.21.15.29:0/361503998 shutdown_connections 2023-10-07T09:29:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.198+0000 7fda26dff700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda200ce8a0 0x7fda200c5460 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.198+0000 7fda26dff700 1 --2- 172.21.15.29:0/361503998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda200d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.198+0000 7fda26dff700 1 -- 172.21.15.29:0/361503998 >> 172.21.15.29:0/361503998 conn(0x7fda20063a10 msgr2=0x7fda200d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.198+0000 7fda26dff700 1 -- 172.21.15.29:0/361503998 shutdown_connections 2023-10-07T09:29:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.198+0000 7fda26dff700 1 -- 172.21.15.29:0/361503998 wait complete. 2023-10-07T09:29:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.199+0000 7fda26dff700 1 Processor -- start 2023-10-07T09:29:27.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.199+0000 7fda26dff700 1 -- start start 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda26dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda200ce8a0 0x7fda2014c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda26dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 0x7fda200c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda26dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda201549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda26dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20150eb0 con 0x7fda20159750 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda26dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200c90b0 con 0x7fda200ce8a0 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda26dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200c9390 con 0x7fda20159b20 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 0x7fda200c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 0x7fda200c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58472/0 (socket says 172.21.15.29:58472) 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda1ffff700 1 -- 172.21.15.29:0/130381174 learned_addr learned my addr 172.21.15.29:0/130381174 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda2539c700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda201549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.200+0000 7fda24b9b700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda200ce8a0 0x7fda2014c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1ffff700 1 -- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 msgr2=0x7fda201549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1ffff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda201549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1ffff700 1 -- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda200ce8a0 msgr2=0x7fda2014c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1ffff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda200ce8a0 0x7fda2014c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1ffff700 1 -- 172.21.15.29:0/130381174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18009a20 con 0x7fda20159750 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1ffff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 0x7fda200c86b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00d860 tx=0x7fda0c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0125f0 con 0x7fda20159750 2023-10-07T09:29:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c012750 con 0x7fda20159750 2023-10-07T09:29:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200c9670 con 0x7fda20159750 2023-10-07T09:29:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c012af0 con 0x7fda20159750 2023-10-07T09:29:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.201+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200c8bf0 con 0x7fda20159750 2023-10-07T09:29:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.203+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2015e280 con 0x7fda20159750 2023-10-07T09:29:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.203+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fda0c013480 con 0x7fda20159750 2023-10-07T09:29:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.204+0000 7fda1cff9700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fda04066f00 0x7fda040693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.204+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fda0c084770 con 0x7fda20159750 2023-10-07T09:29:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.205+0000 7fda24b9b700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fda04066f00 0x7fda040693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.205+0000 7fda24b9b700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fda04066f00 0x7fda040693b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda14001360 tx=0x7fda14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:27.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.208+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda0c051940 con 0x7fda20159750 2023-10-07T09:29:27.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.405+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda2004c3c0 con 0x7fda04066f00 2023-10-07T09:29:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.413+0000 7fda1cff9700 1 -- 172.21.15.29:0/130381174 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+263941 (secure 0 0 0) 0x7fda2004c3c0 con 0x7fda04066f00 2023-10-07T09:29:27.418 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fda04066f00 msgr2=0x7fda040693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fda04066f00 0x7fda040693b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda14001360 tx=0x7fda14009040 comp rx=0 tx=0).stop 2023-10-07T09:29:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 msgr2=0x7fda200c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 0x7fda200c86b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00d860 tx=0x7fda0c008040 comp rx=0 tx=0).stop 2023-10-07T09:29:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 shutdown_connections 2023-10-07T09:29:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fda04066f00 0x7fda040693b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda20159b20 0x7fda201549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda200ce8a0 0x7fda2014c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 --2- 172.21.15.29:0/130381174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda20159750 0x7fda200c86b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 >> 172.21.15.29:0/130381174 conn(0x7fda20063a10 msgr2=0x7fda200ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 shutdown_connections 2023-10-07T09:29:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.417+0000 7fda26dff700 1 -- 172.21.15.29:0/130381174 wait complete. 2023-10-07T09:29:27.423 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:27.435 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-07T09:29:26.017629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":196,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385312,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589408,"statfs":{"total":773094113280,"available":770651553792,"internally_reserved":0,"allocated":2981888,"data_stored":1785288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.937640"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.099417+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:16.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.097265+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:17:45.553586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.096393+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:22.795278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098781+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:35:34.023110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.099332+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:50:37.063563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:20.073275+0000","last_change":"2023-10-07T09:29:20.089824+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:20.073275+0000","last_undegraded":"2023-10-07T09:29:20.073275+0000","last_fullsized":"2023-10-07T09:29:20.073275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:16:03.391338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101453+0000","last_change":"2023-10-07T09:29:21.101590+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101453+0000","last_undegraded":"2023-10-07T09:29:21.101453+0000","last_fullsized":"2023-10-07T09:29:21.101453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098675+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:14:38.048638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.095975+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:38.123274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.095699+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:08.598857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.096497+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:07.659133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":161,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-07T09:29:20.090620+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:20.090620+0000","last_peered":"2023-10-07T09:29:20.090620+0000","last_clean":"2023-10-07T09:29:20.090620+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:20.090620+0000","last_undegraded":"2023-10-07T09:29:20.090620+0000","last_fullsized":"2023-10-07T09:29:20.090620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101516+0000","last_change":"2023-10-07T09:29:21.101650+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101516+0000","last_undegraded":"2023-10-07T09:29:21.101516+0000","last_fullsized":"2023-10-07T09:29:21.101516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.105215+0000","last_change":"2023-10-07T09:29:21.105367+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.105215+0000","last_undegraded":"2023-10-07T09:29:21.105215+0000","last_fullsized":"2023-10-07T09:29:21.105215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923235+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:21.923235+0000","last_peered":"2023-10-07T09:29:21.923235+0000","last_clean":"2023-10-07T09:29:21.923235+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:21.923235+0000","last_undegraded":"2023-10-07T09:29:21.923235+0000","last_fullsized":"2023-10-07T09:29:21.923235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.101823+0000","last_change":"2023-10-07T09:29:21.101996+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.101823+0000","last_undegraded":"2023-10-07T09:29:21.101823+0000","last_fullsized":"2023-10-07T09:29:21.101823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2,"reported_epoch":20,"state":"creating+activating","last_fresh":"2023-10-07T09:29:21.102130+0000","last_change":"2023-10-07T09:29:21.102289+0000","last_active":"2023-10-07T09:29:20.073275+0000","last_peered":"2023-10-07T09:29:20.073275+0000","last_clean":"2023-10-07T09:29:20.073275+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.102130+0000","last_undegraded":"2023-10-07T09:29:21.102130+0000","last_fullsized":"2023-10-07T09:29:21.102130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.098055+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:31:45.651295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:29:21.079033+0000","last_change":"2023-10-07T09:29:21.097549+0000","last_active":"2023-10-07T09:29:21.079033+0000","last_peered":"2023-10-07T09:29:21.079033+0000","last_clean":"2023-10-07T09:29:21.079033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:29:21.079033+0000","last_undegraded":"2023-10-07T09:29:21.079033+0000","last_fullsized":"2023-10-07T09:29:21.079033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:35:30.084095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298048,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073792,"statfs":{"total":96636764160,"available":96331563008,"internally_reserved":0,"allocated":778240,"data_stored":648530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:27.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3f is not active+clean 2023-10-07T09:29:27.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.099417+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T15:05:16.968710+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3b is not active+clean 2023-10-07T09:29:27.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.097265+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T15:17:45.553586+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.39 is not active+clean 2023-10-07T09:29:27.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.096393+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:23:22.795278+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.37 is not active+clean 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.098781+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T20:35:34.023110+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2d is not active+clean 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.099332+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T09:50:37.063563+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1d is not active+clean 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:20.073275+0000', 'last_change': '2023-10-07T09:29:20.089824+0000', 'last_active': '2023-10-07T09:29:20.073275+0000', 'last_peered': '2023-10-07T09:29:20.073275+0000', 'last_clean': '2023-10-07T09:29:20.073275+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:20.073275+0000', 'last_undegraded': '2023-10-07T09:29:20.073275+0000', 'last_fullsized': '2023-10-07T09:29:20.073275+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:16:03.391338+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1c is not active+clean 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1c', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 20, 'state': 'creating+activating', 'last_fresh': '2023-10-07T09:29:21.101453+0000', 'last_change': '2023-10-07T09:29:21.101590+0000', 'last_active': '2023-10-07T09:29:20.073275+0000', 'last_peered': '2023-10-07T09:29:20.073275+0000', 'last_clean': '2023-10-07T09:29:20.073275+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.101453+0000', 'last_undegraded': '2023-10-07T09:29:21.101453+0000', 'last_fullsized': '2023-10-07T09:29:21.101453+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': ['7', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.15 is not active+clean 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.098675+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T21:14:38.048638+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.13 is not active+clean 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.095975+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T19:31:38.123274+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.11 is not active+clean 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.095699+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T20:15:08.598857+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.e is not active+clean 2023-10-07T09:29:27.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.096497+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T10:25:07.659133+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.0 is not active+clean 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 20, 'state': 'creating+activating', 'last_fresh': '2023-10-07T09:29:21.101516+0000', 'last_change': '2023-10-07T09:29:21.101650+0000', 'last_active': '2023-10-07T09:29:20.073275+0000', 'last_peered': '2023-10-07T09:29:20.073275+0000', 'last_clean': '2023-10-07T09:29:20.073275+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.101516+0000', 'last_undegraded': '2023-10-07T09:29:21.101516+0000', 'last_fullsized': '2023-10-07T09:29:21.101516+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': ['7', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.5 is not active+clean 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 20, 'state': 'creating+activating', 'last_fresh': '2023-10-07T09:29:21.105215+0000', 'last_change': '2023-10-07T09:29:21.105367+0000', 'last_active': '2023-10-07T09:29:20.073275+0000', 'last_peered': '2023-10-07T09:29:20.073275+0000', 'last_clean': '2023-10-07T09:29:20.073275+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.105215+0000', 'last_undegraded': '2023-10-07T09:29:21.105215+0000', 'last_fullsized': '2023-10-07T09:29:21.105215+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': ['7', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.8 is not active+clean 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.8', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 20, 'state': 'creating+activating', 'last_fresh': '2023-10-07T09:29:21.101823+0000', 'last_change': '2023-10-07T09:29:21.101996+0000', 'last_active': '2023-10-07T09:29:20.073275+0000', 'last_peered': '2023-10-07T09:29:20.073275+0000', 'last_clean': '2023-10-07T09:29:20.073275+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.101823+0000', 'last_undegraded': '2023-10-07T09:29:21.101823+0000', 'last_fullsized': '2023-10-07T09:29:21.101823+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': ['7', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.b is not active+clean 2023-10-07T09:29:27.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.b', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 20, 'state': 'creating+activating', 'last_fresh': '2023-10-07T09:29:21.102130+0000', 'last_change': '2023-10-07T09:29:21.102289+0000', 'last_active': '2023-10-07T09:29:20.073275+0000', 'last_peered': '2023-10-07T09:29:20.073275+0000', 'last_clean': '2023-10-07T09:29:20.073275+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.102130+0000', 'last_undegraded': '2023-10-07T09:29:21.102130+0000', 'last_fullsized': '2023-10-07T09:29:21.102130+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:20.073275+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': ['7', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.446 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.d is not active+clean 2023-10-07T09:29:27.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.098055+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:31:45.651295+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.446 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.f is not active+clean 2023-10-07T09:29:27.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:29:21.079033+0000', 'last_change': '2023-10-07T09:29:21.097549+0000', 'last_active': '2023-10-07T09:29:21.079033+0000', 'last_peered': '2023-10-07T09:29:21.079033+0000', 'last_clean': '2023-10-07T09:29:21.079033+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:29:21.079033+0000', 'last_undegraded': '2023-10-07T09:29:21.079033+0000', 'last_fullsized': '2023-10-07T09:29:21.079033+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 20, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'last_clean_scrub_stamp': '2023-10-07T09:29:21.079033+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:35:30.084095+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:29:27.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-07T09:29:27.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.669+0000 7f3440bff700 1 Processor -- start 2023-10-07T09:29:27.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.669+0000 7f3440bff700 1 -- start start 2023-10-07T09:29:27.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f3440bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f343c0cf2a0 0x7f343c0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f3440bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0cbf70 con 0x7f343c0c6660 2023-10-07T09:29:27.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f3440bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0cc250 con 0x7f343c0b2ce0 2023-10-07T09:29:27.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f3440bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c0cc530 con 0x7f343c0cf2a0 2023-10-07T09:29:27.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f3439d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f343c0cf2a0 0x7f343c0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f343ad9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f343c0c6660 0x7f343c0cb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36908/0 (socket says 172.21.15.29:36908) 2023-10-07T09:29:27.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f3439d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f343c0cf2a0 0x7f343c0cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58486/0 (socket says 172.21.15.29:58486) 2023-10-07T09:29:27.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.671+0000 7f343ad9d700 1 -- 172.21.15.29:0/2494785245 learned_addr learned my addr 172.21.15.29:0/2494785245 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f3439d9b700 1 -- 172.21.15.29:0/2494785245 >> v1:172.21.15.144:6789/0 conn(0x7f343c0b2ce0 legacy=0x7f343c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f3439d9b700 1 -- 172.21.15.29:0/2494785245 >> v1:172.21.15.29:6789/0 conn(0x7f343c0c6660 legacy=0x7f343c0cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f3439d9b700 1 -- 172.21.15.29:0/2494785245 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0cc810 con 0x7f343c0cf2a0 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f343959a700 1 -- 172.21.15.29:0/2494785245 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407861846 0 0) 0x7f343c0cbf70 con 0x7f343c0c6660 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f3439d9b700 1 --2- 172.21.15.29:0/2494785245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0cf2a0 0x7f343c0cb1d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34300098e0 tx=0x7f34300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a90abd015bf73cd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f343959a700 1 -- 172.21.15.29:0/2494785245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430009e80 con 0x7f343c0cf2a0 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f343959a700 1 -- 172.21.15.29:0/2494785245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000b860 con 0x7f343c0cf2a0 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.672+0000 7f343959a700 1 -- 172.21.15.29:0/2494785245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000bbe0 con 0x7f343c0cf2a0 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 -- 172.21.15.29:0/2494785245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0cf2a0 msgr2=0x7f343c0cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 --2- 172.21.15.29:0/2494785245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0cf2a0 0x7f343c0cb1d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34300098e0 tx=0x7f34300091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 -- 172.21.15.29:0/2494785245 shutdown_connections 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 --2- 172.21.15.29:0/2494785245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0cf2a0 0x7f343c0cb1d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 -- 172.21.15.29:0/2494785245 >> 172.21.15.29:0/2494785245 conn(0x7f343c099820 msgr2=0x7f343c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 -- 172.21.15.29:0/2494785245 shutdown_connections 2023-10-07T09:29:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.673+0000 7f3440bff700 1 -- 172.21.15.29:0/2494785245 wait complete. 2023-10-07T09:29:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.674+0000 7f3440bff700 1 Processor -- start 2023-10-07T09:29:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.675+0000 7f3440bff700 1 -- start start 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.675+0000 7f3440bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 0x7f343c1571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f3440bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f343c0c6660 0x7f343c09a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f3440bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c0cf2a0 0x7f343c0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f3440bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c15cb00 con 0x7f343c0b2ce0 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f3440bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c157720 con 0x7f343c0cf2a0 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f3440bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c15cd50 con 0x7f343c0c6660 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 0x7f343c1571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 0x7f343c1571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58494/0 (socket says 172.21.15.29:58494) 2023-10-07T09:29:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 -- 172.21.15.29:0/739656223 learned_addr learned my addr 172.21.15.29:0/739656223 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f3439d9b700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f343c0c6660 0x7f343c09a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343ad9d700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c0cf2a0 0x7f343c0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 -- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f343c0c6660 msgr2=0x7f343c09a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f343c0c6660 0x7f343c09a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 -- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c0cf2a0 msgr2=0x7f343c0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c0cf2a0 0x7f343c0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 -- 172.21.15.29:0/739656223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3430009420 con 0x7f343c0b2ce0 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343ad9d700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c0cf2a0 0x7f343c0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.676+0000 7f343a59c700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 0x7f343c1571e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f342c00c540 tx=0x7f342c001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.677+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011590 con 0x7f343c0b2ce0 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.677+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c0116f0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.677+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011ab0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.677+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c15d030 con 0x7f343c0b2ce0 2023-10-07T09:29:27.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.677+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c15d5f0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.678+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c04c3c0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.679+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f342c01b3b0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.680+0000 7f342a7fc700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f341c066f30 0x7f341c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:27.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.680+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f342c083970 con 0x7f343c0b2ce0 2023-10-07T09:29:27.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.680+0000 7f3439d9b700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f341c066f30 0x7f341c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:27.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.681+0000 7f3439d9b700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f341c066f30 0x7f341c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f34300093f0 tx=0x7f3430017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:27.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.684+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f342c050bf0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.965+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f343c06a1e0 con 0x7f343c0b2ce0 2023-10-07T09:29:27.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.965+0000 7f342a7fc700 1 -- 172.21.15.29:0/739656223 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1146 (secure 0 0 0) 0x7f342c017020 con 0x7f343c0b2ce0 2023-10-07T09:29:27.966 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:27.967 INFO:teuthology.orchestra.run.smithi029.stdout:{"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":30,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":22,"num_osds":8,"num_up_osds":8,"osd_up_since":1696670950,"num_in_osds":8,"osd_in_since":1696670940,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":80},{"state_name":"creating+peering","count":12},{"state_name":"creating+activating","count":5}],"num_pgs":97,"num_pools":3,"num_objects":20,"data_bytes":592674,"bytes_used":2442559488,"bytes_avail":770651553792,"bytes_total":773094113280,"inactive_pgs_ratio":0.17525772750377655,"write_bytes_sec":2241,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":8,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"c","status":"up:active","gid":4284}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-10-07T09:28:57.119255+0000","services":{}},"progress_events":{}} 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f341c066f30 msgr2=0x7f341c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f341c066f30 0x7f341c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f34300093f0 tx=0x7f3430017000 comp rx=0 tx=0).stop 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 msgr2=0x7f343c1571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 0x7f343c1571e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f342c00c540 tx=0x7f342c001070 comp rx=0 tx=0).stop 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 shutdown_connections 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f341c066f30 0x7f341c0693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c0cf2a0 0x7f343c0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f343c0b2ce0 0x7f343c1571e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 --2- 172.21.15.29:0/739656223 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f343c0c6660 0x7f343c09a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:27.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.967+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 >> 172.21.15.29:0/739656223 conn(0x7f343c099820 msgr2=0x7f343c159090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:27.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.968+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 shutdown_connections 2023-10-07T09:29:27.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:27.968+0000 7f3440bff700 1 -- 172.21.15.29:0/739656223 wait complete. 2023-10-07T09:29:27.986 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-07T09:29:27.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.181+0000 7f8b5f5ff700 1 Processor -- start 2023-10-07T09:29:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.181+0000 7f8b5f5ff700 1 -- start start 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5f5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b58159750 0x7f8b580d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5f5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b580ce8a0 0x7f8b580c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5f5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b580cbf20 con 0x7f8b58159b20 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5f5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b580cc200 con 0x7f8b580ce8a0 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5f5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580cc4e0 con 0x7f8b58159750 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5cb9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8b58159b20 0x7f8b580ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36920/0 (socket says 172.21.15.29:36920) 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5cb9a700 1 -- 172.21.15.29:0/3171112280 learned_addr learned my addr 172.21.15.29:0/3171112280 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5d39b700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b58159750 0x7f8b580d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b5db9c700 1 --2- 172.21.15.29:0/3171112280 >> v2:172.21.15.144:3300/0 conn(0x7f8b580ce8a0 0x7f8b580c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b4ffff700 1 -- 172.21.15.29:0/3171112280 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200125345 0 0) 0x7f8b580cbf20 con 0x7f8b58159b20 2023-10-07T09:29:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.183+0000 7f8b4ffff700 1 -- 172.21.15.29:0/3171112280 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44003570 con 0x7f8b58159b20 2023-10-07T09:29:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b5db9c700 1 -- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b58159750 msgr2=0x7f8b580d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b5db9c700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b58159750 0x7f8b580d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b5db9c700 1 -- 172.21.15.29:0/3171112280 >> v1:172.21.15.29:6789/0 conn(0x7f8b58159b20 legacy=0x7f8b580ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b5db9c700 1 -- 172.21.15.29:0/3171112280 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580cc7c0 con 0x7f8b580ce8a0 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b5d39b700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b58159750 0x7f8b580d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b5db9c700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b580ce8a0 0x7f8b580c5460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8b50009e60 tx=0x7f8b50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3177ea48298d46da server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b4ffff700 1 -- 172.21.15.29:0/3171112280 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000e070 con 0x7f8b580ce8a0 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b4ffff700 1 -- 172.21.15.29:0/3171112280 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5000bab0 con 0x7f8b580ce8a0 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.184+0000 7f8b4ffff700 1 -- 172.21.15.29:0/3171112280 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000be70 con 0x7f8b580ce8a0 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b580ce8a0 msgr2=0x7f8b580c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b580ce8a0 0x7f8b580c5460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8b50009e60 tx=0x7f8b50009780 comp rx=0 tx=0).stop 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/3171112280 shutdown_connections 2023-10-07T09:29:28.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b580ce8a0 0x7f8b580c5460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/3171112280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b58159750 0x7f8b580d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/3171112280 >> 172.21.15.29:0/3171112280 conn(0x7f8b58063a10 msgr2=0x7f8b580d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/3171112280 shutdown_connections 2023-10-07T09:29:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.185+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/3171112280 wait complete. 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.186+0000 7f8b5f5ff700 1 Processor -- start 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.187+0000 7f8b5f5ff700 1 -- start start 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5f5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b580ce8a0 0x7f8b581535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5f5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b58159750 0x7f8b581518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5f5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 0x7f8b58169540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5f5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58153ca0 con 0x7f8b58159b20 2023-10-07T09:29:28.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5f5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58153f80 con 0x7f8b58159750 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5f5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58169d90 con 0x7f8b580ce8a0 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 0x7f8b58169540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 0x7f8b58169540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58510/0 (socket says 172.21.15.29:58510) 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 -- 172.21.15.29:0/1673341719 learned_addr learned my addr 172.21.15.29:0/1673341719 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5cb9a700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b58159750 0x7f8b581518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5d39b700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b580ce8a0 0x7f8b581535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 -- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b580ce8a0 msgr2=0x7f8b581535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b580ce8a0 0x7f8b581535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 -- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b58159750 msgr2=0x7f8b581518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b58159750 0x7f8b581518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.188+0000 7f8b5db9c700 1 -- 172.21.15.29:0/1673341719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540093c0 con 0x7f8b58159b20 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.189+0000 7f8b5db9c700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 0x7f8b58169540 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8b58088a40 tx=0x7f8b50019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.189+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50012e10 con 0x7f8b58159b20 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.189+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50027470 con 0x7f8b58159b20 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.189+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50027830 con 0x7f8b58159b20 2023-10-07T09:29:28.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.189+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b50009a20 con 0x7f8b58159b20 2023-10-07T09:29:28.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.189+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5816a350 con 0x7f8b58159b20 2023-10-07T09:29:28.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.190+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b58154130 con 0x7f8b58159b20 2023-10-07T09:29:28.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.191+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8b50024070 con 0x7f8b58159b20 2023-10-07T09:29:28.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.192+0000 7f8b4d7fa700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8b3c066f30 0x7f8b3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:28.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.192+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8b50086f50 con 0x7f8b58159b20 2023-10-07T09:29:28.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.192+0000 7f8b5d39b700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8b3c066f30 0x7f8b3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:28.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.193+0000 7f8b5d39b700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8b3c066f30 0x7f8b3c0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009f30 tx=0x7f8b54009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:28.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.196+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b58154130 con 0x7f8b58159b20 2023-10-07T09:29:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.394+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8b581540f0 con 0x7f8b3c066f30 2023-10-07T09:29:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.403+0000 7f8b4d7fa700 1 -- 172.21.15.29:0/1673341719 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264637 (secure 0 0 0) 0x7f8b581540f0 con 0x7f8b3c066f30 2023-10-07T09:29:28.407 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8b3c066f30 msgr2=0x7f8b3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8b3c066f30 0x7f8b3c0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009f30 tx=0x7f8b54009120 comp rx=0 tx=0).stop 2023-10-07T09:29:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 msgr2=0x7f8b58169540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 0x7f8b58169540 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8b58088a40 tx=0x7f8b50019000 comp rx=0 tx=0).stop 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 shutdown_connections 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8b3c066f30 0x7f8b3c0693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159b20 0x7f8b58169540 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b580ce8a0 0x7f8b581535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 --2- 172.21.15.29:0/1673341719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b58159750 0x7f8b581518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 >> 172.21.15.29:0/1673341719 conn(0x7f8b58063a10 msgr2=0x7f8b580c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 shutdown_connections 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:28.407+0000 7f8b5f5ff700 1 -- 172.21.15.29:0/1673341719 wait complete. 2023-10-07T09:29:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:28.425 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-07T09:29:28.018425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385352,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589368,"statfs":{"total":773094113280,"available":770651512832,"internally_reserved":0,"allocated":3022848,"data_stored":1805156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.932156"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956059+0000","last_change":"2023-10-07T09:29:21.945132+0000","last_active":"2023-10-07T09:29:21.956059+0000","last_peered":"2023-10-07T09:29:21.956059+0000","last_clean":"2023-10-07T09:29:21.956059+0000","last_became_active":"2023-10-07T09:29:21.944887+0000","last_became_peered":"2023-10-07T09:29:21.944887+0000","last_unstale":"2023-10-07T09:29:21.956059+0000","last_undegraded":"2023-10-07T09:29:21.956059+0000","last_fullsized":"2023-10-07T09:29:21.956059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:41:47.844527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.940936+0000","last_change":"2023-10-07T09:29:21.935576+0000","last_active":"2023-10-07T09:29:21.940936+0000","last_peered":"2023-10-07T09:29:21.940936+0000","last_clean":"2023-10-07T09:29:21.940936+0000","last_became_active":"2023-10-07T09:29:21.935250+0000","last_became_peered":"2023-10-07T09:29:21.935250+0000","last_unstale":"2023-10-07T09:29:21.940936+0000","last_undegraded":"2023-10-07T09:29:21.940936+0000","last_fullsized":"2023-10-07T09:29:21.940936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:16.004086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954621+0000","last_change":"2023-10-07T09:29:21.941623+0000","last_active":"2023-10-07T09:29:21.954621+0000","last_peered":"2023-10-07T09:29:21.954621+0000","last_clean":"2023-10-07T09:29:21.954621+0000","last_became_active":"2023-10-07T09:29:21.941343+0000","last_became_peered":"2023-10-07T09:29:21.941343+0000","last_unstale":"2023-10-07T09:29:21.954621+0000","last_undegraded":"2023-10-07T09:29:21.954621+0000","last_fullsized":"2023-10-07T09:29:21.954621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:17:35.494089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945939+0000","last_change":"2023-10-07T09:29:21.940295+0000","last_active":"2023-10-07T09:29:21.945939+0000","last_peered":"2023-10-07T09:29:21.945939+0000","last_clean":"2023-10-07T09:29:21.945939+0000","last_became_active":"2023-10-07T09:29:21.940011+0000","last_became_peered":"2023-10-07T09:29:21.940011+0000","last_unstale":"2023-10-07T09:29:21.945939+0000","last_undegraded":"2023-10-07T09:29:21.945939+0000","last_fullsized":"2023-10-07T09:29:21.945939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:09:48.552431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943577+0000","last_change":"2023-10-07T09:29:21.938712+0000","last_active":"2023-10-07T09:29:21.943577+0000","last_peered":"2023-10-07T09:29:21.943577+0000","last_clean":"2023-10-07T09:29:21.943577+0000","last_became_active":"2023-10-07T09:29:21.938428+0000","last_became_peered":"2023-10-07T09:29:21.938428+0000","last_unstale":"2023-10-07T09:29:21.943577+0000","last_undegraded":"2023-10-07T09:29:21.943577+0000","last_fullsized":"2023-10-07T09:29:21.943577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:54:39.120171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103275+0000","last_change":"2023-10-07T09:29:21.114399+0000","last_active":"2023-10-07T09:29:22.103275+0000","last_peered":"2023-10-07T09:29:22.103275+0000","last_clean":"2023-10-07T09:29:22.103275+0000","last_became_active":"2023-10-07T09:29:21.114130+0000","last_became_peered":"2023-10-07T09:29:21.114130+0000","last_unstale":"2023-10-07T09:29:22.103275+0000","last_undegraded":"2023-10-07T09:29:22.103275+0000","last_fullsized":"2023-10-07T09:29:22.103275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:05:03.306731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103349+0000","last_change":"2023-10-07T09:29:21.111992+0000","last_active":"2023-10-07T09:29:22.103349+0000","last_peered":"2023-10-07T09:29:22.103349+0000","last_clean":"2023-10-07T09:29:22.103349+0000","last_became_active":"2023-10-07T09:29:21.111721+0000","last_became_peered":"2023-10-07T09:29:21.111721+0000","last_unstale":"2023-10-07T09:29:22.103349+0000","last_undegraded":"2023-10-07T09:29:22.103349+0000","last_fullsized":"2023-10-07T09:29:22.103349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943597+0000","last_change":"2023-10-07T09:29:21.939060+0000","last_active":"2023-10-07T09:29:21.943597+0000","last_peered":"2023-10-07T09:29:21.943597+0000","last_clean":"2023-10-07T09:29:21.943597+0000","last_became_active":"2023-10-07T09:29:21.938700+0000","last_became_peered":"2023-10-07T09:29:21.938700+0000","last_unstale":"2023-10-07T09:29:21.943597+0000","last_undegraded":"2023-10-07T09:29:21.943597+0000","last_fullsized":"2023-10-07T09:29:21.943597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:35.811275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945677+0000","last_change":"2023-10-07T09:29:21.939840+0000","last_active":"2023-10-07T09:29:21.945677+0000","last_peered":"2023-10-07T09:29:21.945677+0000","last_clean":"2023-10-07T09:29:21.945677+0000","last_became_active":"2023-10-07T09:29:21.939674+0000","last_became_peered":"2023-10-07T09:29:21.939674+0000","last_unstale":"2023-10-07T09:29:21.945677+0000","last_undegraded":"2023-10-07T09:29:21.945677+0000","last_fullsized":"2023-10-07T09:29:21.945677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:40.024706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.947776+0000","last_change":"2023-10-07T09:29:21.940515+0000","last_active":"2023-10-07T09:29:21.947776+0000","last_peered":"2023-10-07T09:29:21.947776+0000","last_clean":"2023-10-07T09:29:21.947776+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.947776+0000","last_undegraded":"2023-10-07T09:29:21.947776+0000","last_fullsized":"2023-10-07T09:29:21.947776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:01.738981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945596+0000","last_change":"2023-10-07T09:29:21.939851+0000","last_active":"2023-10-07T09:29:21.945596+0000","last_peered":"2023-10-07T09:29:21.945596+0000","last_clean":"2023-10-07T09:29:21.945596+0000","last_became_active":"2023-10-07T09:29:21.939540+0000","last_became_peered":"2023-10-07T09:29:21.939540+0000","last_unstale":"2023-10-07T09:29:21.945596+0000","last_undegraded":"2023-10-07T09:29:21.945596+0000","last_fullsized":"2023-10-07T09:29:21.945596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:23:02.797155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929892+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:21.929892+0000","last_peered":"2023-10-07T09:29:21.929892+0000","last_clean":"2023-10-07T09:29:21.929892+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:21.929892+0000","last_undegraded":"2023-10-07T09:29:21.929892+0000","last_fullsized":"2023-10-07T09:29:21.929892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930015+0000","last_change":"2023-10-07T09:29:21.111256+0000","last_active":"2023-10-07T09:29:21.930015+0000","last_peered":"2023-10-07T09:29:21.930015+0000","last_clean":"2023-10-07T09:29:21.930015+0000","last_became_active":"2023-10-07T09:29:21.110731+0000","last_became_peered":"2023-10-07T09:29:21.110731+0000","last_unstale":"2023-10-07T09:29:21.930015+0000","last_undegraded":"2023-10-07T09:29:21.930015+0000","last_fullsized":"2023-10-07T09:29:21.930015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929075+0000","last_change":"2023-10-07T09:29:21.112973+0000","last_active":"2023-10-07T09:29:21.929075+0000","last_peered":"2023-10-07T09:29:21.929075+0000","last_clean":"2023-10-07T09:29:21.929075+0000","last_became_active":"2023-10-07T09:29:21.112720+0000","last_became_peered":"2023-10-07T09:29:21.112720+0000","last_unstale":"2023-10-07T09:29:21.929075+0000","last_undegraded":"2023-10-07T09:29:21.929075+0000","last_fullsized":"2023-10-07T09:29:21.929075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923235+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:21.923235+0000","last_peered":"2023-10-07T09:29:21.923235+0000","last_clean":"2023-10-07T09:29:21.923235+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:21.923235+0000","last_undegraded":"2023-10-07T09:29:21.923235+0000","last_fullsized":"2023-10-07T09:29:21.923235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930385+0000","last_change":"2023-10-07T09:29:21.111416+0000","last_active":"2023-10-07T09:29:21.930385+0000","last_peered":"2023-10-07T09:29:21.930385+0000","last_clean":"2023-10-07T09:29:21.930385+0000","last_became_active":"2023-10-07T09:29:21.111010+0000","last_became_peered":"2023-10-07T09:29:21.111010+0000","last_unstale":"2023-10-07T09:29:21.930385+0000","last_undegraded":"2023-10-07T09:29:21.930385+0000","last_fullsized":"2023-10-07T09:29:21.930385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103269+0000","last_change":"2023-10-07T09:29:21.112583+0000","last_active":"2023-10-07T09:29:22.103269+0000","last_peered":"2023-10-07T09:29:22.103269+0000","last_clean":"2023-10-07T09:29:22.103269+0000","last_became_active":"2023-10-07T09:29:21.112254+0000","last_became_peered":"2023-10-07T09:29:21.112254+0000","last_unstale":"2023-10-07T09:29:22.103269+0000","last_undegraded":"2023-10-07T09:29:22.103269+0000","last_fullsized":"2023-10-07T09:29:22.103269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956315+0000","last_change":"2023-10-07T09:29:21.952857+0000","last_active":"2023-10-07T09:29:21.956315+0000","last_peered":"2023-10-07T09:29:21.956315+0000","last_clean":"2023-10-07T09:29:21.956315+0000","last_became_active":"2023-10-07T09:29:21.952580+0000","last_became_peered":"2023-10-07T09:29:21.952580+0000","last_unstale":"2023-10-07T09:29:21.956315+0000","last_undegraded":"2023-10-07T09:29:21.956315+0000","last_fullsized":"2023-10-07T09:29:21.956315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:48.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956258+0000","last_change":"2023-10-07T09:29:21.952342+0000","last_active":"2023-10-07T09:29:21.956258+0000","last_peered":"2023-10-07T09:29:21.956258+0000","last_clean":"2023-10-07T09:29:21.956258+0000","last_became_active":"2023-10-07T09:29:21.952058+0000","last_became_peered":"2023-10-07T09:29:21.952058+0000","last_unstale":"2023-10-07T09:29:21.956258+0000","last_undegraded":"2023-10-07T09:29:21.956258+0000","last_fullsized":"2023-10-07T09:29:21.956258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:47:56.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476742,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298088,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073752,"statfs":{"total":96636764160,"available":96331522048,"internally_reserved":0,"allocated":819200,"data_stored":668398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:31.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.637+0000 7f7fa47ff700 1 Processor -- start 2023-10-07T09:29:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.637+0000 7f7fa47ff700 1 -- start start 2023-10-07T09:29:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.638+0000 7f7fa47ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f9c0ae8b0 0x7f7f9c0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.638+0000 7f7fa47ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f9c0b1840 0x7f7f9c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.638+0000 7f7fa47ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c158ad0 con 0x7f7f9c157290 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.638+0000 7f7fa47ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9c0b1cc0 con 0x7f7f9c0b1840 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.638+0000 7f7fa47ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9c15b140 con 0x7f7f9c0ae8b0 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa259b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7f9c157290 0x7f7f9c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36932/0 (socket says 172.21.15.29:36932) 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa259b700 1 -- 172.21.15.29:0/1086067822 learned_addr learned my addr 172.21.15.29:0/1086067822 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa2d9c700 1 --2- 172.21.15.29:0/1086067822 >> v2:172.21.15.29:3300/0 conn(0x7f7f9c0b1840 0x7f7f9c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa1d9a700 1 --2- 172.21.15.29:0/1086067822 >> v2:172.21.15.144:3300/0 conn(0x7f7f9c0ae8b0 0x7f7f9c0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa1599700 1 -- 172.21.15.29:0/1086067822 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768786825 0 0) 0x7f7f9c158ad0 con 0x7f7f9c157290 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa1599700 1 -- 172.21.15.29:0/1086067822 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f88003570 con 0x7f7f9c157290 2023-10-07T09:29:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa2d9c700 1 -- 172.21.15.29:0/1086067822 >> v2:172.21.15.144:3300/0 conn(0x7f7f9c0ae8b0 msgr2=0x7f7f9c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa2d9c700 1 --2- 172.21.15.29:0/1086067822 >> v2:172.21.15.144:3300/0 conn(0x7f7f9c0ae8b0 0x7f7f9c0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa2d9c700 1 -- 172.21.15.29:0/1086067822 >> v1:172.21.15.29:6789/0 conn(0x7f7f9c157290 legacy=0x7f7f9c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.639+0000 7f7fa2d9c700 1 -- 172.21.15.29:0/1086067822 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c15b420 con 0x7f7f9c0b1840 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa2d9c700 1 --2- 172.21.15.29:0/1086067822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c157660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0c8d90 tx=0x7f7f94009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ebc38aa337036c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa1599700 1 -- 172.21.15.29:0/1086067822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400e070 con 0x7f7f9c0b1840 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa1599700 1 -- 172.21.15.29:0/1086067822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9400bab0 con 0x7f7f9c0b1840 2023-10-07T09:29:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa1599700 1 -- 172.21.15.29:0/1086067822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400be70 con 0x7f7f9c0b1840 2023-10-07T09:29:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 -- 172.21.15.29:0/1086067822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 msgr2=0x7f7f9c157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/1086067822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c157660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0c8d90 tx=0x7f7f94009780 comp rx=0 tx=0).stop 2023-10-07T09:29:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 -- 172.21.15.29:0/1086067822 shutdown_connections 2023-10-07T09:29:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/1086067822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c157660 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/1086067822 >> v2:172.21.15.144:3300/0 conn(0x7f7f9c0ae8b0 0x7f7f9c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 -- 172.21.15.29:0/1086067822 >> 172.21.15.29:0/1086067822 conn(0x7f7f9c0680a0 msgr2=0x7f7f9c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.640+0000 7f7fa47ff700 1 -- 172.21.15.29:0/1086067822 shutdown_connections 2023-10-07T09:29:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.641+0000 7f7fa47ff700 1 -- 172.21.15.29:0/1086067822 wait complete. 2023-10-07T09:29:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.642+0000 7f7fa47ff700 1 Processor -- start 2023-10-07T09:29:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.642+0000 7f7fa47ff700 1 -- start start 2023-10-07T09:29:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa47ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c15daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa47ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f9c157290 0x7f7f9c152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa47ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9c155960 0x7f7f9c155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa47ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c156340 con 0x7f7f9c0b1840 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa47ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c152e00 con 0x7f7f9c155960 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa47ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c158670 con 0x7f7f9c157290 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa259b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c15daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa259b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c15daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58530/0 (socket says 172.21.15.29:58530) 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa259b700 1 -- 172.21.15.29:0/2529054949 learned_addr learned my addr 172.21.15.29:0/2529054949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa1d9a700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f9c157290 0x7f7f9c152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.643+0000 7f7fa2d9c700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9c155960 0x7f7f9c155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa259b700 1 -- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f9c157290 msgr2=0x7f7f9c152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa259b700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f9c157290 0x7f7f9c152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa259b700 1 -- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9c155960 msgr2=0x7f7f9c155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa259b700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9c155960 0x7f7f9c155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa259b700 1 -- 172.21.15.29:0/2529054949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f94009a20 con 0x7f7f9c0b1840 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa259b700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c15daf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800c7c0 tx=0x7f7f98007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98005cd0 con 0x7f7f9c0b1840 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f98005e30 con 0x7f7f9c0b1840 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9c15e030 con 0x7f7f9c0b1840 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98007320 con 0x7f7f9c0b1840 2023-10-07T09:29:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.644+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9c157d40 con 0x7f7f9c0b1840 2023-10-07T09:29:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.645+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9c06e830 con 0x7f7f9c0b1840 2023-10-07T09:29:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.646+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7f9801b3b0 con 0x7f7f9c0b1840 2023-10-07T09:29:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.647+0000 7f7f927fc700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7f80066f30 0x7f7f800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.647+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7f98083c40 con 0x7f7f9c0b1840 2023-10-07T09:29:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.648+0000 7f7fa1d9a700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7f80066f30 0x7f7f800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.648+0000 7f7fa1d9a700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7f80066f30 0x7f7f800693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0c8d10 tx=0x7f7f8c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.651+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f98050e10 con 0x7f7f9c0b1840 2023-10-07T09:29:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.850+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f9c04c3c0 con 0x7f7f80066f30 2023-10-07T09:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.858+0000 7f7f927fc700 1 -- 172.21.15.29:0/2529054949 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264549 (secure 0 0 0) 0x7f7f9c04c3c0 con 0x7f7f80066f30 2023-10-07T09:29:31.862 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7f80066f30 msgr2=0x7f7f800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7f80066f30 0x7f7f800693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0c8d10 tx=0x7f7f8c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 msgr2=0x7f7f9c15daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c15daf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800c7c0 tx=0x7f7f98007e00 comp rx=0 tx=0).stop 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 shutdown_connections 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7f80066f30 0x7f7f800693e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9c155960 0x7f7f9c155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9c0b1840 0x7f7f9c15daf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 --2- 172.21.15.29:0/2529054949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f9c157290 0x7f7f9c152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 >> 172.21.15.29:0/2529054949 conn(0x7f7f9c0680a0 msgr2=0x7f7f9c0ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 shutdown_connections 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:31.862+0000 7f7fa47ff700 1 -- 172.21.15.29:0/2529054949 wait complete. 2023-10-07T09:29:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:31.881 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-07T09:29:30.018891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385352,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589368,"statfs":{"total":773094113280,"available":770651512832,"internally_reserved":0,"allocated":3022848,"data_stored":1805156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.097328"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956059+0000","last_change":"2023-10-07T09:29:21.945132+0000","last_active":"2023-10-07T09:29:21.956059+0000","last_peered":"2023-10-07T09:29:21.956059+0000","last_clean":"2023-10-07T09:29:21.956059+0000","last_became_active":"2023-10-07T09:29:21.944887+0000","last_became_peered":"2023-10-07T09:29:21.944887+0000","last_unstale":"2023-10-07T09:29:21.956059+0000","last_undegraded":"2023-10-07T09:29:21.956059+0000","last_fullsized":"2023-10-07T09:29:21.956059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:41:47.844527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.940936+0000","last_change":"2023-10-07T09:29:21.935576+0000","last_active":"2023-10-07T09:29:21.940936+0000","last_peered":"2023-10-07T09:29:21.940936+0000","last_clean":"2023-10-07T09:29:21.940936+0000","last_became_active":"2023-10-07T09:29:21.935250+0000","last_became_peered":"2023-10-07T09:29:21.935250+0000","last_unstale":"2023-10-07T09:29:21.940936+0000","last_undegraded":"2023-10-07T09:29:21.940936+0000","last_fullsized":"2023-10-07T09:29:21.940936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:16.004086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954621+0000","last_change":"2023-10-07T09:29:21.941623+0000","last_active":"2023-10-07T09:29:21.954621+0000","last_peered":"2023-10-07T09:29:21.954621+0000","last_clean":"2023-10-07T09:29:21.954621+0000","last_became_active":"2023-10-07T09:29:21.941343+0000","last_became_peered":"2023-10-07T09:29:21.941343+0000","last_unstale":"2023-10-07T09:29:21.954621+0000","last_undegraded":"2023-10-07T09:29:21.954621+0000","last_fullsized":"2023-10-07T09:29:21.954621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:17:35.494089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945939+0000","last_change":"2023-10-07T09:29:21.940295+0000","last_active":"2023-10-07T09:29:21.945939+0000","last_peered":"2023-10-07T09:29:21.945939+0000","last_clean":"2023-10-07T09:29:21.945939+0000","last_became_active":"2023-10-07T09:29:21.940011+0000","last_became_peered":"2023-10-07T09:29:21.940011+0000","last_unstale":"2023-10-07T09:29:21.945939+0000","last_undegraded":"2023-10-07T09:29:21.945939+0000","last_fullsized":"2023-10-07T09:29:21.945939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:09:48.552431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943577+0000","last_change":"2023-10-07T09:29:21.938712+0000","last_active":"2023-10-07T09:29:21.943577+0000","last_peered":"2023-10-07T09:29:21.943577+0000","last_clean":"2023-10-07T09:29:21.943577+0000","last_became_active":"2023-10-07T09:29:21.938428+0000","last_became_peered":"2023-10-07T09:29:21.938428+0000","last_unstale":"2023-10-07T09:29:21.943577+0000","last_undegraded":"2023-10-07T09:29:21.943577+0000","last_fullsized":"2023-10-07T09:29:21.943577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:54:39.120171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103275+0000","last_change":"2023-10-07T09:29:21.114399+0000","last_active":"2023-10-07T09:29:22.103275+0000","last_peered":"2023-10-07T09:29:22.103275+0000","last_clean":"2023-10-07T09:29:22.103275+0000","last_became_active":"2023-10-07T09:29:21.114130+0000","last_became_peered":"2023-10-07T09:29:21.114130+0000","last_unstale":"2023-10-07T09:29:22.103275+0000","last_undegraded":"2023-10-07T09:29:22.103275+0000","last_fullsized":"2023-10-07T09:29:22.103275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:05:03.306731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103349+0000","last_change":"2023-10-07T09:29:21.111992+0000","last_active":"2023-10-07T09:29:22.103349+0000","last_peered":"2023-10-07T09:29:22.103349+0000","last_clean":"2023-10-07T09:29:22.103349+0000","last_became_active":"2023-10-07T09:29:21.111721+0000","last_became_peered":"2023-10-07T09:29:21.111721+0000","last_unstale":"2023-10-07T09:29:22.103349+0000","last_undegraded":"2023-10-07T09:29:22.103349+0000","last_fullsized":"2023-10-07T09:29:22.103349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943597+0000","last_change":"2023-10-07T09:29:21.939060+0000","last_active":"2023-10-07T09:29:21.943597+0000","last_peered":"2023-10-07T09:29:21.943597+0000","last_clean":"2023-10-07T09:29:21.943597+0000","last_became_active":"2023-10-07T09:29:21.938700+0000","last_became_peered":"2023-10-07T09:29:21.938700+0000","last_unstale":"2023-10-07T09:29:21.943597+0000","last_undegraded":"2023-10-07T09:29:21.943597+0000","last_fullsized":"2023-10-07T09:29:21.943597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:35.811275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945677+0000","last_change":"2023-10-07T09:29:21.939840+0000","last_active":"2023-10-07T09:29:21.945677+0000","last_peered":"2023-10-07T09:29:21.945677+0000","last_clean":"2023-10-07T09:29:21.945677+0000","last_became_active":"2023-10-07T09:29:21.939674+0000","last_became_peered":"2023-10-07T09:29:21.939674+0000","last_unstale":"2023-10-07T09:29:21.945677+0000","last_undegraded":"2023-10-07T09:29:21.945677+0000","last_fullsized":"2023-10-07T09:29:21.945677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:40.024706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.947776+0000","last_change":"2023-10-07T09:29:21.940515+0000","last_active":"2023-10-07T09:29:21.947776+0000","last_peered":"2023-10-07T09:29:21.947776+0000","last_clean":"2023-10-07T09:29:21.947776+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.947776+0000","last_undegraded":"2023-10-07T09:29:21.947776+0000","last_fullsized":"2023-10-07T09:29:21.947776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:01.738981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945596+0000","last_change":"2023-10-07T09:29:21.939851+0000","last_active":"2023-10-07T09:29:21.945596+0000","last_peered":"2023-10-07T09:29:21.945596+0000","last_clean":"2023-10-07T09:29:21.945596+0000","last_became_active":"2023-10-07T09:29:21.939540+0000","last_became_peered":"2023-10-07T09:29:21.939540+0000","last_unstale":"2023-10-07T09:29:21.945596+0000","last_undegraded":"2023-10-07T09:29:21.945596+0000","last_fullsized":"2023-10-07T09:29:21.945596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:23:02.797155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929892+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:21.929892+0000","last_peered":"2023-10-07T09:29:21.929892+0000","last_clean":"2023-10-07T09:29:21.929892+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:21.929892+0000","last_undegraded":"2023-10-07T09:29:21.929892+0000","last_fullsized":"2023-10-07T09:29:21.929892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930015+0000","last_change":"2023-10-07T09:29:21.111256+0000","last_active":"2023-10-07T09:29:21.930015+0000","last_peered":"2023-10-07T09:29:21.930015+0000","last_clean":"2023-10-07T09:29:21.930015+0000","last_became_active":"2023-10-07T09:29:21.110731+0000","last_became_peered":"2023-10-07T09:29:21.110731+0000","last_unstale":"2023-10-07T09:29:21.930015+0000","last_undegraded":"2023-10-07T09:29:21.930015+0000","last_fullsized":"2023-10-07T09:29:21.930015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929075+0000","last_change":"2023-10-07T09:29:21.112973+0000","last_active":"2023-10-07T09:29:21.929075+0000","last_peered":"2023-10-07T09:29:21.929075+0000","last_clean":"2023-10-07T09:29:21.929075+0000","last_became_active":"2023-10-07T09:29:21.112720+0000","last_became_peered":"2023-10-07T09:29:21.112720+0000","last_unstale":"2023-10-07T09:29:21.929075+0000","last_undegraded":"2023-10-07T09:29:21.929075+0000","last_fullsized":"2023-10-07T09:29:21.929075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.927905+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:22.927905+0000","last_peered":"2023-10-07T09:29:22.927905+0000","last_clean":"2023-10-07T09:29:22.927905+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:22.927905+0000","last_undegraded":"2023-10-07T09:29:22.927905+0000","last_fullsized":"2023-10-07T09:29:22.927905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930385+0000","last_change":"2023-10-07T09:29:21.111416+0000","last_active":"2023-10-07T09:29:21.930385+0000","last_peered":"2023-10-07T09:29:21.930385+0000","last_clean":"2023-10-07T09:29:21.930385+0000","last_became_active":"2023-10-07T09:29:21.111010+0000","last_became_peered":"2023-10-07T09:29:21.111010+0000","last_unstale":"2023-10-07T09:29:21.930385+0000","last_undegraded":"2023-10-07T09:29:21.930385+0000","last_fullsized":"2023-10-07T09:29:21.930385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103269+0000","last_change":"2023-10-07T09:29:21.112583+0000","last_active":"2023-10-07T09:29:22.103269+0000","last_peered":"2023-10-07T09:29:22.103269+0000","last_clean":"2023-10-07T09:29:22.103269+0000","last_became_active":"2023-10-07T09:29:21.112254+0000","last_became_peered":"2023-10-07T09:29:21.112254+0000","last_unstale":"2023-10-07T09:29:22.103269+0000","last_undegraded":"2023-10-07T09:29:22.103269+0000","last_fullsized":"2023-10-07T09:29:22.103269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956315+0000","last_change":"2023-10-07T09:29:21.952857+0000","last_active":"2023-10-07T09:29:21.956315+0000","last_peered":"2023-10-07T09:29:21.956315+0000","last_clean":"2023-10-07T09:29:21.956315+0000","last_became_active":"2023-10-07T09:29:21.952580+0000","last_became_peered":"2023-10-07T09:29:21.952580+0000","last_unstale":"2023-10-07T09:29:21.956315+0000","last_undegraded":"2023-10-07T09:29:21.956315+0000","last_fullsized":"2023-10-07T09:29:21.956315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:48.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956258+0000","last_change":"2023-10-07T09:29:21.952342+0000","last_active":"2023-10-07T09:29:21.956258+0000","last_peered":"2023-10-07T09:29:21.956258+0000","last_clean":"2023-10-07T09:29:21.956258+0000","last_became_active":"2023-10-07T09:29:21.952058+0000","last_became_peered":"2023-10-07T09:29:21.952058+0000","last_unstale":"2023-10-07T09:29:21.956258+0000","last_undegraded":"2023-10-07T09:29:21.956258+0000","last_fullsized":"2023-10-07T09:29:21.956258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:47:56.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476742,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298088,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073752,"statfs":{"total":96636764160,"available":96331522048,"internally_reserved":0,"allocated":819200,"data_stored":668398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:31.993 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:29:32.000 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-07T09:29:32.000 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-07T09:29:32.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-07T09:29:32.001 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-07T09:29:32.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.195+0000 7fa2fe5ff700 1 Processor -- start 2023-10-07T09:29:32.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.195+0000 7fa2fe5ff700 1 -- start start 2023-10-07T09:29:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2fe5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2f8159750 0x7fa2f80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2fe5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 0x7fa2f80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2fe5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80cbf20 con 0x7fa2f8159b20 2023-10-07T09:29:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2fe5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80cc200 con 0x7fa2f8159750 2023-10-07T09:29:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2fe5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80cc4e0 con 0x7fa2f80ce8a0 2023-10-07T09:29:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2f77fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa2f8159b20 0x7fa2f80ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36938/0 (socket says 172.21.15.29:36938) 2023-10-07T09:29:32.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2f77fe700 1 -- 172.21.15.29:0/998832504 learned_addr learned my addr 172.21.15.29:0/998832504 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:32.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386707775 0 0) 0x7fa2f80cbf20 con 0x7fa2f8159b20 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/998832504 >> v2:172.21.15.144:3300/0 conn(0x7fa2f8159750 0x7fa2f80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e0003570 con 0x7fa2f8159b20 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.197+0000 7fa2fcb9c700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 0x7fa2f80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158593986 0 0) 0x7fa2e0003570 con 0x7fa2f8159b20 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 msgr2=0x7fa2f80c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 0x7fa2f80c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 >> v2:172.21.15.144:3300/0 conn(0x7fa2f8159750 msgr2=0x7fa2f80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 --2- 172.21.15.29:0/998832504 >> v2:172.21.15.144:3300/0 conn(0x7fa2f8159750 0x7fa2f80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f80cc7c0 con 0x7fa2f8159b20 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fa2ec001470 con 0x7fa2f8159b20 2023-10-07T09:29:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 >> v1:172.21.15.29:6789/0 conn(0x7fa2f8159b20 legacy=0x7fa2f80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e0003a60 0x7fa2e0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2e0004810 0x7fa2e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 0x7fa2e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80cc200 con 0x7fa2e0003a60 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80cc4e0 con 0x7fa2e0004810 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e000ba60 con 0x7fa2f80ce8a0 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e0003a60 0x7fa2e0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2f77fe700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2e0004810 0x7fa2e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.198+0000 7fa2fcb9c700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 0x7fa2e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f7fff700 1 -- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 msgr2=0x7fa2e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2f80ce8a0 0x7fa2e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f7fff700 1 -- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2e0004810 msgr2=0x7fa2e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2e0004810 0x7fa2e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f7fff700 1 -- 172.21.15.29:0/998832504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e0003570 con 0x7fa2e0003a60 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e0003a60 0x7fa2e0003e70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009c20 tx=0x7fa2e8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead8f016e594eeae server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8005a00 con 0x7fa2e0003a60 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e8005b60 con 0x7fa2e0003a60 2023-10-07T09:29:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.199+0000 7fa2f6ffd700 1 -- 172.21.15.29:0/998832504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e800e710 con 0x7fa2e0003a60 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/998832504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e0003a60 msgr2=0x7fa2e0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/998832504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e0003a60 0x7fa2e0003e70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009c20 tx=0x7fa2e8009120 comp rx=0 tx=0).stop 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fcb9c700 1 -- 172.21.15.29:0/998832504 reap_dead start 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/998832504 shutdown_connections 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/998832504 >> 172.21.15.29:0/998832504 conn(0x7fa2f8063a10 msgr2=0x7fa2f80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/998832504 shutdown_connections 2023-10-07T09:29:32.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.200+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/998832504 wait complete. 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.201+0000 7fa2fe5ff700 1 Processor -- start 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.202+0000 7fa2fe5ff700 1 -- start start 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.202+0000 7fa2fe5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e0003a60 0x7fa2f80b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.202+0000 7fa2fe5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 0x7fa2f8146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2fe5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2f8158050 0x7fa2f81584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2fe5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8158a00 con 0x7fa2f8159f10 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2fe5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f815b310 con 0x7fa2f8158050 2023-10-07T09:29:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2fe5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80b38a0 con 0x7fa2e0003a60 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 0x7fa2f8146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 0x7fa2f8146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58544/0 (socket says 172.21.15.29:58544) 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 -- 172.21.15.29:0/1582643596 learned_addr learned my addr 172.21.15.29:0/1582643596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2fcb9c700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2f8158050 0x7fa2f81584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e0003a60 0x7fa2f80b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 -- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e0003a60 msgr2=0x7fa2f80b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e0003a60 0x7fa2f80b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 -- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2f8158050 msgr2=0x7fa2f81584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2f8158050 0x7fa2f81584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 -- 172.21.15.29:0/1582643596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f815a4d0 con 0x7fa2f8159f10 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.203+0000 7fa2f77fe700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 0x7fa2f8146fd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec007390 tx=0x7fa2ec001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.204+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec018030 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.204+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec00f920 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.204+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00fce0 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.204+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f815a7b0 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.204+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f815ad70 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.205+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f804c3c0 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.206+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa2ec01b070 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.207+0000 7fa2dffff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa2e4066f30 0x7fa2e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.207+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa2ec019070 con 0x7fa2f8159f10 2023-10-07T09:29:32.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.208+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa2e4066f30 0x7fa2e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.208+0000 7fa2f7fff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa2e4066f30 0x7fa2e40693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009160 tx=0x7fa2e8019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:32.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.211+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2ec0516e0 con 0x7fa2f8159f10 2023-10-07T09:29:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.475+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa2f8064040 con 0x7fa2f8159f10 2023-10-07T09:29:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.479+0000 7fa2dffff700 1 -- 172.21.15.29:0/1582643596 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163707 (secure 0 0 0) 0x7fa2ec054d00 con 0x7fa2f8159f10 2023-10-07T09:29:32.481 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa2e4066f30 msgr2=0x7fa2e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa2e4066f30 0x7fa2e40693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009160 tx=0x7fa2e8019000 comp rx=0 tx=0).stop 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 msgr2=0x7fa2f8146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 0x7fa2f8146fd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec007390 tx=0x7fa2ec001e90 comp rx=0 tx=0).stop 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 shutdown_connections 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa2e4066f30 0x7fa2e40693e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2f8158050 0x7fa2f81584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e0003a60 0x7fa2f80b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 --2- 172.21.15.29:0/1582643596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f8159f10 0x7fa2f8146fd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 >> 172.21.15.29:0/1582643596 conn(0x7fa2f8063a10 msgr2=0x7fa2f8152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 shutdown_connections 2023-10-07T09:29:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.482+0000 7fa2fe5ff700 1 -- 172.21.15.29:0/1582643596 wait complete. 2023-10-07T09:29:32.500 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":114714},{"type":"v1","addr":"172.21.15.144:6833","nonce":114714}]},"active_addr":"172.21.15.144:6833/114714","active_change":"2023-10-07T09:29:04.002303+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4111,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":441038837}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":2779235645}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":465296683}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3859209094}]}]} 2023-10-07T09:29:32.514 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-07T09:29:32.514 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-07T09:29:32.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.708+0000 7f0f1c5ff700 1 Processor -- start 2023-10-07T09:29:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.708+0000 7f0f1c5ff700 1 -- start start 2023-10-07T09:29:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.709+0000 7f0f1c5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0f140ad220 0x7f0f14159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.709+0000 7f0f1c5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f14068620 0x7f0f14068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.709+0000 7f0f1c5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1415af00 con 0x7f0f140ad5f0 2023-10-07T09:29:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.709+0000 7f0f1c5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f14068f80 con 0x7f0f14068620 2023-10-07T09:29:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.709+0000 7f0f1c5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f140ada40 con 0x7f0f140ad220 2023-10-07T09:29:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1ab9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f14068620 0x7f0f14068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f19b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0f140ad5f0 0x7f0f140ac490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36954/0 (socket says 172.21.15.29:36954) 2023-10-07T09:29:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f19b9a700 1 -- 172.21.15.29:0/3699702633 learned_addr learned my addr 172.21.15.29:0/3699702633 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1a39b700 1 --2- 172.21.15.29:0/3699702633 >> v2:172.21.15.144:3300/0 conn(0x7f0f140ad220 0x7f0f14159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1ab9c700 1 -- 172.21.15.29:0/3699702633 >> v2:172.21.15.144:3300/0 conn(0x7f0f140ad220 msgr2=0x7f0f14159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1ab9c700 1 --2- 172.21.15.29:0/3699702633 >> v2:172.21.15.144:3300/0 conn(0x7f0f140ad220 0x7f0f14159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1ab9c700 1 -- 172.21.15.29:0/3699702633 >> v1:172.21.15.29:6789/0 conn(0x7f0f140ad5f0 legacy=0x7f0f140ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1ab9c700 1 -- 172.21.15.29:0/3699702633 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1415a250 con 0x7f0f14068620 2023-10-07T09:29:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f19399700 1 -- 172.21.15.29:0/3699702633 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309523946 0 0) 0x7f0f1415af00 con 0x7f0f140ad5f0 2023-10-07T09:29:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.710+0000 7f0f1ab9c700 1 --2- 172.21.15.29:0/3699702633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14068a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0f140691f0 tx=0x7f0f0c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb1c3d1e09b2ad server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f19399700 1 -- 172.21.15.29:0/3699702633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00e070 con 0x7f0f14068620 2023-10-07T09:29:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f19399700 1 -- 172.21.15.29:0/3699702633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c00bc30 con 0x7f0f14068620 2023-10-07T09:29:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f19399700 1 -- 172.21.15.29:0/3699702633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c012430 con 0x7f0f14068620 2023-10-07T09:29:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/3699702633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 msgr2=0x7f0f14068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/3699702633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14068a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0f140691f0 tx=0x7f0f0c009780 comp rx=0 tx=0).stop 2023-10-07T09:29:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/3699702633 shutdown_connections 2023-10-07T09:29:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/3699702633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14068a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/3699702633 >> v2:172.21.15.144:3300/0 conn(0x7f0f140ad220 0x7f0f14159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/3699702633 >> 172.21.15.29:0/3699702633 conn(0x7f0f1406dd00 msgr2=0x7f0f1406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/3699702633 shutdown_connections 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.711+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/3699702633 wait complete. 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.713+0000 7f0f1c5ff700 1 Processor -- start 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.713+0000 7f0f1c5ff700 1 -- start start 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1c5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f140ad220 0x7f0f140cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1c5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f140ad5f0 0x7f0f140cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1c5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14154290 con 0x7f0f14068620 2023-10-07T09:29:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1c5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14154b30 con 0x7f0f140ad5f0 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1416ab20 con 0x7f0f140ad220 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58554/0 (socket says 172.21.15.29:58554) 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 -- 172.21.15.29:0/1550739242 learned_addr learned my addr 172.21.15.29:0/1550739242 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f19b9a700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f140ad220 0x7f0f140cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1ab9c700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f140ad5f0 0x7f0f140cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 -- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f140ad220 msgr2=0x7f0f140cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f140ad220 0x7f0f140cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 -- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f140ad5f0 msgr2=0x7f0f140cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f140ad5f0 0x7f0f140cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.714+0000 7f0f1a39b700 1 -- 172.21.15.29:0/1550739242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c009a20 con 0x7f0f14068620 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.715+0000 7f0f1a39b700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14154fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000d8c0 tx=0x7f0f10008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.715+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000de60 con 0x7f0f14068620 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.715+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1000b780 con 0x7f0f14068620 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.715+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f141544e0 con 0x7f0f14068620 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.715+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000bb20 con 0x7f0f14068620 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.715+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f141533c0 con 0x7f0f14068620 2023-10-07T09:29:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.716+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f14153640 con 0x7f0f14068620 2023-10-07T09:29:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.717+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0f1001e460 con 0x7f0f14068620 2023-10-07T09:29:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.718+0000 7f0f0a7fc700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0ef8066f30 0x7f0ef80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.718+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0f10084c80 con 0x7f0f14068620 2023-10-07T09:29:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.718+0000 7f0f19b9a700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0ef8066f30 0x7f0ef80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.719+0000 7f0f19b9a700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0ef8066f30 0x7f0ef80693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0f14155d80 tx=0x7f0f04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.722+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f10051e50 con 0x7f0f14068620 2023-10-07T09:29:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.922+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f1404c3c0 con 0x7f0f14068620 2023-10-07T09:29:32.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.924+0000 7f0f0a7fc700 1 -- 172.21.15.29:0/1550739242 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f0f10055470 con 0x7f0f14068620 2023-10-07T09:29:32.949 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:32.949 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:32.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0ef8066f30 msgr2=0x7f0ef80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0ef8066f30 0x7f0ef80693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0f14155d80 tx=0x7f0f04009040 comp rx=0 tx=0).stop 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 msgr2=0x7f0f14154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14154fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000d8c0 tx=0x7f0f10008040 comp rx=0 tx=0).stop 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 shutdown_connections 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0ef8066f30 0x7f0ef80693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f140ad5f0 0x7f0f140cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f14068620 0x7f0f14154fc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 --2- 172.21.15.29:0/1550739242 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f140ad220 0x7f0f140cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:32.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 >> 172.21.15.29:0/1550739242 conn(0x7f0f1406dd00 msgr2=0x7f0f1406b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:32.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 shutdown_connections 2023-10-07T09:29:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:32.927+0000 7f0f1c5ff700 1 -- 172.21.15.29:0/1550739242 wait complete. 2023-10-07T09:29:33.024 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-07T09:29:33.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.220+0000 7fd17f9ff700 1 Processor -- start 2023-10-07T09:29:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.220+0000 7fd17f9ff700 1 -- start start 2023-10-07T09:29:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17f9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd178159750 0x7fd1780d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17f9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780cbf20 con 0x7fd178159b20 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17f9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780cc200 con 0x7fd1780ce8a0 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17f9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1780cc4e0 con 0x7fd178159750 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17d79b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd178159750 0x7fd1780d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17d79b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd178159750 0x7fd1780d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56284/0 (socket says 172.21.15.29:56284) 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17d79b700 1 -- 172.21.15.29:0/2867520570 learned_addr learned my addr 172.21.15.29:0/2867520570 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.221+0000 7fd17cf9a700 1 --1- 172.21.15.29:0/2867520570 >> v1:172.21.15.29:6789/0 conn(0x7fd178159b20 0x7fd1780ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58592/0 (socket says 172.21.15.29:58592) 2023-10-07T09:29:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd17d79b700 1 -- 172.21.15.29:0/2867520570 >> v1:172.21.15.144:6789/0 conn(0x7fd1780ce8a0 legacy=0x7fd1780c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd17d79b700 1 -- 172.21.15.29:0/2867520570 >> v1:172.21.15.29:6789/0 conn(0x7fd178159b20 legacy=0x7fd1780ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd17d79b700 1 -- 172.21.15.29:0/2867520570 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780cc7c0 con 0x7fd178159750 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd17d79b700 1 --2- 172.21.15.29:0/2867520570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159750 0x7fd1780d10a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd174009990 tx=0x7fd174009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949e4f92787d5bd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd16ffff700 1 -- 172.21.15.29:0/2867520570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174005c10 con 0x7fd178159750 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd16ffff700 1 -- 172.21.15.29:0/2867520570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd174005d70 con 0x7fd178159750 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.222+0000 7fd16ffff700 1 -- 172.21.15.29:0/2867520570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400b880 con 0x7fd178159750 2023-10-07T09:29:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2867520570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159750 msgr2=0x7fd1780d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2867520570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159750 0x7fd1780d10a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd174009990 tx=0x7fd174009220 comp rx=0 tx=0).stop 2023-10-07T09:29:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2867520570 shutdown_connections 2023-10-07T09:29:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2867520570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159750 0x7fd1780d10a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2867520570 >> 172.21.15.29:0/2867520570 conn(0x7fd178063a10 msgr2=0x7fd1780d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2867520570 shutdown_connections 2023-10-07T09:29:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.223+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2867520570 wait complete. 2023-10-07T09:29:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.224+0000 7fd17f9ff700 1 Processor -- start 2023-10-07T09:29:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.225+0000 7fd17f9ff700 1 -- start start 2023-10-07T09:29:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17f9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1780ce8a0 0x7fd1780b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17f9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd178159750 0x7fd1780b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17f9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 0x7fd178144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17f9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780b5be0 con 0x7fd178159b20 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17f9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd178159f10 con 0x7fd178159750 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17f9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17815a4d0 con 0x7fd1780ce8a0 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17df9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 0x7fd178144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17df9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 0x7fd178144fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56294/0 (socket says 172.21.15.29:56294) 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17df9c700 1 -- 172.21.15.29:0/2372680351 learned_addr learned my addr 172.21.15.29:0/2372680351 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17cf9a700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd178159750 0x7fd1780b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.226+0000 7fd17d79b700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1780ce8a0 0x7fd1780b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17df9c700 1 -- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1780ce8a0 msgr2=0x7fd1780b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17df9c700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1780ce8a0 0x7fd1780b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17df9c700 1 -- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd178159750 msgr2=0x7fd1780b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17df9c700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd178159750 0x7fd1780b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17df9c700 1 -- 172.21.15.29:0/2372680351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd174009480 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17d79b700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1780ce8a0 0x7fd1780b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17cf9a700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd178159750 0x7fd1780b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17df9c700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 0x7fd178144fb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd17000c960 tx=0x7fd1700026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000cea0 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17000a7a0 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1780b42b0 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000ab60 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.227+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1780c8240 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.228+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17804c3c0 con 0x7fd178159b20 2023-10-07T09:29:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.229+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd170012420 con 0x7fd178159b20 2023-10-07T09:29:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.230+0000 7fd16d7fa700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd15c066f30 0x7fd15c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.230+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fd1700849b0 con 0x7fd178159b20 2023-10-07T09:29:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.230+0000 7fd17d79b700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd15c066f30 0x7fd15c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.231+0000 7fd17d79b700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd15c066f30 0x7fd15c0693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd174007fd0 tx=0x7fd174015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.234+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd170051b80 con 0x7fd178159b20 2023-10-07T09:29:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.438+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1780b48b0 con 0x7fd178159b20 2023-10-07T09:29:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.439+0000 7fd16d7fa700 1 -- 172.21.15.29:0/2372680351 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7fd17006be60 con 0x7fd178159b20 2023-10-07T09:29:33.611 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:33.619 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd15c066f30 msgr2=0x7fd15c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd15c066f30 0x7fd15c0693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd174007fd0 tx=0x7fd174015000 comp rx=0 tx=0).stop 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 msgr2=0x7fd178144fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 0x7fd178144fb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd17000c960 tx=0x7fd1700026e0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 shutdown_connections 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd15c066f30 0x7fd15c0693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd178159b20 0x7fd178144fb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1780ce8a0 0x7fd1780b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 --2- 172.21.15.29:0/2372680351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd178159750 0x7fd1780b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.441+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 >> 172.21.15.29:0/2372680351 conn(0x7fd178063a10 msgr2=0x7fd1780c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.442+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 shutdown_connections 2023-10-07T09:29:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.442+0000 7fd17f9ff700 1 -- 172.21.15.29:0/2372680351 wait complete. 2023-10-07T09:29:33.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-07T09:29:33.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-07T09:29:33.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-07T09:29:33.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-07T09:29:33.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-07T09:29:33.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-07T09:29:33.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-07T09:29:33.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-07T09:29:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.992+0000 7fcb933ff700 1 Processor -- start 2023-10-07T09:29:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.992+0000 7f38e07ff700 1 Processor -- start 2023-10-07T09:29:33.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.992+0000 7f38e07ff700 1 -- start start 2023-10-07T09:29:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.992+0000 7fcb933ff700 1 -- start start 2023-10-07T09:29:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb933ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb933ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcb8c0cc870 0x7fcb8c15d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb933ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb8c0b1bc0 0x7fcb8c15da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb933ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c15eeb0 con 0x7fcb8c0cc870 2023-10-07T09:29:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb933ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c0aebe0 con 0x7fcb8c0b1bc0 2023-10-07T09:29:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb933ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c15e2a0 con 0x7fcb8c0b17f0 2023-10-07T09:29:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.995+0000 7fcb9199c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb8c0b1bc0 0x7fcb8c15da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9199c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb8c0b1bc0 0x7fcb8c15da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52678/0 (socket says 172.21.15.29:52678) 2023-10-07T09:29:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9199c700 1 -- 172.21.15.29:0/1357436844 learned_addr learned my addr 172.21.15.29:0/1357436844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9119b700 1 --2- 172.21.15.29:0/1357436844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7f38e07ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d80b1bc0 0x7f38d80afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7f38e07ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d815eeb0 con 0x7f38d80cc870 2023-10-07T09:29:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7f38e07ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d80aebe0 con 0x7f38d80b17f0 2023-10-07T09:29:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7f38e07ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d815e2a0 con 0x7f38d80b1bc0 2023-10-07T09:29:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9119b700 1 -- 172.21.15.29:0/1357436844 >> v2:172.21.15.144:3300/0 conn(0x7fcb8c0b1bc0 msgr2=0x7fcb8c15da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9119b700 1 --2- 172.21.15.29:0/1357436844 >> v2:172.21.15.144:3300/0 conn(0x7fcb8c0b1bc0 0x7fcb8c15da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9119b700 1 -- 172.21.15.29:0/1357436844 >> v2:172.21.15.29:3300/0 conn(0x7fcb8c0cc870 msgr2=0x7fcb8c15d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-07T09:29:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9119b700 1 --2- 172.21.15.29:0/1357436844 >> v2:172.21.15.29:3300/0 conn(0x7fcb8c0cc870 0x7fcb8c15d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7fcb9119b700 1 -- 172.21.15.29:0/1357436844 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c15e580 con 0x7fcb8c0b17f0 2023-10-07T09:29:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7fcb9119b700 1 --2- 172.21.15.29:0/1357436844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c0afcb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098e0 tx=0x7fcb880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83ae55344627cbe server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7f2821dff700 1 Processor -- start 2023-10-07T09:29:34.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7f38ddd9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f38d80cc870 0x7f38d815f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58604/0 (socket says 172.21.15.29:58604) 2023-10-07T09:29:34.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7f38ddd9a700 1 -- 172.21.15.29:0/1600929029 learned_addr learned my addr 172.21.15.29:0/1600929029 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7fcb83fff700 1 -- 172.21.15.29:0/1357436844 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88011630 con 0x7fcb8c0b17f0 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7fcb83fff700 1 -- 172.21.15.29:0/1357436844 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88011790 con 0x7fcb8c0b17f0 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7fcb83fff700 1 -- 172.21.15.29:0/1357436844 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88011b50 con 0x7fcb8c0b17f0 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 -- 172.21.15.29:0/1357436844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 msgr2=0x7fcb8c0afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 --2- 172.21.15.29:0/1357436844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c0afcb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098e0 tx=0x7fcb880091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 -- 172.21.15.29:0/1357436844 shutdown_connections 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 --2- 172.21.15.29:0/1357436844 >> v2:172.21.15.144:3300/0 conn(0x7fcb8c0b1bc0 0x7fcb8c15da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 --2- 172.21.15.29:0/1357436844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c0afcb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 --2- 172.21.15.29:0/1357436844 >> v2:172.21.15.29:3300/0 conn(0x7fcb8c0cc870 0x7fcb8c15d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7fcb933ff700 1 -- 172.21.15.29:0/1357436844 >> 172.21.15.29:0/1357436844 conn(0x7fcb8c0680a0 msgr2=0x7fcb8c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59457842 0 0) 0x7f38d815eeb0 con 0x7f38d80cc870 2023-10-07T09:29:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c4003570 con 0x7f38d80cc870 2023-10-07T09:29:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168797534 0 0) 0x7f38c4003570 con 0x7f38d80cc870 2023-10-07T09:29:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d80b1bc0 msgr2=0x7f38d80afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38ded9c700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d80b1bc0 0x7f38d80afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d80b1bc0 0x7f38d80afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 >> v1:172.21.15.144:6789/0 conn(0x7f38d80b17f0 legacy=0x7f38d815d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-07T09:29:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d815e580 con 0x7f38d80cc870 2023-10-07T09:29:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7f2821dff700 1 -- start start 2023-10-07T09:29:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f38d4001470 con 0x7f38d80cc870 2023-10-07T09:29:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.994+0000 7ff225bff700 1 Processor -- start 2023-10-07T09:29:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.994+0000 7ff225bff700 1 -- start start 2023-10-07T09:29:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7ff225bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff2200afd50 0x7ff2200b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 >> v1:172.21.15.29:6789/0 conn(0x7f38d80cc870 legacy=0x7f38d815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38c4003b10 0x7f38c4003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38c4004810 0x7f38c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38c400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d815e2a0 con 0x7f38c4005370 2023-10-07T09:29:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c400c300 con 0x7f38c4004810 2023-10-07T09:29:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.999+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c400c4c0 con 0x7f38c4003b10 2023-10-07T09:29:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f38ded9c700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d80b1bc0 0x7f38d80afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.996+0000 7ff225bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 0x7ff220152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7ff21effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 0x7ff220152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7ff225bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff220157bd0 con 0x7ff2200b0120 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7ff225bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff220156fc0 con 0x7ff2200afd50 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7ff225bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2201572a0 con 0x7ff220156980 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7ff21effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 0x7ff220152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44218/0 (socket says 172.21.15.29:44218) 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.997+0000 7ff21effd700 1 -- 172.21.15.29:0/805406638 learned_addr learned my addr 172.21.15.29:0/805406638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7ff21effd700 1 -- 172.21.15.29:0/805406638 >> v2:172.21.15.144:3300/0 conn(0x7ff2200afd50 msgr2=0x7ff2200b1a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7ff21effd700 1 --2- 172.21.15.29:0/805406638 >> v2:172.21.15.144:3300/0 conn(0x7ff2200afd50 0x7ff2200b1a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7ff21effd700 1 -- 172.21.15.29:0/805406638 >> v1:172.21.15.29:6789/0 conn(0x7ff2200b0120 legacy=0x7ff220159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7ff21effd700 1 -- 172.21.15.29:0/805406638 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff220157580 con 0x7ff220156980 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:33.998+0000 7ff21effd700 1 --2- 172.21.15.29:0/805406638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 0x7ff220152ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff2140099b0 tx=0x7ff214009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73260200bc04d5a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7ff21e7fc700 1 -- 172.21.15.29:0/805406638 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21400b820 con 0x7ff220156980 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7ff21e7fc700 1 -- 172.21.15.29:0/805406638 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21400b980 con 0x7ff220156980 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7ff21e7fc700 1 -- 172.21.15.29:0/805406638 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21400bd20 con 0x7ff220156980 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f38ddd9a700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38c4004810 0x7f38c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f38ded9c700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38c400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f2821dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f281c0b1bc0 0x7f281c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f2821dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c15eeb0 con 0x7f281c0cc870 2023-10-07T09:29:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f2821dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c0aebe0 con 0x7f281c0b17f0 2023-10-07T09:29:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f2821dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c15e2a0 con 0x7f281c0b1bc0 2023-10-07T09:29:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f281b7fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f281c0b17f0 0x7f281c15d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39352/0 (socket says 172.21.15.29:39352) 2023-10-07T09:29:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.000+0000 7f281b7fe700 1 -- 172.21.15.29:0/207770261 learned_addr learned my addr 172.21.15.29:0/207770261 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f0d103ff700 1 Processor -- start 2023-10-07T09:29:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7fcb933ff700 1 -- 172.21.15.29:0/1357436844 shutdown_connections 2023-10-07T09:29:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7fcb933ff700 1 -- 172.21.15.29:0/1357436844 wait complete. 2023-10-07T09:29:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7ff225bff700 1 -- 172.21.15.29:0/805406638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 msgr2=0x7ff220152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7ff225bff700 1 --2- 172.21.15.29:0/805406638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 0x7ff220152ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff2140099b0 tx=0x7ff214009250 comp rx=0 tx=0).stop 2023-10-07T09:29:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7ff225bff700 1 -- 172.21.15.29:0/805406638 shutdown_connections 2023-10-07T09:29:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7ff225bff700 1 --2- 172.21.15.29:0/805406638 >> v2:172.21.15.144:3300/0 conn(0x7ff2200afd50 0x7ff2200b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7ff225bff700 1 --2- 172.21.15.29:0/805406638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff220156980 0x7ff220152ce0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7ff225bff700 1 -- 172.21.15.29:0/805406638 >> 172.21.15.29:0/805406638 conn(0x7ff2200680a0 msgr2=0x7ff2200cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f0d103ff700 1 -- start start 2023-10-07T09:29:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509230963 0 0) 0x7f281c0aebe0 con 0x7f281c0b17f0 2023-10-07T09:29:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2808003570 con 0x7f281c0b17f0 2023-10-07T09:29:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f38ded9c700 1 -- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38c4003b10 msgr2=0x7f38c4003f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-07T09:29:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f38ded9c700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38c4003b10 0x7f38c4003f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f38ded9c700 1 -- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38c4004810 msgr2=0x7f38c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f38ded9c700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38c4004810 0x7f38c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.001+0000 7f38ded9c700 1 -- 172.21.15.29:0/1600929029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4003570 con 0x7f38c4005370 2023-10-07T09:29:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281bfff700 1 --2- 172.21.15.29:0/207770261 >> v2:172.21.15.29:3300/0 conn(0x7f281c0b1bc0 0x7f281c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690172620 0 0) 0x7f2808003570 con 0x7f281c0b17f0 2023-10-07T09:29:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 >> v2:172.21.15.29:3300/0 conn(0x7f281c0b1bc0 msgr2=0x7f281c0afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 --2- 172.21.15.29:0/207770261 >> v2:172.21.15.29:3300/0 conn(0x7f281c0b1bc0 0x7f281c0afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 >> v1:172.21.15.29:6789/0 conn(0x7f281c0cc870 legacy=0x7f281c15f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c15e580 con 0x7f281c0b17f0 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f280c001470 con 0x7f281c0b17f0 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 >> v1:172.21.15.144:6789/0 conn(0x7f281c0b17f0 legacy=0x7f281c15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2808003ae0 0x7f2808003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808004810 0x7f2808004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28080053c0 0x7f280800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c15e2a0 con 0x7f2808004810 2023-10-07T09:29:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f38ded9c700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38c400ba60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38d80b1f90 tx=0x7f38d00075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff6633d7202e35e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280800c320 con 0x7f28080053c0 2023-10-07T09:29:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280800c4e0 con 0x7f2808003ae0 2023-10-07T09:29:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.002+0000 7fcb933ff700 1 Processor -- start 2023-10-07T09:29:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d00078f0 con 0x7f38c4005370 2023-10-07T09:29:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d000cc00 con 0x7f38c4005370 2023-10-07T09:29:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f38dd599700 1 -- 172.21.15.29:0/1600929029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d000d440 con 0x7f38c4005370 2023-10-07T09:29:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f0d103ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0d08159ec0 0x7f0d0815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f0d103ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0814d790 con 0x7f0d080ad370 2023-10-07T09:29:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f0d103ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080ae5d0 con 0x7f0d080af400 2023-10-07T09:29:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f0d103ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0815a880 con 0x7f0d08159ec0 2023-10-07T09:29:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f281b7fe700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2808003ae0 0x7f2808003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7f281bfff700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28080053c0 0x7f280800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f38e07ff700 1 -- 172.21.15.29:0/1600929029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 msgr2=0x7f38c400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f38e07ff700 1 --2- 172.21.15.29:0/1600929029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38c400ba60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38d80b1f90 tx=0x7f38d00075c0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f38ded9c700 1 -- 172.21.15.29:0/1600929029 reap_dead start 2023-10-07T09:29:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.003+0000 7fcb933ff700 1 -- start start 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281b7fe700 1 -- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28080053c0 msgr2=0x7f280800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281b7fe700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28080053c0 0x7f280800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281b7fe700 1 -- 172.21.15.29:0/207770261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808004810 msgr2=0x7f2808004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281affd700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808004810 0x7f2808004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281b7fe700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808004810 0x7f2808004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281bfff700 1 -- 172.21.15.29:0/207770261 reap_dead start 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281b7fe700 1 -- 172.21.15.29:0/207770261 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c15e580 con 0x7f2808003ae0 2023-10-07T09:29:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281affd700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808004810 0x7f2808004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb933ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c13a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb933ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c0b1bc0 0x7fcb8c138540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb933ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c138a80 0x7fcb8c138f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb933ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c13c8e0 con 0x7fcb8c138a80 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb933ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c13d6e0 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb933ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c13cad0 con 0x7fcb8c0b17f0 2023-10-07T09:29:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f38e07ff700 1 -- 172.21.15.29:0/1600929029 shutdown_connections 2023-10-07T09:29:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f38e07ff700 1 -- 172.21.15.29:0/1600929029 >> 172.21.15.29:0/1600929029 conn(0x7f38d80680a0 msgr2=0x7f38d80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f281b7fe700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2808003ae0 0x7f2808003ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c3e0 tx=0x7f280c001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb2f967175e79ac server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f38e07ff700 1 -- 172.21.15.29:0/1600929029 shutdown_connections 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb9119b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c13a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb9199c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c138a80 0x7fcb8c138f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb9199c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c138a80 0x7fcb8c138f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56330/0 (socket says 172.21.15.29:56330) 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7fcb9199c700 1 -- 172.21.15.29:0/3175986811 learned_addr learned my addr 172.21.15.29:0/3175986811 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7fcb9099a700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c0b1bc0 0x7fcb8c138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7fcb9099a700 1 -- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 msgr2=0x7fcb8c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7fcb9099a700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b17f0 0x7fcb8c13a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7fcb9099a700 1 -- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c138a80 msgr2=0x7fcb8c138f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7fcb9099a700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c138a80 0x7fcb8c138f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7fcb9099a700 1 -- 172.21.15.29:0/3175986811 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88009420 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f38e07ff700 1 -- 172.21.15.29:0/1600929029 wait complete. 2023-10-07T09:29:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f0d0d99a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0d08159ec0 0x7f0d0815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f0d0d99a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0d08159ec0 0x7f0d0815a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52712/0 (socket says 172.21.15.29:52712) 2023-10-07T09:29:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f0d0d99a700 1 -- 172.21.15.29:0/1784850793 learned_addr learned my addr 172.21.15.29:0/1784850793 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f0d0d99a700 1 -- 172.21.15.29:0/1784850793 >> v1:172.21.15.144:6789/0 conn(0x7f0d080af400 legacy=0x7f0d080af840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f0d0d99a700 1 -- 172.21.15.29:0/1784850793 >> v1:172.21.15.29:6789/0 conn(0x7f0d080ad370 legacy=0x7f0d080cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c00cc30 con 0x7f2808003ae0 2023-10-07T09:29:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c00cd90 con 0x7f2808003ae0 2023-10-07T09:29:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f281a7fc700 1 -- 172.21.15.29:0/207770261 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c005b70 con 0x7f2808003ae0 2023-10-07T09:29:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f2821dff700 1 -- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2808003ae0 msgr2=0x7f2808003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f2821dff700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2808003ae0 0x7f2808003ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c3e0 tx=0x7f280c001e60 comp rx=0 tx=0).stop 2023-10-07T09:29:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.004+0000 7f0d0d99a700 1 -- 172.21.15.29:0/1784850793 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080ad740 con 0x7f0d08159ec0 2023-10-07T09:29:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f0d0d99a700 1 --2- 172.21.15.29:0/1784850793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d08159ec0 0x7f0d0815a340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000bfc0 tx=0x7f0d000093d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb5a95ef830fdb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f0d0d199700 1 -- 172.21.15.29:0/1784850793 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0000b2e0 con 0x7f0d08159ec0 2023-10-07T09:29:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f0d0d199700 1 -- 172.21.15.29:0/1784850793 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0000b440 con 0x7f0d08159ec0 2023-10-07T09:29:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.005+0000 7f0d0d199700 1 -- 172.21.15.29:0/1784850793 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00008fc0 con 0x7f0d08159ec0 2023-10-07T09:29:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f2821dff700 1 -- 172.21.15.29:0/207770261 shutdown_connections 2023-10-07T09:29:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f2821dff700 1 --2- 172.21.15.29:0/207770261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2808003ae0 0x7f2808003ed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f2821dff700 1 -- 172.21.15.29:0/207770261 >> 172.21.15.29:0/207770261 conn(0x7f281c0680a0 msgr2=0x7f281c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7fcb9099a700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c0b1bc0 0x7fcb8c138540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c06d2a0 tx=0x7fcb7c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00f510 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c00f690 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00fa50 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f3f253ff700 1 Processor -- start 2023-10-07T09:29:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.006+0000 7f3f253ff700 1 -- start start 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.008+0000 7ff225bff700 1 -- 172.21.15.29:0/805406638 shutdown_connections 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.008+0000 7ff225bff700 1 -- 172.21.15.29:0/805406638 wait complete. 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7ff225bff700 1 Processor -- start 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7f38e07ff700 1 Processor -- start 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.008+0000 7f38e07ff700 1 -- start start 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f38e07ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38d80c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f38e07ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d8160dd0 0x7f38d8156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f38e07ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8157f20 0x7f38d8158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f38e07ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d80c9750 con 0x7f38c4005370 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f38e07ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d81588d0 con 0x7f38d8160dd0 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f38e07ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d80c88e0 con 0x7f38d8157f20 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f38ded9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8157f20 0x7f38d8158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f38ded9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8157f20 0x7f38d8158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44260/0 (socket says 172.21.15.29:44260) 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f38ded9c700 1 -- 172.21.15.29:0/505291652 learned_addr learned my addr 172.21.15.29:0/505291652 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f38de59b700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38d80c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f38ded9c700 1 -- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d8160dd0 msgr2=0x7f38d8156520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f38ded9c700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d8160dd0 0x7f38d8156520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f38ded9c700 1 -- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 msgr2=0x7f38d80c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f38ded9c700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38c4005370 0x7f38d80c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f38ded9c700 1 -- 172.21.15.29:0/505291652 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d80c8b40 con 0x7f38d8157f20 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f38ded9c700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8157f20 0x7f38d8158390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f38c400c2d0 tx=0x7f38d0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f0d103ff700 1 -- 172.21.15.29:0/1784850793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d08159ec0 msgr2=0x7f0d0815a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f0d103ff700 1 --2- 172.21.15.29:0/1784850793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d08159ec0 0x7f0d0815a340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000bfc0 tx=0x7f0d000093d0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f0d103ff700 1 -- 172.21.15.29:0/1784850793 shutdown_connections 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f0d103ff700 1 --2- 172.21.15.29:0/1784850793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d08159ec0 0x7f0d0815a340 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f0d103ff700 1 -- 172.21.15.29:0/1784850793 >> 172.21.15.29:0/1784850793 conn(0x7f0d080680a0 msgr2=0x7f0d080cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f0d103ff700 1 -- 172.21.15.29:0/1784850793 shutdown_connections 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c13cdb0 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c13d370 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.008+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb8c04c3c0 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7f2821dff700 1 -- 172.21.15.29:0/207770261 shutdown_connections 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.007+0000 7f2821dff700 1 -- 172.21.15.29:0/207770261 wait complete. 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f2821dff700 1 Processor -- start 2023-10-07T09:29:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f2821dff700 1 -- start start 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f2821dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808003ae0 0x7f281c0c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f2821dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c0c9370 0x7f281c159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f2821dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f281c06dc70 0x7f281c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.012+0000 7f2821dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0c2360 con 0x7f2808003ae0 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.012+0000 7f2821dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c06e620 con 0x7f281c0c9370 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.012+0000 7f2821dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0c14b0 con 0x7f281c06dc70 2023-10-07T09:29:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.012+0000 7f5f9499d700 1 Processor -- start 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7ff225bff700 1 -- start start 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7ff225bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2200afd50 0x7ff220144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7ff225bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2200b0120 0x7ff2200c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7ff225bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22016e4e0 0x7ff220151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7ff225bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22014bc90 con 0x7ff2200afd50 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7ff225bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff220144870 con 0x7ff22016e4e0 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7ff225bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2200c4d20 con 0x7ff2200b0120 2023-10-07T09:29:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.010+0000 7f0d103ff700 1 -- 172.21.15.29:0/1784850793 wait complete. 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f0d103ff700 1 Processor -- start 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f3f253ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3f200ae7f0 0x7f3f200aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f3f253ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3f200b1c90 0x7f3f20166340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f3f253ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f200af5d0 con 0x7f3f200afac0 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f3f253ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f20166880 con 0x7f3f200ae7f0 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.009+0000 7f3f253ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f20167de0 con 0x7f3f200b1c90 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f3f1e7fc700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3f200afac0 0x7f3f200b1580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39376/0 (socket says 172.21.15.29:39376) 2023-10-07T09:29:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.011+0000 7f3f1e7fc700 1 -- 172.21.15.29:0/2836726238 learned_addr learned my addr 172.21.15.29:0/2836726238 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f281affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c0c9370 0x7f281c159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c0c9370 0x7f281c159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52750/0 (socket says 172.21.15.29:52750) 2023-10-07T09:29:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 -- 172.21.15.29:0/3112558356 learned_addr learned my addr 172.21.15.29:0/3112558356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281bfff700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f281c06dc70 0x7f281c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2200b0120 0x7ff2200c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2200b0120 0x7ff2200c4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44274/0 (socket says 172.21.15.29:44274) 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 -- 172.21.15.29:0/1343323998 learned_addr learned my addr 172.21.15.29:0/1343323998 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21ffff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22016e4e0 0x7ff220151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 -- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f281c06dc70 msgr2=0x7f281c06e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f281c06dc70 0x7f281c06e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 -- 172.21.15.29:0/3112558356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808003ae0 msgr2=0x7f281c0c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2808003ae0 0x7f281c0c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7f281affd700 1 -- 172.21.15.29:0/3112558356 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0c1750 con 0x7f281c0c9370 2023-10-07T09:29:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f5f9499d700 1 -- start start 2023-10-07T09:29:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 -- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22016e4e0 msgr2=0x7ff220151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22016e4e0 0x7ff220151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 -- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2200afd50 msgr2=0x7ff220144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2200afd50 0x7ff220144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.014+0000 7ff21effd700 1 -- 172.21.15.29:0/1343323998 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140094f0 con 0x7ff2200b0120 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7ff21ffff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22016e4e0 0x7ff220151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f281affd700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c0c9370 0x7f281c159450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f280400d520 tx=0x7f2804009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f5f9499d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f900c6940 0x7f5f90164210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f5f9499d700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f900c5da0 con 0x7f5f90159530 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f5f9499d700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f900c6d40 con 0x7f5f900d0e60 2023-10-07T09:29:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f5f9499d700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f900cca90 con 0x7f5f900c6940 2023-10-07T09:29:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7ff21effd700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2200b0120 0x7ff2200c4420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff214003080 tx=0x7ff214016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f0d103ff700 1 -- start start 2023-10-07T09:29:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f0d103ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d080ad370 0x7f0d08177240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f0d103ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d080af400 0x7f0d080c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f0d103ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d08159ec0 0x7f0d080ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f0d103ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0814d290 con 0x7f0d08159ec0 2023-10-07T09:29:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f0d103ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08178860 con 0x7f0d080ad370 2023-10-07T09:29:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f0d103ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08177c50 con 0x7f0d080af400 2023-10-07T09:29:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280400b070 con 0x7f281c0c9370 2023-10-07T09:29:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2804011580 con 0x7f281c0c9370 2023-10-07T09:29:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804011940 con 0x7f281c0c9370 2023-10-07T09:29:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f3f1f7fe700 1 --2- 172.21.15.29:0/2836726238 >> v2:172.21.15.144:3300/0 conn(0x7f3f200b1c90 0x7f3f20166340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f5f8effd700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f5f900d0e60 0x7f5f900c6230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39384/0 (socket says 172.21.15.29:39384) 2023-10-07T09:29:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f5f8effd700 1 -- 172.21.15.29:0/3683478204 learned_addr learned my addr 172.21.15.29:0/3683478204 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d080af400 0x7f0d080c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d080af400 0x7f0d080c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44286/0 (socket says 172.21.15.29:44286) 2023-10-07T09:29:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 -- 172.21.15.29:0/2058947923 learned_addr learned my addr 172.21.15.29:0/2058947923 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c0c1a30 con 0x7f281c0c9370 2023-10-07T09:29:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c0c1ff0 con 0x7f281c0c9370 2023-10-07T09:29:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.015+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214011bd0 con 0x7ff2200b0120 2023-10-07T09:29:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff214020070 con 0x7ff2200b0120 2023-10-07T09:29:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21401b450 con 0x7ff2200b0120 2023-10-07T09:29:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f281c04c3c0 con 0x7f281c0c9370 2023-10-07T09:29:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d000d9c0 con 0x7f38d8157f20 2023-10-07T09:29:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d000db20 con 0x7f38d8157f20 2023-10-07T09:29:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.013+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d001f410 con 0x7f38d8157f20 2023-10-07T09:29:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 -- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d080ad370 msgr2=0x7f0d08177240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d080ad370 0x7f0d08177240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 -- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d08159ec0 msgr2=0x7f0d080ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d08159ec0 0x7f0d080ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7f0d0d99a700 1 -- 172.21.15.29:0/2058947923 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0000bb00 con 0x7f0d080af400 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591139672 0 0) 0x7f5f900c5da0 con 0x7f5f90159530 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f78003570 con 0x7f5f90159530 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22014b640 con 0x7ff2200b0120 2023-10-07T09:29:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2200c4960 con 0x7ff2200b0120 2023-10-07T09:29:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f0d0d99a700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d080af400 0x7f0d080c9f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d000090c0 tx=0x7f0d0001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f3f1effd700 1 --2- 172.21.15.29:0/2836726238 >> v2:172.21.15.29:3300/0 conn(0x7f3f200ae7f0 0x7f3f200aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1358584705 0 0) 0x7f5f78003570 con 0x7f5f90159530 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 >> v2:172.21.15.144:3300/0 conn(0x7f5f900c6940 msgr2=0x7f5f90164210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 --2- 172.21.15.29:0/3683478204 >> v2:172.21.15.144:3300/0 conn(0x7f5f900c6940 0x7f5f90164210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 >> v1:172.21.15.144:6789/0 conn(0x7f5f900d0e60 legacy=0x7f5f900c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900cc4d0 con 0x7f5f90159530 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f5f800014c0 con 0x7f5f90159530 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 >> v1:172.21.15.29:6789/0 conn(0x7f5f90159530 legacy=0x7f5f900b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f78003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f78004810 0x7f5f78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f780053c0 0x7f5f78014270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f900cca90 con 0x7f5f780053c0 2023-10-07T09:29:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78014b00 con 0x7f5f78003a60 2023-10-07T09:29:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78014cc0 con 0x7f5f78004810 2023-10-07T09:29:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d80c8e20 con 0x7f38d8157f20 2023-10-07T09:29:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d80c93e0 con 0x7f38d8157f20 2023-10-07T09:29:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.016+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcb7c011400 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7fcb817fa700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fcb70066fb0 0x7fcb70069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.017+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fcb7c084290 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7fcb817fa700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7fcb7006f1c0 0x7fcb700715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 --> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb70071c80 con 0x7fcb7006f1c0 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb7c0845c0 con 0x7fcb8c0b1bc0 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f3f1effd700 1 -- 172.21.15.29:0/2836726238 >> v2:172.21.15.144:3300/0 conn(0x7f3f200b1c90 msgr2=0x7f3f20166340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f3f1effd700 1 --2- 172.21.15.29:0/2836726238 >> v2:172.21.15.144:3300/0 conn(0x7f3f200b1c90 0x7f3f20166340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f3f1effd700 1 -- 172.21.15.29:0/2836726238 >> v1:172.21.15.144:6789/0 conn(0x7f3f200afac0 legacy=0x7f3f200b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f3f1effd700 1 -- 172.21.15.29:0/2836726238 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20167790 con 0x7f3f200ae7f0 2023-10-07T09:29:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff22014b4a0 con 0x7ff2200b0120 2023-10-07T09:29:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f38d816e100 con 0x7f38d8157f20 2023-10-07T09:29:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7f5f8ffff700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f780053c0 0x7f5f78014270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00019440 con 0x7f0d080af400 2023-10-07T09:29:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.018+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d000195a0 con 0x7f0d080af400 2023-10-07T09:29:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0000a980 con 0x7f0d080af400 2023-10-07T09:29:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f78003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7f5f8effd700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f78004810 0x7f5f78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d08177f30 con 0x7f0d080af400 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d081784f0 con 0x7f0d080af400 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f5f8f7fe700 1 -- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f78004810 msgr2=0x7f5f78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f78004810 0x7f5f78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f5f8f7fe700 1 -- 172.21.15.29:0/3683478204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f780053c0 msgr2=0x7f5f78014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f780053c0 0x7f5f78014270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f5f8f7fe700 1 -- 172.21.15.29:0/3683478204 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900cc4d0 con 0x7f5f78003a60 2023-10-07T09:29:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f5f8ffff700 1 -- 172.21.15.29:0/3683478204 reap_dead start 2023-10-07T09:29:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2804011cf0 con 0x7f281c0c9370 2023-10-07T09:29:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f78003e70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007fd0 tx=0x7f5f80002e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5b1dffbeb108b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d0804c3c0 con 0x7f0d080af400 2023-10-07T09:29:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.019+0000 7fcb9119b700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fcb70066fb0 0x7fcb70069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.020+0000 7fcb9119b700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fcb70066fb0 0x7fcb70069460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb88003080 tx=0x7fcb88017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8000cb30 con 0x7f5f78003a60 2023-10-07T09:29:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8000cc90 con 0x7f5f78003a60 2023-10-07T09:29:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7f5f8e7fc700 1 -- 172.21.15.29:0/3683478204 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80006a70 con 0x7f5f78003a60 2023-10-07T09:29:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff214022030 con 0x7ff2200b0120 2023-10-07T09:29:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7fcb9199c700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7fcb7006f1c0 0x7fcb700715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f3f1effd700 1 --2- 172.21.15.29:0/2836726238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 0x7f3f200aeff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3f2006f8b0 tx=0x7f3f180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3c0a0bec80d42c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.021+0000 7f28137fe700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f27fc067000 0x7f27fc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2804084aa0 con 0x7f281c0c9370 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f28137fe700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f27fc06f210 0x7f27fc071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 --> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc071cd0 con 0x7f27fc06f210 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2804084d90 con 0x7f281c0c9370 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f281b7fe700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f27fc067000 0x7f27fc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7f281bfff700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f27fc06f210 0x7f27fc071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f5f9499d700 1 -- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 msgr2=0x7f5f78003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f5f9499d700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f78003e70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007fd0 tx=0x7f5f80002e50 comp rx=0 tx=0).stop 2023-10-07T09:29:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7ff2077fe700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff20c066f30 0x7ff20c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7f5f9499d700 1 -- 172.21.15.29:0/3683478204 shutdown_connections 2023-10-07T09:29:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7f5f9499d700 1 --2- 172.21.15.29:0/3683478204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f78003e70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7f5f9499d700 1 -- 172.21.15.29:0/3683478204 >> 172.21.15.29:0/3683478204 conn(0x7f5f90063a10 msgr2=0x7f5f900d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff214086a70 con 0x7ff2200b0120 2023-10-07T09:29:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7f281b7fe700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f27fc067000 0x7f27fc0694b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f280c006fd0 tx=0x7f280c012000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7fcb9199c700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7fcb7006f1c0 0x7fcb700715d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.023+0000 7ff2077fe700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7ff20c06f140 0x7ff20c071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 --> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff20c071c00 con 0x7ff20c06f140 2023-10-07T09:29:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff214086d60 con 0x7ff2200b0120 2023-10-07T09:29:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.022+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0d0000ad30 con 0x7f0d080af400 2023-10-07T09:29:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7ff21f7fe700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff20c066f30 0x7ff20c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f3f1dffb700 1 -- 172.21.15.29:0/2836726238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800b740 con 0x7f3f200ae7f0 2023-10-07T09:29:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f3f1dffb700 1 -- 172.21.15.29:0/2836726238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1800b8a0 con 0x7f3f200ae7f0 2023-10-07T09:29:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f3f1dffb700 1 -- 172.21.15.29:0/2836726238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800bc40 con 0x7f3f200ae7f0 2023-10-07T09:29:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f38d0003020 con 0x7f38d8157f20 2023-10-07T09:29:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f5f9499d700 1 -- 172.21.15.29:0/3683478204 shutdown_connections 2023-10-07T09:29:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== osd.3 v2:172.21.15.29:6820/115201 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fcb70071c80 con 0x7fcb7006f1c0 2023-10-07T09:29:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f0cfe7fc700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0cec067080 0x7f0cec069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0d00085fc0 con 0x7f0d080af400 2023-10-07T09:29:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f0cfe7fc700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f0cec06f290 0x7f0cec0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 --> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cec071d50 con 0x7f0cec06f290 2023-10-07T09:29:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d00088500 con 0x7f0d080af400 2023-10-07T09:29:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7ff21ffff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7ff20c06f140 0x7ff20c071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f3f253ff700 1 -- 172.21.15.29:0/2836726238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 msgr2=0x7f3f200aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f3f253ff700 1 --2- 172.21.15.29:0/2836726238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 0x7f3f200aeff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3f2006f8b0 tx=0x7f3f180091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f0d0e19b700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0cec067080 0x7f0cec069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f3f253ff700 1 -- 172.21.15.29:0/2836726238 shutdown_connections 2023-10-07T09:29:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f3f253ff700 1 --2- 172.21.15.29:0/2836726238 >> v2:172.21.15.144:3300/0 conn(0x7f3f200b1c90 0x7f3f20166340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f3f253ff700 1 --2- 172.21.15.29:0/2836726238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 0x7f3f200aeff0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f3f253ff700 1 -- 172.21.15.29:0/2836726238 >> 172.21.15.29:0/2836726238 conn(0x7f3f200ccc30 msgr2=0x7f3f20069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f38ce7fc700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f38bc067080 0x7f38bc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.025+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f38d00862b0 con 0x7f38d8157f20 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f38ce7fc700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f38bc06f290 0x7f38bc0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 --> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38bc071d50 con 0x7f38bc06f290 2023-10-07T09:29:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38d00865e0 con 0x7f38d8157f20 2023-10-07T09:29:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7ff21ffff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7ff20c06f140 0x7ff20c071550 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f0d0e19b700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0cec067080 0x7f0cec069530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80097c0 tx=0x7f0cf8006ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f38de59b700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f38bc067080 0x7f38bc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f38ddd9a700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f38bc06f290 0x7f38bc0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.027+0000 7f0d0e99c700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f0cec06f290 0x7f0cec0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.024+0000 7f5f9499d700 1 -- 172.21.15.29:0/3683478204 wait complete. 2023-10-07T09:29:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.027+0000 7f5f9499d700 1 Processor -- start 2023-10-07T09:29:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.027+0000 7f38de59b700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f38bc067080 0x7f38bc069530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f38c8006fd0 tx=0x7f38c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.028+0000 7f0d0e99c700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f0cec06f290 0x7f0cec0716a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.028+0000 7f38ddd9a700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f38bc06f290 0x7f38bc0716a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.026+0000 7f281bfff700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f27fc06f210 0x7f27fc071620 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.027+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== osd.4 v2:172.21.15.144:6800/115417 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f27fc071cd0 con 0x7f27fc06f210 2023-10-07T09:29:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== osd.0 v2:172.21.15.29:6800/115181 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff20c071c00 con 0x7ff20c06f140 2023-10-07T09:29:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.027+0000 7f5f9499d700 1 -- start start 2023-10-07T09:29:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.028+0000 7f5f9499d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f9014b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7f5f9499d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f900c9040 0x7f5f900c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7f5f9499d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f900c7a20 0x7f5f901573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7f5f9499d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f900c7fa0 con 0x7f5f900c9040 2023-10-07T09:29:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7f5f9499d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f900c8280 con 0x7f5f78003a60 2023-10-07T09:29:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7f5f9499d700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f900c8560 con 0x7f5f900c7a20 2023-10-07T09:29:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.029+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== osd.2 v2:172.21.15.29:6816/115200 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0cec071d50 con 0x7f0cec06f290 2023-10-07T09:29:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f900c7a20 0x7f5f901573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f900c7a20 0x7f5f901573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44304/0 (socket says 172.21.15.29:44304) 2023-10-07T09:29:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 -- 172.21.15.29:0/4005027724 learned_addr learned my addr 172.21.15.29:0/4005027724 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8effd700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f900c9040 0x7f5f900c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f9014b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7ff21f7fe700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff20c066f30 0x7ff20c0693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff2200cc060 tx=0x7ff218009370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== osd.1 v2:172.21.15.29:6808/115184 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f38bc071d50 con 0x7f38bc06f290 2023-10-07T09:29:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 -- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 msgr2=0x7f5f9014b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f78003a60 0x7f5f9014b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 -- 172.21.15.29:0/4005027724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f900c9040 msgr2=0x7f5f900c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f900c9040 0x7f5f900c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.030+0000 7f5f8ffff700 1 -- 172.21.15.29:0/4005027724 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900c8840 con 0x7f5f900c7a20 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.031+0000 7f5f8ffff700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f900c7a20 0x7f5f901573a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5f900885c0 tx=0x7f5f880095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.031+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88011070 con 0x7f5f900c7a20 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.031+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8800bd70 con 0x7f5f900c7a20 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.031+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88016670 con 0x7f5f900c7a20 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.031+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f900c8b20 con 0x7f5f900c7a20 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.032+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9014b7d0 con 0x7f5f900c7a20 2023-10-07T09:29:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.033+0000 7f3f253ff700 1 -- 172.21.15.29:0/2836726238 shutdown_connections 2023-10-07T09:29:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.034+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f70000fd0 con 0x7f5f900c7a20 2023-10-07T09:29:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.034+0000 7f3f253ff700 1 -- 172.21.15.29:0/2836726238 wait complete. 2023-10-07T09:29:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.035+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5f88016a60 con 0x7f5f900c7a20 2023-10-07T09:29:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.036+0000 7f3f253ff700 1 Processor -- start 2023-10-07T09:29:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.037+0000 7f5f777fe700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5f78068b10 0x7f5f7806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.037+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5f88084c90 con 0x7f5f900c7a20 2023-10-07T09:29:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.037+0000 7f5f777fe700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7f5f78070ca0 0x7f5f780730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.037+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 --> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f78073760 con 0x7f5f78070ca0 2023-10-07T09:29:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f88051ee0 con 0x7f5f900c7a20 2023-10-07T09:29:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.037+0000 7f3f253ff700 1 -- start start 2023-10-07T09:29:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f5f8effd700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7f5f78070ca0 0x7f5f780730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5f78068b10 0x7f5f7806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f3f253ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 0x7f3f200c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f3f253ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f200afac0 0x7f3f201830b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f3f253ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f200b1c90 0x7f3f201835f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f3f253ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f20080e10 con 0x7f3f200ae7f0 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f3f253ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f20084520 con 0x7f3f200b1c90 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.038+0000 7f3f253ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200850a0 con 0x7f3f200afac0 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.039+0000 7f3f1e7fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f200afac0 0x7f3f201830b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.039+0000 7f3f1e7fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f200afac0 0x7f3f201830b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44310/0 (socket says 172.21.15.29:44310) 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.039+0000 7f3f1e7fc700 1 -- 172.21.15.29:0/3718230428 learned_addr learned my addr 172.21.15.29:0/3718230428 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.039+0000 7f3f1f7fe700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f200b1c90 0x7f3f201835f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.039+0000 7f5f8f7fe700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5f78068b10 0x7f5f7806afc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f80000c00 tx=0x7f5f80016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f5f8effd700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7f5f78070ca0 0x7f5f780730b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f1f7fe700 1 -- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f200afac0 msgr2=0x7f3f201830b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f1f7fe700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f200afac0 0x7f3f201830b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f1f7fe700 1 -- 172.21.15.29:0/3718230428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 msgr2=0x7f3f200c72d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f1f7fe700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f200ae7f0 0x7f3f200c72d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f1f7fe700 1 -- 172.21.15.29:0/3718230428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18009420 con 0x7f3f200b1c90 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f1f7fe700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f200b1c90 0x7f3f201835f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000d8c0 tx=0x7f3f10008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10018070 con 0x7f3f200b1c90 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.041+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100125f0 con 0x7f3f200b1c90 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.041+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100129b0 con 0x7f3f200b1c90 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.042+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f20084770 con 0x7f3f200b1c90 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.042+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f20084d30 con 0x7f3f200b1c90 2023-10-07T09:29:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.043+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== osd.6 v2:172.21.15.144:6824/115448 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f5f78073760 con 0x7f5f78070ca0 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7ff0ac1ff700 1 Processor -- start 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.040+0000 7ff0ac1ff700 1 -- start start 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.042+0000 7ff0ac1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0a415d8c0 0x7ff0a40afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.043+0000 7ff0ac1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a415f2a0 con 0x7ff0a40b1c90 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.043+0000 7ff0ac1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a406a3f0 con 0x7ff0a415dc90 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.043+0000 7ff0ac1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a40b0000 con 0x7ff0a415d8c0 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.043+0000 7ff0a979a700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff0a415dc90 0x7ff0a40b1580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39388/0 (socket says 172.21.15.29:39388) 2023-10-07T09:29:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.043+0000 7ff0a979a700 1 -- 172.21.15.29:0/3539803292 learned_addr learned my addr 172.21.15.29:0/3539803292 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822180550 0 0) 0x7ff0a406a3f0 con 0x7ff0a415dc90 2023-10-07T09:29:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff090003570 con 0x7ff0a415dc90 2023-10-07T09:29:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.046+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f2004c3c0 con 0x7f3f200b1c90 2023-10-07T09:29:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.048+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3f10012d20 con 0x7f3f200b1c90 2023-10-07T09:29:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2010767243 0 0) 0x7ff090003570 con 0x7ff0a415dc90 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 >> v2:172.21.15.29:3300/0 conn(0x7ff0a415d8c0 msgr2=0x7ff0a40afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 --2- 172.21.15.29:0/3539803292 >> v2:172.21.15.29:3300/0 conn(0x7ff0a415d8c0 0x7ff0a40afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 >> v1:172.21.15.29:6789/0 conn(0x7ff0a40b1c90 legacy=0x7ff0a415f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a40af3d0 con 0x7ff0a415dc90 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7ff094001470 con 0x7ff0a415dc90 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 >> v1:172.21.15.144:6789/0 conn(0x7ff0a415dc90 legacy=0x7ff0a40b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff090003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff090004810 0x7ff090004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0900053c0 0x7ff09000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a40b0000 con 0x7ff090003ae0 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09000c2c0 con 0x7ff0900053c0 2023-10-07T09:29:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.044+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09000c480 con 0x7ff090004810 2023-10-07T09:29:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.046+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff090003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.047+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/3539803292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff090004810 msgr2=0x7ff090004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.047+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff090004810 0x7ff090004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.047+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/3539803292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0900053c0 msgr2=0x7ff09000ba80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-07T09:29:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.047+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0900053c0 0x7ff09000ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.047+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/3539803292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a40af3d0 con 0x7ff090003ae0 2023-10-07T09:29:34.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.047+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff090003ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00099e0 tx=0x7ff0a0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f24e1f8c081e86 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.048+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0011420 con 0x7ff090003ae0 2023-10-07T09:29:34.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.048+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a0011580 con 0x7ff090003ae0 2023-10-07T09:29:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.048+0000 7ff0a8f99700 1 -- 172.21.15.29:0/3539803292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0011920 con 0x7ff090003ae0 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.050+0000 7f3f16ffd700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3f00067080 0x7f3f00069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.050+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3f100845b0 con 0x7f3f200b1c90 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.050+0000 7f3f16ffd700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f3f0006f290 0x7f3f000716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.050+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 --> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f00071d50 con 0x7f3f0006f290 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.050+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f10086b50 con 0x7f3f200b1c90 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.051+0000 7f3f1effd700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3f00067080 0x7f3f00069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.051+0000 7ff0aa79c700 1 -- 172.21.15.29:0/3539803292 reap_dead start 2023-10-07T09:29:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.051+0000 7f3f1e7fc700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f3f0006f290 0x7f3f000716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.052+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/3539803292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 msgr2=0x7ff090003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.052+0000 7ff0ac1ff700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff090003ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00099e0 tx=0x7ff0a0009060 comp rx=0 tx=0).stop 2023-10-07T09:29:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.052+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/3539803292 shutdown_connections 2023-10-07T09:29:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.052+0000 7ff0ac1ff700 1 --2- 172.21.15.29:0/3539803292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff090003ed0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.052+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/3539803292 >> 172.21.15.29:0/3539803292 conn(0x7ff0a40ccc30 msgr2=0x7ff0a4069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.053+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/3539803292 shutdown_connections 2023-10-07T09:29:34.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.053+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/3539803292 wait complete. 2023-10-07T09:29:34.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.055+0000 7f3f1effd700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3f00067080 0x7f3f00069530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800c010 tx=0x7f3f18017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.055+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 --> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb8c13d530 con 0x7fcb7006f1c0 2023-10-07T09:29:34.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.055+0000 7ff0ac1ff700 1 Processor -- start 2023-10-07T09:29:34.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.056+0000 7fcb817fa700 1 -- 172.21.15.29:0/3175986811 <== osd.3 v2:172.21.15.29:6820/115201 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcb8c13d530 con 0x7fcb7006f1c0 2023-10-07T09:29:34.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.056+0000 7ff0ac1ff700 1 -- start start 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7fcb7006f1c0 msgr2=0x7fcb700715d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7fcb7006f1c0 0x7fcb700715d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fcb70066fb0 msgr2=0x7fcb70069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fcb70066fb0 0x7fcb70069460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb88003080 tx=0x7fcb88017000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c0b1bc0 msgr2=0x7fcb8c138540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 --2- 172.21.15.29:0/3175986811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c0b1bc0 0x7fcb8c138540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c06d2a0 tx=0x7fcb7c0090f0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb9199c700 1 -- 172.21.15.29:0/3175986811 reap_dead start 2023-10-07T09:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff0ac1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff0a40c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff0ac1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0a4145b50 0x7ff0a4146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff0ac1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0a40ca140 0x7ff0a4148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff0ac1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a40c3770 con 0x7ff090003ae0 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff0ac1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a414a0d0 con 0x7ff0a4145b50 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff0ac1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a4148870 con 0x7ff0a40ca140 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 --> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff220151fb0 con 0x7ff20c06f140 2023-10-07T09:29:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 shutdown_connections 2023-10-07T09:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.057+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 >> 172.21.15.29:0/3175986811 conn(0x7fcb8c0680a0 msgr2=0x7fcb8c06d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff0a40c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff0a40c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56396/0 (socket says 172.21.15.29:56396) 2023-10-07T09:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/1005312949 learned_addr learned my addr 172.21.15.29:0/1005312949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7f3f1e7fc700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f3f0006f290 0x7f3f000716a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0a40ca140 msgr2=0x7ff0a4148d00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0a40ca140 0x7ff0a4148d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0a4145b50 msgr2=0x7ff0a4146c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0a4145b50 0x7ff0a4146c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.058+0000 7ff0a9f9b700 1 -- 172.21.15.29:0/1005312949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a4146110 con 0x7ff090003ae0 2023-10-07T09:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 shutdown_connections 2023-10-07T09:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 --> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d08064150 con 0x7f0cec06f290 2023-10-07T09:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7fcb933ff700 1 -- 172.21.15.29:0/3175986811 wait complete. 2023-10-07T09:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== osd.5 v2:172.21.15.144:6806/115418 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3f00071d50 con 0x7f3f0006f290 2023-10-07T09:29:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7ff0a9f9b700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff0a40c3230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff09000c290 tx=0x7ff0a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7ff2077fe700 1 -- 172.21.15.29:0/1343323998 <== osd.0 v2:172.21.15.29:6800/115181 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff220151fb0 con 0x7ff20c06f140 2023-10-07T09:29:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a001bc50 con 0x7ff090003ae0 2023-10-07T09:29:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a001bdd0 con 0x7ff090003ae0 2023-10-07T09:29:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0023660 con 0x7ff090003ae0 2023-10-07T09:29:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.059+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a41463f0 con 0x7ff090003ae0 2023-10-07T09:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.060+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a41469b0 con 0x7ff090003ae0 2023-10-07T09:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.060+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7ff20c06f140 msgr2=0x7ff20c071550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.060+0000 7ff225bff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7ff20c06f140 0x7ff20c071550 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff20c066f30 msgr2=0x7ff20c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff225bff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff20c066f30 0x7ff20c0693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff2200cc060 tx=0x7ff218009370 comp rx=0 tx=0).stop 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2200b0120 msgr2=0x7ff2200c4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff225bff700 1 --2- 172.21.15.29:0/1343323998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2200b0120 0x7ff2200c4420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff214003080 tx=0x7ff214016000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff21ffff700 1 -- 172.21.15.29:0/1343323998 reap_dead start 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 shutdown_connections 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 >> 172.21.15.29:0/1343323998 conn(0x7ff2200680a0 msgr2=0x7ff22014db30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.064+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff0a002c440 con 0x7ff090003ae0 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.065+0000 7ff099ffb700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff088067000 0x7ff0880694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.065+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff0a0085a40 con 0x7ff090003ae0 2023-10-07T09:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.066+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 shutdown_connections 2023-10-07T09:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.066+0000 7ff0a979a700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff088067000 0x7ff0880694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.066+0000 7ff0ac1ff700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7ff0a4149240 0x7ff0a414e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.066+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 --> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0a40cbf40 con 0x7ff0a4149240 2023-10-07T09:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.067+0000 7ff0aa79c700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7ff0a4149240 0x7ff0a414e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.067+0000 7ff0a979a700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff088067000 0x7ff0880694b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40c1fe0 tx=0x7ff094001d80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.067+0000 7ff225bff700 1 -- 172.21.15.29:0/1343323998 wait complete. 2023-10-07T09:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.067+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 --> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f38d804c3c0 con 0x7f38bc06f290 2023-10-07T09:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.068+0000 7f38ce7fc700 1 -- 172.21.15.29:0/505291652 <== osd.1 v2:172.21.15.29:6808/115184 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f38d804c3c0 con 0x7f38bc06f290 2023-10-07T09:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.068+0000 7ff0aa79c700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7ff0a4149240 0x7ff0a414e830 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.069+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== osd.7 v2:172.21.15.144:6816/115419 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff0a40cbf40 con 0x7ff0a4149240 2023-10-07T09:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.068+0000 7f0cfe7fc700 1 -- 172.21.15.29:0/2058947923 <== osd.2 v2:172.21.15.29:6816/115200 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0d08064150 con 0x7f0cec06f290 2023-10-07T09:29:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.068+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 --> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f281c15e860 con 0x7f27fc06f210 2023-10-07T09:29:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.069+0000 7f28137fe700 1 -- 172.21.15.29:0/3112558356 <== osd.4 v2:172.21.15.144:6800/115417 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f281c15e860 con 0x7f27fc06f210 2023-10-07T09:29:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f38bc06f290 msgr2=0x7f38bc0716a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f38bc06f290 0x7f38bc0716a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f27fc06f210 msgr2=0x7f27fc071620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f2821dff700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f27fc06f210 0x7f27fc071620 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f38bc067080 msgr2=0x7f38bc069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f38bc067080 0x7f38bc069530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f38c8006fd0 tx=0x7f38c8008040 comp rx=0 tx=0).stop 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8157f20 msgr2=0x7f38d8158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 --2- 172.21.15.29:0/505291652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8157f20 0x7f38d8158390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f38c400c2d0 tx=0x7f38d0016000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38ded9c700 1 -- 172.21.15.29:0/505291652 reap_dead start 2023-10-07T09:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 shutdown_connections 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 >> 172.21.15.29:0/505291652 conn(0x7f38d80680a0 msgr2=0x7f38d8159490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f27fc067000 msgr2=0x7f27fc0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f2821dff700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f27fc067000 0x7f27fc0694b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f280c006fd0 tx=0x7f280c012000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c0c9370 msgr2=0x7f281c159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f2821dff700 1 --2- 172.21.15.29:0/3112558356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c0c9370 0x7f281c159450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f280400d520 tx=0x7f2804009040 comp rx=0 tx=0).stop 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f281bfff700 1 -- 172.21.15.29:0/3112558356 reap_dead start 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 shutdown_connections 2023-10-07T09:29:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 >> 172.21.15.29:0/3112558356 conn(0x7f281c0680a0 msgr2=0x7f281c15f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f0cec06f290 msgr2=0x7f0cec0716a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.070+0000 7f0d103ff700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f0cec06f290 0x7f0cec0716a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0cec067080 msgr2=0x7f0cec069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d103ff700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0cec067080 0x7f0cec069530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80097c0 tx=0x7f0cf8006ce0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d080af400 msgr2=0x7f0d080c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d103ff700 1 --2- 172.21.15.29:0/2058947923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d080af400 0x7f0d080c9f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d000090c0 tx=0x7f0d0001d000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d0e99c700 1 -- 172.21.15.29:0/2058947923 reap_dead start 2023-10-07T09:29:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 shutdown_connections 2023-10-07T09:29:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.071+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 >> 172.21.15.29:0/2058947923 conn(0x7f0d080680a0 msgr2=0x7f0d08069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.072+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 shutdown_connections 2023-10-07T09:29:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.072+0000 7f38e07ff700 1 -- 172.21.15.29:0/505291652 wait complete. 2023-10-07T09:29:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.072+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 shutdown_connections 2023-10-07T09:29:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.072+0000 7f2821dff700 1 -- 172.21.15.29:0/3112558356 wait complete. 2023-10-07T09:29:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.072+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 shutdown_connections 2023-10-07T09:29:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.072+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 --> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f70002d30 con 0x7f5f78070ca0 2023-10-07T09:29:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.073+0000 7f5f777fe700 1 -- 172.21.15.29:0/4005027724 <== osd.6 v2:172.21.15.144:6824/115448 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f70002d30 con 0x7f5f78070ca0 2023-10-07T09:29:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.075+0000 7f0d103ff700 1 -- 172.21.15.29:0/2058947923 wait complete. 2023-10-07T09:29:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.076+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7f5f78070ca0 msgr2=0x7f5f780730b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.076+0000 7f5f9499d700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7f5f78070ca0 0x7f5f780730b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5f78068b10 msgr2=0x7f5f7806afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5f78068b10 0x7f5f7806afc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f80000c00 tx=0x7f5f80016000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f900c7a20 msgr2=0x7f5f901573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 --2- 172.21.15.29:0/4005027724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f900c7a20 0x7f5f901573a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5f900885c0 tx=0x7f5f880095f0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f8ffff700 1 -- 172.21.15.29:0/4005027724 reap_dead start 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 shutdown_connections 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 >> 172.21.15.29:0/4005027724 conn(0x7f5f90063a10 msgr2=0x7f5f900cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 shutdown_connections 2023-10-07T09:29:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.077+0000 7f5f9499d700 1 -- 172.21.15.29:0/4005027724 wait complete. 2023-10-07T09:29:34.146 INFO:teuthology.orchestra.run.smithi029.stdout:60129542152 2023-10-07T09:29:34.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-07T09:29:34.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.090+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 --> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3f20167a70 con 0x7f3f0006f290 2023-10-07T09:29:34.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.090+0000 7f3f16ffd700 1 -- 172.21.15.29:0/3718230428 <== osd.5 v2:172.21.15.144:6806/115418 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3f20167a70 con 0x7f3f0006f290 2023-10-07T09:29:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.091+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f3f0006f290 msgr2=0x7f3f000716a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.091+0000 7f3f253ff700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f3f0006f290 0x7f3f000716a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.091+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3f00067080 msgr2=0x7f3f00069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.091+0000 7f3f253ff700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3f00067080 0x7f3f00069530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800c010 tx=0x7f3f18017000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.091+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f200b1c90 msgr2=0x7f3f201835f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.091+0000 7f3f253ff700 1 --2- 172.21.15.29:0/3718230428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f200b1c90 0x7f3f201835f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000d8c0 tx=0x7f3f10008040 comp rx=0 tx=0).stop 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.092+0000 7f3f1f7fe700 1 -- 172.21.15.29:0/3718230428 reap_dead start 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.092+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 shutdown_connections 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.092+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 >> 172.21.15.29:0/3718230428 conn(0x7f3f200ccc30 msgr2=0x7f3f20168270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.092+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 shutdown_connections 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.092+0000 7f3f253ff700 1 -- 172.21.15.29:0/3718230428 wait complete. 2023-10-07T09:29:34.149 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-10-07T09:29:34.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-07T09:29:34.150 INFO:teuthology.orchestra.run.smithi029.stdout:64424509448 2023-10-07T09:29:34.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-07T09:29:34.151 INFO:teuthology.orchestra.run.smithi029.stdout:60129542151 2023-10-07T09:29:34.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-07T09:29:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.107+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 --> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff0a406f8e0 con 0x7ff0a4149240 2023-10-07T09:29:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.107+0000 7ff099ffb700 1 -- 172.21.15.29:0/1005312949 <== osd.7 v2:172.21.15.144:6816/115419 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff0a406f8e0 con 0x7ff0a4149240 2023-10-07T09:29:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7ff0a4149240 msgr2=0x7ff0a414e830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7ff0a4149240 0x7ff0a414e830 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff088067000 msgr2=0x7ff0880694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff088067000 0x7ff0880694b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40c1fe0 tx=0x7ff094001d80 comp rx=0 tx=0).stop 2023-10-07T09:29:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 msgr2=0x7ff0a40c3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 --2- 172.21.15.29:0/1005312949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff090003ae0 0x7ff0a40c3230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff09000c290 tx=0x7ff0a0017000 comp rx=0 tx=0).stop 2023-10-07T09:29:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0aa79c700 1 -- 172.21.15.29:0/1005312949 reap_dead start 2023-10-07T09:29:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 shutdown_connections 2023-10-07T09:29:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.108+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 >> 172.21.15.29:0/1005312949 conn(0x7ff0a40ccc30 msgr2=0x7ff0a414e120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.154 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-10-07T09:29:34.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-07T09:29:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.109+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 shutdown_connections 2023-10-07T09:29:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.109+0000 7ff0ac1ff700 1 -- 172.21.15.29:0/1005312949 wait complete. 2023-10-07T09:29:34.156 INFO:teuthology.orchestra.run.smithi029.stdout:64424509448 2023-10-07T09:29:34.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-07T09:29:34.157 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-10-07T09:29:34.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-07T09:29:34.158 INFO:teuthology.orchestra.run.smithi029.stdout:68719476744 2023-10-07T09:29:34.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-07T09:29:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.529+0000 7f48b35ff700 1 Processor -- start 2023-10-07T09:29:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.530+0000 7f48b35ff700 1 -- start start 2023-10-07T09:29:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.532+0000 7f48b35ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f48ac0cc870 0x7f48ac0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.532+0000 7f48b35ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f48ac0b1bc0 0x7f48ac15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.532+0000 7f48b35ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac15eeb0 con 0x7f48ac0b17f0 2023-10-07T09:29:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.532+0000 7f48b35ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac0aebe0 con 0x7f48ac0cc870 2023-10-07T09:29:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.532+0000 7f48b35ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac15e2a0 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.534+0000 7f48b1b9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f48ac0b1bc0 0x7f48ac15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.534+0000 7f48b0b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f48ac0cc870 0x7f48ac0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.534+0000 7f48b0b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f48ac0cc870 0x7f48ac0afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56398/0 (socket says 172.21.15.29:56398) 2023-10-07T09:29:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.534+0000 7f48b0b9a700 1 -- 172.21.15.29:0/844016199 learned_addr learned my addr 172.21.15.29:0/844016199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.535+0000 7f48b0b9a700 1 -- 172.21.15.29:0/844016199 >> v2:172.21.15.144:3300/0 conn(0x7f48ac0b1bc0 msgr2=0x7f48ac15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.535+0000 7f48b0b9a700 1 --2- 172.21.15.29:0/844016199 >> v2:172.21.15.144:3300/0 conn(0x7f48ac0b1bc0 0x7f48ac15dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.535+0000 7f48b0b9a700 1 -- 172.21.15.29:0/844016199 >> v1:172.21.15.144:6789/0 conn(0x7f48ac0b17f0 legacy=0x7f48ac15d500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.535+0000 7f48b0b9a700 1 -- 172.21.15.29:0/844016199 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac15e580 con 0x7f48ac0cc870 2023-10-07T09:29:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.536+0000 7f48b0b9a700 1 --2- 172.21.15.29:0/844016199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac0afcb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0ccfd0 tx=0x7f48a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9439c7ad69e3e43d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.537+0000 7f48a3fff700 1 -- 172.21.15.29:0/844016199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a400e070 con 0x7f48ac0cc870 2023-10-07T09:29:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.537+0000 7f48a3fff700 1 -- 172.21.15.29:0/844016199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a400ba10 con 0x7f48ac0cc870 2023-10-07T09:29:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.537+0000 7f48a3fff700 1 -- 172.21.15.29:0/844016199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40092e0 con 0x7f48ac0cc870 2023-10-07T09:29:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.538+0000 7f48b35ff700 1 -- 172.21.15.29:0/844016199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 msgr2=0x7f48ac0afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.538+0000 7f48b35ff700 1 --2- 172.21.15.29:0/844016199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac0afcb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0ccfd0 tx=0x7f48a4009840 comp rx=0 tx=0).stop 2023-10-07T09:29:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.539+0000 7f48b35ff700 1 -- 172.21.15.29:0/844016199 shutdown_connections 2023-10-07T09:29:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.539+0000 7f48b35ff700 1 --2- 172.21.15.29:0/844016199 >> v2:172.21.15.144:3300/0 conn(0x7f48ac0b1bc0 0x7f48ac15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.539+0000 7f48b35ff700 1 --2- 172.21.15.29:0/844016199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac0afcb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.539+0000 7f48b35ff700 1 -- 172.21.15.29:0/844016199 >> 172.21.15.29:0/844016199 conn(0x7f48ac0680a0 msgr2=0x7f48ac0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.539+0000 7f48b35ff700 1 -- 172.21.15.29:0/844016199 shutdown_connections 2023-10-07T09:29:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.539+0000 7f48b35ff700 1 -- 172.21.15.29:0/844016199 wait complete. 2023-10-07T09:29:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.542+0000 7f48b35ff700 1 Processor -- start 2023-10-07T09:29:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.542+0000 7f48b35ff700 1 -- start start 2023-10-07T09:29:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.544+0000 7f48b35ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48ac0b17f0 0x7f48ac15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.544+0000 7f48b35ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 0x7f48ac130d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.544+0000 7f48b35ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac131e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.544+0000 7f48b35ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac1330d0 con 0x7f48ac0cc870 2023-10-07T09:29:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.544+0000 7f48b35ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac131280 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.544+0000 7f48b35ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac1319d0 con 0x7f48ac0b17f0 2023-10-07T09:29:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.545+0000 7f48b0b9a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 0x7f48ac130d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.545+0000 7f48b0b9a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 0x7f48ac130d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52826/0 (socket says 172.21.15.29:52826) 2023-10-07T09:29:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.545+0000 7f48b0b9a700 1 -- 172.21.15.29:0/917338572 learned_addr learned my addr 172.21.15.29:0/917338572 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.545+0000 7f48b1b9c700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac131e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.546+0000 7f48b0b9a700 1 -- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48ac0b17f0 msgr2=0x7f48ac15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.546+0000 7f48b0b9a700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48ac0b17f0 0x7f48ac15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.546+0000 7f48b0b9a700 1 -- 172.21.15.29:0/917338572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 msgr2=0x7f48ac131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.546+0000 7f48b0b9a700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac131e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.546+0000 7f48b0b9a700 1 -- 172.21.15.29:0/917338572 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a4009ae0 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.547+0000 7f48b0b9a700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 0x7f48ac130d40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009fd0 tx=0x7f48a4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.548+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4012850 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.548+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a40129b0 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.548+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4012d50 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.548+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac132b10 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.548+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac131660 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.550+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c0052b0 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.550+0000 7f681cdff700 1 Processor -- start 2023-10-07T09:29:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.553+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f48a401e640 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.552+0000 7f681cdff700 1 -- start start 2023-10-07T09:29:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f681cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 0x7f68180b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f681cdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68181558e0 0x7f68180c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f681cdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68181572e0 con 0x7f68180afa70 2023-10-07T09:29:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f681cdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68181567c0 con 0x7f68181558e0 2023-10-07T09:29:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f681cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68181569b0 con 0x7f6818155510 2023-10-07T09:29:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.555+0000 7f681659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 0x7f68180b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.555+0000 7f681659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 0x7f68180b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44342/0 (socket says 172.21.15.29:44342) 2023-10-07T09:29:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.555+0000 7f681659c700 1 -- 172.21.15.29:0/532709750 learned_addr learned my addr 172.21.15.29:0/532709750 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f48a17fa700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4890067080 0x7f4890069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.554+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f48a4086830 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.555+0000 7f48b139b700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4890067080 0x7f4890069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.556+0000 7f681659c700 1 -- 172.21.15.29:0/532709750 >> v2:172.21.15.29:3300/0 conn(0x7f68181558e0 msgr2=0x7f68180c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.556+0000 7f681659c700 1 --2- 172.21.15.29:0/532709750 >> v2:172.21.15.29:3300/0 conn(0x7f68181558e0 0x7f68180c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.556+0000 7f681659c700 1 -- 172.21.15.29:0/532709750 >> v1:172.21.15.144:6789/0 conn(0x7f68180afa70 legacy=0x7f6818159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.556+0000 7f681659c700 1 -- 172.21.15.29:0/532709750 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818156c90 con 0x7f6818155510 2023-10-07T09:29:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.557+0000 7f681659c700 1 --2- 172.21.15.29:0/532709750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 0x7f68180b1a10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68080089b0 tx=0x7f68080082d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f8d2be45c460629 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.557+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48a4053b00 con 0x7f48ac0b1bc0 2023-10-07T09:29:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.558+0000 7f681559a700 1 -- 172.21.15.29:0/532709750 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808010070 con 0x7f6818155510 2023-10-07T09:29:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.558+0000 7f681559a700 1 -- 172.21.15.29:0/532709750 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680800db80 con 0x7f6818155510 2023-10-07T09:29:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.558+0000 7f681559a700 1 -- 172.21.15.29:0/532709750 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808016440 con 0x7f6818155510 2023-10-07T09:29:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.559+0000 7f681cdff700 1 -- 172.21.15.29:0/532709750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 msgr2=0x7f68180b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.559+0000 7f681cdff700 1 --2- 172.21.15.29:0/532709750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 0x7f68180b1a10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68080089b0 tx=0x7f68080082d0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.559+0000 7f681cdff700 1 -- 172.21.15.29:0/532709750 shutdown_connections 2023-10-07T09:29:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.559+0000 7f681cdff700 1 --2- 172.21.15.29:0/532709750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6818155510 0x7f68180b1a10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.559+0000 7f681cdff700 1 --2- 172.21.15.29:0/532709750 >> v2:172.21.15.29:3300/0 conn(0x7f68181558e0 0x7f68180c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.559+0000 7f681cdff700 1 -- 172.21.15.29:0/532709750 >> 172.21.15.29:0/532709750 conn(0x7f68180ccc30 msgr2=0x7f6818069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.560+0000 7f681cdff700 1 -- 172.21.15.29:0/532709750 shutdown_connections 2023-10-07T09:29:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.561+0000 7f196e3ff700 1 Processor -- start 2023-10-07T09:29:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.560+0000 7f681cdff700 1 -- 172.21.15.29:0/532709750 wait complete. 2023-10-07T09:29:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.561+0000 7f48b139b700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4890067080 0x7f4890069530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f48ac14d380 tx=0x7f489c006ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.562+0000 7f681cdff700 1 Processor -- start 2023-10-07T09:29:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.563+0000 7f681cdff700 1 -- start start 2023-10-07T09:29:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.561+0000 7f196e3ff700 1 -- start start 2023-10-07T09:29:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f196e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 0x7f19680afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f196e3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196815eeb0 con 0x7f19680b1bc0 2023-10-07T09:29:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f196e3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680aebe0 con 0x7f19680b17f0 2023-10-07T09:29:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f196e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196815e2a0 con 0x7f19680cc870 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f681cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68180afa70 0x7f6818164820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f681cdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6818155510 0x7f6818163070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f681cdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68181558e0 0x7f68180caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f681cdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6818164390 con 0x7f6818155510 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f681cdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cb440 con 0x7f68181558e0 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.564+0000 7f681cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6818163780 con 0x7f68180afa70 2023-10-07T09:29:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.565+0000 7f19677fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 0x7f19680afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.565+0000 7f19677fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 0x7f19680afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44370/0 (socket says 172.21.15.29:44370) 2023-10-07T09:29:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.565+0000 7f19677fe700 1 -- 172.21.15.29:0/383845992 learned_addr learned my addr 172.21.15.29:0/383845992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.565+0000 7f196c99c700 1 --1- 172.21.15.29:0/383845992 >> v1:172.21.15.29:6789/0 conn(0x7f19680b1bc0 0x7f196815f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58648/0 (socket says 172.21.15.29:58648) 2023-10-07T09:29:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.566+0000 7f6816d9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68181558e0 0x7f68180caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.566+0000 7f6816d9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68181558e0 0x7f68180caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52842/0 (socket says 172.21.15.29:52842) 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.566+0000 7f6816d9d700 1 -- 172.21.15.29:0/39448496 learned_addr learned my addr 172.21.15.29:0/39448496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.566+0000 7f681659c700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68180afa70 0x7f6818164820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.566+0000 7f6815d9b700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6818155510 0x7f6818163070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f681659c700 1 -- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68181558e0 msgr2=0x7f68180caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f681659c700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68181558e0 0x7f68180caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f681659c700 1 -- 172.21.15.29:0/39448496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6818155510 msgr2=0x7f6818163070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f681659c700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6818155510 0x7f6818163070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f681659c700 1 -- 172.21.15.29:0/39448496 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6808008570 con 0x7f68180afa70 2023-10-07T09:29:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f83b29ff700 1 Processor -- start 2023-10-07T09:29:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f1966ffd700 1 -- 172.21.15.29:0/383845992 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497864252 0 0) 0x7f196815eeb0 con 0x7f19680b1bc0 2023-10-07T09:29:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f1966ffd700 1 -- 172.21.15.29:0/383845992 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1954003570 con 0x7f19680b1bc0 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f681659c700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68180afa70 0x7f6818164820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f681815a3e0 tx=0x7f680801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f19677fe700 1 -- 172.21.15.29:0/383845992 >> v1:172.21.15.144:6789/0 conn(0x7f19680b17f0 legacy=0x7f196815d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f19677fe700 1 -- 172.21.15.29:0/383845992 >> v1:172.21.15.29:6789/0 conn(0x7f19680b1bc0 legacy=0x7f196815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f19677fe700 1 -- 172.21.15.29:0/383845992 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196815e580 con 0x7f19680cc870 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808016c50 con 0x7f68180afa70 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6808016dd0 con 0x7f68180afa70 2023-10-07T09:29:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680800b310 con 0x7f68180afa70 2023-10-07T09:29:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.568+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6818163a60 con 0x7f68180afa70 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.569+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6818164020 con 0x7f68180afa70 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.570+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6818164670 con 0x7f68180afa70 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f680801d040 con 0x7f68180afa70 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f68067fc700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f67f8067080 0x7f67f8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6808084c40 con 0x7f68180afa70 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f6815d9b700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f67f8067080 0x7f67f8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f6815d9b700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f67f8067080 0x7f67f8069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f68180c8970 tx=0x7f6810009780 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.567+0000 7f83b29ff700 1 -- start start 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.569+0000 7f83b29ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b17f0 0x7f83ac0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.569+0000 7f83b29ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f83ac0b1bc0 0x7f83ac15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.569+0000 7f83b29ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac15eeb0 con 0x7f83ac0cc870 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.569+0000 7f83b29ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac0aebe0 con 0x7f83ac0b1bc0 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.569+0000 7f83b29ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac15e2a0 con 0x7f83ac0b17f0 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.570+0000 7f83ab7fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f83ac0cc870 0x7f83ac15d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39428/0 (socket says 172.21.15.29:39428) 2023-10-07T09:29:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.570+0000 7f83ab7fe700 1 -- 172.21.15.29:0/762662279 learned_addr learned my addr 172.21.15.29:0/762662279 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.570+0000 7f83b0f9c700 1 --2- 172.21.15.29:0/762662279 >> v2:172.21.15.144:3300/0 conn(0x7f83ac0b1bc0 0x7f83ac15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.570+0000 7f83abfff700 1 --2- 172.21.15.29:0/762662279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b17f0 0x7f83ac0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f83aaffd700 1 -- 172.21.15.29:0/762662279 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566894802 0 0) 0x7f83ac15eeb0 con 0x7f83ac0cc870 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f83aaffd700 1 -- 172.21.15.29:0/762662279 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8394003570 con 0x7f83ac0cc870 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f83abfff700 1 -- 172.21.15.29:0/762662279 >> v2:172.21.15.144:3300/0 conn(0x7f83ac0b1bc0 msgr2=0x7f83ac15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f83abfff700 1 --2- 172.21.15.29:0/762662279 >> v2:172.21.15.144:3300/0 conn(0x7f83ac0b1bc0 0x7f83ac15dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f83abfff700 1 -- 172.21.15.29:0/762662279 >> v1:172.21.15.144:6789/0 conn(0x7f83ac0cc870 legacy=0x7f83ac15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f83abfff700 1 -- 172.21.15.29:0/762662279 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac15e580 con 0x7f83ac0b17f0 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f83abfff700 1 --2- 172.21.15.29:0/762662279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b17f0 0x7f83ac0afcb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009910 tx=0x7f83a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a3e654463dca60e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f83aaffd700 1 -- 172.21.15.29:0/762662279 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0011630 con 0x7f83ac0b17f0 2023-10-07T09:29:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f83aaffd700 1 -- 172.21.15.29:0/762662279 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0011790 con 0x7f83ac0b17f0 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f83aaffd700 1 -- 172.21.15.29:0/762662279 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0011b50 con 0x7f83ac0b17f0 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.573+0000 7f83b29ff700 1 -- 172.21.15.29:0/762662279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b17f0 msgr2=0x7f83ac0afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.573+0000 7f83b29ff700 1 --2- 172.21.15.29:0/762662279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b17f0 0x7f83ac0afcb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009910 tx=0x7f83a00091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f83b29ff700 1 -- 172.21.15.29:0/762662279 shutdown_connections 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f83b29ff700 1 --2- 172.21.15.29:0/762662279 >> v2:172.21.15.144:3300/0 conn(0x7f83ac0b1bc0 0x7f83ac15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f83b29ff700 1 --2- 172.21.15.29:0/762662279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b17f0 0x7f83ac0afcb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f83b29ff700 1 -- 172.21.15.29:0/762662279 >> 172.21.15.29:0/762662279 conn(0x7f83ac0680a0 msgr2=0x7f83ac0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f83b29ff700 1 -- 172.21.15.29:0/762662279 shutdown_connections 2023-10-07T09:29:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f83b29ff700 1 -- 172.21.15.29:0/762662279 wait complete. 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.571+0000 7f07439ff700 1 Processor -- start 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f07439ff700 1 -- start start 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f07439ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 0x7f073c0b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f07439ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f073c1558e0 0x7f073c0c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f07439ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f073c0afa70 0x7f073c0afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f07439ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c1567c0 con 0x7f073c0afa70 2023-10-07T09:29:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f07439ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c0b0430 con 0x7f073c1558e0 2023-10-07T09:29:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f07439ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c1572e0 con 0x7f073c155510 2023-10-07T09:29:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f074179b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 0x7f073c0b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f074179b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 0x7f073c0b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44392/0 (socket says 172.21.15.29:44392) 2023-10-07T09:29:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f074179b700 1 -- 172.21.15.29:0/2622013000 learned_addr learned my addr 172.21.15.29:0/2622013000 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f074179b700 1 -- 172.21.15.29:0/2622013000 >> v2:172.21.15.144:3300/0 conn(0x7f073c1558e0 msgr2=0x7f073c0c81f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f0740f9a700 1 --2- 172.21.15.29:0/2622013000 >> v2:172.21.15.144:3300/0 conn(0x7f073c1558e0 0x7f073c0c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f5477fff700 1 Processor -- start 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f5477fff700 1 -- start start 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f5477fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5470156980 0x7f54700b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.574+0000 7f5477fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f547006b590 0x7f54700afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f5477fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5470157bd0 con 0x7f54700b0040 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f5477fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547015c5d0 con 0x7f547006b590 2023-10-07T09:29:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f5477fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54700b0430 con 0x7f5470156980 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f5475d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5470156980 0x7f54700b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f5475d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5470156980 0x7f54700b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52862/0 (socket says 172.21.15.29:52862) 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.575+0000 7f5475d9b700 1 -- 172.21.15.29:0/3494149182 learned_addr learned my addr 172.21.15.29:0/3494149182 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f5475d9b700 1 -- 172.21.15.29:0/3494149182 >> v2:172.21.15.29:3300/0 conn(0x7f547006b590 msgr2=0x7f54700afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f5475d9b700 1 --2- 172.21.15.29:0/3494149182 >> v2:172.21.15.29:3300/0 conn(0x7f547006b590 0x7f54700afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f5475d9b700 1 -- 172.21.15.29:0/3494149182 >> v1:172.21.15.29:6789/0 conn(0x7f54700b0040 legacy=0x7f5470152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f5475d9b700 1 -- 172.21.15.29:0/3494149182 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54701572a0 con 0x7f5470156980 2023-10-07T09:29:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f5475d9b700 1 --2- 172.21.15.29:0/3494149182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5470156980 0x7f54700b1a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5470063cd0 tx=0x7f546c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccef68debfe18e84 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f074179b700 1 --2- 172.21.15.29:0/2622013000 >> v2:172.21.15.144:3300/0 conn(0x7f073c1558e0 0x7f073c0c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f074179b700 1 -- 172.21.15.29:0/2622013000 >> v2:172.21.15.29:3300/0 conn(0x7f073c0afa70 msgr2=0x7f073c0afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f074179b700 1 --2- 172.21.15.29:0/2622013000 >> v2:172.21.15.29:3300/0 conn(0x7f073c0afa70 0x7f073c0afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f074179b700 1 -- 172.21.15.29:0/2622013000 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c1569b0 con 0x7f073c155510 2023-10-07T09:29:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.576+0000 7f0740f9a700 1 --2- 172.21.15.29:0/2622013000 >> v2:172.21.15.144:3300/0 conn(0x7f073c1558e0 0x7f073c0c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f291d3ff700 1 Processor -- start 2023-10-07T09:29:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f074179b700 1 --2- 172.21.15.29:0/2622013000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 0x7f073c0b1a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f073c06ceb0 tx=0x7f0738009ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffc54862e3156e1c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.572+0000 7f19677fe700 1 --2- 172.21.15.29:0/383845992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 0x7f19680afcb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1958009990 tx=0x7f1958009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cb882fa185d1f8f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f1966ffd700 1 -- 172.21.15.29:0/383845992 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600017f0 con 0x7f19680cc870 2023-10-07T09:29:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f1966ffd700 1 -- 172.21.15.29:0/383845992 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1958009e80 con 0x7f19680cc870 2023-10-07T09:29:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f1966ffd700 1 -- 172.21.15.29:0/383845992 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800b7a0 con 0x7f19680cc870 2023-10-07T09:29:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f196e3ff700 1 -- 172.21.15.29:0/383845992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 msgr2=0x7f19680afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f196e3ff700 1 --2- 172.21.15.29:0/383845992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 0x7f19680afcb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1958009990 tx=0x7f1958009250 comp rx=0 tx=0).stop 2023-10-07T09:29:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f83b29ff700 1 Processor -- start 2023-10-07T09:29:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f5474d99700 1 -- 172.21.15.29:0/3494149182 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00b8b0 con 0x7f5470156980 2023-10-07T09:29:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f5474d99700 1 -- 172.21.15.29:0/3494149182 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00ba10 con 0x7f5470156980 2023-10-07T09:29:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f5474d99700 1 -- 172.21.15.29:0/3494149182 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00bdb0 con 0x7f5470156980 2023-10-07T09:29:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.577+0000 7f291d3ff700 1 -- start start 2023-10-07T09:29:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f5477fff700 1 -- 172.21.15.29:0/3494149182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5470156980 msgr2=0x7f54700b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f5477fff700 1 --2- 172.21.15.29:0/3494149182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5470156980 0x7f54700b1a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5470063cd0 tx=0x7f546c009250 comp rx=0 tx=0).stop 2023-10-07T09:29:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f5477fff700 1 -- 172.21.15.29:0/3494149182 shutdown_connections 2023-10-07T09:29:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f5477fff700 1 --2- 172.21.15.29:0/3494149182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5470156980 0x7f54700b1a10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f5477fff700 1 --2- 172.21.15.29:0/3494149182 >> v2:172.21.15.29:3300/0 conn(0x7f547006b590 0x7f54700afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f5477fff700 1 -- 172.21.15.29:0/3494149182 >> 172.21.15.29:0/3494149182 conn(0x7f54700680a0 msgr2=0x7f54700cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f291d3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2918155510 0x7f29180b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f291d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291806bae0 0x7f2918157770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f291d3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29181572e0 con 0x7f29181558e0 2023-10-07T09:29:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f291d3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29181567c0 con 0x7f2918155510 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f291d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29181569b0 con 0x7f291806bae0 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f29167fc700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f29181558e0 0x7f29180c81f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39432/0 (socket says 172.21.15.29:39432) 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f29167fc700 1 -- 172.21.15.29:0/2122214577 learned_addr learned my addr 172.21.15.29:0/2122214577 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f29177fe700 1 --2- 172.21.15.29:0/2122214577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291806bae0 0x7f2918157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f0733fff700 1 -- 172.21.15.29:0/2622013000 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073800ca00 con 0x7f073c155510 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f0733fff700 1 -- 172.21.15.29:0/2622013000 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073800cb60 con 0x7f073c155510 2023-10-07T09:29:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f0733fff700 1 -- 172.21.15.29:0/2622013000 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738018400 con 0x7f073c155510 2023-10-07T09:29:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.578+0000 7f83b29ff700 1 -- start start 2023-10-07T09:29:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f83b29ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83ac0b17f0 0x7f83ac13af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f5477fff700 1 -- 172.21.15.29:0/3494149182 shutdown_connections 2023-10-07T09:29:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f5477fff700 1 -- 172.21.15.29:0/3494149182 wait complete. 2023-10-07T09:29:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f83b29ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b1bc0 0x7f83ac1384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f83b29ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83ac0cc870 0x7f83ac1389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f83b29ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac164040 con 0x7f83ac0b17f0 2023-10-07T09:29:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f83b29ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac160a70 con 0x7f83ac0cc870 2023-10-07T09:29:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.579+0000 7f83b29ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac13b4c0 con 0x7f83ac0b1bc0 2023-10-07T09:29:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6808051e90 con 0x7f68180afa70 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f83ab7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b1bc0 0x7f83ac1384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f83ab7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b1bc0 0x7f83ac1384b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44406/0 (socket says 172.21.15.29:44406) 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f83ab7fe700 1 -- 172.21.15.29:0/631763933 learned_addr learned my addr 172.21.15.29:0/631763933 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f83b0f9c700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83ac0cc870 0x7f83ac1389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f29177fe700 1 -- 172.21.15.29:0/2122214577 >> v2:172.21.15.144:3300/0 conn(0x7f2918155510 msgr2=0x7f29180b1a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f29177fe700 1 --2- 172.21.15.29:0/2122214577 >> v2:172.21.15.144:3300/0 conn(0x7f2918155510 0x7f29180b1a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f29177fe700 1 -- 172.21.15.29:0/2122214577 >> v1:172.21.15.144:6789/0 conn(0x7f29181558e0 legacy=0x7f29180c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f29177fe700 1 -- 172.21.15.29:0/2122214577 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2918156c90 con 0x7f291806bae0 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 -- 172.21.15.29:0/2622013000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 msgr2=0x7f073c0b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 --2- 172.21.15.29:0/2622013000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 0x7f073c0b1a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f073c06ceb0 tx=0x7f0738009ae0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 -- 172.21.15.29:0/2622013000 shutdown_connections 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 --2- 172.21.15.29:0/2622013000 >> v2:172.21.15.29:3300/0 conn(0x7f073c0afa70 0x7f073c0afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 --2- 172.21.15.29:0/2622013000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c155510 0x7f073c0b1a10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 --2- 172.21.15.29:0/2622013000 >> v2:172.21.15.144:3300/0 conn(0x7f073c1558e0 0x7f073c0c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 -- 172.21.15.29:0/2622013000 >> 172.21.15.29:0/2622013000 conn(0x7f073c0ccc30 msgr2=0x7f073c069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.580+0000 7f07439ff700 1 -- 172.21.15.29:0/2622013000 shutdown_connections 2023-10-07T09:29:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f29177fe700 1 --2- 172.21.15.29:0/2122214577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291806bae0 0x7f2918157770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f290c0098e0 tx=0x7f290c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5918586136785e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f2915ffb700 1 -- 172.21.15.29:0/2122214577 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c009e80 con 0x7f291806bae0 2023-10-07T09:29:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f2915ffb700 1 -- 172.21.15.29:0/2122214577 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c00b8b0 con 0x7f291806bae0 2023-10-07T09:29:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f2915ffb700 1 -- 172.21.15.29:0/2122214577 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c00bc70 con 0x7f291806bae0 2023-10-07T09:29:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f07439ff700 1 -- 172.21.15.29:0/2622013000 wait complete. 2023-10-07T09:29:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f196e3ff700 1 -- 172.21.15.29:0/383845992 shutdown_connections 2023-10-07T09:29:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f196e3ff700 1 --2- 172.21.15.29:0/383845992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680cc870 0x7f19680afcb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f196e3ff700 1 -- 172.21.15.29:0/383845992 >> 172.21.15.29:0/383845992 conn(0x7f19680680a0 msgr2=0x7f19680cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f5477fff700 1 Processor -- start 2023-10-07T09:29:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f5477fff700 1 -- start start 2023-10-07T09:29:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f291d3ff700 1 -- 172.21.15.29:0/2122214577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291806bae0 msgr2=0x7f2918157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f291d3ff700 1 --2- 172.21.15.29:0/2122214577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291806bae0 0x7f2918157770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f290c0098e0 tx=0x7f290c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f07439ff700 1 Processor -- start 2023-10-07T09:29:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f291d3ff700 1 -- 172.21.15.29:0/2122214577 shutdown_connections 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f291d3ff700 1 --2- 172.21.15.29:0/2122214577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291806bae0 0x7f2918157770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f291d3ff700 1 --2- 172.21.15.29:0/2122214577 >> v2:172.21.15.144:3300/0 conn(0x7f2918155510 0x7f29180b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f291d3ff700 1 -- 172.21.15.29:0/2122214577 >> 172.21.15.29:0/2122214577 conn(0x7f29180ccc30 msgr2=0x7f2918069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f5477fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f547006b590 0x7f5470144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f5477fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 0x7f5470158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f5477fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5470156980 0x7f54701592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f5477fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5470158060 con 0x7f54700b0040 2023-10-07T09:29:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f5477fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5470158900 con 0x7f547006b590 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f5477fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547013ef40 con 0x7f5470156980 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f83b0f9c700 1 -- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b1bc0 msgr2=0x7f83ac1384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f83b0f9c700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b1bc0 0x7f83ac1384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f83b0f9c700 1 -- 172.21.15.29:0/631763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83ac0b17f0 msgr2=0x7f83ac13af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f83b0f9c700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83ac0b17f0 0x7f83ac13af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f83b0f9c700 1 -- 172.21.15.29:0/631763933 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a0009420 con 0x7f83ac0cc870 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.581+0000 7f83b0f9c700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83ac0cc870 0x7f83ac1389f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f839c00e950 tx=0x7f839c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c00c6a0 con 0x7f83ac0cc870 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c00c840 con 0x7f83ac0cc870 2023-10-07T09:29:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.582+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c00cc00 con 0x7f83ac0cc870 2023-10-07T09:29:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f547559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 0x7f5470158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f547559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 0x7f5470158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56452/0 (socket says 172.21.15.29:56452) 2023-10-07T09:29:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f547559a700 1 -- 172.21.15.29:0/2519824118 learned_addr learned my addr 172.21.15.29:0/2519824118 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac1634c0 con 0x7f83ac0cc870 2023-10-07T09:29:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac163a80 con 0x7f83ac0cc870 2023-10-07T09:29:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f196e3ff700 1 -- 172.21.15.29:0/383845992 shutdown_connections 2023-10-07T09:29:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f196e3ff700 1 -- 172.21.15.29:0/383845992 wait complete. 2023-10-07T09:29:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.583+0000 7f07439ff700 1 -- start start 2023-10-07T09:29:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f07439ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 0x7f073c147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f07439ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c155510 0x7f073c152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f07439ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c1558e0 0x7f073c153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f07439ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15a750 con 0x7f073c155510 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f07439ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15e1b0 con 0x7f073c0afa70 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.584+0000 7f07439ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15e490 con 0x7f073c1558e0 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f074179b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 0x7f073c147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f074179b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 0x7f073c147fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52902/0 (socket says 172.21.15.29:52902) 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f074179b700 1 -- 172.21.15.29:0/3001644606 learned_addr learned my addr 172.21.15.29:0/3001644606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f547559a700 1 -- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5470156980 msgr2=0x7f54701592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f547559a700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5470156980 0x7f54701592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f547559a700 1 -- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f547006b590 msgr2=0x7f5470144480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f547559a700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f547006b590 0x7f5470144480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f547559a700 1 -- 172.21.15.29:0/2519824118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c0094f0 con 0x7f54700b0040 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.585+0000 7f0741f9c700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c1558e0 0x7f073c153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f074179b700 1 -- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c1558e0 msgr2=0x7f073c153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f074179b700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c1558e0 0x7f073c153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f074179b700 1 -- 172.21.15.29:0/3001644606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c155510 msgr2=0x7f073c152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f074179b700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c155510 0x7f073c152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f074179b700 1 -- 172.21.15.29:0/3001644606 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0738009d80 con 0x7f073c0afa70 2023-10-07T09:29:34.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f196e3ff700 1 Processor -- start 2023-10-07T09:29:34.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f196e3ff700 1 -- start start 2023-10-07T09:29:34.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac04c3c0 con 0x7f83ac0cc870 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f074179b700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 0x7f073c147fe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0738009b20 tx=0x7f0738018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.586+0000 7f547559a700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 0x7f5470158d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f546000b840 tx=0x7f5460009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546000f570 con 0x7f54700b0040 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546000f6f0 con 0x7f54700b0040 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546000fab0 con 0x7f54700b0040 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5470141e40 con 0x7f54700b0040 2023-10-07T09:29:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5470158590 con 0x7f54700b0040 2023-10-07T09:29:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738010600 con 0x7f073c0afa70 2023-10-07T09:29:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073800c040 con 0x7f073c0afa70 2023-10-07T09:29:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.587+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380109b0 con 0x7f073c0afa70 2023-10-07T09:29:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f196e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680b17f0 0x7f196816ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f196e3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 0x7f1968160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f196e3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19680cc870 0x7f1968163810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f196e3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19681596e0 con 0x7f19680b1bc0 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f196e3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19681615e0 con 0x7f19680cc870 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f196e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19681618c0 con 0x7f19680b17f0 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.589+0000 7f19677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 0x7f1968160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.589+0000 7f19677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 0x7f1968160b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56470/0 (socket says 172.21.15.29:56470) 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.589+0000 7f19677fe700 1 -- 172.21.15.29:0/1889694644 learned_addr learned my addr 172.21.15.29:0/1889694644 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.588+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f839c013470 con 0x7f83ac0cc870 2023-10-07T09:29:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f1967fff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680b17f0 0x7f196816ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f19677fe700 1 -- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680b17f0 msgr2=0x7f196816ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f19677fe700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680b17f0 0x7f196816ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f19677fe700 1 -- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19680cc870 msgr2=0x7f1968163810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f196c99c700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19680cc870 0x7f1968163810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f291d3ff700 1 -- 172.21.15.29:0/2122214577 shutdown_connections 2023-10-07T09:29:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f19677fe700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19680cc870 0x7f1968163810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f19677fe700 1 -- 172.21.15.29:0/1889694644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19580094f0 con 0x7f19680b1bc0 2023-10-07T09:29:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f196c99c700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19680cc870 0x7f1968163810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.590+0000 7f291d3ff700 1 -- 172.21.15.29:0/2122214577 wait complete. 2023-10-07T09:29:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54500052b0 con 0x7f54700b0040 2023-10-07T09:29:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f19677fe700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 0x7f1968160b40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f195800a010 tx=0x7f1958017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.589+0000 7f8393fff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8398067080 0x7f8398069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.589+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f839c084490 con 0x7f83ac0cc870 2023-10-07T09:29:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f83abfff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8398067080 0x7f8398069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c15e770 con 0x7f073c0afa70 2023-10-07T09:29:34.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c148520 con 0x7f073c0afa70 2023-10-07T09:29:34.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.591+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f546001a460 con 0x7f54700b0040 2023-10-07T09:29:34.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.592+0000 7f291d3ff700 1 Processor -- start 2023-10-07T09:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.592+0000 7f5465ffb700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5454067080 0x7f5454069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.593+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5460084220 con 0x7f54700b0040 2023-10-07T09:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.593+0000 7f291d3ff700 1 -- start start 2023-10-07T09:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.594+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f839c055170 con 0x7f83ac0cc870 2023-10-07T09:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.594+0000 7fa760bff700 1 Processor -- start 2023-10-07T09:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.594+0000 7f291d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 0x7f29180c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.594+0000 7f291d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2918155510 0x7f2918143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f291d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29181558e0 0x7f291813f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f291d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180c2280 con 0x7f291806bae0 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f291d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29181466a0 con 0x7f2918155510 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f291d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29181443d0 con 0x7f29181558e0 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f2916ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 0x7f29180c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f2916ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 0x7f29180c1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56476/0 (socket says 172.21.15.29:56476) 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.595+0000 7f2916ffd700 1 -- 172.21.15.29:0/4148603183 learned_addr learned my addr 172.21.15.29:0/4148603183 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f5475d9b700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5454067080 0x7f5454069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f2916ffd700 1 -- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29181558e0 msgr2=0x7f291813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f2916ffd700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29181558e0 0x7f291813f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f2916ffd700 1 -- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2918155510 msgr2=0x7f2918143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f2916ffd700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2918155510 0x7f2918143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f2916ffd700 1 -- 172.21.15.29:0/4148603183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c009420 con 0x7f291806bae0 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.594+0000 7fa760bff700 1 -- start start 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7fa760bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 0x7fa75c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7fa760bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa75c0b1bc0 0x7fa75c15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7fa760bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c15eeb0 con 0x7fa75c0cc870 2023-10-07T09:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7fa760bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c0aebe0 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7fa760bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15e2a0 con 0x7fa75c0b17f0 2023-10-07T09:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.596+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0738022400 con 0x7f073c0afa70 2023-10-07T09:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7f07317fa700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0720067080 0x7f0720069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0738085c90 con 0x7f073c0afa70 2023-10-07T09:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7f2916ffd700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 0x7f29180c1d40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f291000bbb0 tx=0x7f2910009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.597+0000 7fa75a59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 0x7fa75c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7fa75a59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 0x7fa75c0afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44460/0 (socket says 172.21.15.29:44460) 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7fa75a59c700 1 -- 172.21.15.29:0/124175410 learned_addr learned my addr 172.21.15.29:0/124175410 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7fa75a59c700 1 -- 172.21.15.29:0/124175410 >> v2:172.21.15.29:3300/0 conn(0x7fa75c0b1bc0 msgr2=0x7fa75c15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7fa75a59c700 1 --2- 172.21.15.29:0/124175410 >> v2:172.21.15.29:3300/0 conn(0x7fa75c0b1bc0 0x7fa75c15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7fa75a59c700 1 -- 172.21.15.29:0/124175410 >> v1:172.21.15.29:6789/0 conn(0x7fa75c0cc870 legacy=0x7fa75c15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7fa75a59c700 1 -- 172.21.15.29:0/124175410 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c15e580 con 0x7fa75c0b17f0 2023-10-07T09:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7f83abfff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8398067080 0x7f8398069530 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f83ac149770 tx=0x7f83a0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7fa75a59c700 1 --2- 172.21.15.29:0/124175410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 0x7fa75c0afcb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099e0 tx=0x7fa750009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfa5f45c9207afc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7f0740f9a700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0720067080 0x7f0720069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800bed0 con 0x7f19680b1bc0 2023-10-07T09:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1958000c90 con 0x7f19680b1bc0 2023-10-07T09:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195801c420 con 0x7f19680b1bc0 2023-10-07T09:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968161ba0 con 0x7f19680b1bc0 2023-10-07T09:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c04c3c0 con 0x7f073c0afa70 2023-10-07T09:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7fa75959a700 1 -- 172.21.15.29:0/124175410 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75000b7e0 con 0x7fa75c0b17f0 2023-10-07T09:29:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7fa75959a700 1 -- 172.21.15.29:0/124175410 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75000b940 con 0x7fa75c0b17f0 2023-10-07T09:29:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.599+0000 7fa75959a700 1 -- 172.21.15.29:0/124175410 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75000bce0 con 0x7fa75c0b17f0 2023-10-07T09:29:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7fa760bff700 1 -- 172.21.15.29:0/124175410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 msgr2=0x7fa75c0afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7fa760bff700 1 --2- 172.21.15.29:0/124175410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 0x7fa75c0afcb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099e0 tx=0x7fa750009250 comp rx=0 tx=0).stop 2023-10-07T09:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7f0740f9a700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0720067080 0x7f0720069530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f073c154030 tx=0x7f0734007430 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29100092e0 con 0x7f291806bae0 2023-10-07T09:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000c7a0 con 0x7f291806bae0 2023-10-07T09:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000cb40 con 0x7f291806bae0 2023-10-07T09:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1968162160 con 0x7f19680b1bc0 2023-10-07T09:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2918144650 con 0x7f291806bae0 2023-10-07T09:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2918146050 con 0x7f291806bae0 2023-10-07T09:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.603+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291804c3c0 con 0x7f291806bae0 2023-10-07T09:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7fa760bff700 1 -- 172.21.15.29:0/124175410 shutdown_connections 2023-10-07T09:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7fa760bff700 1 --2- 172.21.15.29:0/124175410 >> v2:172.21.15.29:3300/0 conn(0x7fa75c0b1bc0 0x7fa75c15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7fa760bff700 1 --2- 172.21.15.29:0/124175410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b17f0 0x7fa75c0afcb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.601+0000 7fa760bff700 1 -- 172.21.15.29:0/124175410 >> 172.21.15.29:0/124175410 conn(0x7fa75c0680a0 msgr2=0x7fa75c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.603+0000 7fa760bff700 1 -- 172.21.15.29:0/124175410 shutdown_connections 2023-10-07T09:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.602+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19680cc680 con 0x7f19680b1bc0 2023-10-07T09:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.605+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0738052fa0 con 0x7f073c0afa70 2023-10-07T09:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.603+0000 7fa760bff700 1 -- 172.21.15.29:0/124175410 wait complete. 2023-10-07T09:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.605+0000 7fa760bff700 1 Processor -- start 2023-10-07T09:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.598+0000 7f5475d9b700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5454067080 0x7f5454069530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5470063cd0 tx=0x7f546c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.600+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54600514b0 con 0x7f54700b0040 2023-10-07T09:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.605+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f195801c850 con 0x7f19680b1bc0 2023-10-07T09:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.606+0000 7f195ffff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1948067080 0x7f1948069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.606+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1958085ed0 con 0x7f19680b1bc0 2023-10-07T09:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.612+0000 7f1967fff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1948067080 0x7f1948069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.613+0000 7f1967fff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1948067080 0x7f1948069530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1950007fd0 tx=0x7f1950009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.608+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2910012070 con 0x7f291806bae0 2023-10-07T09:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.609+0000 7f28feffd700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2904067080 0x7f2904069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.609+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2910084880 con 0x7f291806bae0 2023-10-07T09:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.610+0000 7f29167fc700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2904067080 0x7f2904069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.612+0000 7f29167fc700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2904067080 0x7f2904069530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29180aed30 tx=0x7f2908006c60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.606+0000 7fa760bff700 1 -- start start 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.607+0000 7fa760bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa75c0b17f0 0x7fa75c147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.607+0000 7fa760bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 0x7fa75c147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.607+0000 7fa760bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c0cc870 0x7fa75c136e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.607+0000 7fa760bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c149810 con 0x7fa75c0cc870 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.607+0000 7fa760bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c148c90 con 0x7fa75c0b17f0 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.607+0000 7fa760bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c148f70 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.610+0000 7fa759d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 0x7fa75c147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.610+0000 7fa759d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 0x7fa75c147580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44468/0 (socket says 172.21.15.29:44468) 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.610+0000 7fa759d9b700 1 -- 172.21.15.29:0/1922312373 learned_addr learned my addr 172.21.15.29:0/1922312373 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.610+0000 7fa75a59c700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa75c0b17f0 0x7fa75c147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa75ad9d700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c0cc870 0x7fa75c136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa759d9b700 1 -- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa75c0b17f0 msgr2=0x7fa75c147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa759d9b700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa75c0b17f0 0x7fa75c147040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa759d9b700 1 -- 172.21.15.29:0/1922312373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c0cc870 msgr2=0x7fa75c136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa759d9b700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c0cc870 0x7fa75c136e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa759d9b700 1 -- 172.21.15.29:0/1922312373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7500094f0 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.611+0000 7fa759d9b700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 0x7fa75c147580 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa75400bc30 tx=0x7fa754009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.612+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75400c9b0 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.612+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75400cb30 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.612+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754014410 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.613+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c149250 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.613+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c147ac0 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.617+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa754014800 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.618+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1958053120 con 0x7f19680b1bc0 2023-10-07T09:29:34.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.616+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2910051b50 con 0x7f291806bae0 2023-10-07T09:29:34.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.618+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c133900 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.619+0000 7fa74a7fc700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa73c067080 0x7fa73c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:34.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.619+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa7540845e0 con 0x7fa75c0b1bc0 2023-10-07T09:29:34.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.619+0000 7fa75a59c700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa73c067080 0x7fa73c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.620+0000 7fa75a59c700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa73c067080 0x7fa73c069530 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099e0 tx=0x7fa750016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:34.624+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa75404c5a0 con 0x7fa75c0b1bc0 2023-10-07T09:29:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.015+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f488c005830 con 0x7f48ac0b1bc0 2023-10-07T09:29:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.018+0000 7f48a17fa700 1 -- 172.21.15.29:0/917338572 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f48a4057120 con 0x7f48ac0b1bc0 2023-10-07T09:29:35.023 INFO:teuthology.orchestra.run.smithi029.stdout:60129542151 2023-10-07T09:29:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.024+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4890067080 msgr2=0x7f4890069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.024+0000 7f48b35ff700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4890067080 0x7f4890069530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f48ac14d380 tx=0x7f489c006ce0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.024+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 msgr2=0x7f48ac130d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.024+0000 7f48b35ff700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 0x7f48ac130d40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009fd0 tx=0x7f48a4017000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.025+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 shutdown_connections 2023-10-07T09:29:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.025+0000 7f48b35ff700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4890067080 0x7f4890069530 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.025+0000 7f48b35ff700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ac0cc870 0x7f48ac131e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.025+0000 7f48b35ff700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48ac0b17f0 0x7f48ac15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.025+0000 7f48b35ff700 1 --2- 172.21.15.29:0/917338572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48ac0b1bc0 0x7f48ac130d40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.025+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 >> 172.21.15.29:0/917338572 conn(0x7f48ac0680a0 msgr2=0x7f48ac14a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.026+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 shutdown_connections 2023-10-07T09:29:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.026+0000 7f48b35ff700 1 -- 172.21.15.29:0/917338572 wait complete. 2023-10-07T09:29:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.032+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f83ac15e860 con 0x7f83ac0cc870 2023-10-07T09:29:35.036 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-10-07T09:29:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.032+0000 7f8393fff700 1 -- 172.21.15.29:0/631763933 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f839c054d00 con 0x7f83ac0cc870 2023-10-07T09:29:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.035+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f681804c3c0 con 0x7f68180afa70 2023-10-07T09:29:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8398067080 msgr2=0x7f8398069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8398067080 0x7f8398069530 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f83ac149770 tx=0x7f83a0017000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83ac0cc870 msgr2=0x7f83ac1389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83ac0cc870 0x7f83ac1389f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f839c00e950 tx=0x7f839c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 shutdown_connections 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8398067080 0x7f8398069530 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83ac0cc870 0x7f83ac1389f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83ac0b17f0 0x7f83ac13af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 --2- 172.21.15.29:0/631763933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83ac0b1bc0 0x7f83ac1384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.036+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 >> 172.21.15.29:0/631763933 conn(0x7f83ac0680a0 msgr2=0x7f83ac15c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.037+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 shutdown_connections 2023-10-07T09:29:35.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.037+0000 7f83b29ff700 1 -- 172.21.15.29:0/631763933 wait complete. 2023-10-07T09:29:35.042 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-10-07T09:29:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.038+0000 7f68067fc700 1 -- 172.21.15.29:0/39448496 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f68080554b0 con 0x7f68180afa70 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.042+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f67f8067080 msgr2=0x7f67f8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.042+0000 7f681cdff700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f67f8067080 0x7f67f8069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f68180c8970 tx=0x7f6810009780 comp rx=0 tx=0).stop 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.042+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68180afa70 msgr2=0x7f6818164820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.042+0000 7f681cdff700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68180afa70 0x7f6818164820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f681815a3e0 tx=0x7f680801b000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.043+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 shutdown_connections 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.043+0000 7f681cdff700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f67f8067080 0x7f67f8069530 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.043+0000 7f681cdff700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68181558e0 0x7f68180caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.043+0000 7f681cdff700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68180afa70 0x7f6818164820 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.043+0000 7f681cdff700 1 --2- 172.21.15.29:0/39448496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6818155510 0x7f6818163070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.043+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 >> 172.21.15.29:0/39448496 conn(0x7f68180ccc30 msgr2=0x7f681806a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.047+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 shutdown_connections 2023-10-07T09:29:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.047+0000 7f681cdff700 1 -- 172.21.15.29:0/39448496 wait complete. 2023-10-07T09:29:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.051+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5450005560 con 0x7f54700b0040 2023-10-07T09:29:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.053+0000 7f5465ffb700 1 -- 172.21.15.29:0/2519824118 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5460054ab0 con 0x7f54700b0040 2023-10-07T09:29:35.056 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2023-10-07T09:29:35.056 INFO:teuthology.orchestra.run.smithi029.stdout:64424509447 2023-10-07T09:29:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.057+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f073c156c90 con 0x7f073c0afa70 2023-10-07T09:29:35.062 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-10-07T09:29:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.057+0000 7f07317fa700 1 -- 172.21.15.29:0/3001644606 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07380565c0 con 0x7f073c0afa70 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5454067080 msgr2=0x7f5454069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5454067080 0x7f5454069530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5470063cd0 tx=0x7f546c016000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 msgr2=0x7f5470158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 0x7f5470158d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f546000b840 tx=0x7f5460009220 comp rx=0 tx=0).stop 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 shutdown_connections 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5454067080 0x7f5454069530 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5470156980 0x7f54701592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f547006b590 0x7f5470144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 --2- 172.21.15.29:0/2519824118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54700b0040 0x7f5470158d90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 >> 172.21.15.29:0/2519824118 conn(0x7f54700680a0 msgr2=0x7f5470142a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.058+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 shutdown_connections 2023-10-07T09:29:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.059+0000 7f5477fff700 1 -- 172.21.15.29:0/2519824118 wait complete. 2023-10-07T09:29:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.061+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0720067080 msgr2=0x7f0720069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.061+0000 7f07439ff700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0720067080 0x7f0720069530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f073c154030 tx=0x7f0734007430 comp rx=0 tx=0).stop 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 msgr2=0x7f073c147fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 0x7f073c147fe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0738009b20 tx=0x7f0738018000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 shutdown_connections 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0720067080 0x7f0720069530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f073c1558e0 0x7f073c153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f073c0afa70 0x7f073c147fe0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 --2- 172.21.15.29:0/3001644606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c155510 0x7f073c152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.062+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 >> 172.21.15.29:0/3001644606 conn(0x7f073c0ccc30 msgr2=0x7f073c06a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.063+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 shutdown_connections 2023-10-07T09:29:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.063+0000 7f07439ff700 1 -- 172.21.15.29:0/3001644606 wait complete. 2023-10-07T09:29:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.069+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1968162350 con 0x7f19680b1bc0 2023-10-07T09:29:35.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.2 2023-10-07T09:29:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.070+0000 7f195ffff700 1 -- 172.21.15.29:0/1889694644 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1958056740 con 0x7f19680b1bc0 2023-10-07T09:29:35.072 INFO:teuthology.orchestra.run.smithi029.stdout:64424509447 2023-10-07T09:29:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1948067080 msgr2=0x7f1948069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1948067080 0x7f1948069530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1950007fd0 tx=0x7f1950009040 comp rx=0 tx=0).stop 2023-10-07T09:29:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 msgr2=0x7f1968160b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 0x7f1968160b40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f195800a010 tx=0x7f1958017000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 shutdown_connections 2023-10-07T09:29:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1948067080 0x7f1948069530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19680cc870 0x7f1968163810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19680b17f0 0x7f196816ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 --2- 172.21.15.29:0/1889694644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19680b1bc0 0x7f1968160b40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 >> 172.21.15.29:0/1889694644 conn(0x7f19680680a0 msgr2=0x7f196815b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 shutdown_connections 2023-10-07T09:29:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.074+0000 7f196e3ff700 1 -- 172.21.15.29:0/1889694644 wait complete. 2023-10-07T09:29:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.075+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa75c04c3c0 con 0x7fa75c0b1bc0 2023-10-07T09:29:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.076+0000 7fa74a7fc700 1 -- 172.21.15.29:0/1922312373 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa75c133900 con 0x7fa75c0b1bc0 2023-10-07T09:29:35.078 INFO:teuthology.orchestra.run.smithi029.stdout:68719476743 2023-10-07T09:29:35.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.0 2023-10-07T09:29:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.079+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2918157000 con 0x7f291806bae0 2023-10-07T09:29:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.079+0000 7f28feffd700 1 -- 172.21.15.29:0/4148603183 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2910055170 con 0x7f291806bae0 2023-10-07T09:29:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.079+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa73c067080 msgr2=0x7fa73c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.079+0000 7fa760bff700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa73c067080 0x7fa73c069530 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099e0 tx=0x7fa750016000 comp rx=0 tx=0).stop 2023-10-07T09:29:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 msgr2=0x7fa75c147580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 0x7fa75c147580 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa75400bc30 tx=0x7fa754009820 comp rx=0 tx=0).stop 2023-10-07T09:29:35.082 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-10-07T09:29:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 shutdown_connections 2023-10-07T09:29:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa73c067080 0x7fa73c069530 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c0cc870 0x7fa75c136e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa75c0b17f0 0x7fa75c147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 --2- 172.21.15.29:0/1922312373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa75c0b1bc0 0x7fa75c147580 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 >> 172.21.15.29:0/1922312373 conn(0x7fa75c0680a0 msgr2=0x7fa75c15b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 shutdown_connections 2023-10-07T09:29:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.080+0000 7fa760bff700 1 -- 172.21.15.29:0/1922312373 wait complete. 2023-10-07T09:29:35.084 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.6 2023-10-07T09:29:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.083+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2904067080 msgr2=0x7f2904069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.083+0000 7f291d3ff700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2904067080 0x7f2904069530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29180aed30 tx=0x7f2908006c60 comp rx=0 tx=0).stop 2023-10-07T09:29:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.083+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 msgr2=0x7f29180c1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.083+0000 7f291d3ff700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 0x7f29180c1d40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f291000bbb0 tx=0x7f2910009840 comp rx=0 tx=0).stop 2023-10-07T09:29:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 shutdown_connections 2023-10-07T09:29:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2904067080 0x7f2904069530 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29181558e0 0x7f291813f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f291806bae0 0x7f29180c1d40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 --2- 172.21.15.29:0/4148603183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2918155510 0x7f2918143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 >> 172.21.15.29:0/4148603183 conn(0x7f29180ccc30 msgr2=0x7f29180af2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 shutdown_connections 2023-10-07T09:29:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:35.084+0000 7f291d3ff700 1 -- 172.21.15.29:0/4148603183 wait complete. 2023-10-07T09:29:35.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.4 2023-10-07T09:29:35.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.1 2023-10-07T09:29:35.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2023-10-07T09:29:35.106 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476743 for osd.7 2023-10-07T09:29:36.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-07T09:29:36.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-07T09:29:36.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-07T09:29:36.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-07T09:29:36.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-07T09:29:36.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-07T09:29:36.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-07T09:29:36.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-07T09:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.371+0000 7f01d9dff700 1 Processor -- start 2023-10-07T09:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.371+0000 7f01d9dff700 1 -- start start 2023-10-07T09:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.376+0000 7f01d9dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01d40ae7f0 0x7f01d40aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.376+0000 7f01d9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4166340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.376+0000 7f01d9dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40af5d0 con 0x7f01d40b1580 2023-10-07T09:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.376+0000 7f01d9dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d4166880 con 0x7f01d40ae7f0 2023-10-07T09:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.376+0000 7f01d9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d4167de0 con 0x7f01d40afac0 2023-10-07T09:29:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.378+0000 7f01d2ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4166340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.378+0000 7f01d2ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4166340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44476/0 (socket says 172.21.15.29:44476) 2023-10-07T09:29:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.378+0000 7f01d2ffd700 1 -- 172.21.15.29:0/4202954529 learned_addr learned my addr 172.21.15.29:0/4202954529 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.379+0000 7f01d2ffd700 1 -- 172.21.15.29:0/4202954529 >> v2:172.21.15.29:3300/0 conn(0x7f01d40ae7f0 msgr2=0x7f01d40aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.379+0000 7f01d2ffd700 1 --2- 172.21.15.29:0/4202954529 >> v2:172.21.15.29:3300/0 conn(0x7f01d40ae7f0 0x7f01d40aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.379+0000 7f01d2ffd700 1 -- 172.21.15.29:0/4202954529 >> v1:172.21.15.29:6789/0 conn(0x7f01d40b1580 legacy=0x7f01d40b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.379+0000 7f01d2ffd700 1 -- 172.21.15.29:0/4202954529 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d416d230 con 0x7f01d40afac0 2023-10-07T09:29:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.380+0000 7f01d2ffd700 1 --2- 172.21.15.29:0/4202954529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4166340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f01c80099b0 tx=0x7f01c8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=793889c378d8a199 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.382+0000 7f01d27fc700 1 -- 172.21.15.29:0/4202954529 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800b860 con 0x7f01d40afac0 2023-10-07T09:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.382+0000 7f01d27fc700 1 -- 172.21.15.29:0/4202954529 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c800b9c0 con 0x7f01d40afac0 2023-10-07T09:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.382+0000 7f01d27fc700 1 -- 172.21.15.29:0/4202954529 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800bd60 con 0x7f01d40afac0 2023-10-07T09:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.383+0000 7f01d9dff700 1 -- 172.21.15.29:0/4202954529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 msgr2=0x7f01d4166340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.383+0000 7f01d9dff700 1 --2- 172.21.15.29:0/4202954529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4166340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f01c80099b0 tx=0x7f01c8009250 comp rx=0 tx=0).stop 2023-10-07T09:29:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.383+0000 7f01d9dff700 1 -- 172.21.15.29:0/4202954529 shutdown_connections 2023-10-07T09:29:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.383+0000 7f01d9dff700 1 --2- 172.21.15.29:0/4202954529 >> v2:172.21.15.29:3300/0 conn(0x7f01d40ae7f0 0x7f01d40aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.383+0000 7f01d9dff700 1 --2- 172.21.15.29:0/4202954529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4166340 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.383+0000 7f01d9dff700 1 -- 172.21.15.29:0/4202954529 >> 172.21.15.29:0/4202954529 conn(0x7f01d40ccc30 msgr2=0x7f01d4069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.384+0000 7f01d9dff700 1 -- 172.21.15.29:0/4202954529 shutdown_connections 2023-10-07T09:29:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.384+0000 7f01d9dff700 1 -- 172.21.15.29:0/4202954529 wait complete. 2023-10-07T09:29:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.386+0000 7f01d9dff700 1 Processor -- start 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.387+0000 7f01d9dff700 1 -- start start 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.389+0000 7f01d9dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 0x7f01d416d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.389+0000 7f01d9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.389+0000 7f01d9dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01d40b1580 0x7f01d4159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.389+0000 7f01d9dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d415cce0 con 0x7f01d40ae7f0 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.389+0000 7f01d9dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d415c0d0 con 0x7f01d40b1580 2023-10-07T09:29:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.389+0000 7f01d9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d415c3b0 con 0x7f01d40afac0 2023-10-07T09:29:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.390+0000 7f01d37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 0x7f01d416d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.390+0000 7f01d37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 0x7f01d416d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56512/0 (socket says 172.21.15.29:56512) 2023-10-07T09:29:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.390+0000 7f01d37fe700 1 -- 172.21.15.29:0/3794349934 learned_addr learned my addr 172.21.15.29:0/3794349934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.391+0000 7f01d37fe700 1 -- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 msgr2=0x7f01d4159750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.391+0000 7f01d37fe700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4159750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.391+0000 7f01d37fe700 1 -- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01d40b1580 msgr2=0x7f01d4159c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.391+0000 7f01d37fe700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01d40b1580 0x7f01d4159c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.391+0000 7f01d37fe700 1 -- 172.21.15.29:0/3794349934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c80094f0 con 0x7f01d40ae7f0 2023-10-07T09:29:36.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.392+0000 7f01d37fe700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 0x7f01d416d6c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01c400b810 tx=0x7f01c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.395+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c400f570 con 0x7f01d40ae7f0 2023-10-07T09:29:36.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.395+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c400f6d0 con 0x7f01d40ae7f0 2023-10-07T09:29:36.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.395+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c400fa90 con 0x7f01d40ae7f0 2023-10-07T09:29:36.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.395+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d415c690 con 0x7f01d40ae7f0 2023-10-07T09:29:36.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.395+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d4151400 con 0x7f01d40ae7f0 2023-10-07T09:29:36.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.399+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d404c3c0 con 0x7f01d40ae7f0 2023-10-07T09:29:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.412+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f01c401a460 con 0x7f01d40ae7f0 2023-10-07T09:29:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.414+0000 7f01bb7fe700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f01c0067080 0x7f01c0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.414+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f01c40841e0 con 0x7f01d40ae7f0 2023-10-07T09:29:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.414+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01c4086780 con 0x7f01d40ae7f0 2023-10-07T09:29:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.415+0000 7f01d2ffd700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f01c0067080 0x7f01c0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.433+0000 7f01d2ffd700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f01c0067080 0x7f01c0069530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01c80099b0 tx=0x7f01c8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.450+0000 7ff8f15ff700 1 Processor -- start 2023-10-07T09:29:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.450+0000 7ff8f15ff700 1 -- start start 2023-10-07T09:29:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.453+0000 7ff8f15ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8ec15dc90 0x7ff8ec0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.453+0000 7ff8f15ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8ec0b1ac0 0x7ff8ec15f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.453+0000 7ff8f15ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec15f2a0 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.453+0000 7ff8f15ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec06a3f0 con 0x7ff8ec15dc90 2023-10-07T09:29:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.453+0000 7ff8f15ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec0af3d0 con 0x7ff8ec0b1ac0 2023-10-07T09:29:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff8eaffd700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff8ec15d8c0 0x7ff8ec0afac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39442/0 (socket says 172.21.15.29:39442) 2023-10-07T09:29:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff8eaffd700 1 -- 172.21.15.29:0/1905767176 learned_addr learned my addr 172.21.15.29:0/1905767176 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff8eb7fe700 1 --2- 172.21.15.29:0/1905767176 >> v2:172.21.15.144:3300/0 conn(0x7ff8ec0b1ac0 0x7ff8ec15f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff8e9ffb700 1 -- 172.21.15.29:0/1905767176 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399339183 0 0) 0x7ff8ec15f2a0 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff8e9ffb700 1 -- 172.21.15.29:0/1905767176 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d4003570 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff0b1fff700 1 Processor -- start 2023-10-07T09:29:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.454+0000 7ff0b1fff700 1 -- start start 2023-10-07T09:29:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.455+0000 7ff8ea7fc700 1 --2- 172.21.15.29:0/1905767176 >> v2:172.21.15.29:3300/0 conn(0x7ff8ec15dc90 0x7ff8ec0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.455+0000 7ff8eb7fe700 1 -- 172.21.15.29:0/1905767176 >> v2:172.21.15.29:3300/0 conn(0x7ff8ec15dc90 msgr2=0x7ff8ec0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.455+0000 7ff8eb7fe700 1 --2- 172.21.15.29:0/1905767176 >> v2:172.21.15.29:3300/0 conn(0x7ff8ec15dc90 0x7ff8ec0b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.455+0000 7ff8eb7fe700 1 -- 172.21.15.29:0/1905767176 >> v1:172.21.15.144:6789/0 conn(0x7ff8ec15d8c0 legacy=0x7ff8ec0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.455+0000 7ff8eb7fe700 1 -- 172.21.15.29:0/1905767176 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec15ec50 con 0x7ff8ec0b1ac0 2023-10-07T09:29:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.456+0000 7ff8eb7fe700 1 --2- 172.21.15.29:0/1905767176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec0b1ac0 0x7ff8ec15f730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0c8450 tx=0x7ff8e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20f3efa85a26e3a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7ff0b1fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0ac0cc870 0x7ff0ac0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7ff0b1fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0ac0b1bc0 0x7ff0ac15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7ff0b1fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac15eeb0 con 0x7ff0ac0b17f0 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7ff0b1fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ac0aebe0 con 0x7ff0ac0cc870 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7ff0b1fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ac15e2a0 con 0x7ff0ac0b1bc0 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7ff0abfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0ac0b1bc0 0x7ff0ac15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.458+0000 7ff0abfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0ac0b1bc0 0x7ff0ac15dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52968/0 (socket says 172.21.15.29:52968) 2023-10-07T09:29:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.458+0000 7ff0abfff700 1 -- 172.21.15.29:0/1953816395 learned_addr learned my addr 172.21.15.29:0/1953816395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.458+0000 7ff0aaffd700 1 --2- 172.21.15.29:0/1953816395 >> v2:172.21.15.29:3300/0 conn(0x7ff0ac0cc870 0x7ff0ac0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff0aaffd700 1 -- 172.21.15.29:0/1953816395 >> v2:172.21.15.144:3300/0 conn(0x7ff0ac0b1bc0 msgr2=0x7ff0ac15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff0aaffd700 1 --2- 172.21.15.29:0/1953816395 >> v2:172.21.15.144:3300/0 conn(0x7ff0ac0b1bc0 0x7ff0ac15dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff0aaffd700 1 -- 172.21.15.29:0/1953816395 >> v1:172.21.15.29:6789/0 conn(0x7ff0ac0b17f0 legacy=0x7ff0ac15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff0aaffd700 1 -- 172.21.15.29:0/1953816395 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac15e580 con 0x7ff0ac0cc870 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff8e9ffb700 1 -- 172.21.15.29:0/1905767176 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0011630 con 0x7ff8ec0b1ac0 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff8e9ffb700 1 -- 172.21.15.29:0/1905767176 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e0011790 con 0x7ff8ec0b1ac0 2023-10-07T09:29:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff8e9ffb700 1 -- 172.21.15.29:0/1905767176 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0011b30 con 0x7ff8ec0b1ac0 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.456+0000 7fbf959ff700 1 Processor -- start 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.457+0000 7fbf959ff700 1 -- start start 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7fbf959ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf9015d8c0 0x7fbf900afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7fbf959ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf9015dc90 0x7fbf900b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7fbf959ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9015f2a0 con 0x7fbf900b1ac0 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7fbf959ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9006a3f0 con 0x7fbf9015dc90 2023-10-07T09:29:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7fbf959ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900b0000 con 0x7fbf9015d8c0 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.459+0000 7ff0aaffd700 1 --2- 172.21.15.29:0/1953816395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac0afcb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0ccfd0 tx=0x7ff0a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37590a7810192ac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1905767176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec0b1ac0 msgr2=0x7ff8ec15f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1905767176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec0b1ac0 0x7ff8ec15f730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0c8450 tx=0x7ff8e00091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1905767176 shutdown_connections 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1905767176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec0b1ac0 0x7ff8ec15f730 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1905767176 >> v2:172.21.15.29:3300/0 conn(0x7ff8ec15dc90 0x7ff8ec0b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1905767176 >> 172.21.15.29:0/1905767176 conn(0x7ff8ec0ccc30 msgr2=0x7ff8ec069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff0aa7fc700 1 -- 172.21.15.29:0/1953816395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400e070 con 0x7ff0ac0cc870 2023-10-07T09:29:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff0aa7fc700 1 -- 172.21.15.29:0/1953816395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a400bb20 con 0x7ff0ac0cc870 2023-10-07T09:29:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff0aa7fc700 1 -- 172.21.15.29:0/1953816395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4012430 con 0x7ff0ac0cc870 2023-10-07T09:29:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1905767176 shutdown_connections 2023-10-07T09:29:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.461+0000 7ff0b1fff700 1 -- 172.21.15.29:0/1953816395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 msgr2=0x7ff0ac0afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.461+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/1953816395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac0afcb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0ccfd0 tx=0x7ff0a4009780 comp rx=0 tx=0).stop 2023-10-07T09:29:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.460+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1905767176 wait complete. 2023-10-07T09:29:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff8f15ff700 1 Processor -- start 2023-10-07T09:29:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7fbf8effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf9015d8c0 0x7fbf900afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7fbf8e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf9015dc90 0x7fbf900b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7fbf8e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf9015dc90 0x7fbf900b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56552/0 (socket says 172.21.15.29:56552) 2023-10-07T09:29:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7fbf8e7fc700 1 -- 172.21.15.29:0/1679248468 learned_addr learned my addr 172.21.15.29:0/1679248468 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff0b1fff700 1 -- 172.21.15.29:0/1953816395 shutdown_connections 2023-10-07T09:29:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/1953816395 >> v2:172.21.15.144:3300/0 conn(0x7ff0ac0b1bc0 0x7ff0ac15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/1953816395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac0afcb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff0b1fff700 1 -- 172.21.15.29:0/1953816395 >> 172.21.15.29:0/1953816395 conn(0x7ff0ac0680a0 msgr2=0x7ff0ac0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff0b1fff700 1 -- 172.21.15.29:0/1953816395 shutdown_connections 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.462+0000 7ff0b1fff700 1 -- 172.21.15.29:0/1953816395 wait complete. 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.463+0000 7ff8f15ff700 1 -- start start 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.464+0000 7ff8f15ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8ec0b1ac0 0x7ff8ec171040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.464+0000 7ff8f15ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 0x7ff8ec161b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.464+0000 7ff8f15ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec15dc90 0x7ff8ec162040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.464+0000 7ff8f15ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0c2ae0 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.464+0000 7ff8f15ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0c1e60 con 0x7ff8ec15dc90 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.464+0000 7ff8f15ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0c2140 con 0x7ff8ec0b1ac0 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.463+0000 7fbf8e7fc700 1 -- 172.21.15.29:0/1679248468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf9015d8c0 msgr2=0x7fbf900afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.463+0000 7fbf8e7fc700 1 --2- 172.21.15.29:0/1679248468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf9015d8c0 0x7fbf900afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.463+0000 7fbf8e7fc700 1 -- 172.21.15.29:0/1679248468 >> v1:172.21.15.29:6789/0 conn(0x7fbf900b1ac0 legacy=0x7fbf901615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.463+0000 7fbf8e7fc700 1 -- 172.21.15.29:0/1679248468 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900af3d0 con 0x7fbf9015dc90 2023-10-07T09:29:36.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.463+0000 7fbf8e7fc700 1 --2- 172.21.15.29:0/1679248468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9015dc90 0x7fbf900b1580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf780099b0 tx=0x7fbf78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b956c74efb3a3f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.465+0000 7ff0b1fff700 1 Processor -- start 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.465+0000 7ff0b1fff700 1 -- start start 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff0b1fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0ac0b17f0 0x7ff0ac152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff0b1fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0ac0b1bc0 0x7ff0ac13ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff0b1fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac06db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff0b1fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac161320 con 0x7ff0ac0cc870 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff0b1fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac156810 con 0x7ff0ac0b1bc0 2023-10-07T09:29:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff0b1fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac152020 con 0x7ff0ac0b17f0 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff8eb7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec15dc90 0x7ff8ec162040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff8eaffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8ec0b1ac0 0x7ff8ec171040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff8ea7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 0x7ff8ec161b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff8ea7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 0x7ff8ec161b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56558/0 (socket says 172.21.15.29:56558) 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.466+0000 7ff8ea7fc700 1 -- 172.21.15.29:0/1634152046 learned_addr learned my addr 172.21.15.29:0/1634152046 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff8ea7fc700 1 -- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8ec0b1ac0 msgr2=0x7ff8ec171040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff8ea7fc700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8ec0b1ac0 0x7ff8ec171040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff8ea7fc700 1 -- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec15dc90 msgr2=0x7ff8ec162040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff8ea7fc700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec15dc90 0x7ff8ec162040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff8ea7fc700 1 -- 172.21.15.29:0/1634152046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e0009420 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.467+0000 7ff8ea7fc700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 0x7ff8ec161b00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d860 tx=0x7ff8dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0125f0 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc012750 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc012b10 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff0abfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff0abfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56564/0 (socket says 172.21.15.29:56564) 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff0abfff700 1 -- 172.21.15.29:0/2822467796 learned_addr learned my addr 172.21.15.29:0/2822467796 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7ff0ab7fe700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0ac0b17f0 0x7ff0ac152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec0c2420 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec171580 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7fbf8dffb700 1 -- 172.21.15.29:0/1679248468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7800b820 con 0x7fbf9015dc90 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7fbf8dffb700 1 -- 172.21.15.29:0/1679248468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7800b980 con 0x7fbf9015dc90 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.468+0000 7fbf8dffb700 1 -- 172.21.15.29:0/1679248468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7800bd20 con 0x7fbf9015dc90 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7fbf959ff700 1 -- 172.21.15.29:0/1679248468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9015dc90 msgr2=0x7fbf900b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7fbf959ff700 1 --2- 172.21.15.29:0/1679248468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9015dc90 0x7fbf900b1580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf780099b0 tx=0x7fbf78009250 comp rx=0 tx=0).stop 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7fbf959ff700 1 -- 172.21.15.29:0/1679248468 shutdown_connections 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7fbf959ff700 1 --2- 172.21.15.29:0/1679248468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf9015d8c0 0x7fbf900afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7fbf959ff700 1 --2- 172.21.15.29:0/1679248468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9015dc90 0x7fbf900b1580 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf780099b0 tx=0x7fbf78009250 comp rx=0 tx=0).stop 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7fbf959ff700 1 -- 172.21.15.29:0/1679248468 >> 172.21.15.29:0/1679248468 conn(0x7fbf900ccc30 msgr2=0x7fbf90069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff0abfff700 1 -- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0ac0b17f0 msgr2=0x7ff0ac152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff0abfff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0ac0b17f0 0x7ff0ac152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff0abfff700 1 -- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0ac0b1bc0 msgr2=0x7ff0ac13ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff0abfff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0ac0b1bc0 0x7ff0ac13ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.469+0000 7ff0abfff700 1 -- 172.21.15.29:0/2822467796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a4009a20 con 0x7ff0ac0cc870 2023-10-07T09:29:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.470+0000 7ff0abfff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac06db20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00e960 tx=0x7ff09c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.470+0000 7fbf959ff700 1 -- 172.21.15.29:0/1679248468 shutdown_connections 2023-10-07T09:29:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.470+0000 7fbf959ff700 1 -- 172.21.15.29:0/1679248468 wait complete. 2023-10-07T09:29:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.471+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00c850 con 0x7ff0ac0cc870 2023-10-07T09:29:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.471+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c00c9b0 con 0x7ff0ac0cc870 2023-10-07T09:29:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.471+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00cd30 con 0x7ff0ac0cc870 2023-10-07T09:29:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.471+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec0c2c20 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.471+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ac151a60 con 0x7ff0ac0cc870 2023-10-07T09:29:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.471+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ac1564a0 con 0x7ff0ac0cc870 2023-10-07T09:29:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.473+0000 7fbf959ff700 1 Processor -- start 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.474+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ac1569a0 con 0x7ff0ac0cc870 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.473+0000 7fbf959ff700 1 -- start start 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbf959ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf900b1ac0 0x7fbf9016f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbf959ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf9015d8c0 0x7fbf90162090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbf959ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 0x7fbf900c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbf959ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf901610e0 con 0x7fbf90157580 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbf959ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90157a00 con 0x7fbf9015d8c0 2023-10-07T09:29:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbf959ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90162790 con 0x7fbf900b1ac0 2023-10-07T09:29:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.476+0000 7fbf8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 0x7fbf900c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.476+0000 7fbf8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 0x7fbf900c1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56572/0 (socket says 172.21.15.29:56572) 2023-10-07T09:29:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.476+0000 7fbf8f7fe700 1 -- 172.21.15.29:0/948844139 learned_addr learned my addr 172.21.15.29:0/948844139 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.475+0000 7fbfe5fff700 1 Processor -- start 2023-10-07T09:29:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.476+0000 7fbfe5fff700 1 -- start start 2023-10-07T09:29:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.477+0000 7fbf8f7fe700 1 -- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf900b1ac0 msgr2=0x7fbf9016f210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.477+0000 7fbf8f7fe700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf900b1ac0 0x7fbf9016f210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.477+0000 7fbf8f7fe700 1 -- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf9015d8c0 msgr2=0x7fbf90162090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.477+0000 7fbf8f7fe700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf9015d8c0 0x7fbf90162090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.477+0000 7fbf8f7fe700 1 -- 172.21.15.29:0/948844139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf780094f0 con 0x7fbf90157580 2023-10-07T09:29:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.477+0000 7fbf8f7fe700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 0x7fbf900c1da0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b5e0 tx=0x7fbf88009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.478+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff8dc01e460 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.478+0000 7fbfe5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 0x7fbfe00cb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.478+0000 7fbfe5fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfe00ce8a0 0x7fbfe00c5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.478+0000 7fbfe5fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00cc560 con 0x7fbfe015df30 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.478+0000 7fbfe5fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00cb950 con 0x7fbfe00ce8a0 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.478+0000 7fbfe5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00cbc30 con 0x7fbfe015db60 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.479+0000 7fbfdf7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 0x7fbfe00cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.479+0000 7fbfdf7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 0x7fbfe00cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44540/0 (socket says 172.21.15.29:44540) 2023-10-07T09:29:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.479+0000 7fbfdf7fe700 1 -- 172.21.15.29:0/521438843 learned_addr learned my addr 172.21.15.29:0/521438843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.479+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff09c013680 con 0x7ff0ac0cc870 2023-10-07T09:29:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7ff8d2ffd700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff8d8067080 0x7ff8d8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff8dc084ae0 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7fbfdf7fe700 1 -- 172.21.15.29:0/521438843 >> v2:172.21.15.29:3300/0 conn(0x7fbfe00ce8a0 msgr2=0x7fbfe00c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/521438843 >> v2:172.21.15.29:3300/0 conn(0x7fbfe00ce8a0 0x7fbfe00c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7fbfdf7fe700 1 -- 172.21.15.29:0/521438843 >> v1:172.21.15.29:6789/0 conn(0x7fbfe015df30 legacy=0x7fbfe00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7fbfdf7fe700 1 -- 172.21.15.29:0/521438843 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe00cbf10 con 0x7fbfe015db60 2023-10-07T09:29:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.480+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/521438843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 0x7fbfe00cb300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009990 tx=0x7fbfd0009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e037aa214c29c8aa server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.481+0000 7ff8eaffd700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff8d8067080 0x7ff8d8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.481+0000 7fbfde7fc700 1 -- 172.21.15.29:0/521438843 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0005c10 con 0x7fbfe015db60 2023-10-07T09:29:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.481+0000 7fbfde7fc700 1 -- 172.21.15.29:0/521438843 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0005d70 con 0x7fbfe015db60 2023-10-07T09:29:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.481+0000 7fbfde7fc700 1 -- 172.21.15.29:0/521438843 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000b840 con 0x7fbfe015db60 2023-10-07T09:29:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.481+0000 7ff0937fe700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff098067080 0x7ff098069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.481+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff09c084790 con 0x7ff0ac0cc870 2023-10-07T09:29:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7ff0ab7fe700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff098067080 0x7ff098069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 -- 172.21.15.29:0/521438843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 msgr2=0x7fbfe00cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/521438843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 0x7fbfe00cb300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009990 tx=0x7fbfd0009220 comp rx=0 tx=0).stop 2023-10-07T09:29:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7ff8eaffd700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff8d8067080 0x7ff8d8069530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4005c50 tx=0x7ff8e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 -- 172.21.15.29:0/521438843 shutdown_connections 2023-10-07T09:29:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/521438843 >> v2:172.21.15.29:3300/0 conn(0x7fbfe00ce8a0 0x7fbfe00c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/521438843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015db60 0x7fbfe00cb300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 -- 172.21.15.29:0/521438843 >> 172.21.15.29:0/521438843 conn(0x7fbfe0063a10 msgr2=0x7fbfe00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 -- 172.21.15.29:0/521438843 shutdown_connections 2023-10-07T09:29:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.482+0000 7fbfe5fff700 1 -- 172.21.15.29:0/521438843 wait complete. 2023-10-07T09:29:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.483+0000 7ff0ab7fe700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff098067080 0x7ff098069530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0003870 tx=0x7ff0a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.484+0000 7fbfe5fff700 1 Processor -- start 2023-10-07T09:29:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.485+0000 7fbfe5fff700 1 -- start start 2023-10-07T09:29:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.485+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8ec0c2c20 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8800f510 con 0x7fbf90157580 2023-10-07T09:29:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf90162a70 con 0x7fbf90157580 2023-10-07T09:29:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf90163030 con 0x7fbf90157580 2023-10-07T09:29:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbfe5fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfe00ce8a0 0x7fbfe016dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbfe5fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 0x7fbfe0160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbfe5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015df30 0x7fbfe00cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbfe5fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe016e520 con 0x7fbfe015db60 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbfe5fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b3cd0 con 0x7fbfe00ce8a0 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.486+0000 7fbfe5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b3f90 con 0x7fbfe015df30 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.487+0000 7fbfdffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015df30 0x7fbfe00cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.487+0000 7fbfdeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 0x7fbfe0160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.487+0000 7fbfdeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 0x7fbfe0160d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56588/0 (socket says 172.21.15.29:56588) 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.487+0000 7fbfdeffd700 1 -- 172.21.15.29:0/2463488360 learned_addr learned my addr 172.21.15.29:0/2463488360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.488+0000 7fbfdeffd700 1 -- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015df30 msgr2=0x7fbfe00cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.488+0000 7fbfdeffd700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015df30 0x7fbfe00cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.488+0000 7fbfdeffd700 1 -- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfe00ce8a0 msgr2=0x7fbfe016dfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.488+0000 7fbfdeffd700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfe00ce8a0 0x7fbfe016dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.488+0000 7fbfdeffd700 1 -- 172.21.15.29:0/2463488360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0009480 con 0x7fbfe015db60 2023-10-07T09:29:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.488+0000 7fbfdeffd700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 0x7fbfe0160d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800bbb0 tx=0x7fbfd8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.489+0000 7f7c251ff700 1 Processor -- start 2023-10-07T09:29:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.489+0000 7f7c251ff700 1 -- start start 2023-10-07T09:29:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.490+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9004c3c0 con 0x7fbf90157580 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.491+0000 7f7c251ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b17f0 0x7f7c200afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.491+0000 7f7c251ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c200b1bc0 0x7f7c2015dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.491+0000 7f7c251ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2015eeb0 con 0x7f7c200cc870 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.491+0000 7f7c251ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200aebe0 con 0x7f7c200b1bc0 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.491+0000 7f7c251ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2015e2a0 con 0x7f7c200b17f0 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.492+0000 7f7c1659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7c200cc870 0x7f7c2015d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58706/0 (socket says 172.21.15.29:58706) 2023-10-07T09:29:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.492+0000 7f7c1659c700 1 -- 172.21.15.29:0/1027252261 learned_addr learned my addr 172.21.15.29:0/1027252261 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7f7c1f59e700 1 --2- 172.21.15.29:0/1027252261 >> v2:172.21.15.144:3300/0 conn(0x7f7c200b1bc0 0x7f7c2015dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7f7c1ed9d700 1 --2- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b17f0 0x7f7c200afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8800f880 con 0x7fbf90157580 2023-10-07T09:29:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8800fa10 con 0x7fbf90157580 2023-10-07T09:29:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80092e0 con 0x7fbfe015db60 2023-10-07T09:29:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe00b4270 con 0x7fbfe015db60 2023-10-07T09:29:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.493+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe00b4830 con 0x7fbfe015db60 2023-10-07T09:29:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.495+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd8009980 con 0x7fbfe015db60 2023-10-07T09:29:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.495+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd800c9b0 con 0x7fbfe015db60 2023-10-07T09:29:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.495+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe00b4a50 con 0x7fbfe015db60 2023-10-07T09:29:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.496+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbf8800fb70 con 0x7fbf90157580 2023-10-07T09:29:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.496+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff09c0519e0 con 0x7ff0ac0cc870 2023-10-07T09:29:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.498+0000 7fbf86ffd700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbf70066f30 0x7fbf700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.498+0000 7fbf8effd700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbf70066f30 0x7fbf700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.498+0000 7f7c1f59e700 1 -- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b17f0 msgr2=0x7f7c200afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.498+0000 7f7c1f59e700 1 --2- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b17f0 0x7f7c200afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.498+0000 7f7c1f59e700 1 -- 172.21.15.29:0/1027252261 >> v1:172.21.15.29:6789/0 conn(0x7f7c200cc870 legacy=0x7f7c2015d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.498+0000 7f7c1f59e700 1 -- 172.21.15.29:0/1027252261 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2015e580 con 0x7f7c200b1bc0 2023-10-07T09:29:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.499+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbf88019360 con 0x7fbf90157580 2023-10-07T09:29:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.500+0000 7f7c1f59e700 1 --2- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b1bc0 0x7f7c2015dc10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009e90 tx=0x7f7c18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d27ec263b641151 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.500+0000 7f7c1e59c700 1 -- 172.21.15.29:0/1027252261 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800e070 con 0x7f7c200b1bc0 2023-10-07T09:29:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.500+0000 7f7c1e59c700 1 -- 172.21.15.29:0/1027252261 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1800bbf0 con 0x7f7c200b1bc0 2023-10-07T09:29:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.500+0000 7f7c1e59c700 1 -- 172.21.15.29:0/1027252261 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18012430 con 0x7f7c200b1bc0 2023-10-07T09:29:36.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.501+0000 7f7c251ff700 1 -- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b1bc0 msgr2=0x7f7c2015dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.501+0000 7f7c251ff700 1 --2- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b1bc0 0x7f7c2015dc10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009e90 tx=0x7f7c18009780 comp rx=0 tx=0).stop 2023-10-07T09:29:36.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.502+0000 7f7c251ff700 1 -- 172.21.15.29:0/1027252261 shutdown_connections 2023-10-07T09:29:36.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.502+0000 7f7c251ff700 1 --2- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b1bc0 0x7f7c2015dc10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.502+0000 7f7c251ff700 1 --2- 172.21.15.29:0/1027252261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b17f0 0x7f7c200afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.502+0000 7f7c251ff700 1 -- 172.21.15.29:0/1027252261 >> 172.21.15.29:0/1027252261 conn(0x7f7c200680a0 msgr2=0x7f7c200cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.502+0000 7f7c251ff700 1 -- 172.21.15.29:0/1027252261 shutdown_connections 2023-10-07T09:29:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.503+0000 7f7c251ff700 1 -- 172.21.15.29:0/1027252261 wait complete. 2023-10-07T09:29:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.503+0000 7fbf8effd700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbf70066f30 0x7fbf700693e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf80007460 tx=0x7fbf80009a10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.504+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf880556b0 con 0x7fbf90157580 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.505+0000 7f7c251ff700 1 Processor -- start 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.506+0000 7f7c251ff700 1 -- start start 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.507+0000 7f7c251ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b17f0 0x7f7c20142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.507+0000 7f7c251ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 0x7f7c20155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.507+0000 7f7c251ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c200cc870 0x7f7c2015be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.507+0000 7f7c251ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c201525a0 con 0x7f7c200cc870 2023-10-07T09:29:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.507+0000 7f7c251ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20156170 con 0x7f7c200b17f0 2023-10-07T09:29:36.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.507+0000 7f7c251ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20156450 con 0x7f7c200b1bc0 2023-10-07T09:29:36.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.510+0000 7f7c1659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 0x7f7c20155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.510+0000 7f7c1659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 0x7f7c20155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44552/0 (socket says 172.21.15.29:44552) 2023-10-07T09:29:36.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.510+0000 7f7c1659c700 1 -- 172.21.15.29:0/198487940 learned_addr learned my addr 172.21.15.29:0/198487940 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.510+0000 7f7c1ed9d700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b17f0 0x7f7c20142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.511+0000 7f7c1f59e700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c200cc870 0x7f7c2015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.513+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbfd8012070 con 0x7fbfe015db60 2023-10-07T09:29:36.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.513+0000 7f7c1659c700 1 -- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b17f0 msgr2=0x7f7c20142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.513+0000 7f7c1659c700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b17f0 0x7f7c20142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.513+0000 7f7c1659c700 1 -- 172.21.15.29:0/198487940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c200cc870 msgr2=0x7f7c2015be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.513+0000 7f7c1659c700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c200cc870 0x7f7c2015be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.514+0000 7f7c1659c700 1 -- 172.21.15.29:0/198487940 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18009a20 con 0x7f7c200b1bc0 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.514+0000 7f7c1659c700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 0x7f7c20155720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000c560 tx=0x7f7c10001020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.515+0000 7fbfc77fe700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbfcc066fb0 0x7fbfcc069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.515+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbfd80842f0 con 0x7fbfe015db60 2023-10-07T09:29:36.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.515+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfd8051600 con 0x7fbfe015db60 2023-10-07T09:29:36.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.516+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10005b30 con 0x7f7c200b1bc0 2023-10-07T09:29:36.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.516+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10005c90 con 0x7f7c200b1bc0 2023-10-07T09:29:36.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.516+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10007300 con 0x7f7c200b1bc0 2023-10-07T09:29:36.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.516+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c20156730 con 0x7f7c200b1bc0 2023-10-07T09:29:36.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.516+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2014f880 con 0x7f7c200b1bc0 2023-10-07T09:29:36.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.516+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbfcc066fb0 0x7fbfcc069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.519+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c040052b0 con 0x7f7c200b1bc0 2023-10-07T09:29:36.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.520+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbfcc066fb0 0x7fbfcc069460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009260 tx=0x7fbfd0015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.520+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c1001b3b0 con 0x7f7c200b1bc0 2023-10-07T09:29:36.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.522+0000 7f7c177fe700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c00067080 0x7f7c00069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.522+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7c10083940 con 0x7f7c200b1bc0 2023-10-07T09:29:36.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.523+0000 7f7c1ed9d700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c00067080 0x7f7c00069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.524+0000 7f7c1ed9d700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c00067080 0x7f7c00069530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009d40 tx=0x7f7c08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.526+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c1004b5a0 con 0x7f7c200b1bc0 2023-10-07T09:29:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.545+0000 7f2fa15ff700 1 Processor -- start 2023-10-07T09:29:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.545+0000 7f2fa15ff700 1 -- start start 2023-10-07T09:29:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.549+0000 7f5245bff700 1 Processor -- start 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.550+0000 7f2fa15ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f9c0b1a10 0x7f2f9c1552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.550+0000 7f2fa15ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0b1e60 con 0x7f2f9c155800 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.550+0000 7f2fa15ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c1572e0 con 0x7f2f9c06bae0 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.550+0000 7f2fa15ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c1567c0 con 0x7f2f9c0b1a10 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.551+0000 7f2f9b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2f9c155800 0x7f2f9c0afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58720/0 (socket says 172.21.15.29:58720) 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.551+0000 7f2f9b7fe700 1 -- 172.21.15.29:0/4085250484 learned_addr learned my addr 172.21.15.29:0/4085250484 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483590230 0 0) 0x7f2f9c0b1e60 con 0x7f2f9c155800 2023-10-07T09:29:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f9a7fc700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f9c0b1a10 0x7f2f9c1552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f84003570 con 0x7f2f9c155800 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 443753780 0 0) 0x7f2f84003570 con 0x7f2f9c155800 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f9c0b1a10 msgr2=0x7f2f9c1552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f9c0b1a10 0x7f2f9c1552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 >> v1:172.21.15.144:6789/0 conn(0x7f2f9c06bae0 legacy=0x7f2f9c0c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c15c640 con 0x7f2f9c155800 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f2f8c001470 con 0x7f2f9c155800 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.552+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 >> v1:172.21.15.29:6789/0 conn(0x7f2f9c155800 legacy=0x7f2f9c0afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f2f99ffb700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f84003b10 0x7f2f84003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f2f99ffb700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f84004810 0x7f2f84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f2f99ffb700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f8400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8400c130 con 0x7f2f84004810 2023-10-07T09:29:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8400c270 con 0x7f2f84003b10 2023-10-07T09:29:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8400c430 con 0x7f2f84005290 2023-10-07T09:29:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.556+0000 7f2f9b7fe700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f8400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.556+0000 7f2f9a7fc700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f84004810 0x7f2f84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.551+0000 7f5245bff700 1 -- start start 2023-10-07T09:29:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f5245bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f524015d8c0 0x7f52400afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f5245bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f524015dc90 0x7f52400b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f5245bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401615b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f5245bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524015f2a0 con 0x7f524015dc90 2023-10-07T09:29:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f5245bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524006a3f0 con 0x7f524015d8c0 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.553+0000 7f5245bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52400b0000 con 0x7f52400b1ac0 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.554+0000 7f523effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f524015dc90 0x7f52400b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.554+0000 7f523effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f524015dc90 0x7f52400b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56604/0 (socket says 172.21.15.29:56604) 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.554+0000 7f523effd700 1 -- 172.21.15.29:0/1647164096 learned_addr learned my addr 172.21.15.29:0/1647164096 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.554+0000 7f523ffff700 1 --2- 172.21.15.29:0/1647164096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.554+0000 7f523f7fe700 1 --2- 172.21.15.29:0/1647164096 >> v2:172.21.15.144:3300/0 conn(0x7f524015d8c0 0x7f52400afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.555+0000 7f523effd700 1 -- 172.21.15.29:0/1647164096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 msgr2=0x7f52401615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.555+0000 7f523effd700 1 --2- 172.21.15.29:0/1647164096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401615b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.555+0000 7f523effd700 1 -- 172.21.15.29:0/1647164096 >> v2:172.21.15.144:3300/0 conn(0x7f524015d8c0 msgr2=0x7f52400afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.555+0000 7f523effd700 1 --2- 172.21.15.29:0/1647164096 >> v2:172.21.15.144:3300/0 conn(0x7f524015d8c0 0x7f52400afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.555+0000 7f523effd700 1 -- 172.21.15.29:0/1647164096 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52400af3d0 con 0x7f524015dc90 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.555+0000 7f523effd700 1 --2- 172.21.15.29:0/1647164096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015dc90 0x7f52400b1580 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5240068eb0 tx=0x7f5234009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faeac2c9298d3e43 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.556+0000 7f523e7fc700 1 -- 172.21.15.29:0/1647164096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523400b860 con 0x7f524015dc90 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.556+0000 7f523e7fc700 1 -- 172.21.15.29:0/1647164096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523400b9c0 con 0x7f524015dc90 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.556+0000 7f523e7fc700 1 -- 172.21.15.29:0/1647164096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523400bd60 con 0x7f524015dc90 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f5245bff700 1 -- 172.21.15.29:0/1647164096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015dc90 msgr2=0x7f52400b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f5245bff700 1 --2- 172.21.15.29:0/1647164096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015dc90 0x7f52400b1580 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5240068eb0 tx=0x7f5234009250 comp rx=0 tx=0).stop 2023-10-07T09:29:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f2f9b7fe700 1 -- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f84003b10 msgr2=0x7f2f84003f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f2f9b7fe700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f84003b10 0x7f2f84003f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f2f9b7fe700 1 -- 172.21.15.29:0/4085250484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f84004810 msgr2=0x7f2f84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f2f9b7fe700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f84004810 0x7f2f84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.557+0000 7f2f9b7fe700 1 -- 172.21.15.29:0/4085250484 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c15c640 con 0x7f2f84005290 2023-10-07T09:29:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 -- 172.21.15.29:0/1647164096 shutdown_connections 2023-10-07T09:29:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 --2- 172.21.15.29:0/1647164096 >> v2:172.21.15.144:3300/0 conn(0x7f524015d8c0 0x7f52400afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 --2- 172.21.15.29:0/1647164096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 --2- 172.21.15.29:0/1647164096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015dc90 0x7f52400b1580 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 -- 172.21.15.29:0/1647164096 >> 172.21.15.29:0/1647164096 conn(0x7f52400ccc30 msgr2=0x7f5240069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 -- 172.21.15.29:0/1647164096 shutdown_connections 2023-10-07T09:29:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.558+0000 7f5245bff700 1 -- 172.21.15.29:0/1647164096 wait complete. 2023-10-07T09:29:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.560+0000 7f2f9b7fe700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f8400ba60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00c3e0 tx=0x7f2f8c001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503080b7ffe4b60a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.561+0000 7f5245bff700 1 Processor -- start 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.560+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c00ccb0 con 0x7f2f84005290 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.561+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c00ce30 con 0x7f2f84005290 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.561+0000 7f2f99ffb700 1 -- 172.21.15.29:0/4085250484 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c005b70 con 0x7f2f84005290 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f2fa15ff700 1 -- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 msgr2=0x7f2f8400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/4085250484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f8400ba60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00c3e0 tx=0x7f2f8c001e60 comp rx=0 tx=0).stop 2023-10-07T09:29:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f2f9b7fe700 1 -- 172.21.15.29:0/4085250484 reap_dead start 2023-10-07T09:29:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f2fa15ff700 1 -- 172.21.15.29:0/4085250484 shutdown_connections 2023-10-07T09:29:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f2fa15ff700 1 -- 172.21.15.29:0/4085250484 >> 172.21.15.29:0/4085250484 conn(0x7f2f9c0ccc30 msgr2=0x7f2f9c069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.561+0000 7f5245bff700 1 -- start start 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f5245bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f5245bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 0x7f5240148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f5245bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f524015dc90 0x7f524014bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f5245bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5240150200 con 0x7f524015d8c0 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f5245bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52400cbd30 con 0x7f524015dc90 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.562+0000 7f5245bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52400cb790 con 0x7f52400b1ac0 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.564+0000 7f523effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 0x7f5240148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.564+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.564+0000 7f523effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 0x7f5240148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56616/0 (socket says 172.21.15.29:56616) 2023-10-07T09:29:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.564+0000 7f523effd700 1 -- 172.21.15.29:0/2479295852 learned_addr learned my addr 172.21.15.29:0/2479295852 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.564+0000 7f523ffff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f524015dc90 0x7f524014bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.564+0000 7f2fa15ff700 1 -- 172.21.15.29:0/4085250484 shutdown_connections 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.565+0000 7f523effd700 1 -- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 msgr2=0x7f52401690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.565+0000 7f523effd700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.565+0000 7f523effd700 1 -- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f524015dc90 msgr2=0x7f524014bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.565+0000 7f523effd700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f524015dc90 0x7f524014bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.565+0000 7f523effd700 1 -- 172.21.15.29:0/2479295852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52340094f0 con 0x7f524015d8c0 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.566+0000 7f523effd700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 0x7f5240148ef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5234009290 tx=0x7f5234039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.566+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234033b70 con 0x7f524015d8c0 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.566+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234033cd0 con 0x7f524015d8c0 2023-10-07T09:29:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.566+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523403c6a0 con 0x7f524015d8c0 2023-10-07T09:29:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.567+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52400cba50 con 0x7f524015d8c0 2023-10-07T09:29:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.567+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524014aec0 con 0x7f524015d8c0 2023-10-07T09:29:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.565+0000 7f2fa15ff700 1 -- 172.21.15.29:0/4085250484 wait complete. 2023-10-07T09:29:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.567+0000 7f2fa15ff700 1 Processor -- start 2023-10-07T09:29:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.567+0000 7f2fa15ff700 1 -- start start 2023-10-07T09:29:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.569+0000 7f2fa15ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f9c144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.569+0000 7f2fa15ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 0x7f2f9c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.569+0000 7f2fa15ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f9c159930 0x7f2f9c159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.569+0000 7f2fa15ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c152390 con 0x7f2f9c0caad0 2023-10-07T09:29:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.569+0000 7f2fa15ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0cb480 con 0x7f2f9c159930 2023-10-07T09:29:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.569+0000 7f2fa15ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c15b2e0 con 0x7f2f84005290 2023-10-07T09:29:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.570+0000 7f2f9b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f9c159930 0x7f2f9c159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.570+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f523403cad0 con 0x7f524015d8c0 2023-10-07T09:29:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.571+0000 7f2f9a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 0x7f2f9c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.571+0000 7f2f9a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 0x7f2f9c0caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56620/0 (socket says 172.21.15.29:56620) 2023-10-07T09:29:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.571+0000 7f2f9a7fc700 1 -- 172.21.15.29:0/1404845616 learned_addr learned my addr 172.21.15.29:0/1404845616 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f52277fe700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f522c067080 0x7f522c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f52340a7df0 con 0x7f524015d8c0 2023-10-07T09:29:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f2f9a7fc700 1 -- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 msgr2=0x7f2f9c144450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:29:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f2f9a7fc700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f9c144450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f2f9a7fc700 1 -- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f9c159930 msgr2=0x7f2f9c159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f2f9a7fc700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f9c159930 0x7f2f9c159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.572+0000 7f2f9a7fc700 1 -- 172.21.15.29:0/1404845616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c144990 con 0x7f2f9c0caad0 2023-10-07T09:29:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.573+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52200052b0 con 0x7f524015d8c0 2023-10-07T09:29:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.573+0000 7f2f9a7fc700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 0x7f2f9c0caf40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000f5e0 tx=0x7f2f90008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.577+0000 7f523f7fe700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f522c067080 0x7f522c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.578+0000 7f523f7fe700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f522c067080 0x7f522c069530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5238007460 tx=0x7f523800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.583+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5234075040 con 0x7f524015d8c0 2023-10-07T09:29:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.582+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f90013610 con 0x7f2f9c0caad0 2023-10-07T09:29:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.582+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f90013770 con 0x7f2f9c0caad0 2023-10-07T09:29:36.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.582+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f90013b30 con 0x7f2f9c0caad0 2023-10-07T09:29:36.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.582+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c15b770 con 0x7f2f9c0caad0 2023-10-07T09:29:36.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.582+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c15a9b0 con 0x7f2f9c0caad0 2023-10-07T09:29:36.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.588+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f90015410 con 0x7f2f9c0caad0 2023-10-07T09:29:36.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.590+0000 7f2f82ffd700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2f84068cb0 0x7f2f8406b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:36.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.590+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2f90084580 con 0x7f2f9c0caad0 2023-10-07T09:29:36.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.591+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c0052b0 con 0x7f2f9c0caad0 2023-10-07T09:29:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.598+0000 7f2f9affd700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2f84068cb0 0x7f2f8406b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.600+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f900517d0 con 0x7f2f9c0caad0 2023-10-07T09:29:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.611+0000 7f2f9affd700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2f84068cb0 0x7f2f8406b160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f94002d10 tx=0x7f2f94007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.854+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f01d4167790 con 0x7f01d40ae7f0 2023-10-07T09:29:36.859 INFO:teuthology.orchestra.run.smithi029.stdout:60129542151 2023-10-07T09:29:36.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.855+0000 7f01bb7fe700 1 -- 172.21.15.29:0/3794349934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f01c40514b0 con 0x7f01d40ae7f0 2023-10-07T09:29:36.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f01c0067080 msgr2=0x7f01c0069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f01c0067080 0x7f01c0069530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01c80099b0 tx=0x7f01c8017000 comp rx=0 tx=0).stop 2023-10-07T09:29:36.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 msgr2=0x7f01d416d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 0x7f01d416d6c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01c400b810 tx=0x7f01c40091d0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 shutdown_connections 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f01c0067080 0x7f01c0069530 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01c80099b0 tx=0x7f01c8017000 comp rx=0 tx=0).stop 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01d40b1580 0x7f01d4159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d40ae7f0 0x7f01d416d6c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 --2- 172.21.15.29:0/3794349934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01d40afac0 0x7f01d4159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.860+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 >> 172.21.15.29:0/3794349934 conn(0x7f01d40ccc30 msgr2=0x7f01d406d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.861+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 shutdown_connections 2023-10-07T09:29:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.861+0000 7f01d9dff700 1 -- 172.21.15.29:0/3794349934 wait complete. 2023-10-07T09:29:36.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2023-10-07T09:29:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.921+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff8ec0c2c20 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.922+0000 7ff8d2ffd700 1 -- 172.21.15.29:0/1634152046 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff8ec0c2c20 con 0x7ff8ec15d8c0 2023-10-07T09:29:36.926 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-10-07T09:29:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.927+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff8d8067080 msgr2=0x7ff8d8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.927+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff8d8067080 0x7ff8d8069530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4005c50 tx=0x7ff8e4009040 comp rx=0 tx=0).stop 2023-10-07T09:29:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.927+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 msgr2=0x7ff8ec161b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.927+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 0x7ff8ec161b00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d860 tx=0x7ff8dc008040 comp rx=0 tx=0).stop 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 shutdown_connections 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff8d8067080 0x7ff8d8069530 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8ec15dc90 0x7ff8ec162040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8ec0b1ac0 0x7ff8ec171040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 --2- 172.21.15.29:0/1634152046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ec15d8c0 0x7ff8ec161b00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 >> 172.21.15.29:0/1634152046 conn(0x7ff8ec0ccc30 msgr2=0x7ff8ec06e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 shutdown_connections 2023-10-07T09:29:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.928+0000 7ff8f15ff700 1 -- 172.21.15.29:0/1634152046 wait complete. 2023-10-07T09:29:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.934+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fbfe004c3c0 con 0x7fbfe015db60 2023-10-07T09:29:36.936 INFO:teuthology.orchestra.run.smithi029.stdout:64424509447 2023-10-07T09:29:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.935+0000 7fbfc77fe700 1 -- 172.21.15.29:0/2463488360 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbfd8054c20 con 0x7fbfe015db60 2023-10-07T09:29:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.936+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff0ac04c3c0 con 0x7ff0ac0cc870 2023-10-07T09:29:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.937+0000 7ff0937fe700 1 -- 172.21.15.29:0/2822467796 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff09c055000 con 0x7ff0ac0cc870 2023-10-07T09:29:36.938 INFO:teuthology.orchestra.run.smithi029.stdout:64424509447 2023-10-07T09:29:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.939+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbfcc066fb0 msgr2=0x7fbfcc069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.939+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbfcc066fb0 0x7fbfcc069460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009260 tx=0x7fbfd0015000 comp rx=0 tx=0).stop 2023-10-07T09:29:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.939+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 msgr2=0x7fbfe0160d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 0x7fbfe0160d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800bbb0 tx=0x7fbfd8009840 comp rx=0 tx=0).stop 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 shutdown_connections 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbfcc066fb0 0x7fbfcc069460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfe015df30 0x7fbfe00cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfe00ce8a0 0x7fbfe016dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 --2- 172.21.15.29:0/2463488360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe015db60 0x7fbfe0160d90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 >> 172.21.15.29:0/2463488360 conn(0x7fbfe0063a10 msgr2=0x7fbfe015a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 shutdown_connections 2023-10-07T09:29:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7fbfe5fff700 1 -- 172.21.15.29:0/2463488360 wait complete. 2023-10-07T09:29:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff098067080 msgr2=0x7ff098069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff098067080 0x7ff098069530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0003870 tx=0x7ff0a0009040 comp rx=0 tx=0).stop 2023-10-07T09:29:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 msgr2=0x7ff0ac06db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.940+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac06db20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00e960 tx=0x7ff09c009250 comp rx=0 tx=0).stop 2023-10-07T09:29:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.941+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 shutdown_connections 2023-10-07T09:29:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.941+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff098067080 0x7ff098069530 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.941+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ac0cc870 0x7ff0ac06db20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.941+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0ac0b17f0 0x7ff0ac152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.941+0000 7ff0b1fff700 1 --2- 172.21.15.29:0/2822467796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0ac0b1bc0 0x7ff0ac13ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.941+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 >> 172.21.15.29:0/2822467796 conn(0x7ff0ac0680a0 msgr2=0x7ff0ac15a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.943+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 shutdown_connections 2023-10-07T09:29:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.943+0000 7ff0b1fff700 1 -- 172.21.15.29:0/2822467796 wait complete. 2023-10-07T09:29:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.953+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbf900ae920 con 0x7fbf90157580 2023-10-07T09:29:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.954+0000 7fbf86ffd700 1 -- 172.21.15.29:0/948844139 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbf88055240 con 0x7fbf90157580 2023-10-07T09:29:36.958 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-10-07T09:29:36.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.2 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbf70066f30 msgr2=0x7fbf700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbf70066f30 0x7fbf700693e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf80007460 tx=0x7fbf80009a10 comp rx=0 tx=0).stop 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 msgr2=0x7fbf900c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 0x7fbf900c1da0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b5e0 tx=0x7fbf88009120 comp rx=0 tx=0).stop 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 shutdown_connections 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbf70066f30 0x7fbf700693e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf90157580 0x7fbf900c1da0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf900b1ac0 0x7fbf9016f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 --2- 172.21.15.29:0/948844139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf9015d8c0 0x7fbf90162090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 >> 172.21.15.29:0/948844139 conn(0x7fbf900ccc30 msgr2=0x7fbf901592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 shutdown_connections 2023-10-07T09:29:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.959+0000 7fbf959ff700 1 -- 172.21.15.29:0/948844139 wait complete. 2023-10-07T09:29:36.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.4 2023-10-07T09:29:36.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.973+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7c04005560 con 0x7f7c200b1bc0 2023-10-07T09:29:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.974+0000 7f7c177fe700 1 -- 172.21.15.29:0/198487940 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c10017090 con 0x7f7c200b1bc0 2023-10-07T09:29:36.975 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-10-07T09:29:36.976 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.6 2023-10-07T09:29:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c00067080 msgr2=0x7f7c00069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c00067080 0x7f7c00069530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009d40 tx=0x7f7c08008040 comp rx=0 tx=0).stop 2023-10-07T09:29:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 msgr2=0x7f7c20155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 0x7f7c20155720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000c560 tx=0x7f7c10001020 comp rx=0 tx=0).stop 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 shutdown_connections 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c00067080 0x7f7c00069530 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c200cc870 0x7f7c2015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c200b17f0 0x7f7c20142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 --2- 172.21.15.29:0/198487940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c200b1bc0 0x7f7c20155720 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 >> 172.21.15.29:0/198487940 conn(0x7f7c200680a0 msgr2=0x7f7c2015a550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 shutdown_connections 2023-10-07T09:29:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.977+0000 7f7c251ff700 1 -- 172.21.15.29:0/198487940 wait complete. 2023-10-07T09:29:36.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.0 2023-10-07T09:29:36.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.1 2023-10-07T09:29:37.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.999+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5220005560 con 0x7f524015d8c0 2023-10-07T09:29:37.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:36.999+0000 7f52277fe700 1 -- 172.21.15.29:0/2479295852 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5234037020 con 0x7f524015d8c0 2023-10-07T09:29:37.000 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-10-07T09:29:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.001+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f522c067080 msgr2=0x7f522c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.001+0000 7f5245bff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f522c067080 0x7f522c069530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5238007460 tx=0x7f523800b040 comp rx=0 tx=0).stop 2023-10-07T09:29:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.001+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 msgr2=0x7f5240148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.001+0000 7f5245bff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 0x7f5240148ef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5234009290 tx=0x7f5234039000 comp rx=0 tx=0).stop 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.001+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 shutdown_connections 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f522c067080 0x7f522c069530 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f524015dc90 0x7f524014bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52400b1ac0 0x7f52401690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 --2- 172.21.15.29:0/2479295852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f524015d8c0 0x7f5240148ef0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 >> 172.21.15.29:0/2479295852 conn(0x7f52400ccc30 msgr2=0x7f524006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 shutdown_connections 2023-10-07T09:29:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.002+0000 7f5245bff700 1 -- 172.21.15.29:0/2479295852 wait complete. 2023-10-07T09:29:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.008+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2f7c005560 con 0x7f2f9c0caad0 2023-10-07T09:29:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.008+0000 7f2f82ffd700 1 -- 172.21.15.29:0/1404845616 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f90054df0 con 0x7f2f9c0caad0 2023-10-07T09:29:37.008 INFO:teuthology.orchestra.run.smithi029.stdout:68719476743 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2f84068cb0 msgr2=0x7f2f8406b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2f84068cb0 0x7f2f8406b160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f94002d10 tx=0x7f2f94007ea0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 msgr2=0x7f2f9c0caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 0x7f2f9c0caf40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000f5e0 tx=0x7f2f90008040 comp rx=0 tx=0).stop 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 shutdown_connections 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2f84068cb0 0x7f2f8406b160 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f9c159930 0x7f2f9c159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f84005290 0x7f2f9c144450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 --2- 172.21.15.29:0/1404845616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f9c0caad0 0x7f2f9c0caf40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 >> 172.21.15.29:0/1404845616 conn(0x7f2f9c0ccc30 msgr2=0x7f2f9c1547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 shutdown_connections 2023-10-07T09:29:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:37.010+0000 7f2fa15ff700 1 -- 172.21.15.29:0/1404845616 wait complete. 2023-10-07T09:29:37.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2023-10-07T09:29:37.029 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476743 for osd.7 2023-10-07T09:29:37.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-07T09:29:37.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-07T09:29:37.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-07T09:29:37.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-07T09:29:37.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-07T09:29:37.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-07T09:29:38.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-07T09:29:38.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-07T09:29:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.172+0000 7f4c919ff700 1 Processor -- start 2023-10-07T09:29:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.172+0000 7f4c919ff700 1 -- start start 2023-10-07T09:29:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.174+0000 7f4c919ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b17f0 0x7f4c8c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.174+0000 7f4c919ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4c8c0cc870 0x7f4c8c15d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.174+0000 7f4c919ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c15eeb0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.174+0000 7f4c919ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c0aebe0 con 0x7f4c8c0cc870 2023-10-07T09:29:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.174+0000 7f4c919ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c15e2a0 con 0x7f4c8c0b17f0 2023-10-07T09:29:38.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.178+0000 7f4c8a7fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4c8c0cc870 0x7f4c8c15d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.178+0000 7f4c8a7fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4c8c0cc870 0x7f4c8c15d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53064/0 (socket says 172.21.15.29:53064) 2023-10-07T09:29:38.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.178+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/784057674 learned_addr learned my addr 172.21.15.29:0/784057674 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.179+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b17f0 msgr2=0x7f4c8c0afcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.179+0000 7f4c8a7fc700 1 --2- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b17f0 0x7f4c8c0afcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.179+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/784057674 >> v1:172.21.15.29:6789/0 conn(0x7f4c8c0b1bc0 legacy=0x7f4c8c15da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.179+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/784057674 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c15e580 con 0x7f4c8c0cc870 2023-10-07T09:29:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.179+0000 7f4c8a7fc700 1 --2- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0cc870 0x7f4c8c15d500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0099e0 tx=0x7f4c7c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e6dd2c77ed2043 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.180+0000 7f4c89ffb700 1 -- 172.21.15.29:0/784057674 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c00b7e0 con 0x7f4c8c0cc870 2023-10-07T09:29:38.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.180+0000 7f4c89ffb700 1 -- 172.21.15.29:0/784057674 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c00b940 con 0x7f4c8c0cc870 2023-10-07T09:29:38.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.180+0000 7f4c89ffb700 1 -- 172.21.15.29:0/784057674 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c00bce0 con 0x7f4c8c0cc870 2023-10-07T09:29:38.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.181+0000 7f4c919ff700 1 -- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0cc870 msgr2=0x7f4c8c15d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.181+0000 7f4c919ff700 1 --2- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0cc870 0x7f4c8c15d500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0099e0 tx=0x7f4c7c009250 comp rx=0 tx=0).stop 2023-10-07T09:29:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.185+0000 7f4c919ff700 1 -- 172.21.15.29:0/784057674 shutdown_connections 2023-10-07T09:29:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.185+0000 7f4c919ff700 1 --2- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b17f0 0x7f4c8c0afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.185+0000 7f4c919ff700 1 --2- 172.21.15.29:0/784057674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0cc870 0x7f4c8c15d500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.185+0000 7f4c919ff700 1 -- 172.21.15.29:0/784057674 >> 172.21.15.29:0/784057674 conn(0x7f4c8c0680a0 msgr2=0x7f4c8c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.185+0000 7f4c919ff700 1 -- 172.21.15.29:0/784057674 shutdown_connections 2023-10-07T09:29:38.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.186+0000 7f4c919ff700 1 -- 172.21.15.29:0/784057674 wait complete. 2023-10-07T09:29:38.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.188+0000 7f4c919ff700 1 Processor -- start 2023-10-07T09:29:38.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.189+0000 7f4c919ff700 1 -- start start 2023-10-07T09:29:38.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.190+0000 7f4c919ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0b17f0 0x7f4c8c14ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.190+0000 7f4c919ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b1bc0 0x7f4c8c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.190+0000 7f4c919ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c8c0cc870 0x7f4c8c0c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.190+0000 7f4c919ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c14b290 con 0x7f4c8c0cc870 2023-10-07T09:29:38.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.190+0000 7f4c919ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0c2010 con 0x7f4c8c0b17f0 2023-10-07T09:29:38.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.190+0000 7f4c919ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0c1400 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.191+0000 7f4c8affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0b17f0 0x7f4c8c14ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.191+0000 7f4c8affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0b17f0 0x7f4c8c14ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53074/0 (socket says 172.21.15.29:53074) 2023-10-07T09:29:38.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.191+0000 7f4c8affd700 1 -- 172.21.15.29:0/1297577733 learned_addr learned my addr 172.21.15.29:0/1297577733 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.191+0000 7f4c8a7fc700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b1bc0 0x7f4c8c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.192+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0b17f0 msgr2=0x7f4c8c14ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.192+0000 7f4c8a7fc700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0b17f0 0x7f4c8c14ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.192+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/1297577733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c8c0cc870 msgr2=0x7f4c8c0c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.192+0000 7f4c8a7fc700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c8c0cc870 0x7f4c8c0c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.192+0000 7f4c8a7fc700 1 -- 172.21.15.29:0/1297577733 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0094f0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.193+0000 7f4c8a7fc700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b1bc0 0x7f4c8c0cb430 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0099b0 tx=0x7f4c7c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.199+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c011cc0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.199+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c020070 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.199+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c01b6a0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.200+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c0c16e0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.200+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c0c1ca0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.203+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c0c21a0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.207+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4c7c022030 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.209+0000 7f4c72ffd700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4c78067080 0x7f4c78069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.209+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4c7c0864e0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.211+0000 7f4c8affd700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4c78067080 0x7f4c78069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.212+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c7c0537b0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.241+0000 7f4c8affd700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4c78067080 0x7f4c78069530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009b80 tx=0x7f4c800091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.283+0000 7f3a9cdff700 1 Processor -- start 2023-10-07T09:29:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.283+0000 7f3a9cdff700 1 -- start start 2023-10-07T09:29:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.285+0000 7f3a9cdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3a9815dc90 0x7f3a980b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.285+0000 7f3a9cdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9815f2a0 con 0x7f3a9815d8c0 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.285+0000 7f3a9cdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9806a3f0 con 0x7f3a980b1ac0 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.285+0000 7f3a9cdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a980af3d0 con 0x7f3a9815dc90 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a95d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3a9815dc90 0x7f3a980b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a9659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3a9815d8c0 0x7f3a980afac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58734/0 (socket says 172.21.15.29:58734) 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a96d9d700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3a980b1ac0 0x7f3a9815f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39454/0 (socket says 172.21.15.29:39454) 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a9659c700 1 -- 172.21.15.29:0/1787477753 learned_addr learned my addr 172.21.15.29:0/1787477753 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a95d9b700 1 -- 172.21.15.29:0/1787477753 >> v1:172.21.15.144:6789/0 conn(0x7f3a980b1ac0 legacy=0x7f3a9815f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a95d9b700 1 -- 172.21.15.29:0/1787477753 >> v1:172.21.15.29:6789/0 conn(0x7f3a9815d8c0 legacy=0x7f3a980afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.287+0000 7f3a95d9b700 1 -- 172.21.15.29:0/1787477753 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9815ec50 con 0x7f3a9815dc90 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.288+0000 7f3a95d9b700 1 --2- 172.21.15.29:0/1787477753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815dc90 0x7f3a980b1580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009e60 tx=0x7f3a90009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f927c1095d6ff24 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.289+0000 7f3a9559a700 1 -- 172.21.15.29:0/1787477753 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9000e070 con 0x7f3a9815dc90 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.289+0000 7f3a9559a700 1 -- 172.21.15.29:0/1787477753 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9000bbb0 con 0x7f3a9815dc90 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.289+0000 7f3a9559a700 1 -- 172.21.15.29:0/1787477753 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90012430 con 0x7f3a9815dc90 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.290+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1787477753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815dc90 msgr2=0x7f3a980b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.290+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1787477753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815dc90 0x7f3a980b1580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009e60 tx=0x7f3a90009780 comp rx=0 tx=0).stop 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.290+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1787477753 shutdown_connections 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.290+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1787477753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815dc90 0x7f3a980b1580 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.290+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1787477753 >> 172.21.15.29:0/1787477753 conn(0x7f3a980ccc30 msgr2=0x7f3a98069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.291+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1787477753 shutdown_connections 2023-10-07T09:29:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.291+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1787477753 wait complete. 2023-10-07T09:29:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.293+0000 7f3a9cdff700 1 Processor -- start 2023-10-07T09:29:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.294+0000 7f3a9cdff700 1 -- start start 2023-10-07T09:29:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.295+0000 7f3a9cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a980b1ac0 0x7f3a9813a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.295+0000 7f3a9cdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815d8c0 0x7f3a981777e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.295+0000 7f3a9cdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9815dc90 0x7f3a980cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.295+0000 7f3a9cdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a980c8960 con 0x7f3a9815dc90 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.295+0000 7f3a9cdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a98138320 con 0x7f3a9815d8c0 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.295+0000 7f3a9cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a980cb310 con 0x7f3a980b1ac0 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.296+0000 7f3a9659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a980b1ac0 0x7f3a9813a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.296+0000 7f3a9659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a980b1ac0 0x7f3a9813a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44626/0 (socket says 172.21.15.29:44626) 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.296+0000 7f3a9659c700 1 -- 172.21.15.29:0/1967021577 learned_addr learned my addr 172.21.15.29:0/1967021577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.296+0000 7f3a95d9b700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815d8c0 0x7f3a981777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.296+0000 7f3a96d9d700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9815dc90 0x7f3a980cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.297+0000 7f3a95d9b700 1 -- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a980b1ac0 msgr2=0x7f3a9813a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.297+0000 7f3a95d9b700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a980b1ac0 0x7f3a9813a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.297+0000 7f3a95d9b700 1 -- 172.21.15.29:0/1967021577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9815dc90 msgr2=0x7f3a980cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.297+0000 7f3a95d9b700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9815dc90 0x7f3a980cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.297+0000 7f3a95d9b700 1 -- 172.21.15.29:0/1967021577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90009a20 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.298+0000 7f3a95d9b700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815d8c0 0x7f3a981777e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a90015000 tx=0x7f3a90018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.298+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90023070 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.298+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9001e430 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.298+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9001e7f0 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.298+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a980cb5b0 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.299+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a980c8e20 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.303+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3a9001ec20 con 0x7f3a9815d8c0 2023-10-07T09:29:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.305+0000 7f3a867fc700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3a78067080 0x7f3a78069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.305+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3a90085e30 con 0x7f3a9815d8c0 2023-10-07T09:29:38.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.306+0000 7f3a9659c700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3a78067080 0x7f3a78069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.306+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c0052b0 con 0x7f3a9815d8c0 2023-10-07T09:29:38.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.308+0000 7f3a9659c700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3a78067080 0x7f3a78069530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007880 tx=0x7f3a8c004a80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.315+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a90053080 con 0x7f3a9815d8c0 2023-10-07T09:29:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.354+0000 7f3184bff700 1 Processor -- start 2023-10-07T09:29:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.354+0000 7f3184bff700 1 -- start start 2023-10-07T09:29:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.356+0000 7f3184bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f318015d730 0x7f318015db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.356+0000 7f3184bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318015ee30 con 0x7f31800cc870 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.356+0000 7f3184bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318015e220 con 0x7f31800b1b20 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.356+0000 7f3184bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318015e040 con 0x7f318015d730 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.357+0000 7f317ed9d700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f31800b1b20 0x7f318015f2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39458/0 (socket says 172.21.15.29:39458) 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.357+0000 7f317ed9d700 1 -- 172.21.15.29:0/3191740167 learned_addr learned my addr 172.21.15.29:0/3191740167 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.358+0000 7f317e59c700 1 --2- 172.21.15.29:0/3191740167 >> v2:172.21.15.29:3300/0 conn(0x7f318015d730 0x7f318015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.358+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755135260 0 0) 0x7f318015e220 con 0x7f31800b1b20 2023-10-07T09:29:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.358+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3168003570 con 0x7f31800b1b20 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 734079781 0 0) 0x7f3168003570 con 0x7f31800b1b20 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 >> v2:172.21.15.29:3300/0 conn(0x7f318015d730 msgr2=0x7f318015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 --2- 172.21.15.29:0/3191740167 >> v2:172.21.15.29:3300/0 conn(0x7f318015d730 0x7f318015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 >> v1:172.21.15.29:6789/0 conn(0x7f31800cc870 legacy=0x7f31800b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318015e500 con 0x7f31800b1b20 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f31780017f0 con 0x7f31800b1b20 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 >> v1:172.21.15.144:6789/0 conn(0x7f31800b1b20 legacy=0x7f318015f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3168003a60 0x7f3168003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3168004810 0x7f3168004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31680051c0 0x7f316800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318015e040 con 0x7f31680051c0 2023-10-07T09:29:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316800c0d0 con 0x7f3168004810 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.359+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316800c290 con 0x7f3168003a60 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.360+0000 7f317ed9d700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31680051c0 0x7f316800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.360+0000 7f317e59c700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3168003a60 0x7f3168003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 -- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3168003a60 msgr2=0x7f3168003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3168003a60 0x7f3168003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 -- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3168004810 msgr2=0x7f3168004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3168004810 0x7f3168004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 -- 172.21.15.29:0/3191740167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318015e500 con 0x7f31680051c0 2023-10-07T09:29:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 -- 172.21.15.29:0/3191740167 reap_dead start 2023-10-07T09:29:38.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.357+0000 7f3d117ff700 1 Processor -- start 2023-10-07T09:29:38.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.361+0000 7f317ed9d700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31680051c0 0x7f316800ba60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f317800a990 tx=0x7f31780021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19065bf37ce48a35 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.362+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31780070d0 con 0x7f31680051c0 2023-10-07T09:29:38.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.362+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178007270 con 0x7f31680051c0 2023-10-07T09:29:38.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.362+0000 7f317d59a700 1 -- 172.21.15.29:0/3191740167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178007630 con 0x7f31680051c0 2023-10-07T09:29:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.363+0000 7f3d117ff700 1 -- start start 2023-10-07T09:29:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3184bff700 1 -- 172.21.15.29:0/3191740167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31680051c0 msgr2=0x7f316800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3184bff700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31680051c0 0x7f316800ba60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f317800a990 tx=0x7f31780021c0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3184bff700 1 -- 172.21.15.29:0/3191740167 shutdown_connections 2023-10-07T09:29:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3184bff700 1 --2- 172.21.15.29:0/3191740167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31680051c0 0x7f316800ba60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3184bff700 1 -- 172.21.15.29:0/3191740167 >> 172.21.15.29:0/3191740167 conn(0x7f31800680a0 msgr2=0x7f31800cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3d117ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c0b0180 0x7f3d0c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3d117ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c1572e0 con 0x7f3d0c155480 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3d117ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c1569b0 con 0x7f3d0c06bae0 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f3d117ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c1567c0 con 0x7f3d0c0b0180 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.366+0000 7f3d0a7fc700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3d0c06bae0 0x7f3d0c0afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39462/0 (socket says 172.21.15.29:39462) 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.366+0000 7f3d0a7fc700 1 -- 172.21.15.29:0/3290270341 learned_addr learned my addr 172.21.15.29:0/3290270341 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.366+0000 7f3d0affd700 1 --1- 172.21.15.29:0/3290270341 >> v1:172.21.15.29:6789/0 conn(0x7f3d0c155480 0x7f3d0c0c81f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58766/0 (socket says 172.21.15.29:58766) 2023-10-07T09:29:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.366+0000 7f3d0b7fe700 1 --2- 172.21.15.29:0/3290270341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c0b0180 0x7f3d0c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.365+0000 7f8746fff700 1 Processor -- start 2023-10-07T09:29:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.366+0000 7f8746fff700 1 -- start start 2023-10-07T09:29:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.367+0000 7f3d0b7fe700 1 -- 172.21.15.29:0/3290270341 >> v1:172.21.15.144:6789/0 conn(0x7f3d0c06bae0 legacy=0x7f3d0c0afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.367+0000 7f3d09ffb700 1 -- 172.21.15.29:0/3290270341 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284882313 0 0) 0x7f3d0c1569b0 con 0x7f3d0c06bae0 2023-10-07T09:29:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.367+0000 7f3d0b7fe700 1 -- 172.21.15.29:0/3290270341 >> v1:172.21.15.29:6789/0 conn(0x7f3d0c155480 legacy=0x7f3d0c0c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.367+0000 7f3d0b7fe700 1 -- 172.21.15.29:0/3290270341 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c156c90 con 0x7f3d0c0b0180 2023-10-07T09:29:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.366+0000 7f3184bff700 1 -- 172.21.15.29:0/3191740167 shutdown_connections 2023-10-07T09:29:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.368+0000 7f3184bff700 1 -- 172.21.15.29:0/3191740167 wait complete. 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.368+0000 7f8746fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8740156980 0x7f87400b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.368+0000 7f8746fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f874006b590 0x7f87400afd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.368+0000 7f8746fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8740157bd0 con 0x7f8740152ce0 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.368+0000 7f8746fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8740156fc0 con 0x7f874006b590 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.368+0000 7f8746fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87401572a0 con 0x7f8740156980 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8740156980 0x7f87400b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8740156980 0x7f87400b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53118/0 (socket says 172.21.15.29:53118) 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 -- 172.21.15.29:0/3206028442 learned_addr learned my addr 172.21.15.29:0/3206028442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 -- 172.21.15.29:0/3206028442 >> v2:172.21.15.29:3300/0 conn(0x7f874006b590 msgr2=0x7f87400afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 --2- 172.21.15.29:0/3206028442 >> v2:172.21.15.29:3300/0 conn(0x7f874006b590 0x7f87400afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 -- 172.21.15.29:0/3206028442 >> v1:172.21.15.29:6789/0 conn(0x7f8740152ce0 legacy=0x7f8740153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.369+0000 7f8744d9b700 1 -- 172.21.15.29:0/3206028442 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740157580 con 0x7f8740156980 2023-10-07T09:29:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.370+0000 7f3184bff700 1 Processor -- start 2023-10-07T09:29:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.370+0000 7f8744d9b700 1 --2- 172.21.15.29:0/3206028442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8740156980 0x7f87400b1a10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8740063cd0 tx=0x7f87380143c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f313f1b3789bf49 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.371+0000 7f3184bff700 1 -- start start 2023-10-07T09:29:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.370+0000 7f3d0b7fe700 1 --2- 172.21.15.29:0/3290270341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c0b0180 0x7f3d0c155850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009910 tx=0x7f3cfc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=431d2130ce68b4f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.371+0000 7f3d09ffb700 1 -- 172.21.15.29:0/3290270341 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00b760 con 0x7f3d0c0b0180 2023-10-07T09:29:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.371+0000 7f3d09ffb700 1 -- 172.21.15.29:0/3290270341 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc00b8c0 con 0x7f3d0c0b0180 2023-10-07T09:29:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.371+0000 7f3d09ffb700 1 -- 172.21.15.29:0/3290270341 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00bc60 con 0x7f3d0c0b0180 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3184bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31680051c0 0x7f31801434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3184bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 0x7f3180159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3184bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31801542f0 0x7f3180154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3184bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31801439e0 con 0x7f3180158f90 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3184bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800c9e00 con 0x7f31801542f0 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3184bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3180159940 con 0x7f31680051c0 2023-10-07T09:29:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3d117ff700 1 -- 172.21.15.29:0/3290270341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c0b0180 msgr2=0x7f3d0c155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.372+0000 7f3d117ff700 1 --2- 172.21.15.29:0/3290270341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c0b0180 0x7f3d0c155850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009910 tx=0x7f3cfc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f3d117ff700 1 -- 172.21.15.29:0/3290270341 shutdown_connections 2023-10-07T09:29:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f3d117ff700 1 --2- 172.21.15.29:0/3290270341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c0b0180 0x7f3d0c155850 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f3d117ff700 1 -- 172.21.15.29:0/3290270341 >> 172.21.15.29:0/3290270341 conn(0x7f3d0c0ccc30 msgr2=0x7f3d0c069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f317dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 0x7f3180159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f317dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 0x7f3180159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56672/0 (socket says 172.21.15.29:56672) 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f317dd9b700 1 -- 172.21.15.29:0/3675942100 learned_addr learned my addr 172.21.15.29:0/3675942100 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f317e59c700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31680051c0 0x7f31801434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317e59c700 1 -- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31801542f0 msgr2=0x7f3180154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317e59c700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31801542f0 0x7f3180154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317e59c700 1 -- 172.21.15.29:0/3675942100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 msgr2=0x7f3180159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317e59c700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 0x7f3180159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317e59c700 1 -- 172.21.15.29:0/3675942100 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31801550a0 con 0x7f31680051c0 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317dd9b700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 0x7f3180159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.374+0000 7f317e59c700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31680051c0 0x7f31801434a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f31800650f0 tx=0x7f31740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f3d117ff700 1 -- 172.21.15.29:0/3290270341 shutdown_connections 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.373+0000 7f3d117ff700 1 -- 172.21.15.29:0/3290270341 wait complete. 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.375+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317400f830 con 0x7f31680051c0 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.375+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317400f990 con 0x7f31680051c0 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.375+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317400fd10 con 0x7f31680051c0 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.376+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3180155380 con 0x7f31680051c0 2023-10-07T09:29:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.376+0000 7f3d117ff700 1 Processor -- start 2023-10-07T09:29:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.376+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3180155940 con 0x7f31680051c0 2023-10-07T09:29:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.376+0000 7f3d117ff700 1 -- start start 2023-10-07T09:29:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31800c9bd0 con 0x7f31680051c0 2023-10-07T09:29:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.377+0000 7f3d117ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c06bae0 0x7f3d0c148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.377+0000 7f3d117ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c0b0180 0x7f3d0c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.377+0000 7f3d117ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c155480 0x7f3d0c1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d117ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c146840 con 0x7f3d0c06bae0 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d117ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c145c70 con 0x7f3d0c0b0180 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d117ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c145f10 con 0x7f3d0c155480 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d0a7fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c0b0180 0x7f3d0c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d0a7fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c0b0180 0x7f3d0c151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53140/0 (socket says 172.21.15.29:53140) 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d0a7fc700 1 -- 172.21.15.29:0/2543469586 learned_addr learned my addr 172.21.15.29:0/2543469586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.378+0000 7f3d0b7fe700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c155480 0x7f3d0c1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.379+0000 7f3d0affd700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c06bae0 0x7f3d0c148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f3d0b7fe700 1 -- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c0b0180 msgr2=0x7f3d0c151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f3d0b7fe700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c0b0180 0x7f3d0c151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f3d0b7fe700 1 -- 172.21.15.29:0/2543469586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c06bae0 msgr2=0x7f3d0c148220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f3d0b7fe700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c06bae0 0x7f3d0c148220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f3d0b7fe700 1 -- 172.21.15.29:0/2543469586 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc009420 con 0x7f3d0c155480 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.377+0000 7f873f7fe700 1 -- 172.21.15.29:0/3206028442 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738013b90 con 0x7f8740156980 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.377+0000 7f873f7fe700 1 -- 172.21.15.29:0/3206028442 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738013cf0 con 0x7f8740156980 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.377+0000 7f873f7fe700 1 -- 172.21.15.29:0/3206028442 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738014090 con 0x7f8740156980 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.379+0000 7f8746fff700 1 -- 172.21.15.29:0/3206028442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8740156980 msgr2=0x7f87400b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.379+0000 7f8746fff700 1 --2- 172.21.15.29:0/3206028442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8740156980 0x7f87400b1a10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8740063cd0 tx=0x7f87380143c0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f8746fff700 1 -- 172.21.15.29:0/3206028442 shutdown_connections 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f8746fff700 1 --2- 172.21.15.29:0/3206028442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8740156980 0x7f87400b1a10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f8746fff700 1 --2- 172.21.15.29:0/3206028442 >> v2:172.21.15.29:3300/0 conn(0x7f874006b590 0x7f87400afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f8746fff700 1 -- 172.21.15.29:0/3206028442 >> 172.21.15.29:0/3206028442 conn(0x7f87400680a0 msgr2=0x7f87400cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3174014440 con 0x7f31680051c0 2023-10-07T09:29:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f3d0b7fe700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c155480 0x7f3d0c1522b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc015000 tx=0x7f3cfc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.380+0000 7f8746fff700 1 -- 172.21.15.29:0/3206028442 shutdown_connections 2023-10-07T09:29:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.381+0000 7f8746fff700 1 -- 172.21.15.29:0/3206028442 wait complete. 2023-10-07T09:29:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.382+0000 7f316e7fc700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3160067080 0x7f3160069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.382+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3174018070 con 0x7f31680051c0 2023-10-07T09:29:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.382+0000 7f317dd9b700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3160067080 0x7f3160069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.383+0000 7f8746fff700 1 Processor -- start 2023-10-07T09:29:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.383+0000 7f317dd9b700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3160067080 0x7f3160069530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3170007fd0 tx=0x7f3170009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.383+0000 7f8746fff700 1 -- start start 2023-10-07T09:29:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.386+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31800c9bd0 con 0x7f31680051c0 2023-10-07T09:29:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc01c460 con 0x7f3d0c155480 2023-10-07T09:29:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc01c5c0 con 0x7f3d0c155480 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc01c960 con 0x7f3d0c155480 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f8746fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f874006b590 0x7f8740144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f8746fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 0x7f8740166270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f8746fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8740156980 0x7f8740155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f8746fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87401667b0 con 0x7f8740156980 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f8746fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8740156720 con 0x7f874006b590 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.385+0000 7f8746fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8740154220 con 0x7f8740152ce0 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.386+0000 7f873ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 0x7f8740166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.386+0000 7f873ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 0x7f8740166270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44678/0 (socket says 172.21.15.29:44678) 2023-10-07T09:29:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.386+0000 7f873ffff700 1 -- 172.21.15.29:0/1849904927 learned_addr learned my addr 172.21.15.29:0/1849904927 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.386+0000 7f8744d9b700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f874006b590 0x7f8740144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.386+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c1461f0 con 0x7f3d0c155480 2023-10-07T09:29:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.387+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c146cd0 con 0x7f3d0c155480 2023-10-07T09:29:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.387+0000 7f873ffff700 1 -- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f874006b590 msgr2=0x7f8740144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.387+0000 7f873ffff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f874006b590 0x7f8740144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.387+0000 7f873ffff700 1 -- 172.21.15.29:0/1849904927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8740156980 msgr2=0x7f8740155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.387+0000 7f873ffff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8740156980 0x7f8740155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.387+0000 7f873ffff700 1 -- 172.21.15.29:0/1849904927 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738014600 con 0x7f8740152ce0 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f873ffff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 0x7f8740166270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f873000b840 tx=0x7f8730009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f99495ff700 1 Processor -- start 2023-10-07T09:29:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f99495ff700 1 -- start start 2023-10-07T09:29:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c1469b0 con 0x7f3d0c155480 2023-10-07T09:29:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000f510 con 0x7f8740152ce0 2023-10-07T09:29:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873000f670 con 0x7f8740152ce0 2023-10-07T09:29:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.388+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000fa30 con 0x7f8740152ce0 2023-10-07T09:29:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.389+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8740156460 con 0x7f8740152ce0 2023-10-07T09:29:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.390+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8740153bd0 con 0x7f8740152ce0 2023-10-07T09:29:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.390+0000 7f99495ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99440c6470 0x7f99440d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.390+0000 7f99495ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99440cefb0 con 0x7f9944159750 2023-10-07T09:29:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.390+0000 7f99495ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99440cbf20 con 0x7f9944159b20 2023-10-07T09:29:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.390+0000 7f99495ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99440cc200 con 0x7f99440c6470 2023-10-07T09:29:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.391+0000 7f9942ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9944159750 0x7f99440ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58770/0 (socket says 172.21.15.29:58770) 2023-10-07T09:29:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.391+0000 7f9942ffd700 1 -- 172.21.15.29:0/2740055110 learned_addr learned my addr 172.21.15.29:0/2740055110 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.392+0000 7f99437fe700 1 --2- 172.21.15.29:0/2740055110 >> v2:172.21.15.144:3300/0 conn(0x7f99440c6470 0x7f99440d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.393+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87401543b0 con 0x7f8740152ce0 2023-10-07T09:29:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.391+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3cfc01ccd0 con 0x7f3d0c155480 2023-10-07T09:29:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.393+0000 7f3cf2ffd700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3cf8067080 0x7f3cf8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.393+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3cfc086bf0 con 0x7f3d0c155480 2023-10-07T09:29:38.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.393+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276183210 0 0) 0x7f99440cefb0 con 0x7f9944159750 2023-10-07T09:29:38.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.393+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992c003570 con 0x7f9944159750 2023-10-07T09:29:38.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.394+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492375743 0 0) 0x7f99440cbf20 con 0x7f9944159b20 2023-10-07T09:29:38.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.394+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99440cefb0 con 0x7f9944159b20 2023-10-07T09:29:38.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.394+0000 7f3d0affd700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3cf8067080 0x7f3cf8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.394+0000 7f99437fe700 1 -- 172.21.15.29:0/2740055110 >> v1:172.21.15.144:6789/0 conn(0x7f9944159b20 legacy=0x7f99440ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.394+0000 7f99437fe700 1 -- 172.21.15.29:0/2740055110 >> v1:172.21.15.29:6789/0 conn(0x7f9944159750 legacy=0x7f99440ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.394+0000 7f99437fe700 1 -- 172.21.15.29:0/2740055110 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440cc4e0 con 0x7f99440c6470 2023-10-07T09:29:38.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.395+0000 7f99437fe700 1 --2- 172.21.15.29:0/2740055110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99440c6470 0x7f99440d10a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99340098e0 tx=0x7f99340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b95c9d39f94313b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.395+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3cfc053ec0 con 0x7f3d0c155480 2023-10-07T09:29:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.395+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c001840 con 0x7f99440c6470 2023-10-07T09:29:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.395+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9934011420 con 0x7f99440c6470 2023-10-07T09:29:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.395+0000 7f9941ffb700 1 -- 172.21.15.29:0/2740055110 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99340117c0 con 0x7f99440c6470 2023-10-07T09:29:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.396+0000 7f99495ff700 1 -- 172.21.15.29:0/2740055110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99440c6470 msgr2=0x7f99440d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.396+0000 7f99495ff700 1 --2- 172.21.15.29:0/2740055110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99440c6470 0x7f99440d10a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99340098e0 tx=0x7f99340091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.396+0000 7f99495ff700 1 -- 172.21.15.29:0/2740055110 shutdown_connections 2023-10-07T09:29:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.396+0000 7f99495ff700 1 --2- 172.21.15.29:0/2740055110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99440c6470 0x7f99440d10a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.396+0000 7f99495ff700 1 -- 172.21.15.29:0/2740055110 >> 172.21.15.29:0/2740055110 conn(0x7f9944063a10 msgr2=0x7f99440d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.397+0000 7f3d0affd700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3cf8067080 0x7f3cf8069530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c145aa0 tx=0x7f3d00007cd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.397+0000 7f99495ff700 1 -- 172.21.15.29:0/2740055110 shutdown_connections 2023-10-07T09:29:38.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.397+0000 7f99495ff700 1 -- 172.21.15.29:0/2740055110 wait complete. 2023-10-07T09:29:38.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.397+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8730024430 con 0x7f8740152ce0 2023-10-07T09:29:38.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.399+0000 7f99495ff700 1 Processor -- start 2023-10-07T09:29:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.399+0000 7f873cff9700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f872c067080 0x7f872c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.400+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8730084c00 con 0x7f8740152ce0 2023-10-07T09:29:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.401+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8730055960 con 0x7f8740152ce0 2023-10-07T09:29:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.408+0000 7f8744d9b700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f872c067080 0x7f872c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.408+0000 7f99495ff700 1 -- start start 2023-10-07T09:29:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.410+0000 7f99495ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99440c6470 0x7f994414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.410+0000 7f99495ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9944159750 0x7f99441558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.410+0000 7f99495ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 0x7f9944155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.410+0000 7f99495ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99440b4260 con 0x7f99440c6470 2023-10-07T09:29:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.410+0000 7f99495ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99440b50c0 con 0x7f9944159b20 2023-10-07T09:29:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.410+0000 7f99495ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99440b36e0 con 0x7f9944159750 2023-10-07T09:29:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.412+0000 7f99437fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 0x7f9944155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.413+0000 7f8744d9b700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f872c067080 0x7f872c069530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8738014400 tx=0x7f873801e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.413+0000 7f99427fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9944159750 0x7f99441558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.413+0000 7f99437fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 0x7f9944155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53168/0 (socket says 172.21.15.29:53168) 2023-10-07T09:29:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.413+0000 7f99437fe700 1 -- 172.21.15.29:0/1481903912 learned_addr learned my addr 172.21.15.29:0/1481903912 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.414+0000 7f99437fe700 1 -- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9944159750 msgr2=0x7f99441558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.414+0000 7f99437fe700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9944159750 0x7f99441558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.414+0000 7f99437fe700 1 -- 172.21.15.29:0/1481903912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99440c6470 msgr2=0x7f994414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.414+0000 7f99437fe700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99440c6470 0x7f994414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.414+0000 7f99437fe700 1 -- 172.21.15.29:0/1481903912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9934009420 con 0x7f9944159b20 2023-10-07T09:29:38.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.415+0000 7f99437fe700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 0x7f9944155e10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9944088a40 tx=0x7f9934017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.415+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934010c40 con 0x7f9944159b20 2023-10-07T09:29:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.416+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9934010da0 con 0x7f9944159b20 2023-10-07T09:29:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.416+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993401d660 con 0x7f9944159b20 2023-10-07T09:29:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.416+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440b39c0 con 0x7f9944159b20 2023-10-07T09:29:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.416+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99440b3f80 con 0x7f9944159b20 2023-10-07T09:29:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.418+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994404c3c0 con 0x7f9944159b20 2023-10-07T09:29:38.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.422+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f993401d9d0 con 0x7f9944159b20 2023-10-07T09:29:38.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.424+0000 7f992affd700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9930067080 0x7f9930069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.425+0000 7f9942ffd700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9930067080 0x7f9930069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.425+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f9934086490 con 0x7f9944159b20 2023-10-07T09:29:38.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.425+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99340571f0 con 0x7f9944159b20 2023-10-07T09:29:38.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.426+0000 7f9942ffd700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9930067080 0x7f9930069530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f993c0015b0 tx=0x7f993c001470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.443+0000 7f3877dff700 1 Processor -- start 2023-10-07T09:29:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.443+0000 7f3877dff700 1 -- start start 2023-10-07T09:29:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.445+0000 7f3877dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f38700afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.445+0000 7f3877dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387015eeb0 con 0x7f38700cc870 2023-10-07T09:29:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.445+0000 7f3877dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38700aebe0 con 0x7f38700b17f0 2023-10-07T09:29:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.445+0000 7f3877dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387015e2a0 con 0x7f38700b1bc0 2023-10-07T09:29:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.446+0000 7f387639c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f38700afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.446+0000 7f387639c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f38700afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44694/0 (socket says 172.21.15.29:44694) 2023-10-07T09:29:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.446+0000 7f387639c700 1 -- 172.21.15.29:0/540426331 learned_addr learned my addr 172.21.15.29:0/540426331 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.447+0000 7f387639c700 1 -- 172.21.15.29:0/540426331 >> v1:172.21.15.144:6789/0 conn(0x7f38700b17f0 legacy=0x7f387015d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.447+0000 7f387639c700 1 -- 172.21.15.29:0/540426331 >> v1:172.21.15.29:6789/0 conn(0x7f38700cc870 legacy=0x7f387015f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.447+0000 7f387639c700 1 -- 172.21.15.29:0/540426331 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387015e580 con 0x7f38700b1bc0 2023-10-07T09:29:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.447+0000 7f387639c700 1 --2- 172.21.15.29:0/540426331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f38700afcb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38600099e0 tx=0x7f3860009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c8702115516f311 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.448+0000 7f3874b99700 1 -- 172.21.15.29:0/540426331 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386000b7e0 con 0x7f38700b1bc0 2023-10-07T09:29:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.448+0000 7f3874b99700 1 -- 172.21.15.29:0/540426331 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386000b940 con 0x7f38700b1bc0 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.448+0000 7f3874b99700 1 -- 172.21.15.29:0/540426331 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386000bce0 con 0x7f38700b1bc0 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.449+0000 7f3877dff700 1 -- 172.21.15.29:0/540426331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 msgr2=0x7f38700afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.449+0000 7f3877dff700 1 --2- 172.21.15.29:0/540426331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f38700afcb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38600099e0 tx=0x7f3860009250 comp rx=0 tx=0).stop 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.449+0000 7f3877dff700 1 -- 172.21.15.29:0/540426331 shutdown_connections 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.449+0000 7f3877dff700 1 --2- 172.21.15.29:0/540426331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f38700afcb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.449+0000 7f3877dff700 1 -- 172.21.15.29:0/540426331 >> 172.21.15.29:0/540426331 conn(0x7f38700680a0 msgr2=0x7f38700cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.449+0000 7fa5ef7ff700 1 Processor -- start 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.450+0000 7f3877dff700 1 -- 172.21.15.29:0/540426331 shutdown_connections 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.450+0000 7fa5ef7ff700 1 -- start start 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.452+0000 7fa5ef7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5e8159b20 0x7fa5e80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.452+0000 7fa5ef7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5e80ce8a0 0x7fa5e80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.452+0000 7fa5ef7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80cbf20 con 0x7fa5e8159750 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.452+0000 7fa5ef7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80cc200 con 0x7fa5e8159b20 2023-10-07T09:29:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.452+0000 7fa5ef7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80cc4e0 con 0x7fa5e80ce8a0 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.453+0000 7fa5edd9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5e80ce8a0 0x7fa5e80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.453+0000 7fa5edd9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5e80ce8a0 0x7fa5e80c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53176/0 (socket says 172.21.15.29:53176) 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.453+0000 7fa5edd9c700 1 -- 172.21.15.29:0/986387723 learned_addr learned my addr 172.21.15.29:0/986387723 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.453+0000 7fa5ecd9a700 1 --2- 172.21.15.29:0/986387723 >> v2:172.21.15.29:3300/0 conn(0x7fa5e8159b20 0x7fa5e80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.454+0000 7fa5edd9c700 1 -- 172.21.15.29:0/986387723 >> v2:172.21.15.29:3300/0 conn(0x7fa5e8159b20 msgr2=0x7fa5e80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.454+0000 7fa5edd9c700 1 --2- 172.21.15.29:0/986387723 >> v2:172.21.15.29:3300/0 conn(0x7fa5e8159b20 0x7fa5e80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.454+0000 7fa5edd9c700 1 -- 172.21.15.29:0/986387723 >> v1:172.21.15.144:6789/0 conn(0x7fa5e8159750 legacy=0x7fa5e80ce190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-07T09:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.454+0000 7fa5edd9c700 1 -- 172.21.15.29:0/986387723 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80cc7c0 con 0x7fa5e80ce8a0 2023-10-07T09:29:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.451+0000 7f3877dff700 1 -- 172.21.15.29:0/540426331 wait complete. 2023-10-07T09:29:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.453+0000 7f3877dff700 1 Processor -- start 2023-10-07T09:29:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.455+0000 7f3877dff700 1 -- start start 2023-10-07T09:29:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.456+0000 7fa5edd9c700 1 --2- 172.21.15.29:0/986387723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e80c5460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5e815e510 tx=0x7fa5e4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c146f2bbacbb7371 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.456+0000 7fa5dffff700 1 -- 172.21.15.29:0/986387723 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e400b820 con 0x7fa5e80ce8a0 2023-10-07T09:29:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.456+0000 7fa5dffff700 1 -- 172.21.15.29:0/986387723 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e400b980 con 0x7fa5e80ce8a0 2023-10-07T09:29:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.456+0000 7fa5dffff700 1 -- 172.21.15.29:0/986387723 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e400bd20 con 0x7fa5e80ce8a0 2023-10-07T09:29:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7f3877dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 0x7f387016f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7f3877dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f3870157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7f3877dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38700cc870 0x7f387006de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7f3877dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387016ed50 con 0x7f38700cc870 2023-10-07T09:29:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7f3877dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38700ca4c0 con 0x7f38700b17f0 2023-10-07T09:29:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7f3877dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38700ca7a0 con 0x7f38700b1bc0 2023-10-07T09:29:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.458+0000 7f3875b9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 0x7f387016f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/986387723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e80ce8a0 msgr2=0x7fa5e80c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.457+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/986387723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e80c5460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5e815e510 tx=0x7fa5e4009250 comp rx=0 tx=0).stop 2023-10-07T09:29:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.458+0000 7f3875b9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 0x7f387016f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53192/0 (socket says 172.21.15.29:53192) 2023-10-07T09:29:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.458+0000 7f3875b9b700 1 -- 172.21.15.29:0/3106816201 learned_addr learned my addr 172.21.15.29:0/3106816201 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.458+0000 7f387639c700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38700cc870 0x7f387006de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.458+0000 7f387539a700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f3870157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3875b9b700 1 -- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 msgr2=0x7f3870157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3875b9b700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f3870157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3875b9b700 1 -- 172.21.15.29:0/3106816201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38700cc870 msgr2=0x7f387006de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3875b9b700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38700cc870 0x7f387006de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3875b9b700 1 -- 172.21.15.29:0/3106816201 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38600094f0 con 0x7f38700b17f0 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3875b9b700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 0x7f387016f050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f386c00c5d0 tx=0x7f386c007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/986387723 shutdown_connections 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/986387723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e80c5460 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/986387723 >> v2:172.21.15.29:3300/0 conn(0x7fa5e8159b20 0x7fa5e80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/986387723 >> 172.21.15.29:0/986387723 conn(0x7fa5e8063a10 msgr2=0x7fa5e80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c011590 con 0x7f38700b17f0 2023-10-07T09:29:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386c011710 con 0x7f38700b17f0 2023-10-07T09:29:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c011ad0 con 0x7f38700b17f0 2023-10-07T09:29:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.460+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38700caa80 con 0x7f38700b17f0 2023-10-07T09:29:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.460+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3870157ac0 con 0x7f38700b17f0 2023-10-07T09:29:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.459+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/986387723 shutdown_connections 2023-10-07T09:29:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.460+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/986387723 wait complete. 2023-10-07T09:29:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.462+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387004c3c0 con 0x7f38700b17f0 2023-10-07T09:29:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.462+0000 7fa5ef7ff700 1 Processor -- start 2023-10-07T09:29:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.462+0000 7fa5ef7ff700 1 -- start start 2023-10-07T09:29:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.464+0000 7fa5ef7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.464+0000 7fa5ef7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5e8159750 0x7fa5e8154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.464+0000 7fa5ef7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e8159b20 0x7fa5e80c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.464+0000 7fa5ef7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b55f0 con 0x7fa5e80ce8a0 2023-10-07T09:29:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.464+0000 7fa5ef7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b37c0 con 0x7fa5e8159b20 2023-10-07T09:29:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.464+0000 7fa5ef7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80c97e0 con 0x7fa5e8159750 2023-10-07T09:29:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.465+0000 7fa5ed59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.465+0000 7fa5ed59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56728/0 (socket says 172.21.15.29:56728) 2023-10-07T09:29:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.465+0000 7fa5ed59b700 1 -- 172.21.15.29:0/1579322918 learned_addr learned my addr 172.21.15.29:0/1579322918 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.465+0000 7fa5ecd9a700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5e8159750 0x7fa5e8154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.466+0000 7fa5edd9c700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e8159b20 0x7fa5e80c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.466+0000 7fa5ecd9a700 1 -- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e8159b20 msgr2=0x7fa5e80c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.466+0000 7fa5ecd9a700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e8159b20 0x7fa5e80c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.466+0000 7fa5ecd9a700 1 -- 172.21.15.29:0/1579322918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5e80ce8a0 msgr2=0x7fa5e815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.466+0000 7fa5ecd9a700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.466+0000 7fa5ecd9a700 1 -- 172.21.15.29:0/1579322918 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e40094f0 con 0x7fa5e8159750 2023-10-07T09:29:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.467+0000 7fa5ecd9a700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5e8159750 0x7fa5e8154190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0007f50 tx=0x7fa5e00097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.468+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e000ebc0 con 0x7fa5e8159750 2023-10-07T09:29:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.468+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e000ed20 con 0x7fa5e8159750 2023-10-07T09:29:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.468+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0014650 con 0x7fa5e8159750 2023-10-07T09:29:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.468+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e80b3490 con 0x7fa5e8159750 2023-10-07T09:29:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.468+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e80c9220 con 0x7fa5e8159750 2023-10-07T09:29:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.470+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f386c01c460 con 0x7f38700b17f0 2023-10-07T09:29:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.472+0000 7f3865ffb700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3854067080 0x7f3854069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.472+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f386c084cd0 con 0x7f38700b17f0 2023-10-07T09:29:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.472+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f386c085160 con 0x7f38700b17f0 2023-10-07T09:29:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.472+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa5e0014a80 con 0x7fa5e8159750 2023-10-07T09:29:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.474+0000 7f387539a700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3854067080 0x7f3854069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.474+0000 7fa5dd7fa700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5cc067080 0x7fa5cc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.474+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa5e0084b50 con 0x7fa5e8159750 2023-10-07T09:29:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.475+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c80052b0 con 0x7fa5e8159750 2023-10-07T09:29:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.475+0000 7fa5ed59b700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5cc067080 0x7fa5cc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.475+0000 7f387539a700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3854067080 0x7f3854069530 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38700b05c0 tx=0x7f3868009620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.476+0000 7fa5ed59b700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5cc067080 0x7fa5cc069530 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8006fd0 tx=0x7fa5d80091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:38.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.480+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5e0055830 con 0x7fa5e8159750 2023-10-07T09:29:38.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.633+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4c8c0c2150 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.634+0000 7f4c72ffd700 1 -- 172.21.15.29:0/1297577733 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c7c056dd0 con 0x7f4c8c0b1bc0 2023-10-07T09:29:38.637 INFO:teuthology.orchestra.run.smithi029.stdout:60129542153 2023-10-07T09:29:38.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.639+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4c78067080 msgr2=0x7f4c78069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.639+0000 7f4c919ff700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4c78067080 0x7f4c78069530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009b80 tx=0x7f4c800091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.639+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b1bc0 msgr2=0x7f4c8c0cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.639+0000 7f4c919ff700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b1bc0 0x7f4c8c0cb430 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0099b0 tx=0x7f4c7c016000 comp rx=0 tx=0).stop 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 shutdown_connections 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4c78067080 0x7f4c78069530 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c8c0cc870 0x7f4c8c0c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0b17f0 0x7f4c8c14ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 --2- 172.21.15.29:0/1297577733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0b1bc0 0x7f4c8c0cb430 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 >> 172.21.15.29:0/1297577733 conn(0x7f4c8c0680a0 msgr2=0x7f4c8c15c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 shutdown_connections 2023-10-07T09:29:38.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.640+0000 7f4c919ff700 1 -- 172.21.15.29:0/1297577733 wait complete. 2023-10-07T09:29:38.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.3 2023-10-07T09:29:38.671 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.761+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3a7c005560 con 0x7f3a9815d8c0 2023-10-07T09:29:38.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.762+0000 7f3a867fc700 1 -- 172.21.15.29:0/1967021577 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a900566a0 con 0x7f3a9815d8c0 2023-10-07T09:29:38.762 INFO:teuthology.orchestra.run.smithi029.stdout:60129542152 2023-10-07T09:29:38.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3a78067080 msgr2=0x7f3a78069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3a78067080 0x7f3a78069530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007880 tx=0x7f3a8c004a80 comp rx=0 tx=0).stop 2023-10-07T09:29:38.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815d8c0 msgr2=0x7f3a981777e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815d8c0 0x7f3a981777e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a90015000 tx=0x7f3a90018000 comp rx=0 tx=0).stop 2023-10-07T09:29:38.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 shutdown_connections 2023-10-07T09:29:38.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3a78067080 0x7f3a78069530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9815dc90 0x7f3a980cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a980b1ac0 0x7f3a9813a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 --2- 172.21.15.29:0/1967021577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a9815d8c0 0x7f3a981777e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.766+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 >> 172.21.15.29:0/1967021577 conn(0x7f3a980ccc30 msgr2=0x7f3a9815e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.767+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 shutdown_connections 2023-10-07T09:29:38.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.767+0000 7f3a9cdff700 1 -- 172.21.15.29:0/1967021577 wait complete. 2023-10-07T09:29:38.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.787+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa5c8005560 con 0x7fa5e8159750 2023-10-07T09:29:38.788 INFO:teuthology.orchestra.run.smithi029.stdout:68719476745 2023-10-07T09:29:38.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.788+0000 7fa5dd7fa700 1 -- 172.21.15.29:0/1579322918 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa5e0009900 con 0x7fa5e8159750 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.790+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5cc067080 msgr2=0x7fa5cc069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.790+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5cc067080 0x7fa5cc069530 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8006fd0 tx=0x7fa5d80091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.790+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5e8159750 msgr2=0x7fa5e8154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.790+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5e8159750 0x7fa5e8154190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0007f50 tx=0x7fa5e00097e0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 shutdown_connections 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5cc067080 0x7fa5cc069530 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5e8159b20 0x7fa5e80c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5e80ce8a0 0x7fa5e815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 --2- 172.21.15.29:0/1579322918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5e8159750 0x7fa5e8154190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 >> 172.21.15.29:0/1579322918 conn(0x7fa5e8063a10 msgr2=0x7fa5e80cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 shutdown_connections 2023-10-07T09:29:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.791+0000 7fa5ef7ff700 1 -- 172.21.15.29:0/1579322918 wait complete. 2023-10-07T09:29:38.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542152 for osd.2 2023-10-07T09:29:38.796 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.810 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476745 for osd.7 2023-10-07T09:29:38.810 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.817+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f874004c3c0 con 0x7f8740152ce0 2023-10-07T09:29:38.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.818+0000 7f873cff9700 1 -- 172.21.15.29:0/1849904927 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f87300554f0 con 0x7f8740152ce0 2023-10-07T09:29:38.818 INFO:teuthology.orchestra.run.smithi029.stdout:60129542151 2023-10-07T09:29:38.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f872c067080 msgr2=0x7f872c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f872c067080 0x7f872c069530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8738014400 tx=0x7f873801e000 comp rx=0 tx=0).stop 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 msgr2=0x7f8740166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 0x7f8740166270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f873000b840 tx=0x7f8730009220 comp rx=0 tx=0).stop 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 shutdown_connections 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f872c067080 0x7f872c069530 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8740156980 0x7f8740155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f874006b590 0x7f8740144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 --2- 172.21.15.29:0/1849904927 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8740152ce0 0x7f8740166270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 >> 172.21.15.29:0/1849904927 conn(0x7f87400680a0 msgr2=0x7f8740148b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 shutdown_connections 2023-10-07T09:29:38.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.820+0000 7f8746fff700 1 -- 172.21.15.29:0/1849904927 wait complete. 2023-10-07T09:29:38.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.823+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f318015de50 con 0x7f31680051c0 2023-10-07T09:29:38.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.824+0000 7f316e7fc700 1 -- 172.21.15.29:0/3675942100 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3174052590 con 0x7f31680051c0 2023-10-07T09:29:38.824 INFO:teuthology.orchestra.run.smithi029.stdout:64424509449 2023-10-07T09:29:38.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3160067080 msgr2=0x7f3160069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3160067080 0x7f3160069530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3170007fd0 tx=0x7f3170009040 comp rx=0 tx=0).stop 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31680051c0 msgr2=0x7f31801434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31680051c0 0x7f31801434a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f31800650f0 tx=0x7f31740091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 shutdown_connections 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3160067080 0x7f3160069530 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31801542f0 0x7f3180154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31680051c0 0x7f31801434a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 --2- 172.21.15.29:0/3675942100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3180158f90 0x7f3180159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.827+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 >> 172.21.15.29:0/3675942100 conn(0x7f31800680a0 msgr2=0x7f318015b860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.828+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 shutdown_connections 2023-10-07T09:29:38.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.828+0000 7f3184bff700 1 -- 172.21.15.29:0/3675942100 wait complete. 2023-10-07T09:29:38.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.0 2023-10-07T09:29:38.840 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.850+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3d0c1469b0 con 0x7f3d0c155480 2023-10-07T09:29:38.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.850+0000 7f3cf2ffd700 1 -- 172.21.15.29:0/2543469586 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3cfc0574e0 con 0x7f3d0c155480 2023-10-07T09:29:38.850 INFO:teuthology.orchestra.run.smithi029.stdout:64424509449 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3cf8067080 msgr2=0x7f3cf8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3cf8067080 0x7f3cf8069530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c145aa0 tx=0x7f3d00007cd0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c155480 msgr2=0x7f3d0c1522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c155480 0x7f3d0c1522b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc015000 tx=0x7f3cfc016000 comp rx=0 tx=0).stop 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 shutdown_connections 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3cf8067080 0x7f3cf8069530 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d0c155480 0x7f3d0c1522b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c06bae0 0x7f3d0c148220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 --2- 172.21.15.29:0/2543469586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0c0b0180 0x7f3d0c151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 >> 172.21.15.29:0/2543469586 conn(0x7f3d0c0ccc30 msgr2=0x7f3d0c06b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 shutdown_connections 2023-10-07T09:29:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.854+0000 7f3d117ff700 1 -- 172.21.15.29:0/2543469586 wait complete. 2023-10-07T09:29:38.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.4 2023-10-07T09:29:38.856 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.868+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9944064040 con 0x7f9944159b20 2023-10-07T09:29:38.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.868+0000 7f992affd700 1 -- 172.21.15.29:0/1481903912 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f993401dc80 con 0x7f9944159b20 2023-10-07T09:29:38.869 INFO:teuthology.orchestra.run.smithi029.stdout:60129542151 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.871+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9930067080 msgr2=0x7f9930069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.871+0000 7f99495ff700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9930067080 0x7f9930069530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f993c0015b0 tx=0x7f993c001470 comp rx=0 tx=0).stop 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 msgr2=0x7f9944155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 0x7f9944155e10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9944088a40 tx=0x7f9934017000 comp rx=0 tx=0).stop 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 shutdown_connections 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9930067080 0x7f9930069530 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9944159b20 0x7f9944155e10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99440c6470 0x7f994414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 --2- 172.21.15.29:0/1481903912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9944159750 0x7f99441558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 >> 172.21.15.29:0/1481903912 conn(0x7f9944063a10 msgr2=0x7f9944070530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 shutdown_connections 2023-10-07T09:29:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.872+0000 7f99495ff700 1 -- 172.21.15.29:0/1481903912 wait complete. 2023-10-07T09:29:38.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.6 2023-10-07T09:29:38.874 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.882+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f387015e860 con 0x7f38700b17f0 2023-10-07T09:29:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.883+0000 7f3865ffb700 1 -- 172.21.15.29:0/3106816201 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f386c052050 con 0x7f38700b17f0 2023-10-07T09:29:38.883 INFO:teuthology.orchestra.run.smithi029.stdout:60129542151 2023-10-07T09:29:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3854067080 msgr2=0x7f3854069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3854067080 0x7f3854069530 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38700b05c0 tx=0x7f3868009620 comp rx=0 tx=0).stop 2023-10-07T09:29:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 msgr2=0x7f387016f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 0x7f387016f050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f386c00c5d0 tx=0x7f386c007f60 comp rx=0 tx=0).stop 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 shutdown_connections 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3854067080 0x7f3854069530 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38700cc870 0x7f387006de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.885+0000 7f3877dff700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38700b17f0 0x7f387016f050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.886+0000 7f3877dff700 1 --2- 172.21.15.29:0/3106816201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38700b1bc0 0x7f3870157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.886+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 >> 172.21.15.29:0/3106816201 conn(0x7f38700680a0 msgr2=0x7f3870159490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.886+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 shutdown_connections 2023-10-07T09:29:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:38.886+0000 7f3877dff700 1 -- 172.21.15.29:0/3106816201 wait complete. 2023-10-07T09:29:38.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.1 2023-10-07T09:29:38.890 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.5 2023-10-07T09:29:38.904 DEBUG:teuthology.parallel:result is None 2023-10-07T09:29:38.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-07T09:29:38.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:39.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.139+0000 7fb1693ff700 1 Processor -- start 2023-10-07T09:29:39.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.139+0000 7fb1693ff700 1 -- start start 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1693ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1640b07c0 0x7fb1640ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1693ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1641578a0 con 0x7fb1640ce390 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1693ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1640cc830 con 0x7fb1640b3e60 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1693ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb164157ce0 con 0x7fb1640b07c0 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1637fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1640b07c0 0x7fb1640ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1637fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1640b07c0 0x7fb1640ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56730/0 (socket says 172.21.15.29:56730) 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.140+0000 7fb1637fe700 1 -- 172.21.15.29:0/1413463922 learned_addr learned my addr 172.21.15.29:0/1413463922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb1637fe700 1 -- 172.21.15.29:0/1413463922 >> v1:172.21.15.144:6789/0 conn(0x7fb1640b3e60 legacy=0x7fb164158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb1637fe700 1 -- 172.21.15.29:0/1413463922 >> v1:172.21.15.29:6789/0 conn(0x7fb1640ce390 legacy=0x7fb1640b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb1637fe700 1 -- 172.21.15.29:0/1413463922 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1640cee50 con 0x7fb1640b07c0 2023-10-07T09:29:39.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb1637fe700 1 --2- 172.21.15.29:0/1413463922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb1640ce830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009f20 tx=0x7fb15c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32ec821bbf204d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:39.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb161ffb700 1 -- 172.21.15.29:0/1413463922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00e070 con 0x7fb1640b07c0 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb161ffb700 1 -- 172.21.15.29:0/1413463922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c00ba80 con 0x7fb1640b07c0 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.141+0000 7fb161ffb700 1 -- 172.21.15.29:0/1413463922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0092e0 con 0x7fb1640b07c0 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 -- 172.21.15.29:0/1413463922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 msgr2=0x7fb1640ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1413463922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb1640ce830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009f20 tx=0x7fb15c009840 comp rx=0 tx=0).stop 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 -- 172.21.15.29:0/1413463922 shutdown_connections 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1413463922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb1640ce830 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 -- 172.21.15.29:0/1413463922 >> 172.21.15.29:0/1413463922 conn(0x7fb164067610 msgr2=0x7fb164067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 -- 172.21.15.29:0/1413463922 shutdown_connections 2023-10-07T09:29:39.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.142+0000 7fb1693ff700 1 -- 172.21.15.29:0/1413463922 wait complete. 2023-10-07T09:29:39.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.143+0000 7fb1693ff700 1 Processor -- start 2023-10-07T09:29:39.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.144+0000 7fb1693ff700 1 -- start start 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1693ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb164155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1693ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1640b3e60 0x7fb1640690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1693ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1640ce390 0x7fb1640c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1693ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164069610 con 0x7fb1640b07c0 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1693ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164169550 con 0x7fb1640ce390 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1693ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16406a0d0 con 0x7fb1640b3e60 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb164155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb164155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56736/0 (socket says 172.21.15.29:56736) 2023-10-07T09:29:39.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 -- 172.21.15.29:0/1894524618 learned_addr learned my addr 172.21.15.29:0/1894524618 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1627fc700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1640b3e60 0x7fb1640690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb1637fe700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1640ce390 0x7fb1640c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 -- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1640b3e60 msgr2=0x7fb1640690d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1640b3e60 0x7fb1640690d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 -- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1640ce390 msgr2=0x7fb1640c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1640ce390 0x7fb1640c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.145+0000 7fb162ffd700 1 -- 172.21.15.29:0/1894524618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c009ae0 con 0x7fb1640b07c0 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.146+0000 7fb162ffd700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb164155c80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c5b0 tx=0x7fb154007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.146+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154011590 con 0x7fb1640b07c0 2023-10-07T09:29:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.146+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1540116f0 con 0x7fb1640b07c0 2023-10-07T09:29:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.146+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb164169820 con 0x7fb1640b07c0 2023-10-07T09:29:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.146+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154011af0 con 0x7fb1640b07c0 2023-10-07T09:29:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.146+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb164069830 con 0x7fb1640b07c0 2023-10-07T09:29:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.147+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16404c3c0 con 0x7fb1640b07c0 2023-10-07T09:29:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.148+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb15401b3b0 con 0x7fb1640b07c0 2023-10-07T09:29:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.149+0000 7fb14affd700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb150066f30 0x7fb1500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.149+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb154083a60 con 0x7fb1640b07c0 2023-10-07T09:29:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.150+0000 7fb1627fc700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb150066f30 0x7fb1500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.150+0000 7fb1627fc700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb150066f30 0x7fb1500693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb16406a330 tx=0x7fb158009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:39.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.154+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb154050ce0 con 0x7fb1640b07c0 2023-10-07T09:29:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.350+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1640cf130 con 0x7fb150066f30 2023-10-07T09:29:39.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.359+0000 7fb14affd700 1 -- 172.21.15.29:0/1894524618 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264470 (secure 0 0 0) 0x7fb1640cf130 con 0x7fb150066f30 2023-10-07T09:29:39.362 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:39.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb150066f30 msgr2=0x7fb1500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb150066f30 0x7fb1500693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb16406a330 tx=0x7fb158009040 comp rx=0 tx=0).stop 2023-10-07T09:29:39.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 msgr2=0x7fb164155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb164155c80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c5b0 tx=0x7fb154007f30 comp rx=0 tx=0).stop 2023-10-07T09:29:39.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 shutdown_connections 2023-10-07T09:29:39.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb150066f30 0x7fb1500693e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1640ce390 0x7fb1640c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b07c0 0x7fb164155c80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 --2- 172.21.15.29:0/1894524618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1640b3e60 0x7fb1640690d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 >> 172.21.15.29:0/1894524618 conn(0x7fb164067610 msgr2=0x7fb1640ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 shutdown_connections 2023-10-07T09:29:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.363+0000 7fb1693ff700 1 -- 172.21.15.29:0/1894524618 wait complete. 2023-10-07T09:29:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:39.381 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-07T09:29:38.021119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385352,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589368,"statfs":{"total":773094113280,"available":770651512832,"internally_reserved":0,"allocated":3022848,"data_stored":1805156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003489"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956059+0000","last_change":"2023-10-07T09:29:21.945132+0000","last_active":"2023-10-07T09:29:21.956059+0000","last_peered":"2023-10-07T09:29:21.956059+0000","last_clean":"2023-10-07T09:29:21.956059+0000","last_became_active":"2023-10-07T09:29:21.944887+0000","last_became_peered":"2023-10-07T09:29:21.944887+0000","last_unstale":"2023-10-07T09:29:21.956059+0000","last_undegraded":"2023-10-07T09:29:21.956059+0000","last_fullsized":"2023-10-07T09:29:21.956059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:41:47.844527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.940936+0000","last_change":"2023-10-07T09:29:21.935576+0000","last_active":"2023-10-07T09:29:21.940936+0000","last_peered":"2023-10-07T09:29:21.940936+0000","last_clean":"2023-10-07T09:29:21.940936+0000","last_became_active":"2023-10-07T09:29:21.935250+0000","last_became_peered":"2023-10-07T09:29:21.935250+0000","last_unstale":"2023-10-07T09:29:21.940936+0000","last_undegraded":"2023-10-07T09:29:21.940936+0000","last_fullsized":"2023-10-07T09:29:21.940936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:16.004086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954621+0000","last_change":"2023-10-07T09:29:21.941623+0000","last_active":"2023-10-07T09:29:21.954621+0000","last_peered":"2023-10-07T09:29:21.954621+0000","last_clean":"2023-10-07T09:29:21.954621+0000","last_became_active":"2023-10-07T09:29:21.941343+0000","last_became_peered":"2023-10-07T09:29:21.941343+0000","last_unstale":"2023-10-07T09:29:21.954621+0000","last_undegraded":"2023-10-07T09:29:21.954621+0000","last_fullsized":"2023-10-07T09:29:21.954621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:17:35.494089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945939+0000","last_change":"2023-10-07T09:29:21.940295+0000","last_active":"2023-10-07T09:29:21.945939+0000","last_peered":"2023-10-07T09:29:21.945939+0000","last_clean":"2023-10-07T09:29:21.945939+0000","last_became_active":"2023-10-07T09:29:21.940011+0000","last_became_peered":"2023-10-07T09:29:21.940011+0000","last_unstale":"2023-10-07T09:29:21.945939+0000","last_undegraded":"2023-10-07T09:29:21.945939+0000","last_fullsized":"2023-10-07T09:29:21.945939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:09:48.552431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943577+0000","last_change":"2023-10-07T09:29:21.938712+0000","last_active":"2023-10-07T09:29:21.943577+0000","last_peered":"2023-10-07T09:29:21.943577+0000","last_clean":"2023-10-07T09:29:21.943577+0000","last_became_active":"2023-10-07T09:29:21.938428+0000","last_became_peered":"2023-10-07T09:29:21.938428+0000","last_unstale":"2023-10-07T09:29:21.943577+0000","last_undegraded":"2023-10-07T09:29:21.943577+0000","last_fullsized":"2023-10-07T09:29:21.943577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:54:39.120171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103275+0000","last_change":"2023-10-07T09:29:21.114399+0000","last_active":"2023-10-07T09:29:22.103275+0000","last_peered":"2023-10-07T09:29:22.103275+0000","last_clean":"2023-10-07T09:29:22.103275+0000","last_became_active":"2023-10-07T09:29:21.114130+0000","last_became_peered":"2023-10-07T09:29:21.114130+0000","last_unstale":"2023-10-07T09:29:22.103275+0000","last_undegraded":"2023-10-07T09:29:22.103275+0000","last_fullsized":"2023-10-07T09:29:22.103275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:05:03.306731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103349+0000","last_change":"2023-10-07T09:29:21.111992+0000","last_active":"2023-10-07T09:29:22.103349+0000","last_peered":"2023-10-07T09:29:22.103349+0000","last_clean":"2023-10-07T09:29:22.103349+0000","last_became_active":"2023-10-07T09:29:21.111721+0000","last_became_peered":"2023-10-07T09:29:21.111721+0000","last_unstale":"2023-10-07T09:29:22.103349+0000","last_undegraded":"2023-10-07T09:29:22.103349+0000","last_fullsized":"2023-10-07T09:29:22.103349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943597+0000","last_change":"2023-10-07T09:29:21.939060+0000","last_active":"2023-10-07T09:29:21.943597+0000","last_peered":"2023-10-07T09:29:21.943597+0000","last_clean":"2023-10-07T09:29:21.943597+0000","last_became_active":"2023-10-07T09:29:21.938700+0000","last_became_peered":"2023-10-07T09:29:21.938700+0000","last_unstale":"2023-10-07T09:29:21.943597+0000","last_undegraded":"2023-10-07T09:29:21.943597+0000","last_fullsized":"2023-10-07T09:29:21.943597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:35.811275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945677+0000","last_change":"2023-10-07T09:29:21.939840+0000","last_active":"2023-10-07T09:29:21.945677+0000","last_peered":"2023-10-07T09:29:21.945677+0000","last_clean":"2023-10-07T09:29:21.945677+0000","last_became_active":"2023-10-07T09:29:21.939674+0000","last_became_peered":"2023-10-07T09:29:21.939674+0000","last_unstale":"2023-10-07T09:29:21.945677+0000","last_undegraded":"2023-10-07T09:29:21.945677+0000","last_fullsized":"2023-10-07T09:29:21.945677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:40.024706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.947776+0000","last_change":"2023-10-07T09:29:21.940515+0000","last_active":"2023-10-07T09:29:21.947776+0000","last_peered":"2023-10-07T09:29:21.947776+0000","last_clean":"2023-10-07T09:29:21.947776+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.947776+0000","last_undegraded":"2023-10-07T09:29:21.947776+0000","last_fullsized":"2023-10-07T09:29:21.947776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:01.738981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945596+0000","last_change":"2023-10-07T09:29:21.939851+0000","last_active":"2023-10-07T09:29:21.945596+0000","last_peered":"2023-10-07T09:29:21.945596+0000","last_clean":"2023-10-07T09:29:21.945596+0000","last_became_active":"2023-10-07T09:29:21.939540+0000","last_became_peered":"2023-10-07T09:29:21.939540+0000","last_unstale":"2023-10-07T09:29:21.945596+0000","last_undegraded":"2023-10-07T09:29:21.945596+0000","last_fullsized":"2023-10-07T09:29:21.945596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:23:02.797155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929892+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:21.929892+0000","last_peered":"2023-10-07T09:29:21.929892+0000","last_clean":"2023-10-07T09:29:21.929892+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:21.929892+0000","last_undegraded":"2023-10-07T09:29:21.929892+0000","last_fullsized":"2023-10-07T09:29:21.929892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930015+0000","last_change":"2023-10-07T09:29:21.111256+0000","last_active":"2023-10-07T09:29:21.930015+0000","last_peered":"2023-10-07T09:29:21.930015+0000","last_clean":"2023-10-07T09:29:21.930015+0000","last_became_active":"2023-10-07T09:29:21.110731+0000","last_became_peered":"2023-10-07T09:29:21.110731+0000","last_unstale":"2023-10-07T09:29:21.930015+0000","last_undegraded":"2023-10-07T09:29:21.930015+0000","last_fullsized":"2023-10-07T09:29:21.930015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929075+0000","last_change":"2023-10-07T09:29:21.112973+0000","last_active":"2023-10-07T09:29:21.929075+0000","last_peered":"2023-10-07T09:29:21.929075+0000","last_clean":"2023-10-07T09:29:21.929075+0000","last_became_active":"2023-10-07T09:29:21.112720+0000","last_became_peered":"2023-10-07T09:29:21.112720+0000","last_unstale":"2023-10-07T09:29:21.929075+0000","last_undegraded":"2023-10-07T09:29:21.929075+0000","last_fullsized":"2023-10-07T09:29:21.929075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.927905+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:22.927905+0000","last_peered":"2023-10-07T09:29:22.927905+0000","last_clean":"2023-10-07T09:29:22.927905+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:22.927905+0000","last_undegraded":"2023-10-07T09:29:22.927905+0000","last_fullsized":"2023-10-07T09:29:22.927905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930385+0000","last_change":"2023-10-07T09:29:21.111416+0000","last_active":"2023-10-07T09:29:21.930385+0000","last_peered":"2023-10-07T09:29:21.930385+0000","last_clean":"2023-10-07T09:29:21.930385+0000","last_became_active":"2023-10-07T09:29:21.111010+0000","last_became_peered":"2023-10-07T09:29:21.111010+0000","last_unstale":"2023-10-07T09:29:21.930385+0000","last_undegraded":"2023-10-07T09:29:21.930385+0000","last_fullsized":"2023-10-07T09:29:21.930385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103269+0000","last_change":"2023-10-07T09:29:21.112583+0000","last_active":"2023-10-07T09:29:22.103269+0000","last_peered":"2023-10-07T09:29:22.103269+0000","last_clean":"2023-10-07T09:29:22.103269+0000","last_became_active":"2023-10-07T09:29:21.112254+0000","last_became_peered":"2023-10-07T09:29:21.112254+0000","last_unstale":"2023-10-07T09:29:22.103269+0000","last_undegraded":"2023-10-07T09:29:22.103269+0000","last_fullsized":"2023-10-07T09:29:22.103269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956315+0000","last_change":"2023-10-07T09:29:21.952857+0000","last_active":"2023-10-07T09:29:21.956315+0000","last_peered":"2023-10-07T09:29:21.956315+0000","last_clean":"2023-10-07T09:29:21.956315+0000","last_became_active":"2023-10-07T09:29:21.952580+0000","last_became_peered":"2023-10-07T09:29:21.952580+0000","last_unstale":"2023-10-07T09:29:21.956315+0000","last_undegraded":"2023-10-07T09:29:21.956315+0000","last_fullsized":"2023-10-07T09:29:21.956315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:48.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956258+0000","last_change":"2023-10-07T09:29:21.952342+0000","last_active":"2023-10-07T09:29:21.956258+0000","last_peered":"2023-10-07T09:29:21.956258+0000","last_clean":"2023-10-07T09:29:21.956258+0000","last_became_active":"2023-10-07T09:29:21.952058+0000","last_became_peered":"2023-10-07T09:29:21.952058+0000","last_unstale":"2023-10-07T09:29:21.956258+0000","last_undegraded":"2023-10-07T09:29:21.956258+0000","last_fullsized":"2023-10-07T09:29:21.956258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:47:56.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476745,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298088,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073752,"statfs":{"total":96636764160,"available":96331522048,"internally_reserved":0,"allocated":819200,"data_stored":668398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.819}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1920000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:39.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:29:39.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.617+0000 7f40b81ff700 1 Processor -- start 2023-10-07T09:29:39.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.617+0000 7f40b81ff700 1 -- start start 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.618+0000 7f40b81ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40b015ade0 0x7f40b00b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.618+0000 7f40b81ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40b00b0540 0x7f40b015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.618+0000 7f40b81ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b0154090 con 0x7f40b0153cc0 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.618+0000 7f40b81ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b00b09e0 con 0x7f40b00b0540 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b81ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b0159f60 con 0x7f40b015ade0 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40b00b0540 0x7f40b015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40b00b0540 0x7f40b015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56742/0 (socket says 172.21.15.29:56742) 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 -- 172.21.15.29:0/3717785794 learned_addr learned my addr 172.21.15.29:0/3717785794 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:39.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b579a700 1 --2- 172.21.15.29:0/3717785794 >> v2:172.21.15.144:3300/0 conn(0x7f40b015ade0 0x7f40b00b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 -- 172.21.15.29:0/3717785794 >> v2:172.21.15.144:3300/0 conn(0x7f40b015ade0 msgr2=0x7f40b00b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 --2- 172.21.15.29:0/3717785794 >> v2:172.21.15.144:3300/0 conn(0x7f40b015ade0 0x7f40b00b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 -- 172.21.15.29:0/3717785794 >> v1:172.21.15.144:6789/0 conn(0x7f40b0153cc0 legacy=0x7f40b00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.619+0000 7f40b679c700 1 -- 172.21.15.29:0/3717785794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b015a240 con 0x7f40b00b0540 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b679c700 1 --2- 172.21.15.29:0/3717785794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b00b0540 0x7f40b015c240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f40b00add60 tx=0x7f40a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f514cf3c7e1ce5e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b4f99700 1 -- 172.21.15.29:0/3717785794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a800e070 con 0x7f40b00b0540 2023-10-07T09:29:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b4f99700 1 -- 172.21.15.29:0/3717785794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a800ba20 con 0x7f40b00b0540 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b4f99700 1 -- 172.21.15.29:0/3717785794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a80092e0 con 0x7f40b00b0540 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b81ff700 1 -- 172.21.15.29:0/3717785794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b00b0540 msgr2=0x7f40b015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b81ff700 1 --2- 172.21.15.29:0/3717785794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b00b0540 0x7f40b015c240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f40b00add60 tx=0x7f40a8009840 comp rx=0 tx=0).stop 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b81ff700 1 -- 172.21.15.29:0/3717785794 shutdown_connections 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b81ff700 1 --2- 172.21.15.29:0/3717785794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b00b0540 0x7f40b015c240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b81ff700 1 --2- 172.21.15.29:0/3717785794 >> v2:172.21.15.144:3300/0 conn(0x7f40b015ade0 0x7f40b00b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.620+0000 7f40b81ff700 1 -- 172.21.15.29:0/3717785794 >> 172.21.15.29:0/3717785794 conn(0x7f40b0067610 msgr2=0x7f40b0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.621+0000 7f40b81ff700 1 -- 172.21.15.29:0/3717785794 shutdown_connections 2023-10-07T09:29:39.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.621+0000 7f40b81ff700 1 -- 172.21.15.29:0/3717785794 wait complete. 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.622+0000 7f40b81ff700 1 Processor -- start 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.622+0000 7f40b81ff700 1 -- start start 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b81ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40b00b0540 0x7f40b015e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b81ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b0153cc0 0x7f40b015e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b81ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 0x7f40b0156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b81ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b00c7900 con 0x7f40b015ade0 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b81ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b0157ea0 con 0x7f40b0153cc0 2023-10-07T09:29:39.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b81ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b0157290 con 0x7f40b00b0540 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 0x7f40b0156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 0x7f40b0156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56744/0 (socket says 172.21.15.29:56744) 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b679c700 1 -- 172.21.15.29:0/2515635818 learned_addr learned my addr 172.21.15.29:0/2515635818 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b5f9b700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40b00b0540 0x7f40b015e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.623+0000 7f40b579a700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b0153cc0 0x7f40b015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b679c700 1 -- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40b00b0540 msgr2=0x7f40b015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b679c700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40b00b0540 0x7f40b015e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b679c700 1 -- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b0153cc0 msgr2=0x7f40b015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b679c700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b0153cc0 0x7f40b015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b679c700 1 -- 172.21.15.29:0/2515635818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a8009ae0 con 0x7f40b015ade0 2023-10-07T09:29:39.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b679c700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 0x7f40b0156680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009880 tx=0x7f40a8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:39.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a8008a80 con 0x7f40b015ade0 2023-10-07T09:29:39.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a8008be0 con 0x7f40b015ade0 2023-10-07T09:29:39.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b0157570 con 0x7f40b015ade0 2023-10-07T09:29:39.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.624+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a8026460 con 0x7f40b015ade0 2023-10-07T09:29:39.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.625+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b0157b30 con 0x7f40b015ade0 2023-10-07T09:29:39.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.626+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b0158050 con 0x7f40b015ade0 2023-10-07T09:29:39.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.626+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f40a8003040 con 0x7f40b015ade0 2023-10-07T09:29:39.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.627+0000 7f40a5ffb700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4094066f30 0x7f40940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:39.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.627+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f40a8086370 con 0x7f40b015ade0 2023-10-07T09:29:39.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.628+0000 7f40b5f9b700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4094066f30 0x7f40940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:39.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.628+0000 7f40b5f9b700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4094066f30 0x7f40940693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f40b00cf300 tx=0x7f40ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:39.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.631+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40b0158050 con 0x7f40b015ade0 2023-10-07T09:29:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.828+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40b0158010 con 0x7f4094066f30 2023-10-07T09:29:39.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.836+0000 7f40a5ffb700 1 -- 172.21.15.29:0/2515635818 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264470 (secure 0 0 0) 0x7f40b0158010 con 0x7f4094066f30 2023-10-07T09:29:39.840 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4094066f30 msgr2=0x7f40940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4094066f30 0x7f40940693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f40b00cf300 tx=0x7f40ac009040 comp rx=0 tx=0).stop 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 msgr2=0x7f40b0156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 0x7f40b0156680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009880 tx=0x7f40a8018000 comp rx=0 tx=0).stop 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 shutdown_connections 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4094066f30 0x7f40940693e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015ade0 0x7f40b0156680 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40b00b0540 0x7f40b015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 --2- 172.21.15.29:0/2515635818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b0153cc0 0x7f40b015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 >> 172.21.15.29:0/2515635818 conn(0x7f40b0067610 msgr2=0x7f40b0152d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 shutdown_connections 2023-10-07T09:29:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:39.840+0000 7f40b81ff700 1 -- 172.21.15.29:0/2515635818 wait complete. 2023-10-07T09:29:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:29:39.857 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-07T09:29:38.021119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2385352,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752589368,"statfs":{"total":773094113280,"available":770651512832,"internally_reserved":0,"allocated":3022848,"data_stored":1805156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003489"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956059+0000","last_change":"2023-10-07T09:29:21.945132+0000","last_active":"2023-10-07T09:29:21.956059+0000","last_peered":"2023-10-07T09:29:21.956059+0000","last_clean":"2023-10-07T09:29:21.956059+0000","last_became_active":"2023-10-07T09:29:21.944887+0000","last_became_peered":"2023-10-07T09:29:21.944887+0000","last_unstale":"2023-10-07T09:29:21.956059+0000","last_undegraded":"2023-10-07T09:29:21.956059+0000","last_fullsized":"2023-10-07T09:29:21.956059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:41:47.844527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945207+0000","last_change":"2023-10-07T09:29:21.942021+0000","last_active":"2023-10-07T09:29:21.945207+0000","last_peered":"2023-10-07T09:29:21.945207+0000","last_clean":"2023-10-07T09:29:21.945207+0000","last_became_active":"2023-10-07T09:29:21.941510+0000","last_became_peered":"2023-10-07T09:29:21.941510+0000","last_unstale":"2023-10-07T09:29:21.945207+0000","last_undegraded":"2023-10-07T09:29:21.945207+0000","last_fullsized":"2023-10-07T09:29:21.945207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943036+0000","last_change":"2023-10-07T09:29:21.940520+0000","last_active":"2023-10-07T09:29:21.943036+0000","last_peered":"2023-10-07T09:29:21.943036+0000","last_clean":"2023-10-07T09:29:21.943036+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.943036+0000","last_undegraded":"2023-10-07T09:29:21.943036+0000","last_fullsized":"2023-10-07T09:29:21.943036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:25:51.524938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945606+0000","last_change":"2023-10-07T09:29:21.943338+0000","last_active":"2023-10-07T09:29:21.945606+0000","last_peered":"2023-10-07T09:29:21.945606+0000","last_clean":"2023-10-07T09:29:21.945606+0000","last_became_active":"2023-10-07T09:29:21.943017+0000","last_became_peered":"2023-10-07T09:29:21.943017+0000","last_unstale":"2023-10-07T09:29:21.945606+0000","last_undegraded":"2023-10-07T09:29:21.945606+0000","last_fullsized":"2023-10-07T09:29:21.945606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:24.081280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.940936+0000","last_change":"2023-10-07T09:29:21.935576+0000","last_active":"2023-10-07T09:29:21.940936+0000","last_peered":"2023-10-07T09:29:21.940936+0000","last_clean":"2023-10-07T09:29:21.940936+0000","last_became_active":"2023-10-07T09:29:21.935250+0000","last_became_peered":"2023-10-07T09:29:21.935250+0000","last_unstale":"2023-10-07T09:29:21.940936+0000","last_undegraded":"2023-10-07T09:29:21.940936+0000","last_fullsized":"2023-10-07T09:29:21.940936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:16.004086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944618+0000","last_change":"2023-10-07T09:29:21.938344+0000","last_active":"2023-10-07T09:29:21.944618+0000","last_peered":"2023-10-07T09:29:21.944618+0000","last_clean":"2023-10-07T09:29:21.944618+0000","last_became_active":"2023-10-07T09:29:21.938038+0000","last_became_peered":"2023-10-07T09:29:21.938038+0000","last_unstale":"2023-10-07T09:29:21.944618+0000","last_undegraded":"2023-10-07T09:29:21.944618+0000","last_fullsized":"2023-10-07T09:29:21.944618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954621+0000","last_change":"2023-10-07T09:29:21.941623+0000","last_active":"2023-10-07T09:29:21.954621+0000","last_peered":"2023-10-07T09:29:21.954621+0000","last_clean":"2023-10-07T09:29:21.954621+0000","last_became_active":"2023-10-07T09:29:21.941343+0000","last_became_peered":"2023-10-07T09:29:21.941343+0000","last_unstale":"2023-10-07T09:29:21.954621+0000","last_undegraded":"2023-10-07T09:29:21.954621+0000","last_fullsized":"2023-10-07T09:29:21.954621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:17:35.494089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948713+0000","last_change":"2023-10-07T09:29:21.946979+0000","last_active":"2023-10-07T09:29:21.948713+0000","last_peered":"2023-10-07T09:29:21.948713+0000","last_clean":"2023-10-07T09:29:21.948713+0000","last_became_active":"2023-10-07T09:29:21.946385+0000","last_became_peered":"2023-10-07T09:29:21.946385+0000","last_unstale":"2023-10-07T09:29:21.948713+0000","last_undegraded":"2023-10-07T09:29:21.948713+0000","last_fullsized":"2023-10-07T09:29:21.948713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945939+0000","last_change":"2023-10-07T09:29:21.940295+0000","last_active":"2023-10-07T09:29:21.945939+0000","last_peered":"2023-10-07T09:29:21.945939+0000","last_clean":"2023-10-07T09:29:21.945939+0000","last_became_active":"2023-10-07T09:29:21.940011+0000","last_became_peered":"2023-10-07T09:29:21.940011+0000","last_unstale":"2023-10-07T09:29:21.945939+0000","last_undegraded":"2023-10-07T09:29:21.945939+0000","last_fullsized":"2023-10-07T09:29:21.945939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:09:48.552431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953043+0000","last_change":"2023-10-07T09:29:21.942515+0000","last_active":"2023-10-07T09:29:21.953043+0000","last_peered":"2023-10-07T09:29:21.953043+0000","last_clean":"2023-10-07T09:29:21.953043+0000","last_became_active":"2023-10-07T09:29:21.942235+0000","last_became_peered":"2023-10-07T09:29:21.942235+0000","last_unstale":"2023-10-07T09:29:21.953043+0000","last_undegraded":"2023-10-07T09:29:21.953043+0000","last_fullsized":"2023-10-07T09:29:21.953043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.952994+0000","last_change":"2023-10-07T09:29:21.948053+0000","last_active":"2023-10-07T09:29:21.952994+0000","last_peered":"2023-10-07T09:29:21.952994+0000","last_clean":"2023-10-07T09:29:21.952994+0000","last_became_active":"2023-10-07T09:29:21.947725+0000","last_became_peered":"2023-10-07T09:29:21.947725+0000","last_unstale":"2023-10-07T09:29:21.952994+0000","last_undegraded":"2023-10-07T09:29:21.952994+0000","last_fullsized":"2023-10-07T09:29:21.952994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950882+0000","last_change":"2023-10-07T09:29:21.945908+0000","last_active":"2023-10-07T09:29:21.950882+0000","last_peered":"2023-10-07T09:29:21.950882+0000","last_clean":"2023-10-07T09:29:21.950882+0000","last_became_active":"2023-10-07T09:29:21.945637+0000","last_became_peered":"2023-10-07T09:29:21.945637+0000","last_unstale":"2023-10-07T09:29:21.950882+0000","last_undegraded":"2023-10-07T09:29:21.950882+0000","last_fullsized":"2023-10-07T09:29:21.950882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942054+0000","last_change":"2023-10-07T09:29:21.939403+0000","last_active":"2023-10-07T09:29:21.942054+0000","last_peered":"2023-10-07T09:29:21.942054+0000","last_clean":"2023-10-07T09:29:21.942054+0000","last_became_active":"2023-10-07T09:29:21.939115+0000","last_became_peered":"2023-10-07T09:29:21.939115+0000","last_unstale":"2023-10-07T09:29:21.942054+0000","last_undegraded":"2023-10-07T09:29:21.942054+0000","last_fullsized":"2023-10-07T09:29:21.942054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942813+0000","last_change":"2023-10-07T09:29:21.940284+0000","last_active":"2023-10-07T09:29:21.942813+0000","last_peered":"2023-10-07T09:29:21.942813+0000","last_clean":"2023-10-07T09:29:21.942813+0000","last_became_active":"2023-10-07T09:29:21.939984+0000","last_became_peered":"2023-10-07T09:29:21.939984+0000","last_unstale":"2023-10-07T09:29:21.942813+0000","last_undegraded":"2023-10-07T09:29:21.942813+0000","last_fullsized":"2023-10-07T09:29:21.942813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.948670+0000","last_change":"2023-10-07T09:29:21.946364+0000","last_active":"2023-10-07T09:29:21.948670+0000","last_peered":"2023-10-07T09:29:21.948670+0000","last_clean":"2023-10-07T09:29:21.948670+0000","last_became_active":"2023-10-07T09:29:21.946081+0000","last_became_peered":"2023-10-07T09:29:21.946081+0000","last_unstale":"2023-10-07T09:29:21.948670+0000","last_undegraded":"2023-10-07T09:29:21.948670+0000","last_fullsized":"2023-10-07T09:29:21.948670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944651+0000","last_change":"2023-10-07T09:29:21.939221+0000","last_active":"2023-10-07T09:29:21.944651+0000","last_peered":"2023-10-07T09:29:21.944651+0000","last_clean":"2023-10-07T09:29:21.944651+0000","last_became_active":"2023-10-07T09:29:21.938879+0000","last_became_peered":"2023-10-07T09:29:21.938879+0000","last_unstale":"2023-10-07T09:29:21.944651+0000","last_undegraded":"2023-10-07T09:29:21.944651+0000","last_fullsized":"2023-10-07T09:29:21.944651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945634+0000","last_change":"2023-10-07T09:29:21.942715+0000","last_active":"2023-10-07T09:29:21.945634+0000","last_peered":"2023-10-07T09:29:21.945634+0000","last_clean":"2023-10-07T09:29:21.945634+0000","last_became_active":"2023-10-07T09:29:21.942045+0000","last_became_peered":"2023-10-07T09:29:21.942045+0000","last_unstale":"2023-10-07T09:29:21.945634+0000","last_undegraded":"2023-10-07T09:29:21.945634+0000","last_fullsized":"2023-10-07T09:29:21.945634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945374+0000","last_change":"2023-10-07T09:29:21.942093+0000","last_active":"2023-10-07T09:29:21.945374+0000","last_peered":"2023-10-07T09:29:21.945374+0000","last_clean":"2023-10-07T09:29:21.945374+0000","last_became_active":"2023-10-07T09:29:21.941903+0000","last_became_peered":"2023-10-07T09:29:21.941903+0000","last_unstale":"2023-10-07T09:29:21.945374+0000","last_undegraded":"2023-10-07T09:29:21.945374+0000","last_fullsized":"2023-10-07T09:29:21.945374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:47:50.272015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943577+0000","last_change":"2023-10-07T09:29:21.938712+0000","last_active":"2023-10-07T09:29:21.943577+0000","last_peered":"2023-10-07T09:29:21.943577+0000","last_clean":"2023-10-07T09:29:21.943577+0000","last_became_active":"2023-10-07T09:29:21.938428+0000","last_became_peered":"2023-10-07T09:29:21.938428+0000","last_unstale":"2023-10-07T09:29:21.943577+0000","last_undegraded":"2023-10-07T09:29:21.943577+0000","last_fullsized":"2023-10-07T09:29:21.943577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:54:39.120171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.104035+0000","last_change":"2023-10-07T09:29:21.111912+0000","last_active":"2023-10-07T09:29:22.104035+0000","last_peered":"2023-10-07T09:29:22.104035+0000","last_clean":"2023-10-07T09:29:22.104035+0000","last_became_active":"2023-10-07T09:29:21.111578+0000","last_became_peered":"2023-10-07T09:29:21.111578+0000","last_unstale":"2023-10-07T09:29:22.104035+0000","last_undegraded":"2023-10-07T09:29:22.104035+0000","last_fullsized":"2023-10-07T09:29:22.104035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:10.791446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945687+0000","last_change":"2023-10-07T09:29:21.942229+0000","last_active":"2023-10-07T09:29:21.945687+0000","last_peered":"2023-10-07T09:29:21.945687+0000","last_clean":"2023-10-07T09:29:21.945687+0000","last_became_active":"2023-10-07T09:29:21.941927+0000","last_became_peered":"2023-10-07T09:29:21.941927+0000","last_unstale":"2023-10-07T09:29:21.945687+0000","last_undegraded":"2023-10-07T09:29:21.945687+0000","last_fullsized":"2023-10-07T09:29:21.945687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:59.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101077+0000","last_change":"2023-10-07T09:29:21.114987+0000","last_active":"2023-10-07T09:29:22.101077+0000","last_peered":"2023-10-07T09:29:22.101077+0000","last_clean":"2023-10-07T09:29:22.101077+0000","last_became_active":"2023-10-07T09:29:21.114692+0000","last_became_peered":"2023-10-07T09:29:21.114692+0000","last_unstale":"2023-10-07T09:29:22.101077+0000","last_undegraded":"2023-10-07T09:29:22.101077+0000","last_fullsized":"2023-10-07T09:29:22.101077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:04.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950681+0000","last_change":"2023-10-07T09:29:21.948169+0000","last_active":"2023-10-07T09:29:21.950681+0000","last_peered":"2023-10-07T09:29:21.950681+0000","last_clean":"2023-10-07T09:29:21.950681+0000","last_became_active":"2023-10-07T09:29:21.947899+0000","last_became_peered":"2023-10-07T09:29:21.947899+0000","last_unstale":"2023-10-07T09:29:21.950681+0000","last_undegraded":"2023-10-07T09:29:21.950681+0000","last_fullsized":"2023-10-07T09:29:21.950681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:23.352281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103275+0000","last_change":"2023-10-07T09:29:21.114399+0000","last_active":"2023-10-07T09:29:22.103275+0000","last_peered":"2023-10-07T09:29:22.103275+0000","last_clean":"2023-10-07T09:29:22.103275+0000","last_became_active":"2023-10-07T09:29:21.114130+0000","last_became_peered":"2023-10-07T09:29:21.114130+0000","last_unstale":"2023-10-07T09:29:22.103275+0000","last_undegraded":"2023-10-07T09:29:22.103275+0000","last_fullsized":"2023-10-07T09:29:22.103275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:05:03.306731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954720+0000","last_change":"2023-10-07T09:29:21.952219+0000","last_active":"2023-10-07T09:29:21.954720+0000","last_peered":"2023-10-07T09:29:21.954720+0000","last_clean":"2023-10-07T09:29:21.954720+0000","last_became_active":"2023-10-07T09:29:21.951849+0000","last_became_peered":"2023-10-07T09:29:21.951849+0000","last_unstale":"2023-10-07T09:29:21.954720+0000","last_undegraded":"2023-10-07T09:29:21.954720+0000","last_fullsized":"2023-10-07T09:29:21.954720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:01:57.914950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103349+0000","last_change":"2023-10-07T09:29:21.111992+0000","last_active":"2023-10-07T09:29:22.103349+0000","last_peered":"2023-10-07T09:29:22.103349+0000","last_clean":"2023-10-07T09:29:22.103349+0000","last_became_active":"2023-10-07T09:29:21.111721+0000","last_became_peered":"2023-10-07T09:29:21.111721+0000","last_unstale":"2023-10-07T09:29:22.103349+0000","last_undegraded":"2023-10-07T09:29:22.103349+0000","last_fullsized":"2023-10-07T09:29:22.103349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:25.179074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.954709+0000","last_change":"2023-10-07T09:29:21.951568+0000","last_active":"2023-10-07T09:29:21.954709+0000","last_peered":"2023-10-07T09:29:21.954709+0000","last_clean":"2023-10-07T09:29:21.954709+0000","last_became_active":"2023-10-07T09:29:21.951273+0000","last_became_peered":"2023-10-07T09:29:21.951273+0000","last_unstale":"2023-10-07T09:29:21.954709+0000","last_undegraded":"2023-10-07T09:29:21.954709+0000","last_fullsized":"2023-10-07T09:29:21.954709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:28.857956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103655+0000","last_change":"2023-10-07T09:29:21.117185+0000","last_active":"2023-10-07T09:29:22.103655+0000","last_peered":"2023-10-07T09:29:22.103655+0000","last_clean":"2023-10-07T09:29:22.103655+0000","last_became_active":"2023-10-07T09:29:21.116889+0000","last_became_peered":"2023-10-07T09:29:21.116889+0000","last_unstale":"2023-10-07T09:29:22.103655+0000","last_undegraded":"2023-10-07T09:29:22.103655+0000","last_fullsized":"2023-10-07T09:29:22.103655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949981+0000","last_change":"2023-10-07T09:29:21.946567+0000","last_active":"2023-10-07T09:29:21.949981+0000","last_peered":"2023-10-07T09:29:21.949981+0000","last_clean":"2023-10-07T09:29:21.949981+0000","last_became_active":"2023-10-07T09:29:21.946266+0000","last_became_peered":"2023-10-07T09:29:21.946266+0000","last_unstale":"2023-10-07T09:29:21.949981+0000","last_undegraded":"2023-10-07T09:29:21.949981+0000","last_fullsized":"2023-10-07T09:29:21.949981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:41.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100941+0000","last_change":"2023-10-07T09:29:21.110311+0000","last_active":"2023-10-07T09:29:22.100941+0000","last_peered":"2023-10-07T09:29:22.100941+0000","last_clean":"2023-10-07T09:29:22.100941+0000","last_became_active":"2023-10-07T09:29:21.109875+0000","last_became_peered":"2023-10-07T09:29:21.109875+0000","last_unstale":"2023-10-07T09:29:22.100941+0000","last_undegraded":"2023-10-07T09:29:22.100941+0000","last_fullsized":"2023-10-07T09:29:22.100941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950707+0000","last_change":"2023-10-07T09:29:21.947736+0000","last_active":"2023-10-07T09:29:21.950707+0000","last_peered":"2023-10-07T09:29:21.950707+0000","last_clean":"2023-10-07T09:29:21.950707+0000","last_became_active":"2023-10-07T09:29:21.947472+0000","last_became_peered":"2023-10-07T09:29:21.947472+0000","last_unstale":"2023-10-07T09:29:21.950707+0000","last_undegraded":"2023-10-07T09:29:21.950707+0000","last_fullsized":"2023-10-07T09:29:21.950707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:14.849927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928421+0000","last_change":"2023-10-07T09:29:21.117995+0000","last_active":"2023-10-07T09:29:21.928421+0000","last_peered":"2023-10-07T09:29:21.928421+0000","last_clean":"2023-10-07T09:29:21.928421+0000","last_became_active":"2023-10-07T09:29:21.117713+0000","last_became_peered":"2023-10-07T09:29:21.117713+0000","last_unstale":"2023-10-07T09:29:21.928421+0000","last_undegraded":"2023-10-07T09:29:21.928421+0000","last_fullsized":"2023-10-07T09:29:21.928421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942268+0000","last_change":"2023-10-07T09:29:21.939366+0000","last_active":"2023-10-07T09:29:21.942268+0000","last_peered":"2023-10-07T09:29:21.942268+0000","last_clean":"2023-10-07T09:29:21.942268+0000","last_became_active":"2023-10-07T09:29:21.939057+0000","last_became_peered":"2023-10-07T09:29:21.939057+0000","last_unstale":"2023-10-07T09:29:21.942268+0000","last_undegraded":"2023-10-07T09:29:21.942268+0000","last_fullsized":"2023-10-07T09:29:21.942268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.928202+0000","last_change":"2023-10-07T09:29:21.115173+0000","last_active":"2023-10-07T09:29:21.928202+0000","last_peered":"2023-10-07T09:29:21.928202+0000","last_clean":"2023-10-07T09:29:21.928202+0000","last_became_active":"2023-10-07T09:29:21.114808+0000","last_became_peered":"2023-10-07T09:29:21.114808+0000","last_unstale":"2023-10-07T09:29:21.928202+0000","last_undegraded":"2023-10-07T09:29:21.928202+0000","last_fullsized":"2023-10-07T09:29:21.928202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937225+0000","last_change":"2023-10-07T09:29:21.934277+0000","last_active":"2023-10-07T09:29:21.937225+0000","last_peered":"2023-10-07T09:29:21.937225+0000","last_clean":"2023-10-07T09:29:21.937225+0000","last_became_active":"2023-10-07T09:29:21.933960+0000","last_became_peered":"2023-10-07T09:29:21.933960+0000","last_unstale":"2023-10-07T09:29:21.937225+0000","last_undegraded":"2023-10-07T09:29:21.937225+0000","last_fullsized":"2023-10-07T09:29:21.937225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937843+0000","last_change":"2023-10-07T09:29:21.112554+0000","last_active":"2023-10-07T09:29:21.937843+0000","last_peered":"2023-10-07T09:29:21.937843+0000","last_clean":"2023-10-07T09:29:21.937843+0000","last_became_active":"2023-10-07T09:29:21.112255+0000","last_became_peered":"2023-10-07T09:29:21.112255+0000","last_unstale":"2023-10-07T09:29:21.937843+0000","last_undegraded":"2023-10-07T09:29:21.937843+0000","last_fullsized":"2023-10-07T09:29:21.937843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944169+0000","last_change":"2023-10-07T09:29:21.942224+0000","last_active":"2023-10-07T09:29:21.944169+0000","last_peered":"2023-10-07T09:29:21.944169+0000","last_clean":"2023-10-07T09:29:21.944169+0000","last_became_active":"2023-10-07T09:29:21.941838+0000","last_became_peered":"2023-10-07T09:29:21.941838+0000","last_unstale":"2023-10-07T09:29:21.944169+0000","last_undegraded":"2023-10-07T09:29:21.944169+0000","last_fullsized":"2023-10-07T09:29:21.944169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936542+0000","last_change":"2023-10-07T09:29:21.110469+0000","last_active":"2023-10-07T09:29:21.936542+0000","last_peered":"2023-10-07T09:29:21.936542+0000","last_clean":"2023-10-07T09:29:21.936542+0000","last_became_active":"2023-10-07T09:29:21.110080+0000","last_became_peered":"2023-10-07T09:29:21.110080+0000","last_unstale":"2023-10-07T09:29:21.936542+0000","last_undegraded":"2023-10-07T09:29:21.936542+0000","last_fullsized":"2023-10-07T09:29:21.936542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950664+0000","last_change":"2023-10-07T09:29:21.947926+0000","last_active":"2023-10-07T09:29:21.950664+0000","last_peered":"2023-10-07T09:29:21.950664+0000","last_clean":"2023-10-07T09:29:21.950664+0000","last_became_active":"2023-10-07T09:29:21.947671+0000","last_became_peered":"2023-10-07T09:29:21.947671+0000","last_unstale":"2023-10-07T09:29:21.950664+0000","last_undegraded":"2023-10-07T09:29:21.950664+0000","last_fullsized":"2023-10-07T09:29:21.950664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101103+0000","last_change":"2023-10-07T09:29:21.101685+0000","last_active":"2023-10-07T09:29:22.101103+0000","last_peered":"2023-10-07T09:29:22.101103+0000","last_clean":"2023-10-07T09:29:22.101103+0000","last_became_active":"2023-10-07T09:29:21.101327+0000","last_became_peered":"2023-10-07T09:29:21.101327+0000","last_unstale":"2023-10-07T09:29:22.101103+0000","last_undegraded":"2023-10-07T09:29:22.101103+0000","last_fullsized":"2023-10-07T09:29:22.101103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950760+0000","last_change":"2023-10-07T09:29:21.944565+0000","last_active":"2023-10-07T09:29:21.950760+0000","last_peered":"2023-10-07T09:29:21.950760+0000","last_clean":"2023-10-07T09:29:21.950760+0000","last_became_active":"2023-10-07T09:29:21.942552+0000","last_became_peered":"2023-10-07T09:29:21.942552+0000","last_unstale":"2023-10-07T09:29:21.950760+0000","last_undegraded":"2023-10-07T09:29:21.950760+0000","last_fullsized":"2023-10-07T09:29:21.950760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.105844+0000","last_change":"2023-10-07T09:29:21.116578+0000","last_active":"2023-10-07T09:29:22.105844+0000","last_peered":"2023-10-07T09:29:22.105844+0000","last_clean":"2023-10-07T09:29:22.105844+0000","last_became_active":"2023-10-07T09:29:21.116342+0000","last_became_peered":"2023-10-07T09:29:21.116342+0000","last_unstale":"2023-10-07T09:29:22.105844+0000","last_undegraded":"2023-10-07T09:29:22.105844+0000","last_fullsized":"2023-10-07T09:29:22.105844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943597+0000","last_change":"2023-10-07T09:29:21.939060+0000","last_active":"2023-10-07T09:29:21.943597+0000","last_peered":"2023-10-07T09:29:21.943597+0000","last_clean":"2023-10-07T09:29:21.943597+0000","last_became_active":"2023-10-07T09:29:21.938700+0000","last_became_peered":"2023-10-07T09:29:21.938700+0000","last_unstale":"2023-10-07T09:29:21.943597+0000","last_undegraded":"2023-10-07T09:29:21.943597+0000","last_fullsized":"2023-10-07T09:29:21.943597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:35.811275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103094+0000","last_change":"2023-10-07T09:29:21.100478+0000","last_active":"2023-10-07T09:29:22.103094+0000","last_peered":"2023-10-07T09:29:22.103094+0000","last_clean":"2023-10-07T09:29:22.103094+0000","last_became_active":"2023-10-07T09:29:21.099101+0000","last_became_peered":"2023-10-07T09:29:21.099101+0000","last_unstale":"2023-10-07T09:29:22.103094+0000","last_undegraded":"2023-10-07T09:29:22.103094+0000","last_fullsized":"2023-10-07T09:29:22.103094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946550+0000","last_change":"2023-10-07T09:29:21.944679+0000","last_active":"2023-10-07T09:29:21.946550+0000","last_peered":"2023-10-07T09:29:21.946550+0000","last_clean":"2023-10-07T09:29:21.946550+0000","last_became_active":"2023-10-07T09:29:21.944373+0000","last_became_peered":"2023-10-07T09:29:21.944373+0000","last_unstale":"2023-10-07T09:29:21.946550+0000","last_undegraded":"2023-10-07T09:29:21.946550+0000","last_fullsized":"2023-10-07T09:29:21.946550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936344+0000","last_change":"2023-10-07T09:29:21.118404+0000","last_active":"2023-10-07T09:29:21.936344+0000","last_peered":"2023-10-07T09:29:21.936344+0000","last_clean":"2023-10-07T09:29:21.936344+0000","last_became_active":"2023-10-07T09:29:21.117974+0000","last_became_peered":"2023-10-07T09:29:21.117974+0000","last_unstale":"2023-10-07T09:29:21.936344+0000","last_undegraded":"2023-10-07T09:29:21.936344+0000","last_fullsized":"2023-10-07T09:29:21.936344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945677+0000","last_change":"2023-10-07T09:29:21.939840+0000","last_active":"2023-10-07T09:29:21.945677+0000","last_peered":"2023-10-07T09:29:21.945677+0000","last_clean":"2023-10-07T09:29:21.945677+0000","last_became_active":"2023-10-07T09:29:21.939674+0000","last_became_peered":"2023-10-07T09:29:21.939674+0000","last_unstale":"2023-10-07T09:29:21.945677+0000","last_undegraded":"2023-10-07T09:29:21.945677+0000","last_fullsized":"2023-10-07T09:29:21.945677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:40.024706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937422+0000","last_change":"2023-10-07T09:29:21.109388+0000","last_active":"2023-10-07T09:29:21.937422+0000","last_peered":"2023-10-07T09:29:21.937422+0000","last_clean":"2023-10-07T09:29:21.937422+0000","last_became_active":"2023-10-07T09:29:21.109042+0000","last_became_peered":"2023-10-07T09:29:21.109042+0000","last_unstale":"2023-10-07T09:29:21.937422+0000","last_undegraded":"2023-10-07T09:29:21.937422+0000","last_fullsized":"2023-10-07T09:29:21.937422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953003+0000","last_change":"2023-10-07T09:29:21.946258+0000","last_active":"2023-10-07T09:29:21.953003+0000","last_peered":"2023-10-07T09:29:21.953003+0000","last_clean":"2023-10-07T09:29:21.953003+0000","last_became_active":"2023-10-07T09:29:21.945610+0000","last_became_peered":"2023-10-07T09:29:21.945610+0000","last_unstale":"2023-10-07T09:29:21.953003+0000","last_undegraded":"2023-10-07T09:29:21.953003+0000","last_fullsized":"2023-10-07T09:29:21.953003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.102037+0000","last_change":"2023-10-07T09:29:21.104200+0000","last_active":"2023-10-07T09:29:22.102037+0000","last_peered":"2023-10-07T09:29:22.102037+0000","last_clean":"2023-10-07T09:29:22.102037+0000","last_became_active":"2023-10-07T09:29:21.103872+0000","last_became_peered":"2023-10-07T09:29:21.103872+0000","last_unstale":"2023-10-07T09:29:22.102037+0000","last_undegraded":"2023-10-07T09:29:22.102037+0000","last_fullsized":"2023-10-07T09:29:22.102037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.947776+0000","last_change":"2023-10-07T09:29:21.940515+0000","last_active":"2023-10-07T09:29:21.947776+0000","last_peered":"2023-10-07T09:29:21.947776+0000","last_clean":"2023-10-07T09:29:21.947776+0000","last_became_active":"2023-10-07T09:29:21.940218+0000","last_became_peered":"2023-10-07T09:29:21.940218+0000","last_unstale":"2023-10-07T09:29:21.947776+0000","last_undegraded":"2023-10-07T09:29:21.947776+0000","last_fullsized":"2023-10-07T09:29:21.947776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:01.738981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.101082+0000","last_change":"2023-10-07T09:29:21.104429+0000","last_active":"2023-10-07T09:29:22.101082+0000","last_peered":"2023-10-07T09:29:22.101082+0000","last_clean":"2023-10-07T09:29:22.101082+0000","last_became_active":"2023-10-07T09:29:21.104059+0000","last_became_peered":"2023-10-07T09:29:21.104059+0000","last_unstale":"2023-10-07T09:29:22.101082+0000","last_undegraded":"2023-10-07T09:29:22.101082+0000","last_fullsized":"2023-10-07T09:29:22.101082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:50.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945596+0000","last_change":"2023-10-07T09:29:21.939851+0000","last_active":"2023-10-07T09:29:21.945596+0000","last_peered":"2023-10-07T09:29:21.945596+0000","last_clean":"2023-10-07T09:29:21.945596+0000","last_became_active":"2023-10-07T09:29:21.939540+0000","last_became_peered":"2023-10-07T09:29:21.939540+0000","last_unstale":"2023-10-07T09:29:21.945596+0000","last_undegraded":"2023-10-07T09:29:21.945596+0000","last_fullsized":"2023-10-07T09:29:21.945596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:23:02.797155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923330+0000","last_change":"2023-10-07T09:29:21.104032+0000","last_active":"2023-10-07T09:29:21.923330+0000","last_peered":"2023-10-07T09:29:21.923330+0000","last_clean":"2023-10-07T09:29:21.923330+0000","last_became_active":"2023-10-07T09:29:21.103676+0000","last_became_peered":"2023-10-07T09:29:21.103676+0000","last_unstale":"2023-10-07T09:29:21.923330+0000","last_undegraded":"2023-10-07T09:29:21.923330+0000","last_fullsized":"2023-10-07T09:29:21.923330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945489+0000","last_change":"2023-10-07T09:29:21.943214+0000","last_active":"2023-10-07T09:29:21.945489+0000","last_peered":"2023-10-07T09:29:21.945489+0000","last_clean":"2023-10-07T09:29:21.945489+0000","last_became_active":"2023-10-07T09:29:21.942921+0000","last_became_peered":"2023-10-07T09:29:21.942921+0000","last_unstale":"2023-10-07T09:29:21.945489+0000","last_undegraded":"2023-10-07T09:29:21.945489+0000","last_fullsized":"2023-10-07T09:29:21.945489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936205+0000","last_change":"2023-10-07T09:29:21.106852+0000","last_active":"2023-10-07T09:29:21.936205+0000","last_peered":"2023-10-07T09:29:21.936205+0000","last_clean":"2023-10-07T09:29:21.936205+0000","last_became_active":"2023-10-07T09:29:21.106486+0000","last_became_peered":"2023-10-07T09:29:21.106486+0000","last_unstale":"2023-10-07T09:29:21.936205+0000","last_undegraded":"2023-10-07T09:29:21.936205+0000","last_fullsized":"2023-10-07T09:29:21.936205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:24:40.968437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950853+0000","last_change":"2023-10-07T09:29:21.948010+0000","last_active":"2023-10-07T09:29:21.950853+0000","last_peered":"2023-10-07T09:29:21.950853+0000","last_clean":"2023-10-07T09:29:21.950853+0000","last_became_active":"2023-10-07T09:29:21.947744+0000","last_became_peered":"2023-10-07T09:29:21.947744+0000","last_unstale":"2023-10-07T09:29:21.950853+0000","last_undegraded":"2023-10-07T09:29:21.950853+0000","last_fullsized":"2023-10-07T09:29:21.950853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937318+0000","last_change":"2023-10-07T09:29:21.933486+0000","last_active":"2023-10-07T09:29:21.937318+0000","last_peered":"2023-10-07T09:29:21.937318+0000","last_clean":"2023-10-07T09:29:21.937318+0000","last_became_active":"2023-10-07T09:29:21.933219+0000","last_became_peered":"2023-10-07T09:29:21.933219+0000","last_unstale":"2023-10-07T09:29:21.937318+0000","last_undegraded":"2023-10-07T09:29:21.937318+0000","last_fullsized":"2023-10-07T09:29:21.937318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926085+0000","last_change":"2023-10-07T09:29:21.118247+0000","last_active":"2023-10-07T09:29:21.926085+0000","last_peered":"2023-10-07T09:29:21.926085+0000","last_clean":"2023-10-07T09:29:21.926085+0000","last_became_active":"2023-10-07T09:29:21.117907+0000","last_became_peered":"2023-10-07T09:29:21.117907+0000","last_unstale":"2023-10-07T09:29:21.926085+0000","last_undegraded":"2023-10-07T09:29:21.926085+0000","last_fullsized":"2023-10-07T09:29:21.926085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:26:32.636064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.950071+0000","last_change":"2023-10-07T09:29:21.946650+0000","last_active":"2023-10-07T09:29:21.950071+0000","last_peered":"2023-10-07T09:29:21.950071+0000","last_clean":"2023-10-07T09:29:21.950071+0000","last_became_active":"2023-10-07T09:29:21.946398+0000","last_became_peered":"2023-10-07T09:29:21.946398+0000","last_unstale":"2023-10-07T09:29:21.950071+0000","last_undegraded":"2023-10-07T09:29:21.950071+0000","last_fullsized":"2023-10-07T09:29:21.950071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.935372+0000","last_change":"2023-10-07T09:29:21.108389+0000","last_active":"2023-10-07T09:29:21.935372+0000","last_peered":"2023-10-07T09:29:21.935372+0000","last_clean":"2023-10-07T09:29:21.935372+0000","last_became_active":"2023-10-07T09:29:21.108101+0000","last_became_peered":"2023-10-07T09:29:21.108101+0000","last_unstale":"2023-10-07T09:29:21.935372+0000","last_undegraded":"2023-10-07T09:29:21.935372+0000","last_fullsized":"2023-10-07T09:29:21.935372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929892+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:29:21.929892+0000","last_peered":"2023-10-07T09:29:21.929892+0000","last_clean":"2023-10-07T09:29:21.929892+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:29:21.929892+0000","last_undegraded":"2023-10-07T09:29:21.929892+0000","last_fullsized":"2023-10-07T09:29:21.929892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951324+0000","last_change":"2023-10-07T09:29:21.950355+0000","last_active":"2023-10-07T09:29:21.951324+0000","last_peered":"2023-10-07T09:29:21.951324+0000","last_clean":"2023-10-07T09:29:21.951324+0000","last_became_active":"2023-10-07T09:29:21.950114+0000","last_became_peered":"2023-10-07T09:29:21.950114+0000","last_unstale":"2023-10-07T09:29:21.951324+0000","last_undegraded":"2023-10-07T09:29:21.951324+0000","last_fullsized":"2023-10-07T09:29:21.951324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:14.387087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930015+0000","last_change":"2023-10-07T09:29:21.111256+0000","last_active":"2023-10-07T09:29:21.930015+0000","last_peered":"2023-10-07T09:29:21.930015+0000","last_clean":"2023-10-07T09:29:21.930015+0000","last_became_active":"2023-10-07T09:29:21.110731+0000","last_became_peered":"2023-10-07T09:29:21.110731+0000","last_unstale":"2023-10-07T09:29:21.930015+0000","last_undegraded":"2023-10-07T09:29:21.930015+0000","last_fullsized":"2023-10-07T09:29:21.930015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:18.263832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.929075+0000","last_change":"2023-10-07T09:29:21.112973+0000","last_active":"2023-10-07T09:29:21.929075+0000","last_peered":"2023-10-07T09:29:21.929075+0000","last_clean":"2023-10-07T09:29:21.929075+0000","last_became_active":"2023-10-07T09:29:21.112720+0000","last_became_peered":"2023-10-07T09:29:21.112720+0000","last_unstale":"2023-10-07T09:29:21.929075+0000","last_undegraded":"2023-10-07T09:29:21.929075+0000","last_fullsized":"2023-10-07T09:29:21.929075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:03:12.427984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936181+0000","last_change":"2023-10-07T09:29:21.929678+0000","last_active":"2023-10-07T09:29:21.936181+0000","last_peered":"2023-10-07T09:29:21.936181+0000","last_clean":"2023-10-07T09:29:21.936181+0000","last_became_active":"2023-10-07T09:29:21.929422+0000","last_became_peered":"2023-10-07T09:29:21.929422+0000","last_unstale":"2023-10-07T09:29:21.936181+0000","last_undegraded":"2023-10-07T09:29:21.936181+0000","last_fullsized":"2023-10-07T09:29:21.936181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.927905+0000","last_change":"2023-10-07T09:29:21.110743+0000","last_active":"2023-10-07T09:29:22.927905+0000","last_peered":"2023-10-07T09:29:22.927905+0000","last_clean":"2023-10-07T09:29:22.927905+0000","last_became_active":"2023-10-07T09:29:21.110368+0000","last_became_peered":"2023-10-07T09:29:21.110368+0000","last_unstale":"2023-10-07T09:29:22.927905+0000","last_undegraded":"2023-10-07T09:29:22.927905+0000","last_fullsized":"2023-10-07T09:29:22.927905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.951085+0000","last_change":"2023-10-07T09:29:21.949764+0000","last_active":"2023-10-07T09:29:21.951085+0000","last_peered":"2023-10-07T09:29:21.951085+0000","last_clean":"2023-10-07T09:29:21.951085+0000","last_became_active":"2023-10-07T09:29:21.949454+0000","last_became_peered":"2023-10-07T09:29:21.949454+0000","last_unstale":"2023-10-07T09:29:21.951085+0000","last_undegraded":"2023-10-07T09:29:21.951085+0000","last_fullsized":"2023-10-07T09:29:21.951085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:13.734414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103632+0000","last_change":"2023-10-07T09:29:21.113734+0000","last_active":"2023-10-07T09:29:22.103632+0000","last_peered":"2023-10-07T09:29:22.103632+0000","last_clean":"2023-10-07T09:29:22.103632+0000","last_became_active":"2023-10-07T09:29:21.113444+0000","last_became_peered":"2023-10-07T09:29:21.113444+0000","last_unstale":"2023-10-07T09:29:22.103632+0000","last_undegraded":"2023-10-07T09:29:22.103632+0000","last_fullsized":"2023-10-07T09:29:22.103632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:29.451309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942844+0000","last_change":"2023-10-07T09:29:21.939989+0000","last_active":"2023-10-07T09:29:21.942844+0000","last_peered":"2023-10-07T09:29:21.942844+0000","last_clean":"2023-10-07T09:29:21.942844+0000","last_became_active":"2023-10-07T09:29:21.939722+0000","last_became_peered":"2023-10-07T09:29:21.939722+0000","last_unstale":"2023-10-07T09:29:21.942844+0000","last_undegraded":"2023-10-07T09:29:21.942844+0000","last_fullsized":"2023-10-07T09:29:21.942844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.930385+0000","last_change":"2023-10-07T09:29:21.111416+0000","last_active":"2023-10-07T09:29:21.930385+0000","last_peered":"2023-10-07T09:29:21.930385+0000","last_clean":"2023-10-07T09:29:21.930385+0000","last_became_active":"2023-10-07T09:29:21.111010+0000","last_became_peered":"2023-10-07T09:29:21.111010+0000","last_unstale":"2023-10-07T09:29:21.930385+0000","last_undegraded":"2023-10-07T09:29:21.930385+0000","last_fullsized":"2023-10-07T09:29:21.930385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:09:17.095419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949983+0000","last_change":"2023-10-07T09:29:21.946309+0000","last_active":"2023-10-07T09:29:21.949983+0000","last_peered":"2023-10-07T09:29:21.949983+0000","last_clean":"2023-10-07T09:29:21.949983+0000","last_became_active":"2023-10-07T09:29:21.946022+0000","last_became_peered":"2023-10-07T09:29:21.946022+0000","last_unstale":"2023-10-07T09:29:21.949983+0000","last_undegraded":"2023-10-07T09:29:21.949983+0000","last_fullsized":"2023-10-07T09:29:21.949983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:30.457067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923059+0000","last_change":"2023-10-07T09:29:21.110732+0000","last_active":"2023-10-07T09:29:21.923059+0000","last_peered":"2023-10-07T09:29:21.923059+0000","last_clean":"2023-10-07T09:29:21.923059+0000","last_became_active":"2023-10-07T09:29:21.110336+0000","last_became_peered":"2023-10-07T09:29:21.110336+0000","last_unstale":"2023-10-07T09:29:21.923059+0000","last_undegraded":"2023-10-07T09:29:21.923059+0000","last_fullsized":"2023-10-07T09:29:21.923059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:06.684478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936313+0000","last_change":"2023-10-07T09:29:21.929247+0000","last_active":"2023-10-07T09:29:21.936313+0000","last_peered":"2023-10-07T09:29:21.936313+0000","last_clean":"2023-10-07T09:29:21.936313+0000","last_became_active":"2023-10-07T09:29:21.928939+0000","last_became_peered":"2023-10-07T09:29:21.928939+0000","last_unstale":"2023-10-07T09:29:21.936313+0000","last_undegraded":"2023-10-07T09:29:21.936313+0000","last_fullsized":"2023-10-07T09:29:21.936313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:07.690236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.923343+0000","last_change":"2023-10-07T09:29:21.118298+0000","last_active":"2023-10-07T09:29:21.923343+0000","last_peered":"2023-10-07T09:29:21.923343+0000","last_clean":"2023-10-07T09:29:21.923343+0000","last_became_active":"2023-10-07T09:29:21.117997+0000","last_became_peered":"2023-10-07T09:29:21.117997+0000","last_unstale":"2023-10-07T09:29:21.923343+0000","last_undegraded":"2023-10-07T09:29:21.923343+0000","last_fullsized":"2023-10-07T09:29:21.923343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:32.105046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.946951+0000","last_change":"2023-10-07T09:29:21.943738+0000","last_active":"2023-10-07T09:29:21.946951+0000","last_peered":"2023-10-07T09:29:21.946951+0000","last_clean":"2023-10-07T09:29:21.946951+0000","last_became_active":"2023-10-07T09:29:21.943415+0000","last_became_peered":"2023-10-07T09:29:21.943415+0000","last_unstale":"2023-10-07T09:29:21.946951+0000","last_undegraded":"2023-10-07T09:29:21.946951+0000","last_fullsized":"2023-10-07T09:29:21.946951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:38.839266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.103269+0000","last_change":"2023-10-07T09:29:21.112583+0000","last_active":"2023-10-07T09:29:22.103269+0000","last_peered":"2023-10-07T09:29:22.103269+0000","last_clean":"2023-10-07T09:29:22.103269+0000","last_became_active":"2023-10-07T09:29:21.112254+0000","last_became_peered":"2023-10-07T09:29:21.112254+0000","last_unstale":"2023-10-07T09:29:22.103269+0000","last_undegraded":"2023-10-07T09:29:22.103269+0000","last_fullsized":"2023-10-07T09:29:22.103269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:17:30.473974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955515+0000","last_change":"2023-10-07T09:29:21.950233+0000","last_active":"2023-10-07T09:29:21.955515+0000","last_peered":"2023-10-07T09:29:21.955515+0000","last_clean":"2023-10-07T09:29:21.955515+0000","last_became_active":"2023-10-07T09:29:21.949828+0000","last_became_peered":"2023-10-07T09:29:21.949828+0000","last_unstale":"2023-10-07T09:29:21.955515+0000","last_undegraded":"2023-10-07T09:29:21.955515+0000","last_fullsized":"2023-10-07T09:29:21.955515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:59.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.926187+0000","last_change":"2023-10-07T09:29:21.102750+0000","last_active":"2023-10-07T09:29:21.926187+0000","last_peered":"2023-10-07T09:29:21.926187+0000","last_clean":"2023-10-07T09:29:21.926187+0000","last_became_active":"2023-10-07T09:29:21.102379+0000","last_became_peered":"2023-10-07T09:29:21.102379+0000","last_unstale":"2023-10-07T09:29:21.926187+0000","last_undegraded":"2023-10-07T09:29:21.926187+0000","last_fullsized":"2023-10-07T09:29:21.926187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:16.159093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956315+0000","last_change":"2023-10-07T09:29:21.952857+0000","last_active":"2023-10-07T09:29:21.956315+0000","last_peered":"2023-10-07T09:29:21.956315+0000","last_clean":"2023-10-07T09:29:21.956315+0000","last_became_active":"2023-10-07T09:29:21.952580+0000","last_became_peered":"2023-10-07T09:29:21.952580+0000","last_unstale":"2023-10-07T09:29:21.956315+0000","last_undegraded":"2023-10-07T09:29:21.956315+0000","last_fullsized":"2023-10-07T09:29:21.956315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:48.448429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100949+0000","last_change":"2023-10-07T09:29:21.101266+0000","last_active":"2023-10-07T09:29:22.100949+0000","last_peered":"2023-10-07T09:29:22.100949+0000","last_clean":"2023-10-07T09:29:22.100949+0000","last_became_active":"2023-10-07T09:29:21.100874+0000","last_became_peered":"2023-10-07T09:29:21.100874+0000","last_unstale":"2023-10-07T09:29:22.100949+0000","last_undegraded":"2023-10-07T09:29:22.100949+0000","last_fullsized":"2023-10-07T09:29:22.100949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:33:45.356930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953651+0000","last_change":"2023-10-07T09:29:21.943704+0000","last_active":"2023-10-07T09:29:21.953651+0000","last_peered":"2023-10-07T09:29:21.953651+0000","last_clean":"2023-10-07T09:29:21.953651+0000","last_became_active":"2023-10-07T09:29:21.943437+0000","last_became_peered":"2023-10-07T09:29:21.943437+0000","last_unstale":"2023-10-07T09:29:21.953651+0000","last_undegraded":"2023-10-07T09:29:21.953651+0000","last_fullsized":"2023-10-07T09:29:21.953651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:18:38.139766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:22.100341+0000","last_change":"2023-10-07T09:29:21.115221+0000","last_active":"2023-10-07T09:29:22.100341+0000","last_peered":"2023-10-07T09:29:22.100341+0000","last_clean":"2023-10-07T09:29:22.100341+0000","last_became_active":"2023-10-07T09:29:21.114931+0000","last_became_peered":"2023-10-07T09:29:21.114931+0000","last_unstale":"2023-10-07T09:29:22.100341+0000","last_undegraded":"2023-10-07T09:29:22.100341+0000","last_fullsized":"2023-10-07T09:29:22.100341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:20.073275+0000","last_clean_scrub_stamp":"2023-10-07T09:29:20.073275+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:59.644928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.956258+0000","last_change":"2023-10-07T09:29:21.952342+0000","last_active":"2023-10-07T09:29:21.956258+0000","last_peered":"2023-10-07T09:29:21.956258+0000","last_clean":"2023-10-07T09:29:21.956258+0000","last_became_active":"2023-10-07T09:29:21.952058+0000","last_became_peered":"2023-10-07T09:29:21.952058+0000","last_unstale":"2023-10-07T09:29:21.956258+0000","last_undegraded":"2023-10-07T09:29:21.956258+0000","last_fullsized":"2023-10-07T09:29:21.956258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:47:56.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953018+0000","last_change":"2023-10-07T09:29:21.946532+0000","last_active":"2023-10-07T09:29:21.953018+0000","last_peered":"2023-10-07T09:29:21.953018+0000","last_clean":"2023-10-07T09:29:21.953018+0000","last_became_active":"2023-10-07T09:29:21.945958+0000","last_became_peered":"2023-10-07T09:29:21.945958+0000","last_unstale":"2023-10-07T09:29:21.953018+0000","last_undegraded":"2023-10-07T09:29:21.953018+0000","last_fullsized":"2023-10-07T09:29:21.953018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.937119+0000","last_change":"2023-10-07T09:29:21.934190+0000","last_active":"2023-10-07T09:29:21.937119+0000","last_peered":"2023-10-07T09:29:21.937119+0000","last_clean":"2023-10-07T09:29:21.937119+0000","last_became_active":"2023-10-07T09:29:21.933905+0000","last_became_peered":"2023-10-07T09:29:21.933905+0000","last_unstale":"2023-10-07T09:29:21.937119+0000","last_undegraded":"2023-10-07T09:29:21.937119+0000","last_fullsized":"2023-10-07T09:29:21.937119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:33.110804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953676+0000","last_change":"2023-10-07T09:29:21.944496+0000","last_active":"2023-10-07T09:29:21.953676+0000","last_peered":"2023-10-07T09:29:21.953676+0000","last_clean":"2023-10-07T09:29:21.953676+0000","last_became_active":"2023-10-07T09:29:21.944221+0000","last_became_peered":"2023-10-07T09:29:21.944221+0000","last_unstale":"2023-10-07T09:29:21.953676+0000","last_undegraded":"2023-10-07T09:29:21.953676+0000","last_fullsized":"2023-10-07T09:29:21.953676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:08.622635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945010+0000","last_change":"2023-10-07T09:29:21.943328+0000","last_active":"2023-10-07T09:29:21.945010+0000","last_peered":"2023-10-07T09:29:21.945010+0000","last_clean":"2023-10-07T09:29:21.945010+0000","last_became_active":"2023-10-07T09:29:21.943037+0000","last_became_peered":"2023-10-07T09:29:21.943037+0000","last_unstale":"2023-10-07T09:29:21.945010+0000","last_undegraded":"2023-10-07T09:29:21.945010+0000","last_fullsized":"2023-10-07T09:29:21.945010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:30.034144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.942949+0000","last_change":"2023-10-07T09:29:21.941347+0000","last_active":"2023-10-07T09:29:21.942949+0000","last_peered":"2023-10-07T09:29:21.942949+0000","last_clean":"2023-10-07T09:29:21.942949+0000","last_became_active":"2023-10-07T09:29:21.940507+0000","last_became_peered":"2023-10-07T09:29:21.940507+0000","last_unstale":"2023-10-07T09:29:21.942949+0000","last_undegraded":"2023-10-07T09:29:21.942949+0000","last_fullsized":"2023-10-07T09:29:21.942949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:03.458227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.936124+0000","last_change":"2023-10-07T09:29:21.931357+0000","last_active":"2023-10-07T09:29:21.936124+0000","last_peered":"2023-10-07T09:29:21.936124+0000","last_clean":"2023-10-07T09:29:21.936124+0000","last_became_active":"2023-10-07T09:29:21.931068+0000","last_became_peered":"2023-10-07T09:29:21.931068+0000","last_unstale":"2023-10-07T09:29:21.936124+0000","last_undegraded":"2023-10-07T09:29:21.936124+0000","last_fullsized":"2023-10-07T09:29:21.936124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:58:42.897086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.949491+0000","last_change":"2023-10-07T09:29:21.942196+0000","last_active":"2023-10-07T09:29:21.949491+0000","last_peered":"2023-10-07T09:29:21.949491+0000","last_clean":"2023-10-07T09:29:21.949491+0000","last_became_active":"2023-10-07T09:29:21.941850+0000","last_became_peered":"2023-10-07T09:29:21.941850+0000","last_unstale":"2023-10-07T09:29:21.949491+0000","last_undegraded":"2023-10-07T09:29:21.949491+0000","last_fullsized":"2023-10-07T09:29:21.949491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:39.523562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943070+0000","last_change":"2023-10-07T09:29:21.940614+0000","last_active":"2023-10-07T09:29:21.943070+0000","last_peered":"2023-10-07T09:29:21.943070+0000","last_clean":"2023-10-07T09:29:21.943070+0000","last_became_active":"2023-10-07T09:29:21.940329+0000","last_became_peered":"2023-10-07T09:29:21.940329+0000","last_unstale":"2023-10-07T09:29:21.943070+0000","last_undegraded":"2023-10-07T09:29:21.943070+0000","last_fullsized":"2023-10-07T09:29:21.943070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:27.060722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.953717+0000","last_change":"2023-10-07T09:29:21.944772+0000","last_active":"2023-10-07T09:29:21.953717+0000","last_peered":"2023-10-07T09:29:21.953717+0000","last_clean":"2023-10-07T09:29:21.953717+0000","last_became_active":"2023-10-07T09:29:21.944458+0000","last_became_peered":"2023-10-07T09:29:21.944458+0000","last_unstale":"2023-10-07T09:29:21.953717+0000","last_undegraded":"2023-10-07T09:29:21.953717+0000","last_fullsized":"2023-10-07T09:29:21.953717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:49:31.884897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.943114+0000","last_change":"2023-10-07T09:29:21.941126+0000","last_active":"2023-10-07T09:29:21.943114+0000","last_peered":"2023-10-07T09:29:21.943114+0000","last_clean":"2023-10-07T09:29:21.943114+0000","last_became_active":"2023-10-07T09:29:21.940838+0000","last_became_peered":"2023-10-07T09:29:21.940838+0000","last_unstale":"2023-10-07T09:29:21.943114+0000","last_undegraded":"2023-10-07T09:29:21.943114+0000","last_fullsized":"2023-10-07T09:29:21.943114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.944719+0000","last_change":"2023-10-07T09:29:21.941751+0000","last_active":"2023-10-07T09:29:21.944719+0000","last_peered":"2023-10-07T09:29:21.944719+0000","last_clean":"2023-10-07T09:29:21.944719+0000","last_became_active":"2023-10-07T09:29:21.941468+0000","last_became_peered":"2023-10-07T09:29:21.941468+0000","last_unstale":"2023-10-07T09:29:21.944719+0000","last_undegraded":"2023-10-07T09:29:21.944719+0000","last_fullsized":"2023-10-07T09:29:21.944719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.945447+0000","last_change":"2023-10-07T09:29:21.944238+0000","last_active":"2023-10-07T09:29:21.945447+0000","last_peered":"2023-10-07T09:29:21.945447+0000","last_clean":"2023-10-07T09:29:21.945447+0000","last_became_active":"2023-10-07T09:29:21.943918+0000","last_became_peered":"2023-10-07T09:29:21.943918+0000","last_unstale":"2023-10-07T09:29:21.945447+0000","last_undegraded":"2023-10-07T09:29:21.945447+0000","last_fullsized":"2023-10-07T09:29:21.945447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:38:11.774727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-07T09:29:21.955506+0000","last_change":"2023-10-07T09:29:21.950178+0000","last_active":"2023-10-07T09:29:21.955506+0000","last_peered":"2023-10-07T09:29:21.955506+0000","last_clean":"2023-10-07T09:29:21.955506+0000","last_became_active":"2023-10-07T09:29:21.949902+0000","last_became_peered":"2023-10-07T09:29:21.949902+0000","last_unstale":"2023-10-07T09:29:21.955506+0000","last_undegraded":"2023-10-07T09:29:21.955506+0000","last_fullsized":"2023-10-07T09:29:21.955506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:21.079033+0000","last_clean_scrub_stamp":"2023-10-07T09:29:21.079033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:08:55.478261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476745,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298088,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073752,"statfs":{"total":96636764160,"available":96331522048,"internally_reserved":0,"allocated":819200,"data_stored":668398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.819}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1920000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":77636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":668454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":77476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":79012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":77566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":233472,"data_stored":77602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:29:39.864 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-07T09:29:39.865 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-07T09:29:39.865 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-07T09:29:39.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-07T09:29:40.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.092+0000 7f18b17ff700 1 Processor -- start 2023-10-07T09:29:40.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.092+0000 7f18b17ff700 1 -- start start 2023-10-07T09:29:40.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.093+0000 7f18b17ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f18ac0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.093+0000 7f18b17ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0cbf70 con 0x7f18ac0b2ce0 2023-10-07T09:29:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.093+0000 7f18b17ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0cc250 con 0x7f18ac0cf2a0 2023-10-07T09:29:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.093+0000 7f18b17ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0cc530 con 0x7f18ac0c6660 2023-10-07T09:29:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.093+0000 7f18aaffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f18ac0b2ce0 0x7f18ac0ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58800/0 (socket says 172.21.15.29:58800) 2023-10-07T09:29:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18aaffd700 1 -- 172.21.15.29:0/3019223899 learned_addr learned my addr 172.21.15.29:0/3019223899 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18ab7fe700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f18ac0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115797100 0 0) 0x7f18ac0cbf70 con 0x7f18ac0b2ce0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1894003570 con 0x7f18ac0b2ce0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066923151 0 0) 0x7f18ac0cc250 con 0x7f18ac0cf2a0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ac0cbf70 con 0x7f18ac0cf2a0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367706032 0 0) 0x7f1894003570 con 0x7f18ac0b2ce0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 msgr2=0x7f18ac0cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f18ac0cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 >> v1:172.21.15.144:6789/0 conn(0x7f18ac0cf2a0 legacy=0x7f18ac0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac0cc810 con 0x7f18ac0b2ce0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f189c0015f0 con 0x7f18ac0b2ce0 2023-10-07T09:29:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 >> v1:172.21.15.29:6789/0 conn(0x7f18ac0b2ce0 legacy=0x7f18ac0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f1894004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.094+0000 7f18a9ffb700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f1894004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18a9ffb700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1894004d50 0x7f18940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0cc530 con 0x7f1894003c20 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189400bb40 con 0x7f1894004d50 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189400bd00 con 0x7f18ac0c6660 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f1894004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18ab7fe700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1894004d50 0x7f18940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aa7fc700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f1894004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 -- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 msgr2=0x7f1894004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f1894004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 -- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1894004d50 msgr2=0x7f18940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1894004d50 0x7f18940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 -- 172.21.15.29:0/3019223899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894003570 con 0x7f1894003c20 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aa7fc700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac0c6660 0x7f1894004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.095+0000 7f18aaffd700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f1894004010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c2a0 tx=0x7f189c0078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c776377c2693daee server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c00cb70 con 0x7f1894003c20 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c00ccf0 con 0x7f1894003c20 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18a9ffb700 1 -- 172.21.15.29:0/3019223899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c007320 con 0x7f1894003c20 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18b17ff700 1 -- 172.21.15.29:0/3019223899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 msgr2=0x7f1894004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18b17ff700 1 --2- 172.21.15.29:0/3019223899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f1894004010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c2a0 tx=0x7f189c0078a0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18ab7fe700 1 -- 172.21.15.29:0/3019223899 reap_dead start 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18b17ff700 1 -- 172.21.15.29:0/3019223899 shutdown_connections 2023-10-07T09:29:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18b17ff700 1 -- 172.21.15.29:0/3019223899 >> 172.21.15.29:0/3019223899 conn(0x7f18ac099820 msgr2=0x7f18ac063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18b17ff700 1 -- 172.21.15.29:0/3019223899 shutdown_connections 2023-10-07T09:29:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.096+0000 7f18b17ff700 1 -- 172.21.15.29:0/3019223899 wait complete. 2023-10-07T09:29:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.098+0000 7f18b17ff700 1 Processor -- start 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.098+0000 7f18b17ff700 1 -- start start 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18b17ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f18ac157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18b17ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac15d280 0x7f18ac15e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18b17ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18ac15e770 0x7f18ac0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18b17ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0b5d00 con 0x7f1894003c20 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18b17ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac15ebc0 con 0x7f18ac15e770 2023-10-07T09:29:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18b17ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac157180 con 0x7f18ac15d280 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18aaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f18ac157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18aaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f18ac157f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56774/0 (socket says 172.21.15.29:56774) 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18aaffd700 1 -- 172.21.15.29:0/4113667262 learned_addr learned my addr 172.21.15.29:0/4113667262 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18ab7fe700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18ac15e770 0x7f18ac0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.099+0000 7f18aa7fc700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac15d280 0x7f18ac15e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aaffd700 1 -- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac15d280 msgr2=0x7f18ac15e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aaffd700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac15d280 0x7f18ac15e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aaffd700 1 -- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18ac15e770 msgr2=0x7f18ac0b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aaffd700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18ac15e770 0x7f18ac0b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aaffd700 1 -- 172.21.15.29:0/4113667262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac157460 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aa7fc700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac15d280 0x7f18ac15e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18aaffd700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f18ac157f40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f189c000ef0 tx=0x7f189c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c002cc0 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c0070d0 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c0057f0 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac157740 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.100+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac156e90 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.101+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac04c3c0 con 0x7f1894003c20 2023-10-07T09:29:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.102+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f189c005b80 con 0x7f1894003c20 2023-10-07T09:29:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.103+0000 7f1892ffd700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1898066f30 0x7f18980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.104+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f189c085af0 con 0x7f1894003c20 2023-10-07T09:29:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.104+0000 7f18aa7fc700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1898066f30 0x7f18980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.104+0000 7f18aa7fc700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1898066f30 0x7f18980693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18a0007bc0 tx=0x7f18a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.107+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f189c052cc0 con 0x7f1894003c20 2023-10-07T09:29:40.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.351+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f18ac09e1d0 con 0x7f1894003c20 2023-10-07T09:29:40.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.351+0000 7f1892ffd700 1 -- 172.21.15.29:0/4113667262 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f189c023040 con 0x7f1894003c20 2023-10-07T09:29:40.352 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:40.352 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1898066f30 msgr2=0x7f18980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1898066f30 0x7f18980693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18a0007bc0 tx=0x7f18a0009040 comp rx=0 tx=0).stop 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 msgr2=0x7f18ac157f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f18ac157f40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f189c000ef0 tx=0x7f189c015000 comp rx=0 tx=0).stop 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 shutdown_connections 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1898066f30 0x7f18980693e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18ac15e770 0x7f18ac0b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1894003c20 0x7f18ac157f40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 --2- 172.21.15.29:0/4113667262 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18ac15d280 0x7f18ac15e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.353+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 >> 172.21.15.29:0/4113667262 conn(0x7f18ac099820 msgr2=0x7f18ac0cb190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.354+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 shutdown_connections 2023-10-07T09:29:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.354+0000 7f18b17ff700 1 -- 172.21.15.29:0/4113667262 wait complete. 2023-10-07T09:29:40.372 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-07T09:29:40.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.566+0000 7f236e5ff700 1 Processor -- start 2023-10-07T09:29:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.567+0000 7f236e5ff700 1 -- start start 2023-10-07T09:29:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236e5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2368159750 0x7f23680d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236e5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2368159b20 0x7f23680ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236e5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23680ce6d0 0x7f23680ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236e5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23680cbf20 con 0x7f2368159b20 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236e5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23680cc200 con 0x7f2368159750 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236e5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23680cc4e0 con 0x7f23680ce6d0 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f23677fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2368159b20 0x7f23680ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f23677fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2368159b20 0x7f23680ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56788/0 (socket says 172.21.15.29:56788) 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f23677fe700 1 -- 172.21.15.29:0/3724365844 learned_addr learned my addr 172.21.15.29:0/3724365844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.568+0000 7f236cb9c700 1 --2- 172.21.15.29:0/3724365844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23680ce6d0 0x7f23680ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f2367fff700 1 --2- 172.21.15.29:0/3724365844 >> v2:172.21.15.144:3300/0 conn(0x7f2368159750 0x7f23680d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f23677fe700 1 -- 172.21.15.29:0/3724365844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23680ce6d0 msgr2=0x7f23680ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f23677fe700 1 --2- 172.21.15.29:0/3724365844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23680ce6d0 0x7f23680ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f23677fe700 1 -- 172.21.15.29:0/3724365844 >> v2:172.21.15.144:3300/0 conn(0x7f2368159750 msgr2=0x7f23680d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f23677fe700 1 --2- 172.21.15.29:0/3724365844 >> v2:172.21.15.144:3300/0 conn(0x7f2368159750 0x7f23680d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f23677fe700 1 -- 172.21.15.29:0/3724365844 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680cc7c0 con 0x7f2368159b20 2023-10-07T09:29:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f23677fe700 1 --2- 172.21.15.29:0/3724365844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368159b20 0x7f23680ce190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2368089840 tx=0x7f235c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5443a0a276c0f381 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f2366ffd700 1 -- 172.21.15.29:0/3724365844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00b860 con 0x7f2368159b20 2023-10-07T09:29:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f2366ffd700 1 -- 172.21.15.29:0/3724365844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c00b9c0 con 0x7f2368159b20 2023-10-07T09:29:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.569+0000 7f2366ffd700 1 -- 172.21.15.29:0/3724365844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00bd60 con 0x7f2368159b20 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 -- 172.21.15.29:0/3724365844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368159b20 msgr2=0x7f23680ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 --2- 172.21.15.29:0/3724365844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368159b20 0x7f23680ce190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2368089840 tx=0x7f235c009250 comp rx=0 tx=0).stop 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 -- 172.21.15.29:0/3724365844 shutdown_connections 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 --2- 172.21.15.29:0/3724365844 >> v2:172.21.15.144:3300/0 conn(0x7f2368159750 0x7f23680d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 --2- 172.21.15.29:0/3724365844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23680ce6d0 0x7f23680ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 --2- 172.21.15.29:0/3724365844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368159b20 0x7f23680ce190 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 -- 172.21.15.29:0/3724365844 >> 172.21.15.29:0/3724365844 conn(0x7f2368063a10 msgr2=0x7f23680d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:40.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 -- 172.21.15.29:0/3724365844 shutdown_connections 2023-10-07T09:29:40.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.570+0000 7f236e5ff700 1 -- 172.21.15.29:0/3724365844 wait complete. 2023-10-07T09:29:40.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.571+0000 7f236e5ff700 1 Processor -- start 2023-10-07T09:29:40.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.572+0000 7f236e5ff700 1 -- start start 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236e5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 0x7f23680b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236e5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2368159750 0x7f236814d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236e5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2368159b20 0x7f2368157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236e5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236815a7f0 con 0x7f23680ce6d0 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236e5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236814d980 con 0x7f2368159750 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236e5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236815a980 con 0x7f2368159b20 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 0x7f23680b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 0x7f23680b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56802/0 (socket says 172.21.15.29:56802) 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 -- 172.21.15.29:0/560452205 learned_addr learned my addr 172.21.15.29:0/560452205 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f23677fe700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2368159750 0x7f236814d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236cb9c700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2368159b20 0x7f2368157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 -- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2368159b20 msgr2=0x7f2368157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2368159b20 0x7f2368157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 -- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2368159750 msgr2=0x7f236814d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2368159750 0x7f236814d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 -- 172.21.15.29:0/560452205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c0094f0 con 0x7f23680ce6d0 2023-10-07T09:29:40.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f236cb9c700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2368159b20 0x7f2368157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:40.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.573+0000 7f2367fff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 0x7f23680b3360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f235800d840 tx=0x7f2358008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:40.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.574+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23580125f0 con 0x7f23680ce6d0 2023-10-07T09:29:40.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.574+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2358012750 con 0x7f23680ce6d0 2023-10-07T09:29:40.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.574+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358012b10 con 0x7f23680ce6d0 2023-10-07T09:29:40.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.574+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23680b39f0 con 0x7f23680ce6d0 2023-10-07T09:29:40.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.574+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23680b3fb0 con 0x7f23680ce6d0 2023-10-07T09:29:40.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.575+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236804c3c0 con 0x7f23680ce6d0 2023-10-07T09:29:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.576+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f235801e460 con 0x7f23680ce6d0 2023-10-07T09:29:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.577+0000 7f234ffff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2354066fb0 0x7f2354069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.577+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2358084bc0 con 0x7f23680ce6d0 2023-10-07T09:29:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.578+0000 7f23677fe700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2354066fb0 0x7f2354069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.578+0000 7f23677fe700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2354066fb0 0x7f2354069460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2368089840 tx=0x7f235c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.581+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2358051d90 con 0x7f23680ce6d0 2023-10-07T09:29:40.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.785+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23680caf00 con 0x7f23680ce6d0 2023-10-07T09:29:40.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.786+0000 7f234ffff700 1 -- 172.21.15.29:0/560452205 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f23580553b0 con 0x7f23680ce6d0 2023-10-07T09:29:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:40.787 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2354066fb0 msgr2=0x7f2354069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2354066fb0 0x7f2354069460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2368089840 tx=0x7f235c017000 comp rx=0 tx=0).stop 2023-10-07T09:29:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 msgr2=0x7f23680b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 0x7f23680b3360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f235800d840 tx=0x7f2358008040 comp rx=0 tx=0).stop 2023-10-07T09:29:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 shutdown_connections 2023-10-07T09:29:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2354066fb0 0x7f2354069460 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2368159b20 0x7f2368157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23680ce6d0 0x7f23680b3360 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 --2- 172.21.15.29:0/560452205 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2368159750 0x7f236814d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 >> 172.21.15.29:0/560452205 conn(0x7f2368063a10 msgr2=0x7f2368068440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 shutdown_connections 2023-10-07T09:29:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:40.789+0000 7f236e5ff700 1 -- 172.21.15.29:0/560452205 wait complete. 2023-10-07T09:29:40.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:29:40.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:29:41.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.004+0000 7f2b883ff700 1 Processor -- start 2023-10-07T09:29:41.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.004+0000 7f2b883ff700 1 -- start start 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b883ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b80153cc0 0x7f2b800b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b883ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2b8015ade0 0x7f2b800b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b883ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b80154090 con 0x7f2b800b0000 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b883ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b80159f60 con 0x7f2b80153cc0 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b883ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8015a240 con 0x7f2b8015ade0 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b8619b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b80153cc0 0x7f2b800b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b8699c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2b800b0000 0x7f2b8015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58804/0 (socket says 172.21.15.29:58804) 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b8619b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b80153cc0 0x7f2b800b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56808/0 (socket says 172.21.15.29:56808) 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b8619b700 1 -- 172.21.15.29:0/31400443 learned_addr learned my addr 172.21.15.29:0/31400443 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.006+0000 7f2b8599a700 1 --2- 172.21.15.29:0/31400443 >> v2:172.21.15.144:3300/0 conn(0x7f2b8015ade0 0x7f2b800b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b8619b700 1 -- 172.21.15.29:0/31400443 >> v2:172.21.15.144:3300/0 conn(0x7f2b8015ade0 msgr2=0x7f2b800b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b8619b700 1 --2- 172.21.15.29:0/31400443 >> v2:172.21.15.144:3300/0 conn(0x7f2b8015ade0 0x7f2b800b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b8619b700 1 -- 172.21.15.29:0/31400443 >> v1:172.21.15.29:6789/0 conn(0x7f2b800b0000 legacy=0x7f2b8015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b8619b700 1 -- 172.21.15.29:0/31400443 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8015a520 con 0x7f2b80153cc0 2023-10-07T09:29:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b8619b700 1 --2- 172.21.15.29:0/31400443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800b17c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2b800add60 tx=0x7f2b7c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9f190df9d1d545 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b85199700 1 -- 172.21.15.29:0/31400443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c005c10 con 0x7f2b80153cc0 2023-10-07T09:29:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b85199700 1 -- 172.21.15.29:0/31400443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c005d70 con 0x7f2b80153cc0 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.007+0000 7f2b85199700 1 -- 172.21.15.29:0/31400443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00b880 con 0x7f2b80153cc0 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 -- 172.21.15.29:0/31400443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 msgr2=0x7f2b800b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 --2- 172.21.15.29:0/31400443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800b17c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2b800add60 tx=0x7f2b7c009220 comp rx=0 tx=0).stop 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 -- 172.21.15.29:0/31400443 shutdown_connections 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 --2- 172.21.15.29:0/31400443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800b17c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 --2- 172.21.15.29:0/31400443 >> v2:172.21.15.144:3300/0 conn(0x7f2b8015ade0 0x7f2b800b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 -- 172.21.15.29:0/31400443 >> 172.21.15.29:0/31400443 conn(0x7f2b80067610 msgr2=0x7f2b80067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 -- 172.21.15.29:0/31400443 shutdown_connections 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.008+0000 7f2b883ff700 1 -- 172.21.15.29:0/31400443 wait complete. 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.009+0000 7f2b883ff700 1 Processor -- start 2023-10-07T09:29:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.009+0000 7f2b883ff700 1 -- start start 2023-10-07T09:29:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.010+0000 7f2b883ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b800b0000 0x7f2b8016a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.010+0000 7f2b883ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.010+0000 7f2b883ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b8015ade0 0x7f2b80145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.010+0000 7f2b883ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800c8f10 con 0x7f2b80153cc0 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.010+0000 7f2b883ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80158510 con 0x7f2b800b0000 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.010+0000 7f2b883ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800c86b0 con 0x7f2b8015ade0 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56822/0 (socket says 172.21.15.29:56822) 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 -- 172.21.15.29:0/1021607700 learned_addr learned my addr 172.21.15.29:0/1021607700 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8619b700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b800b0000 0x7f2b8016a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8699c700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b8015ade0 0x7f2b80145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 -- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b8015ade0 msgr2=0x7f2b80145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b8015ade0 0x7f2b80145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 -- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b800b0000 msgr2=0x7f2b8016a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b800b0000 0x7f2b8016a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 -- 172.21.15.29:0/1021607700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c009480 con 0x7f2b80153cc0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b8599a700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800c8170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000d890 tx=0x7f2b70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.011+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70012630 con 0x7f2b80153cc0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.012+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70012790 con 0x7f2b80153cc0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.012+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70012b30 con 0x7f2b80153cc0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.012+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b800c8950 con 0x7f2b80153cc0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.012+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b801581a0 con 0x7f2b80153cc0 2023-10-07T09:29:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.013+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80158000 con 0x7f2b80153cc0 2023-10-07T09:29:41.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.013+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2b7001e460 con 0x7f2b80153cc0 2023-10-07T09:29:41.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.014+0000 7f2b767fc700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2b64066f30 0x7f2b640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.014+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2b70084b20 con 0x7f2b80153cc0 2023-10-07T09:29:41.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.015+0000 7f2b8619b700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2b64066f30 0x7f2b640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.016+0000 7f2b8619b700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2b64066f30 0x7f2b640693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009260 tx=0x7f2b7c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.018+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b80158000 con 0x7f2b80153cc0 2023-10-07T09:29:41.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.219+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b80158000 con 0x7f2b80153cc0 2023-10-07T09:29:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.220+0000 7f2b767fc700 1 -- 172.21.15.29:0/1021607700 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b80158000 con 0x7f2b80153cc0 2023-10-07T09:29:41.220 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:29:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2b64066f30 msgr2=0x7f2b640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2b64066f30 0x7f2b640693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009260 tx=0x7f2b7c016000 comp rx=0 tx=0).stop 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 msgr2=0x7f2b800c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800c8170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000d890 tx=0x7f2b70008040 comp rx=0 tx=0).stop 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 shutdown_connections 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2b64066f30 0x7f2b640693e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b8015ade0 0x7f2b80145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b800b0000 0x7f2b8016a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 --2- 172.21.15.29:0/1021607700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b80153cc0 0x7f2b800c8170 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 >> 172.21.15.29:0/1021607700 conn(0x7f2b80067610 msgr2=0x7f2b8015ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 shutdown_connections 2023-10-07T09:29:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.222+0000 7f2b883ff700 1 -- 172.21.15.29:0/1021607700 wait complete. 2023-10-07T09:29:41.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:29:41.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.436+0000 7f8aa5bff700 1 Processor -- start 2023-10-07T09:29:41.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.436+0000 7f8aa5bff700 1 -- start start 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.437+0000 7f8aa5bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa0159750 0x7f8aa00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.437+0000 7f8aa5bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8aa0159b20 0x7f8aa00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.437+0000 7f8aa5bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8aa00ce6d0 0x7f8aa00ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.437+0000 7f8aa5bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa00cbf20 con 0x7f8aa0159b20 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.437+0000 7f8aa5bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa00cc200 con 0x7f8aa00ce6d0 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.437+0000 7f8aa5bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00cc4e0 con 0x7f8aa0159750 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8aa0159b20 0x7f8aa00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8aa0159b20 0x7f8aa00ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56824/0 (socket says 172.21.15.29:56824) 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 -- 172.21.15.29:0/1477525908 learned_addr learned my addr 172.21.15.29:0/1477525908 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9f7fe700 1 --2- 172.21.15.29:0/1477525908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa0159750 0x7f8aa00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9ffff700 1 --2- 172.21.15.29:0/1477525908 >> v2:172.21.15.144:3300/0 conn(0x7f8aa00ce6d0 0x7f8aa00ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 -- 172.21.15.29:0/1477525908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa0159750 msgr2=0x7f8aa00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 --2- 172.21.15.29:0/1477525908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa0159750 0x7f8aa00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 -- 172.21.15.29:0/1477525908 >> v2:172.21.15.144:3300/0 conn(0x7f8aa00ce6d0 msgr2=0x7f8aa00ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 --2- 172.21.15.29:0/1477525908 >> v2:172.21.15.144:3300/0 conn(0x7f8aa00ce6d0 0x7f8aa00ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9effd700 1 -- 172.21.15.29:0/1477525908 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa00cc7c0 con 0x7f8aa0159b20 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.438+0000 7f8a9ffff700 1 --2- 172.21.15.29:0/1477525908 >> v2:172.21.15.144:3300/0 conn(0x7f8aa00ce6d0 0x7f8aa00ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8a9effd700 1 --2- 172.21.15.29:0/1477525908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159b20 0x7f8aa00ce190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00d0d60 tx=0x7f8a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae3a0eae1257439 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8a9e7fc700 1 -- 172.21.15.29:0/1477525908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94009e70 con 0x7f8aa0159b20 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8a9e7fc700 1 -- 172.21.15.29:0/1477525908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400b860 con 0x7f8aa0159b20 2023-10-07T09:29:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8a9e7fc700 1 -- 172.21.15.29:0/1477525908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400bc00 con 0x7f8aa0159b20 2023-10-07T09:29:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 -- 172.21.15.29:0/1477525908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159b20 msgr2=0x7f8aa00ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/1477525908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159b20 0x7f8aa00ce190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00d0d60 tx=0x7f8a94009250 comp rx=0 tx=0).stop 2023-10-07T09:29:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 -- 172.21.15.29:0/1477525908 shutdown_connections 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/1477525908 >> v2:172.21.15.144:3300/0 conn(0x7f8aa00ce6d0 0x7f8aa00ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/1477525908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa0159750 0x7f8aa00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/1477525908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159b20 0x7f8aa00ce190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 -- 172.21.15.29:0/1477525908 >> 172.21.15.29:0/1477525908 conn(0x7f8aa0063a10 msgr2=0x7f8aa00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.439+0000 7f8aa5bff700 1 -- 172.21.15.29:0/1477525908 shutdown_connections 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.440+0000 7f8aa5bff700 1 -- 172.21.15.29:0/1477525908 wait complete. 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.441+0000 7f8aa5bff700 1 Processor -- start 2023-10-07T09:29:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.441+0000 7f8aa5bff700 1 -- start start 2023-10-07T09:29:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8aa5bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa00ce6d0 0x7f8aa0159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8aa5bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 0x7f8aa00c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8aa5bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0159b20 0x7f8aa00c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8aa5bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00c8930 con 0x7f8aa0159750 2023-10-07T09:29:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8aa5bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00c8650 con 0x7f8aa0159b20 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8aa5bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00b49b0 con 0x7f8aa00ce6d0 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8a9effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 0x7f8aa00c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8a9effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 0x7f8aa00c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56832/0 (socket says 172.21.15.29:56832) 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.442+0000 7f8a9effd700 1 -- 172.21.15.29:0/3170999624 learned_addr learned my addr 172.21.15.29:0/3170999624 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9f7fe700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa00ce6d0 0x7f8aa0159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9ffff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0159b20 0x7f8aa00c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9effd700 1 -- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa00ce6d0 msgr2=0x7f8aa0159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9effd700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa00ce6d0 0x7f8aa0159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9effd700 1 -- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0159b20 msgr2=0x7f8aa00c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9effd700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0159b20 0x7f8aa00c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9effd700 1 -- 172.21.15.29:0/3170999624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a940094f0 con 0x7f8aa0159750 2023-10-07T09:29:41.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9f7fe700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa00ce6d0 0x7f8aa0159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9ffff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0159b20 0x7f8aa00c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a9effd700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 0x7f8aa00c7fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009290 tx=0x7f8a94017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400bbe0 con 0x7f8aa0159750 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9401f070 con 0x7f8aa0159750 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94026690 con 0x7f8aa0159750 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa00b4c90 con 0x7f8aa0159750 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.443+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0153f60 con 0x7f8aa0159750 2023-10-07T09:29:41.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.444+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa004c3c0 con 0x7f8aa0159750 2023-10-07T09:29:41.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.445+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8a9401a430 con 0x7f8aa0159750 2023-10-07T09:29:41.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.446+0000 7f8a877fe700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8a8c066f30 0x7f8a8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.446+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8a94021070 con 0x7f8aa0159750 2023-10-07T09:29:41.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.446+0000 7f8a9f7fe700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8a8c066f30 0x7f8a8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.447+0000 7f8a9f7fe700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8a8c066f30 0x7f8a8c0693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a90006ea0 tx=0x7f8a90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.450+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a94053550 con 0x7f8aa0159750 2023-10-07T09:29:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.652+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8aa0064040 con 0x7f8aa0159750 2023-10-07T09:29:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.653+0000 7f8a877fe700 1 -- 172.21.15.29:0/3170999624 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a94056b70 con 0x7f8aa0159750 2023-10-07T09:29:41.653 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:29:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8a8c066f30 msgr2=0x7f8a8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8a8c066f30 0x7f8a8c0693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a90006ea0 tx=0x7f8a90008040 comp rx=0 tx=0).stop 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 msgr2=0x7f8aa00c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 0x7f8aa00c7fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009290 tx=0x7f8a94017000 comp rx=0 tx=0).stop 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 shutdown_connections 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8a8c066f30 0x7f8a8c0693e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0159b20 0x7f8aa00c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8aa00ce6d0 0x7f8aa0159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 --2- 172.21.15.29:0/3170999624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa0159750 0x7f8aa00c7fc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 >> 172.21.15.29:0/3170999624 conn(0x7f8aa0063a10 msgr2=0x7f8aa00ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 shutdown_connections 2023-10-07T09:29:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.655+0000 7f8aa5bff700 1 -- 172.21.15.29:0/3170999624 wait complete. 2023-10-07T09:29:41.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:29:41.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.871+0000 7fe39d1ff700 1 Processor -- start 2023-10-07T09:29:41.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.871+0000 7fe39d1ff700 1 -- start start 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.872+0000 7fe39d1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159750 0x7fe3980d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.872+0000 7fe39d1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3980ce8a0 0x7fe3980c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.872+0000 7fe39d1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3980cbf20 con 0x7fe398159b20 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.872+0000 7fe39d1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3980cc200 con 0x7fe3980ce8a0 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.872+0000 7fe39d1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3980cc4e0 con 0x7fe398159750 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.872+0000 7fe39759e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3980ce8a0 0x7fe3980c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3980ce8a0 0x7fe3980c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56844/0 (socket says 172.21.15.29:56844) 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 -- 172.21.15.29:0/1441913427 learned_addr learned my addr 172.21.15.29:0/1441913427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe396d9d700 1 --2- 172.21.15.29:0/1441913427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159750 0x7fe3980d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 -- 172.21.15.29:0/1441913427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159750 msgr2=0x7fe3980d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:41.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 --2- 172.21.15.29:0/1441913427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159750 0x7fe3980d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 -- 172.21.15.29:0/1441913427 >> v1:172.21.15.144:6789/0 conn(0x7fe398159b20 legacy=0x7fe3980ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 -- 172.21.15.29:0/1441913427 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980cc7c0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe39759e700 1 --2- 172.21.15.29:0/1441913427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe3980c5460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe390009f20 tx=0x7fe390009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3475a04b5e4026 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe395d9b700 1 -- 172.21.15.29:0/1441913427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39000e070 con 0x7fe3980ce8a0 2023-10-07T09:29:41.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe395d9b700 1 -- 172.21.15.29:0/1441913427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39000b9d0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.873+0000 7fe395d9b700 1 -- 172.21.15.29:0/1441913427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900092e0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 -- 172.21.15.29:0/1441913427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 msgr2=0x7fe3980c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/1441913427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe3980c5460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe390009f20 tx=0x7fe390009840 comp rx=0 tx=0).stop 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 -- 172.21.15.29:0/1441913427 shutdown_connections 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/1441913427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe3980c5460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/1441913427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159750 0x7fe3980d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 -- 172.21.15.29:0/1441913427 >> 172.21.15.29:0/1441913427 conn(0x7fe398063a10 msgr2=0x7fe3980d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:41.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 -- 172.21.15.29:0/1441913427 shutdown_connections 2023-10-07T09:29:41.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.874+0000 7fe39d1ff700 1 -- 172.21.15.29:0/1441913427 wait complete. 2023-10-07T09:29:41.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.876+0000 7fe39d1ff700 1 Processor -- start 2023-10-07T09:29:41.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.876+0000 7fe39d1ff700 1 -- start start 2023-10-07T09:29:41.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39d1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe39815fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39d1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe398159750 0x7fe398152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39d1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159b20 0x7fe3981525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39d1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39816a9e0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39d1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39816ae70 con 0x7fe398159750 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39d1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39816a0b0 con 0x7fe398159b20 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe39815fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe39815fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56846/0 (socket says 172.21.15.29:56846) 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 -- 172.21.15.29:0/915831664 learned_addr learned my addr 172.21.15.29:0/915831664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39759e700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159b20 0x7fe3981525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39659c700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe398159750 0x7fe398152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 -- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159b20 msgr2=0x7fe3981525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159b20 0x7fe3981525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 -- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe398159750 msgr2=0x7fe398152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe398159750 0x7fe398152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe396d9d700 1 -- 172.21.15.29:0/915831664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe390009ae0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.878+0000 7fe39659c700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe398159750 0x7fe398152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.879+0000 7fe396d9d700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe39815fb50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe38800ec80 tx=0x7fe388009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.879+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38800c850 con 0x7fe3980ce8a0 2023-10-07T09:29:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.879+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38800c9b0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.879+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38800cd30 con 0x7fe3980ce8a0 2023-10-07T09:29:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.879+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39816a390 con 0x7fe3980ce8a0 2023-10-07T09:29:41.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.879+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39816e330 con 0x7fe3980ce8a0 2023-10-07T09:29:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.880+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39816acc0 con 0x7fe3980ce8a0 2023-10-07T09:29:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.881+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe388013680 con 0x7fe3980ce8a0 2023-10-07T09:29:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.882+0000 7fe37effd700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe384066f30 0x7fe3840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.882+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe388084c50 con 0x7fe3980ce8a0 2023-10-07T09:29:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.883+0000 7fe39659c700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe384066f30 0x7fe3840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:41.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.883+0000 7fe39659c700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe384066f30 0x7fe3840693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe38c007fd0 tx=0x7fe38c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:41.886+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe388051e20 con 0x7fe3980ce8a0 2023-10-07T09:29:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.088+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe39816a1f0 con 0x7fe3980ce8a0 2023-10-07T09:29:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.089+0000 7fe37effd700 1 -- 172.21.15.29:0/915831664 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe39816a1f0 con 0x7fe3980ce8a0 2023-10-07T09:29:42.089 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe384066f30 msgr2=0x7fe3840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe384066f30 0x7fe3840693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe38c007fd0 tx=0x7fe38c009040 comp rx=0 tx=0).stop 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 msgr2=0x7fe39815fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe39815fb50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe38800ec80 tx=0x7fe388009120 comp rx=0 tx=0).stop 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 shutdown_connections 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe384066f30 0x7fe3840693e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe398159b20 0x7fe3981525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3980ce8a0 0x7fe39815fb50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 --2- 172.21.15.29:0/915831664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe398159750 0x7fe398152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.091+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 >> 172.21.15.29:0/915831664 conn(0x7fe398063a10 msgr2=0x7fe398154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.092+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 shutdown_connections 2023-10-07T09:29:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.092+0000 7fe39d1ff700 1 -- 172.21.15.29:0/915831664 wait complete. 2023-10-07T09:29:42.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:29:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.305+0000 7fa56ffff700 1 Processor -- start 2023-10-07T09:29:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.305+0000 7fa56ffff700 1 -- start start 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5680d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa568159b20 0x7fa5680ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56ffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5680cbf20 con 0x7fa5680ce6d0 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56ffff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5680cc200 con 0x7fa568159b20 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680cc4e0 con 0x7fa568159750 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa568159b20 0x7fa5680ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa568159b20 0x7fa5680ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56860/0 (socket says 172.21.15.29:56860) 2023-10-07T09:29:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 -- 172.21.15.29:0/2061714510 learned_addr learned my addr 172.21.15.29:0/2061714510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56e59c700 1 --1- 172.21.15.29:0/2061714510 >> v1:172.21.15.29:6789/0 conn(0x7fa5680ce6d0 0x7fa56815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58808/0 (socket says 172.21.15.29:58808) 2023-10-07T09:29:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56dd9b700 1 --2- 172.21.15.29:0/2061714510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5680d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 -- 172.21.15.29:0/2061714510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 msgr2=0x7fa5680d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 --2- 172.21.15.29:0/2061714510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5680d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 -- 172.21.15.29:0/2061714510 >> v1:172.21.15.29:6789/0 conn(0x7fa5680ce6d0 legacy=0x7fa56815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56d59a700 1 -- 172.21.15.29:0/2061714510 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5680cc7c0 con 0x7fa568159b20 2023-10-07T09:29:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.307+0000 7fa56cd99700 1 -- 172.21.15.29:0/2061714510 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572880935 0 0) 0x7fa5680cbf20 con 0x7fa5680ce6d0 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56d59a700 1 --2- 172.21.15.29:0/2061714510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa5680ce190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa5580099b0 tx=0x7fa558009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550d07c5c0bfef13 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56cd99700 1 -- 172.21.15.29:0/2061714510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558009ea0 con 0x7fa568159b20 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56cd99700 1 -- 172.21.15.29:0/2061714510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55800b860 con 0x7fa568159b20 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56cd99700 1 -- 172.21.15.29:0/2061714510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55800bc00 con 0x7fa568159b20 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56ffff700 1 -- 172.21.15.29:0/2061714510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 msgr2=0x7fa5680ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56ffff700 1 --2- 172.21.15.29:0/2061714510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa5680ce190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa5580099b0 tx=0x7fa558009250 comp rx=0 tx=0).stop 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56ffff700 1 -- 172.21.15.29:0/2061714510 shutdown_connections 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56ffff700 1 --2- 172.21.15.29:0/2061714510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5680d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.308+0000 7fa56ffff700 1 --2- 172.21.15.29:0/2061714510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa5680ce190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.309+0000 7fa56ffff700 1 -- 172.21.15.29:0/2061714510 >> 172.21.15.29:0/2061714510 conn(0x7fa568063a10 msgr2=0x7fa5680d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:42.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.309+0000 7fa56ffff700 1 -- 172.21.15.29:0/2061714510 shutdown_connections 2023-10-07T09:29:42.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.309+0000 7fa56ffff700 1 -- 172.21.15.29:0/2061714510 wait complete. 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.310+0000 7fa56ffff700 1 Processor -- start 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.310+0000 7fa56ffff700 1 -- start start 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5680ce6d0 0x7fa568154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5681470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa56815ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681475f0 con 0x7fa568159b20 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56815f2d0 con 0x7fa5680ce6d0 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56815e9a0 con 0x7fa568159750 2023-10-07T09:29:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa56815ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa56815ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56870/0 (socket says 172.21.15.29:56870) 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.311+0000 7fa56e59c700 1 -- 172.21.15.29:0/3216096764 learned_addr learned my addr 172.21.15.29:0/3216096764 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56d59a700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5681470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56dd9b700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5680ce6d0 0x7fa568154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56e59c700 1 -- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 msgr2=0x7fa5681470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56e59c700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5681470b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56e59c700 1 -- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5680ce6d0 msgr2=0x7fa568154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56e59c700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5680ce6d0 0x7fa568154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56e59c700 1 -- 172.21.15.29:0/3216096764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580094f0 con 0x7fa568159b20 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56dd9b700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5680ce6d0 0x7fa568154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56e59c700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa56815ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa56000cd50 tx=0x7fa560001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560005df0 con 0x7fa568159b20 2023-10-07T09:29:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56000a960 con 0x7fa568159b20 2023-10-07T09:29:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56815ec80 con 0x7fa568159b20 2023-10-07T09:29:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000ad20 con 0x7fa568159b20 2023-10-07T09:29:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.312+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56815b280 con 0x7fa568159b20 2023-10-07T09:29:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.313+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56815f440 con 0x7fa568159b20 2023-10-07T09:29:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.314+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa560007300 con 0x7fa568159b20 2023-10-07T09:29:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.315+0000 7fa55dffb700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa54c066f30 0x7fa54c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.315+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa5600847a0 con 0x7fa568159b20 2023-10-07T09:29:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.315+0000 7fa56dd9b700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa54c066f30 0x7fa54c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.316+0000 7fa56dd9b700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa54c066f30 0x7fa54c0693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa564006ea0 tx=0x7fa564008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.319+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5600519f0 con 0x7fa568159b20 2023-10-07T09:29:42.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.565+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa56804c3c0 con 0x7fa568159b20 2023-10-07T09:29:42.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.566+0000 7fa55dffb700 1 -- 172.21.15.29:0/3216096764 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7fa560055010 con 0x7fa568159b20 2023-10-07T09:29:42.566 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:42.566 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:29:42.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa54c066f30 msgr2=0x7fa54c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa54c066f30 0x7fa54c0693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa564006ea0 tx=0x7fa564008040 comp rx=0 tx=0).stop 2023-10-07T09:29:42.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 msgr2=0x7fa56815ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa56815ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa56000cd50 tx=0x7fa560001b10 comp rx=0 tx=0).stop 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 shutdown_connections 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa54c066f30 0x7fa54c0693e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa568159b20 0x7fa56815ad40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5680ce6d0 0x7fa568154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 --2- 172.21.15.29:0/3216096764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa568159750 0x7fa5681470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 >> 172.21.15.29:0/3216096764 conn(0x7fa568063a10 msgr2=0x7fa5680cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 shutdown_connections 2023-10-07T09:29:42.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.568+0000 7fa56ffff700 1 -- 172.21.15.29:0/3216096764 wait complete. 2023-10-07T09:29:42.570 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:29:42.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.780+0000 7f1ec29ff700 1 Processor -- start 2023-10-07T09:29:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.781+0000 7f1ec29ff700 1 -- start start 2023-10-07T09:29:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ec29ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ebc0ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ec29ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc15af00 con 0x7f1ebc0ad220 2023-10-07T09:29:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ec29ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc15a250 con 0x7f1ebc159c00 2023-10-07T09:29:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ec29ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc15a050 con 0x7f1ebc068d30 2023-10-07T09:29:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ebbfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1ebc0ad220 0x7f1ebc0ac490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58812/0 (socket says 172.21.15.29:58812) 2023-10-07T09:29:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ebbfff700 1 -- 172.21.15.29:0/3309235208 learned_addr learned my addr 172.21.15.29:0/3309235208 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.782+0000 7f1ec0f9c700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ebc0ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430903756 0 0) 0x7f1ebc15af00 con 0x7f1ebc0ad220 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea4003570 con 0x7f1ebc0ad220 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318763348 0 0) 0x7f1ebc15a250 con 0x7f1ebc159c00 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc15af00 con 0x7f1ebc159c00 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281682707 0 0) 0x7f1ea4003570 con 0x7f1ebc0ad220 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 msgr2=0x7f1ebc0ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ebc0ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 >> v1:172.21.15.144:6789/0 conn(0x7f1ebc159c00 legacy=0x7f1ebc068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc15a530 con 0x7f1ebc0ad220 2023-10-07T09:29:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f1eac001470 con 0x7f1ebc0ad220 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 >> v1:172.21.15.29:6789/0 conn(0x7f1ebc0ad220 legacy=0x7f1ebc0ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea4003c20 0x7f1ea4004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4004810 0x7f1ea4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ea4005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc15a050 con 0x7f1ea4003c20 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea400bbf0 con 0x7f1ea4004810 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea400bdb0 con 0x7f1ebc068d30 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea4003c20 0x7f1ea4004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ebb7fe700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4004810 0x7f1ea4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.783+0000 7f1ec0f9c700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ea4005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebbfff700 1 -- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 msgr2=0x7f1ea4005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ea4005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebbfff700 1 -- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4004810 msgr2=0x7f1ea4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4004810 0x7f1ea4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebbfff700 1 -- 172.21.15.29:0/3309235208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4003570 con 0x7f1ea4003c20 2023-10-07T09:29:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ec0f9c700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc068d30 0x7f1ea4005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebb7fe700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4004810 0x7f1ea4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea4003c20 0x7f1ea4004010 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c1e0 tx=0x7f1eac001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660ff644aba3ee3c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00cb10 con 0x7f1ea4003c20 2023-10-07T09:29:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00cc90 con 0x7f1ea4003c20 2023-10-07T09:29:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.784+0000 7f1ebaffd700 1 -- 172.21.15.29:0/3309235208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac005b70 con 0x7f1ea4003c20 2023-10-07T09:29:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec29ff700 1 -- 172.21.15.29:0/3309235208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea4003c20 msgr2=0x7f1ea4004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/3309235208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea4003c20 0x7f1ea4004010 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c1e0 tx=0x7f1eac001d40 comp rx=0 tx=0).stop 2023-10-07T09:29:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec0f9c700 1 -- 172.21.15.29:0/3309235208 reap_dead start 2023-10-07T09:29:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec29ff700 1 -- 172.21.15.29:0/3309235208 shutdown_connections 2023-10-07T09:29:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec29ff700 1 -- 172.21.15.29:0/3309235208 >> 172.21.15.29:0/3309235208 conn(0x7f1ebc06dd00 msgr2=0x7f1ebc06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec29ff700 1 -- 172.21.15.29:0/3309235208 shutdown_connections 2023-10-07T09:29:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.785+0000 7f1ec29ff700 1 -- 172.21.15.29:0/3309235208 wait complete. 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.786+0000 7f1ec29ff700 1 Processor -- start 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.786+0000 7f1ec29ff700 1 -- start start 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec29ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4003c20 0x7f1ebc0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec29ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0c8c30 0x7f1ebc15c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec29ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 0x7f1ebc1454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec29ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0b2650 con 0x7f1ebc145070 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec29ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc145a20 con 0x7f1ea4003c20 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec29ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c91f0 con 0x7f1ebc0c8c30 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.787+0000 7f1ec0f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 0x7f1ebc1454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 0x7f1ebc1454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56894/0 (socket says 172.21.15.29:56894) 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 -- 172.21.15.29:0/2112854468 learned_addr learned my addr 172.21.15.29:0/2112854468 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4003c20 0x7f1ebc0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ebb7fe700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0c8c30 0x7f1ebc15c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 -- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0c8c30 msgr2=0x7f1ebc15c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0c8c30 0x7f1ebc15c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 -- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4003c20 msgr2=0x7f1ebc0b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4003c20 0x7f1ebc0b2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 -- 172.21.15.29:0/2112854468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0c94d0 con 0x7f1ebc145070 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ebb7fe700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0c8c30 0x7f1ebc15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec0f9c700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 0x7f1ebc1454e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc069130 tx=0x7f1eb400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb400e030 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb4011d00 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0c97b0 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.788+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40166c0 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.789+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc0c9d70 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.790+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc04c3c0 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.791+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1eb4016ab0 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.791+0000 7f1ea3fff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.792+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1eb400f070 con 0x7f1ebc145070 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.792+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.792+0000 7f1ebbfff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1eac007ee0 tx=0x7f1eac015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.796+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eb4089070 con 0x7f1ebc145070 2023-10-07T09:29:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.995+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ebc15a810 con 0x7f1ebc145070 2023-10-07T09:29:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.996+0000 7f1ea3fff700 1 -- 172.21.15.29:0/2112854468 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f1eb4055bd0 con 0x7f1ebc145070 2023-10-07T09:29:42.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:42.996 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1ea8066f30 msgr2=0x7f1ea80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1eac007ee0 tx=0x7f1eac015000 comp rx=0 tx=0).stop 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 msgr2=0x7f1ebc1454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 0x7f1ebc1454e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc069130 tx=0x7f1eb400b040 comp rx=0 tx=0).stop 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 shutdown_connections 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc145070 0x7f1ebc1454e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4003c20 0x7f1ebc0b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 --2- 172.21.15.29:0/2112854468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0c8c30 0x7f1ebc15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.998+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 >> 172.21.15.29:0/2112854468 conn(0x7f1ebc06dd00 msgr2=0x7f1ebc1533a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.999+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 shutdown_connections 2023-10-07T09:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:42.999+0000 7f1ec29ff700 1 -- 172.21.15.29:0/2112854468 wait complete. 2023-10-07T09:29:43.016 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:29:43.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:29:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.215+0000 7f22df7ff700 1 Processor -- start 2023-10-07T09:29:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.215+0000 7f22df7ff700 1 -- start start 2023-10-07T09:29:43.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.216+0000 7f22df7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22d815ade0 0x7f22d80b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.216+0000 7f22df7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f22d80b0540 0x7f22d815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.216+0000 7f22df7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d8154090 con 0x7f22d8153cc0 2023-10-07T09:29:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.216+0000 7f22df7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d80b09e0 con 0x7f22d815ade0 2023-10-07T09:29:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.216+0000 7f22df7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d8159f60 con 0x7f22d80b0540 2023-10-07T09:29:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22d815ade0 0x7f22d80b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22d815ade0 0x7f22d80b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35150/0 (socket says 172.21.15.29:35150) 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 -- 172.21.15.29:0/507607770 learned_addr learned my addr 172.21.15.29:0/507607770 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22ddd9c700 1 --2- 172.21.15.29:0/507607770 >> v2:172.21.15.144:3300/0 conn(0x7f22d80b0540 0x7f22d815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 -- 172.21.15.29:0/507607770 >> v2:172.21.15.144:3300/0 conn(0x7f22d80b0540 msgr2=0x7f22d815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 --2- 172.21.15.29:0/507607770 >> v2:172.21.15.144:3300/0 conn(0x7f22d80b0540 0x7f22d815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 -- 172.21.15.29:0/507607770 >> v1:172.21.15.144:6789/0 conn(0x7f22d8153cc0 legacy=0x7f22d80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 -- 172.21.15.29:0/507607770 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d815a240 con 0x7f22d815ade0 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22dcd9a700 1 --2- 172.21.15.29:0/507607770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d815ade0 0x7f22d80b0000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f22d806c400 tx=0x7f22c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da65cf28ba734d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.217+0000 7f22cffff700 1 -- 172.21.15.29:0/507607770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c800b860 con 0x7f22d815ade0 2023-10-07T09:29:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22cffff700 1 -- 172.21.15.29:0/507607770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c800b9c0 con 0x7f22d815ade0 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22cffff700 1 -- 172.21.15.29:0/507607770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c800bd60 con 0x7f22d815ade0 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 -- 172.21.15.29:0/507607770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d815ade0 msgr2=0x7f22d80b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 --2- 172.21.15.29:0/507607770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d815ade0 0x7f22d80b0000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f22d806c400 tx=0x7f22c8009250 comp rx=0 tx=0).stop 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 -- 172.21.15.29:0/507607770 shutdown_connections 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 --2- 172.21.15.29:0/507607770 >> v2:172.21.15.144:3300/0 conn(0x7f22d80b0540 0x7f22d815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 --2- 172.21.15.29:0/507607770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d815ade0 0x7f22d80b0000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 -- 172.21.15.29:0/507607770 >> 172.21.15.29:0/507607770 conn(0x7f22d8067610 msgr2=0x7f22d8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 -- 172.21.15.29:0/507607770 shutdown_connections 2023-10-07T09:29:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.218+0000 7f22df7ff700 1 -- 172.21.15.29:0/507607770 wait complete. 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.220+0000 7f22df7ff700 1 Processor -- start 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.220+0000 7f22df7ff700 1 -- start start 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22df7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 0x7f22d8158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22df7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22d8153cc0 0x7f22d80ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22df7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22d815ade0 0x7f22d8156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22df7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d813e760 con 0x7f22d80b0540 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22df7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d80ce9a0 con 0x7f22d8153cc0 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22df7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d81566c0 con 0x7f22d815ade0 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dd59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 0x7f22d8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dd59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 0x7f22d8158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35166/0 (socket says 172.21.15.29:35166) 2023-10-07T09:29:43.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dd59b700 1 -- 172.21.15.29:0/93413618 learned_addr learned my addr 172.21.15.29:0/93413618 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dcd9a700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22d8153cc0 0x7f22d80ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22ddd9c700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22d815ade0 0x7f22d8156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dd59b700 1 -- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22d815ade0 msgr2=0x7f22d8156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dd59b700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22d815ade0 0x7f22d8156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.221+0000 7f22dd59b700 1 -- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22d8153cc0 msgr2=0x7f22d80ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22dd59b700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22d8153cc0 0x7f22d80ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22dd59b700 1 -- 172.21.15.29:0/93413618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80094f0 con 0x7f22d80b0540 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22ddd9c700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22d815ade0 0x7f22d8156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22dd59b700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 0x7f22d8158320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c7c0 tx=0x7f22d4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4011590 con 0x7f22d80b0540 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4011710 con 0x7f22d80b0540 2023-10-07T09:29:43.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d8156950 con 0x7f22d80b0540 2023-10-07T09:29:43.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4011ad0 con 0x7f22d80b0540 2023-10-07T09:29:43.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.222+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d8156f10 con 0x7f22d80b0540 2023-10-07T09:29:43.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.223+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d804c3c0 con 0x7f22d80b0540 2023-10-07T09:29:43.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.224+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f22d401b3b0 con 0x7f22d80b0540 2023-10-07T09:29:43.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.225+0000 7f22cd7fa700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f22bc066f30 0x7f22bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.225+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f22d4083ad0 con 0x7f22d80b0540 2023-10-07T09:29:43.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.225+0000 7f22dcd9a700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f22bc066f30 0x7f22bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.226+0000 7f22dcd9a700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f22bc066f30 0x7f22bc0693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f22d806c400 tx=0x7f22c8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:43.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.229+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22d4050d50 con 0x7f22d80b0540 2023-10-07T09:29:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.431+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22d80689d0 con 0x7f22d80b0540 2023-10-07T09:29:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.432+0000 7f22cd7fa700 1 -- 172.21.15.29:0/93413618 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22d4017070 con 0x7f22d80b0540 2023-10-07T09:29:43.433 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:29:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f22bc066f30 msgr2=0x7f22bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f22bc066f30 0x7f22bc0693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f22d806c400 tx=0x7f22c8017000 comp rx=0 tx=0).stop 2023-10-07T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 msgr2=0x7f22d8158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 0x7f22d8158320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c7c0 tx=0x7f22d4007f70 comp rx=0 tx=0).stop 2023-10-07T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 shutdown_connections 2023-10-07T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f22bc066f30 0x7f22bc0693e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22d815ade0 0x7f22d8156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d80b0540 0x7f22d8158320 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 --2- 172.21.15.29:0/93413618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22d8153cc0 0x7f22d80ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 >> 172.21.15.29:0/93413618 conn(0x7f22d8067610 msgr2=0x7f22d815b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 shutdown_connections 2023-10-07T09:29:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.434+0000 7f22df7ff700 1 -- 172.21.15.29:0/93413618 wait complete. 2023-10-07T09:29:43.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:29:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.650+0000 7f6a6ebff700 1 Processor -- start 2023-10-07T09:29:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.650+0000 7f6a6ebff700 1 -- start start 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.651+0000 7f6a6ebff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a68159b20 0x7f6a680d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.651+0000 7f6a6ebff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 0x7f6a680c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.651+0000 7f6a6ebff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680cbf20 con 0x7f6a68159750 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.651+0000 7f6a6ebff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680cc200 con 0x7f6a68159b20 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.651+0000 7f6a6ebff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680cc4e0 con 0x7f6a680ce8a0 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a6c99b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6a68159750 0x7f6a680ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34288/0 (socket says 172.21.15.29:34288) 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a6c99b700 1 -- 172.21.15.29:0/2336756730 learned_addr learned my addr 172.21.15.29:0/2336756730 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a67fff700 1 --2- 172.21.15.29:0/2336756730 >> v2:172.21.15.144:3300/0 conn(0x7f6a68159b20 0x7f6a680d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a6d19c700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 0x7f6a680c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938581955 0 0) 0x7f6a680cbf20 con 0x7f6a68159750 2023-10-07T09:29:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a58003570 con 0x7f6a68159750 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698792827 0 0) 0x7f6a58003570 con 0x7f6a68159750 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 msgr2=0x7f6a680c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 0x7f6a680c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 >> v2:172.21.15.144:3300/0 conn(0x7f6a68159b20 msgr2=0x7f6a680d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 --2- 172.21.15.29:0/2336756730 >> v2:172.21.15.144:3300/0 conn(0x7f6a68159b20 0x7f6a680d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680cc7c0 con 0x7f6a68159750 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f6a5c0015f0 con 0x7f6a68159750 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a67fff700 1 --2- 172.21.15.29:0/2336756730 >> v2:172.21.15.144:3300/0 conn(0x7f6a68159b20 0x7f6a680d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 >> v1:172.21.15.29:6789/0 conn(0x7f6a68159750 legacy=0x7f6a680ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a58003a60 0x7f6a58003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 0x7f6a58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a58004d50 0x7f6a580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680cc200 con 0x7f6a58003a60 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680cc4e0 con 0x7f6a58004d50 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.652+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5800ba60 con 0x7f6a680ce8a0 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a58003a60 0x7f6a58003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a67fff700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 0x7f6a58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6d19c700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a58004d50 0x7f6a580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 -- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 msgr2=0x7f6a58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a680ce8a0 0x7f6a58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 -- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a58004d50 msgr2=0x7f6a580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a58004d50 0x7f6a580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 -- 172.21.15.29:0/2336756730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58003570 con 0x7f6a58003a60 2023-10-07T09:29:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.653+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a58003a60 0x7f6a58003e70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00c3c0 tx=0x7f6a5c007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9918039b8acc896 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00cc30 con 0x7f6a58003a60 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c00cd90 con 0x7f6a58003a60 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a677fe700 1 -- 172.21.15.29:0/2336756730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c007320 con 0x7f6a58003a60 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6ebff700 1 -- 172.21.15.29:0/2336756730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a58003a60 msgr2=0x7f6a58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/2336756730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a58003a60 0x7f6a58003e70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00c3c0 tx=0x7f6a5c007920 comp rx=0 tx=0).stop 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6d19c700 1 -- 172.21.15.29:0/2336756730 reap_dead start 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6ebff700 1 -- 172.21.15.29:0/2336756730 shutdown_connections 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6ebff700 1 -- 172.21.15.29:0/2336756730 >> 172.21.15.29:0/2336756730 conn(0x7f6a68063a10 msgr2=0x7f6a680d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6ebff700 1 -- 172.21.15.29:0/2336756730 shutdown_connections 2023-10-07T09:29:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.654+0000 7f6a6ebff700 1 -- 172.21.15.29:0/2336756730 wait complete. 2023-10-07T09:29:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.656+0000 7f6a6ebff700 1 Processor -- start 2023-10-07T09:29:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.657+0000 7f6a6ebff700 1 -- start start 2023-10-07T09:29:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6ebff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a58003a60 0x7f6a68153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6ebff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 0x7f6a680c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6ebff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a680c8570 0x7f6a680c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6ebff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b3360 con 0x7f6a680c9f10 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6ebff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b3640 con 0x7f6a680c8570 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6ebff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b3920 con 0x7f6a58003a60 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 0x7f6a680c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 0x7f6a680c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35182/0 (socket says 172.21.15.29:35182) 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 -- 172.21.15.29:0/3902028362 learned_addr learned my addr 172.21.15.29:0/3902028362 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a58003a60 0x7f6a68153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6d19c700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a680c8570 0x7f6a680c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 -- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a58003a60 msgr2=0x7f6a68153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a58003a60 0x7f6a68153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 -- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a680c8570 msgr2=0x7f6a680c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a680c8570 0x7f6a680c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a67fff700 1 -- 172.21.15.29:0/3902028362 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680b3c00 con 0x7f6a680c9f10 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a58003a60 0x7f6a68153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.658+0000 7f6a6d19c700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a680c8570 0x7f6a680c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.659+0000 7f6a67fff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 0x7f6a680c8030 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6a54011d70 tx=0x7f6a540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.659+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5400fcd0 con 0x7f6a680c9f10 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.659+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5400fe30 con 0x7f6a680c9f10 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.659+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54014670 con 0x7f6a680c9f10 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.659+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a680b3ee0 con 0x7f6a680c9f10 2023-10-07T09:29:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.659+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a680c58a0 con 0x7f6a680c9f10 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.660+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6804c3c0 con 0x7f6a680c9f10 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.661+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6a54014aa0 con 0x7f6a680c9f10 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.662+0000 7f6a64ff9700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6a4c066f30 0x7f6a4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.662+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6a54018030 con 0x7f6a680c9f10 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.662+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6a4c066f30 0x7f6a4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.663+0000 7f6a6c99b700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6a4c066f30 0x7f6a4c0693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c008fd0 tx=0x7f6a5c015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.666+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a54052720 con 0x7f6a680c9f10 2023-10-07T09:29:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.871+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a680cb450 con 0x7f6a680c9f10 2023-10-07T09:29:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.871+0000 7f6a64ff9700 1 -- 172.21.15.29:0/3902028362 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a54006cb0 con 0x7f6a680c9f10 2023-10-07T09:29:43.872 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:29:43.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6a4c066f30 msgr2=0x7f6a4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6a4c066f30 0x7f6a4c0693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c008fd0 tx=0x7f6a5c015000 comp rx=0 tx=0).stop 2023-10-07T09:29:43.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 msgr2=0x7f6a680c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 0x7f6a680c8030 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6a54011d70 tx=0x7f6a540091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 shutdown_connections 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6a4c066f30 0x7f6a4c0693e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a680c8570 0x7f6a680c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a58003a60 0x7f6a68153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 --2- 172.21.15.29:0/3902028362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a680c9f10 0x7f6a680c8030 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 >> 172.21.15.29:0/3902028362 conn(0x7f6a68063a10 msgr2=0x7f6a680cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 shutdown_connections 2023-10-07T09:29:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:43.874+0000 7f6a6ebff700 1 -- 172.21.15.29:0/3902028362 wait complete. 2023-10-07T09:29:43.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:29:44.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.090+0000 7f155e3ff700 1 Processor -- start 2023-10-07T09:29:44.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.090+0000 7f155e3ff700 1 -- start start 2023-10-07T09:29:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 0x7f155806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155e3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f155806b3f0 0x7f15580c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155e3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155806a430 con 0x7f155806a8c0 2023-10-07T09:29:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155e3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155806b830 con 0x7f155806b3f0 2023-10-07T09:29:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1558159b60 con 0x7f1558159790 2023-10-07T09:29:44.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f1557fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 0x7f155806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155c99c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f155806b3f0 0x7f15580c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155c99c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f155806b3f0 0x7f15580c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:39232/0 (socket says 172.21.15.29:39232) 2023-10-07T09:29:44.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f155c99c700 1 -- 172.21.15.29:0/380469378 learned_addr learned my addr 172.21.15.29:0/380469378 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.092+0000 7f1557fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 0x7f155806aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:58438/0 (socket says 172.21.15.29:58438) 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f155c99c700 1 -- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 msgr2=0x7f155806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f155c99c700 1 --2- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 0x7f155806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f155c99c700 1 -- 172.21.15.29:0/380469378 >> v1:172.21.15.144:6789/0 conn(0x7f155806a8c0 legacy=0x7f15580c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f155c99c700 1 -- 172.21.15.29:0/380469378 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1558069820 con 0x7f155806b3f0 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f1557fff700 1 --2- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 0x7f155806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f1556ffd700 1 -- 172.21.15.29:0/380469378 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146160424 0 0) 0x7f155806a430 con 0x7f155806a8c0 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f155c99c700 1 --2- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155806b3f0 0x7f15580c7650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1550009f20 tx=0x7f1550009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291c3702d24edbb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f1556ffd700 1 -- 172.21.15.29:0/380469378 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000e070 con 0x7f155806b3f0 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f1556ffd700 1 -- 172.21.15.29:0/380469378 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155000b9d0 con 0x7f155806b3f0 2023-10-07T09:29:44.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.093+0000 7f1556ffd700 1 -- 172.21.15.29:0/380469378 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15500092e0 con 0x7f155806b3f0 2023-10-07T09:29:44.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 -- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155806b3f0 msgr2=0x7f15580c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 --2- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155806b3f0 0x7f15580c7650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1550009f20 tx=0x7f1550009840 comp rx=0 tx=0).stop 2023-10-07T09:29:44.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 -- 172.21.15.29:0/380469378 shutdown_connections 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 --2- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155806b3f0 0x7f15580c7650 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 --2- 172.21.15.29:0/380469378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1558159790 0x7f155806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 -- 172.21.15.29:0/380469378 >> 172.21.15.29:0/380469378 conn(0x7f15580c9410 msgr2=0x7f15580c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 -- 172.21.15.29:0/380469378 shutdown_connections 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.094+0000 7f155e3ff700 1 -- 172.21.15.29:0/380469378 wait complete. 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.095+0000 7f155e3ff700 1 Processor -- start 2023-10-07T09:29:44.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 -- start start 2023-10-07T09:29:44.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 0x7f15580aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1558159790 0x7f15580c2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15580aa2f0 0x7f155815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580ad780 con 0x7f155806b3f0 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580ae5a0 con 0x7f1558159790 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.096+0000 7f155e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580aef70 con 0x7f15580aa2f0 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 0x7f15580aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 0x7f15580aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35188/0 (socket says 172.21.15.29:35188) 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 -- 172.21.15.29:0/3087701908 learned_addr learned my addr 172.21.15.29:0/3087701908 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f155c99c700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15580aa2f0 0x7f155815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f15577fe700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1558159790 0x7f15580c2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 -- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15580aa2f0 msgr2=0x7f155815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15580aa2f0 0x7f155815b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 -- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1558159790 msgr2=0x7f15580c2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1558159790 0x7f15580c2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 -- 172.21.15.29:0/3087701908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548009420 con 0x7f155806b3f0 2023-10-07T09:29:44.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f155c99c700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15580aa2f0 0x7f155815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:44.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.097+0000 7f1557fff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 0x7f15580aea30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f154800ec80 tx=0x7f15480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.098+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154800c9d0 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.098+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154800cb30 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.098+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154800ceb0 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.098+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1550009ae0 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.098+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15580adc70 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.099+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155804c3c0 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.100+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1548013680 con 0x7f155806b3f0 2023-10-07T09:29:44.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.100+0000 7f154ffff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1538066f30 0x7f15380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.101+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1548084a20 con 0x7f155806b3f0 2023-10-07T09:29:44.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.101+0000 7f15577fe700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1538066f30 0x7f15380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.101+0000 7f15577fe700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1538066f30 0x7f15380693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1558068ca0 tx=0x7f1540009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:44.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.105+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1548051bf0 con 0x7f155806b3f0 2023-10-07T09:29:44.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.308+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1558068710 con 0x7f155806b3f0 2023-10-07T09:29:44.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.308+0000 7f154ffff700 1 -- 172.21.15.29:0/3087701908 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1548055210 con 0x7f155806b3f0 2023-10-07T09:29:44.308 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1538066f30 msgr2=0x7f15380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1538066f30 0x7f15380693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1558068ca0 tx=0x7f1540009040 comp rx=0 tx=0).stop 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 msgr2=0x7f15580aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 0x7f15580aea30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f154800ec80 tx=0x7f15480091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 shutdown_connections 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1538066f30 0x7f15380693e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15580aa2f0 0x7f155815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f155806b3f0 0x7f15580aea30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 --2- 172.21.15.29:0/3087701908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1558159790 0x7f15580c2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.310+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 >> 172.21.15.29:0/3087701908 conn(0x7f15580c9410 msgr2=0x7f1558067610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:44.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.311+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 shutdown_connections 2023-10-07T09:29:44.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.311+0000 7f155e3ff700 1 -- 172.21.15.29:0/3087701908 wait complete. 2023-10-07T09:29:44.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:29:44.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.524+0000 7f894d9ff700 1 Processor -- start 2023-10-07T09:29:44.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.524+0000 7f894d9ff700 1 -- start start 2023-10-07T09:29:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f894d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8948159750 0x7f89480d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f894d9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8948159b20 0x7f89480ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f894d9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480cbf20 con 0x7f89480ce6d0 2023-10-07T09:29:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f894d9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89480cc200 con 0x7f8948159b20 2023-10-07T09:29:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f894d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89480cc4e0 con 0x7f8948159750 2023-10-07T09:29:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89477fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f89480ce6d0 0x7f894815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34292/0 (socket says 172.21.15.29:34292) 2023-10-07T09:29:44.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89477fe700 1 -- 172.21.15.29:0/1246355144 learned_addr learned my addr 172.21.15.29:0/1246355144 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:44.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f8946ffd700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8948159750 0x7f89480d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89467fc700 1 --2- 172.21.15.29:0/1246355144 >> v2:172.21.15.144:3300/0 conn(0x7f8948159b20 0x7f89480ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f8945ffb700 1 -- 172.21.15.29:0/1246355144 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011898231 0 0) 0x7f89480cbf20 con 0x7f89480ce6d0 2023-10-07T09:29:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f8945ffb700 1 -- 172.21.15.29:0/1246355144 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8930003570 con 0x7f89480ce6d0 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89467fc700 1 -- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8948159750 msgr2=0x7f89480d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89467fc700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8948159750 0x7f89480d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89467fc700 1 -- 172.21.15.29:0/1246355144 >> v1:172.21.15.29:6789/0 conn(0x7f89480ce6d0 legacy=0x7f894815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.526+0000 7f89467fc700 1 -- 172.21.15.29:0/1246355144 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480cc7c0 con 0x7f8948159b20 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.527+0000 7f8946ffd700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8948159750 0x7f89480d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.527+0000 7f89467fc700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159b20 0x7f89480ce190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f893c0098e0 tx=0x7f893c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=943f4f08bfc0d736 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.527+0000 7f8945ffb700 1 -- 172.21.15.29:0/1246355144 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c011630 con 0x7f8948159b20 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.527+0000 7f8945ffb700 1 -- 172.21.15.29:0/1246355144 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c011790 con 0x7f8948159b20 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.527+0000 7f8945ffb700 1 -- 172.21.15.29:0/1246355144 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c011b30 con 0x7f8948159b20 2023-10-07T09:29:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 -- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159b20 msgr2=0x7f89480ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159b20 0x7f89480ce190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f893c0098e0 tx=0x7f893c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 -- 172.21.15.29:0/1246355144 shutdown_connections 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8948159750 0x7f89480d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 --2- 172.21.15.29:0/1246355144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159b20 0x7f89480ce190 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 -- 172.21.15.29:0/1246355144 >> 172.21.15.29:0/1246355144 conn(0x7f8948063a10 msgr2=0x7f89480d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 -- 172.21.15.29:0/1246355144 shutdown_connections 2023-10-07T09:29:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.528+0000 7f894d9ff700 1 -- 172.21.15.29:0/1246355144 wait complete. 2023-10-07T09:29:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.529+0000 7f894d9ff700 1 Processor -- start 2023-10-07T09:29:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 -- start start 2023-10-07T09:29:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89480ce6d0 0x7f89480c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 0x7f89480c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 0x7f894814b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89480c8fa0 con 0x7f8948159b20 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89480c9e00 con 0x7f8948159750 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.530+0000 7f894d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89480c7d00 con 0x7f89480ce6d0 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89477fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 0x7f894814b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 0x7f89480c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f8946ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89480ce6d0 0x7f89480c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 0x7f89480c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:39256/0 (socket says 172.21.15.29:39256) 2023-10-07T09:29:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 -- 172.21.15.29:0/2802519094 learned_addr learned my addr 172.21.15.29:0/2802519094 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89477fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 0x7f894814b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35196/0 (socket says 172.21.15.29:35196) 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 -- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89480ce6d0 msgr2=0x7f89480c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89480ce6d0 0x7f89480c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 -- 172.21.15.29:0/2802519094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 msgr2=0x7f894814b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 0x7f894814b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89467fc700 1 -- 172.21.15.29:0/2802519094 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380093c0 con 0x7f8948159750 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.531+0000 7f89477fe700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 0x7f894814b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.532+0000 7f89467fc700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 0x7f89480c8a60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f893c009ed0 tx=0x7f893c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.532+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c011900 con 0x7f8948159750 2023-10-07T09:29:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.532+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c011a60 con 0x7f8948159750 2023-10-07T09:29:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.532+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c009420 con 0x7f8948159750 2023-10-07T09:29:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.532+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c025640 con 0x7f8948159750 2023-10-07T09:29:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.532+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89480c97b0 con 0x7f8948159750 2023-10-07T09:29:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.533+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894804c3c0 con 0x7f8948159750 2023-10-07T09:29:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.534+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f893c025a30 con 0x7f8948159750 2023-10-07T09:29:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.535+0000 7f892effd700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8934066f30 0x7f89340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.535+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f893c086140 con 0x7f8948159750 2023-10-07T09:29:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.535+0000 7f8946ffd700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8934066f30 0x7f89340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.536+0000 7f8946ffd700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8934066f30 0x7f89340693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8938009ea0 tx=0x7f8938009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.539+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f893c053390 con 0x7f8948159750 2023-10-07T09:29:44.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.785+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89480c9f40 con 0x7f8948159750 2023-10-07T09:29:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.786+0000 7f892effd700 1 -- 172.21.15.29:0/2802519094 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f893c0569b0 con 0x7f8948159750 2023-10-07T09:29:44.786 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:44.786 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8934066f30 msgr2=0x7f89340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8934066f30 0x7f89340693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8938009ea0 tx=0x7f8938009120 comp rx=0 tx=0).stop 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 msgr2=0x7f89480c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 0x7f89480c8a60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f893c009ed0 tx=0x7f893c017000 comp rx=0 tx=0).stop 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 shutdown_connections 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8934066f30 0x7f89340693e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8948159b20 0x7f894814b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89480ce6d0 0x7f89480c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 --2- 172.21.15.29:0/2802519094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8948159750 0x7f89480c8a60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 >> 172.21.15.29:0/2802519094 conn(0x7f8948063a10 msgr2=0x7f894815e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 shutdown_connections 2023-10-07T09:29:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:44.788+0000 7f894d9ff700 1 -- 172.21.15.29:0/2802519094 wait complete. 2023-10-07T09:29:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:29:44.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:45.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.001+0000 7f374c5ff700 1 Processor -- start 2023-10-07T09:29:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.001+0000 7f374c5ff700 1 -- start start 2023-10-07T09:29:45.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.002+0000 7f374c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f37440b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.002+0000 7f374c5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37440b0980 con 0x7f37440b0540 2023-10-07T09:29:45.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.002+0000 7f374c5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3744154090 con 0x7f3744153cc0 2023-10-07T09:29:45.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.002+0000 7f374c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3744159f60 con 0x7f374415ade0 2023-10-07T09:29:45.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.002+0000 7f374ab9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f37440b0540 0x7f374415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34294/0 (socket says 172.21.15.29:34294) 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f374ab9c700 1 -- 172.21.15.29:0/2939614020 learned_addr learned my addr 172.21.15.29:0/2939614020 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749b9a700 1 --2- 172.21.15.29:0/2939614020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f37440b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726097595 0 0) 0x7f37440b0980 con 0x7f37440b0540 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3730003570 con 0x7f37440b0540 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822885421 0 0) 0x7f3744154090 con 0x7f3744153cc0 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37440b0980 con 0x7f3744153cc0 2023-10-07T09:29:45.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749b9a700 1 -- 172.21.15.29:0/2939614020 >> v1:172.21.15.144:6789/0 conn(0x7f3744153cc0 legacy=0x7f37440b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749b9a700 1 -- 172.21.15.29:0/2939614020 >> v1:172.21.15.29:6789/0 conn(0x7f37440b0540 legacy=0x7f374415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706380863 0 0) 0x7f3730003570 con 0x7f37440b0540 2023-10-07T09:29:45.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.003+0000 7f3749b9a700 1 -- 172.21.15.29:0/2939614020 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374415a240 con 0x7f374415ade0 2023-10-07T09:29:45.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f3749b9a700 1 --2- 172.21.15.29:0/2939614020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f37440b0000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3734009910 tx=0x7f37340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7b3f69391357a59 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:45.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c0017f0 con 0x7f374415ade0 2023-10-07T09:29:45.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3734011420 con 0x7f374415ade0 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f3749399700 1 -- 172.21.15.29:0/2939614020 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37340117c0 con 0x7f374415ade0 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 -- 172.21.15.29:0/2939614020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 msgr2=0x7f37440b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 --2- 172.21.15.29:0/2939614020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f37440b0000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3734009910 tx=0x7f37340091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 -- 172.21.15.29:0/2939614020 shutdown_connections 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 --2- 172.21.15.29:0/2939614020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f37440b0000 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 -- 172.21.15.29:0/2939614020 >> 172.21.15.29:0/2939614020 conn(0x7f3744067610 msgr2=0x7f3744067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 -- 172.21.15.29:0/2939614020 shutdown_connections 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.004+0000 7f374c5ff700 1 -- 172.21.15.29:0/2939614020 wait complete. 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.006+0000 7f374c5ff700 1 Processor -- start 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.006+0000 7f374c5ff700 1 -- start start 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374c5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 0x7f3744152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374c5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744153cc0 0x7f3744154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f3744154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374c5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3744169c90 con 0x7f37440b0540 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374c5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3744169950 con 0x7f3744153cc0 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37441580f0 con 0x7f374415ade0 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374a39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 0x7f3744152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374a39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 0x7f3744152ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35202/0 (socket says 172.21.15.29:35202) 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374a39b700 1 -- 172.21.15.29:0/333039558 learned_addr learned my addr 172.21.15.29:0/333039558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f3749b9a700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744153cc0 0x7f3744154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.007+0000 7f374ab9c700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f3744154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374a39b700 1 -- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 msgr2=0x7f3744154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374a39b700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f3744154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374a39b700 1 -- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744153cc0 msgr2=0x7f3744154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374a39b700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744153cc0 0x7f3744154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374a39b700 1 -- 172.21.15.29:0/333039558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3734009420 con 0x7f37440b0540 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374a39b700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 0x7f3744152ee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f374000c7c0 tx=0x7f3740007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:45.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740001530 con 0x7f37440b0540 2023-10-07T09:29:45.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37400016b0 con 0x7f37440b0540 2023-10-07T09:29:45.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740001a70 con 0x7f37440b0540 2023-10-07T09:29:45.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37441583d0 con 0x7f37440b0540 2023-10-07T09:29:45.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.008+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37440cea40 con 0x7f37440b0540 2023-10-07T09:29:45.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.009+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374404c3c0 con 0x7f37440b0540 2023-10-07T09:29:45.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.010+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f374001b3b0 con 0x7f37440b0540 2023-10-07T09:29:45.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.011+0000 7f373a7fc700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3728066f30 0x7f37280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.011+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3740083db0 con 0x7f37440b0540 2023-10-07T09:29:45.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.012+0000 7f3749b9a700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3728066f30 0x7f37280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.012+0000 7f3749b9a700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3728066f30 0x7f37280693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f37340030c0 tx=0x7f3734016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:45.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.015+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3740088070 con 0x7f37440b0540 2023-10-07T09:29:45.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.220+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f374415a520 con 0x7f37440b0540 2023-10-07T09:29:45.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.221+0000 7f373a7fc700 1 -- 172.21.15.29:0/333039558 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f3740017020 con 0x7f37440b0540 2023-10-07T09:29:45.223 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:45.223 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.223+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3728066f30 msgr2=0x7f37280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.223+0000 7f374c5ff700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3728066f30 0x7f37280693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f37340030c0 tx=0x7f3734016000 comp rx=0 tx=0).stop 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.223+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 msgr2=0x7f3744152ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.223+0000 7f374c5ff700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 0x7f3744152ee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f374000c7c0 tx=0x7f3740007e00 comp rx=0 tx=0).stop 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 shutdown_connections 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3728066f30 0x7f37280693e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f374415ade0 0x7f3744154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440b0540 0x7f3744152ee0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 --2- 172.21.15.29:0/333039558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744153cc0 0x7f3744154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 >> 172.21.15.29:0/333039558 conn(0x7f3744067610 msgr2=0x7f37440ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 shutdown_connections 2023-10-07T09:29:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.224+0000 7f374c5ff700 1 -- 172.21.15.29:0/333039558 wait complete. 2023-10-07T09:29:45.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:29:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.438+0000 7fb3af7ff700 1 Processor -- start 2023-10-07T09:29:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.438+0000 7fb3af7ff700 1 -- start start 2023-10-07T09:29:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.439+0000 7fb3af7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb3a8159750 0x7fb3a80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.439+0000 7fb3af7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3a8159b20 0x7fb3a80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3af7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a80cbf20 con 0x7fb3a80ce6d0 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3af7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a80cc200 con 0x7fb3a8159b20 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3af7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a80cc4e0 con 0x7fb3a8159750 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3a8159b20 0x7fb3a80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3a8159b20 0x7fb3a80ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35218/0 (socket says 172.21.15.29:35218) 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 -- 172.21.15.29:0/420335723 learned_addr learned my addr 172.21.15.29:0/420335723 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3ad59b700 1 --2- 172.21.15.29:0/420335723 >> v2:172.21.15.144:3300/0 conn(0x7fb3a8159750 0x7fb3a80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 -- 172.21.15.29:0/420335723 >> v2:172.21.15.144:3300/0 conn(0x7fb3a8159750 msgr2=0x7fb3a80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 --2- 172.21.15.29:0/420335723 >> v2:172.21.15.144:3300/0 conn(0x7fb3a8159750 0x7fb3a80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 -- 172.21.15.29:0/420335723 >> v1:172.21.15.144:6789/0 conn(0x7fb3a80ce6d0 legacy=0x7fb3a815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 -- 172.21.15.29:0/420335723 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a80cc7c0 con 0x7fb3a8159b20 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.440+0000 7fb3acd9a700 1 --2- 172.21.15.29:0/420335723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159b20 0x7fb3a80ce190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8089840 tx=0x7fb398009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741527446137269e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb39ffff700 1 -- 172.21.15.29:0/420335723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398009e70 con 0x7fb3a8159b20 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb39ffff700 1 -- 172.21.15.29:0/420335723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39800b860 con 0x7fb3a8159b20 2023-10-07T09:29:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb39ffff700 1 -- 172.21.15.29:0/420335723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800bc00 con 0x7fb3a8159b20 2023-10-07T09:29:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb3af7ff700 1 -- 172.21.15.29:0/420335723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159b20 msgr2=0x7fb3a80ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/420335723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159b20 0x7fb3a80ce190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8089840 tx=0x7fb398009250 comp rx=0 tx=0).stop 2023-10-07T09:29:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb3af7ff700 1 -- 172.21.15.29:0/420335723 shutdown_connections 2023-10-07T09:29:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/420335723 >> v2:172.21.15.144:3300/0 conn(0x7fb3a8159750 0x7fb3a80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/420335723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159b20 0x7fb3a80ce190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.441+0000 7fb3af7ff700 1 -- 172.21.15.29:0/420335723 >> 172.21.15.29:0/420335723 conn(0x7fb3a8063a10 msgr2=0x7fb3a80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.442+0000 7fb3af7ff700 1 -- 172.21.15.29:0/420335723 shutdown_connections 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.442+0000 7fb3af7ff700 1 -- 172.21.15.29:0/420335723 wait complete. 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.443+0000 7fb3af7ff700 1 Processor -- start 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.443+0000 7fb3af7ff700 1 -- start start 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3af7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3a80ce6d0 0x7fb3a8144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3af7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 0x7fb3a80b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3af7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3a8159b20 0x7fb3a80b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3af7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8153750 con 0x7fb3a8159750 2023-10-07T09:29:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3af7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80b4ac0 con 0x7fb3a80ce6d0 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3af7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80b3f40 con 0x7fb3a8159b20 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3acd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 0x7fb3a80b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3acd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 0x7fb3a80b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35220/0 (socket says 172.21.15.29:35220) 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.444+0000 7fb3acd9a700 1 -- 172.21.15.29:0/4072244466 learned_addr learned my addr 172.21.15.29:0/4072244466 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3acd9a700 1 -- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3a8159b20 msgr2=0x7fb3a80b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3acd9a700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3a8159b20 0x7fb3a80b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3acd9a700 1 -- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3a80ce6d0 msgr2=0x7fb3a8144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3acd9a700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3a80ce6d0 0x7fb3a8144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3acd9a700 1 -- 172.21.15.29:0/4072244466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980094f0 con 0x7fb3a8159750 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3acd9a700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 0x7fb3a80b3360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8088a40 tx=0x7fb398017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398011420 con 0x7fb3a8159750 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3980115a0 con 0x7fb3a8159750 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398011960 con 0x7fb3a8159750 2023-10-07T09:29:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a80b4220 con 0x7fb3a8159750 2023-10-07T09:29:45.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.445+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a80b47e0 con 0x7fb3a8159750 2023-10-07T09:29:45.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.446+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a8159ef0 con 0x7fb3a8159750 2023-10-07T09:29:45.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.447+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb3980253f0 con 0x7fb3a8159750 2023-10-07T09:29:45.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.449+0000 7fb39d7fa700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb38c066f30 0x7fb38c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:45.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.449+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb398085f20 con 0x7fb3a8159750 2023-10-07T09:29:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.449+0000 7fb3ad59b700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb38c066f30 0x7fb38c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.450+0000 7fb3ad59b700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb38c066f30 0x7fb38c0693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009ba0 tx=0x7fb3a4009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.453+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb3980530f0 con 0x7fb3a8159750 2023-10-07T09:29:45.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.698+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3a804c3c0 con 0x7fb3a8159750 2023-10-07T09:29:45.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.698+0000 7fb39d7fa700 1 -- 172.21.15.29:0/4072244466 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7fb398056710 con 0x7fb3a8159750 2023-10-07T09:29:45.699 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:45.699 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:29:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.700+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb38c066f30 msgr2=0x7fb38c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.700+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb38c066f30 0x7fb38c0693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009ba0 tx=0x7fb3a4009120 comp rx=0 tx=0).stop 2023-10-07T09:29:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 msgr2=0x7fb3a80b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 0x7fb3a80b3360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8088a40 tx=0x7fb398017000 comp rx=0 tx=0).stop 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 shutdown_connections 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb38c066f30 0x7fb38c0693e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3a8159b20 0x7fb3a80b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3a80ce6d0 0x7fb3a8144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 --2- 172.21.15.29:0/4072244466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a8159750 0x7fb3a80b3360 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 >> 172.21.15.29:0/4072244466 conn(0x7fb3a8063a10 msgr2=0x7fb3a8154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 shutdown_connections 2023-10-07T09:29:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:45.701+0000 7fb3af7ff700 1 -- 172.21.15.29:0/4072244466 wait complete. 2023-10-07T09:29:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:29:46.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:29:21.598754+0000', 'modified': '2023-10-07T09:29:25.105471+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4284}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4284': {'gid': 4284, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6834/3263907411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 3263907411}, {'type': 'v1', 'addr': '172.21.15.29:6834', 'nonce': 3263907411}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:29:46.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:29:46.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-07T09:29:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.320+0000 7f875f3ff700 1 Processor -- start 2023-10-07T09:29:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.320+0000 7f875f3ff700 1 -- start start 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.321+0000 7f875f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8758159b20 0x7f87580d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.321+0000 7f875f3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87580cbf20 con 0x7f8758159750 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.321+0000 7f875f3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87580cc200 con 0x7f87580ce8a0 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.321+0000 7f875f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87580cc4e0 con 0x7f8758159b20 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f875d19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8758159750 0x7f87580ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34310/0 (socket says 172.21.15.29:34310) 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f875d19b700 1 -- 172.21.15.29:0/3366679705 learned_addr learned my addr 172.21.15.29:0/3366679705 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f875c99a700 1 --2- 172.21.15.29:0/3366679705 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8758159b20 0x7f87580d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699963341 0 0) 0x7f87580cbf20 con 0x7f8758159750 2023-10-07T09:29:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8744003570 con 0x7f8758159750 2023-10-07T09:29:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261316705 0 0) 0x7f87580cc200 con 0x7f87580ce8a0 2023-10-07T09:29:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87580cbf20 con 0x7f87580ce8a0 2023-10-07T09:29:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3229663623 0 0) 0x7f8744003570 con 0x7f8758159750 2023-10-07T09:29:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f875c99a700 1 -- 172.21.15.29:0/3366679705 >> v1:172.21.15.144:6789/0 conn(0x7f87580ce8a0 legacy=0x7f87580c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f875c99a700 1 -- 172.21.15.29:0/3366679705 >> v1:172.21.15.29:6789/0 conn(0x7f8758159750 legacy=0x7f87580ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.322+0000 7f875c99a700 1 -- 172.21.15.29:0/3366679705 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87580cc7c0 con 0x7f8758159b20 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.323+0000 7f875c99a700 1 --2- 172.21.15.29:0/3366679705 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8758159b20 0x7f87580d10a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8748009910 tx=0x7f87480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514e30e2839f62f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.323+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540015f0 con 0x7f8758159b20 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.323+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8748011420 con 0x7f8758159b20 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.323+0000 7f874ffff700 1 -- 172.21.15.29:0/3366679705 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87480117c0 con 0x7f8758159b20 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 -- 172.21.15.29:0/3366679705 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8758159b20 msgr2=0x7f87580d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 --2- 172.21.15.29:0/3366679705 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8758159b20 0x7f87580d10a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8748009910 tx=0x7f87480091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 -- 172.21.15.29:0/3366679705 shutdown_connections 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 --2- 172.21.15.29:0/3366679705 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8758159b20 0x7f87580d10a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 -- 172.21.15.29:0/3366679705 >> 172.21.15.29:0/3366679705 conn(0x7f8758063a10 msgr2=0x7f87580d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 -- 172.21.15.29:0/3366679705 shutdown_connections 2023-10-07T09:29:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.324+0000 7f875f3ff700 1 -- 172.21.15.29:0/3366679705 wait complete. 2023-10-07T09:29:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.325+0000 7f875f3ff700 1 Processor -- start 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.325+0000 7f875f3ff700 1 -- start start 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.326+0000 7f875f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87580ce8a0 0x7f8758151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.326+0000 7f875f3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 0x7f87581520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.326+0000 7f875f3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8758159b20 0x7f87580c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.326+0000 7f875f3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87581531d0 con 0x7f8758159750 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.326+0000 7f875f3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580b3950 con 0x7f8758159b20 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.326+0000 7f875f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875813fb70 con 0x7f87580ce8a0 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 0x7f87581520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 0x7f87581520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35236/0 (socket says 172.21.15.29:35236) 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 -- 172.21.15.29:0/4058582152 learned_addr learned my addr 172.21.15.29:0/4058582152 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875d19b700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87580ce8a0 0x7f8758151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875d99c700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8758159b20 0x7f87580c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 -- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87580ce8a0 msgr2=0x7f8758151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87580ce8a0 0x7f8758151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 -- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8758159b20 msgr2=0x7f87580c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8758159b20 0x7f87580c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 -- 172.21.15.29:0/4058582152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748009420 con 0x7f8758159750 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f875c99a700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 0x7f87581520c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8748009860 tx=0x7f8748016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.327+0000 7f874d7fa700 1 -- 172.21.15.29:0/4058582152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874801cbe0 con 0x7f8758159750 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.328+0000 7f874d7fa700 1 -- 172.21.15.29:0/4058582152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874801cd40 con 0x7f8758159750 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.328+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8758152930 con 0x7f8758159750 2023-10-07T09:29:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.328+0000 7f874d7fa700 1 -- 172.21.15.29:0/4058582152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874801a6a0 con 0x7f8758159750 2023-10-07T09:29:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.328+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8758152ef0 con 0x7f8758159750 2023-10-07T09:29:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.330+0000 7f874d7fa700 1 -- 172.21.15.29:0/4058582152 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f874801aad0 con 0x7f8758159750 2023-10-07T09:29:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.330+0000 7f874d7fa700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f873c066f30 0x7f873c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.331+0000 7f874d7fa700 1 -- 172.21.15.29:0/4058582152 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f87480863e0 con 0x7f8758159750 2023-10-07T09:29:46.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.331+0000 7f875d19b700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f873c066f30 0x7f873c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.332+0000 7f875d19b700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f873c066f30 0x7f873c0693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f87540017e0 tx=0x7f8754001770 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.333+0000 7f876dafeb80 1 Processor -- start 2023-10-07T09:29:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.333+0000 7f876dafeb80 1 -- start start 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 0x562008fe1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fc7530 0x562008fea680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008feabc0 0x562008feb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f876dafeb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008eb4310 con 0x562008fe1a40 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008ef1130 con 0x562008feabc0 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562008eeeaf0 con 0x562008fc7530 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 0x562008fe1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 0x562008fe1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35246/0 (socket says 172.21.15.29:35246) 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 -- 172.21.15.29:0/478262970 learned_addr learned my addr 172.21.15.29:0/478262970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875c99a700 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fc7530 0x562008fea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d99c700 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008feabc0 0x562008feb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 -- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fc7530 msgr2=0x562008fea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fc7530 0x562008fea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 -- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008feabc0 msgr2=0x562008feb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008feabc0 0x562008feb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 -- 172.21.15.29:0/478262970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562008ddd0a0 con 0x562008fe1a40 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d99c700 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008feabc0 0x562008feb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.334+0000 7f875d19b700 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 0x562008fe1e10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f875400fd70 tx=0x7f8754015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c2fe0b865ea208 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f87427fc700 1 -- 172.21.15.29:0/478262970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754006a40 con 0x562008fe1a40 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f87427fc700 1 -- 172.21.15.29:0/478262970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8754014af0 con 0x562008fe1a40 2023-10-07T09:29:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f87427fc700 1 -- 172.21.15.29:0/478262970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754020410 con 0x562008fe1a40 2023-10-07T09:29:46.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 -- 172.21.15.29:0/478262970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 msgr2=0x562008fe1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 0x562008fe1e10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f875400fd70 tx=0x7f8754015000 comp rx=0 tx=0).stop 2023-10-07T09:29:46.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 -- 172.21.15.29:0/478262970 shutdown_connections 2023-10-07T09:29:46.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008feabc0 0x562008feb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fe1a40 0x562008fe1e10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 --2- 172.21.15.29:0/478262970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fc7530 0x562008fea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.335+0000 7f876dafeb80 1 -- 172.21.15.29:0/478262970 >> 172.21.15.29:0/478262970 conn(0x562008fe4e70 msgr2=0x562008fe5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.336+0000 7f876dafeb80 1 -- 172.21.15.29:0/478262970 shutdown_connections 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.336+0000 7f876dafeb80 1 -- 172.21.15.29:0/478262970 wait complete. 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.336+0000 7f876dafeb80 1 -- start start 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.337+0000 7f876dafeb80 1 Processor -- start 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.337+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fc7530 0x562008fe9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.337+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fe1a40 0x562008fc54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.337+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 0x562008fc5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.337+0000 7f876dafeb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008eea8e0 con 0x562008feabc0 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008e96980 con 0x562008fc7530 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562008e96620 con 0x562008fe1a40 2023-10-07T09:29:46.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f876dafeb80 10 client.0 _ll_register_callbacks cb 0x562008cf9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 0x562008fc5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 0x562008fc5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35260/0 (socket says 172.21.15.29:35260) 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 -- 172.21.15.29:0/2639568722 learned_addr learned my addr 172.21.15.29:0/2639568722 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d19b700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fc7530 0x562008fe9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875c99a700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fe1a40 0x562008fc54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 -- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fe1a40 msgr2=0x562008fc54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fe1a40 0x562008fc54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 -- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fc7530 msgr2=0x562008fe9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fc7530 0x562008fe9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 -- 172.21.15.29:0/2639568722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875400f9b0 con 0x562008feabc0 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d19b700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fc7530 0x562008fe9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:46.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f875d99c700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 0x562008fc5a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8750003780 tx=0x7f8750009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad82562d59c8f0d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.338+0000 7f87377fe700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f87377fe700 1 -- 172.21.15.29:0/2639568722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87500034e0 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f87377fe700 1 -- 172.21.15.29:0/2639568722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8750004100 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562008f8f560 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f87377fe700 1 -- 172.21.15.29:0/2639568722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8750013780 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 10 client.4701 fetch_fsmap learned FSMap version 8 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562008eb93c0 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f87377fe700 1 -- 172.21.15.29:0/2639568722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562008f8f560 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f87377fe700 1 -- 172.21.15.29:0/2639568722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f87500138e0 con 0x562008feabc0 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 10 client.4701 fetch_fsmap finished waiting for FSMap version 8 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 10 client.4701 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:29:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 20 client.4701 populate_metadata read hostname 'smithi029' 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008fc0fe0 0x562008fc13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 4 client.4701 mds_command: new command op to 4284 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.339+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 --> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562008f6c0e0 con 0x562008fc0fe0 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.340+0000 7f875d19b700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008fc0fe0 0x562008fc13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.340+0000 7f875d19b700 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008fc0fe0 0x562008fc13b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=801599fb940c8420 server_cookie=7aa4e943d70affb4 in_seq=0 out_seq=0 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.340+0000 7f87377fe700 10 client.4701 ms_handle_connect on v2:172.21.15.29:6832/3263907411 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f87377fe700 1 -- 172.21.15.29:0/2639568722 <== mds.0 v2:172.21.15.29:6832/3263907411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x562008f6c0e0 con 0x562008fc0fe0 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f87377fe700 10 client.4701 handle_command_reply: tid=0 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 client.4701 shutdown 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 msgr2=0x562008fc5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 0x562008fc5a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8750003780 tx=0x7f8750009040 comp rx=0 tx=0).stop 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 shutdown_connections 2023-10-07T09:29:46.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008feabc0 0x562008fc5a30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fc7530 0x562008fe9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008fc0fe0 0x562008fc13b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 --2- 172.21.15.29:0/2639568722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008fe1a40 0x562008fc54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 >> 172.21.15.29:0/2639568722 conn(0x562008fe4e70 msgr2=0x562008f97250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f87377fe700 0 client.4701 ms_handle_reset on v2:172.21.15.29:6832/3263907411 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 shutdown_connections 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 1 -- 172.21.15.29:0/2639568722 wait complete. 2023-10-07T09:29:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.341+0000 7f876dafeb80 20 client.4701 trim_cache size 0 max 16384 2023-10-07T09:29:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.359+0000 7f876dafeb80 1 Processor -- start 2023-10-07T09:29:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.359+0000 7f876dafeb80 1 -- start start 2023-10-07T09:29:46.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562008f9c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562008fc8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f876dafeb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008d821e0 con 0x562008fc7530 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008fc4ad0 con 0x562008fe1a40 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562008fc1810 con 0x562008feabc0 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f875d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562008f9c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f875d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562008f9c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35262/0 (socket says 172.21.15.29:35262) 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.360+0000 7f875d19b700 1 -- 172.21.15.29:0/759145866 learned_addr learned my addr 172.21.15.29:0/759145866 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875c99a700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562008fc8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d19b700 1 -- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 msgr2=0x562008fc1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d99c700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d19b700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d19b700 1 -- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 msgr2=0x562008fc8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d19b700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562008fc8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d19b700 1 -- 172.21.15.29:0/759145866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562008d511d0 con 0x562008fc7530 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875c99a700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562008fc8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d99c700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f875d19b700 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562008f9c440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8754020000 tx=0x7f8754033250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e7761c6e5a1c8e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f8742ffd700 1 -- 172.21.15.29:0/759145866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754032070 con 0x562008fc7530 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f8742ffd700 1 -- 172.21.15.29:0/759145866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8754001810 con 0x562008fc7530 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.361+0000 7f8742ffd700 1 -- 172.21.15.29:0/759145866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754006a40 con 0x562008fc7530 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 -- 172.21.15.29:0/759145866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 msgr2=0x562008f9c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562008f9c440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8754020000 tx=0x7f8754033250 comp rx=0 tx=0).stop 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 -- 172.21.15.29:0/759145866 shutdown_connections 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562008f9c440 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 --2- 172.21.15.29:0/759145866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562008fc8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 -- 172.21.15.29:0/759145866 >> 172.21.15.29:0/759145866 conn(0x562008fe4e70 msgr2=0x562008f98800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 -- 172.21.15.29:0/759145866 shutdown_connections 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.362+0000 7f876dafeb80 1 -- 172.21.15.29:0/759145866 wait complete. 2023-10-07T09:29:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.363+0000 7f876dafeb80 1 -- start start 2023-10-07T09:29:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.363+0000 7f876dafeb80 1 Processor -- start 2023-10-07T09:29:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562009049b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562009047550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562009047fc0 con 0x562008fc7530 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562008fc1e70 con 0x562008fe1a40 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562008f91b80 con 0x562008feabc0 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f876dafeb80 10 client.0 _ll_register_callbacks cb 0x562008cf9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562009049b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562009049b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35264/0 (socket says 172.21.15.29:35264) 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 -- 172.21.15.29:0/200753240 learned_addr learned my addr 172.21.15.29:0/200753240 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 -- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 msgr2=0x562008fc8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875c99a700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562009047550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 -- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 msgr2=0x562009047550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562009047550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875d19b700 1 -- 172.21.15.29:0/200753240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562008ddd0a0 con 0x562008fc7530 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.364+0000 7f875c99a700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562009047550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f875d19b700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562009049b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x562008fc74b0 tx=0x7f8754032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0b32e7a04de8f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f87377fe700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f87377fe700 1 -- 172.21.15.29:0/200753240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875403b070 con 0x562008fc7530 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f87377fe700 1 -- 172.21.15.29:0/200753240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f875401c040 con 0x562008fc7530 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8754014af0 con 0x562008fc7530 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f87377fe700 1 -- 172.21.15.29:0/200753240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87540209d0 con 0x562008fc7530 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 10 client.4707 fetch_fsmap learned FSMap version 8 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562008d2ad60 con 0x562008fc7530 2023-10-07T09:29:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f87377fe700 1 -- 172.21.15.29:0/200753240 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8754014af0 con 0x562008fc7530 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f87377fe700 1 -- 172.21.15.29:0/200753240 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8754012070 con 0x562008fc7530 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 10 client.4707 fetch_fsmap finished waiting for FSMap version 8 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 10 client.4707 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 20 client.4707 populate_metadata read hostname 'smithi029' 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008f9a6b0 0x562008f9aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 4 client.4707 mds_command: new command op to 4284 tid=0[{"prefix": "status"}] 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.365+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 --> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562008e8d8d0 con 0x562008f9a6b0 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.366+0000 7f875c99a700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008f9a6b0 0x562008f9aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.366+0000 7f875c99a700 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008f9a6b0 0x562008f9aa80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38bc6eccd32babc1 server_cookie=1096471ab1d36fd3 in_seq=0 out_seq=0 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.366+0000 7f87377fe700 10 client.4707 ms_handle_connect on v2:172.21.15.29:6832/3263907411 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.366+0000 7f87377fe700 1 -- 172.21.15.29:0/200753240 <== mds.0 v2:172.21.15.29:6832/3263907411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x562008e8d8d0 con 0x562008f9a6b0 2023-10-07T09:29:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.366+0000 7f87377fe700 10 client.4707 handle_command_reply: tid=0 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.366+0000 7f876dafeb80 1 client.4707 shutdown 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 msgr2=0x562009049b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562009049b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x562008fc74b0 tx=0x7f8754032000 comp rx=0 tx=0).stop 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 shutdown_connections 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:6832/3263907411,v1:172.21.15.29:6834/3263907411] conn(0x562008f9a6b0 0x562008f9aa80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562008feabc0 0x562008fc8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562008fc7530 0x562009049b20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f87377fe700 0 client.4707 ms_handle_reset on v2:172.21.15.29:6832/3263907411 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 --2- 172.21.15.29:0/200753240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562008fe1a40 0x562009047550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 >> 172.21.15.29:0/200753240 conn(0x562008fe4e70 msgr2=0x562008fc7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 shutdown_connections 2023-10-07T09:29:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 1 -- 172.21.15.29:0/200753240 wait complete. 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f876dafeb80 20 client.4707 trim_cache size 0 max 16384 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4284, 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:29:46.375 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 24.280535919999998, 2023-10-07T09:29:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 8, 2023-10-07T09:29:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 22, 2023-10-07T09:29:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 22, 2023-10-07T09:29:46.376 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 29.732840152000001 2023-10-07T09:29:46.376 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f873c066f30 msgr2=0x7f873c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f875f3ff700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f873c066f30 0x7f873c0693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f87540017e0 tx=0x7f8754001770 comp rx=0 tx=0).stop 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 msgr2=0x7f87581520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.367+0000 7f875f3ff700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 0x7f87581520c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8748009860 tx=0x7f8748016000 comp rx=0 tx=0).stop 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 shutdown_connections 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f873c066f30 0x7f873c0693e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8758159b20 0x7f87580c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87580ce8a0 0x7f8758151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 --2- 172.21.15.29:0/4058582152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8758159750 0x7f87581520c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 >> 172.21.15.29:0/4058582152 conn(0x7f8758063a10 msgr2=0x7f8758154390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:46.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 shutdown_connections 2023-10-07T09:29:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:46.368+0000 7f875f3ff700 1 -- 172.21.15.29:0/4058582152 wait complete. 2023-10-07T09:29:46.385 INFO:teuthology.run_tasks:Running task kclient... 2023-10-07T09:29:46.409 INFO:tasks.kclient:Mounting kernel clients... 2023-10-07T09:29:46.409 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-07T09:29:46.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.409 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:29:46.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-07T09:29:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-07T09:29:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:29:46.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.574 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:29:46.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-07T09:29:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-07T09:29:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:29:46.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.736 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-07T09:29:46.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-07T09:29:46.864 INFO:teuthology.orchestra.run.smithi144.stderr:Cannot find device "ceph-brx" 2023-10-07T09:29:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:29:46.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.868 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-07T09:29:46.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:46.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-07T09:29:46.987 INFO:teuthology.orchestra.run.smithi144.stderr:Cannot find device "ceph-brx" 2023-10-07T09:29:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T09:29:46.990 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-07T09:29:46.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:29:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.185+0000 7f8b1ffff700 1 Processor -- start 2023-10-07T09:29:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.185+0000 7f8b1ffff700 1 -- start start 2023-10-07T09:29:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.186+0000 7f8b1ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 0x7f8b180d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.186+0000 7f8b1ffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b180cbf20 con 0x7f8b18159b20 2023-10-07T09:29:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.186+0000 7f8b1ffff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b180cc200 con 0x7f8b180ce8a0 2023-10-07T09:29:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.186+0000 7f8b1ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180cc4e0 con 0x7f8b18159750 2023-10-07T09:29:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1d59a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8b18159b20 0x7f8b180ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34318/0 (socket says 172.21.15.29:34318) 2023-10-07T09:29:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1d59a700 1 -- 172.21.15.29:0/3724305104 learned_addr learned my addr 172.21.15.29:0/3724305104 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 0x7f8b180d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353181517 0 0) 0x7f8b180cbf20 con 0x7f8b18159b20 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b04003570 con 0x7f8b18159b20 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602844745 0 0) 0x7f8b180cc200 con 0x7f8b180ce8a0 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b180cbf20 con 0x7f8b180ce8a0 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1502903976 0 0) 0x7f8b04003570 con 0x7f8b18159b20 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 msgr2=0x7f8b180d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 0x7f8b180d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 >> v1:172.21.15.144:6789/0 conn(0x7f8b180ce8a0 legacy=0x7f8b180c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b180cc7c0 con 0x7f8b18159b20 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f8b08001470 con 0x7f8b18159b20 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 >> v1:172.21.15.29:6789/0 conn(0x7f8b18159b20 legacy=0x7f8b180ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.187+0000 7f8b1cd99700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b04004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1cd99700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04004810 0x7f8b04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1cd99700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 0x7f8b04005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180cc4e0 con 0x7f8b04003c20 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400bbf0 con 0x7f8b04004810 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400bdb0 con 0x7f8b18159750 2023-10-07T09:29:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b04004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1d59a700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04004810 0x7f8b04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1e59c700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 0x7f8b04005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 msgr2=0x7f8b04005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b18159750 0x7f8b04005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04004810 msgr2=0x7f8b04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04004810 0x7f8b04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/3724305104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b04003570 con 0x7f8b04003c20 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.188+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b04004010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009bf0 tx=0x7f8b14009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa210fd738af377 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14005a00 con 0x7f8b04003c20 2023-10-07T09:29:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14005b60 con 0x7f8b04003c20 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1cd99700 1 -- 172.21.15.29:0/3724305104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400e710 con 0x7f8b04003c20 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1ffff700 1 -- 172.21.15.29:0/3724305104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 msgr2=0x7f8b04004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/3724305104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b04004010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009bf0 tx=0x7f8b14009120 comp rx=0 tx=0).stop 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1e59c700 1 -- 172.21.15.29:0/3724305104 reap_dead start 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1ffff700 1 -- 172.21.15.29:0/3724305104 shutdown_connections 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1ffff700 1 -- 172.21.15.29:0/3724305104 >> 172.21.15.29:0/3724305104 conn(0x7f8b18063a10 msgr2=0x7f8b180d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1ffff700 1 -- 172.21.15.29:0/3724305104 shutdown_connections 2023-10-07T09:29:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.189+0000 7f8b1ffff700 1 -- 172.21.15.29:0/3724305104 wait complete. 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.191+0000 7f8b1ffff700 1 Processor -- start 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.191+0000 7f8b1ffff700 1 -- start start 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b181450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b181545b0 0x7f8b18152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b180c83d0 0x7f8b18158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181455f0 con 0x7f8b04003c20 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18152fe0 con 0x7f8b180c83d0 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181532c0 con 0x7f8b181545b0 2023-10-07T09:29:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b181450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b181450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35286/0 (socket says 172.21.15.29:35286) 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/1035199270 learned_addr learned my addr 172.21.15.29:0/1035199270 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1d59a700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b181545b0 0x7f8b18152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1e59c700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b180c83d0 0x7f8b18158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b181545b0 msgr2=0x7f8b18152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.192+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b181545b0 0x7f8b18152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b180c83d0 msgr2=0x7f8b18158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b180c83d0 0x7f8b18158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b1dd9b700 1 -- 172.21.15.29:0/1035199270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b181535a0 con 0x7f8b04003c20 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b1dd9b700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b181450b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8b040057f0 tx=0x7f8b14019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14005a00 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14006240 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b140065e0 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b180b3400 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.193+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b18152bd0 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.194+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1804c3c0 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.195+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8b14013470 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.196+0000 7f8b0dffb700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8afc066f60 0x7f8afc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.196+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8b14085c50 con 0x7f8b04003c20 2023-10-07T09:29:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.196+0000 7f8b1d59a700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8afc066f60 0x7f8afc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:47.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.197+0000 7f8b1d59a700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8afc066f60 0x7f8afc069410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8b18154980 tx=0x7f8b08001d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.200+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b1408b020 con 0x7f8b04003c20 2023-10-07T09:29:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.452+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8b18064040 con 0x7f8b04003c20 2023-10-07T09:29:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.452+0000 7f8b0dffb700 1 -- 172.21.15.29:0/1035199270 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8b14052ec0 con 0x7f8b04003c20 2023-10-07T09:29:47.454 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:29:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8afc066f60 msgr2=0x7f8afc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8afc066f60 0x7f8afc069410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8b18154980 tx=0x7f8b08001d40 comp rx=0 tx=0).stop 2023-10-07T09:29:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 msgr2=0x7f8b181450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b181450b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8b040057f0 tx=0x7f8b14019000 comp rx=0 tx=0).stop 2023-10-07T09:29:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 shutdown_connections 2023-10-07T09:29:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8afc066f60 0x7f8afc069410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b180c83d0 0x7f8b18158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b04003c20 0x7f8b181450b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 --2- 172.21.15.29:0/1035199270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b181545b0 0x7f8b18152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 >> 172.21.15.29:0/1035199270 conn(0x7f8b18063a10 msgr2=0x7f8b180c60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 shutdown_connections 2023-10-07T09:29:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:47.455+0000 7f8b1ffff700 1 -- 172.21.15.29:0/1035199270 wait complete. 2023-10-07T09:29:47.473 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-07T09:29:47.473 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:29:47.474 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:29:47.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:47.474 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:29:47.520 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:47.520 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:47.520 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:47.520 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:47.520 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:29:47.521 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42241sec preferred_lft 42241sec 2023-10-07T09:29:47.522 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:29:47.522 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:47.522 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:47.522 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:47.522 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:29:47.523 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-07T09:29:47.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:47.523 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:47.523 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:47.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2023-10-07T09:29:47.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2023-10-07T09:29:47.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2023-10-07T09:29:47.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-07T09:29:47.524 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:47.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:29:47.570 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:29:48.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:48.184 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:29:48.303 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:29:48.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:48.306 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:29:48.382 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:29:48.383 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:29:48.383 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:29:48.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:29:48.383 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:29:48.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:48.384 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:48.384 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:48.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:29:48.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:29:48.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:29:48.384 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:49.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:49.410 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:29:49.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:49.458 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2023-10-07T09:29:49.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:49.545 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:49.545 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:49.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:29:49.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-07T09:29:49.546 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:49.730 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-07T09:29:49.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:49.730 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:49.730 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:49.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-07T09:29:49.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-07T09:29:49.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-07T09:29:49.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-07T09:29:49.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-07T09:29:49.731 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:50.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:50.010 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:50.011 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:50.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:29:50.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2023-10-07T09:29:50.011 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:50.186 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:50.186 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:29:50.233 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:29:50.234 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:50.234 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:29:50.280 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:29:50.280 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:29:50.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:29:50.420 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:29:50.420 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:29:50.420 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:29:51.216 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:29:51.217 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:29:51.217 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:29:51.217 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:51.217 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:51.242 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:29:51.242 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:29:51.242 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:29:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:29:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:29:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:29:51.250 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:29:51.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:51.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:29:51.323 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-10-07T09:29:51.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:29:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.516+0000 7fa7511ff700 1 Processor -- start 2023-10-07T09:29:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.517+0000 7fa7511ff700 1 -- start start 2023-10-07T09:29:51.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa7511ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa74c0ad370 0x7fa74c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:51.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa7511ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa74c0b0410 0x7fa74c15a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:51.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa7511ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74c14d790 con 0x7fa74c159ec0 2023-10-07T09:29:51.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa7511ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c15a880 con 0x7fa74c0ad370 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa7511ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c0ae5d0 con 0x7fa74c0b0410 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa74ad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa74c0ad370 0x7fa74c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa74ad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa74c0ad370 0x7fa74c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35294/0 (socket says 172.21.15.29:35294) 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.518+0000 7fa74ad9d700 1 -- 172.21.15.29:0/1894352686 learned_addr learned my addr 172.21.15.29:0/1894352686 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa74b59e700 1 --2- 172.21.15.29:0/1894352686 >> v2:172.21.15.144:3300/0 conn(0x7fa74c0b0410 0x7fa74c15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa74ad9d700 1 -- 172.21.15.29:0/1894352686 >> v2:172.21.15.144:3300/0 conn(0x7fa74c0b0410 msgr2=0x7fa74c15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa74ad9d700 1 --2- 172.21.15.29:0/1894352686 >> v2:172.21.15.144:3300/0 conn(0x7fa74c0b0410 0x7fa74c15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa74ad9d700 1 -- 172.21.15.29:0/1894352686 >> v1:172.21.15.144:6789/0 conn(0x7fa74c159ec0 legacy=0x7fa74c1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa74ad9d700 1 -- 172.21.15.29:0/1894352686 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0ad740 con 0x7fa74c0ad370 2023-10-07T09:29:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa74ad9d700 1 --2- 172.21.15.29:0/1894352686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0ad370 0x7fa74c0cc870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0099b0 tx=0x7fa73c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65185c84b08bb9a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.519+0000 7fa749d9b700 1 -- 172.21.15.29:0/1894352686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c009ea0 con 0x7fa74c0ad370 2023-10-07T09:29:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa749d9b700 1 -- 172.21.15.29:0/1894352686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c00b820 con 0x7fa74c0ad370 2023-10-07T09:29:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa749d9b700 1 -- 172.21.15.29:0/1894352686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c00bbc0 con 0x7fa74c0ad370 2023-10-07T09:29:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 -- 172.21.15.29:0/1894352686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0ad370 msgr2=0x7fa74c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 --2- 172.21.15.29:0/1894352686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0ad370 0x7fa74c0cc870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0099b0 tx=0x7fa73c009250 comp rx=0 tx=0).stop 2023-10-07T09:29:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 -- 172.21.15.29:0/1894352686 shutdown_connections 2023-10-07T09:29:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 --2- 172.21.15.29:0/1894352686 >> v2:172.21.15.144:3300/0 conn(0x7fa74c0b0410 0x7fa74c15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 --2- 172.21.15.29:0/1894352686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0ad370 0x7fa74c0cc870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 -- 172.21.15.29:0/1894352686 >> 172.21.15.29:0/1894352686 conn(0x7fa74c0680a0 msgr2=0x7fa74c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 -- 172.21.15.29:0/1894352686 shutdown_connections 2023-10-07T09:29:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.520+0000 7fa7511ff700 1 -- 172.21.15.29:0/1894352686 wait complete. 2023-10-07T09:29:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.522+0000 7fa7511ff700 1 Processor -- start 2023-10-07T09:29:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.522+0000 7fa7511ff700 1 -- start start 2023-10-07T09:29:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa7511ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa74c0ad370 0x7fa74c1484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa7511ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 0x7fa74c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa7511ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c159ec0 0x7fa74c1478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa7511ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1489e0 con 0x7fa74c0b0410 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa7511ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c148c10 con 0x7fa74c159ec0 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa7511ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c148ef0 con 0x7fa74c0ad370 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa74a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 0x7fa74c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa74a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 0x7fa74c06d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35298/0 (socket says 172.21.15.29:35298) 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa74a59c700 1 -- 172.21.15.29:0/262905004 learned_addr learned my addr 172.21.15.29:0/262905004 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa74b59e700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c159ec0 0x7fa74c1478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.523+0000 7fa74ad9d700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa74c0ad370 0x7fa74c1484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa74a59c700 1 -- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa74c0ad370 msgr2=0x7fa74c1484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa74a59c700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa74c0ad370 0x7fa74c1484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa74a59c700 1 -- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c159ec0 msgr2=0x7fa74c1478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa74a59c700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c159ec0 0x7fa74c1478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa74a59c700 1 -- 172.21.15.29:0/262905004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c0094f0 con 0x7fa74c0b0410 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa74a59c700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 0x7fa74c06d6f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa74000c4e0 tx=0x7fa74000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740011590 con 0x7fa74c0b0410 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa740011710 con 0x7fa74c0b0410 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740011ad0 con 0x7fa74c0b0410 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c146e10 con 0x7fa74c0b0410 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.524+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c1473d0 con 0x7fa74c0b0410 2023-10-07T09:29:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.525+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c1475f0 con 0x7fa74c0b0410 2023-10-07T09:29:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.526+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa74001b3b0 con 0x7fa74c0b0410 2023-10-07T09:29:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.527+0000 7fa732ffd700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa738066f30 0x7fa7380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.528+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa740083990 con 0x7fa74c0b0410 2023-10-07T09:29:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.528+0000 7fa74ad9d700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa738066f30 0x7fa7380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.529+0000 7fa74ad9d700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa738066f30 0x7fa7380693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0099b0 tx=0x7fa73c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.531+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa740050b60 con 0x7fa74c0b0410 2023-10-07T09:29:51.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.782+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa74c04c3c0 con 0x7fa74c0b0410 2023-10-07T09:29:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.783+0000 7fa732ffd700 1 -- 172.21.15.29:0/262905004 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa740017070 con 0x7fa74c0b0410 2023-10-07T09:29:51.784 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:29:51.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa738066f30 msgr2=0x7fa7380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa738066f30 0x7fa7380693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0099b0 tx=0x7fa73c017000 comp rx=0 tx=0).stop 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 msgr2=0x7fa74c06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 0x7fa74c06d6f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa74000c4e0 tx=0x7fa74000c7f0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 shutdown_connections 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa738066f30 0x7fa7380693e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c159ec0 0x7fa74c1478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa74c0ad370 0x7fa74c1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 --2- 172.21.15.29:0/262905004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa74c0b0410 0x7fa74c06d6f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 >> 172.21.15.29:0/262905004 conn(0x7fa74c0680a0 msgr2=0x7fa74c0cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 shutdown_connections 2023-10-07T09:29:51.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:51.785+0000 7fa7511ff700 1 -- 172.21.15.29:0/262905004 wait complete. 2023-10-07T09:29:51.803 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-07T09:29:51.803 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:29:51.803 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:29:51.803 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-07T09:29:51.803 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-07T09:29:51.804 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-07T09:29:51.804 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-07T09:29:51.804 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:29:51.804 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-07T09:29:51.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:51.804 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:29:51.851 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:29:51.852 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42237sec preferred_lft 42237sec 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:29:51.853 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:29:51.854 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:51.855 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:29:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:29:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:29:51.855 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:51.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:51.855 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:29:51.924 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:29:51.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:51.927 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:29:51.976 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:29:51.977 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:29:51.977 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:29:51.977 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:29:51.977 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:29:51.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:51.978 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:51.978 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:51.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:29:51.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:29:51.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:29:51.979 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:52.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:52.208 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:29:52.255 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:29:52.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:52.256 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2023-10-07T09:29:52.342 INFO:teuthology.orchestra.run.smithi029.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-07T09:29:52.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:52.343 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:52.343 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:52.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:29:52.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-07T09:29:52.344 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:52.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:52.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-07T09:29:52.599 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.632 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 1e:72:31:41:dd:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::1c72:31ff:fe41:dd3f/64 scope link 2023-10-07T09:29:52.633 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:52.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-07T09:29:52.706 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:52.748 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 1e:72:31:41:dd:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::1c72:31ff:fe41:dd3f/64 scope link 2023-10-07T09:29:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:52.751 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-07T09:29:52.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:52.751 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:52.751 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:52.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-07T09:29:52.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-07T09:29:52.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-07T09:29:52.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-07T09:29:52.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-07T09:29:52.752 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:53.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:53.011 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:29:53.011 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:29:53.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.1 up 2023-10-07T09:29:53.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.1 master ceph-brx 2023-10-07T09:29:53.011 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:29:53.187 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:53.188 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-07T09:29:53.234 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-07T09:29:53.234 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:53.235 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-07T09:29:53.320 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:29:53.320 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:29:53.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-07T09:29:53.461 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-07T09:29:53.462 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-07T09:29:53.462 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:29:53.476 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-07T09:29:53.476 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-07T09:29:53.476 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:29:53.476 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:53.476 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-07T09:29:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-07T09:29:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:29:53.498 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:29:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:29:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:29:53.503 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:29:53.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:53.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-07T09:29:53.573 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-10-07T09:29:53.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:29:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.767+0000 7f47269ff700 1 Processor -- start 2023-10-07T09:29:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.767+0000 7f47269ff700 1 -- start start 2023-10-07T09:29:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f47269ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4720157290 0x7f47200cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f47269ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47200ae8b0 0x7f47200b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f47269ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4720158ad0 con 0x7f47200b1840 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f47269ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47200b1cc0 con 0x7f47200ae8b0 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f47269ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472015b140 con 0x7f4720157290 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f471f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47200ae8b0 0x7f47200b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f471f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47200ae8b0 0x7f47200b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48702/0 (socket says 172.21.15.29:48702) 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f471f7fe700 1 -- 172.21.15.29:0/2724491352 learned_addr learned my addr 172.21.15.29:0/2724491352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.768+0000 7f4724f9c700 1 --1- 172.21.15.29:0/2724491352 >> v1:172.21.15.29:6789/0 conn(0x7f47200b1840 0x7f47200ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50838/0 (socket says 172.21.15.29:50838) 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471ffff700 1 --2- 172.21.15.29:0/2724491352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4720157290 0x7f47200cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471f7fe700 1 -- 172.21.15.29:0/2724491352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4720157290 msgr2=0x7f47200cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471f7fe700 1 --2- 172.21.15.29:0/2724491352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4720157290 0x7f47200cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471f7fe700 1 -- 172.21.15.29:0/2724491352 >> v1:172.21.15.29:6789/0 conn(0x7f47200b1840 legacy=0x7f47200ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471f7fe700 1 -- 172.21.15.29:0/2724491352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472015b420 con 0x7f47200ae8b0 2023-10-07T09:29:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471f7fe700 1 --2- 172.21.15.29:0/2724491352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200ae8b0 0x7f47200b1300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f47140099b0 tx=0x7f4714009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89c1b6fdf2d9ca4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471effd700 1 -- 172.21.15.29:0/2724491352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714009ea0 con 0x7f47200ae8b0 2023-10-07T09:29:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.769+0000 7f471effd700 1 -- 172.21.15.29:0/2724491352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471400b860 con 0x7f47200ae8b0 2023-10-07T09:29:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f471effd700 1 -- 172.21.15.29:0/2724491352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471400bc00 con 0x7f47200ae8b0 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 -- 172.21.15.29:0/2724491352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200ae8b0 msgr2=0x7f47200b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 --2- 172.21.15.29:0/2724491352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200ae8b0 0x7f47200b1300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f47140099b0 tx=0x7f4714009250 comp rx=0 tx=0).stop 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 -- 172.21.15.29:0/2724491352 shutdown_connections 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 --2- 172.21.15.29:0/2724491352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4720157290 0x7f47200cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 --2- 172.21.15.29:0/2724491352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200ae8b0 0x7f47200b1300 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 -- 172.21.15.29:0/2724491352 >> 172.21.15.29:0/2724491352 conn(0x7f47200680a0 msgr2=0x7f47200cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 -- 172.21.15.29:0/2724491352 shutdown_connections 2023-10-07T09:29:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.770+0000 7f47269ff700 1 -- 172.21.15.29:0/2724491352 wait complete. 2023-10-07T09:29:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.772+0000 7f47269ff700 1 Processor -- start 2023-10-07T09:29:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.772+0000 7f47269ff700 1 -- start start 2023-10-07T09:29:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f47269ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 0x7f4720152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f47269ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4720157290 0x7f47200c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f47269ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47200c6750 0x7f4720155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f47269ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47200c7500 con 0x7f47200b1840 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f47269ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4720153330 con 0x7f4720157290 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f47269ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472015bf00 con 0x7f47200c6750 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 0x7f4720152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 0x7f4720152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48712/0 (socket says 172.21.15.29:48712) 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 -- 172.21.15.29:0/2639290142 learned_addr learned my addr 172.21.15.29:0/2639290142 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f4724f9c700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47200c6750 0x7f4720155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471f7fe700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4720157290 0x7f47200c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 -- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47200c6750 msgr2=0x7f4720155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47200c6750 0x7f4720155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 -- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4720157290 msgr2=0x7f47200c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4720157290 0x7f47200c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471ffff700 1 -- 172.21.15.29:0/2639290142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47140094f0 con 0x7f47200b1840 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f4724f9c700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47200c6750 0x7f4720155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.773+0000 7f471f7fe700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4720157290 0x7f47200c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.774+0000 7f471ffff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 0x7f4720152df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f471000e960 tx=0x7f47100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.774+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000c850 con 0x7f47200b1840 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.774+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471000c9b0 con 0x7f47200b1840 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.774+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000cd50 con 0x7f47200b1840 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.774+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47200c7750 con 0x7f47200b1840 2023-10-07T09:29:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.774+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47200c7d10 con 0x7f47200b1840 2023-10-07T09:29:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.775+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47201548f0 con 0x7f47200b1840 2023-10-07T09:29:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.776+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4710013680 con 0x7f47200b1840 2023-10-07T09:29:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.777+0000 7f4707fff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f470c066f30 0x7f470c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.777+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4710084800 con 0x7f47200b1840 2023-10-07T09:29:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.778+0000 7f471f7fe700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f470c066f30 0x7f470c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.778+0000 7f471f7fe700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f470c066f30 0x7f470c0693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f47140099b0 tx=0x7f4714017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:53.781+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47100519d0 con 0x7f47200b1840 2023-10-07T09:29:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.035+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f472004c3c0 con 0x7f47200b1840 2023-10-07T09:29:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.036+0000 7f4707fff700 1 -- 172.21.15.29:0/2639290142 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4710054ff0 con 0x7f47200b1840 2023-10-07T09:29:54.037 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:29:54.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f470c066f30 msgr2=0x7f470c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:54.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f470c066f30 0x7f470c0693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f47140099b0 tx=0x7f4714017000 comp rx=0 tx=0).stop 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 msgr2=0x7f4720152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 0x7f4720152df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f471000e960 tx=0x7f47100091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 shutdown_connections 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f470c066f30 0x7f470c0693e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47200c6750 0x7f4720155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47200b1840 0x7f4720152df0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 --2- 172.21.15.29:0/2639290142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4720157290 0x7f47200c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 >> 172.21.15.29:0/2639290142 conn(0x7f47200680a0 msgr2=0x7f472015a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 shutdown_connections 2023-10-07T09:29:54.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:54.038+0000 7f47269ff700 1 -- 172.21.15.29:0/2639290142 wait complete. 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi144.front.sepia.ceph.com 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:29:54.057 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-07T09:29:54.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:54.058 DEBUG:teuthology.orchestra.run.smithi144:> ip addr 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:54.112 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:22 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f0 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:23 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f1 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5e brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5f brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: inet 172.21.15.144/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft 42241sec preferred_lft 42241sec 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2023-10-07T09:29:54.113 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:54.114 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-07T09:29:54.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:54.114 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:54.114 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:54.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add name ceph-brx type bridge 2023-10-07T09:29:54.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr flush dev ceph-brx 2023-10-07T09:29:54.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set ceph-brx up 2023-10-07T09:29:54.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-07T09:29:54.115 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:54.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:54.554 DEBUG:teuthology.orchestra.run.smithi144:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:29:54.664 INFO:teuthology.orchestra.run.smithi144.stdout:1 2023-10-07T09:29:54.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:54.667 DEBUG:teuthology.orchestra.run.smithi144:> route 2023-10-07T09:29:54.736 INFO:teuthology.orchestra.run.smithi144.stdout:Kernel IP routing table 2023-10-07T09:29:54.737 INFO:teuthology.orchestra.run.smithi144.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:29:54.737 INFO:teuthology.orchestra.run.smithi144.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-07T09:29:54.737 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-07T09:29:54.737 INFO:teuthology.orchestra.run.smithi144.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:29:54.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:54.737 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:54.737 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:54.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-07T09:29:54.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-07T09:29:54.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-07T09:29:54.738 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:55.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:55.450 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-07T09:29:55.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:55.496 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list-id 2023-10-07T09:29:55.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:55.542 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:55.543 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:55.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-07T09:29:55.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-07T09:29:55.543 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:55.727 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-07T09:29:55.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-07T09:29:55.728 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:56.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:56.004 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:56.004 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:56.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set brx.0 up 2023-10-07T09:29:56.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set dev brx.0 master ceph-brx 2023-10-07T09:29:56.005 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:56.179 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:56.179 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-07T09:29:56.225 INFO:teuthology.orchestra.run.smithi144.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-07T09:29:56.225 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:56.225 DEBUG:teuthology.orchestra.run.smithi144:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-07T09:29:56.309 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:29:56.309 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:29:56.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-07T09:29:56.449 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-07T09:29:56.449 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-07T09:29:56.450 INFO:teuthology.orchestra.run.smithi144.stdout:invalid new device string format 2023-10-07T09:29:56.882 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-07T09:29:56.882 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-07T09:29:56.882 INFO:teuthology.orchestra.run.smithi144.stdout:invalid new device string format 2023-10-07T09:29:56.882 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:56.882 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:56.906 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:invalid new device string format 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:29:56.907 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:29:56.912 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:29:56.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:56.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-07T09:29:56.981 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-10-07T09:29:56.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:29:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.187+0000 7f1c0e3ff700 1 Processor -- start 2023-10-07T09:29:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.187+0000 7f1c0e3ff700 1 -- start start 2023-10-07T09:29:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.188+0000 7f1c0e3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1c08157290 0x7f1c080cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.188+0000 7f1c0e3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c08158ad0 con 0x7f1c080b1a10 2023-10-07T09:29:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.188+0000 7f1c0e3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0815b140 con 0x7f1c080ae8b0 2023-10-07T09:29:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.188+0000 7f1c0e3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0815b420 con 0x7f1c08157290 2023-10-07T09:29:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.189+0000 7f1c0c99c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1c080b1a10 0x7f1c08157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50852/0 (socket says 172.21.15.29:50852) 2023-10-07T09:29:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.189+0000 7f1c0c99c700 1 -- 172.21.15.29:0/686341664 learned_addr learned my addr 172.21.15.29:0/686341664 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.189+0000 7f1c07fff700 1 --2- 172.21.15.29:0/686341664 >> v2:172.21.15.144:3300/0 conn(0x7f1c08157290 0x7f1c080cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.189+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528690500 0 0) 0x7f1c08158ad0 con 0x7f1c080b1a10 2023-10-07T09:29:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.189+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf0003570 con 0x7f1c080b1a10 2023-10-07T09:29:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399454915 0 0) 0x7f1c0815b140 con 0x7f1c080ae8b0 2023-10-07T09:29:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c08158ad0 con 0x7f1c080ae8b0 2023-10-07T09:29:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c07fff700 1 -- 172.21.15.29:0/686341664 >> v1:172.21.15.144:6789/0 conn(0x7f1c080ae8b0 legacy=0x7f1c080b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c07fff700 1 -- 172.21.15.29:0/686341664 >> v1:172.21.15.29:6789/0 conn(0x7f1c080b1a10 legacy=0x7f1c08157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984479931 0 0) 0x7f1bf0003570 con 0x7f1c080b1a10 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c07fff700 1 -- 172.21.15.29:0/686341664 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0815b700 con 0x7f1c08157290 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c07fff700 1 --2- 172.21.15.29:0/686341664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c08157290 0x7f1c080cc870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80098e0 tx=0x7f1bf80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8be280135742638 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c000017f0 con 0x7f1c08157290 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf8011420 con 0x7f1c08157290 2023-10-07T09:29:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.190+0000 7f1c06ffd700 1 -- 172.21.15.29:0/686341664 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80117c0 con 0x7f1c08157290 2023-10-07T09:29:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/686341664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c08157290 msgr2=0x7f1c080cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/686341664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c08157290 0x7f1c080cc870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80098e0 tx=0x7f1bf80091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/686341664 shutdown_connections 2023-10-07T09:29:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/686341664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c08157290 0x7f1c080cc870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/686341664 >> 172.21.15.29:0/686341664 conn(0x7f1c080680a0 msgr2=0x7f1c080cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/686341664 shutdown_connections 2023-10-07T09:29:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.191+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/686341664 wait complete. 2023-10-07T09:29:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.192+0000 7f1c0e3ff700 1 Processor -- start 2023-10-07T09:29:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 -- start start 2023-10-07T09:29:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c080b1a10 0x7f1c080c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c08157290 0x7f1c0816afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 0x7f1c080af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08155fd0 con 0x7f1c08154f70 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08153520 con 0x7f1c080b1a10 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.193+0000 7f1c0e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08153800 con 0x7f1c08157290 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 0x7f1c080af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 0x7f1c080af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48720/0 (socket says 172.21.15.29:48720) 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 -- 172.21.15.29:0/1302539067 learned_addr learned my addr 172.21.15.29:0/1302539067 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c07fff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c080b1a10 0x7f1c080c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c077fe700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c08157290 0x7f1c0816afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 -- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c08157290 msgr2=0x7f1c0816afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c08157290 0x7f1c0816afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 -- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c080b1a10 msgr2=0x7f1c080c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c080b1a10 0x7f1c080c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 -- 172.21.15.29:0/1302539067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf8009420 con 0x7f1c08154f70 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c07fff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c080b1a10 0x7f1c080c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.194+0000 7f1c0c99c700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 0x7f1c080af180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000cb40 tx=0x7f1c00007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.195+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00004dc0 con 0x7f1c08154f70 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.195+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000a6a0 con 0x7f1c08154f70 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.195+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000aa60 con 0x7f1c08154f70 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.195+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c08155730 con 0x7f1c08154f70 2023-10-07T09:29:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.195+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c08155cf0 con 0x7f1c08154f70 2023-10-07T09:29:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.196+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0804c3c0 con 0x7f1c08154f70 2023-10-07T09:29:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.197+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1c00007150 con 0x7f1c08154f70 2023-10-07T09:29:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.197+0000 7f1beffff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.198+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1c000840e0 con 0x7f1c08154f70 2023-10-07T09:29:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.198+0000 7f1c07fff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.198+0000 7f1c07fff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80093f0 tx=0x7f1bf8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.202+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c00051330 con 0x7f1c08154f70 2023-10-07T09:29:57.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.448+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1c0815b9e0 con 0x7f1c08154f70 2023-10-07T09:29:57.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.449+0000 7f1beffff700 1 -- 172.21.15.29:0/1302539067 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1c00018070 con 0x7f1c08154f70 2023-10-07T09:29:57.450 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:29:57.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 msgr2=0x7f1bf40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80093f0 tx=0x7f1bf8016000 comp rx=0 tx=0).stop 2023-10-07T09:29:57.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 msgr2=0x7f1c080af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:57.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 0x7f1c080af180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000cb40 tx=0x7f1c00007c70 comp rx=0 tx=0).stop 2023-10-07T09:29:57.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 shutdown_connections 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08154f70 0x7f1c080af180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c080b1a10 0x7f1c080c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 --2- 172.21.15.29:0/1302539067 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c08157290 0x7f1c0816afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 >> 172.21.15.29:0/1302539067 conn(0x7f1c080680a0 msgr2=0x7f1c080cb800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 shutdown_connections 2023-10-07T09:29:57.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:57.451+0000 7f1c0e3ff700 1 -- 172.21.15.29:0/1302539067 wait complete. 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi144.front.sepia.ceph.com 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:29:57.470 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-07T09:29:57.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.471 DEBUG:teuthology.orchestra.run.smithi144:> ip addr 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:57.519 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:22 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f0 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:23 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f1 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5e brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:57.520 INFO:teuthology.orchestra.run.smithi144.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:29:57.521 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5f brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:57.521 INFO:teuthology.orchestra.run.smithi144.stdout: inet 172.21.15.144/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft 42237sec preferred_lft 42237sec 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether f6:b6:b8:86:c6:58 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::7849:45ff:fe03:bab4/64 scope link 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:29:57.522 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether f6:b6:b8:86:c6:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-07T09:29:57.523 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::f4b6:b8ff:fe86:c658/64 scope link tentative 2023-10-07T09:29:57.523 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:57.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.524 DEBUG:teuthology.orchestra.run.smithi144:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:29:57.592 INFO:teuthology.orchestra.run.smithi144.stdout:1 2023-10-07T09:29:57.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.595 DEBUG:teuthology.orchestra.run.smithi144:> route 2023-10-07T09:29:57.644 INFO:teuthology.orchestra.run.smithi144.stdout:Kernel IP routing table 2023-10-07T09:29:57.644 INFO:teuthology.orchestra.run.smithi144.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:29:57.644 INFO:teuthology.orchestra.run.smithi144.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-07T09:29:57.644 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-07T09:29:57.644 INFO:teuthology.orchestra.run.smithi144.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:29:57.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.645 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:57.645 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:57.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-07T09:29:57.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-07T09:29:57.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-07T09:29:57.645 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:57.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.814 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-07T09:29:57.861 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-07T09:29:57.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.861 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list-id 2023-10-07T09:29:57.946 INFO:teuthology.orchestra.run.smithi144.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-07T09:29:57.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:57.948 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:57.948 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:57.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-07T09:29:57.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-07T09:29:57.948 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:58.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:58.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-07T09:29:58.200 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:58.200 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:58.200 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 3e:7d:e7:45:e9:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::3c7d:e7ff:fe45:e96d/64 scope link 2023-10-07T09:29:58.201 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:58.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:29:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 3e:7d:e7:45:e9:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-07T09:29:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-07T09:29:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::3c7d:e7ff:fe45:e96d/64 scope link 2023-10-07T09:29:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:29:58.276 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-07T09:29:58.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-07T09:29:58.277 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:58.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:58.525 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-07T09:29:58.525 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-07T09:29:58.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set brx.1 up 2023-10-07T09:29:58.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set dev brx.1 master ceph-brx 2023-10-07T09:29:58.525 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-07T09:29:58.697 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:58.697 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-07T09:29:58.744 INFO:teuthology.orchestra.run.smithi144.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-07T09:29:58.744 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:29:58.745 DEBUG:teuthology.orchestra.run.smithi144:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-07T09:29:58.830 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:29:58.830 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:29:58.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-07T09:29:58.969 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-07T09:29:58.969 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-07T09:29:58.969 INFO:teuthology.orchestra.run.smithi144.stdout:invalid new device string format 2023-10-07T09:29:58.984 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-07T09:29:58.984 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-07T09:29:58.984 INFO:teuthology.orchestra.run.smithi144.stdout:invalid new device string format 2023-10-07T09:29:58.984 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:58.984 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:59.005 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-07T09:29:59.005 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-07T09:29:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:invalid new device string format 2023-10-07T09:29:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:29:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:29:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:29:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:29:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:29:59.010 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:29:59.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:29:59.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-07T09:29:59.079 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-07T09:29:59.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:29:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.304+0000 7fc31cfff700 1 Processor -- start 2023-10-07T09:29:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.304+0000 7fc31cfff700 1 -- start start 2023-10-07T09:29:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31cfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 0x7fc3180cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31cfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 0x7fc318157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31cfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc318158ad0 con 0x7fc3180ae8b0 2023-10-07T09:29:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31cfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31815b140 con 0x7fc3180b1a10 2023-10-07T09:29:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31cfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31815b420 con 0x7fc318157290 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 0x7fc3180cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc316d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 0x7fc318157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc316d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 0x7fc318157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:45158/0 (socket says 172.21.15.29:45158) 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc316d9d700 1 -- 172.21.15.29:0/3321772883 learned_addr learned my addr 172.21.15.29:0/3321772883 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc31659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 0x7fc3180cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:40224/0 (socket says 172.21.15.29:40224) 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.306+0000 7fc315d9b700 1 --1- 172.21.15.29:0/3321772883 >> v1:172.21.15.144:6789/0 conn(0x7fc3180ae8b0 0x7fc3180b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:33190/0 (socket says 172.21.15.29:33190) 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31659c700 1 -- 172.21.15.29:0/3321772883 >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 msgr2=0x7fc318157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31659c700 1 --2- 172.21.15.29:0/3321772883 >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 0x7fc318157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31659c700 1 -- 172.21.15.29:0/3321772883 >> v1:172.21.15.144:6789/0 conn(0x7fc3180ae8b0 legacy=0x7fc3180b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31659c700 1 -- 172.21.15.29:0/3321772883 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31815b700 con 0x7fc318157290 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc316d9d700 1 --2- 172.21.15.29:0/3321772883 >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 0x7fc318157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31659c700 1 --2- 172.21.15.29:0/3321772883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 0x7fc3180cc870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3080098e0 tx=0x7fc3080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e86060507d2ff77 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31559a700 1 -- 172.21.15.29:0/3321772883 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30800b640 con 0x7fc318157290 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31559a700 1 -- 172.21.15.29:0/3321772883 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30800b7a0 con 0x7fc318157290 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.307+0000 7fc31559a700 1 -- 172.21.15.29:0/3321772883 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30800bb40 con 0x7fc318157290 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 -- 172.21.15.29:0/3321772883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 msgr2=0x7fc3180cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 --2- 172.21.15.29:0/3321772883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 0x7fc3180cc870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3080098e0 tx=0x7fc3080091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 -- 172.21.15.29:0/3321772883 shutdown_connections 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 --2- 172.21.15.29:0/3321772883 >> v2:172.21.15.144:3300/0 conn(0x7fc3180b1a10 0x7fc318157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 --2- 172.21.15.29:0/3321772883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc318157290 0x7fc3180cc870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 -- 172.21.15.29:0/3321772883 >> 172.21.15.29:0/3321772883 conn(0x7fc3180680a0 msgr2=0x7fc3180cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 -- 172.21.15.29:0/3321772883 shutdown_connections 2023-10-07T09:29:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.308+0000 7fc31cfff700 1 -- 172.21.15.29:0/3321772883 wait complete. 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.309+0000 7fc31cfff700 1 Processor -- start 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.309+0000 7fc31cfff700 1 -- start start 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.310+0000 7fc31cfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 0x7fc31816a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.310+0000 7fc31cfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc318157290 0x7fc31806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.310+0000 7fc31cfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31816d8f0 0x7fc318154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.310+0000 7fc31cfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31806eef0 con 0x7fc31816d8f0 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.310+0000 7fc31cfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180afec0 con 0x7fc318157290 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.310+0000 7fc31cfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31816a090 con 0x7fc3180b1a10 2023-10-07T09:29:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 0x7fc31816a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc316d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31816d8f0 0x7fc318154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 0x7fc31816a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:40226/0 (socket says 172.21.15.29:40226) 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 -- 172.21.15.29:0/491134141 learned_addr learned my addr 172.21.15.29:0/491134141 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc315d9b700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc318157290 0x7fc31806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 -- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc318157290 msgr2=0x7fc31806e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc318157290 0x7fc31806e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 -- 172.21.15.29:0/491134141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31816d8f0 msgr2=0x7fc318154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31816d8f0 0x7fc318154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.311+0000 7fc31659c700 1 -- 172.21.15.29:0/491134141 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc310009ae0 con 0x7fc3180b1a10 2023-10-07T09:29:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.312+0000 7fc31659c700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 0x7fc31816a520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc30800a010 tx=0x7fc308017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:59.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.312+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30801bcc0 con 0x7fc3180b1a10 2023-10-07T09:29:59.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.312+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30801be60 con 0x7fc3180b1a10 2023-10-07T09:29:59.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.312+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc308009420 con 0x7fc3180b1a10 2023-10-07T09:29:59.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.312+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc308011670 con 0x7fc3180b1a10 2023-10-07T09:29:59.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.312+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc318169a40 con 0x7fc3180b1a10 2023-10-07T09:29:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.313+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31804c3c0 con 0x7fc3180b1a10 2023-10-07T09:29:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.314+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc308011a60 con 0x7fc3180b1a10 2023-10-07T09:29:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.315+0000 7fc3067fc700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc2f8066f30 0x7fc2f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.315+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc308085e20 con 0x7fc3180b1a10 2023-10-07T09:29:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.315+0000 7fc315d9b700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc2f8066f30 0x7fc2f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.315+0000 7fc315d9b700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc2f8066f30 0x7fc2f80693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc318153e50 tx=0x7fc30c007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.319+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc308052ff0 con 0x7fc3180b1a10 2023-10-07T09:29:59.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.521+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc31815b9e0 con 0x7fc3180b1a10 2023-10-07T09:29:59.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.523+0000 7fc3067fc700 1 -- 172.21.15.29:0/491134141 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7fc308056610 con 0x7fc3180b1a10 2023-10-07T09:29:59.524 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:29:59.524 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:29:59.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc2f8066f30 msgr2=0x7fc2f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc2f8066f30 0x7fc2f80693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc318153e50 tx=0x7fc30c007eb0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 msgr2=0x7fc31816a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 0x7fc31816a520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc30800a010 tx=0x7fc308017000 comp rx=0 tx=0).stop 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 shutdown_connections 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc2f8066f30 0x7fc2f80693e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31816d8f0 0x7fc318154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3180b1a10 0x7fc31816a520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 --2- 172.21.15.29:0/491134141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc318157290 0x7fc31806e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 >> 172.21.15.29:0/491134141 conn(0x7fc3180680a0 msgr2=0x7fc3181564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 shutdown_connections 2023-10-07T09:29:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.525+0000 7fc31cfff700 1 -- 172.21.15.29:0/491134141 wait complete. 2023-10-07T09:29:59.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:29:59.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:29:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.742+0000 7f7c8a7ff700 1 Processor -- start 2023-10-07T09:29:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.742+0000 7f7c8a7ff700 1 -- start start 2023-10-07T09:29:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.743+0000 7f7c8a7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c840d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.743+0000 7f7c8a7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c840cbf20 con 0x7f7c84159750 2023-10-07T09:29:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.743+0000 7f7c8a7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c840cc200 con 0x7f7c840ce8a0 2023-10-07T09:29:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.743+0000 7f7c8a7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c840cc4e0 con 0x7f7c84159b20 2023-10-07T09:29:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c83fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7c84159750 0x7f7c840ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50868/0 (socket says 172.21.15.29:50868) 2023-10-07T09:29:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c83fff700 1 -- 172.21.15.29:0/1866238756 learned_addr learned my addr 172.21.15.29:0/1866238756 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c837fe700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c840d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871685309 0 0) 0x7f7c840cbf20 con 0x7f7c84159750 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c6c003570 con 0x7f7c84159750 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2796757341 0 0) 0x7f7c6c003570 con 0x7f7c84159750 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 msgr2=0x7f7c840d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c840d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 >> v1:172.21.15.144:6789/0 conn(0x7f7c840ce8a0 legacy=0x7f7c840c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c840cc7c0 con 0x7f7c84159750 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f7c740015f0 con 0x7f7c84159750 2023-10-07T09:29:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.744+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 >> v1:172.21.15.29:6789/0 conn(0x7f7c84159750 legacy=0x7f7c840ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c82ffd700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c6c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c82ffd700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c003fd0 0x7f7c6c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c82ffd700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c6c004d50 0x7f7c6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c840cc4e0 con 0x7f7c6c004d50 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c00bba0 con 0x7f7c6c003fd0 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c00bd60 con 0x7f7c84159b20 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c837fe700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c003fd0 0x7f7c6c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c88d9c700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c6c004d50 0x7f7c6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c83fff700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c6c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c88d9c700 1 -- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 msgr2=0x7f7c6c003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c88d9c700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c6c003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c88d9c700 1 -- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c003fd0 msgr2=0x7f7c6c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c88d9c700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c003fd0 0x7f7c6c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c88d9c700 1 -- 172.21.15.29:0/1866238756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6c003570 con 0x7f7c6c004d50 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.745+0000 7f7c83fff700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c84159b20 0x7f7c6c003a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c88d9c700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c6c004d50 0x7f7c6c0051c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0020a0 tx=0x7f7c7c0016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6287134f99f204a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0019e0 con 0x7f7c6c004d50 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7c00c8b0 con 0x7f7c6c004d50 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c82ffd700 1 -- 172.21.15.29:0/1866238756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c00cc50 con 0x7f7c6c004d50 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/1866238756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c6c004d50 msgr2=0x7f7c6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/1866238756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c6c004d50 0x7f7c6c0051c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0020a0 tx=0x7f7c7c0016d0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c88d9c700 1 -- 172.21.15.29:0/1866238756 reap_dead start 2023-10-07T09:29:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/1866238756 shutdown_connections 2023-10-07T09:29:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.746+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/1866238756 >> 172.21.15.29:0/1866238756 conn(0x7f7c84063a10 msgr2=0x7f7c840d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.747+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/1866238756 shutdown_connections 2023-10-07T09:29:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.747+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/1866238756 wait complete. 2023-10-07T09:29:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.749+0000 7f7c8a7ff700 1 Processor -- start 2023-10-07T09:29:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.749+0000 7f7c8a7ff700 1 -- start start 2023-10-07T09:29:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c8a7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c004d50 0x7f7c840b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c8a7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 0x7f7c841500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c8a7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c841505e0 0x7f7c84169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c8a7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c84150a80 con 0x7f7c8415f270 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c8a7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8415e540 con 0x7f7c6c004d50 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c8a7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8415ede0 con 0x7f7c841505e0 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 0x7f7c841500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 0x7f7c841500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48748/0 (socket says 172.21.15.29:48748) 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c837fe700 1 -- 172.21.15.29:0/2691703322 learned_addr learned my addr 172.21.15.29:0/2691703322 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.750+0000 7f7c83fff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c004d50 0x7f7c840b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c88d9c700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c841505e0 0x7f7c84169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c837fe700 1 -- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c841505e0 msgr2=0x7f7c84169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c837fe700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c841505e0 0x7f7c84169990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c837fe700 1 -- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c004d50 msgr2=0x7f7c840b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c837fe700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c004d50 0x7f7c840b3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c837fe700 1 -- 172.21.15.29:0/2691703322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c840b3c60 con 0x7f7c8415f270 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c88d9c700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c841505e0 0x7f7c84169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c837fe700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 0x7f7c841500a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c78011b40 tx=0x7f7c780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800fa10 con 0x7f7c8415f270 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7800fb70 con 0x7f7c8415f270 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8416a710 con 0x7f7c8415f270 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78014440 con 0x7f7c8415f270 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.751+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8415ea70 con 0x7f7c8415f270 2023-10-07T09:29:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.752+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8415ef50 con 0x7f7c8415f270 2023-10-07T09:29:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.753+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c78014830 con 0x7f7c8415f270 2023-10-07T09:29:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.754+0000 7f7c6bfff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c70066f30 0x7f7c700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:29:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.754+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7c78018030 con 0x7f7c8415f270 2023-10-07T09:29:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.755+0000 7f7c83fff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c70066f30 0x7f7c700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:29:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.755+0000 7f7c83fff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c70066f30 0x7f7c700693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c74000e10 tx=0x7f7c7400c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:29:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.758+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c78052210 con 0x7f7c8415f270 2023-10-07T09:29:59.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.959+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c8404c3c0 con 0x7f7c8415f270 2023-10-07T09:29:59.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.960+0000 7f7c6bfff700 1 -- 172.21.15.29:0/2691703322 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c78059e60 con 0x7f7c8415f270 2023-10-07T09:29:59.961 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c70066f30 msgr2=0x7f7c700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c70066f30 0x7f7c700693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c74000e10 tx=0x7f7c7400c040 comp rx=0 tx=0).stop 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 msgr2=0x7f7c841500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 0x7f7c841500a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c78011b40 tx=0x7f7c780091a0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 shutdown_connections 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c70066f30 0x7f7c700693e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c841505e0 0x7f7c84169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c6c004d50 0x7f7c840b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 --2- 172.21.15.29:0/2691703322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8415f270 0x7f7c841500a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 >> 172.21.15.29:0/2691703322 conn(0x7f7c84063a10 msgr2=0x7f7c840704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:29:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 shutdown_connections 2023-10-07T09:29:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:29:59.962+0000 7f7c8a7ff700 1 -- 172.21.15.29:0/2691703322 wait complete. 2023-10-07T09:29:59.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:30:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.193+0000 7f399719d700 1 Processor -- start 2023-10-07T09:30:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.193+0000 7f399719d700 1 -- start start 2023-10-07T09:30:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.194+0000 7f399719d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f399006a050 0x7f399015c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.194+0000 7f399719d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39900c8fc0 0x7f399015cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.194+0000 7f399719d700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900a98e0 con 0x7f39900c8bf0 2023-10-07T09:30:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.194+0000 7f399719d700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900aabd0 con 0x7f399006a050 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.194+0000 7f399719d700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900a9b30 con 0x7f39900c8fc0 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.194+0000 7f399599a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f399006a050 0x7f399015c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f399006a050 0x7f399015c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48756/0 (socket says 172.21.15.29:48756) 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 -- 172.21.15.29:0/2891035038 learned_addr learned my addr 172.21.15.29:0/2891035038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399699c700 1 --2- 172.21.15.29:0/2891035038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39900c8fc0 0x7f399015cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 -- 172.21.15.29:0/2891035038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39900c8fc0 msgr2=0x7f399015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 --2- 172.21.15.29:0/2891035038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39900c8fc0 0x7f399015cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 -- 172.21.15.29:0/2891035038 >> v1:172.21.15.144:6789/0 conn(0x7f39900c8bf0 legacy=0x7f39900c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 -- 172.21.15.29:0/2891035038 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900a9e10 con 0x7f399006a050 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f3995199700 1 -- 172.21.15.29:0/2891035038 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965639849 0 0) 0x7f39900a98e0 con 0x7f39900c8bf0 2023-10-07T09:30:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f399599a700 1 --2- 172.21.15.29:0/2891035038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f399006a050 0x7f399015c690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f39800099b0 tx=0x7f3980009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd1a93cbc35e765 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f3995199700 1 -- 172.21.15.29:0/2891035038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980009ea0 con 0x7f399006a050 2023-10-07T09:30:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f3995199700 1 -- 172.21.15.29:0/2891035038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398000b8f0 con 0x7f399006a050 2023-10-07T09:30:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.195+0000 7f3995199700 1 -- 172.21.15.29:0/2891035038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000bc90 con 0x7f399006a050 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 -- 172.21.15.29:0/2891035038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f399006a050 msgr2=0x7f399015c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 --2- 172.21.15.29:0/2891035038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f399006a050 0x7f399015c690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f39800099b0 tx=0x7f3980009250 comp rx=0 tx=0).stop 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 -- 172.21.15.29:0/2891035038 shutdown_connections 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 --2- 172.21.15.29:0/2891035038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39900c8fc0 0x7f399015cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 --2- 172.21.15.29:0/2891035038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f399006a050 0x7f399015c690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 -- 172.21.15.29:0/2891035038 >> 172.21.15.29:0/2891035038 conn(0x7f3990067610 msgr2=0x7f3990067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 -- 172.21.15.29:0/2891035038 shutdown_connections 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.196+0000 7f399719d700 1 -- 172.21.15.29:0/2891035038 wait complete. 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.198+0000 7f399719d700 1 Processor -- start 2023-10-07T09:30:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.198+0000 7f399719d700 1 -- start start 2023-10-07T09:30:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399719d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f399006a050 0x7f39900ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399719d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 0x7f39900b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399719d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39900c8fc0 0x7f3990150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399719d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900af0c0 con 0x7f39900c8bf0 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399719d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900af3a0 con 0x7f39900c8fc0 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399719d700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900af680 con 0x7f399006a050 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 0x7f39900b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 0x7f39900b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48760/0 (socket says 172.21.15.29:48760) 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399599a700 1 -- 172.21.15.29:0/2787125112 learned_addr learned my addr 172.21.15.29:0/2787125112 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399699c700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39900c8fc0 0x7f3990150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.199+0000 7f399619b700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f399006a050 0x7f39900ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399599a700 1 -- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f399006a050 msgr2=0x7f39900ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399599a700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f399006a050 0x7f39900ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399599a700 1 -- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39900c8fc0 msgr2=0x7f3990150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399599a700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39900c8fc0 0x7f3990150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399599a700 1 -- 172.21.15.29:0/2787125112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800094f0 con 0x7f39900c8bf0 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399619b700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f399006a050 0x7f39900ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399599a700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 0x7f39900b0d10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f399006cdc0 tx=0x7f3980017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000bc70 con 0x7f39900c8bf0 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980011420 con 0x7f39900c8bf0 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39900af960 con 0x7f39900c8bf0 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39800117e0 con 0x7f39900c8bf0 2023-10-07T09:30:00.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.200+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39900ca9a0 con 0x7f39900c8bf0 2023-10-07T09:30:00.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.202+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39900afba0 con 0x7f39900c8bf0 2023-10-07T09:30:00.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.202+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3980011b90 con 0x7f39900c8bf0 2023-10-07T09:30:00.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.203+0000 7f39867fc700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3974066f30 0x7f39740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.203+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3980085f70 con 0x7f39900c8bf0 2023-10-07T09:30:00.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.204+0000 7f399619b700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3974066f30 0x7f39740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.204+0000 7f399619b700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3974066f30 0x7f39740693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f398c0011e0 tx=0x7f398c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:00.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.207+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3980053140 con 0x7f39900c8bf0 2023-10-07T09:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.408+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f39900683e0 con 0x7f39900c8bf0 2023-10-07T09:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.409+0000 7f39867fc700 1 -- 172.21.15.29:0/2787125112 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3980056760 con 0x7f39900c8bf0 2023-10-07T09:30:00.410 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3974066f30 msgr2=0x7f39740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3974066f30 0x7f39740693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f398c0011e0 tx=0x7f398c009040 comp rx=0 tx=0).stop 2023-10-07T09:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 msgr2=0x7f39900b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 0x7f39900b0d10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f399006cdc0 tx=0x7f3980017000 comp rx=0 tx=0).stop 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 shutdown_connections 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3974066f30 0x7f39740693e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39900c8fc0 0x7f3990150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f399006a050 0x7f39900ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 --2- 172.21.15.29:0/2787125112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39900c8bf0 0x7f39900b0d10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 >> 172.21.15.29:0/2787125112 conn(0x7f3990067610 msgr2=0x7f399015bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 shutdown_connections 2023-10-07T09:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.411+0000 7f399719d700 1 -- 172.21.15.29:0/2787125112 wait complete. 2023-10-07T09:30:00.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:30:00.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.627+0000 7f0a85bff700 1 Processor -- start 2023-10-07T09:30:00.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.628+0000 7f0a85bff700 1 -- start start 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a85bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a80153cc0 0x7f0a800b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a85bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a8015ade0 0x7f0a800b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a85bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a80154090 con 0x7f0a800b0000 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a85bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a80159f60 con 0x7f0a80153cc0 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a85bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8015a240 con 0x7f0a8015ade0 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a7f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a80153cc0 0x7f0a800b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a7ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0a800b0000 0x7f0a8015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50884/0 (socket says 172.21.15.29:50884) 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a7ffff700 1 -- 172.21.15.29:0/3026159557 learned_addr learned my addr 172.21.15.29:0/3026159557 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.629+0000 7f0a7effd700 1 --2- 172.21.15.29:0/3026159557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a8015ade0 0x7f0a800b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7f7fe700 1 -- 172.21.15.29:0/3026159557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a8015ade0 msgr2=0x7f0a800b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7f7fe700 1 --2- 172.21.15.29:0/3026159557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a8015ade0 0x7f0a800b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7f7fe700 1 -- 172.21.15.29:0/3026159557 >> v1:172.21.15.29:6789/0 conn(0x7f0a800b0000 legacy=0x7f0a8015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7f7fe700 1 -- 172.21.15.29:0/3026159557 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8015a520 con 0x7f0a80153cc0 2023-10-07T09:30:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7f7fe700 1 --2- 172.21.15.29:0/3026159557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a800b17c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a700098d0 tx=0x7f0a70009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657228bcbdd9ec4a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7e7fc700 1 -- 172.21.15.29:0/3026159557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a70009e70 con 0x7f0a80153cc0 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7e7fc700 1 -- 172.21.15.29:0/3026159557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a70005c10 con 0x7f0a80153cc0 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.630+0000 7f0a7e7fc700 1 -- 172.21.15.29:0/3026159557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7000b650 con 0x7f0a80153cc0 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 -- 172.21.15.29:0/3026159557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 msgr2=0x7f0a800b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 --2- 172.21.15.29:0/3026159557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a800b17c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a700098d0 tx=0x7f0a70009120 comp rx=0 tx=0).stop 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 -- 172.21.15.29:0/3026159557 shutdown_connections 2023-10-07T09:30:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 --2- 172.21.15.29:0/3026159557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a800b17c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 --2- 172.21.15.29:0/3026159557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a8015ade0 0x7f0a800b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 -- 172.21.15.29:0/3026159557 >> 172.21.15.29:0/3026159557 conn(0x7f0a80067610 msgr2=0x7f0a80067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:00.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 -- 172.21.15.29:0/3026159557 shutdown_connections 2023-10-07T09:30:00.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.631+0000 7f0a85bff700 1 -- 172.21.15.29:0/3026159557 wait complete. 2023-10-07T09:30:00.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.632+0000 7f0a85bff700 1 Processor -- start 2023-10-07T09:30:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 -- start start 2023-10-07T09:30:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a800b0000 0x7f0a80145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a80158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a8015ade0 0x7f0a801699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a80156110 con 0x7f0a80153cc0 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a801563f0 con 0x7f0a8015ade0 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.633+0000 7f0a85bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a801566d0 con 0x7f0a800b0000 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a80158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a80158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48780/0 (socket says 172.21.15.29:48780) 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 -- 172.21.15.29:0/1603062592 learned_addr learned my addr 172.21.15.29:0/1603062592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7f7fe700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a800b0000 0x7f0a80145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7ffff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a8015ade0 0x7f0a801699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 -- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a800b0000 msgr2=0x7f0a80145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a800b0000 0x7f0a80145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 -- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a8015ade0 msgr2=0x7f0a801699b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a8015ade0 0x7f0a801699b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 -- 172.21.15.29:0/1603062592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a700093c0 con 0x7f0a80153cc0 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7ffff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a8015ade0 0x7f0a801699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:00.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.634+0000 7f0a7effd700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a80158990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400d860 tx=0x7f0a74008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.635+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a740125f0 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.635+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74012750 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.635+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a801569b0 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.635+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74012b10 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.635+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a80146370 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.636+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8004c3c0 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.637+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0a7401e460 con 0x7f0a80153cc0 2023-10-07T09:30:00.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.637+0000 7f0a677fe700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0a6c066f30 0x7f0a6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.638+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0a74084b70 con 0x7f0a80153cc0 2023-10-07T09:30:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.638+0000 7f0a7f7fe700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0a6c066f30 0x7f0a6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.638+0000 7f0a7f7fe700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0a6c066f30 0x7f0a6c0693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009160 tx=0x7f0a70016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:00.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.642+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a74051d40 con 0x7f0a80153cc0 2023-10-07T09:30:00.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.842+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a8015a800 con 0x7f0a80153cc0 2023-10-07T09:30:00.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.842+0000 7f0a677fe700 1 -- 172.21.15.29:0/1603062592 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a74055360 con 0x7f0a80153cc0 2023-10-07T09:30:00.843 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0a6c066f30 msgr2=0x7f0a6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0a6c066f30 0x7f0a6c0693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009160 tx=0x7f0a70016000 comp rx=0 tx=0).stop 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 msgr2=0x7f0a80158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a80158990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400d860 tx=0x7f0a74008040 comp rx=0 tx=0).stop 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 shutdown_connections 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0a6c066f30 0x7f0a6c0693e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a8015ade0 0x7f0a801699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a800b0000 0x7f0a80145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 --2- 172.21.15.29:0/1603062592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a80153cc0 0x7f0a80158990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.844+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 >> 172.21.15.29:0/1603062592 conn(0x7f0a80067610 msgr2=0x7f0a800b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.845+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 shutdown_connections 2023-10-07T09:30:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:00.845+0000 7f0a85bff700 1 -- 172.21.15.29:0/1603062592 wait complete. 2023-10-07T09:30:00.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:01.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.057+0000 7f23c35ff700 1 Processor -- start 2023-10-07T09:30:01.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.057+0000 7f23c35ff700 1 -- start start 2023-10-07T09:30:01.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c35ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f23bc159750 0x7f23bc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c35ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc0ce8a0 0x7f23bc0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c35ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23bc0cbf20 con 0x7f23bc159b20 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c35ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc0cc200 con 0x7f23bc159750 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c35ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0cc4e0 con 0x7f23bc0ce8a0 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c0b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f23bc159b20 0x7f23bc0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50890/0 (socket says 172.21.15.29:50890) 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.058+0000 7f23c0b9a700 1 -- 172.21.15.29:0/527974656 learned_addr learned my addr 172.21.15.29:0/527974656 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23c1b9c700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc0ce8a0 0x7f23bc0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23c139b700 1 --2- 172.21.15.29:0/527974656 >> v2:172.21.15.144:3300/0 conn(0x7f23bc159750 0x7f23bc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287761285 0 0) 0x7f23bc0cbf20 con 0x7f23bc159b20 2023-10-07T09:30:01.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a8003570 con 0x7f23bc159b20 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 797755382 0 0) 0x7f23a8003570 con 0x7f23bc159b20 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc0ce8a0 msgr2=0x7f23bc0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc0ce8a0 0x7f23bc0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 >> v2:172.21.15.144:3300/0 conn(0x7f23bc159750 msgr2=0x7f23bc0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 --2- 172.21.15.29:0/527974656 >> v2:172.21.15.144:3300/0 conn(0x7f23bc159750 0x7f23bc0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0cc7c0 con 0x7f23bc159b20 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f23ac001470 con 0x7f23bc159b20 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 >> v1:172.21.15.29:6789/0 conn(0x7f23bc159b20 legacy=0x7f23bc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23a8003ae0 0x7f23a8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23a8004810 0x7f23a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a80051c0 0x7f23a800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0cc200 con 0x7f23a8003ae0 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0cc4e0 con 0x7f23a80051c0 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a800bfd0 con 0x7f23a8004810 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23c139b700 1 --2- 172.21.15.29:0/527974656 >> v2:172.21.15.144:3300/0 conn(0x7f23bc159750 0x7f23bc0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.059+0000 7f23c1b9c700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc0ce8a0 0x7f23bc0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c0b9a700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23a8004810 0x7f23a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23a8003ae0 0x7f23a8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c1b9c700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a80051c0 0x7f23a800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 -- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23a8004810 msgr2=0x7f23a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:01.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23a8004810 0x7f23a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 -- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a80051c0 msgr2=0x7f23a800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a80051c0 0x7f23a800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 -- 172.21.15.29:0/527974656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8003570 con 0x7f23a8003ae0 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.060+0000 7f23c139b700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23a8003ae0 0x7f23a8003ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009da0 tx=0x7f23b8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d050d0f393fe0b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8009730 con 0x7f23a8003ae0 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b800e8c0 con 0x7f23a8003ae0 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23b3fff700 1 -- 172.21.15.29:0/527974656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b800ec60 con 0x7f23a8003ae0 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c35ff700 1 -- 172.21.15.29:0/527974656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23a8003ae0 msgr2=0x7f23a8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c35ff700 1 --2- 172.21.15.29:0/527974656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23a8003ae0 0x7f23a8003ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009da0 tx=0x7f23b8009120 comp rx=0 tx=0).stop 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c1b9c700 1 -- 172.21.15.29:0/527974656 reap_dead start 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c35ff700 1 -- 172.21.15.29:0/527974656 shutdown_connections 2023-10-07T09:30:01.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c35ff700 1 -- 172.21.15.29:0/527974656 >> 172.21.15.29:0/527974656 conn(0x7f23bc063a10 msgr2=0x7f23bc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:01.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c35ff700 1 -- 172.21.15.29:0/527974656 shutdown_connections 2023-10-07T09:30:01.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.061+0000 7f23c35ff700 1 -- 172.21.15.29:0/527974656 wait complete. 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.063+0000 7f23c35ff700 1 Processor -- start 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.063+0000 7f23c35ff700 1 -- start start 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c35ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a8003ae0 0x7f23bc149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c35ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 0x7f23bc152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c35ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc152c90 0x7f23bc0c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c35ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc153110 con 0x7f23bc0b38b0 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c35ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc068540 con 0x7f23a8003ae0 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c35ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc068b00 con 0x7f23bc152c90 2023-10-07T09:30:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c0b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 0x7f23bc152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c0b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 0x7f23bc152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48794/0 (socket says 172.21.15.29:48794) 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c0b9a700 1 -- 172.21.15.29:0/2699910802 learned_addr learned my addr 172.21.15.29:0/2699910802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.064+0000 7f23c139b700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a8003ae0 0x7f23bc149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c1b9c700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc152c90 0x7f23bc0c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c0b9a700 1 -- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc152c90 msgr2=0x7f23bc0c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c0b9a700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc152c90 0x7f23bc0c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c0b9a700 1 -- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a8003ae0 msgr2=0x7f23bc149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c0b9a700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a8003ae0 0x7f23bc149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c0b9a700 1 -- 172.21.15.29:0/2699910802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0c83a0 con 0x7f23bc0b38b0 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c0b9a700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 0x7f23bc152750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0073c0 tx=0x7f23ac001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac018030 con 0x7f23bc0b38b0 2023-10-07T09:30:01.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00f920 con 0x7f23bc0b38b0 2023-10-07T09:30:01.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac00fce0 con 0x7f23bc0b38b0 2023-10-07T09:30:01.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc0c8680 con 0x7f23bc0b38b0 2023-10-07T09:30:01.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.065+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc1496b0 con 0x7f23bc0b38b0 2023-10-07T09:30:01.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.066+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc04c3c0 con 0x7f23bc0b38b0 2023-10-07T09:30:01.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.067+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f23ac01b070 con 0x7f23bc0b38b0 2023-10-07T09:30:01.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.068+0000 7f23b17fa700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23a0066f30 0x7f23a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.068+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f23ac019070 con 0x7f23bc0b38b0 2023-10-07T09:30:01.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.069+0000 7f23c139b700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23a0066f30 0x7f23a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.069+0000 7f23c139b700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23a0066f30 0x7f23a00693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009570 tx=0x7f23b8019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.072+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23ac0551d0 con 0x7f23bc0b38b0 2023-10-07T09:30:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.279+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23bc068c40 con 0x7f23bc0b38b0 2023-10-07T09:30:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.280+0000 7f23b17fa700 1 -- 172.21.15.29:0/2699910802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f23ac054d60 con 0x7f23bc0b38b0 2023-10-07T09:30:01.282 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:01.282 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23a0066f30 msgr2=0x7f23a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23a0066f30 0x7f23a00693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009570 tx=0x7f23b8019000 comp rx=0 tx=0).stop 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 msgr2=0x7f23bc152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 0x7f23bc152750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0073c0 tx=0x7f23ac001dc0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 shutdown_connections 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23a0066f30 0x7f23a00693e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23bc152c90 0x7f23bc0c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23a8003ae0 0x7f23bc149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 --2- 172.21.15.29:0/2699910802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23bc0b38b0 0x7f23bc152750 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 >> 172.21.15.29:0/2699910802 conn(0x7f23bc063a10 msgr2=0x7f23bc149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 shutdown_connections 2023-10-07T09:30:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.283+0000 7f23c35ff700 1 -- 172.21.15.29:0/2699910802 wait complete. 2023-10-07T09:30:01.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:30:01.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:30:01.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.502+0000 7f74f87ff700 1 Processor -- start 2023-10-07T09:30:01.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.502+0000 7f74f87ff700 1 -- start start 2023-10-07T09:30:01.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f87ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f74f0159790 0x7f74f006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f006a8c0 0x7f74f006b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f87ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f006a430 con 0x7f74f00c6f40 2023-10-07T09:30:01.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f87ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f0159b60 con 0x7f74f0159790 2023-10-07T09:30:01.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00c7a70 con 0x7f74f006a8c0 2023-10-07T09:30:01.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f6d9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f74f00c6f40 0x7f74f00c7360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50894/0 (socket says 172.21.15.29:50894) 2023-10-07T09:30:01.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.503+0000 7f74f6d9c700 1 -- 172.21.15.29:0/1791250361 learned_addr learned my addr 172.21.15.29:0/1791250361 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:01.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f659b700 1 --2- 172.21.15.29:0/1791250361 >> v2:172.21.15.144:3300/0 conn(0x7f74f0159790 0x7f74f006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f5d9a700 1 --2- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f006a8c0 0x7f74f006b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f5599700 1 -- 172.21.15.29:0/1791250361 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009842339 0 0) 0x7f74f006a430 con 0x7f74f00c6f40 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f5599700 1 -- 172.21.15.29:0/1791250361 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc003570 con 0x7f74f00c6f40 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f659b700 1 -- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f006a8c0 msgr2=0x7f74f006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f659b700 1 --2- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f006a8c0 0x7f74f006b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f659b700 1 -- 172.21.15.29:0/1791250361 >> v1:172.21.15.29:6789/0 conn(0x7f74f00c6f40 legacy=0x7f74f00c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f659b700 1 -- 172.21.15.29:0/1791250361 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0069820 con 0x7f74f0159790 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.504+0000 7f74f659b700 1 --2- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f006aeb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0098e0 tx=0x7f74ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c10294699cb43977 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f5599700 1 -- 172.21.15.29:0/1791250361 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec011630 con 0x7f74f0159790 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f5599700 1 -- 172.21.15.29:0/1791250361 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec011790 con 0x7f74f0159790 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f5599700 1 -- 172.21.15.29:0/1791250361 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec011b30 con 0x7f74f0159790 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 -- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 msgr2=0x7f74f006aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 --2- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f006aeb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0098e0 tx=0x7f74ec0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 -- 172.21.15.29:0/1791250361 shutdown_connections 2023-10-07T09:30:01.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 --2- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f006aeb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 --2- 172.21.15.29:0/1791250361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f006a8c0 0x7f74f006b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 -- 172.21.15.29:0/1791250361 >> 172.21.15.29:0/1791250361 conn(0x7f74f00c9410 msgr2=0x7f74f00c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:01.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.505+0000 7f74f87ff700 1 -- 172.21.15.29:0/1791250361 shutdown_connections 2023-10-07T09:30:01.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.506+0000 7f74f87ff700 1 -- 172.21.15.29:0/1791250361 wait complete. 2023-10-07T09:30:01.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.507+0000 7f74f87ff700 1 Processor -- start 2023-10-07T09:30:01.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.507+0000 7f74f87ff700 1 -- start start 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f87ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 0x7f74f00aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f87ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f00acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f015e260 0x7f74f0152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f87ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00ad770 con 0x7f74f00c6f40 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f87ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00ae5d0 con 0x7f74f0159790 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00ad9c0 con 0x7f74f015e260 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f659b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 0x7f74f00aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f659b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 0x7f74f00aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48808/0 (socket says 172.21.15.29:48808) 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f659b700 1 -- 172.21.15.29:0/1383331533 learned_addr learned my addr 172.21.15.29:0/1383331533 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:01.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.508+0000 7f74f5d9a700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f00acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f6d9c700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f015e260 0x7f74f0152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f659b700 1 -- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f015e260 msgr2=0x7f74f0152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f659b700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f015e260 0x7f74f0152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f659b700 1 -- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 msgr2=0x7f74f00acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f659b700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f00acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f659b700 1 -- 172.21.15.29:0/1383331533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec009420 con 0x7f74f00c6f40 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f6d9c700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f015e260 0x7f74f0152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f5d9a700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f00acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f659b700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 0x7f74f00aea60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009ed0 tx=0x7f74ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec011900 con 0x7f74f00c6f40 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec011a60 con 0x7f74f00c6f40 2023-10-07T09:30:01.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec026680 con 0x7f74f00c6f40 2023-10-07T09:30:01.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f87ff700 1 -- 172.21.15.29:0/1383331533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f00adca0 con 0x7f74f00c6f40 2023-10-07T09:30:01.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.509+0000 7f74f87ff700 1 -- 172.21.15.29:0/1383331533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f00ae260 con 0x7f74f00c6f40 2023-10-07T09:30:01.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.511+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f004c3c0 con 0x7f74f00c6f40 2023-10-07T09:30:01.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.512+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f74ec023030 con 0x7f74f00c6f40 2023-10-07T09:30:01.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.512+0000 7f74e67fc700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74d4066f60 0x7f74d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.513+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f74ec085d60 con 0x7f74f00c6f40 2023-10-07T09:30:01.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.513+0000 7f74f5d9a700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74d4066f60 0x7f74d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.514+0000 7f74f5d9a700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74d4066f60 0x7f74d4069410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f74f016db00 tx=0x7f74e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.517+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74ec052fb0 con 0x7f74f00c6f40 2023-10-07T09:30:01.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.717+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74f0063640 con 0x7f74f00c6f40 2023-10-07T09:30:01.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.717+0000 7f74e67fc700 1 -- 172.21.15.29:0/1383331533 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f74ec0565d0 con 0x7f74f00c6f40 2023-10-07T09:30:01.719 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:30:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74d4066f60 msgr2=0x7f74d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74d4066f60 0x7f74d4069410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f74f016db00 tx=0x7f74e0008040 comp rx=0 tx=0).stop 2023-10-07T09:30:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 msgr2=0x7f74f00aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 0x7f74f00aea60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009ed0 tx=0x7f74ec017000 comp rx=0 tx=0).stop 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 shutdown_connections 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f74d4066f60 0x7f74d4069410 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74f015e260 0x7f74f0152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f00c6f40 0x7f74f00aea60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 --2- 172.21.15.29:0/1383331533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74f0159790 0x7f74f00acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 >> 172.21.15.29:0/1383331533 conn(0x7f74f00c9410 msgr2=0x7f74f0153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 shutdown_connections 2023-10-07T09:30:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.720+0000 7f74d3fff700 1 -- 172.21.15.29:0/1383331533 wait complete. 2023-10-07T09:30:01.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:30:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.937+0000 7fc92d3ff700 1 Processor -- start 2023-10-07T09:30:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.938+0000 7fc92d3ff700 1 -- start start 2023-10-07T09:30:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc92d3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc9280c6470 0x7fc9280d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc92d3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9280cefb0 con 0x7fc928159b20 2023-10-07T09:30:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc92d3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9280cbf20 con 0x7fc928159750 2023-10-07T09:30:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc92d3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9280cc200 con 0x7fc9280c6470 2023-10-07T09:30:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc926ffd700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc928159750 0x7fc9280ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:33206/0 (socket says 172.21.15.29:33206) 2023-10-07T09:30:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc926ffd700 1 -- 172.21.15.29:0/1161968959 learned_addr learned my addr 172.21.15.29:0/1161968959 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc9267fc700 1 --1- 172.21.15.29:0/1161968959 >> v1:172.21.15.29:6789/0 conn(0x7fc928159b20 0x7fc9280ce8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50900/0 (socket says 172.21.15.29:50900) 2023-10-07T09:30:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc9277fe700 1 --2- 172.21.15.29:0/1161968959 >> v2:172.21.15.144:3300/0 conn(0x7fc9280c6470 0x7fc9280d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.940+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928443121 0 0) 0x7fc9280cbf20 con 0x7fc928159750 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc910003570 con 0x7fc928159750 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094874305 0 0) 0x7fc9280cefb0 con 0x7fc928159b20 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9280cbf20 con 0x7fc928159b20 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc9277fe700 1 -- 172.21.15.29:0/1161968959 >> v1:172.21.15.144:6789/0 conn(0x7fc928159750 legacy=0x7fc9280ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc9277fe700 1 -- 172.21.15.29:0/1161968959 >> v1:172.21.15.29:6789/0 conn(0x7fc928159b20 legacy=0x7fc9280ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc9277fe700 1 -- 172.21.15.29:0/1161968959 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280cc4e0 con 0x7fc9280c6470 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.941+0000 7fc9277fe700 1 --2- 172.21.15.29:0/1161968959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280d10a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc920009e60 tx=0x7fc920009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ffaacbcc4db6a8f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92000e070 con 0x7fc9280c6470 2023-10-07T09:30:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92000bab0 con 0x7fc9280c6470 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc925ffb700 1 -- 172.21.15.29:0/1161968959 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92000be70 con 0x7fc9280c6470 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1161968959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 msgr2=0x7fc9280d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1161968959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280d10a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc920009e60 tx=0x7fc920009780 comp rx=0 tx=0).stop 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1161968959 shutdown_connections 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1161968959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280d10a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.942+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1161968959 >> 172.21.15.29:0/1161968959 conn(0x7fc928063a10 msgr2=0x7fc9280d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.943+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1161968959 shutdown_connections 2023-10-07T09:30:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.943+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1161968959 wait complete. 2023-10-07T09:30:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.944+0000 7fc92d3ff700 1 Processor -- start 2023-10-07T09:30:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.945+0000 7fc92d3ff700 1 -- start start 2023-10-07T09:30:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc92d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc92d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc928159750 0x7fc9280c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc92d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc928159b20 0x7fc9281517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc92d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280ca550 con 0x7fc928159b20 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc92d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281548c0 con 0x7fc9280c6470 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc92d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280c7bb0 con 0x7fc928159750 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc926ffd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc9277fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc928159b20 0x7fc9281517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc9267fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc928159750 0x7fc9280c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc926ffd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:45242/0 (socket says 172.21.15.29:45242) 2023-10-07T09:30:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.946+0000 7fc926ffd700 1 -- 172.21.15.29:0/1911726532 learned_addr learned my addr 172.21.15.29:0/1911726532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc926ffd700 1 -- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc928159750 msgr2=0x7fc9280c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc926ffd700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc928159750 0x7fc9280c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc926ffd700 1 -- 172.21.15.29:0/1911726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc928159b20 msgr2=0x7fc9281517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc926ffd700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc928159b20 0x7fc9281517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc926ffd700 1 -- 172.21.15.29:0/1911726532 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc920009a20 con 0x7fc9280c6470 2023-10-07T09:30:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc9277fe700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc928159b20 0x7fc9281517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc926ffd700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280c9f70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc918001170 tx=0x7fc918007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc918005d20 con 0x7fc9280c6470 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc918005e80 con 0x7fc9280c6470 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9180072e0 con 0x7fc9280c6470 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.947+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc928151d20 con 0x7fc9280c6470 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.948+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9280b60d0 con 0x7fc9280c6470 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.949+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc928154690 con 0x7fc9280c6470 2023-10-07T09:30:01.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.950+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc918007690 con 0x7fc9280c6470 2023-10-07T09:30:01.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.951+0000 7fc90effd700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc914066f30 0x7fc9140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:01.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.951+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc918083b10 con 0x7fc9280c6470 2023-10-07T09:30:01.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.951+0000 7fc9267fc700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc914066f30 0x7fc9140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:01.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.952+0000 7fc9267fc700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc914066f30 0x7fc9140693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc9280ccb50 tx=0x7fc91c007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:01.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:01.956+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc91804b5a0 con 0x7fc9280c6470 2023-10-07T09:30:02.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.159+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc92804c3c0 con 0x7fc9280c6470 2023-10-07T09:30:02.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.160+0000 7fc90effd700 1 -- 172.21.15.29:0/1911726532 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9180170a0 con 0x7fc9280c6470 2023-10-07T09:30:02.161 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:30:02.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc914066f30 msgr2=0x7fc9140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc914066f30 0x7fc9140693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc9280ccb50 tx=0x7fc91c007d40 comp rx=0 tx=0).stop 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 msgr2=0x7fc9280c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280c9f70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc918001170 tx=0x7fc918007e00 comp rx=0 tx=0).stop 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 shutdown_connections 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc914066f30 0x7fc9140693e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc928159b20 0x7fc9281517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9280c6470 0x7fc9280c9f70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 --2- 172.21.15.29:0/1911726532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc928159750 0x7fc9280c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 >> 172.21.15.29:0/1911726532 conn(0x7fc928063a10 msgr2=0x7fc928153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 shutdown_connections 2023-10-07T09:30:02.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.162+0000 7fc92d3ff700 1 -- 172.21.15.29:0/1911726532 wait complete. 2023-10-07T09:30:02.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:30:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.378+0000 7f169d3ff700 1 Processor -- start 2023-10-07T09:30:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.378+0000 7f169d3ff700 1 -- start start 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.379+0000 7f169d3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f169815ade0 0x7f16980b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.379+0000 7f169d3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1698154090 con 0x7f1698153cc0 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.379+0000 7f169d3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1698159f60 con 0x7f16980b0540 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.379+0000 7f169d3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16980b09e0 con 0x7f169815ade0 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f1696ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1698153cc0 0x7f16980b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50902/0 (socket says 172.21.15.29:50902) 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f1696ffd700 1 -- 172.21.15.29:0/1585495293 learned_addr learned my addr 172.21.15.29:0/1585495293 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f16967fc700 1 --2- 172.21.15.29:0/1585495293 >> v2:172.21.15.144:3300/0 conn(0x7f169815ade0 0x7f16980b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f1695ffb700 1 -- 172.21.15.29:0/1585495293 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774866762 0 0) 0x7f1698154090 con 0x7f1698153cc0 2023-10-07T09:30:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f1695ffb700 1 -- 172.21.15.29:0/1585495293 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1680003570 con 0x7f1698153cc0 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f16967fc700 1 -- 172.21.15.29:0/1585495293 >> v1:172.21.15.144:6789/0 conn(0x7f16980b0540 legacy=0x7f169815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f16967fc700 1 -- 172.21.15.29:0/1585495293 >> v1:172.21.15.29:6789/0 conn(0x7f1698153cc0 legacy=0x7f16980b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f1695ffb700 1 -- 172.21.15.29:0/1585495293 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241012098 0 0) 0x7f1680003570 con 0x7f1698153cc0 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.380+0000 7f16967fc700 1 -- 172.21.15.29:0/1585495293 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169815a240 con 0x7f169815ade0 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.381+0000 7f16967fc700 1 --2- 172.21.15.29:0/1585495293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f169815ade0 0x7f16980b0000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f168c0098e0 tx=0x7f168c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15efafa3fe2a098f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.381+0000 7f1695ffb700 1 -- 172.21.15.29:0/1585495293 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16880015f0 con 0x7f169815ade0 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.381+0000 7f1695ffb700 1 -- 172.21.15.29:0/1585495293 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c011420 con 0x7f169815ade0 2023-10-07T09:30:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.381+0000 7f1695ffb700 1 -- 172.21.15.29:0/1585495293 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c0117c0 con 0x7f169815ade0 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.381+0000 7f169d3ff700 1 -- 172.21.15.29:0/1585495293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f169815ade0 msgr2=0x7f16980b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.381+0000 7f169d3ff700 1 --2- 172.21.15.29:0/1585495293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f169815ade0 0x7f16980b0000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f168c0098e0 tx=0x7f168c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.382+0000 7f169d3ff700 1 -- 172.21.15.29:0/1585495293 shutdown_connections 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.382+0000 7f169d3ff700 1 --2- 172.21.15.29:0/1585495293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f169815ade0 0x7f16980b0000 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.382+0000 7f169d3ff700 1 -- 172.21.15.29:0/1585495293 >> 172.21.15.29:0/1585495293 conn(0x7f1698067610 msgr2=0x7f1698067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.382+0000 7f169d3ff700 1 -- 172.21.15.29:0/1585495293 shutdown_connections 2023-10-07T09:30:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.382+0000 7f169d3ff700 1 -- 172.21.15.29:0/1585495293 wait complete. 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.383+0000 7f169d3ff700 1 Processor -- start 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.383+0000 7f169d3ff700 1 -- start start 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f169d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16980b0540 0x7f16981581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f169d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 0x7f1698150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f169d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f169815ade0 0x7f1698151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f169d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981586f0 con 0x7f1698153cc0 2023-10-07T09:30:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f169d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981510b0 con 0x7f16980b0540 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f169d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169813e920 con 0x7f169815ade0 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.384+0000 7f16967fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 0x7f1698150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 0x7f1698150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48818/0 (socket says 172.21.15.29:48818) 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 -- 172.21.15.29:0/4188384015 learned_addr learned my addr 172.21.15.29:0/4188384015 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f1696ffd700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16980b0540 0x7f16981581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16977fe700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f169815ade0 0x7f1698151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 -- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f169815ade0 msgr2=0x7f1698151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f169815ade0 0x7f1698151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 -- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16980b0540 msgr2=0x7f16981581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16980b0540 0x7f16981581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 -- 172.21.15.29:0/4188384015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c009420 con 0x7f1698153cc0 2023-10-07T09:30:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16977fe700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f169815ade0 0x7f1698151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:02.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f16967fc700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 0x7f1698150b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f168c0091e0 tx=0x7f168c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:02.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.385+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c01cbe0 con 0x7f1698153cc0 2023-10-07T09:30:02.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.386+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c01cd40 con 0x7f1698153cc0 2023-10-07T09:30:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.386+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c01a6a0 con 0x7f1698153cc0 2023-10-07T09:30:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.386+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1698157d20 con 0x7f1698153cc0 2023-10-07T09:30:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.386+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16981573f0 con 0x7f1698153cc0 2023-10-07T09:30:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.387+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169804c3c0 con 0x7f1698153cc0 2023-10-07T09:30:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.388+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f168c01aa10 con 0x7f1698153cc0 2023-10-07T09:30:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.389+0000 7f167effd700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1684066f30 0x7f16840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.389+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f168c085f30 con 0x7f1698153cc0 2023-10-07T09:30:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.389+0000 7f1696ffd700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1684066f30 0x7f16840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.390+0000 7f1696ffd700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1684066f30 0x7f16840693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1688000c00 tx=0x7f1688001320 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.393+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f168c053100 con 0x7f1698153cc0 2023-10-07T09:30:02.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.594+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16980689b0 con 0x7f1698153cc0 2023-10-07T09:30:02.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.595+0000 7f167effd700 1 -- 172.21.15.29:0/4188384015 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f168c0290a0 con 0x7f1698153cc0 2023-10-07T09:30:02.596 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:30:02.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1684066f30 msgr2=0x7f16840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1684066f30 0x7f16840693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1688000c00 tx=0x7f1688001320 comp rx=0 tx=0).stop 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 msgr2=0x7f1698150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 0x7f1698150b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f168c0091e0 tx=0x7f168c016000 comp rx=0 tx=0).stop 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 shutdown_connections 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1684066f30 0x7f16840693e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f169815ade0 0x7f1698151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16980b0540 0x7f16981581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 --2- 172.21.15.29:0/4188384015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1698153cc0 0x7f1698150b70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 >> 172.21.15.29:0/4188384015 conn(0x7f1698067610 msgr2=0x7f16980af380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 shutdown_connections 2023-10-07T09:30:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.597+0000 7f169d3ff700 1 -- 172.21.15.29:0/4188384015 wait complete. 2023-10-07T09:30:02.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.809+0000 7efc57dff700 1 Processor -- start 2023-10-07T09:30:02.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.809+0000 7efc57dff700 1 -- start start 2023-10-07T09:30:02.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc57dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efc50159ec0 0x7efc5015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc57dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc500af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc57dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5014d790 con 0x7efc500ad370 2023-10-07T09:30:02.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc57dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5015a880 con 0x7efc50159ec0 2023-10-07T09:30:02.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc57dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500ae5d0 con 0x7efc500af400 2023-10-07T09:30:02.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc55b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc500ad370 0x7efc500cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50916/0 (socket says 172.21.15.29:50916) 2023-10-07T09:30:02.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc55b9b700 1 -- 172.21.15.29:0/3080320247 learned_addr learned my addr 172.21.15.29:0/3080320247 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.811+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539079833 0 0) 0x7efc5014d790 con 0x7efc500ad370 2023-10-07T09:30:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc5639c700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc500af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003570 con 0x7efc500ad370 2023-10-07T09:30:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc5539a700 1 --2- 172.21.15.29:0/3080320247 >> v2:172.21.15.144:3300/0 conn(0x7efc50159ec0 0x7efc5015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956716719 0 0) 0x7efc3c003570 con 0x7efc500ad370 2023-10-07T09:30:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 msgr2=0x7efc500af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc500af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 >> v2:172.21.15.144:3300/0 conn(0x7efc50159ec0 msgr2=0x7efc5015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 --2- 172.21.15.29:0/3080320247 >> v2:172.21.15.144:3300/0 conn(0x7efc50159ec0 0x7efc5015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500ad740 con 0x7efc500ad370 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7efc4c001470 con 0x7efc500ad370 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 >> v1:172.21.15.29:6789/0 conn(0x7efc500ad370 legacy=0x7efc500cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc3c003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc3c0040b0 0x7efc3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5015a880 con 0x7efc3c004d50 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500ae5d0 con 0x7efc3c0040b0 2023-10-07T09:30:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c00bb60 con 0x7efc500af400 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc5639c700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc55b9b700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc3c003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.812+0000 7efc5539a700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc3c0040b0 0x7efc3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5639c700 1 -- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 msgr2=0x7efc3c003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5639c700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc3c003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5639c700 1 -- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc3c0040b0 msgr2=0x7efc3c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5639c700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc3c0040b0 0x7efc3c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5639c700 1 -- 172.21.15.29:0/3080320247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c003570 con 0x7efc3c004d50 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc55b9b700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500af400 0x7efc3c003ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5539a700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc3c0040b0 0x7efc3c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc5639c700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc3c0051c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc480094d0 tx=0x7efc4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7c2c17e677c50e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48010d60 con 0x7efc3c004d50 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc48010ec0 con 0x7efc3c004d50 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.813+0000 7efc54b99700 1 -- 172.21.15.29:0/3080320247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48012680 con 0x7efc3c004d50 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc57dff700 1 -- 172.21.15.29:0/3080320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 msgr2=0x7efc3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc57dff700 1 --2- 172.21.15.29:0/3080320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc3c0051c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc480094d0 tx=0x7efc4800b040 comp rx=0 tx=0).stop 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc5639c700 1 -- 172.21.15.29:0/3080320247 reap_dead start 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc57dff700 1 -- 172.21.15.29:0/3080320247 shutdown_connections 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc57dff700 1 -- 172.21.15.29:0/3080320247 >> 172.21.15.29:0/3080320247 conn(0x7efc500680a0 msgr2=0x7efc500cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc57dff700 1 -- 172.21.15.29:0/3080320247 shutdown_connections 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.814+0000 7efc57dff700 1 -- 172.21.15.29:0/3080320247 wait complete. 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.816+0000 7efc57dff700 1 Processor -- start 2023-10-07T09:30:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.816+0000 7efc57dff700 1 -- start start 2023-10-07T09:30:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc57dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc500a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc57dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500a6ca0 0x7efc500a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc57dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc500abb20 0x7efc500abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc57dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500ac4e0 con 0x7efc3c004d50 2023-10-07T09:30:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc57dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500a7ec0 con 0x7efc500abb20 2023-10-07T09:30:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc57dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500a81a0 con 0x7efc500a6ca0 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc500a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc500a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48834/0 (socket says 172.21.15.29:48834) 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 -- 172.21.15.29:0/1403330647 learned_addr learned my addr 172.21.15.29:0/1403330647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 -- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500a6ca0 msgr2=0x7efc500a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500a6ca0 0x7efc500a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 -- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc500abb20 msgr2=0x7efc500abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc500abb20 0x7efc500abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.817+0000 7efc55b9b700 1 -- 172.21.15.29:0/1403330647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500a8480 con 0x7efc3c004d50 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.818+0000 7efc55b9b700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc500a9ce0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc4c001200 tx=0x7efc501578d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.818+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c018030 con 0x7efc3c004d50 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.818+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c00fa70 con 0x7efc3c004d50 2023-10-07T09:30:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.818+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c00fdf0 con 0x7efc3c004d50 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.818+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc500a8760 con 0x7efc3c004d50 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.818+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc500a54c0 con 0x7efc3c004d50 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.819+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5004c3c0 con 0x7efc3c004d50 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.820+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc4c01b070 con 0x7efc3c004d50 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.821+0000 7efc45ffb700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efc34066f30 0x7efc340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.821+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7efc4c019070 con 0x7efc3c004d50 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.822+0000 7efc5539a700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efc34066f30 0x7efc340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.822+0000 7efc5539a700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efc34066f30 0x7efc340693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc40009170 tx=0x7efc400091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:02.825+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc4c051870 con 0x7efc3c004d50 2023-10-07T09:30:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.031+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc5006b590 con 0x7efc3c004d50 2023-10-07T09:30:03.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.032+0000 7efc45ffb700 1 -- 172.21.15.29:0/1403330647 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15446 (secure 0 0 0) 0x7efc4c054e90 con 0x7efc3c004d50 2023-10-07T09:30:03.033 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:03.033 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:29:22.920871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efc34066f30 msgr2=0x7efc340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efc34066f30 0x7efc340693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc40009170 tx=0x7efc400091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 msgr2=0x7efc500a9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc500a9ce0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc4c001200 tx=0x7efc501578d0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 shutdown_connections 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efc34066f30 0x7efc340693e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc500abb20 0x7efc500abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc3c004d50 0x7efc500a9ce0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 --2- 172.21.15.29:0/1403330647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc500a6ca0 0x7efc500a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 >> 172.21.15.29:0/1403330647 conn(0x7efc500680a0 msgr2=0x7efc5014cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 shutdown_connections 2023-10-07T09:30:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.035+0000 7efc57dff700 1 -- 172.21.15.29:0/1403330647 wait complete. 2023-10-07T09:30:03.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:30:03.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:30:03.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.252+0000 7fe6c5fff700 1 Processor -- start 2023-10-07T09:30:03.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.252+0000 7fe6c5fff700 1 -- start start 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6c5fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6c0159750 0x7fe6c00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6c5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c00ce8a0 0x7fe6c00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6c5fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00cbf20 con 0x7fe6c0159b20 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6c5fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00cc200 con 0x7fe6c0159750 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6c5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00cc4e0 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6beffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe6c0159b20 0x7fe6c00ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37382/0 (socket says 172.21.15.29:37382) 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.253+0000 7fe6beffd700 1 -- 172.21.15.29:0/2017728535 learned_addr learned my addr 172.21.15.29:0/2017728535 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:03.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bf7fe700 1 --2- 172.21.15.29:0/2017728535 >> v2:172.21.15.144:3300/0 conn(0x7fe6c0159750 0x7fe6c00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bffff700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c00ce8a0 0x7fe6c00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6be7fc700 1 -- 172.21.15.29:0/2017728535 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841316691 0 0) 0x7fe6c00cbf20 con 0x7fe6c0159b20 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6be7fc700 1 -- 172.21.15.29:0/2017728535 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a8003570 con 0x7fe6c0159b20 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c00ce8a0 msgr2=0x7fe6c00c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bf7fe700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c00ce8a0 0x7fe6c00c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2017728535 >> v1:172.21.15.29:6789/0 conn(0x7fe6c0159b20 legacy=0x7fe6c00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2017728535 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00cc7c0 con 0x7fe6c0159750 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.254+0000 7fe6bffff700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c00ce8a0 0x7fe6c00c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6bf7fe700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159750 0x7fe6c00d10a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00098d0 tx=0x7fe6b0009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69aa223b1ab3caaf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6be7fc700 1 -- 172.21.15.29:0/2017728535 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0005c10 con 0x7fe6c0159750 2023-10-07T09:30:03.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6be7fc700 1 -- 172.21.15.29:0/2017728535 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b0005d70 con 0x7fe6c0159750 2023-10-07T09:30:03.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6be7fc700 1 -- 172.21.15.29:0/2017728535 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0006260 con 0x7fe6c0159750 2023-10-07T09:30:03.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159750 msgr2=0x7fe6c00d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159750 0x7fe6c00d10a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00098d0 tx=0x7fe6b0009120 comp rx=0 tx=0).stop 2023-10-07T09:30:03.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2017728535 shutdown_connections 2023-10-07T09:30:03.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c00ce8a0 0x7fe6c00c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2017728535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159750 0x7fe6c00d10a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.255+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2017728535 >> 172.21.15.29:0/2017728535 conn(0x7fe6c0063a10 msgr2=0x7fe6c00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.256+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2017728535 shutdown_connections 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.256+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2017728535 wait complete. 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.257+0000 7fe6c5fff700 1 Processor -- start 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.257+0000 7fe6c5fff700 1 -- start start 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6c5fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 0x7fe6c014b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6c5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c0159750 0x7fe6c00c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6c5fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159b20 0x7fe6c00c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6c5fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c015ac50 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6c5fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c94a0 con 0x7fe6c0159b20 2023-10-07T09:30:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6c5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c014b7e0 con 0x7fe6c0159750 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6bf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 0x7fe6c014b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6bf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 0x7fe6c014b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40538/0 (socket says 172.21.15.29:40538) 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2950897917 learned_addr learned my addr 172.21.15.29:0/2950897917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6bffff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159b20 0x7fe6c00c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.258+0000 7fe6beffd700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c0159750 0x7fe6c00c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c0159750 msgr2=0x7fe6c00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6bf7fe700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c0159750 0x7fe6c00c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159b20 msgr2=0x7fe6c00c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6bf7fe700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159b20 0x7fe6c00c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6bf7fe700 1 -- 172.21.15.29:0/2950897917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b8009a20 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6bf7fe700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 0x7fe6c014b2a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009ec0 tx=0x7fe6b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b001d9c0 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b001db20 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b001b480 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b00093c0 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.259+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c015a320 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.260+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c00ca8b0 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.261+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe6b001b870 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.262+0000 7fe6a77fe700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe6ac066f30 0x7fe6ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.262+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe6b00864e0 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.263+0000 7fe6beffd700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe6ac066f30 0x7fe6ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.263+0000 7fe6beffd700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe6ac066f30 0x7fe6ac0693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00c64e0 tx=0x7fe6b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.266+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6b0053770 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.467+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6c004c3c0 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.467+0000 7fe6a77fe700 1 -- 172.21.15.29:0/2950897917 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6b0056d90 con 0x7fe6c00ce8a0 2023-10-07T09:30:03.468 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe6ac066f30 msgr2=0x7fe6ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe6ac066f30 0x7fe6ac0693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00c64e0 tx=0x7fe6b4009040 comp rx=0 tx=0).stop 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 msgr2=0x7fe6c014b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 0x7fe6c014b2a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009ec0 tx=0x7fe6b0016000 comp rx=0 tx=0).stop 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 shutdown_connections 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe6ac066f30 0x7fe6ac0693e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6c0159b20 0x7fe6c00c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6c00ce8a0 0x7fe6c014b2a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 --2- 172.21.15.29:0/2950897917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6c0159750 0x7fe6c00c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.469+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 >> 172.21.15.29:0/2950897917 conn(0x7fe6c0063a10 msgr2=0x7fe6c01563c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:03.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.470+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 shutdown_connections 2023-10-07T09:30:03.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.470+0000 7fe6c5fff700 1 -- 172.21.15.29:0/2950897917 wait complete. 2023-10-07T09:30:03.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:30:03.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.685+0000 7f41d3bff700 1 Processor -- start 2023-10-07T09:30:03.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.685+0000 7f41d3bff700 1 -- start start 2023-10-07T09:30:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d3bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f41cc15ade0 0x7f41cc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d3bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc0b0540 0x7f41cc15c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d3bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc0b09b0 con 0x7f41cc153cc0 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d3bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41cc154090 con 0x7f41cc15ade0 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d3bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc159f60 con 0x7f41cc0b0540 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d199b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41cc153cc0 0x7f41cc0b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37390/0 (socket says 172.21.15.29:37390) 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.686+0000 7f41d199b700 1 -- 172.21.15.29:0/502718469 learned_addr learned my addr 172.21.15.29:0/502718469 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d119a700 1 --2- 172.21.15.29:0/502718469 >> v2:172.21.15.144:3300/0 conn(0x7f41cc15ade0 0x7f41cc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d219c700 1 --2- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc0b0540 0x7f41cc15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d0999700 1 -- 172.21.15.29:0/502718469 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457318766 0 0) 0x7f41cc0b09b0 con 0x7f41cc153cc0 2023-10-07T09:30:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d0999700 1 -- 172.21.15.29:0/502718469 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b8003570 con 0x7f41cc153cc0 2023-10-07T09:30:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d119a700 1 -- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc0b0540 msgr2=0x7f41cc15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d119a700 1 --2- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc0b0540 0x7f41cc15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d119a700 1 -- 172.21.15.29:0/502718469 >> v1:172.21.15.29:6789/0 conn(0x7f41cc153cc0 legacy=0x7f41cc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d119a700 1 -- 172.21.15.29:0/502718469 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc15a240 con 0x7f41cc15ade0 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.687+0000 7f41d119a700 1 --2- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc15ade0 0x7f41cc0b0000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0098e0 tx=0x7f41bc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51f1a314564dfc14 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d0999700 1 -- 172.21.15.29:0/502718469 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc011630 con 0x7f41cc15ade0 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d0999700 1 -- 172.21.15.29:0/502718469 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc011790 con 0x7f41cc15ade0 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d0999700 1 -- 172.21.15.29:0/502718469 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc011b30 con 0x7f41cc15ade0 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 -- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc15ade0 msgr2=0x7f41cc0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 --2- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc15ade0 0x7f41cc0b0000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0098e0 tx=0x7f41bc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 -- 172.21.15.29:0/502718469 shutdown_connections 2023-10-07T09:30:03.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 --2- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc0b0540 0x7f41cc15c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 --2- 172.21.15.29:0/502718469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc15ade0 0x7f41cc0b0000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 -- 172.21.15.29:0/502718469 >> 172.21.15.29:0/502718469 conn(0x7f41cc067610 msgr2=0x7f41cc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:03.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 -- 172.21.15.29:0/502718469 shutdown_connections 2023-10-07T09:30:03.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.688+0000 7f41d3bff700 1 -- 172.21.15.29:0/502718469 wait complete. 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.690+0000 7f41d3bff700 1 Processor -- start 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.690+0000 7f41d3bff700 1 -- start start 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d3bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc0b0540 0x7f41cc152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d3bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 0x7f41cc155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d3bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc15ade0 0x7f41cc0c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d3bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc169800 con 0x7f41cc153cc0 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d3bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc1694c0 con 0x7f41cc0b0540 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d3bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc155c50 con 0x7f41cc15ade0 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 0x7f41cc155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 0x7f41cc155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40540/0 (socket says 172.21.15.29:40540) 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 -- 172.21.15.29:0/1974116514 learned_addr learned my addr 172.21.15.29:0/1974116514 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d199b700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc0b0540 0x7f41cc152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d219c700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc15ade0 0x7f41cc0c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 -- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc15ade0 msgr2=0x7f41cc0c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc15ade0 0x7f41cc0c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 -- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc0b0540 msgr2=0x7f41cc152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc0b0540 0x7f41cc152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.691+0000 7f41d119a700 1 -- 172.21.15.29:0/1974116514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc009420 con 0x7f41cc153cc0 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.692+0000 7f41d119a700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 0x7f41cc155320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41bc013000 tx=0x7f41bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.692+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc011900 con 0x7f41cc153cc0 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.692+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc011a60 con 0x7f41cc153cc0 2023-10-07T09:30:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.692+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc155f30 con 0x7f41cc153cc0 2023-10-07T09:30:03.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.692+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc025640 con 0x7f41cc153cc0 2023-10-07T09:30:03.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.692+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc152740 con 0x7f41cc153cc0 2023-10-07T09:30:03.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.693+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41cc169970 con 0x7f41cc153cc0 2023-10-07T09:30:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.694+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f41bc025a30 con 0x7f41cc153cc0 2023-10-07T09:30:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.695+0000 7f41c1ffb700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f41b0066f30 0x7f41b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.695+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f41bc085c40 con 0x7f41cc153cc0 2023-10-07T09:30:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.695+0000 7f41d199b700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f41b0066f30 0x7f41b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.696+0000 7f41d199b700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f41b0066f30 0x7f41b00693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f41c8001360 tx=0x7f41c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:03.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.700+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41bc052e10 con 0x7f41cc153cc0 2023-10-07T09:30:03.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.900+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f41cc04c3c0 con 0x7f41cc153cc0 2023-10-07T09:30:03.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.900+0000 7f41c1ffb700 1 -- 172.21.15.29:0/1974116514 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f41bc056430 con 0x7f41cc153cc0 2023-10-07T09:30:03.902 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f41b0066f30 msgr2=0x7f41b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f41b0066f30 0x7f41b00693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f41c8001360 tx=0x7f41c8009040 comp rx=0 tx=0).stop 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 msgr2=0x7f41cc155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 0x7f41cc155320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41bc013000 tx=0x7f41bc017000 comp rx=0 tx=0).stop 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 shutdown_connections 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f41b0066f30 0x7f41b00693e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41cc15ade0 0x7f41cc0c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41cc0b0540 0x7f41cc152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 --2- 172.21.15.29:0/1974116514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41cc153cc0 0x7f41cc155320 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:03.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 >> 172.21.15.29:0/1974116514 conn(0x7f41cc067610 msgr2=0x7f41cc15b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:03.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 shutdown_connections 2023-10-07T09:30:03.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:03.903+0000 7f41d3bff700 1 -- 172.21.15.29:0/1974116514 wait complete. 2023-10-07T09:30:03.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:30:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.124+0000 7f85e99ff700 1 Processor -- start 2023-10-07T09:30:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.124+0000 7f85e99ff700 1 -- start start 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.125+0000 7f85e99ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85e4153cc0 0x7f85e40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.125+0000 7f85e99ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85e414cd70 0x7f85e4159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.125+0000 7f85e99ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e4157790 con 0x7f85e414c8d0 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.125+0000 7f85e99ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e40b0950 con 0x7f85e414cd70 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.125+0000 7f85e99ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e4157580 con 0x7f85e4153cc0 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85e414cd70 0x7f85e4159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85e414cd70 0x7f85e4159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40554/0 (socket says 172.21.15.29:40554) 2023-10-07T09:30:04.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 -- 172.21.15.29:0/1971791217 learned_addr learned my addr 172.21.15.29:0/1971791217 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:04.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e2ffd700 1 --2- 172.21.15.29:0/1971791217 >> v2:172.21.15.144:3300/0 conn(0x7f85e4153cc0 0x7f85e40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 -- 172.21.15.29:0/1971791217 >> v2:172.21.15.144:3300/0 conn(0x7f85e4153cc0 msgr2=0x7f85e40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:04.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 --2- 172.21.15.29:0/1971791217 >> v2:172.21.15.144:3300/0 conn(0x7f85e4153cc0 0x7f85e40cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 -- 172.21.15.29:0/1971791217 >> v1:172.21.15.144:6789/0 conn(0x7f85e414c8d0 legacy=0x7f85e40afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 -- 172.21.15.29:0/1971791217 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e4157a70 con 0x7f85e414cd70 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.126+0000 7f85e37fe700 1 --2- 172.21.15.29:0/1971791217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e414cd70 0x7f85e4159680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f85e406d0b0 tx=0x7f85dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90999e7868b521f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e1ffb700 1 -- 172.21.15.29:0/1971791217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc00e070 con 0x7f85e414cd70 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e1ffb700 1 -- 172.21.15.29:0/1971791217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc00ba20 con 0x7f85e414cd70 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e1ffb700 1 -- 172.21.15.29:0/1971791217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0092e0 con 0x7f85e414cd70 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e99ff700 1 -- 172.21.15.29:0/1971791217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e414cd70 msgr2=0x7f85e4159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1971791217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e414cd70 0x7f85e4159680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f85e406d0b0 tx=0x7f85dc009840 comp rx=0 tx=0).stop 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e99ff700 1 -- 172.21.15.29:0/1971791217 shutdown_connections 2023-10-07T09:30:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1971791217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e414cd70 0x7f85e4159680 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1971791217 >> v2:172.21.15.144:3300/0 conn(0x7f85e4153cc0 0x7f85e40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.127+0000 7f85e99ff700 1 -- 172.21.15.29:0/1971791217 >> 172.21.15.29:0/1971791217 conn(0x7f85e4067610 msgr2=0x7f85e4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:04.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.128+0000 7f85e99ff700 1 -- 172.21.15.29:0/1971791217 shutdown_connections 2023-10-07T09:30:04.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.128+0000 7f85e99ff700 1 -- 172.21.15.29:0/1971791217 wait complete. 2023-10-07T09:30:04.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.129+0000 7f85e99ff700 1 Processor -- start 2023-10-07T09:30:04.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.129+0000 7f85e99ff700 1 -- start start 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e99ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85e414c8d0 0x7f85e40ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e99ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85e414cd70 0x7f85e4151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e99ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 0x7f85e4151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e99ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e40ca5e0 con 0x7f85e4153cc0 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e99ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e41508f0 con 0x7f85e414c8d0 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e99ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e4150bd0 con 0x7f85e414cd70 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 0x7f85e4151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 0x7f85e4151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40568/0 (socket says 172.21.15.29:40568) 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.130+0000 7f85e37fe700 1 -- 172.21.15.29:0/1145434341 learned_addr learned my addr 172.21.15.29:0/1145434341 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e2ffd700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85e414c8d0 0x7f85e40ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e27fc700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85e414cd70 0x7f85e4151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e37fe700 1 -- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85e414cd70 msgr2=0x7f85e4151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e37fe700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85e414cd70 0x7f85e4151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e37fe700 1 -- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85e414c8d0 msgr2=0x7f85e40ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e37fe700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85e414c8d0 0x7f85e40ca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e37fe700 1 -- 172.21.15.29:0/1145434341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc009ae0 con 0x7f85e4153cc0 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85e37fe700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 0x7f85e4151ed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009880 tx=0x7f85dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.131+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc008a80 con 0x7f85e4153cc0 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.132+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc008c00 con 0x7f85e4153cc0 2023-10-07T09:30:04.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.132+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc026460 con 0x7f85e4153cc0 2023-10-07T09:30:04.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.132+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e4150eb0 con 0x7f85e4153cc0 2023-10-07T09:30:04.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.132+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e4151470 con 0x7f85e4153cc0 2023-10-07T09:30:04.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.133+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e404c3c0 con 0x7f85e4153cc0 2023-10-07T09:30:04.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.134+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f85dc003040 con 0x7f85e4153cc0 2023-10-07T09:30:04.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.135+0000 7f85caffd700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85d0066f30 0x7f85d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.135+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f85dc085e80 con 0x7f85e4153cc0 2023-10-07T09:30:04.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.135+0000 7f85e2ffd700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85d0066f30 0x7f85d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.136+0000 7f85e2ffd700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85d0066f30 0x7f85d00693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009d10 tx=0x7f85d40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.139+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f85dc053080 con 0x7f85e4153cc0 2023-10-07T09:30:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.339+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f85e40ccd00 con 0x7f85e4153cc0 2023-10-07T09:30:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.339+0000 7f85caffd700 1 -- 172.21.15.29:0/1145434341 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f85dc0566a0 con 0x7f85e4153cc0 2023-10-07T09:30:04.341 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:30:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85d0066f30 msgr2=0x7f85d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85d0066f30 0x7f85d00693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009d10 tx=0x7f85d40091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 msgr2=0x7f85e4151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 0x7f85e4151ed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009880 tx=0x7f85dc018000 comp rx=0 tx=0).stop 2023-10-07T09:30:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 shutdown_connections 2023-10-07T09:30:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85d0066f30 0x7f85d00693e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85e4153cc0 0x7f85e4151ed0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85e414c8d0 0x7f85e40ca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 --2- 172.21.15.29:0/1145434341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85e414cd70 0x7f85e4151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 >> 172.21.15.29:0/1145434341 conn(0x7f85e4067610 msgr2=0x7f85e406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 shutdown_connections 2023-10-07T09:30:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.342+0000 7f85e99ff700 1 -- 172.21.15.29:0/1145434341 wait complete. 2023-10-07T09:30:04.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.554+0000 7fd9f91ff700 1 Processor -- start 2023-10-07T09:30:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.555+0000 7fd9f91ff700 1 -- start start 2023-10-07T09:30:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f91ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9f4159750 0x7fd9f40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f91ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f40ce8a0 0x7fd9f40c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f91ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f40cbf20 con 0x7fd9f4159b20 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f91ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f40cc200 con 0x7fd9f4159750 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f91ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40cc4e0 con 0x7fd9f40ce8a0 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f2d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9f4159750 0x7fd9f40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f2d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9f4159750 0x7fd9f40d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40582/0 (socket says 172.21.15.29:40582) 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f2d9d700 1 -- 172.21.15.29:0/4112337598 learned_addr learned my addr 172.21.15.29:0/4112337598 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.556+0000 7fd9f359e700 1 --2- 172.21.15.29:0/4112337598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f40ce8a0 0x7fd9f40c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f2d9d700 1 -- 172.21.15.29:0/4112337598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f40ce8a0 msgr2=0x7fd9f40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f2d9d700 1 --2- 172.21.15.29:0/4112337598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f40ce8a0 0x7fd9f40c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f2d9d700 1 -- 172.21.15.29:0/4112337598 >> v1:172.21.15.144:6789/0 conn(0x7fd9f4159b20 legacy=0x7fd9f40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f2d9d700 1 -- 172.21.15.29:0/4112337598 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40cc7c0 con 0x7fd9f4159750 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f2d9d700 1 --2- 172.21.15.29:0/4112337598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159750 0x7fd9f40d10a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009990 tx=0x7fd9e4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8436bae43d4ae0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f1d9b700 1 -- 172.21.15.29:0/4112337598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4009e80 con 0x7fd9f4159750 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f1d9b700 1 -- 172.21.15.29:0/4112337598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4005c10 con 0x7fd9f4159750 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.557+0000 7fd9f1d9b700 1 -- 172.21.15.29:0/4112337598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e400b650 con 0x7fd9f4159750 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 -- 172.21.15.29:0/4112337598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159750 msgr2=0x7fd9f40d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/4112337598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159750 0x7fd9f40d10a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009990 tx=0x7fd9e4009220 comp rx=0 tx=0).stop 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 -- 172.21.15.29:0/4112337598 shutdown_connections 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/4112337598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f40ce8a0 0x7fd9f40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/4112337598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159750 0x7fd9f40d10a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 -- 172.21.15.29:0/4112337598 >> 172.21.15.29:0/4112337598 conn(0x7fd9f4063a10 msgr2=0x7fd9f40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 -- 172.21.15.29:0/4112337598 shutdown_connections 2023-10-07T09:30:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.558+0000 7fd9f91ff700 1 -- 172.21.15.29:0/4112337598 wait complete. 2023-10-07T09:30:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.559+0000 7fd9f91ff700 1 Processor -- start 2023-10-07T09:30:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.559+0000 7fd9f91ff700 1 -- start start 2023-10-07T09:30:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.560+0000 7fd9f91ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9f40ce8a0 0x7fd9f414b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.560+0000 7fd9f91ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f4159750 0x7fd9f4153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.560+0000 7fd9f91ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f4153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.560+0000 7fd9f91ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c7670 con 0x7fd9f4159b20 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.560+0000 7fd9f91ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c7950 con 0x7fd9f40ce8a0 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.560+0000 7fd9f91ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f4158310 con 0x7fd9f4159750 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f4153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f4153fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40588/0 (socket says 172.21.15.29:40588) 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 -- 172.21.15.29:0/3054369812 learned_addr learned my addr 172.21.15.29:0/3054369812 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f2d9d700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9f40ce8a0 0x7fd9f414b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f259c700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f4159750 0x7fd9f4153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 -- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f4159750 msgr2=0x7fd9f4153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f4159750 0x7fd9f4153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 -- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9f40ce8a0 msgr2=0x7fd9f414b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9f40ce8a0 0x7fd9f414b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 -- 172.21.15.29:0/3054369812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4009480 con 0x7fd9f4159b20 2023-10-07T09:30:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.561+0000 7fd9f359e700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f4153fd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009e90 tx=0x7fd9ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.562+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00cb70 con 0x7fd9f4159b20 2023-10-07T09:30:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.562+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec00f040 con 0x7fd9f4159b20 2023-10-07T09:30:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.562+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f41585f0 con 0x7fd9f4159b20 2023-10-07T09:30:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.562+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec014480 con 0x7fd9f4159b20 2023-10-07T09:30:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.562+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f414b6e0 con 0x7fd9f4159b20 2023-10-07T09:30:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.563+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f40c7ac0 con 0x7fd9f4159b20 2023-10-07T09:30:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.564+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd9ec014830 con 0x7fd9f4159b20 2023-10-07T09:30:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.565+0000 7fd9daffd700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd9e0066f30 0x7fd9e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.565+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fd9ec0852f0 con 0x7fd9f4159b20 2023-10-07T09:30:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.565+0000 7fd9f2d9d700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd9e0066f30 0x7fd9e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.566+0000 7fd9f2d9d700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd9e0066f30 0x7fd9e00693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009260 tx=0x7fd9e4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.569+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9ec052450 con 0x7fd9f4159b20 2023-10-07T09:30:04.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.769+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9f404c3c0 con 0x7fd9f4159b20 2023-10-07T09:30:04.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.770+0000 7fd9daffd700 1 -- 172.21.15.29:0/3054369812 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+15636 (secure 0 0 0) 0x7fd9ec055a70 con 0x7fd9f4159b20 2023-10-07T09:30:04.771 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:04.771 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":23,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:04.090627+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd9e0066f30 msgr2=0x7fd9e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd9e0066f30 0x7fd9e00693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009260 tx=0x7fd9e4016000 comp rx=0 tx=0).stop 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 msgr2=0x7fd9f4153fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f4153fd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009e90 tx=0x7fd9ec00b040 comp rx=0 tx=0).stop 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 shutdown_connections 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd9e0066f30 0x7fd9e00693e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f4153fd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9f40ce8a0 0x7fd9f414b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 --2- 172.21.15.29:0/3054369812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9f4159750 0x7fd9f4153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.772+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 >> 172.21.15.29:0/3054369812 conn(0x7fd9f4063a10 msgr2=0x7fd9f40c61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.773+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 shutdown_connections 2023-10-07T09:30:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.773+0000 7fd9f91ff700 1 -- 172.21.15.29:0/3054369812 wait complete. 2023-10-07T09:30:04.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:30:04.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:30:04.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.992+0000 7fdaeefff700 1 Processor -- start 2023-10-07T09:30:04.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.992+0000 7fdaeefff700 1 -- start start 2023-10-07T09:30:04.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdaeefff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159b20 0x7fdae80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:04.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdaeefff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80cbf20 con 0x7fdae80ce8a0 2023-10-07T09:30:04.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdaeefff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80cc200 con 0x7fdae8159750 2023-10-07T09:30:04.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdaeefff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80cc4e0 con 0x7fdae8159b20 2023-10-07T09:30:04.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdaed59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdae80ce8a0 0x7fdae80c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37406/0 (socket says 172.21.15.29:37406) 2023-10-07T09:30:04.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdaed59c700 1 -- 172.21.15.29:0/1237886854 learned_addr learned my addr 172.21.15.29:0/1237886854 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:04.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.994+0000 7fdae7fff700 1 --2- 172.21.15.29:0/1237886854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159b20 0x7fdae80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485743978 0 0) 0x7fdae80cc200 con 0x7fdae8159750 2023-10-07T09:30:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad8003570 con 0x7fdae8159750 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730018412 0 0) 0x7fdae80cbf20 con 0x7fdae80ce8a0 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae80cc200 con 0x7fdae80ce8a0 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae7fff700 1 -- 172.21.15.29:0/1237886854 >> v1:172.21.15.144:6789/0 conn(0x7fdae8159750 legacy=0x7fdae80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae7fff700 1 -- 172.21.15.29:0/1237886854 >> v1:172.21.15.29:6789/0 conn(0x7fdae80ce8a0 legacy=0x7fdae80c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae7fff700 1 -- 172.21.15.29:0/1237886854 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80cc7c0 con 0x7fdae8159b20 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae7fff700 1 --2- 172.21.15.29:0/1237886854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159b20 0x7fdae80d10a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009910 tx=0x7fdad40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1b9c4c1af654c0c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4011630 con 0x7fdae8159b20 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad4011790 con 0x7fdae8159b20 2023-10-07T09:30:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.995+0000 7fdae77fe700 1 -- 172.21.15.29:0/1237886854 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4011b30 con 0x7fdae8159b20 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 -- 172.21.15.29:0/1237886854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159b20 msgr2=0x7fdae80d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 --2- 172.21.15.29:0/1237886854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159b20 0x7fdae80d10a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009910 tx=0x7fdad40091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 -- 172.21.15.29:0/1237886854 shutdown_connections 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 --2- 172.21.15.29:0/1237886854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159b20 0x7fdae80d10a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 -- 172.21.15.29:0/1237886854 >> 172.21.15.29:0/1237886854 conn(0x7fdae8063a10 msgr2=0x7fdae80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 -- 172.21.15.29:0/1237886854 shutdown_connections 2023-10-07T09:30:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.996+0000 7fdaeefff700 1 -- 172.21.15.29:0/1237886854 wait complete. 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.998+0000 7fdaeefff700 1 Processor -- start 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:04.998+0000 7fdaeefff700 1 -- start start 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaeefff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 0x7fdae80c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaeefff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159750 0x7fdae80b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaeefff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdae8159b20 0x7fdae814c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaeefff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae814ca00 con 0x7fdae80ce8a0 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaeefff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae8150ed0 con 0x7fdae8159b20 2023-10-07T09:30:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaeefff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80c8640 con 0x7fdae8159750 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 0x7fdae80c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 0x7fdae80c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40600/0 (socket says 172.21.15.29:40600) 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 -- 172.21.15.29:0/3487586736 learned_addr learned my addr 172.21.15.29:0/3487586736 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdae7fff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159750 0x7fdae80b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaed59c700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdae8159b20 0x7fdae814c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 -- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159750 msgr2=0x7fdae80b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159750 0x7fdae80b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 -- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdae8159b20 msgr2=0x7fdae814c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdae8159b20 0x7fdae814c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.000+0000 7fdaecd9b700 1 -- 172.21.15.29:0/3487586736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad4009420 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.001+0000 7fdaecd9b700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 0x7fdae80c8030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c7c0 tx=0x7fdadc007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.001+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc001530 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.001+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc001690 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.001+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc001a50 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.001+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae8150880 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.001+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae81548a0 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.002+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae81506e0 con 0x7fdae80ce8a0 2023-10-07T09:30:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.003+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdadc01b3b0 con 0x7fdae80ce8a0 2023-10-07T09:30:05.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.004+0000 7fdae4ff9700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdacc066f30 0x7fdacc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.004+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fdadc083e50 con 0x7fdae80ce8a0 2023-10-07T09:30:05.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.004+0000 7fdae7fff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdacc066f30 0x7fdacc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.005+0000 7fdae7fff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdacc066f30 0x7fdacc0693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009860 tx=0x7fdad4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.008+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdae81506e0 con 0x7fdae80ce8a0 2023-10-07T09:30:05.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.218+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdae81506e0 con 0x7fdae80ce8a0 2023-10-07T09:30:05.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.218+0000 7fdae4ff9700 1 -- 172.21.15.29:0/3487586736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdae81506e0 con 0x7fdae80ce8a0 2023-10-07T09:30:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:30:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdacc066f30 msgr2=0x7fdacc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdacc066f30 0x7fdacc0693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009860 tx=0x7fdad4017000 comp rx=0 tx=0).stop 2023-10-07T09:30:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 msgr2=0x7fdae80c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 0x7fdae80c8030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c7c0 tx=0x7fdadc007e20 comp rx=0 tx=0).stop 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 shutdown_connections 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdacc066f30 0x7fdacc0693e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdae8159b20 0x7fdae814c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdae80ce8a0 0x7fdae80c8030 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 --2- 172.21.15.29:0/3487586736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdae8159750 0x7fdae80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 >> 172.21.15.29:0/3487586736 conn(0x7fdae8063a10 msgr2=0x7fdae8155b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 shutdown_connections 2023-10-07T09:30:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.221+0000 7fdaeefff700 1 -- 172.21.15.29:0/3487586736 wait complete. 2023-10-07T09:30:05.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:30:05.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.438+0000 7f4014dff700 1 Processor -- start 2023-10-07T09:30:05.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.439+0000 7f4014dff700 1 -- start start 2023-10-07T09:30:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.440+0000 7f4014dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 0x7f40100d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.440+0000 7f4014dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40100cbf20 con 0x7f4010159b20 2023-10-07T09:30:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.440+0000 7f4014dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40100cc200 con 0x7f40100ce8a0 2023-10-07T09:30:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.440+0000 7f4014dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40100cc4e0 con 0x7f4010159750 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4010159b20 0x7f40100ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37416/0 (socket says 172.21.15.29:37416) 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400dd9b700 1 -- 172.21.15.29:0/2641714936 learned_addr learned my addr 172.21.15.29:0/2641714936 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400e59c700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 0x7f40100d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566158291 0 0) 0x7f40100cbf20 con 0x7f4010159b20 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff8003570 con 0x7f4010159b20 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256713980 0 0) 0x7f40100cc200 con 0x7f40100ce8a0 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40100cbf20 con 0x7f40100ce8a0 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57191654 0 0) 0x7f3ff8003570 con 0x7f4010159b20 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 msgr2=0x7f40100d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 0x7f40100d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 >> v1:172.21.15.144:6789/0 conn(0x7f40100ce8a0 legacy=0x7f40100c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40100cc7c0 con 0x7f4010159b20 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f4004001470 con 0x7f4010159b20 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 >> v1:172.21.15.29:6789/0 conn(0x7f4010159b20 legacy=0x7f40100ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.441+0000 7f400d59a700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 0x7f3ff8003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400d59a700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ff8004180 0x7f3ff8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400d59a700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f3ff80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100cc4e0 con 0x7f3ff8004d50 2023-10-07T09:30:05.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff800bba0 con 0x7f3ff8004180 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff800bd60 con 0x7f4010159750 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f3ff80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400e59c700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 0x7f3ff8003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400dd9b700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ff8004180 0x7f3ff8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 -- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 msgr2=0x7f3ff8003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4010159750 0x7f3ff8003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 -- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ff8004180 msgr2=0x7f3ff8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ff8004180 0x7f3ff8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 -- 172.21.15.29:0/2641714936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8003570 con 0x7f3ff8004d50 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.442+0000 7f400ed9d700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f3ff80051c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4008002120 tx=0x7f40080017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442d9eb18d660057 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008001ab0 con 0x7f3ff8004d50 2023-10-07T09:30:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400800c980 con 0x7f3ff8004d50 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f400d59a700 1 -- 172.21.15.29:0/2641714936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400800cd20 con 0x7f3ff8004d50 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f4014dff700 1 -- 172.21.15.29:0/2641714936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 msgr2=0x7f3ff80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f4014dff700 1 --2- 172.21.15.29:0/2641714936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f3ff80051c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4008002120 tx=0x7f40080017a0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f400ed9d700 1 -- 172.21.15.29:0/2641714936 reap_dead start 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f4014dff700 1 -- 172.21.15.29:0/2641714936 shutdown_connections 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f4014dff700 1 -- 172.21.15.29:0/2641714936 >> 172.21.15.29:0/2641714936 conn(0x7f4010063a10 msgr2=0x7f40100d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f4014dff700 1 -- 172.21.15.29:0/2641714936 shutdown_connections 2023-10-07T09:30:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.443+0000 7f4014dff700 1 -- 172.21.15.29:0/2641714936 wait complete. 2023-10-07T09:30:05.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.445+0000 7f4014dff700 1 Processor -- start 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.445+0000 7f4014dff700 1 -- start start 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f4014dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f4010141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f4014dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401009c6d0 0x7f401009cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f4014dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40100c5360 0x7f40100c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f4014dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100c5d40 con 0x7f3ff8004d50 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f4014dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4010153520 con 0x7f401009c6d0 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f4014dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401009dda0 con 0x7f40100c5360 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f400e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f4010141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f400e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f4010141960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40620/0 (socket says 172.21.15.29:40620) 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f400e59c700 1 -- 172.21.15.29:0/2199976943 learned_addr learned my addr 172.21.15.29:0/2199976943 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f400dd9b700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401009c6d0 0x7f401009cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.446+0000 7f400ed9d700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40100c5360 0x7f40100c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400e59c700 1 -- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40100c5360 msgr2=0x7f40100c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400e59c700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40100c5360 0x7f40100c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400e59c700 1 -- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401009c6d0 msgr2=0x7f401009cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400e59c700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401009c6d0 0x7f401009cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400e59c700 1 -- 172.21.15.29:0/2199976943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401009d190 con 0x7f3ff8004d50 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400dd9b700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401009c6d0 0x7f401009cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f400e59c700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f4010141960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4000011b40 tx=0x7f4000009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4000005c10 con 0x7f3ff8004d50 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4000005d70 con 0x7f3ff8004d50 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400000fa90 con 0x7f3ff8004d50 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401009d470 con 0x7f3ff8004d50 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.447+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401009da30 con 0x7f3ff8004d50 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.449+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401004c3c0 con 0x7f3ff8004d50 2023-10-07T09:30:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.449+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4000006050 con 0x7f3ff8004d50 2023-10-07T09:30:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.450+0000 7f3ffe7fc700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ff0066f30 0x7f3ff00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.450+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f4000085000 con 0x7f3ff8004d50 2023-10-07T09:30:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.451+0000 7f400dd9b700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ff0066f30 0x7f3ff00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.452+0000 7f400dd9b700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ff0066f30 0x7f3ff00693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4004001c80 tx=0x7f4004009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.454+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4000052200 con 0x7f3ff8004d50 2023-10-07T09:30:05.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.658+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4010064040 con 0x7f3ff8004d50 2023-10-07T09:30:05.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.658+0000 7f3ffe7fc700 1 -- 172.21.15.29:0/2199976943 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4000055820 con 0x7f3ff8004d50 2023-10-07T09:30:05.660 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:30:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ff0066f30 msgr2=0x7f3ff00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ff0066f30 0x7f3ff00693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4004001c80 tx=0x7f4004009040 comp rx=0 tx=0).stop 2023-10-07T09:30:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 msgr2=0x7f4010141960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f4010141960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4000011b40 tx=0x7f4000009120 comp rx=0 tx=0).stop 2023-10-07T09:30:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 shutdown_connections 2023-10-07T09:30:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ff0066f30 0x7f3ff00693e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40100c5360 0x7f40100c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff8004d50 0x7f4010141960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 --2- 172.21.15.29:0/2199976943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401009c6d0 0x7f401009cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 >> 172.21.15.29:0/2199976943 conn(0x7f4010063a10 msgr2=0x7f4010068440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 shutdown_connections 2023-10-07T09:30:05.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.661+0000 7f4014dff700 1 -- 172.21.15.29:0/2199976943 wait complete. 2023-10-07T09:30:05.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:30:05.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.876+0000 7f649ddff700 1 Processor -- start 2023-10-07T09:30:05.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.876+0000 7f649ddff700 1 -- start start 2023-10-07T09:30:05.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.877+0000 7f649ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6498156180 0x7f64980afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.877+0000 7f649ddff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64980b0520 0x7f64980b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f649ddff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64980ad4d0 con 0x7f64980ae570 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f649ddff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64980b0ee0 con 0x7f64980b0520 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f649ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64980ad7b0 con 0x7f6498156180 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64980b0520 0x7f64980b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64980b0520 0x7f64980b09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40624/0 (socket says 172.21.15.29:40624) 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 -- 172.21.15.29:0/3205970788 learned_addr learned my addr 172.21.15.29:0/3205970788 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f64977fe700 1 --2- 172.21.15.29:0/3205970788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6498156180 0x7f64980afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 -- 172.21.15.29:0/3205970788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6498156180 msgr2=0x7f64980afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 --2- 172.21.15.29:0/3205970788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6498156180 0x7f64980afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 -- 172.21.15.29:0/3205970788 >> v1:172.21.15.144:6789/0 conn(0x7f64980ae570 legacy=0x7f64980accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.878+0000 7f6497fff700 1 -- 172.21.15.29:0/3205970788 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64980ada90 con 0x7f64980b0520 2023-10-07T09:30:05.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.879+0000 7f6497fff700 1 --2- 172.21.15.29:0/3205970788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64980b0520 0x7f64980b09a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f649000d1a0 tx=0x7f649000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db1ae7821a488db server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.879+0000 7f64967fc700 1 -- 172.21.15.29:0/3205970788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490010070 con 0x7f64980b0520 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.879+0000 7f64967fc700 1 -- 172.21.15.29:0/3205970788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649000b0d0 con 0x7f64980b0520 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.879+0000 7f64967fc700 1 -- 172.21.15.29:0/3205970788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649000b490 con 0x7f64980b0520 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.879+0000 7f649ddff700 1 -- 172.21.15.29:0/3205970788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64980b0520 msgr2=0x7f64980b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.879+0000 7f649ddff700 1 --2- 172.21.15.29:0/3205970788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64980b0520 0x7f64980b09a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f649000d1a0 tx=0x7f649000ba50 comp rx=0 tx=0).stop 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.880+0000 7f649ddff700 1 -- 172.21.15.29:0/3205970788 shutdown_connections 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.880+0000 7f649ddff700 1 --2- 172.21.15.29:0/3205970788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64980b0520 0x7f64980b09a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.880+0000 7f649ddff700 1 --2- 172.21.15.29:0/3205970788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6498156180 0x7f64980afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.880+0000 7f649ddff700 1 -- 172.21.15.29:0/3205970788 >> 172.21.15.29:0/3205970788 conn(0x7f6498067610 msgr2=0x7f6498067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.880+0000 7f649ddff700 1 -- 172.21.15.29:0/3205970788 shutdown_connections 2023-10-07T09:30:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.880+0000 7f649ddff700 1 -- 172.21.15.29:0/3205970788 wait complete. 2023-10-07T09:30:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.881+0000 7f649ddff700 1 Processor -- start 2023-10-07T09:30:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.881+0000 7f649ddff700 1 -- start start 2023-10-07T09:30:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f649ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64980ae570 0x7f649814ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f649ddff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64980b0520 0x7f64981548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f649ddff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 0x7f649814def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f649ddff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64980cbe30 con 0x7f6498156180 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f649ddff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64980c5640 con 0x7f64980b0520 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f649ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64980c5920 con 0x7f64980ae570 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.882+0000 7f6497fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 0x7f649814def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 0x7f649814def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40632/0 (socket says 172.21.15.29:40632) 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 -- 172.21.15.29:0/1987211180 learned_addr learned my addr 172.21.15.29:0/1987211180 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f64977fe700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64980ae570 0x7f649814ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6496ffd700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64980b0520 0x7f64981548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 -- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64980ae570 msgr2=0x7f649814ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64980ae570 0x7f649814ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 -- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64980b0520 msgr2=0x7f64981548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64980b0520 0x7f64981548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 -- 172.21.15.29:0/1987211180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649000bcf0 con 0x7f6498156180 2023-10-07T09:30:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f64977fe700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64980ae570 0x7f649814ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.883+0000 7f6497fff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 0x7f649814def0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64980641e0 tx=0x7f649001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.884+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649000acf0 con 0x7f6498156180 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.884+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649000ae90 con 0x7f6498156180 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.884+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649001e690 con 0x7f6498156180 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.884+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64980c5c00 con 0x7f6498156180 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.884+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64980c61c0 con 0x7f6498156180 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.885+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649804c3c0 con 0x7f6498156180 2023-10-07T09:30:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.886+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6490003040 con 0x7f6498156180 2023-10-07T09:30:05.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.886+0000 7f647f7fe700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6484066f30 0x7f64840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:05.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.887+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f6490086420 con 0x7f6498156180 2023-10-07T09:30:05.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.887+0000 7f64977fe700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6484066f30 0x7f64840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:05.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.887+0000 7f64977fe700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6484066f30 0x7f64840693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6488006fd0 tx=0x7f6488008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:05.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:05.891+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6490053580 con 0x7f6498156180 2023-10-07T09:30:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.094+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64980add70 con 0x7f6498156180 2023-10-07T09:30:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.095+0000 7f647f7fe700 1 -- 172.21.15.29:0/1987211180 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6490056ba0 con 0x7f6498156180 2023-10-07T09:30:06.096 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6484066f30 msgr2=0x7f64840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6484066f30 0x7f64840693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6488006fd0 tx=0x7f6488008040 comp rx=0 tx=0).stop 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 msgr2=0x7f649814def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 0x7f649814def0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64980641e0 tx=0x7f649001a000 comp rx=0 tx=0).stop 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 shutdown_connections 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6484066f30 0x7f64840693e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6498156180 0x7f649814def0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64980ae570 0x7f649814ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 --2- 172.21.15.29:0/1987211180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64980b0520 0x7f64981548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 >> 172.21.15.29:0/1987211180 conn(0x7f6498067610 msgr2=0x7f64980aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 shutdown_connections 2023-10-07T09:30:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.097+0000 7f649ddff700 1 -- 172.21.15.29:0/1987211180 wait complete. 2023-10-07T09:30:06.126 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-07T09:30:06.127 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:30:06.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-07T09:30:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.321+0000 7f47a63ff700 1 Processor -- start 2023-10-07T09:30:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.321+0000 7f47a63ff700 1 -- start start 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a63ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47a015a340 0x7f47a00afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a63ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a014d790 con 0x7f47a00ad370 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a63ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00ae5d0 con 0x7f47a0159ec0 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a63ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00ad740 con 0x7f47a015a340 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a499c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47a015a340 0x7f47a00afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f479ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f47a00ad370 0x7f47a00cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37422/0 (socket says 172.21.15.29:37422) 2023-10-07T09:30:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f479ffff700 1 -- 172.21.15.29:0/3588892277 learned_addr learned my addr 172.21.15.29:0/3588892277 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a499c700 1 -- 172.21.15.29:0/3588892277 >> v1:172.21.15.144:6789/0 conn(0x7f47a0159ec0 legacy=0x7f47a00af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a499c700 1 -- 172.21.15.29:0/3588892277 >> v1:172.21.15.29:6789/0 conn(0x7f47a00ad370 legacy=0x7f47a00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.323+0000 7f47a499c700 1 -- 172.21.15.29:0/3588892277 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00ad9c0 con 0x7f47a015a340 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.324+0000 7f47a499c700 1 --2- 172.21.15.29:0/3588892277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a015a340 0x7f47a00afb10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4798009e60 tx=0x7f4798009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837f07bf83b18e48 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.324+0000 7f479effd700 1 -- 172.21.15.29:0/3588892277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479800e070 con 0x7f47a015a340 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.324+0000 7f479effd700 1 -- 172.21.15.29:0/3588892277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479800bc30 con 0x7f47a015a340 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.324+0000 7f479effd700 1 -- 172.21.15.29:0/3588892277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798012430 con 0x7f47a015a340 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.324+0000 7f47a63ff700 1 -- 172.21.15.29:0/3588892277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a015a340 msgr2=0x7f47a00afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.324+0000 7f47a63ff700 1 --2- 172.21.15.29:0/3588892277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a015a340 0x7f47a00afb10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4798009e60 tx=0x7f4798009780 comp rx=0 tx=0).stop 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.325+0000 7f47a63ff700 1 -- 172.21.15.29:0/3588892277 shutdown_connections 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.325+0000 7f47a63ff700 1 --2- 172.21.15.29:0/3588892277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a015a340 0x7f47a00afb10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.325+0000 7f47a63ff700 1 -- 172.21.15.29:0/3588892277 >> 172.21.15.29:0/3588892277 conn(0x7f47a00680a0 msgr2=0x7f47a00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.325+0000 7f47a63ff700 1 -- 172.21.15.29:0/3588892277 shutdown_connections 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.325+0000 7f47a63ff700 1 -- 172.21.15.29:0/3588892277 wait complete. 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.326+0000 7f47a63ff700 1 Processor -- start 2023-10-07T09:30:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.326+0000 7f47a63ff700 1 -- start start 2023-10-07T09:30:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f47a63ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 0x7f47a00c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f47a63ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47a0159ec0 0x7f47a01692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f47a63ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47a015a340 0x7f47a0169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f47a63ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a016aef0 con 0x7f47a00ad370 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f47a63ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00ca4a0 con 0x7f47a015a340 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f47a63ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a016a550 con 0x7f47a0159ec0 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f479ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 0x7f47a00c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f479ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 0x7f47a00c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40652/0 (socket says 172.21.15.29:40652) 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.327+0000 7f479ffff700 1 -- 172.21.15.29:0/565711940 learned_addr learned my addr 172.21.15.29:0/565711940 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f47a499c700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47a015a340 0x7f47a0169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479ffff700 1 -- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47a0159ec0 msgr2=0x7f47a01692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479f7fe700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47a0159ec0 0x7f47a01692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479ffff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47a0159ec0 0x7f47a01692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479ffff700 1 -- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47a015a340 msgr2=0x7f47a0169820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479ffff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47a015a340 0x7f47a0169820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479ffff700 1 -- 172.21.15.29:0/565711940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009a20 con 0x7f47a00ad370 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f47a499c700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47a015a340 0x7f47a0169820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479f7fe700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47a0159ec0 0x7f47a01692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f479ffff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 0x7f47a00c9b50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f479000c7c0 tx=0x7f4790007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790005b30 con 0x7f47a00ad370 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790005cb0 con 0x7f47a00ad370 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790004a80 con 0x7f47a00ad370 2023-10-07T09:30:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.328+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a016a830 con 0x7f47a00ad370 2023-10-07T09:30:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.329+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a00ca090 con 0x7f47a00ad370 2023-10-07T09:30:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.330+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a004c3c0 con 0x7f47a00ad370 2023-10-07T09:30:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.330+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f47900070f0 con 0x7f47a00ad370 2023-10-07T09:30:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.331+0000 7f4787fff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f478c066f30 0x7f478c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.331+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f4790083ac0 con 0x7f47a00ad370 2023-10-07T09:30:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.332+0000 7f479f7fe700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f478c066f30 0x7f478c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.332+0000 7f479f7fe700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f478c066f30 0x7f478c0693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f47a00c85c0 tx=0x7f4794009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.335+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4790050ca0 con 0x7f47a00ad370 2023-10-07T09:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:06.615+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f47a00c85f0 con 0x7f47a00ad370 2023-10-07T09:30:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.110+0000 7f4787fff700 1 -- 172.21.15.29:0/565711940 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v64) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4790018070 con 0x7f47a00ad370 2023-10-07T09:30:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f478c066f30 msgr2=0x7f478c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f478c066f30 0x7f478c0693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f47a00c85c0 tx=0x7f4794009040 comp rx=0 tx=0).stop 2023-10-07T09:30:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 msgr2=0x7f47a00c9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 0x7f47a00c9b50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f479000c7c0 tx=0x7f4790007f70 comp rx=0 tx=0).stop 2023-10-07T09:30:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 shutdown_connections 2023-10-07T09:30:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f478c066f30 0x7f478c0693e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47a015a340 0x7f47a0169820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47a00ad370 0x7f47a00c9b50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 --2- 172.21.15.29:0/565711940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47a0159ec0 0x7f47a01692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 >> 172.21.15.29:0/565711940 conn(0x7f47a00680a0 msgr2=0x7f47a00ac4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 shutdown_connections 2023-10-07T09:30:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.113+0000 7f47a63ff700 1 -- 172.21.15.29:0/565711940 wait complete. 2023-10-07T09:30:07.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:30:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.335+0000 7f800fbff700 1 Processor -- start 2023-10-07T09:30:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.335+0000 7f800fbff700 1 -- start start 2023-10-07T09:30:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.336+0000 7f800fbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f800815ade0 0x7f80080b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.336+0000 7f800fbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8008154090 con 0x7f80080b0540 2023-10-07T09:30:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.336+0000 7f800fbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8008159f60 con 0x7f8008153cc0 2023-10-07T09:30:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.336+0000 7f800fbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080b09e0 con 0x7f800815ade0 2023-10-07T09:30:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.336+0000 7f800e19c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f80080b0540 0x7f800815c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37426/0 (socket says 172.21.15.29:37426) 2023-10-07T09:30:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.336+0000 7f800e19c700 1 -- 172.21.15.29:0/3470218348 learned_addr learned my addr 172.21.15.29:0/3470218348 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800d19a700 1 --2- 172.21.15.29:0/3470218348 >> v2:172.21.15.144:3300/0 conn(0x7f800815ade0 0x7f80080b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641951328 0 0) 0x7f8008154090 con 0x7f80080b0540 2023-10-07T09:30:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff4003570 con 0x7f80080b0540 2023-10-07T09:30:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513491562 0 0) 0x7f8008159f60 con 0x7f8008153cc0 2023-10-07T09:30:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8008154090 con 0x7f8008153cc0 2023-10-07T09:30:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1648485033 0 0) 0x7f7ff4003570 con 0x7f80080b0540 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 >> v2:172.21.15.144:3300/0 conn(0x7f800815ade0 msgr2=0x7f80080b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 --2- 172.21.15.29:0/3470218348 >> v2:172.21.15.144:3300/0 conn(0x7f800815ade0 0x7f80080b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 >> v1:172.21.15.144:6789/0 conn(0x7f8008153cc0 legacy=0x7f80080b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800815a240 con 0x7f80080b0540 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f80000017f0 con 0x7f80080b0540 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 >> v1:172.21.15.29:6789/0 conn(0x7f80080b0540 legacy=0x7f800815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ff4003bf0 0x7f7ff4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff4004810 0x7f7ff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff40051c0 0x7f7ff400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080b09e0 con 0x7f7ff40051c0 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff400c0b0 con 0x7f7ff4004810 2023-10-07T09:30:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.337+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff400c270 con 0x7f7ff4003bf0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff40051c0 0x7f7ff400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800d99b700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ff4003bf0 0x7f7ff4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800d19a700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff4004810 0x7f7ff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 -- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ff4003bf0 msgr2=0x7f7ff4003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ff4003bf0 0x7f7ff4003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 -- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff4004810 msgr2=0x7f7ff4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff4004810 0x7f7ff4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 -- 172.21.15.29:0/3470218348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff4003570 con 0x7f7ff40051c0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 -- 172.21.15.29:0/3470218348 reap_dead start 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.338+0000 7f800e19c700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff40051c0 0x7f7ff400ba60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f800000a8d0 tx=0x7f8000002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5165186a4429ed server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000070d0 con 0x7f7ff40051c0 2023-10-07T09:30:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8000007270 con 0x7f7ff40051c0 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800c999700 1 -- 172.21.15.29:0/3470218348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000007630 con 0x7f7ff40051c0 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 -- 172.21.15.29:0/3470218348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff40051c0 msgr2=0x7f7ff400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff40051c0 0x7f7ff400ba60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f800000a8d0 tx=0x7f8000002190 comp rx=0 tx=0).stop 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 -- 172.21.15.29:0/3470218348 shutdown_connections 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 --2- 172.21.15.29:0/3470218348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff40051c0 0x7f7ff400ba60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 -- 172.21.15.29:0/3470218348 >> 172.21.15.29:0/3470218348 conn(0x7f8008067610 msgr2=0x7f8008067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 -- 172.21.15.29:0/3470218348 shutdown_connections 2023-10-07T09:30:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.339+0000 7f800fbff700 1 -- 172.21.15.29:0/3470218348 wait complete. 2023-10-07T09:30:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.341+0000 7f800fbff700 1 Processor -- start 2023-10-07T09:30:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.341+0000 7f800fbff700 1 -- start start 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800fbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff40051c0 0x7f800814eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800fbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f800816d5e0 0x7f8008155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800fbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 0x7f80080c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800fbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080c77c0 con 0x7f8008146f20 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800fbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008158af0 con 0x7f7ff40051c0 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800fbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8008158dd0 con 0x7f800816d5e0 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800e19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 0x7f80080c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800e19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 0x7f80080c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40678/0 (socket says 172.21.15.29:40678) 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800e19c700 1 -- 172.21.15.29:0/1895827269 learned_addr learned my addr 172.21.15.29:0/1895827269 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800d99b700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff40051c0 0x7f800814eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.342+0000 7f800d19a700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f800816d5e0 0x7f8008155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800e19c700 1 -- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f800816d5e0 msgr2=0x7f8008155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800e19c700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f800816d5e0 0x7f8008155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800e19c700 1 -- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff40051c0 msgr2=0x7f800814eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800e19c700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff40051c0 0x7f800814eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800e19c700 1 -- 172.21.15.29:0/1895827269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008147390 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800e19c700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 0x7f80080c92e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8008068230 tx=0x7f8000016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000004dc0 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800001d420 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800001d7e0 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800814f5a0 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.343+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80080c99e0 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.344+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80080c7930 con 0x7f8008146f20 2023-10-07T09:30:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.345+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f800001db90 con 0x7f8008146f20 2023-10-07T09:30:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.346+0000 7f7ffdffb700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7fec066f30 0x7f7fec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.346+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f8000085f50 con 0x7f8008146f20 2023-10-07T09:30:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.347+0000 7f800d99b700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7fec066f30 0x7f7fec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.347+0000 7f800d99b700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7fec066f30 0x7f7fec0693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8004001130 tx=0x7f8004007d80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:07.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.350+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80000530b0 con 0x7f8008146f20 2023-10-07T09:30:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.551+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f800814f790 con 0x7f8008146f20 2023-10-07T09:30:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.555+0000 7f7ffdffb700 1 -- 172.21.15.29:0/1895827269 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f800814f790 con 0x7f8008146f20 2023-10-07T09:30:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7fec066f30 msgr2=0x7f7fec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7fec066f30 0x7f7fec0693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8004001130 tx=0x7f8004007d80 comp rx=0 tx=0).stop 2023-10-07T09:30:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 msgr2=0x7f80080c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 0x7f80080c92e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8008068230 tx=0x7f8000016000 comp rx=0 tx=0).stop 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 shutdown_connections 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7fec066f30 0x7f7fec0693e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8008146f20 0x7f80080c92e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ff40051c0 0x7f800814eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 --2- 172.21.15.29:0/1895827269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f800816d5e0 0x7f8008155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 >> 172.21.15.29:0/1895827269 conn(0x7f8008067610 msgr2=0x7f80080af000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 shutdown_connections 2023-10-07T09:30:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:07.558+0000 7f800fbff700 1 -- 172.21.15.29:0/1895827269 wait complete. 2023-10-07T09:30:07.577 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:30:07.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:30:07.671 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:30:07.785 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:30:07.785 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:30:07.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:07.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-07T09:30:07.874 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:30:07.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:07.874 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:30:07.926 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-07T09:30:07.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:07.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-07T09:30:08.017 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:30:08.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:08.017 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-07T09:30:08.074 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-07T09:30:08.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:08.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-07T09:30:08.161 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:30:08.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:08.161 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-07T09:30:08.207 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-07T09:30:08.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:08.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-07T09:30:08.304 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T09:30:08.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:08.305 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-07T09:30:08.351 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:30:08.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:30:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.545+0000 7f094c7ff700 1 Processor -- start 2023-10-07T09:30:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.545+0000 7f094c7ff700 1 -- start start 2023-10-07T09:30:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f09440cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094c7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0944159ec0 0x7f094415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094c7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09441568c0 0x7f0944156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094c7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094414d790 con 0x7f09441568c0 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094c7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09440ae5d0 con 0x7f0944159ec0 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094415a850 con 0x7f09440ad370 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094ad9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09441568c0 0x7f0944156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094ad9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09441568c0 0x7f0944156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40694/0 (socket says 172.21.15.29:40694) 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094ad9c700 1 -- 172.21.15.29:0/2079075577 learned_addr learned my addr 172.21.15.29:0/2079075577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f0949d9a700 1 --2- 172.21.15.29:0/2079075577 >> v2:172.21.15.144:3300/0 conn(0x7f0944159ec0 0x7f094415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.547+0000 7f094a59b700 1 --2- 172.21.15.29:0/2079075577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f09440cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f094ad9c700 1 -- 172.21.15.29:0/2079075577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 msgr2=0x7f09440cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f094ad9c700 1 --2- 172.21.15.29:0/2079075577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f09440cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f094ad9c700 1 -- 172.21.15.29:0/2079075577 >> v2:172.21.15.144:3300/0 conn(0x7f0944159ec0 msgr2=0x7f094415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f094ad9c700 1 --2- 172.21.15.29:0/2079075577 >> v2:172.21.15.144:3300/0 conn(0x7f0944159ec0 0x7f094415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f094ad9c700 1 -- 172.21.15.29:0/2079075577 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440ad740 con 0x7f09441568c0 2023-10-07T09:30:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f094ad9c700 1 --2- 172.21.15.29:0/2079075577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09441568c0 0x7f0944156d00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0944063d00 tx=0x7f093c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262306cd729bca52 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f0949599700 1 -- 172.21.15.29:0/2079075577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c00e070 con 0x7f09441568c0 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f0949599700 1 -- 172.21.15.29:0/2079075577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00b970 con 0x7f09441568c0 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.548+0000 7f0949599700 1 -- 172.21.15.29:0/2079075577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c0092e0 con 0x7f09441568c0 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 -- 172.21.15.29:0/2079075577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09441568c0 msgr2=0x7f0944156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 --2- 172.21.15.29:0/2079075577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09441568c0 0x7f0944156d00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0944063d00 tx=0x7f093c009840 comp rx=0 tx=0).stop 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 -- 172.21.15.29:0/2079075577 shutdown_connections 2023-10-07T09:30:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 --2- 172.21.15.29:0/2079075577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09441568c0 0x7f0944156d00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 --2- 172.21.15.29:0/2079075577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f09440cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 --2- 172.21.15.29:0/2079075577 >> v2:172.21.15.144:3300/0 conn(0x7f0944159ec0 0x7f094415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 -- 172.21.15.29:0/2079075577 >> 172.21.15.29:0/2079075577 conn(0x7f09440680a0 msgr2=0x7f09440cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 -- 172.21.15.29:0/2079075577 shutdown_connections 2023-10-07T09:30:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.549+0000 7f094c7ff700 1 -- 172.21.15.29:0/2079075577 wait complete. 2023-10-07T09:30:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.550+0000 7f094c7ff700 1 Processor -- start 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 -- start start 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f0944146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09441568c0 0x7f09440a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 0x7f09440a71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440aa2d0 con 0x7f0944159ec0 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440a6620 con 0x7f09441568c0 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.551+0000 7f094c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09440a9c80 con 0x7f09440ad370 2023-10-07T09:30:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 0x7f09440a71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 0x7f09440a71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40708/0 (socket says 172.21.15.29:40708) 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 -- 172.21.15.29:0/1536951835 learned_addr learned my addr 172.21.15.29:0/1536951835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094a59b700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f0944146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f0949d9a700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09441568c0 0x7f09440a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 -- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 msgr2=0x7f0944146d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f0944146d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 -- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09441568c0 msgr2=0x7f09440a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09441568c0 0x7f09440a5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 -- 172.21.15.29:0/1536951835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c009ae0 con 0x7f0944159ec0 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.552+0000 7f094ad9c700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 0x7f09440a71c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f093c009880 tx=0x7f093c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.553+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c008bc0 con 0x7f0944159ec0 2023-10-07T09:30:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.553+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c008d40 con 0x7f0944159ec0 2023-10-07T09:30:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.553+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c0266b0 con 0x7f0944159ec0 2023-10-07T09:30:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.553+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09440a9f60 con 0x7f0944159ec0 2023-10-07T09:30:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.553+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09440c9fa0 con 0x7f0944159ec0 2023-10-07T09:30:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.554+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09440aa480 con 0x7f0944159ec0 2023-10-07T09:30:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.555+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f093c003040 con 0x7f0944159ec0 2023-10-07T09:30:08.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.556+0000 7f093a7fc700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0928066f30 0x7f09280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:08.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.556+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f093c0864e0 con 0x7f0944159ec0 2023-10-07T09:30:08.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.556+0000 7f094a59b700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0928066f30 0x7f09280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:08.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.557+0000 7f094a59b700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0928066f30 0x7f09280693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d40 tx=0x7f0940008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:08.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.561+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f093c053790 con 0x7f0944159ec0 2023-10-07T09:30:08.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.805+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f094404c3c0 con 0x7f0944159ec0 2023-10-07T09:30:08.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.806+0000 7f093a7fc700 1 -- 172.21.15.29:0/1536951835 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f093c056db0 con 0x7f0944159ec0 2023-10-07T09:30:08.807 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:08.807 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0928066f30 msgr2=0x7f09280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0928066f30 0x7f09280693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d40 tx=0x7f0940008040 comp rx=0 tx=0).stop 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 msgr2=0x7f09440a71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 0x7f09440a71c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f093c009880 tx=0x7f093c018000 comp rx=0 tx=0).stop 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 shutdown_connections 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0928066f30 0x7f09280693e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0944159ec0 0x7f09440a71c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09440ad370 0x7f0944146d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 --2- 172.21.15.29:0/1536951835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09441568c0 0x7f09440a5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.808+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 >> 172.21.15.29:0/1536951835 conn(0x7f09440680a0 msgr2=0x7f09441592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.809+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 shutdown_connections 2023-10-07T09:30:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:08.809+0000 7f094c7ff700 1 -- 172.21.15.29:0/1536951835 wait complete. 2023-10-07T09:30:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:30:08.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:09.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.021+0000 7ff8353ff700 1 Processor -- start 2023-10-07T09:30:09.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.021+0000 7ff8353ff700 1 -- start start 2023-10-07T09:30:09.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff8353ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8300ad370 0x7ff8300cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff8353ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff830159ec0 0x7ff83015a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff8353ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8301568c0 0x7ff830156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff8353ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83014d790 con 0x7ff830159ec0 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff8353ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8300ae5d0 con 0x7ff8301568c0 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff8353ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83015a850 con 0x7ff8300ad370 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff830159ec0 0x7ff83015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff830159ec0 0x7ff83015a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40716/0 (socket says 172.21.15.29:40716) 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82e7fc700 1 -- 172.21.15.29:0/2350822571 learned_addr learned my addr 172.21.15.29:0/2350822571 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82effd700 1 --2- 172.21.15.29:0/2350822571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8300ad370 0x7ff8300cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82f7fe700 1 --2- 172.21.15.29:0/2350822571 >> v2:172.21.15.144:3300/0 conn(0x7ff8301568c0 0x7ff830156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82e7fc700 1 -- 172.21.15.29:0/2350822571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8300ad370 msgr2=0x7ff8300cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82e7fc700 1 --2- 172.21.15.29:0/2350822571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8300ad370 0x7ff8300cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.023+0000 7ff82e7fc700 1 -- 172.21.15.29:0/2350822571 >> v2:172.21.15.144:3300/0 conn(0x7ff8301568c0 msgr2=0x7ff830156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82e7fc700 1 --2- 172.21.15.29:0/2350822571 >> v2:172.21.15.144:3300/0 conn(0x7ff8301568c0 0x7ff830156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82e7fc700 1 -- 172.21.15.29:0/2350822571 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8300ad740 con 0x7ff830159ec0 2023-10-07T09:30:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82f7fe700 1 --2- 172.21.15.29:0/2350822571 >> v2:172.21.15.144:3300/0 conn(0x7ff8301568c0 0x7ff830156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82e7fc700 1 --2- 172.21.15.29:0/2350822571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff830159ec0 0x7ff83015a310 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff83006a6a0 tx=0x7ff824009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4773be8ae1f7436c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82dffb700 1 -- 172.21.15.29:0/2350822571 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82400b860 con 0x7ff830159ec0 2023-10-07T09:30:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82dffb700 1 -- 172.21.15.29:0/2350822571 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82400b9c0 con 0x7ff830159ec0 2023-10-07T09:30:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.024+0000 7ff82dffb700 1 -- 172.21.15.29:0/2350822571 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82400bd60 con 0x7ff830159ec0 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 -- 172.21.15.29:0/2350822571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff830159ec0 msgr2=0x7ff83015a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 --2- 172.21.15.29:0/2350822571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff830159ec0 0x7ff83015a310 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff83006a6a0 tx=0x7ff824009250 comp rx=0 tx=0).stop 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 -- 172.21.15.29:0/2350822571 shutdown_connections 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 --2- 172.21.15.29:0/2350822571 >> v2:172.21.15.144:3300/0 conn(0x7ff8301568c0 0x7ff830156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 --2- 172.21.15.29:0/2350822571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8300ad370 0x7ff8300cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 --2- 172.21.15.29:0/2350822571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff830159ec0 0x7ff83015a310 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 -- 172.21.15.29:0/2350822571 >> 172.21.15.29:0/2350822571 conn(0x7ff8300680a0 msgr2=0x7ff8300cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 -- 172.21.15.29:0/2350822571 shutdown_connections 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.025+0000 7ff8353ff700 1 -- 172.21.15.29:0/2350822571 wait complete. 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.026+0000 7ff8353ff700 1 Processor -- start 2023-10-07T09:30:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.026+0000 7ff8353ff700 1 -- start start 2023-10-07T09:30:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.027+0000 7ff8353ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 0x7ff83016a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.027+0000 7ff8353ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8301568c0 0x7ff8300c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.027+0000 7ff8353ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff830159ec0 0x7ff8300aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.027+0000 7ff8353ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300ca490 con 0x7ff8300ad370 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.027+0000 7ff8353ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300ca730 con 0x7ff830159ec0 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.027+0000 7ff8353ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300caa10 con 0x7ff8301568c0 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 0x7ff83016a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 0x7ff83016a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40732/0 (socket says 172.21.15.29:40732) 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 -- 172.21.15.29:0/366361583 learned_addr learned my addr 172.21.15.29:0/366361583 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82e7fc700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8301568c0 0x7ff8300c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82f7fe700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff830159ec0 0x7ff8300aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 -- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8301568c0 msgr2=0x7ff8300c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8301568c0 0x7ff8300c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 -- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff830159ec0 msgr2=0x7ff8300aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff830159ec0 0x7ff8300aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 -- 172.21.15.29:0/366361583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240094f0 con 0x7ff8300ad370 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82f7fe700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff830159ec0 0x7ff8300aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff82effd700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 0x7ff83016a1b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff82000d890 tx=0x7ff820008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.028+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82000b600 con 0x7ff8300ad370 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.029+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82000b760 con 0x7ff8300ad370 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.029+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8300cacf0 con 0x7ff8300ad370 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.029+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82000bae0 con 0x7ff8300ad370 2023-10-07T09:30:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.029+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8300cb2b0 con 0x7ff8300ad370 2023-10-07T09:30:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.030+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8300cb470 con 0x7ff8300ad370 2023-10-07T09:30:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.031+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff82001e460 con 0x7ff8300ad370 2023-10-07T09:30:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.032+0000 7ff816ffd700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff81c066f30 0x7ff81c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.032+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7ff820084cf0 con 0x7ff8300ad370 2023-10-07T09:30:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.032+0000 7ff82e7fc700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff81c066f30 0x7ff81c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.033+0000 7ff82e7fc700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff81c066f30 0x7ff81c0693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff83006a6a0 tx=0x7ff824017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.036+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff820051e50 con 0x7ff8300ad370 2023-10-07T09:30:09.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.243+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8300cb620 con 0x7ff8300ad370 2023-10-07T09:30:09.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.244+0000 7ff816ffd700 1 -- 172.21.15.29:0/366361583 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+15636 (secure 0 0 0) 0x7ff820055470 con 0x7ff8300ad370 2023-10-07T09:30:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:09.246 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:05.095626+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:09.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff81c066f30 msgr2=0x7ff81c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff81c066f30 0x7ff81c0693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff83006a6a0 tx=0x7ff824017000 comp rx=0 tx=0).stop 2023-10-07T09:30:09.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 msgr2=0x7ff83016a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 0x7ff83016a1b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff82000d890 tx=0x7ff820008040 comp rx=0 tx=0).stop 2023-10-07T09:30:09.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 shutdown_connections 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff81c066f30 0x7ff81c0693e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff830159ec0 0x7ff8300aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8300ad370 0x7ff83016a1b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 --2- 172.21.15.29:0/366361583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8301568c0 0x7ff8300c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 >> 172.21.15.29:0/366361583 conn(0x7ff8300680a0 msgr2=0x7ff8300abc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 shutdown_connections 2023-10-07T09:30:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.247+0000 7ff8353ff700 1 -- 172.21.15.29:0/366361583 wait complete. 2023-10-07T09:30:09.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:29:19.636918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:29:20.573395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:30:09.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:30:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.464+0000 7f1c0d9ff700 1 Processor -- start 2023-10-07T09:30:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.464+0000 7f1c0d9ff700 1 -- start start 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.465+0000 7f1c0d9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c0815ade0 0x7f1c080b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.465+0000 7f1c0d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c0815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.465+0000 7f1c0d9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080b09b0 con 0x7f1c08153cc0 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.465+0000 7f1c0d9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c08154090 con 0x7f1c0815ade0 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.465+0000 7f1c0d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08159f60 con 0x7f1c080b0540 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c067fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c0815ade0 0x7f1c080b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c06ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1c08153cc0 0x7f1c080b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37432/0 (socket says 172.21.15.29:37432) 2023-10-07T09:30:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c06ffd700 1 -- 172.21.15.29:0/3040078937 learned_addr learned my addr 172.21.15.29:0/3040078937 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c077fe700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c0815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c05ffb700 1 -- 172.21.15.29:0/3040078937 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101674468 0 0) 0x7f1c080b09b0 con 0x7f1c08153cc0 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c067fc700 1 -- 172.21.15.29:0/3040078937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 msgr2=0x7f1c0815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c067fc700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c0815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c067fc700 1 -- 172.21.15.29:0/3040078937 >> v1:172.21.15.29:6789/0 conn(0x7f1c08153cc0 legacy=0x7f1c080b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c067fc700 1 -- 172.21.15.29:0/3040078937 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0815a240 con 0x7f1c0815ade0 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.466+0000 7f1c077fe700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c0815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c067fc700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c0815ade0 0x7f1c080b0000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0098e0 tx=0x7f1bfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487a184dedbc417d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c05ffb700 1 -- 172.21.15.29:0/3040078937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc009e80 con 0x7f1c0815ade0 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c05ffb700 1 -- 172.21.15.29:0/3040078937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc00b860 con 0x7f1c0815ade0 2023-10-07T09:30:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c05ffb700 1 -- 172.21.15.29:0/3040078937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00bbe0 con 0x7f1c0815ade0 2023-10-07T09:30:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/3040078937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c0815ade0 msgr2=0x7f1c080b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c0815ade0 0x7f1c080b0000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0098e0 tx=0x7f1bfc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/3040078937 shutdown_connections 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c0815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/3040078937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c0815ade0 0x7f1c080b0000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.467+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/3040078937 >> 172.21.15.29:0/3040078937 conn(0x7f1c08067610 msgr2=0x7f1c08067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.468+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/3040078937 shutdown_connections 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.468+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/3040078937 wait complete. 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.469+0000 7f1c0d9ff700 1 Processor -- start 2023-10-07T09:30:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.469+0000 7f1c0d9ff700 1 -- start start 2023-10-07T09:30:09.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c0d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c080ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c0d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 0x7f1c08152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c0d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c0815ade0 0x7f1c0815c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c0d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0815c3b0 con 0x7f1c08153cc0 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c0d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080cf040 con 0x7f1c0815ade0 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c0d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0815b7a0 con 0x7f1c080b0540 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 0x7f1c08152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 0x7f1c08152f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40752/0 (socket says 172.21.15.29:40752) 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.470+0000 7f1c067fc700 1 -- 172.21.15.29:0/1535192180 learned_addr learned my addr 172.21.15.29:0/1535192180 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c06ffd700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c080ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c067fc700 1 -- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 msgr2=0x7f1c080ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c077fe700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c0815ade0 0x7f1c0815c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c067fc700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c080ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c067fc700 1 -- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c0815ade0 msgr2=0x7f1c0815c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c067fc700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c0815ade0 0x7f1c0815c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c067fc700 1 -- 172.21.15.29:0/1535192180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc009420 con 0x7f1c08153cc0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c06ffd700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c080ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c077fe700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c0815ade0 0x7f1c0815c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c067fc700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 0x7f1c08152f00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c080b0be0 tx=0x7f1bfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00bbe0 con 0x7f1c08153cc0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc000cc0 con 0x7f1c08153cc0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0815ba80 con 0x7f1c08153cc0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc011630 con 0x7f1c08153cc0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.471+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0815c040 con 0x7f1c08153cc0 2023-10-07T09:30:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.472+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0815c560 con 0x7f1c08153cc0 2023-10-07T09:30:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.473+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1bfc0119e0 con 0x7f1c08153cc0 2023-10-07T09:30:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.474+0000 7f1beeffd700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.474+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f1bfc085f50 con 0x7f1c08153cc0 2023-10-07T09:30:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.475+0000 7f1c06ffd700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.475+0000 7f1c06ffd700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8001860 tx=0x7f1bf8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.478+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c0815c560 con 0x7f1c08153cc0 2023-10-07T09:30:09.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.681+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c0815c560 con 0x7f1c08153cc0 2023-10-07T09:30:09.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.682+0000 7f1beeffd700 1 -- 172.21.15.29:0/1535192180 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c0815c560 con 0x7f1c08153cc0 2023-10-07T09:30:09.683 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:30:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 msgr2=0x7f1bf40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8001860 tx=0x7f1bf8009040 comp rx=0 tx=0).stop 2023-10-07T09:30:09.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 msgr2=0x7f1c08152f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 0x7f1c08152f00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c080b0be0 tx=0x7f1bfc017000 comp rx=0 tx=0).stop 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 shutdown_connections 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1bf4066f30 0x7f1bf40693e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c0815ade0 0x7f1c0815c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c080b0540 0x7f1c080ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 --2- 172.21.15.29:0/1535192180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c08153cc0 0x7f1c08152f00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 >> 172.21.15.29:0/1535192180 conn(0x7f1c08067610 msgr2=0x7f1c080aee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 shutdown_connections 2023-10-07T09:30:09.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.684+0000 7f1c0d9ff700 1 -- 172.21.15.29:0/1535192180 wait complete. 2023-10-07T09:30:09.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:30:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.901+0000 7f9288fff700 1 Processor -- start 2023-10-07T09:30:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.901+0000 7f9288fff700 1 -- start start 2023-10-07T09:30:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.902+0000 7f9288fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9284159750 0x7f92840d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.902+0000 7f9288fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9284159b20 0x7f92840ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.902+0000 7f9288fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f92840ce6d0 0x7f92840ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.902+0000 7f9288fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92840cbf20 con 0x7f9284159b20 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.902+0000 7f9288fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92840cc200 con 0x7f92840ce6d0 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.902+0000 7f9288fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92840cc4e0 con 0x7f9284159750 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9284159b20 0x7f92840ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9284159b20 0x7f92840ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40760/0 (socket says 172.21.15.29:40760) 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 -- 172.21.15.29:0/1769851113 learned_addr learned my addr 172.21.15.29:0/1769851113 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f928259c700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9284159750 0x7f92840d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9282d9d700 1 --2- 172.21.15.29:0/1769851113 >> v2:172.21.15.144:3300/0 conn(0x7f92840ce6d0 0x7f92840ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 -- 172.21.15.29:0/1769851113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9284159750 msgr2=0x7f92840d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9284159750 0x7f92840d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 -- 172.21.15.29:0/1769851113 >> v2:172.21.15.144:3300/0 conn(0x7f92840ce6d0 msgr2=0x7f92840ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 --2- 172.21.15.29:0/1769851113 >> v2:172.21.15.144:3300/0 conn(0x7f92840ce6d0 0x7f92840ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9281d9b700 1 -- 172.21.15.29:0/1769851113 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92840cc7c0 con 0x7f9284159b20 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f9282d9d700 1 --2- 172.21.15.29:0/1769851113 >> v2:172.21.15.144:3300/0 conn(0x7f92840ce6d0 0x7f92840ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.903+0000 7f928259c700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9284159750 0x7f92840d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9281d9b700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f92840ce190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f92840d0d60 tx=0x7f926c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94122a9e194d4f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f928159a700 1 -- 172.21.15.29:0/1769851113 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c00b860 con 0x7f9284159b20 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f928159a700 1 -- 172.21.15.29:0/1769851113 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c00b9c0 con 0x7f9284159b20 2023-10-07T09:30:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f928159a700 1 -- 172.21.15.29:0/1769851113 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c00bd60 con 0x7f9284159b20 2023-10-07T09:30:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 -- 172.21.15.29:0/1769851113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 msgr2=0x7f92840ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f92840ce190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f92840d0d60 tx=0x7f926c009250 comp rx=0 tx=0).stop 2023-10-07T09:30:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 -- 172.21.15.29:0/1769851113 shutdown_connections 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 --2- 172.21.15.29:0/1769851113 >> v2:172.21.15.144:3300/0 conn(0x7f92840ce6d0 0x7f92840ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9284159750 0x7f92840d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 --2- 172.21.15.29:0/1769851113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f92840ce190 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.904+0000 7f9288fff700 1 -- 172.21.15.29:0/1769851113 >> 172.21.15.29:0/1769851113 conn(0x7f9284063a10 msgr2=0x7f92840d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.905+0000 7f9288fff700 1 -- 172.21.15.29:0/1769851113 shutdown_connections 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.905+0000 7f9288fff700 1 -- 172.21.15.29:0/1769851113 wait complete. 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.906+0000 7f9288fff700 1 Processor -- start 2023-10-07T09:30:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.906+0000 7f9288fff700 1 -- start start 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9288fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92840ce6d0 0x7f9284152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9288fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9284159750 0x7f92841579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9288fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f9284158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9288fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284152e50 con 0x7f9284159b20 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9288fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92840c93c0 con 0x7f9284159750 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9288fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92840c96a0 con 0x7f92840ce6d0 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9282d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f9284158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9282d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f9284158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40770/0 (socket says 172.21.15.29:40770) 2023-10-07T09:30:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.907+0000 7f9282d9d700 1 -- 172.21.15.29:0/2071017164 learned_addr learned my addr 172.21.15.29:0/2071017164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f928259c700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92840ce6d0 0x7f9284152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9281d9b700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9284159750 0x7f92841579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9282d9d700 1 -- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92840ce6d0 msgr2=0x7f9284152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9282d9d700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92840ce6d0 0x7f9284152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9282d9d700 1 -- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9284159750 msgr2=0x7f92841579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9282d9d700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9284159750 0x7f92841579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9282d9d700 1 -- 172.21.15.29:0/2071017164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0094f0 con 0x7f9284159b20 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f928259c700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92840ce6d0 0x7f9284152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9282d9d700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f9284158350 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f927c00d180 tx=0x7f927c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c00de20 con 0x7f9284159b20 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c00f040 con 0x7f9284159b20 2023-10-07T09:30:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c015690 con 0x7f9284159b20 2023-10-07T09:30:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928415ea10 con 0x7f9284159b20 2023-10-07T09:30:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.908+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9284151090 con 0x7f9284159b20 2023-10-07T09:30:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.910+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928415e850 con 0x7f9284159b20 2023-10-07T09:30:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.911+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f927c01e3b0 con 0x7f9284159b20 2023-10-07T09:30:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.911+0000 7f927a7fc700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9264066f30 0x7f92640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.911+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f927c084bd0 con 0x7f9284159b20 2023-10-07T09:30:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.912+0000 7f928259c700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9264066f30 0x7f92640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.913+0000 7f928259c700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9264066f30 0x7f92640693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9284153440 tx=0x7f9274008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:09.915+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f927c051d30 con 0x7f9284159b20 2023-10-07T09:30:10.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.118+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f92840cb660 con 0x7f9284159b20 2023-10-07T09:30:10.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.119+0000 7f927a7fc700 1 -- 172.21.15.29:0/2071017164 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f927c01a020 con 0x7f9284159b20 2023-10-07T09:30:10.120 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:30:10.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9264066f30 msgr2=0x7f92640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9264066f30 0x7f92640693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9284153440 tx=0x7f9274008040 comp rx=0 tx=0).stop 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 msgr2=0x7f9284158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f9284158350 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f927c00d180 tx=0x7f927c009a50 comp rx=0 tx=0).stop 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 shutdown_connections 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9264066f30 0x7f92640693e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9284159b20 0x7f9284158350 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92840ce6d0 0x7f9284152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 --2- 172.21.15.29:0/2071017164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9284159750 0x7f92841579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 >> 172.21.15.29:0/2071017164 conn(0x7f9284063a10 msgr2=0x7f92840cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 shutdown_connections 2023-10-07T09:30:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.121+0000 7f9288fff700 1 -- 172.21.15.29:0/2071017164 wait complete. 2023-10-07T09:30:10.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:30:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.337+0000 7f26c87ff700 1 Processor -- start 2023-10-07T09:30:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.337+0000 7f26c87ff700 1 -- start start 2023-10-07T09:30:10.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.338+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c0156180 0x7f26c00afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c87ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00ad4d0 con 0x7f26c00ae570 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c87ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00ad7b0 con 0x7f26c00b0520 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c00ada90 con 0x7f26c0156180 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c5d9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f26c00ae570 0x7f26c00accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37440/0 (socket says 172.21.15.29:37440) 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c5d9a700 1 -- 172.21.15.29:0/2451477868 learned_addr learned my addr 172.21.15.29:0/2451477868 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c659b700 1 --2- 172.21.15.29:0/2451477868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c0156180 0x7f26c00afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438673971 0 0) 0x7f26c00ad4d0 con 0x7f26c00ae570 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac003570 con 0x7f26c00ae570 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99689146 0 0) 0x7f26c00ad7b0 con 0x7f26c00b0520 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.339+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c00ad4d0 con 0x7f26c00b0520 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c659b700 1 -- 172.21.15.29:0/2451477868 >> v1:172.21.15.144:6789/0 conn(0x7f26c00b0520 legacy=0x7f26c0156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c659b700 1 -- 172.21.15.29:0/2451477868 >> v1:172.21.15.29:6789/0 conn(0x7f26c00ae570 legacy=0x7f26c00accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c659b700 1 -- 172.21.15.29:0/2451477868 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00add70 con 0x7f26c0156180 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c659b700 1 --2- 172.21.15.29:0/2451477868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c0156180 0x7f26c00afc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009910 tx=0x7f26bc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f4772d2ba6aa2bc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc011630 con 0x7f26c0156180 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc011790 con 0x7f26c0156180 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.340+0000 7f26c5599700 1 -- 172.21.15.29:0/2451477868 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc011b30 con 0x7f26c0156180 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 -- 172.21.15.29:0/2451477868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c0156180 msgr2=0x7f26c00afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 --2- 172.21.15.29:0/2451477868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c0156180 0x7f26c00afc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009910 tx=0x7f26bc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 -- 172.21.15.29:0/2451477868 shutdown_connections 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 --2- 172.21.15.29:0/2451477868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c0156180 0x7f26c00afc00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 -- 172.21.15.29:0/2451477868 >> 172.21.15.29:0/2451477868 conn(0x7f26c0067610 msgr2=0x7f26c0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 -- 172.21.15.29:0/2451477868 shutdown_connections 2023-10-07T09:30:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.341+0000 7f26c87ff700 1 -- 172.21.15.29:0/2451477868 wait complete. 2023-10-07T09:30:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.342+0000 7f26c87ff700 1 Processor -- start 2023-10-07T09:30:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.342+0000 7f26c87ff700 1 -- start start 2023-10-07T09:30:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.343+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c00ae570 0x7f26c016cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.343+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c00b0520 0x7f26c00c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.343+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 0x7f26c016a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.343+0000 7f26c87ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c0158260 con 0x7f26c0156180 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.343+0000 7f26c87ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c016d4b0 con 0x7f26c00b0520 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.343+0000 7f26c87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c0169940 con 0x7f26c00ae570 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 0x7f26c016a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 0x7f26c016a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40778/0 (socket says 172.21.15.29:40778) 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 -- 172.21.15.29:0/1606489389 learned_addr learned my addr 172.21.15.29:0/1606489389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c659b700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c00ae570 0x7f26c016cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c5d9a700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c00b0520 0x7f26c00c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 -- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c00ae570 msgr2=0x7f26c016cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c00ae570 0x7f26c016cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 -- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c00b0520 msgr2=0x7f26c00c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c00b0520 0x7f26c00c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 -- 172.21.15.29:0/1606489389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc009420 con 0x7f26c0156180 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.344+0000 7f26c6d9c700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 0x7f26c016a130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c910 tx=0x7f26b80027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.345+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8012630 con 0x7f26c0156180 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.345+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b80127b0 con 0x7f26c0156180 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.345+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8012b70 con 0x7f26c0156180 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.345+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c016d6c0 con 0x7f26c0156180 2023-10-07T09:30:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.345+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c016dc80 con 0x7f26c0156180 2023-10-07T09:30:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.346+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c00c8fa0 con 0x7f26c0156180 2023-10-07T09:30:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.347+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f26b801b400 con 0x7f26c0156180 2023-10-07T09:30:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.347+0000 7f26b67fc700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26a4066f30 0x7f26a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.348+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f26b8084950 con 0x7f26c0156180 2023-10-07T09:30:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.348+0000 7f26c659b700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26a4066f30 0x7f26a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.349+0000 7f26c659b700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26a4066f30 0x7f26a40693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0091e0 tx=0x7f26bc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.351+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26b8051ab0 con 0x7f26c0156180 2023-10-07T09:30:10.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.555+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f26c004c3c0 con 0x7f26c0156180 2023-10-07T09:30:10.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.555+0000 7f26b67fc700 1 -- 172.21.15.29:0/1606489389 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f26b80550d0 con 0x7f26c0156180 2023-10-07T09:30:10.556 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26a4066f30 msgr2=0x7f26a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26a4066f30 0x7f26a40693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0091e0 tx=0x7f26bc017000 comp rx=0 tx=0).stop 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 msgr2=0x7f26c016a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 0x7f26c016a130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c910 tx=0x7f26b80027a0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 shutdown_connections 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26a4066f30 0x7f26a40693e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c0156180 0x7f26c016a130 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c00ae570 0x7f26c016cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 --2- 172.21.15.29:0/1606489389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c00b0520 0x7f26c00c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 >> 172.21.15.29:0/1606489389 conn(0x7f26c0067610 msgr2=0x7f26c0154f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 shutdown_connections 2023-10-07T09:30:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.558+0000 7f26c87ff700 1 -- 172.21.15.29:0/1606489389 wait complete. 2023-10-07T09:30:10.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:30:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.771+0000 7f3e9bdff700 1 Processor -- start 2023-10-07T09:30:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.771+0000 7f3e9bdff700 1 -- start start 2023-10-07T09:30:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.772+0000 7f3e9bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e94159ec0 0x7f3e9415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.772+0000 7f3e9bdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e940af400 0x7f3e940af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.772+0000 7f3e9bdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9414d790 con 0x7f3e940ad370 2023-10-07T09:30:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.772+0000 7f3e9bdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e940ae5d0 con 0x7f3e940af400 2023-10-07T09:30:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.772+0000 7f3e9bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9415a850 con 0x7f3e94159ec0 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e940af400 0x7f3e940af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e940af400 0x7f3e940af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40782/0 (socket says 172.21.15.29:40782) 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 -- 172.21.15.29:0/821328497 learned_addr learned my addr 172.21.15.29:0/821328497 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9939a700 1 --2- 172.21.15.29:0/821328497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e94159ec0 0x7f3e9415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 -- 172.21.15.29:0/821328497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e94159ec0 msgr2=0x7f3e9415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 --2- 172.21.15.29:0/821328497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e94159ec0 0x7f3e9415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 -- 172.21.15.29:0/821328497 >> v1:172.21.15.144:6789/0 conn(0x7f3e940ad370 legacy=0x7f3e940cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 -- 172.21.15.29:0/821328497 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940ad740 con 0x7f3e940af400 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.773+0000 7f3e9a39c700 1 --2- 172.21.15.29:0/821328497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e940af400 0x7f3e940af810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009f20 tx=0x7f3e8c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f41ea94641986cb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e98b99700 1 -- 172.21.15.29:0/821328497 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c00e070 con 0x7f3e940af400 2023-10-07T09:30:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e98b99700 1 -- 172.21.15.29:0/821328497 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c00b9d0 con 0x7f3e940af400 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e98b99700 1 -- 172.21.15.29:0/821328497 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0092e0 con 0x7f3e940af400 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 -- 172.21.15.29:0/821328497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e940af400 msgr2=0x7f3e940af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/821328497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e940af400 0x7f3e940af810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009f20 tx=0x7f3e8c009840 comp rx=0 tx=0).stop 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 -- 172.21.15.29:0/821328497 shutdown_connections 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/821328497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e940af400 0x7f3e940af810 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/821328497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e94159ec0 0x7f3e9415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 -- 172.21.15.29:0/821328497 >> 172.21.15.29:0/821328497 conn(0x7f3e940680a0 msgr2=0x7f3e940cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 -- 172.21.15.29:0/821328497 shutdown_connections 2023-10-07T09:30:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.774+0000 7f3e9bdff700 1 -- 172.21.15.29:0/821328497 wait complete. 2023-10-07T09:30:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.776+0000 7f3e9bdff700 1 Processor -- start 2023-10-07T09:30:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.776+0000 7f3e9bdff700 1 -- start start 2023-10-07T09:30:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e940ad370 0x7f3e940cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9bdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e940af400 0x7f3e94147100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9bdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 0x7f3e9414b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9bdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e940caf10 con 0x7f3e94159ec0 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9bdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e940a6610 con 0x7f3e940af400 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e940a6050 con 0x7f3e940ad370 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9a39c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 0x7f3e9414b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9a39c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 0x7f3e9414b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40790/0 (socket says 172.21.15.29:40790) 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9a39c700 1 -- 172.21.15.29:0/1773007117 learned_addr learned my addr 172.21.15.29:0/1773007117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e9939a700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e940af400 0x7f3e94147100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.777+0000 7f3e99b9b700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e940ad370 0x7f3e940cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9a39c700 1 -- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e940ad370 msgr2=0x7f3e940cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9a39c700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e940ad370 0x7f3e940cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9a39c700 1 -- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e940af400 msgr2=0x7f3e94147100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9a39c700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e940af400 0x7f3e94147100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9a39c700 1 -- 172.21.15.29:0/1773007117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c009ae0 con 0x7f3e94159ec0 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9a39c700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 0x7f3e9414b0d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e94068dd0 tx=0x7f3e8c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c008c70 con 0x7f3e94159ec0 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c008e10 con 0x7f3e94159ec0 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e940a6330 con 0x7f3e94159ec0 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0266b0 con 0x7f3e94159ec0 2023-10-07T09:30:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.778+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e94147cd0 con 0x7f3e94159ec0 2023-10-07T09:30:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.779+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9404c3c0 con 0x7f3e94159ec0 2023-10-07T09:30:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.780+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3e8c003040 con 0x7f3e94159ec0 2023-10-07T09:30:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.782+0000 7f3e89ffb700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3e78066f30 0x7f3e780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.782+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f3e8c086070 con 0x7f3e94159ec0 2023-10-07T09:30:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.782+0000 7f3e99b9b700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3e78066f30 0x7f3e780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.783+0000 7f3e99b9b700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3e78066f30 0x7f3e780693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3e940c8360 tx=0x7f3e90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:10.786+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e8c053280 con 0x7f3e94159ec0 2023-10-07T09:30:11.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.029+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e940ccfe0 con 0x7f3e94159ec0 2023-10-07T09:30:11.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.029+0000 7f3e89ffb700 1 -- 172.21.15.29:0/1773007117 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f3e8c0568a0 con 0x7f3e94159ec0 2023-10-07T09:30:11.031 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:11.031 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3e78066f30 msgr2=0x7f3e780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3e78066f30 0x7f3e780693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3e940c8360 tx=0x7f3e90009040 comp rx=0 tx=0).stop 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 msgr2=0x7f3e9414b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 0x7f3e9414b0d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e94068dd0 tx=0x7f3e8c018000 comp rx=0 tx=0).stop 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 shutdown_connections 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3e78066f30 0x7f3e780693e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e94159ec0 0x7f3e9414b0d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e940ad370 0x7f3e940cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 --2- 172.21.15.29:0/1773007117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e940af400 0x7f3e94147100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 >> 172.21.15.29:0/1773007117 conn(0x7f3e940680a0 msgr2=0x7f3e94157fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:11.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 shutdown_connections 2023-10-07T09:30:11.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.032+0000 7f3e9bdff700 1 -- 172.21.15.29:0/1773007117 wait complete. 2023-10-07T09:30:11.034 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:30:11.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.245+0000 7fe9153ff700 1 Processor -- start 2023-10-07T09:30:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.245+0000 7fe9153ff700 1 -- start start 2023-10-07T09:30:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.246+0000 7fe9153ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe910159750 0x7fe9100d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.246+0000 7fe9153ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9100ce8a0 0x7fe9100c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.246+0000 7fe9153ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9100cbf20 con 0x7fe910159b20 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.246+0000 7fe9153ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100cc200 con 0x7fe910159750 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.246+0000 7fe9153ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100cc4e0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90e7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe910159b20 0x7fe9100ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37442/0 (socket says 172.21.15.29:37442) 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90e7fc700 1 -- 172.21.15.29:0/1753988951 learned_addr learned my addr 172.21.15.29:0/1753988951 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90f7fe700 1 --2- 172.21.15.29:0/1753988951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9100ce8a0 0x7fe9100c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90effd700 1 --2- 172.21.15.29:0/1753988951 >> v2:172.21.15.144:3300/0 conn(0x7fe910159750 0x7fe9100d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90dffb700 1 -- 172.21.15.29:0/1753988951 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335995526 0 0) 0x7fe9100cbf20 con 0x7fe910159b20 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90dffb700 1 -- 172.21.15.29:0/1753988951 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f8003570 con 0x7fe910159b20 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90f7fe700 1 -- 172.21.15.29:0/1753988951 >> v2:172.21.15.144:3300/0 conn(0x7fe910159750 msgr2=0x7fe9100d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90f7fe700 1 --2- 172.21.15.29:0/1753988951 >> v2:172.21.15.144:3300/0 conn(0x7fe910159750 0x7fe9100d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90f7fe700 1 -- 172.21.15.29:0/1753988951 >> v1:172.21.15.29:6789/0 conn(0x7fe910159b20 legacy=0x7fe9100ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90f7fe700 1 -- 172.21.15.29:0/1753988951 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9100cc7c0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.247+0000 7fe90effd700 1 --2- 172.21.15.29:0/1753988951 >> v2:172.21.15.144:3300/0 conn(0x7fe910159750 0x7fe9100d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.248+0000 7fe90f7fe700 1 --2- 172.21.15.29:0/1753988951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9100ce8a0 0x7fe9100c5460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe908009e60 tx=0x7fe908009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e79ba08f962b21a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.248+0000 7fe90dffb700 1 -- 172.21.15.29:0/1753988951 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800e070 con 0x7fe9100ce8a0 2023-10-07T09:30:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.248+0000 7fe90dffb700 1 -- 172.21.15.29:0/1753988951 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90800bab0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.248+0000 7fe90dffb700 1 -- 172.21.15.29:0/1753988951 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800be70 con 0x7fe9100ce8a0 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.248+0000 7fe9153ff700 1 -- 172.21.15.29:0/1753988951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9100ce8a0 msgr2=0x7fe9100c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.248+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1753988951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9100ce8a0 0x7fe9100c5460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe908009e60 tx=0x7fe908009780 comp rx=0 tx=0).stop 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.249+0000 7fe9153ff700 1 -- 172.21.15.29:0/1753988951 shutdown_connections 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.249+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1753988951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9100ce8a0 0x7fe9100c5460 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.249+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1753988951 >> v2:172.21.15.144:3300/0 conn(0x7fe910159750 0x7fe9100d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.249+0000 7fe9153ff700 1 -- 172.21.15.29:0/1753988951 >> 172.21.15.29:0/1753988951 conn(0x7fe910063a10 msgr2=0x7fe9100d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.249+0000 7fe9153ff700 1 -- 172.21.15.29:0/1753988951 shutdown_connections 2023-10-07T09:30:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.249+0000 7fe9153ff700 1 -- 172.21.15.29:0/1753988951 wait complete. 2023-10-07T09:30:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.250+0000 7fe9153ff700 1 Processor -- start 2023-10-07T09:30:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.250+0000 7fe9153ff700 1 -- start start 2023-10-07T09:30:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe9153ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 0x7fe910155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe9153ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe910159750 0x7fe910140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe9153ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe910159b20 0x7fe9100c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe9153ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9101540e0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe9153ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9101543c0 con 0x7fe910159b20 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe9153ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9101546a0 con 0x7fe910159750 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe90effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 0x7fe910155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.251+0000 7fe90effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 0x7fe910155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40792/0 (socket says 172.21.15.29:40792) 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 -- 172.21.15.29:0/1436358906 learned_addr learned my addr 172.21.15.29:0/1436358906 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90f7fe700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe910159b20 0x7fe9100c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90e7fc700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe910159750 0x7fe910140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 -- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe910159750 msgr2=0x7fe910140e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe910159750 0x7fe910140e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 -- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe910159b20 msgr2=0x7fe9100c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe910159b20 0x7fe9100c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 -- 172.21.15.29:0/1436358906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000093c0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90f7fe700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe910159b20 0x7fe9100c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe90effd700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 0x7fe910155180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe90000ec80 tx=0x7fe900009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.252+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900005a00 con 0x7fe9100ce8a0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.253+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe900005b60 con 0x7fe9100ce8a0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.253+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908009a20 con 0x7fe9100ce8a0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.253+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90000c9d0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.253+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe910154c60 con 0x7fe9100ce8a0 2023-10-07T09:30:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.254+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe910154fd0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.255+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe900006030 con 0x7fe9100ce8a0 2023-10-07T09:30:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.256+0000 7fe8f6ffd700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe8fc066f30 0x7fe8fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.256+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fe900084b80 con 0x7fe9100ce8a0 2023-10-07T09:30:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.256+0000 7fe90e7fc700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe8fc066f30 0x7fe8fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.257+0000 7fe90e7fc700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe8fc066f30 0x7fe8fc0693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe91016cfc0 tx=0x7fe904009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.260+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe900051ce0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.461+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe91004c3c0 con 0x7fe9100ce8a0 2023-10-07T09:30:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.462+0000 7fe8f6ffd700 1 -- 172.21.15.29:0/1436358906 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+15636 (secure 0 0 0) 0x7fe900055300 con 0x7fe9100ce8a0 2023-10-07T09:30:11.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:11.463 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:05.095626+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe8fc066f30 msgr2=0x7fe8fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe8fc066f30 0x7fe8fc0693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe91016cfc0 tx=0x7fe904009040 comp rx=0 tx=0).stop 2023-10-07T09:30:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 msgr2=0x7fe910155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 0x7fe910155180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe90000ec80 tx=0x7fe900009120 comp rx=0 tx=0).stop 2023-10-07T09:30:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 shutdown_connections 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe8fc066f30 0x7fe8fc0693e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe910159b20 0x7fe9100c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9100ce8a0 0x7fe910155180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 --2- 172.21.15.29:0/1436358906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe910159750 0x7fe910140e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 >> 172.21.15.29:0/1436358906 conn(0x7fe910063a10 msgr2=0x7fe9100cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 shutdown_connections 2023-10-07T09:30:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.464+0000 7fe9153ff700 1 -- 172.21.15.29:0/1436358906 wait complete. 2023-10-07T09:30:11.482 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:30:11.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:30:11.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.678+0000 7f7b1b7ff700 1 Processor -- start 2023-10-07T09:30:11.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.678+0000 7f7b1b7ff700 1 -- start start 2023-10-07T09:30:11.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.679+0000 7f7b1b7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b140c6470 0x7f7b140d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.679+0000 7f7b1b7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b140cefb0 con 0x7f7b14159b20 2023-10-07T09:30:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.679+0000 7f7b1b7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b140cbf20 con 0x7f7b14159750 2023-10-07T09:30:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.679+0000 7f7b1b7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b140cc200 con 0x7f7b140c6470 2023-10-07T09:30:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.679+0000 7f7b18d9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7b14159b20 0x7f7b140ce8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37444/0 (socket says 172.21.15.29:37444) 2023-10-07T09:30:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.679+0000 7f7b18d9a700 1 -- 172.21.15.29:0/116916898 learned_addr learned my addr 172.21.15.29:0/116916898 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b19d9c700 1 --2- 172.21.15.29:0/116916898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b140c6470 0x7f7b140d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070548461 0 0) 0x7f7b140cefb0 con 0x7f7b14159b20 2023-10-07T09:30:11.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00003570 con 0x7f7b14159b20 2023-10-07T09:30:11.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239447833 0 0) 0x7f7b140cbf20 con 0x7f7b14159750 2023-10-07T09:30:11.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b140cefb0 con 0x7f7b14159750 2023-10-07T09:30:11.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b19d9c700 1 -- 172.21.15.29:0/116916898 >> v1:172.21.15.144:6789/0 conn(0x7f7b14159750 legacy=0x7f7b140ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b19d9c700 1 -- 172.21.15.29:0/116916898 >> v1:172.21.15.29:6789/0 conn(0x7f7b14159b20 legacy=0x7f7b140ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b19d9c700 1 -- 172.21.15.29:0/116916898 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b140cc4e0 con 0x7f7b140c6470 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.680+0000 7f7b19d9c700 1 --2- 172.21.15.29:0/116916898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b140c6470 0x7f7b140d10a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009e90 tx=0x7f7b0c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ce9d45d63449105 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c00e070 con 0x7f7b140c6470 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c00bb80 con 0x7f7b140c6470 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b0bfff700 1 -- 172.21.15.29:0/116916898 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c012430 con 0x7f7b140c6470 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/116916898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b140c6470 msgr2=0x7f7b140d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/116916898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b140c6470 0x7f7b140d10a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009e90 tx=0x7f7b0c009780 comp rx=0 tx=0).stop 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/116916898 shutdown_connections 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/116916898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b140c6470 0x7f7b140d10a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/116916898 >> 172.21.15.29:0/116916898 conn(0x7f7b14063a10 msgr2=0x7f7b140d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:11.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.681+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/116916898 shutdown_connections 2023-10-07T09:30:11.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.682+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/116916898 wait complete. 2023-10-07T09:30:11.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.684+0000 7f7b1b7ff700 1 Processor -- start 2023-10-07T09:30:11.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.684+0000 7f7b1b7ff700 1 -- start start 2023-10-07T09:30:11.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1b7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b140c6470 0x7f7b140ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1b7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 0x7f7b14168ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1b7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b14159b20 0x7f7b140b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1b7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14157b80 con 0x7f7b14159750 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1b7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b140cedb0 con 0x7f7b140c6470 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1b7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14157220 con 0x7f7b14159b20 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b18d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 0x7f7b14168ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b18d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 0x7f7b14168ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40796/0 (socket says 172.21.15.29:40796) 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b18d9a700 1 -- 172.21.15.29:0/1203588675 learned_addr learned my addr 172.21.15.29:0/1203588675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b19d9c700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b14159b20 0x7f7b140b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.685+0000 7f7b1959b700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b140c6470 0x7f7b140ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b18d9a700 1 -- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b14159b20 msgr2=0x7f7b140b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b18d9a700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b14159b20 0x7f7b140b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b18d9a700 1 -- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b140c6470 msgr2=0x7f7b140ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b18d9a700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b140c6470 0x7f7b140ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b18d9a700 1 -- 172.21.15.29:0/1203588675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c009a20 con 0x7f7b14159750 2023-10-07T09:30:11.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b18d9a700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 0x7f7b14168ec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400c7c0 tx=0x7f7b04007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:11.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04005b30 con 0x7f7b14159750 2023-10-07T09:30:11.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b04005c90 con 0x7f7b14159750 2023-10-07T09:30:11.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04007300 con 0x7f7b14159750 2023-10-07T09:30:11.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b141574c0 con 0x7f7b14159750 2023-10-07T09:30:11.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.686+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b140ce9f0 con 0x7f7b14159750 2023-10-07T09:30:11.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.688+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b14157cc0 con 0x7f7b14159750 2023-10-07T09:30:11.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.688+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b0401b3b0 con 0x7f7b14159750 2023-10-07T09:30:11.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.689+0000 7f7b097fa700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7af8066f30 0x7f7af80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:11.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.689+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f7b04083a90 con 0x7f7b14159750 2023-10-07T09:30:11.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.690+0000 7f7b1959b700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7af8066f30 0x7f7af80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:11.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.691+0000 7f7b1959b700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7af8066f30 0x7f7af80693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007fd0 tx=0x7f7b10007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:11.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.693+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b14157cc0 con 0x7f7b14159750 2023-10-07T09:30:11.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.944+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b14157cc0 con 0x7f7b14159750 2023-10-07T09:30:11.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.944+0000 7f7b097fa700 1 -- 172.21.15.29:0/1203588675 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f7b14157cc0 con 0x7f7b14159750 2023-10-07T09:30:11.945 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:11.946 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7af8066f30 msgr2=0x7f7af80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7af8066f30 0x7f7af80693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007fd0 tx=0x7f7b10007d30 comp rx=0 tx=0).stop 2023-10-07T09:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 msgr2=0x7f7b14168ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 0x7f7b14168ec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400c7c0 tx=0x7f7b04007ea0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 shutdown_connections 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7af8066f30 0x7f7af80693e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b14159b20 0x7f7b140b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b140c6470 0x7f7b140ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 --2- 172.21.15.29:0/1203588675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b14159750 0x7f7b14168ec0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 >> 172.21.15.29:0/1203588675 conn(0x7f7b14063a10 msgr2=0x7f7b140d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 shutdown_connections 2023-10-07T09:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:11.947+0000 7f7b1b7ff700 1 -- 172.21.15.29:0/1203588675 wait complete. 2023-10-07T09:30:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:30:11.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:30:12.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.161+0000 7f0947fff700 1 Processor -- start 2023-10-07T09:30:12.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.161+0000 7f0947fff700 1 -- start start 2023-10-07T09:30:12.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.162+0000 7f0947fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f094015ade0 0x7f09400b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.162+0000 7f0947fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0940154090 con 0x7f0940153cc0 2023-10-07T09:30:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.162+0000 7f0947fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0940159f60 con 0x7f09400b0540 2023-10-07T09:30:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.162+0000 7f0947fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09400b09e0 con 0x7f094015ade0 2023-10-07T09:30:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f0945d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0940153cc0 0x7f09400b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37452/0 (socket says 172.21.15.29:37452) 2023-10-07T09:30:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f0945d9b700 1 -- 172.21.15.29:0/3022094450 learned_addr learned my addr 172.21.15.29:0/3022094450 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f094559a700 1 --2- 172.21.15.29:0/3022094450 >> v2:172.21.15.144:3300/0 conn(0x7f094015ade0 0x7f09400b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f0944d99700 1 -- 172.21.15.29:0/3022094450 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284552412 0 0) 0x7f0940154090 con 0x7f0940153cc0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f0944d99700 1 -- 172.21.15.29:0/3022094450 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092c003570 con 0x7f0940153cc0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f094559a700 1 -- 172.21.15.29:0/3022094450 >> v1:172.21.15.144:6789/0 conn(0x7f09400b0540 legacy=0x7f094015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f094559a700 1 -- 172.21.15.29:0/3022094450 >> v1:172.21.15.29:6789/0 conn(0x7f0940153cc0 legacy=0x7f09400b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.163+0000 7f094559a700 1 -- 172.21.15.29:0/3022094450 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094015a240 con 0x7f094015ade0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f094559a700 1 --2- 172.21.15.29:0/3022094450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f094015ade0 0x7f09400b0000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09300098e0 tx=0x7f09300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8689ff8401a6bdb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f0944d99700 1 -- 172.21.15.29:0/3022094450 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000b640 con 0x7f094015ade0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f0944d99700 1 -- 172.21.15.29:0/3022094450 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093000b7a0 con 0x7f094015ade0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f0944d99700 1 -- 172.21.15.29:0/3022094450 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000bb40 con 0x7f094015ade0 2023-10-07T09:30:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f0947fff700 1 -- 172.21.15.29:0/3022094450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f094015ade0 msgr2=0x7f09400b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f0947fff700 1 --2- 172.21.15.29:0/3022094450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f094015ade0 0x7f09400b0000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09300098e0 tx=0x7f09300091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.164+0000 7f0947fff700 1 -- 172.21.15.29:0/3022094450 shutdown_connections 2023-10-07T09:30:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.165+0000 7f0947fff700 1 --2- 172.21.15.29:0/3022094450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f094015ade0 0x7f09400b0000 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.165+0000 7f0947fff700 1 -- 172.21.15.29:0/3022094450 >> 172.21.15.29:0/3022094450 conn(0x7f0940067610 msgr2=0x7f0940067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.165+0000 7f0947fff700 1 -- 172.21.15.29:0/3022094450 shutdown_connections 2023-10-07T09:30:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.165+0000 7f0947fff700 1 -- 172.21.15.29:0/3022094450 wait complete. 2023-10-07T09:30:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.167+0000 7f0947fff700 1 Processor -- start 2023-10-07T09:30:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.167+0000 7f0947fff700 1 -- start start 2023-10-07T09:30:12.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0947fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 0x7f09400cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0947fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0940153cc0 0x7f0940157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0947fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f094015ade0 0x7f094015e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0947fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0940169370 con 0x7f09400b0540 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0947fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0940169600 con 0x7f0940153cc0 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0947fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09400ce380 con 0x7f094015ade0 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0945d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 0x7f09400cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0945d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 0x7f09400cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40808/0 (socket says 172.21.15.29:40808) 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.168+0000 7f0945d9b700 1 -- 172.21.15.29:0/1183755613 learned_addr learned my addr 172.21.15.29:0/1183755613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f094559a700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0940153cc0 0x7f0940157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f094659c700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f094015ade0 0x7f094015e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0945d9b700 1 -- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f094015ade0 msgr2=0x7f094015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0945d9b700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f094015ade0 0x7f094015e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0945d9b700 1 -- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0940153cc0 msgr2=0x7f0940157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0945d9b700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0940153cc0 0x7f0940157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0945d9b700 1 -- 172.21.15.29:0/1183755613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930009420 con 0x7f09400b0540 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0945d9b700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 0x7f09400cde40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f093c00c790 tx=0x7f093c007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c0072e0 con 0x7f09400b0540 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c007440 con 0x7f09400b0540 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09401588a0 con 0x7f09400b0540 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.169+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c0057f0 con 0x7f09400b0540 2023-10-07T09:30:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.170+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0940159140 con 0x7f09400b0540 2023-10-07T09:30:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.170+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094004c3c0 con 0x7f09400b0540 2023-10-07T09:30:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.171+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f093c005c20 con 0x7f09400b0540 2023-10-07T09:30:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.172+0000 7f0935ffb700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0924066f30 0x7f09240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.172+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f093c083960 con 0x7f09400b0540 2023-10-07T09:30:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.173+0000 7f094559a700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0924066f30 0x7f09240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.173+0000 7f094559a700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0924066f30 0x7f09240693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f094015b200 tx=0x7f0930017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.176+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f093c04b5a0 con 0x7f09400b0540 2023-10-07T09:30:12.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.423+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f094015a550 con 0x7f09400b0540 2023-10-07T09:30:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.424+0000 7f0935ffb700 1 -- 172.21.15.29:0/1183755613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f093c017020 con 0x7f09400b0540 2023-10-07T09:30:12.425 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:12.425 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1664297612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1664297612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1664297612}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1265043464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1265043464},{"type":"v1","addr":"172.21.15.144:6835","nonce":1265043464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/423954529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":423954529},{"type":"v1","addr":"172.21.15.144:6837","nonce":423954529}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:29:21.598754+0000","modified":"2023-10-07T09:29:25.105471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/3263907411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3263907411},{"type":"v1","addr":"172.21.15.29:6834","nonce":3263907411}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.426+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0924066f30 msgr2=0x7f09240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.426+0000 7f0947fff700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0924066f30 0x7f09240693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f094015b200 tx=0x7f0930017000 comp rx=0 tx=0).stop 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.426+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 msgr2=0x7f09400cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.426+0000 7f0947fff700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 0x7f09400cde40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f093c00c790 tx=0x7f093c007c80 comp rx=0 tx=0).stop 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 shutdown_connections 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0924066f30 0x7f09240693e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f094015ade0 0x7f094015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400b0540 0x7f09400cde40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 --2- 172.21.15.29:0/1183755613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0940153cc0 0x7f0940157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 >> 172.21.15.29:0/1183755613 conn(0x7f0940067610 msgr2=0x7f0940152da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 shutdown_connections 2023-10-07T09:30:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.427+0000 7f0947fff700 1 -- 172.21.15.29:0/1183755613 wait complete. 2023-10-07T09:30:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-07T09:30:12.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.639+0000 7f7733dff700 1 Processor -- start 2023-10-07T09:30:12.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.639+0000 7f7733dff700 1 -- start start 2023-10-07T09:30:12.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7733dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f772c158670 0x7f772c162060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7733dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c158ad0 con 0x7f772c157290 2023-10-07T09:30:12.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7733dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c15b140 con 0x7f772c0ae8b0 2023-10-07T09:30:12.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7733dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772c15b420 con 0x7f772c158670 2023-10-07T09:30:12.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7731b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f772c157290 0x7f772c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37458/0 (socket says 172.21.15.29:37458) 2023-10-07T09:30:12.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7731b9b700 1 -- 172.21.15.29:0/3395086240 learned_addr learned my addr 172.21.15.29:0/3395086240 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:12.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f773239c700 1 --2- 172.21.15.29:0/3395086240 >> v2:172.21.15.29:3300/0 conn(0x7f772c158670 0x7f772c162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.641+0000 7f7730b99700 1 -- 172.21.15.29:0/3395086240 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716096794 0 0) 0x7f772c158ad0 con 0x7f772c157290 2023-10-07T09:30:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f7730b99700 1 -- 172.21.15.29:0/3395086240 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7718003570 con 0x7f772c157290 2023-10-07T09:30:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f773239c700 1 -- 172.21.15.29:0/3395086240 >> v1:172.21.15.144:6789/0 conn(0x7f772c0ae8b0 legacy=0x7f772c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f773239c700 1 -- 172.21.15.29:0/3395086240 >> v1:172.21.15.29:6789/0 conn(0x7f772c157290 legacy=0x7f772c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f773239c700 1 -- 172.21.15.29:0/3395086240 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c15b700 con 0x7f772c158670 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f773239c700 1 --2- 172.21.15.29:0/3395086240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c158670 0x7f772c162060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7724009e60 tx=0x7f7724009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c92bdc03986753c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f7730b99700 1 -- 172.21.15.29:0/3395086240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772400e070 con 0x7f772c158670 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f7730b99700 1 -- 172.21.15.29:0/3395086240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772400bb60 con 0x7f772c158670 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.642+0000 7f7730b99700 1 -- 172.21.15.29:0/3395086240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724012430 con 0x7f772c158670 2023-10-07T09:30:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 -- 172.21.15.29:0/3395086240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c158670 msgr2=0x7f772c162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 --2- 172.21.15.29:0/3395086240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c158670 0x7f772c162060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7724009e60 tx=0x7f7724009780 comp rx=0 tx=0).stop 2023-10-07T09:30:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 -- 172.21.15.29:0/3395086240 shutdown_connections 2023-10-07T09:30:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 --2- 172.21.15.29:0/3395086240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c158670 0x7f772c162060 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 -- 172.21.15.29:0/3395086240 >> 172.21.15.29:0/3395086240 conn(0x7f772c0680a0 msgr2=0x7f772c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 -- 172.21.15.29:0/3395086240 shutdown_connections 2023-10-07T09:30:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.643+0000 7f7733dff700 1 -- 172.21.15.29:0/3395086240 wait complete. 2023-10-07T09:30:12.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.644+0000 7f7733dff700 1 Processor -- start 2023-10-07T09:30:12.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 -- start start 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772c157290 0x7f772c16a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772c158670 0x7f772c15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c16cbd0 0x7f772c0c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c16a590 con 0x7f772c16cbd0 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c151de0 con 0x7f772c158670 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.645+0000 7f7733dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772c15c570 con 0x7f772c157290 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c16cbd0 0x7f772c0c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f7731b9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772c157290 0x7f772c16a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773139a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772c158670 0x7f772c15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f7731b9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772c157290 0x7f772c16a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:56696/0 (socket says 172.21.15.29:56696) 2023-10-07T09:30:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f7731b9b700 1 -- 172.21.15.29:0/2074225802 learned_addr learned my addr 172.21.15.29:0/2074225802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773239c700 1 -- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772c157290 msgr2=0x7f772c16a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773239c700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772c157290 0x7f772c16a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773239c700 1 -- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772c158670 msgr2=0x7f772c15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773239c700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772c158670 0x7f772c15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.646+0000 7f773239c700 1 -- 172.21.15.29:0/2074225802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7724009a20 con 0x7f772c16cbd0 2023-10-07T09:30:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.647+0000 7f773239c700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c16cbd0 0x7f772c0c7da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7724016000 tx=0x7f7724019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.647+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724027470 con 0x7f772c16cbd0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.647+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7724027630 con 0x7f772c16cbd0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.647+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77240279f0 con 0x7f772c16cbd0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.647+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c16d0b0 con 0x7f772c16cbd0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.647+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c16d670 con 0x7f772c16cbd0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.648+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c04c3c0 con 0x7f772c16cbd0 2023-10-07T09:30:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.649+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7724024070 con 0x7f772c16cbd0 2023-10-07T09:30:12.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.650+0000 7f7721ffb700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7710066f30 0x7f77100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:12.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.650+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f7724086bf0 con 0x7f772c16cbd0 2023-10-07T09:30:12.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.650+0000 7f7731b9b700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7710066f30 0x7f77100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:12.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.651+0000 7f7731b9b700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7710066f30 0x7f77100693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7728001a30 tx=0x7f7728009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:12.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.654+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7724053d50 con 0x7f772c16cbd0 2023-10-07T09:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.857+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f772c0ca390 con 0x7f772c16cbd0 2023-10-07T09:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.858+0000 7f7721ffb700 1 -- 172.21.15.29:0/2074225802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+15636 (secure 0 0 0) 0x7f7724057370 con 0x7f772c16cbd0 2023-10-07T09:30:12.859 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:12.859 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:05.095626+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:29:19.636918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-07T09:29:20.573395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7710066f30 msgr2=0x7f77100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7710066f30 0x7f77100693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7728001a30 tx=0x7f7728009040 comp rx=0 tx=0).stop 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c16cbd0 msgr2=0x7f772c0c7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c16cbd0 0x7f772c0c7da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7724016000 tx=0x7f7724019000 comp rx=0 tx=0).stop 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 shutdown_connections 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7710066f30 0x7f77100693e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f772c16cbd0 0x7f772c0c7da0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772c157290 0x7f772c16a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 --2- 172.21.15.29:0/2074225802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772c158670 0x7f772c15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.860+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 >> 172.21.15.29:0/2074225802 conn(0x7f772c0680a0 msgr2=0x7f772c0cb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.861+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 shutdown_connections 2023-10-07T09:30:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:12.861+0000 7f7733dff700 1 -- 172.21.15.29:0/2074225802 wait complete. 2023-10-07T09:30:12.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:30:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.074+0000 7f5acc1ff700 1 Processor -- start 2023-10-07T09:30:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.074+0000 7f5acc1ff700 1 -- start start 2023-10-07T09:30:13.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5acc1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ac4159ec0 0x7f5ac415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5acc1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5ac40af400 0x7f5ac40af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5acc1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac414d790 con 0x7f5ac40ad370 2023-10-07T09:30:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5acc1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac415a880 con 0x7f5ac4159ec0 2023-10-07T09:30:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5acc1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac40ae5d0 con 0x7f5ac40af400 2023-10-07T09:30:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5ac979a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ac4159ec0 0x7f5ac415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5ac979a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ac4159ec0 0x7f5ac415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48534/0 (socket says 172.21.15.29:48534) 2023-10-07T09:30:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5ac979a700 1 -- 172.21.15.29:0/1892513866 learned_addr learned my addr 172.21.15.29:0/1892513866 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.076+0000 7f5aca79c700 1 --2- 172.21.15.29:0/1892513866 >> v2:172.21.15.144:3300/0 conn(0x7f5ac40af400 0x7f5ac40af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac979a700 1 -- 172.21.15.29:0/1892513866 >> v2:172.21.15.144:3300/0 conn(0x7f5ac40af400 msgr2=0x7f5ac40af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac979a700 1 --2- 172.21.15.29:0/1892513866 >> v2:172.21.15.144:3300/0 conn(0x7f5ac40af400 0x7f5ac40af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac979a700 1 -- 172.21.15.29:0/1892513866 >> v1:172.21.15.29:6789/0 conn(0x7f5ac40ad370 legacy=0x7f5ac40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac979a700 1 -- 172.21.15.29:0/1892513866 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40ad740 con 0x7f5ac4159ec0 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac8f99700 1 -- 172.21.15.29:0/1892513866 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099922634 0 0) 0x7f5ac414d790 con 0x7f5ac40ad370 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac979a700 1 --2- 172.21.15.29:0/1892513866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac4159ec0 0x7f5ac415a340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40b1270 tx=0x7f5ab400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd72cf452377ec8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac8f99700 1 -- 172.21.15.29:0/1892513866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab400dba0 con 0x7f5ac4159ec0 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac8f99700 1 -- 172.21.15.29:0/1892513866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab400dd00 con 0x7f5ac4159ec0 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.077+0000 7f5ac8f99700 1 -- 172.21.15.29:0/1892513866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40196d0 con 0x7f5ac4159ec0 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 -- 172.21.15.29:0/1892513866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac4159ec0 msgr2=0x7f5ac415a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/1892513866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac4159ec0 0x7f5ac415a340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40b1270 tx=0x7f5ab400b4f0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 -- 172.21.15.29:0/1892513866 shutdown_connections 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/1892513866 >> v2:172.21.15.144:3300/0 conn(0x7f5ac40af400 0x7f5ac40af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/1892513866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac4159ec0 0x7f5ac415a340 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 -- 172.21.15.29:0/1892513866 >> 172.21.15.29:0/1892513866 conn(0x7f5ac40680a0 msgr2=0x7f5ac40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 -- 172.21.15.29:0/1892513866 shutdown_connections 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.078+0000 7f5acc1ff700 1 -- 172.21.15.29:0/1892513866 wait complete. 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.079+0000 7f5acc1ff700 1 Processor -- start 2023-10-07T09:30:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.080+0000 7f5acc1ff700 1 -- start start 2023-10-07T09:30:13.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5acc1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 0x7f5ac4169ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5acc1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ac40af400 0x7f5ac414cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5acc1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ac4159ec0 0x7f5ac4151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5acc1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac414d560 con 0x7f5ac40ad370 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5acc1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac414d7b0 con 0x7f5ac40af400 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5acc1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac414da90 con 0x7f5ac4159ec0 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 0x7f5ac4169ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 0x7f5ac4169ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48544/0 (socket says 172.21.15.29:48544) 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 -- 172.21.15.29:0/2494378821 learned_addr learned my addr 172.21.15.29:0/2494378821 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5aca79c700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ac4159ec0 0x7f5ac4151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac979a700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ac40af400 0x7f5ac414cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 -- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ac4159ec0 msgr2=0x7f5ac4151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ac4159ec0 0x7f5ac4151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 -- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ac40af400 msgr2=0x7f5ac414cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ac40af400 0x7f5ac414cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac9f9b700 1 -- 172.21.15.29:0/2494378821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab400b790 con 0x7f5ac40ad370 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.081+0000 7f5ac979a700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ac40af400 0x7f5ac414cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.082+0000 7f5ac9f9b700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 0x7f5ac4169ab0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000c7e0 tx=0x7f5ac0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.082+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0012700 con 0x7f5ac40ad370 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.082+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0012880 con 0x7f5ac40ad370 2023-10-07T09:30:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.082+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0012c40 con 0x7f5ac40ad370 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.082+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac414dd70 con 0x7f5ac40ad370 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.082+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac4169ff0 con 0x7f5ac40ad370 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.083+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac414e290 con 0x7f5ac40ad370 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.084+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5ac001b400 con 0x7f5ac40ad370 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.084+0000 7f5ab9ffb700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5aa8066f30 0x7f5aa80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.085+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f5ac0084210 con 0x7f5ac40ad370 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.085+0000 7f5ac979a700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5aa8066f30 0x7f5aa80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.086+0000 7f5ac979a700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5aa8066f30 0x7f5aa80693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40b1270 tx=0x7f5ab401b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:13.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.089+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ac414e290 con 0x7f5ac40ad370 2023-10-07T09:30:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.335+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5ac414e290 con 0x7f5ac40ad370 2023-10-07T09:30:13.337 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:30:13.336+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:30:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.340+0000 7f5ab9ffb700 1 -- 172.21.15.29:0/2494378821 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5ac414e290 con 0x7f5ac40ad370 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.343+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5aa8066f30 msgr2=0x7f5aa80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.343+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5aa8066f30 0x7f5aa80693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40b1270 tx=0x7f5ab401b000 comp rx=0 tx=0).stop 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.343+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 msgr2=0x7f5ac4169ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.343+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 0x7f5ac4169ab0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000c7e0 tx=0x7f5ac0007df0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 shutdown_connections 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5aa8066f30 0x7f5aa80693e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ac4159ec0 0x7f5ac4151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ac40ad370 0x7f5ac4169ab0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 --2- 172.21.15.29:0/2494378821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ac40af400 0x7f5ac414cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 >> 172.21.15.29:0/2494378821 conn(0x7f5ac40680a0 msgr2=0x7f5ac40abc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 shutdown_connections 2023-10-07T09:30:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.344+0000 7f5acc1ff700 1 -- 172.21.15.29:0/2494378821 wait complete. 2023-10-07T09:30:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:30:13.348 INFO:tasks.ceph.mds.c.smithi029.stderr: -2216> 2023-10-07T09:29:26.647+0000 7f37eb7f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:30:13.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:30:13.390 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:30:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.562+0000 7fe743fff700 1 Processor -- start 2023-10-07T09:30:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.562+0000 7fe743fff700 1 -- start start 2023-10-07T09:30:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe743fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c15a960 0x7fe73c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe743fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c0acb40 con 0x7fe73c0ae8a0 2023-10-07T09:30:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe743fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c0abf30 con 0x7fe73c15ad30 2023-10-07T09:30:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe743fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c158c60 con 0x7fe73c15a960 2023-10-07T09:30:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe74259c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe73c0ae8a0 0x7fe73c0b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47480/0 (socket says 172.21.15.29:47480) 2023-10-07T09:30:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe74259c700 1 -- 172.21.15.29:0/3518317135 learned_addr learned my addr 172.21.15.29:0/3518317135 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe741d9b700 1 --2- 172.21.15.29:0/3518317135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c15a960 0x7fe73c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.564+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808557983 0 0) 0x7fe73c0acb40 con 0x7fe73c0ae8a0 2023-10-07T09:30:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe728003570 con 0x7fe73c0ae8a0 2023-10-07T09:30:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180274679 0 0) 0x7fe73c0abf30 con 0x7fe73c15ad30 2023-10-07T09:30:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73c0acb40 con 0x7fe73c15ad30 2023-10-07T09:30:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe741d9b700 1 -- 172.21.15.29:0/3518317135 >> v1:172.21.15.144:6789/0 conn(0x7fe73c15ad30 legacy=0x7fe73c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe741d9b700 1 -- 172.21.15.29:0/3518317135 >> v1:172.21.15.29:6789/0 conn(0x7fe73c0ae8a0 legacy=0x7fe73c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe741d9b700 1 -- 172.21.15.29:0/3518317135 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c0ac210 con 0x7fe73c15a960 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe741d9b700 1 --2- 172.21.15.29:0/3518317135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c15a960 0x7fe73c158e20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe738009910 tx=0x7fe7380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6963bf2101a4d6e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738011630 con 0x7fe73c15a960 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe738011790 con 0x7fe73c15a960 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.565+0000 7fe740d99700 1 -- 172.21.15.29:0/3518317135 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738011b30 con 0x7fe73c15a960 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 -- 172.21.15.29:0/3518317135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c15a960 msgr2=0x7fe73c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 --2- 172.21.15.29:0/3518317135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c15a960 0x7fe73c158e20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe738009910 tx=0x7fe7380091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 -- 172.21.15.29:0/3518317135 shutdown_connections 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 --2- 172.21.15.29:0/3518317135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c15a960 0x7fe73c158e20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 -- 172.21.15.29:0/3518317135 >> 172.21.15.29:0/3518317135 conn(0x7fe73c067610 msgr2=0x7fe73c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 -- 172.21.15.29:0/3518317135 shutdown_connections 2023-10-07T09:30:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.566+0000 7fe743fff700 1 -- 172.21.15.29:0/3518317135 wait complete. 2023-10-07T09:30:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.568+0000 7fe743fff700 1 Processor -- start 2023-10-07T09:30:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.568+0000 7fe743fff700 1 -- start start 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe743fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c0ae8a0 0x7fe73c14c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe743fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 0x7fe73c167a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe743fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe73c15ad30 0x7fe73c14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe743fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c14dc90 con 0x7fe73c15a960 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe743fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c1682b0 con 0x7fe73c15ad30 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe743fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c168590 con 0x7fe73c0ae8a0 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe74159a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 0x7fe73c167a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe74159a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 0x7fe73c167a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48566/0 (socket says 172.21.15.29:48566) 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe74159a700 1 -- 172.21.15.29:0/2381914249 learned_addr learned my addr 172.21.15.29:0/2381914249 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe741d9b700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c0ae8a0 0x7fe73c14c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.569+0000 7fe74259c700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe73c15ad30 0x7fe73c14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe74159a700 1 -- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c0ae8a0 msgr2=0x7fe73c14c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe74159a700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c0ae8a0 0x7fe73c14c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe74159a700 1 -- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe73c15ad30 msgr2=0x7fe73c14fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe74159a700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe73c15ad30 0x7fe73c14fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe74159a700 1 -- 172.21.15.29:0/2381914249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738009420 con 0x7fe73c15a960 2023-10-07T09:30:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe74159a700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 0x7fe73c167a10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00c580 tx=0x7fe72c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c010510 con 0x7fe73c15a960 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c010670 con 0x7fe73c15a960 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73c168870 con 0x7fe73c15a960 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c010a30 con 0x7fe73c15a960 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.570+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73c14c5f0 con 0x7fe73c15a960 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.571+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73c14de40 con 0x7fe73c15a960 2023-10-07T09:30:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.572+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe72c01b400 con 0x7fe73c15a960 2023-10-07T09:30:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.573+0000 7fe731ffb700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe720066f30 0x7fe7200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.573+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fe72c084190 con 0x7fe73c15a960 2023-10-07T09:30:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.574+0000 7fe741d9b700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe720066f30 0x7fe7200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.574+0000 7fe741d9b700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe720066f30 0x7fe7200693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe738009860 tx=0x7fe738017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.577+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe73c14de40 con 0x7fe73c15a960 2023-10-07T09:30:13.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:13.824+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe73c04c3c0 con 0x7fe73c15a960 2023-10-07T09:30:14.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.345+0000 7fe731ffb700 1 -- 172.21.15.29:0/2381914249 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe72c051290 con 0x7fe73c15a960 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe720066f30 msgr2=0x7fe7200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe720066f30 0x7fe7200693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe738009860 tx=0x7fe738017000 comp rx=0 tx=0).stop 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 msgr2=0x7fe73c167a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 0x7fe73c167a10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00c580 tx=0x7fe72c007e10 comp rx=0 tx=0).stop 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 shutdown_connections 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe720066f30 0x7fe7200693e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe73c15ad30 0x7fe73c14fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe73c0ae8a0 0x7fe73c14c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 --2- 172.21.15.29:0/2381914249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe73c15a960 0x7fe73c167a10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.348+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 >> 172.21.15.29:0/2381914249 conn(0x7fe73c067610 msgr2=0x7fe73c14cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.349+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 shutdown_connections 2023-10-07T09:30:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.349+0000 7fe743fff700 1 -- 172.21.15.29:0/2381914249 wait complete. 2023-10-07T09:30:14.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:30:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.607+0000 7f898bdff700 1 Processor -- start 2023-10-07T09:30:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.608+0000 7f898bdff700 1 -- start start 2023-10-07T09:30:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f898bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f8984164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f898bdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89840cc940 con 0x7f89840b2a20 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f898bdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89840b2df0 con 0x7f89840cb8c0 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f898bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840cbd30 con 0x7f89840cf2c0 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f8989b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f89840b2a20 0x7f89840cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47492/0 (socket says 172.21.15.29:47492) 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f8989b9b700 1 -- 172.21.15.29:0/836275225 learned_addr learned my addr 172.21.15.29:0/836275225 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.609+0000 7f898a39c700 1 --2- 172.21.15.29:0/836275225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f8984164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174922394 0 0) 0x7f89840cc940 con 0x7f89840b2a20 2023-10-07T09:30:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8970003570 con 0x7f89840b2a20 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703129723 0 0) 0x7f89840b2df0 con 0x7f89840cb8c0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89840cc940 con 0x7f89840cb8c0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f898a39c700 1 -- 172.21.15.29:0/836275225 >> v1:172.21.15.144:6789/0 conn(0x7f89840cb8c0 legacy=0x7f89840cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f898a39c700 1 -- 172.21.15.29:0/836275225 >> v1:172.21.15.29:6789/0 conn(0x7f89840b2a20 legacy=0x7f89840cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f898a39c700 1 -- 172.21.15.29:0/836275225 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840cc010 con 0x7f89840cf2c0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.610+0000 7f898a39c700 1 --2- 172.21.15.29:0/836275225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f8984164330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f897c009e90 tx=0x7f897c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8b89fd13f28dfab server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c00e070 con 0x7f89840cf2c0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897c00bb80 con 0x7f89840cf2c0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f8988b99700 1 -- 172.21.15.29:0/836275225 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c012430 con 0x7f89840cf2c0 2023-10-07T09:30:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 -- 172.21.15.29:0/836275225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 msgr2=0x7f8984164330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 --2- 172.21.15.29:0/836275225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f8984164330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f897c009e90 tx=0x7f897c009780 comp rx=0 tx=0).stop 2023-10-07T09:30:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 -- 172.21.15.29:0/836275225 shutdown_connections 2023-10-07T09:30:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 --2- 172.21.15.29:0/836275225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f8984164330 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 -- 172.21.15.29:0/836275225 >> 172.21.15.29:0/836275225 conn(0x7f8984099820 msgr2=0x7f8984063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 -- 172.21.15.29:0/836275225 shutdown_connections 2023-10-07T09:30:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.611+0000 7f898bdff700 1 -- 172.21.15.29:0/836275225 wait complete. 2023-10-07T09:30:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.613+0000 7f898bdff700 1 Processor -- start 2023-10-07T09:30:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.614+0000 7f898bdff700 1 -- start start 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898bdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89840b2a20 0x7f8984142e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898bdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 0x7f89840c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f89840c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898bdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840ca310 con 0x7f89840cb8c0 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898bdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840caad0 con 0x7f89840b2a20 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840ca510 con 0x7f89840cf2c0 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898939a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 0x7f89840c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f8989b9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89840b2a20 0x7f8984142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898939a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 0x7f89840c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48582/0 (socket says 172.21.15.29:48582) 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898939a700 1 -- 172.21.15.29:0/3406781996 learned_addr learned my addr 172.21.15.29:0/3406781996 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.615+0000 7f898a39c700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f89840c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898939a700 1 -- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 msgr2=0x7f89840c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898939a700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f89840c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898939a700 1 -- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89840b2a20 msgr2=0x7f8984142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898939a700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89840b2a20 0x7f8984142e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898939a700 1 -- 172.21.15.29:0/3406781996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c009a20 con 0x7f89840cb8c0 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898939a700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 0x7f89840c9890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f897400c590 tx=0x7f8974007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974011590 con 0x7f89840cb8c0 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89740116f0 con 0x7f89840cb8c0 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89840ca7f0 con 0x7f89840cb8c0 2023-10-07T09:30:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974011ab0 con 0x7f89840cb8c0 2023-10-07T09:30:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.616+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89840b3760 con 0x7f89840cb8c0 2023-10-07T09:30:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.617+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89840b3980 con 0x7f89840cb8c0 2023-10-07T09:30:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.618+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f897401c460 con 0x7f89840cb8c0 2023-10-07T09:30:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.619+0000 7f8979ffb700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8968066f30 0x7f89680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.619+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8974084210 con 0x7f89840cb8c0 2023-10-07T09:30:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.619+0000 7f8989b9b700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8968066f30 0x7f89680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.620+0000 7f8989b9b700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8968066f30 0x7f89680693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8980001360 tx=0x7f8980009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.623+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8974051310 con 0x7f89840cb8c0 2023-10-07T09:30:14.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:14.825+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f89840718b0 con 0x7f89840cb8c0 2023-10-07T09:30:15.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.348+0000 7f8979ffb700 1 -- 172.21.15.29:0/3406781996 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8974054930 con 0x7f89840cb8c0 2023-10-07T09:30:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8968066f30 msgr2=0x7f89680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8968066f30 0x7f89680693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8980001360 tx=0x7f8980009040 comp rx=0 tx=0).stop 2023-10-07T09:30:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 msgr2=0x7f89840c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 0x7f89840c9890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f897400c590 tx=0x7f8974007d60 comp rx=0 tx=0).stop 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 shutdown_connections 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8968066f30 0x7f89680693e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89840cf2c0 0x7f89840c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89840b2a20 0x7f8984142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 --2- 172.21.15.29:0/3406781996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89840cb8c0 0x7f89840c9890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 >> 172.21.15.29:0/3406781996 conn(0x7f8984099820 msgr2=0x7f8984157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 shutdown_connections 2023-10-07T09:30:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.352+0000 7f898bdff700 1 -- 172.21.15.29:0/3406781996 wait complete. 2023-10-07T09:30:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:30:15.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:30:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.573+0000 7f242e9ff700 1 Processor -- start 2023-10-07T09:30:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.573+0000 7f242e9ff700 1 -- start start 2023-10-07T09:30:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.575+0000 7f242e9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280b0610 0x7f24280ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.575+0000 7f242e9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280690f0 con 0x7f24280b09e0 2023-10-07T09:30:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.575+0000 7f242e9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280cf2e0 con 0x7f24280c8a20 2023-10-07T09:30:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.575+0000 7f242e9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280687c0 con 0x7f24280b0610 2023-10-07T09:30:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f24277fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f24280b09e0 0x7f24280c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47504/0 (socket says 172.21.15.29:47504) 2023-10-07T09:30:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f24277fe700 1 -- 172.21.15.29:0/291224679 learned_addr learned my addr 172.21.15.29:0/291224679 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2427fff700 1 --2- 172.21.15.29:0/291224679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280b0610 0x7f24280ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571857945 0 0) 0x7f24280690f0 con 0x7f24280b09e0 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2410003570 con 0x7f24280b09e0 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290480591 0 0) 0x7f24280cf2e0 con 0x7f24280c8a20 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24280690f0 con 0x7f24280c8a20 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3915244194 0 0) 0x7f2410003570 con 0x7f24280b09e0 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2427fff700 1 -- 172.21.15.29:0/291224679 >> v1:172.21.15.144:6789/0 conn(0x7f24280c8a20 legacy=0x7f24280cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2427fff700 1 -- 172.21.15.29:0/291224679 >> v1:172.21.15.29:6789/0 conn(0x7f24280b09e0 legacy=0x7f24280c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.576+0000 7f2427fff700 1 -- 172.21.15.29:0/291224679 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428068aa0 con 0x7f24280b0610 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.577+0000 7f2427fff700 1 --2- 172.21.15.29:0/291224679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280b0610 0x7f24280ced00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2418009910 tx=0x7f24180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8966c7f1d008b8ca server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.577+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c001470 con 0x7f24280b0610 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.577+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418011420 con 0x7f24280b0610 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.577+0000 7f2426ffd700 1 -- 172.21.15.29:0/291224679 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24180117c0 con 0x7f24280b0610 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.577+0000 7f242e9ff700 1 -- 172.21.15.29:0/291224679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280b0610 msgr2=0x7f24280ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.577+0000 7f242e9ff700 1 --2- 172.21.15.29:0/291224679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280b0610 0x7f24280ced00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2418009910 tx=0x7f24180091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.578+0000 7f242e9ff700 1 -- 172.21.15.29:0/291224679 shutdown_connections 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.578+0000 7f242e9ff700 1 --2- 172.21.15.29:0/291224679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280b0610 0x7f24280ced00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.578+0000 7f242e9ff700 1 -- 172.21.15.29:0/291224679 >> 172.21.15.29:0/291224679 conn(0x7f2428063a10 msgr2=0x7f24280ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:15.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.578+0000 7f242e9ff700 1 -- 172.21.15.29:0/291224679 shutdown_connections 2023-10-07T09:30:15.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.578+0000 7f242e9ff700 1 -- 172.21.15.29:0/291224679 wait complete. 2023-10-07T09:30:15.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.580+0000 7f242e9ff700 1 Processor -- start 2023-10-07T09:30:15.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.580+0000 7f242e9ff700 1 -- start start 2023-10-07T09:30:15.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f242e9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 0x7f2428156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f242e9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24280b09e0 0x7f24280c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f242e9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280c8a20 0x7f24280c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f242e9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2428098f40 con 0x7f24280b0610 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f242e9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b1fe0 con 0x7f24280b09e0 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f242e9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b22c0 con 0x7f24280c8a20 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.581+0000 7f2427fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 0x7f2428156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 0x7f2428156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48594/0 (socket says 172.21.15.29:48594) 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 -- 172.21.15.29:0/3926600013 learned_addr learned my addr 172.21.15.29:0/3926600013 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:15.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f242cf9c700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280c8a20 0x7f24280c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f24277fe700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24280b09e0 0x7f24280c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 -- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280c8a20 msgr2=0x7f24280c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280c8a20 0x7f24280c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 -- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24280b09e0 msgr2=0x7f24280c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24280b09e0 0x7f24280c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 -- 172.21.15.29:0/3926600013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2418009420 con 0x7f24280b0610 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f24277fe700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24280b09e0 0x7f24280c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f2427fff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 0x7f2428156600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f24180091e0 tx=0x7f2418016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241801cbe0 con 0x7f24280b0610 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241801cd40 con 0x7f24280b0610 2023-10-07T09:30:15.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.582+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241801a680 con 0x7f24280b0610 2023-10-07T09:30:15.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.583+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24280b25a0 con 0x7f24280b0610 2023-10-07T09:30:15.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.583+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2428156b40 con 0x7f24280b0610 2023-10-07T09:30:15.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.584+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24280718b0 con 0x7f24280b0610 2023-10-07T09:30:15.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.584+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f241801aab0 con 0x7f24280b0610 2023-10-07T09:30:15.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.585+0000 7f240ffff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2414066f30 0x7f24140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:15.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.585+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5469+0+0 (secure 0 0 0) 0x7f2418086310 con 0x7f24280b0610 2023-10-07T09:30:15.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.586+0000 7f24277fe700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2414066f30 0x7f24140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:15.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.587+0000 7f24277fe700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2414066f30 0x7f24140693e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f241c0011e0 tx=0x7f241c000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:15.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.590+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24180535d0 con 0x7f24280b0610 2023-10-07T09:30:15.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:15.793+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2428068d80 con 0x7f24280b0610 2023-10-07T09:30:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.354+0000 7f240ffff700 1 -- 172.21.15.29:0/3926600013 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2418056bf0 con 0x7f24280b0610 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.358+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2414066f30 msgr2=0x7f24140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.358+0000 7f242e9ff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2414066f30 0x7f24140693e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f241c0011e0 tx=0x7f241c000bc0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.358+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 msgr2=0x7f2428156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.358+0000 7f242e9ff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 0x7f2428156600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f24180091e0 tx=0x7f2418016000 comp rx=0 tx=0).stop 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.359+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 shutdown_connections 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.359+0000 7f242e9ff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2414066f30 0x7f24140693e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.359+0000 7f242e9ff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24280c8a20 0x7f24280c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.359+0000 7f242e9ff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24280b0610 0x7f2428156600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.359+0000 7f242e9ff700 1 --2- 172.21.15.29:0/3926600013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24280b09e0 0x7f24280c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.359+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 >> 172.21.15.29:0/3926600013 conn(0x7f2428063a10 msgr2=0x7f242815f140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.360+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 shutdown_connections 2023-10-07T09:30:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.360+0000 7f242e9ff700 1 -- 172.21.15.29:0/3926600013 wait complete. 2023-10-07T09:30:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:30:16.382 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:30:16.382 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:30:16.382 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:30:16.382 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:30:16.382 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:30:16.382 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:30:16.383 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:30:16.383 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:30:16.383 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:30:16.383 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:30:16.383 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:30:16.383 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:30:16.383 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:30:16.384 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:30:16.384 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:30:16.384 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:30:16.397 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:30:16.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:30:16.398 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:30:16.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:30:16.403 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:30:16.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:30:16.404 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:30:16.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:30:16.442 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:30:16.442 DEBUG:teuthology.parallel:result is None 2023-10-07T09:30:16.445 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:30:16.445 DEBUG:teuthology.parallel:result is None 2023-10-07T09:30:16.449 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:30:16.449 DEBUG:teuthology.parallel:result is None 2023-10-07T09:30:16.451 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:30:16.452 DEBUG:teuthology.parallel:result is None 2023-10-07T09:30:16.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:30:16.591 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:30:16.592 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:30:16.596 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:30:16.597 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:30:16.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.661+0000 7fa99e7ff700 1 Processor -- start 2023-10-07T09:30:16.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.661+0000 7fa99e7ff700 1 -- start start 2023-10-07T09:30:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9980ad370 0x7fa9980cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99e7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa998159ec0 0x7fa99815a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99e7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99814d790 con 0x7fa9981568c0 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99e7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9980ae5d0 con 0x7fa998159ec0 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99815a850 con 0x7fa9980ad370 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99cd9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa9981568c0 0x7fa998156d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47516/0 (socket says 172.21.15.29:47516) 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.662+0000 7fa99cd9c700 1 -- 172.21.15.29:0/2520333381 learned_addr learned my addr 172.21.15.29:0/2520333381 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa9977fe700 1 --2- 172.21.15.29:0/2520333381 >> v2:172.21.15.144:3300/0 conn(0x7fa998159ec0 0x7fa99815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa997fff700 1 --2- 172.21.15.29:0/2520333381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9980ad370 0x7fa9980cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:16.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa996ffd700 1 -- 172.21.15.29:0/2520333381 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785920613 0 0) 0x7fa99814d790 con 0x7fa9981568c0 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa996ffd700 1 -- 172.21.15.29:0/2520333381 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa980003570 con 0x7fa9981568c0 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa997fff700 1 -- 172.21.15.29:0/2520333381 >> v2:172.21.15.144:3300/0 conn(0x7fa998159ec0 msgr2=0x7fa99815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa997fff700 1 --2- 172.21.15.29:0/2520333381 >> v2:172.21.15.144:3300/0 conn(0x7fa998159ec0 0x7fa99815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa997fff700 1 -- 172.21.15.29:0/2520333381 >> v1:172.21.15.29:6789/0 conn(0x7fa9981568c0 legacy=0x7fa998156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa997fff700 1 -- 172.21.15.29:0/2520333381 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980ad740 con 0x7fa9980ad370 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa9977fe700 1 --2- 172.21.15.29:0/2520333381 >> v2:172.21.15.144:3300/0 conn(0x7fa998159ec0 0x7fa99815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.663+0000 7fa997fff700 1 --2- 172.21.15.29:0/2520333381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9980ad370 0x7fa9980cc870 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa9880098e0 tx=0x7fa9880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16f89a2bdaf684 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa996ffd700 1 -- 172.21.15.29:0/2520333381 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988011630 con 0x7fa9980ad370 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa996ffd700 1 -- 172.21.15.29:0/2520333381 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988011790 con 0x7fa9980ad370 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa996ffd700 1 -- 172.21.15.29:0/2520333381 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988011b30 con 0x7fa9980ad370 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 -- 172.21.15.29:0/2520333381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9980ad370 msgr2=0x7fa9980cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/2520333381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9980ad370 0x7fa9980cc870 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa9880098e0 tx=0x7fa9880091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 -- 172.21.15.29:0/2520333381 shutdown_connections 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/2520333381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9980ad370 0x7fa9980cc870 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/2520333381 >> v2:172.21.15.144:3300/0 conn(0x7fa998159ec0 0x7fa99815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 -- 172.21.15.29:0/2520333381 >> 172.21.15.29:0/2520333381 conn(0x7fa9980680a0 msgr2=0x7fa9980cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 -- 172.21.15.29:0/2520333381 shutdown_connections 2023-10-07T09:30:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.664+0000 7fa99e7ff700 1 -- 172.21.15.29:0/2520333381 wait complete. 2023-10-07T09:30:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.666+0000 7fa99e7ff700 1 Processor -- start 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.666+0000 7fa99e7ff700 1 -- start start 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 0x7fa99814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9981568c0 0x7fa99814fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99e7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa998159ec0 0x7fa9980cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa998169110 con 0x7fa9980ad370 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa998150ed0 con 0x7fa998159ec0 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa998168500 con 0x7fa9981568c0 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa997fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 0x7fa99814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa997fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 0x7fa99814c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48620/0 (socket says 172.21.15.29:48620) 2023-10-07T09:30:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa997fff700 1 -- 172.21.15.29:0/1194943058 learned_addr learned my addr 172.21.15.29:0/1194943058 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa9977fe700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9981568c0 0x7fa99814fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.667+0000 7fa99cd9c700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa998159ec0 0x7fa9980cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa997fff700 1 -- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9981568c0 msgr2=0x7fa99814fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa997fff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9981568c0 0x7fa99814fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa997fff700 1 -- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa998159ec0 msgr2=0x7fa9980cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa997fff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa998159ec0 0x7fa9980cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa997fff700 1 -- 172.21.15.29:0/1194943058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c009420 con 0x7fa9980ad370 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa997fff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 0x7fa99814c250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa99815a990 tx=0x7fa988017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988000d10 con 0x7fa9980ad370 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988011b10 con 0x7fa9980ad370 2023-10-07T09:30:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa988009420 con 0x7fa9980ad370 2023-10-07T09:30:16.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.668+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988025660 con 0x7fa9980ad370 2023-10-07T09:30:16.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.669+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa998168ac0 con 0x7fa9980ad370 2023-10-07T09:30:16.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.670+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9981694a0 con 0x7fa9980ad370 2023-10-07T09:30:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.670+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa988025a50 con 0x7fa9980ad370 2023-10-07T09:30:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.671+0000 7fa97ffff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa984066f30 0x7fa9840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.671+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4939+0+0 (secure 0 0 0) 0x7fa988085a20 con 0x7fa9980ad370 2023-10-07T09:30:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.672+0000 7fa9977fe700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa984066f30 0x7fa9840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.672+0000 7fa9977fe700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa984066f30 0x7fa9840693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa998150df0 tx=0x7fa98c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.676+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9981694a0 con 0x7fa9980ad370 2023-10-07T09:30:16.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:16.894+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa9981694a0 con 0x7fa9980ad370 2023-10-07T09:30:17.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:30:17.687 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:30:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.955+0000 7fa97ffff700 1 -- 172.21.15.29:0/1194943058 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa9981694a0 con 0x7fa9980ad370 2023-10-07T09:30:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.957+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa984066f30 msgr2=0x7fa9840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.957+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa984066f30 0x7fa9840693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa998150df0 tx=0x7fa98c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.957+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 msgr2=0x7fa99814c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.957+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 0x7fa99814c250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa99815a990 tx=0x7fa988017000 comp rx=0 tx=0).stop 2023-10-07T09:30:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 shutdown_connections 2023-10-07T09:30:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa984066f30 0x7fa9840693e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa998159ec0 0x7fa9980cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9980ad370 0x7fa99814c250 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 --2- 172.21.15.29:0/1194943058 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9981568c0 0x7fa99814fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 >> 172.21.15.29:0/1194943058 conn(0x7fa9980680a0 msgr2=0x7fa9980af400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 shutdown_connections 2023-10-07T09:30:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:17.958+0000 7fa99e7ff700 1 -- 172.21.15.29:0/1194943058 wait complete. 2023-10-07T09:30:17.962 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:30:17.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:30:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.175+0000 7fd2799ff700 1 Processor -- start 2023-10-07T09:30:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.175+0000 7fd2799ff700 1 -- start start 2023-10-07T09:30:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.176+0000 7fd2799ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740afd20 0x7fd2741596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.176+0000 7fd2799ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd2740b1250 0x7fd2740acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.176+0000 7fd2799ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2740b0dc0 con 0x7fd274159320 2023-10-07T09:30:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.176+0000 7fd2799ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2740cd0d0 con 0x7fd2740b1250 2023-10-07T09:30:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.176+0000 7fd2799ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740b01b0 con 0x7fd2740afd20 2023-10-07T09:30:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.176+0000 7fd272ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd274159320 0x7fd2740cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47528/0 (socket says 172.21.15.29:47528) 2023-10-07T09:30:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd272ffd700 1 -- 172.21.15.29:0/2323324040 learned_addr learned my addr 172.21.15.29:0/2323324040 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2737fe700 1 --2- 172.21.15.29:0/2323324040 >> v2:172.21.15.144:3300/0 conn(0x7fd2740b1250 0x7fd2740acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2727fc700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740afd20 0x7fd2741596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd271ffb700 1 -- 172.21.15.29:0/2323324040 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363354717 0 0) 0x7fd2740b0dc0 con 0x7fd274159320 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd271ffb700 1 -- 172.21.15.29:0/2323324040 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c003570 con 0x7fd274159320 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd271ffb700 1 -- 172.21.15.29:0/2323324040 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292448721 0 0) 0x7fd25c003570 con 0x7fd274159320 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2737fe700 1 -- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740afd20 msgr2=0x7fd2741596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2737fe700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740afd20 0x7fd2741596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2737fe700 1 -- 172.21.15.29:0/2323324040 >> v1:172.21.15.29:6789/0 conn(0x7fd274159320 legacy=0x7fd2740cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2737fe700 1 -- 172.21.15.29:0/2323324040 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740b0490 con 0x7fd2740b1250 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.177+0000 7fd2727fc700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740afd20 0x7fd2741596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.178+0000 7fd2737fe700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2740b1250 0x7fd2740acd80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009e60 tx=0x7fd26c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2b4ef93fee96567 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.178+0000 7fd271ffb700 1 -- 172.21.15.29:0/2323324040 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c00e070 con 0x7fd2740b1250 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.178+0000 7fd271ffb700 1 -- 172.21.15.29:0/2323324040 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c00b830 con 0x7fd2740b1250 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.178+0000 7fd271ffb700 1 -- 172.21.15.29:0/2323324040 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264001470 con 0x7fd2740b1250 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 -- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2740b1250 msgr2=0x7fd2740acd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2740b1250 0x7fd2740acd80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009e60 tx=0x7fd26c009780 comp rx=0 tx=0).stop 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 -- 172.21.15.29:0/2323324040 shutdown_connections 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2740b1250 0x7fd2740acd80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 --2- 172.21.15.29:0/2323324040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740afd20 0x7fd2741596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 -- 172.21.15.29:0/2323324040 >> 172.21.15.29:0/2323324040 conn(0x7fd2740680a0 msgr2=0x7fd2740cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 -- 172.21.15.29:0/2323324040 shutdown_connections 2023-10-07T09:30:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.179+0000 7fd2799ff700 1 -- 172.21.15.29:0/2323324040 wait complete. 2023-10-07T09:30:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.181+0000 7fd2799ff700 1 Processor -- start 2023-10-07T09:30:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.181+0000 7fd2799ff700 1 -- start start 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.182+0000 7fd2799ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 0x7fd2740c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.182+0000 7fd2799ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b1250 0x7fd274153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.182+0000 7fd2799ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd274159320 0x7fd2740c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.182+0000 7fd2799ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c4420 con 0x7fd2740afd20 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.182+0000 7fd2799ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c3810 con 0x7fd274159320 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.182+0000 7fd2799ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c3af0 con 0x7fd2740b1250 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 0x7fd2740c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 0x7fd2740c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48630/0 (socket says 172.21.15.29:48630) 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 -- 172.21.15.29:0/1017019134 learned_addr learned my addr 172.21.15.29:0/1017019134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd2737fe700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd274159320 0x7fd2740c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd2727fc700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b1250 0x7fd274153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 -- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b1250 msgr2=0x7fd274153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b1250 0x7fd274153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 -- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd274159320 msgr2=0x7fd2740c48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd274159320 0x7fd2740c48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 -- 172.21.15.29:0/1017019134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268009420 con 0x7fd2740afd20 2023-10-07T09:30:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.183+0000 7fd272ffd700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 0x7fd2740c2950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd26400c7f0 tx=0x7fd2640019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.184+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264007360 con 0x7fd2740afd20 2023-10-07T09:30:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.184+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2640074c0 con 0x7fd2740afd20 2023-10-07T09:30:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.184+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c009a20 con 0x7fd2740afd20 2023-10-07T09:30:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.184+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264004830 con 0x7fd2740afd20 2023-10-07T09:30:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.184+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2740c40b0 con 0x7fd2740afd20 2023-10-07T09:30:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.185+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2740c4700 con 0x7fd2740afd20 2023-10-07T09:30:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.186+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd264004c20 con 0x7fd2740afd20 2023-10-07T09:30:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.186+0000 7fd25affd700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd260066f30 0x7fd2600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.186+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fd264083510 con 0x7fd2740afd20 2023-10-07T09:30:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.187+0000 7fd2727fc700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd260066f30 0x7fd2600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.188+0000 7fd2727fc700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd260066f30 0x7fd2600693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2741580a0 tx=0x7fd2680091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.191+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd264050a30 con 0x7fd2740afd20 2023-10-07T09:30:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.467+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd27404c3c0 con 0x7fd2740afd20 2023-10-07T09:30:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.468+0000 7fd25affd700 1 -- 172.21.15.29:0/1017019134 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fd264018090 con 0x7fd2740afd20 2023-10-07T09:30:18.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:18.469 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:30:18.469 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:30:18.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:30:18.470 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.471 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.472 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:30:18.473 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.474 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:30:18.476 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:30:18.477 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.479 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:30:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:30:18.482 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:30:18.483 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-mds", 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDZJCFlTaQQBxAAN4E/zSIB543WGT0H8lEYNg==", 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-mds" 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-mgr", 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDZJCFl1bAQBxAAjU+/+3k5OO/8UeoOULKyig==", 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-osd", 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDZJCFlnroQBxAAlDtjMTg70b+KRRJbFolneA==", 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-osd" 2023-10-07T09:30:18.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rbd", 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDZJCFlScMQBxAA8P3U6+dW4TpgzXzE/peIcA==", 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDZJCFl78sQBxAA4oC6DQclEUcgkdebifkFnQ==", 2023-10-07T09:30:18.486 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rgw", 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDZJCFllNQQBxAAKOrSsz0EUBPYn3FQxAgVmg==", 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.487 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:30:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:30:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:30:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:30:18.489 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:30:18.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd260066f30 msgr2=0x7fd2600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd260066f30 0x7fd2600693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2741580a0 tx=0x7fd2680091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 msgr2=0x7fd2740c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 0x7fd2740c2950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd26400c7f0 tx=0x7fd2640019e0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 shutdown_connections 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd260066f30 0x7fd2600693e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd274159320 0x7fd2740c48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2740afd20 0x7fd2740c2950 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 --2- 172.21.15.29:0/1017019134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b1250 0x7fd274153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 >> 172.21.15.29:0/1017019134 conn(0x7fd2740680a0 msgr2=0x7fd27406de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 shutdown_connections 2023-10-07T09:30:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.471+0000 7fd2799ff700 1 -- 172.21.15.29:0/1017019134 wait complete. 2023-10-07T09:30:18.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-07T09:30:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.687+0000 7fc5d1fff700 1 Processor -- start 2023-10-07T09:30:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.687+0000 7fc5d1fff700 1 -- start start 2023-10-07T09:30:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.688+0000 7fc5d1fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 0x7fc5cc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.688+0000 7fc5d1fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0cefb0 con 0x7fc5cc159b20 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.688+0000 7fc5d1fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0cbf20 con 0x7fc5cc159750 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.688+0000 7fc5d1fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0cc200 con 0x7fc5cc0c6470 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5caffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5cc159b20 0x7fc5cc0ce8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47534/0 (socket says 172.21.15.29:47534) 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5caffd700 1 -- 172.21.15.29:0/2575729295 learned_addr learned my addr 172.21.15.29:0/2575729295 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5cbfff700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 0x7fc5cc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967498530 0 0) 0x7fc5cc0cefb0 con 0x7fc5cc159b20 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b4003570 con 0x7fc5cc159b20 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342302936 0 0) 0x7fc5cc0cbf20 con 0x7fc5cc159750 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5cc0cefb0 con 0x7fc5cc159750 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123540429 0 0) 0x7fc5b4003570 con 0x7fc5cc159b20 2023-10-07T09:30:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 msgr2=0x7fc5cc0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 0x7fc5cc0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 >> v1:172.21.15.144:6789/0 conn(0x7fc5cc159750 legacy=0x7fc5cc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0cc4e0 con 0x7fc5cc159b20 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.689+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fc5c0001470 con 0x7fc5cc159b20 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 >> v1:172.21.15.29:6789/0 conn(0x7fc5cc159b20 legacy=0x7fc5cc0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 0x7fc5b4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b4004180 0x7fc5b4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0cc200 con 0x7fc5b4004d50 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b400bba0 con 0x7fc5b4004180 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b400bd60 con 0x7fc5cc0c6470 2023-10-07T09:30:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cbfff700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cb7fe700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 0x7fc5b4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5caffd700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b4004180 0x7fc5b4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cbfff700 1 -- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 msgr2=0x7fc5b4003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cbfff700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc0c6470 0x7fc5b4003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cbfff700 1 -- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b4004180 msgr2=0x7fc5b4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cbfff700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b4004180 0x7fc5b4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.690+0000 7fc5cbfff700 1 -- 172.21.15.29:0/2575729295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b4003570 con 0x7fc5b4004d50 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5cbfff700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5b40051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400e210 tx=0x7fc5c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99294fe06a0cd25 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400eac0 con 0x7fc5b4004d50 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c400f040 con 0x7fc5b4004d50 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5ca7fc700 1 -- 172.21.15.29:0/2575729295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400ee70 con 0x7fc5b4004d50 2023-10-07T09:30:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2575729295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 msgr2=0x7fc5b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2575729295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5b40051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400e210 tx=0x7fc5c400b040 comp rx=0 tx=0).stop 2023-10-07T09:30:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5cbfff700 1 -- 172.21.15.29:0/2575729295 reap_dead start 2023-10-07T09:30:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2575729295 shutdown_connections 2023-10-07T09:30:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2575729295 >> 172.21.15.29:0/2575729295 conn(0x7fc5cc063a10 msgr2=0x7fc5cc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.691+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2575729295 shutdown_connections 2023-10-07T09:30:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.692+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2575729295 wait complete. 2023-10-07T09:30:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.693+0000 7fc5d1fff700 1 Processor -- start 2023-10-07T09:30:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.693+0000 7fc5d1fff700 1 -- start start 2023-10-07T09:30:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.694+0000 7fc5d1fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5cc151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.694+0000 7fc5d1fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc1551a0 0x7fc5cc159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.694+0000 7fc5d1fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5cc155610 0x7fc5cc15abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.694+0000 7fc5d1fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc154d10 con 0x7fc5b4004d50 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.694+0000 7fc5d1fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc153e60 con 0x7fc5cc155610 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.694+0000 7fc5d1fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0b5ed0 con 0x7fc5cc1551a0 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5cc151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5cc151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48640/0 (socket says 172.21.15.29:48640) 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 -- 172.21.15.29:0/2844235029 learned_addr learned my addr 172.21.15.29:0/2844235029 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5caffd700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc1551a0 0x7fc5cc159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cbfff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5cc155610 0x7fc5cc15abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 -- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc1551a0 msgr2=0x7fc5cc159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc1551a0 0x7fc5cc159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 -- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5cc155610 msgr2=0x7fc5cc15abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5cc155610 0x7fc5cc15abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 -- 172.21.15.29:0/2844235029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc154100 con 0x7fc5b4004d50 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5cb7fe700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5cc151a00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc007ad0 tx=0x7fc5bc007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00fe70 con 0x7fc5b4004d50 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc00d850 con 0x7fc5b4004d50 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.695+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00dbd0 con 0x7fc5b4004d50 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.696+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc1543e0 con 0x7fc5b4004d50 2023-10-07T09:30:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.696+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc1549a0 con 0x7fc5b4004d50 2023-10-07T09:30:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.697+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc154ff0 con 0x7fc5b4004d50 2023-10-07T09:30:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.698+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc5bc01c400 con 0x7fc5b4004d50 2023-10-07T09:30:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.699+0000 7fc5b37fe700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc5b8066f30 0x7fc5b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.699+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fc5bc0846d0 con 0x7fc5b4004d50 2023-10-07T09:30:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.699+0000 7fc5caffd700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc5b8066f30 0x7fc5b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.700+0000 7fc5caffd700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc5b8066f30 0x7fc5b80693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc068ff0 tx=0x7fc5c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.703+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5bc051d40 con 0x7fc5b4004d50 2023-10-07T09:30:18.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.987+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fc5cc04c3c0 con 0x7fc5b4004d50 2023-10-07T09:30:18.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.991+0000 7fc5b37fe700 1 -- 172.21.15.29:0/2844235029 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc5bc017020 con 0x7fc5b4004d50 2023-10-07T09:30:18.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc5b8066f30 msgr2=0x7fc5b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc5b8066f30 0x7fc5b80693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc068ff0 tx=0x7fc5c0009040 comp rx=0 tx=0).stop 2023-10-07T09:30:18.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 msgr2=0x7fc5cc151a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:18.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5cc151a00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc007ad0 tx=0x7fc5bc007d80 comp rx=0 tx=0).stop 2023-10-07T09:30:18.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 shutdown_connections 2023-10-07T09:30:18.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc5b8066f30 0x7fc5b80693e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5cc155610 0x7fc5cc15abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b4004d50 0x7fc5cc151a00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 --2- 172.21.15.29:0/2844235029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5cc1551a0 0x7fc5cc159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 >> 172.21.15.29:0/2844235029 conn(0x7fc5cc063a10 msgr2=0x7fc5cc155b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 shutdown_connections 2023-10-07T09:30:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:18.994+0000 7fc5d1fff700 1 -- 172.21.15.29:0/2844235029 wait complete. 2023-10-07T09:30:19.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-07T09:30:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.208+0000 7f8a20bff700 1 Processor -- start 2023-10-07T09:30:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.208+0000 7f8a20bff700 1 -- start start 2023-10-07T09:30:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.209+0000 7f8a20bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 0x7f8a1c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.209+0000 7f8a20bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a1c0ce8a0 0x7f8a1c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.209+0000 7f8a20bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0cbf20 con 0x7f8a1c159750 2023-10-07T09:30:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.209+0000 7f8a20bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cc200 con 0x7f8a1c0ce8a0 2023-10-07T09:30:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.209+0000 7f8a20bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cc4e0 con 0x7f8a1c159b20 2023-10-07T09:30:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1a59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8a1c159750 0x7f8a1c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47548/0 (socket says 172.21.15.29:47548) 2023-10-07T09:30:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1a59c700 1 -- 172.21.15.29:0/3256437613 learned_addr learned my addr 172.21.15.29:0/3256437613 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3256437613 >> v2:172.21.15.144:3300/0 conn(0x7f8a1c0ce8a0 0x7f8a1c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a19d9b700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 0x7f8a1c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659602629 0 0) 0x7f8a1c0cbf20 con 0x7f8a1c159750 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a04003570 con 0x7f8a1c159750 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2981964314 0 0) 0x7f8a04003570 con 0x7f8a1c159750 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 msgr2=0x7f8a1c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 0x7f8a1c0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 >> v2:172.21.15.144:3300/0 conn(0x7f8a1c0ce8a0 msgr2=0x7f8a1c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 --2- 172.21.15.29:0/3256437613 >> v2:172.21.15.144:3300/0 conn(0x7f8a1c0ce8a0 0x7f8a1c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0cc7c0 con 0x7f8a1c159750 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f8a0c0015f0 con 0x7f8a1c159750 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 >> v1:172.21.15.29:6789/0 conn(0x7f8a1c159750 legacy=0x7f8a1c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 0x7f8a04003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04003fe0 0x7f8a04004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a04004d50 0x7f8a040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cc200 con 0x7f8a04004d50 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cc4e0 con 0x7f8a04003fe0 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.210+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0400ba90 con 0x7f8a1c159b20 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a04004d50 0x7f8a040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a19d9b700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04003fe0 0x7f8a04004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1a59c700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 0x7f8a04003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 -- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 msgr2=0x7f8a04003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c159b20 0x7f8a04003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 -- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04003fe0 msgr2=0x7f8a04004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04003fe0 0x7f8a04004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 -- 172.21.15.29:0/3256437613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a04003570 con 0x7f8a04004d50 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.211+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a04004d50 0x7f8a040051c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e210 tx=0x7f8a1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e54eee13acad273 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1400ea60 con 0x7f8a04004d50 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1400f040 con 0x7f8a04004d50 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a1959a700 1 -- 172.21.15.29:0/3256437613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1400ee10 con 0x7f8a04004d50 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a20bff700 1 -- 172.21.15.29:0/3256437613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a04004d50 msgr2=0x7f8a040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3256437613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a04004d50 0x7f8a040051c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e210 tx=0x7f8a1400b040 comp rx=0 tx=0).stop 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a1ad9d700 1 -- 172.21.15.29:0/3256437613 reap_dead start 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a20bff700 1 -- 172.21.15.29:0/3256437613 shutdown_connections 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a20bff700 1 -- 172.21.15.29:0/3256437613 >> 172.21.15.29:0/3256437613 conn(0x7f8a1c063a10 msgr2=0x7f8a1c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a20bff700 1 -- 172.21.15.29:0/3256437613 shutdown_connections 2023-10-07T09:30:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.212+0000 7f8a20bff700 1 -- 172.21.15.29:0/3256437613 wait complete. 2023-10-07T09:30:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.214+0000 7f8a20bff700 1 Processor -- start 2023-10-07T09:30:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.214+0000 7f8a20bff700 1 -- start start 2023-10-07T09:30:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a20bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04004d50 0x7f8a1c15d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a20bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 0x7f8a1c159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a20bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c154260 0x7f8a1c15a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a20bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c15dd40 con 0x7f8a1c153df0 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a20bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0c8c60 con 0x7f8a04004d50 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a20bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0c8f40 con 0x7f8a1c154260 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a19d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 0x7f8a1c159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a19d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 0x7f8a1c159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48664/0 (socket says 172.21.15.29:48664) 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a19d9b700 1 -- 172.21.15.29:0/3247154105 learned_addr learned my addr 172.21.15.29:0/3247154105 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a1a59c700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04004d50 0x7f8a1c15d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c154260 0x7f8a1c15a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a19d9b700 1 -- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c154260 msgr2=0x7f8a1c15a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a19d9b700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c154260 0x7f8a1c15a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.215+0000 7f8a19d9b700 1 -- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04004d50 msgr2=0x7f8a1c15d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a19d9b700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04004d50 0x7f8a1c15d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a19d9b700 1 -- 172.21.15.29:0/3247154105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0c9220 con 0x7f8a1c153df0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a1ad9d700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c154260 0x7f8a1c15a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a19d9b700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 0x7f8a1c159ef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006ac0 tx=0x7f8a100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1000f9d0 con 0x7f8a1c153df0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1000fb30 con 0x7f8a1c153df0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1c0c9500 con 0x7f8a1c153df0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1000feb0 con 0x7f8a1c153df0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.216+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1c0c9d00 con 0x7f8a1c153df0 2023-10-07T09:30:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.217+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1c04c3c0 con 0x7f8a1c153df0 2023-10-07T09:30:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.218+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8a10014650 con 0x7f8a1c153df0 2023-10-07T09:30:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.220+0000 7f8a0a7fc700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f89fc066f30 0x7f89fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.220+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f8a10018030 con 0x7f8a1c153df0 2023-10-07T09:30:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.221+0000 7f8a1a59c700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f89fc066f30 0x7f89fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.221+0000 7f8a1a59c700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f89fc066f30 0x7f89fc0693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c000ef0 tx=0x7f8a0c00c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.224+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a100522a0 con 0x7f8a1c153df0 2023-10-07T09:30:19.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.506+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f8a1c0d1000 con 0x7f8a1c153df0 2023-10-07T09:30:19.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.510+0000 7f8a0a7fc700 1 -- 172.21.15.29:0/3247154105 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8a100220a0 con 0x7f8a1c153df0 2023-10-07T09:30:19.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f89fc066f30 msgr2=0x7f89fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f89fc066f30 0x7f89fc0693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c000ef0 tx=0x7f8a0c00c040 comp rx=0 tx=0).stop 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 msgr2=0x7f8a1c159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 0x7f8a1c159ef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006ac0 tx=0x7f8a100091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 shutdown_connections 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f89fc066f30 0x7f89fc0693e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a1c154260 0x7f8a1c15a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a04004d50 0x7f8a1c15d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 --2- 172.21.15.29:0/3247154105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a1c153df0 0x7f8a1c159ef0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 >> 172.21.15.29:0/3247154105 conn(0x7f8a1c063a10 msgr2=0x7f8a1c157080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 shutdown_connections 2023-10-07T09:30:19.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.512+0000 7f8a20bff700 1 -- 172.21.15.29:0/3247154105 wait complete. 2023-10-07T09:30:19.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-07T09:30:19.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.726+0000 7f9bdefff700 1 Processor -- start 2023-10-07T09:30:19.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.726+0000 7f9bdefff700 1 -- start start 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdefff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9bd8159750 0x7f9bd80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdefff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159b20 0x7f9bd80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdefff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80cbf20 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdefff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80cc200 con 0x7f9bd8159750 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdefff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80cc4e0 con 0x7f9bd8159b20 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdd59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9bd80ce6d0 0x7f9bd815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47556/0 (socket says 172.21.15.29:47556) 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdd59c700 1 -- 172.21.15.29:0/4258901485 learned_addr learned my addr 172.21.15.29:0/4258901485 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:19.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bdcd9b700 1 --2- 172.21.15.29:0/4258901485 >> v2:172.21.15.144:3300/0 conn(0x7f9bd8159750 0x7f9bd80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.728+0000 7f9bd7fff700 1 --2- 172.21.15.29:0/4258901485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159b20 0x7f9bd80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd77fe700 1 -- 172.21.15.29:0/4258901485 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637767108 0 0) 0x7f9bd80cbf20 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd77fe700 1 -- 172.21.15.29:0/4258901485 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc8003570 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd7fff700 1 -- 172.21.15.29:0/4258901485 >> v2:172.21.15.144:3300/0 conn(0x7f9bd8159750 msgr2=0x7f9bd80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd7fff700 1 --2- 172.21.15.29:0/4258901485 >> v2:172.21.15.144:3300/0 conn(0x7f9bd8159750 0x7f9bd80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd7fff700 1 -- 172.21.15.29:0/4258901485 >> v1:172.21.15.29:6789/0 conn(0x7f9bd80ce6d0 legacy=0x7f9bd815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd7fff700 1 -- 172.21.15.29:0/4258901485 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80cc7c0 con 0x7f9bd8159b20 2023-10-07T09:30:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bdcd9b700 1 --2- 172.21.15.29:0/4258901485 >> v2:172.21.15.144:3300/0 conn(0x7f9bd8159750 0x7f9bd80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd7fff700 1 --2- 172.21.15.29:0/4258901485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159b20 0x7f9bd80ce190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40098e0 tx=0x7f9bc40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36919322c4602c64 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd77fe700 1 -- 172.21.15.29:0/4258901485 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4011630 con 0x7f9bd8159b20 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd77fe700 1 -- 172.21.15.29:0/4258901485 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc4011790 con 0x7f9bd8159b20 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.729+0000 7f9bd77fe700 1 -- 172.21.15.29:0/4258901485 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4011b30 con 0x7f9bd8159b20 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 -- 172.21.15.29:0/4258901485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159b20 msgr2=0x7f9bd80ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 --2- 172.21.15.29:0/4258901485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159b20 0x7f9bd80ce190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40098e0 tx=0x7f9bc40091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 -- 172.21.15.29:0/4258901485 shutdown_connections 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 --2- 172.21.15.29:0/4258901485 >> v2:172.21.15.144:3300/0 conn(0x7f9bd8159750 0x7f9bd80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 --2- 172.21.15.29:0/4258901485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159b20 0x7f9bd80ce190 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 -- 172.21.15.29:0/4258901485 >> 172.21.15.29:0/4258901485 conn(0x7f9bd8063a10 msgr2=0x7f9bd80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:19.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 -- 172.21.15.29:0/4258901485 shutdown_connections 2023-10-07T09:30:19.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.730+0000 7f9bdefff700 1 -- 172.21.15.29:0/4258901485 wait complete. 2023-10-07T09:30:19.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.732+0000 7f9bdefff700 1 Processor -- start 2023-10-07T09:30:19.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.732+0000 7f9bdefff700 1 -- start start 2023-10-07T09:30:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdefff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 0x7f9bd814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdefff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159750 0x7f9bd80b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdefff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bd8159b20 0x7f9bd80c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdefff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80c9be0 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdefff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd81573f0 con 0x7f9bd8159b20 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdefff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd81576d0 con 0x7f9bd8159750 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 0x7f9bd814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 0x7f9bd814c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48672/0 (socket says 172.21.15.29:48672) 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 -- 172.21.15.29:0/936841118 learned_addr learned my addr 172.21.15.29:0/936841118 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdd59c700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bd8159b20 0x7f9bd80c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bd7fff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159750 0x7f9bd80b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 -- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159750 msgr2=0x7f9bd80b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159750 0x7f9bd80b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 -- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bd8159b20 msgr2=0x7f9bd80c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bd8159b20 0x7f9bd80c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.733+0000 7f9bdcd9b700 1 -- 172.21.15.29:0/936841118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0093c0 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.734+0000 7f9bdcd9b700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 0x7f9bd814c350 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00ec80 tx=0x7f9bcc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.734+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc005a00 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.734+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc005b60 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.734+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00c850 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.734+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc4009420 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.734+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd8157c90 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.735+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8157e50 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.736+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9bcc00cbc0 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.737+0000 7f9bd4ff9700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9bbc066f30 0x7f9bbc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:19.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.737+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f9bcc084780 con 0x7f9bd80ce6d0 2023-10-07T09:30:19.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.738+0000 7f9bd7fff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9bbc066f30 0x7f9bbc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:19.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.738+0000 7f9bd7fff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9bbc066f30 0x7f9bbc0693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40091e0 tx=0x7f9bc4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:19.741+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9bcc055730 con 0x7f9bd80ce6d0 2023-10-07T09:30:20.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.024+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f9bd8158000 con 0x7f9bd80ce6d0 2023-10-07T09:30:20.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.027+0000 7f9bd4ff9700 1 -- 172.21.15.29:0/936841118 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9bcc0552c0 con 0x7f9bd80ce6d0 2023-10-07T09:30:20.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9bbc066f30 msgr2=0x7f9bbc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9bbc066f30 0x7f9bbc0693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40091e0 tx=0x7f9bc4017000 comp rx=0 tx=0).stop 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 msgr2=0x7f9bd814c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 0x7f9bd814c350 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00ec80 tx=0x7f9bcc009120 comp rx=0 tx=0).stop 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 shutdown_connections 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9bbc066f30 0x7f9bbc0693e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bd8159b20 0x7f9bd80c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd80ce6d0 0x7f9bd814c350 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 --2- 172.21.15.29:0/936841118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bd8159750 0x7f9bd80b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 >> 172.21.15.29:0/936841118 conn(0x7f9bd8063a10 msgr2=0x7f9bd80c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 shutdown_connections 2023-10-07T09:30:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.030+0000 7f9bdefff700 1 -- 172.21.15.29:0/936841118 wait complete. 2023-10-07T09:30:20.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-07T09:30:20.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.244+0000 7fd47dfff700 1 Processor -- start 2023-10-07T09:30:20.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.244+0000 7fd47dfff700 1 -- start start 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.245+0000 7fd47dfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4780b2670 0x7fd4780b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.245+0000 7fd47dfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47806f4e0 con 0x7fd4780a98d0 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.245+0000 7fd47dfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780b1120 con 0x7fd478159550 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.245+0000 7fd47dfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47806e8d0 con 0x7fd4780b2670 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.245+0000 7fd476ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4780b2670 0x7fd4780b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd477fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd4780a98d0 0x7fd4780a9d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47570/0 (socket says 172.21.15.29:47570) 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd477fff700 1 -- 172.21.15.29:0/2333308993 learned_addr learned my addr 172.21.15.29:0/2333308993 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd476ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4780b2670 0x7fd4780b2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48686/0 (socket says 172.21.15.29:48686) 2023-10-07T09:30:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd476ffd700 1 -- 172.21.15.29:0/2333308993 >> v1:172.21.15.144:6789/0 conn(0x7fd478159550 legacy=0x7fd4780b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd476ffd700 1 -- 172.21.15.29:0/2333308993 >> v1:172.21.15.29:6789/0 conn(0x7fd4780a98d0 legacy=0x7fd4780a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd476ffd700 1 -- 172.21.15.29:0/2333308993 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47806ebb0 con 0x7fd4780b2670 2023-10-07T09:30:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd4767fc700 1 -- 172.21.15.29:0/2333308993 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170341619 0 0) 0x7fd47806f4e0 con 0x7fd4780a98d0 2023-10-07T09:30:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.246+0000 7fd476ffd700 1 --2- 172.21.15.29:0/2333308993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780b2670 0x7fd4780b2ae0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0098e0 tx=0x7fd46c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15c61180a686af server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd4767fc700 1 -- 172.21.15.29:0/2333308993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c009e80 con 0x7fd4780b2670 2023-10-07T09:30:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd4767fc700 1 -- 172.21.15.29:0/2333308993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c00b860 con 0x7fd4780b2670 2023-10-07T09:30:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd4767fc700 1 -- 172.21.15.29:0/2333308993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00bbe0 con 0x7fd4780b2670 2023-10-07T09:30:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 -- 172.21.15.29:0/2333308993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780b2670 msgr2=0x7fd4780b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 --2- 172.21.15.29:0/2333308993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780b2670 0x7fd4780b2ae0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0098e0 tx=0x7fd46c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 -- 172.21.15.29:0/2333308993 shutdown_connections 2023-10-07T09:30:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 --2- 172.21.15.29:0/2333308993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780b2670 0x7fd4780b2ae0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 -- 172.21.15.29:0/2333308993 >> 172.21.15.29:0/2333308993 conn(0x7fd478069190 msgr2=0x7fd47806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 -- 172.21.15.29:0/2333308993 shutdown_connections 2023-10-07T09:30:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.247+0000 7fd47dfff700 1 -- 172.21.15.29:0/2333308993 wait complete. 2023-10-07T09:30:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.249+0000 7fd47dfff700 1 Processor -- start 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.249+0000 7fd47dfff700 1 -- start start 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd47dfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 0x7fd47815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd47dfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4780b2670 0x7fd47814cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd47dfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd478159550 0x7fd478158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd47dfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47814b640 con 0x7fd4780a98d0 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd47dfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd478158b40 con 0x7fd4780b2670 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd47dfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4781599c0 con 0x7fd478159550 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 0x7fd47815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 0x7fd47815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48704/0 (socket says 172.21.15.29:48704) 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd4777fe700 1 -- 172.21.15.29:0/364036203 learned_addr learned my addr 172.21.15.29:0/364036203 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd476ffd700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4780b2670 0x7fd47814cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.250+0000 7fd477fff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd478159550 0x7fd478158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd4777fe700 1 -- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd478159550 msgr2=0x7fd478158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd4777fe700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd478159550 0x7fd478158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd4777fe700 1 -- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4780b2670 msgr2=0x7fd47814cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd4777fe700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4780b2670 0x7fd47814cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd4777fe700 1 -- 172.21.15.29:0/364036203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c009420 con 0x7fd4780a98d0 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd477fff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd478159550 0x7fd478158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd4777fe700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 0x7fd47815aa60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd46800c590 tx=0x7fd46800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468011590 con 0x7fd4780a98d0 2023-10-07T09:30:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468011710 con 0x7fd4780a98d0 2023-10-07T09:30:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468011ad0 con 0x7fd4780a98d0 2023-10-07T09:30:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd478159ca0 con 0x7fd4780a98d0 2023-10-07T09:30:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.251+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47815a260 con 0x7fd4780a98d0 2023-10-07T09:30:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.252+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47804c3c0 con 0x7fd4780a98d0 2023-10-07T09:30:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.253+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd46801b3b0 con 0x7fd4780a98d0 2023-10-07T09:30:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.254+0000 7fd45f7fe700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd464066f30 0x7fd4640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.254+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fd4680837b0 con 0x7fd4780a98d0 2023-10-07T09:30:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.255+0000 7fd476ffd700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd464066f30 0x7fd4640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.255+0000 7fd476ffd700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd464066f30 0x7fd4640693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0093f0 tx=0x7fd46c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:20.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.258+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd468050cd0 con 0x7fd4780a98d0 2023-10-07T09:30:20.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.541+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fd478158d80 con 0x7fd4780a98d0 2023-10-07T09:30:20.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.545+0000 7fd45f7fe700 1 -- 172.21.15.29:0/364036203 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd468017020 con 0x7fd4780a98d0 2023-10-07T09:30:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.547+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd464066f30 msgr2=0x7fd4640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.547+0000 7fd47dfff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd464066f30 0x7fd4640693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0093f0 tx=0x7fd46c017000 comp rx=0 tx=0).stop 2023-10-07T09:30:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.547+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 msgr2=0x7fd47815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.547+0000 7fd47dfff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 0x7fd47815aa60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd46800c590 tx=0x7fd46800c8a0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 shutdown_connections 2023-10-07T09:30:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd464066f30 0x7fd4640693e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd478159550 0x7fd478158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4780a98d0 0x7fd47815aa60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 --2- 172.21.15.29:0/364036203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4780b2670 0x7fd47814cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 >> 172.21.15.29:0/364036203 conn(0x7fd478069190 msgr2=0x7fd4780aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:20.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 shutdown_connections 2023-10-07T09:30:20.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.548+0000 7fd47dfff700 1 -- 172.21.15.29:0/364036203 wait complete. 2023-10-07T09:30:20.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-07T09:30:20.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.761+0000 7f0f241ff700 1 Processor -- start 2023-10-07T09:30:20.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.761+0000 7f0f241ff700 1 -- start start 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.762+0000 7f0f241ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0ad370 0x7f0f1c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.762+0000 7f0f241ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f1c0b0410 0x7f0f1c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.762+0000 7f0f241ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c14d790 con 0x7f0f1c159ec0 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.762+0000 7f0f241ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c0ae5d0 con 0x7f0f1c0b0410 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.762+0000 7f0f241ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c15a850 con 0x7f0f1c0ad370 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f1c0b0410 0x7f0f1c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f1c0b0410 0x7f0f1c15a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48716/0 (socket says 172.21.15.29:48716) 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 -- 172.21.15.29:0/2023140046 learned_addr learned my addr 172.21.15.29:0/2023140046 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:20.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2179a700 1 --1- 172.21.15.29:0/2023140046 >> v1:172.21.15.144:6789/0 conn(0x7f0f1c159ec0 0x7f0f1c1568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:37654/0 (socket says 172.21.15.29:37654) 2023-10-07T09:30:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f21f9b700 1 --2- 172.21.15.29:0/2023140046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0ad370 0x7f0f1c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 -- 172.21.15.29:0/2023140046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0ad370 msgr2=0x7f0f1c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 --2- 172.21.15.29:0/2023140046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0ad370 0x7f0f1c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 -- 172.21.15.29:0/2023140046 >> v1:172.21.15.144:6789/0 conn(0x7f0f1c159ec0 legacy=0x7f0f1c1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f2279c700 1 -- 172.21.15.29:0/2023140046 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0ad740 con 0x7f0f1c0b0410 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.763+0000 7f0f20f99700 1 -- 172.21.15.29:0/2023140046 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557221726 0 0) 0x7f0f1c14d790 con 0x7f0f1c159ec0 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f2279c700 1 --2- 172.21.15.29:0/2023140046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c0b0410 0x7f0f1c15a310 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009f20 tx=0x7f0f14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518b2a2d48d0d3e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f20f99700 1 -- 172.21.15.29:0/2023140046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400e070 con 0x7f0f1c0b0410 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f20f99700 1 -- 172.21.15.29:0/2023140046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400b9d0 con 0x7f0f1c0b0410 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f20f99700 1 -- 172.21.15.29:0/2023140046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f140092e0 con 0x7f0f1c0b0410 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f241ff700 1 -- 172.21.15.29:0/2023140046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c0b0410 msgr2=0x7f0f1c15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f241ff700 1 --2- 172.21.15.29:0/2023140046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c0b0410 0x7f0f1c15a310 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009f20 tx=0x7f0f14009840 comp rx=0 tx=0).stop 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f241ff700 1 -- 172.21.15.29:0/2023140046 shutdown_connections 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f241ff700 1 --2- 172.21.15.29:0/2023140046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c0b0410 0x7f0f1c15a310 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f241ff700 1 --2- 172.21.15.29:0/2023140046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0ad370 0x7f0f1c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.764+0000 7f0f241ff700 1 -- 172.21.15.29:0/2023140046 >> 172.21.15.29:0/2023140046 conn(0x7f0f1c0680a0 msgr2=0x7f0f1c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.765+0000 7f0f241ff700 1 -- 172.21.15.29:0/2023140046 shutdown_connections 2023-10-07T09:30:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.765+0000 7f0f241ff700 1 -- 172.21.15.29:0/2023140046 wait complete. 2023-10-07T09:30:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.766+0000 7f0f241ff700 1 Processor -- start 2023-10-07T09:30:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.766+0000 7f0f241ff700 1 -- start start 2023-10-07T09:30:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f241ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c0ad370 0x7f0f1c168c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f241ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0b0410 0x7f0f1c06de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f241ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 0x7f0f1c069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f241ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c14e6e0 con 0x7f0f1c159ec0 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f241ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c14f260 con 0x7f0f1c0ad370 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f241ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c06e3a0 con 0x7f0f1c0b0410 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f2279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 0x7f0f1c069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f2279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 0x7f0f1c069740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48730/0 (socket says 172.21.15.29:48730) 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.767+0000 7f0f2279c700 1 -- 172.21.15.29:0/1556989725 learned_addr learned my addr 172.21.15.29:0/1556989725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f21f9b700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c0ad370 0x7f0f1c168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2179a700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0b0410 0x7f0f1c06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2279c700 1 -- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0b0410 msgr2=0x7f0f1c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2279c700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0b0410 0x7f0f1c06de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2279c700 1 -- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c0ad370 msgr2=0x7f0f1c168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2279c700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c0ad370 0x7f0f1c168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2279c700 1 -- 172.21.15.29:0/1556989725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14009ae0 con 0x7f0f1c159ec0 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2179a700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0b0410 0x7f0f1c06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f2279c700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 0x7f0f1c069740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c068dd0 tx=0x7f0f14018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14008c70 con 0x7f0f1c159ec0 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14008df0 con 0x7f0f1c159ec0 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f140266b0 con 0x7f0f1c159ec0 2023-10-07T09:30:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.768+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c14e930 con 0x7f0f1c159ec0 2023-10-07T09:30:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.769+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c14eef0 con 0x7f0f1c159ec0 2023-10-07T09:30:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.770+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c14f410 con 0x7f0f1c159ec0 2023-10-07T09:30:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.770+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0f14003040 con 0x7f0f1c159ec0 2023-10-07T09:30:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.771+0000 7f0f11ffb700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0f00066f30 0x7f0f000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.771+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f0f14086240 con 0x7f0f1c159ec0 2023-10-07T09:30:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.772+0000 7f0f21f9b700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0f00066f30 0x7f0f000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.772+0000 7f0f21f9b700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0f00066f30 0x7f0f000693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009d40 tx=0x7f0f18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:20.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:20.775+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f1c14f410 con 0x7f0f1c159ec0 2023-10-07T09:30:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.063+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f0f1c14f3d0 con 0x7f0f1c159ec0 2023-10-07T09:30:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.066+0000 7f0f11ffb700 1 -- 172.21.15.29:0/1556989725 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0f14053760 con 0x7f0f1c159ec0 2023-10-07T09:30:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0f00066f30 msgr2=0x7f0f000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0f00066f30 0x7f0f000693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009d40 tx=0x7f0f18008040 comp rx=0 tx=0).stop 2023-10-07T09:30:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 msgr2=0x7f0f1c069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 0x7f0f1c069740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c068dd0 tx=0x7f0f14018000 comp rx=0 tx=0).stop 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 shutdown_connections 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0f00066f30 0x7f0f000693e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f1c159ec0 0x7f0f1c069740 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c0ad370 0x7f0f1c168c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 --2- 172.21.15.29:0/1556989725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f1c0b0410 0x7f0f1c06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 >> 172.21.15.29:0/1556989725 conn(0x7f0f1c0680a0 msgr2=0x7f0f1c1591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 shutdown_connections 2023-10-07T09:30:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.069+0000 7f0f241ff700 1 -- 172.21.15.29:0/1556989725 wait complete. 2023-10-07T09:30:21.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-07T09:30:21.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.283+0000 7f8d1bbff700 1 Processor -- start 2023-10-07T09:30:21.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.283+0000 7f8d1bbff700 1 -- start start 2023-10-07T09:30:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.284+0000 7f8d1bbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8d14159750 0x7f8d140d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.284+0000 7f8d1bbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d140ce8a0 0x7f8d140c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.284+0000 7f8d1bbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d140cbf20 con 0x7f8d14159b20 2023-10-07T09:30:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.284+0000 7f8d1bbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d140cc200 con 0x7f8d140ce8a0 2023-10-07T09:30:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.284+0000 7f8d1bbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d140cc4e0 con 0x7f8d14159750 2023-10-07T09:30:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d140ce8a0 0x7f8d140c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d140ce8a0 0x7f8d140c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48738/0 (socket says 172.21.15.29:48738) 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 -- 172.21.15.29:0/1463810216 learned_addr learned my addr 172.21.15.29:0/1463810216 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1999b700 1 --2- 172.21.15.29:0/1463810216 >> v2:172.21.15.144:3300/0 conn(0x7f8d14159750 0x7f8d140d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 -- 172.21.15.29:0/1463810216 >> v2:172.21.15.144:3300/0 conn(0x7f8d14159750 msgr2=0x7f8d140d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 --2- 172.21.15.29:0/1463810216 >> v2:172.21.15.144:3300/0 conn(0x7f8d14159750 0x7f8d140d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 -- 172.21.15.29:0/1463810216 >> v1:172.21.15.144:6789/0 conn(0x7f8d14159b20 legacy=0x7f8d140ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.285+0000 7f8d1a19c700 1 -- 172.21.15.29:0/1463810216 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140cc7c0 con 0x7f8d140ce8a0 2023-10-07T09:30:21.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1a19c700 1 --2- 172.21.15.29:0/1463810216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d140ce8a0 0x7f8d140c5460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d14089840 tx=0x7f8d0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70879669e964a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:21.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d18999700 1 -- 172.21.15.29:0/1463810216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c00e070 con 0x7f8d140ce8a0 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d18999700 1 -- 172.21.15.29:0/1463810216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c00ba50 con 0x7f8d140ce8a0 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d18999700 1 -- 172.21.15.29:0/1463810216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c0092e0 con 0x7f8d140ce8a0 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 -- 172.21.15.29:0/1463810216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d140ce8a0 msgr2=0x7f8d140c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/1463810216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d140ce8a0 0x7f8d140c5460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d14089840 tx=0x7f8d0c009840 comp rx=0 tx=0).stop 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 -- 172.21.15.29:0/1463810216 shutdown_connections 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/1463810216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d140ce8a0 0x7f8d140c5460 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/1463810216 >> v2:172.21.15.144:3300/0 conn(0x7f8d14159750 0x7f8d140d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 -- 172.21.15.29:0/1463810216 >> 172.21.15.29:0/1463810216 conn(0x7f8d14063a10 msgr2=0x7f8d140d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.286+0000 7f8d1bbff700 1 -- 172.21.15.29:0/1463810216 shutdown_connections 2023-10-07T09:30:21.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.287+0000 7f8d1bbff700 1 -- 172.21.15.29:0/1463810216 wait complete. 2023-10-07T09:30:21.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.289+0000 7f8d1bbff700 1 Processor -- start 2023-10-07T09:30:21.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.289+0000 7f8d1bbff700 1 -- start start 2023-10-07T09:30:21.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1bbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d140ce8a0 0x7f8d1415b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1bbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 0x7f8d1415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1bbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d14159b20 0x7f8d1415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1bbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1415c1f0 con 0x7f8d14159750 2023-10-07T09:30:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1bbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140b3ae0 con 0x7f8d140ce8a0 2023-10-07T09:30:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1bbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1415bf10 con 0x7f8d14159b20 2023-10-07T09:30:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1919a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 0x7f8d1415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1919a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 0x7f8d1415d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48750/0 (socket says 172.21.15.29:48750) 2023-10-07T09:30:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.290+0000 7f8d1919a700 1 -- 172.21.15.29:0/3347886957 learned_addr learned my addr 172.21.15.29:0/3347886957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1a19c700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d14159b20 0x7f8d1415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1999b700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d140ce8a0 0x7f8d1415b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1919a700 1 -- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d14159b20 msgr2=0x7f8d1415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1919a700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d14159b20 0x7f8d1415cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1919a700 1 -- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d140ce8a0 msgr2=0x7f8d1415b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1919a700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d140ce8a0 0x7f8d1415b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1919a700 1 -- 172.21.15.29:0/3347886957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c009ae0 con 0x7f8d14159750 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1999b700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d140ce8a0 0x7f8d1415b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1919a700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 0x7f8d1415d4f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400c4e0 tx=0x7f8d0400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04011590 con 0x7f8d14159750 2023-10-07T09:30:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04011710 con 0x7f8d14159750 2023-10-07T09:30:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04011ad0 con 0x7f8d14159750 2023-10-07T09:30:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d140b3490 con 0x7f8d14159750 2023-10-07T09:30:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.291+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1415d110 con 0x7f8d14159750 2023-10-07T09:30:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.293+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d140b3c50 con 0x7f8d14159750 2023-10-07T09:30:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.293+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8d0401b3b0 con 0x7f8d14159750 2023-10-07T09:30:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.294+0000 7f8d09ffb700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8cf8066f30 0x7f8cf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.294+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f8d040837b0 con 0x7f8d14159750 2023-10-07T09:30:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.295+0000 7f8d1999b700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8cf8066f30 0x7f8cf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.295+0000 7f8d1999b700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8cf8066f30 0x7f8cf80693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8d10006aa0 tx=0x7f8d10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:21.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.298+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d04050cd0 con 0x7f8d14159750 2023-10-07T09:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.579+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f8d1404c3c0 con 0x7f8d14159750 2023-10-07T09:30:21.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.583+0000 7f8d09ffb700 1 -- 172.21.15.29:0/3347886957 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8d04017020 con 0x7f8d14159750 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8cf8066f30 msgr2=0x7f8cf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8cf8066f30 0x7f8cf80693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8d10006aa0 tx=0x7f8d10008040 comp rx=0 tx=0).stop 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 msgr2=0x7f8d1415d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 0x7f8d1415d4f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400c4e0 tx=0x7f8d0400c8a0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 shutdown_connections 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8cf8066f30 0x7f8cf80693e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d14159b20 0x7f8d1415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d140ce8a0 0x7f8d1415b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 --2- 172.21.15.29:0/3347886957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d14159750 0x7f8d1415d4f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 >> 172.21.15.29:0/3347886957 conn(0x7f8d14063a10 msgr2=0x7f8d140704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.585+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 shutdown_connections 2023-10-07T09:30:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.586+0000 7f8d1bbff700 1 -- 172.21.15.29:0/3347886957 wait complete. 2023-10-07T09:30:21.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.799+0000 7fc0781ff700 1 Processor -- start 2023-10-07T09:30:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.799+0000 7fc0781ff700 1 -- start start 2023-10-07T09:30:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.800+0000 7fc0781ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc0700cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.800+0000 7fc0781ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc0700ae8b0 0x7fc0700b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.800+0000 7fc0781ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0700b1840 0x7fc0700ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.800+0000 7fc0781ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0700b1cc0 con 0x7fc0700b1840 2023-10-07T09:30:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.800+0000 7fc0781ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc070158ad0 con 0x7fc0700ae8b0 2023-10-07T09:30:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.800+0000 7fc0781ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07015b140 con 0x7fc070157290 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0700b1840 0x7fc0700ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0700b1840 0x7fc0700ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48758/0 (socket says 172.21.15.29:48758) 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 -- 172.21.15.29:0/1062825267 learned_addr learned my addr 172.21.15.29:0/1062825267 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc075f9b700 1 --2- 172.21.15.29:0/1062825267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc0700cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07579a700 1 --2- 172.21.15.29:0/1062825267 >> v2:172.21.15.144:3300/0 conn(0x7fc0700ae8b0 0x7fc0700b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 -- 172.21.15.29:0/1062825267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 msgr2=0x7fc0700cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 --2- 172.21.15.29:0/1062825267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc0700cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 -- 172.21.15.29:0/1062825267 >> v2:172.21.15.144:3300/0 conn(0x7fc0700ae8b0 msgr2=0x7fc0700b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 --2- 172.21.15.29:0/1062825267 >> v2:172.21.15.144:3300/0 conn(0x7fc0700ae8b0 0x7fc0700b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 -- 172.21.15.29:0/1062825267 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07015b420 con 0x7fc0700b1840 2023-10-07T09:30:21.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.801+0000 7fc07679c700 1 --2- 172.21.15.29:0/1062825267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc0700ca310 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc070063d00 tx=0x7fc068009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3961f4aeb68705bf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc074f99700 1 -- 172.21.15.29:0/1062825267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06800e070 con 0x7fc0700b1840 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc074f99700 1 -- 172.21.15.29:0/1062825267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06800b970 con 0x7fc0700b1840 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc074f99700 1 -- 172.21.15.29:0/1062825267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0680092e0 con 0x7fc0700b1840 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 -- 172.21.15.29:0/1062825267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 msgr2=0x7fc0700ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 --2- 172.21.15.29:0/1062825267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc0700ca310 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc070063d00 tx=0x7fc068009840 comp rx=0 tx=0).stop 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 -- 172.21.15.29:0/1062825267 shutdown_connections 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 --2- 172.21.15.29:0/1062825267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc0700ca310 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 --2- 172.21.15.29:0/1062825267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc0700cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 --2- 172.21.15.29:0/1062825267 >> v2:172.21.15.144:3300/0 conn(0x7fc0700ae8b0 0x7fc0700b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.802+0000 7fc0781ff700 1 -- 172.21.15.29:0/1062825267 >> 172.21.15.29:0/1062825267 conn(0x7fc0700680a0 msgr2=0x7fc0700cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:21.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.803+0000 7fc0781ff700 1 -- 172.21.15.29:0/1062825267 shutdown_connections 2023-10-07T09:30:21.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.803+0000 7fc0781ff700 1 -- 172.21.15.29:0/1062825267 wait complete. 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.804+0000 7fc0781ff700 1 Processor -- start 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.804+0000 7fc0781ff700 1 -- start start 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc0781ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc070149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc0781ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc07015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc0781ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc07015cdf0 0x7fc07014bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc0781ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07015d270 con 0x7fc0700b1840 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc0781ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07006dbc0 con 0x7fc07015cdf0 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc0781ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07006d8c0 con 0x7fc070157290 2023-10-07T09:30:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc075f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc070149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc075f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc070149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48764/0 (socket says 172.21.15.29:48764) 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.805+0000 7fc075f9b700 1 -- 172.21.15.29:0/41489308 learned_addr learned my addr 172.21.15.29:0/41489308 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc07579a700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc07015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc07679c700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc07015cdf0 0x7fc07014bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc075f9b700 1 -- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 msgr2=0x7fc07015c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc075f9b700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc07015c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc075f9b700 1 -- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc07015cdf0 msgr2=0x7fc07014bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc075f9b700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc07015cdf0 0x7fc07014bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc075f9b700 1 -- 172.21.15.29:0/41489308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068009ae0 con 0x7fc0700b1840 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc07579a700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc07015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc075f9b700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc070149bd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00e960 tx=0x7fc06c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c00c850 con 0x7fc0700b1840 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c00c9b0 con 0x7fc0700b1840 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c00cd50 con 0x7fc0700b1840 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.806+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07014e040 con 0x7fc0700b1840 2023-10-07T09:30:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.807+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07015d760 con 0x7fc0700b1840 2023-10-07T09:30:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.808+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07004c3c0 con 0x7fc0700b1840 2023-10-07T09:30:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.808+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc06c013680 con 0x7fc0700b1840 2023-10-07T09:30:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.810+0000 7fc065ffb700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc054066f30 0x7fc0540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.810+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fc06c084500 con 0x7fc0700b1840 2023-10-07T09:30:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.810+0000 7fc07579a700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc054066f30 0x7fc0540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:21.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.811+0000 7fc07579a700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc054066f30 0x7fc0540693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc070152580 tx=0x7fc060008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:21.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:21.814+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc06c051a20 con 0x7fc0700b1840 2023-10-07T09:30:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.019+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc07015b700 con 0x7fc0700b1840 2023-10-07T09:30:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.020+0000 7fc065ffb700 1 -- 172.21.15.29:0/41489308 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+12230 (secure 0 0 0) 0x7fc06c055040 con 0x7fc0700b1840 2023-10-07T09:30:22.021 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:22.021 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:17.952560+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.022+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc054066f30 msgr2=0x7fc0540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc054066f30 0x7fc0540693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc070152580 tx=0x7fc060008040 comp rx=0 tx=0).stop 2023-10-07T09:30:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 msgr2=0x7fc070149bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc070149bd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00e960 tx=0x7fc06c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 shutdown_connections 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc054066f30 0x7fc0540693e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc07015cdf0 0x7fc07014bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0700b1840 0x7fc070149bd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 --2- 172.21.15.29:0/41489308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc070157290 0x7fc07015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 >> 172.21.15.29:0/41489308 conn(0x7fc0700680a0 msgr2=0x7fc070152ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 shutdown_connections 2023-10-07T09:30:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.023+0000 7fc0781ff700 1 -- 172.21.15.29:0/41489308 wait complete. 2023-10-07T09:30:22.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:30:22.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:30:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.241+0000 7f95e01ff700 1 Processor -- start 2023-10-07T09:30:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.241+0000 7f95e01ff700 1 -- start start 2023-10-07T09:30:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.242+0000 7f95e01ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f95d8159750 0x7f95d80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95e01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159b20 0x7f95d80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95e01ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d80cbf20 con 0x7f95d80ce6d0 2023-10-07T09:30:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95e01ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d80cc200 con 0x7f95d8159750 2023-10-07T09:30:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95e01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80cc4e0 con 0x7f95d8159b20 2023-10-07T09:30:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f95d8159750 0x7f95d80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f95d8159750 0x7f95d80d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48778/0 (socket says 172.21.15.29:48778) 2023-10-07T09:30:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 -- 172.21.15.29:0/1582849911 learned_addr learned my addr 172.21.15.29:0/1582849911 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95dd79a700 1 --2- 172.21.15.29:0/1582849911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159b20 0x7f95d80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 -- 172.21.15.29:0/1582849911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159b20 msgr2=0x7f95d80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 --2- 172.21.15.29:0/1582849911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159b20 0x7f95d80ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 -- 172.21.15.29:0/1582849911 >> v1:172.21.15.144:6789/0 conn(0x7f95d80ce6d0 legacy=0x7f95d815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.243+0000 7f95ddf9b700 1 -- 172.21.15.29:0/1582849911 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d80cc7c0 con 0x7f95d8159750 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.244+0000 7f95ddf9b700 1 --2- 172.21.15.29:0/1582849911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d8159750 0x7f95d80d10a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009990 tx=0x7f95d4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d4e076486747e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.244+0000 7f95dcf99700 1 -- 172.21.15.29:0/1582849911 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4009e80 con 0x7f95d8159750 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.244+0000 7f95dcf99700 1 -- 172.21.15.29:0/1582849911 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d4005c10 con 0x7f95d8159750 2023-10-07T09:30:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.244+0000 7f95dcf99700 1 -- 172.21.15.29:0/1582849911 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d400b650 con 0x7f95d8159750 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.244+0000 7f95e01ff700 1 -- 172.21.15.29:0/1582849911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d8159750 msgr2=0x7f95d80d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.244+0000 7f95e01ff700 1 --2- 172.21.15.29:0/1582849911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d8159750 0x7f95d80d10a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009990 tx=0x7f95d4009220 comp rx=0 tx=0).stop 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.245+0000 7f95e01ff700 1 -- 172.21.15.29:0/1582849911 shutdown_connections 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.245+0000 7f95e01ff700 1 --2- 172.21.15.29:0/1582849911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d8159750 0x7f95d80d10a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.245+0000 7f95e01ff700 1 --2- 172.21.15.29:0/1582849911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159b20 0x7f95d80ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.245+0000 7f95e01ff700 1 -- 172.21.15.29:0/1582849911 >> 172.21.15.29:0/1582849911 conn(0x7f95d8063a10 msgr2=0x7f95d80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.245+0000 7f95e01ff700 1 -- 172.21.15.29:0/1582849911 shutdown_connections 2023-10-07T09:30:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.245+0000 7f95e01ff700 1 -- 172.21.15.29:0/1582849911 wait complete. 2023-10-07T09:30:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.246+0000 7f95e01ff700 1 Processor -- start 2023-10-07T09:30:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.246+0000 7f95e01ff700 1 -- start start 2023-10-07T09:30:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95e01ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 0x7f95d8143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95e01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159750 0x7f95d80b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95e01ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95d8159b20 0x7f95d8169d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95e01ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80b4c40 con 0x7f95d80ce6d0 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95e01ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80b5200 con 0x7f95d8159b20 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95e01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d816a800 con 0x7f95d8159750 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.247+0000 7f95ddf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 0x7f95d8143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 0x7f95d8143fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48784/0 (socket says 172.21.15.29:48784) 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 -- 172.21.15.29:0/974748706 learned_addr learned my addr 172.21.15.29:0/974748706 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95de79c700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95d8159b20 0x7f95d8169d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95dd79a700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159750 0x7f95d80b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 -- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159750 msgr2=0x7f95d80b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159750 0x7f95d80b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 -- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95d8159b20 msgr2=0x7f95d8169d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95d8159b20 0x7f95d8169d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 -- 172.21.15.29:0/974748706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d4009480 con 0x7f95d80ce6d0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95de79c700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95d8159b20 0x7f95d8169d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95dd79a700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159750 0x7f95d80b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95ddf9b700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 0x7f95d8143fb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f95d4011750 tx=0x7f95d4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4011ce0 con 0x7f95d80ce6d0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d4011e40 con 0x7f95d80ce6d0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d401b660 con 0x7f95d80ce6d0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.248+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d816aae0 con 0x7f95d80ce6d0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.249+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d81444f0 con 0x7f95d80ce6d0 2023-10-07T09:30:22.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.250+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d8154a00 con 0x7f95d80ce6d0 2023-10-07T09:30:22.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.251+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f95d401ba90 con 0x7f95d80ce6d0 2023-10-07T09:30:22.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.252+0000 7f95cdffb700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95bc066f30 0x7f95bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.252+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f95d4085b80 con 0x7f95d80ce6d0 2023-10-07T09:30:22.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.252+0000 7f95dd79a700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95bc066f30 0x7f95bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.253+0000 7f95dd79a700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95bc066f30 0x7f95bc0693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f95c8006f00 tx=0x7f95c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:22.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.257+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95d4053140 con 0x7f95d80ce6d0 2023-10-07T09:30:22.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.459+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95d80b54e0 con 0x7f95d80ce6d0 2023-10-07T09:30:22.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.459+0000 7f95cdffb700 1 -- 172.21.15.29:0/974748706 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95d402a0a0 con 0x7f95d80ce6d0 2023-10-07T09:30:22.460 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95bc066f30 msgr2=0x7f95bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95bc066f30 0x7f95bc0693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f95c8006f00 tx=0x7f95c8008040 comp rx=0 tx=0).stop 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 msgr2=0x7f95d8143fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 0x7f95d8143fb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f95d4011750 tx=0x7f95d4016000 comp rx=0 tx=0).stop 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 shutdown_connections 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f95bc066f30 0x7f95bc0693e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95d8159b20 0x7f95d8169d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95d80ce6d0 0x7f95d8143fb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 --2- 172.21.15.29:0/974748706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95d8159750 0x7f95d80b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 >> 172.21.15.29:0/974748706 conn(0x7f95d8063a10 msgr2=0x7f95d80c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 shutdown_connections 2023-10-07T09:30:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.462+0000 7f95e01ff700 1 -- 172.21.15.29:0/974748706 wait complete. 2023-10-07T09:30:22.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:30:22.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.674+0000 7fb7e71ff700 1 Processor -- start 2023-10-07T09:30:22.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.674+0000 7fb7e71ff700 1 -- start start 2023-10-07T09:30:22.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7e71ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7e015ade0 0x7fb7e00b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7e71ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7e00b0540 0x7fb7e015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7e71ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e0154090 con 0x7fb7e0153cc0 2023-10-07T09:30:22.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7e71ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e00b09e0 con 0x7fb7e015ade0 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7e71ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e0159f60 con 0x7fb7e00b0540 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7e015ade0 0x7fb7e00b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7e015ade0 0x7fb7e00b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48796/0 (socket says 172.21.15.29:48796) 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 -- 172.21.15.29:0/1698596637 learned_addr learned my addr 172.21.15.29:0/1698596637 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7e579c700 1 --2- 172.21.15.29:0/1698596637 >> v2:172.21.15.144:3300/0 conn(0x7fb7e00b0540 0x7fb7e015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 -- 172.21.15.29:0/1698596637 >> v2:172.21.15.144:3300/0 conn(0x7fb7e00b0540 msgr2=0x7fb7e015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 --2- 172.21.15.29:0/1698596637 >> v2:172.21.15.144:3300/0 conn(0x7fb7e00b0540 0x7fb7e015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 -- 172.21.15.29:0/1698596637 >> v1:172.21.15.144:6789/0 conn(0x7fb7e0153cc0 legacy=0x7fb7e00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.676+0000 7fb7dffff700 1 -- 172.21.15.29:0/1698596637 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e015a240 con 0x7fb7e015ade0 2023-10-07T09:30:22.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.677+0000 7fb7dffff700 1 --2- 172.21.15.29:0/1698596637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00b0000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb7e006c400 tx=0x7fb7cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0e71860c471e5c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:22.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.677+0000 7fb7df7fe700 1 -- 172.21.15.29:0/1698596637 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00b860 con 0x7fb7e015ade0 2023-10-07T09:30:22.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.677+0000 7fb7df7fe700 1 -- 172.21.15.29:0/1698596637 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc00b9c0 con 0x7fb7e015ade0 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.677+0000 7fb7df7fe700 1 -- 172.21.15.29:0/1698596637 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00bd60 con 0x7fb7e015ade0 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.677+0000 7fb7e71ff700 1 -- 172.21.15.29:0/1698596637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 msgr2=0x7fb7e00b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.677+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/1698596637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00b0000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb7e006c400 tx=0x7fb7cc009250 comp rx=0 tx=0).stop 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.678+0000 7fb7e71ff700 1 -- 172.21.15.29:0/1698596637 shutdown_connections 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.678+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/1698596637 >> v2:172.21.15.144:3300/0 conn(0x7fb7e00b0540 0x7fb7e015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.678+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/1698596637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00b0000 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.678+0000 7fb7e71ff700 1 -- 172.21.15.29:0/1698596637 >> 172.21.15.29:0/1698596637 conn(0x7fb7e0067610 msgr2=0x7fb7e0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.678+0000 7fb7e71ff700 1 -- 172.21.15.29:0/1698596637 shutdown_connections 2023-10-07T09:30:22.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.678+0000 7fb7e71ff700 1 -- 172.21.15.29:0/1698596637 wait complete. 2023-10-07T09:30:22.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.679+0000 7fb7e71ff700 1 Processor -- start 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.679+0000 7fb7e71ff700 1 -- start start 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.680+0000 7fb7e71ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7e00b0540 0x7fb7e013e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.680+0000 7fb7e71ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e0153cc0 0x7fb7e0156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.680+0000 7fb7e71ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.680+0000 7fb7e71ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e016a820 con 0x7fb7e015ade0 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.680+0000 7fb7e71ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e01557a0 con 0x7fb7e0153cc0 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.680+0000 7fb7e71ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e0155a80 con 0x7fb7e00b0540 2023-10-07T09:30:22.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48800/0 (socket says 172.21.15.29:48800) 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 -- 172.21.15.29:0/3535086579 learned_addr learned my addr 172.21.15.29:0/3535086579 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7dffff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e0153cc0 0x7fb7e0156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e4f9b700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7e00b0540 0x7fb7e013e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 -- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7e00b0540 msgr2=0x7fb7e013e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7e00b0540 0x7fb7e013e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 -- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e0153cc0 msgr2=0x7fb7e0156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e0153cc0 0x7fb7e0156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 -- 172.21.15.29:0/3535086579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0094f0 con 0x7fb7e015ade0 2023-10-07T09:30:22.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7e579c700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00c75b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009fa0 tx=0x7fb7d8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.681+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800ec70 con 0x7fb7e015ade0 2023-10-07T09:30:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.682+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d800edd0 con 0x7fb7e015ade0 2023-10-07T09:30:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.682+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8014670 con 0x7fb7e015ade0 2023-10-07T09:30:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.682+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e0155d60 con 0x7fb7e015ade0 2023-10-07T09:30:22.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.682+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e013ece0 con 0x7fb7e015ade0 2023-10-07T09:30:22.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.683+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e004c3c0 con 0x7fb7e015ade0 2023-10-07T09:30:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.683+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb7d80149e0 con 0x7fb7e015ade0 2023-10-07T09:30:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.685+0000 7fb7dcff9700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb7c4066f30 0x7fb7c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.685+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fb7d8084970 con 0x7fb7e015ade0 2023-10-07T09:30:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.685+0000 7fb7e4f9b700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb7c4066f30 0x7fb7c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.686+0000 7fb7e4f9b700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb7c4066f30 0x7fb7c40693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003890 tx=0x7fb7d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.689+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7d8051e90 con 0x7fb7e015ade0 2023-10-07T09:30:22.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.895+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb7e015a520 con 0x7fb7e015ade0 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.895+0000 7fb7dcff9700 1 -- 172.21.15.29:0/3535086579 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v32) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb7d80554b0 con 0x7fb7e015ade0 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:30:22.897 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:22.898 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:30:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb7c4066f30 msgr2=0x7fb7c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb7c4066f30 0x7fb7c40693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003890 tx=0x7fb7d4009040 comp rx=0 tx=0).stop 2023-10-07T09:30:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 msgr2=0x7fb7e00c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00c75b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009fa0 tx=0x7fb7d8009900 comp rx=0 tx=0).stop 2023-10-07T09:30:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 shutdown_connections 2023-10-07T09:30:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb7c4066f30 0x7fb7c40693e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e015ade0 0x7fb7e00c75b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7e00b0540 0x7fb7e013e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 --2- 172.21.15.29:0/3535086579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e0153cc0 0x7fb7e0156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 >> 172.21.15.29:0/3535086579 conn(0x7fb7e0067610 msgr2=0x7fb7e0152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 shutdown_connections 2023-10-07T09:30:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:22.898+0000 7fb7e71ff700 1 -- 172.21.15.29:0/3535086579 wait complete. 2023-10-07T09:30:22.917 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:30:22.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:30:23.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.115+0000 7f9fa13ff700 1 Processor -- start 2023-10-07T09:30:23.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.116+0000 7f9fa13ff700 1 -- start start 2023-10-07T09:30:23.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9fa13ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0cb8c0 0x7f9f9c0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:23.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9fa13ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0cf0f0 0x7f9f9c0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9fa13ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c0cc940 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9fa13ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0cf530 con 0x7f9f9c0cb8c0 2023-10-07T09:30:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9fa13ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0b2df0 con 0x7f9f9c0cf0f0 2023-10-07T09:30:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9f9a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0cb8c0 0x7f9f9c0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9f9affd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9f9c0b2a20 0x7f9f9c0cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33426/0 (socket says 172.21.15.29:33426) 2023-10-07T09:30:23.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9f9affd700 1 -- 172.21.15.29:0/404423230 learned_addr learned my addr 172.21.15.29:0/404423230 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.117+0000 7f9f9a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0cb8c0 0x7f9f9c0cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39576/0 (socket says 172.21.15.29:39576) 2023-10-07T09:30:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f9b7fe700 1 --2- 172.21.15.29:0/404423230 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0cf0f0 0x7f9f9c0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f9a7fc700 1 -- 172.21.15.29:0/404423230 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0cf0f0 msgr2=0x7f9f9c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f9a7fc700 1 --2- 172.21.15.29:0/404423230 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0cf0f0 0x7f9f9c0c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f9a7fc700 1 -- 172.21.15.29:0/404423230 >> v1:172.21.15.29:6789/0 conn(0x7f9f9c0b2a20 legacy=0x7f9f9c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f9a7fc700 1 -- 172.21.15.29:0/404423230 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c0cbd30 con 0x7f9f9c0cb8c0 2023-10-07T09:30:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f99ffb700 1 -- 172.21.15.29:0/404423230 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116403689 0 0) 0x7f9f9c0cc940 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f9a7fc700 1 --2- 172.21.15.29:0/404423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0cb8c0 0x7f9f9c0cebb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0b3110 tx=0x7f9f900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9903a518988bab14 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f99ffb700 1 -- 172.21.15.29:0/404423230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9000b6b0 con 0x7f9f9c0cb8c0 2023-10-07T09:30:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f99ffb700 1 -- 172.21.15.29:0/404423230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9000b810 con 0x7f9f9c0cb8c0 2023-10-07T09:30:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.118+0000 7f9f99ffb700 1 -- 172.21.15.29:0/404423230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9000bbb0 con 0x7f9f9c0cb8c0 2023-10-07T09:30:23.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 -- 172.21.15.29:0/404423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0cb8c0 msgr2=0x7f9f9c0cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:23.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/404423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0cb8c0 0x7f9f9c0cebb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0b3110 tx=0x7f9f900091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 -- 172.21.15.29:0/404423230 shutdown_connections 2023-10-07T09:30:23.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/404423230 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0cf0f0 0x7f9f9c0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/404423230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0cb8c0 0x7f9f9c0cebb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 -- 172.21.15.29:0/404423230 >> 172.21.15.29:0/404423230 conn(0x7f9f9c099820 msgr2=0x7f9f9c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:23.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 -- 172.21.15.29:0/404423230 shutdown_connections 2023-10-07T09:30:23.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.119+0000 7f9fa13ff700 1 -- 172.21.15.29:0/404423230 wait complete. 2023-10-07T09:30:23.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.120+0000 7f9fa13ff700 1 Processor -- start 2023-10-07T09:30:23.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.121+0000 7f9fa13ff700 1 -- start start 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.121+0000 7f9fa13ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 0x7f9f9c16a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 0x7f9f9c16a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 0x7f9f9c16a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39584/0 (socket says 172.21.15.29:39584) 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9fa13ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0cb8c0 0x7f9f9c0c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 -- 172.21.15.29:0/2543558758 learned_addr learned my addr 172.21.15.29:0/2543558758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0cf0f0 0x7f9f9c0b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c16ab00 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c15afa0 con 0x7f9f9c0cb8c0 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0b3370 con 0x7f9f9c0cf0f0 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 -- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0cf0f0 msgr2=0x7f9f9c0b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0cf0f0 0x7f9f9c0b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 -- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0cb8c0 msgr2=0x7f9f9c0c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0cb8c0 0x7f9f9c0c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.122+0000 7f9f9affd700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f90009420 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.123+0000 7f9f9affd700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 0x7f9f9c16a5c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00c7e0 tx=0x7f9f8c007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.123+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c007300 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.123+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c007460 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.123+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c0b3650 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.123+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0057f0 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.123+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c0b3c10 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.124+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c0718b0 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.125+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f8c005c00 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.126+0000 7f9f82ffd700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9f88066f30 0x7f9f880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.126+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f9f8c083de0 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.126+0000 7f9f9a7fc700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9f88066f30 0x7f9f880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.127+0000 7f9f9a7fc700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9f88066f30 0x7f9f880693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0b3110 tx=0x7f9f90017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.130+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f8c051380 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.333+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9f9c0cc010 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.940+0000 7f9f82ffd700 1 -- 172.21.15.29:0/2543558758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v33) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9f8c0549a0 con 0x7f9f9c0b2a20 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9f88066f30 msgr2=0x7f9f880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9f88066f30 0x7f9f880693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0b3110 tx=0x7f9f90017000 comp rx=0 tx=0).stop 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 msgr2=0x7f9f9c16a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 0x7f9f9c16a5c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00c7e0 tx=0x7f9f8c007c80 comp rx=0 tx=0).stop 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 shutdown_connections 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9f88066f30 0x7f9f880693e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0cf0f0 0x7f9f9c0b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0b2a20 0x7f9f9c16a5c0 secure :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00c7e0 tx=0x7f9f8c007c80 comp rx=0 tx=0).stop 2023-10-07T09:30:23.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 --2- 172.21.15.29:0/2543558758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0cb8c0 0x7f9f9c0c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:23.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 >> 172.21.15.29:0/2543558758 conn(0x7f9f9c099820 msgr2=0x7f9f9c0c6360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:23.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 shutdown_connections 2023-10-07T09:30:23.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:23.946+0000 7f9fa13ff700 1 -- 172.21.15.29:0/2543558758 wait complete. 2023-10-07T09:30:23.949 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:30:23.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:30:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.172+0000 7f7c61bff700 1 Processor -- start 2023-10-07T09:30:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.172+0000 7f7c61bff700 1 -- start start 2023-10-07T09:30:24.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.173+0000 7f7c61bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 0x7f7c5c15a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:24.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c61bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c5c0b0690 0x7f7c5c15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c61bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c15ab80 con 0x7f7c5c153ac0 2023-10-07T09:30:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c61bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c069d40 con 0x7f7c5c0b0690 2023-10-07T09:30:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c61bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c068ca0 con 0x7f7c5c153e90 2023-10-07T09:30:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 0x7f7c5c15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c5bfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c5c0b0690 0x7f7c5c15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 0x7f7c5c15a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:49232/0 (socket says 172.21.15.29:49232) 2023-10-07T09:30:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c5affd700 1 -- 172.21.15.29:0/2332360574 learned_addr learned my addr 172.21.15.29:0/2332360574 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c5b7fe700 1 --1- 172.21.15.29:0/2332360574 >> v1:172.21.15.144:6789/0 conn(0x7f7c5c153ac0 0x7f7c5c0aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:46620/0 (socket says 172.21.15.29:46620) 2023-10-07T09:30:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.174+0000 7f7c5affd700 1 -- 172.21.15.29:0/2332360574 >> v2:172.21.15.144:3300/0 conn(0x7f7c5c0b0690 msgr2=0x7f7c5c15ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5affd700 1 --2- 172.21.15.29:0/2332360574 >> v2:172.21.15.144:3300/0 conn(0x7f7c5c0b0690 0x7f7c5c15ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5affd700 1 -- 172.21.15.29:0/2332360574 >> v1:172.21.15.144:6789/0 conn(0x7f7c5c153ac0 legacy=0x7f7c5c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5affd700 1 -- 172.21.15.29:0/2332360574 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c068f80 con 0x7f7c5c153e90 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5bfff700 1 --2- 172.21.15.29:0/2332360574 >> v2:172.21.15.144:3300/0 conn(0x7f7c5c0b0690 0x7f7c5c15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5a7fc700 1 -- 172.21.15.29:0/2332360574 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131062125 0 0) 0x7f7c4c001110 con 0x7f7c5c153ac0 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5affd700 1 --2- 172.21.15.29:0/2332360574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 0x7f7c5c15a640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c500099b0 tx=0x7f7c50009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b845d9450261e727 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5a7fc700 1 -- 172.21.15.29:0/2332360574 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50009ea0 con 0x7f7c5c153e90 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5a7fc700 1 -- 172.21.15.29:0/2332360574 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5000b860 con 0x7f7c5c153e90 2023-10-07T09:30:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.175+0000 7f7c5a7fc700 1 -- 172.21.15.29:0/2332360574 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5000bc00 con 0x7f7c5c153e90 2023-10-07T09:30:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 -- 172.21.15.29:0/2332360574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 msgr2=0x7f7c5c15a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 --2- 172.21.15.29:0/2332360574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 0x7f7c5c15a640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c500099b0 tx=0x7f7c50009250 comp rx=0 tx=0).stop 2023-10-07T09:30:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 -- 172.21.15.29:0/2332360574 shutdown_connections 2023-10-07T09:30:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 --2- 172.21.15.29:0/2332360574 >> v2:172.21.15.144:3300/0 conn(0x7f7c5c0b0690 0x7f7c5c15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 --2- 172.21.15.29:0/2332360574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c153e90 0x7f7c5c15a640 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 -- 172.21.15.29:0/2332360574 >> 172.21.15.29:0/2332360574 conn(0x7f7c5c067610 msgr2=0x7f7c5c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 -- 172.21.15.29:0/2332360574 shutdown_connections 2023-10-07T09:30:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.176+0000 7f7c61bff700 1 -- 172.21.15.29:0/2332360574 wait complete. 2023-10-07T09:30:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.178+0000 7f7c61bff700 1 Processor -- start 2023-10-07T09:30:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.178+0000 7f7c61bff700 1 -- start start 2023-10-07T09:30:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.179+0000 7f7c61bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c0b0690 0x7f7c5c0c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.179+0000 7f7c61bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 0x7f7c5c154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.179+0000 7f7c61bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c5c1551c0 0x7f7c5c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.179+0000 7f7c61bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c15cc00 con 0x7f7c5c153ac0 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.179+0000 7f7c61bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0c86b0 con 0x7f7c5c1551c0 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.179+0000 7f7c61bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0c88e0 con 0x7f7c5c0b0690 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 0x7f7c5c154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 0x7f7c5c154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39590/0 (socket says 172.21.15.29:39590) 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 -- 172.21.15.29:0/1413436819 learned_addr learned my addr 172.21.15.29:0/1413436819 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5b7fe700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c0b0690 0x7f7c5c0c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5bfff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c5c1551c0 0x7f7c5c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 -- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c0b0690 msgr2=0x7f7c5c0c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c0b0690 0x7f7c5c0c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 -- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c5c1551c0 msgr2=0x7f7c5c15ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c5c1551c0 0x7f7c5c15ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.180+0000 7f7c5affd700 1 -- 172.21.15.29:0/1413436819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54009a20 con 0x7f7c5c153ac0 2023-10-07T09:30:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.181+0000 7f7c5affd700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 0x7f7c5c154c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009290 tx=0x7f7c50017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.181+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5000bbe0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.181+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5001a450 con 0x7f7c5c153ac0 2023-10-07T09:30:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.181+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c500094f0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.181+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5001a7f0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.181+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c0c8ea0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.182+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0718b0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.183+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c5001abe0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.184+0000 7f7c437fe700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c48066f30 0x7f7c480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.184+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f7c50085cc0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.184+0000 7f7c5b7fe700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c48066f30 0x7f7c480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.185+0000 7f7c5b7fe700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c48066f30 0x7f7c480693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c007fd0 tx=0x7f7c4c007b90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.189+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c5008c080 con 0x7f7c5c153ac0 2023-10-07T09:30:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.394+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7c5c15c5b0 con 0x7f7c5c153ac0 2023-10-07T09:30:24.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.947+0000 7f7c437fe700 1 -- 172.21.15.29:0/1413436819 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v34) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7c50053130 con 0x7f7c5c153ac0 2023-10-07T09:30:24.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.956+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c48066f30 msgr2=0x7f7c480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:24.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.956+0000 7f7c61bff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c48066f30 0x7f7c480693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c007fd0 tx=0x7f7c4c007b90 comp rx=0 tx=0).stop 2023-10-07T09:30:24.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.956+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 msgr2=0x7f7c5c154c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:24.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.956+0000 7f7c61bff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 0x7f7c5c154c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009290 tx=0x7f7c50017000 comp rx=0 tx=0).stop 2023-10-07T09:30:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.959+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 shutdown_connections 2023-10-07T09:30:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.959+0000 7f7c61bff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7c48066f30 0x7f7c480693e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.959+0000 7f7c61bff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c5c1551c0 0x7f7c5c15ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.959+0000 7f7c61bff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c5c0b0690 0x7f7c5c0c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.959+0000 7f7c61bff700 1 --2- 172.21.15.29:0/1413436819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c5c153ac0 0x7f7c5c154c80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.959+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 >> 172.21.15.29:0/1413436819 conn(0x7f7c5c067610 msgr2=0x7f7c5c0af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.960+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 shutdown_connections 2023-10-07T09:30:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:24.960+0000 7f7c61bff700 1 -- 172.21.15.29:0/1413436819 wait complete. 2023-10-07T09:30:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:30:24.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:30:25.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.188+0000 7fb89c1ff700 1 Processor -- start 2023-10-07T09:30:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.188+0000 7fb89c1ff700 1 -- start start 2023-10-07T09:30:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89c1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 0x7fb8940afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89c1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb8940ae570 0x7fb8940accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89c1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8940ad230 con 0x7fb8940b0520 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89c1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8940ad4d0 con 0x7fb8940ae570 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89c1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940ad7b0 con 0x7fb894156180 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89a79c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb8940b0520 0x7fb894159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33438/0 (socket says 172.21.15.29:33438) 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89a79c700 1 -- 172.21.15.29:0/105783674 learned_addr learned my addr 172.21.15.29:0/105783674 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb899f9b700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 0x7fb8940afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb89979a700 1 --2- 172.21.15.29:0/105783674 >> v2:172.21.15.144:3300/0 conn(0x7fb8940ae570 0x7fb8940accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.190+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25758608 0 0) 0x7fb8940ad230 con 0x7fb8940b0520 2023-10-07T09:30:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb880003570 con 0x7fb8940b0520 2023-10-07T09:30:25.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2712765417 0 0) 0x7fb880003570 con 0x7fb8940b0520 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 msgr2=0x7fb8940afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 0x7fb8940afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 >> v2:172.21.15.144:3300/0 conn(0x7fb8940ae570 msgr2=0x7fb8940accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 --2- 172.21.15.29:0/105783674 >> v2:172.21.15.144:3300/0 conn(0x7fb8940ae570 0x7fb8940accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940ada90 con 0x7fb8940b0520 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fb88c0017f0 con 0x7fb8940b0520 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 >> v1:172.21.15.29:6789/0 conn(0x7fb8940b0520 legacy=0x7fb894159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb880003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 0x7fb880004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb880004d50 0x7fb8800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940ad4d0 con 0x7fb880003a60 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940ad7b0 con 0x7fb880004d50 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88000ba60 con 0x7fb894156180 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb899f9b700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb880003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb89979a700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 0x7fb880004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.191+0000 7fb89a79c700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb880004d50 0x7fb8800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb899f9b700 1 -- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 msgr2=0x7fb880004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb899f9b700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb894156180 0x7fb880004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb899f9b700 1 -- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb880004d50 msgr2=0x7fb8800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb899f9b700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb880004d50 0x7fb8800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb899f9b700 1 -- 172.21.15.29:0/105783674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb880003570 con 0x7fb880003a60 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb899f9b700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb880003e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb890009c00 tx=0x7fb8900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4345247e53aa5205 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000e6b0 con 0x7fb880003a60 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89000e810 con 0x7fb880003a60 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.192+0000 7fb898f99700 1 -- 172.21.15.29:0/105783674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000ebd0 con 0x7fb880003a60 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89c1ff700 1 -- 172.21.15.29:0/105783674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 msgr2=0x7fb880003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/105783674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb880003e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb890009c00 tx=0x7fb8900091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89a79c700 1 -- 172.21.15.29:0/105783674 reap_dead start 2023-10-07T09:30:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89c1ff700 1 -- 172.21.15.29:0/105783674 shutdown_connections 2023-10-07T09:30:25.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89c1ff700 1 -- 172.21.15.29:0/105783674 >> 172.21.15.29:0/105783674 conn(0x7fb894067610 msgr2=0x7fb894067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:25.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89c1ff700 1 -- 172.21.15.29:0/105783674 shutdown_connections 2023-10-07T09:30:25.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.193+0000 7fb89c1ff700 1 -- 172.21.15.29:0/105783674 wait complete. 2023-10-07T09:30:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.195+0000 7fb89c1ff700 1 Processor -- start 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.195+0000 7fb89c1ff700 1 -- start start 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.196+0000 7fb89c1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb8940c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89c1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8940cbd20 0x7fb894169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89c1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8940cc190 0x7fb89416a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89c1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940caa20 con 0x7fb880003a60 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89c1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c9da0 con 0x7fb8940cc190 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89c1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940ca080 con 0x7fb8940cbd20 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb8940c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb8940c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39610/0 (socket says 172.21.15.29:39610) 2023-10-07T09:30:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 -- 172.21.15.29:0/3215362529 learned_addr learned my addr 172.21.15.29:0/3215362529 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89979a700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8940cbd20 0x7fb894169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb89a79c700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8940cc190 0x7fb89416a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 -- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8940cbd20 msgr2=0x7fb894169ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8940cbd20 0x7fb894169ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 -- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8940cc190 msgr2=0x7fb89416a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8940cc190 0x7fb89416a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.197+0000 7fb899f9b700 1 -- 172.21.15.29:0/3215362529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940ca360 con 0x7fb880003a60 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.198+0000 7fb899f9b700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb8940c9790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb880004730 tx=0x7fb89001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.198+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890022070 con 0x7fb880003a60 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.198+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89000e980 con 0x7fb880003a60 2023-10-07T09:30:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.198+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8940ca640 con 0x7fb880003a60 2023-10-07T09:30:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.198+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000e6b0 con 0x7fb880003a60 2023-10-07T09:30:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.198+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89416a920 con 0x7fb880003a60 2023-10-07T09:30:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.199+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89416ab10 con 0x7fb880003a60 2023-10-07T09:30:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.200+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb890027680 con 0x7fb880003a60 2023-10-07T09:30:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.201+0000 7fb889ffb700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb878066f30 0x7fb8780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.201+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5605+0+0 (secure 0 0 0) 0x7fb890085eb0 con 0x7fb880003a60 2023-10-07T09:30:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.201+0000 7fb89979a700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb878066f30 0x7fb8780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.202+0000 7fb89979a700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb878066f30 0x7fb8780693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8940ada10 tx=0x7fb8840091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:25.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.206+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb890053190 con 0x7fb880003a60 2023-10-07T09:30:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:25.452+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb89404c3c0 con 0x7fb880003a60 2023-10-07T09:30:26.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.391+0000 7fb889ffb700 1 -- 172.21.15.29:0/3215362529 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb8900170c0 con 0x7fb880003a60 2023-10-07T09:30:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.400+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb878066f30 msgr2=0x7fb8780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.400+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb878066f30 0x7fb8780693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8940ada10 tx=0x7fb8840091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.400+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 msgr2=0x7fb8940c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.400+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb8940c9790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb880004730 tx=0x7fb89001a000 comp rx=0 tx=0).stop 2023-10-07T09:30:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.404+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 shutdown_connections 2023-10-07T09:30:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.404+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb878066f30 0x7fb8780693e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.404+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8940cc190 0x7fb89416a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.404+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb880003a60 0x7fb8940c9790 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.404+0000 7fb89c1ff700 1 --2- 172.21.15.29:0/3215362529 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8940cbd20 0x7fb894169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.404+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 >> 172.21.15.29:0/3215362529 conn(0x7fb894067610 msgr2=0x7fb8941577a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.406+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 shutdown_connections 2023-10-07T09:30:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.406+0000 7fb89c1ff700 1 -- 172.21.15.29:0/3215362529 wait complete. 2023-10-07T09:30:26.421 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 4 and data pool 5 2023-10-07T09:30:26.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.639+0000 7f73ccfff700 1 Processor -- start 2023-10-07T09:30:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.639+0000 7f73ccfff700 1 -- start start 2023-10-07T09:30:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.641+0000 7f73ccfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73c8157290 0x7f73c80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.641+0000 7f73ccfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c8158ad0 con 0x7f73c80b1a10 2023-10-07T09:30:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.641+0000 7f73ccfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c815b140 con 0x7f73c80ae8b0 2023-10-07T09:30:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.641+0000 7f73ccfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c815b420 con 0x7f73c8157290 2023-10-07T09:30:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.641+0000 7f73c6d9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f73c80b1a10 0x7f73c8157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33442/0 (socket says 172.21.15.29:33442) 2023-10-07T09:30:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.641+0000 7f73c6d9d700 1 -- 172.21.15.29:0/49380892 learned_addr learned my addr 172.21.15.29:0/49380892 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c659c700 1 --2- 172.21.15.29:0/49380892 >> v2:172.21.15.144:3300/0 conn(0x7f73c8157290 0x7f73c80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201135305 0 0) 0x7f73c8158ad0 con 0x7f73c80b1a10 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b0003570 con 0x7f73c80b1a10 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228415112 0 0) 0x7f73c815b140 con 0x7f73c80ae8b0 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c8158ad0 con 0x7f73c80ae8b0 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1399276756 0 0) 0x7f73b0003570 con 0x7f73c80b1a10 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 >> v2:172.21.15.144:3300/0 conn(0x7f73c8157290 msgr2=0x7f73c80cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 --2- 172.21.15.29:0/49380892 >> v2:172.21.15.144:3300/0 conn(0x7f73c8157290 0x7f73c80cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 >> v1:172.21.15.144:6789/0 conn(0x7f73c80ae8b0 legacy=0x7f73c80b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c815b700 con 0x7f73c80b1a10 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f73c00017f0 con 0x7f73c80b1a10 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 >> v1:172.21.15.29:6789/0 conn(0x7f73c80b1a10 legacy=0x7f73c8157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73b0003c40 0x7f73b0004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b0004810 0x7f73b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0005210 0x7f73b000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c815b420 con 0x7f73b0004810 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b000c1b0 con 0x7f73b0003c40 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.642+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b000c370 con 0x7f73b0005210 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b0004810 0x7f73b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c659c700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73b0003c40 0x7f73b0004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c6d9d700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0005210 0x7f73b000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 -- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0005210 msgr2=0x7f73b000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0005210 0x7f73b000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 -- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73b0003c40 msgr2=0x7f73b0004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73b0003c40 0x7f73b0004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 -- 172.21.15.29:0/49380892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0003570 con 0x7f73b0004810 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c6d9d700 1 -- 172.21.15.29:0/49380892 reap_dead start 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b0004810 0x7f73b0004c80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0019e0 tx=0x7f73bc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aabede970730812 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc00c5d0 con 0x7f73b0004810 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc00c730 con 0x7f73b0004810 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.643+0000 7f73c559a700 1 -- 172.21.15.29:0/49380892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc00caf0 con 0x7f73b0004810 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 -- 172.21.15.29:0/49380892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b0004810 msgr2=0x7f73b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b0004810 0x7f73b0004c80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0019e0 tx=0x7f73bc007d30 comp rx=0 tx=0).stop 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 -- 172.21.15.29:0/49380892 shutdown_connections 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 --2- 172.21.15.29:0/49380892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b0004810 0x7f73b0004c80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 -- 172.21.15.29:0/49380892 >> 172.21.15.29:0/49380892 conn(0x7f73c80680a0 msgr2=0x7f73c80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 -- 172.21.15.29:0/49380892 shutdown_connections 2023-10-07T09:30:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.644+0000 7f73ccfff700 1 -- 172.21.15.29:0/49380892 wait complete. 2023-10-07T09:30:26.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.646+0000 7f73ccfff700 1 Processor -- start 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.646+0000 7f73ccfff700 1 -- start start 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73ccfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0004810 0x7f73c80c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73ccfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 0x7f73c80c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73ccfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c80c74c0 0x7f73c80c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73ccfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80c5bb0 con 0x7f73c8153840 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73ccfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80afb00 con 0x7f73c80c74c0 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73ccfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80b0640 con 0x7f73b0004810 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73c5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 0x7f73c80c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73c5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 0x7f73c80c6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39620/0 (socket says 172.21.15.29:39620) 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73c5d9b700 1 -- 172.21.15.29:0/497064370 learned_addr learned my addr 172.21.15.29:0/497064370 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73c659c700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0004810 0x7f73c80c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.647+0000 7f73c6d9d700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c80c74c0 0x7f73c80c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73c5d9b700 1 -- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0004810 msgr2=0x7f73c80c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0004810 0x7f73c80c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73c5d9b700 1 -- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c80c74c0 msgr2=0x7f73c80c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c80c74c0 0x7f73c80c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73c5d9b700 1 -- 172.21.15.29:0/497064370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c80c7e70 con 0x7f73c8153840 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73c5d9b700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 0x7f73c80c6f80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f73b000c180 tx=0x7f73bc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc00c5d0 con 0x7f73c8153840 2023-10-07T09:30:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc00cab0 con 0x7f73c8153840 2023-10-07T09:30:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c80afd10 con 0x7f73c8153840 2023-10-07T09:30:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc007b10 con 0x7f73c8153840 2023-10-07T09:30:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.648+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c80b02d0 con 0x7f73c8153840 2023-10-07T09:30:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.649+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c80b0490 con 0x7f73c8153840 2023-10-07T09:30:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.650+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f73bc019030 con 0x7f73c8153840 2023-10-07T09:30:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.651+0000 7f73b67fc700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73a8066f30 0x7f73a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.651+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f73bc085c10 con 0x7f73c8153840 2023-10-07T09:30:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.651+0000 7f73c659c700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73a8066f30 0x7f73a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.652+0000 7f73c659c700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73a8066f30 0x7f73a80693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f73c8061160 tx=0x7f73b80091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.655+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73bc052de0 con 0x7f73c8153840 2023-10-07T09:30:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.856+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c80b07b0 con 0x7f73c8153840 2023-10-07T09:30:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.857+0000 7f73b67fc700 1 -- 172.21.15.29:0/497064370 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+15459 (secure 0 0 0) 0x7f73bc022090 con 0x7f73c8153840 2023-10-07T09:30:26.858 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:26.858 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":35,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:25.453568+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:30:23.373202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-07T09:30:24.433861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:26.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73a8066f30 msgr2=0x7f73a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73a8066f30 0x7f73a80693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f73c8061160 tx=0x7f73b80091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 msgr2=0x7f73c80c6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 0x7f73c80c6f80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f73b000c180 tx=0x7f73bc015000 comp rx=0 tx=0).stop 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 shutdown_connections 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73a8066f30 0x7f73a80693e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c80c74c0 0x7f73c80c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73b0004810 0x7f73c80c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 --2- 172.21.15.29:0/497064370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73c8153840 0x7f73c80c6f80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 >> 172.21.15.29:0/497064370 conn(0x7f73c80680a0 msgr2=0x7f73c8159f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 shutdown_connections 2023-10-07T09:30:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:26.859+0000 7f73ccfff700 1 -- 172.21.15.29:0/497064370 wait complete. 2023-10-07T09:30:26.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.075+0000 7f10e07ff700 1 Processor -- start 2023-10-07T09:30:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.076+0000 7f10e07ff700 1 -- start start 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10e07ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f10d8159750 0x7f10d80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10e07ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159b20 0x7f10d80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10e07ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10d80ce6d0 0x7f10d80ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10e07ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d80cbf20 con 0x7f10d80ce6d0 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10e07ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d80cc200 con 0x7f10d8159750 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10e07ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80cc4e0 con 0x7f10d8159b20 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10ded9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10d80ce6d0 0x7f10d80ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10ded9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10d80ce6d0 0x7f10d80ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39632/0 (socket says 172.21.15.29:39632) 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.077+0000 7f10ded9c700 1 -- 172.21.15.29:0/602611777 learned_addr learned my addr 172.21.15.29:0/602611777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10ded9c700 1 -- 172.21.15.29:0/602611777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159b20 msgr2=0x7f10d80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10ded9c700 1 --2- 172.21.15.29:0/602611777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159b20 0x7f10d80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10ded9c700 1 -- 172.21.15.29:0/602611777 >> v2:172.21.15.144:3300/0 conn(0x7f10d8159750 msgr2=0x7f10d80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10de59b700 1 --2- 172.21.15.29:0/602611777 >> v2:172.21.15.144:3300/0 conn(0x7f10d8159750 0x7f10d80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10ded9c700 1 --2- 172.21.15.29:0/602611777 >> v2:172.21.15.144:3300/0 conn(0x7f10d8159750 0x7f10d80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10ded9c700 1 -- 172.21.15.29:0/602611777 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d80cc7c0 con 0x7f10d80ce6d0 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10de59b700 1 --2- 172.21.15.29:0/602611777 >> v2:172.21.15.144:3300/0 conn(0x7f10d8159750 0x7f10d80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:27.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10ded9c700 1 --2- 172.21.15.29:0/602611777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d80ceb50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f10d8089840 tx=0x7f10d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc8862ecea22bf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:27.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10dd599700 1 -- 172.21.15.29:0/602611777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d000e070 con 0x7f10d80ce6d0 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10dd599700 1 -- 172.21.15.29:0/602611777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d000ba50 con 0x7f10d80ce6d0 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.078+0000 7f10dd599700 1 -- 172.21.15.29:0/602611777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d00092e0 con 0x7f10d80ce6d0 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 -- 172.21.15.29:0/602611777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 msgr2=0x7f10d80ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 --2- 172.21.15.29:0/602611777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d80ceb50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f10d8089840 tx=0x7f10d0009840 comp rx=0 tx=0).stop 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 -- 172.21.15.29:0/602611777 shutdown_connections 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 --2- 172.21.15.29:0/602611777 >> v2:172.21.15.144:3300/0 conn(0x7f10d8159750 0x7f10d80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 --2- 172.21.15.29:0/602611777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d80ceb50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 --2- 172.21.15.29:0/602611777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159b20 0x7f10d80ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 -- 172.21.15.29:0/602611777 >> 172.21.15.29:0/602611777 conn(0x7f10d8063a10 msgr2=0x7f10d80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 -- 172.21.15.29:0/602611777 shutdown_connections 2023-10-07T09:30:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.079+0000 7f10e07ff700 1 -- 172.21.15.29:0/602611777 wait complete. 2023-10-07T09:30:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.081+0000 7f10e07ff700 1 Processor -- start 2023-10-07T09:30:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.081+0000 7f10e07ff700 1 -- start start 2023-10-07T09:30:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10e07ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d814c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10e07ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159750 0x7f10d8157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10e07ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10d8159b20 0x7f10d81575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10e07ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80c7810 con 0x7f10d80ce6d0 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10e07ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d814c540 con 0x7f10d8159b20 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10e07ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8158870 con 0x7f10d8159750 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d814c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d814c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39636/0 (socket says 172.21.15.29:39636) 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 -- 172.21.15.29:0/2449376695 learned_addr learned my addr 172.21.15.29:0/2449376695 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10ddd9a700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159750 0x7f10d8157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10ded9c700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10d8159b20 0x7f10d81575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 -- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159750 msgr2=0x7f10d8157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159750 0x7f10d8157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 -- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10d8159b20 msgr2=0x7f10d81575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10d8159b20 0x7f10d81575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.082+0000 7f10de59b700 1 -- 172.21.15.29:0/2449376695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d0009ae0 con 0x7f10d80ce6d0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.083+0000 7f10de59b700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d814c000 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f10d400c830 tx=0x7f10d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.083+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4005a00 con 0x7f10d80ce6d0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.083+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d4005b60 con 0x7f10d80ce6d0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.083+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4011600 con 0x7f10d80ce6d0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.083+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8157c60 con 0x7f10d80ce6d0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.083+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d8158220 con 0x7f10d80ce6d0 2023-10-07T09:30:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.084+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d804c3c0 con 0x7f10d80ce6d0 2023-10-07T09:30:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.085+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f10d4011970 con 0x7f10d80ce6d0 2023-10-07T09:30:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.086+0000 7f10ce7fc700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f10bc066f30 0x7f10bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.086+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f10d4084320 con 0x7f10d80ce6d0 2023-10-07T09:30:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.086+0000 7f10ddd9a700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f10bc066f30 0x7f10bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.087+0000 7f10ddd9a700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f10bc066f30 0x7f10bc0693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009b40 tx=0x7f10c80090f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.090+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10d4054f80 con 0x7f10d80ce6d0 2023-10-07T09:30:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.293+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10d8156b10 con 0x7f10d80ce6d0 2023-10-07T09:30:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.294+0000 7f10ce7fc700 1 -- 172.21.15.29:0/2449376695 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+15459 (secure 0 0 0) 0x7f10d4054b10 con 0x7f10d80ce6d0 2023-10-07T09:30:27.295 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:27.295 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":35,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:25.453568+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:30:23.373202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-07T09:30:24.433861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f10bc066f30 msgr2=0x7f10bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f10bc066f30 0x7f10bc0693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009b40 tx=0x7f10c80090f0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 msgr2=0x7f10d814c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d814c000 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f10d400c830 tx=0x7f10d4008040 comp rx=0 tx=0).stop 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 shutdown_connections 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f10bc066f30 0x7f10bc0693e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10d8159b20 0x7f10d81575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10d80ce6d0 0x7f10d814c000 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 --2- 172.21.15.29:0/2449376695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10d8159750 0x7f10d8157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 >> 172.21.15.29:0/2449376695 conn(0x7f10d8063a10 msgr2=0x7f10d80c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 shutdown_connections 2023-10-07T09:30:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.296+0000 7f10e07ff700 1 -- 172.21.15.29:0/2449376695 wait complete. 2023-10-07T09:30:27.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:30:27.363 INFO:tasks.ceph.mds.d.smithi144.stderr:2023-10-07T09:30:27.362+0000 7f62d49f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:30:27.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.513+0000 7f71d73ff700 1 Processor -- start 2023-10-07T09:30:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.513+0000 7f71d73ff700 1 -- start start 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d73ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71d0159750 0x7f71d00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d73ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f71d0159b20 0x7f71d00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d73ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d00cbf20 con 0x7f71d00ce6d0 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d73ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00cc200 con 0x7f71d0159750 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d73ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00cc4e0 con 0x7f71d0159b20 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71d0159750 0x7f71d00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71d0159750 0x7f71d00d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39644/0 (socket says 172.21.15.29:39644) 2023-10-07T09:30:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 -- 172.21.15.29:0/1446478066 learned_addr learned my addr 172.21.15.29:0/1446478066 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 -- 172.21.15.29:0/1446478066 >> v2:172.21.15.144:3300/0 conn(0x7f71d0159b20 msgr2=0x7f71d00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 --2- 172.21.15.29:0/1446478066 >> v2:172.21.15.144:3300/0 conn(0x7f71d0159b20 0x7f71d00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 -- 172.21.15.29:0/1446478066 >> v1:172.21.15.144:6789/0 conn(0x7f71d00ce6d0 legacy=0x7f71d015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.515+0000 7f71d519b700 1 -- 172.21.15.29:0/1446478066 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00cc7c0 con 0x7f71d0159750 2023-10-07T09:30:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.516+0000 7f71d519b700 1 --2- 172.21.15.29:0/1446478066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159750 0x7f71d00d10a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f71d0088a70 tx=0x7f71cc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8068d74ac47cb0f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.516+0000 7f71c7fff700 1 -- 172.21.15.29:0/1446478066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc005c10 con 0x7f71d0159750 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.516+0000 7f71c7fff700 1 -- 172.21.15.29:0/1446478066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc005d70 con 0x7f71d0159750 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.516+0000 7f71c7fff700 1 -- 172.21.15.29:0/1446478066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc00b840 con 0x7f71d0159750 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.516+0000 7f71d73ff700 1 -- 172.21.15.29:0/1446478066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159750 msgr2=0x7f71d00d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.516+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1446478066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159750 0x7f71d00d10a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f71d0088a70 tx=0x7f71cc009220 comp rx=0 tx=0).stop 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.517+0000 7f71d73ff700 1 -- 172.21.15.29:0/1446478066 shutdown_connections 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.517+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1446478066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159750 0x7f71d00d10a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.517+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1446478066 >> v2:172.21.15.144:3300/0 conn(0x7f71d0159b20 0x7f71d00ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.517+0000 7f71d73ff700 1 -- 172.21.15.29:0/1446478066 >> 172.21.15.29:0/1446478066 conn(0x7f71d0063a10 msgr2=0x7f71d00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.517+0000 7f71d73ff700 1 -- 172.21.15.29:0/1446478066 shutdown_connections 2023-10-07T09:30:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.517+0000 7f71d73ff700 1 -- 172.21.15.29:0/1446478066 wait complete. 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.518+0000 7f71d73ff700 1 Processor -- start 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.518+0000 7f71d73ff700 1 -- start start 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.519+0000 7f71d73ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71d00ce6d0 0x7f71d00c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.519+0000 7f71d73ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71d0159750 0x7f71d01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.519+0000 7f71d73ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 0x7f71d00b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.519+0000 7f71d73ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00ca680 con 0x7f71d0159b20 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.519+0000 7f71d73ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00b3400 con 0x7f71d00ce6d0 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.519+0000 7f71d73ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00b36e0 con 0x7f71d0159750 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 0x7f71d00b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 0x7f71d00b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39648/0 (socket says 172.21.15.29:39648) 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 -- 172.21.15.29:0/1873561391 learned_addr learned my addr 172.21.15.29:0/1873561391 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d519b700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71d00ce6d0 0x7f71d00c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d499a700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71d0159750 0x7f71d01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 -- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71d0159750 msgr2=0x7f71d01545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71d0159750 0x7f71d01545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 -- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71d00ce6d0 msgr2=0x7f71d00c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71d00ce6d0 0x7f71d00c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 -- 172.21.15.29:0/1873561391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc009480 con 0x7f71d0159b20 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.520+0000 7f71d599c700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 0x7f71d00b44a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f71c800bbb0 tx=0x7f71c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.521+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c800c630 con 0x7f71d0159b20 2023-10-07T09:30:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.521+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c800c7b0 con 0x7f71d0159b20 2023-10-07T09:30:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.521+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c800cb70 con 0x7f71d0159b20 2023-10-07T09:30:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.521+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d00b39c0 con 0x7f71d0159b20 2023-10-07T09:30:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.521+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d00b3f80 con 0x7f71d0159b20 2023-10-07T09:30:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.522+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d004c3c0 con 0x7f71d0159b20 2023-10-07T09:30:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.523+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f71c8012070 con 0x7f71d0159b20 2023-10-07T09:30:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.524+0000 7f71c57fa700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f71b4066f30 0x7f71b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.524+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f71c8084150 con 0x7f71d0159b20 2023-10-07T09:30:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.524+0000 7f71d519b700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f71b4066f30 0x7f71b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.525+0000 7f71d519b700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f71b4066f30 0x7f71b40693e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f71cc003080 tx=0x7f71cc015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.528+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71c8051320 con 0x7f71d0159b20 2023-10-07T09:30:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:27.777+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f71d0064040 con 0x7f71d0159b20 2023-10-07T09:30:28.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.396+0000 7f71c57fa700 1 -- 172.21.15.29:0/1873561391 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f71c8054940 con 0x7f71d0159b20 2023-10-07T09:30:28.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f71b4066f30 msgr2=0x7f71b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f71b4066f30 0x7f71b40693e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f71cc003080 tx=0x7f71cc015000 comp rx=0 tx=0).stop 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 msgr2=0x7f71d00b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 0x7f71d00b44a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f71c800bbb0 tx=0x7f71c8009840 comp rx=0 tx=0).stop 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 shutdown_connections 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f71b4066f30 0x7f71b40693e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71d0159b20 0x7f71d00b44a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71d00ce6d0 0x7f71d00c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 --2- 172.21.15.29:0/1873561391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71d0159750 0x7f71d01545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 >> 172.21.15.29:0/1873561391 conn(0x7f71d0063a10 msgr2=0x7f71d00694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 shutdown_connections 2023-10-07T09:30:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.399+0000 7f71d73ff700 1 -- 172.21.15.29:0/1873561391 wait complete. 2023-10-07T09:30:28.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:30:28.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.619+0000 7fa7351ff700 1 Processor -- start 2023-10-07T09:30:28.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.619+0000 7fa7351ff700 1 -- start start 2023-10-07T09:30:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa7351ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa730156180 0x7fa7300afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa7351ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7300ae570 0x7fa7300accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa7351ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7300ad230 con 0x7fa7300b0520 2023-10-07T09:30:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa7351ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300ad4d0 con 0x7fa7300ae570 2023-10-07T09:30:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa7351ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300ad7b0 con 0x7fa730156180 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa72e59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7300ae570 0x7fa7300accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa72e59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7300ae570 0x7fa7300accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39650/0 (socket says 172.21.15.29:39650) 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa72e59c700 1 -- 172.21.15.29:0/1466961160 learned_addr learned my addr 172.21.15.29:0/1466961160 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa72f59e700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa7300b0520 0x7fa7300b0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33444/0 (socket says 172.21.15.29:33444) 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa72ed9d700 1 --2- 172.21.15.29:0/1466961160 >> v2:172.21.15.144:3300/0 conn(0x7fa730156180 0x7fa7300afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.621+0000 7fa72e59c700 1 -- 172.21.15.29:0/1466961160 >> v2:172.21.15.144:3300/0 conn(0x7fa730156180 msgr2=0x7fa7300afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72e59c700 1 --2- 172.21.15.29:0/1466961160 >> v2:172.21.15.144:3300/0 conn(0x7fa730156180 0x7fa7300afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72dd9b700 1 -- 172.21.15.29:0/1466961160 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382607760 0 0) 0x7fa7300ad230 con 0x7fa7300b0520 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72e59c700 1 -- 172.21.15.29:0/1466961160 >> v1:172.21.15.29:6789/0 conn(0x7fa7300b0520 legacy=0x7fa7300b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72e59c700 1 -- 172.21.15.29:0/1466961160 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300ada90 con 0x7fa7300ae570 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72ed9d700 1 --2- 172.21.15.29:0/1466961160 >> v2:172.21.15.144:3300/0 conn(0x7fa730156180 0x7fa7300afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72e59c700 1 --2- 172.21.15.29:0/1466961160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300ae570 0x7fa7300accf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7300b2250 tx=0x7fa724009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c139e06c7b0ee2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72dd9b700 1 -- 172.21.15.29:0/1466961160 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72400b860 con 0x7fa7300ae570 2023-10-07T09:30:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72dd9b700 1 -- 172.21.15.29:0/1466961160 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72400b9c0 con 0x7fa7300ae570 2023-10-07T09:30:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.622+0000 7fa72dd9b700 1 -- 172.21.15.29:0/1466961160 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72400bd60 con 0x7fa7300ae570 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 -- 172.21.15.29:0/1466961160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300ae570 msgr2=0x7fa7300accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 --2- 172.21.15.29:0/1466961160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300ae570 0x7fa7300accf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7300b2250 tx=0x7fa724009250 comp rx=0 tx=0).stop 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 -- 172.21.15.29:0/1466961160 shutdown_connections 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 --2- 172.21.15.29:0/1466961160 >> v2:172.21.15.144:3300/0 conn(0x7fa730156180 0x7fa7300afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 --2- 172.21.15.29:0/1466961160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300ae570 0x7fa7300accf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 -- 172.21.15.29:0/1466961160 >> 172.21.15.29:0/1466961160 conn(0x7fa730067610 msgr2=0x7fa730067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 -- 172.21.15.29:0/1466961160 shutdown_connections 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.623+0000 7fa7351ff700 1 -- 172.21.15.29:0/1466961160 wait complete. 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.625+0000 7fa7351ff700 1 Processor -- start 2023-10-07T09:30:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.625+0000 7fa7351ff700 1 -- start start 2023-10-07T09:30:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.626+0000 7fa7351ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7300ae570 0x7fa7300c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.626+0000 7fa7351ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 0x7fa730147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.626+0000 7fa7351ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa730156180 0x7fa7300a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.626+0000 7fa7351ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300c5b60 con 0x7fa7300b0520 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.626+0000 7fa7351ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300c5e40 con 0x7fa7300ae570 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.626+0000 7fa7351ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300c6120 con 0x7fa730156180 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 0x7fa730147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 0x7fa730147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39660/0 (socket says 172.21.15.29:39660) 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 -- 172.21.15.29:0/3603857830 learned_addr learned my addr 172.21.15.29:0/3603857830 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72ed9d700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7300ae570 0x7fa7300c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72f59e700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa730156180 0x7fa7300a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 -- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa730156180 msgr2=0x7fa7300a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa730156180 0x7fa7300a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 -- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7300ae570 msgr2=0x7fa7300c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7300ae570 0x7fa7300c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:28.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 -- 172.21.15.29:0/3603857830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7240094f0 con 0x7fa7300b0520 2023-10-07T09:30:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.627+0000 7fa72e59c700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 0x7fa730147b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa724009290 tx=0x7fa724017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.628+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724026450 con 0x7fa7300b0520 2023-10-07T09:30:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.628+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7240265b0 con 0x7fa7300b0520 2023-10-07T09:30:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.628+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7300c6400 con 0x7fa7300b0520 2023-10-07T09:30:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.628+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724026950 con 0x7fa7300b0520 2023-10-07T09:30:28.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.628+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7300c5850 con 0x7fa7300b0520 2023-10-07T09:30:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.629+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7300c6920 con 0x7fa7300b0520 2023-10-07T09:30:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.630+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa72401a420 con 0x7fa7300b0520 2023-10-07T09:30:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.630+0000 7fa716ffd700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa71c066f30 0x7fa71c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.630+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa724086470 con 0x7fa7300b0520 2023-10-07T09:30:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.631+0000 7fa72ed9d700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa71c066f30 0x7fa71c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.632+0000 7fa72ed9d700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa71c066f30 0x7fa71c0693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa720006fd0 tx=0x7fa720008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.634+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7300c6920 con 0x7fa7300b0520 2023-10-07T09:30:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:28.881+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa7300c6920 con 0x7fa7300b0520 2023-10-07T09:30:29.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.402+0000 7fa716ffd700 1 -- 172.21.15.29:0/3603857830 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7300c6920 con 0x7fa7300b0520 2023-10-07T09:30:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa71c066f30 msgr2=0x7fa71c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa71c066f30 0x7fa71c0693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa720006fd0 tx=0x7fa720008040 comp rx=0 tx=0).stop 2023-10-07T09:30:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 msgr2=0x7fa730147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 0x7fa730147b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa724009290 tx=0x7fa724017000 comp rx=0 tx=0).stop 2023-10-07T09:30:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 shutdown_connections 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa71c066f30 0x7fa71c0693e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa730156180 0x7fa7300a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7300ae570 0x7fa7300c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 --2- 172.21.15.29:0/3603857830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7300b0520 0x7fa730147b10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 >> 172.21.15.29:0/3603857830 conn(0x7fa730067610 msgr2=0x7fa730152d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 shutdown_connections 2023-10-07T09:30:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.405+0000 7fa7351ff700 1 -- 172.21.15.29:0/3603857830 wait complete. 2023-10-07T09:30:29.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:30:29.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.623+0000 7f618efff700 1 Processor -- start 2023-10-07T09:30:29.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.624+0000 7f618efff700 1 -- start start 2023-10-07T09:30:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f618efff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61880ae8b0 0x7f61880b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f618efff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f61880b1840 0x7f6188157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f618efff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188158ad0 con 0x7f6188157290 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f618efff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61880b1cc0 con 0x7f61880ae8b0 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f618efff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618815b140 con 0x7f61880b1840 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f6187fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61880ae8b0 0x7f61880b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f6187fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61880ae8b0 0x7f61880b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39662/0 (socket says 172.21.15.29:39662) 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f6187fff700 1 -- 172.21.15.29:0/2809862022 learned_addr learned my addr 172.21.15.29:0/2809862022 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f618d59c700 1 --2- 172.21.15.29:0/2809862022 >> v2:172.21.15.144:3300/0 conn(0x7f61880b1840 0x7f6188157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f6187fff700 1 -- 172.21.15.29:0/2809862022 >> v2:172.21.15.144:3300/0 conn(0x7f61880b1840 msgr2=0x7f6188157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.625+0000 7f6187fff700 1 --2- 172.21.15.29:0/2809862022 >> v2:172.21.15.144:3300/0 conn(0x7f61880b1840 0x7f6188157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.626+0000 7f6187fff700 1 -- 172.21.15.29:0/2809862022 >> v1:172.21.15.144:6789/0 conn(0x7f6188157290 legacy=0x7f61880cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.626+0000 7f6187fff700 1 -- 172.21.15.29:0/2809862022 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618815b420 con 0x7f61880ae8b0 2023-10-07T09:30:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.626+0000 7f6187fff700 1 --2- 172.21.15.29:0/2809862022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880ae8b0 0x7f61880b1300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f61880cbff0 tx=0x7f6174009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3149411ebf865 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.626+0000 7f61877fe700 1 -- 172.21.15.29:0/2809862022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617400b860 con 0x7f61880ae8b0 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.626+0000 7f61877fe700 1 -- 172.21.15.29:0/2809862022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617400b9c0 con 0x7f61880ae8b0 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.626+0000 7f61877fe700 1 -- 172.21.15.29:0/2809862022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617400bd60 con 0x7f61880ae8b0 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 -- 172.21.15.29:0/2809862022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880ae8b0 msgr2=0x7f61880b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 --2- 172.21.15.29:0/2809862022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880ae8b0 0x7f61880b1300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f61880cbff0 tx=0x7f6174009250 comp rx=0 tx=0).stop 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 -- 172.21.15.29:0/2809862022 shutdown_connections 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 --2- 172.21.15.29:0/2809862022 >> v2:172.21.15.144:3300/0 conn(0x7f61880b1840 0x7f6188157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 --2- 172.21.15.29:0/2809862022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880ae8b0 0x7f61880b1300 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 -- 172.21.15.29:0/2809862022 >> 172.21.15.29:0/2809862022 conn(0x7f61880680a0 msgr2=0x7f61880cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 -- 172.21.15.29:0/2809862022 shutdown_connections 2023-10-07T09:30:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.627+0000 7f618efff700 1 -- 172.21.15.29:0/2809862022 wait complete. 2023-10-07T09:30:29.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.628+0000 7f618efff700 1 Processor -- start 2023-10-07T09:30:29.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.628+0000 7f618efff700 1 -- start start 2023-10-07T09:30:29.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.629+0000 7f618efff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61880b1840 0x7f61880b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:29.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.629+0000 7f618efff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6188157290 0x7f6188155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:29.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.629+0000 7f618efff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 0x7f6188152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.629+0000 7f618efff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618816dc20 con 0x7f61880b0ee0 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.629+0000 7f618efff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61880b0550 con 0x7f61880b1840 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.629+0000 7f618efff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618814b070 con 0x7f6188157290 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 0x7f6188152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 0x7f6188152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39678/0 (socket says 172.21.15.29:39678) 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 -- 172.21.15.29:0/788582606 learned_addr learned my addr 172.21.15.29:0/788582606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618cd9b700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61880b1840 0x7f61880b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f6187fff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6188157290 0x7f6188155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:29.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 -- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6188157290 msgr2=0x7f6188155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6188157290 0x7f6188155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 -- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61880b1840 msgr2=0x7f61880b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61880b1840 0x7f61880b09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 -- 172.21.15.29:0/788582606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740094f0 con 0x7f61880b0ee0 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.630+0000 7f618d59c700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 0x7f6188152910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6180009fa0 tx=0x7f6180009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.631+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618000ebc0 con 0x7f61880b0ee0 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.631+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618000ed20 con 0x7f61880b0ee0 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.631+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180014670 con 0x7f61880b0ee0 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.631+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618816d010 con 0x7f61880b0ee0 2023-10-07T09:30:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.631+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618816d5d0 con 0x7f61880b0ee0 2023-10-07T09:30:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.632+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618804c3c0 con 0x7f61880b0ee0 2023-10-07T09:30:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.633+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f61800149e0 con 0x7f61880b0ee0 2023-10-07T09:30:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.634+0000 7f6184ff9700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f616c066f30 0x7f616c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:29.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.634+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6180084c30 con 0x7f61880b0ee0 2023-10-07T09:30:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.634+0000 7f618cd9b700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f616c066f30 0x7f616c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.635+0000 7f618cd9b700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f616c066f30 0x7f616c0693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f617c003a80 tx=0x7f617c007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.638+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6180051e00 con 0x7f61880b0ee0 2023-10-07T09:30:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.885+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61880cbc50 con 0x7f61880b0ee0 2023-10-07T09:30:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.886+0000 7f6184ff9700 1 -- 172.21.15.29:0/788582606 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f6180055420 con 0x7f61880b0ee0 2023-10-07T09:30:29.887 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:29.887 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:30:25.453869+0000","modified":"2023-10-07T09:30:29.396464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6836/3596704883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3596704883},{"type":"v1","addr":"172.21.15.144:6836","nonce":3596704883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.888+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f616c066f30 msgr2=0x7f616c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.888+0000 7f618efff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f616c066f30 0x7f616c0693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f617c003a80 tx=0x7f617c007ec0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.888+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 msgr2=0x7f6188152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.888+0000 7f618efff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 0x7f6188152910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6180009fa0 tx=0x7f6180009900 comp rx=0 tx=0).stop 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 shutdown_connections 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f616c066f30 0x7f616c0693e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61880b0ee0 0x7f6188152910 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61880b1840 0x7f61880b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 --2- 172.21.15.29:0/788582606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6188157290 0x7f6188155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 >> 172.21.15.29:0/788582606 conn(0x7f61880680a0 msgr2=0x7f61880c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 shutdown_connections 2023-10-07T09:30:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:29.889+0000 7f618efff700 1 -- 172.21.15.29:0/788582606 wait complete. 2023-10-07T09:30:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-07T09:30:29.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:30:30.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.102+0000 7f4aff9ff700 1 Processor -- start 2023-10-07T09:30:30.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.103+0000 7f4aff9ff700 1 -- start start 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4aff9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4af815ade0 0x7f4af80b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4aff9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0540 0x7f4af815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4aff9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af80b09b0 con 0x7f4af8153cc0 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4aff9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af8154090 con 0x7f4af815ade0 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4aff9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8159f60 con 0x7f4af80b0540 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4afd79b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4af8153cc0 0x7f4af80b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33446/0 (socket says 172.21.15.29:33446) 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4afd79b700 1 -- 172.21.15.29:0/4084675448 learned_addr learned my addr 172.21.15.29:0/4084675448 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4afcf9a700 1 --2- 172.21.15.29:0/4084675448 >> v2:172.21.15.29:3300/0 conn(0x7f4af815ade0 0x7f4af80b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4afdf9c700 1 --2- 172.21.15.29:0/4084675448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0540 0x7f4af815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.104+0000 7f4aeffff700 1 -- 172.21.15.29:0/4084675448 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374849757 0 0) 0x7f4af80b09b0 con 0x7f4af8153cc0 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4afcf9a700 1 -- 172.21.15.29:0/4084675448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0540 msgr2=0x7f4af815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4afcf9a700 1 --2- 172.21.15.29:0/4084675448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0540 0x7f4af815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4afcf9a700 1 -- 172.21.15.29:0/4084675448 >> v1:172.21.15.29:6789/0 conn(0x7f4af8153cc0 legacy=0x7f4af80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4afcf9a700 1 -- 172.21.15.29:0/4084675448 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af815a240 con 0x7f4af815ade0 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4afcf9a700 1 --2- 172.21.15.29:0/4084675448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af815ade0 0x7f4af80b0000 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80098e0 tx=0x7f4ae80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4276720dd4ee5e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:30.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4aeffff700 1 -- 172.21.15.29:0/4084675448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8009e80 con 0x7f4af815ade0 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4aeffff700 1 -- 172.21.15.29:0/4084675448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae800b860 con 0x7f4af815ade0 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.105+0000 7f4aeffff700 1 -- 172.21.15.29:0/4084675448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800bbe0 con 0x7f4af815ade0 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 -- 172.21.15.29:0/4084675448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af815ade0 msgr2=0x7f4af80b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/4084675448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af815ade0 0x7f4af80b0000 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80098e0 tx=0x7f4ae80091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 -- 172.21.15.29:0/4084675448 shutdown_connections 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/4084675448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0540 0x7f4af815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/4084675448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af815ade0 0x7f4af80b0000 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 -- 172.21.15.29:0/4084675448 >> 172.21.15.29:0/4084675448 conn(0x7f4af8067610 msgr2=0x7f4af8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 -- 172.21.15.29:0/4084675448 shutdown_connections 2023-10-07T09:30:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.106+0000 7f4aff9ff700 1 -- 172.21.15.29:0/4084675448 wait complete. 2023-10-07T09:30:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.107+0000 7f4aff9ff700 1 Processor -- start 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.107+0000 7f4aff9ff700 1 -- start start 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.108+0000 7f4aff9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 0x7f4af8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.108+0000 7f4aff9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8153cc0 0x7f4af8155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.108+0000 7f4aff9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af815ade0 0x7f4af8158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.108+0000 7f4aff9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af81591d0 con 0x7f4af80b0540 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.108+0000 7f4aff9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af81567e0 con 0x7f4af815ade0 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.108+0000 7f4aff9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8155bd0 con 0x7f4af8153cc0 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 0x7f4af8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 0x7f4af8156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39692/0 (socket says 172.21.15.29:39692) 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 -- 172.21.15.29:0/3085871826 learned_addr learned my addr 172.21.15.29:0/3085871826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afcf9a700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8153cc0 0x7f4af8155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afdf9c700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af815ade0 0x7f4af8158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 -- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8153cc0 msgr2=0x7f4af8155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8153cc0 0x7f4af8155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 -- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af815ade0 msgr2=0x7f4af8158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af815ade0 0x7f4af8158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 -- 172.21.15.29:0/3085871826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8009420 con 0x7f4af80b0540 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.109+0000 7f4afd79b700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 0x7f4af8156c70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4af400c7e0 tx=0x7f4af4007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.110+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af40072e0 con 0x7f4af80b0540 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.110+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af4007460 con 0x7f4af80b0540 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.110+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af40057f0 con 0x7f4af80b0540 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.110+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af8155eb0 con 0x7f4af80b0540 2023-10-07T09:30:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.110+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af8156470 con 0x7f4af80b0540 2023-10-07T09:30:30.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.111+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af8156ac0 con 0x7f4af80b0540 2023-10-07T09:30:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.112+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4af4005b60 con 0x7f4af80b0540 2023-10-07T09:30:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.113+0000 7f4aed7fa700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4adc066f30 0x7f4adc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.113+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4af4001e40 con 0x7f4af80b0540 2023-10-07T09:30:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.114+0000 7f4afcf9a700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4adc066f30 0x7f4adc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.114+0000 7f4afcf9a700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4adc066f30 0x7f4adc0693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4af815b200 tx=0x7f4ae8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:30.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.117+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4af404b5a0 con 0x7f4af80b0540 2023-10-07T09:30:30.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.323+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4af804c3c0 con 0x7f4af80b0540 2023-10-07T09:30:30.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.324+0000 7f4aed7fa700 1 -- 172.21.15.29:0/3085871826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+15446 (secure 0 0 0) 0x7f4af4054750 con 0x7f4af80b0540 2023-10-07T09:30:30.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:30.325 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":36,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:30:27.388881+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:30:23.373202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-07T09:30:24.433861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.326+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4adc066f30 msgr2=0x7f4adc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.326+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4adc066f30 0x7f4adc0693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4af815b200 tx=0x7f4ae8017000 comp rx=0 tx=0).stop 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.326+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 msgr2=0x7f4af8156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.326+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 0x7f4af8156c70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4af400c7e0 tx=0x7f4af4007c80 comp rx=0 tx=0).stop 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 shutdown_connections 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4adc066f30 0x7f4adc0693e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af815ade0 0x7f4af8158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b0540 0x7f4af8156c70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 --2- 172.21.15.29:0/3085871826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8153cc0 0x7f4af8155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 >> 172.21.15.29:0/3085871826 conn(0x7f4af8067610 msgr2=0x7f4af80aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:30.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 shutdown_connections 2023-10-07T09:30:30.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.327+0000 7f4aff9ff700 1 -- 172.21.15.29:0/3085871826 wait complete. 2023-10-07T09:30:30.345 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:30:30.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:30:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.540+0000 7f072f3ff700 1 Processor -- start 2023-10-07T09:30:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.540+0000 7f072f3ff700 1 -- start start 2023-10-07T09:30:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07280ad370 0x7f07280cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072f3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0728159ec0 0x7f072815a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072f3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072814d790 con 0x7f07281568c0 2023-10-07T09:30:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072f3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07280ae5d0 con 0x7f0728159ec0 2023-10-07T09:30:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072815a850 con 0x7f07280ad370 2023-10-07T09:30:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072d99c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f07281568c0 0x7f0728156d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33460/0 (socket says 172.21.15.29:33460) 2023-10-07T09:30:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.541+0000 7f072d99c700 1 -- 172.21.15.29:0/2389984849 learned_addr learned my addr 172.21.15.29:0/2389984849 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072c99a700 1 --2- 172.21.15.29:0/2389984849 >> v2:172.21.15.144:3300/0 conn(0x7f0728159ec0 0x7f072815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072d19b700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07280ad370 0x7f07280cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f071ffff700 1 -- 172.21.15.29:0/2389984849 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379109878 0 0) 0x7f072814d790 con 0x7f07281568c0 2023-10-07T09:30:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f071ffff700 1 -- 172.21.15.29:0/2389984849 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0714003570 con 0x7f07281568c0 2023-10-07T09:30:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f071ffff700 1 -- 172.21.15.29:0/2389984849 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 892400015 0 0) 0x7f0714003570 con 0x7f07281568c0 2023-10-07T09:30:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072c99a700 1 -- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07280ad370 msgr2=0x7f07280cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072c99a700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07280ad370 0x7f07280cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072c99a700 1 -- 172.21.15.29:0/2389984849 >> v1:172.21.15.29:6789/0 conn(0x7f07281568c0 legacy=0x7f0728156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072c99a700 1 -- 172.21.15.29:0/2389984849 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07280ad740 con 0x7f0728159ec0 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.542+0000 7f072d19b700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07280ad370 0x7f07280cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.543+0000 7f072c99a700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072815a310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f07180098e0 tx=0x7f07180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce1326985e4c9574 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.543+0000 7f071ffff700 1 -- 172.21.15.29:0/2389984849 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07200017f0 con 0x7f0728159ec0 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.543+0000 7f071ffff700 1 -- 172.21.15.29:0/2389984849 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0718011420 con 0x7f0728159ec0 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.543+0000 7f071ffff700 1 -- 172.21.15.29:0/2389984849 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07180117c0 con 0x7f0728159ec0 2023-10-07T09:30:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.543+0000 7f072f3ff700 1 -- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 msgr2=0x7f072815a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.543+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072815a310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f07180098e0 tx=0x7f07180091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.544+0000 7f072f3ff700 1 -- 172.21.15.29:0/2389984849 shutdown_connections 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.544+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07280ad370 0x7f07280cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.544+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2389984849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072815a310 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.544+0000 7f072f3ff700 1 -- 172.21.15.29:0/2389984849 >> 172.21.15.29:0/2389984849 conn(0x7f07280680a0 msgr2=0x7f07280cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.544+0000 7f072f3ff700 1 -- 172.21.15.29:0/2389984849 shutdown_connections 2023-10-07T09:30:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.544+0000 7f072f3ff700 1 -- 172.21.15.29:0/2389984849 wait complete. 2023-10-07T09:30:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.545+0000 7f072f3ff700 1 Processor -- start 2023-10-07T09:30:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.545+0000 7f072f3ff700 1 -- start start 2023-10-07T09:30:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072f3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 0x7f07281441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07281568c0 0x7f072816ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072f3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072816d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072f3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0728143100 con 0x7f07280ad370 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072f3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07281433e0 con 0x7f0728159ec0 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07281436c0 con 0x7f07281568c0 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 0x7f07281441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 0x7f07281441a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39694/0 (socket says 172.21.15.29:39694) 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.546+0000 7f072d19b700 1 -- 172.21.15.29:0/2821449101 learned_addr learned my addr 172.21.15.29:0/2821449101 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d99c700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072816d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072c99a700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07281568c0 0x7f072816ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d19b700 1 -- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07281568c0 msgr2=0x7f072816ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d19b700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07281568c0 0x7f072816ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d19b700 1 -- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 msgr2=0x7f072816d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d19b700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072816d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d19b700 1 -- 172.21.15.29:0/2821449101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0724009420 con 0x7f07280ad370 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072c99a700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07281568c0 0x7f072816ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d99c700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072816d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f072d19b700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 0x7f07281441a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f072400ec80 tx=0x7f07240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.547+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072400c9d0 con 0x7f07280ad370 2023-10-07T09:30:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.548+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072400cb30 con 0x7f07280ad370 2023-10-07T09:30:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.548+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0718009420 con 0x7f07280ad370 2023-10-07T09:30:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.548+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072400ceb0 con 0x7f07280ad370 2023-10-07T09:30:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.548+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0728143c80 con 0x7f07280ad370 2023-10-07T09:30:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.549+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0728143ff0 con 0x7f07280ad370 2023-10-07T09:30:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.550+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0724013680 con 0x7f07280ad370 2023-10-07T09:30:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.551+0000 7f071d7fa700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f070c066f30 0x7f070c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.551+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0724084a30 con 0x7f07280ad370 2023-10-07T09:30:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.552+0000 7f072c99a700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f070c066f30 0x7f070c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.552+0000 7f072c99a700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f070c066f30 0x7f070c0693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f07180030c0 tx=0x7f0718016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:30.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.555+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0724051c00 con 0x7f07280ad370 2023-10-07T09:30:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.752+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f072804c3c0 con 0x7f070c066f30 2023-10-07T09:30:30.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.763+0000 7f071d7fa700 1 -- 172.21.15.29:0/2821449101 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287592 (secure 0 0 0) 0x7f072804c3c0 con 0x7f070c066f30 2023-10-07T09:30:30.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f070c066f30 msgr2=0x7f070c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f070c066f30 0x7f070c0693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f07180030c0 tx=0x7f0718016000 comp rx=0 tx=0).stop 2023-10-07T09:30:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 msgr2=0x7f07281441a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 0x7f07281441a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f072400ec80 tx=0x7f07240091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 shutdown_connections 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f070c066f30 0x7f070c0693e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0728159ec0 0x7f072816d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ad370 0x7f07281441a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 --2- 172.21.15.29:0/2821449101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07281568c0 0x7f072816ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 >> 172.21.15.29:0/2821449101 conn(0x7f07280680a0 msgr2=0x7f07280afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 shutdown_connections 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:30.767+0000 7f072f3ff700 1 -- 172.21.15.29:0/2821449101 wait complete. 2023-10-07T09:30:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:30:30.786 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-10-07T09:30:30.034161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592016,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":192,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2386644,"kb_used_data":4244,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752588076,"statfs":{"total":773094113280,"available":770650189824,"internally_reserved":0,"allocated":4345856,"data_stored":2399343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1648,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001688"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956469+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415669+0000","last_change":"2023-10-07T09:30:26.408750+0000","last_active":"2023-10-07T09:30:26.415669+0000","last_peered":"2023-10-07T09:30:26.415669+0000","last_clean":"2023-10-07T09:30:26.415669+0000","last_became_active":"2023-10-07T09:30:26.407823+0000","last_became_peered":"2023-10-07T09:30:26.407823+0000","last_unstale":"2023-10-07T09:30:26.415669+0000","last_undegraded":"2023-10-07T09:30:26.415669+0000","last_fullsized":"2023-10-07T09:30:26.415669+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:58.672867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414461+0000","last_change":"2023-10-07T09:30:26.408350+0000","last_active":"2023-10-07T09:30:26.414461+0000","last_peered":"2023-10-07T09:30:26.414461+0000","last_clean":"2023-10-07T09:30:26.414461+0000","last_became_active":"2023-10-07T09:30:26.408029+0000","last_became_peered":"2023-10-07T09:30:26.408029+0000","last_unstale":"2023-10-07T09:30:26.414461+0000","last_undegraded":"2023-10-07T09:30:26.414461+0000","last_fullsized":"2023-10-07T09:30:26.414461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407832+0000","last_change":"2023-10-07T09:30:26.403115+0000","last_active":"2023-10-07T09:30:26.407832+0000","last_peered":"2023-10-07T09:30:26.407832+0000","last_clean":"2023-10-07T09:30:26.407832+0000","last_became_active":"2023-10-07T09:30:26.402699+0000","last_became_peered":"2023-10-07T09:30:26.402699+0000","last_unstale":"2023-10-07T09:30:26.407832+0000","last_undegraded":"2023-10-07T09:30:26.407832+0000","last_fullsized":"2023-10-07T09:30:26.407832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:06:53.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414913+0000","last_change":"2023-10-07T09:30:26.410770+0000","last_active":"2023-10-07T09:30:26.414913+0000","last_peered":"2023-10-07T09:30:26.414913+0000","last_clean":"2023-10-07T09:30:26.414913+0000","last_became_active":"2023-10-07T09:30:26.410465+0000","last_became_peered":"2023-10-07T09:30:26.410465+0000","last_unstale":"2023-10-07T09:30:26.414913+0000","last_undegraded":"2023-10-07T09:30:26.414913+0000","last_fullsized":"2023-10-07T09:30:26.414913+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.403447+0000","last_change":"2023-10-07T09:30:26.395345+0000","last_active":"2023-10-07T09:30:26.403447+0000","last_peered":"2023-10-07T09:30:26.403447+0000","last_clean":"2023-10-07T09:30:26.403447+0000","last_became_active":"2023-10-07T09:30:26.395042+0000","last_became_peered":"2023-10-07T09:30:26.395042+0000","last_unstale":"2023-10-07T09:30:26.403447+0000","last_undegraded":"2023-10-07T09:30:26.403447+0000","last_fullsized":"2023-10-07T09:30:26.403447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956440+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:09.240847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957353+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:20:14.772461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951211+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.399392+0000","last_change":"2023-10-07T09:30:26.396209+0000","last_active":"2023-10-07T09:30:26.399392+0000","last_peered":"2023-10-07T09:30:26.399392+0000","last_clean":"2023-10-07T09:30:26.399392+0000","last_became_active":"2023-10-07T09:30:26.395573+0000","last_became_peered":"2023-10-07T09:30:26.395573+0000","last_unstale":"2023-10-07T09:30:26.399392+0000","last_undegraded":"2023-10-07T09:30:26.399392+0000","last_fullsized":"2023-10-07T09:30:26.399392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:54.021183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957373+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:17.334679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413491+0000","last_change":"2023-10-07T09:30:26.409670+0000","last_active":"2023-10-07T09:30:26.413491+0000","last_peered":"2023-10-07T09:30:26.413491+0000","last_clean":"2023-10-07T09:30:26.413491+0000","last_became_active":"2023-10-07T09:30:26.408687+0000","last_became_peered":"2023-10-07T09:30:26.408687+0000","last_unstale":"2023-10-07T09:30:26.413491+0000","last_undegraded":"2023-10-07T09:30:26.413491+0000","last_fullsized":"2023-10-07T09:30:26.413491+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413335+0000","last_change":"2023-10-07T09:30:26.406082+0000","last_active":"2023-10-07T09:30:26.413335+0000","last_peered":"2023-10-07T09:30:26.413335+0000","last_clean":"2023-10-07T09:30:26.413335+0000","last_became_active":"2023-10-07T09:30:26.405724+0000","last_became_peered":"2023-10-07T09:30:26.405724+0000","last_unstale":"2023-10-07T09:30:26.413335+0000","last_undegraded":"2023-10-07T09:30:26.413335+0000","last_fullsized":"2023-10-07T09:30:26.413335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411102+0000","last_change":"2023-10-07T09:30:26.403342+0000","last_active":"2023-10-07T09:30:26.411102+0000","last_peered":"2023-10-07T09:30:26.411102+0000","last_clean":"2023-10-07T09:30:26.411102+0000","last_became_active":"2023-10-07T09:30:26.403138+0000","last_became_peered":"2023-10-07T09:30:26.403138+0000","last_unstale":"2023-10-07T09:30:26.411102+0000","last_undegraded":"2023-10-07T09:30:26.411102+0000","last_fullsized":"2023-10-07T09:30:26.411102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408435+0000","last_change":"2023-10-07T09:30:26.395877+0000","last_active":"2023-10-07T09:30:26.408435+0000","last_peered":"2023-10-07T09:30:26.408435+0000","last_clean":"2023-10-07T09:30:26.408435+0000","last_became_active":"2023-10-07T09:30:26.395596+0000","last_became_peered":"2023-10-07T09:30:26.395596+0000","last_unstale":"2023-10-07T09:30:26.408435+0000","last_undegraded":"2023-10-07T09:30:26.408435+0000","last_fullsized":"2023-10-07T09:30:26.408435+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:20.489580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956822+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:20:47.855402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957600+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:19.142808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409109+0000","last_change":"2023-10-07T09:30:26.394655+0000","last_active":"2023-10-07T09:30:26.409109+0000","last_peered":"2023-10-07T09:30:26.409109+0000","last_clean":"2023-10-07T09:30:26.409109+0000","last_became_active":"2023-10-07T09:30:26.394372+0000","last_became_peered":"2023-10-07T09:30:26.394372+0000","last_unstale":"2023-10-07T09:30:26.409109+0000","last_undegraded":"2023-10-07T09:30:26.409109+0000","last_fullsized":"2023-10-07T09:30:26.409109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:35.770412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405481+0000","last_change":"2023-10-07T09:30:26.402289+0000","last_active":"2023-10-07T09:30:26.405481+0000","last_peered":"2023-10-07T09:30:26.405481+0000","last_clean":"2023-10-07T09:30:26.405481+0000","last_became_active":"2023-10-07T09:30:26.401826+0000","last_became_peered":"2023-10-07T09:30:26.401826+0000","last_unstale":"2023-10-07T09:30:26.405481+0000","last_undegraded":"2023-10-07T09:30:26.405481+0000","last_fullsized":"2023-10-07T09:30:26.405481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:46:15.007773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414990+0000","last_change":"2023-10-07T09:30:26.411654+0000","last_active":"2023-10-07T09:30:26.414990+0000","last_peered":"2023-10-07T09:30:26.414990+0000","last_clean":"2023-10-07T09:30:26.414990+0000","last_became_active":"2023-10-07T09:30:26.411407+0000","last_became_peered":"2023-10-07T09:30:26.411407+0000","last_unstale":"2023-10-07T09:30:26.414990+0000","last_undegraded":"2023-10-07T09:30:26.414990+0000","last_fullsized":"2023-10-07T09:30:26.414990+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951679+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.949829+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:03.694836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410992+0000","last_change":"2023-10-07T09:30:26.403259+0000","last_active":"2023-10-07T09:30:26.410992+0000","last_peered":"2023-10-07T09:30:26.410992+0000","last_clean":"2023-10-07T09:30:26.410992+0000","last_became_active":"2023-10-07T09:30:26.402926+0000","last_became_peered":"2023-10-07T09:30:26.402926+0000","last_unstale":"2023-10-07T09:30:26.410992+0000","last_undegraded":"2023-10-07T09:30:26.410992+0000","last_fullsized":"2023-10-07T09:30:26.410992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413211+0000","last_change":"2023-10-07T09:30:26.409044+0000","last_active":"2023-10-07T09:30:26.413211+0000","last_peered":"2023-10-07T09:30:26.413211+0000","last_clean":"2023-10-07T09:30:26.413211+0000","last_became_active":"2023-10-07T09:30:26.408764+0000","last_became_peered":"2023-10-07T09:30:26.408764+0000","last_unstale":"2023-10-07T09:30:26.413211+0000","last_undegraded":"2023-10-07T09:30:26.413211+0000","last_fullsized":"2023-10-07T09:30:26.413211+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:47.675483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422810+0000","last_change":"2023-10-07T09:30:26.413627+0000","last_active":"2023-10-07T09:30:26.422810+0000","last_peered":"2023-10-07T09:30:26.422810+0000","last_clean":"2023-10-07T09:30:26.422810+0000","last_became_active":"2023-10-07T09:30:26.413371+0000","last_became_peered":"2023-10-07T09:30:26.413371+0000","last_unstale":"2023-10-07T09:30:26.422810+0000","last_undegraded":"2023-10-07T09:30:26.422810+0000","last_fullsized":"2023-10-07T09:30:26.422810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:34.133968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415691+0000","last_change":"2023-10-07T09:30:26.408999+0000","last_active":"2023-10-07T09:30:26.415691+0000","last_peered":"2023-10-07T09:30:26.415691+0000","last_clean":"2023-10-07T09:30:26.415691+0000","last_became_active":"2023-10-07T09:30:26.408201+0000","last_became_peered":"2023-10-07T09:30:26.408201+0000","last_unstale":"2023-10-07T09:30:26.415691+0000","last_undegraded":"2023-10-07T09:30:26.415691+0000","last_fullsized":"2023-10-07T09:30:26.415691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421410+0000","last_change":"2023-10-07T09:30:26.411198+0000","last_active":"2023-10-07T09:30:26.421410+0000","last_peered":"2023-10-07T09:30:26.421410+0000","last_clean":"2023-10-07T09:30:26.421410+0000","last_became_active":"2023-10-07T09:30:26.410735+0000","last_became_peered":"2023-10-07T09:30:26.410735+0000","last_unstale":"2023-10-07T09:30:26.421410+0000","last_undegraded":"2023-10-07T09:30:26.421410+0000","last_fullsized":"2023-10-07T09:30:26.421410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:24.740454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413473+0000","last_change":"2023-10-07T09:30:26.409340+0000","last_active":"2023-10-07T09:30:26.413473+0000","last_peered":"2023-10-07T09:30:26.413473+0000","last_clean":"2023-10-07T09:30:26.413473+0000","last_became_active":"2023-10-07T09:30:26.408986+0000","last_became_peered":"2023-10-07T09:30:26.408986+0000","last_unstale":"2023-10-07T09:30:26.413473+0000","last_undegraded":"2023-10-07T09:30:26.413473+0000","last_fullsized":"2023-10-07T09:30:26.413473+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950988+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955210+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:25.231695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.954033+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955840+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:30.437449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418432+0000","last_change":"2023-10-07T09:30:26.413698+0000","last_active":"2023-10-07T09:30:26.418432+0000","last_peered":"2023-10-07T09:30:26.418432+0000","last_clean":"2023-10-07T09:30:26.418432+0000","last_became_active":"2023-10-07T09:30:26.413219+0000","last_became_peered":"2023-10-07T09:30:26.413219+0000","last_unstale":"2023-10-07T09:30:26.418432+0000","last_undegraded":"2023-10-07T09:30:26.418432+0000","last_fullsized":"2023-10-07T09:30:26.418432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.970001+0000","last_change":"2023-10-07T09:30:24.970001+0000","last_active":"2023-10-07T09:30:24.970001+0000","last_peered":"2023-10-07T09:30:24.970001+0000","last_clean":"2023-10-07T09:30:24.970001+0000","last_became_active":"2023-10-07T09:30:24.969692+0000","last_became_peered":"2023-10-07T09:30:24.969692+0000","last_unstale":"2023-10-07T09:30:24.970001+0000","last_undegraded":"2023-10-07T09:30:24.970001+0000","last_fullsized":"2023-10-07T09:30:24.970001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956067+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421779+0000","last_change":"2023-10-07T09:30:24.961003+0000","last_active":"2023-10-07T09:30:26.421779+0000","last_peered":"2023-10-07T09:30:26.421779+0000","last_clean":"2023-10-07T09:30:26.421779+0000","last_became_active":"2023-10-07T09:30:24.960711+0000","last_became_peered":"2023-10-07T09:30:24.960711+0000","last_unstale":"2023-10-07T09:30:26.421779+0000","last_undegraded":"2023-10-07T09:30:26.421779+0000","last_fullsized":"2023-10-07T09:30:26.421779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:06.545101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950450+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:07.548280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424146+0000","last_change":"2023-10-07T09:30:24.967737+0000","last_active":"2023-10-07T09:30:26.424146+0000","last_peered":"2023-10-07T09:30:26.424146+0000","last_clean":"2023-10-07T09:30:26.424146+0000","last_became_active":"2023-10-07T09:30:24.966888+0000","last_became_peered":"2023-10-07T09:30:24.966888+0000","last_unstale":"2023-10-07T09:30:26.424146+0000","last_undegraded":"2023-10-07T09:30:26.424146+0000","last_fullsized":"2023-10-07T09:30:26.424146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421389+0000","last_change":"2023-10-07T09:30:26.411013+0000","last_active":"2023-10-07T09:30:26.421389+0000","last_peered":"2023-10-07T09:30:26.421389+0000","last_clean":"2023-10-07T09:30:26.421389+0000","last_became_active":"2023-10-07T09:30:26.410413+0000","last_became_peered":"2023-10-07T09:30:26.410413+0000","last_unstale":"2023-10-07T09:30:26.421389+0000","last_undegraded":"2023-10-07T09:30:26.421389+0000","last_fullsized":"2023-10-07T09:30:26.421389+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:06.340913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423552+0000","last_change":"2023-10-07T09:30:24.966147+0000","last_active":"2023-10-07T09:30:26.423552+0000","last_peered":"2023-10-07T09:30:26.423552+0000","last_clean":"2023-10-07T09:30:26.423552+0000","last_became_active":"2023-10-07T09:30:24.965853+0000","last_became_peered":"2023-10-07T09:30:24.965853+0000","last_unstale":"2023-10-07T09:30:26.423552+0000","last_undegraded":"2023-10-07T09:30:26.423552+0000","last_fullsized":"2023-10-07T09:30:26.423552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.954591+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:38:03.049978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967094+0000","last_change":"2023-10-07T09:30:24.967094+0000","last_active":"2023-10-07T09:30:24.967094+0000","last_peered":"2023-10-07T09:30:24.967094+0000","last_clean":"2023-10-07T09:30:24.967094+0000","last_became_active":"2023-10-07T09:30:24.966810+0000","last_became_peered":"2023-10-07T09:30:24.966810+0000","last_unstale":"2023-10-07T09:30:24.967094+0000","last_undegraded":"2023-10-07T09:30:24.967094+0000","last_fullsized":"2023-10-07T09:30:24.967094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:24:57.676848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414299+0000","last_change":"2023-10-07T09:30:26.407749+0000","last_active":"2023-10-07T09:30:26.414299+0000","last_peered":"2023-10-07T09:30:26.414299+0000","last_clean":"2023-10-07T09:30:26.414299+0000","last_became_active":"2023-10-07T09:30:26.407433+0000","last_became_peered":"2023-10-07T09:30:26.407433+0000","last_unstale":"2023-10-07T09:30:26.414299+0000","last_undegraded":"2023-10-07T09:30:26.414299+0000","last_fullsized":"2023-10-07T09:30:26.414299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.963628+0000","last_change":"2023-10-07T09:30:24.963628+0000","last_active":"2023-10-07T09:30:24.963628+0000","last_peered":"2023-10-07T09:30:24.963628+0000","last_clean":"2023-10-07T09:30:24.963628+0000","last_became_active":"2023-10-07T09:30:24.963306+0000","last_became_peered":"2023-10-07T09:30:24.963306+0000","last_unstale":"2023-10-07T09:30:24.963628+0000","last_undegraded":"2023-10-07T09:30:24.963628+0000","last_fullsized":"2023-10-07T09:30:24.963628+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:12.068751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.949818+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:26.104877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.383857+0000","last_change":"2023-10-07T09:30:24.965572+0000","last_active":"2023-10-07T09:30:26.383857+0000","last_peered":"2023-10-07T09:30:26.383857+0000","last_clean":"2023-10-07T09:30:26.383857+0000","last_became_active":"2023-10-07T09:30:24.965320+0000","last_became_peered":"2023-10-07T09:30:24.965320+0000","last_unstale":"2023-10-07T09:30:26.383857+0000","last_undegraded":"2023-10-07T09:30:26.383857+0000","last_fullsized":"2023-10-07T09:30:26.383857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:49.978481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414174+0000","last_change":"2023-10-07T09:30:26.409216+0000","last_active":"2023-10-07T09:30:26.414174+0000","last_peered":"2023-10-07T09:30:26.414174+0000","last_clean":"2023-10-07T09:30:26.414174+0000","last_became_active":"2023-10-07T09:30:26.408947+0000","last_became_peered":"2023-10-07T09:30:26.408947+0000","last_unstale":"2023-10-07T09:30:26.414174+0000","last_undegraded":"2023-10-07T09:30:26.414174+0000","last_fullsized":"2023-10-07T09:30:26.414174+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.962784+0000","last_change":"2023-10-07T09:30:24.962784+0000","last_active":"2023-10-07T09:30:24.962784+0000","last_peered":"2023-10-07T09:30:24.962784+0000","last_clean":"2023-10-07T09:30:24.962784+0000","last_became_active":"2023-10-07T09:30:24.962409+0000","last_became_peered":"2023-10-07T09:30:24.962409+0000","last_unstale":"2023-10-07T09:30:24.962784+0000","last_undegraded":"2023-10-07T09:30:24.962784+0000","last_fullsized":"2023-10-07T09:30:24.962784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:04.344607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413097+0000","last_change":"2023-10-07T09:30:26.406965+0000","last_active":"2023-10-07T09:30:26.413097+0000","last_peered":"2023-10-07T09:30:26.413097+0000","last_clean":"2023-10-07T09:30:26.413097+0000","last_became_active":"2023-10-07T09:30:26.406607+0000","last_became_peered":"2023-10-07T09:30:26.406607+0000","last_unstale":"2023-10-07T09:30:26.413097+0000","last_undegraded":"2023-10-07T09:30:26.413097+0000","last_fullsized":"2023-10-07T09:30:26.413097+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384959+0000","last_change":"2023-10-07T09:30:24.967714+0000","last_active":"2023-10-07T09:30:26.384959+0000","last_peered":"2023-10-07T09:30:26.384959+0000","last_clean":"2023-10-07T09:30:26.384959+0000","last_became_active":"2023-10-07T09:30:24.967416+0000","last_became_peered":"2023-10-07T09:30:24.967416+0000","last_unstale":"2023-10-07T09:30:26.384959+0000","last_undegraded":"2023-10-07T09:30:26.384959+0000","last_fullsized":"2023-10-07T09:30:26.384959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:25.781083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402385+0000","last_change":"2023-10-07T09:30:26.395987+0000","last_active":"2023-10-07T09:30:26.402385+0000","last_peered":"2023-10-07T09:30:26.402385+0000","last_clean":"2023-10-07T09:30:26.402385+0000","last_became_active":"2023-10-07T09:30:26.395709+0000","last_became_peered":"2023-10-07T09:30:26.395709+0000","last_unstale":"2023-10-07T09:30:26.402385+0000","last_undegraded":"2023-10-07T09:30:26.402385+0000","last_fullsized":"2023-10-07T09:30:26.402385+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390349+0000","last_change":"2023-10-07T09:30:24.962930+0000","last_active":"2023-10-07T09:30:26.390349+0000","last_peered":"2023-10-07T09:30:26.390349+0000","last_clean":"2023-10-07T09:30:26.390349+0000","last_became_active":"2023-10-07T09:30:24.962586+0000","last_became_peered":"2023-10-07T09:30:24.962586+0000","last_unstale":"2023-10-07T09:30:26.390349+0000","last_undegraded":"2023-10-07T09:30:26.390349+0000","last_fullsized":"2023-10-07T09:30:26.390349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.417232+0000","last_change":"2023-10-07T09:30:26.412395+0000","last_active":"2023-10-07T09:30:26.417232+0000","last_peered":"2023-10-07T09:30:26.417232+0000","last_clean":"2023-10-07T09:30:26.417232+0000","last_became_active":"2023-10-07T09:30:26.412085+0000","last_became_peered":"2023-10-07T09:30:26.412085+0000","last_unstale":"2023-10-07T09:30:26.417232+0000","last_undegraded":"2023-10-07T09:30:26.417232+0000","last_fullsized":"2023-10-07T09:30:26.417232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421318+0000","last_change":"2023-10-07T09:30:24.967809+0000","last_active":"2023-10-07T09:30:26.421318+0000","last_peered":"2023-10-07T09:30:26.421318+0000","last_clean":"2023-10-07T09:30:26.421318+0000","last_became_active":"2023-10-07T09:30:24.967550+0000","last_became_peered":"2023-10-07T09:30:24.967550+0000","last_unstale":"2023-10-07T09:30:26.421318+0000","last_undegraded":"2023-10-07T09:30:26.421318+0000","last_fullsized":"2023-10-07T09:30:26.421318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414443+0000","last_change":"2023-10-07T09:30:26.408025+0000","last_active":"2023-10-07T09:30:26.414443+0000","last_peered":"2023-10-07T09:30:26.414443+0000","last_clean":"2023-10-07T09:30:26.414443+0000","last_became_active":"2023-10-07T09:30:26.407736+0000","last_became_peered":"2023-10-07T09:30:26.407736+0000","last_unstale":"2023-10-07T09:30:26.414443+0000","last_undegraded":"2023-10-07T09:30:26.414443+0000","last_fullsized":"2023-10-07T09:30:26.414443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.425863+0000","last_change":"2023-10-07T09:30:24.966563+0000","last_active":"2023-10-07T09:30:26.425863+0000","last_peered":"2023-10-07T09:30:26.425863+0000","last_clean":"2023-10-07T09:30:26.425863+0000","last_became_active":"2023-10-07T09:30:24.966273+0000","last_became_peered":"2023-10-07T09:30:24.966273+0000","last_unstale":"2023-10-07T09:30:26.425863+0000","last_undegraded":"2023-10-07T09:30:26.425863+0000","last_fullsized":"2023-10-07T09:30:26.425863+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:41.648356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421340+0000","last_change":"2023-10-07T09:30:26.409745+0000","last_active":"2023-10-07T09:30:26.421340+0000","last_peered":"2023-10-07T09:30:26.421340+0000","last_clean":"2023-10-07T09:30:26.421340+0000","last_became_active":"2023-10-07T09:30:26.409489+0000","last_became_peered":"2023-10-07T09:30:26.409489+0000","last_unstale":"2023-10-07T09:30:26.421340+0000","last_undegraded":"2023-10-07T09:30:26.421340+0000","last_fullsized":"2023-10-07T09:30:26.421340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.964382+0000","last_change":"2023-10-07T09:30:24.964382+0000","last_active":"2023-10-07T09:30:24.964382+0000","last_peered":"2023-10-07T09:30:24.964382+0000","last_clean":"2023-10-07T09:30:24.964382+0000","last_became_active":"2023-10-07T09:30:24.964076+0000","last_became_peered":"2023-10-07T09:30:24.964076+0000","last_unstale":"2023-10-07T09:30:24.964382+0000","last_undegraded":"2023-10-07T09:30:24.964382+0000","last_fullsized":"2023-10-07T09:30:24.964382+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:19:17.590704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402685+0000","last_change":"2023-10-07T09:30:26.396545+0000","last_active":"2023-10-07T09:30:26.402685+0000","last_peered":"2023-10-07T09:30:26.402685+0000","last_clean":"2023-10-07T09:30:26.402685+0000","last_became_active":"2023-10-07T09:30:26.395816+0000","last_became_peered":"2023-10-07T09:30:26.395816+0000","last_unstale":"2023-10-07T09:30:26.402685+0000","last_undegraded":"2023-10-07T09:30:26.402685+0000","last_fullsized":"2023-10-07T09:30:26.402685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.391928+0000","last_change":"2023-10-07T09:30:24.964522+0000","last_active":"2023-10-07T09:30:26.391928+0000","last_peered":"2023-10-07T09:30:26.391928+0000","last_clean":"2023-10-07T09:30:26.391928+0000","last_became_active":"2023-10-07T09:30:24.964221+0000","last_became_peered":"2023-10-07T09:30:24.964221+0000","last_unstale":"2023-10-07T09:30:26.391928+0000","last_undegraded":"2023-10-07T09:30:26.391928+0000","last_fullsized":"2023-10-07T09:30:26.391928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:35.312512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956354+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:12.844376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384014+0000","last_change":"2023-10-07T09:30:24.967550+0000","last_active":"2023-10-07T09:30:26.384014+0000","last_peered":"2023-10-07T09:30:26.384014+0000","last_clean":"2023-10-07T09:30:26.384014+0000","last_became_active":"2023-10-07T09:30:24.967265+0000","last_became_peered":"2023-10-07T09:30:24.967265+0000","last_unstale":"2023-10-07T09:30:26.384014+0000","last_undegraded":"2023-10-07T09:30:26.384014+0000","last_fullsized":"2023-10-07T09:30:26.384014+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:12.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.953547+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424346+0000","last_change":"2023-10-07T09:30:24.971275+0000","last_active":"2023-10-07T09:30:26.424346+0000","last_peered":"2023-10-07T09:30:26.424346+0000","last_clean":"2023-10-07T09:30:26.424346+0000","last_became_active":"2023-10-07T09:30:24.970929+0000","last_became_peered":"2023-10-07T09:30:24.970929+0000","last_unstale":"2023-10-07T09:30:26.424346+0000","last_undegraded":"2023-10-07T09:30:26.424346+0000","last_fullsized":"2023-10-07T09:30:26.424346+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951700+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423122+0000","last_change":"2023-10-07T09:30:24.968081+0000","last_active":"2023-10-07T09:30:26.423122+0000","last_peered":"2023-10-07T09:30:26.423122+0000","last_clean":"2023-10-07T09:30:26.423122+0000","last_became_active":"2023-10-07T09:30:24.967047+0000","last_became_peered":"2023-10-07T09:30:24.967047+0000","last_unstale":"2023-10-07T09:30:26.423122+0000","last_undegraded":"2023-10-07T09:30:26.423122+0000","last_fullsized":"2023-10-07T09:30:26.423122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:00.381039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413417+0000","last_change":"2023-10-07T09:30:26.407336+0000","last_active":"2023-10-07T09:30:26.413417+0000","last_peered":"2023-10-07T09:30:26.413417+0000","last_clean":"2023-10-07T09:30:26.413417+0000","last_became_active":"2023-10-07T09:30:26.407027+0000","last_became_peered":"2023-10-07T09:30:26.407027+0000","last_unstale":"2023-10-07T09:30:26.413417+0000","last_undegraded":"2023-10-07T09:30:26.413417+0000","last_fullsized":"2023-10-07T09:30:26.413417+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.962904+0000","last_change":"2023-10-07T09:30:24.962904+0000","last_active":"2023-10-07T09:30:24.962904+0000","last_peered":"2023-10-07T09:30:24.962904+0000","last_clean":"2023-10-07T09:30:24.962904+0000","last_became_active":"2023-10-07T09:30:24.962519+0000","last_became_peered":"2023-10-07T09:30:24.962519+0000","last_unstale":"2023-10-07T09:30:24.962904+0000","last_undegraded":"2023-10-07T09:30:24.962904+0000","last_fullsized":"2023-10-07T09:30:24.962904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:00.363653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407852+0000","last_change":"2023-10-07T09:30:26.403114+0000","last_active":"2023-10-07T09:30:26.407852+0000","last_peered":"2023-10-07T09:30:26.407852+0000","last_clean":"2023-10-07T09:30:26.407852+0000","last_became_active":"2023-10-07T09:30:26.402830+0000","last_became_peered":"2023-10-07T09:30:26.402830+0000","last_unstale":"2023-10-07T09:30:26.407852+0000","last_undegraded":"2023-10-07T09:30:26.407852+0000","last_fullsized":"2023-10-07T09:30:26.407852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.961853+0000","last_change":"2023-10-07T09:30:24.961853+0000","last_active":"2023-10-07T09:30:24.961853+0000","last_peered":"2023-10-07T09:30:24.961853+0000","last_clean":"2023-10-07T09:30:24.961853+0000","last_became_active":"2023-10-07T09:30:24.961591+0000","last_became_peered":"2023-10-07T09:30:24.961591+0000","last_unstale":"2023-10-07T09:30:24.961853+0000","last_undegraded":"2023-10-07T09:30:24.961853+0000","last_fullsized":"2023-10-07T09:30:24.961853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415642+0000","last_change":"2023-10-07T09:30:26.413595+0000","last_active":"2023-10-07T09:30:26.415642+0000","last_peered":"2023-10-07T09:30:26.415642+0000","last_clean":"2023-10-07T09:30:26.415642+0000","last_became_active":"2023-10-07T09:30:26.413270+0000","last_became_peered":"2023-10-07T09:30:26.413270+0000","last_unstale":"2023-10-07T09:30:26.415642+0000","last_undegraded":"2023-10-07T09:30:26.415642+0000","last_fullsized":"2023-10-07T09:30:26.415642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967115+0000","last_change":"2023-10-07T09:30:24.967115+0000","last_active":"2023-10-07T09:30:24.967115+0000","last_peered":"2023-10-07T09:30:24.967115+0000","last_clean":"2023-10-07T09:30:24.967115+0000","last_became_active":"2023-10-07T09:30:24.966832+0000","last_became_peered":"2023-10-07T09:30:24.966832+0000","last_unstale":"2023-10-07T09:30:24.967115+0000","last_undegraded":"2023-10-07T09:30:24.967115+0000","last_fullsized":"2023-10-07T09:30:24.967115+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:11:11.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413302+0000","last_change":"2023-10-07T09:30:26.408606+0000","last_active":"2023-10-07T09:30:26.413302+0000","last_peered":"2023-10-07T09:30:26.413302+0000","last_clean":"2023-10-07T09:30:26.413302+0000","last_became_active":"2023-10-07T09:30:26.408343+0000","last_became_peered":"2023-10-07T09:30:26.408343+0000","last_unstale":"2023-10-07T09:30:26.413302+0000","last_undegraded":"2023-10-07T09:30:26.413302+0000","last_fullsized":"2023-10-07T09:30:26.413302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423847+0000","last_change":"2023-10-07T09:30:24.963726+0000","last_active":"2023-10-07T09:30:26.423847+0000","last_peered":"2023-10-07T09:30:26.423847+0000","last_clean":"2023-10-07T09:30:26.423847+0000","last_became_active":"2023-10-07T09:30:24.963442+0000","last_became_peered":"2023-10-07T09:30:24.963442+0000","last_unstale":"2023-10-07T09:30:26.423847+0000","last_undegraded":"2023-10-07T09:30:26.423847+0000","last_fullsized":"2023-10-07T09:30:26.423847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:38.051902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413355+0000","last_change":"2023-10-07T09:30:26.406759+0000","last_active":"2023-10-07T09:30:26.413355+0000","last_peered":"2023-10-07T09:30:26.413355+0000","last_clean":"2023-10-07T09:30:26.413355+0000","last_became_active":"2023-10-07T09:30:26.406449+0000","last_became_peered":"2023-10-07T09:30:26.406449+0000","last_unstale":"2023-10-07T09:30:26.413355+0000","last_undegraded":"2023-10-07T09:30:26.413355+0000","last_fullsized":"2023-10-07T09:30:26.413355+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.970510+0000","last_change":"2023-10-07T09:30:24.970510+0000","last_active":"2023-10-07T09:30:24.970510+0000","last_peered":"2023-10-07T09:30:24.970510+0000","last_clean":"2023-10-07T09:30:24.970510+0000","last_became_active":"2023-10-07T09:30:24.970183+0000","last_became_peered":"2023-10-07T09:30:24.970183+0000","last_unstale":"2023-10-07T09:30:24.970510+0000","last_undegraded":"2023-10-07T09:30:24.970510+0000","last_fullsized":"2023-10-07T09:30:24.970510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412638+0000","last_change":"2023-10-07T09:30:26.404546+0000","last_active":"2023-10-07T09:30:26.412638+0000","last_peered":"2023-10-07T09:30:26.412638+0000","last_clean":"2023-10-07T09:30:26.412638+0000","last_became_active":"2023-10-07T09:30:26.404284+0000","last_became_peered":"2023-10-07T09:30:26.404284+0000","last_unstale":"2023-10-07T09:30:26.412638+0000","last_undegraded":"2023-10-07T09:30:26.412638+0000","last_fullsized":"2023-10-07T09:30:26.412638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.965098+0000","last_change":"2023-10-07T09:30:24.965098+0000","last_active":"2023-10-07T09:30:24.965098+0000","last_peered":"2023-10-07T09:30:24.965098+0000","last_clean":"2023-10-07T09:30:24.965098+0000","last_became_active":"2023-10-07T09:30:24.964594+0000","last_became_peered":"2023-10-07T09:30:24.964594+0000","last_unstale":"2023-10-07T09:30:24.965098+0000","last_undegraded":"2023-10-07T09:30:24.965098+0000","last_fullsized":"2023-10-07T09:30:24.965098+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:56.143330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955888+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:11:46.955055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.393481+0000","last_change":"2023-10-07T09:30:24.967401+0000","last_active":"2023-10-07T09:30:26.393481+0000","last_peered":"2023-10-07T09:30:26.393481+0000","last_clean":"2023-10-07T09:30:26.393481+0000","last_became_active":"2023-10-07T09:30:24.967138+0000","last_became_peered":"2023-10-07T09:30:24.967138+0000","last_unstale":"2023-10-07T09:30:26.393481+0000","last_undegraded":"2023-10-07T09:30:26.393481+0000","last_fullsized":"2023-10-07T09:30:26.393481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414869+0000","last_change":"2023-10-07T09:30:26.411476+0000","last_active":"2023-10-07T09:30:26.414869+0000","last_peered":"2023-10-07T09:30:26.414869+0000","last_clean":"2023-10-07T09:30:26.414869+0000","last_became_active":"2023-10-07T09:30:26.411198+0000","last_became_peered":"2023-10-07T09:30:26.411198+0000","last_unstale":"2023-10-07T09:30:26.414869+0000","last_undegraded":"2023-10-07T09:30:26.414869+0000","last_fullsized":"2023-10-07T09:30:26.414869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423028+0000","last_change":"2023-10-07T09:30:24.971777+0000","last_active":"2023-10-07T09:30:26.423028+0000","last_peered":"2023-10-07T09:30:26.423028+0000","last_clean":"2023-10-07T09:30:26.423028+0000","last_became_active":"2023-10-07T09:30:24.971526+0000","last_became_peered":"2023-10-07T09:30:24.971526+0000","last_unstale":"2023-10-07T09:30:26.423028+0000","last_undegraded":"2023-10-07T09:30:26.423028+0000","last_fullsized":"2023-10-07T09:30:26.423028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:00.376299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410968+0000","last_change":"2023-10-07T09:30:26.404652+0000","last_active":"2023-10-07T09:30:26.410968+0000","last_peered":"2023-10-07T09:30:26.410968+0000","last_clean":"2023-10-07T09:30:26.410968+0000","last_became_active":"2023-10-07T09:30:26.404410+0000","last_became_peered":"2023-10-07T09:30:26.404410+0000","last_unstale":"2023-10-07T09:30:26.410968+0000","last_undegraded":"2023-10-07T09:30:26.410968+0000","last_fullsized":"2023-10-07T09:30:26.410968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390466+0000","last_change":"2023-10-07T09:30:24.963621+0000","last_active":"2023-10-07T09:30:26.390466+0000","last_peered":"2023-10-07T09:30:26.390466+0000","last_clean":"2023-10-07T09:30:26.390466+0000","last_became_active":"2023-10-07T09:30:24.963315+0000","last_became_peered":"2023-10-07T09:30:24.963315+0000","last_unstale":"2023-10-07T09:30:26.390466+0000","last_undegraded":"2023-10-07T09:30:26.390466+0000","last_fullsized":"2023-10-07T09:30:26.390466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410981+0000","last_change":"2023-10-07T09:30:26.404627+0000","last_active":"2023-10-07T09:30:26.410981+0000","last_peered":"2023-10-07T09:30:26.410981+0000","last_clean":"2023-10-07T09:30:26.410981+0000","last_became_active":"2023-10-07T09:30:26.404362+0000","last_became_peered":"2023-10-07T09:30:26.404362+0000","last_unstale":"2023-10-07T09:30:26.410981+0000","last_undegraded":"2023-10-07T09:30:26.410981+0000","last_fullsized":"2023-10-07T09:30:26.410981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392082+0000","last_change":"2023-10-07T09:30:24.972623+0000","last_active":"2023-10-07T09:30:26.392082+0000","last_peered":"2023-10-07T09:30:26.392082+0000","last_clean":"2023-10-07T09:30:26.392082+0000","last_became_active":"2023-10-07T09:30:24.972377+0000","last_became_peered":"2023-10-07T09:30:24.972377+0000","last_unstale":"2023-10-07T09:30:26.392082+0000","last_undegraded":"2023-10-07T09:30:26.392082+0000","last_fullsized":"2023-10-07T09:30:26.392082+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:49.176868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955354+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390656+0000","last_change":"2023-10-07T09:30:24.968835+0000","last_active":"2023-10-07T09:30:26.390656+0000","last_peered":"2023-10-07T09:30:26.390656+0000","last_clean":"2023-10-07T09:30:26.390656+0000","last_became_active":"2023-10-07T09:30:24.968531+0000","last_became_peered":"2023-10-07T09:30:24.968531+0000","last_unstale":"2023-10-07T09:30:26.390656+0000","last_undegraded":"2023-10-07T09:30:26.390656+0000","last_fullsized":"2023-10-07T09:30:26.390656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:06:46.058261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418399+0000","last_change":"2023-10-07T09:30:26.413894+0000","last_active":"2023-10-07T09:30:26.418399+0000","last_peered":"2023-10-07T09:30:26.418399+0000","last_clean":"2023-10-07T09:30:26.418399+0000","last_became_active":"2023-10-07T09:30:26.413276+0000","last_became_peered":"2023-10-07T09:30:26.413276+0000","last_unstale":"2023-10-07T09:30:26.418399+0000","last_undegraded":"2023-10-07T09:30:26.418399+0000","last_fullsized":"2023-10-07T09:30:26.418399+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392045+0000","last_change":"2023-10-07T09:30:24.971933+0000","last_active":"2023-10-07T09:30:26.392045+0000","last_peered":"2023-10-07T09:30:26.392045+0000","last_clean":"2023-10-07T09:30:26.392045+0000","last_became_active":"2023-10-07T09:30:24.971478+0000","last_became_peered":"2023-10-07T09:30:24.971478+0000","last_unstale":"2023-10-07T09:30:26.392045+0000","last_undegraded":"2023-10-07T09:30:26.392045+0000","last_fullsized":"2023-10-07T09:30:26.392045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:19.116181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410100+0000","last_change":"2023-10-07T09:30:26.402235+0000","last_active":"2023-10-07T09:30:26.410100+0000","last_peered":"2023-10-07T09:30:26.410100+0000","last_clean":"2023-10-07T09:30:26.410100+0000","last_became_active":"2023-10-07T09:30:26.401915+0000","last_became_peered":"2023-10-07T09:30:26.401915+0000","last_unstale":"2023-10-07T09:30:26.410100+0000","last_undegraded":"2023-10-07T09:30:26.410100+0000","last_fullsized":"2023-10-07T09:30:26.410100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:01.010896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967230+0000","last_change":"2023-10-07T09:30:24.967230+0000","last_active":"2023-10-07T09:30:24.967230+0000","last_peered":"2023-10-07T09:30:24.967230+0000","last_clean":"2023-10-07T09:30:24.967230+0000","last_became_active":"2023-10-07T09:30:24.966740+0000","last_became_peered":"2023-10-07T09:30:24.966740+0000","last_unstale":"2023-10-07T09:30:24.967230+0000","last_undegraded":"2023-10-07T09:30:24.967230+0000","last_fullsized":"2023-10-07T09:30:24.967230+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:31:56.775021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950417+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384013+0000","last_change":"2023-10-07T09:30:24.967499+0000","last_active":"2023-10-07T09:30:26.384013+0000","last_peered":"2023-10-07T09:30:26.384013+0000","last_clean":"2023-10-07T09:30:26.384013+0000","last_became_active":"2023-10-07T09:30:24.967231+0000","last_became_peered":"2023-10-07T09:30:24.967231+0000","last_unstale":"2023-10-07T09:30:26.384013+0000","last_undegraded":"2023-10-07T09:30:26.384013+0000","last_fullsized":"2023-10-07T09:30:26.384013+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:23.529707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956851+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:23.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.971597+0000","last_change":"2023-10-07T09:30:24.971597+0000","last_active":"2023-10-07T09:30:24.971597+0000","last_peered":"2023-10-07T09:30:24.971597+0000","last_clean":"2023-10-07T09:30:24.971597+0000","last_became_active":"2023-10-07T09:30:24.971341+0000","last_became_peered":"2023-10-07T09:30:24.971341+0000","last_unstale":"2023-10-07T09:30:24.971597+0000","last_undegraded":"2023-10-07T09:30:24.971597+0000","last_fullsized":"2023-10-07T09:30:24.971597+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:15.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-07T09:30:23.945970+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:30:23.945970+0000","last_peered":"2023-10-07T09:30:23.945970+0000","last_clean":"2023-10-07T09:30:23.945970+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:30:23.945970+0000","last_undegraded":"2023-10-07T09:30:23.945970+0000","last_fullsized":"2023-10-07T09:30:23.945970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1648,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298236,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073604,"statfs":{"total":96636764160,"available":96331370496,"internally_reserved":0,"allocated":970752,"data_stored":735622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1,"5min":1,"15min":1},"last":0.88},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71399999999999997}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.89800000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.93300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.628}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.94899999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.81599999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074196,"statfs":{"total":96636764160,"available":96331976704,"internally_reserved":0,"allocated":364544,"data_stored":138999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.984,"5min":2.984,"15min":2.984},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":7.867,"5min":7.867,"15min":7.867},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":8.0399999999999991},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.61099999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.79500000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.93200000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.31},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.755}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.69699999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302356,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069484,"statfs":{"total":96636764160,"available":96327151616,"internally_reserved":0,"allocated":995328,"data_stored":748673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.726,"5min":2.726,"15min":2.726},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.57399999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.92200000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.66700000000000004}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.8199999999999998,"5min":3.8199999999999998,"15min":3.8199999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.627},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":3.855,"5min":3.855,"15min":3.855},"last":1.036}]},{"osd":7,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":3.778,"5min":3.778,"15min":3.778},"last":1.097}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297684,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074156,"statfs":{"total":96636764160,"available":96331935744,"internally_reserved":0,"allocated":405504,"data_stored":158375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.439}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":1.0609999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.876},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.94299999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.91400000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":0.314},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.40000000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.498},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.42699999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.877}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.85299999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.88}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":1.2889999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.276}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":6.0620000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074196,"statfs":{"total":96636764160,"available":96331976704,"internally_reserved":0,"allocated":364544,"data_stored":138999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.623}]},{"osd":3,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.80600000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.83799999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.76600000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.82299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:30:30.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:30:31.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.020+0000 7f773ddff700 1 Processor -- start 2023-10-07T09:30:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.020+0000 7f773ddff700 1 -- start start 2023-10-07T09:30:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f773ddff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7738159750 0x7f77380d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f773ddff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77380cbf20 con 0x7f77380ce8a0 2023-10-07T09:30:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f773ddff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77380cc200 con 0x7f7738159b20 2023-10-07T09:30:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f773ddff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77380cc4e0 con 0x7f7738159750 2023-10-07T09:30:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f77377fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7738159750 0x7f77380d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f7737fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f77380ce8a0 0x7f77380c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33472/0 (socket says 172.21.15.29:33472) 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f7737fff700 1 -- 172.21.15.29:0/2235413385 learned_addr learned my addr 172.21.15.29:0/2235413385 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.022+0000 7f77367fc700 1 -- 172.21.15.29:0/2235413385 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455063661 0 0) 0x7f77380cbf20 con 0x7f77380ce8a0 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77377fe700 1 -- 172.21.15.29:0/2235413385 >> v1:172.21.15.144:6789/0 conn(0x7f7738159b20 legacy=0x7f77380ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77377fe700 1 -- 172.21.15.29:0/2235413385 >> v1:172.21.15.29:6789/0 conn(0x7f77380ce8a0 legacy=0x7f77380c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77377fe700 1 -- 172.21.15.29:0/2235413385 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77380cc7c0 con 0x7f7738159750 2023-10-07T09:30:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77377fe700 1 --2- 172.21.15.29:0/2235413385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159750 0x7f77380d10a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f77280098d0 tx=0x7f77280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b66b06e982f1c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77367fc700 1 -- 172.21.15.29:0/2235413385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728009e70 con 0x7f7738159750 2023-10-07T09:30:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77367fc700 1 -- 172.21.15.29:0/2235413385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772800b860 con 0x7f7738159750 2023-10-07T09:30:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.023+0000 7f77367fc700 1 -- 172.21.15.29:0/2235413385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772800bc00 con 0x7f7738159750 2023-10-07T09:30:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 -- 172.21.15.29:0/2235413385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159750 msgr2=0x7f77380d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 --2- 172.21.15.29:0/2235413385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159750 0x7f77380d10a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f77280098d0 tx=0x7f77280091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 -- 172.21.15.29:0/2235413385 shutdown_connections 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 --2- 172.21.15.29:0/2235413385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159750 0x7f77380d10a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 -- 172.21.15.29:0/2235413385 >> 172.21.15.29:0/2235413385 conn(0x7f7738063a10 msgr2=0x7f77380d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 -- 172.21.15.29:0/2235413385 shutdown_connections 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.024+0000 7f773ddff700 1 -- 172.21.15.29:0/2235413385 wait complete. 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.025+0000 7f773ddff700 1 Processor -- start 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.025+0000 7f773ddff700 1 -- start start 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.026+0000 7f773ddff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77380ce8a0 0x7f77380b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.026+0000 7f773ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7738159750 0x7f7738147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.026+0000 7f773ddff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 0x7f7738069580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.026+0000 7f773ddff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77380b4f90 con 0x7f7738159b20 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.026+0000 7f773ddff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773815a970 con 0x7f77380ce8a0 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.026+0000 7f773ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77380b38d0 con 0x7f7738159750 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 0x7f7738069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 0x7f7738069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39712/0 (socket says 172.21.15.29:39712) 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 -- 172.21.15.29:0/1552153086 learned_addr learned my addr 172.21.15.29:0/1552153086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f77377fe700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77380ce8a0 0x7f77380b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7736ffd700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7738159750 0x7f7738147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 -- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7738159750 msgr2=0x7f7738147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7738159750 0x7f7738147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 -- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77380ce8a0 msgr2=0x7f77380b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77380ce8a0 0x7f77380b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 -- 172.21.15.29:0/1552153086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280093c0 con 0x7f7738159b20 2023-10-07T09:30:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.027+0000 7f7737fff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 0x7f7738069580 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f773000cc90 tx=0x7f7730001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.028+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773000cea0 con 0x7f7738159b20 2023-10-07T09:30:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.028+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773000ab70 con 0x7f7738159b20 2023-10-07T09:30:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.028+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77380b3bb0 con 0x7f7738159b20 2023-10-07T09:30:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.028+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77300070f0 con 0x7f7738159b20 2023-10-07T09:30:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.028+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77380b41f0 con 0x7f7738159b20 2023-10-07T09:30:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.029+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77380b5860 con 0x7f7738159b20 2023-10-07T09:30:31.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.030+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f77300074a0 con 0x7f7738159b20 2023-10-07T09:30:31.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.031+0000 7f771f7fe700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7724066f30 0x7f77240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.031+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7730084a40 con 0x7f7738159b20 2023-10-07T09:30:31.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.032+0000 7f77377fe700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7724066f30 0x7f77240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.032+0000 7f77377fe700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7724066f30 0x7f77240693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7728009390 tx=0x7f7728017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.035+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7730051c90 con 0x7f7738159b20 2023-10-07T09:30:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.232+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f773815a740 con 0x7f7724066f30 2023-10-07T09:30:31.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.241+0000 7f771f7fe700 1 -- 172.21.15.29:0/1552153086 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287592 (secure 0 0 0) 0x7f773815a740 con 0x7f7724066f30 2023-10-07T09:30:31.245 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7724066f30 msgr2=0x7f77240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7724066f30 0x7f77240693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7728009390 tx=0x7f7728017000 comp rx=0 tx=0).stop 2023-10-07T09:30:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 msgr2=0x7f7738069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 0x7f7738069580 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f773000cc90 tx=0x7f7730001a80 comp rx=0 tx=0).stop 2023-10-07T09:30:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 shutdown_connections 2023-10-07T09:30:31.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7724066f30 0x7f77240693e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7738159b20 0x7f7738069580 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77380ce8a0 0x7f77380b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 --2- 172.21.15.29:0/1552153086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7738159750 0x7f7738147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 >> 172.21.15.29:0/1552153086 conn(0x7f7738063a10 msgr2=0x7f7738155c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 shutdown_connections 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.245+0000 7f773ddff700 1 -- 172.21.15.29:0/1552153086 wait complete. 2023-10-07T09:30:31.249 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:30:31.263 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-10-07T09:30:30.034161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592016,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":192,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2386644,"kb_used_data":4244,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752588076,"statfs":{"total":773094113280,"available":770650189824,"internally_reserved":0,"allocated":4345856,"data_stored":2399343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1648,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001688"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956469+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415669+0000","last_change":"2023-10-07T09:30:26.408750+0000","last_active":"2023-10-07T09:30:26.415669+0000","last_peered":"2023-10-07T09:30:26.415669+0000","last_clean":"2023-10-07T09:30:26.415669+0000","last_became_active":"2023-10-07T09:30:26.407823+0000","last_became_peered":"2023-10-07T09:30:26.407823+0000","last_unstale":"2023-10-07T09:30:26.415669+0000","last_undegraded":"2023-10-07T09:30:26.415669+0000","last_fullsized":"2023-10-07T09:30:26.415669+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:58.672867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414461+0000","last_change":"2023-10-07T09:30:26.408350+0000","last_active":"2023-10-07T09:30:26.414461+0000","last_peered":"2023-10-07T09:30:26.414461+0000","last_clean":"2023-10-07T09:30:26.414461+0000","last_became_active":"2023-10-07T09:30:26.408029+0000","last_became_peered":"2023-10-07T09:30:26.408029+0000","last_unstale":"2023-10-07T09:30:26.414461+0000","last_undegraded":"2023-10-07T09:30:26.414461+0000","last_fullsized":"2023-10-07T09:30:26.414461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407832+0000","last_change":"2023-10-07T09:30:26.403115+0000","last_active":"2023-10-07T09:30:26.407832+0000","last_peered":"2023-10-07T09:30:26.407832+0000","last_clean":"2023-10-07T09:30:26.407832+0000","last_became_active":"2023-10-07T09:30:26.402699+0000","last_became_peered":"2023-10-07T09:30:26.402699+0000","last_unstale":"2023-10-07T09:30:26.407832+0000","last_undegraded":"2023-10-07T09:30:26.407832+0000","last_fullsized":"2023-10-07T09:30:26.407832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:06:53.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414913+0000","last_change":"2023-10-07T09:30:26.410770+0000","last_active":"2023-10-07T09:30:26.414913+0000","last_peered":"2023-10-07T09:30:26.414913+0000","last_clean":"2023-10-07T09:30:26.414913+0000","last_became_active":"2023-10-07T09:30:26.410465+0000","last_became_peered":"2023-10-07T09:30:26.410465+0000","last_unstale":"2023-10-07T09:30:26.414913+0000","last_undegraded":"2023-10-07T09:30:26.414913+0000","last_fullsized":"2023-10-07T09:30:26.414913+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.403447+0000","last_change":"2023-10-07T09:30:26.395345+0000","last_active":"2023-10-07T09:30:26.403447+0000","last_peered":"2023-10-07T09:30:26.403447+0000","last_clean":"2023-10-07T09:30:26.403447+0000","last_became_active":"2023-10-07T09:30:26.395042+0000","last_became_peered":"2023-10-07T09:30:26.395042+0000","last_unstale":"2023-10-07T09:30:26.403447+0000","last_undegraded":"2023-10-07T09:30:26.403447+0000","last_fullsized":"2023-10-07T09:30:26.403447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956440+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:09.240847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957353+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:20:14.772461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951211+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.399392+0000","last_change":"2023-10-07T09:30:26.396209+0000","last_active":"2023-10-07T09:30:26.399392+0000","last_peered":"2023-10-07T09:30:26.399392+0000","last_clean":"2023-10-07T09:30:26.399392+0000","last_became_active":"2023-10-07T09:30:26.395573+0000","last_became_peered":"2023-10-07T09:30:26.395573+0000","last_unstale":"2023-10-07T09:30:26.399392+0000","last_undegraded":"2023-10-07T09:30:26.399392+0000","last_fullsized":"2023-10-07T09:30:26.399392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:54.021183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957373+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:17.334679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413491+0000","last_change":"2023-10-07T09:30:26.409670+0000","last_active":"2023-10-07T09:30:26.413491+0000","last_peered":"2023-10-07T09:30:26.413491+0000","last_clean":"2023-10-07T09:30:26.413491+0000","last_became_active":"2023-10-07T09:30:26.408687+0000","last_became_peered":"2023-10-07T09:30:26.408687+0000","last_unstale":"2023-10-07T09:30:26.413491+0000","last_undegraded":"2023-10-07T09:30:26.413491+0000","last_fullsized":"2023-10-07T09:30:26.413491+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413335+0000","last_change":"2023-10-07T09:30:26.406082+0000","last_active":"2023-10-07T09:30:26.413335+0000","last_peered":"2023-10-07T09:30:26.413335+0000","last_clean":"2023-10-07T09:30:26.413335+0000","last_became_active":"2023-10-07T09:30:26.405724+0000","last_became_peered":"2023-10-07T09:30:26.405724+0000","last_unstale":"2023-10-07T09:30:26.413335+0000","last_undegraded":"2023-10-07T09:30:26.413335+0000","last_fullsized":"2023-10-07T09:30:26.413335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411102+0000","last_change":"2023-10-07T09:30:26.403342+0000","last_active":"2023-10-07T09:30:26.411102+0000","last_peered":"2023-10-07T09:30:26.411102+0000","last_clean":"2023-10-07T09:30:26.411102+0000","last_became_active":"2023-10-07T09:30:26.403138+0000","last_became_peered":"2023-10-07T09:30:26.403138+0000","last_unstale":"2023-10-07T09:30:26.411102+0000","last_undegraded":"2023-10-07T09:30:26.411102+0000","last_fullsized":"2023-10-07T09:30:26.411102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408435+0000","last_change":"2023-10-07T09:30:26.395877+0000","last_active":"2023-10-07T09:30:26.408435+0000","last_peered":"2023-10-07T09:30:26.408435+0000","last_clean":"2023-10-07T09:30:26.408435+0000","last_became_active":"2023-10-07T09:30:26.395596+0000","last_became_peered":"2023-10-07T09:30:26.395596+0000","last_unstale":"2023-10-07T09:30:26.408435+0000","last_undegraded":"2023-10-07T09:30:26.408435+0000","last_fullsized":"2023-10-07T09:30:26.408435+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:20.489580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956822+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:20:47.855402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957600+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:19.142808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409109+0000","last_change":"2023-10-07T09:30:26.394655+0000","last_active":"2023-10-07T09:30:26.409109+0000","last_peered":"2023-10-07T09:30:26.409109+0000","last_clean":"2023-10-07T09:30:26.409109+0000","last_became_active":"2023-10-07T09:30:26.394372+0000","last_became_peered":"2023-10-07T09:30:26.394372+0000","last_unstale":"2023-10-07T09:30:26.409109+0000","last_undegraded":"2023-10-07T09:30:26.409109+0000","last_fullsized":"2023-10-07T09:30:26.409109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:35.770412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405481+0000","last_change":"2023-10-07T09:30:26.402289+0000","last_active":"2023-10-07T09:30:26.405481+0000","last_peered":"2023-10-07T09:30:26.405481+0000","last_clean":"2023-10-07T09:30:26.405481+0000","last_became_active":"2023-10-07T09:30:26.401826+0000","last_became_peered":"2023-10-07T09:30:26.401826+0000","last_unstale":"2023-10-07T09:30:26.405481+0000","last_undegraded":"2023-10-07T09:30:26.405481+0000","last_fullsized":"2023-10-07T09:30:26.405481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:46:15.007773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414990+0000","last_change":"2023-10-07T09:30:26.411654+0000","last_active":"2023-10-07T09:30:26.414990+0000","last_peered":"2023-10-07T09:30:26.414990+0000","last_clean":"2023-10-07T09:30:26.414990+0000","last_became_active":"2023-10-07T09:30:26.411407+0000","last_became_peered":"2023-10-07T09:30:26.411407+0000","last_unstale":"2023-10-07T09:30:26.414990+0000","last_undegraded":"2023-10-07T09:30:26.414990+0000","last_fullsized":"2023-10-07T09:30:26.414990+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951679+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.949829+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:03.694836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410992+0000","last_change":"2023-10-07T09:30:26.403259+0000","last_active":"2023-10-07T09:30:26.410992+0000","last_peered":"2023-10-07T09:30:26.410992+0000","last_clean":"2023-10-07T09:30:26.410992+0000","last_became_active":"2023-10-07T09:30:26.402926+0000","last_became_peered":"2023-10-07T09:30:26.402926+0000","last_unstale":"2023-10-07T09:30:26.410992+0000","last_undegraded":"2023-10-07T09:30:26.410992+0000","last_fullsized":"2023-10-07T09:30:26.410992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413211+0000","last_change":"2023-10-07T09:30:26.409044+0000","last_active":"2023-10-07T09:30:26.413211+0000","last_peered":"2023-10-07T09:30:26.413211+0000","last_clean":"2023-10-07T09:30:26.413211+0000","last_became_active":"2023-10-07T09:30:26.408764+0000","last_became_peered":"2023-10-07T09:30:26.408764+0000","last_unstale":"2023-10-07T09:30:26.413211+0000","last_undegraded":"2023-10-07T09:30:26.413211+0000","last_fullsized":"2023-10-07T09:30:26.413211+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:47.675483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422810+0000","last_change":"2023-10-07T09:30:26.413627+0000","last_active":"2023-10-07T09:30:26.422810+0000","last_peered":"2023-10-07T09:30:26.422810+0000","last_clean":"2023-10-07T09:30:26.422810+0000","last_became_active":"2023-10-07T09:30:26.413371+0000","last_became_peered":"2023-10-07T09:30:26.413371+0000","last_unstale":"2023-10-07T09:30:26.422810+0000","last_undegraded":"2023-10-07T09:30:26.422810+0000","last_fullsized":"2023-10-07T09:30:26.422810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:34.133968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415691+0000","last_change":"2023-10-07T09:30:26.408999+0000","last_active":"2023-10-07T09:30:26.415691+0000","last_peered":"2023-10-07T09:30:26.415691+0000","last_clean":"2023-10-07T09:30:26.415691+0000","last_became_active":"2023-10-07T09:30:26.408201+0000","last_became_peered":"2023-10-07T09:30:26.408201+0000","last_unstale":"2023-10-07T09:30:26.415691+0000","last_undegraded":"2023-10-07T09:30:26.415691+0000","last_fullsized":"2023-10-07T09:30:26.415691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421410+0000","last_change":"2023-10-07T09:30:26.411198+0000","last_active":"2023-10-07T09:30:26.421410+0000","last_peered":"2023-10-07T09:30:26.421410+0000","last_clean":"2023-10-07T09:30:26.421410+0000","last_became_active":"2023-10-07T09:30:26.410735+0000","last_became_peered":"2023-10-07T09:30:26.410735+0000","last_unstale":"2023-10-07T09:30:26.421410+0000","last_undegraded":"2023-10-07T09:30:26.421410+0000","last_fullsized":"2023-10-07T09:30:26.421410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:24.740454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413473+0000","last_change":"2023-10-07T09:30:26.409340+0000","last_active":"2023-10-07T09:30:26.413473+0000","last_peered":"2023-10-07T09:30:26.413473+0000","last_clean":"2023-10-07T09:30:26.413473+0000","last_became_active":"2023-10-07T09:30:26.408986+0000","last_became_peered":"2023-10-07T09:30:26.408986+0000","last_unstale":"2023-10-07T09:30:26.413473+0000","last_undegraded":"2023-10-07T09:30:26.413473+0000","last_fullsized":"2023-10-07T09:30:26.413473+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950988+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955210+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:25.231695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.954033+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955840+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:30.437449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418432+0000","last_change":"2023-10-07T09:30:26.413698+0000","last_active":"2023-10-07T09:30:26.418432+0000","last_peered":"2023-10-07T09:30:26.418432+0000","last_clean":"2023-10-07T09:30:26.418432+0000","last_became_active":"2023-10-07T09:30:26.413219+0000","last_became_peered":"2023-10-07T09:30:26.413219+0000","last_unstale":"2023-10-07T09:30:26.418432+0000","last_undegraded":"2023-10-07T09:30:26.418432+0000","last_fullsized":"2023-10-07T09:30:26.418432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.970001+0000","last_change":"2023-10-07T09:30:24.970001+0000","last_active":"2023-10-07T09:30:24.970001+0000","last_peered":"2023-10-07T09:30:24.970001+0000","last_clean":"2023-10-07T09:30:24.970001+0000","last_became_active":"2023-10-07T09:30:24.969692+0000","last_became_peered":"2023-10-07T09:30:24.969692+0000","last_unstale":"2023-10-07T09:30:24.970001+0000","last_undegraded":"2023-10-07T09:30:24.970001+0000","last_fullsized":"2023-10-07T09:30:24.970001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956067+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421779+0000","last_change":"2023-10-07T09:30:24.961003+0000","last_active":"2023-10-07T09:30:26.421779+0000","last_peered":"2023-10-07T09:30:26.421779+0000","last_clean":"2023-10-07T09:30:26.421779+0000","last_became_active":"2023-10-07T09:30:24.960711+0000","last_became_peered":"2023-10-07T09:30:24.960711+0000","last_unstale":"2023-10-07T09:30:26.421779+0000","last_undegraded":"2023-10-07T09:30:26.421779+0000","last_fullsized":"2023-10-07T09:30:26.421779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:06.545101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950450+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:07.548280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424146+0000","last_change":"2023-10-07T09:30:24.967737+0000","last_active":"2023-10-07T09:30:26.424146+0000","last_peered":"2023-10-07T09:30:26.424146+0000","last_clean":"2023-10-07T09:30:26.424146+0000","last_became_active":"2023-10-07T09:30:24.966888+0000","last_became_peered":"2023-10-07T09:30:24.966888+0000","last_unstale":"2023-10-07T09:30:26.424146+0000","last_undegraded":"2023-10-07T09:30:26.424146+0000","last_fullsized":"2023-10-07T09:30:26.424146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421389+0000","last_change":"2023-10-07T09:30:26.411013+0000","last_active":"2023-10-07T09:30:26.421389+0000","last_peered":"2023-10-07T09:30:26.421389+0000","last_clean":"2023-10-07T09:30:26.421389+0000","last_became_active":"2023-10-07T09:30:26.410413+0000","last_became_peered":"2023-10-07T09:30:26.410413+0000","last_unstale":"2023-10-07T09:30:26.421389+0000","last_undegraded":"2023-10-07T09:30:26.421389+0000","last_fullsized":"2023-10-07T09:30:26.421389+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:06.340913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423552+0000","last_change":"2023-10-07T09:30:24.966147+0000","last_active":"2023-10-07T09:30:26.423552+0000","last_peered":"2023-10-07T09:30:26.423552+0000","last_clean":"2023-10-07T09:30:26.423552+0000","last_became_active":"2023-10-07T09:30:24.965853+0000","last_became_peered":"2023-10-07T09:30:24.965853+0000","last_unstale":"2023-10-07T09:30:26.423552+0000","last_undegraded":"2023-10-07T09:30:26.423552+0000","last_fullsized":"2023-10-07T09:30:26.423552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.954591+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:38:03.049978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967094+0000","last_change":"2023-10-07T09:30:24.967094+0000","last_active":"2023-10-07T09:30:24.967094+0000","last_peered":"2023-10-07T09:30:24.967094+0000","last_clean":"2023-10-07T09:30:24.967094+0000","last_became_active":"2023-10-07T09:30:24.966810+0000","last_became_peered":"2023-10-07T09:30:24.966810+0000","last_unstale":"2023-10-07T09:30:24.967094+0000","last_undegraded":"2023-10-07T09:30:24.967094+0000","last_fullsized":"2023-10-07T09:30:24.967094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:24:57.676848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414299+0000","last_change":"2023-10-07T09:30:26.407749+0000","last_active":"2023-10-07T09:30:26.414299+0000","last_peered":"2023-10-07T09:30:26.414299+0000","last_clean":"2023-10-07T09:30:26.414299+0000","last_became_active":"2023-10-07T09:30:26.407433+0000","last_became_peered":"2023-10-07T09:30:26.407433+0000","last_unstale":"2023-10-07T09:30:26.414299+0000","last_undegraded":"2023-10-07T09:30:26.414299+0000","last_fullsized":"2023-10-07T09:30:26.414299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.963628+0000","last_change":"2023-10-07T09:30:24.963628+0000","last_active":"2023-10-07T09:30:24.963628+0000","last_peered":"2023-10-07T09:30:24.963628+0000","last_clean":"2023-10-07T09:30:24.963628+0000","last_became_active":"2023-10-07T09:30:24.963306+0000","last_became_peered":"2023-10-07T09:30:24.963306+0000","last_unstale":"2023-10-07T09:30:24.963628+0000","last_undegraded":"2023-10-07T09:30:24.963628+0000","last_fullsized":"2023-10-07T09:30:24.963628+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:12.068751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.949818+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:26.104877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.383857+0000","last_change":"2023-10-07T09:30:24.965572+0000","last_active":"2023-10-07T09:30:26.383857+0000","last_peered":"2023-10-07T09:30:26.383857+0000","last_clean":"2023-10-07T09:30:26.383857+0000","last_became_active":"2023-10-07T09:30:24.965320+0000","last_became_peered":"2023-10-07T09:30:24.965320+0000","last_unstale":"2023-10-07T09:30:26.383857+0000","last_undegraded":"2023-10-07T09:30:26.383857+0000","last_fullsized":"2023-10-07T09:30:26.383857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:49.978481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414174+0000","last_change":"2023-10-07T09:30:26.409216+0000","last_active":"2023-10-07T09:30:26.414174+0000","last_peered":"2023-10-07T09:30:26.414174+0000","last_clean":"2023-10-07T09:30:26.414174+0000","last_became_active":"2023-10-07T09:30:26.408947+0000","last_became_peered":"2023-10-07T09:30:26.408947+0000","last_unstale":"2023-10-07T09:30:26.414174+0000","last_undegraded":"2023-10-07T09:30:26.414174+0000","last_fullsized":"2023-10-07T09:30:26.414174+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.962784+0000","last_change":"2023-10-07T09:30:24.962784+0000","last_active":"2023-10-07T09:30:24.962784+0000","last_peered":"2023-10-07T09:30:24.962784+0000","last_clean":"2023-10-07T09:30:24.962784+0000","last_became_active":"2023-10-07T09:30:24.962409+0000","last_became_peered":"2023-10-07T09:30:24.962409+0000","last_unstale":"2023-10-07T09:30:24.962784+0000","last_undegraded":"2023-10-07T09:30:24.962784+0000","last_fullsized":"2023-10-07T09:30:24.962784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:04.344607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413097+0000","last_change":"2023-10-07T09:30:26.406965+0000","last_active":"2023-10-07T09:30:26.413097+0000","last_peered":"2023-10-07T09:30:26.413097+0000","last_clean":"2023-10-07T09:30:26.413097+0000","last_became_active":"2023-10-07T09:30:26.406607+0000","last_became_peered":"2023-10-07T09:30:26.406607+0000","last_unstale":"2023-10-07T09:30:26.413097+0000","last_undegraded":"2023-10-07T09:30:26.413097+0000","last_fullsized":"2023-10-07T09:30:26.413097+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384959+0000","last_change":"2023-10-07T09:30:24.967714+0000","last_active":"2023-10-07T09:30:26.384959+0000","last_peered":"2023-10-07T09:30:26.384959+0000","last_clean":"2023-10-07T09:30:26.384959+0000","last_became_active":"2023-10-07T09:30:24.967416+0000","last_became_peered":"2023-10-07T09:30:24.967416+0000","last_unstale":"2023-10-07T09:30:26.384959+0000","last_undegraded":"2023-10-07T09:30:26.384959+0000","last_fullsized":"2023-10-07T09:30:26.384959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:25.781083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402385+0000","last_change":"2023-10-07T09:30:26.395987+0000","last_active":"2023-10-07T09:30:26.402385+0000","last_peered":"2023-10-07T09:30:26.402385+0000","last_clean":"2023-10-07T09:30:26.402385+0000","last_became_active":"2023-10-07T09:30:26.395709+0000","last_became_peered":"2023-10-07T09:30:26.395709+0000","last_unstale":"2023-10-07T09:30:26.402385+0000","last_undegraded":"2023-10-07T09:30:26.402385+0000","last_fullsized":"2023-10-07T09:30:26.402385+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390349+0000","last_change":"2023-10-07T09:30:24.962930+0000","last_active":"2023-10-07T09:30:26.390349+0000","last_peered":"2023-10-07T09:30:26.390349+0000","last_clean":"2023-10-07T09:30:26.390349+0000","last_became_active":"2023-10-07T09:30:24.962586+0000","last_became_peered":"2023-10-07T09:30:24.962586+0000","last_unstale":"2023-10-07T09:30:26.390349+0000","last_undegraded":"2023-10-07T09:30:26.390349+0000","last_fullsized":"2023-10-07T09:30:26.390349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.417232+0000","last_change":"2023-10-07T09:30:26.412395+0000","last_active":"2023-10-07T09:30:26.417232+0000","last_peered":"2023-10-07T09:30:26.417232+0000","last_clean":"2023-10-07T09:30:26.417232+0000","last_became_active":"2023-10-07T09:30:26.412085+0000","last_became_peered":"2023-10-07T09:30:26.412085+0000","last_unstale":"2023-10-07T09:30:26.417232+0000","last_undegraded":"2023-10-07T09:30:26.417232+0000","last_fullsized":"2023-10-07T09:30:26.417232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421318+0000","last_change":"2023-10-07T09:30:24.967809+0000","last_active":"2023-10-07T09:30:26.421318+0000","last_peered":"2023-10-07T09:30:26.421318+0000","last_clean":"2023-10-07T09:30:26.421318+0000","last_became_active":"2023-10-07T09:30:24.967550+0000","last_became_peered":"2023-10-07T09:30:24.967550+0000","last_unstale":"2023-10-07T09:30:26.421318+0000","last_undegraded":"2023-10-07T09:30:26.421318+0000","last_fullsized":"2023-10-07T09:30:26.421318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414443+0000","last_change":"2023-10-07T09:30:26.408025+0000","last_active":"2023-10-07T09:30:26.414443+0000","last_peered":"2023-10-07T09:30:26.414443+0000","last_clean":"2023-10-07T09:30:26.414443+0000","last_became_active":"2023-10-07T09:30:26.407736+0000","last_became_peered":"2023-10-07T09:30:26.407736+0000","last_unstale":"2023-10-07T09:30:26.414443+0000","last_undegraded":"2023-10-07T09:30:26.414443+0000","last_fullsized":"2023-10-07T09:30:26.414443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.425863+0000","last_change":"2023-10-07T09:30:24.966563+0000","last_active":"2023-10-07T09:30:26.425863+0000","last_peered":"2023-10-07T09:30:26.425863+0000","last_clean":"2023-10-07T09:30:26.425863+0000","last_became_active":"2023-10-07T09:30:24.966273+0000","last_became_peered":"2023-10-07T09:30:24.966273+0000","last_unstale":"2023-10-07T09:30:26.425863+0000","last_undegraded":"2023-10-07T09:30:26.425863+0000","last_fullsized":"2023-10-07T09:30:26.425863+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:41.648356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421340+0000","last_change":"2023-10-07T09:30:26.409745+0000","last_active":"2023-10-07T09:30:26.421340+0000","last_peered":"2023-10-07T09:30:26.421340+0000","last_clean":"2023-10-07T09:30:26.421340+0000","last_became_active":"2023-10-07T09:30:26.409489+0000","last_became_peered":"2023-10-07T09:30:26.409489+0000","last_unstale":"2023-10-07T09:30:26.421340+0000","last_undegraded":"2023-10-07T09:30:26.421340+0000","last_fullsized":"2023-10-07T09:30:26.421340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.964382+0000","last_change":"2023-10-07T09:30:24.964382+0000","last_active":"2023-10-07T09:30:24.964382+0000","last_peered":"2023-10-07T09:30:24.964382+0000","last_clean":"2023-10-07T09:30:24.964382+0000","last_became_active":"2023-10-07T09:30:24.964076+0000","last_became_peered":"2023-10-07T09:30:24.964076+0000","last_unstale":"2023-10-07T09:30:24.964382+0000","last_undegraded":"2023-10-07T09:30:24.964382+0000","last_fullsized":"2023-10-07T09:30:24.964382+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:19:17.590704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402685+0000","last_change":"2023-10-07T09:30:26.396545+0000","last_active":"2023-10-07T09:30:26.402685+0000","last_peered":"2023-10-07T09:30:26.402685+0000","last_clean":"2023-10-07T09:30:26.402685+0000","last_became_active":"2023-10-07T09:30:26.395816+0000","last_became_peered":"2023-10-07T09:30:26.395816+0000","last_unstale":"2023-10-07T09:30:26.402685+0000","last_undegraded":"2023-10-07T09:30:26.402685+0000","last_fullsized":"2023-10-07T09:30:26.402685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.391928+0000","last_change":"2023-10-07T09:30:24.964522+0000","last_active":"2023-10-07T09:30:26.391928+0000","last_peered":"2023-10-07T09:30:26.391928+0000","last_clean":"2023-10-07T09:30:26.391928+0000","last_became_active":"2023-10-07T09:30:24.964221+0000","last_became_peered":"2023-10-07T09:30:24.964221+0000","last_unstale":"2023-10-07T09:30:26.391928+0000","last_undegraded":"2023-10-07T09:30:26.391928+0000","last_fullsized":"2023-10-07T09:30:26.391928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:35.312512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956354+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:12.844376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384014+0000","last_change":"2023-10-07T09:30:24.967550+0000","last_active":"2023-10-07T09:30:26.384014+0000","last_peered":"2023-10-07T09:30:26.384014+0000","last_clean":"2023-10-07T09:30:26.384014+0000","last_became_active":"2023-10-07T09:30:24.967265+0000","last_became_peered":"2023-10-07T09:30:24.967265+0000","last_unstale":"2023-10-07T09:30:26.384014+0000","last_undegraded":"2023-10-07T09:30:26.384014+0000","last_fullsized":"2023-10-07T09:30:26.384014+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:12.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.953547+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424346+0000","last_change":"2023-10-07T09:30:24.971275+0000","last_active":"2023-10-07T09:30:26.424346+0000","last_peered":"2023-10-07T09:30:26.424346+0000","last_clean":"2023-10-07T09:30:26.424346+0000","last_became_active":"2023-10-07T09:30:24.970929+0000","last_became_peered":"2023-10-07T09:30:24.970929+0000","last_unstale":"2023-10-07T09:30:26.424346+0000","last_undegraded":"2023-10-07T09:30:26.424346+0000","last_fullsized":"2023-10-07T09:30:26.424346+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951700+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423122+0000","last_change":"2023-10-07T09:30:24.968081+0000","last_active":"2023-10-07T09:30:26.423122+0000","last_peered":"2023-10-07T09:30:26.423122+0000","last_clean":"2023-10-07T09:30:26.423122+0000","last_became_active":"2023-10-07T09:30:24.967047+0000","last_became_peered":"2023-10-07T09:30:24.967047+0000","last_unstale":"2023-10-07T09:30:26.423122+0000","last_undegraded":"2023-10-07T09:30:26.423122+0000","last_fullsized":"2023-10-07T09:30:26.423122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:00.381039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413417+0000","last_change":"2023-10-07T09:30:26.407336+0000","last_active":"2023-10-07T09:30:26.413417+0000","last_peered":"2023-10-07T09:30:26.413417+0000","last_clean":"2023-10-07T09:30:26.413417+0000","last_became_active":"2023-10-07T09:30:26.407027+0000","last_became_peered":"2023-10-07T09:30:26.407027+0000","last_unstale":"2023-10-07T09:30:26.413417+0000","last_undegraded":"2023-10-07T09:30:26.413417+0000","last_fullsized":"2023-10-07T09:30:26.413417+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.962904+0000","last_change":"2023-10-07T09:30:24.962904+0000","last_active":"2023-10-07T09:30:24.962904+0000","last_peered":"2023-10-07T09:30:24.962904+0000","last_clean":"2023-10-07T09:30:24.962904+0000","last_became_active":"2023-10-07T09:30:24.962519+0000","last_became_peered":"2023-10-07T09:30:24.962519+0000","last_unstale":"2023-10-07T09:30:24.962904+0000","last_undegraded":"2023-10-07T09:30:24.962904+0000","last_fullsized":"2023-10-07T09:30:24.962904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:00.363653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407852+0000","last_change":"2023-10-07T09:30:26.403114+0000","last_active":"2023-10-07T09:30:26.407852+0000","last_peered":"2023-10-07T09:30:26.407852+0000","last_clean":"2023-10-07T09:30:26.407852+0000","last_became_active":"2023-10-07T09:30:26.402830+0000","last_became_peered":"2023-10-07T09:30:26.402830+0000","last_unstale":"2023-10-07T09:30:26.407852+0000","last_undegraded":"2023-10-07T09:30:26.407852+0000","last_fullsized":"2023-10-07T09:30:26.407852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.961853+0000","last_change":"2023-10-07T09:30:24.961853+0000","last_active":"2023-10-07T09:30:24.961853+0000","last_peered":"2023-10-07T09:30:24.961853+0000","last_clean":"2023-10-07T09:30:24.961853+0000","last_became_active":"2023-10-07T09:30:24.961591+0000","last_became_peered":"2023-10-07T09:30:24.961591+0000","last_unstale":"2023-10-07T09:30:24.961853+0000","last_undegraded":"2023-10-07T09:30:24.961853+0000","last_fullsized":"2023-10-07T09:30:24.961853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415642+0000","last_change":"2023-10-07T09:30:26.413595+0000","last_active":"2023-10-07T09:30:26.415642+0000","last_peered":"2023-10-07T09:30:26.415642+0000","last_clean":"2023-10-07T09:30:26.415642+0000","last_became_active":"2023-10-07T09:30:26.413270+0000","last_became_peered":"2023-10-07T09:30:26.413270+0000","last_unstale":"2023-10-07T09:30:26.415642+0000","last_undegraded":"2023-10-07T09:30:26.415642+0000","last_fullsized":"2023-10-07T09:30:26.415642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967115+0000","last_change":"2023-10-07T09:30:24.967115+0000","last_active":"2023-10-07T09:30:24.967115+0000","last_peered":"2023-10-07T09:30:24.967115+0000","last_clean":"2023-10-07T09:30:24.967115+0000","last_became_active":"2023-10-07T09:30:24.966832+0000","last_became_peered":"2023-10-07T09:30:24.966832+0000","last_unstale":"2023-10-07T09:30:24.967115+0000","last_undegraded":"2023-10-07T09:30:24.967115+0000","last_fullsized":"2023-10-07T09:30:24.967115+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:11:11.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413302+0000","last_change":"2023-10-07T09:30:26.408606+0000","last_active":"2023-10-07T09:30:26.413302+0000","last_peered":"2023-10-07T09:30:26.413302+0000","last_clean":"2023-10-07T09:30:26.413302+0000","last_became_active":"2023-10-07T09:30:26.408343+0000","last_became_peered":"2023-10-07T09:30:26.408343+0000","last_unstale":"2023-10-07T09:30:26.413302+0000","last_undegraded":"2023-10-07T09:30:26.413302+0000","last_fullsized":"2023-10-07T09:30:26.413302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423847+0000","last_change":"2023-10-07T09:30:24.963726+0000","last_active":"2023-10-07T09:30:26.423847+0000","last_peered":"2023-10-07T09:30:26.423847+0000","last_clean":"2023-10-07T09:30:26.423847+0000","last_became_active":"2023-10-07T09:30:24.963442+0000","last_became_peered":"2023-10-07T09:30:24.963442+0000","last_unstale":"2023-10-07T09:30:26.423847+0000","last_undegraded":"2023-10-07T09:30:26.423847+0000","last_fullsized":"2023-10-07T09:30:26.423847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:38.051902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413355+0000","last_change":"2023-10-07T09:30:26.406759+0000","last_active":"2023-10-07T09:30:26.413355+0000","last_peered":"2023-10-07T09:30:26.413355+0000","last_clean":"2023-10-07T09:30:26.413355+0000","last_became_active":"2023-10-07T09:30:26.406449+0000","last_became_peered":"2023-10-07T09:30:26.406449+0000","last_unstale":"2023-10-07T09:30:26.413355+0000","last_undegraded":"2023-10-07T09:30:26.413355+0000","last_fullsized":"2023-10-07T09:30:26.413355+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.970510+0000","last_change":"2023-10-07T09:30:24.970510+0000","last_active":"2023-10-07T09:30:24.970510+0000","last_peered":"2023-10-07T09:30:24.970510+0000","last_clean":"2023-10-07T09:30:24.970510+0000","last_became_active":"2023-10-07T09:30:24.970183+0000","last_became_peered":"2023-10-07T09:30:24.970183+0000","last_unstale":"2023-10-07T09:30:24.970510+0000","last_undegraded":"2023-10-07T09:30:24.970510+0000","last_fullsized":"2023-10-07T09:30:24.970510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412638+0000","last_change":"2023-10-07T09:30:26.404546+0000","last_active":"2023-10-07T09:30:26.412638+0000","last_peered":"2023-10-07T09:30:26.412638+0000","last_clean":"2023-10-07T09:30:26.412638+0000","last_became_active":"2023-10-07T09:30:26.404284+0000","last_became_peered":"2023-10-07T09:30:26.404284+0000","last_unstale":"2023-10-07T09:30:26.412638+0000","last_undegraded":"2023-10-07T09:30:26.412638+0000","last_fullsized":"2023-10-07T09:30:26.412638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.965098+0000","last_change":"2023-10-07T09:30:24.965098+0000","last_active":"2023-10-07T09:30:24.965098+0000","last_peered":"2023-10-07T09:30:24.965098+0000","last_clean":"2023-10-07T09:30:24.965098+0000","last_became_active":"2023-10-07T09:30:24.964594+0000","last_became_peered":"2023-10-07T09:30:24.964594+0000","last_unstale":"2023-10-07T09:30:24.965098+0000","last_undegraded":"2023-10-07T09:30:24.965098+0000","last_fullsized":"2023-10-07T09:30:24.965098+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:56.143330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955888+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:11:46.955055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.393481+0000","last_change":"2023-10-07T09:30:24.967401+0000","last_active":"2023-10-07T09:30:26.393481+0000","last_peered":"2023-10-07T09:30:26.393481+0000","last_clean":"2023-10-07T09:30:26.393481+0000","last_became_active":"2023-10-07T09:30:24.967138+0000","last_became_peered":"2023-10-07T09:30:24.967138+0000","last_unstale":"2023-10-07T09:30:26.393481+0000","last_undegraded":"2023-10-07T09:30:26.393481+0000","last_fullsized":"2023-10-07T09:30:26.393481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414869+0000","last_change":"2023-10-07T09:30:26.411476+0000","last_active":"2023-10-07T09:30:26.414869+0000","last_peered":"2023-10-07T09:30:26.414869+0000","last_clean":"2023-10-07T09:30:26.414869+0000","last_became_active":"2023-10-07T09:30:26.411198+0000","last_became_peered":"2023-10-07T09:30:26.411198+0000","last_unstale":"2023-10-07T09:30:26.414869+0000","last_undegraded":"2023-10-07T09:30:26.414869+0000","last_fullsized":"2023-10-07T09:30:26.414869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423028+0000","last_change":"2023-10-07T09:30:24.971777+0000","last_active":"2023-10-07T09:30:26.423028+0000","last_peered":"2023-10-07T09:30:26.423028+0000","last_clean":"2023-10-07T09:30:26.423028+0000","last_became_active":"2023-10-07T09:30:24.971526+0000","last_became_peered":"2023-10-07T09:30:24.971526+0000","last_unstale":"2023-10-07T09:30:26.423028+0000","last_undegraded":"2023-10-07T09:30:26.423028+0000","last_fullsized":"2023-10-07T09:30:26.423028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:00.376299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410968+0000","last_change":"2023-10-07T09:30:26.404652+0000","last_active":"2023-10-07T09:30:26.410968+0000","last_peered":"2023-10-07T09:30:26.410968+0000","last_clean":"2023-10-07T09:30:26.410968+0000","last_became_active":"2023-10-07T09:30:26.404410+0000","last_became_peered":"2023-10-07T09:30:26.404410+0000","last_unstale":"2023-10-07T09:30:26.410968+0000","last_undegraded":"2023-10-07T09:30:26.410968+0000","last_fullsized":"2023-10-07T09:30:26.410968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390466+0000","last_change":"2023-10-07T09:30:24.963621+0000","last_active":"2023-10-07T09:30:26.390466+0000","last_peered":"2023-10-07T09:30:26.390466+0000","last_clean":"2023-10-07T09:30:26.390466+0000","last_became_active":"2023-10-07T09:30:24.963315+0000","last_became_peered":"2023-10-07T09:30:24.963315+0000","last_unstale":"2023-10-07T09:30:26.390466+0000","last_undegraded":"2023-10-07T09:30:26.390466+0000","last_fullsized":"2023-10-07T09:30:26.390466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410981+0000","last_change":"2023-10-07T09:30:26.404627+0000","last_active":"2023-10-07T09:30:26.410981+0000","last_peered":"2023-10-07T09:30:26.410981+0000","last_clean":"2023-10-07T09:30:26.410981+0000","last_became_active":"2023-10-07T09:30:26.404362+0000","last_became_peered":"2023-10-07T09:30:26.404362+0000","last_unstale":"2023-10-07T09:30:26.410981+0000","last_undegraded":"2023-10-07T09:30:26.410981+0000","last_fullsized":"2023-10-07T09:30:26.410981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392082+0000","last_change":"2023-10-07T09:30:24.972623+0000","last_active":"2023-10-07T09:30:26.392082+0000","last_peered":"2023-10-07T09:30:26.392082+0000","last_clean":"2023-10-07T09:30:26.392082+0000","last_became_active":"2023-10-07T09:30:24.972377+0000","last_became_peered":"2023-10-07T09:30:24.972377+0000","last_unstale":"2023-10-07T09:30:26.392082+0000","last_undegraded":"2023-10-07T09:30:26.392082+0000","last_fullsized":"2023-10-07T09:30:26.392082+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:49.176868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955354+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390656+0000","last_change":"2023-10-07T09:30:24.968835+0000","last_active":"2023-10-07T09:30:26.390656+0000","last_peered":"2023-10-07T09:30:26.390656+0000","last_clean":"2023-10-07T09:30:26.390656+0000","last_became_active":"2023-10-07T09:30:24.968531+0000","last_became_peered":"2023-10-07T09:30:24.968531+0000","last_unstale":"2023-10-07T09:30:26.390656+0000","last_undegraded":"2023-10-07T09:30:26.390656+0000","last_fullsized":"2023-10-07T09:30:26.390656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:06:46.058261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418399+0000","last_change":"2023-10-07T09:30:26.413894+0000","last_active":"2023-10-07T09:30:26.418399+0000","last_peered":"2023-10-07T09:30:26.418399+0000","last_clean":"2023-10-07T09:30:26.418399+0000","last_became_active":"2023-10-07T09:30:26.413276+0000","last_became_peered":"2023-10-07T09:30:26.413276+0000","last_unstale":"2023-10-07T09:30:26.418399+0000","last_undegraded":"2023-10-07T09:30:26.418399+0000","last_fullsized":"2023-10-07T09:30:26.418399+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392045+0000","last_change":"2023-10-07T09:30:24.971933+0000","last_active":"2023-10-07T09:30:26.392045+0000","last_peered":"2023-10-07T09:30:26.392045+0000","last_clean":"2023-10-07T09:30:26.392045+0000","last_became_active":"2023-10-07T09:30:24.971478+0000","last_became_peered":"2023-10-07T09:30:24.971478+0000","last_unstale":"2023-10-07T09:30:26.392045+0000","last_undegraded":"2023-10-07T09:30:26.392045+0000","last_fullsized":"2023-10-07T09:30:26.392045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:19.116181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410100+0000","last_change":"2023-10-07T09:30:26.402235+0000","last_active":"2023-10-07T09:30:26.410100+0000","last_peered":"2023-10-07T09:30:26.410100+0000","last_clean":"2023-10-07T09:30:26.410100+0000","last_became_active":"2023-10-07T09:30:26.401915+0000","last_became_peered":"2023-10-07T09:30:26.401915+0000","last_unstale":"2023-10-07T09:30:26.410100+0000","last_undegraded":"2023-10-07T09:30:26.410100+0000","last_fullsized":"2023-10-07T09:30:26.410100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:01.010896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967230+0000","last_change":"2023-10-07T09:30:24.967230+0000","last_active":"2023-10-07T09:30:24.967230+0000","last_peered":"2023-10-07T09:30:24.967230+0000","last_clean":"2023-10-07T09:30:24.967230+0000","last_became_active":"2023-10-07T09:30:24.966740+0000","last_became_peered":"2023-10-07T09:30:24.966740+0000","last_unstale":"2023-10-07T09:30:24.967230+0000","last_undegraded":"2023-10-07T09:30:24.967230+0000","last_fullsized":"2023-10-07T09:30:24.967230+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:31:56.775021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950417+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384013+0000","last_change":"2023-10-07T09:30:24.967499+0000","last_active":"2023-10-07T09:30:26.384013+0000","last_peered":"2023-10-07T09:30:26.384013+0000","last_clean":"2023-10-07T09:30:26.384013+0000","last_became_active":"2023-10-07T09:30:24.967231+0000","last_became_peered":"2023-10-07T09:30:24.967231+0000","last_unstale":"2023-10-07T09:30:26.384013+0000","last_undegraded":"2023-10-07T09:30:26.384013+0000","last_fullsized":"2023-10-07T09:30:26.384013+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:23.529707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956851+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:23.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.971597+0000","last_change":"2023-10-07T09:30:24.971597+0000","last_active":"2023-10-07T09:30:24.971597+0000","last_peered":"2023-10-07T09:30:24.971597+0000","last_clean":"2023-10-07T09:30:24.971597+0000","last_became_active":"2023-10-07T09:30:24.971341+0000","last_became_peered":"2023-10-07T09:30:24.971341+0000","last_unstale":"2023-10-07T09:30:24.971597+0000","last_undegraded":"2023-10-07T09:30:24.971597+0000","last_fullsized":"2023-10-07T09:30:24.971597+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:15.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-07T09:30:23.945970+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:30:23.945970+0000","last_peered":"2023-10-07T09:30:23.945970+0000","last_clean":"2023-10-07T09:30:23.945970+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:30:23.945970+0000","last_undegraded":"2023-10-07T09:30:23.945970+0000","last_fullsized":"2023-10-07T09:30:23.945970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1648,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298236,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073604,"statfs":{"total":96636764160,"available":96331370496,"internally_reserved":0,"allocated":970752,"data_stored":735622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1,"5min":1,"15min":1},"last":0.88},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71399999999999997}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.89800000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.93300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.628}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.94899999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.81599999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074196,"statfs":{"total":96636764160,"available":96331976704,"internally_reserved":0,"allocated":364544,"data_stored":138999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.984,"5min":2.984,"15min":2.984},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":7.867,"5min":7.867,"15min":7.867},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":8.0399999999999991},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.61099999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.79500000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.93200000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.31},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.755}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.69699999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302356,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069484,"statfs":{"total":96636764160,"available":96327151616,"internally_reserved":0,"allocated":995328,"data_stored":748673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.726,"5min":2.726,"15min":2.726},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.57399999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.92200000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.66700000000000004}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.8199999999999998,"5min":3.8199999999999998,"15min":3.8199999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.627},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":3.855,"5min":3.855,"15min":3.855},"last":1.036}]},{"osd":7,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":3.778,"5min":3.778,"15min":3.778},"last":1.097}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297684,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074156,"statfs":{"total":96636764160,"available":96331935744,"internally_reserved":0,"allocated":405504,"data_stored":158375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.439}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":1.0609999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.876},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.94299999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.91400000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":0.314},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.40000000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.498},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.42699999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.877}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.85299999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.88}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":1.2889999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.276}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":6.0620000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074196,"statfs":{"total":96636764160,"available":96331976704,"internally_reserved":0,"allocated":364544,"data_stored":138999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.623}]},{"osd":3,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.80600000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.83799999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.76600000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.82299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:30:31.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:30:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.499+0000 7f5657bff700 1 Processor -- start 2023-10-07T09:30:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.500+0000 7f5657bff700 1 -- start start 2023-10-07T09:30:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.501+0000 7f5657bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 0x7f56500d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.501+0000 7f5657bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56500cefb0 con 0x7f5650159750 2023-10-07T09:30:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.501+0000 7f5657bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56500cbf20 con 0x7f5650159b20 2023-10-07T09:30:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.501+0000 7f5657bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56500cc200 con 0x7f56500c6470 2023-10-07T09:30:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.501+0000 7f565599b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5650159750 0x7f56500ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33488/0 (socket says 172.21.15.29:33488) 2023-10-07T09:30:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.501+0000 7f565599b700 1 -- 172.21.15.29:0/559293250 learned_addr learned my addr 172.21.15.29:0/559293250 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:31.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f565619c700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 0x7f56500d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008986452 0 0) 0x7f56500cefb0 con 0x7f5650159750 2023-10-07T09:30:31.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f563c003570 con 0x7f5650159750 2023-10-07T09:30:31.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288292802 0 0) 0x7f56500cbf20 con 0x7f5650159b20 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56500cefb0 con 0x7f5650159b20 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142491316 0 0) 0x7f563c003570 con 0x7f5650159750 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 msgr2=0x7f56500d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 0x7f56500d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 >> v1:172.21.15.144:6789/0 conn(0x7f5650159b20 legacy=0x7f56500ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500cc4e0 con 0x7f5650159750 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f564c0015f0 con 0x7f5650159750 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 >> v1:172.21.15.29:6789/0 conn(0x7f5650159750 legacy=0x7f56500ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 0x7f563c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c0041d0 0x7f563c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f563c004d50 0x7f563c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500cc200 con 0x7f563c004d50 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c00bbf0 con 0x7f563c0041d0 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.502+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c00bdb0 con 0x7f56500c6470 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f563c004d50 0x7f563c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565599b700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 0x7f563c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565519a700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c0041d0 0x7f563c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 -- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 msgr2=0x7f563c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c6470 0x7f563c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 -- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c0041d0 msgr2=0x7f563c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c0041d0 0x7f563c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 -- 172.21.15.29:0/559293250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c003570 con 0x7f563c004d50 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f565619c700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f563c004d50 0x7f563c0051c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f564800e210 tx=0x7f564800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da59b91f07add06c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.503+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564800eac0 con 0x7f563c004d50 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564800f040 con 0x7f563c004d50 2023-10-07T09:30:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5654999700 1 -- 172.21.15.29:0/559293250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564800ee70 con 0x7f563c004d50 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5657bff700 1 -- 172.21.15.29:0/559293250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f563c004d50 msgr2=0x7f563c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5657bff700 1 --2- 172.21.15.29:0/559293250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f563c004d50 0x7f563c0051c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f564800e210 tx=0x7f564800b040 comp rx=0 tx=0).stop 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f565619c700 1 -- 172.21.15.29:0/559293250 reap_dead start 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5657bff700 1 -- 172.21.15.29:0/559293250 shutdown_connections 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5657bff700 1 -- 172.21.15.29:0/559293250 >> 172.21.15.29:0/559293250 conn(0x7f5650063a10 msgr2=0x7f56500d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5657bff700 1 -- 172.21.15.29:0/559293250 shutdown_connections 2023-10-07T09:30:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.504+0000 7f5657bff700 1 -- 172.21.15.29:0/559293250 wait complete. 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.506+0000 7f5657bff700 1 Processor -- start 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.506+0000 7f5657bff700 1 -- start start 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f5657bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c004d50 0x7f565013c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f5657bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 0x7f56500c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f5657bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c7670 0x7f56500c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f5657bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565016b0e0 con 0x7f56500c9540 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f5657bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500c8020 con 0x7f563c004d50 2023-10-07T09:30:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f5657bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56500ca020 con 0x7f56500c7670 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f565519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 0x7f56500c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f565519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 0x7f56500c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39728/0 (socket says 172.21.15.29:39728) 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f565519a700 1 -- 172.21.15.29:0/1266436955 learned_addr learned my addr 172.21.15.29:0/1266436955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f565599b700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c004d50 0x7f565013c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.507+0000 7f565619c700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c7670 0x7f56500c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565519a700 1 -- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c7670 msgr2=0x7f56500c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565519a700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c7670 0x7f56500c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565519a700 1 -- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c004d50 msgr2=0x7f565013c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565519a700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c004d50 0x7f565013c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565519a700 1 -- 172.21.15.29:0/1266436955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500ca300 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565599b700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c004d50 0x7f565013c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f565519a700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 0x7f56500c99b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f564000f8e0 tx=0x7f5640007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000d640 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564000d7a0 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56500ca5e0 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000db60 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.508+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5650152cb0 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.509+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565004c3c0 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.510+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f564001c610 con 0x7f56500c9540 2023-10-07T09:30:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.511+0000 7f5645ffb700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5634066f30 0x7f56340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.511+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5640084b20 con 0x7f56500c9540 2023-10-07T09:30:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.511+0000 7f565599b700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5634066f30 0x7f56340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.512+0000 7f565599b700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5634066f30 0x7f56340693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f564c000e10 tx=0x7f564c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.515+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5640051da0 con 0x7f56500c9540 2023-10-07T09:30:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.712+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f565015d1a0 con 0x7f5634066f30 2023-10-07T09:30:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.721+0000 7f5645ffb700 1 -- 172.21.15.29:0/1266436955 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287592 (secure 0 0 0) 0x7f565015d1a0 con 0x7f5634066f30 2023-10-07T09:30:31.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5634066f30 msgr2=0x7f56340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5634066f30 0x7f56340693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f564c000e10 tx=0x7f564c009040 comp rx=0 tx=0).stop 2023-10-07T09:30:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 msgr2=0x7f56500c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 0x7f56500c99b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f564000f8e0 tx=0x7f5640007d30 comp rx=0 tx=0).stop 2023-10-07T09:30:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 shutdown_connections 2023-10-07T09:30:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5634066f30 0x7f56340693e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56500c7670 0x7f56500c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c004d50 0x7f565013c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 --2- 172.21.15.29:0/1266436955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56500c9540 0x7f56500c99b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 >> 172.21.15.29:0/1266436955 conn(0x7f5650063a10 msgr2=0x7f5650068440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 shutdown_connections 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.725+0000 7f5657bff700 1 -- 172.21.15.29:0/1266436955 wait complete. 2023-10-07T09:30:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:30:31.743 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-10-07T09:30:30.034161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592016,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":192,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2386644,"kb_used_data":4244,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752588076,"statfs":{"total":773094113280,"available":770650189824,"internally_reserved":0,"allocated":4345856,"data_stored":2399343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1648,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001688"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956469+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415669+0000","last_change":"2023-10-07T09:30:26.408750+0000","last_active":"2023-10-07T09:30:26.415669+0000","last_peered":"2023-10-07T09:30:26.415669+0000","last_clean":"2023-10-07T09:30:26.415669+0000","last_became_active":"2023-10-07T09:30:26.407823+0000","last_became_peered":"2023-10-07T09:30:26.407823+0000","last_unstale":"2023-10-07T09:30:26.415669+0000","last_undegraded":"2023-10-07T09:30:26.415669+0000","last_fullsized":"2023-10-07T09:30:26.415669+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:58.672867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414461+0000","last_change":"2023-10-07T09:30:26.408350+0000","last_active":"2023-10-07T09:30:26.414461+0000","last_peered":"2023-10-07T09:30:26.414461+0000","last_clean":"2023-10-07T09:30:26.414461+0000","last_became_active":"2023-10-07T09:30:26.408029+0000","last_became_peered":"2023-10-07T09:30:26.408029+0000","last_unstale":"2023-10-07T09:30:26.414461+0000","last_undegraded":"2023-10-07T09:30:26.414461+0000","last_fullsized":"2023-10-07T09:30:26.414461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407832+0000","last_change":"2023-10-07T09:30:26.403115+0000","last_active":"2023-10-07T09:30:26.407832+0000","last_peered":"2023-10-07T09:30:26.407832+0000","last_clean":"2023-10-07T09:30:26.407832+0000","last_became_active":"2023-10-07T09:30:26.402699+0000","last_became_peered":"2023-10-07T09:30:26.402699+0000","last_unstale":"2023-10-07T09:30:26.407832+0000","last_undegraded":"2023-10-07T09:30:26.407832+0000","last_fullsized":"2023-10-07T09:30:26.407832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:06:53.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414913+0000","last_change":"2023-10-07T09:30:26.410770+0000","last_active":"2023-10-07T09:30:26.414913+0000","last_peered":"2023-10-07T09:30:26.414913+0000","last_clean":"2023-10-07T09:30:26.414913+0000","last_became_active":"2023-10-07T09:30:26.410465+0000","last_became_peered":"2023-10-07T09:30:26.410465+0000","last_unstale":"2023-10-07T09:30:26.414913+0000","last_undegraded":"2023-10-07T09:30:26.414913+0000","last_fullsized":"2023-10-07T09:30:26.414913+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.403447+0000","last_change":"2023-10-07T09:30:26.395345+0000","last_active":"2023-10-07T09:30:26.403447+0000","last_peered":"2023-10-07T09:30:26.403447+0000","last_clean":"2023-10-07T09:30:26.403447+0000","last_became_active":"2023-10-07T09:30:26.395042+0000","last_became_peered":"2023-10-07T09:30:26.395042+0000","last_unstale":"2023-10-07T09:30:26.403447+0000","last_undegraded":"2023-10-07T09:30:26.403447+0000","last_fullsized":"2023-10-07T09:30:26.403447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956440+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:09.240847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957353+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:20:14.772461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951211+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.399392+0000","last_change":"2023-10-07T09:30:26.396209+0000","last_active":"2023-10-07T09:30:26.399392+0000","last_peered":"2023-10-07T09:30:26.399392+0000","last_clean":"2023-10-07T09:30:26.399392+0000","last_became_active":"2023-10-07T09:30:26.395573+0000","last_became_peered":"2023-10-07T09:30:26.395573+0000","last_unstale":"2023-10-07T09:30:26.399392+0000","last_undegraded":"2023-10-07T09:30:26.399392+0000","last_fullsized":"2023-10-07T09:30:26.399392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:54.021183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957373+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:17.334679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413491+0000","last_change":"2023-10-07T09:30:26.409670+0000","last_active":"2023-10-07T09:30:26.413491+0000","last_peered":"2023-10-07T09:30:26.413491+0000","last_clean":"2023-10-07T09:30:26.413491+0000","last_became_active":"2023-10-07T09:30:26.408687+0000","last_became_peered":"2023-10-07T09:30:26.408687+0000","last_unstale":"2023-10-07T09:30:26.413491+0000","last_undegraded":"2023-10-07T09:30:26.413491+0000","last_fullsized":"2023-10-07T09:30:26.413491+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413335+0000","last_change":"2023-10-07T09:30:26.406082+0000","last_active":"2023-10-07T09:30:26.413335+0000","last_peered":"2023-10-07T09:30:26.413335+0000","last_clean":"2023-10-07T09:30:26.413335+0000","last_became_active":"2023-10-07T09:30:26.405724+0000","last_became_peered":"2023-10-07T09:30:26.405724+0000","last_unstale":"2023-10-07T09:30:26.413335+0000","last_undegraded":"2023-10-07T09:30:26.413335+0000","last_fullsized":"2023-10-07T09:30:26.413335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411102+0000","last_change":"2023-10-07T09:30:26.403342+0000","last_active":"2023-10-07T09:30:26.411102+0000","last_peered":"2023-10-07T09:30:26.411102+0000","last_clean":"2023-10-07T09:30:26.411102+0000","last_became_active":"2023-10-07T09:30:26.403138+0000","last_became_peered":"2023-10-07T09:30:26.403138+0000","last_unstale":"2023-10-07T09:30:26.411102+0000","last_undegraded":"2023-10-07T09:30:26.411102+0000","last_fullsized":"2023-10-07T09:30:26.411102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408435+0000","last_change":"2023-10-07T09:30:26.395877+0000","last_active":"2023-10-07T09:30:26.408435+0000","last_peered":"2023-10-07T09:30:26.408435+0000","last_clean":"2023-10-07T09:30:26.408435+0000","last_became_active":"2023-10-07T09:30:26.395596+0000","last_became_peered":"2023-10-07T09:30:26.395596+0000","last_unstale":"2023-10-07T09:30:26.408435+0000","last_undegraded":"2023-10-07T09:30:26.408435+0000","last_fullsized":"2023-10-07T09:30:26.408435+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:20.489580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956822+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:20:47.855402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.957600+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:19.142808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409109+0000","last_change":"2023-10-07T09:30:26.394655+0000","last_active":"2023-10-07T09:30:26.409109+0000","last_peered":"2023-10-07T09:30:26.409109+0000","last_clean":"2023-10-07T09:30:26.409109+0000","last_became_active":"2023-10-07T09:30:26.394372+0000","last_became_peered":"2023-10-07T09:30:26.394372+0000","last_unstale":"2023-10-07T09:30:26.409109+0000","last_undegraded":"2023-10-07T09:30:26.409109+0000","last_fullsized":"2023-10-07T09:30:26.409109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:35.770412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405481+0000","last_change":"2023-10-07T09:30:26.402289+0000","last_active":"2023-10-07T09:30:26.405481+0000","last_peered":"2023-10-07T09:30:26.405481+0000","last_clean":"2023-10-07T09:30:26.405481+0000","last_became_active":"2023-10-07T09:30:26.401826+0000","last_became_peered":"2023-10-07T09:30:26.401826+0000","last_unstale":"2023-10-07T09:30:26.405481+0000","last_undegraded":"2023-10-07T09:30:26.405481+0000","last_fullsized":"2023-10-07T09:30:26.405481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:46:15.007773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414990+0000","last_change":"2023-10-07T09:30:26.411654+0000","last_active":"2023-10-07T09:30:26.414990+0000","last_peered":"2023-10-07T09:30:26.414990+0000","last_clean":"2023-10-07T09:30:26.414990+0000","last_became_active":"2023-10-07T09:30:26.411407+0000","last_became_peered":"2023-10-07T09:30:26.411407+0000","last_unstale":"2023-10-07T09:30:26.414990+0000","last_undegraded":"2023-10-07T09:30:26.414990+0000","last_fullsized":"2023-10-07T09:30:26.414990+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951679+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.949829+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:03.694836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410992+0000","last_change":"2023-10-07T09:30:26.403259+0000","last_active":"2023-10-07T09:30:26.410992+0000","last_peered":"2023-10-07T09:30:26.410992+0000","last_clean":"2023-10-07T09:30:26.410992+0000","last_became_active":"2023-10-07T09:30:26.402926+0000","last_became_peered":"2023-10-07T09:30:26.402926+0000","last_unstale":"2023-10-07T09:30:26.410992+0000","last_undegraded":"2023-10-07T09:30:26.410992+0000","last_fullsized":"2023-10-07T09:30:26.410992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413211+0000","last_change":"2023-10-07T09:30:26.409044+0000","last_active":"2023-10-07T09:30:26.413211+0000","last_peered":"2023-10-07T09:30:26.413211+0000","last_clean":"2023-10-07T09:30:26.413211+0000","last_became_active":"2023-10-07T09:30:26.408764+0000","last_became_peered":"2023-10-07T09:30:26.408764+0000","last_unstale":"2023-10-07T09:30:26.413211+0000","last_undegraded":"2023-10-07T09:30:26.413211+0000","last_fullsized":"2023-10-07T09:30:26.413211+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:47.675483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422810+0000","last_change":"2023-10-07T09:30:26.413627+0000","last_active":"2023-10-07T09:30:26.422810+0000","last_peered":"2023-10-07T09:30:26.422810+0000","last_clean":"2023-10-07T09:30:26.422810+0000","last_became_active":"2023-10-07T09:30:26.413371+0000","last_became_peered":"2023-10-07T09:30:26.413371+0000","last_unstale":"2023-10-07T09:30:26.422810+0000","last_undegraded":"2023-10-07T09:30:26.422810+0000","last_fullsized":"2023-10-07T09:30:26.422810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:34.133968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415691+0000","last_change":"2023-10-07T09:30:26.408999+0000","last_active":"2023-10-07T09:30:26.415691+0000","last_peered":"2023-10-07T09:30:26.415691+0000","last_clean":"2023-10-07T09:30:26.415691+0000","last_became_active":"2023-10-07T09:30:26.408201+0000","last_became_peered":"2023-10-07T09:30:26.408201+0000","last_unstale":"2023-10-07T09:30:26.415691+0000","last_undegraded":"2023-10-07T09:30:26.415691+0000","last_fullsized":"2023-10-07T09:30:26.415691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421410+0000","last_change":"2023-10-07T09:30:26.411198+0000","last_active":"2023-10-07T09:30:26.421410+0000","last_peered":"2023-10-07T09:30:26.421410+0000","last_clean":"2023-10-07T09:30:26.421410+0000","last_became_active":"2023-10-07T09:30:26.410735+0000","last_became_peered":"2023-10-07T09:30:26.410735+0000","last_unstale":"2023-10-07T09:30:26.421410+0000","last_undegraded":"2023-10-07T09:30:26.421410+0000","last_fullsized":"2023-10-07T09:30:26.421410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:24.740454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413473+0000","last_change":"2023-10-07T09:30:26.409340+0000","last_active":"2023-10-07T09:30:26.413473+0000","last_peered":"2023-10-07T09:30:26.413473+0000","last_clean":"2023-10-07T09:30:26.413473+0000","last_became_active":"2023-10-07T09:30:26.408986+0000","last_became_peered":"2023-10-07T09:30:26.408986+0000","last_unstale":"2023-10-07T09:30:26.413473+0000","last_undegraded":"2023-10-07T09:30:26.413473+0000","last_fullsized":"2023-10-07T09:30:26.413473+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950988+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955210+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:25.231695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.954033+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955840+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:30.437449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418432+0000","last_change":"2023-10-07T09:30:26.413698+0000","last_active":"2023-10-07T09:30:26.418432+0000","last_peered":"2023-10-07T09:30:26.418432+0000","last_clean":"2023-10-07T09:30:26.418432+0000","last_became_active":"2023-10-07T09:30:26.413219+0000","last_became_peered":"2023-10-07T09:30:26.413219+0000","last_unstale":"2023-10-07T09:30:26.418432+0000","last_undegraded":"2023-10-07T09:30:26.418432+0000","last_fullsized":"2023-10-07T09:30:26.418432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.970001+0000","last_change":"2023-10-07T09:30:24.970001+0000","last_active":"2023-10-07T09:30:24.970001+0000","last_peered":"2023-10-07T09:30:24.970001+0000","last_clean":"2023-10-07T09:30:24.970001+0000","last_became_active":"2023-10-07T09:30:24.969692+0000","last_became_peered":"2023-10-07T09:30:24.969692+0000","last_unstale":"2023-10-07T09:30:24.970001+0000","last_undegraded":"2023-10-07T09:30:24.970001+0000","last_fullsized":"2023-10-07T09:30:24.970001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956067+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421779+0000","last_change":"2023-10-07T09:30:24.961003+0000","last_active":"2023-10-07T09:30:26.421779+0000","last_peered":"2023-10-07T09:30:26.421779+0000","last_clean":"2023-10-07T09:30:26.421779+0000","last_became_active":"2023-10-07T09:30:24.960711+0000","last_became_peered":"2023-10-07T09:30:24.960711+0000","last_unstale":"2023-10-07T09:30:26.421779+0000","last_undegraded":"2023-10-07T09:30:26.421779+0000","last_fullsized":"2023-10-07T09:30:26.421779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:06.545101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950450+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:07.548280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424146+0000","last_change":"2023-10-07T09:30:24.967737+0000","last_active":"2023-10-07T09:30:26.424146+0000","last_peered":"2023-10-07T09:30:26.424146+0000","last_clean":"2023-10-07T09:30:26.424146+0000","last_became_active":"2023-10-07T09:30:24.966888+0000","last_became_peered":"2023-10-07T09:30:24.966888+0000","last_unstale":"2023-10-07T09:30:26.424146+0000","last_undegraded":"2023-10-07T09:30:26.424146+0000","last_fullsized":"2023-10-07T09:30:26.424146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421389+0000","last_change":"2023-10-07T09:30:26.411013+0000","last_active":"2023-10-07T09:30:26.421389+0000","last_peered":"2023-10-07T09:30:26.421389+0000","last_clean":"2023-10-07T09:30:26.421389+0000","last_became_active":"2023-10-07T09:30:26.410413+0000","last_became_peered":"2023-10-07T09:30:26.410413+0000","last_unstale":"2023-10-07T09:30:26.421389+0000","last_undegraded":"2023-10-07T09:30:26.421389+0000","last_fullsized":"2023-10-07T09:30:26.421389+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:06.340913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423552+0000","last_change":"2023-10-07T09:30:24.966147+0000","last_active":"2023-10-07T09:30:26.423552+0000","last_peered":"2023-10-07T09:30:26.423552+0000","last_clean":"2023-10-07T09:30:26.423552+0000","last_became_active":"2023-10-07T09:30:24.965853+0000","last_became_peered":"2023-10-07T09:30:24.965853+0000","last_unstale":"2023-10-07T09:30:26.423552+0000","last_undegraded":"2023-10-07T09:30:26.423552+0000","last_fullsized":"2023-10-07T09:30:26.423552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.954591+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:38:03.049978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967094+0000","last_change":"2023-10-07T09:30:24.967094+0000","last_active":"2023-10-07T09:30:24.967094+0000","last_peered":"2023-10-07T09:30:24.967094+0000","last_clean":"2023-10-07T09:30:24.967094+0000","last_became_active":"2023-10-07T09:30:24.966810+0000","last_became_peered":"2023-10-07T09:30:24.966810+0000","last_unstale":"2023-10-07T09:30:24.967094+0000","last_undegraded":"2023-10-07T09:30:24.967094+0000","last_fullsized":"2023-10-07T09:30:24.967094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:24:57.676848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414299+0000","last_change":"2023-10-07T09:30:26.407749+0000","last_active":"2023-10-07T09:30:26.414299+0000","last_peered":"2023-10-07T09:30:26.414299+0000","last_clean":"2023-10-07T09:30:26.414299+0000","last_became_active":"2023-10-07T09:30:26.407433+0000","last_became_peered":"2023-10-07T09:30:26.407433+0000","last_unstale":"2023-10-07T09:30:26.414299+0000","last_undegraded":"2023-10-07T09:30:26.414299+0000","last_fullsized":"2023-10-07T09:30:26.414299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.963628+0000","last_change":"2023-10-07T09:30:24.963628+0000","last_active":"2023-10-07T09:30:24.963628+0000","last_peered":"2023-10-07T09:30:24.963628+0000","last_clean":"2023-10-07T09:30:24.963628+0000","last_became_active":"2023-10-07T09:30:24.963306+0000","last_became_peered":"2023-10-07T09:30:24.963306+0000","last_unstale":"2023-10-07T09:30:24.963628+0000","last_undegraded":"2023-10-07T09:30:24.963628+0000","last_fullsized":"2023-10-07T09:30:24.963628+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:12.068751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.949818+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:26.104877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.383857+0000","last_change":"2023-10-07T09:30:24.965572+0000","last_active":"2023-10-07T09:30:26.383857+0000","last_peered":"2023-10-07T09:30:26.383857+0000","last_clean":"2023-10-07T09:30:26.383857+0000","last_became_active":"2023-10-07T09:30:24.965320+0000","last_became_peered":"2023-10-07T09:30:24.965320+0000","last_unstale":"2023-10-07T09:30:26.383857+0000","last_undegraded":"2023-10-07T09:30:26.383857+0000","last_fullsized":"2023-10-07T09:30:26.383857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:49.978481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414174+0000","last_change":"2023-10-07T09:30:26.409216+0000","last_active":"2023-10-07T09:30:26.414174+0000","last_peered":"2023-10-07T09:30:26.414174+0000","last_clean":"2023-10-07T09:30:26.414174+0000","last_became_active":"2023-10-07T09:30:26.408947+0000","last_became_peered":"2023-10-07T09:30:26.408947+0000","last_unstale":"2023-10-07T09:30:26.414174+0000","last_undegraded":"2023-10-07T09:30:26.414174+0000","last_fullsized":"2023-10-07T09:30:26.414174+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.962784+0000","last_change":"2023-10-07T09:30:24.962784+0000","last_active":"2023-10-07T09:30:24.962784+0000","last_peered":"2023-10-07T09:30:24.962784+0000","last_clean":"2023-10-07T09:30:24.962784+0000","last_became_active":"2023-10-07T09:30:24.962409+0000","last_became_peered":"2023-10-07T09:30:24.962409+0000","last_unstale":"2023-10-07T09:30:24.962784+0000","last_undegraded":"2023-10-07T09:30:24.962784+0000","last_fullsized":"2023-10-07T09:30:24.962784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:04.344607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413097+0000","last_change":"2023-10-07T09:30:26.406965+0000","last_active":"2023-10-07T09:30:26.413097+0000","last_peered":"2023-10-07T09:30:26.413097+0000","last_clean":"2023-10-07T09:30:26.413097+0000","last_became_active":"2023-10-07T09:30:26.406607+0000","last_became_peered":"2023-10-07T09:30:26.406607+0000","last_unstale":"2023-10-07T09:30:26.413097+0000","last_undegraded":"2023-10-07T09:30:26.413097+0000","last_fullsized":"2023-10-07T09:30:26.413097+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384959+0000","last_change":"2023-10-07T09:30:24.967714+0000","last_active":"2023-10-07T09:30:26.384959+0000","last_peered":"2023-10-07T09:30:26.384959+0000","last_clean":"2023-10-07T09:30:26.384959+0000","last_became_active":"2023-10-07T09:30:24.967416+0000","last_became_peered":"2023-10-07T09:30:24.967416+0000","last_unstale":"2023-10-07T09:30:26.384959+0000","last_undegraded":"2023-10-07T09:30:26.384959+0000","last_fullsized":"2023-10-07T09:30:26.384959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:25.781083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402385+0000","last_change":"2023-10-07T09:30:26.395987+0000","last_active":"2023-10-07T09:30:26.402385+0000","last_peered":"2023-10-07T09:30:26.402385+0000","last_clean":"2023-10-07T09:30:26.402385+0000","last_became_active":"2023-10-07T09:30:26.395709+0000","last_became_peered":"2023-10-07T09:30:26.395709+0000","last_unstale":"2023-10-07T09:30:26.402385+0000","last_undegraded":"2023-10-07T09:30:26.402385+0000","last_fullsized":"2023-10-07T09:30:26.402385+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390349+0000","last_change":"2023-10-07T09:30:24.962930+0000","last_active":"2023-10-07T09:30:26.390349+0000","last_peered":"2023-10-07T09:30:26.390349+0000","last_clean":"2023-10-07T09:30:26.390349+0000","last_became_active":"2023-10-07T09:30:24.962586+0000","last_became_peered":"2023-10-07T09:30:24.962586+0000","last_unstale":"2023-10-07T09:30:26.390349+0000","last_undegraded":"2023-10-07T09:30:26.390349+0000","last_fullsized":"2023-10-07T09:30:26.390349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.417232+0000","last_change":"2023-10-07T09:30:26.412395+0000","last_active":"2023-10-07T09:30:26.417232+0000","last_peered":"2023-10-07T09:30:26.417232+0000","last_clean":"2023-10-07T09:30:26.417232+0000","last_became_active":"2023-10-07T09:30:26.412085+0000","last_became_peered":"2023-10-07T09:30:26.412085+0000","last_unstale":"2023-10-07T09:30:26.417232+0000","last_undegraded":"2023-10-07T09:30:26.417232+0000","last_fullsized":"2023-10-07T09:30:26.417232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421318+0000","last_change":"2023-10-07T09:30:24.967809+0000","last_active":"2023-10-07T09:30:26.421318+0000","last_peered":"2023-10-07T09:30:26.421318+0000","last_clean":"2023-10-07T09:30:26.421318+0000","last_became_active":"2023-10-07T09:30:24.967550+0000","last_became_peered":"2023-10-07T09:30:24.967550+0000","last_unstale":"2023-10-07T09:30:26.421318+0000","last_undegraded":"2023-10-07T09:30:26.421318+0000","last_fullsized":"2023-10-07T09:30:26.421318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414443+0000","last_change":"2023-10-07T09:30:26.408025+0000","last_active":"2023-10-07T09:30:26.414443+0000","last_peered":"2023-10-07T09:30:26.414443+0000","last_clean":"2023-10-07T09:30:26.414443+0000","last_became_active":"2023-10-07T09:30:26.407736+0000","last_became_peered":"2023-10-07T09:30:26.407736+0000","last_unstale":"2023-10-07T09:30:26.414443+0000","last_undegraded":"2023-10-07T09:30:26.414443+0000","last_fullsized":"2023-10-07T09:30:26.414443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.425863+0000","last_change":"2023-10-07T09:30:24.966563+0000","last_active":"2023-10-07T09:30:26.425863+0000","last_peered":"2023-10-07T09:30:26.425863+0000","last_clean":"2023-10-07T09:30:26.425863+0000","last_became_active":"2023-10-07T09:30:24.966273+0000","last_became_peered":"2023-10-07T09:30:24.966273+0000","last_unstale":"2023-10-07T09:30:26.425863+0000","last_undegraded":"2023-10-07T09:30:26.425863+0000","last_fullsized":"2023-10-07T09:30:26.425863+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:41.648356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421340+0000","last_change":"2023-10-07T09:30:26.409745+0000","last_active":"2023-10-07T09:30:26.421340+0000","last_peered":"2023-10-07T09:30:26.421340+0000","last_clean":"2023-10-07T09:30:26.421340+0000","last_became_active":"2023-10-07T09:30:26.409489+0000","last_became_peered":"2023-10-07T09:30:26.409489+0000","last_unstale":"2023-10-07T09:30:26.421340+0000","last_undegraded":"2023-10-07T09:30:26.421340+0000","last_fullsized":"2023-10-07T09:30:26.421340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.964382+0000","last_change":"2023-10-07T09:30:24.964382+0000","last_active":"2023-10-07T09:30:24.964382+0000","last_peered":"2023-10-07T09:30:24.964382+0000","last_clean":"2023-10-07T09:30:24.964382+0000","last_became_active":"2023-10-07T09:30:24.964076+0000","last_became_peered":"2023-10-07T09:30:24.964076+0000","last_unstale":"2023-10-07T09:30:24.964382+0000","last_undegraded":"2023-10-07T09:30:24.964382+0000","last_fullsized":"2023-10-07T09:30:24.964382+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:19:17.590704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402685+0000","last_change":"2023-10-07T09:30:26.396545+0000","last_active":"2023-10-07T09:30:26.402685+0000","last_peered":"2023-10-07T09:30:26.402685+0000","last_clean":"2023-10-07T09:30:26.402685+0000","last_became_active":"2023-10-07T09:30:26.395816+0000","last_became_peered":"2023-10-07T09:30:26.395816+0000","last_unstale":"2023-10-07T09:30:26.402685+0000","last_undegraded":"2023-10-07T09:30:26.402685+0000","last_fullsized":"2023-10-07T09:30:26.402685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.391928+0000","last_change":"2023-10-07T09:30:24.964522+0000","last_active":"2023-10-07T09:30:26.391928+0000","last_peered":"2023-10-07T09:30:26.391928+0000","last_clean":"2023-10-07T09:30:26.391928+0000","last_became_active":"2023-10-07T09:30:24.964221+0000","last_became_peered":"2023-10-07T09:30:24.964221+0000","last_unstale":"2023-10-07T09:30:26.391928+0000","last_undegraded":"2023-10-07T09:30:26.391928+0000","last_fullsized":"2023-10-07T09:30:26.391928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:35.312512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956354+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:12.844376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384014+0000","last_change":"2023-10-07T09:30:24.967550+0000","last_active":"2023-10-07T09:30:26.384014+0000","last_peered":"2023-10-07T09:30:26.384014+0000","last_clean":"2023-10-07T09:30:26.384014+0000","last_became_active":"2023-10-07T09:30:24.967265+0000","last_became_peered":"2023-10-07T09:30:24.967265+0000","last_unstale":"2023-10-07T09:30:26.384014+0000","last_undegraded":"2023-10-07T09:30:26.384014+0000","last_fullsized":"2023-10-07T09:30:26.384014+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:12.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.953547+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424346+0000","last_change":"2023-10-07T09:30:24.971275+0000","last_active":"2023-10-07T09:30:26.424346+0000","last_peered":"2023-10-07T09:30:26.424346+0000","last_clean":"2023-10-07T09:30:26.424346+0000","last_became_active":"2023-10-07T09:30:24.970929+0000","last_became_peered":"2023-10-07T09:30:24.970929+0000","last_unstale":"2023-10-07T09:30:26.424346+0000","last_undegraded":"2023-10-07T09:30:26.424346+0000","last_fullsized":"2023-10-07T09:30:26.424346+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.951700+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423122+0000","last_change":"2023-10-07T09:30:24.968081+0000","last_active":"2023-10-07T09:30:26.423122+0000","last_peered":"2023-10-07T09:30:26.423122+0000","last_clean":"2023-10-07T09:30:26.423122+0000","last_became_active":"2023-10-07T09:30:24.967047+0000","last_became_peered":"2023-10-07T09:30:24.967047+0000","last_unstale":"2023-10-07T09:30:26.423122+0000","last_undegraded":"2023-10-07T09:30:26.423122+0000","last_fullsized":"2023-10-07T09:30:26.423122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:00.381039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413417+0000","last_change":"2023-10-07T09:30:26.407336+0000","last_active":"2023-10-07T09:30:26.413417+0000","last_peered":"2023-10-07T09:30:26.413417+0000","last_clean":"2023-10-07T09:30:26.413417+0000","last_became_active":"2023-10-07T09:30:26.407027+0000","last_became_peered":"2023-10-07T09:30:26.407027+0000","last_unstale":"2023-10-07T09:30:26.413417+0000","last_undegraded":"2023-10-07T09:30:26.413417+0000","last_fullsized":"2023-10-07T09:30:26.413417+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.962904+0000","last_change":"2023-10-07T09:30:24.962904+0000","last_active":"2023-10-07T09:30:24.962904+0000","last_peered":"2023-10-07T09:30:24.962904+0000","last_clean":"2023-10-07T09:30:24.962904+0000","last_became_active":"2023-10-07T09:30:24.962519+0000","last_became_peered":"2023-10-07T09:30:24.962519+0000","last_unstale":"2023-10-07T09:30:24.962904+0000","last_undegraded":"2023-10-07T09:30:24.962904+0000","last_fullsized":"2023-10-07T09:30:24.962904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:00.363653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407852+0000","last_change":"2023-10-07T09:30:26.403114+0000","last_active":"2023-10-07T09:30:26.407852+0000","last_peered":"2023-10-07T09:30:26.407852+0000","last_clean":"2023-10-07T09:30:26.407852+0000","last_became_active":"2023-10-07T09:30:26.402830+0000","last_became_peered":"2023-10-07T09:30:26.402830+0000","last_unstale":"2023-10-07T09:30:26.407852+0000","last_undegraded":"2023-10-07T09:30:26.407852+0000","last_fullsized":"2023-10-07T09:30:26.407852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.961853+0000","last_change":"2023-10-07T09:30:24.961853+0000","last_active":"2023-10-07T09:30:24.961853+0000","last_peered":"2023-10-07T09:30:24.961853+0000","last_clean":"2023-10-07T09:30:24.961853+0000","last_became_active":"2023-10-07T09:30:24.961591+0000","last_became_peered":"2023-10-07T09:30:24.961591+0000","last_unstale":"2023-10-07T09:30:24.961853+0000","last_undegraded":"2023-10-07T09:30:24.961853+0000","last_fullsized":"2023-10-07T09:30:24.961853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415642+0000","last_change":"2023-10-07T09:30:26.413595+0000","last_active":"2023-10-07T09:30:26.415642+0000","last_peered":"2023-10-07T09:30:26.415642+0000","last_clean":"2023-10-07T09:30:26.415642+0000","last_became_active":"2023-10-07T09:30:26.413270+0000","last_became_peered":"2023-10-07T09:30:26.413270+0000","last_unstale":"2023-10-07T09:30:26.415642+0000","last_undegraded":"2023-10-07T09:30:26.415642+0000","last_fullsized":"2023-10-07T09:30:26.415642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967115+0000","last_change":"2023-10-07T09:30:24.967115+0000","last_active":"2023-10-07T09:30:24.967115+0000","last_peered":"2023-10-07T09:30:24.967115+0000","last_clean":"2023-10-07T09:30:24.967115+0000","last_became_active":"2023-10-07T09:30:24.966832+0000","last_became_peered":"2023-10-07T09:30:24.966832+0000","last_unstale":"2023-10-07T09:30:24.967115+0000","last_undegraded":"2023-10-07T09:30:24.967115+0000","last_fullsized":"2023-10-07T09:30:24.967115+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:11:11.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413302+0000","last_change":"2023-10-07T09:30:26.408606+0000","last_active":"2023-10-07T09:30:26.413302+0000","last_peered":"2023-10-07T09:30:26.413302+0000","last_clean":"2023-10-07T09:30:26.413302+0000","last_became_active":"2023-10-07T09:30:26.408343+0000","last_became_peered":"2023-10-07T09:30:26.408343+0000","last_unstale":"2023-10-07T09:30:26.413302+0000","last_undegraded":"2023-10-07T09:30:26.413302+0000","last_fullsized":"2023-10-07T09:30:26.413302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423847+0000","last_change":"2023-10-07T09:30:24.963726+0000","last_active":"2023-10-07T09:30:26.423847+0000","last_peered":"2023-10-07T09:30:26.423847+0000","last_clean":"2023-10-07T09:30:26.423847+0000","last_became_active":"2023-10-07T09:30:24.963442+0000","last_became_peered":"2023-10-07T09:30:24.963442+0000","last_unstale":"2023-10-07T09:30:26.423847+0000","last_undegraded":"2023-10-07T09:30:26.423847+0000","last_fullsized":"2023-10-07T09:30:26.423847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:38.051902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413355+0000","last_change":"2023-10-07T09:30:26.406759+0000","last_active":"2023-10-07T09:30:26.413355+0000","last_peered":"2023-10-07T09:30:26.413355+0000","last_clean":"2023-10-07T09:30:26.413355+0000","last_became_active":"2023-10-07T09:30:26.406449+0000","last_became_peered":"2023-10-07T09:30:26.406449+0000","last_unstale":"2023-10-07T09:30:26.413355+0000","last_undegraded":"2023-10-07T09:30:26.413355+0000","last_fullsized":"2023-10-07T09:30:26.413355+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.970510+0000","last_change":"2023-10-07T09:30:24.970510+0000","last_active":"2023-10-07T09:30:24.970510+0000","last_peered":"2023-10-07T09:30:24.970510+0000","last_clean":"2023-10-07T09:30:24.970510+0000","last_became_active":"2023-10-07T09:30:24.970183+0000","last_became_peered":"2023-10-07T09:30:24.970183+0000","last_unstale":"2023-10-07T09:30:24.970510+0000","last_undegraded":"2023-10-07T09:30:24.970510+0000","last_fullsized":"2023-10-07T09:30:24.970510+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412638+0000","last_change":"2023-10-07T09:30:26.404546+0000","last_active":"2023-10-07T09:30:26.412638+0000","last_peered":"2023-10-07T09:30:26.412638+0000","last_clean":"2023-10-07T09:30:26.412638+0000","last_became_active":"2023-10-07T09:30:26.404284+0000","last_became_peered":"2023-10-07T09:30:26.404284+0000","last_unstale":"2023-10-07T09:30:26.412638+0000","last_undegraded":"2023-10-07T09:30:26.412638+0000","last_fullsized":"2023-10-07T09:30:26.412638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.965098+0000","last_change":"2023-10-07T09:30:24.965098+0000","last_active":"2023-10-07T09:30:24.965098+0000","last_peered":"2023-10-07T09:30:24.965098+0000","last_clean":"2023-10-07T09:30:24.965098+0000","last_became_active":"2023-10-07T09:30:24.964594+0000","last_became_peered":"2023-10-07T09:30:24.964594+0000","last_unstale":"2023-10-07T09:30:24.965098+0000","last_undegraded":"2023-10-07T09:30:24.965098+0000","last_fullsized":"2023-10-07T09:30:24.965098+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:56.143330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955888+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:11:46.955055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.393481+0000","last_change":"2023-10-07T09:30:24.967401+0000","last_active":"2023-10-07T09:30:26.393481+0000","last_peered":"2023-10-07T09:30:26.393481+0000","last_clean":"2023-10-07T09:30:26.393481+0000","last_became_active":"2023-10-07T09:30:24.967138+0000","last_became_peered":"2023-10-07T09:30:24.967138+0000","last_unstale":"2023-10-07T09:30:26.393481+0000","last_undegraded":"2023-10-07T09:30:26.393481+0000","last_fullsized":"2023-10-07T09:30:26.393481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414869+0000","last_change":"2023-10-07T09:30:26.411476+0000","last_active":"2023-10-07T09:30:26.414869+0000","last_peered":"2023-10-07T09:30:26.414869+0000","last_clean":"2023-10-07T09:30:26.414869+0000","last_became_active":"2023-10-07T09:30:26.411198+0000","last_became_peered":"2023-10-07T09:30:26.411198+0000","last_unstale":"2023-10-07T09:30:26.414869+0000","last_undegraded":"2023-10-07T09:30:26.414869+0000","last_fullsized":"2023-10-07T09:30:26.414869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423028+0000","last_change":"2023-10-07T09:30:24.971777+0000","last_active":"2023-10-07T09:30:26.423028+0000","last_peered":"2023-10-07T09:30:26.423028+0000","last_clean":"2023-10-07T09:30:26.423028+0000","last_became_active":"2023-10-07T09:30:24.971526+0000","last_became_peered":"2023-10-07T09:30:24.971526+0000","last_unstale":"2023-10-07T09:30:26.423028+0000","last_undegraded":"2023-10-07T09:30:26.423028+0000","last_fullsized":"2023-10-07T09:30:26.423028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:00.376299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410968+0000","last_change":"2023-10-07T09:30:26.404652+0000","last_active":"2023-10-07T09:30:26.410968+0000","last_peered":"2023-10-07T09:30:26.410968+0000","last_clean":"2023-10-07T09:30:26.410968+0000","last_became_active":"2023-10-07T09:30:26.404410+0000","last_became_peered":"2023-10-07T09:30:26.404410+0000","last_unstale":"2023-10-07T09:30:26.410968+0000","last_undegraded":"2023-10-07T09:30:26.410968+0000","last_fullsized":"2023-10-07T09:30:26.410968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390466+0000","last_change":"2023-10-07T09:30:24.963621+0000","last_active":"2023-10-07T09:30:26.390466+0000","last_peered":"2023-10-07T09:30:26.390466+0000","last_clean":"2023-10-07T09:30:26.390466+0000","last_became_active":"2023-10-07T09:30:24.963315+0000","last_became_peered":"2023-10-07T09:30:24.963315+0000","last_unstale":"2023-10-07T09:30:26.390466+0000","last_undegraded":"2023-10-07T09:30:26.390466+0000","last_fullsized":"2023-10-07T09:30:26.390466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410981+0000","last_change":"2023-10-07T09:30:26.404627+0000","last_active":"2023-10-07T09:30:26.410981+0000","last_peered":"2023-10-07T09:30:26.410981+0000","last_clean":"2023-10-07T09:30:26.410981+0000","last_became_active":"2023-10-07T09:30:26.404362+0000","last_became_peered":"2023-10-07T09:30:26.404362+0000","last_unstale":"2023-10-07T09:30:26.410981+0000","last_undegraded":"2023-10-07T09:30:26.410981+0000","last_fullsized":"2023-10-07T09:30:26.410981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392082+0000","last_change":"2023-10-07T09:30:24.972623+0000","last_active":"2023-10-07T09:30:26.392082+0000","last_peered":"2023-10-07T09:30:26.392082+0000","last_clean":"2023-10-07T09:30:26.392082+0000","last_became_active":"2023-10-07T09:30:24.972377+0000","last_became_peered":"2023-10-07T09:30:24.972377+0000","last_unstale":"2023-10-07T09:30:26.392082+0000","last_undegraded":"2023-10-07T09:30:26.392082+0000","last_fullsized":"2023-10-07T09:30:26.392082+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:49.176868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.955354+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390656+0000","last_change":"2023-10-07T09:30:24.968835+0000","last_active":"2023-10-07T09:30:26.390656+0000","last_peered":"2023-10-07T09:30:26.390656+0000","last_clean":"2023-10-07T09:30:26.390656+0000","last_became_active":"2023-10-07T09:30:24.968531+0000","last_became_peered":"2023-10-07T09:30:24.968531+0000","last_unstale":"2023-10-07T09:30:26.390656+0000","last_undegraded":"2023-10-07T09:30:26.390656+0000","last_fullsized":"2023-10-07T09:30:26.390656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:06:46.058261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418399+0000","last_change":"2023-10-07T09:30:26.413894+0000","last_active":"2023-10-07T09:30:26.418399+0000","last_peered":"2023-10-07T09:30:26.418399+0000","last_clean":"2023-10-07T09:30:26.418399+0000","last_became_active":"2023-10-07T09:30:26.413276+0000","last_became_peered":"2023-10-07T09:30:26.413276+0000","last_unstale":"2023-10-07T09:30:26.418399+0000","last_undegraded":"2023-10-07T09:30:26.418399+0000","last_fullsized":"2023-10-07T09:30:26.418399+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392045+0000","last_change":"2023-10-07T09:30:24.971933+0000","last_active":"2023-10-07T09:30:26.392045+0000","last_peered":"2023-10-07T09:30:26.392045+0000","last_clean":"2023-10-07T09:30:26.392045+0000","last_became_active":"2023-10-07T09:30:24.971478+0000","last_became_peered":"2023-10-07T09:30:24.971478+0000","last_unstale":"2023-10-07T09:30:26.392045+0000","last_undegraded":"2023-10-07T09:30:26.392045+0000","last_fullsized":"2023-10-07T09:30:26.392045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:19.116181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410100+0000","last_change":"2023-10-07T09:30:26.402235+0000","last_active":"2023-10-07T09:30:26.410100+0000","last_peered":"2023-10-07T09:30:26.410100+0000","last_clean":"2023-10-07T09:30:26.410100+0000","last_became_active":"2023-10-07T09:30:26.401915+0000","last_became_peered":"2023-10-07T09:30:26.401915+0000","last_unstale":"2023-10-07T09:30:26.410100+0000","last_undegraded":"2023-10-07T09:30:26.410100+0000","last_fullsized":"2023-10-07T09:30:26.410100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:01.010896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.967230+0000","last_change":"2023-10-07T09:30:24.967230+0000","last_active":"2023-10-07T09:30:24.967230+0000","last_peered":"2023-10-07T09:30:24.967230+0000","last_clean":"2023-10-07T09:30:24.967230+0000","last_became_active":"2023-10-07T09:30:24.966740+0000","last_became_peered":"2023-10-07T09:30:24.966740+0000","last_unstale":"2023-10-07T09:30:24.967230+0000","last_undegraded":"2023-10-07T09:30:24.967230+0000","last_fullsized":"2023-10-07T09:30:24.967230+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:31:56.775021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.950417+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384013+0000","last_change":"2023-10-07T09:30:24.967499+0000","last_active":"2023-10-07T09:30:26.384013+0000","last_peered":"2023-10-07T09:30:26.384013+0000","last_clean":"2023-10-07T09:30:26.384013+0000","last_became_active":"2023-10-07T09:30:24.967231+0000","last_became_peered":"2023-10-07T09:30:24.967231+0000","last_unstale":"2023-10-07T09:30:26.384013+0000","last_undegraded":"2023-10-07T09:30:26.384013+0000","last_fullsized":"2023-10-07T09:30:26.384013+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:23.529707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T09:30:24.939563+0000","last_change":"2023-10-07T09:30:24.956851+0000","last_active":"2023-10-07T09:30:24.939563+0000","last_peered":"2023-10-07T09:30:24.939563+0000","last_clean":"2023-10-07T09:30:24.939563+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T09:30:24.939563+0000","last_undegraded":"2023-10-07T09:30:24.939563+0000","last_fullsized":"2023-10-07T09:30:24.939563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:23.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-07T09:30:24.971597+0000","last_change":"2023-10-07T09:30:24.971597+0000","last_active":"2023-10-07T09:30:24.971597+0000","last_peered":"2023-10-07T09:30:24.971597+0000","last_clean":"2023-10-07T09:30:24.971597+0000","last_became_active":"2023-10-07T09:30:24.971341+0000","last_became_peered":"2023-10-07T09:30:24.971341+0000","last_unstale":"2023-10-07T09:30:24.971597+0000","last_undegraded":"2023-10-07T09:30:24.971597+0000","last_fullsized":"2023-10-07T09:30:24.971597+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:15.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-07T09:30:23.945970+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:30:23.945970+0000","last_peered":"2023-10-07T09:30:23.945970+0000","last_clean":"2023-10-07T09:30:23.945970+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:30:23.945970+0000","last_undegraded":"2023-10-07T09:30:23.945970+0000","last_fullsized":"2023-10-07T09:30:23.945970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1648,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298236,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073604,"statfs":{"total":96636764160,"available":96331370496,"internally_reserved":0,"allocated":970752,"data_stored":735622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1,"5min":1,"15min":1},"last":0.88},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71399999999999997}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.89800000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.93300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.628}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.94899999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.81599999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074196,"statfs":{"total":96636764160,"available":96331976704,"internally_reserved":0,"allocated":364544,"data_stored":138999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.984,"5min":2.984,"15min":2.984},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":7.867,"5min":7.867,"15min":7.867},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":8.0399999999999991},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.61099999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.79500000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.93200000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.31},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.755}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.69699999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302356,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069484,"statfs":{"total":96636764160,"available":96327151616,"internally_reserved":0,"allocated":995328,"data_stored":748673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.726,"5min":2.726,"15min":2.726},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.57399999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.92200000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.66700000000000004}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.8199999999999998,"5min":3.8199999999999998,"15min":3.8199999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.627},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":3.855,"5min":3.855,"15min":3.855},"last":1.036}]},{"osd":7,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":3.778,"5min":3.778,"15min":3.778},"last":1.097}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297684,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074156,"statfs":{"total":96636764160,"available":96331935744,"internally_reserved":0,"allocated":405504,"data_stored":158375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.439}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":1.0609999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.876},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.94299999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.91400000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":0.314},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.40000000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.498},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.42699999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.877}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.85299999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.88}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":1.2889999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.276}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":6.0620000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074196,"statfs":{"total":96636764160,"available":96331976704,"internally_reserved":0,"allocated":364544,"data_stored":138999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.623}]},{"osd":3,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.80600000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.83799999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.76600000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.82299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:30:31.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2023-10-07T09:30:31.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.956469+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.956440+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T13:36:09.240847+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.957353+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T11:20:14.772461+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.951211+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2023-10-07T09:30:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.957373+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T10:37:17.334679+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2023-10-07T09:30:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.956822+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:20:47.855402+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2023-10-07T09:30:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.957600+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:13:19.142808+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.951679+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.949829+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T12:32:03.694836+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.950988+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2023-10-07T09:30:31.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.955210+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:58:25.231695+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.754 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2023-10-07T09:30:31.754 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.954033+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T09:30:31.754 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2023-10-07T09:30:31.754 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.955840+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:16:30.437449+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.754 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.956067+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.950450+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:25:07.548280+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.954591+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T12:38:03.049978+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2023-10-07T09:30:31.755 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.949818+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T09:58:26.104877+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.756 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-10-07T09:30:31.756 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.956354+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:58:12.844376+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.756 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2023-10-07T09:30:31.756 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.953547+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T09:30:31.756 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-10-07T09:30:31.756 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.951700+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.757 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-10-07T09:30:31.757 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.955888+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T13:11:46.955055+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.757 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-10-07T09:30:31.757 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.955354+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T09:30:31.757 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2023-10-07T09:30:31.757 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.950417+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T09:30:31.758 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-10-07T09:30:31.758 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T09:30:24.939563+0000', 'last_change': '2023-10-07T09:30:24.956851+0000', 'last_active': '2023-10-07T09:30:24.939563+0000', 'last_peered': '2023-10-07T09:30:24.939563+0000', 'last_clean': '2023-10-07T09:30:24.939563+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T09:30:24.939563+0000', 'last_undegraded': '2023-10-07T09:30:24.939563+0000', 'last_fullsized': '2023-10-07T09:30:24.939563+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'last_clean_scrub_stamp': '2023-10-07T09:30:24.939563+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:24:23.467369+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T09:30:31.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-07T09:30:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.979+0000 7f4f4afff700 1 Processor -- start 2023-10-07T09:30:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.980+0000 7f4f4afff700 1 -- start start 2023-10-07T09:30:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4afff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f440aa520 0x7f4f440aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440a9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4afff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4415d6e0 con 0x7f4f44068620 2023-10-07T09:30:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4afff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4415d9c0 con 0x7f4f440aa520 2023-10-07T09:30:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4415dca0 con 0x7f4f4415d200 2023-10-07T09:30:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4959c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f44068620 0x7f4f44159be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33500/0 (socket says 172.21.15.29:33500) 2023-10-07T09:30:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f4959c700 1 -- 172.21.15.29:0/1900983764 learned_addr learned my addr 172.21.15.29:0/1900983764 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f43fff700 1 --2- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.981+0000 7f4f48d9b700 1 --2- 172.21.15.29:0/1900983764 >> v2:172.21.15.144:3300/0 conn(0x7f4f440aa520 0x7f4f440aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f437fe700 1 -- 172.21.15.29:0/1900983764 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482075352 0 0) 0x7f4f4415d6e0 con 0x7f4f44068620 2023-10-07T09:30:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f437fe700 1 -- 172.21.15.29:0/1900983764 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f34003570 con 0x7f4f44068620 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f48d9b700 1 -- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 msgr2=0x7f4f440a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f48d9b700 1 --2- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440a9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f48d9b700 1 -- 172.21.15.29:0/1900983764 >> v1:172.21.15.29:6789/0 conn(0x7f4f44068620 legacy=0x7f4f44159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f48d9b700 1 -- 172.21.15.29:0/1900983764 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4415df80 con 0x7f4f440aa520 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f48d9b700 1 --2- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f440aa520 0x7f4f440aa8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4f380098e0 tx=0x7f4f380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71cb553ff1682178 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f437fe700 1 -- 172.21.15.29:0/1900983764 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38011630 con 0x7f4f440aa520 2023-10-07T09:30:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f437fe700 1 -- 172.21.15.29:0/1900983764 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f38011790 con 0x7f4f440aa520 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.982+0000 7f4f437fe700 1 -- 172.21.15.29:0/1900983764 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38011b30 con 0x7f4f440aa520 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 -- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f440aa520 msgr2=0x7f4f440aa8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 --2- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f440aa520 0x7f4f440aa8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4f380098e0 tx=0x7f4f380091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 -- 172.21.15.29:0/1900983764 shutdown_connections 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 --2- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f440aa520 0x7f4f440aa8f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 --2- 172.21.15.29:0/1900983764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 -- 172.21.15.29:0/1900983764 >> 172.21.15.29:0/1900983764 conn(0x7f4f440c9f00 msgr2=0x7f4f4406dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 -- 172.21.15.29:0/1900983764 shutdown_connections 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.983+0000 7f4f4afff700 1 -- 172.21.15.29:0/1900983764 wait complete. 2023-10-07T09:30:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.985+0000 7f4f4afff700 1 Processor -- start 2023-10-07T09:30:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.985+0000 7f4f4afff700 1 -- start start 2023-10-07T09:30:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4afff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f44068620 0x7f4f440ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4afff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 0x7f4f44158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4afff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c38a0 con 0x7f4f440aa520 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4afff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c3090 con 0x7f4f44068620 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440aedf0 con 0x7f4f4415d200 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f43fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 0x7f4f44158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f43fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 0x7f4f44158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39736/0 (socket says 172.21.15.29:39736) 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f43fff700 1 -- 172.21.15.29:0/293997188 learned_addr learned my addr 172.21.15.29:0/293997188 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f48d9b700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f44068620 0x7f4f440ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.986+0000 7f4f4959c700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f43fff700 1 -- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 msgr2=0x7f4f440c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f43fff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f43fff700 1 -- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f44068620 msgr2=0x7f4f440ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f43fff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f44068620 0x7f4f440ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f43fff700 1 -- 172.21.15.29:0/293997188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f38009420 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f43fff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 0x7f4f44158140 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000ec50 tx=0x7f4f300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000c850 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3000c9b0 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000cd70 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f44158680 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.987+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f440c35c0 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.988+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f440c3a50 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.989+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4f30013680 con 0x7f4f440aa520 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.990+0000 7f4f40ff9700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4f28066f30 0x7f4f280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.990+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4f30084b70 con 0x7f4f440aa520 2023-10-07T09:30:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.991+0000 7f4f48d9b700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4f28066f30 0x7f4f280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.991+0000 7f4f48d9b700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4f28066f30 0x7f4f280693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f380091e0 tx=0x7f4f38017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:31.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:31.994+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f440c3a50 con 0x7f4f440aa520 2023-10-07T09:30:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.273+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4f440c3a50 con 0x7f4f440aa520 2023-10-07T09:30:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.273+0000 7f4f40ff9700 1 -- 172.21.15.29:0/293997188 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1518 (secure 0 0 0) 0x7f4f30017600 con 0x7f4f440aa520 2023-10-07T09:30:32.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:32.274 INFO:teuthology.orchestra.run.smithi029.stdout:{"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":95,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":36,"num_osds":8,"num_up_osds":8,"osd_up_since":1696670950,"num_in_osds":8,"osd_in_since":1696670940,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":73},{"state_name":"creating+peering","count":24}],"num_pgs":97,"num_pools":3,"num_objects":17,"data_bytes":592016,"bytes_used":2443923456,"bytes_avail":770650189824,"bytes_total":773094113280,"inactive_pgs_ratio":0.24742268025875092,"write_bytes_sec":853,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":4959}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-10-07T09:30:14.030514+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-07T09:30:32.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.275+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4f28066f30 msgr2=0x7f4f280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.275+0000 7f4f4afff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4f28066f30 0x7f4f280693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f380091e0 tx=0x7f4f38017000 comp rx=0 tx=0).stop 2023-10-07T09:30:32.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.275+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 msgr2=0x7f4f44158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.275+0000 7f4f4afff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 0x7f4f44158140 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000ec50 tx=0x7f4f300091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 shutdown_connections 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f4f28066f30 0x7f4f280693e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f4415d200 0x7f4f440c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f44068620 0x7f4f440ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 --2- 172.21.15.29:0/293997188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f440aa520 0x7f4f44158140 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 >> 172.21.15.29:0/293997188 conn(0x7f4f440c9f00 msgr2=0x7f4f44158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 shutdown_connections 2023-10-07T09:30:32.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.276+0000 7f4f4afff700 1 -- 172.21.15.29:0/293997188 wait complete. 2023-10-07T09:30:32.294 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-07T09:30:32.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:30:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.488+0000 7fb0ed3ff700 1 Processor -- start 2023-10-07T09:30:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.488+0000 7fb0ed3ff700 1 -- start start 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0ed3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb0e8159750 0x7fb0e80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0ed3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0e8159b20 0x7fb0e80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0ed3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e80ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0ed3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e80cbf20 con 0x7fb0e8159b20 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0ed3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e80cc200 con 0x7fb0e8159750 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0ed3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80cc4e0 con 0x7fb0e80ce6d0 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0e67fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0e8159b20 0x7fb0e80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0e6ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb0e8159750 0x7fb0e80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0e6ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb0e8159750 0x7fb0e80d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36650/0 (socket says 172.21.15.29:36650) 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0e6ffd700 1 -- 172.21.15.29:0/3418198158 learned_addr learned my addr 172.21.15.29:0/3418198158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.490+0000 7fb0e77fe700 1 --2- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e80ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e6ffd700 1 -- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 msgr2=0x7fb0e80ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e6ffd700 1 --2- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e80ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e6ffd700 1 -- 172.21.15.29:0/3418198158 >> v2:172.21.15.29:3300/0 conn(0x7fb0e8159b20 msgr2=0x7fb0e80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e6ffd700 1 --2- 172.21.15.29:0/3418198158 >> v2:172.21.15.29:3300/0 conn(0x7fb0e8159b20 0x7fb0e80ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e6ffd700 1 -- 172.21.15.29:0/3418198158 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80cc7c0 con 0x7fb0e8159750 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e67fc700 1 --2- 172.21.15.29:0/3418198158 >> v2:172.21.15.29:3300/0 conn(0x7fb0e8159b20 0x7fb0e80ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e6ffd700 1 --2- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e80d10a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8089840 tx=0x7fb0d8009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d3340c2eb07ba3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e5ffb700 1 -- 172.21.15.29:0/3418198158 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8005c10 con 0x7fb0e8159750 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e5ffb700 1 -- 172.21.15.29:0/3418198158 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d8005d70 con 0x7fb0e8159750 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.491+0000 7fb0e5ffb700 1 -- 172.21.15.29:0/3418198158 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8006260 con 0x7fb0e8159750 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 msgr2=0x7fb0e80d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e80d10a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8089840 tx=0x7fb0d8009220 comp rx=0 tx=0).stop 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/3418198158 shutdown_connections 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e80d10a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/3418198158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e80ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/3418198158 >> v2:172.21.15.29:3300/0 conn(0x7fb0e8159b20 0x7fb0e80ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/3418198158 >> 172.21.15.29:0/3418198158 conn(0x7fb0e8063a10 msgr2=0x7fb0e80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/3418198158 shutdown_connections 2023-10-07T09:30:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.492+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/3418198158 wait complete. 2023-10-07T09:30:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.493+0000 7fb0ed3ff700 1 Processor -- start 2023-10-07T09:30:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.494+0000 7fb0ed3ff700 1 -- start start 2023-10-07T09:30:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.494+0000 7fb0ed3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e8140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.494+0000 7fb0ed3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e8141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0ed3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 0x7fb0e80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0ed3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80b55a0 con 0x7fb0e8159b20 2023-10-07T09:30:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0ed3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e815acd0 con 0x7fb0e8159750 2023-10-07T09:30:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0ed3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80b4c70 con 0x7fb0e80ce6d0 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 0x7fb0e80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 0x7fb0e80b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39742/0 (socket says 172.21.15.29:39742) 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e6ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e8140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 -- 172.21.15.29:0/408882072 learned_addr learned my addr 172.21.15.29:0/408882072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e67fc700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e8141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 -- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 msgr2=0x7fb0e8140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e8140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 -- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 msgr2=0x7fb0e8141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e8141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.495+0000 7fb0e77fe700 1 -- 172.21.15.29:0/408882072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc009420 con 0x7fb0e8159b20 2023-10-07T09:30:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.496+0000 7fb0e77fe700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 0x7fb0e80b5a30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009e90 tx=0x7fb0e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.496+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000cb70 con 0x7fb0e8159b20 2023-10-07T09:30:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.496+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e000f040 con 0x7fb0e8159b20 2023-10-07T09:30:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.496+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0014480 con 0x7fb0e8159b20 2023-10-07T09:30:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.496+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d8009480 con 0x7fb0e8159b20 2023-10-07T09:30:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.496+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e80b5230 con 0x7fb0e8159b20 2023-10-07T09:30:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.497+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e80b5750 con 0x7fb0e8159b20 2023-10-07T09:30:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.498+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb0e0014830 con 0x7fb0e8159b20 2023-10-07T09:30:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.500+0000 7fb0ceffd700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb0d4066f60 0x7fb0d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.500+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb0e0085240 con 0x7fb0e8159b20 2023-10-07T09:30:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.500+0000 7fb0e6ffd700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb0d4066f60 0x7fb0d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.501+0000 7fb0e6ffd700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb0d4066f60 0x7fb0d4069410 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009260 tx=0x7fb0d8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.504+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0e80b5750 con 0x7fb0e8159b20 2023-10-07T09:30:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.698+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0e804c3c0 con 0x7fb0d4066f60 2023-10-07T09:30:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.708+0000 7fb0ceffd700 1 -- 172.21.15.29:0/408882072 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288694 (secure 0 0 0) 0x7fb0e804c3c0 con 0x7fb0d4066f60 2023-10-07T09:30:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb0d4066f60 msgr2=0x7fb0d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb0d4066f60 0x7fb0d4069410 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009260 tx=0x7fb0d8016000 comp rx=0 tx=0).stop 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 msgr2=0x7fb0e80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 0x7fb0e80b5a30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009e90 tx=0x7fb0e000b040 comp rx=0 tx=0).stop 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 shutdown_connections 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb0d4066f60 0x7fb0d4069410 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e8159b20 0x7fb0e80b5a30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0e80ce6d0 0x7fb0e8140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 --2- 172.21.15.29:0/408882072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0e8159750 0x7fb0e8141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 >> 172.21.15.29:0/408882072 conn(0x7fb0e8063a10 msgr2=0x7fb0e80687b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 shutdown_connections 2023-10-07T09:30:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:32.712+0000 7fb0ed3ff700 1 -- 172.21.15.29:0/408882072 wait complete. 2023-10-07T09:30:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:30:32.731 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2023-10-07T09:30:32.034806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2386760,"kb_used_data":4360,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752587960,"statfs":{"total":773094113280,"available":770650071040,"internally_reserved":0,"allocated":4464640,"data_stored":2451788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.088508"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412600+0000","last_change":"2023-10-07T09:30:26.410317+0000","last_active":"2023-10-07T09:30:26.412600+0000","last_peered":"2023-10-07T09:30:26.412600+0000","last_clean":"2023-10-07T09:30:26.412600+0000","last_became_active":"2023-10-07T09:30:26.409965+0000","last_became_peered":"2023-10-07T09:30:26.409965+0000","last_unstale":"2023-10-07T09:30:26.412600+0000","last_undegraded":"2023-10-07T09:30:26.412600+0000","last_fullsized":"2023-10-07T09:30:26.412600+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415669+0000","last_change":"2023-10-07T09:30:26.408750+0000","last_active":"2023-10-07T09:30:26.415669+0000","last_peered":"2023-10-07T09:30:26.415669+0000","last_clean":"2023-10-07T09:30:26.415669+0000","last_became_active":"2023-10-07T09:30:26.407823+0000","last_became_peered":"2023-10-07T09:30:26.407823+0000","last_unstale":"2023-10-07T09:30:26.415669+0000","last_undegraded":"2023-10-07T09:30:26.415669+0000","last_fullsized":"2023-10-07T09:30:26.415669+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:58.672867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414461+0000","last_change":"2023-10-07T09:30:26.408350+0000","last_active":"2023-10-07T09:30:26.414461+0000","last_peered":"2023-10-07T09:30:26.414461+0000","last_clean":"2023-10-07T09:30:26.414461+0000","last_became_active":"2023-10-07T09:30:26.408029+0000","last_became_peered":"2023-10-07T09:30:26.408029+0000","last_unstale":"2023-10-07T09:30:26.414461+0000","last_undegraded":"2023-10-07T09:30:26.414461+0000","last_fullsized":"2023-10-07T09:30:26.414461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407832+0000","last_change":"2023-10-07T09:30:26.403115+0000","last_active":"2023-10-07T09:30:26.407832+0000","last_peered":"2023-10-07T09:30:26.407832+0000","last_clean":"2023-10-07T09:30:26.407832+0000","last_became_active":"2023-10-07T09:30:26.402699+0000","last_became_peered":"2023-10-07T09:30:26.402699+0000","last_unstale":"2023-10-07T09:30:26.407832+0000","last_undegraded":"2023-10-07T09:30:26.407832+0000","last_fullsized":"2023-10-07T09:30:26.407832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:06:53.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414913+0000","last_change":"2023-10-07T09:30:26.410770+0000","last_active":"2023-10-07T09:30:26.414913+0000","last_peered":"2023-10-07T09:30:26.414913+0000","last_clean":"2023-10-07T09:30:26.414913+0000","last_became_active":"2023-10-07T09:30:26.410465+0000","last_became_peered":"2023-10-07T09:30:26.410465+0000","last_unstale":"2023-10-07T09:30:26.414913+0000","last_undegraded":"2023-10-07T09:30:26.414913+0000","last_fullsized":"2023-10-07T09:30:26.414913+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.403447+0000","last_change":"2023-10-07T09:30:26.395345+0000","last_active":"2023-10-07T09:30:26.403447+0000","last_peered":"2023-10-07T09:30:26.403447+0000","last_clean":"2023-10-07T09:30:26.403447+0000","last_became_active":"2023-10-07T09:30:26.395042+0000","last_became_peered":"2023-10-07T09:30:26.395042+0000","last_unstale":"2023-10-07T09:30:26.403447+0000","last_undegraded":"2023-10-07T09:30:26.403447+0000","last_fullsized":"2023-10-07T09:30:26.403447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409954+0000","last_change":"2023-10-07T09:30:26.390280+0000","last_active":"2023-10-07T09:30:26.409954+0000","last_peered":"2023-10-07T09:30:26.409954+0000","last_clean":"2023-10-07T09:30:26.409954+0000","last_became_active":"2023-10-07T09:30:26.390003+0000","last_became_peered":"2023-10-07T09:30:26.390003+0000","last_unstale":"2023-10-07T09:30:26.409954+0000","last_undegraded":"2023-10-07T09:30:26.409954+0000","last_fullsized":"2023-10-07T09:30:26.409954+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:36:52.517291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408954+0000","last_change":"2023-10-07T09:30:26.402274+0000","last_active":"2023-10-07T09:30:26.408954+0000","last_peered":"2023-10-07T09:30:26.408954+0000","last_clean":"2023-10-07T09:30:26.408954+0000","last_became_active":"2023-10-07T09:30:26.401927+0000","last_became_peered":"2023-10-07T09:30:26.401927+0000","last_unstale":"2023-10-07T09:30:26.408954+0000","last_undegraded":"2023-10-07T09:30:26.408954+0000","last_fullsized":"2023-10-07T09:30:26.408954+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:15:36.666517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.416845+0000","last_change":"2023-10-07T09:30:26.412014+0000","last_active":"2023-10-07T09:30:26.416845+0000","last_peered":"2023-10-07T09:30:26.416845+0000","last_clean":"2023-10-07T09:30:26.416845+0000","last_became_active":"2023-10-07T09:30:26.411754+0000","last_became_peered":"2023-10-07T09:30:26.411754+0000","last_unstale":"2023-10-07T09:30:26.416845+0000","last_undegraded":"2023-10-07T09:30:26.416845+0000","last_fullsized":"2023-10-07T09:30:26.416845+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:42.651535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.399392+0000","last_change":"2023-10-07T09:30:26.396209+0000","last_active":"2023-10-07T09:30:26.399392+0000","last_peered":"2023-10-07T09:30:26.399392+0000","last_clean":"2023-10-07T09:30:26.399392+0000","last_became_active":"2023-10-07T09:30:26.395573+0000","last_became_peered":"2023-10-07T09:30:26.395573+0000","last_unstale":"2023-10-07T09:30:26.399392+0000","last_undegraded":"2023-10-07T09:30:26.399392+0000","last_fullsized":"2023-10-07T09:30:26.399392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:54.021183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409352+0000","last_change":"2023-10-07T09:30:26.402704+0000","last_active":"2023-10-07T09:30:26.409352+0000","last_peered":"2023-10-07T09:30:26.409352+0000","last_clean":"2023-10-07T09:30:26.409352+0000","last_became_active":"2023-10-07T09:30:26.402452+0000","last_became_peered":"2023-10-07T09:30:26.402452+0000","last_unstale":"2023-10-07T09:30:26.409352+0000","last_undegraded":"2023-10-07T09:30:26.409352+0000","last_fullsized":"2023-10-07T09:30:26.409352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:31:05.985248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413491+0000","last_change":"2023-10-07T09:30:26.409670+0000","last_active":"2023-10-07T09:30:26.413491+0000","last_peered":"2023-10-07T09:30:26.413491+0000","last_clean":"2023-10-07T09:30:26.413491+0000","last_became_active":"2023-10-07T09:30:26.408687+0000","last_became_peered":"2023-10-07T09:30:26.408687+0000","last_unstale":"2023-10-07T09:30:26.413491+0000","last_undegraded":"2023-10-07T09:30:26.413491+0000","last_fullsized":"2023-10-07T09:30:26.413491+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413335+0000","last_change":"2023-10-07T09:30:26.406082+0000","last_active":"2023-10-07T09:30:26.413335+0000","last_peered":"2023-10-07T09:30:26.413335+0000","last_clean":"2023-10-07T09:30:26.413335+0000","last_became_active":"2023-10-07T09:30:26.405724+0000","last_became_peered":"2023-10-07T09:30:26.405724+0000","last_unstale":"2023-10-07T09:30:26.413335+0000","last_undegraded":"2023-10-07T09:30:26.413335+0000","last_fullsized":"2023-10-07T09:30:26.413335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411102+0000","last_change":"2023-10-07T09:30:26.403342+0000","last_active":"2023-10-07T09:30:26.411102+0000","last_peered":"2023-10-07T09:30:26.411102+0000","last_clean":"2023-10-07T09:30:26.411102+0000","last_became_active":"2023-10-07T09:30:26.403138+0000","last_became_peered":"2023-10-07T09:30:26.403138+0000","last_unstale":"2023-10-07T09:30:26.411102+0000","last_undegraded":"2023-10-07T09:30:26.411102+0000","last_fullsized":"2023-10-07T09:30:26.411102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408435+0000","last_change":"2023-10-07T09:30:26.395877+0000","last_active":"2023-10-07T09:30:26.408435+0000","last_peered":"2023-10-07T09:30:26.408435+0000","last_clean":"2023-10-07T09:30:26.408435+0000","last_became_active":"2023-10-07T09:30:26.395596+0000","last_became_peered":"2023-10-07T09:30:26.395596+0000","last_unstale":"2023-10-07T09:30:26.408435+0000","last_undegraded":"2023-10-07T09:30:26.408435+0000","last_fullsized":"2023-10-07T09:30:26.408435+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:20.489580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408761+0000","last_change":"2023-10-07T09:30:26.398142+0000","last_active":"2023-10-07T09:30:26.408761+0000","last_peered":"2023-10-07T09:30:26.408761+0000","last_clean":"2023-10-07T09:30:26.408761+0000","last_became_active":"2023-10-07T09:30:26.397863+0000","last_became_peered":"2023-10-07T09:30:26.397863+0000","last_unstale":"2023-10-07T09:30:26.408761+0000","last_undegraded":"2023-10-07T09:30:26.408761+0000","last_fullsized":"2023-10-07T09:30:26.408761+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:59.225017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408428+0000","last_change":"2023-10-07T09:30:26.394555+0000","last_active":"2023-10-07T09:30:26.408428+0000","last_peered":"2023-10-07T09:30:26.408428+0000","last_clean":"2023-10-07T09:30:26.408428+0000","last_became_active":"2023-10-07T09:30:26.393660+0000","last_became_peered":"2023-10-07T09:30:26.393660+0000","last_unstale":"2023-10-07T09:30:26.408428+0000","last_undegraded":"2023-10-07T09:30:26.408428+0000","last_fullsized":"2023-10-07T09:30:26.408428+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:46:14.675050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409109+0000","last_change":"2023-10-07T09:30:26.394655+0000","last_active":"2023-10-07T09:30:26.409109+0000","last_peered":"2023-10-07T09:30:26.409109+0000","last_clean":"2023-10-07T09:30:26.409109+0000","last_became_active":"2023-10-07T09:30:26.394372+0000","last_became_peered":"2023-10-07T09:30:26.394372+0000","last_unstale":"2023-10-07T09:30:26.409109+0000","last_undegraded":"2023-10-07T09:30:26.409109+0000","last_fullsized":"2023-10-07T09:30:26.409109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:35.770412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405481+0000","last_change":"2023-10-07T09:30:26.402289+0000","last_active":"2023-10-07T09:30:26.405481+0000","last_peered":"2023-10-07T09:30:26.405481+0000","last_clean":"2023-10-07T09:30:26.405481+0000","last_became_active":"2023-10-07T09:30:26.401826+0000","last_became_peered":"2023-10-07T09:30:26.401826+0000","last_unstale":"2023-10-07T09:30:26.405481+0000","last_undegraded":"2023-10-07T09:30:26.405481+0000","last_fullsized":"2023-10-07T09:30:26.405481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:46:15.007773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414990+0000","last_change":"2023-10-07T09:30:26.411654+0000","last_active":"2023-10-07T09:30:26.414990+0000","last_peered":"2023-10-07T09:30:26.414990+0000","last_clean":"2023-10-07T09:30:26.414990+0000","last_became_active":"2023-10-07T09:30:26.411407+0000","last_became_peered":"2023-10-07T09:30:26.411407+0000","last_unstale":"2023-10-07T09:30:26.414990+0000","last_undegraded":"2023-10-07T09:30:26.414990+0000","last_fullsized":"2023-10-07T09:30:26.414990+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414072+0000","last_change":"2023-10-07T09:30:26.405589+0000","last_active":"2023-10-07T09:30:26.414072+0000","last_peered":"2023-10-07T09:30:26.414072+0000","last_clean":"2023-10-07T09:30:26.414072+0000","last_became_active":"2023-10-07T09:30:26.405334+0000","last_became_peered":"2023-10-07T09:30:26.405334+0000","last_unstale":"2023-10-07T09:30:26.414072+0000","last_undegraded":"2023-10-07T09:30:26.414072+0000","last_fullsized":"2023-10-07T09:30:26.414072+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411311+0000","last_change":"2023-10-07T09:30:26.404743+0000","last_active":"2023-10-07T09:30:26.411311+0000","last_peered":"2023-10-07T09:30:26.411311+0000","last_clean":"2023-10-07T09:30:26.411311+0000","last_became_active":"2023-10-07T09:30:26.404395+0000","last_became_peered":"2023-10-07T09:30:26.404395+0000","last_unstale":"2023-10-07T09:30:26.411311+0000","last_undegraded":"2023-10-07T09:30:26.411311+0000","last_fullsized":"2023-10-07T09:30:26.411311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410992+0000","last_change":"2023-10-07T09:30:26.403259+0000","last_active":"2023-10-07T09:30:26.410992+0000","last_peered":"2023-10-07T09:30:26.410992+0000","last_clean":"2023-10-07T09:30:26.410992+0000","last_became_active":"2023-10-07T09:30:26.402926+0000","last_became_peered":"2023-10-07T09:30:26.402926+0000","last_unstale":"2023-10-07T09:30:26.410992+0000","last_undegraded":"2023-10-07T09:30:26.410992+0000","last_fullsized":"2023-10-07T09:30:26.410992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413211+0000","last_change":"2023-10-07T09:30:26.409044+0000","last_active":"2023-10-07T09:30:26.413211+0000","last_peered":"2023-10-07T09:30:26.413211+0000","last_clean":"2023-10-07T09:30:26.413211+0000","last_became_active":"2023-10-07T09:30:26.408764+0000","last_became_peered":"2023-10-07T09:30:26.408764+0000","last_unstale":"2023-10-07T09:30:26.413211+0000","last_undegraded":"2023-10-07T09:30:26.413211+0000","last_fullsized":"2023-10-07T09:30:26.413211+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:47.675483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422810+0000","last_change":"2023-10-07T09:30:26.413627+0000","last_active":"2023-10-07T09:30:26.422810+0000","last_peered":"2023-10-07T09:30:26.422810+0000","last_clean":"2023-10-07T09:30:26.422810+0000","last_became_active":"2023-10-07T09:30:26.413371+0000","last_became_peered":"2023-10-07T09:30:26.413371+0000","last_unstale":"2023-10-07T09:30:26.422810+0000","last_undegraded":"2023-10-07T09:30:26.422810+0000","last_fullsized":"2023-10-07T09:30:26.422810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:34.133968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415691+0000","last_change":"2023-10-07T09:30:26.408999+0000","last_active":"2023-10-07T09:30:26.415691+0000","last_peered":"2023-10-07T09:30:26.415691+0000","last_clean":"2023-10-07T09:30:26.415691+0000","last_became_active":"2023-10-07T09:30:26.408201+0000","last_became_peered":"2023-10-07T09:30:26.408201+0000","last_unstale":"2023-10-07T09:30:26.415691+0000","last_undegraded":"2023-10-07T09:30:26.415691+0000","last_fullsized":"2023-10-07T09:30:26.415691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421410+0000","last_change":"2023-10-07T09:30:26.411198+0000","last_active":"2023-10-07T09:30:26.421410+0000","last_peered":"2023-10-07T09:30:26.421410+0000","last_clean":"2023-10-07T09:30:26.421410+0000","last_became_active":"2023-10-07T09:30:26.410735+0000","last_became_peered":"2023-10-07T09:30:26.410735+0000","last_unstale":"2023-10-07T09:30:26.421410+0000","last_undegraded":"2023-10-07T09:30:26.421410+0000","last_fullsized":"2023-10-07T09:30:26.421410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:24.740454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413473+0000","last_change":"2023-10-07T09:30:26.409340+0000","last_active":"2023-10-07T09:30:26.413473+0000","last_peered":"2023-10-07T09:30:26.413473+0000","last_clean":"2023-10-07T09:30:26.413473+0000","last_became_active":"2023-10-07T09:30:26.408986+0000","last_became_peered":"2023-10-07T09:30:26.408986+0000","last_unstale":"2023-10-07T09:30:26.413473+0000","last_undegraded":"2023-10-07T09:30:26.413473+0000","last_fullsized":"2023-10-07T09:30:26.413473+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413566+0000","last_change":"2023-10-07T09:30:26.404629+0000","last_active":"2023-10-07T09:30:26.413566+0000","last_peered":"2023-10-07T09:30:26.413566+0000","last_clean":"2023-10-07T09:30:26.413566+0000","last_became_active":"2023-10-07T09:30:26.404320+0000","last_became_peered":"2023-10-07T09:30:26.404320+0000","last_unstale":"2023-10-07T09:30:26.413566+0000","last_undegraded":"2023-10-07T09:30:26.413566+0000","last_fullsized":"2023-10-07T09:30:26.413566+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:13.869543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408330+0000","last_change":"2023-10-07T09:30:26.392465+0000","last_active":"2023-10-07T09:30:26.408330+0000","last_peered":"2023-10-07T09:30:26.408330+0000","last_clean":"2023-10-07T09:30:26.408330+0000","last_became_active":"2023-10-07T09:30:26.392140+0000","last_became_peered":"2023-10-07T09:30:26.392140+0000","last_unstale":"2023-10-07T09:30:26.408330+0000","last_undegraded":"2023-10-07T09:30:26.408330+0000","last_fullsized":"2023-10-07T09:30:26.408330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:34:46.460732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405565+0000","last_change":"2023-10-07T09:30:26.401922+0000","last_active":"2023-10-07T09:30:26.405565+0000","last_peered":"2023-10-07T09:30:26.405565+0000","last_clean":"2023-10-07T09:30:26.405565+0000","last_became_active":"2023-10-07T09:30:26.401633+0000","last_became_peered":"2023-10-07T09:30:26.401633+0000","last_unstale":"2023-10-07T09:30:26.405565+0000","last_undegraded":"2023-10-07T09:30:26.405565+0000","last_fullsized":"2023-10-07T09:30:26.405565+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408933+0000","last_change":"2023-10-07T09:30:26.401905+0000","last_active":"2023-10-07T09:30:26.408933+0000","last_peered":"2023-10-07T09:30:26.408933+0000","last_clean":"2023-10-07T09:30:26.408933+0000","last_became_active":"2023-10-07T09:30:26.401318+0000","last_became_peered":"2023-10-07T09:30:26.401318+0000","last_unstale":"2023-10-07T09:30:26.408933+0000","last_undegraded":"2023-10-07T09:30:26.408933+0000","last_fullsized":"2023-10-07T09:30:26.408933+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:38:16.579339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418432+0000","last_change":"2023-10-07T09:30:26.413698+0000","last_active":"2023-10-07T09:30:26.418432+0000","last_peered":"2023-10-07T09:30:26.418432+0000","last_clean":"2023-10-07T09:30:26.418432+0000","last_became_active":"2023-10-07T09:30:26.413219+0000","last_became_peered":"2023-10-07T09:30:26.413219+0000","last_unstale":"2023-10-07T09:30:26.418432+0000","last_undegraded":"2023-10-07T09:30:26.418432+0000","last_fullsized":"2023-10-07T09:30:26.418432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422004+0000","last_change":"2023-10-07T09:30:24.970001+0000","last_active":"2023-10-07T09:30:26.422004+0000","last_peered":"2023-10-07T09:30:26.422004+0000","last_clean":"2023-10-07T09:30:26.422004+0000","last_became_active":"2023-10-07T09:30:24.969692+0000","last_became_peered":"2023-10-07T09:30:24.969692+0000","last_unstale":"2023-10-07T09:30:26.422004+0000","last_undegraded":"2023-10-07T09:30:26.422004+0000","last_fullsized":"2023-10-07T09:30:26.422004+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405616+0000","last_change":"2023-10-07T09:30:26.402818+0000","last_active":"2023-10-07T09:30:26.405616+0000","last_peered":"2023-10-07T09:30:26.405616+0000","last_clean":"2023-10-07T09:30:26.405616+0000","last_became_active":"2023-10-07T09:30:26.402490+0000","last_became_peered":"2023-10-07T09:30:26.402490+0000","last_unstale":"2023-10-07T09:30:26.405616+0000","last_undegraded":"2023-10-07T09:30:26.405616+0000","last_fullsized":"2023-10-07T09:30:26.405616+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:13.869543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421779+0000","last_change":"2023-10-07T09:30:24.961003+0000","last_active":"2023-10-07T09:30:26.421779+0000","last_peered":"2023-10-07T09:30:26.421779+0000","last_clean":"2023-10-07T09:30:26.421779+0000","last_became_active":"2023-10-07T09:30:24.960711+0000","last_became_peered":"2023-10-07T09:30:24.960711+0000","last_unstale":"2023-10-07T09:30:26.421779+0000","last_undegraded":"2023-10-07T09:30:26.421779+0000","last_fullsized":"2023-10-07T09:30:26.421779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:06.545101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.416884+0000","last_change":"2023-10-07T09:30:26.413169+0000","last_active":"2023-10-07T09:30:26.416884+0000","last_peered":"2023-10-07T09:30:26.416884+0000","last_clean":"2023-10-07T09:30:26.416884+0000","last_became_active":"2023-10-07T09:30:26.412855+0000","last_became_peered":"2023-10-07T09:30:26.412855+0000","last_unstale":"2023-10-07T09:30:26.416884+0000","last_undegraded":"2023-10-07T09:30:26.416884+0000","last_fullsized":"2023-10-07T09:30:26.416884+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:24.532886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424146+0000","last_change":"2023-10-07T09:30:24.967737+0000","last_active":"2023-10-07T09:30:26.424146+0000","last_peered":"2023-10-07T09:30:26.424146+0000","last_clean":"2023-10-07T09:30:26.424146+0000","last_became_active":"2023-10-07T09:30:24.966888+0000","last_became_peered":"2023-10-07T09:30:24.966888+0000","last_unstale":"2023-10-07T09:30:26.424146+0000","last_undegraded":"2023-10-07T09:30:26.424146+0000","last_fullsized":"2023-10-07T09:30:26.424146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421389+0000","last_change":"2023-10-07T09:30:26.411013+0000","last_active":"2023-10-07T09:30:26.421389+0000","last_peered":"2023-10-07T09:30:26.421389+0000","last_clean":"2023-10-07T09:30:26.421389+0000","last_became_active":"2023-10-07T09:30:26.410413+0000","last_became_peered":"2023-10-07T09:30:26.410413+0000","last_unstale":"2023-10-07T09:30:26.421389+0000","last_undegraded":"2023-10-07T09:30:26.421389+0000","last_fullsized":"2023-10-07T09:30:26.421389+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:06.340913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423552+0000","last_change":"2023-10-07T09:30:24.966147+0000","last_active":"2023-10-07T09:30:26.423552+0000","last_peered":"2023-10-07T09:30:26.423552+0000","last_clean":"2023-10-07T09:30:26.423552+0000","last_became_active":"2023-10-07T09:30:24.965853+0000","last_became_peered":"2023-10-07T09:30:24.965853+0000","last_unstale":"2023-10-07T09:30:26.423552+0000","last_undegraded":"2023-10-07T09:30:26.423552+0000","last_fullsized":"2023-10-07T09:30:26.423552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409239+0000","last_change":"2023-10-07T09:30:26.402741+0000","last_active":"2023-10-07T09:30:26.409239+0000","last_peered":"2023-10-07T09:30:26.409239+0000","last_clean":"2023-10-07T09:30:26.409239+0000","last_became_active":"2023-10-07T09:30:26.402366+0000","last_became_peered":"2023-10-07T09:30:26.402366+0000","last_unstale":"2023-10-07T09:30:26.409239+0000","last_undegraded":"2023-10-07T09:30:26.409239+0000","last_fullsized":"2023-10-07T09:30:26.409239+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:56:23.358835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421752+0000","last_change":"2023-10-07T09:30:24.967094+0000","last_active":"2023-10-07T09:30:26.421752+0000","last_peered":"2023-10-07T09:30:26.421752+0000","last_clean":"2023-10-07T09:30:26.421752+0000","last_became_active":"2023-10-07T09:30:24.966810+0000","last_became_peered":"2023-10-07T09:30:24.966810+0000","last_unstale":"2023-10-07T09:30:26.421752+0000","last_undegraded":"2023-10-07T09:30:26.421752+0000","last_fullsized":"2023-10-07T09:30:26.421752+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:24:57.676848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414299+0000","last_change":"2023-10-07T09:30:26.407749+0000","last_active":"2023-10-07T09:30:26.414299+0000","last_peered":"2023-10-07T09:30:26.414299+0000","last_clean":"2023-10-07T09:30:26.414299+0000","last_became_active":"2023-10-07T09:30:26.407433+0000","last_became_peered":"2023-10-07T09:30:26.407433+0000","last_unstale":"2023-10-07T09:30:26.414299+0000","last_undegraded":"2023-10-07T09:30:26.414299+0000","last_fullsized":"2023-10-07T09:30:26.414299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421806+0000","last_change":"2023-10-07T09:30:24.963628+0000","last_active":"2023-10-07T09:30:26.421806+0000","last_peered":"2023-10-07T09:30:26.421806+0000","last_clean":"2023-10-07T09:30:26.421806+0000","last_became_active":"2023-10-07T09:30:24.963306+0000","last_became_peered":"2023-10-07T09:30:24.963306+0000","last_unstale":"2023-10-07T09:30:26.421806+0000","last_undegraded":"2023-10-07T09:30:26.421806+0000","last_fullsized":"2023-10-07T09:30:26.421806+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:12.068751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413311+0000","last_change":"2023-10-07T09:30:26.406943+0000","last_active":"2023-10-07T09:30:26.413311+0000","last_peered":"2023-10-07T09:30:26.413311+0000","last_clean":"2023-10-07T09:30:26.413311+0000","last_became_active":"2023-10-07T09:30:26.406650+0000","last_became_peered":"2023-10-07T09:30:26.406650+0000","last_unstale":"2023-10-07T09:30:26.413311+0000","last_undegraded":"2023-10-07T09:30:26.413311+0000","last_fullsized":"2023-10-07T09:30:26.413311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.383857+0000","last_change":"2023-10-07T09:30:24.965572+0000","last_active":"2023-10-07T09:30:26.383857+0000","last_peered":"2023-10-07T09:30:26.383857+0000","last_clean":"2023-10-07T09:30:26.383857+0000","last_became_active":"2023-10-07T09:30:24.965320+0000","last_became_peered":"2023-10-07T09:30:24.965320+0000","last_unstale":"2023-10-07T09:30:26.383857+0000","last_undegraded":"2023-10-07T09:30:26.383857+0000","last_fullsized":"2023-10-07T09:30:26.383857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:49.978481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414174+0000","last_change":"2023-10-07T09:30:26.409216+0000","last_active":"2023-10-07T09:30:26.414174+0000","last_peered":"2023-10-07T09:30:26.414174+0000","last_clean":"2023-10-07T09:30:26.414174+0000","last_became_active":"2023-10-07T09:30:26.408947+0000","last_became_peered":"2023-10-07T09:30:26.408947+0000","last_unstale":"2023-10-07T09:30:26.414174+0000","last_undegraded":"2023-10-07T09:30:26.414174+0000","last_fullsized":"2023-10-07T09:30:26.414174+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.394513+0000","last_change":"2023-10-07T09:30:24.962784+0000","last_active":"2023-10-07T09:30:26.394513+0000","last_peered":"2023-10-07T09:30:26.394513+0000","last_clean":"2023-10-07T09:30:26.394513+0000","last_became_active":"2023-10-07T09:30:24.962409+0000","last_became_peered":"2023-10-07T09:30:24.962409+0000","last_unstale":"2023-10-07T09:30:26.394513+0000","last_undegraded":"2023-10-07T09:30:26.394513+0000","last_fullsized":"2023-10-07T09:30:26.394513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:04.344607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413097+0000","last_change":"2023-10-07T09:30:26.406965+0000","last_active":"2023-10-07T09:30:26.413097+0000","last_peered":"2023-10-07T09:30:26.413097+0000","last_clean":"2023-10-07T09:30:26.413097+0000","last_became_active":"2023-10-07T09:30:26.406607+0000","last_became_peered":"2023-10-07T09:30:26.406607+0000","last_unstale":"2023-10-07T09:30:26.413097+0000","last_undegraded":"2023-10-07T09:30:26.413097+0000","last_fullsized":"2023-10-07T09:30:26.413097+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384959+0000","last_change":"2023-10-07T09:30:24.967714+0000","last_active":"2023-10-07T09:30:26.384959+0000","last_peered":"2023-10-07T09:30:26.384959+0000","last_clean":"2023-10-07T09:30:26.384959+0000","last_became_active":"2023-10-07T09:30:24.967416+0000","last_became_peered":"2023-10-07T09:30:24.967416+0000","last_unstale":"2023-10-07T09:30:26.384959+0000","last_undegraded":"2023-10-07T09:30:26.384959+0000","last_fullsized":"2023-10-07T09:30:26.384959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:25.781083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402385+0000","last_change":"2023-10-07T09:30:26.395987+0000","last_active":"2023-10-07T09:30:26.402385+0000","last_peered":"2023-10-07T09:30:26.402385+0000","last_clean":"2023-10-07T09:30:26.402385+0000","last_became_active":"2023-10-07T09:30:26.395709+0000","last_became_peered":"2023-10-07T09:30:26.395709+0000","last_unstale":"2023-10-07T09:30:26.402385+0000","last_undegraded":"2023-10-07T09:30:26.402385+0000","last_fullsized":"2023-10-07T09:30:26.402385+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390349+0000","last_change":"2023-10-07T09:30:24.962930+0000","last_active":"2023-10-07T09:30:26.390349+0000","last_peered":"2023-10-07T09:30:26.390349+0000","last_clean":"2023-10-07T09:30:26.390349+0000","last_became_active":"2023-10-07T09:30:24.962586+0000","last_became_peered":"2023-10-07T09:30:24.962586+0000","last_unstale":"2023-10-07T09:30:26.390349+0000","last_undegraded":"2023-10-07T09:30:26.390349+0000","last_fullsized":"2023-10-07T09:30:26.390349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.417232+0000","last_change":"2023-10-07T09:30:26.412395+0000","last_active":"2023-10-07T09:30:26.417232+0000","last_peered":"2023-10-07T09:30:26.417232+0000","last_clean":"2023-10-07T09:30:26.417232+0000","last_became_active":"2023-10-07T09:30:26.412085+0000","last_became_peered":"2023-10-07T09:30:26.412085+0000","last_unstale":"2023-10-07T09:30:26.417232+0000","last_undegraded":"2023-10-07T09:30:26.417232+0000","last_fullsized":"2023-10-07T09:30:26.417232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421318+0000","last_change":"2023-10-07T09:30:24.967809+0000","last_active":"2023-10-07T09:30:26.421318+0000","last_peered":"2023-10-07T09:30:26.421318+0000","last_clean":"2023-10-07T09:30:26.421318+0000","last_became_active":"2023-10-07T09:30:24.967550+0000","last_became_peered":"2023-10-07T09:30:24.967550+0000","last_unstale":"2023-10-07T09:30:26.421318+0000","last_undegraded":"2023-10-07T09:30:26.421318+0000","last_fullsized":"2023-10-07T09:30:26.421318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414443+0000","last_change":"2023-10-07T09:30:26.408025+0000","last_active":"2023-10-07T09:30:26.414443+0000","last_peered":"2023-10-07T09:30:26.414443+0000","last_clean":"2023-10-07T09:30:26.414443+0000","last_became_active":"2023-10-07T09:30:26.407736+0000","last_became_peered":"2023-10-07T09:30:26.407736+0000","last_unstale":"2023-10-07T09:30:26.414443+0000","last_undegraded":"2023-10-07T09:30:26.414443+0000","last_fullsized":"2023-10-07T09:30:26.414443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.425863+0000","last_change":"2023-10-07T09:30:24.966563+0000","last_active":"2023-10-07T09:30:26.425863+0000","last_peered":"2023-10-07T09:30:26.425863+0000","last_clean":"2023-10-07T09:30:26.425863+0000","last_became_active":"2023-10-07T09:30:24.966273+0000","last_became_peered":"2023-10-07T09:30:24.966273+0000","last_unstale":"2023-10-07T09:30:26.425863+0000","last_undegraded":"2023-10-07T09:30:26.425863+0000","last_fullsized":"2023-10-07T09:30:26.425863+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:41.648356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421340+0000","last_change":"2023-10-07T09:30:26.409745+0000","last_active":"2023-10-07T09:30:26.421340+0000","last_peered":"2023-10-07T09:30:26.421340+0000","last_clean":"2023-10-07T09:30:26.421340+0000","last_became_active":"2023-10-07T09:30:26.409489+0000","last_became_peered":"2023-10-07T09:30:26.409489+0000","last_unstale":"2023-10-07T09:30:26.421340+0000","last_undegraded":"2023-10-07T09:30:26.421340+0000","last_fullsized":"2023-10-07T09:30:26.421340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421862+0000","last_change":"2023-10-07T09:30:24.964382+0000","last_active":"2023-10-07T09:30:26.421862+0000","last_peered":"2023-10-07T09:30:26.421862+0000","last_clean":"2023-10-07T09:30:26.421862+0000","last_became_active":"2023-10-07T09:30:24.964076+0000","last_became_peered":"2023-10-07T09:30:24.964076+0000","last_unstale":"2023-10-07T09:30:26.421862+0000","last_undegraded":"2023-10-07T09:30:26.421862+0000","last_fullsized":"2023-10-07T09:30:26.421862+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:19:17.590704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402685+0000","last_change":"2023-10-07T09:30:26.396545+0000","last_active":"2023-10-07T09:30:26.402685+0000","last_peered":"2023-10-07T09:30:26.402685+0000","last_clean":"2023-10-07T09:30:26.402685+0000","last_became_active":"2023-10-07T09:30:26.395816+0000","last_became_peered":"2023-10-07T09:30:26.395816+0000","last_unstale":"2023-10-07T09:30:26.402685+0000","last_undegraded":"2023-10-07T09:30:26.402685+0000","last_fullsized":"2023-10-07T09:30:26.402685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.391928+0000","last_change":"2023-10-07T09:30:24.964522+0000","last_active":"2023-10-07T09:30:26.391928+0000","last_peered":"2023-10-07T09:30:26.391928+0000","last_clean":"2023-10-07T09:30:26.391928+0000","last_became_active":"2023-10-07T09:30:24.964221+0000","last_became_peered":"2023-10-07T09:30:24.964221+0000","last_unstale":"2023-10-07T09:30:26.391928+0000","last_undegraded":"2023-10-07T09:30:26.391928+0000","last_fullsized":"2023-10-07T09:30:26.391928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:35.312512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408972+0000","last_change":"2023-10-07T09:30:26.402134+0000","last_active":"2023-10-07T09:30:26.408972+0000","last_peered":"2023-10-07T09:30:26.408972+0000","last_clean":"2023-10-07T09:30:26.408972+0000","last_became_active":"2023-10-07T09:30:26.401552+0000","last_became_peered":"2023-10-07T09:30:26.401552+0000","last_unstale":"2023-10-07T09:30:26.408972+0000","last_undegraded":"2023-10-07T09:30:26.408972+0000","last_fullsized":"2023-10-07T09:30:26.408972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:08:44.978453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384014+0000","last_change":"2023-10-07T09:30:24.967550+0000","last_active":"2023-10-07T09:30:26.384014+0000","last_peered":"2023-10-07T09:30:26.384014+0000","last_clean":"2023-10-07T09:30:26.384014+0000","last_became_active":"2023-10-07T09:30:24.967265+0000","last_became_peered":"2023-10-07T09:30:24.967265+0000","last_unstale":"2023-10-07T09:30:26.384014+0000","last_undegraded":"2023-10-07T09:30:26.384014+0000","last_fullsized":"2023-10-07T09:30:26.384014+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:12.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405587+0000","last_change":"2023-10-07T09:30:26.402637+0000","last_active":"2023-10-07T09:30:26.405587+0000","last_peered":"2023-10-07T09:30:26.405587+0000","last_clean":"2023-10-07T09:30:26.405587+0000","last_became_active":"2023-10-07T09:30:26.402364+0000","last_became_peered":"2023-10-07T09:30:26.402364+0000","last_unstale":"2023-10-07T09:30:26.405587+0000","last_undegraded":"2023-10-07T09:30:26.405587+0000","last_fullsized":"2023-10-07T09:30:26.405587+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424346+0000","last_change":"2023-10-07T09:30:24.971275+0000","last_active":"2023-10-07T09:30:26.424346+0000","last_peered":"2023-10-07T09:30:26.424346+0000","last_clean":"2023-10-07T09:30:26.424346+0000","last_became_active":"2023-10-07T09:30:24.970929+0000","last_became_peered":"2023-10-07T09:30:24.970929+0000","last_unstale":"2023-10-07T09:30:26.424346+0000","last_undegraded":"2023-10-07T09:30:26.424346+0000","last_fullsized":"2023-10-07T09:30:26.424346+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421090+0000","last_change":"2023-10-07T09:30:26.415671+0000","last_active":"2023-10-07T09:30:26.421090+0000","last_peered":"2023-10-07T09:30:26.421090+0000","last_clean":"2023-10-07T09:30:26.421090+0000","last_became_active":"2023-10-07T09:30:26.415360+0000","last_became_peered":"2023-10-07T09:30:26.415360+0000","last_unstale":"2023-10-07T09:30:26.421090+0000","last_undegraded":"2023-10-07T09:30:26.421090+0000","last_fullsized":"2023-10-07T09:30:26.421090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423122+0000","last_change":"2023-10-07T09:30:24.968081+0000","last_active":"2023-10-07T09:30:26.423122+0000","last_peered":"2023-10-07T09:30:26.423122+0000","last_clean":"2023-10-07T09:30:26.423122+0000","last_became_active":"2023-10-07T09:30:24.967047+0000","last_became_peered":"2023-10-07T09:30:24.967047+0000","last_unstale":"2023-10-07T09:30:26.423122+0000","last_undegraded":"2023-10-07T09:30:26.423122+0000","last_fullsized":"2023-10-07T09:30:26.423122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:00.381039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413417+0000","last_change":"2023-10-07T09:30:26.407336+0000","last_active":"2023-10-07T09:30:26.413417+0000","last_peered":"2023-10-07T09:30:26.413417+0000","last_clean":"2023-10-07T09:30:26.413417+0000","last_became_active":"2023-10-07T09:30:26.407027+0000","last_became_peered":"2023-10-07T09:30:26.407027+0000","last_unstale":"2023-10-07T09:30:26.413417+0000","last_undegraded":"2023-10-07T09:30:26.413417+0000","last_fullsized":"2023-10-07T09:30:26.413417+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422008+0000","last_change":"2023-10-07T09:30:24.962904+0000","last_active":"2023-10-07T09:30:26.422008+0000","last_peered":"2023-10-07T09:30:26.422008+0000","last_clean":"2023-10-07T09:30:26.422008+0000","last_became_active":"2023-10-07T09:30:24.962519+0000","last_became_peered":"2023-10-07T09:30:24.962519+0000","last_unstale":"2023-10-07T09:30:26.422008+0000","last_undegraded":"2023-10-07T09:30:26.422008+0000","last_fullsized":"2023-10-07T09:30:26.422008+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:00.363653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407852+0000","last_change":"2023-10-07T09:30:26.403114+0000","last_active":"2023-10-07T09:30:26.407852+0000","last_peered":"2023-10-07T09:30:26.407852+0000","last_clean":"2023-10-07T09:30:26.407852+0000","last_became_active":"2023-10-07T09:30:26.402830+0000","last_became_peered":"2023-10-07T09:30:26.402830+0000","last_unstale":"2023-10-07T09:30:26.407852+0000","last_undegraded":"2023-10-07T09:30:26.407852+0000","last_fullsized":"2023-10-07T09:30:26.407852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421820+0000","last_change":"2023-10-07T09:30:24.961853+0000","last_active":"2023-10-07T09:30:26.421820+0000","last_peered":"2023-10-07T09:30:26.421820+0000","last_clean":"2023-10-07T09:30:26.421820+0000","last_became_active":"2023-10-07T09:30:24.961591+0000","last_became_peered":"2023-10-07T09:30:24.961591+0000","last_unstale":"2023-10-07T09:30:26.421820+0000","last_undegraded":"2023-10-07T09:30:26.421820+0000","last_fullsized":"2023-10-07T09:30:26.421820+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415642+0000","last_change":"2023-10-07T09:30:26.413595+0000","last_active":"2023-10-07T09:30:26.415642+0000","last_peered":"2023-10-07T09:30:26.415642+0000","last_clean":"2023-10-07T09:30:26.415642+0000","last_became_active":"2023-10-07T09:30:26.413270+0000","last_became_peered":"2023-10-07T09:30:26.413270+0000","last_unstale":"2023-10-07T09:30:26.415642+0000","last_undegraded":"2023-10-07T09:30:26.415642+0000","last_fullsized":"2023-10-07T09:30:26.415642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392150+0000","last_change":"2023-10-07T09:30:24.967115+0000","last_active":"2023-10-07T09:30:26.392150+0000","last_peered":"2023-10-07T09:30:26.392150+0000","last_clean":"2023-10-07T09:30:26.392150+0000","last_became_active":"2023-10-07T09:30:24.966832+0000","last_became_peered":"2023-10-07T09:30:24.966832+0000","last_unstale":"2023-10-07T09:30:26.392150+0000","last_undegraded":"2023-10-07T09:30:26.392150+0000","last_fullsized":"2023-10-07T09:30:26.392150+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:11:11.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413302+0000","last_change":"2023-10-07T09:30:26.408606+0000","last_active":"2023-10-07T09:30:26.413302+0000","last_peered":"2023-10-07T09:30:26.413302+0000","last_clean":"2023-10-07T09:30:26.413302+0000","last_became_active":"2023-10-07T09:30:26.408343+0000","last_became_peered":"2023-10-07T09:30:26.408343+0000","last_unstale":"2023-10-07T09:30:26.413302+0000","last_undegraded":"2023-10-07T09:30:26.413302+0000","last_fullsized":"2023-10-07T09:30:26.413302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423847+0000","last_change":"2023-10-07T09:30:24.963726+0000","last_active":"2023-10-07T09:30:26.423847+0000","last_peered":"2023-10-07T09:30:26.423847+0000","last_clean":"2023-10-07T09:30:26.423847+0000","last_became_active":"2023-10-07T09:30:24.963442+0000","last_became_peered":"2023-10-07T09:30:24.963442+0000","last_unstale":"2023-10-07T09:30:26.423847+0000","last_undegraded":"2023-10-07T09:30:26.423847+0000","last_fullsized":"2023-10-07T09:30:26.423847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:38.051902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413355+0000","last_change":"2023-10-07T09:30:26.406759+0000","last_active":"2023-10-07T09:30:26.413355+0000","last_peered":"2023-10-07T09:30:26.413355+0000","last_clean":"2023-10-07T09:30:26.413355+0000","last_became_active":"2023-10-07T09:30:26.406449+0000","last_became_peered":"2023-10-07T09:30:26.406449+0000","last_unstale":"2023-10-07T09:30:26.413355+0000","last_undegraded":"2023-10-07T09:30:26.413355+0000","last_fullsized":"2023-10-07T09:30:26.413355+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.385246+0000","last_change":"2023-10-07T09:30:24.970510+0000","last_active":"2023-10-07T09:30:26.385246+0000","last_peered":"2023-10-07T09:30:26.385246+0000","last_clean":"2023-10-07T09:30:26.385246+0000","last_became_active":"2023-10-07T09:30:24.970183+0000","last_became_peered":"2023-10-07T09:30:24.970183+0000","last_unstale":"2023-10-07T09:30:26.385246+0000","last_undegraded":"2023-10-07T09:30:26.385246+0000","last_fullsized":"2023-10-07T09:30:26.385246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412638+0000","last_change":"2023-10-07T09:30:26.404546+0000","last_active":"2023-10-07T09:30:26.412638+0000","last_peered":"2023-10-07T09:30:26.412638+0000","last_clean":"2023-10-07T09:30:26.412638+0000","last_became_active":"2023-10-07T09:30:26.404284+0000","last_became_peered":"2023-10-07T09:30:26.404284+0000","last_unstale":"2023-10-07T09:30:26.412638+0000","last_undegraded":"2023-10-07T09:30:26.412638+0000","last_fullsized":"2023-10-07T09:30:26.412638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.394753+0000","last_change":"2023-10-07T09:30:24.965098+0000","last_active":"2023-10-07T09:30:26.394753+0000","last_peered":"2023-10-07T09:30:26.394753+0000","last_clean":"2023-10-07T09:30:26.394753+0000","last_became_active":"2023-10-07T09:30:24.964594+0000","last_became_peered":"2023-10-07T09:30:24.964594+0000","last_unstale":"2023-10-07T09:30:26.394753+0000","last_undegraded":"2023-10-07T09:30:26.394753+0000","last_fullsized":"2023-10-07T09:30:26.394753+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:56.143330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409978+0000","last_change":"2023-10-07T09:30:26.393757+0000","last_active":"2023-10-07T09:30:26.409978+0000","last_peered":"2023-10-07T09:30:26.409978+0000","last_clean":"2023-10-07T09:30:26.409978+0000","last_became_active":"2023-10-07T09:30:26.393474+0000","last_became_peered":"2023-10-07T09:30:26.393474+0000","last_unstale":"2023-10-07T09:30:26.409978+0000","last_undegraded":"2023-10-07T09:30:26.409978+0000","last_fullsized":"2023-10-07T09:30:26.409978+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:06.050510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.393481+0000","last_change":"2023-10-07T09:30:24.967401+0000","last_active":"2023-10-07T09:30:26.393481+0000","last_peered":"2023-10-07T09:30:26.393481+0000","last_clean":"2023-10-07T09:30:26.393481+0000","last_became_active":"2023-10-07T09:30:24.967138+0000","last_became_peered":"2023-10-07T09:30:24.967138+0000","last_unstale":"2023-10-07T09:30:26.393481+0000","last_undegraded":"2023-10-07T09:30:26.393481+0000","last_fullsized":"2023-10-07T09:30:26.393481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414869+0000","last_change":"2023-10-07T09:30:26.411476+0000","last_active":"2023-10-07T09:30:26.414869+0000","last_peered":"2023-10-07T09:30:26.414869+0000","last_clean":"2023-10-07T09:30:26.414869+0000","last_became_active":"2023-10-07T09:30:26.411198+0000","last_became_peered":"2023-10-07T09:30:26.411198+0000","last_unstale":"2023-10-07T09:30:26.414869+0000","last_undegraded":"2023-10-07T09:30:26.414869+0000","last_fullsized":"2023-10-07T09:30:26.414869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423028+0000","last_change":"2023-10-07T09:30:24.971777+0000","last_active":"2023-10-07T09:30:26.423028+0000","last_peered":"2023-10-07T09:30:26.423028+0000","last_clean":"2023-10-07T09:30:26.423028+0000","last_became_active":"2023-10-07T09:30:24.971526+0000","last_became_peered":"2023-10-07T09:30:24.971526+0000","last_unstale":"2023-10-07T09:30:26.423028+0000","last_undegraded":"2023-10-07T09:30:26.423028+0000","last_fullsized":"2023-10-07T09:30:26.423028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:00.376299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410968+0000","last_change":"2023-10-07T09:30:26.404652+0000","last_active":"2023-10-07T09:30:26.410968+0000","last_peered":"2023-10-07T09:30:26.410968+0000","last_clean":"2023-10-07T09:30:26.410968+0000","last_became_active":"2023-10-07T09:30:26.404410+0000","last_became_peered":"2023-10-07T09:30:26.404410+0000","last_unstale":"2023-10-07T09:30:26.410968+0000","last_undegraded":"2023-10-07T09:30:26.410968+0000","last_fullsized":"2023-10-07T09:30:26.410968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390466+0000","last_change":"2023-10-07T09:30:24.963621+0000","last_active":"2023-10-07T09:30:26.390466+0000","last_peered":"2023-10-07T09:30:26.390466+0000","last_clean":"2023-10-07T09:30:26.390466+0000","last_became_active":"2023-10-07T09:30:24.963315+0000","last_became_peered":"2023-10-07T09:30:24.963315+0000","last_unstale":"2023-10-07T09:30:26.390466+0000","last_undegraded":"2023-10-07T09:30:26.390466+0000","last_fullsized":"2023-10-07T09:30:26.390466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410981+0000","last_change":"2023-10-07T09:30:26.404627+0000","last_active":"2023-10-07T09:30:26.410981+0000","last_peered":"2023-10-07T09:30:26.410981+0000","last_clean":"2023-10-07T09:30:26.410981+0000","last_became_active":"2023-10-07T09:30:26.404362+0000","last_became_peered":"2023-10-07T09:30:26.404362+0000","last_unstale":"2023-10-07T09:30:26.410981+0000","last_undegraded":"2023-10-07T09:30:26.410981+0000","last_fullsized":"2023-10-07T09:30:26.410981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392082+0000","last_change":"2023-10-07T09:30:24.972623+0000","last_active":"2023-10-07T09:30:26.392082+0000","last_peered":"2023-10-07T09:30:26.392082+0000","last_clean":"2023-10-07T09:30:26.392082+0000","last_became_active":"2023-10-07T09:30:24.972377+0000","last_became_peered":"2023-10-07T09:30:24.972377+0000","last_unstale":"2023-10-07T09:30:26.392082+0000","last_undegraded":"2023-10-07T09:30:26.392082+0000","last_fullsized":"2023-10-07T09:30:26.392082+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:49.176868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409562+0000","last_change":"2023-10-07T09:30:26.398256+0000","last_active":"2023-10-07T09:30:26.409562+0000","last_peered":"2023-10-07T09:30:26.409562+0000","last_clean":"2023-10-07T09:30:26.409562+0000","last_became_active":"2023-10-07T09:30:26.397968+0000","last_became_peered":"2023-10-07T09:30:26.397968+0000","last_unstale":"2023-10-07T09:30:26.409562+0000","last_undegraded":"2023-10-07T09:30:26.409562+0000","last_fullsized":"2023-10-07T09:30:26.409562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390656+0000","last_change":"2023-10-07T09:30:24.968835+0000","last_active":"2023-10-07T09:30:26.390656+0000","last_peered":"2023-10-07T09:30:26.390656+0000","last_clean":"2023-10-07T09:30:26.390656+0000","last_became_active":"2023-10-07T09:30:24.968531+0000","last_became_peered":"2023-10-07T09:30:24.968531+0000","last_unstale":"2023-10-07T09:30:26.390656+0000","last_undegraded":"2023-10-07T09:30:26.390656+0000","last_fullsized":"2023-10-07T09:30:26.390656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:06:46.058261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418399+0000","last_change":"2023-10-07T09:30:26.413894+0000","last_active":"2023-10-07T09:30:26.418399+0000","last_peered":"2023-10-07T09:30:26.418399+0000","last_clean":"2023-10-07T09:30:26.418399+0000","last_became_active":"2023-10-07T09:30:26.413276+0000","last_became_peered":"2023-10-07T09:30:26.413276+0000","last_unstale":"2023-10-07T09:30:26.418399+0000","last_undegraded":"2023-10-07T09:30:26.418399+0000","last_fullsized":"2023-10-07T09:30:26.418399+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392045+0000","last_change":"2023-10-07T09:30:24.971933+0000","last_active":"2023-10-07T09:30:26.392045+0000","last_peered":"2023-10-07T09:30:26.392045+0000","last_clean":"2023-10-07T09:30:26.392045+0000","last_became_active":"2023-10-07T09:30:24.971478+0000","last_became_peered":"2023-10-07T09:30:24.971478+0000","last_unstale":"2023-10-07T09:30:26.392045+0000","last_undegraded":"2023-10-07T09:30:26.392045+0000","last_fullsized":"2023-10-07T09:30:26.392045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:19.116181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410100+0000","last_change":"2023-10-07T09:30:26.402235+0000","last_active":"2023-10-07T09:30:26.410100+0000","last_peered":"2023-10-07T09:30:26.410100+0000","last_clean":"2023-10-07T09:30:26.410100+0000","last_became_active":"2023-10-07T09:30:26.401915+0000","last_became_peered":"2023-10-07T09:30:26.401915+0000","last_unstale":"2023-10-07T09:30:26.410100+0000","last_undegraded":"2023-10-07T09:30:26.410100+0000","last_fullsized":"2023-10-07T09:30:26.410100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:01.010896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.394842+0000","last_change":"2023-10-07T09:30:24.967230+0000","last_active":"2023-10-07T09:30:26.394842+0000","last_peered":"2023-10-07T09:30:26.394842+0000","last_clean":"2023-10-07T09:30:26.394842+0000","last_became_active":"2023-10-07T09:30:24.966740+0000","last_became_peered":"2023-10-07T09:30:24.966740+0000","last_unstale":"2023-10-07T09:30:26.394842+0000","last_undegraded":"2023-10-07T09:30:26.394842+0000","last_fullsized":"2023-10-07T09:30:26.394842+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:31:56.775021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413060+0000","last_change":"2023-10-07T09:30:26.407227+0000","last_active":"2023-10-07T09:30:26.413060+0000","last_peered":"2023-10-07T09:30:26.413060+0000","last_clean":"2023-10-07T09:30:26.413060+0000","last_became_active":"2023-10-07T09:30:26.406801+0000","last_became_peered":"2023-10-07T09:30:26.406801+0000","last_unstale":"2023-10-07T09:30:26.413060+0000","last_undegraded":"2023-10-07T09:30:26.413060+0000","last_fullsized":"2023-10-07T09:30:26.413060+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:26.784262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384013+0000","last_change":"2023-10-07T09:30:24.967499+0000","last_active":"2023-10-07T09:30:26.384013+0000","last_peered":"2023-10-07T09:30:26.384013+0000","last_clean":"2023-10-07T09:30:26.384013+0000","last_became_active":"2023-10-07T09:30:24.967231+0000","last_became_peered":"2023-10-07T09:30:24.967231+0000","last_unstale":"2023-10-07T09:30:26.384013+0000","last_undegraded":"2023-10-07T09:30:26.384013+0000","last_fullsized":"2023-10-07T09:30:26.384013+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:23.529707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408925+0000","last_change":"2023-10-07T09:30:26.402511+0000","last_active":"2023-10-07T09:30:26.408925+0000","last_peered":"2023-10-07T09:30:26.408925+0000","last_clean":"2023-10-07T09:30:26.408925+0000","last_became_active":"2023-10-07T09:30:26.402144+0000","last_became_peered":"2023-10-07T09:30:26.402144+0000","last_unstale":"2023-10-07T09:30:26.408925+0000","last_undegraded":"2023-10-07T09:30:26.408925+0000","last_fullsized":"2023-10-07T09:30:26.408925+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:47:36.538323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.385241+0000","last_change":"2023-10-07T09:30:24.971597+0000","last_active":"2023-10-07T09:30:26.385241+0000","last_peered":"2023-10-07T09:30:26.385241+0000","last_clean":"2023-10-07T09:30:26.385241+0000","last_became_active":"2023-10-07T09:30:24.971341+0000","last_became_peered":"2023-10-07T09:30:24.971341+0000","last_unstale":"2023-10-07T09:30:26.385241+0000","last_undegraded":"2023-10-07T09:30:26.385241+0000","last_fullsized":"2023-10-07T09:30:26.385241+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:15.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.386477+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:30:26.386477+0000","last_peered":"2023-10-07T09:30:26.386477+0000","last_clean":"2023-10-07T09:30:26.386477+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:30:26.386477+0000","last_undegraded":"2023-10-07T09:30:26.386477+0000","last_fullsized":"2023-10-07T09:30:26.386477+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476756,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298264,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073576,"statfs":{"total":96636764160,"available":96331341824,"internally_reserved":0,"allocated":999424,"data_stored":748763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1,"5min":1,"15min":1},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.71099999999999997}]},{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.89200000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.78100000000000003}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.84799999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.64200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509460,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.984,"5min":2.984,"15min":2.984},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.65800000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.748},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":7.867,"5min":7.867,"15min":7.867},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.63900000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":8.0399999999999991},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81999999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.94399999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.31},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.755}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.69699999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302356,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069484,"statfs":{"total":96636764160,"available":96327151616,"internally_reserved":0,"allocated":995328,"data_stored":748673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.726,"5min":2.726,"15min":2.726},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.57399999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.92200000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.66700000000000004}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.8199999999999998,"5min":3.8199999999999998,"15min":3.8199999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.627},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":3.855,"5min":3.855,"15min":3.855},"last":1.036}]},{"osd":7,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":3.778,"5min":3.778,"15min":3.778},"last":1.097}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297684,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074156,"statfs":{"total":96636764160,"available":96331935744,"internally_reserved":0,"allocated":405504,"data_stored":158375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.439}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":1.0609999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.876},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.94299999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.91400000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":0.314},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.40000000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.498},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.42699999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.877}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.85299999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.88}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":1.2889999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.276}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":6.0620000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509460,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.77800000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.72599999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.61499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:30:35.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:30:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:35.999+0000 7fbcb65ff700 1 Processor -- start 2023-10-07T09:30:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:35.999+0000 7fbcb65ff700 1 -- start start 2023-10-07T09:30:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.000+0000 7fbcb65ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbcb00ad220 0x7fbcb0159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcb65ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb015af00 con 0x7fbcb0068620 2023-10-07T09:30:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcb65ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb00ada40 con 0x7fbcb00ad5f0 2023-10-07T09:30:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcb65ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb015a250 con 0x7fbcb00ad220 2023-10-07T09:30:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcb4b9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbcb0068620 0x7fbcb0068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41008/0 (socket says 172.21.15.29:41008) 2023-10-07T09:30:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcb4b9c700 1 -- 172.21.15.29:0/4201768956 learned_addr learned my addr 172.21.15.29:0/4201768956 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:36.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcaffff700 1 --2- 172.21.15.29:0/4201768956 >> v2:172.21.15.144:3300/0 conn(0x7fbcb00ad220 0x7fbcb0159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119853354 0 0) 0x7fbcb015af00 con 0x7fbcb0068620 2023-10-07T09:30:36.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.001+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc98003570 con 0x7fbcb0068620 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392492856 0 0) 0x7fbc98003570 con 0x7fbcb0068620 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 >> v2:172.21.15.144:3300/0 conn(0x7fbcb00ad220 msgr2=0x7fbcb0159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 --2- 172.21.15.29:0/4201768956 >> v2:172.21.15.144:3300/0 conn(0x7fbcb00ad220 0x7fbcb0159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 >> v1:172.21.15.144:6789/0 conn(0x7fbcb00ad5f0 legacy=0x7fbcb00ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb015a530 con 0x7fbcb0068620 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fbca80017f0 con 0x7fbcb0068620 2023-10-07T09:30:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 >> v1:172.21.15.29:6789/0 conn(0x7fbcb0068620 legacy=0x7fbcb0068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc98003a60 0x7fbc98003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc98004810 0x7fbc98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc980051c0 0x7fbc9800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb015a250 con 0x7fbc980051c0 2023-10-07T09:30:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9800c130 con 0x7fbc98004810 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9800c2f0 con 0x7fbc98003a60 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcb4b9c700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc980051c0 0x7fbc9800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaffff700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc98003a60 0x7fbc98003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.002+0000 7fbcaf7fe700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc98004810 0x7fbc98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 -- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc98003a60 msgr2=0x7fbc98003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc98003a60 0x7fbc98003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 -- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc98004810 msgr2=0x7fbc98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc98004810 0x7fbc98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 -- 172.21.15.29:0/4201768956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc98003570 con 0x7fbc980051c0 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 -- 172.21.15.29:0/4201768956 reap_dead start 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcb4b9c700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc980051c0 0x7fbc9800ba60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbca800a850 tx=0x7fbca8002120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62fb260f213eab2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca800a210 con 0x7fbc980051c0 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca80070f0 con 0x7fbc980051c0 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.003+0000 7fbcaeffd700 1 -- 172.21.15.29:0/4201768956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca80074b0 con 0x7fbc980051c0 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 -- 172.21.15.29:0/4201768956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc980051c0 msgr2=0x7fbc9800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc980051c0 0x7fbc9800ba60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbca800a850 tx=0x7fbca8002120 comp rx=0 tx=0).stop 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 -- 172.21.15.29:0/4201768956 shutdown_connections 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/4201768956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc980051c0 0x7fbc9800ba60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 -- 172.21.15.29:0/4201768956 >> 172.21.15.29:0/4201768956 conn(0x7fbcb006dd00 msgr2=0x7fbcb006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:36.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 -- 172.21.15.29:0/4201768956 shutdown_connections 2023-10-07T09:30:36.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.004+0000 7fbcb65ff700 1 -- 172.21.15.29:0/4201768956 wait complete. 2023-10-07T09:30:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.005+0000 7fbcb65ff700 1 Processor -- start 2023-10-07T09:30:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.005+0000 7fbcb65ff700 1 -- start start 2023-10-07T09:30:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.006+0000 7fbcb65ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980051c0 0x7fbcb0158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.006+0000 7fbcb65ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 0x7fbcb00c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.006+0000 7fbcb65ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcb0145f10 0x7fbcb00c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.006+0000 7fbcb65ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb0142c90 con 0x7fbcb0145aa0 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.006+0000 7fbcb65ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb0145610 con 0x7fbc980051c0 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.006+0000 7fbcb65ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb0144a00 con 0x7fbcb0145f10 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 0x7fbcb00c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 0x7fbcb00c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41352/0 (socket says 172.21.15.29:41352) 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 -- 172.21.15.29:0/3694111090 learned_addr learned my addr 172.21.15.29:0/3694111090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaffff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980051c0 0x7fbcb0158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcb4b9c700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcb0145f10 0x7fbcb00c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 -- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcb0145f10 msgr2=0x7fbcb00c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcb0145f10 0x7fbcb00c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 -- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980051c0 msgr2=0x7fbcb0158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980051c0 0x7fbcb0158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 -- 172.21.15.29:0/3694111090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb0144ce0 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcb4b9c700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcb0145f10 0x7fbcb00c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.007+0000 7fbcaf7fe700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 0x7fbcb00c94c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbca4002ab0 tx=0x7fbca4007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.008+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca400f9a0 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.008+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca400fb00 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.008+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb0144fc0 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.008+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca40133f0 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.008+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb0159380 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.009+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb0145780 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.010+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbca4013820 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.010+0000 7fbc97fff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc9c066f30 0x7fbc9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.010+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbca4084c70 con 0x7fbcb0145aa0 2023-10-07T09:30:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.011+0000 7fbcaffff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc9c066f30 0x7fbc9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.012+0000 7fbcaffff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc9c066f30 0x7fbc9c0693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbca0006ed0 tx=0x7fbca0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:36.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.014+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbca4051e40 con 0x7fbcb0145aa0 2023-10-07T09:30:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.218+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbcb004c3c0 con 0x7fbc9c066f30 2023-10-07T09:30:36.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.227+0000 7fbc97fff700 1 -- 172.21.15.29:0/3694111090 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288641 (secure 0 0 0) 0x7fbcb004c3c0 con 0x7fbc9c066f30 2023-10-07T09:30:36.232 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.231+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc9c066f30 msgr2=0x7fbc9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.231+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc9c066f30 0x7fbc9c0693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbca0006ed0 tx=0x7fbca0008040 comp rx=0 tx=0).stop 2023-10-07T09:30:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.231+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 msgr2=0x7fbcb00c94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.231+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 0x7fbcb00c94c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbca4002ab0 tx=0x7fbca4007d10 comp rx=0 tx=0).stop 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 shutdown_connections 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc9c066f30 0x7fbc9c0693e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcb0145f10 0x7fbcb00c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980051c0 0x7fbcb0158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 --2- 172.21.15.29:0/3694111090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcb0145aa0 0x7fbcb00c94c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 >> 172.21.15.29:0/3694111090 conn(0x7fbcb006dd00 msgr2=0x7fbcb006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 shutdown_connections 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.232+0000 7fbcb65ff700 1 -- 172.21.15.29:0/3694111090 wait complete. 2023-10-07T09:30:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:30:36.250 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2023-10-07T09:30:36.035576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2386760,"kb_used_data":4360,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752587960,"statfs":{"total":773094113280,"available":770650071040,"internally_reserved":0,"allocated":4464640,"data_stored":2451788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.653052"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412600+0000","last_change":"2023-10-07T09:30:26.410317+0000","last_active":"2023-10-07T09:30:26.412600+0000","last_peered":"2023-10-07T09:30:26.412600+0000","last_clean":"2023-10-07T09:30:26.412600+0000","last_became_active":"2023-10-07T09:30:26.409965+0000","last_became_peered":"2023-10-07T09:30:26.409965+0000","last_unstale":"2023-10-07T09:30:26.412600+0000","last_undegraded":"2023-10-07T09:30:26.412600+0000","last_fullsized":"2023-10-07T09:30:26.412600+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415669+0000","last_change":"2023-10-07T09:30:26.408750+0000","last_active":"2023-10-07T09:30:26.415669+0000","last_peered":"2023-10-07T09:30:26.415669+0000","last_clean":"2023-10-07T09:30:26.415669+0000","last_became_active":"2023-10-07T09:30:26.407823+0000","last_became_peered":"2023-10-07T09:30:26.407823+0000","last_unstale":"2023-10-07T09:30:26.415669+0000","last_undegraded":"2023-10-07T09:30:26.415669+0000","last_fullsized":"2023-10-07T09:30:26.415669+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:06:58.672867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414461+0000","last_change":"2023-10-07T09:30:26.408350+0000","last_active":"2023-10-07T09:30:26.414461+0000","last_peered":"2023-10-07T09:30:26.414461+0000","last_clean":"2023-10-07T09:30:26.414461+0000","last_became_active":"2023-10-07T09:30:26.408029+0000","last_became_peered":"2023-10-07T09:30:26.408029+0000","last_unstale":"2023-10-07T09:30:26.414461+0000","last_undegraded":"2023-10-07T09:30:26.414461+0000","last_fullsized":"2023-10-07T09:30:26.414461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407832+0000","last_change":"2023-10-07T09:30:26.403115+0000","last_active":"2023-10-07T09:30:26.407832+0000","last_peered":"2023-10-07T09:30:26.407832+0000","last_clean":"2023-10-07T09:30:26.407832+0000","last_became_active":"2023-10-07T09:30:26.402699+0000","last_became_peered":"2023-10-07T09:30:26.402699+0000","last_unstale":"2023-10-07T09:30:26.407832+0000","last_undegraded":"2023-10-07T09:30:26.407832+0000","last_fullsized":"2023-10-07T09:30:26.407832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:06:53.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414913+0000","last_change":"2023-10-07T09:30:26.410770+0000","last_active":"2023-10-07T09:30:26.414913+0000","last_peered":"2023-10-07T09:30:26.414913+0000","last_clean":"2023-10-07T09:30:26.414913+0000","last_became_active":"2023-10-07T09:30:26.410465+0000","last_became_peered":"2023-10-07T09:30:26.410465+0000","last_unstale":"2023-10-07T09:30:26.414913+0000","last_undegraded":"2023-10-07T09:30:26.414913+0000","last_fullsized":"2023-10-07T09:30:26.414913+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:09.027232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.403447+0000","last_change":"2023-10-07T09:30:26.395345+0000","last_active":"2023-10-07T09:30:26.403447+0000","last_peered":"2023-10-07T09:30:26.403447+0000","last_clean":"2023-10-07T09:30:26.403447+0000","last_became_active":"2023-10-07T09:30:26.395042+0000","last_became_peered":"2023-10-07T09:30:26.395042+0000","last_unstale":"2023-10-07T09:30:26.403447+0000","last_undegraded":"2023-10-07T09:30:26.403447+0000","last_fullsized":"2023-10-07T09:30:26.403447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409954+0000","last_change":"2023-10-07T09:30:26.390280+0000","last_active":"2023-10-07T09:30:26.409954+0000","last_peered":"2023-10-07T09:30:26.409954+0000","last_clean":"2023-10-07T09:30:26.409954+0000","last_became_active":"2023-10-07T09:30:26.390003+0000","last_became_peered":"2023-10-07T09:30:26.390003+0000","last_unstale":"2023-10-07T09:30:26.409954+0000","last_undegraded":"2023-10-07T09:30:26.409954+0000","last_fullsized":"2023-10-07T09:30:26.409954+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:36:52.517291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408954+0000","last_change":"2023-10-07T09:30:26.402274+0000","last_active":"2023-10-07T09:30:26.408954+0000","last_peered":"2023-10-07T09:30:26.408954+0000","last_clean":"2023-10-07T09:30:26.408954+0000","last_became_active":"2023-10-07T09:30:26.401927+0000","last_became_peered":"2023-10-07T09:30:26.401927+0000","last_unstale":"2023-10-07T09:30:26.408954+0000","last_undegraded":"2023-10-07T09:30:26.408954+0000","last_fullsized":"2023-10-07T09:30:26.408954+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:15:36.666517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.416845+0000","last_change":"2023-10-07T09:30:26.412014+0000","last_active":"2023-10-07T09:30:26.416845+0000","last_peered":"2023-10-07T09:30:26.416845+0000","last_clean":"2023-10-07T09:30:26.416845+0000","last_became_active":"2023-10-07T09:30:26.411754+0000","last_became_peered":"2023-10-07T09:30:26.411754+0000","last_unstale":"2023-10-07T09:30:26.416845+0000","last_undegraded":"2023-10-07T09:30:26.416845+0000","last_fullsized":"2023-10-07T09:30:26.416845+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:42.651535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.399392+0000","last_change":"2023-10-07T09:30:26.396209+0000","last_active":"2023-10-07T09:30:26.399392+0000","last_peered":"2023-10-07T09:30:26.399392+0000","last_clean":"2023-10-07T09:30:26.399392+0000","last_became_active":"2023-10-07T09:30:26.395573+0000","last_became_peered":"2023-10-07T09:30:26.395573+0000","last_unstale":"2023-10-07T09:30:26.399392+0000","last_undegraded":"2023-10-07T09:30:26.399392+0000","last_fullsized":"2023-10-07T09:30:26.399392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:54.021183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409352+0000","last_change":"2023-10-07T09:30:26.402704+0000","last_active":"2023-10-07T09:30:26.409352+0000","last_peered":"2023-10-07T09:30:26.409352+0000","last_clean":"2023-10-07T09:30:26.409352+0000","last_became_active":"2023-10-07T09:30:26.402452+0000","last_became_peered":"2023-10-07T09:30:26.402452+0000","last_unstale":"2023-10-07T09:30:26.409352+0000","last_undegraded":"2023-10-07T09:30:26.409352+0000","last_fullsized":"2023-10-07T09:30:26.409352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:31:05.985248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413491+0000","last_change":"2023-10-07T09:30:26.409670+0000","last_active":"2023-10-07T09:30:26.413491+0000","last_peered":"2023-10-07T09:30:26.413491+0000","last_clean":"2023-10-07T09:30:26.413491+0000","last_became_active":"2023-10-07T09:30:26.408687+0000","last_became_peered":"2023-10-07T09:30:26.408687+0000","last_unstale":"2023-10-07T09:30:26.413491+0000","last_undegraded":"2023-10-07T09:30:26.413491+0000","last_fullsized":"2023-10-07T09:30:26.413491+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413335+0000","last_change":"2023-10-07T09:30:26.406082+0000","last_active":"2023-10-07T09:30:26.413335+0000","last_peered":"2023-10-07T09:30:26.413335+0000","last_clean":"2023-10-07T09:30:26.413335+0000","last_became_active":"2023-10-07T09:30:26.405724+0000","last_became_peered":"2023-10-07T09:30:26.405724+0000","last_unstale":"2023-10-07T09:30:26.413335+0000","last_undegraded":"2023-10-07T09:30:26.413335+0000","last_fullsized":"2023-10-07T09:30:26.413335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411102+0000","last_change":"2023-10-07T09:30:26.403342+0000","last_active":"2023-10-07T09:30:26.411102+0000","last_peered":"2023-10-07T09:30:26.411102+0000","last_clean":"2023-10-07T09:30:26.411102+0000","last_became_active":"2023-10-07T09:30:26.403138+0000","last_became_peered":"2023-10-07T09:30:26.403138+0000","last_unstale":"2023-10-07T09:30:26.411102+0000","last_undegraded":"2023-10-07T09:30:26.411102+0000","last_fullsized":"2023-10-07T09:30:26.411102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408435+0000","last_change":"2023-10-07T09:30:26.395877+0000","last_active":"2023-10-07T09:30:26.408435+0000","last_peered":"2023-10-07T09:30:26.408435+0000","last_clean":"2023-10-07T09:30:26.408435+0000","last_became_active":"2023-10-07T09:30:26.395596+0000","last_became_peered":"2023-10-07T09:30:26.395596+0000","last_unstale":"2023-10-07T09:30:26.408435+0000","last_undegraded":"2023-10-07T09:30:26.408435+0000","last_fullsized":"2023-10-07T09:30:26.408435+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:20.489580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408761+0000","last_change":"2023-10-07T09:30:26.398142+0000","last_active":"2023-10-07T09:30:26.408761+0000","last_peered":"2023-10-07T09:30:26.408761+0000","last_clean":"2023-10-07T09:30:26.408761+0000","last_became_active":"2023-10-07T09:30:26.397863+0000","last_became_peered":"2023-10-07T09:30:26.397863+0000","last_unstale":"2023-10-07T09:30:26.408761+0000","last_undegraded":"2023-10-07T09:30:26.408761+0000","last_fullsized":"2023-10-07T09:30:26.408761+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:59.225017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408428+0000","last_change":"2023-10-07T09:30:26.394555+0000","last_active":"2023-10-07T09:30:26.408428+0000","last_peered":"2023-10-07T09:30:26.408428+0000","last_clean":"2023-10-07T09:30:26.408428+0000","last_became_active":"2023-10-07T09:30:26.393660+0000","last_became_peered":"2023-10-07T09:30:26.393660+0000","last_unstale":"2023-10-07T09:30:26.408428+0000","last_undegraded":"2023-10-07T09:30:26.408428+0000","last_fullsized":"2023-10-07T09:30:26.408428+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:46:14.675050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409109+0000","last_change":"2023-10-07T09:30:26.394655+0000","last_active":"2023-10-07T09:30:26.409109+0000","last_peered":"2023-10-07T09:30:26.409109+0000","last_clean":"2023-10-07T09:30:26.409109+0000","last_became_active":"2023-10-07T09:30:26.394372+0000","last_became_peered":"2023-10-07T09:30:26.394372+0000","last_unstale":"2023-10-07T09:30:26.409109+0000","last_undegraded":"2023-10-07T09:30:26.409109+0000","last_fullsized":"2023-10-07T09:30:26.409109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:35.770412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405481+0000","last_change":"2023-10-07T09:30:26.402289+0000","last_active":"2023-10-07T09:30:26.405481+0000","last_peered":"2023-10-07T09:30:26.405481+0000","last_clean":"2023-10-07T09:30:26.405481+0000","last_became_active":"2023-10-07T09:30:26.401826+0000","last_became_peered":"2023-10-07T09:30:26.401826+0000","last_unstale":"2023-10-07T09:30:26.405481+0000","last_undegraded":"2023-10-07T09:30:26.405481+0000","last_fullsized":"2023-10-07T09:30:26.405481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:46:15.007773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414990+0000","last_change":"2023-10-07T09:30:26.411654+0000","last_active":"2023-10-07T09:30:26.414990+0000","last_peered":"2023-10-07T09:30:26.414990+0000","last_clean":"2023-10-07T09:30:26.414990+0000","last_became_active":"2023-10-07T09:30:26.411407+0000","last_became_peered":"2023-10-07T09:30:26.411407+0000","last_unstale":"2023-10-07T09:30:26.414990+0000","last_undegraded":"2023-10-07T09:30:26.414990+0000","last_fullsized":"2023-10-07T09:30:26.414990+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:20.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414072+0000","last_change":"2023-10-07T09:30:26.405589+0000","last_active":"2023-10-07T09:30:26.414072+0000","last_peered":"2023-10-07T09:30:26.414072+0000","last_clean":"2023-10-07T09:30:26.414072+0000","last_became_active":"2023-10-07T09:30:26.405334+0000","last_became_peered":"2023-10-07T09:30:26.405334+0000","last_unstale":"2023-10-07T09:30:26.414072+0000","last_undegraded":"2023-10-07T09:30:26.414072+0000","last_fullsized":"2023-10-07T09:30:26.414072+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.411311+0000","last_change":"2023-10-07T09:30:26.404743+0000","last_active":"2023-10-07T09:30:26.411311+0000","last_peered":"2023-10-07T09:30:26.411311+0000","last_clean":"2023-10-07T09:30:26.411311+0000","last_became_active":"2023-10-07T09:30:26.404395+0000","last_became_peered":"2023-10-07T09:30:26.404395+0000","last_unstale":"2023-10-07T09:30:26.411311+0000","last_undegraded":"2023-10-07T09:30:26.411311+0000","last_fullsized":"2023-10-07T09:30:26.411311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410992+0000","last_change":"2023-10-07T09:30:26.403259+0000","last_active":"2023-10-07T09:30:26.410992+0000","last_peered":"2023-10-07T09:30:26.410992+0000","last_clean":"2023-10-07T09:30:26.410992+0000","last_became_active":"2023-10-07T09:30:26.402926+0000","last_became_peered":"2023-10-07T09:30:26.402926+0000","last_unstale":"2023-10-07T09:30:26.410992+0000","last_undegraded":"2023-10-07T09:30:26.410992+0000","last_fullsized":"2023-10-07T09:30:26.410992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413211+0000","last_change":"2023-10-07T09:30:26.409044+0000","last_active":"2023-10-07T09:30:26.413211+0000","last_peered":"2023-10-07T09:30:26.413211+0000","last_clean":"2023-10-07T09:30:26.413211+0000","last_became_active":"2023-10-07T09:30:26.408764+0000","last_became_peered":"2023-10-07T09:30:26.408764+0000","last_unstale":"2023-10-07T09:30:26.413211+0000","last_undegraded":"2023-10-07T09:30:26.413211+0000","last_fullsized":"2023-10-07T09:30:26.413211+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:47.675483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422810+0000","last_change":"2023-10-07T09:30:26.413627+0000","last_active":"2023-10-07T09:30:26.422810+0000","last_peered":"2023-10-07T09:30:26.422810+0000","last_clean":"2023-10-07T09:30:26.422810+0000","last_became_active":"2023-10-07T09:30:26.413371+0000","last_became_peered":"2023-10-07T09:30:26.413371+0000","last_unstale":"2023-10-07T09:30:26.422810+0000","last_undegraded":"2023-10-07T09:30:26.422810+0000","last_fullsized":"2023-10-07T09:30:26.422810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:34.133968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415691+0000","last_change":"2023-10-07T09:30:26.408999+0000","last_active":"2023-10-07T09:30:26.415691+0000","last_peered":"2023-10-07T09:30:26.415691+0000","last_clean":"2023-10-07T09:30:26.415691+0000","last_became_active":"2023-10-07T09:30:26.408201+0000","last_became_peered":"2023-10-07T09:30:26.408201+0000","last_unstale":"2023-10-07T09:30:26.415691+0000","last_undegraded":"2023-10-07T09:30:26.415691+0000","last_fullsized":"2023-10-07T09:30:26.415691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421410+0000","last_change":"2023-10-07T09:30:26.411198+0000","last_active":"2023-10-07T09:30:26.421410+0000","last_peered":"2023-10-07T09:30:26.421410+0000","last_clean":"2023-10-07T09:30:26.421410+0000","last_became_active":"2023-10-07T09:30:26.410735+0000","last_became_peered":"2023-10-07T09:30:26.410735+0000","last_unstale":"2023-10-07T09:30:26.421410+0000","last_undegraded":"2023-10-07T09:30:26.421410+0000","last_fullsized":"2023-10-07T09:30:26.421410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:24.740454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413473+0000","last_change":"2023-10-07T09:30:26.409340+0000","last_active":"2023-10-07T09:30:26.413473+0000","last_peered":"2023-10-07T09:30:26.413473+0000","last_clean":"2023-10-07T09:30:26.413473+0000","last_became_active":"2023-10-07T09:30:26.408986+0000","last_became_peered":"2023-10-07T09:30:26.408986+0000","last_unstale":"2023-10-07T09:30:26.413473+0000","last_undegraded":"2023-10-07T09:30:26.413473+0000","last_fullsized":"2023-10-07T09:30:26.413473+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:39.624255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413566+0000","last_change":"2023-10-07T09:30:26.404629+0000","last_active":"2023-10-07T09:30:26.413566+0000","last_peered":"2023-10-07T09:30:26.413566+0000","last_clean":"2023-10-07T09:30:26.413566+0000","last_became_active":"2023-10-07T09:30:26.404320+0000","last_became_peered":"2023-10-07T09:30:26.404320+0000","last_unstale":"2023-10-07T09:30:26.413566+0000","last_undegraded":"2023-10-07T09:30:26.413566+0000","last_fullsized":"2023-10-07T09:30:26.413566+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:13.869543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408330+0000","last_change":"2023-10-07T09:30:26.392465+0000","last_active":"2023-10-07T09:30:26.408330+0000","last_peered":"2023-10-07T09:30:26.408330+0000","last_clean":"2023-10-07T09:30:26.408330+0000","last_became_active":"2023-10-07T09:30:26.392140+0000","last_became_peered":"2023-10-07T09:30:26.392140+0000","last_unstale":"2023-10-07T09:30:26.408330+0000","last_undegraded":"2023-10-07T09:30:26.408330+0000","last_fullsized":"2023-10-07T09:30:26.408330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:34:46.460732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405565+0000","last_change":"2023-10-07T09:30:26.401922+0000","last_active":"2023-10-07T09:30:26.405565+0000","last_peered":"2023-10-07T09:30:26.405565+0000","last_clean":"2023-10-07T09:30:26.405565+0000","last_became_active":"2023-10-07T09:30:26.401633+0000","last_became_peered":"2023-10-07T09:30:26.401633+0000","last_unstale":"2023-10-07T09:30:26.405565+0000","last_undegraded":"2023-10-07T09:30:26.405565+0000","last_fullsized":"2023-10-07T09:30:26.405565+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408933+0000","last_change":"2023-10-07T09:30:26.401905+0000","last_active":"2023-10-07T09:30:26.408933+0000","last_peered":"2023-10-07T09:30:26.408933+0000","last_clean":"2023-10-07T09:30:26.408933+0000","last_became_active":"2023-10-07T09:30:26.401318+0000","last_became_peered":"2023-10-07T09:30:26.401318+0000","last_unstale":"2023-10-07T09:30:26.408933+0000","last_undegraded":"2023-10-07T09:30:26.408933+0000","last_fullsized":"2023-10-07T09:30:26.408933+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:38:16.579339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418432+0000","last_change":"2023-10-07T09:30:26.413698+0000","last_active":"2023-10-07T09:30:26.418432+0000","last_peered":"2023-10-07T09:30:26.418432+0000","last_clean":"2023-10-07T09:30:26.418432+0000","last_became_active":"2023-10-07T09:30:26.413219+0000","last_became_peered":"2023-10-07T09:30:26.413219+0000","last_unstale":"2023-10-07T09:30:26.418432+0000","last_undegraded":"2023-10-07T09:30:26.418432+0000","last_fullsized":"2023-10-07T09:30:26.418432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422004+0000","last_change":"2023-10-07T09:30:24.970001+0000","last_active":"2023-10-07T09:30:26.422004+0000","last_peered":"2023-10-07T09:30:26.422004+0000","last_clean":"2023-10-07T09:30:26.422004+0000","last_became_active":"2023-10-07T09:30:24.969692+0000","last_became_peered":"2023-10-07T09:30:24.969692+0000","last_unstale":"2023-10-07T09:30:26.422004+0000","last_undegraded":"2023-10-07T09:30:26.422004+0000","last_fullsized":"2023-10-07T09:30:26.422004+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405616+0000","last_change":"2023-10-07T09:30:26.402818+0000","last_active":"2023-10-07T09:30:26.405616+0000","last_peered":"2023-10-07T09:30:26.405616+0000","last_clean":"2023-10-07T09:30:26.405616+0000","last_became_active":"2023-10-07T09:30:26.402490+0000","last_became_peered":"2023-10-07T09:30:26.402490+0000","last_unstale":"2023-10-07T09:30:26.405616+0000","last_undegraded":"2023-10-07T09:30:26.405616+0000","last_fullsized":"2023-10-07T09:30:26.405616+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:13.869543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421779+0000","last_change":"2023-10-07T09:30:24.961003+0000","last_active":"2023-10-07T09:30:26.421779+0000","last_peered":"2023-10-07T09:30:26.421779+0000","last_clean":"2023-10-07T09:30:26.421779+0000","last_became_active":"2023-10-07T09:30:24.960711+0000","last_became_peered":"2023-10-07T09:30:24.960711+0000","last_unstale":"2023-10-07T09:30:26.421779+0000","last_undegraded":"2023-10-07T09:30:26.421779+0000","last_fullsized":"2023-10-07T09:30:26.421779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:25:06.545101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.416884+0000","last_change":"2023-10-07T09:30:26.413169+0000","last_active":"2023-10-07T09:30:26.416884+0000","last_peered":"2023-10-07T09:30:26.416884+0000","last_clean":"2023-10-07T09:30:26.416884+0000","last_became_active":"2023-10-07T09:30:26.412855+0000","last_became_peered":"2023-10-07T09:30:26.412855+0000","last_unstale":"2023-10-07T09:30:26.416884+0000","last_undegraded":"2023-10-07T09:30:26.416884+0000","last_fullsized":"2023-10-07T09:30:26.416884+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:24.532886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424146+0000","last_change":"2023-10-07T09:30:24.967737+0000","last_active":"2023-10-07T09:30:26.424146+0000","last_peered":"2023-10-07T09:30:26.424146+0000","last_clean":"2023-10-07T09:30:26.424146+0000","last_became_active":"2023-10-07T09:30:24.966888+0000","last_became_peered":"2023-10-07T09:30:24.966888+0000","last_unstale":"2023-10-07T09:30:26.424146+0000","last_undegraded":"2023-10-07T09:30:26.424146+0000","last_fullsized":"2023-10-07T09:30:26.424146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421389+0000","last_change":"2023-10-07T09:30:26.411013+0000","last_active":"2023-10-07T09:30:26.421389+0000","last_peered":"2023-10-07T09:30:26.421389+0000","last_clean":"2023-10-07T09:30:26.421389+0000","last_became_active":"2023-10-07T09:30:26.410413+0000","last_became_peered":"2023-10-07T09:30:26.410413+0000","last_unstale":"2023-10-07T09:30:26.421389+0000","last_undegraded":"2023-10-07T09:30:26.421389+0000","last_fullsized":"2023-10-07T09:30:26.421389+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:06.340913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423552+0000","last_change":"2023-10-07T09:30:24.966147+0000","last_active":"2023-10-07T09:30:26.423552+0000","last_peered":"2023-10-07T09:30:26.423552+0000","last_clean":"2023-10-07T09:30:26.423552+0000","last_became_active":"2023-10-07T09:30:24.965853+0000","last_became_peered":"2023-10-07T09:30:24.965853+0000","last_unstale":"2023-10-07T09:30:26.423552+0000","last_undegraded":"2023-10-07T09:30:26.423552+0000","last_fullsized":"2023-10-07T09:30:26.423552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409239+0000","last_change":"2023-10-07T09:30:26.402741+0000","last_active":"2023-10-07T09:30:26.409239+0000","last_peered":"2023-10-07T09:30:26.409239+0000","last_clean":"2023-10-07T09:30:26.409239+0000","last_became_active":"2023-10-07T09:30:26.402366+0000","last_became_peered":"2023-10-07T09:30:26.402366+0000","last_unstale":"2023-10-07T09:30:26.409239+0000","last_undegraded":"2023-10-07T09:30:26.409239+0000","last_fullsized":"2023-10-07T09:30:26.409239+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:56:23.358835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421752+0000","last_change":"2023-10-07T09:30:24.967094+0000","last_active":"2023-10-07T09:30:26.421752+0000","last_peered":"2023-10-07T09:30:26.421752+0000","last_clean":"2023-10-07T09:30:26.421752+0000","last_became_active":"2023-10-07T09:30:24.966810+0000","last_became_peered":"2023-10-07T09:30:24.966810+0000","last_unstale":"2023-10-07T09:30:26.421752+0000","last_undegraded":"2023-10-07T09:30:26.421752+0000","last_fullsized":"2023-10-07T09:30:26.421752+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:24:57.676848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414299+0000","last_change":"2023-10-07T09:30:26.407749+0000","last_active":"2023-10-07T09:30:26.414299+0000","last_peered":"2023-10-07T09:30:26.414299+0000","last_clean":"2023-10-07T09:30:26.414299+0000","last_became_active":"2023-10-07T09:30:26.407433+0000","last_became_peered":"2023-10-07T09:30:26.407433+0000","last_unstale":"2023-10-07T09:30:26.414299+0000","last_undegraded":"2023-10-07T09:30:26.414299+0000","last_fullsized":"2023-10-07T09:30:26.414299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:52.164831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421806+0000","last_change":"2023-10-07T09:30:24.963628+0000","last_active":"2023-10-07T09:30:26.421806+0000","last_peered":"2023-10-07T09:30:26.421806+0000","last_clean":"2023-10-07T09:30:26.421806+0000","last_became_active":"2023-10-07T09:30:24.963306+0000","last_became_peered":"2023-10-07T09:30:24.963306+0000","last_unstale":"2023-10-07T09:30:26.421806+0000","last_undegraded":"2023-10-07T09:30:26.421806+0000","last_fullsized":"2023-10-07T09:30:26.421806+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:12.068751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413311+0000","last_change":"2023-10-07T09:30:26.406943+0000","last_active":"2023-10-07T09:30:26.413311+0000","last_peered":"2023-10-07T09:30:26.413311+0000","last_clean":"2023-10-07T09:30:26.413311+0000","last_became_active":"2023-10-07T09:30:26.406650+0000","last_became_peered":"2023-10-07T09:30:26.406650+0000","last_unstale":"2023-10-07T09:30:26.413311+0000","last_undegraded":"2023-10-07T09:30:26.413311+0000","last_fullsized":"2023-10-07T09:30:26.413311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.383857+0000","last_change":"2023-10-07T09:30:24.965572+0000","last_active":"2023-10-07T09:30:26.383857+0000","last_peered":"2023-10-07T09:30:26.383857+0000","last_clean":"2023-10-07T09:30:26.383857+0000","last_became_active":"2023-10-07T09:30:24.965320+0000","last_became_peered":"2023-10-07T09:30:24.965320+0000","last_unstale":"2023-10-07T09:30:26.383857+0000","last_undegraded":"2023-10-07T09:30:26.383857+0000","last_fullsized":"2023-10-07T09:30:26.383857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:49.978481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414174+0000","last_change":"2023-10-07T09:30:26.409216+0000","last_active":"2023-10-07T09:30:26.414174+0000","last_peered":"2023-10-07T09:30:26.414174+0000","last_clean":"2023-10-07T09:30:26.414174+0000","last_became_active":"2023-10-07T09:30:26.408947+0000","last_became_peered":"2023-10-07T09:30:26.408947+0000","last_unstale":"2023-10-07T09:30:26.414174+0000","last_undegraded":"2023-10-07T09:30:26.414174+0000","last_fullsized":"2023-10-07T09:30:26.414174+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.394513+0000","last_change":"2023-10-07T09:30:24.962784+0000","last_active":"2023-10-07T09:30:26.394513+0000","last_peered":"2023-10-07T09:30:26.394513+0000","last_clean":"2023-10-07T09:30:26.394513+0000","last_became_active":"2023-10-07T09:30:24.962409+0000","last_became_peered":"2023-10-07T09:30:24.962409+0000","last_unstale":"2023-10-07T09:30:26.394513+0000","last_undegraded":"2023-10-07T09:30:26.394513+0000","last_fullsized":"2023-10-07T09:30:26.394513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:04.344607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413097+0000","last_change":"2023-10-07T09:30:26.406965+0000","last_active":"2023-10-07T09:30:26.413097+0000","last_peered":"2023-10-07T09:30:26.413097+0000","last_clean":"2023-10-07T09:30:26.413097+0000","last_became_active":"2023-10-07T09:30:26.406607+0000","last_became_peered":"2023-10-07T09:30:26.406607+0000","last_unstale":"2023-10-07T09:30:26.413097+0000","last_undegraded":"2023-10-07T09:30:26.413097+0000","last_fullsized":"2023-10-07T09:30:26.413097+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384959+0000","last_change":"2023-10-07T09:30:24.967714+0000","last_active":"2023-10-07T09:30:26.384959+0000","last_peered":"2023-10-07T09:30:26.384959+0000","last_clean":"2023-10-07T09:30:26.384959+0000","last_became_active":"2023-10-07T09:30:24.967416+0000","last_became_peered":"2023-10-07T09:30:24.967416+0000","last_unstale":"2023-10-07T09:30:26.384959+0000","last_undegraded":"2023-10-07T09:30:26.384959+0000","last_fullsized":"2023-10-07T09:30:26.384959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:25.781083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402385+0000","last_change":"2023-10-07T09:30:26.395987+0000","last_active":"2023-10-07T09:30:26.402385+0000","last_peered":"2023-10-07T09:30:26.402385+0000","last_clean":"2023-10-07T09:30:26.402385+0000","last_became_active":"2023-10-07T09:30:26.395709+0000","last_became_peered":"2023-10-07T09:30:26.395709+0000","last_unstale":"2023-10-07T09:30:26.402385+0000","last_undegraded":"2023-10-07T09:30:26.402385+0000","last_fullsized":"2023-10-07T09:30:26.402385+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:30.920124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390349+0000","last_change":"2023-10-07T09:30:24.962930+0000","last_active":"2023-10-07T09:30:26.390349+0000","last_peered":"2023-10-07T09:30:26.390349+0000","last_clean":"2023-10-07T09:30:26.390349+0000","last_became_active":"2023-10-07T09:30:24.962586+0000","last_became_peered":"2023-10-07T09:30:24.962586+0000","last_unstale":"2023-10-07T09:30:26.390349+0000","last_undegraded":"2023-10-07T09:30:26.390349+0000","last_fullsized":"2023-10-07T09:30:26.390349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:02.691657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.417232+0000","last_change":"2023-10-07T09:30:26.412395+0000","last_active":"2023-10-07T09:30:26.417232+0000","last_peered":"2023-10-07T09:30:26.417232+0000","last_clean":"2023-10-07T09:30:26.417232+0000","last_became_active":"2023-10-07T09:30:26.412085+0000","last_became_peered":"2023-10-07T09:30:26.412085+0000","last_unstale":"2023-10-07T09:30:26.417232+0000","last_undegraded":"2023-10-07T09:30:26.417232+0000","last_fullsized":"2023-10-07T09:30:26.417232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421318+0000","last_change":"2023-10-07T09:30:24.967809+0000","last_active":"2023-10-07T09:30:26.421318+0000","last_peered":"2023-10-07T09:30:26.421318+0000","last_clean":"2023-10-07T09:30:26.421318+0000","last_became_active":"2023-10-07T09:30:24.967550+0000","last_became_peered":"2023-10-07T09:30:24.967550+0000","last_unstale":"2023-10-07T09:30:26.421318+0000","last_undegraded":"2023-10-07T09:30:26.421318+0000","last_fullsized":"2023-10-07T09:30:26.421318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:43.746945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414443+0000","last_change":"2023-10-07T09:30:26.408025+0000","last_active":"2023-10-07T09:30:26.414443+0000","last_peered":"2023-10-07T09:30:26.414443+0000","last_clean":"2023-10-07T09:30:26.414443+0000","last_became_active":"2023-10-07T09:30:26.407736+0000","last_became_peered":"2023-10-07T09:30:26.407736+0000","last_unstale":"2023-10-07T09:30:26.414443+0000","last_undegraded":"2023-10-07T09:30:26.414443+0000","last_fullsized":"2023-10-07T09:30:26.414443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:21.654894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.425863+0000","last_change":"2023-10-07T09:30:24.966563+0000","last_active":"2023-10-07T09:30:26.425863+0000","last_peered":"2023-10-07T09:30:26.425863+0000","last_clean":"2023-10-07T09:30:26.425863+0000","last_became_active":"2023-10-07T09:30:24.966273+0000","last_became_peered":"2023-10-07T09:30:24.966273+0000","last_unstale":"2023-10-07T09:30:26.425863+0000","last_undegraded":"2023-10-07T09:30:26.425863+0000","last_fullsized":"2023-10-07T09:30:26.425863+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:41.648356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421340+0000","last_change":"2023-10-07T09:30:26.409745+0000","last_active":"2023-10-07T09:30:26.421340+0000","last_peered":"2023-10-07T09:30:26.421340+0000","last_clean":"2023-10-07T09:30:26.421340+0000","last_became_active":"2023-10-07T09:30:26.409489+0000","last_became_peered":"2023-10-07T09:30:26.409489+0000","last_unstale":"2023-10-07T09:30:26.421340+0000","last_undegraded":"2023-10-07T09:30:26.421340+0000","last_fullsized":"2023-10-07T09:30:26.421340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:56:40.081357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421862+0000","last_change":"2023-10-07T09:30:24.964382+0000","last_active":"2023-10-07T09:30:26.421862+0000","last_peered":"2023-10-07T09:30:26.421862+0000","last_clean":"2023-10-07T09:30:26.421862+0000","last_became_active":"2023-10-07T09:30:24.964076+0000","last_became_peered":"2023-10-07T09:30:24.964076+0000","last_unstale":"2023-10-07T09:30:26.421862+0000","last_undegraded":"2023-10-07T09:30:26.421862+0000","last_fullsized":"2023-10-07T09:30:26.421862+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:19:17.590704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.402685+0000","last_change":"2023-10-07T09:30:26.396545+0000","last_active":"2023-10-07T09:30:26.402685+0000","last_peered":"2023-10-07T09:30:26.402685+0000","last_clean":"2023-10-07T09:30:26.402685+0000","last_became_active":"2023-10-07T09:30:26.395816+0000","last_became_peered":"2023-10-07T09:30:26.395816+0000","last_unstale":"2023-10-07T09:30:26.402685+0000","last_undegraded":"2023-10-07T09:30:26.402685+0000","last_fullsized":"2023-10-07T09:30:26.402685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:30.042953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.391928+0000","last_change":"2023-10-07T09:30:24.964522+0000","last_active":"2023-10-07T09:30:26.391928+0000","last_peered":"2023-10-07T09:30:26.391928+0000","last_clean":"2023-10-07T09:30:26.391928+0000","last_became_active":"2023-10-07T09:30:24.964221+0000","last_became_peered":"2023-10-07T09:30:24.964221+0000","last_unstale":"2023-10-07T09:30:26.391928+0000","last_undegraded":"2023-10-07T09:30:26.391928+0000","last_fullsized":"2023-10-07T09:30:26.391928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:35.312512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408972+0000","last_change":"2023-10-07T09:30:26.402134+0000","last_active":"2023-10-07T09:30:26.408972+0000","last_peered":"2023-10-07T09:30:26.408972+0000","last_clean":"2023-10-07T09:30:26.408972+0000","last_became_active":"2023-10-07T09:30:26.401552+0000","last_became_peered":"2023-10-07T09:30:26.401552+0000","last_unstale":"2023-10-07T09:30:26.408972+0000","last_undegraded":"2023-10-07T09:30:26.408972+0000","last_fullsized":"2023-10-07T09:30:26.408972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:08:44.978453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384014+0000","last_change":"2023-10-07T09:30:24.967550+0000","last_active":"2023-10-07T09:30:26.384014+0000","last_peered":"2023-10-07T09:30:26.384014+0000","last_clean":"2023-10-07T09:30:26.384014+0000","last_became_active":"2023-10-07T09:30:24.967265+0000","last_became_peered":"2023-10-07T09:30:24.967265+0000","last_unstale":"2023-10-07T09:30:26.384014+0000","last_undegraded":"2023-10-07T09:30:26.384014+0000","last_fullsized":"2023-10-07T09:30:26.384014+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:12.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.405587+0000","last_change":"2023-10-07T09:30:26.402637+0000","last_active":"2023-10-07T09:30:26.405587+0000","last_peered":"2023-10-07T09:30:26.405587+0000","last_clean":"2023-10-07T09:30:26.405587+0000","last_became_active":"2023-10-07T09:30:26.402364+0000","last_became_peered":"2023-10-07T09:30:26.402364+0000","last_unstale":"2023-10-07T09:30:26.405587+0000","last_undegraded":"2023-10-07T09:30:26.405587+0000","last_fullsized":"2023-10-07T09:30:26.405587+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.424346+0000","last_change":"2023-10-07T09:30:24.971275+0000","last_active":"2023-10-07T09:30:26.424346+0000","last_peered":"2023-10-07T09:30:26.424346+0000","last_clean":"2023-10-07T09:30:26.424346+0000","last_became_active":"2023-10-07T09:30:24.970929+0000","last_became_peered":"2023-10-07T09:30:24.970929+0000","last_unstale":"2023-10-07T09:30:26.424346+0000","last_undegraded":"2023-10-07T09:30:26.424346+0000","last_fullsized":"2023-10-07T09:30:26.424346+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421090+0000","last_change":"2023-10-07T09:30:26.415671+0000","last_active":"2023-10-07T09:30:26.421090+0000","last_peered":"2023-10-07T09:30:26.421090+0000","last_clean":"2023-10-07T09:30:26.421090+0000","last_became_active":"2023-10-07T09:30:26.415360+0000","last_became_peered":"2023-10-07T09:30:26.415360+0000","last_unstale":"2023-10-07T09:30:26.421090+0000","last_undegraded":"2023-10-07T09:30:26.421090+0000","last_fullsized":"2023-10-07T09:30:26.421090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:44.750124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423122+0000","last_change":"2023-10-07T09:30:24.968081+0000","last_active":"2023-10-07T09:30:26.423122+0000","last_peered":"2023-10-07T09:30:26.423122+0000","last_clean":"2023-10-07T09:30:26.423122+0000","last_became_active":"2023-10-07T09:30:24.967047+0000","last_became_peered":"2023-10-07T09:30:24.967047+0000","last_unstale":"2023-10-07T09:30:26.423122+0000","last_undegraded":"2023-10-07T09:30:26.423122+0000","last_fullsized":"2023-10-07T09:30:26.423122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:00.381039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413417+0000","last_change":"2023-10-07T09:30:26.407336+0000","last_active":"2023-10-07T09:30:26.413417+0000","last_peered":"2023-10-07T09:30:26.413417+0000","last_clean":"2023-10-07T09:30:26.413417+0000","last_became_active":"2023-10-07T09:30:26.407027+0000","last_became_peered":"2023-10-07T09:30:26.407027+0000","last_unstale":"2023-10-07T09:30:26.413417+0000","last_undegraded":"2023-10-07T09:30:26.413417+0000","last_fullsized":"2023-10-07T09:30:26.413417+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.422008+0000","last_change":"2023-10-07T09:30:24.962904+0000","last_active":"2023-10-07T09:30:26.422008+0000","last_peered":"2023-10-07T09:30:26.422008+0000","last_clean":"2023-10-07T09:30:26.422008+0000","last_became_active":"2023-10-07T09:30:24.962519+0000","last_became_peered":"2023-10-07T09:30:24.962519+0000","last_unstale":"2023-10-07T09:30:26.422008+0000","last_undegraded":"2023-10-07T09:30:26.422008+0000","last_fullsized":"2023-10-07T09:30:26.422008+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:00.363653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.407852+0000","last_change":"2023-10-07T09:30:26.403114+0000","last_active":"2023-10-07T09:30:26.407852+0000","last_peered":"2023-10-07T09:30:26.407852+0000","last_clean":"2023-10-07T09:30:26.407852+0000","last_became_active":"2023-10-07T09:30:26.402830+0000","last_became_peered":"2023-10-07T09:30:26.402830+0000","last_unstale":"2023-10-07T09:30:26.407852+0000","last_undegraded":"2023-10-07T09:30:26.407852+0000","last_fullsized":"2023-10-07T09:30:26.407852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:06.343485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.421820+0000","last_change":"2023-10-07T09:30:24.961853+0000","last_active":"2023-10-07T09:30:26.421820+0000","last_peered":"2023-10-07T09:30:26.421820+0000","last_clean":"2023-10-07T09:30:26.421820+0000","last_became_active":"2023-10-07T09:30:24.961591+0000","last_became_peered":"2023-10-07T09:30:24.961591+0000","last_unstale":"2023-10-07T09:30:26.421820+0000","last_undegraded":"2023-10-07T09:30:26.421820+0000","last_fullsized":"2023-10-07T09:30:26.421820+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:20:53.217133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.415642+0000","last_change":"2023-10-07T09:30:26.413595+0000","last_active":"2023-10-07T09:30:26.415642+0000","last_peered":"2023-10-07T09:30:26.415642+0000","last_clean":"2023-10-07T09:30:26.415642+0000","last_became_active":"2023-10-07T09:30:26.413270+0000","last_became_peered":"2023-10-07T09:30:26.413270+0000","last_unstale":"2023-10-07T09:30:26.415642+0000","last_undegraded":"2023-10-07T09:30:26.415642+0000","last_fullsized":"2023-10-07T09:30:26.415642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:50.180047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392150+0000","last_change":"2023-10-07T09:30:24.967115+0000","last_active":"2023-10-07T09:30:26.392150+0000","last_peered":"2023-10-07T09:30:26.392150+0000","last_clean":"2023-10-07T09:30:26.392150+0000","last_became_active":"2023-10-07T09:30:24.966832+0000","last_became_peered":"2023-10-07T09:30:24.966832+0000","last_unstale":"2023-10-07T09:30:26.392150+0000","last_undegraded":"2023-10-07T09:30:26.392150+0000","last_fullsized":"2023-10-07T09:30:26.392150+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:11:11.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413302+0000","last_change":"2023-10-07T09:30:26.408606+0000","last_active":"2023-10-07T09:30:26.413302+0000","last_peered":"2023-10-07T09:30:26.413302+0000","last_clean":"2023-10-07T09:30:26.413302+0000","last_became_active":"2023-10-07T09:30:26.408343+0000","last_became_peered":"2023-10-07T09:30:26.408343+0000","last_unstale":"2023-10-07T09:30:26.413302+0000","last_undegraded":"2023-10-07T09:30:26.413302+0000","last_fullsized":"2023-10-07T09:30:26.413302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:36.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423847+0000","last_change":"2023-10-07T09:30:24.963726+0000","last_active":"2023-10-07T09:30:26.423847+0000","last_peered":"2023-10-07T09:30:26.423847+0000","last_clean":"2023-10-07T09:30:26.423847+0000","last_became_active":"2023-10-07T09:30:24.963442+0000","last_became_peered":"2023-10-07T09:30:24.963442+0000","last_unstale":"2023-10-07T09:30:26.423847+0000","last_undegraded":"2023-10-07T09:30:26.423847+0000","last_fullsized":"2023-10-07T09:30:26.423847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:38.051902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413355+0000","last_change":"2023-10-07T09:30:26.406759+0000","last_active":"2023-10-07T09:30:26.413355+0000","last_peered":"2023-10-07T09:30:26.413355+0000","last_clean":"2023-10-07T09:30:26.413355+0000","last_became_active":"2023-10-07T09:30:26.406449+0000","last_became_peered":"2023-10-07T09:30:26.406449+0000","last_unstale":"2023-10-07T09:30:26.413355+0000","last_undegraded":"2023-10-07T09:30:26.413355+0000","last_fullsized":"2023-10-07T09:30:26.413355+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:01.379478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.385246+0000","last_change":"2023-10-07T09:30:24.970510+0000","last_active":"2023-10-07T09:30:26.385246+0000","last_peered":"2023-10-07T09:30:26.385246+0000","last_clean":"2023-10-07T09:30:26.385246+0000","last_became_active":"2023-10-07T09:30:24.970183+0000","last_became_peered":"2023-10-07T09:30:24.970183+0000","last_unstale":"2023-10-07T09:30:26.385246+0000","last_undegraded":"2023-10-07T09:30:26.385246+0000","last_fullsized":"2023-10-07T09:30:26.385246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.412638+0000","last_change":"2023-10-07T09:30:26.404546+0000","last_active":"2023-10-07T09:30:26.412638+0000","last_peered":"2023-10-07T09:30:26.412638+0000","last_clean":"2023-10-07T09:30:26.412638+0000","last_became_active":"2023-10-07T09:30:26.404284+0000","last_became_peered":"2023-10-07T09:30:26.404284+0000","last_unstale":"2023-10-07T09:30:26.412638+0000","last_undegraded":"2023-10-07T09:30:26.412638+0000","last_fullsized":"2023-10-07T09:30:26.412638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:39.055081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.394753+0000","last_change":"2023-10-07T09:30:24.965098+0000","last_active":"2023-10-07T09:30:26.394753+0000","last_peered":"2023-10-07T09:30:26.394753+0000","last_clean":"2023-10-07T09:30:26.394753+0000","last_became_active":"2023-10-07T09:30:24.964594+0000","last_became_peered":"2023-10-07T09:30:24.964594+0000","last_unstale":"2023-10-07T09:30:26.394753+0000","last_undegraded":"2023-10-07T09:30:26.394753+0000","last_fullsized":"2023-10-07T09:30:26.394753+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:56.143330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409978+0000","last_change":"2023-10-07T09:30:26.393757+0000","last_active":"2023-10-07T09:30:26.409978+0000","last_peered":"2023-10-07T09:30:26.409978+0000","last_clean":"2023-10-07T09:30:26.409978+0000","last_became_active":"2023-10-07T09:30:26.393474+0000","last_became_peered":"2023-10-07T09:30:26.393474+0000","last_unstale":"2023-10-07T09:30:26.409978+0000","last_undegraded":"2023-10-07T09:30:26.409978+0000","last_fullsized":"2023-10-07T09:30:26.409978+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:06.050510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.393481+0000","last_change":"2023-10-07T09:30:24.967401+0000","last_active":"2023-10-07T09:30:26.393481+0000","last_peered":"2023-10-07T09:30:26.393481+0000","last_clean":"2023-10-07T09:30:26.393481+0000","last_became_active":"2023-10-07T09:30:24.967138+0000","last_became_peered":"2023-10-07T09:30:24.967138+0000","last_unstale":"2023-10-07T09:30:26.393481+0000","last_undegraded":"2023-10-07T09:30:26.393481+0000","last_fullsized":"2023-10-07T09:30:26.393481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:54.102941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.414869+0000","last_change":"2023-10-07T09:30:26.411476+0000","last_active":"2023-10-07T09:30:26.414869+0000","last_peered":"2023-10-07T09:30:26.414869+0000","last_clean":"2023-10-07T09:30:26.414869+0000","last_became_active":"2023-10-07T09:30:26.411198+0000","last_became_peered":"2023-10-07T09:30:26.411198+0000","last_unstale":"2023-10-07T09:30:26.414869+0000","last_undegraded":"2023-10-07T09:30:26.414869+0000","last_fullsized":"2023-10-07T09:30:26.414869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.423028+0000","last_change":"2023-10-07T09:30:24.971777+0000","last_active":"2023-10-07T09:30:26.423028+0000","last_peered":"2023-10-07T09:30:26.423028+0000","last_clean":"2023-10-07T09:30:26.423028+0000","last_became_active":"2023-10-07T09:30:24.971526+0000","last_became_peered":"2023-10-07T09:30:24.971526+0000","last_unstale":"2023-10-07T09:30:26.423028+0000","last_undegraded":"2023-10-07T09:30:26.423028+0000","last_fullsized":"2023-10-07T09:30:26.423028+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:30:00.376299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410968+0000","last_change":"2023-10-07T09:30:26.404652+0000","last_active":"2023-10-07T09:30:26.410968+0000","last_peered":"2023-10-07T09:30:26.410968+0000","last_clean":"2023-10-07T09:30:26.410968+0000","last_became_active":"2023-10-07T09:30:26.404410+0000","last_became_peered":"2023-10-07T09:30:26.404410+0000","last_unstale":"2023-10-07T09:30:26.410968+0000","last_undegraded":"2023-10-07T09:30:26.410968+0000","last_fullsized":"2023-10-07T09:30:26.410968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.232972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"36'1","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-07T09:30:28.424850+0000","last_change":"2023-10-07T09:30:24.963621+0000","last_active":"2023-10-07T09:30:28.424850+0000","last_peered":"2023-10-07T09:30:28.424850+0000","last_clean":"2023-10-07T09:30:28.424850+0000","last_became_active":"2023-10-07T09:30:24.963315+0000","last_became_peered":"2023-10-07T09:30:24.963315+0000","last_unstale":"2023-10-07T09:30:28.424850+0000","last_undegraded":"2023-10-07T09:30:28.424850+0000","last_fullsized":"2023-10-07T09:30:28.424850+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:58:25.101698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410981+0000","last_change":"2023-10-07T09:30:26.404627+0000","last_active":"2023-10-07T09:30:26.410981+0000","last_peered":"2023-10-07T09:30:26.410981+0000","last_clean":"2023-10-07T09:30:26.410981+0000","last_became_active":"2023-10-07T09:30:26.404362+0000","last_became_peered":"2023-10-07T09:30:26.404362+0000","last_unstale":"2023-10-07T09:30:26.410981+0000","last_undegraded":"2023-10-07T09:30:26.410981+0000","last_fullsized":"2023-10-07T09:30:26.410981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:04:01.384218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392082+0000","last_change":"2023-10-07T09:30:24.972623+0000","last_active":"2023-10-07T09:30:26.392082+0000","last_peered":"2023-10-07T09:30:26.392082+0000","last_clean":"2023-10-07T09:30:26.392082+0000","last_became_active":"2023-10-07T09:30:24.972377+0000","last_became_peered":"2023-10-07T09:30:24.972377+0000","last_unstale":"2023-10-07T09:30:26.392082+0000","last_undegraded":"2023-10-07T09:30:26.392082+0000","last_fullsized":"2023-10-07T09:30:26.392082+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:49.176868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.409562+0000","last_change":"2023-10-07T09:30:26.398256+0000","last_active":"2023-10-07T09:30:26.409562+0000","last_peered":"2023-10-07T09:30:26.409562+0000","last_clean":"2023-10-07T09:30:26.409562+0000","last_became_active":"2023-10-07T09:30:26.397968+0000","last_became_peered":"2023-10-07T09:30:26.397968+0000","last_unstale":"2023-10-07T09:30:26.409562+0000","last_undegraded":"2023-10-07T09:30:26.409562+0000","last_fullsized":"2023-10-07T09:30:26.409562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.390656+0000","last_change":"2023-10-07T09:30:24.968835+0000","last_active":"2023-10-07T09:30:26.390656+0000","last_peered":"2023-10-07T09:30:26.390656+0000","last_clean":"2023-10-07T09:30:26.390656+0000","last_became_active":"2023-10-07T09:30:24.968531+0000","last_became_peered":"2023-10-07T09:30:24.968531+0000","last_unstale":"2023-10-07T09:30:26.390656+0000","last_undegraded":"2023-10-07T09:30:26.390656+0000","last_fullsized":"2023-10-07T09:30:26.390656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:06:46.058261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.418399+0000","last_change":"2023-10-07T09:30:26.413894+0000","last_active":"2023-10-07T09:30:26.418399+0000","last_peered":"2023-10-07T09:30:26.418399+0000","last_clean":"2023-10-07T09:30:26.418399+0000","last_became_active":"2023-10-07T09:30:26.413276+0000","last_became_peered":"2023-10-07T09:30:26.413276+0000","last_unstale":"2023-10-07T09:30:26.418399+0000","last_undegraded":"2023-10-07T09:30:26.418399+0000","last_fullsized":"2023-10-07T09:30:26.418399+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:55.106120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.392045+0000","last_change":"2023-10-07T09:30:24.971933+0000","last_active":"2023-10-07T09:30:26.392045+0000","last_peered":"2023-10-07T09:30:26.392045+0000","last_clean":"2023-10-07T09:30:26.392045+0000","last_became_active":"2023-10-07T09:30:24.971478+0000","last_became_peered":"2023-10-07T09:30:24.971478+0000","last_unstale":"2023-10-07T09:30:26.392045+0000","last_undegraded":"2023-10-07T09:30:26.392045+0000","last_fullsized":"2023-10-07T09:30:26.392045+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:19.116181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.410100+0000","last_change":"2023-10-07T09:30:26.402235+0000","last_active":"2023-10-07T09:30:26.410100+0000","last_peered":"2023-10-07T09:30:26.410100+0000","last_clean":"2023-10-07T09:30:26.410100+0000","last_became_active":"2023-10-07T09:30:26.401915+0000","last_became_peered":"2023-10-07T09:30:26.401915+0000","last_unstale":"2023-10-07T09:30:26.410100+0000","last_undegraded":"2023-10-07T09:30:26.410100+0000","last_fullsized":"2023-10-07T09:30:26.410100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:39:01.010896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.394842+0000","last_change":"2023-10-07T09:30:24.967230+0000","last_active":"2023-10-07T09:30:26.394842+0000","last_peered":"2023-10-07T09:30:26.394842+0000","last_clean":"2023-10-07T09:30:26.394842+0000","last_became_active":"2023-10-07T09:30:24.966740+0000","last_became_peered":"2023-10-07T09:30:24.966740+0000","last_unstale":"2023-10-07T09:30:26.394842+0000","last_undegraded":"2023-10-07T09:30:26.394842+0000","last_fullsized":"2023-10-07T09:30:26.394842+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:31:56.775021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.413060+0000","last_change":"2023-10-07T09:30:26.407227+0000","last_active":"2023-10-07T09:30:26.413060+0000","last_peered":"2023-10-07T09:30:26.413060+0000","last_clean":"2023-10-07T09:30:26.413060+0000","last_became_active":"2023-10-07T09:30:26.406801+0000","last_became_peered":"2023-10-07T09:30:26.406801+0000","last_unstale":"2023-10-07T09:30:26.413060+0000","last_undegraded":"2023-10-07T09:30:26.413060+0000","last_fullsized":"2023-10-07T09:30:26.413060+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:26.784262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.384013+0000","last_change":"2023-10-07T09:30:24.967499+0000","last_active":"2023-10-07T09:30:26.384013+0000","last_peered":"2023-10-07T09:30:26.384013+0000","last_clean":"2023-10-07T09:30:26.384013+0000","last_became_active":"2023-10-07T09:30:24.967231+0000","last_became_peered":"2023-10-07T09:30:24.967231+0000","last_unstale":"2023-10-07T09:30:26.384013+0000","last_undegraded":"2023-10-07T09:30:26.384013+0000","last_fullsized":"2023-10-07T09:30:26.384013+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:23.529707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.408925+0000","last_change":"2023-10-07T09:30:26.402511+0000","last_active":"2023-10-07T09:30:26.408925+0000","last_peered":"2023-10-07T09:30:26.408925+0000","last_clean":"2023-10-07T09:30:26.408925+0000","last_became_active":"2023-10-07T09:30:26.402144+0000","last_became_peered":"2023-10-07T09:30:26.402144+0000","last_unstale":"2023-10-07T09:30:26.408925+0000","last_undegraded":"2023-10-07T09:30:26.408925+0000","last_fullsized":"2023-10-07T09:30:26.408925+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:24.939563+0000","last_clean_scrub_stamp":"2023-10-07T09:30:24.939563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:47:36.538323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.385241+0000","last_change":"2023-10-07T09:30:24.971597+0000","last_active":"2023-10-07T09:30:26.385241+0000","last_peered":"2023-10-07T09:30:26.385241+0000","last_clean":"2023-10-07T09:30:26.385241+0000","last_became_active":"2023-10-07T09:30:24.971341+0000","last_became_peered":"2023-10-07T09:30:24.971341+0000","last_unstale":"2023-10-07T09:30:26.385241+0000","last_undegraded":"2023-10-07T09:30:26.385241+0000","last_fullsized":"2023-10-07T09:30:26.385241+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:30:23.936384+0000","last_clean_scrub_stamp":"2023-10-07T09:30:23.936384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:48:15.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-07T09:30:26.386477+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:30:26.386477+0000","last_peered":"2023-10-07T09:30:26.386477+0000","last_clean":"2023-10-07T09:30:26.386477+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:30:26.386477+0000","last_undegraded":"2023-10-07T09:30:26.386477+0000","last_fullsized":"2023-10-07T09:30:26.386477+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476756,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298264,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073576,"statfs":{"total":96636764160,"available":96331341824,"internally_reserved":0,"allocated":999424,"data_stored":748763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1,"5min":1,"15min":1},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.71099999999999997}]},{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.89200000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.78100000000000003}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.84799999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.64200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.984,"5min":2.984,"15min":2.984},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.54100000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":7.867,"5min":7.867,"15min":7.867},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.497}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":8.0399999999999991},"last":0.622},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.71499999999999997}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.85699999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.749}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81000000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.443},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.47699999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.85099999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.92900000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.88100000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302356,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069484,"statfs":{"total":96636764160,"available":96327151616,"internally_reserved":0,"allocated":995328,"data_stored":748673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.726,"5min":2.726,"15min":2.726},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6909999999999998,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.65500000000000003}]},{"osd":5,"last update":"Sat Oct 7 09:30:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.8199999999999998,"5min":3.8199999999999998,"15min":3.8199999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":3.855,"5min":3.855,"15min":3.855},"last":0.876}]},{"osd":7,"last update":"Sat Oct 7 09:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":3.778,"5min":3.778,"15min":3.778},"last":0.83399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297684,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074156,"statfs":{"total":96636764160,"available":96331935744,"internally_reserved":0,"allocated":405504,"data_stored":158375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.442}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.75700000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.74099999999999999}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.92800000000000005}]}]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":408,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":417792,"data_stored":159863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.502},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.495}]},{"osd":2,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.754},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.81799999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.78400000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.82799999999999996}]},{"osd":1,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.753}]},{"osd":2,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.72099999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":6.0620000000000003},"last":0.76400000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.505}]},{"osd":6,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.623}]}]},{"osd":4,"up_from":15,"seq":64424509461,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":158409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64700000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59999999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:30:36.259 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:30:36.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:30:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.492+0000 7f87b93ff700 1 Processor -- start 2023-10-07T09:30:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.492+0000 7f87b93ff700 1 -- start start 2023-10-07T09:30:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.493+0000 7f87b93ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 0x7f87b415a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.493+0000 7f87b93ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b415ab80 con 0x7f87b4153e90 2023-10-07T09:30:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.493+0000 7f87b93ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b4069d40 con 0x7f87b40b0690 2023-10-07T09:30:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.493+0000 7f87b93ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b4068ca0 con 0x7f87b4153ac0 2023-10-07T09:30:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b27fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f87b4153e90 0x7f87b40aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41024/0 (socket says 172.21.15.29:41024) 2023-10-07T09:30:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b27fc700 1 -- 172.21.15.29:0/3423389603 learned_addr learned my addr 172.21.15.29:0/3423389603 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b2ffd700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 0x7f87b415a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321029810 0 0) 0x7f87a8001110 con 0x7f87b4153e90 2023-10-07T09:30:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879c003570 con 0x7f87b4153e90 2023-10-07T09:30:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516365757 0 0) 0x7f87b4069d40 con 0x7f87b40b0690 2023-10-07T09:30:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a8001110 con 0x7f87b40b0690 2023-10-07T09:30:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352790224 0 0) 0x7f879c003570 con 0x7f87b4153e90 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 msgr2=0x7f87b415a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 0x7f87b415a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 >> v1:172.21.15.144:6789/0 conn(0x7f87b40b0690 legacy=0x7f87b415ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b4068f80 con 0x7f87b4153e90 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.494+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f87a8001600 con 0x7f87b4153e90 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 >> v1:172.21.15.29:6789/0 conn(0x7f87b4153e90 legacy=0x7f87b40aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f879c003c20 0x7f879c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 0x7f879c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f879c004d50 0x7f879c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b4068ca0 con 0x7f879c004d50 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c00bbf0 con 0x7f879c003c20 2023-10-07T09:30:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879c00bd60 con 0x7f87b4153ac0 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b37fe700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f879c004d50 0x7f879c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b27fc700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 0x7f879c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b2ffd700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f879c003c20 0x7f879c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b37fe700 1 -- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 msgr2=0x7f879c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b37fe700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b4153ac0 0x7f879c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b37fe700 1 -- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f879c003c20 msgr2=0x7f879c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b37fe700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f879c003c20 0x7f879c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.495+0000 7f87b37fe700 1 -- 172.21.15.29:0/3423389603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c003570 con 0x7f879c004d50 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b37fe700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f879c004d50 0x7f879c0051c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f87b40b0a90 tx=0x7f87ac0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809cd00aebcbd200 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac001ab0 con 0x7f879c004d50 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ac00c8b0 con 0x7f879c004d50 2023-10-07T09:30:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b1ffb700 1 -- 172.21.15.29:0/3423389603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac00cc70 con 0x7f879c004d50 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b93ff700 1 -- 172.21.15.29:0/3423389603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f879c004d50 msgr2=0x7f879c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b93ff700 1 --2- 172.21.15.29:0/3423389603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f879c004d50 0x7f879c0051c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f87b40b0a90 tx=0x7f87ac0017a0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b37fe700 1 -- 172.21.15.29:0/3423389603 reap_dead start 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b93ff700 1 -- 172.21.15.29:0/3423389603 shutdown_connections 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.496+0000 7f87b93ff700 1 -- 172.21.15.29:0/3423389603 >> 172.21.15.29:0/3423389603 conn(0x7f87b4067610 msgr2=0x7f87b4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.497+0000 7f87b93ff700 1 -- 172.21.15.29:0/3423389603 shutdown_connections 2023-10-07T09:30:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.497+0000 7f87b93ff700 1 -- 172.21.15.29:0/3423389603 wait complete. 2023-10-07T09:30:36.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.498+0000 7f87b93ff700 1 Processor -- start 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.498+0000 7f87b93ff700 1 -- start start 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b93ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879c004d50 0x7f87b4150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b93ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 0x7f87b41503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b93ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b415b160 0x7f87b415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b93ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b4150910 con 0x7f87b416d2f0 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b93ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40c9c00 con 0x7f87b415b160 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b93ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b416d760 con 0x7f879c004d50 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 0x7f87b41503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 0x7f87b41503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41366/0 (socket says 172.21.15.29:41366) 2023-10-07T09:30:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.499+0000 7f87b27fc700 1 -- 172.21.15.29:0/2043374941 learned_addr learned my addr 172.21.15.29:0/2043374941 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b2ffd700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879c004d50 0x7f87b4150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b37fe700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b415b160 0x7f87b415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b27fc700 1 -- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879c004d50 msgr2=0x7f87b4150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b27fc700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879c004d50 0x7f87b4150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b27fc700 1 -- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b415b160 msgr2=0x7f87b415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b27fc700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b415b160 0x7f87b415b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b27fc700 1 -- 172.21.15.29:0/2043374941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b40c9030 con 0x7f87b416d2f0 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b37fe700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b415b160 0x7f87b415b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b27fc700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 0x7f87b41503d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f87a8007390 tx=0x7f87a8007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8018030 con 0x7f87b416d2f0 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8007750 con 0x7f87b416d2f0 2023-10-07T09:30:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b4151270 con 0x7f87b416d2f0 2023-10-07T09:30:36.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.500+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800fb00 con 0x7f87b416d2f0 2023-10-07T09:30:36.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.501+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b40c95b0 con 0x7f87b416d2f0 2023-10-07T09:30:36.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.502+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b40718b0 con 0x7f87b416d2f0 2023-10-07T09:30:36.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.502+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f87a801b070 con 0x7f87b416d2f0 2023-10-07T09:30:36.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.503+0000 7f879affd700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f87a0066f30 0x7f87a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:36.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.503+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f87a8019070 con 0x7f87b416d2f0 2023-10-07T09:30:36.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.504+0000 7f87b2ffd700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f87a0066f30 0x7f87a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:36.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.504+0000 7f87b2ffd700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f87a0066f30 0x7f87a00693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f87a4006f00 tx=0x7f87a40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:36.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.507+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87a8051530 con 0x7f87b416d2f0 2023-10-07T09:30:36.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.789+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f87b415ab80 con 0x7f87b416d2f0 2023-10-07T09:30:36.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.793+0000 7f879affd700 1 -- 172.21.15.29:0/2043374941 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f87b415ab80 con 0x7f87b416d2f0 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.795+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f87a0066f30 msgr2=0x7f87a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.795+0000 7f87b93ff700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f87a0066f30 0x7f87a00693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f87a4006f00 tx=0x7f87a40091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 msgr2=0x7f87b41503d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 0x7f87b41503d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f87a8007390 tx=0x7f87a8007940 comp rx=0 tx=0).stop 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 shutdown_connections 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f87a0066f30 0x7f87a00693e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b415b160 0x7f87b415b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879c004d50 0x7f87b4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 --2- 172.21.15.29:0/2043374941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87b416d2f0 0x7f87b41503d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:36.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 >> 172.21.15.29:0/2043374941 conn(0x7f87b4067610 msgr2=0x7f87b41592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:36.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 shutdown_connections 2023-10-07T09:30:36.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:36.796+0000 7f87b93ff700 1 -- 172.21.15.29:0/2043374941 wait complete. 2023-10-07T09:30:36.798 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:30:36.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:30:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.009+0000 7fdd3a5ff700 1 Processor -- start 2023-10-07T09:30:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.009+0000 7fdd3a5ff700 1 -- start start 2023-10-07T09:30:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd3a5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd34159b20 0x7fdd340d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd3a5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340cbf20 con 0x7fdd340ce8a0 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd3a5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340cc200 con 0x7fdd34159750 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd3a5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd340cc4e0 con 0x7fdd34159b20 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd337fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd34159b20 0x7fdd340d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd337fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd34159b20 0x7fdd340d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41380/0 (socket says 172.21.15.29:41380) 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd337fe700 1 -- 172.21.15.29:0/3011106037 learned_addr learned my addr 172.21.15.29:0/3011106037 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.011+0000 7fdd38b9c700 1 --1- 172.21.15.29:0/3011106037 >> v1:172.21.15.29:6789/0 conn(0x7fdd340ce8a0 0x7fdd340c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41028/0 (socket says 172.21.15.29:41028) 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd337fe700 1 -- 172.21.15.29:0/3011106037 >> v1:172.21.15.144:6789/0 conn(0x7fdd34159750 legacy=0x7fdd340ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd337fe700 1 -- 172.21.15.29:0/3011106037 >> v1:172.21.15.29:6789/0 conn(0x7fdd340ce8a0 legacy=0x7fdd340c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd337fe700 1 -- 172.21.15.29:0/3011106037 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340cc7c0 con 0x7fdd34159b20 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd337fe700 1 --2- 172.21.15.29:0/3011106037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd340d10a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdd280099b0 tx=0x7fdd28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc0998a2840aee5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd32ffd700 1 -- 172.21.15.29:0/3011106037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800b860 con 0x7fdd34159b20 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd32ffd700 1 -- 172.21.15.29:0/3011106037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2800b9c0 con 0x7fdd34159b20 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.012+0000 7fdd32ffd700 1 -- 172.21.15.29:0/3011106037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800bd60 con 0x7fdd34159b20 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/3011106037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 msgr2=0x7fdd340d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/3011106037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd340d10a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdd280099b0 tx=0x7fdd28009250 comp rx=0 tx=0).stop 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/3011106037 shutdown_connections 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/3011106037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd340d10a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/3011106037 >> 172.21.15.29:0/3011106037 conn(0x7fdd34063a10 msgr2=0x7fdd340d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/3011106037 shutdown_connections 2023-10-07T09:30:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.013+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/3011106037 wait complete. 2023-10-07T09:30:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.015+0000 7fdd3a5ff700 1 Processor -- start 2023-10-07T09:30:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.015+0000 7fdd3a5ff700 1 -- start start 2023-10-07T09:30:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd3a5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd340ce8a0 0x7fdd34169b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd3a5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd34159750 0x7fdd34156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd3a5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd34156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd3a5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340ca430 con 0x7fdd34159b20 2023-10-07T09:30:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd3a5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340c9820 con 0x7fdd34159750 2023-10-07T09:30:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd3a5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340c9b00 con 0x7fdd340ce8a0 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd34156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd34156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41396/0 (socket says 172.21.15.29:41396) 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 -- 172.21.15.29:0/2064928773 learned_addr learned my addr 172.21.15.29:0/2064928773 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd33fff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd340ce8a0 0x7fdd34169b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd337fe700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd34159750 0x7fdd34156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 -- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd340ce8a0 msgr2=0x7fdd34169b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd340ce8a0 0x7fdd34169b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 -- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd34159750 msgr2=0x7fdd34156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd34159750 0x7fdd34156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.016+0000 7fdd38b9c700 1 -- 172.21.15.29:0/2064928773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280094f0 con 0x7fdd34159b20 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.017+0000 7fdd38b9c700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd34156590 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00cd00 tx=0x7fdd2c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.017+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c00ce60 con 0x7fdd34159b20 2023-10-07T09:30:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.017+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2c005df0 con 0x7fdd34159b20 2023-10-07T09:30:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.017+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c00ab00 con 0x7fdd34159b20 2023-10-07T09:30:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.017+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd340c9de0 con 0x7fdd34159b20 2023-10-07T09:30:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.017+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3416a040 con 0x7fdd34159b20 2023-10-07T09:30:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.018+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3404c3c0 con 0x7fdd34159b20 2023-10-07T09:30:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.019+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdd2c0070f0 con 0x7fdd34159b20 2023-10-07T09:30:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.020+0000 7fdd1bfff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdd20066f30 0x7fdd200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.020+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fdd2c084580 con 0x7fdd34159b20 2023-10-07T09:30:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.020+0000 7fdd33fff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdd20066f30 0x7fdd200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.021+0000 7fdd33fff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdd20066f30 0x7fdd200693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd24003b00 tx=0x7fdd24007ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.024+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd2c0517d0 con 0x7fdd34159b20 2023-10-07T09:30:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.275+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd340709e0 con 0x7fdd34159b20 2023-10-07T09:30:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.275+0000 7fdd1bfff700 1 -- 172.21.15.29:0/2064928773 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4600 (secure 0 0 0) 0x7fdd2c054df0 con 0x7fdd34159b20 2023-10-07T09:30:37.277 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:30:37.277 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:30:25.453869+0000","modified":"2023-10-07T09:30:29.396464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6836/3596704883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3596704883},{"type":"v1","addr":"172.21.15.144:6836","nonce":3596704883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-07T09:30:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdd20066f30 msgr2=0x7fdd200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdd20066f30 0x7fdd200693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd24003b00 tx=0x7fdd24007ef0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 msgr2=0x7fdd34156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd34156590 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00cd00 tx=0x7fdd2c001b10 comp rx=0 tx=0).stop 2023-10-07T09:30:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 shutdown_connections 2023-10-07T09:30:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fdd20066f30 0x7fdd200693e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd34159b20 0x7fdd34156590 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd340ce8a0 0x7fdd34169b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 --2- 172.21.15.29:0/2064928773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd34159750 0x7fdd34156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 >> 172.21.15.29:0/2064928773 conn(0x7fdd34063a10 msgr2=0x7fdd340d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 shutdown_connections 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.278+0000 7fdd3a5ff700 1 -- 172.21.15.29:0/2064928773 wait complete. 2023-10-07T09:30:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-07T09:30:37.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:30:25.453869+0000', 'modified': '2023-10-07T09:30:29.396464+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4959}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4959': {'gid': 4959, 'name': 'd', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.144:6836/3596704883', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6834', 'nonce': 3596704883}, {'type': 'v1', 'addr': '172.21.15.144:6836', 'nonce': 3596704883}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:30:37.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:30:37.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-07T09:30:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.896+0000 7fa52cdff700 1 Processor -- start 2023-10-07T09:30:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.896+0000 7fa52cdff700 1 -- start start 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.897+0000 7fa52cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa528159750 0x7fa5280d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.897+0000 7fa52cdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa528159b20 0x7fa5280ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.897+0000 7fa52cdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5280ce6d0 0x7fa5280ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa52cdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5280cbf20 con 0x7fa5280ce6d0 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa52cdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5280cc200 con 0x7fa528159b20 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa52cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280cc4e0 con 0x7fa528159750 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5280ce6d0 0x7fa5280ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5280ce6d0 0x7fa5280ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41404/0 (socket says 172.21.15.29:41404) 2023-10-07T09:30:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 -- 172.21.15.29:0/4054775691 learned_addr learned my addr 172.21.15.29:0/4054775691 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa525d9b700 1 --2- 172.21.15.29:0/4054775691 >> v2:172.21.15.144:3300/0 conn(0x7fa528159b20 0x7fa5280ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa52659c700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa528159750 0x7fa5280d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 -- 172.21.15.29:0/4054775691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa528159750 msgr2=0x7fa5280d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa528159750 0x7fa5280d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 -- 172.21.15.29:0/4054775691 >> v2:172.21.15.144:3300/0 conn(0x7fa528159b20 msgr2=0x7fa5280ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 --2- 172.21.15.29:0/4054775691 >> v2:172.21.15.144:3300/0 conn(0x7fa528159b20 0x7fa5280ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa526d9d700 1 -- 172.21.15.29:0/4054775691 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5280cc7c0 con 0x7fa5280ce6d0 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa525d9b700 1 --2- 172.21.15.29:0/4054775691 >> v2:172.21.15.144:3300/0 conn(0x7fa528159b20 0x7fa5280ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.898+0000 7fa52659c700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa528159750 0x7fa5280d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.899+0000 7fa526d9d700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5280ce6d0 0x7fa5280ceb50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa528088830 tx=0x7fa520009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eceb3324d72c7cda server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.899+0000 7fa52559a700 1 -- 172.21.15.29:0/4054775691 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000e070 con 0x7fa5280ce6d0 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.899+0000 7fa52559a700 1 -- 172.21.15.29:0/4054775691 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52000ba50 con 0x7fa5280ce6d0 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.899+0000 7fa52559a700 1 -- 172.21.15.29:0/4054775691 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5200092e0 con 0x7fa5280ce6d0 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.899+0000 7fa52cdff700 1 -- 172.21.15.29:0/4054775691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5280ce6d0 msgr2=0x7fa5280ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.899+0000 7fa52cdff700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5280ce6d0 0x7fa5280ceb50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa528088830 tx=0x7fa520009840 comp rx=0 tx=0).stop 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 -- 172.21.15.29:0/4054775691 shutdown_connections 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5280ce6d0 0x7fa5280ceb50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 --2- 172.21.15.29:0/4054775691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa528159750 0x7fa5280d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 --2- 172.21.15.29:0/4054775691 >> v2:172.21.15.144:3300/0 conn(0x7fa528159b20 0x7fa5280ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 -- 172.21.15.29:0/4054775691 >> 172.21.15.29:0/4054775691 conn(0x7fa528063a10 msgr2=0x7fa5280d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 -- 172.21.15.29:0/4054775691 shutdown_connections 2023-10-07T09:30:37.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.900+0000 7fa52cdff700 1 -- 172.21.15.29:0/4054775691 wait complete. 2023-10-07T09:30:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 Processor -- start 2023-10-07T09:30:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 -- start start 2023-10-07T09:30:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5280ce6d0 0x7fa5280697b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa528159750 0x7fa528154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 0x7fa52815f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280b3ee0 con 0x7fa528159b20 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa528153750 con 0x7fa528159750 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.902+0000 7fa52cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa528153a30 con 0x7fa5280ce6d0 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 0x7fa52815f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 0x7fa52815f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41406/0 (socket says 172.21.15.29:41406) 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 -- 172.21.15.29:0/3968912664 learned_addr learned my addr 172.21.15.29:0/3968912664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa52659c700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5280ce6d0 0x7fa5280697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa525d9b700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa528159750 0x7fa528154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 -- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5280ce6d0 msgr2=0x7fa5280697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5280ce6d0 0x7fa5280697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 -- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa528159750 msgr2=0x7fa528154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa528159750 0x7fa528154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 -- 172.21.15.29:0/3968912664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa520009ae0 con 0x7fa528159b20 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa525d9b700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa528159750 0x7fa528154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa52659c700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5280ce6d0 0x7fa5280697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa526d9d700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 0x7fa52815f510 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa528088a40 tx=0x7fa520018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa5167fc700 1 -- 172.21.15.29:0/3968912664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520008e80 con 0x7fa528159b20 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.903+0000 7fa5167fc700 1 -- 172.21.15.29:0/3968912664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520026460 con 0x7fa528159b20 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.904+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa528153d10 con 0x7fa528159b20 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.904+0000 7fa5167fc700 1 -- 172.21.15.29:0/3968912664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520026800 con 0x7fa528159b20 2023-10-07T09:30:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.904+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa528153370 con 0x7fa528159b20 2023-10-07T09:30:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.906+0000 7fa5167fc700 1 -- 172.21.15.29:0/3968912664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa520003040 con 0x7fa528159b20 2023-10-07T09:30:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.907+0000 7fa5167fc700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa508066f30 0x7fa5080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.907+0000 7fa5167fc700 1 -- 172.21.15.29:0/3968912664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa520086100 con 0x7fa528159b20 2023-10-07T09:30:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.907+0000 7fa52659c700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa508066f30 0x7fa5080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.908+0000 7fa52659c700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa508066f30 0x7fa5080693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 Processor -- start 2023-10-07T09:30:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 -- start start 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad10f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad10fbc0 0x5563ad110010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563acfd9310 con 0x5563ad0ec530 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ad016130 con 0x5563ad10fbc0 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ad013af0 con 0x5563ad106a40 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad10f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad10f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41412/0 (socket says 172.21.15.29:41412) 2023-10-07T09:30:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 -- 172.21.15.29:0/4191853568 learned_addr learned my addr 172.21.15.29:0/4191853568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa52659c700 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa526d9d700 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad10fbc0 0x5563ad110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 -- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 msgr2=0x5563ad106e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad106e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 -- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad10fbc0 msgr2=0x5563ad110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.910+0000 7fa525d9b700 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad10fbc0 0x5563ad110010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.911+0000 7fa525d9b700 1 -- 172.21.15.29:0/4191853568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563acf020a0 con 0x5563ad0ec530 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.911+0000 7fa525d9b700 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad10f680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00ba10 tx=0x7fa51c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa0d042af104787 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.911+0000 7fa5077fe700 1 -- 172.21.15.29:0/4191853568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c00d6b0 con 0x5563ad0ec530 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.911+0000 7fa5077fe700 1 -- 172.21.15.29:0/4191853568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c00c040 con 0x5563ad0ec530 2023-10-07T09:30:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.911+0000 7fa5077fe700 1 -- 172.21.15.29:0/4191853568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c00da60 con 0x5563ad0ec530 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/4191853568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 msgr2=0x5563ad10f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad10f680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00ba10 tx=0x7fa51c008040 comp rx=0 tx=0).stop 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/4191853568 shutdown_connections 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad10fbc0 0x5563ad110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/4191853568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad10f680 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/4191853568 >> 172.21.15.29:0/4191853568 conn(0x5563ad109e70 msgr2=0x5563ad10a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/4191853568 shutdown_connections 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.912+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/4191853568 wait complete. 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.913+0000 7fa53b5d1b80 1 -- start start 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.913+0000 7fa53b5d1b80 1 Processor -- start 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad0ec530 0x5563ad0e6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad0bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad0c0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ad00f8e0 con 0x5563ad10fbc0 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563acfbb980 con 0x5563ad0ec530 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563acfbb620 con 0x5563ad106a40 2023-10-07T09:30:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa53b5d1b80 10 client.0 _ll_register_callbacks cb 0x5563ace1ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad0c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad0c0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41416/0 (socket says 172.21.15.29:41416) 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 -- 172.21.15.29:0/2293933608 learned_addr learned my addr 172.21.15.29:0/2293933608 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa52659c700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad0ec530 0x5563ad0e6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa525d9b700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad0bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 -- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 msgr2=0x5563ad0bfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad0bfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 -- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad0ec530 msgr2=0x5563ad0e6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad0ec530 0x5563ad0e6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.914+0000 7fa526d9d700 1 -- 172.21.15.29:0/2293933608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c00b670 con 0x5563ad10fbc0 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa526d9d700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad0c0370 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa520056c90 tx=0x7fa52005c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64875e0b54427f8a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa504ff9700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa504ff9700 1 -- 172.21.15.29:0/2293933608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000e040 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa504ff9700 1 -- 172.21.15.29:0/2293933608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa52005a1e0 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563ad0b4560 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa504ff9700 1 -- 172.21.15.29:0/2293933608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa52005c2c0 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa53b5d1b80 10 client.5157 fetch_fsmap learned FSMap version 15 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x5563acfde3c0 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa504ff9700 1 -- 172.21.15.29:0/2293933608 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520082080 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa504ff9700 1 -- 172.21.15.29:0/2293933608 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 15) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fa52006d170 con 0x5563ad10fbc0 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.915+0000 7fa53b5d1b80 10 client.5157 fetch_fsmap finished waiting for FSMap version 15 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.916+0000 7fa53b5d1b80 10 client.5157 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-07T09:30:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.916+0000 7fa53b5d1b80 20 client.5157 populate_metadata read hostname 'smithi029' 2023-10-07T09:30:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.916+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad119600 0x5563ad1199d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.916+0000 7fa53b5d1b80 4 client.5157 mds_command: new command op to 4959 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:30:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.916+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 --> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563ad0910e0 con 0x5563ad119600 2023-10-07T09:30:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.916+0000 7fa52659c700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad119600 0x5563ad1199d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.917+0000 7fa504ff9700 10 client.5157 ms_handle_connect on v2:172.21.15.144:6834/3596704883 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.917+0000 7fa52659c700 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad119600 0x5563ad1199d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=597f8758a1210d21 server_cookie=a206a6653d147b95 in_seq=0 out_seq=0 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.917+0000 7fa504ff9700 1 -- 172.21.15.29:0/2293933608 <== mds.0 v2:172.21.15.144:6834/3596704883 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5563ad0910e0 con 0x5563ad119600 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.917+0000 7fa504ff9700 10 client.5157 handle_command_reply: tid=0 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.917+0000 7fa53b5d1b80 1 client.5157 shutdown 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 msgr2=0x5563ad0c0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad0c0370 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa520056c90 tx=0x7fa52005c040 comp rx=0 tx=0).stop 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 shutdown_connections 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad0c0370 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad119600 0x5563ad1199d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad0ec530 0x5563ad0e6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/2293933608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad106a40 0x5563ad0bfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 >> 172.21.15.29:0/2293933608 conn(0x5563ad109e70 msgr2=0x5563ad0bf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa504ff9700 0 client.5157 ms_handle_reset on v2:172.21.15.144:6834/3596704883 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 shutdown_connections 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/2293933608 wait complete. 2023-10-07T09:30:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.918+0000 7fa53b5d1b80 20 client.5157 trim_cache size 0 max 16384 2023-10-07T09:30:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.936+0000 7fa53b5d1b80 1 Processor -- start 2023-10-07T09:30:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.936+0000 7fa53b5d1b80 1 -- start start 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.936+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad0ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.936+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad10a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.936+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad10fbc0 0x5563ad0ef440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563acea71e0 con 0x5563ad0ec530 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ad127410 con 0x5563ad106a40 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ad0e6d60 con 0x5563ad10fbc0 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad0ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad0ee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41422/0 (socket says 172.21.15.29:41422) 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 -- 172.21.15.29:0/3894284535 learned_addr learned my addr 172.21.15.29:0/3894284535 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 -- 172.21.15.29:0/3894284535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad10fbc0 msgr2=0x5563ad0ef440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad10fbc0 0x5563ad0ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 -- 172.21.15.29:0/3894284535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 msgr2=0x5563ad10a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad10a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 -- 172.21.15.29:0/3894284535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563ace761d0 con 0x5563ad0ec530 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.937+0000 7fa52659c700 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad0ee9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa518008460 tx=0x7fa518027250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89feda1b4d76166 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa507fff700 1 -- 172.21.15.29:0/3894284535 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518026070 con 0x5563ad0ec530 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa507fff700 1 -- 172.21.15.29:0/3894284535 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51800c040 con 0x5563ad0ec530 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa507fff700 1 -- 172.21.15.29:0/3894284535 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51800b050 con 0x5563ad0ec530 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/3894284535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 msgr2=0x5563ad0ee9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad0ee9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa518008460 tx=0x7fa518027250 comp rx=0 tx=0).stop 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/3894284535 shutdown_connections 2023-10-07T09:30:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad10fbc0 0x5563ad0ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad0ec530 0x5563ad0ee9a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/3894284535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/3894284535 >> 172.21.15.29:0/3894284535 conn(0x5563ad109e70 msgr2=0x5563ad10c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/3894284535 shutdown_connections 2023-10-07T09:30:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.938+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/3894284535 wait complete. 2023-10-07T09:30:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.939+0000 7fa53b5d1b80 1 -- start start 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.939+0000 7fa53b5d1b80 1 Processor -- start 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad0ec530 0x5563ad10c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad0e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad117fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ad0e7fa0 con 0x5563ad10fbc0 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ad10c660 con 0x5563ad106a40 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ad0eb100 con 0x5563ad0ec530 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa53b5d1b80 10 client.0 _ll_register_callbacks cb 0x5563ace1ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:30:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa526d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad117fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa526d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad117fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41434/0 (socket says 172.21.15.29:41434) 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.940+0000 7fa526d9d700 1 -- 172.21.15.29:0/1118826493 learned_addr learned my addr 172.21.15.29:0/1118826493 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa525d9b700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad0e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa52659c700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad0ec530 0x5563ad10c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa526d9d700 1 -- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad0ec530 msgr2=0x5563ad10c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa526d9d700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad0ec530 0x5563ad10c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa526d9d700 1 -- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 msgr2=0x5563ad0e87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa526d9d700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad0e87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa526d9d700 1 -- 172.21.15.29:0/1118826493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51800dc70 con 0x5563ad10fbc0 2023-10-07T09:30:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa52659c700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad0ec530 0x5563ad10c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa526d9d700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad117fd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa520009880 tx=0x7fa52006c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942cf846580e3b6a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa504ff9700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa504ff9700 1 -- 172.21.15.29:0/1118826493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000e040 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa51c00c040 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa504ff9700 1 -- 172.21.15.29:0/1118826493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa52005a1e0 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.941+0000 7fa504ff9700 1 -- 172.21.15.29:0/1118826493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa52005c2c0 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 10 client.5163 fetch_fsmap learned FSMap version 15 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x5563ace4fd60 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa504ff9700 1 -- 172.21.15.29:0/1118826493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c00c040 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa504ff9700 1 -- 172.21.15.29:0/1118826493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 15) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fa52005daa0 con 0x5563ad10fbc0 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 10 client.5163 fetch_fsmap finished waiting for FSMap version 15 2023-10-07T09:30:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 10 client.5163 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 20 client.5163 populate_metadata read hostname 'smithi029' 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad0e6eb0 0x5563ad0e7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 4 client.5163 mds_command: new command op to 4959 tid=0[{"prefix": "status"}] 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 --> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5563acfb28d0 con 0x5563ad0e6eb0 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.942+0000 7fa52659c700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad0e6eb0 0x5563ad0e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52659c700 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad0e6eb0 0x5563ad0e7280 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e066d9e1ff3d59a5 server_cookie=1eee181d472a7268 in_seq=0 out_seq=0 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa504ff9700 10 client.5163 ms_handle_connect on v2:172.21.15.144:6834/3596704883 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa504ff9700 1 -- 172.21.15.29:0/1118826493 <== mds.0 v2:172.21.15.144:6834/3596704883 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5563acfb28d0 con 0x5563ad0e6eb0 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa504ff9700 10 client.5163 handle_command_reply: tid=0 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 client.5163 shutdown 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 msgr2=0x5563ad117fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad117fd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa520009880 tx=0x7fa52006c000 comp rx=0 tx=0).stop 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 shutdown_connections 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5563ad10fbc0 0x5563ad117fd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5563ad0ec530 0x5563ad10c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:6834/3596704883,v1:172.21.15.144:6836/3596704883] conn(0x5563ad0e6eb0 0x5563ad0e7280 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 --2- 172.21.15.29:0/1118826493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5563ad106a40 0x5563ad0e87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa504ff9700 0 client.5163 ms_handle_reset on v2:172.21.15.144:6834/3596704883 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 >> 172.21.15.29:0/1118826493 conn(0x5563ad109e70 msgr2=0x5563ad0ef830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 shutdown_connections 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 1 -- 172.21.15.29:0/1118826493 wait complete. 2023-10-07T09:30:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa53b5d1b80 20 client.5163 trim_cache size 0 max 16384 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4959, 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 11.546354001999999, 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 15, 2023-10-07T09:30:37.951 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 36, 2023-10-07T09:30:37.952 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 36, 2023-10-07T09:30:37.952 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 21.343612869000001 2023-10-07T09:30:37.952 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa508066f30 msgr2=0x7fa5080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa508066f30 0x7fa5080693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518008040 comp rx=0 tx=0).stop 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 msgr2=0x7fa52815f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 0x7fa52815f510 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa528088a40 tx=0x7fa520018000 comp rx=0 tx=0).stop 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 shutdown_connections 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa508066f30 0x7fa5080693e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa528159b20 0x7fa52815f510 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5280ce6d0 0x7fa5280697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 --2- 172.21.15.29:0/3968912664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa528159750 0x7fa528154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 >> 172.21.15.29:0/3968912664 conn(0x7fa528063a10 msgr2=0x7fa5280ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:37.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.944+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 shutdown_connections 2023-10-07T09:30:37.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:37.945+0000 7fa52cdff700 1 -- 172.21.15.29:0/3968912664 wait complete. 2023-10-07T09:30:37.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:30:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.160+0000 7fa720bff700 1 Processor -- start 2023-10-07T09:30:38.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.160+0000 7fa720bff700 1 -- start start 2023-10-07T09:30:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa720bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa720bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa71c0b0410 0x7fa71c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa720bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71c14d790 con 0x7fa71c159ec0 2023-10-07T09:30:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa720bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa71c0ae5d0 con 0x7fa71c0b0410 2023-10-07T09:30:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa720bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c15a850 con 0x7fa71c0ad370 2023-10-07T09:30:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53086/0 (socket says 172.21.15.29:53086) 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 -- 172.21.15.29:0/1075460256 learned_addr learned my addr 172.21.15.29:0/1075460256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa719d9b700 1 --1- 172.21.15.29:0/1075460256 >> v1:172.21.15.144:6789/0 conn(0x7fa71c159ec0 0x7fa71c1568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:45262/0 (socket says 172.21.15.29:45262) 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71ad9d700 1 --2- 172.21.15.29:0/1075460256 >> v2:172.21.15.144:3300/0 conn(0x7fa71c0b0410 0x7fa71c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71959a700 1 -- 172.21.15.29:0/1075460256 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131747773 0 0) 0x7fa71c14d790 con 0x7fa71c159ec0 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71959a700 1 -- 172.21.15.29:0/1075460256 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa704003570 con 0x7fa71c159ec0 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 -- 172.21.15.29:0/1075460256 >> v2:172.21.15.144:3300/0 conn(0x7fa71c0b0410 msgr2=0x7fa71c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 --2- 172.21.15.29:0/1075460256 >> v2:172.21.15.144:3300/0 conn(0x7fa71c0b0410 0x7fa71c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 -- 172.21.15.29:0/1075460256 >> v1:172.21.15.144:6789/0 conn(0x7fa71c159ec0 legacy=0x7fa71c1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71a59c700 1 -- 172.21.15.29:0/1075460256 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c0ad740 con 0x7fa71c0ad370 2023-10-07T09:30:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.162+0000 7fa71ad9d700 1 --2- 172.21.15.29:0/1075460256 >> v2:172.21.15.144:3300/0 conn(0x7fa71c0b0410 0x7fa71c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa71a59c700 1 --2- 172.21.15.29:0/1075460256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c0cc870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0099b0 tx=0x7fa70c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6727541d716b1b15 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa71959a700 1 -- 172.21.15.29:0/1075460256 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c00f6b0 con 0x7fa71c0ad370 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa71959a700 1 -- 172.21.15.29:0/1075460256 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c00f810 con 0x7fa71c0ad370 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa71959a700 1 -- 172.21.15.29:0/1075460256 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c00fbb0 con 0x7fa71c0ad370 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa720bff700 1 -- 172.21.15.29:0/1075460256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 msgr2=0x7fa71c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa720bff700 1 --2- 172.21.15.29:0/1075460256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c0cc870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0099b0 tx=0x7fa70c009250 comp rx=0 tx=0).stop 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa720bff700 1 -- 172.21.15.29:0/1075460256 shutdown_connections 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa720bff700 1 --2- 172.21.15.29:0/1075460256 >> v2:172.21.15.144:3300/0 conn(0x7fa71c0b0410 0x7fa71c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa720bff700 1 --2- 172.21.15.29:0/1075460256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c0cc870 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.163+0000 7fa720bff700 1 -- 172.21.15.29:0/1075460256 >> 172.21.15.29:0/1075460256 conn(0x7fa71c0680a0 msgr2=0x7fa71c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.164+0000 7fa720bff700 1 -- 172.21.15.29:0/1075460256 shutdown_connections 2023-10-07T09:30:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.164+0000 7fa720bff700 1 -- 172.21.15.29:0/1075460256 wait complete. 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.166+0000 7fa720bff700 1 Processor -- start 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.166+0000 7fa720bff700 1 -- start start 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa720bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c14d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa720bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 0x7fa71c150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa720bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa71c159ec0 0x7fa71c169780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa720bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c155c10 con 0x7fa71c0b0410 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa720bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c1507c0 con 0x7fa71c159ec0 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa720bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c150a40 con 0x7fa71c0ad370 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 0x7fa71c150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 0x7fa71c150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41440/0 (socket says 172.21.15.29:41440) 2023-10-07T09:30:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 -- 172.21.15.29:0/3193796953 learned_addr learned my addr 172.21.15.29:0/3193796953 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa71ad9d700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa71c159ec0 0x7fa71c169780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa71a59c700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 -- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 msgr2=0x7fa71c14d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c14d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 -- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa71c159ec0 msgr2=0x7fa71c169780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa71c159ec0 0x7fa71c169780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 -- 172.21.15.29:0/3193796953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa714009a20 con 0x7fa71c0b0410 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa71ad9d700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa71c159ec0 0x7fa71c169780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa719d9b700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 0x7fa71c150280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c5c0 tx=0x7fa710007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710005a70 con 0x7fa71c0b0410 2023-10-07T09:30:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.167+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa710005bf0 con 0x7fa71c0b0410 2023-10-07T09:30:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.168+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c0094f0 con 0x7fa71c0b0410 2023-10-07T09:30:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.168+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7100070d0 con 0x7fa71c0b0410 2023-10-07T09:30:38.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.168+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71c1555c0 con 0x7fa71c0b0410 2023-10-07T09:30:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.169+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71c04c3c0 con 0x7fa71c0b0410 2023-10-07T09:30:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.170+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa710007480 con 0x7fa71c0b0410 2023-10-07T09:30:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.170+0000 7fa70a7fc700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6fc066f30 0x7fa6fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.170+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa710083b80 con 0x7fa71c0b0410 2023-10-07T09:30:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.171+0000 7fa71a59c700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6fc066f30 0x7fa6fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:38.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.171+0000 7fa71a59c700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6fc066f30 0x7fa6fc0693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0099b0 tx=0x7fa70c019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:38.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.175+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa710050d50 con 0x7fa71c0b0410 2023-10-07T09:30:38.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.420+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa71c0c8210 con 0x7fa71c0b0410 2023-10-07T09:30:38.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.420+0000 7fa70a7fc700 1 -- 172.21.15.29:0/3193796953 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa710083e60 con 0x7fa71c0b0410 2023-10-07T09:30:38.421 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6fc066f30 msgr2=0x7fa6fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6fc066f30 0x7fa6fc0693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0099b0 tx=0x7fa70c019000 comp rx=0 tx=0).stop 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 msgr2=0x7fa71c150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 0x7fa71c150280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c5c0 tx=0x7fa710007e00 comp rx=0 tx=0).stop 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 shutdown_connections 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6fc066f30 0x7fa6fc0693e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa71c159ec0 0x7fa71c169780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa71c0ad370 0x7fa71c14d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 --2- 172.21.15.29:0/3193796953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa71c0b0410 0x7fa71c150280 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:38.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.422+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 >> 172.21.15.29:0/3193796953 conn(0x7fa71c0680a0 msgr2=0x7fa71c06d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:38.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.423+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 shutdown_connections 2023-10-07T09:30:38.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:38.423+0000 7fa720bff700 1 -- 172.21.15.29:0/3193796953 wait complete. 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:30:38.441 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:30:38.442 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:30:38.442 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:30:38.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:38.442 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:30:38.490 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:30:38.490 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:30:38.490 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:30:38.490 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:30:38.491 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:30:38.492 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:30:38.492 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:30:38.492 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:30:38.492 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:30:38.492 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42190sec preferred_lft 42190sec 2023-10-07T09:30:38.492 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:30:38.493 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:30:38.494 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.495 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:30:38.495 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:30:38.495 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:30:38.495 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:30:38.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:38.496 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:30:38.598 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:30:38.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:38.601 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:30:38.650 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:30:38.651 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:30:38.651 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:30:38.651 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:30:38.651 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:30:38.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:38.652 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:30:38.652 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:30:38.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:30:38.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:30:38.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:30:38.653 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:30:38.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:38.825 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:30:38.872 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:30:38.872 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:30:38.873 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:30:38.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:38.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:30:38.944 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:30:38.944 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:30:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:30:38.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:30:38.991 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:30:39.076 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:30:39.076 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:30:39.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:30:39.217 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:30:39.218 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:30:39.218 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:30:39.232 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:30:39.233 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:30:39.233 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:30:39.233 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:30:39.233 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:30:39.255 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:30:39.255 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:30:39.255 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:30:39.255 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:30:39.255 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:30:39.256 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:30:39.256 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:30:39.256 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:30:39.261 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:30:39.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:30:39.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:30:39.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:30:39.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.536+0000 7f393e3ff700 1 Processor -- start 2023-10-07T09:30:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.536+0000 7f393e3ff700 1 -- start start 2023-10-07T09:30:39.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f393e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39381537c0 0x7f3938156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f393e3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3938159670 con 0x7f39380b04b0 2023-10-07T09:30:39.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f393e3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393815a1f0 con 0x7f39381533f0 2023-10-07T09:30:39.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f393e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393815a680 con 0x7f39381537c0 2023-10-07T09:30:39.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f39377fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f39380b04b0 0x7f3938157240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41030/0 (socket says 172.21.15.29:41030) 2023-10-07T09:30:39.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f39377fe700 1 -- 172.21.15.29:0/3140101249 learned_addr learned my addr 172.21.15.29:0/3140101249 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:39.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f393c99c700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39381537c0 0x7f3938156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.538+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608473190 0 0) 0x7f3938159670 con 0x7f39380b04b0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3920004680 con 0x7f39380b04b0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864742946 0 0) 0x7f393815a1f0 con 0x7f39381533f0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3938159670 con 0x7f39381533f0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944204094 0 0) 0x7f3920004680 con 0x7f39380b04b0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39381537c0 msgr2=0x7f3938156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39381537c0 0x7f3938156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 >> v1:172.21.15.144:6789/0 conn(0x7f39381533f0 legacy=0x7f39380afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39381598c0 con 0x7f39380b04b0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f392c001470 con 0x7f39380b04b0 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 >> v1:172.21.15.29:6789/0 conn(0x7f39380b04b0 legacy=0x7f3938157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3920004d30 0x7f3920005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3920005920 0x7f3920005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39200062d0 0x7f392000cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392000d190 con 0x7f3920004d30 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392000d300 con 0x7f39200062d0 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392000d4c0 con 0x7f3920005920 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f393c99c700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39381537c0 0x7f3938156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f3937fff700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3920004d30 0x7f3920005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f39377fe700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3920005920 0x7f3920005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.539+0000 7f393c99c700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39200062d0 0x7f392000cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3937fff700 1 -- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3920005920 msgr2=0x7f3920005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3937fff700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3920005920 0x7f3920005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3937fff700 1 -- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39200062d0 msgr2=0x7f392000cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3937fff700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39200062d0 0x7f392000cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3937fff700 1 -- 172.21.15.29:0/3140101249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920004680 con 0x7f3920004d30 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3937fff700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3920004d30 0x7f3920005120 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3928001a10 tx=0x7f3928007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deeec4749f901dfc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392800c5d0 con 0x7f3920004d30 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392800c730 con 0x7f3920004d30 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.540+0000 7f3936ffd700 1 -- 172.21.15.29:0/3140101249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392800cad0 con 0x7f3920004d30 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393e3ff700 1 -- 172.21.15.29:0/3140101249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3920004d30 msgr2=0x7f3920005120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3140101249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3920004d30 0x7f3920005120 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3928001a10 tx=0x7f3928007d30 comp rx=0 tx=0).stop 2023-10-07T09:30:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393c99c700 1 -- 172.21.15.29:0/3140101249 reap_dead start 2023-10-07T09:30:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393e3ff700 1 -- 172.21.15.29:0/3140101249 shutdown_connections 2023-10-07T09:30:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393e3ff700 1 -- 172.21.15.29:0/3140101249 >> 172.21.15.29:0/3140101249 conn(0x7f39380680a0 msgr2=0x7f39380cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393e3ff700 1 -- 172.21.15.29:0/3140101249 shutdown_connections 2023-10-07T09:30:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.541+0000 7f393e3ff700 1 -- 172.21.15.29:0/3140101249 wait complete. 2023-10-07T09:30:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.542+0000 7f393e3ff700 1 Processor -- start 2023-10-07T09:30:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.542+0000 7f393e3ff700 1 -- start start 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.543+0000 7f393e3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3920004d30 0x7f393814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.543+0000 7f393e3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 0x7f393816a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.543+0000 7f393e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f393814f1c0 0x7f39380a5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.543+0000 7f393e3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393816bba0 con 0x7f39381698b0 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.543+0000 7f393e3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393814c2f0 con 0x7f3920004d30 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.543+0000 7f393e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393816af90 con 0x7f393814f1c0 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 0x7f393816a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 0x7f393816a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41464/0 (socket says 172.21.15.29:41464) 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 -- 172.21.15.29:0/3144905226 learned_addr learned my addr 172.21.15.29:0/3144905226 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f393c99c700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f393814f1c0 0x7f39380a5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f3937fff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3920004d30 0x7f393814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 -- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f393814f1c0 msgr2=0x7f39380a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f393814f1c0 0x7f39380a5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 -- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3920004d30 msgr2=0x7f393814bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3920004d30 0x7f393814bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 -- 172.21.15.29:0/3144905226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393816b270 con 0x7f39381698b0 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f3937fff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3920004d30 0x7f393814bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.544+0000 7f39377fe700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 0x7f393816a710 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f392c000fd0 tx=0x7f393815a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.545+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c018030 con 0x7f39381698b0 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.545+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c00f690 con 0x7f39381698b0 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.545+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c00fa50 con 0x7f39381698b0 2023-10-07T09:30:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.545+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393816b550 con 0x7f39381698b0 2023-10-07T09:30:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.545+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393816ac50 con 0x7f39381698b0 2023-10-07T09:30:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.546+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393816b9f0 con 0x7f39381698b0 2023-10-07T09:30:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.547+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f392c01b070 con 0x7f39381698b0 2023-10-07T09:30:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.548+0000 7f391ffff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3924066f30 0x7f39240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.548+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f392c019070 con 0x7f39381698b0 2023-10-07T09:30:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.548+0000 7f3937fff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3924066f30 0x7f39240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.549+0000 7f3937fff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3924066f30 0x7f39240693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3928008fd0 tx=0x7f3928015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:39.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.552+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f392c0514f0 con 0x7f39381698b0 2023-10-07T09:30:39.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.782+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f393804c3c0 con 0x7f3924066f30 2023-10-07T09:30:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.784+0000 7f391ffff700 1 -- 172.21.15.29:0/3144905226 <== mgr.4100 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f393804c3c0 con 0x7f3924066f30 2023-10-07T09:30:39.784 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:30:39.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:30:39.784 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:30:39.784 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:30:39.784 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:30:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3924066f30 msgr2=0x7f39240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3924066f30 0x7f39240693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3928008fd0 tx=0x7f3928015000 comp rx=0 tx=0).stop 2023-10-07T09:30:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 msgr2=0x7f393816a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 0x7f393816a710 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f392c000fd0 tx=0x7f393815a160 comp rx=0 tx=0).stop 2023-10-07T09:30:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 shutdown_connections 2023-10-07T09:30:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3924066f30 0x7f39240693e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f393814f1c0 0x7f39380a5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3920004d30 0x7f393814bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 --2- 172.21.15.29:0/3144905226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39381698b0 0x7f393816a710 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 >> 172.21.15.29:0/3144905226 conn(0x7f39380680a0 msgr2=0x7f393806d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 shutdown_connections 2023-10-07T09:30:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:39.786+0000 7f393e3ff700 1 -- 172.21.15.29:0/3144905226 wait complete. 2023-10-07T09:30:39.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:30:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.001+0000 7fd2943ff700 1 Processor -- start 2023-10-07T09:30:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.001+0000 7fd2943ff700 1 -- start start 2023-10-07T09:30:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.002+0000 7fd2943ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd28c159750 0x7fd28c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.002+0000 7fd2943ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c0ce8a0 0x7fd28c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.002+0000 7fd2943ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c0cbf20 con 0x7fd28c159b20 2023-10-07T09:30:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.002+0000 7fd2943ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28c0cc200 con 0x7fd28c159750 2023-10-07T09:30:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.002+0000 7fd2943ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0cc4e0 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd28c159750 0x7fd28c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd28c159750 0x7fd28c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41474/0 (socket says 172.21.15.29:41474) 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 -- 172.21.15.29:0/1942625585 learned_addr learned my addr 172.21.15.29:0/1942625585 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29299c700 1 --2- 172.21.15.29:0/1942625585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c0ce8a0 0x7fd28c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 -- 172.21.15.29:0/1942625585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c0ce8a0 msgr2=0x7fd28c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 --2- 172.21.15.29:0/1942625585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c0ce8a0 0x7fd28c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 -- 172.21.15.29:0/1942625585 >> v1:172.21.15.144:6789/0 conn(0x7fd28c159b20 legacy=0x7fd28c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 -- 172.21.15.29:0/1942625585 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c0cc7c0 con 0x7fd28c159750 2023-10-07T09:30:40.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.003+0000 7fd29219b700 1 --2- 172.21.15.29:0/1942625585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c159750 0x7fd28c0d10a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd288009990 tx=0x7fd288009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0a34451db412a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:40.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd291199700 1 -- 172.21.15.29:0/1942625585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288005c10 con 0x7fd28c159750 2023-10-07T09:30:40.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd291199700 1 -- 172.21.15.29:0/1942625585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd288005d70 con 0x7fd28c159750 2023-10-07T09:30:40.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd291199700 1 -- 172.21.15.29:0/1942625585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28800b880 con 0x7fd28c159750 2023-10-07T09:30:40.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 -- 172.21.15.29:0/1942625585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c159750 msgr2=0x7fd28c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:40.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 --2- 172.21.15.29:0/1942625585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c159750 0x7fd28c0d10a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd288009990 tx=0x7fd288009220 comp rx=0 tx=0).stop 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 -- 172.21.15.29:0/1942625585 shutdown_connections 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 --2- 172.21.15.29:0/1942625585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c0ce8a0 0x7fd28c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 --2- 172.21.15.29:0/1942625585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c159750 0x7fd28c0d10a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 -- 172.21.15.29:0/1942625585 >> 172.21.15.29:0/1942625585 conn(0x7fd28c063a10 msgr2=0x7fd28c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 -- 172.21.15.29:0/1942625585 shutdown_connections 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.004+0000 7fd2943ff700 1 -- 172.21.15.29:0/1942625585 wait complete. 2023-10-07T09:30:40.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.006+0000 7fd2943ff700 1 Processor -- start 2023-10-07T09:30:40.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.007+0000 7fd2943ff700 1 -- start start 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd2943ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 0x7fd28c15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd2943ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd28c159750 0x7fd28c13f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd2943ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c159b20 0x7fd28c16d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd2943ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c15a240 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd2943ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c16de60 con 0x7fd28c159750 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd2943ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c15cbd0 con 0x7fd28c159b20 2023-10-07T09:30:40.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29219b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 0x7fd28c15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:40.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29219b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 0x7fd28c15b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41480/0 (socket says 172.21.15.29:41480) 2023-10-07T09:30:40.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29219b700 1 -- 172.21.15.29:0/3177772002 learned_addr learned my addr 172.21.15.29:0/3177772002 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29219b700 1 -- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c159b20 msgr2=0x7fd28c16d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29219b700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c159b20 0x7fd28c16d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29219b700 1 -- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd28c159750 msgr2=0x7fd28c13f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.008+0000 7fd29199a700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd28c159750 0x7fd28c13f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd29219b700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd28c159750 0x7fd28c13f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd29219b700 1 -- 172.21.15.29:0/3177772002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd288009480 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd29199a700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd28c159750 0x7fd28c13f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:40.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd29219b700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 0x7fd28c15b950 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd28c088a40 tx=0x7fd288016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288011a20 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd288011b80 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28801b430 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c15bfc0 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.009+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c15c580 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.010+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c15ceb0 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.011+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd28801b820 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.012+0000 7fd2827fc700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd270066f30 0x7fd2700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:40.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.012+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fd288085f70 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.013+0000 7fd29199a700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd270066f30 0x7fd2700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:40.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.013+0000 7fd29199a700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd270066f30 0x7fd2700693e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd28c154b50 tx=0x7fd27c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.016+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd288053140 con 0x7fd28c0ce8a0 2023-10-07T09:30:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:40.276+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fd28c04c3c0 con 0x7fd28c0ce8a0 2023-10-07T09:30:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.111+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 6) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fd288053510 con 0x7fd28c0ce8a0 2023-10-07T09:30:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.113+0000 7fd2827fc700 1 -- 172.21.15.29:0/3177772002 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7fd288056bd0 con 0x7fd28c0ce8a0 2023-10-07T09:30:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd270066f30 msgr2=0x7fd2700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd270066f30 0x7fd2700693e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd28c154b50 tx=0x7fd27c009040 comp rx=0 tx=0).stop 2023-10-07T09:30:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 msgr2=0x7fd28c15b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 0x7fd28c15b950 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd28c088a40 tx=0x7fd288016000 comp rx=0 tx=0).stop 2023-10-07T09:30:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 shutdown_connections 2023-10-07T09:30:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd270066f30 0x7fd2700693e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd28c159b20 0x7fd28c16d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd28c0ce8a0 0x7fd28c15b950 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 --2- 172.21.15.29:0/3177772002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd28c159750 0x7fd28c13f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 >> 172.21.15.29:0/3177772002 conn(0x7fd28c063a10 msgr2=0x7fd28c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 shutdown_connections 2023-10-07T09:30:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.116+0000 7fd2943ff700 1 -- 172.21.15.29:0/3177772002 wait complete. 2023-10-07T09:30:41.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:30:41.322 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:41.321+0000 7f803796b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:30:41.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:41.325+0000 7f251ae731c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:30:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.346+0000 7f53189ff700 1 Processor -- start 2023-10-07T09:30:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.346+0000 7f53189ff700 1 -- start start 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.347+0000 7f53189ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f53140ce390 0x7f53140ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.347+0000 7f53189ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53141578a0 con 0x7f53140b3e60 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.347+0000 7f53189ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53140cc830 con 0x7f53140b00b0 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f53189ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5314157ce0 con 0x7f53140ce390 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f531259c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f53140b3e60 0x7f5314158b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41036/0 (socket says 172.21.15.29:41036) 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f531259c700 1 -- 172.21.15.29:0/2889075947 learned_addr learned my addr 172.21.15.29:0/2889075947 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:41.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f5311d9b700 1 --2- 172.21.15.29:0/2889075947 >> v2:172.21.15.144:3300/0 conn(0x7f53140ce390 0x7f53140ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f531159a700 1 -- 172.21.15.29:0/2889075947 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660646744 0 0) 0x7f53141578a0 con 0x7f53140b3e60 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f531159a700 1 -- 172.21.15.29:0/2889075947 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52fc003570 con 0x7f53140b3e60 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f5311d9b700 1 -- 172.21.15.29:0/2889075947 >> v1:172.21.15.144:6789/0 conn(0x7f53140b00b0 legacy=0x7f53140cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f5311d9b700 1 -- 172.21.15.29:0/2889075947 >> v1:172.21.15.29:6789/0 conn(0x7f53140b3e60 legacy=0x7f5314158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.348+0000 7f5311d9b700 1 -- 172.21.15.29:0/2889075947 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53140cee50 con 0x7f53140ce390 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.349+0000 7f5311d9b700 1 --2- 172.21.15.29:0/2889075947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140ce390 0x7f53140ce830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53080098e0 tx=0x7f53080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a46a0b63cdf575 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.349+0000 7f531159a700 1 -- 172.21.15.29:0/2889075947 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530800b640 con 0x7f53140ce390 2023-10-07T09:30:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.349+0000 7f531159a700 1 -- 172.21.15.29:0/2889075947 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530800b7a0 con 0x7f53140ce390 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.349+0000 7f531159a700 1 -- 172.21.15.29:0/2889075947 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530800bb40 con 0x7f53140ce390 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.349+0000 7f53189ff700 1 -- 172.21.15.29:0/2889075947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140ce390 msgr2=0x7f53140ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.349+0000 7f53189ff700 1 --2- 172.21.15.29:0/2889075947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140ce390 0x7f53140ce830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53080098e0 tx=0x7f53080091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.350+0000 7f53189ff700 1 -- 172.21.15.29:0/2889075947 shutdown_connections 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.350+0000 7f53189ff700 1 --2- 172.21.15.29:0/2889075947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140ce390 0x7f53140ce830 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.350+0000 7f53189ff700 1 -- 172.21.15.29:0/2889075947 >> 172.21.15.29:0/2889075947 conn(0x7f5314067610 msgr2=0x7f5314067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.350+0000 7f53189ff700 1 -- 172.21.15.29:0/2889075947 shutdown_connections 2023-10-07T09:30:41.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.350+0000 7f53189ff700 1 -- 172.21.15.29:0/2889075947 wait complete. 2023-10-07T09:30:41.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.351+0000 7f53189ff700 1 Processor -- start 2023-10-07T09:30:41.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.351+0000 7f53189ff700 1 -- start start 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.352+0000 7f53189ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 0x7f5314145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.352+0000 7f53189ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140b3e60 0x7f53141696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.352+0000 7f53189ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53140ce390 0x7f5314153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.352+0000 7f53189ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314155b60 con 0x7f53140b00b0 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.352+0000 7f53189ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314146450 con 0x7f53140b3e60 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.352+0000 7f53189ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53140c7aa0 con 0x7f53140ce390 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 0x7f5314145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 0x7f5314145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41516/0 (socket says 172.21.15.29:41516) 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 -- 172.21.15.29:0/955148992 learned_addr learned my addr 172.21.15.29:0/955148992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f5311d9b700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140b3e60 0x7f53141696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f5312d9d700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53140ce390 0x7f5314153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 -- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53140ce390 msgr2=0x7f5314153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53140ce390 0x7f5314153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 -- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140b3e60 msgr2=0x7f53141696a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140b3e60 0x7f53141696a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 -- 172.21.15.29:0/955148992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5308009420 con 0x7f53140b00b0 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.353+0000 7f531259c700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 0x7f5314145f10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f530400c790 tx=0x7f5304007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:41.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.354+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304005c00 con 0x7f53140b00b0 2023-10-07T09:30:41.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.354+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5304005d60 con 0x7f53140b00b0 2023-10-07T09:30:41.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.354+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5314154730 con 0x7f53140b00b0 2023-10-07T09:30:41.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.354+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040072e0 con 0x7f53140b00b0 2023-10-07T09:30:41.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.354+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53140c7730 con 0x7f53140b00b0 2023-10-07T09:30:41.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.355+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531404c3c0 con 0x7f53140b00b0 2023-10-07T09:30:41.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.357+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f5304012400 con 0x7f53140b00b0 2023-10-07T09:30:41.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.358+0000 7f53027fc700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 0x7f52f4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.358+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5304083980 con 0x7f53140b00b0 2023-10-07T09:30:41.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.358+0000 7f5311d9b700 1 -- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 msgr2=0x7f52f4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:41.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.358+0000 7f5311d9b700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 0x7f52f4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:30:41.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.362+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f530404b5a0 con 0x7f53140b00b0 2023-10-07T09:30:41.499 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:41.498+0000 7f803796b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:30:41.517 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:41.517+0000 7f251ae731c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:30:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.559+0000 7f5311d9b700 1 -- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 msgr2=0x7f52f4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.559+0000 7f5311d9b700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 0x7f52f4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:30:41.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.600+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f53140cf130 con 0x7f53140b00b0 2023-10-07T09:30:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.604+0000 7f53027fc700 1 -- 172.21.15.29:0/955148992 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5304017020 con 0x7f53140b00b0 2023-10-07T09:30:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.606+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 msgr2=0x7f52f4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:30:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.606+0000 7f53189ff700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 0x7f52f4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.606+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 msgr2=0x7f5314145f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.606+0000 7f53189ff700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 0x7f5314145f10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f530400c790 tx=0x7f5304007dd0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 shutdown_connections 2023-10-07T09:30:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f52f4066f80 0x7f52f4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53140ce390 0x7f5314153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53140b00b0 0x7f5314145f10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 --2- 172.21.15.29:0/955148992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53140b3e60 0x7f53141696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 >> 172.21.15.29:0/955148992 conn(0x7f5314067610 msgr2=0x7f53140704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 shutdown_connections 2023-10-07T09:30:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.607+0000 7f53189ff700 1 -- 172.21.15.29:0/955148992 wait complete. 2023-10-07T09:30:41.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:30:41.812 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:41.811+0000 7f803796b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.826+0000 7fbbcbdff700 1 Processor -- start 2023-10-07T09:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.826+0000 7fbbcbdff700 1 -- start start 2023-10-07T09:30:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbcbdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbbc415a960 0x7fbbc4158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbcbdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc40ae8a0 0x7fbbc40b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbcbdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc40acb40 con 0x7fbbc415ad30 2023-10-07T09:30:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbcbdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc4158c60 con 0x7fbbc415a960 2023-10-07T09:30:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbcbdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40abf30 con 0x7fbbc40ae8a0 2023-10-07T09:30:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbc9b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbbc415a960 0x7fbbc4158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbc9b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbbc415a960 0x7fbbc4158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41522/0 (socket says 172.21.15.29:41522) 2023-10-07T09:30:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbc9b9b700 1 -- 172.21.15.29:0/1937334453 learned_addr learned my addr 172.21.15.29:0/1937334453 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.828+0000 7fbbca39c700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc40ae8a0 0x7fbbc40b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc9b9b700 1 -- 172.21.15.29:0/1937334453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc40ae8a0 msgr2=0x7fbbc40b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:41.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc9b9b700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc40ae8a0 0x7fbbc40b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc9b9b700 1 -- 172.21.15.29:0/1937334453 >> v1:172.21.15.144:6789/0 conn(0x7fbbc415ad30 legacy=0x7fbbc40ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc9b9b700 1 -- 172.21.15.29:0/1937334453 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc40ac210 con 0x7fbbc415a960 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbca39c700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc40ae8a0 0x7fbbc40b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc9b9b700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc4158e20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00099b0 tx=0x7fbbc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e161f99f49123e48 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc8b99700 1 -- 172.21.15.29:0/1937334453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc000b860 con 0x7fbbc415a960 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc8b99700 1 -- 172.21.15.29:0/1937334453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc000b9c0 con 0x7fbbc415a960 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.829+0000 7fbbc8b99700 1 -- 172.21.15.29:0/1937334453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc000bd60 con 0x7fbbc415a960 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 -- 172.21.15.29:0/1937334453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 msgr2=0x7fbbc4158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc4158e20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00099b0 tx=0x7fbbc0009250 comp rx=0 tx=0).stop 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 -- 172.21.15.29:0/1937334453 shutdown_connections 2023-10-07T09:30:41.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc40ae8a0 0x7fbbc40b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/1937334453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc4158e20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 -- 172.21.15.29:0/1937334453 >> 172.21.15.29:0/1937334453 conn(0x7fbbc4067610 msgr2=0x7fbbc4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 -- 172.21.15.29:0/1937334453 shutdown_connections 2023-10-07T09:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.830+0000 7fbbcbdff700 1 -- 172.21.15.29:0/1937334453 wait complete. 2023-10-07T09:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.832+0000 7fbbcbdff700 1 Processor -- start 2023-10-07T09:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.832+0000 7fbbcbdff700 1 -- start start 2023-10-07T09:30:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbcbdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbc40ae8a0 0x7fbbc4141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbcbdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc40cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbcbdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc415ad30 0x7fbbc40cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbcbdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4156fa0 con 0x7fbbc415a960 2023-10-07T09:30:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbcbdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4156390 con 0x7fbbc40ae8a0 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbcbdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4156670 con 0x7fbbc415ad30 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbc939a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc40cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbc939a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc40cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41530/0 (socket says 172.21.15.29:41530) 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbc939a700 1 -- 172.21.15.29:0/301836603 learned_addr learned my addr 172.21.15.29:0/301836603 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbc9b9b700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbc40ae8a0 0x7fbbc4141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbca39c700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc415ad30 0x7fbbc40cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbc939a700 1 -- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc415ad30 msgr2=0x7fbbc40cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.833+0000 7fbbc939a700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc415ad30 0x7fbbc40cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbc939a700 1 -- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbc40ae8a0 msgr2=0x7fbbc4141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbc939a700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbc40ae8a0 0x7fbbc4141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbc939a700 1 -- 172.21.15.29:0/301836603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc00094f0 con 0x7fbbc415a960 2023-10-07T09:30:41.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbc939a700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc40cbb60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c590 tx=0x7fbbb4007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4011590 con 0x7fbbc415a960 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb4011710 con 0x7fbbc415a960 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4011ad0 con 0x7fbbc415a960 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc4156950 con 0x7fbbc415a960 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.834+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41420b0 con 0x7fbbc415a960 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.836+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc404c3c0 con 0x7fbbc415a960 2023-10-07T09:30:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.836+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fbbb401a360 con 0x7fbbc415a960 2023-10-07T09:30:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.837+0000 7fbbb9ffb700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 0x7fbba8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.837+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbbb4083460 con 0x7fbbc415a960 2023-10-07T09:30:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.837+0000 7fbbc9b9b700 1 -- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 msgr2=0x7fbba8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.837+0000 7fbbc9b9b700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 0x7fbba8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:30:41.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:41.841+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbbb4050630 con 0x7fbbc415a960 2023-10-07T09:30:41.854 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:41.854+0000 7f251ae731c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:30:42.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.038+0000 7fbbc9b9b700 1 -- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 msgr2=0x7fbba8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:42.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.038+0000 7fbbc9b9b700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 0x7fbba8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:30:42.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.047+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fbbc40ac4f0 con 0x7fbbc415a960 2023-10-07T09:30:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.050+0000 7fbbb9ffb700 1 -- 172.21.15.29:0/301836603 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbbb4053c50 con 0x7fbbc415a960 2023-10-07T09:30:42.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.054+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 msgr2=0x7fbba8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:30:42.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.054+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 0x7fbba8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.054+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 msgr2=0x7fbbc40cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.054+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc40cbb60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c590 tx=0x7fbbb4007f50 comp rx=0 tx=0).stop 2023-10-07T09:30:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 shutdown_connections 2023-10-07T09:30:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbba8066f80 0x7fbba8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbc415ad30 0x7fbbc40cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbc40ae8a0 0x7fbbc4141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 --2- 172.21.15.29:0/301836603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc415a960 0x7fbbc40cbb60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 >> 172.21.15.29:0/301836603 conn(0x7fbbc4067610 msgr2=0x7fbbc4158340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 shutdown_connections 2023-10-07T09:30:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.055+0000 7fbbcbdff700 1 -- 172.21.15.29:0/301836603 wait complete. 2023-10-07T09:30:42.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:42.081 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T09:30:42.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:42.134 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-07T09:30:42.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir0 1M 2023-10-07T09:30:42.335 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:42.334+0000 7f803796b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:30:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.388+0000 7f08239ff700 1 Processor -- start 2023-10-07T09:30:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.389+0000 7f08239ff700 1 -- start start 2023-10-07T09:30:42.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f08239ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f081c157a20 0x7f081c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:42.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f08239ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081c0aeae0 con 0x7f081c157650 2023-10-07T09:30:42.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f08239ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081c0ca430 con 0x7f081c0cb4d0 2023-10-07T09:30:42.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f08239ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081c0ca710 con 0x7f081c157a20 2023-10-07T09:30:42.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f082179b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f081c157650 0x7f081c0ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41038/0 (socket says 172.21.15.29:41038) 2023-10-07T09:30:42.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f082179b700 1 -- 172.21.15.29:0/2165679716 learned_addr learned my addr 172.21.15.29:0/2165679716 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:42.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.390+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2165679716 >> v2:172.21.15.144:3300/0 conn(0x7f081c157a20 0x7f081c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0813fff700 1 -- 172.21.15.29:0/2165679716 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828820905 0 0) 0x7f081c0aeae0 con 0x7f081c157650 2023-10-07T09:30:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0813fff700 1 -- 172.21.15.29:0/2165679716 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0808003570 con 0x7f081c157650 2023-10-07T09:30:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0813fff700 1 -- 172.21.15.29:0/2165679716 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643721060 0 0) 0x7f081c0ca430 con 0x7f081c0cb4d0 2023-10-07T09:30:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0820f9a700 1 -- 172.21.15.29:0/2165679716 >> v1:172.21.15.144:6789/0 conn(0x7f081c0cb4d0 legacy=0x7f081c0cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0820f9a700 1 -- 172.21.15.29:0/2165679716 >> v1:172.21.15.29:6789/0 conn(0x7f081c157650 legacy=0x7f081c0ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0820f9a700 1 -- 172.21.15.29:0/2165679716 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c0ca9f0 con 0x7f081c157a20 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2165679716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c157a20 0x7f081c0b1290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f080c0098e0 tx=0x7f080c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=971630c0075ebe05 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.391+0000 7f0813fff700 1 -- 172.21.15.29:0/2165679716 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c00b640 con 0x7f081c157a20 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f0813fff700 1 -- 172.21.15.29:0/2165679716 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080c00b7a0 con 0x7f081c157a20 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f0813fff700 1 -- 172.21.15.29:0/2165679716 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c00bb20 con 0x7f081c157a20 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 -- 172.21.15.29:0/2165679716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c157a20 msgr2=0x7f081c0b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 --2- 172.21.15.29:0/2165679716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c157a20 0x7f081c0b1290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f080c0098e0 tx=0x7f080c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 -- 172.21.15.29:0/2165679716 shutdown_connections 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 --2- 172.21.15.29:0/2165679716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c157a20 0x7f081c0b1290 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 -- 172.21.15.29:0/2165679716 >> 172.21.15.29:0/2165679716 conn(0x7f081c067610 msgr2=0x7f081c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 -- 172.21.15.29:0/2165679716 shutdown_connections 2023-10-07T09:30:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.392+0000 7f08239ff700 1 -- 172.21.15.29:0/2165679716 wait complete. 2023-10-07T09:30:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.394+0000 7f08239ff700 1 Processor -- start 2023-10-07T09:30:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.394+0000 7f08239ff700 1 -- start start 2023-10-07T09:30:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f08239ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 0x7f081c14fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f08239ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f081c157650 0x7f081c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f08239ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f081c157a20 0x7f081c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f08239ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c158b80 con 0x7f081c157a20 2023-10-07T09:30:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f08239ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c0b0510 con 0x7f081c0cb4d0 2023-10-07T09:30:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f08239ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081c0b07f0 con 0x7f081c157650 2023-10-07T09:30:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.395+0000 7f082179b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 0x7f081c14fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 0x7f081c14fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:49554/0 (socket says 172.21.15.29:49554) 2023-10-07T09:30:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 -- 172.21.15.29:0/2245830083 learned_addr learned my addr 172.21.15.29:0/2245830083 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f0821f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f081c157a20 0x7f081c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f081c157650 0x7f081c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f081c157650 msgr2=0x7f081c1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f081c157650 0x7f081c1526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f081c157a20 msgr2=0x7f081c154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f081c157a20 0x7f081c154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 -- 172.21.15.29:0/2245830083 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c009420 con 0x7f081c0cb4d0 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f0821f9c700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f081c157a20 0x7f081c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.396+0000 7f082179b700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 0x7f081c14fa00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f08180011e0 tx=0x7f0818007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.397+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818005b30 con 0x7f081c0cb4d0 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.397+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0818005c90 con 0x7f081c0cb4d0 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.397+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081c0b0ad0 con 0x7f081c0cb4d0 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.397+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818007300 con 0x7f081c0cb4d0 2023-10-07T09:30:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.397+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081c14ff40 con 0x7f081c0cb4d0 2023-10-07T09:30:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.398+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081c158940 con 0x7f081c0cb4d0 2023-10-07T09:30:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.399+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 6) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f0818007670 con 0x7f081c0cb4d0 2023-10-07T09:30:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.400+0000 7f08117fa700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.400+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0818083a50 con 0x7f081c0cb4d0 2023-10-07T09:30:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.400+0000 7f0820f9a700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f0800069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.400+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:30:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.405+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0818050c20 con 0x7f081c0cb4d0 2023-10-07T09:30:42.434 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:42.434+0000 7f251ae731c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:30:42.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:42.506+0000 7f803796b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:30:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.601+0000 7f0820f9a700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f0800069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.601+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:30:42.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:42.610+0000 7f251ae731c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:30:42.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:42.644+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f081c0718b0 con 0x7f0800066f80 2023-10-07T09:30:42.915 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:42.914+0000 7f803796b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:30:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:43.002+0000 7f0820f9a700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f0800069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:43.002+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:30:43.035 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:43.035+0000 7f251ae731c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:30:43.687 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:43.685+0000 7f803796b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:30:43.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:43.772+0000 7f803796b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:30:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:43.803+0000 7f0820f9a700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f0800069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:43.803+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:30:43.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:43.836+0000 7f251ae731c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:30:43.926 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:43.926+0000 7f251ae731c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:30:43.927 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:43.926+0000 7f803796b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:30:44.013 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.011+0000 7f803796b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:30:44.087 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.087+0000 7f251ae731c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:30:44.176 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.176+0000 7f251ae731c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:30:44.227 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.226+0000 7f803796b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:30:44.321 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.320+0000 7f803796b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:30:44.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.394+0000 7f251ae731c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:30:44.414 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.413+0000 7f803796b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:30:44.492 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.492+0000 7f251ae731c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:30:44.508 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.507+0000 7f803796b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:30:44.590 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.590+0000 7f251ae731c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:30:44.688 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.688+0000 7f251ae731c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:30:44.693 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.692+0000 7f803796b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:30:44.788 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:44.787+0000 7f803796b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:30:44.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.881+0000 7f251ae731c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:30:44.981 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:44.981+0000 7f251ae731c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:30:45.125 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:45.124+0000 7f803796b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:30:45.212 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:45.211+0000 7f803796b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:30:45.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:45.326+0000 7f251ae731c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:30:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:45.405+0000 7f0820f9a700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f0800069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:30:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:45.405+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:30:45.416 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:45.417+0000 7f251ae731c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:30:45.743 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:45.742+0000 7f803796b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:30:45.925 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:45.924+0000 7f803796b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:30:45.983 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:45.983+0000 7f251ae731c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:30:46.172 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:46.172+0000 7f251ae731c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:30:46.298 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:46.296+0000 7f803796b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:30:46.550 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:46.550+0000 7f251ae731c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:30:46.710 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:46.709+0000 7f803796b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:30:46.886 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:46.885+0000 7f803796b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:30:46.991 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:46.991+0000 7f251ae731c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:30:47.069 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.068+0000 7f803796b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:30:47.173 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:47.173+0000 7f251ae731c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:30:47.175 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.174+0000 7f803796b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:30:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:47.186+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 7) v1 ==== 84468+0+0 (secure 0 0 0) 0x7f0818050ff0 con 0x7f081c0cb4d0 2023-10-07T09:30:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:47.187+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f0800069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:30:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:47.187+0000 7f08117fa700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f0800069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fab776700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fab776700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fab776700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fab776700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fab776700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fa7f6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fa7f6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fa7f6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fa7f6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.312 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:30:47.310+0000 7f7fa7f6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:30:47.381 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:47.380+0000 7f251ae731c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:30:47.492 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:30:47.491+0000 7f251ae731c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:30:47.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:30:47.808 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:30:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.192+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mgrmap(e 8) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f08180703a0 con 0x7f081c0cb4d0 2023-10-07T09:30:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.192+0000 7f08117fa700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f08000693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.192+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f081c0718b0 con 0x7f0800066f80 2023-10-07T09:30:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.194+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f08000693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.199+0000 7f0820f9a700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f08000693b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f081c0685c0 tx=0x7f080c017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.276+0000 7f08117fa700 1 -- 172.21.15.29:0/2245830083 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f081c0718b0 con 0x7f0800066f80 2023-10-07T09:30:48.278 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-07T09:30:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 msgr2=0x7f08000693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f08000693b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f081c0685c0 tx=0x7f080c017000 comp rx=0 tx=0).stop 2023-10-07T09:30:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 msgr2=0x7f081c14fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 0x7f081c14fa00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f08180011e0 tx=0x7f0818007df0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 shutdown_connections 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f0800066f80 0x7f08000693b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f081c157a20 0x7f081c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f081c0cb4d0 0x7f081c14fa00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 --2- 172.21.15.29:0/2245830083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f081c157650 0x7f081c1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 >> 172.21.15.29:0/2245830083 conn(0x7f081c067610 msgr2=0x7f081c0ac810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 shutdown_connections 2023-10-07T09:30:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.279+0000 7f08239ff700 1 -- 172.21.15.29:0/2245830083 wait complete. 2023-10-07T09:30:48.298 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-07T09:30:48.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:48.299 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-07T09:30:48.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir1 1M 2023-10-07T09:30:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.547+0000 7f02e01ff700 1 Processor -- start 2023-10-07T09:30:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.547+0000 7f02e01ff700 1 -- start start 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.548+0000 7f02e01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.548+0000 7f02e01ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02d80b09e0 0x7f02d80c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.548+0000 7f02e01ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f02d80c8a20 0x7f02d80cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.548+0000 7f02e01ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d80690f0 con 0x7f02d80c8a20 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.548+0000 7f02e01ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d80c4740 con 0x7f02d80b09e0 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.548+0000 7f02e01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80cf2e0 con 0x7f02d80b0610 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f02d80c8a20 0x7f02d80cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f02d80c8a20 0x7f02d80cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57710/0 (socket says 172.21.15.29:57710) 2023-10-07T09:30:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 -- 172.21.15.29:0/3583478482 learned_addr learned my addr 172.21.15.29:0/3583478482 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02ddf9b700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02dd79a700 1 --2- 172.21.15.29:0/3583478482 >> v2:172.21.15.144:3300/0 conn(0x7f02d80b09e0 0x7f02d80c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 -- 172.21.15.29:0/3583478482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 msgr2=0x7f02d80ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 -- 172.21.15.29:0/3583478482 >> v2:172.21.15.144:3300/0 conn(0x7f02d80b09e0 msgr2=0x7f02d80c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 --2- 172.21.15.29:0/3583478482 >> v2:172.21.15.144:3300/0 conn(0x7f02d80b09e0 0x7f02d80c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02de79c700 1 -- 172.21.15.29:0/3583478482 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d80687c0 con 0x7f02d80c8a20 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02ddf9b700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.549+0000 7f02dd79a700 1 --2- 172.21.15.29:0/3583478482 >> v2:172.21.15.144:3300/0 conn(0x7f02d80b09e0 0x7f02d80c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.550+0000 7f02de79c700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d80cc7e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f02d806edd0 tx=0x7f02d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d6d87c36e616c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.550+0000 7f02dcf99700 1 -- 172.21.15.29:0/3583478482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d000e070 con 0x7f02d80c8a20 2023-10-07T09:30:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.550+0000 7f02dcf99700 1 -- 172.21.15.29:0/3583478482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d000ba50 con 0x7f02d80c8a20 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.550+0000 7f02dcf99700 1 -- 172.21.15.29:0/3583478482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00092e0 con 0x7f02d80c8a20 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.550+0000 7f02e01ff700 1 -- 172.21.15.29:0/3583478482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 msgr2=0x7f02d80cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.550+0000 7f02e01ff700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d80cc7e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f02d806edd0 tx=0x7f02d0009840 comp rx=0 tx=0).stop 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 -- 172.21.15.29:0/3583478482 shutdown_connections 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d80cc7e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 --2- 172.21.15.29:0/3583478482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 --2- 172.21.15.29:0/3583478482 >> v2:172.21.15.144:3300/0 conn(0x7f02d80b09e0 0x7f02d80c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 -- 172.21.15.29:0/3583478482 >> 172.21.15.29:0/3583478482 conn(0x7f02d8063a10 msgr2=0x7f02d80ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 -- 172.21.15.29:0/3583478482 shutdown_connections 2023-10-07T09:30:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.551+0000 7f02e01ff700 1 -- 172.21.15.29:0/3583478482 wait complete. 2023-10-07T09:30:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.552+0000 7f02e01ff700 1 Processor -- start 2023-10-07T09:30:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.552+0000 7f02e01ff700 1 -- start start 2023-10-07T09:30:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02e01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02e01ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d80b09e0 0x7f02d8169560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02e01ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d8169aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02e01ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b2700 con 0x7f02d80c8a20 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02e01ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b2b90 con 0x7f02d80b09e0 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02e01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80b1af0 con 0x7f02d80b0610 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02de79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d8169aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02de79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d8169aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57712/0 (socket says 172.21.15.29:57712) 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.553+0000 7f02de79c700 1 -- 172.21.15.29:0/2036152285 learned_addr learned my addr 172.21.15.29:0/2036152285 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02ddf9b700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02dd79a700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d80b09e0 0x7f02d8169560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02de79c700 1 -- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 msgr2=0x7f02d80b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02de79c700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02de79c700 1 -- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d80b09e0 msgr2=0x7f02d8169560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02de79c700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d80b09e0 0x7f02d8169560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02de79c700 1 -- 172.21.15.29:0/2036152285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0009ae0 con 0x7f02d80c8a20 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02dd79a700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d80b09e0 0x7f02d8169560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02de79c700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d8169aa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f02d806ea20 tx=0x7f02d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00032f0 con 0x7f02d80c8a20 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d0008d90 con 0x7f02d80c8a20 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00266b0 con 0x7f02d80c8a20 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.554+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d80b1dd0 con 0x7f02d80c8a20 2023-10-07T09:30:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.555+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d80b2390 con 0x7f02d80c8a20 2023-10-07T09:30:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.556+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d80718b0 con 0x7f02d80c8a20 2023-10-07T09:30:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.556+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f02d0003040 con 0x7f02d80c8a20 2023-10-07T09:30:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.558+0000 7f02cdffb700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f02bc066dd0 0x7f02bc069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.558+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f02d0086240 con 0x7f02d80c8a20 2023-10-07T09:30:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.558+0000 7f02ddf9b700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f02bc066dd0 0x7f02bc069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.559+0000 7f02ddf9b700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f02bc066dd0 0x7f02bc069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f02d806cbf0 tx=0x7f02d4008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.562+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f02d0053590 con 0x7f02d80c8a20 2023-10-07T09:30:48.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.793+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f02d8068aa0 con 0x7f02bc066dd0 2023-10-07T09:30:48.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.855+0000 7f02cdffb700 1 -- 172.21.15.29:0/2036152285 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f02d8068aa0 con 0x7f02bc066dd0 2023-10-07T09:30:48.857 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-07T09:30:48.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f02bc066dd0 msgr2=0x7f02bc069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f02bc066dd0 0x7f02bc069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f02d806cbf0 tx=0x7f02d4008040 comp rx=0 tx=0).stop 2023-10-07T09:30:48.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 msgr2=0x7f02d8169aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:48.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d8169aa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f02d806ea20 tx=0x7f02d0018000 comp rx=0 tx=0).stop 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 shutdown_connections 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f02bc066dd0 0x7f02bc069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02d80c8a20 0x7f02d8169aa0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02d80b0610 0x7f02d80b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 --2- 172.21.15.29:0/2036152285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02d80b09e0 0x7f02d8169560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 >> 172.21.15.29:0/2036152285 conn(0x7f02d8063a10 msgr2=0x7f02d806cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 shutdown_connections 2023-10-07T09:30:48.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:48.858+0000 7f02e01ff700 1 -- 172.21.15.29:0/2036152285 wait complete. 2023-10-07T09:30:48.876 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-07T09:30:48.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:48.877 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-07T09:30:48.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir2 1M 2023-10-07T09:30:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.126+0000 7ff3cc9ff700 1 Processor -- start 2023-10-07T09:30:49.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.127+0000 7ff3cc9ff700 1 -- start start 2023-10-07T09:30:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3cc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80cf2c0 0x7ff3c8164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3cc9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80cc940 con 0x7ff3c80b2a20 2023-10-07T09:30:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3cc9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80b2df0 con 0x7ff3c80cb8c0 2023-10-07T09:30:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3cc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c80cbd30 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3c659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3c80b2a20 0x7ff3c80cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38906/0 (socket says 172.21.15.29:38906) 2023-10-07T09:30:49.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3c659c700 1 -- 172.21.15.29:0/3040322462 learned_addr learned my addr 172.21.15.29:0/3040322462 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:49.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.128+0000 7ff3c6d9d700 1 --2- 172.21.15.29:0/3040322462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80cf2c0 0x7ff3c8164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932580660 0 0) 0x7ff3c80cc940 con 0x7ff3c80b2a20 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b0003570 con 0x7ff3c80b2a20 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411706270 0 0) 0x7ff3c80b2df0 con 0x7ff3c80cb8c0 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c80cc940 con 0x7ff3c80cb8c0 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/3040322462 >> v1:172.21.15.144:6789/0 conn(0x7ff3c80cb8c0 legacy=0x7ff3c80cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/3040322462 >> v1:172.21.15.29:6789/0 conn(0x7ff3c80b2a20 legacy=0x7ff3c80cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1454623493 0 0) 0x7ff3b0003570 con 0x7ff3c80b2a20 2023-10-07T09:30:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/3040322462 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c80cc010 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.129+0000 7ff3c6d9d700 1 --2- 172.21.15.29:0/3040322462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80cf2c0 0x7ff3c8164330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009e90 tx=0x7ff3c0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23bae6d4f8266085 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000e070 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000b870 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3c559a700 1 -- 172.21.15.29:0/3040322462 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b80015f0 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/3040322462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80cf2c0 msgr2=0x7ff3c8164330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/3040322462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80cf2c0 0x7ff3c8164330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009e90 tx=0x7ff3c0009780 comp rx=0 tx=0).stop 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/3040322462 shutdown_connections 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/3040322462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80cf2c0 0x7ff3c8164330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/3040322462 >> 172.21.15.29:0/3040322462 conn(0x7ff3c8099820 msgr2=0x7ff3c8063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.130+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/3040322462 shutdown_connections 2023-10-07T09:30:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.131+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/3040322462 wait complete. 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.132+0000 7ff3cc9ff700 1 Processor -- start 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.132+0000 7ff3cc9ff700 1 -- start start 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3cc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80b2a20 0x7ff3c80b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3cc9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3c80cb8c0 0x7ff3c8156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3cc9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 0x7ff3c816e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3cc9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8157500 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3cc9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c816e670 con 0x7ff3c80cb8c0 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3cc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c80b5820 con 0x7ff3c80b2a20 2023-10-07T09:30:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3c6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 0x7ff3c816e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3c6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 0x7ff3c816e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57720/0 (socket says 172.21.15.29:57720) 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/830263381 learned_addr learned my addr 172.21.15.29:0/830263381 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3c659c700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80b2a20 0x7ff3c80b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.133+0000 7ff3c5d9b700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3c80cb8c0 0x7ff3c8156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80b2a20 msgr2=0x7ff3c80b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3c6d9d700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80b2a20 0x7ff3c80b5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3c80cb8c0 msgr2=0x7ff3c8156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3c6d9d700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3c80cb8c0 0x7ff3c8156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3c6d9d700 1 -- 172.21.15.29:0/830263381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0009a20 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3c6d9d700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 0x7ff3c816e130 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00097c0 tx=0x7ff3c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0012c40 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c0012dc0 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c816a220 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.134+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c001e6a0 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.135+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c816a7e0 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.136+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c80718b0 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.136+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 84595+0+0 (secure 0 0 0) 0x7ff3c001ea90 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.137+0000 7ff3b67fc700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff3a8066e00 0x7ff3a80692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.138+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ff3c0086270 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.138+0000 7ff3c659c700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff3a8066e00 0x7ff3a80692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.139+0000 7ff3c659c700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff3a8066e00 0x7ff3a80692b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff3c815ec30 tx=0x7ff3b8001770 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:49.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.141+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3c00535c0 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.195+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7ff3c0053990 con 0x7ff3c80cf2c0 2023-10-07T09:30:49.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.371+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3c80cc2f0 con 0x7ff3a8066e00 2023-10-07T09:30:49.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.434+0000 7ff3b67fc700 1 -- 172.21.15.29:0/830263381 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ff3c80cc2f0 con 0x7ff3a8066e00 2023-10-07T09:30:49.435 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff3a8066e00 msgr2=0x7ff3a80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff3a8066e00 0x7ff3a80692b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff3c815ec30 tx=0x7ff3b8001770 comp rx=0 tx=0).stop 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 msgr2=0x7ff3c816e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 0x7ff3c816e130 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00097c0 tx=0x7ff3c0017000 comp rx=0 tx=0).stop 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 shutdown_connections 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff3a8066e00 0x7ff3a80692b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c80cf2c0 0x7ff3c816e130 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3c80b2a20 0x7ff3c80b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 --2- 172.21.15.29:0/830263381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3c80cb8c0 0x7ff3c8156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.436+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 >> 172.21.15.29:0/830263381 conn(0x7ff3c8099820 msgr2=0x7ff3c80c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.437+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 shutdown_connections 2023-10-07T09:30:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.437+0000 7ff3cc9ff700 1 -- 172.21.15.29:0/830263381 wait complete. 2023-10-07T09:30:49.455 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-07T09:30:49.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:49.455 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-07T09:30:49.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir3 1M 2023-10-07T09:30:49.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.702+0000 7fb6937ff700 1 Processor -- start 2023-10-07T09:30:49.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.702+0000 7fb6937ff700 1 -- start start 2023-10-07T09:30:49.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb6937ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb68c0b0610 0x7fb68c0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb6937ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb68c0b09e0 0x7fb68c0c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb6937ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb6937ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c0690f0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb6937ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c0c4740 con 0x7fb68c0b0610 2023-10-07T09:30:49.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb6937ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0cf2e0 con 0x7fb68c0c8a20 2023-10-07T09:30:49.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb690d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb68c0b09e0 0x7fb68c0c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb690d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb68c0b09e0 0x7fb68c0c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57726/0 (socket says 172.21.15.29:57726) 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb690d9a700 1 -- 172.21.15.29:0/2308230133 learned_addr learned my addr 172.21.15.29:0/2308230133 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb691d9c700 1 --2- 172.21.15.29:0/2308230133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.704+0000 7fb69159b700 1 --2- 172.21.15.29:0/2308230133 >> v2:172.21.15.144:3300/0 conn(0x7fb68c0b0610 0x7fb68c0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb690d9a700 1 -- 172.21.15.29:0/2308230133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 msgr2=0x7fb68c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb690d9a700 1 --2- 172.21.15.29:0/2308230133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb690d9a700 1 -- 172.21.15.29:0/2308230133 >> v2:172.21.15.144:3300/0 conn(0x7fb68c0b0610 msgr2=0x7fb68c0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:49.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb690d9a700 1 --2- 172.21.15.29:0/2308230133 >> v2:172.21.15.144:3300/0 conn(0x7fb68c0b0610 0x7fb68c0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb690d9a700 1 -- 172.21.15.29:0/2308230133 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0687c0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb690d9a700 1 --2- 172.21.15.29:0/2308230133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c0c4200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0659c0 tx=0x7fb67c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70706a5086e7a859 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb683fff700 1 -- 172.21.15.29:0/2308230133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c00b860 con 0x7fb68c0b09e0 2023-10-07T09:30:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb683fff700 1 -- 172.21.15.29:0/2308230133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c00b9c0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.705+0000 7fb683fff700 1 -- 172.21.15.29:0/2308230133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c00bd60 con 0x7fb68c0b09e0 2023-10-07T09:30:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 -- 172.21.15.29:0/2308230133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 msgr2=0x7fb68c0c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 --2- 172.21.15.29:0/2308230133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c0c4200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0659c0 tx=0x7fb67c009250 comp rx=0 tx=0).stop 2023-10-07T09:30:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 -- 172.21.15.29:0/2308230133 shutdown_connections 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 --2- 172.21.15.29:0/2308230133 >> v2:172.21.15.144:3300/0 conn(0x7fb68c0b0610 0x7fb68c0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 --2- 172.21.15.29:0/2308230133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 --2- 172.21.15.29:0/2308230133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c0c4200 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 -- 172.21.15.29:0/2308230133 >> 172.21.15.29:0/2308230133 conn(0x7fb68c063a10 msgr2=0x7fb68c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 -- 172.21.15.29:0/2308230133 shutdown_connections 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.706+0000 7fb6937ff700 1 -- 172.21.15.29:0/2308230133 wait complete. 2023-10-07T09:30:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.707+0000 7fb6937ff700 1 Processor -- start 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.707+0000 7fb6937ff700 1 -- start start 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.708+0000 7fb6937ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb68c0b0610 0x7fb68c1694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.708+0000 7fb6937ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c169eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.708+0000 7fb6937ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.708+0000 7fb6937ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c16a3f0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.708+0000 7fb6937ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0b1220 con 0x7fb68c0b0610 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.708+0000 7fb6937ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0b1500 con 0x7fb68c0c8a20 2023-10-07T09:30:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c169eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c169eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57732/0 (socket says 172.21.15.29:57732) 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 -- 172.21.15.29:0/1289700134 learned_addr learned my addr 172.21.15.29:0/1289700134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb691d9c700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb69159b700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb68c0b0610 0x7fb68c1694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 -- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 msgr2=0x7fb68c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 -- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb68c0b0610 msgr2=0x7fb68c1694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb68c0b0610 0x7fb68c1694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 -- 172.21.15.29:0/1289700134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0094f0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb691d9c700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.709+0000 7fb690d9a700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c169eb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb68c06ea20 tx=0x7fb67c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.710+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c024440 con 0x7fb68c0b09e0 2023-10-07T09:30:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.710+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c0245a0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.710+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c024960 con 0x7fb68c0b09e0 2023-10-07T09:30:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.710+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c0b17e0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.710+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c0b1da0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.711+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c0b1fc0 con 0x7fb68c0b09e0 2023-10-07T09:30:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.712+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fb67c01a420 con 0x7fb68c0b09e0 2023-10-07T09:30:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.713+0000 7fb6817fa700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb670067130 0x7fb6700695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.713+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fb67c086810 con 0x7fb68c0b09e0 2023-10-07T09:30:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.713+0000 7fb69159b700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb670067130 0x7fb6700695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.714+0000 7fb69159b700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb670067130 0x7fb6700695e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb688006fd0 tx=0x7fb688008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.717+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb67c053780 con 0x7fb68c0b09e0 2023-10-07T09:30:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:49.943+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb68c15d0f0 con 0x7fb670067130 2023-10-07T09:30:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.007+0000 7fb6817fa700 1 -- 172.21.15.29:0/1289700134 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb68c15d0f0 con 0x7fb670067130 2023-10-07T09:30:50.008 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.009+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb670067130 msgr2=0x7fb6700695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.009+0000 7fb6937ff700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb670067130 0x7fb6700695e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb688006fd0 tx=0x7fb688008040 comp rx=0 tx=0).stop 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.009+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 msgr2=0x7fb68c169eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.009+0000 7fb6937ff700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c169eb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb68c06ea20 tx=0x7fb67c017000 comp rx=0 tx=0).stop 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 shutdown_connections 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb670067130 0x7fb6700695e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb68c0c8a20 0x7fb68c0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb68c0b0610 0x7fb68c1694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 --2- 172.21.15.29:0/1289700134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb68c0b09e0 0x7fb68c169eb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 >> 172.21.15.29:0/1289700134 conn(0x7fb68c063a10 msgr2=0x7fb68c0c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 shutdown_connections 2023-10-07T09:30:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.010+0000 7fb6937ff700 1 -- 172.21.15.29:0/1289700134 wait complete. 2023-10-07T09:30:50.029 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-07T09:30:50.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:50.031 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-07T09:30:50.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir4 1M 2023-10-07T09:30:50.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.284+0000 7ff7d5fff700 1 Processor -- start 2023-10-07T09:30:50.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.284+0000 7ff7d5fff700 1 -- start start 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7d5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d00cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7d5fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff7d00cb6f0 0x7ff7d00cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7d5fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00cc940 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7d5fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d00cf530 con 0x7ff7d00cb6f0 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7d5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00b2df0 con 0x7ff7d00b2a20 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7ceffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff7d00cb6f0 0x7ff7d00cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7ceffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff7d00cb6f0 0x7ff7d00cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57748/0 (socket says 172.21.15.29:57748) 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7ceffd700 1 -- 172.21.15.29:0/3290953376 learned_addr learned my addr 172.21.15.29:0/3290953376 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7cffff700 1 --1- 172.21.15.29:0/3290953376 >> v1:172.21.15.29:6789/0 conn(0x7ff7d00cf0f0 0x7ff7d0159ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38908/0 (socket says 172.21.15.29:38908) 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.286+0000 7ff7cf7fe700 1 --2- 172.21.15.29:0/3290953376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d00cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ceffd700 1 -- 172.21.15.29:0/3290953376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 msgr2=0x7ff7d00cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ceffd700 1 --2- 172.21.15.29:0/3290953376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d00cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ceffd700 1 -- 172.21.15.29:0/3290953376 >> v1:172.21.15.29:6789/0 conn(0x7ff7d00cf0f0 legacy=0x7ff7d0159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ceffd700 1 -- 172.21.15.29:0/3290953376 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00cbd30 con 0x7ff7d00cb6f0 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ce7fc700 1 -- 172.21.15.29:0/3290953376 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411238608 0 0) 0x7ff7d00cc940 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ceffd700 1 --2- 172.21.15.29:0/3290953376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d00cebb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40099b0 tx=0x7ff7c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f0e64adf7e6bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ce7fc700 1 -- 172.21.15.29:0/3290953376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400b860 con 0x7ff7d00cb6f0 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ce7fc700 1 -- 172.21.15.29:0/3290953376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c400b9c0 con 0x7ff7d00cb6f0 2023-10-07T09:30:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.287+0000 7ff7ce7fc700 1 -- 172.21.15.29:0/3290953376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400bd60 con 0x7ff7d00cb6f0 2023-10-07T09:30:50.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 -- 172.21.15.29:0/3290953376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cb6f0 msgr2=0x7ff7d00cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/3290953376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d00cebb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40099b0 tx=0x7ff7c4009250 comp rx=0 tx=0).stop 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 -- 172.21.15.29:0/3290953376 shutdown_connections 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/3290953376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d00cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/3290953376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d00cebb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 -- 172.21.15.29:0/3290953376 >> 172.21.15.29:0/3290953376 conn(0x7ff7d0099820 msgr2=0x7ff7d0063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 -- 172.21.15.29:0/3290953376 shutdown_connections 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.288+0000 7ff7d5fff700 1 -- 172.21.15.29:0/3290953376 wait complete. 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.289+0000 7ff7d5fff700 1 Processor -- start 2023-10-07T09:30:50.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 -- start start 2023-10-07T09:30:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d01551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d0156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 0x7ff7d01573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00c9f30 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00c9060 con 0x7ff7d00cb6f0 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.290+0000 7ff7d5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00c9320 con 0x7ff7d00b2a20 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 0x7ff7d01573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 0x7ff7d01573c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57760/0 (socket says 172.21.15.29:57760) 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 -- 172.21.15.29:0/1481020232 learned_addr learned my addr 172.21.15.29:0/1481020232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cf7fe700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d01551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7ceffd700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d0156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 -- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 msgr2=0x7ff7d01551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d01551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 -- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d00cb6f0 msgr2=0x7ff7d0156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d0156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.291+0000 7ff7cffff700 1 -- 172.21.15.29:0/1481020232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c40094f0 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.292+0000 7ff7cffff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 0x7ff7d01573c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800cd50 tx=0x7ff7c8001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.292+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8005df0 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.292+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c800a8b0 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.292+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d00c9600 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.292+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800ac70 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.292+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d00c9bc0 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.293+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d00718b0 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.294+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7ff7c8007300 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.295+0000 7ff7b77fe700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7bc067130 0x7ff7bc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.295+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ff7c8084b30 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.295+0000 7ff7cf7fe700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7bc067130 0x7ff7bc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.296+0000 7ff7cf7fe700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7bc067130 0x7ff7bc0695e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00061f0 tx=0x7ff7c0008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:50.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.299+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7c8051a00 con 0x7ff7d00cf0f0 2023-10-07T09:30:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.530+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7d00ca070 con 0x7ff7bc067130 2023-10-07T09:30:50.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.597+0000 7ff7b77fe700 1 -- 172.21.15.29:0/1481020232 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ff7d00ca070 con 0x7ff7bc067130 2023-10-07T09:30:50.599 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-07T09:30:50.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7bc067130 msgr2=0x7ff7bc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7bc067130 0x7ff7bc0695e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00061f0 tx=0x7ff7c0008040 comp rx=0 tx=0).stop 2023-10-07T09:30:50.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 msgr2=0x7ff7d01573c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 0x7ff7d01573c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800cd50 tx=0x7ff7c8001b10 comp rx=0 tx=0).stop 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 shutdown_connections 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7bc067130 0x7ff7bc0695e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d00cf0f0 0x7ff7d01573c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d00b2a20 0x7ff7d01551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 --2- 172.21.15.29:0/1481020232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d00cb6f0 0x7ff7d0156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 >> 172.21.15.29:0/1481020232 conn(0x7ff7d0099820 msgr2=0x7ff7d00b5380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 shutdown_connections 2023-10-07T09:30:50.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.600+0000 7ff7d5fff700 1 -- 172.21.15.29:0/1481020232 wait complete. 2023-10-07T09:30:50.618 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-07T09:30:50.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:50.619 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-07T09:30:50.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir5 1M 2023-10-07T09:30:50.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.866+0000 7fe26a5ff700 1 Processor -- start 2023-10-07T09:30:50.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.866+0000 7fe26a5ff700 1 -- start start 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe26a5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2640b0610 0x7fe2640ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe26a5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b09e0 0x7fe2640c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe26a5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640690f0 con 0x7fe2640c8a20 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe26a5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2640c4740 con 0x7fe2640b0610 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe26a5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640cf2e0 con 0x7fe2640b09e0 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe263fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2640b0610 0x7fe2640ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe263fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2640b0610 0x7fe2640ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57768/0 (socket says 172.21.15.29:57768) 2023-10-07T09:30:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe263fff700 1 -- 172.21.15.29:0/168599268 learned_addr learned my addr 172.21.15.29:0/168599268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe268b9c700 1 --1- 172.21.15.29:0/168599268 >> v1:172.21.15.29:6789/0 conn(0x7fe2640c8a20 0x7fe2640cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38914/0 (socket says 172.21.15.29:38914) 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe2637fe700 1 --2- 172.21.15.29:0/168599268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b09e0 0x7fe2640c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe263fff700 1 -- 172.21.15.29:0/168599268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b09e0 msgr2=0x7fe2640c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe263fff700 1 --2- 172.21.15.29:0/168599268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b09e0 0x7fe2640c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe263fff700 1 -- 172.21.15.29:0/168599268 >> v1:172.21.15.29:6789/0 conn(0x7fe2640c8a20 legacy=0x7fe2640cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe263fff700 1 -- 172.21.15.29:0/168599268 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2640687c0 con 0x7fe2640b0610 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.868+0000 7fe262ffd700 1 -- 172.21.15.29:0/168599268 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340649766 0 0) 0x7fe2640690f0 con 0x7fe2640c8a20 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe263fff700 1 --2- 172.21.15.29:0/168599268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b0610 0x7fe2640ced00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe2540099b0 tx=0x7fe254009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ffad8c7bba7d23 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe262ffd700 1 -- 172.21.15.29:0/168599268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25400b860 con 0x7fe2640b0610 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe262ffd700 1 -- 172.21.15.29:0/168599268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25400b9c0 con 0x7fe2640b0610 2023-10-07T09:30:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe262ffd700 1 -- 172.21.15.29:0/168599268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25400bd60 con 0x7fe2640b0610 2023-10-07T09:30:50.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe26a5ff700 1 -- 172.21.15.29:0/168599268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b0610 msgr2=0x7fe2640ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.869+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/168599268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b0610 0x7fe2640ced00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe2540099b0 tx=0x7fe254009250 comp rx=0 tx=0).stop 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.870+0000 7fe26a5ff700 1 -- 172.21.15.29:0/168599268 shutdown_connections 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.870+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/168599268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b0610 0x7fe2640ced00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.870+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/168599268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b09e0 0x7fe2640c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.870+0000 7fe26a5ff700 1 -- 172.21.15.29:0/168599268 >> 172.21.15.29:0/168599268 conn(0x7fe264063a10 msgr2=0x7fe2640ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.870+0000 7fe26a5ff700 1 -- 172.21.15.29:0/168599268 shutdown_connections 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.870+0000 7fe26a5ff700 1 -- 172.21.15.29:0/168599268 wait complete. 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.871+0000 7fe26a5ff700 1 Processor -- start 2023-10-07T09:30:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.871+0000 7fe26a5ff700 1 -- start start 2023-10-07T09:30:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.872+0000 7fe26a5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b0610 0x7fe2640c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.872+0000 7fe26a5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 0x7fe2641697d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.872+0000 7fe26a5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2640c8a20 0x7fe2640c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.872+0000 7fe26a5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640ca0b0 con 0x7fe2640b09e0 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.872+0000 7fe26a5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640c5c80 con 0x7fe2640c8a20 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.872+0000 7fe26a5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640c83e0 con 0x7fe2640b0610 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 0x7fe2641697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 0x7fe2641697d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57782/0 (socket says 172.21.15.29:57782) 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 -- 172.21.15.29:0/995104063 learned_addr learned my addr 172.21.15.29:0/995104063 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe263fff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b0610 0x7fe2640c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe268b9c700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2640c8a20 0x7fe2640c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 -- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b0610 msgr2=0x7fe2640c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b0610 0x7fe2640c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 -- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2640c8a20 msgr2=0x7fe2640c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2640c8a20 0x7fe2640c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 -- 172.21.15.29:0/995104063 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2540094f0 con 0x7fe2640b09e0 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.873+0000 7fe2637fe700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 0x7fe2641697d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe25800d890 tx=0x7fe258008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.874+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2580125f0 con 0x7fe2640b09e0 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.874+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258012750 con 0x7fe2640b09e0 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.874+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2640c9dd0 con 0x7fe2640b09e0 2023-10-07T09:30:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.874+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258012ad0 con 0x7fe2640b09e0 2023-10-07T09:30:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.874+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2640c5910 con 0x7fe2640b09e0 2023-10-07T09:30:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.875+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2640c5470 con 0x7fe2640b09e0 2023-10-07T09:30:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.876+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fe25801e460 con 0x7fe2640b09e0 2023-10-07T09:30:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.877+0000 7fe24bfff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe250067130 0x7fe2500695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.877+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fe258085090 con 0x7fe2640b09e0 2023-10-07T09:30:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.877+0000 7fe263fff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe250067130 0x7fe2500695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.878+0000 7fe263fff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe250067130 0x7fe2500695e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2540099b0 tx=0x7fe254017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:50.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:50.881+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe258051ee0 con 0x7fe2640b09e0 2023-10-07T09:30:51.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.107+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2640c5df0 con 0x7fe250067130 2023-10-07T09:30:51.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.170+0000 7fe24bfff700 1 -- 172.21.15.29:0/995104063 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe2640c5df0 con 0x7fe250067130 2023-10-07T09:30:51.171 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-07T09:30:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe250067130 msgr2=0x7fe2500695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe250067130 0x7fe2500695e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2540099b0 tx=0x7fe254017000 comp rx=0 tx=0).stop 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 msgr2=0x7fe2641697d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 0x7fe2641697d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe25800d890 tx=0x7fe258008040 comp rx=0 tx=0).stop 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 shutdown_connections 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe250067130 0x7fe2500695e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2640c8a20 0x7fe2640c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2640b0610 0x7fe2640c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 --2- 172.21.15.29:0/995104063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2640b09e0 0x7fe2641697d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 >> 172.21.15.29:0/995104063 conn(0x7fe264063a10 msgr2=0x7fe2640b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 shutdown_connections 2023-10-07T09:30:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.172+0000 7fe26a5ff700 1 -- 172.21.15.29:0/995104063 wait complete. 2023-10-07T09:30:51.191 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-07T09:30:51.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:51.191 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-07T09:30:51.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir6 1M 2023-10-07T09:30:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.440+0000 7fbc8afff700 1 Processor -- start 2023-10-07T09:30:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.440+0000 7fbc8afff700 1 -- start start 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc8afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc840ca9e0 0x7fbc840681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc8afff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc84067610 con 0x7fbc8415a730 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc8afff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc840c1e70 con 0x7fbc840ca590 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc8afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840c2140 con 0x7fbc840ca9e0 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc88d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbc8415a730 0x7fbc840c1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38930/0 (socket says 172.21.15.29:38930) 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc88d9b700 1 -- 172.21.15.29:0/2026140426 learned_addr learned my addr 172.21.15.29:0/2026140426 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc8959c700 1 --2- 172.21.15.29:0/2026140426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc840ca9e0 0x7fbc840681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123033213 0 0) 0x7fbc84067610 con 0x7fbc8415a730 2023-10-07T09:30:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc74003570 con 0x7fbc8415a730 2023-10-07T09:30:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.442+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580092737 0 0) 0x7fbc840c1e70 con 0x7fbc840ca590 2023-10-07T09:30:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84067610 con 0x7fbc840ca590 2023-10-07T09:30:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc8959c700 1 -- 172.21.15.29:0/2026140426 >> v1:172.21.15.144:6789/0 conn(0x7fbc840ca590 legacy=0x7fbc84067aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc8959c700 1 -- 172.21.15.29:0/2026140426 >> v1:172.21.15.29:6789/0 conn(0x7fbc8415a730 legacy=0x7fbc840c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc8959c700 1 -- 172.21.15.29:0/2026140426 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc840c2420 con 0x7fbc840ca9e0 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc8959c700 1 --2- 172.21.15.29:0/2026140426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc840ca9e0 0x7fbc840681b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009e90 tx=0x7fbc7c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9eed753973d5842 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00e070 con 0x7fbc840ca9e0 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c00bb80 con 0x7fbc840ca9e0 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.443+0000 7fbc837fe700 1 -- 172.21.15.29:0/2026140426 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c012430 con 0x7fbc840ca9e0 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 -- 172.21.15.29:0/2026140426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc840ca9e0 msgr2=0x7fbc840681b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 --2- 172.21.15.29:0/2026140426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc840ca9e0 0x7fbc840681b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009e90 tx=0x7fbc7c009780 comp rx=0 tx=0).stop 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 -- 172.21.15.29:0/2026140426 shutdown_connections 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 --2- 172.21.15.29:0/2026140426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc840ca9e0 0x7fbc840681b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 -- 172.21.15.29:0/2026140426 >> 172.21.15.29:0/2026140426 conn(0x7fbc840c9430 msgr2=0x7fbc840c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 -- 172.21.15.29:0/2026140426 shutdown_connections 2023-10-07T09:30:51.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.444+0000 7fbc8afff700 1 -- 172.21.15.29:0/2026140426 wait complete. 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.445+0000 7fbc8afff700 1 Processor -- start 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.446+0000 7fbc8afff700 1 -- start start 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.446+0000 7fbc8afff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc840ca590 0x7fbc84151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.446+0000 7fbc8afff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 0x7fbc84155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.446+0000 7fbc8afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc8415a730 0x7fbc840c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.446+0000 7fbc8afff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840b0830 con 0x7fbc840ca9e0 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc8afff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840afc20 con 0x7fbc840ca590 2023-10-07T09:30:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc8afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840aff00 con 0x7fbc8415a730 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 0x7fbc84155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 0x7fbc84155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57792/0 (socket says 172.21.15.29:57792) 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 -- 172.21.15.29:0/238101448 learned_addr learned my addr 172.21.15.29:0/238101448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc88d9b700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc840ca590 0x7fbc84151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc8959c700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc8415a730 0x7fbc840c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 -- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc8415a730 msgr2=0x7fbc840c3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc8415a730 0x7fbc840c3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 -- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc840ca590 msgr2=0x7fbc84151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc840ca590 0x7fbc84151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 -- 172.21.15.29:0/238101448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c009a20 con 0x7fbc840ca9e0 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.447+0000 7fbc83fff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 0x7fbc84155df0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c590 tx=0x7fbc70007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.448+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70010510 con 0x7fbc840ca9e0 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.448+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc70010690 con 0x7fbc840ca9e0 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.448+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70010a50 con 0x7fbc840ca9e0 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.448+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc840b01e0 con 0x7fbc840ca9e0 2023-10-07T09:30:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.448+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc84156330 con 0x7fbc840ca9e0 2023-10-07T09:30:51.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.449+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc840b09e0 con 0x7fbc840ca9e0 2023-10-07T09:30:51.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.450+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fbc7001b400 con 0x7fbc840ca9e0 2023-10-07T09:30:51.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.451+0000 7fbc80ff9700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc68067130 0x7fbc680695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:51.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.451+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fbc70084750 con 0x7fbc840ca9e0 2023-10-07T09:30:51.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.451+0000 7fbc88d9b700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc68067130 0x7fbc680695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:51.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.452+0000 7fbc88d9b700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc68067130 0x7fbc680695e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbc840b2760 tx=0x7fbc78009040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:51.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.455+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc840b09e0 con 0x7fbc840ca9e0 2023-10-07T09:30:51.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.682+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc840718b0 con 0x7fbc68067130 2023-10-07T09:30:51.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.746+0000 7fbc80ff9700 1 -- 172.21.15.29:0/238101448 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbc840718b0 con 0x7fbc68067130 2023-10-07T09:30:51.747 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-07T09:30:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc68067130 msgr2=0x7fbc680695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc68067130 0x7fbc680695e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbc840b2760 tx=0x7fbc78009040 comp rx=0 tx=0).stop 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 msgr2=0x7fbc84155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 0x7fbc84155df0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c590 tx=0x7fbc70007dd0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 shutdown_connections 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fbc68067130 0x7fbc680695e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc8415a730 0x7fbc840c3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc840ca590 0x7fbc84151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 --2- 172.21.15.29:0/238101448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc840ca9e0 0x7fbc84155df0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 >> 172.21.15.29:0/238101448 conn(0x7fbc840c9430 msgr2=0x7fbc84153390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 shutdown_connections 2023-10-07T09:30:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:51.748+0000 7fbc8afff700 1 -- 172.21.15.29:0/238101448 wait complete. 2023-10-07T09:30:51.767 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-07T09:30:51.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:51.767 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-07T09:30:51.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir7 1M 2023-10-07T09:30:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.016+0000 7f16ff3ff700 1 Processor -- start 2023-10-07T09:30:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.016+0000 7f16ff3ff700 1 -- start start 2023-10-07T09:30:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.017+0000 7f16ff3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 0x7f16f80ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.017+0000 7f16ff3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80ca430 con 0x7f16f80ae910 2023-10-07T09:30:52.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16ff3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80ca710 con 0x7f16f8157a20 2023-10-07T09:30:52.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16ff3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80ca9f0 con 0x7f16f8157650 2023-10-07T09:30:52.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16fc99a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16f80ae910 0x7f16f80b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38946/0 (socket says 172.21.15.29:38946) 2023-10-07T09:30:52.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16fc99a700 1 -- 172.21.15.29:0/2724889124 learned_addr learned my addr 172.21.15.29:0/2724889124 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16fd19b700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 0x7f16f80ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226445155 0 0) 0x7f16f80ca430 con 0x7f16f80ae910 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e4003570 con 0x7f16f80ae910 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270832984 0 0) 0x7f16f80ca710 con 0x7f16f8157a20 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f80ca430 con 0x7f16f8157a20 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900391826 0 0) 0x7f16e4003570 con 0x7f16f80ae910 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 msgr2=0x7f16f80ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 0x7f16f80ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.018+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 >> v1:172.21.15.144:6789/0 conn(0x7f16f8157a20 legacy=0x7f16f80cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80cacd0 con 0x7f16f80ae910 2023-10-07T09:30:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f16e8001470 con 0x7f16f80ae910 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 >> v1:172.21.15.29:6789/0 conn(0x7f16f80ae910 legacy=0x7f16f80b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4003c20 0x7f16e4004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 0x7f16e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16e4004d50 0x7f16e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80ca9f0 con 0x7f16e4004d50 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e400bc40 con 0x7f16e4003c20 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e400be00 con 0x7f16f8157650 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd19b700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4003c20 0x7f16e4004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd99c700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16e4004d50 0x7f16e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fc99a700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 0x7f16e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd99c700 1 -- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 msgr2=0x7f16e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd99c700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f8157650 0x7f16e4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd99c700 1 -- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4003c20 msgr2=0x7f16e4004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd99c700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4003c20 0x7f16e4004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd99c700 1 -- 172.21.15.29:0/2724889124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4003570 con 0x7f16e4004d50 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.019+0000 7f16fd19b700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4003c20 0x7f16e4004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16fd99c700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16e4004d50 0x7f16e40051c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f16f00020f0 tx=0x7f16f00017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293e1c59b476abf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f000c8b0 con 0x7f16e4004d50 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f000ca10 con 0x7f16e4004d50 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16effff700 1 -- 172.21.15.29:0/2724889124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f000cdb0 con 0x7f16e4004d50 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16ff3ff700 1 -- 172.21.15.29:0/2724889124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16e4004d50 msgr2=0x7f16e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/2724889124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16e4004d50 0x7f16e40051c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f16f00020f0 tx=0x7f16f00017a0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.020+0000 7f16fd99c700 1 -- 172.21.15.29:0/2724889124 reap_dead start 2023-10-07T09:30:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.021+0000 7f16ff3ff700 1 -- 172.21.15.29:0/2724889124 shutdown_connections 2023-10-07T09:30:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.021+0000 7f16ff3ff700 1 -- 172.21.15.29:0/2724889124 >> 172.21.15.29:0/2724889124 conn(0x7f16f8067610 msgr2=0x7f16f8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.021+0000 7f16ff3ff700 1 -- 172.21.15.29:0/2724889124 shutdown_connections 2023-10-07T09:30:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.021+0000 7f16ff3ff700 1 -- 172.21.15.29:0/2724889124 wait complete. 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.023+0000 7f16ff3ff700 1 Processor -- start 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.023+0000 7f16ff3ff700 1 -- start start 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16ff3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4004d50 0x7f16f81597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16ff3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f80ab110 0x7f16f80a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16ff3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 0x7f16f80a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16ff3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f813fb00 con 0x7f16f8159d10 2023-10-07T09:30:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16ff3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f815a250 con 0x7f16e4004d50 2023-10-07T09:30:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16ff3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f815a530 con 0x7f16f80ab110 2023-10-07T09:30:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.024+0000 7f16fd99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 0x7f16f80a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 0x7f16f80a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57814/0 (socket says 172.21.15.29:57814) 2023-10-07T09:30:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 -- 172.21.15.29:0/4240316427 learned_addr learned my addr 172.21.15.29:0/4240316427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd19b700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4004d50 0x7f16f81597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fc99a700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f80ab110 0x7f16f80a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 -- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f80ab110 msgr2=0x7f16f80a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f80ab110 0x7f16f80a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 -- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4004d50 msgr2=0x7f16f81597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4004d50 0x7f16f81597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 -- 172.21.15.29:0/4240316427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f813f790 con 0x7f16f8159d10 2023-10-07T09:30:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16fd99c700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 0x7f16f80a5ca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f16e400bc10 tx=0x7f16f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0001400 con 0x7f16f8159d10 2023-10-07T09:30:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f0023070 con 0x7f16f8159d10 2023-10-07T09:30:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f8152ef0 con 0x7f16f8159d10 2023-10-07T09:30:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.025+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0006db0 con 0x7f16f8159d10 2023-10-07T09:30:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.026+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f8140150 con 0x7f16f8159d10 2023-10-07T09:30:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.027+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f813f920 con 0x7f16f8159d10 2023-10-07T09:30:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.027+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f16f0003030 con 0x7f16f8159d10 2023-10-07T09:30:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.028+0000 7f16ed7fa700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f16dc067130 0x7f16dc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.028+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f16f00862b0 con 0x7f16f8159d10 2023-10-07T09:30:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.029+0000 7f16fd19b700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f16dc067130 0x7f16dc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.029+0000 7f16fd19b700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f16dc067130 0x7f16dc0695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f16f80c82f0 tx=0x7f16f40091a0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:52.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.033+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16f0053100 con 0x7f16f8159d10 2023-10-07T09:30:52.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.265+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f16f813f610 con 0x7f16dc067130 2023-10-07T09:30:52.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.331+0000 7f16ed7fa700 1 -- 172.21.15.29:0/4240316427 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f16f4019070 con 0x7f16dc067130 2023-10-07T09:30:52.333 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-07T09:30:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f16dc067130 msgr2=0x7f16dc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f16dc067130 0x7f16dc0695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f16f80c82f0 tx=0x7f16f40091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 msgr2=0x7f16f80a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 0x7f16f80a5ca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f16e400bc10 tx=0x7f16f0016000 comp rx=0 tx=0).stop 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 shutdown_connections 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f16dc067130 0x7f16dc0695e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f8159d10 0x7f16f80a5ca0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e4004d50 0x7f16f81597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 --2- 172.21.15.29:0/4240316427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16f80ab110 0x7f16f80a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 >> 172.21.15.29:0/4240316427 conn(0x7f16f8067610 msgr2=0x7f16f80b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 shutdown_connections 2023-10-07T09:30:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.334+0000 7f16ff3ff700 1 -- 172.21.15.29:0/4240316427 wait complete. 2023-10-07T09:30:52.353 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-07T09:30:52.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:52.355 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-07T09:30:52.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir8 1M 2023-10-07T09:30:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.604+0000 7fe3e45ff700 1 Processor -- start 2023-10-07T09:30:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.604+0000 7fe3e45ff700 1 -- start start 2023-10-07T09:30:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e45ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe3dc1559f0 0x7fe3dc0b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e45ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc0c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e45ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0c8c70 con 0x7fe3dc0b0660 2023-10-07T09:30:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e45ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc0c98f0 con 0x7fe3dc1559f0 2023-10-07T09:30:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e45ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0c8f50 con 0x7fe3dc159de0 2023-10-07T09:30:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e1b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe3dc0b0660 0x7fe3dc1596d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38950/0 (socket says 172.21.15.29:38950) 2023-10-07T09:30:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e1b9a700 1 -- 172.21.15.29:0/1347733922 learned_addr learned my addr 172.21.15.29:0/1347733922 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e239b700 1 --2- 172.21.15.29:0/1347733922 >> v2:172.21.15.144:3300/0 conn(0x7fe3dc1559f0 0x7fe3dc0b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.605+0000 7fe3e2b9c700 1 --2- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc0c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e1399700 1 -- 172.21.15.29:0/1347733922 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018252440 0 0) 0x7fe3dc0c8c70 con 0x7fe3dc0b0660 2023-10-07T09:30:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e1399700 1 -- 172.21.15.29:0/1347733922 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c8003570 con 0x7fe3dc0b0660 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e239b700 1 -- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 msgr2=0x7fe3dc0c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e239b700 1 --2- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc0c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e239b700 1 -- 172.21.15.29:0/1347733922 >> v1:172.21.15.29:6789/0 conn(0x7fe3dc0b0660 legacy=0x7fe3dc1596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e239b700 1 -- 172.21.15.29:0/1347733922 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0c9230 con 0x7fe3dc1559f0 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.606+0000 7fe3e239b700 1 --2- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc0b0120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe3d80098e0 tx=0x7fe3d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b901955749e3bcc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e1399700 1 -- 172.21.15.29:0/1347733922 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8011630 con 0x7fe3dc1559f0 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e1399700 1 -- 172.21.15.29:0/1347733922 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8011790 con 0x7fe3dc1559f0 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e1399700 1 -- 172.21.15.29:0/1347733922 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8011b30 con 0x7fe3dc1559f0 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 -- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 msgr2=0x7fe3dc0b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc0b0120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe3d80098e0 tx=0x7fe3d80091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 -- 172.21.15.29:0/1347733922 shutdown_connections 2023-10-07T09:30:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc0c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/1347733922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc0b0120 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 -- 172.21.15.29:0/1347733922 >> 172.21.15.29:0/1347733922 conn(0x7fe3dc067610 msgr2=0x7fe3dc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 -- 172.21.15.29:0/1347733922 shutdown_connections 2023-10-07T09:30:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.607+0000 7fe3e45ff700 1 -- 172.21.15.29:0/1347733922 wait complete. 2023-10-07T09:30:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.609+0000 7fe3e45ff700 1 Processor -- start 2023-10-07T09:30:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.610+0000 7fe3e45ff700 1 -- start start 2023-10-07T09:30:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e45ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 0x7fe3dc14c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e45ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e45ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e45ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc14f470 con 0x7fe3dc0b0660 2023-10-07T09:30:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e45ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0c6cd0 con 0x7fe3dc1559f0 2023-10-07T09:30:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e45ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0c63e0 con 0x7fe3dc159de0 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e239b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 0x7fe3dc14c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e239b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 0x7fe3dc14c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57818/0 (socket says 172.21.15.29:57818) 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e239b700 1 -- 172.21.15.29:0/68517506 learned_addr learned my addr 172.21.15.29:0/68517506 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e1b9a700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e2b9c700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.611+0000 7fe3e239b700 1 -- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 msgr2=0x7fe3dc153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e239b700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e239b700 1 -- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 msgr2=0x7fe3dc153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e239b700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e239b700 1 -- 172.21.15.29:0/68517506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8009420 con 0x7fe3dc0b0660 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e1b9a700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e239b700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 0x7fe3dc14c8d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009ed0 tx=0x7fe3d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8011b10 con 0x7fe3dc0b0660 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d80253f0 con 0x7fe3dc0b0660 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc14f190 con 0x7fe3dc0b0660 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80257b0 con 0x7fe3dc0b0660 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.612+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc0c6960 con 0x7fe3dc0b0660 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.613+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc0718b0 con 0x7fe3dc0b0660 2023-10-07T09:30:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.614+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fe3d8025ba0 con 0x7fe3dc0b0660 2023-10-07T09:30:52.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.615+0000 7fe3d27fc700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe3c0067130 0x7fe3c00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:52.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.615+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fe3d8086210 con 0x7fe3dc0b0660 2023-10-07T09:30:52.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.616+0000 7fe3e1b9a700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe3c0067130 0x7fe3c00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:52.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.616+0000 7fe3e1b9a700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe3c0067130 0x7fe3c00695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc007fd0 tx=0x7fe3cc009040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:52.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.619+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3d8053100 con 0x7fe3dc0b0660 2023-10-07T09:30:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.845+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3dc0c9510 con 0x7fe3c0067130 2023-10-07T09:30:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.911+0000 7fe3d27fc700 1 -- 172.21.15.29:0/68517506 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe3dc0c9510 con 0x7fe3c0067130 2023-10-07T09:30:52.912 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe3c0067130 msgr2=0x7fe3c00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe3c0067130 0x7fe3c00695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc007fd0 tx=0x7fe3cc009040 comp rx=0 tx=0).stop 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 msgr2=0x7fe3dc14c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 0x7fe3dc14c8d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009ed0 tx=0x7fe3d8017000 comp rx=0 tx=0).stop 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 shutdown_connections 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe3c0067130 0x7fe3c00695e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3dc159de0 0x7fe3dc153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0660 0x7fe3dc14c8d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 --2- 172.21.15.29:0/68517506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3dc1559f0 0x7fe3dc153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.913+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 >> 172.21.15.29:0/68517506 conn(0x7fe3dc067610 msgr2=0x7fe3dc0c9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.914+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 shutdown_connections 2023-10-07T09:30:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:52.914+0000 7fe3e45ff700 1 -- 172.21.15.29:0/68517506 wait complete. 2023-10-07T09:30:52.932 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-07T09:30:52.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:30:52.933 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-07T09:30:52.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir9 1M 2023-10-07T09:30:53.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.181+0000 7f7e98dff700 1 Processor -- start 2023-10-07T09:30:53.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.181+0000 7f7e98dff700 1 -- start start 2023-10-07T09:30:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e98dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940ca9e0 0x7f7e940681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e98dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e94067610 con 0x7f7e9415a730 2023-10-07T09:30:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e98dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e940c1e70 con 0x7f7e940ca590 2023-10-07T09:30:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e98dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c2140 con 0x7f7e940ca9e0 2023-10-07T09:30:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e9259c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7e9415a730 0x7f7e940c1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39342/0 (socket says 172.21.15.29:39342) 2023-10-07T09:30:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e9259c700 1 -- 172.21.15.29:0/3696778435 learned_addr learned my addr 172.21.15.29:0/3696778435 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e92d9d700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940ca9e0 0x7f7e940681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.183+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179653387 0 0) 0x7f7e94067610 con 0x7f7e9415a730 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e7c003570 con 0x7f7e9415a730 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979751261 0 0) 0x7f7e7c003570 con 0x7f7e9415a730 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940ca9e0 msgr2=0x7f7e940681b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940ca9e0 0x7f7e940681b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 >> v1:172.21.15.144:6789/0 conn(0x7f7e940ca590 legacy=0x7f7e94067aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940c2420 con 0x7f7e9415a730 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f7e84001470 con 0x7f7e9415a730 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 >> v1:172.21.15.29:6789/0 conn(0x7f7e9415a730 legacy=0x7f7e940c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c003a90 0x7f7e7c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e7c004810 0x7f7e7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c0051c0 0x7f7e7c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c2140 con 0x7f7e7c003a90 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c00c0b0 con 0x7f7e7c0051c0 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c00c270 con 0x7f7e7c004810 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e92d9d700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940ca9e0 0x7f7e940681b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e9259c700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c003a90 0x7f7e7c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.184+0000 7f7e91d9b700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e7c004810 0x7f7e7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e92d9d700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c0051c0 0x7f7e7c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9259c700 1 -- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e7c004810 msgr2=0x7f7e7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9259c700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e7c004810 0x7f7e7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9259c700 1 -- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c0051c0 msgr2=0x7f7e7c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9259c700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c0051c0 0x7f7e7c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9259c700 1 -- 172.21.15.29:0/3696778435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c003570 con 0x7f7e7c003a90 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9259c700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c003a90 0x7f7e7c003e60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400c2a0 tx=0x7f7e940c2cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107d3f085b75e486 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8400cb70 con 0x7f7e7c003a90 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8400ccd0 con 0x7f7e7c003a90 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.185+0000 7f7e9159a700 1 -- 172.21.15.29:0/3696778435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84005b70 con 0x7f7e7c003a90 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e98dff700 1 -- 172.21.15.29:0/3696778435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c003a90 msgr2=0x7f7e7c003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e98dff700 1 --2- 172.21.15.29:0/3696778435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7c003a90 0x7f7e7c003e60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400c2a0 tx=0x7f7e940c2cc0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e92d9d700 1 -- 172.21.15.29:0/3696778435 reap_dead start 2023-10-07T09:30:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e98dff700 1 -- 172.21.15.29:0/3696778435 shutdown_connections 2023-10-07T09:30:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e98dff700 1 -- 172.21.15.29:0/3696778435 >> 172.21.15.29:0/3696778435 conn(0x7f7e940c9430 msgr2=0x7f7e940c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e98dff700 1 -- 172.21.15.29:0/3696778435 shutdown_connections 2023-10-07T09:30:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.186+0000 7f7e98dff700 1 -- 172.21.15.29:0/3696778435 wait complete. 2023-10-07T09:30:53.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.187+0000 7f7e98dff700 1 Processor -- start 2023-10-07T09:30:53.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.188+0000 7f7e98dff700 1 -- start start 2023-10-07T09:30:53.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.188+0000 7f7e98dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c003a90 0x7f7e940b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e98dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940c3a70 0x7f7e940c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e98dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 0x7f7e94157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e98dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c4eb0 con 0x7f7e94151200 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e98dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c4230 con 0x7f7e7c003a90 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e98dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c4510 con 0x7f7e940c3a70 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 0x7f7e94157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 0x7f7e94157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52546/0 (socket says 172.21.15.29:52546) 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 -- 172.21.15.29:0/2421815815 learned_addr learned my addr 172.21.15.29:0/2421815815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e91d9b700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940c3a70 0x7f7e940c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e9259c700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c003a90 0x7f7e940b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 -- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940c3a70 msgr2=0x7f7e940c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940c3a70 0x7f7e940c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 -- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c003a90 msgr2=0x7f7e940b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c003a90 0x7f7e940b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.189+0000 7f7e92d9d700 1 -- 172.21.15.29:0/2421815815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940c47f0 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.190+0000 7f7e92d9d700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 0x7f7e94157810 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c004790 tx=0x7f7e8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.190+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00e040 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.190+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00f040 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.190+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00fe00 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.190+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e940c4ad0 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.190+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e940c3ee0 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.191+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e940c4ff0 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.192+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f7e8c015640 con 0x7f7e94151200 2023-10-07T09:30:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.193+0000 7f7e827fc700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e74067130 0x7f7e740695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.193+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f7e8c085100 con 0x7f7e94151200 2023-10-07T09:30:53.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.194+0000 7f7e9259c700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e74067130 0x7f7e740695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.195+0000 7f7e9259c700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e74067130 0x7f7e740695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e940c6af0 tx=0x7f7e84015000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:53.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.197+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e940c4ff0 con 0x7f7e94151200 2023-10-07T09:30:53.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.425+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e940c4ff0 con 0x7f7e74067130 2023-10-07T09:30:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.492+0000 7f7e827fc700 1 -- 172.21.15.29:0/2421815815 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7e940c4ff0 con 0x7f7e74067130 2023-10-07T09:30:53.493 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e74067130 msgr2=0x7f7e740695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e74067130 0x7f7e740695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e940c6af0 tx=0x7f7e84015000 comp rx=0 tx=0).stop 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 msgr2=0x7f7e94157810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 0x7f7e94157810 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c004790 tx=0x7f7e8c00b040 comp rx=0 tx=0).stop 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 shutdown_connections 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e74067130 0x7f7e740695e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e94151200 0x7f7e94157810 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e7c003a90 0x7f7e940b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 --2- 172.21.15.29:0/2421815815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e940c3a70 0x7f7e940c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 >> 172.21.15.29:0/2421815815 conn(0x7f7e940c9430 msgr2=0x7f7e940c3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.494+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 shutdown_connections 2023-10-07T09:30:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.495+0000 7f7e98dff700 1 -- 172.21.15.29:0/2421815815 wait complete. 2023-10-07T09:30:53.513 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-07T09:30:53.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir0 1M 2023-10-07T09:30:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.713+0000 7f5b67bff700 1 Processor -- start 2023-10-07T09:30:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.713+0000 7f5b67bff700 1 -- start start 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.714+0000 7f5b67bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b600ced00 0x7f5b600b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.714+0000 7f5b67bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b600c8a20 0x7f5b600cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.714+0000 7f5b67bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b600690f0 con 0x7f5b600b0610 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.714+0000 7f5b67bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600cf2e0 con 0x7f5b600ced00 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.714+0000 7f5b67bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600687c0 con 0x7f5b600c8a20 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b600ced00 0x7f5b600b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b600ced00 0x7f5b600b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52556/0 (socket says 172.21.15.29:52556) 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 -- 172.21.15.29:0/3894522896 learned_addr learned my addr 172.21.15.29:0/3894522896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:53.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6599b700 1 --1- 172.21.15.29:0/3894522896 >> v1:172.21.15.29:6789/0 conn(0x7f5b600b0610 0x7f5b600c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39346/0 (socket says 172.21.15.29:39346) 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 -- 172.21.15.29:0/3894522896 >> v2:172.21.15.144:3300/0 conn(0x7f5b600c8a20 msgr2=0x7f5b600cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6619c700 1 --2- 172.21.15.29:0/3894522896 >> v2:172.21.15.144:3300/0 conn(0x7f5b600c8a20 0x7f5b600cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 --2- 172.21.15.29:0/3894522896 >> v2:172.21.15.144:3300/0 conn(0x7f5b600c8a20 0x7f5b600cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 -- 172.21.15.29:0/3894522896 >> v1:172.21.15.29:6789/0 conn(0x7f5b600b0610 legacy=0x7f5b600c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 -- 172.21.15.29:0/3894522896 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60068aa0 con 0x7f5b600ced00 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b64999700 1 -- 172.21.15.29:0/3894522896 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643522883 0 0) 0x7f5b600690f0 con 0x7f5b600b0610 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6619c700 1 --2- 172.21.15.29:0/3894522896 >> v2:172.21.15.144:3300/0 conn(0x7f5b600c8a20 0x7f5b600cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.715+0000 7f5b6519a700 1 --2- 172.21.15.29:0/3894522896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600ced00 0x7f5b600b09e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5b60070ea0 tx=0x7f5b50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d210a264bbec0829 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b64999700 1 -- 172.21.15.29:0/3894522896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5000b860 con 0x7f5b600ced00 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b64999700 1 -- 172.21.15.29:0/3894522896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5000b9c0 con 0x7f5b600ced00 2023-10-07T09:30:53.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b64999700 1 -- 172.21.15.29:0/3894522896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5000bd60 con 0x7f5b600ced00 2023-10-07T09:30:53.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 -- 172.21.15.29:0/3894522896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600ced00 msgr2=0x7f5b600b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 --2- 172.21.15.29:0/3894522896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600ced00 0x7f5b600b09e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5b60070ea0 tx=0x7f5b50009250 comp rx=0 tx=0).stop 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 -- 172.21.15.29:0/3894522896 shutdown_connections 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 --2- 172.21.15.29:0/3894522896 >> v2:172.21.15.144:3300/0 conn(0x7f5b600c8a20 0x7f5b600cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 --2- 172.21.15.29:0/3894522896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600ced00 0x7f5b600b09e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 -- 172.21.15.29:0/3894522896 >> 172.21.15.29:0/3894522896 conn(0x7f5b60063a10 msgr2=0x7f5b600ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 -- 172.21.15.29:0/3894522896 shutdown_connections 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.716+0000 7f5b67bff700 1 -- 172.21.15.29:0/3894522896 wait complete. 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.718+0000 7f5b67bff700 1 Processor -- start 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.718+0000 7f5b67bff700 1 -- start start 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b67bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b600b0610 0x7f5b600b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b67bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 0x7f5b601527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b67bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b600ced00 0x7f5b6006c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b67bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60158e40 con 0x7f5b600c8a20 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b67bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6016dee0 con 0x7f5b600b0610 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b67bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6016dc00 con 0x7f5b600ced00 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b6519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 0x7f5b601527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b6519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 0x7f5b601527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52568/0 (socket says 172.21.15.29:52568) 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b6519a700 1 -- 172.21.15.29:0/590185090 learned_addr learned my addr 172.21.15.29:0/590185090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b6599b700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b600b0610 0x7f5b600b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.719+0000 7f5b6619c700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b600ced00 0x7f5b6006c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b6519a700 1 -- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b600ced00 msgr2=0x7f5b6006c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b6519a700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b600ced00 0x7f5b6006c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b6519a700 1 -- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b600b0610 msgr2=0x7f5b600b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b6519a700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b600b0610 0x7f5b600b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b6519a700 1 -- 172.21.15.29:0/590185090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b500094f0 con 0x7f5b600c8a20 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b6519a700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 0x7f5b601527d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5b6006ea20 tx=0x7f5b50017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50024430 con 0x7f5b600c8a20 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b50024590 con 0x7f5b600c8a20 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b600b2350 con 0x7f5b600c8a20 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50024910 con 0x7f5b600c8a20 2023-10-07T09:30:53.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.720+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600b2910 con 0x7f5b600c8a20 2023-10-07T09:30:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.721+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b60152d10 con 0x7f5b600c8a20 2023-10-07T09:30:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.722+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f5b5001a420 con 0x7f5b600c8a20 2023-10-07T09:30:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.723+0000 7f5b55ffb700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5b44067130 0x7f5b440695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.723+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f5b50086660 con 0x7f5b600c8a20 2023-10-07T09:30:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.723+0000 7f5b6599b700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5b44067130 0x7f5b440695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:53.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.724+0000 7f5b6599b700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5b44067130 0x7f5b440695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c007fd0 tx=0x7f5b5c007e10 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:53.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.727+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b50053530 con 0x7f5b600c8a20 2023-10-07T09:30:53.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:53.956+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b600718b0 con 0x7f5b44067130 2023-10-07T09:30:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.009+0000 7f5b55ffb700 1 -- 172.21.15.29:0/590185090 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5b600718b0 con 0x7f5b44067130 2023-10-07T09:30:54.011 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-07T09:30:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5b44067130 msgr2=0x7f5b440695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5b44067130 0x7f5b440695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c007fd0 tx=0x7f5b5c007e10 comp rx=0 tx=0).stop 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 msgr2=0x7f5b601527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 0x7f5b601527d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5b6006ea20 tx=0x7f5b50017000 comp rx=0 tx=0).stop 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 shutdown_connections 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5b44067130 0x7f5b440695e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b600ced00 0x7f5b6006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b600b0610 0x7f5b600b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 --2- 172.21.15.29:0/590185090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b600c8a20 0x7f5b601527d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 >> 172.21.15.29:0/590185090 conn(0x7f5b60063a10 msgr2=0x7f5b60155640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 shutdown_connections 2023-10-07T09:30:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.012+0000 7f5b67bff700 1 -- 172.21.15.29:0/590185090 wait complete. 2023-10-07T09:30:54.030 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-07T09:30:54.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir1 1M 2023-10-07T09:30:54.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.231+0000 7f7e343ff700 1 Processor -- start 2023-10-07T09:30:54.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.231+0000 7f7e343ff700 1 -- start start 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e343ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e2c0b2a20 0x7f7e2c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e343ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0cc940 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e343ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0b2df0 con 0x7f7e2c0cb6f0 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e343ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c0cbd30 con 0x7f7e2c0b2a20 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e3219b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e2c0b2a20 0x7f7e2c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e3299c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7e2c0cf2c0 0x7f7e2c0c5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39356/0 (socket says 172.21.15.29:39356) 2023-10-07T09:30:54.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e3299c700 1 -- 172.21.15.29:0/268414317 learned_addr learned my addr 172.21.15.29:0/268414317 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e3219b700 1 -- 172.21.15.29:0/268414317 >> v1:172.21.15.144:6789/0 conn(0x7f7e2c0cb6f0 legacy=0x7f7e2c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e3219b700 1 -- 172.21.15.29:0/268414317 >> v1:172.21.15.29:6789/0 conn(0x7f7e2c0cf2c0 legacy=0x7f7e2c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.233+0000 7f7e3219b700 1 -- 172.21.15.29:0/268414317 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0cc010 con 0x7f7e2c0b2a20 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.234+0000 7f7e3219b700 1 --2- 172.21.15.29:0/268414317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0b2a20 0x7f7e2c0cb1b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7e280098d0 tx=0x7f7e280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a626dccfed7091 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.234+0000 7f7e31199700 1 -- 172.21.15.29:0/268414317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28009e70 con 0x7f7e2c0b2a20 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.234+0000 7f7e31199700 1 -- 172.21.15.29:0/268414317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2800b860 con 0x7f7e2c0b2a20 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.234+0000 7f7e31199700 1 -- 172.21.15.29:0/268414317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800bc00 con 0x7f7e2c0b2a20 2023-10-07T09:30:54.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.234+0000 7f7e343ff700 1 -- 172.21.15.29:0/268414317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0b2a20 msgr2=0x7f7e2c0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.234+0000 7f7e343ff700 1 --2- 172.21.15.29:0/268414317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0b2a20 0x7f7e2c0cb1b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7e280098d0 tx=0x7f7e280091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.235+0000 7f7e343ff700 1 -- 172.21.15.29:0/268414317 shutdown_connections 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.235+0000 7f7e343ff700 1 --2- 172.21.15.29:0/268414317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0b2a20 0x7f7e2c0cb1b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.235+0000 7f7e343ff700 1 -- 172.21.15.29:0/268414317 >> 172.21.15.29:0/268414317 conn(0x7f7e2c099820 msgr2=0x7f7e2c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.235+0000 7f7e343ff700 1 -- 172.21.15.29:0/268414317 shutdown_connections 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.235+0000 7f7e343ff700 1 -- 172.21.15.29:0/268414317 wait complete. 2023-10-07T09:30:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.237+0000 7f7e343ff700 1 Processor -- start 2023-10-07T09:30:54.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.237+0000 7f7e343ff700 1 -- start start 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.238+0000 7f7e343ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e2c0b2a20 0x7f7e2c14d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.238+0000 7f7e343ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e2c0cb6f0 0x7f7e2c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.238+0000 7f7e343ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 0x7f7e2c0c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.238+0000 7f7e343ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c16d420 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.238+0000 7f7e343ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c16e240 con 0x7f7e2c0cb6f0 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.238+0000 7f7e343ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c16d630 con 0x7f7e2c0b2a20 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 0x7f7e2c0c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 0x7f7e2c0c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52584/0 (socket says 172.21.15.29:52584) 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 -- 172.21.15.29:0/1484389931 learned_addr learned my addr 172.21.15.29:0/1484389931 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3219b700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e2c0b2a20 0x7f7e2c14d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3199a700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e2c0cb6f0 0x7f7e2c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 -- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e2c0b2a20 msgr2=0x7f7e2c14d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e2c0b2a20 0x7f7e2c14d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 -- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e2c0cb6f0 msgr2=0x7f7e2c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e2c0cb6f0 0x7f7e2c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 -- 172.21.15.29:0/1484389931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280093c0 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3199a700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e2c0cb6f0 0x7f7e2c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:30:54.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e3299c700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 0x7f7e2c0c6620 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400cbc0 tx=0x7f7e24001990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:54.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2400a900 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.239+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2400aa60 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.240+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c16d910 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.240+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2400ade0 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.240+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c16ded0 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.241+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c0718b0 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.242+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f7e24007300 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.243+0000 7f7e227fc700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e10067130 0x7f7e100695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.243+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f7e24084d90 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.243+0000 7f7e3219b700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e10067130 0x7f7e100695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.244+0000 7f7e3219b700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e10067130 0x7f7e100695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009390 tx=0x7f7e28017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:54.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.247+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e24051c60 con 0x7f7e2c0cf2c0 2023-10-07T09:30:54.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.479+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e2c16e380 con 0x7f7e10067130 2023-10-07T09:30:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.529+0000 7f7e227fc700 1 -- 172.21.15.29:0/1484389931 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7e2c16e380 con 0x7f7e10067130 2023-10-07T09:30:54.531 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e10067130 msgr2=0x7f7e100695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e10067130 0x7f7e100695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009390 tx=0x7f7e28017000 comp rx=0 tx=0).stop 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 msgr2=0x7f7e2c0c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 0x7f7e2c0c6620 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400cbc0 tx=0x7f7e24001990 comp rx=0 tx=0).stop 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 shutdown_connections 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7e10067130 0x7f7e100695e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e2c0cf2c0 0x7f7e2c0c6620 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e2c0b2a20 0x7f7e2c14d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 --2- 172.21.15.29:0/1484389931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e2c0cb6f0 0x7f7e2c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 >> 172.21.15.29:0/1484389931 conn(0x7f7e2c099820 msgr2=0x7f7e2c159040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 shutdown_connections 2023-10-07T09:30:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.532+0000 7f7e343ff700 1 -- 172.21.15.29:0/1484389931 wait complete. 2023-10-07T09:30:54.550 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-07T09:30:54.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir2 1M 2023-10-07T09:30:54.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.750+0000 7fc0c89ff700 1 Processor -- start 2023-10-07T09:30:54.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.750+0000 7fc0c89ff700 1 -- start start 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c89ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0c40b0610 0x7fc0c40ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c89ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc0c40b09e0 0x7fc0c40c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40c8a20 0x7fc0c40cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c89ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c40690f0 con 0x7fc0c40b0610 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c89ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c40c4740 con 0x7fc0c40b09e0 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c40cf2e0 con 0x7fc0c40c8a20 2023-10-07T09:30:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0c40b0610 0x7fc0c40ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0c40b0610 0x7fc0c40ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52600/0 (socket says 172.21.15.29:52600) 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.751+0000 7fc0c259c700 1 -- 172.21.15.29:0/1395353640 learned_addr learned my addr 172.21.15.29:0/1395353640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c1d9b700 1 --2- 172.21.15.29:0/1395353640 >> v2:172.21.15.144:3300/0 conn(0x7fc0c40b09e0 0x7fc0c40c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c2d9d700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40c8a20 0x7fc0c40cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c259c700 1 -- 172.21.15.29:0/1395353640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40c8a20 msgr2=0x7fc0c40cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c259c700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40c8a20 0x7fc0c40cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c259c700 1 -- 172.21.15.29:0/1395353640 >> v2:172.21.15.144:3300/0 conn(0x7fc0c40b09e0 msgr2=0x7fc0c40c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c259c700 1 --2- 172.21.15.29:0/1395353640 >> v2:172.21.15.144:3300/0 conn(0x7fc0c40b09e0 0x7fc0c40c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c259c700 1 -- 172.21.15.29:0/1395353640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c40687c0 con 0x7fc0c40b0610 2023-10-07T09:30:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c2d9d700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40c8a20 0x7fc0c40cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c1d9b700 1 --2- 172.21.15.29:0/1395353640 >> v2:172.21.15.144:3300/0 conn(0x7fc0c40b09e0 0x7fc0c40c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c259c700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b0610 0x7fc0c40ced00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40cc2f0 tx=0x7fc0b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb7421889cf78ea server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.752+0000 7fc0c159a700 1 -- 172.21.15.29:0/1395353640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b400b860 con 0x7fc0c40b0610 2023-10-07T09:30:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c159a700 1 -- 172.21.15.29:0/1395353640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b400b9c0 con 0x7fc0c40b0610 2023-10-07T09:30:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c159a700 1 -- 172.21.15.29:0/1395353640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b400bd60 con 0x7fc0c40b0610 2023-10-07T09:30:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 -- 172.21.15.29:0/1395353640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b0610 msgr2=0x7fc0c40ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b0610 0x7fc0c40ced00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40cc2f0 tx=0x7fc0b4009250 comp rx=0 tx=0).stop 2023-10-07T09:30:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 -- 172.21.15.29:0/1395353640 shutdown_connections 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40c8a20 0x7fc0c40cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/1395353640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b0610 0x7fc0c40ced00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/1395353640 >> v2:172.21.15.144:3300/0 conn(0x7fc0c40b09e0 0x7fc0c40c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 -- 172.21.15.29:0/1395353640 >> 172.21.15.29:0/1395353640 conn(0x7fc0c4063a10 msgr2=0x7fc0c40ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 -- 172.21.15.29:0/1395353640 shutdown_connections 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.753+0000 7fc0c89ff700 1 -- 172.21.15.29:0/1395353640 wait complete. 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.755+0000 7fc0c89ff700 1 Processor -- start 2023-10-07T09:30:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.755+0000 7fc0c89ff700 1 -- start start 2023-10-07T09:30:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40b0610 0x7fc0c4157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c89ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 0x7fc0c40c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c89ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0c40c8a20 0x7fc0c40c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c89ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c409a8a0 con 0x7fc0c40b09e0 2023-10-07T09:30:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c89ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c40c8690 con 0x7fc0c40c8a20 2023-10-07T09:30:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c40c6380 con 0x7fc0c40b0610 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 0x7fc0c40c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 0x7fc0c40c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52616/0 (socket says 172.21.15.29:52616) 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 -- 172.21.15.29:0/3138324965 learned_addr learned my addr 172.21.15.29:0/3138324965 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c259c700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40b0610 0x7fc0c4157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c2d9d700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0c40c8a20 0x7fc0c40c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 -- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40b0610 msgr2=0x7fc0c4157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40b0610 0x7fc0c4157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 -- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0c40c8a20 msgr2=0x7fc0c40c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.756+0000 7fc0c1d9b700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0c40c8a20 0x7fc0c40c33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0c1d9b700 1 -- 172.21.15.29:0/3138324965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40094f0 con 0x7fc0c40b09e0 2023-10-07T09:30:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0c1d9b700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 0x7fc0c40c7250 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800e900 tx=0x7fc0b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800eea0 con 0x7fc0c40b09e0 2023-10-07T09:30:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b800c5e0 con 0x7fc0c40b09e0 2023-10-07T09:30:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800c980 con 0x7fc0c40b09e0 2023-10-07T09:30:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c40c5490 con 0x7fc0c40b09e0 2023-10-07T09:30:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.757+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c40c5a50 con 0x7fc0c40b09e0 2023-10-07T09:30:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.758+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c40c6510 con 0x7fc0c40b09e0 2023-10-07T09:30:54.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.759+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fc0b800cd30 con 0x7fc0c40b09e0 2023-10-07T09:30:54.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.760+0000 7fc0b27fc700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc0a4067200 0x7fc0a40696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:54.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.760+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fc0b80848f0 con 0x7fc0c40b09e0 2023-10-07T09:30:54.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.761+0000 7fc0c259c700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc0a4067200 0x7fc0a40696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:54.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.761+0000 7fc0c259c700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc0a4067200 0x7fc0a40696b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40cc2f0 tx=0x7fc0b4017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:54.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.764+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc0b80516b0 con 0x7fc0c40b09e0 2023-10-07T09:30:54.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:54.995+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0c40718b0 con 0x7fc0a4067200 2023-10-07T09:30:55.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.046+0000 7fc0b27fc700 1 -- 172.21.15.29:0/3138324965 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc0c40718b0 con 0x7fc0a4067200 2023-10-07T09:30:55.048 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc0a4067200 msgr2=0x7fc0a40696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc0a4067200 0x7fc0a40696b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40cc2f0 tx=0x7fc0b4017000 comp rx=0 tx=0).stop 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 msgr2=0x7fc0c40c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 0x7fc0c40c7250 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800e900 tx=0x7fc0b8008040 comp rx=0 tx=0).stop 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 shutdown_connections 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc0a4067200 0x7fc0a40696b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0c40c8a20 0x7fc0c40c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0c40b0610 0x7fc0c4157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 --2- 172.21.15.29:0/3138324965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c40b09e0 0x7fc0c40c7250 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 >> 172.21.15.29:0/3138324965 conn(0x7fc0c4063a10 msgr2=0x7fc0c41593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:55.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 shutdown_connections 2023-10-07T09:30:55.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.049+0000 7fc0c89ff700 1 -- 172.21.15.29:0/3138324965 wait complete. 2023-10-07T09:30:55.067 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-07T09:30:55.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir3 1M 2023-10-07T09:30:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.268+0000 7f18fa5ff700 1 Processor -- start 2023-10-07T09:30:55.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.269+0000 7f18fa5ff700 1 -- start start 2023-10-07T09:30:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18fa5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f18f40cb8c0 0x7f18f40cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18fa5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18f40cf0f0 0x7f18f40c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18fa5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40cc940 con 0x7f18f40b2a20 2023-10-07T09:30:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18fa5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f40cf530 con 0x7f18f40cf0f0 2023-10-07T09:30:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18fa5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f40b2df0 con 0x7f18f40cb8c0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18f8b9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18f40cf0f0 0x7f18f40c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18f8b9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18f40cf0f0 0x7f18f40c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52620/0 (socket says 172.21.15.29:52620) 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.270+0000 7f18f8b9c700 1 -- 172.21.15.29:0/2097135892 learned_addr learned my addr 172.21.15.29:0/2097135892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18ebfff700 1 --2- 172.21.15.29:0/2097135892 >> v2:172.21.15.144:3300/0 conn(0x7f18f40cb8c0 0x7f18f40cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f8b9c700 1 -- 172.21.15.29:0/2097135892 >> v2:172.21.15.144:3300/0 conn(0x7f18f40cb8c0 msgr2=0x7f18f40cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f8b9c700 1 --2- 172.21.15.29:0/2097135892 >> v2:172.21.15.144:3300/0 conn(0x7f18f40cb8c0 0x7f18f40cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f8b9c700 1 -- 172.21.15.29:0/2097135892 >> v1:172.21.15.144:6789/0 conn(0x7f18f40b2a20 legacy=0x7f18f40cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f8b9c700 1 -- 172.21.15.29:0/2097135892 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f40cbd30 con 0x7f18f40cf0f0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f8b9c700 1 --2- 172.21.15.29:0/2097135892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cf0f0 0x7f18f40c5400 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f18f40889b0 tx=0x7f18ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e809b16a089a46ca server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f37fe700 1 -- 172.21.15.29:0/2097135892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec00e070 con 0x7f18f40cf0f0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f37fe700 1 -- 172.21.15.29:0/2097135892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec00ba20 con 0x7f18f40cf0f0 2023-10-07T09:30:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.271+0000 7f18f37fe700 1 -- 172.21.15.29:0/2097135892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0092e0 con 0x7f18f40cf0f0 2023-10-07T09:30:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 -- 172.21.15.29:0/2097135892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cf0f0 msgr2=0x7f18f40c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/2097135892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cf0f0 0x7f18f40c5400 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f18f40889b0 tx=0x7f18ec009840 comp rx=0 tx=0).stop 2023-10-07T09:30:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 -- 172.21.15.29:0/2097135892 shutdown_connections 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/2097135892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cf0f0 0x7f18f40c5400 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/2097135892 >> v2:172.21.15.144:3300/0 conn(0x7f18f40cb8c0 0x7f18f40cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 -- 172.21.15.29:0/2097135892 >> 172.21.15.29:0/2097135892 conn(0x7f18f4099820 msgr2=0x7f18f4063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 -- 172.21.15.29:0/2097135892 shutdown_connections 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.272+0000 7f18fa5ff700 1 -- 172.21.15.29:0/2097135892 wait complete. 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.274+0000 7f18fa5ff700 1 Processor -- start 2023-10-07T09:30:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.274+0000 7f18fa5ff700 1 -- start start 2023-10-07T09:30:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18fa5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18f40b2a20 0x7f18f40c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18fa5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 0x7f18f4143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18fa5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f40cf0f0 0x7f18f415ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18fa5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40b3a20 con 0x7f18f40cb8c0 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18fa5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40ca440 con 0x7f18f40cf0f0 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18fa5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40ca720 con 0x7f18f40b2a20 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 0x7f18f4143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 0x7f18f4143240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52622/0 (socket says 172.21.15.29:52622) 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 -- 172.21.15.29:0/106879968 learned_addr learned my addr 172.21.15.29:0/106879968 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 -- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18f40b2a20 msgr2=0x7f18f40c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18f40b2a20 0x7f18f40c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 -- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f40cf0f0 msgr2=0x7f18f415ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f40cf0f0 0x7f18f415ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.276+0000 7f18ebfff700 1 -- 172.21.15.29:0/106879968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec009ae0 con 0x7f18f40cb8c0 2023-10-07T09:30:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.277+0000 7f18ebfff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 0x7f18f4143240 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00ea50 tx=0x7f18dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.277+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00c920 con 0x7f18f40cb8c0 2023-10-07T09:30:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.277+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc00ca80 con 0x7f18f40cb8c0 2023-10-07T09:30:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.277+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00ce20 con 0x7f18f40cb8c0 2023-10-07T09:30:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.277+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f40caa00 con 0x7f18f40cb8c0 2023-10-07T09:30:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.277+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f40c7ce0 con 0x7f18f40cb8c0 2023-10-07T09:30:55.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.278+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f40718b0 con 0x7f18f40cb8c0 2023-10-07T09:30:55.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.279+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f18dc013680 con 0x7f18f40cb8c0 2023-10-07T09:30:55.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.280+0000 7f18f0ff9700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f18d4067130 0x7f18d40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.280+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f18dc084ee0 con 0x7f18f40cb8c0 2023-10-07T09:30:55.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.280+0000 7f18f3fff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f18d4067130 0x7f18d40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.281+0000 7f18f3fff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f18d4067130 0x7f18d40695e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f18e40016d0 tx=0x7f18e4007b30 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:55.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.284+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18dc051d30 con 0x7f18f40cb8c0 2023-10-07T09:30:55.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.518+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f18f40b3b60 con 0x7f18d4067130 2023-10-07T09:30:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.571+0000 7f18f0ff9700 1 -- 172.21.15.29:0/106879968 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f18e4001320 con 0x7f18d4067130 2023-10-07T09:30:55.573 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-07T09:30:55.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f18d4067130 msgr2=0x7f18d40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f18d4067130 0x7f18d40695e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f18e40016d0 tx=0x7f18e4007b30 comp rx=0 tx=0).stop 2023-10-07T09:30:55.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 msgr2=0x7f18f4143240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 0x7f18f4143240 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00ea50 tx=0x7f18dc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 shutdown_connections 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f18d4067130 0x7f18d40695e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18f40cf0f0 0x7f18f415ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18f40b2a20 0x7f18f40c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 --2- 172.21.15.29:0/106879968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18f40cb8c0 0x7f18f4143240 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 >> 172.21.15.29:0/106879968 conn(0x7f18f4099820 msgr2=0x7f18f40cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 shutdown_connections 2023-10-07T09:30:55.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.574+0000 7f18fa5ff700 1 -- 172.21.15.29:0/106879968 wait complete. 2023-10-07T09:30:55.592 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-07T09:30:55.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir4 1M 2023-10-07T09:30:55.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.792+0000 7f7c167ff700 1 Processor -- start 2023-10-07T09:30:55.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.793+0000 7f7c167ff700 1 -- start start 2023-10-07T09:30:55.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.794+0000 7f7c167ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c100cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.794+0000 7f7c167ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c100cb6f0 0x7f7c100cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.794+0000 7f7c167ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c100cc940 con 0x7f7c100cf0f0 2023-10-07T09:30:55.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.794+0000 7f7c167ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c100cf530 con 0x7f7c100cb6f0 2023-10-07T09:30:55.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.794+0000 7f7c167ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b2df0 con 0x7f7c100b2a20 2023-10-07T09:30:55.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c100cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0f7fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c100cb6f0 0x7f7c100cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c100cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:41738/0 (socket says 172.21.15.29:41738) 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0ffff700 1 -- 172.21.15.29:0/757103292 learned_addr learned my addr 172.21.15.29:0/757103292 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0f7fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c100cb6f0 0x7f7c100cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:45210/0 (socket says 172.21.15.29:45210) 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0f7fe700 1 -- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 msgr2=0x7f7c100cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0f7fe700 1 --2- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c100cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0f7fe700 1 -- 172.21.15.29:0/757103292 >> v1:172.21.15.144:6789/0 conn(0x7f7c100cf0f0 legacy=0x7f7c10159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0f7fe700 1 -- 172.21.15.29:0/757103292 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100cbd30 con 0x7f7c100cb6f0 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.795+0000 7f7c0ffff700 1 --2- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c100cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:30:55.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.796+0000 7f7c0f7fe700 1 --2- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100cebb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c040098e0 tx=0x7f7c040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=accee0f950ef7312 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:55.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.796+0000 7f7c0effd700 1 -- 172.21.15.29:0/757103292 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400b640 con 0x7f7c100cb6f0 2023-10-07T09:30:55.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.796+0000 7f7c0effd700 1 -- 172.21.15.29:0/757103292 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400b7a0 con 0x7f7c100cb6f0 2023-10-07T09:30:55.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.796+0000 7f7c0effd700 1 -- 172.21.15.29:0/757103292 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400bb40 con 0x7f7c100cb6f0 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.796+0000 7f7c167ff700 1 -- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 msgr2=0x7f7c100cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.796+0000 7f7c167ff700 1 --2- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100cebb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c040098e0 tx=0x7f7c040091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.797+0000 7f7c167ff700 1 -- 172.21.15.29:0/757103292 shutdown_connections 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.797+0000 7f7c167ff700 1 --2- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c100cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.797+0000 7f7c167ff700 1 --2- 172.21.15.29:0/757103292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100cebb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.797+0000 7f7c167ff700 1 -- 172.21.15.29:0/757103292 >> 172.21.15.29:0/757103292 conn(0x7f7c10099820 msgr2=0x7f7c10063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.797+0000 7f7c167ff700 1 -- 172.21.15.29:0/757103292 shutdown_connections 2023-10-07T09:30:55.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.797+0000 7f7c167ff700 1 -- 172.21.15.29:0/757103292 wait complete. 2023-10-07T09:30:55.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.799+0000 7f7c167ff700 1 Processor -- start 2023-10-07T09:30:55.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.799+0000 7f7c167ff700 1 -- start start 2023-10-07T09:30:55.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c167ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c10154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c167ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100b5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c167ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 0x7f7c10156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c167ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b46e0 con 0x7f7c100cf0f0 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c167ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b49c0 con 0x7f7c100cb6f0 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c167ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b4ca0 con 0x7f7c100b2a20 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.800+0000 7f7c14d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 0x7f7c10156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 0x7f7c10156450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52626/0 (socket says 172.21.15.29:52626) 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 -- 172.21.15.29:0/2332451631 learned_addr learned my addr 172.21.15.29:0/2332451631 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c0ffff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c10154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c0f7fe700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 -- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 msgr2=0x7f7c10154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c10154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 -- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 msgr2=0x7f7c100b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 -- 172.21.15.29:0/2332451631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00009480 con 0x7f7c100cf0f0 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.801+0000 7f7c14d9c700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 0x7f7c10156450 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c8e0 tx=0x7f7c08001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.802+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08012630 con 0x7f7c100cf0f0 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.802+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c08012790 con 0x7f7c100cf0f0 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.802+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08012b50 con 0x7f7c100cf0f0 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.802+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c04009420 con 0x7f7c100cf0f0 2023-10-07T09:30:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.802+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c100b5260 con 0x7f7c100cf0f0 2023-10-07T09:30:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.803+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c100718b0 con 0x7f7c100cf0f0 2023-10-07T09:30:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.804+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f7c0801a3a0 con 0x7f7c100cf0f0 2023-10-07T09:30:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.805+0000 7f7bf7fff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7bfc067130 0x7f7bfc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.805+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f7c08084bc0 con 0x7f7c100cf0f0 2023-10-07T09:30:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.805+0000 7f7c0ffff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7bfc067130 0x7f7bfc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.806+0000 7f7c0ffff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7bfc067130 0x7f7bfc0695e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009e90 tx=0x7f7c00009220 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:55.809+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c08051a10 con 0x7f7c100cf0f0 2023-10-07T09:30:56.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.041+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c100b5450 con 0x7f7bfc067130 2023-10-07T09:30:56.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.093+0000 7f7bf7fff700 1 -- 172.21.15.29:0/2332451631 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7c100b5450 con 0x7f7bfc067130 2023-10-07T09:30:56.094 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7bfc067130 msgr2=0x7f7bfc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7bfc067130 0x7f7bfc0695e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009e90 tx=0x7f7c00009220 comp rx=0 tx=0).stop 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 msgr2=0x7f7c10156450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 0x7f7c10156450 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c8e0 tx=0x7f7c08001710 comp rx=0 tx=0).stop 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 shutdown_connections 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7bfc067130 0x7f7bfc0695e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c100cf0f0 0x7f7c10156450 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c100b2a20 0x7f7c10154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 --2- 172.21.15.29:0/2332451631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c100cb6f0 0x7f7c100b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.095+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 >> 172.21.15.29:0/2332451631 conn(0x7f7c10099820 msgr2=0x7f7c100683b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:56.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.096+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 shutdown_connections 2023-10-07T09:30:56.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.096+0000 7f7c167ff700 1 -- 172.21.15.29:0/2332451631 wait complete. 2023-10-07T09:30:56.114 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-07T09:30:56.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir5 1M 2023-10-07T09:30:56.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.315+0000 7fb5fc9ff700 1 Processor -- start 2023-10-07T09:30:56.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.315+0000 7fb5fc9ff700 1 -- start start 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.316+0000 7fb5fc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80cb8c0 0x7fb5f80cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5fc9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5f80cf0f0 0x7fb5f80c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5fc9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f80cc940 con 0x7fb5f80b2a20 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5fc9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f80cf530 con 0x7fb5f80cf0f0 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5fc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80b2df0 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5f80cf0f0 0x7fb5f80c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5f80cf0f0 0x7fb5f80c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52642/0 (socket says 172.21.15.29:52642) 2023-10-07T09:30:56.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 -- 172.21.15.29:0/2652579690 learned_addr learned my addr 172.21.15.29:0/2652579690 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:56.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f5d9b700 1 --2- 172.21.15.29:0/2652579690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80cb8c0 0x7fb5f80cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 -- 172.21.15.29:0/2652579690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80cb8c0 msgr2=0x7fb5f80cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:56.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 --2- 172.21.15.29:0/2652579690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80cb8c0 0x7fb5f80cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 -- 172.21.15.29:0/2652579690 >> v1:172.21.15.144:6789/0 conn(0x7fb5f80b2a20 legacy=0x7fb5f80cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.317+0000 7fb5f6d9d700 1 -- 172.21.15.29:0/2652579690 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f80cbd30 con 0x7fb5f80cf0f0 2023-10-07T09:30:56.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5f6d9d700 1 --2- 172.21.15.29:0/2652579690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f80c5400 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009f20 tx=0x7fb5f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa732524d14e7ebb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:56.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5f559a700 1 -- 172.21.15.29:0/2652579690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f000e070 con 0x7fb5f80cf0f0 2023-10-07T09:30:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5f559a700 1 -- 172.21.15.29:0/2652579690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f000b9d0 con 0x7fb5f80cf0f0 2023-10-07T09:30:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5f559a700 1 -- 172.21.15.29:0/2652579690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00092e0 con 0x7fb5f80cf0f0 2023-10-07T09:30:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/2652579690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cf0f0 msgr2=0x7fb5f80c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/2652579690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f80c5400 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009f20 tx=0x7fb5f0009840 comp rx=0 tx=0).stop 2023-10-07T09:30:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.318+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/2652579690 shutdown_connections 2023-10-07T09:30:56.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.319+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/2652579690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f80c5400 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.319+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/2652579690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80cb8c0 0x7fb5f80cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.319+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/2652579690 >> 172.21.15.29:0/2652579690 conn(0x7fb5f8099820 msgr2=0x7fb5f8063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:56.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.319+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/2652579690 shutdown_connections 2023-10-07T09:30:56.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.319+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/2652579690 wait complete. 2023-10-07T09:30:56.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.320+0000 7fb5fc9ff700 1 Processor -- start 2023-10-07T09:30:56.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.320+0000 7fb5fc9ff700 1 -- start start 2023-10-07T09:30:56.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5fc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80b2a20 0x7fb5f815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5fc9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 0x7fb5f80b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5fc9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5fc9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f81567e0 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5fc9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80b4460 con 0x7fb5f80cf0f0 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5fc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80b4740 con 0x7fb5f80b2a20 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 0x7fb5f80b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 0x7fb5f80b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52654/0 (socket says 172.21.15.29:52654) 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5f5d9b700 1 -- 172.21.15.29:0/1571908314 learned_addr learned my addr 172.21.15.29:0/1571908314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5f659c700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80b2a20 0x7fb5f815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.321+0000 7fb5f6d9d700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f5d9b700 1 -- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80b2a20 msgr2=0x7fb5f815b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f5d9b700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80b2a20 0x7fb5f815b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f5d9b700 1 -- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5f80cf0f0 msgr2=0x7fb5f815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f5d9b700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f5d9b700 1 -- 172.21.15.29:0/1571908314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0009ae0 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f6d9d700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f659c700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80b2a20 0x7fb5f815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5f5d9b700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 0x7fb5f80b3d80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00ec50 tx=0x7fb5ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00c9d0 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec00cb30 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec013470 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f8156190 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.322+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f80c7680 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.323+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f8156950 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.324+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fb5ec013820 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.325+0000 7fb5e67fc700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb5d8067130 0x7fb5d80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.325+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fb5ec084af0 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.326+0000 7fb5f659c700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb5d8067130 0x7fb5d80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.326+0000 7fb5f659c700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb5d8067130 0x7fb5d80695e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80018f0 tx=0x7fb5e8009040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:56.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.329+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5ec051940 con 0x7fb5f80cb8c0 2023-10-07T09:30:56.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.559+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5f80718b0 con 0x7fb5d8067130 2023-10-07T09:30:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.613+0000 7fb5e67fc700 1 -- 172.21.15.29:0/1571908314 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb5f80718b0 con 0x7fb5d8067130 2023-10-07T09:30:56.614 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-07T09:30:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb5d8067130 msgr2=0x7fb5d80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb5d8067130 0x7fb5d80695e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80018f0 tx=0x7fb5e8009040 comp rx=0 tx=0).stop 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 msgr2=0x7fb5f80b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 0x7fb5f80b3d80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00ec50 tx=0x7fb5ec0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 shutdown_connections 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb5d8067130 0x7fb5d80695e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5f80cf0f0 0x7fb5f815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5f80b2a20 0x7fb5f815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 --2- 172.21.15.29:0/1571908314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f80cb8c0 0x7fb5f80b3d80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 >> 172.21.15.29:0/1571908314 conn(0x7fb5f8099820 msgr2=0x7fb5f80699c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 shutdown_connections 2023-10-07T09:30:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.615+0000 7fb5fc9ff700 1 -- 172.21.15.29:0/1571908314 wait complete. 2023-10-07T09:30:56.633 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-07T09:30:56.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir6 1M 2023-10-07T09:30:56.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.832+0000 7f9cf3bff700 1 Processor -- start 2023-10-07T09:30:56.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.832+0000 7f9cf3bff700 1 -- start start 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.833+0000 7f9cf3bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9cec0cc810 0x7f9cec0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.833+0000 7f9cf3bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec15a590 0x7f9cec15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.833+0000 7f9cf3bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0ad400 con 0x7f9cec0afeb0 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.833+0000 7f9cf3bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cec15a140 con 0x7f9cec0cc810 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.833+0000 7f9cf3bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0adca0 con 0x7f9cec15a590 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf199b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9cec0afeb0 0x7f9cec0aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39364/0 (socket says 172.21.15.29:39364) 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf199b700 1 -- 172.21.15.29:0/3645686037 learned_addr learned my addr 172.21.15.29:0/3645686037 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf219c700 1 --2- 172.21.15.29:0/3645686037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec15a590 0x7f9cec15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf119a700 1 --2- 172.21.15.29:0/3645686037 >> v2:172.21.15.144:3300/0 conn(0x7f9cec0cc810 0x7f9cec0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf0999700 1 -- 172.21.15.29:0/3645686037 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337594216 0 0) 0x7f9cec0ad400 con 0x7f9cec0afeb0 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf0999700 1 -- 172.21.15.29:0/3645686037 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd8003570 con 0x7f9cec0afeb0 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf219c700 1 -- 172.21.15.29:0/3645686037 >> v2:172.21.15.144:3300/0 conn(0x7f9cec0cc810 msgr2=0x7f9cec0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf219c700 1 --2- 172.21.15.29:0/3645686037 >> v2:172.21.15.144:3300/0 conn(0x7f9cec0cc810 0x7f9cec0accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf219c700 1 -- 172.21.15.29:0/3645686037 >> v1:172.21.15.29:6789/0 conn(0x7f9cec0afeb0 legacy=0x7f9cec0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.834+0000 7f9cf219c700 1 -- 172.21.15.29:0/3645686037 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0ad6e0 con 0x7f9cec15a590 2023-10-07T09:30:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.835+0000 7f9cf219c700 1 --2- 172.21.15.29:0/3645686037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec15a590 0x7f9cec15a9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009e60 tx=0x7f9ce4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d40258b87d9550 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.835+0000 7f9cf0999700 1 -- 172.21.15.29:0/3645686037 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400e070 con 0x7f9cec15a590 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.835+0000 7f9cf0999700 1 -- 172.21.15.29:0/3645686037 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce400bab0 con 0x7f9cec15a590 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.835+0000 7f9cf0999700 1 -- 172.21.15.29:0/3645686037 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400be70 con 0x7f9cec15a590 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 -- 172.21.15.29:0/3645686037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec15a590 msgr2=0x7f9cec15a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/3645686037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec15a590 0x7f9cec15a9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009e60 tx=0x7f9ce4009780 comp rx=0 tx=0).stop 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 -- 172.21.15.29:0/3645686037 shutdown_connections 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/3645686037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec15a590 0x7f9cec15a9c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/3645686037 >> v2:172.21.15.144:3300/0 conn(0x7f9cec0cc810 0x7f9cec0accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 -- 172.21.15.29:0/3645686037 >> 172.21.15.29:0/3645686037 conn(0x7f9cec067610 msgr2=0x7f9cec067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 -- 172.21.15.29:0/3645686037 shutdown_connections 2023-10-07T09:30:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.836+0000 7f9cf3bff700 1 -- 172.21.15.29:0/3645686037 wait complete. 2023-10-07T09:30:56.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.837+0000 7f9cf3bff700 1 Processor -- start 2023-10-07T09:30:56.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.837+0000 7f9cf3bff700 1 -- start start 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.838+0000 7f9cf3bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 0x7f9cec1543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.838+0000 7f9cf3bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec0cc810 0x7f9cec13d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.838+0000 7f9cf3bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cec15a590 0x7f9cec06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.838+0000 7f9cf3bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec154920 con 0x7f9cec0afeb0 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.838+0000 7f9cf3bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec153430 con 0x7f9cec15a590 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.838+0000 7f9cf3bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec16a8f0 con 0x7f9cec0cc810 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 0x7f9cec1543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 0x7f9cec1543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52662/0 (socket says 172.21.15.29:52662) 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 -- 172.21.15.29:0/1899750950 learned_addr learned my addr 172.21.15.29:0/1899750950 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf119a700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec0cc810 0x7f9cec13d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf219c700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cec15a590 0x7f9cec06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 -- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec0cc810 msgr2=0x7f9cec13d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec0cc810 0x7f9cec13d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 -- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cec15a590 msgr2=0x7f9cec06d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cec15a590 0x7f9cec06d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 -- 172.21.15.29:0/1899750950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4009a20 con 0x7f9cec0afeb0 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9cf199b700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 0x7f9cec1543e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c7c0 tx=0x7f9ce8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.839+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8005b30 con 0x7f9cec0afeb0 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.840+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8005cb0 con 0x7f9cec0afeb0 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.840+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec153660 con 0x7f9cec0afeb0 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.840+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8007320 con 0x7f9cec0afeb0 2023-10-07T09:30:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.840+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec153c60 con 0x7f9cec0afeb0 2023-10-07T09:30:56.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.841+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec0718b0 con 0x7f9cec0afeb0 2023-10-07T09:30:56.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.841+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f9ce8007690 con 0x7f9cec0afeb0 2023-10-07T09:30:56.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.842+0000 7f9ce1ffb700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9cd0067130 0x7f9cd00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:56.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.842+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f9ce8083f00 con 0x7f9cec0afeb0 2023-10-07T09:30:56.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.843+0000 7f9cf119a700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9cd0067130 0x7f9cd00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:56.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.844+0000 7f9cf119a700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9cd0067130 0x7f9cd00695e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9cec157070 tx=0x7f9cdc0091a0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:56.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:56.847+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ce8050df0 con 0x7f9cec0afeb0 2023-10-07T09:30:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.080+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cec1596c0 con 0x7f9cd0067130 2023-10-07T09:30:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.130+0000 7f9ce1ffb700 1 -- 172.21.15.29:0/1899750950 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9cec1596c0 con 0x7f9cd0067130 2023-10-07T09:30:57.131 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-07T09:30:57.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9cd0067130 msgr2=0x7f9cd00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9cd0067130 0x7f9cd00695e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9cec157070 tx=0x7f9cdc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 msgr2=0x7f9cec1543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 0x7f9cec1543e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c7c0 tx=0x7f9ce8007e00 comp rx=0 tx=0).stop 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 shutdown_connections 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9cd0067130 0x7f9cd00695e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cec15a590 0x7f9cec06d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec0afeb0 0x7f9cec1543e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 --2- 172.21.15.29:0/1899750950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cec0cc810 0x7f9cec13d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 >> 172.21.15.29:0/1899750950 conn(0x7f9cec067610 msgr2=0x7f9cec154b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:57.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 shutdown_connections 2023-10-07T09:30:57.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.132+0000 7f9cf3bff700 1 -- 172.21.15.29:0/1899750950 wait complete. 2023-10-07T09:30:57.151 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-07T09:30:57.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir7 1M 2023-10-07T09:30:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.352+0000 7f9cb2bff700 1 Processor -- start 2023-10-07T09:30:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.353+0000 7f9cb2bff700 1 -- start start 2023-10-07T09:30:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cb2bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cac0cb8c0 0x7f9cac0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cb2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cb2bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac0cc940 con 0x7f9cac0b2a20 2023-10-07T09:30:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cb2bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac0cf530 con 0x7f9cac0cb8c0 2023-10-07T09:30:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cb2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0b2df0 con 0x7f9cac0cf0f0 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cabfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cac0cb8c0 0x7f9cac0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cabfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cac0cb8c0 0x7f9cac0cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52676/0 (socket says 172.21.15.29:52676) 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.354+0000 7f9cabfff700 1 -- 172.21.15.29:0/1841516625 learned_addr learned my addr 172.21.15.29:0/1841516625 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cb119c700 1 --2- 172.21.15.29:0/1841516625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cabfff700 1 -- 172.21.15.29:0/1841516625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 msgr2=0x7f9cac0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cabfff700 1 --2- 172.21.15.29:0/1841516625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac0c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cabfff700 1 -- 172.21.15.29:0/1841516625 >> v1:172.21.15.144:6789/0 conn(0x7f9cac0b2a20 legacy=0x7f9cac0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cabfff700 1 -- 172.21.15.29:0/1841516625 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac0cbd30 con 0x7f9cac0cb8c0 2023-10-07T09:30:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cabfff700 1 --2- 172.21.15.29:0/1841516625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0cebb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9c980099b0 tx=0x7f9c98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8125f7fb42e16f92 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cab7fe700 1 -- 172.21.15.29:0/1841516625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98009ea0 con 0x7f9cac0cb8c0 2023-10-07T09:30:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.355+0000 7f9cab7fe700 1 -- 172.21.15.29:0/1841516625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9800b860 con 0x7f9cac0cb8c0 2023-10-07T09:30:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cab7fe700 1 -- 172.21.15.29:0/1841516625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800bc00 con 0x7f9cac0cb8c0 2023-10-07T09:30:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 -- 172.21.15.29:0/1841516625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0cb8c0 msgr2=0x7f9cac0cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/1841516625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0cebb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9c980099b0 tx=0x7f9c98009250 comp rx=0 tx=0).stop 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 -- 172.21.15.29:0/1841516625 shutdown_connections 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/1841516625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/1841516625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0cebb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 -- 172.21.15.29:0/1841516625 >> 172.21.15.29:0/1841516625 conn(0x7f9cac099820 msgr2=0x7f9cac063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 -- 172.21.15.29:0/1841516625 shutdown_connections 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.356+0000 7f9cb2bff700 1 -- 172.21.15.29:0/1841516625 wait complete. 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.358+0000 7f9cb2bff700 1 Processor -- start 2023-10-07T09:30:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.358+0000 7f9cb2bff700 1 -- start start 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb2bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 0x7f9cac0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb2bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb2bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac155420 con 0x7f9cac0b2a20 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb2bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac154810 con 0x7f9cac0cb8c0 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac154af0 con 0x7f9cac0cf0f0 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 0x7f9cac0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 0x7f9cac0c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52678/0 (socket says 172.21.15.29:52678) 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb119c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 -- 172.21.15.29:0/3111653646 learned_addr learned my addr 172.21.15.29:0/3111653646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cabfff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 -- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 msgr2=0x7f9cac155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 -- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0cb8c0 msgr2=0x7f9cac0b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.359+0000 7f9cb099b700 1 -- 172.21.15.29:0/3111653646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980094f0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.360+0000 7f9cb099b700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 0x7f9cac0c7f00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000c7c0 tx=0x7f9ca0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.360+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0011590 con 0x7f9cac0b2a20 2023-10-07T09:30:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.360+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca00116f0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.360+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0011ab0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.360+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac154dd0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.360+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac0c8440 con 0x7f9cac0b2a20 2023-10-07T09:30:57.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.361+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac15ae70 con 0x7f9cac0b2a20 2023-10-07T09:30:57.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.362+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f9ca001b3b0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.363+0000 7f9ca8ff9700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9c90067130 0x7f9c900695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.363+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f9ca0083dd0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.364+0000 7f9cabfff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9c90067130 0x7f9c900695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.364+0000 7f9cabfff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9c90067130 0x7f9c900695e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9cac15ae40 tx=0x7f9c98017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:57.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.367+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ca0050cd0 con 0x7f9cac0b2a20 2023-10-07T09:30:57.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.598+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cac0c8630 con 0x7f9c90067130 2023-10-07T09:30:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.650+0000 7f9ca8ff9700 1 -- 172.21.15.29:0/3111653646 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9cac0c8630 con 0x7f9c90067130 2023-10-07T09:30:57.652 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.652+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9c90067130 msgr2=0x7f9c900695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.652+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9c90067130 0x7f9c900695e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9cac15ae40 tx=0x7f9c98017000 comp rx=0 tx=0).stop 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.652+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 msgr2=0x7f9cac0c7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.652+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 0x7f9cac0c7f00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000c7c0 tx=0x7f9ca0007f70 comp rx=0 tx=0).stop 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 shutdown_connections 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9c90067130 0x7f9c900695e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0cf0f0 0x7f9cac155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0b2a20 0x7f9cac0c7f00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 --2- 172.21.15.29:0/3111653646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0cb8c0 0x7f9cac0b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 >> 172.21.15.29:0/3111653646 conn(0x7f9cac099820 msgr2=0x7f9cac157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 shutdown_connections 2023-10-07T09:30:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.653+0000 7f9cb2bff700 1 -- 172.21.15.29:0/3111653646 wait complete. 2023-10-07T09:30:57.671 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-07T09:30:57.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir8 1M 2023-10-07T09:30:57.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.870+0000 7ff6e0fff700 1 Processor -- start 2023-10-07T09:30:57.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.870+0000 7ff6e0fff700 1 -- start start 2023-10-07T09:30:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.871+0000 7ff6e0fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6dc0afeb0 0x7ff6dc0cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.871+0000 7ff6e0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6e0fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc0ad160 con 0x7ff6dc0af4c0 2023-10-07T09:30:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6e0fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc15a140 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6e0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0adca0 con 0x7ff6dc0accf0 2023-10-07T09:30:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6da59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6dc0afeb0 0x7ff6dc0cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6dad9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6dc0af4c0 0x7ff6dc15a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39372/0 (socket says 172.21.15.29:39372) 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6dad9d700 1 -- 172.21.15.29:0/1711397911 learned_addr learned my addr 172.21.15.29:0/1711397911 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6d9d9b700 1 --2- 172.21.15.29:0/1711397911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6da59c700 1 -- 172.21.15.29:0/1711397911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 msgr2=0x7ff6dc0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6da59c700 1 --2- 172.21.15.29:0/1711397911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6d959a700 1 -- 172.21.15.29:0/1711397911 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353944134 0 0) 0x7ff6dc0ad160 con 0x7ff6dc0af4c0 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6da59c700 1 -- 172.21.15.29:0/1711397911 >> v1:172.21.15.29:6789/0 conn(0x7ff6dc0af4c0 legacy=0x7ff6dc15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.872+0000 7ff6da59c700 1 -- 172.21.15.29:0/1711397911 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0ad400 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.873+0000 7ff6da59c700 1 --2- 172.21.15.29:0/1711397911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cc810 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0098e0 tx=0x7ff6cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55c4d748854be94 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:57.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.873+0000 7ff6d959a700 1 -- 172.21.15.29:0/1711397911 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00b640 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.873+0000 7ff6d959a700 1 -- 172.21.15.29:0/1711397911 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc00b7a0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.873+0000 7ff6d959a700 1 -- 172.21.15.29:0/1711397911 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00bb40 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.873+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1711397911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 msgr2=0x7ff6dc0cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.873+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1711397911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cc810 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0098e0 tx=0x7ff6cc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.874+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1711397911 shutdown_connections 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.874+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1711397911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cc810 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.874+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1711397911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.874+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1711397911 >> 172.21.15.29:0/1711397911 conn(0x7ff6dc067610 msgr2=0x7ff6dc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.874+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1711397911 shutdown_connections 2023-10-07T09:30:57.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.874+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1711397911 wait complete. 2023-10-07T09:30:57.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.876+0000 7ff6e0fff700 1 Processor -- start 2023-10-07T09:30:57.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.876+0000 7ff6e0fff700 1 -- start start 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6e0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6e0fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6dc0af4c0 0x7ff6dc154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6e0fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6e0fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0ca9e0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6e0fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0c9610 con 0x7ff6dc0af4c0 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6e0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0c9870 con 0x7ff6dc0accf0 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.877+0000 7ff6dad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52704/0 (socket says 172.21.15.29:52704) 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 -- 172.21.15.29:0/1408372498 learned_addr learned my addr 172.21.15.29:0/1408372498 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6da59c700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6d9d9b700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6dc0af4c0 0x7ff6dc154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 -- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 msgr2=0x7ff6dc0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 -- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6dc0af4c0 msgr2=0x7ff6dc154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6dc0af4c0 0x7ff6dc154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 -- 172.21.15.29:0/1408372498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc009420 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6da59c700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6d9d9b700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6dc0af4c0 0x7ff6dc154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6dad9d700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cb110 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400cc90 tx=0x7ff6d4001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d400cea0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d400ab30 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40070f0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.878+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc0c9b50 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.879+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc0cabf0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.880+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0718b0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.880+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7ff6d40074a0 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.882+0000 7ff6ca7fc700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff6bc067130 0x7ff6bc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.882+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ff6d4084f10 con 0x7ff6dc0afeb0 2023-10-07T09:30:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.882+0000 7ff6da59c700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff6bc067130 0x7ff6bc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.883+0000 7ff6da59c700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff6bc067130 0x7ff6bc0695e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0091e0 tx=0x7ff6cc017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:57.885+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff6d4051d50 con 0x7ff6dc0afeb0 2023-10-07T09:30:58.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.115+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6dc0cd020 con 0x7ff6bc067130 2023-10-07T09:30:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.169+0000 7ff6ca7fc700 1 -- 172.21.15.29:0/1408372498 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff6dc0cd020 con 0x7ff6bc067130 2023-10-07T09:30:58.170 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.171+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff6bc067130 msgr2=0x7ff6bc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.171+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff6bc067130 0x7ff6bc0695e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0091e0 tx=0x7ff6cc017000 comp rx=0 tx=0).stop 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.171+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 msgr2=0x7ff6dc0cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.171+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cb110 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400cc90 tx=0x7ff6d4001a20 comp rx=0 tx=0).stop 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 shutdown_connections 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff6bc067130 0x7ff6bc0695e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6dc0afeb0 0x7ff6dc0cb110 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6dc0accf0 0x7ff6dc0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 --2- 172.21.15.29:0/1408372498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6dc0af4c0 0x7ff6dc154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 >> 172.21.15.29:0/1408372498 conn(0x7ff6dc067610 msgr2=0x7ff6dc1579c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 shutdown_connections 2023-10-07T09:30:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.172+0000 7ff6e0fff700 1 -- 172.21.15.29:0/1408372498 wait complete. 2023-10-07T09:30:58.189 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-07T09:30:58.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir9 1M 2023-10-07T09:30:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.389+0000 7fa70b3ff700 1 Processor -- start 2023-10-07T09:30:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.389+0000 7fa70b3ff700 1 -- start start 2023-10-07T09:30:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70b3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa70415a730 0x7fa7040c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70b3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7040c1ca0 0x7fa7040ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7040caad0 0x7fa70415d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70b3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040caf70 con 0x7fa7040c1ca0 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70b3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa704067610 con 0x7fa70415a730 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040c2140 con 0x7fa7040caad0 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7040c1ca0 0x7fa7040ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7040c1ca0 0x7fa7040ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52708/0 (socket says 172.21.15.29:52708) 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 -- 172.21.15.29:0/136736992 learned_addr learned my addr 172.21.15.29:0/136736992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70919b700 1 --2- 172.21.15.29:0/136736992 >> v2:172.21.15.144:3300/0 conn(0x7fa70415a730 0x7fa7040c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70999c700 1 --2- 172.21.15.29:0/136736992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7040caad0 0x7fa70415d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 -- 172.21.15.29:0/136736992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7040caad0 msgr2=0x7fa70415d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 --2- 172.21.15.29:0/136736992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7040caad0 0x7fa70415d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 -- 172.21.15.29:0/136736992 >> v2:172.21.15.144:3300/0 conn(0x7fa70415a730 msgr2=0x7fa7040c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:30:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 --2- 172.21.15.29:0/136736992 >> v2:172.21.15.144:3300/0 conn(0x7fa70415a730 0x7fa7040c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.391+0000 7fa70899a700 1 -- 172.21.15.29:0/136736992 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040c2420 con 0x7fa7040c1ca0 2023-10-07T09:30:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.392+0000 7fa70899a700 1 --2- 172.21.15.29:0/136736992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040c1ca0 0x7fa7040ca590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa70406f4b0 tx=0x7fa6f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf0ed8467313e34 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.392+0000 7fa6fbfff700 1 -- 172.21.15.29:0/136736992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4009e70 con 0x7fa7040c1ca0 2023-10-07T09:30:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.392+0000 7fa6fbfff700 1 -- 172.21.15.29:0/136736992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f400b860 con 0x7fa7040c1ca0 2023-10-07T09:30:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.392+0000 7fa6fbfff700 1 -- 172.21.15.29:0/136736992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f400bc00 con 0x7fa7040c1ca0 2023-10-07T09:30:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 -- 172.21.15.29:0/136736992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040c1ca0 msgr2=0x7fa7040ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/136736992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040c1ca0 0x7fa7040ca590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa70406f4b0 tx=0x7fa6f4009250 comp rx=0 tx=0).stop 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 -- 172.21.15.29:0/136736992 shutdown_connections 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/136736992 >> v2:172.21.15.144:3300/0 conn(0x7fa70415a730 0x7fa7040c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/136736992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7040caad0 0x7fa70415d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/136736992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040c1ca0 0x7fa7040ca590 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 -- 172.21.15.29:0/136736992 >> 172.21.15.29:0/136736992 conn(0x7fa7040c9430 msgr2=0x7fa7040c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 -- 172.21.15.29:0/136736992 shutdown_connections 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.393+0000 7fa70b3ff700 1 -- 172.21.15.29:0/136736992 wait complete. 2023-10-07T09:30:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.394+0000 7fa70b3ff700 1 Processor -- start 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.394+0000 7fa70b3ff700 1 -- start start 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.395+0000 7fa70b3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7040c1ca0 0x7fa70414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70b3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 0x7fa7040aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70415a730 0x7fa7040aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70b3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b1490 con 0x7fa7040caad0 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70b3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b16e0 con 0x7fa7040c1ca0 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b19c0 con 0x7fa70415a730 2023-10-07T09:30:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 0x7fa7040aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 0x7fa7040aa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52720/0 (socket says 172.21.15.29:52720) 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 -- 172.21.15.29:0/253645892 learned_addr learned my addr 172.21.15.29:0/253645892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70919b700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7040c1ca0 0x7fa70414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70999c700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70415a730 0x7fa7040aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 -- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70415a730 msgr2=0x7fa7040aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70415a730 0x7fa7040aade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 -- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7040c1ca0 msgr2=0x7fa70414a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7040c1ca0 0x7fa70414a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 -- 172.21.15.29:0/253645892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40094f0 con 0x7fa7040caad0 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.396+0000 7fa70899a700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 0x7fa7040aa8a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa70406d6c0 tx=0x7fa6f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.397+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4000d10 con 0x7fa7040caad0 2023-10-07T09:30:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.397+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f400bbe0 con 0x7fa7040caad0 2023-10-07T09:30:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.397+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4011670 con 0x7fa7040caad0 2023-10-07T09:30:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.397+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7040b1ca0 con 0x7fa7040caad0 2023-10-07T09:30:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.397+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7040b24a0 con 0x7fa7040caad0 2023-10-07T09:30:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.398+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040b21c0 con 0x7fa7040caad0 2023-10-07T09:30:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.399+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fa6f4011aa0 con 0x7fa7040caad0 2023-10-07T09:30:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.400+0000 7fa6f97fa700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6e8067130 0x7fa6e80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:30:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.400+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fa6f4085e20 con 0x7fa7040caad0 2023-10-07T09:30:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.400+0000 7fa70919b700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6e8067130 0x7fa6e80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:30:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.402+0000 7fa70919b700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6e8067130 0x7fa6e80695e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa700009db0 tx=0x7fa7000091a0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:30:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.405+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7040b21c0 con 0x7fa7040caad0 2023-10-07T09:30:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.639+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7040b21c0 con 0x7fa6e8067130 2023-10-07T09:30:58.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.693+0000 7fa6f97fa700 1 -- 172.21.15.29:0/253645892 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa7040b21c0 con 0x7fa6e8067130 2023-10-07T09:30:58.694 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6e8067130 msgr2=0x7fa6e80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6e8067130 0x7fa6e80695e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa700009db0 tx=0x7fa7000091a0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 msgr2=0x7fa7040aa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 0x7fa7040aa8a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa70406d6c0 tx=0x7fa6f4017000 comp rx=0 tx=0).stop 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 shutdown_connections 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6e8067130 0x7fa6e80695e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70415a730 0x7fa7040aade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7040c1ca0 0x7fa70414a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 --2- 172.21.15.29:0/253645892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7040caad0 0x7fa7040aa8a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:30:58.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.695+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 >> 172.21.15.29:0/253645892 conn(0x7fa7040c9430 msgr2=0x7fa704157930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:30:58.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.696+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 shutdown_connections 2023-10-07T09:30:58.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:30:58.696+0000 7fa70b3ff700 1 -- 172.21.15.29:0/253645892 wait complete. 2023-10-07T09:30:58.714 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-07T09:31:00.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:31:00.023+0000 7f7f9674c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:31:00.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:31:00.023+0000 7f7f9674c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:31:00.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:31:00.023+0000 7f7f9674c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:31:00.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:31:00.023+0000 7f7f9674c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:31:00.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:31:00.023+0000 7f7f9674c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:31:17.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:31:18.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:31:48.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:31:48.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:32:18.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:32:18.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:32:48.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:32:48.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:33:18.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:33:18.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:33:48.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:33:48.624 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:34:18.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:34:18.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:34:48.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:34:48.860 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:35:00.450 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:35:00.449+0000 7f7fa1762700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:35:00.450 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:35:00.449+0000 7f7fa1762700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:35:00.450 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:35:00.449+0000 7f7fa1762700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:35:00.451 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:35:00.449+0000 7f7fa1762700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:35:00.451 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:35:00.449+0000 7f7fa1762700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:35:18.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:35:19.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:35:49.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:35:49.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:36:19.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:36:19.253 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:36:49.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:36:49.361 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:37:19.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:37:19.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:37:49.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:37:49.593 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:38:19.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:38:19.709 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:38:49.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:38:49.842 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:39:19.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:39:19.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:39:50.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:39:50.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:40:20.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:40:20.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:40:50.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:40:50.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:41:20.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:41:20.509 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:41:20.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir0 1M 2023-10-07T09:41:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.922+0000 7f6ffbbff700 1 Processor -- start 2023-10-07T09:41:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.922+0000 7f6ffbbff700 1 -- start start 2023-10-07T09:41:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.923+0000 7f6ffbbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ff41559f0 0x7f6ff40b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.923+0000 7f6ffbbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff41596d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.923+0000 7f6ffbbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40c8c70 con 0x7f6ff4159c10 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.923+0000 7f6ffbbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff40c98f0 con 0x7f6ff41559f0 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.923+0000 7f6ffbbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff415a090 con 0x7f6ff40b0660 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff41596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff41596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:41680/0 (socket says 172.21.15.29:41680) 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 -- 172.21.15.29:0/3115625608 learned_addr learned my addr 172.21.15.29:0/3115625608 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ffa19c700 1 --1- 172.21.15.29:0/3115625608 >> v1:172.21.15.144:6789/0 conn(0x7f6ff4159c10 0x7f6ff40adb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:39772/0 (socket says 172.21.15.29:39772) 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff999b700 1 --2- 172.21.15.29:0/3115625608 >> v2:172.21.15.144:3300/0 conn(0x7f6ff41559f0 0x7f6ff40b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff8999700 1 -- 172.21.15.29:0/3115625608 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508200499 0 0) 0x7f6ff40c8c70 con 0x7f6ff4159c10 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 -- 172.21.15.29:0/3115625608 >> v2:172.21.15.144:3300/0 conn(0x7f6ff41559f0 msgr2=0x7f6ff40b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 --2- 172.21.15.29:0/3115625608 >> v2:172.21.15.144:3300/0 conn(0x7f6ff41559f0 0x7f6ff40b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 -- 172.21.15.29:0/3115625608 >> v1:172.21.15.144:6789/0 conn(0x7f6ff4159c10 legacy=0x7f6ff40adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.924+0000 7f6ff919a700 1 -- 172.21.15.29:0/3115625608 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40c8f50 con 0x7f6ff40b0660 2023-10-07T09:41:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.925+0000 7f6ff919a700 1 --2- 172.21.15.29:0/3115625608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff41596d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40099b0 tx=0x7f6fe4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abdbc04aa0ee7282 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.925+0000 7f6ff8999700 1 -- 172.21.15.29:0/3115625608 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4009ea0 con 0x7f6ff40b0660 2023-10-07T09:41:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.925+0000 7f6ff8999700 1 -- 172.21.15.29:0/3115625608 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe400b860 con 0x7f6ff40b0660 2023-10-07T09:41:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.925+0000 7f6ff8999700 1 -- 172.21.15.29:0/3115625608 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400bc20 con 0x7f6ff40b0660 2023-10-07T09:41:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.925+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3115625608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 msgr2=0x7f6ff41596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.925+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3115625608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff41596d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40099b0 tx=0x7f6fe4009250 comp rx=0 tx=0).stop 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.926+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3115625608 shutdown_connections 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.926+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3115625608 >> v2:172.21.15.144:3300/0 conn(0x7f6ff41559f0 0x7f6ff40b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.926+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3115625608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff41596d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.926+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3115625608 >> 172.21.15.29:0/3115625608 conn(0x7f6ff4067610 msgr2=0x7f6ff4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.926+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3115625608 shutdown_connections 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.926+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3115625608 wait complete. 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.927+0000 7f6ffbbff700 1 Processor -- start 2023-10-07T09:41:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.927+0000 7f6ffbbff700 1 -- start start 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffbbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffbbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff41559f0 0x7f6ff415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffbbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 0x7f6ff415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffbbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff414b4d0 con 0x7f6ff4159c10 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffbbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff414b990 con 0x7f6ff41559f0 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffbbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff415a940 con 0x7f6ff40b0660 2023-10-07T09:41:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.928+0000 7f6ffa19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 0x7f6ff415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 0x7f6ff415c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41236/0 (socket says 172.21.15.29:41236) 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 -- 172.21.15.29:0/3558590219 learned_addr learned my addr 172.21.15.29:0/3558590219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ff999b700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ff919a700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff41559f0 0x7f6ff415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 -- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 msgr2=0x7f6ff414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 -- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff41559f0 msgr2=0x7f6ff415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff41559f0 0x7f6ff415b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 -- 172.21.15.29:0/3558590219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40094f0 con 0x7f6ff4159c10 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6ffa19c700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 0x7f6ff415c2c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ccc0 tx=0x7f6fec001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00ce00 con 0x7f6ff4159c10 2023-10-07T09:41:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec005df0 con 0x7f6ff4159c10 2023-10-07T09:41:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.929+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00abb0 con 0x7f6ff4159c10 2023-10-07T09:41:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.930+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff415abe0 con 0x7f6ff4159c10 2023-10-07T09:41:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.930+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff415b1a0 con 0x7f6ff4159c10 2023-10-07T09:41:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.931+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff40718b0 con 0x7f6ff4159c10 2023-10-07T09:41:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.931+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f6fec0070f0 con 0x7f6ff4159c10 2023-10-07T09:41:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.932+0000 7f6fe9ffb700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6fd8067130 0x7f6fd80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.932+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f6fec084bd0 con 0x7f6ff4159c10 2023-10-07T09:41:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.933+0000 7f6ff999b700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6fd8067130 0x7f6fd80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.933+0000 7f6ff999b700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6fd8067130 0x7f6fd80695e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009d40 tx=0x7f6ff0008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:20.937+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fec051aa0 con 0x7f6ff4159c10 2023-10-07T09:41:21.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.170+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ff40c9230 con 0x7f6fd8067130 2023-10-07T09:41:21.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.240+0000 7f6fe9ffb700 1 -- 172.21.15.29:0/3558590219 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6ff40c9230 con 0x7f6fd8067130 2023-10-07T09:41:21.242 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-07T09:41:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.242+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6fd8067130 msgr2=0x7f6fd80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.242+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6fd8067130 0x7f6fd80695e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009d40 tx=0x7f6ff0008040 comp rx=0 tx=0).stop 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.242+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 msgr2=0x7f6ff415c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.242+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 0x7f6ff415c2c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ccc0 tx=0x7f6fec001a70 comp rx=0 tx=0).stop 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 shutdown_connections 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6fd8067130 0x7f6fd80695e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff4159c10 0x7f6ff415c2c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ff40b0660 0x7f6ff414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 --2- 172.21.15.29:0/3558590219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff41559f0 0x7f6ff415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 >> 172.21.15.29:0/3558590219 conn(0x7f6ff4067610 msgr2=0x7f6ff40af030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 shutdown_connections 2023-10-07T09:41:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.243+0000 7f6ffbbff700 1 -- 172.21.15.29:0/3558590219 wait complete. 2023-10-07T09:41:21.261 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-07T09:41:21.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir1 1M 2023-10-07T09:41:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.461+0000 7f20d7dff700 1 Processor -- start 2023-10-07T09:41:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.462+0000 7f20d7dff700 1 -- start start 2023-10-07T09:41:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d7dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20d0157650 0x7f20d00ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d7dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d00ca430 con 0x7f20d0157a20 2023-10-07T09:41:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d7dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d00ca710 con 0x7f20d00ae910 2023-10-07T09:41:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d7dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d00ca9f0 con 0x7f20d0157650 2023-10-07T09:41:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d5b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20d0157650 0x7f20d00ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d639c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f20d0157a20 0x7f20d00cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43754/0 (socket says 172.21.15.29:43754) 2023-10-07T09:41:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.463+0000 7f20d639c700 1 -- 172.21.15.29:0/2065023745 learned_addr learned my addr 172.21.15.29:0/2065023745 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d5b9b700 1 -- 172.21.15.29:0/2065023745 >> v1:172.21.15.144:6789/0 conn(0x7f20d00ae910 legacy=0x7f20d00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d5b9b700 1 -- 172.21.15.29:0/2065023745 >> v1:172.21.15.29:6789/0 conn(0x7f20d0157a20 legacy=0x7f20d00cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d5b9b700 1 -- 172.21.15.29:0/2065023745 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d00cacd0 con 0x7f20d0157650 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d4b99700 1 -- 172.21.15.29:0/2065023745 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783869798 0 0) 0x7f20d00ca430 con 0x7f20d0157a20 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d5b9b700 1 --2- 172.21.15.29:0/2065023745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157650 0x7f20d00ae3d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009910 tx=0x7f20cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8359f9dd02e0a329 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d4b99700 1 -- 172.21.15.29:0/2065023745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc00b820 con 0x7f20d0157650 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d4b99700 1 -- 172.21.15.29:0/2065023745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20cc00b980 con 0x7f20d0157650 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.464+0000 7f20d4b99700 1 -- 172.21.15.29:0/2065023745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc00bd20 con 0x7f20d0157650 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 -- 172.21.15.29:0/2065023745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157650 msgr2=0x7f20d00ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 --2- 172.21.15.29:0/2065023745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157650 0x7f20d00ae3d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009910 tx=0x7f20cc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 -- 172.21.15.29:0/2065023745 shutdown_connections 2023-10-07T09:41:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 --2- 172.21.15.29:0/2065023745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157650 0x7f20d00ae3d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 -- 172.21.15.29:0/2065023745 >> 172.21.15.29:0/2065023745 conn(0x7f20d0067610 msgr2=0x7f20d0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 -- 172.21.15.29:0/2065023745 shutdown_connections 2023-10-07T09:41:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.465+0000 7f20d7dff700 1 -- 172.21.15.29:0/2065023745 wait complete. 2023-10-07T09:41:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.466+0000 7f20d7dff700 1 Processor -- start 2023-10-07T09:41:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.467+0000 7f20d7dff700 1 -- start start 2023-10-07T09:41:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.467+0000 7f20d7dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20d0157650 0x7f20d00afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.467+0000 7f20d7dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 0x7f20d0152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d7dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20d0155ce0 0x7f20d0154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d7dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d015c720 con 0x7f20d0157a20 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d7dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00b0480 con 0x7f20d0155ce0 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d7dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00b0760 con 0x7f20d0157650 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 0x7f20d0152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 0x7f20d0152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41266/0 (socket says 172.21.15.29:41266) 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 -- 172.21.15.29:0/1915301759 learned_addr learned my addr 172.21.15.29:0/1915301759 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d639c700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20d0155ce0 0x7f20d0154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d5b9b700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20d0157650 0x7f20d00afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 -- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20d0157650 msgr2=0x7f20d00afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20d0157650 0x7f20d00afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 -- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20d0155ce0 msgr2=0x7f20d0154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20d0155ce0 0x7f20d0154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d539a700 1 -- 172.21.15.29:0/1915301759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc009420 con 0x7f20d0157a20 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.468+0000 7f20d5b9b700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20d0157650 0x7f20d00afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.469+0000 7f20d539a700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 0x7f20d0152d00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c590 tx=0x7f20c000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.469+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0011590 con 0x7f20d0157a20 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.469+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0011710 con 0x7f20d0157a20 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.469+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d00b0a40 con 0x7f20d0157a20 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.469+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0011ad0 con 0x7f20d0157a20 2023-10-07T09:41:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.469+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d0153240 con 0x7f20d0157a20 2023-10-07T09:41:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.470+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d00c9c40 con 0x7f20d0157a20 2023-10-07T09:41:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.471+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f20c001b3b0 con 0x7f20d0157a20 2023-10-07T09:41:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.472+0000 7f20c5ffb700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f20b4067130 0x7f20b40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.472+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f20c0083f80 con 0x7f20d0157a20 2023-10-07T09:41:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.473+0000 7f20d5b9b700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f20b4067130 0x7f20b40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.473+0000 7f20d5b9b700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f20b4067130 0x7f20b40695e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0093f0 tx=0x7f20cc017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.476+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20c0050dd0 con 0x7f20d0157a20 2023-10-07T09:41:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.709+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d00718b0 con 0x7f20b4067130 2023-10-07T09:41:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.785+0000 7f20c5ffb700 1 -- 172.21.15.29:0/1915301759 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f20d00718b0 con 0x7f20b4067130 2023-10-07T09:41:21.786 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-07T09:41:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f20b4067130 msgr2=0x7f20b40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f20b4067130 0x7f20b40695e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0093f0 tx=0x7f20cc017000 comp rx=0 tx=0).stop 2023-10-07T09:41:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 msgr2=0x7f20d0152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 0x7f20d0152d00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c590 tx=0x7f20c000c8a0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 shutdown_connections 2023-10-07T09:41:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f20b4067130 0x7f20b40695e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20d0155ce0 0x7f20d0154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20d0157650 0x7f20d00afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 --2- 172.21.15.29:0/1915301759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20d0157a20 0x7f20d0152d00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 >> 172.21.15.29:0/1915301759 conn(0x7f20d0067610 msgr2=0x7f20d00b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 shutdown_connections 2023-10-07T09:41:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:21.787+0000 7f20d7dff700 1 -- 172.21.15.29:0/1915301759 wait complete. 2023-10-07T09:41:21.805 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-07T09:41:21.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir2 1M 2023-10-07T09:41:22.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.006+0000 7f873c7ff700 1 Processor -- start 2023-10-07T09:41:22.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.006+0000 7f873c7ff700 1 -- start start 2023-10-07T09:41:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.007+0000 7f873c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.007+0000 7f873c7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87340c8a20 0x7f87340cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.007+0000 7f873c7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87340690f0 con 0x7f87340b09e0 2023-10-07T09:41:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.007+0000 7f873c7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87340cf2e0 con 0x7f87340c8a20 2023-10-07T09:41:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.007+0000 7f873c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87340687c0 con 0x7f87340b0610 2023-10-07T09:41:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87340c8a20 0x7f87340cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87340c8a20 0x7f87340cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41274/0 (socket says 172.21.15.29:41274) 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 -- 172.21.15.29:0/1108963461 learned_addr learned my addr 172.21.15.29:0/1108963461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f8739d9a700 1 --1- 172.21.15.29:0/1108963461 >> v1:172.21.15.29:6789/0 conn(0x7f87340b09e0 0x7f87340c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43762/0 (socket says 172.21.15.29:43762) 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873a59b700 1 --2- 172.21.15.29:0/1108963461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 -- 172.21.15.29:0/1108963461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 msgr2=0x7f87340ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 --2- 172.21.15.29:0/1108963461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 -- 172.21.15.29:0/1108963461 >> v1:172.21.15.29:6789/0 conn(0x7f87340b09e0 legacy=0x7f87340c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f873ad9c700 1 -- 172.21.15.29:0/1108963461 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734068aa0 con 0x7f87340c8a20 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.008+0000 7f8739599700 1 -- 172.21.15.29:0/1108963461 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615498662 0 0) 0x7f87340690f0 con 0x7f87340b09e0 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873ad9c700 1 --2- 172.21.15.29:0/1108963461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340c8a20 0x7f87340cc7e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f872c009f20 tx=0x7f872c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbdfc7d651f96fd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f8739599700 1 -- 172.21.15.29:0/1108963461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00e070 con 0x7f87340c8a20 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f8739599700 1 -- 172.21.15.29:0/1108963461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c00ba80 con 0x7f87340c8a20 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f8739599700 1 -- 172.21.15.29:0/1108963461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0092e0 con 0x7f87340c8a20 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873c7ff700 1 -- 172.21.15.29:0/1108963461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340c8a20 msgr2=0x7f87340cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1108963461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340c8a20 0x7f87340cc7e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f872c009f20 tx=0x7f872c009840 comp rx=0 tx=0).stop 2023-10-07T09:41:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873c7ff700 1 -- 172.21.15.29:0/1108963461 shutdown_connections 2023-10-07T09:41:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1108963461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340c8a20 0x7f87340cc7e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1108963461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.009+0000 7f873c7ff700 1 -- 172.21.15.29:0/1108963461 >> 172.21.15.29:0/1108963461 conn(0x7f8734063a10 msgr2=0x7f87340ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.010+0000 7f873c7ff700 1 -- 172.21.15.29:0/1108963461 shutdown_connections 2023-10-07T09:41:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.010+0000 7f873c7ff700 1 -- 172.21.15.29:0/1108963461 wait complete. 2023-10-07T09:41:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.011+0000 7f873c7ff700 1 Processor -- start 2023-10-07T09:41:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.011+0000 7f873c7ff700 1 -- start start 2023-10-07T09:41:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 0x7f87340caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87340c8a20 0x7f87340c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340cbba0 con 0x7f87340b09e0 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340b1910 con 0x7f87340c8a20 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87340cb300 con 0x7f87340b0610 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f8739d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 0x7f87340caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f8739d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 0x7f87340caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41280/0 (socket says 172.21.15.29:41280) 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f8739d9a700 1 -- 172.21.15.29:0/1589596433 learned_addr learned my addr 172.21.15.29:0/1589596433 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.012+0000 7f873a59b700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f873ad9c700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87340c8a20 0x7f87340c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f8739d9a700 1 -- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 msgr2=0x7f87340ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f8739d9a700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f8739d9a700 1 -- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87340c8a20 msgr2=0x7f87340c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f8739d9a700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87340c8a20 0x7f87340c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f8739d9a700 1 -- 172.21.15.29:0/1589596433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c009ae0 con 0x7f87340b09e0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f8739d9a700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 0x7f87340caad0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f87240011e0 tx=0x7f8724007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724005a70 con 0x7f87340b09e0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724005bd0 con 0x7f87340b09e0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87340cb5e0 con 0x7f87340b09e0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.013+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87240070d0 con 0x7f87340b09e0 2023-10-07T09:41:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.014+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87340cb010 con 0x7f87340b09e0 2023-10-07T09:41:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.014+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87340b1b10 con 0x7f87340b09e0 2023-10-07T09:41:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.015+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f87240074c0 con 0x7f87340b09e0 2023-10-07T09:41:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.016+0000 7f872a7fc700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8718067130 0x7f87180695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.016+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f8724083e60 con 0x7f87340b09e0 2023-10-07T09:41:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.017+0000 7f873a59b700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8718067130 0x7f87180695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.018+0000 7f873a59b700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8718067130 0x7f87180695e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8730006fd0 tx=0x7f8730008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.020+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87340b1b10 con 0x7f87340b09e0 2023-10-07T09:41:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.250+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f87340b1ae0 con 0x7f8718067130 2023-10-07T09:41:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.328+0000 7f872a7fc700 1 -- 172.21.15.29:0/1589596433 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f87340b1ae0 con 0x7f8718067130 2023-10-07T09:41:22.330 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.330+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8718067130 msgr2=0x7f87180695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.330+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8718067130 0x7f87180695e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8730006fd0 tx=0x7f8730008040 comp rx=0 tx=0).stop 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.330+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 msgr2=0x7f87340caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.330+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 0x7f87340caad0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f87240011e0 tx=0x7f8724007e10 comp rx=0 tx=0).stop 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 shutdown_connections 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8718067130 0x7f87180695e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87340c8a20 0x7f87340c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87340b0610 0x7f87340ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 --2- 172.21.15.29:0/1589596433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87340b09e0 0x7f87340caad0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 >> 172.21.15.29:0/1589596433 conn(0x7f8734063a10 msgr2=0x7f8734158910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 shutdown_connections 2023-10-07T09:41:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.331+0000 7f873c7ff700 1 -- 172.21.15.29:0/1589596433 wait complete. 2023-10-07T09:41:22.349 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-07T09:41:22.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir3 1M 2023-10-07T09:41:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.550+0000 7f63671ff700 1 Processor -- start 2023-10-07T09:41:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.550+0000 7f63671ff700 1 -- start start 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.551+0000 7f63671ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600afeb0 0x7f63600cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.551+0000 7f63671ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63600accf0 0x7f63600aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.551+0000 7f63671ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600ad160 con 0x7f63600af4c0 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.551+0000 7f63671ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636015a140 con 0x7f63600accf0 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.551+0000 7f63671ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600adca0 con 0x7f63600afeb0 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63600accf0 0x7f63600aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f636579c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f63600af4c0 0x7f636015a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43772/0 (socket says 172.21.15.29:43772) 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f636579c700 1 -- 172.21.15.29:0/3068548793 learned_addr learned my addr 172.21.15.29:0/3068548793 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63600accf0 0x7f63600aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41286/0 (socket says 172.21.15.29:41286) 2023-10-07T09:41:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f6364f9b700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600afeb0 0x7f63600cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 -- 172.21.15.29:0/3068548793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600afeb0 msgr2=0x7f63600cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600afeb0 0x7f63600cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 -- 172.21.15.29:0/3068548793 >> v1:172.21.15.29:6789/0 conn(0x7f63600af4c0 legacy=0x7f636015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 -- 172.21.15.29:0/3068548793 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600ad400 con 0x7f63600accf0 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f6364f9b700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600afeb0 0x7f63600cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635f7fe700 1 -- 172.21.15.29:0/3068548793 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216181796 0 0) 0x7f63600ad160 con 0x7f63600af4c0 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.552+0000 7f635ffff700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600accf0 0x7f63600aef80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f634c0098e0 tx=0x7f634c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658c5a6a2cd8110c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f635f7fe700 1 -- 172.21.15.29:0/3068548793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c009e80 con 0x7f63600accf0 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f635f7fe700 1 -- 172.21.15.29:0/3068548793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c00b860 con 0x7f63600accf0 2023-10-07T09:41:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f635f7fe700 1 -- 172.21.15.29:0/3068548793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00bbe0 con 0x7f63600accf0 2023-10-07T09:41:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 -- 172.21.15.29:0/3068548793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600accf0 msgr2=0x7f63600aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600accf0 0x7f63600aef80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f634c0098e0 tx=0x7f634c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 -- 172.21.15.29:0/3068548793 shutdown_connections 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600afeb0 0x7f63600cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 --2- 172.21.15.29:0/3068548793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600accf0 0x7f63600aef80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 -- 172.21.15.29:0/3068548793 >> 172.21.15.29:0/3068548793 conn(0x7f6360067610 msgr2=0x7f6360067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 -- 172.21.15.29:0/3068548793 shutdown_connections 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.553+0000 7f63671ff700 1 -- 172.21.15.29:0/3068548793 wait complete. 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.556+0000 7f63671ff700 1 Processor -- start 2023-10-07T09:41:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.556+0000 7f63671ff700 1 -- start start 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f63671ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600accf0 0x7f63600a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f63671ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600af4c0 0x7f63600aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f63671ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 0x7f63600a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f63671ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600a9240 con 0x7f63600afeb0 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f63671ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600a7a20 con 0x7f63600af4c0 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f63671ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600a7050 con 0x7f63600accf0 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f636579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 0x7f63600a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f636579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 0x7f63600a85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41298/0 (socket says 172.21.15.29:41298) 2023-10-07T09:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f636579c700 1 -- 172.21.15.29:0/250628673 learned_addr learned my addr 172.21.15.29:0/250628673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f635ffff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600af4c0 0x7f63600aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.557+0000 7f6364f9b700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600accf0 0x7f63600a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f636579c700 1 -- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600accf0 msgr2=0x7f63600a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f636579c700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600accf0 0x7f63600a74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f636579c700 1 -- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600af4c0 msgr2=0x7f63600aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f636579c700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600af4c0 0x7f63600aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f636579c700 1 -- 172.21.15.29:0/250628673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c009420 con 0x7f63600afeb0 2023-10-07T09:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f636579c700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 0x7f63600a85c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f635800cc40 tx=0x7f6358001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800a810 con 0x7f63600afeb0 2023-10-07T09:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800a990 con 0x7f63600afeb0 2023-10-07T09:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63600a6720 con 0x7f63600afeb0 2023-10-07T09:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800ad50 con 0x7f63600afeb0 2023-10-07T09:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.558+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63600a6ce0 con 0x7f63600afeb0 2023-10-07T09:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.559+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63600718b0 con 0x7f63600afeb0 2023-10-07T09:41:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.560+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f6358007300 con 0x7f63600afeb0 2023-10-07T09:41:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.561+0000 7f635cff9700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6344067130 0x7f63440695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.561+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f6358084c20 con 0x7f63600afeb0 2023-10-07T09:41:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.562+0000 7f6364f9b700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6344067130 0x7f63440695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.562+0000 7f6364f9b700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6344067130 0x7f63440695e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6354006f00 tx=0x7f6354008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.566+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6358051af0 con 0x7f63600afeb0 2023-10-07T09:41:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.796+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f63600cd020 con 0x7f6344067130 2023-10-07T09:41:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.868+0000 7f635cff9700 1 -- 172.21.15.29:0/250628673 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f63600cd020 con 0x7f6344067130 2023-10-07T09:41:22.870 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-07T09:41:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6344067130 msgr2=0x7f63440695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6344067130 0x7f63440695e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6354006f00 tx=0x7f6354008040 comp rx=0 tx=0).stop 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 msgr2=0x7f63600a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 0x7f63600a85c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f635800cc40 tx=0x7f6358001970 comp rx=0 tx=0).stop 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 shutdown_connections 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6344067130 0x7f63440695e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63600afeb0 0x7f63600a85c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600accf0 0x7f63600a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 --2- 172.21.15.29:0/250628673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600af4c0 0x7f63600aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 >> 172.21.15.29:0/250628673 conn(0x7f6360067610 msgr2=0x7f6360153820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 shutdown_connections 2023-10-07T09:41:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:22.871+0000 7f63671ff700 1 -- 172.21.15.29:0/250628673 wait complete. 2023-10-07T09:41:22.890 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-07T09:41:22.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir4 1M 2023-10-07T09:41:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.088+0000 7fe5319ff700 1 Processor -- start 2023-10-07T09:41:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.089+0000 7fe5319ff700 1 -- start start 2023-10-07T09:41:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe5319ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe52c0b2a20 0x7fe52c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe5319ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0cc940 con 0x7fe52c0cf2c0 2023-10-07T09:41:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe5319ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0b2df0 con 0x7fe52c0cb6f0 2023-10-07T09:41:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe5319ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c0cbd30 con 0x7fe52c0b2a20 2023-10-07T09:41:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe52b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe52c0cf2c0 0x7fe52c0c5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51744/0 (socket says 172.21.15.29:51744) 2023-10-07T09:41:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe52b7fe700 1 -- 172.21.15.29:0/1187067978 learned_addr learned my addr 172.21.15.29:0/1187067978 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.090+0000 7fe52affd700 1 --2- 172.21.15.29:0/1187067978 >> v2:172.21.15.144:3300/0 conn(0x7fe52c0b2a20 0x7fe52c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166933157 0 0) 0x7fe52c0cc940 con 0x7fe52c0cf2c0 2023-10-07T09:41:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514003570 con 0x7fe52c0cf2c0 2023-10-07T09:41:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023500342 0 0) 0x7fe52c0b2df0 con 0x7fe52c0cb6f0 2023-10-07T09:41:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe52c0cc940 con 0x7fe52c0cb6f0 2023-10-07T09:41:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3732152430 0 0) 0x7fe514003570 con 0x7fe52c0cf2c0 2023-10-07T09:41:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe52affd700 1 -- 172.21.15.29:0/1187067978 >> v1:172.21.15.144:6789/0 conn(0x7fe52c0cb6f0 legacy=0x7fe52c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe52affd700 1 -- 172.21.15.29:0/1187067978 >> v1:172.21.15.29:6789/0 conn(0x7fe52c0cf2c0 legacy=0x7fe52c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe52affd700 1 -- 172.21.15.29:0/1187067978 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c0cc010 con 0x7fe52c0b2a20 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.091+0000 7fe52affd700 1 --2- 172.21.15.29:0/1187067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0cb1b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0098d0 tx=0x7fe51c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22d70ba91d2180e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240017f0 con 0x7fe52c0b2a20 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c011420 con 0x7fe52c0b2a20 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe529ffb700 1 -- 172.21.15.29:0/1187067978 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0117c0 con 0x7fe52c0b2a20 2023-10-07T09:41:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 -- 172.21.15.29:0/1187067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0b2a20 msgr2=0x7fe52c0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1187067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0cb1b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0098d0 tx=0x7fe51c009120 comp rx=0 tx=0).stop 2023-10-07T09:41:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 -- 172.21.15.29:0/1187067978 shutdown_connections 2023-10-07T09:41:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1187067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0cb1b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 -- 172.21.15.29:0/1187067978 >> 172.21.15.29:0/1187067978 conn(0x7fe52c099820 msgr2=0x7fe52c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 -- 172.21.15.29:0/1187067978 shutdown_connections 2023-10-07T09:41:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.092+0000 7fe5319ff700 1 -- 172.21.15.29:0/1187067978 wait complete. 2023-10-07T09:41:23.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.094+0000 7fe5319ff700 1 Processor -- start 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.094+0000 7fe5319ff700 1 -- start start 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe5319ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe5319ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe52c0cb6f0 0x7fe52c14e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe5319ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0cf2c0 0x7fe52c16d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe5319ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0b4900 con 0x7fe52c0b2a20 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe5319ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c15b310 con 0x7fe52c0cf2c0 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe5319ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0b4340 con 0x7fe52c0cb6f0 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe52affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe52affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49190/0 (socket says 172.21.15.29:49190) 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.095+0000 7fe52affd700 1 -- 172.21.15.29:0/1492199113 learned_addr learned my addr 172.21.15.29:0/1492199113 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52affd700 1 -- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe52c0cb6f0 msgr2=0x7fe52c14e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52affd700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe52c0cb6f0 0x7fe52c14e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52affd700 1 -- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0cf2c0 msgr2=0x7fe52c16d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52b7fe700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0cf2c0 0x7fe52c16d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52affd700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0cf2c0 0x7fe52c16d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52affd700 1 -- 172.21.15.29:0/1492199113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0093c0 con 0x7fe52c0b2a20 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52b7fe700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0cf2c0 0x7fe52c16d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe52affd700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0b3d20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009390 tx=0x7fe51c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c01cd20 con 0x7fe52c0b2a20 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c01ce80 con 0x7fe52c0b2a20 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c0b4620 con 0x7fe52c0b2a20 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c01a6a0 con 0x7fe52c0b2a20 2023-10-07T09:41:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.096+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c14f580 con 0x7fe52c0b2a20 2023-10-07T09:41:23.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.097+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c0718b0 con 0x7fe52c0b2a20 2023-10-07T09:41:23.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.098+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fe51c01aa50 con 0x7fe52c0b2a20 2023-10-07T09:41:23.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.100+0000 7fe512ffd700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe518067130 0x7fe5180695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.100+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fe51c0866f0 con 0x7fe52c0b2a20 2023-10-07T09:41:23.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.100+0000 7fe52a7fc700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe518067130 0x7fe5180695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.101+0000 7fe52a7fc700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe518067130 0x7fe5180695e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe52c16d220 tx=0x7fe5200079d0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.104+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe51c053540 con 0x7fe52c0b2a20 2023-10-07T09:41:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.335+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe52c0cc2f0 con 0x7fe518067130 2023-10-07T09:41:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.403+0000 7fe512ffd700 1 -- 172.21.15.29:0/1492199113 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe52c0cc2f0 con 0x7fe518067130 2023-10-07T09:41:23.405 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-07T09:41:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe518067130 msgr2=0x7fe5180695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe518067130 0x7fe5180695e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe52c16d220 tx=0x7fe5200079d0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 msgr2=0x7fe52c0b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0b3d20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009390 tx=0x7fe51c016000 comp rx=0 tx=0).stop 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 shutdown_connections 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe518067130 0x7fe5180695e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52c0cf2c0 0x7fe52c16d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c0b2a20 0x7fe52c0b3d20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 --2- 172.21.15.29:0/1492199113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe52c0cb6f0 0x7fe52c14e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 >> 172.21.15.29:0/1492199113 conn(0x7fe52c099820 msgr2=0x7fe52c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 shutdown_connections 2023-10-07T09:41:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.406+0000 7fe5319ff700 1 -- 172.21.15.29:0/1492199113 wait complete. 2023-10-07T09:41:23.424 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-07T09:41:23.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir5 1M 2023-10-07T09:41:23.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.624+0000 7fe222bff700 1 Processor -- start 2023-10-07T09:41:23.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.624+0000 7fe222bff700 1 -- start start 2023-10-07T09:41:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.625+0000 7fe222bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe21c0cc810 0x7fe21c0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe222bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 0x7fe21c15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe222bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c0ad400 con 0x7fe21c0afeb0 2023-10-07T09:41:23.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe222bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c15a140 con 0x7fe21c0cc810 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe222bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0adca0 con 0x7fe21c15a590 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe22099b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe21c0afeb0 0x7fe21c0aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51748/0 (socket says 172.21.15.29:51748) 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe22099b700 1 -- 172.21.15.29:0/38457281 learned_addr learned my addr 172.21.15.29:0/38457281 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe22119c700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 0x7fe21c15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe21bfff700 1 --2- 172.21.15.29:0/38457281 >> v2:172.21.15.144:3300/0 conn(0x7fe21c0cc810 0x7fe21c0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587613066 0 0) 0x7fe21c0ad400 con 0x7fe21c0afeb0 2023-10-07T09:41:23.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.626+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c003570 con 0x7fe21c0afeb0 2023-10-07T09:41:23.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3582949593 0 0) 0x7fe20c003570 con 0x7fe21c0afeb0 2023-10-07T09:41:23.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 msgr2=0x7fe21c15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:23.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 0x7fe21c15a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 >> v2:172.21.15.144:3300/0 conn(0x7fe21c0cc810 msgr2=0x7fe21c0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 --2- 172.21.15.29:0/38457281 >> v2:172.21.15.144:3300/0 conn(0x7fe21c0cc810 0x7fe21c0accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0ad6e0 con 0x7fe21c0afeb0 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fe210001470 con 0x7fe21c0afeb0 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 >> v1:172.21.15.29:6789/0 conn(0x7fe21c0afeb0 legacy=0x7fe21c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe20c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20c004810 0x7fe20c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 0x7fe20c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c15a140 con 0x7fe20c003a60 2023-10-07T09:41:23.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0adca0 con 0x7fe20c004810 2023-10-07T09:41:23.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c00dc70 con 0x7fe21c15a590 2023-10-07T09:41:23.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22099b700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe20c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe21bfff700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20c004810 0x7fe20c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22119c700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 0x7fe20c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22099b700 1 -- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 msgr2=0x7fe20c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22099b700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c15a590 0x7fe20c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22099b700 1 -- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20c004810 msgr2=0x7fe20c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22099b700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe20c004810 0x7fe20c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.627+0000 7fe22099b700 1 -- 172.21.15.29:0/38457281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c003570 con 0x7fe20c003a60 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe22099b700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe20c003e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe21000c3f0 tx=0x7fe210001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11325243b8d8f5a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21000cc80 con 0x7fe20c003a60 2023-10-07T09:41:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21000cde0 con 0x7fe20c003a60 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe21b7fe700 1 -- 172.21.15.29:0/38457281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210005b50 con 0x7fe20c003a60 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe222bff700 1 -- 172.21.15.29:0/38457281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 msgr2=0x7fe20c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe222bff700 1 --2- 172.21.15.29:0/38457281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe20c003e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe21000c3f0 tx=0x7fe210001e90 comp rx=0 tx=0).stop 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.628+0000 7fe22119c700 1 -- 172.21.15.29:0/38457281 reap_dead start 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.629+0000 7fe222bff700 1 -- 172.21.15.29:0/38457281 shutdown_connections 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.629+0000 7fe222bff700 1 -- 172.21.15.29:0/38457281 >> 172.21.15.29:0/38457281 conn(0x7fe21c067610 msgr2=0x7fe21c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.629+0000 7fe222bff700 1 -- 172.21.15.29:0/38457281 shutdown_connections 2023-10-07T09:41:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.629+0000 7fe222bff700 1 -- 172.21.15.29:0/38457281 wait complete. 2023-10-07T09:41:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.630+0000 7fe222bff700 1 Processor -- start 2023-10-07T09:41:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.630+0000 7fe222bff700 1 -- start start 2023-10-07T09:41:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe222bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe21c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe222bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c1593a0 0x7fe21c153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe222bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21c0ca560 0x7fe21c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe222bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c15c560 con 0x7fe20c003a60 2023-10-07T09:41:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe222bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c15b050 con 0x7fe21c0ca560 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe222bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c15cd30 con 0x7fe21c1593a0 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe22099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe21c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe22099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe21c1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49212/0 (socket says 172.21.15.29:49212) 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.631+0000 7fe22099b700 1 -- 172.21.15.29:0/3055275068 learned_addr learned my addr 172.21.15.29:0/3055275068 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22119c700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21c0ca560 0x7fe21c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe21bfff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c1593a0 0x7fe21c153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22099b700 1 -- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c1593a0 msgr2=0x7fe21c153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22099b700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c1593a0 0x7fe21c153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22099b700 1 -- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21c0ca560 msgr2=0x7fe21c0ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22099b700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21c0ca560 0x7fe21c0ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22099b700 1 -- 172.21.15.29:0/3055275068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0caf10 con 0x7fe20c003a60 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22119c700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21c0ca560 0x7fe21c0ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe22099b700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe21c1581a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe210007b80 tx=0x7fe210015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210022070 con 0x7fe20c003a60 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2100072e0 con 0x7fe20c003a60 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2100076a0 con 0x7fe20c003a60 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.632+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c15b8b0 con 0x7fe20c003a60 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.633+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c15b260 con 0x7fe20c003a60 2023-10-07T09:41:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.634+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c0718b0 con 0x7fe20c003a60 2023-10-07T09:41:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.634+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fe21001e430 con 0x7fe20c003a60 2023-10-07T09:41:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.635+0000 7fe218ff9700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe200067130 0x7fe2000695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.635+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fe2100857e0 con 0x7fe20c003a60 2023-10-07T09:41:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.636+0000 7fe21bfff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe200067130 0x7fe2000695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.637+0000 7fe21bfff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe200067130 0x7fe2000695e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe208006dc0 tx=0x7fe208008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.639+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe210052630 con 0x7fe20c003a60 2023-10-07T09:41:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.871+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe21c15b600 con 0x7fe200067130 2023-10-07T09:41:23.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.941+0000 7fe218ff9700 1 -- 172.21.15.29:0/3055275068 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe21c15b600 con 0x7fe200067130 2023-10-07T09:41:23.942 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-07T09:41:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe200067130 msgr2=0x7fe2000695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe200067130 0x7fe2000695e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe208006dc0 tx=0x7fe208008040 comp rx=0 tx=0).stop 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 msgr2=0x7fe21c1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe21c1581a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe210007b80 tx=0x7fe210015000 comp rx=0 tx=0).stop 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 shutdown_connections 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe200067130 0x7fe2000695e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21c0ca560 0x7fe21c0ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe20c003a60 0x7fe21c1581a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 --2- 172.21.15.29:0/3055275068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe21c1593a0 0x7fe21c153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 >> 172.21.15.29:0/3055275068 conn(0x7fe21c067610 msgr2=0x7fe21c0abbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 shutdown_connections 2023-10-07T09:41:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:23.943+0000 7fe222bff700 1 -- 172.21.15.29:0/3055275068 wait complete. 2023-10-07T09:41:23.962 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-07T09:41:23.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir6 1M 2023-10-07T09:41:24.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.162+0000 7f89ab9ff700 1 Processor -- start 2023-10-07T09:41:24.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.162+0000 7f89ab9ff700 1 -- start start 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.163+0000 7f89ab9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 0x7f89a40cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.163+0000 7f89ab9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f89a40cb6f0 0x7f89a40cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.163+0000 7f89ab9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40cc940 con 0x7f89a40cf0f0 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.163+0000 7f89ab9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a40cf530 con 0x7f89a40cb6f0 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.163+0000 7f89ab9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40b2df0 con 0x7f89a40b2a20 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 0x7f89a40cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 0x7f89a40cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:35670/0 (socket says 172.21.15.29:35670) 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 -- 172.21.15.29:0/1375934848 learned_addr learned my addr 172.21.15.29:0/1375934848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a8f9a700 1 --2- 172.21.15.29:0/1375934848 >> v2:172.21.15.144:3300/0 conn(0x7f89a40cb6f0 0x7f89a40cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 -- 172.21.15.29:0/1375934848 >> v2:172.21.15.144:3300/0 conn(0x7f89a40cb6f0 msgr2=0x7f89a40cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 --2- 172.21.15.29:0/1375934848 >> v2:172.21.15.144:3300/0 conn(0x7f89a40cb6f0 0x7f89a40cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 -- 172.21.15.29:0/1375934848 >> v1:172.21.15.144:6789/0 conn(0x7f89a40cf0f0 legacy=0x7f89a4159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 -- 172.21.15.29:0/1375934848 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40cbd30 con 0x7f89a40b2a20 2023-10-07T09:41:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.164+0000 7f89a979b700 1 --2- 172.21.15.29:0/1375934848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 0x7f89a40cb1b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009990 tx=0x7f89a0009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81250488c60778f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f899bfff700 1 -- 172.21.15.29:0/1375934848 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0009e80 con 0x7f89a40b2a20 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f899bfff700 1 -- 172.21.15.29:0/1375934848 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a0005c10 con 0x7f89a40b2a20 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f899bfff700 1 -- 172.21.15.29:0/1375934848 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a000b650 con 0x7f89a40b2a20 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 -- 172.21.15.29:0/1375934848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 msgr2=0x7f89a40cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/1375934848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 0x7f89a40cb1b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009990 tx=0x7f89a0009220 comp rx=0 tx=0).stop 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 -- 172.21.15.29:0/1375934848 shutdown_connections 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/1375934848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40b2a20 0x7f89a40cb1b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/1375934848 >> v2:172.21.15.144:3300/0 conn(0x7f89a40cb6f0 0x7f89a40cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 -- 172.21.15.29:0/1375934848 >> 172.21.15.29:0/1375934848 conn(0x7f89a4099820 msgr2=0x7f89a4063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:24.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 -- 172.21.15.29:0/1375934848 shutdown_connections 2023-10-07T09:41:24.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.165+0000 7f89ab9ff700 1 -- 172.21.15.29:0/1375934848 wait complete. 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.167+0000 7f89ab9ff700 1 Processor -- start 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.167+0000 7f89ab9ff700 1 -- start start 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89ab9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 0x7f89a41551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89ab9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89a40cb6f0 0x7f89a4157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89ab9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40cf0f0 0x7f89a4158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89ab9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a409e220 con 0x7f89a40b2a20 2023-10-07T09:41:24.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89ab9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a4156aa0 con 0x7f89a40cb6f0 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89ab9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a4155a00 con 0x7f89a40cf0f0 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89a979b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 0x7f89a41551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89a979b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 0x7f89a41551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49224/0 (socket says 172.21.15.29:49224) 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.168+0000 7f89a979b700 1 -- 172.21.15.29:0/686217417 learned_addr learned my addr 172.21.15.29:0/686217417 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89a979b700 1 -- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40cf0f0 msgr2=0x7f89a4158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89a979b700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40cf0f0 0x7f89a4158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89a979b700 1 -- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89a40cb6f0 msgr2=0x7f89a4157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89a979b700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89a40cb6f0 0x7f89a4157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89a979b700 1 -- 172.21.15.29:0/686217417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0009480 con 0x7f89a40b2a20 2023-10-07T09:41:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89a979b700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 0x7f89a41551d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f89a000b3c0 tx=0x7f89a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0011ae0 con 0x7f89a40b2a20 2023-10-07T09:41:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a0020070 con 0x7f89a40b2a20 2023-10-07T09:41:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0011e70 con 0x7f89a40b2a20 2023-10-07T09:41:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a4155ce0 con 0x7f89a40b2a20 2023-10-07T09:41:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.169+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a41562a0 con 0x7f89a40b2a20 2023-10-07T09:41:24.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.170+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a40718b0 con 0x7f89a40b2a20 2023-10-07T09:41:24.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.171+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f89a0022030 con 0x7f89a40b2a20 2023-10-07T09:41:24.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.173+0000 7f89997fa700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8988067130 0x7f89880695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.173+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f89a0086630 con 0x7f89a40b2a20 2023-10-07T09:41:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.173+0000 7f89a8f9a700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8988067130 0x7f89880695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.174+0000 7f89a8f9a700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8988067130 0x7f89880695e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f89a4154df0 tx=0x7f89940091a0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.177+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89a0053480 con 0x7f89a40b2a20 2023-10-07T09:41:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.412+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f89a40cce70 con 0x7f8988067130 2023-10-07T09:41:24.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.483+0000 7f89997fa700 1 -- 172.21.15.29:0/686217417 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f89a40cce70 con 0x7f8988067130 2023-10-07T09:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-07T09:41:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8988067130 msgr2=0x7f89880695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8988067130 0x7f89880695e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f89a4154df0 tx=0x7f89940091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 msgr2=0x7f89a41551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 0x7f89a41551d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f89a000b3c0 tx=0x7f89a0016000 comp rx=0 tx=0).stop 2023-10-07T09:41:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 shutdown_connections 2023-10-07T09:41:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8988067130 0x7f89880695e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89a40cf0f0 0x7f89a4158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89a40b2a20 0x7f89a41551d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 --2- 172.21.15.29:0/686217417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89a40cb6f0 0x7f89a4157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 >> 172.21.15.29:0/686217417 conn(0x7f89a4099820 msgr2=0x7f89a40683b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 shutdown_connections 2023-10-07T09:41:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.486+0000 7f89ab9ff700 1 -- 172.21.15.29:0/686217417 wait complete. 2023-10-07T09:41:24.504 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-07T09:41:24.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir7 1M 2023-10-07T09:41:24.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.704+0000 7fe0271ff700 1 Processor -- start 2023-10-07T09:41:24.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.704+0000 7fe0271ff700 1 -- start start 2023-10-07T09:41:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.705+0000 7fe0271ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0200b0610 0x7fe0200ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.705+0000 7fe0271ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0200690f0 con 0x7fe0200c8a20 2023-10-07T09:41:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.705+0000 7fe0271ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0200cf2e0 con 0x7fe0200b09e0 2023-10-07T09:41:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.705+0000 7fe0271ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0200687c0 con 0x7fe0200b0610 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe02579c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe0200c8a20 0x7fe0200cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51754/0 (socket says 172.21.15.29:51754) 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe02579c700 1 -- 172.21.15.29:0/3826706623 learned_addr learned my addr 172.21.15.29:0/3826706623 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe024f9b700 1 --2- 172.21.15.29:0/3826706623 >> v2:172.21.15.144:3300/0 conn(0x7fe0200b0610 0x7fe0200ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67673016 0 0) 0x7fe0200690f0 con 0x7fe0200c8a20 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe010003570 con 0x7fe0200c8a20 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158682817 0 0) 0x7fe0200cf2e0 con 0x7fe0200b09e0 2023-10-07T09:41:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0200690f0 con 0x7fe0200b09e0 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3149056027 0 0) 0x7fe010003570 con 0x7fe0200c8a20 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.706+0000 7fe024f9b700 1 -- 172.21.15.29:0/3826706623 >> v1:172.21.15.144:6789/0 conn(0x7fe0200b09e0 legacy=0x7fe0200c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.707+0000 7fe024f9b700 1 -- 172.21.15.29:0/3826706623 >> v1:172.21.15.29:6789/0 conn(0x7fe0200c8a20 legacy=0x7fe0200cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.707+0000 7fe024f9b700 1 -- 172.21.15.29:0/3826706623 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe020068aa0 con 0x7fe0200b0610 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.707+0000 7fe024f9b700 1 --2- 172.21.15.29:0/3826706623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200b0610 0x7fe0200ced00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0140098e0 tx=0x7fe0140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c600991a0a87eb1b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.707+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0180017f0 con 0x7fe0200b0610 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.707+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe014011420 con 0x7fe0200b0610 2023-10-07T09:41:24.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.707+0000 7fe01f7fe700 1 -- 172.21.15.29:0/3826706623 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0140117c0 con 0x7fe0200b0610 2023-10-07T09:41:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 -- 172.21.15.29:0/3826706623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200b0610 msgr2=0x7fe0200ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 --2- 172.21.15.29:0/3826706623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200b0610 0x7fe0200ced00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0140098e0 tx=0x7fe0140091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 -- 172.21.15.29:0/3826706623 shutdown_connections 2023-10-07T09:41:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 --2- 172.21.15.29:0/3826706623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200b0610 0x7fe0200ced00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 -- 172.21.15.29:0/3826706623 >> 172.21.15.29:0/3826706623 conn(0x7fe020063a10 msgr2=0x7fe0200ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 -- 172.21.15.29:0/3826706623 shutdown_connections 2023-10-07T09:41:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.708+0000 7fe0271ff700 1 -- 172.21.15.29:0/3826706623 wait complete. 2023-10-07T09:41:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.709+0000 7fe0271ff700 1 Processor -- start 2023-10-07T09:41:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.709+0000 7fe0271ff700 1 -- start start 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.710+0000 7fe0271ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0200b0610 0x7fe02014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.710+0000 7fe0271ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 0x7fe02009b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.710+0000 7fe0271ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200c8a20 0x7fe0200b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.710+0000 7fe0271ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200b1b60 con 0x7fe0200b09e0 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.710+0000 7fe0271ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200b1e40 con 0x7fe0200c8a20 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.710+0000 7fe0271ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200b2120 con 0x7fe0200b0610 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 0x7fe02009b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 0x7fe02009b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49234/0 (socket says 172.21.15.29:49234) 2023-10-07T09:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 -- 172.21.15.29:0/1136780113 learned_addr learned my addr 172.21.15.29:0/1136780113 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe02579c700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200c8a20 0x7fe0200b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe024f9b700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0200b0610 0x7fe02014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 -- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0200b0610 msgr2=0x7fe02014b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0200b0610 0x7fe02014b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 -- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200c8a20 msgr2=0x7fe0200b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200c8a20 0x7fe0200b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 -- 172.21.15.29:0/1136780113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014009420 con 0x7fe0200b09e0 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe02579c700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200c8a20 0x7fe0200b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe024f9b700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0200b0610 0x7fe02014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.711+0000 7fe01ffff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 0x7fe02009b630 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c790 tx=0x7fe00c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.712+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c010510 con 0x7fe0200b09e0 2023-10-07T09:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.712+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c010670 con 0x7fe0200b09e0 2023-10-07T09:41:24.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.712+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c010a10 con 0x7fe0200b09e0 2023-10-07T09:41:24.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.712+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02009ae30 con 0x7fe0200b09e0 2023-10-07T09:41:24.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.712+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02014b840 con 0x7fe0200b09e0 2023-10-07T09:41:24.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.713+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02009b350 con 0x7fe0200b09e0 2023-10-07T09:41:24.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.714+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fe00c01b400 con 0x7fe0200b09e0 2023-10-07T09:41:24.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.714+0000 7fe01cff9700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe004067130 0x7fe0040695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:24.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.715+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fe00c084880 con 0x7fe0200b09e0 2023-10-07T09:41:24.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.715+0000 7fe024f9b700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe004067130 0x7fe0040695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:24.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.716+0000 7fe024f9b700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe004067130 0x7fe0040695e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe014003080 tx=0x7fe014016000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.719+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe02009b350 con 0x7fe0200b09e0 2023-10-07T09:41:24.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:24.949+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe02009acb0 con 0x7fe004067130 2023-10-07T09:41:25.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.020+0000 7fe01cff9700 1 -- 172.21.15.29:0/1136780113 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe014011700 con 0x7fe004067130 2023-10-07T09:41:25.022 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-07T09:41:25.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe004067130 msgr2=0x7fe0040695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe004067130 0x7fe0040695e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe014003080 tx=0x7fe014016000 comp rx=0 tx=0).stop 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 msgr2=0x7fe02009b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 0x7fe02009b630 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c790 tx=0x7fe00c007e10 comp rx=0 tx=0).stop 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 shutdown_connections 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe004067130 0x7fe0040695e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0200c8a20 0x7fe0200b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0200b0610 0x7fe02014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 --2- 172.21.15.29:0/1136780113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0200b09e0 0x7fe02009b630 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 >> 172.21.15.29:0/1136780113 conn(0x7fe020063a10 msgr2=0x7fe0201593d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:25.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 shutdown_connections 2023-10-07T09:41:25.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.023+0000 7fe0271ff700 1 -- 172.21.15.29:0/1136780113 wait complete. 2023-10-07T09:41:25.041 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-07T09:41:25.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir8 1M 2023-10-07T09:41:25.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.240+0000 7fb6d63ff700 1 Processor -- start 2023-10-07T09:41:25.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.240+0000 7fb6d63ff700 1 -- start start 2023-10-07T09:41:25.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6d63ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6d00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6d63ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d00690f0 con 0x7fb6d00ced00 2023-10-07T09:41:25.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6d63ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d00cf2e0 con 0x7fb6d00b0610 2023-10-07T09:41:25.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6d63ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00687c0 con 0x7fb6d00b09e0 2023-10-07T09:41:25.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6cf7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6d00ced00 0x7fb6d00c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51770/0 (socket says 172.21.15.29:51770) 2023-10-07T09:41:25.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6cf7fe700 1 -- 172.21.15.29:0/1407480572 learned_addr learned my addr 172.21.15.29:0/1407480572 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:25.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6d499c700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6d00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.242+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542409338 0 0) 0x7fb6d00690f0 con 0x7fb6d00ced00 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b8003570 con 0x7fb6d00ced00 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065352334 0 0) 0x7fb6d00cf2e0 con 0x7fb6d00b0610 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d00690f0 con 0x7fb6d00b0610 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 228151571 0 0) 0x7fb6b8003570 con 0x7fb6d00ced00 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 msgr2=0x7fb6d00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6d00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 >> v1:172.21.15.144:6789/0 conn(0x7fb6d00b0610 legacy=0x7fb6d00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0068aa0 con 0x7fb6d00ced00 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fb6c4001470 con 0x7fb6d00ced00 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6d499c700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6d00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:25.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 >> v1:172.21.15.29:6789/0 conn(0x7fb6d00ced00 legacy=0x7fb6d00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b8003c20 0x7fb6b8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b8004810 0x7fb6b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6b80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00687c0 con 0x7fb6b8003c20 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b800bbe0 con 0x7fb6b8004810 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b800bda0 con 0x7fb6d00b09e0 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b8003c20 0x7fb6b8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6d499c700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6b80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.243+0000 7fb6cf7fe700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b8004810 0x7fb6b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6cffff700 1 -- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 msgr2=0x7fb6b80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6b80051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6cffff700 1 -- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b8004810 msgr2=0x7fb6b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b8004810 0x7fb6b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6cffff700 1 -- 172.21.15.29:0/1407480572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c8009a20 con 0x7fb6b8003c20 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6d499c700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6d00b09e0 0x7fb6b80051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b8003c20 0x7fb6b8004010 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0001a50 tx=0x7fb6c0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336a197127b0e5b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c000c5d0 con 0x7fb6b8003c20 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c000c730 con 0x7fb6b8003c20 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.244+0000 7fb6ceffd700 1 -- 172.21.15.29:0/1407480572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c000cad0 con 0x7fb6b8003c20 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1407480572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b8003c20 msgr2=0x7fb6b8004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1407480572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b8003c20 0x7fb6b8004010 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0001a50 tx=0x7fb6c0007d30 comp rx=0 tx=0).stop 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d499c700 1 -- 172.21.15.29:0/1407480572 reap_dead start 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1407480572 shutdown_connections 2023-10-07T09:41:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1407480572 >> 172.21.15.29:0/1407480572 conn(0x7fb6d0063a10 msgr2=0x7fb6d00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1407480572 shutdown_connections 2023-10-07T09:41:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.245+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1407480572 wait complete. 2023-10-07T09:41:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.246+0000 7fb6d63ff700 1 Processor -- start 2023-10-07T09:41:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.246+0000 7fb6d63ff700 1 -- start start 2023-10-07T09:41:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.247+0000 7fb6d63ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b8003c20 0x7fb6d0153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.247+0000 7fb6d63ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 0x7fb6d00c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.247+0000 7fb6d63ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d00ccbe0 0x7fb6d00cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.247+0000 7fb6d63ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00c6ed0 con 0x7fb6d00c6520 2023-10-07T09:41:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.247+0000 7fb6d63ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01538c0 con 0x7fb6d00ccbe0 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.247+0000 7fb6d63ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00cd570 con 0x7fb6b8003c20 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 0x7fb6d00c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 0x7fb6d00c6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49246/0 (socket says 172.21.15.29:49246) 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 -- 172.21.15.29:0/1145159488 learned_addr learned my addr 172.21.15.29:0/1145159488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b8003c20 0x7fb6d0153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6d499c700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d00ccbe0 0x7fb6d00cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 -- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b8003c20 msgr2=0x7fb6d0153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b8003c20 0x7fb6d0153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 -- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d00ccbe0 msgr2=0x7fb6d00cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d00ccbe0 0x7fb6d00cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 -- 172.21.15.29:0/1145159488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d016ae70 con 0x7fb6d00c6520 2023-10-07T09:41:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.248+0000 7fb6cf7fe700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 0x7fb6d00c6990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb6c400e810 tx=0x7fb6c4000c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.249+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4012590 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.249+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c4012710 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.249+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4012ad0 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.249+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d016a260 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.249+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d016a820 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.250+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d00718b0 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.251+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fb6c401d460 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.252+0000 7fb6b7fff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb6bc067130 0x7fb6bc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.252+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fb6c4084830 con 0x7fb6d00c6520 2023-10-07T09:41:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.252+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb6bc067130 0x7fb6bc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.253+0000 7fb6cffff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb6bc067130 0x7fb6bc0695e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0001ed0 tx=0x7fb6c0015000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:25.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.256+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6c4051680 con 0x7fb6d00c6520 2023-10-07T09:41:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.486+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6d0068d80 con 0x7fb6bc067130 2023-10-07T09:41:25.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.556+0000 7fb6b7fff700 1 -- 172.21.15.29:0/1145159488 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb6d0068d80 con 0x7fb6bc067130 2023-10-07T09:41:25.557 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-07T09:41:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb6bc067130 msgr2=0x7fb6bc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb6bc067130 0x7fb6bc0695e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0001ed0 tx=0x7fb6c0015000 comp rx=0 tx=0).stop 2023-10-07T09:41:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 msgr2=0x7fb6d00c6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 0x7fb6d00c6990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb6c400e810 tx=0x7fb6c4000c50 comp rx=0 tx=0).stop 2023-10-07T09:41:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 shutdown_connections 2023-10-07T09:41:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb6bc067130 0x7fb6bc0695e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d00ccbe0 0x7fb6d00cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b8003c20 0x7fb6d0153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 --2- 172.21.15.29:0/1145159488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d00c6520 0x7fb6d00c6990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.558+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 >> 172.21.15.29:0/1145159488 conn(0x7fb6d0063a10 msgr2=0x7fb6d00c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.559+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 shutdown_connections 2023-10-07T09:41:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.559+0000 7fb6d63ff700 1 -- 172.21.15.29:0/1145159488 wait complete. 2023-10-07T09:41:25.577 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-07T09:41:25.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir9 1M 2023-10-07T09:41:25.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.777+0000 7f63cb9ff700 1 Processor -- start 2023-10-07T09:41:25.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.777+0000 7f63cb9ff700 1 -- start start 2023-10-07T09:41:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.779+0000 7f63cb9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c40cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.779+0000 7f63cb9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40cc940 con 0x7f63c40cf2c0 2023-10-07T09:41:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.779+0000 7f63cb9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40b2df0 con 0x7f63c40cb6f0 2023-10-07T09:41:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.779+0000 7f63cb9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40cbd30 con 0x7f63c40b2a20 2023-10-07T09:41:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63c9f9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f63c40cf2c0 0x7f63c40c5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51776/0 (socket says 172.21.15.29:51776) 2023-10-07T09:41:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63c9f9c700 1 -- 172.21.15.29:0/874206025 learned_addr learned my addr 172.21.15.29:0/874206025 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63c979b700 1 --2- 172.21.15.29:0/874206025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c40cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329509942 0 0) 0x7f63c40cc940 con 0x7f63c40cf2c0 2023-10-07T09:41:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b0003570 con 0x7f63c40cf2c0 2023-10-07T09:41:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176105798 0 0) 0x7f63c40b2df0 con 0x7f63c40cb6f0 2023-10-07T09:41:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c40cc940 con 0x7f63c40cb6f0 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722510249 0 0) 0x7f63b0003570 con 0x7f63c40cf2c0 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63c979b700 1 -- 172.21.15.29:0/874206025 >> v1:172.21.15.144:6789/0 conn(0x7f63c40cb6f0 legacy=0x7f63c40cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63c979b700 1 -- 172.21.15.29:0/874206025 >> v1:172.21.15.29:6789/0 conn(0x7f63c40cf2c0 legacy=0x7f63c40c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.780+0000 7f63c979b700 1 -- 172.21.15.29:0/874206025 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40cc010 con 0x7f63c40b2a20 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.781+0000 7f63c979b700 1 --2- 172.21.15.29:0/874206025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c40cb1b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009900 tx=0x7f63c0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1878c72f60d73cfc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.781+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0017f0 con 0x7f63c40b2a20 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.781+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0011420 con 0x7f63c40b2a20 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.781+0000 7f63bbfff700 1 -- 172.21.15.29:0/874206025 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c00117c0 con 0x7f63c40b2a20 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 -- 172.21.15.29:0/874206025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 msgr2=0x7f63c40cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/874206025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c40cb1b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009900 tx=0x7f63c0009120 comp rx=0 tx=0).stop 2023-10-07T09:41:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 -- 172.21.15.29:0/874206025 shutdown_connections 2023-10-07T09:41:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/874206025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c40cb1b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 -- 172.21.15.29:0/874206025 >> 172.21.15.29:0/874206025 conn(0x7f63c4099820 msgr2=0x7f63c4063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 -- 172.21.15.29:0/874206025 shutdown_connections 2023-10-07T09:41:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.782+0000 7f63cb9ff700 1 -- 172.21.15.29:0/874206025 wait complete. 2023-10-07T09:41:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.784+0000 7f63cb9ff700 1 Processor -- start 2023-10-07T09:41:25.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.784+0000 7f63cb9ff700 1 -- start start 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.785+0000 7f63cb9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c409d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.785+0000 7f63cb9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 0x7f63c41497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63cb9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40cf2c0 0x7f63c4149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63cb9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c414a4b0 con 0x7f63c40cb6f0 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63cb9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c414a220 con 0x7f63c40cf2c0 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63cb9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40ca140 con 0x7f63c40b2a20 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 0x7f63c41497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 0x7f63c41497a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49248/0 (socket says 172.21.15.29:49248) 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 -- 172.21.15.29:0/3069524985 learned_addr learned my addr 172.21.15.29:0/3069524985 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c979b700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c409d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c9f9c700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40cf2c0 0x7f63c4149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 -- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 msgr2=0x7f63c409d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c409d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 -- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40cf2c0 msgr2=0x7f63c4149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40cf2c0 0x7f63c4149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 -- 172.21.15.29:0/3069524985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00093c0 con 0x7f63c40cb6f0 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c9f9c700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40cf2c0 0x7f63c4149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.786+0000 7f63c8f9a700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 0x7f63c41497a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c790 tx=0x7f63b4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.787+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4010510 con 0x7f63c40cb6f0 2023-10-07T09:41:25.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.787+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4010670 con 0x7f63c40cb6f0 2023-10-07T09:41:25.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.787+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4010a10 con 0x7f63c40cb6f0 2023-10-07T09:41:25.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.787+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c409b600 con 0x7f63c40cb6f0 2023-10-07T09:41:25.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.787+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c409afb0 con 0x7f63c40cb6f0 2023-10-07T09:41:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.788+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c40ceff0 con 0x7f63c40cb6f0 2023-10-07T09:41:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.789+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f63b401b400 con 0x7f63c40cb6f0 2023-10-07T09:41:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.789+0000 7f63b97fa700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f63a8067130 0x7f63a80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.790+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f63b4084870 con 0x7f63c40cb6f0 2023-10-07T09:41:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.790+0000 7f63c979b700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f63a8067130 0x7f63a80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.791+0000 7f63c979b700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f63a8067130 0x7f63a80695e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009160 tx=0x7f63c0016000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:25.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:25.794+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63b40516c0 con 0x7f63c40cb6f0 2023-10-07T09:41:26.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.025+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f63c409b790 con 0x7f63a8067130 2023-10-07T09:41:26.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.097+0000 7f63b97fa700 1 -- 172.21.15.29:0/3069524985 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f63c409b790 con 0x7f63a8067130 2023-10-07T09:41:26.099 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-07T09:41:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f63a8067130 msgr2=0x7f63a80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f63a8067130 0x7f63a80695e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009160 tx=0x7f63c0016000 comp rx=0 tx=0).stop 2023-10-07T09:41:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 msgr2=0x7f63c41497a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 0x7f63c41497a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c790 tx=0x7f63b4007e10 comp rx=0 tx=0).stop 2023-10-07T09:41:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 shutdown_connections 2023-10-07T09:41:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f63a8067130 0x7f63a80695e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40cf2c0 0x7f63c4149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40b2a20 0x7f63c409d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 --2- 172.21.15.29:0/3069524985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40cb6f0 0x7f63c41497a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 >> 172.21.15.29:0/3069524985 conn(0x7f63c4099820 msgr2=0x7f63c40ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:26.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 shutdown_connections 2023-10-07T09:41:26.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.100+0000 7f63cb9ff700 1 -- 172.21.15.29:0/3069524985 wait complete. 2023-10-07T09:41:26.118 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-07T09:41:26.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:26.119 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-07T09:41:26.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:26.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:26.170 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:26.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir0 2023-10-07T09:41:26.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.372+0000 7f3ad33ff700 1 Processor -- start 2023-10-07T09:41:26.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.372+0000 7f3ad33ff700 1 -- start start 2023-10-07T09:41:26.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.373+0000 7f3ad33ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3acc0ced00 0x7f3acc0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.373+0000 7f3ad33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0c8a20 0x7f3acc0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad33ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3acc0690f0 con 0x7f3acc0b0610 2023-10-07T09:41:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad33ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3acc0cf2e0 con 0x7f3acc0ced00 2023-10-07T09:41:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0687c0 con 0x7f3acc0c8a20 2023-10-07T09:41:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad199c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0c8a20 0x7f3acc0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad099a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3acc0ced00 0x7f3acc0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad119b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3acc0b0610 0x7f3acc0c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:59410/0 (socket says 172.21.15.29:59410) 2023-10-07T09:41:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad119b700 1 -- 172.21.15.29:0/1213614731 learned_addr learned my addr 172.21.15.29:0/1213614731 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad199c700 1 -- 172.21.15.29:0/1213614731 >> v2:172.21.15.144:3300/0 conn(0x7f3acc0ced00 msgr2=0x7f3acc0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad199c700 1 --2- 172.21.15.29:0/1213614731 >> v2:172.21.15.144:3300/0 conn(0x7f3acc0ced00 0x7f3acc0b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad199c700 1 -- 172.21.15.29:0/1213614731 >> v1:172.21.15.144:6789/0 conn(0x7f3acc0b0610 legacy=0x7f3acc0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad199c700 1 -- 172.21.15.29:0/1213614731 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc068aa0 con 0x7f3acc0c8a20 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ac3fff700 1 -- 172.21.15.29:0/1213614731 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153028515 0 0) 0x7f3acc0690f0 con 0x7f3acc0b0610 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.374+0000 7f3ad099a700 1 --2- 172.21.15.29:0/1213614731 >> v2:172.21.15.144:3300/0 conn(0x7f3acc0ced00 0x7f3acc0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.375+0000 7f3ad199c700 1 --2- 172.21.15.29:0/1213614731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0c8a20 0x7f3acc0cc7e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009e60 tx=0x7f3ac4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359a437aef202cd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.375+0000 7f3ac3fff700 1 -- 172.21.15.29:0/1213614731 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400e070 con 0x7f3acc0c8a20 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.375+0000 7f3ac3fff700 1 -- 172.21.15.29:0/1213614731 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac400bb60 con 0x7f3acc0c8a20 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.375+0000 7f3ac3fff700 1 -- 172.21.15.29:0/1213614731 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4012430 con 0x7f3acc0c8a20 2023-10-07T09:41:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 -- 172.21.15.29:0/1213614731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0c8a20 msgr2=0x7f3acc0cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/1213614731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0c8a20 0x7f3acc0cc7e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009e60 tx=0x7f3ac4009780 comp rx=0 tx=0).stop 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 -- 172.21.15.29:0/1213614731 shutdown_connections 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/1213614731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0c8a20 0x7f3acc0cc7e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/1213614731 >> v2:172.21.15.144:3300/0 conn(0x7f3acc0ced00 0x7f3acc0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 -- 172.21.15.29:0/1213614731 >> 172.21.15.29:0/1213614731 conn(0x7f3acc063a10 msgr2=0x7f3acc0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 -- 172.21.15.29:0/1213614731 shutdown_connections 2023-10-07T09:41:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.376+0000 7f3ad33ff700 1 -- 172.21.15.29:0/1213614731 wait complete. 2023-10-07T09:41:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.378+0000 7f3ad33ff700 1 Processor -- start 2023-10-07T09:41:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.378+0000 7f3ad33ff700 1 -- start start 2023-10-07T09:41:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.379+0000 7f3ad33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0b0610 0x7f3acc1692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.379+0000 7f3ad33ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 0x7f3acc14b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.379+0000 7f3ad33ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3acc0ced00 0x7f3acc0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.379+0000 7f3ad33ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0c5530 con 0x7f3acc0c8a20 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.379+0000 7f3ad33ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0c5810 con 0x7f3acc0ced00 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.379+0000 7f3ad33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0c5af0 con 0x7f3acc0b0610 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 0x7f3acc14b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 0x7f3acc14b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49258/0 (socket says 172.21.15.29:49258) 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 -- 172.21.15.29:0/126361296 learned_addr learned my addr 172.21.15.29:0/126361296 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad199c700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3acc0ced00 0x7f3acc0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad119b700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0b0610 0x7f3acc1692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 -- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0b0610 msgr2=0x7f3acc1692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0b0610 0x7f3acc1692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 -- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3acc0ced00 msgr2=0x7f3acc0cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3acc0ced00 0x7f3acc0cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 -- 172.21.15.29:0/126361296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc009420 con 0x7f3acc0c8a20 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad199c700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3acc0ced00 0x7f3acc0cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.380+0000 7f3ad099a700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 0x7f3acc14b200 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00ec50 tx=0x7f3abc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.381+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc00c9d0 con 0x7f3acc0c8a20 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.381+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc00cb30 con 0x7f3acc0c8a20 2023-10-07T09:41:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.381+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac4009a20 con 0x7f3acc0c8a20 2023-10-07T09:41:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.381+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc013470 con 0x7f3acc0c8a20 2023-10-07T09:41:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.381+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3acc14b740 con 0x7f3acc0c8a20 2023-10-07T09:41:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.382+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3acc0c6140 con 0x7f3acc0c8a20 2023-10-07T09:41:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.383+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f3abc0137e0 con 0x7f3acc0c8a20 2023-10-07T09:41:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.384+0000 7f3ac17fa700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ab0067130 0x7f3ab00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.384+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f3abc084dd0 con 0x7f3acc0c8a20 2023-10-07T09:41:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.385+0000 7f3ad119b700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ab0067130 0x7f3ab00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.385+0000 7f3ad119b700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ab0067130 0x7f3ab00695e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c8f0 tx=0x7f3ac8009040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:26.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.388+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3abc051cc0 con 0x7f3acc0c8a20 2023-10-07T09:41:26.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.618+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3acc0c5670 con 0x7f3ab0067130 2023-10-07T09:41:26.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.629+0000 7f3ac17fa700 1 -- 172.21.15.29:0/126361296 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f3acc0c5670 con 0x7f3ab0067130 2023-10-07T09:41:26.631 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:26.631 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:48", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:26.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ab0067130 msgr2=0x7f3ab00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ab0067130 0x7f3ab00695e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c8f0 tx=0x7f3ac8009040 comp rx=0 tx=0).stop 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 msgr2=0x7f3acc14b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 0x7f3acc14b200 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00ec50 tx=0x7f3abc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 shutdown_connections 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f3ab0067130 0x7f3ab00695e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3acc0ced00 0x7f3acc0cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3acc0b0610 0x7f3acc1692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 --2- 172.21.15.29:0/126361296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc0c8a20 0x7f3acc14b200 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 >> 172.21.15.29:0/126361296 conn(0x7f3acc063a10 msgr2=0x7f3acc0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:26.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 shutdown_connections 2023-10-07T09:41:26.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.632+0000 7f3ad33ff700 1 -- 172.21.15.29:0/126361296 wait complete. 2023-10-07T09:41:26.650 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:48", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:26.651 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:48', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:26.651 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:26.651 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:26.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:26.651 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-07T09:41:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:26.703 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:26.703 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:26.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir1 2023-10-07T09:41:26.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.902+0000 7fd8b57ff700 1 Processor -- start 2023-10-07T09:41:26.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.903+0000 7fd8b57ff700 1 -- start start 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8b57ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd8b00ced00 0x7fd8b00b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8b57ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8b00c8a20 0x7fd8b00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8b57ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00690f0 con 0x7fd8b00b0610 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8b57ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00cf2e0 con 0x7fd8b00ced00 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8b57ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00687c0 con 0x7fd8b00c8a20 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8ae7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd8b00ced00 0x7fd8b00b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8aeffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd8b00b0610 0x7fd8b00c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51792/0 (socket says 172.21.15.29:51792) 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.904+0000 7fd8aeffd700 1 -- 172.21.15.29:0/1179752066 learned_addr learned my addr 172.21.15.29:0/1179752066 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8af7fe700 1 --2- 172.21.15.29:0/1179752066 >> v2:172.21.15.144:3300/0 conn(0x7fd8b00c8a20 0x7fd8b00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8ae7fc700 1 -- 172.21.15.29:0/1179752066 >> v2:172.21.15.144:3300/0 conn(0x7fd8b00c8a20 msgr2=0x7fd8b00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8ae7fc700 1 --2- 172.21.15.29:0/1179752066 >> v2:172.21.15.144:3300/0 conn(0x7fd8b00c8a20 0x7fd8b00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8ae7fc700 1 -- 172.21.15.29:0/1179752066 >> v1:172.21.15.29:6789/0 conn(0x7fd8b00b0610 legacy=0x7fd8b00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8adffb700 1 -- 172.21.15.29:0/1179752066 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794521636 0 0) 0x7fd8b00690f0 con 0x7fd8b00b0610 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8ae7fc700 1 -- 172.21.15.29:0/1179752066 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b0068aa0 con 0x7fd8b00ced00 2023-10-07T09:41:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8ae7fc700 1 --2- 172.21.15.29:0/1179752066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00ced00 0x7fd8b00b09e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0070ea0 tx=0x7fd8a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c734235d4337681 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8adffb700 1 -- 172.21.15.29:0/1179752066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400b6b0 con 0x7fd8b00ced00 2023-10-07T09:41:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8adffb700 1 -- 172.21.15.29:0/1179752066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a400b810 con 0x7fd8b00ced00 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.905+0000 7fd8adffb700 1 -- 172.21.15.29:0/1179752066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400bbb0 con 0x7fd8b00ced00 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 -- 172.21.15.29:0/1179752066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00ced00 msgr2=0x7fd8b00b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/1179752066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00ced00 0x7fd8b00b09e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0070ea0 tx=0x7fd8a40091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 -- 172.21.15.29:0/1179752066 shutdown_connections 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/1179752066 >> v2:172.21.15.144:3300/0 conn(0x7fd8b00c8a20 0x7fd8b00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/1179752066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00ced00 0x7fd8b00b09e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 -- 172.21.15.29:0/1179752066 >> 172.21.15.29:0/1179752066 conn(0x7fd8b0063a10 msgr2=0x7fd8b00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 -- 172.21.15.29:0/1179752066 shutdown_connections 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.906+0000 7fd8b57ff700 1 -- 172.21.15.29:0/1179752066 wait complete. 2023-10-07T09:41:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.907+0000 7fd8b57ff700 1 Processor -- start 2023-10-07T09:41:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.908+0000 7fd8b57ff700 1 -- start start 2023-10-07T09:41:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8b57ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 0x7fd8b0156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8b57ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b00c8a20 0x7fd8b009b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8b57ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00ced00 0x7fd8b00cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8b57ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0169960 con 0x7fd8b00b0610 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8b57ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c6c10 con 0x7fd8b00c8a20 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8b57ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01574e0 con 0x7fd8b00ced00 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 0x7fd8b0156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 0x7fd8b0156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49264/0 (socket says 172.21.15.29:49264) 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 -- 172.21.15.29:0/2924253383 learned_addr learned my addr 172.21.15.29:0/2924253383 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8af7fe700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00ced00 0x7fd8b00cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 -- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00ced00 msgr2=0x7fd8b00cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8ae7fc700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b00c8a20 0x7fd8b009b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00ced00 0x7fd8b00cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 -- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b00c8a20 msgr2=0x7fd8b009b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b00c8a20 0x7fd8b009b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8aeffd700 1 -- 172.21.15.29:0/2924253383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a4009420 con 0x7fd8b00b0610 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8af7fe700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00ced00 0x7fd8b00cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.909+0000 7fd8ae7fc700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b00c8a20 0x7fd8b009b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.910+0000 7fd8aeffd700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 0x7fd8b0156fa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c5b0 tx=0x7fd8a0001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.910+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0005b30 con 0x7fd8b00b0610 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.910+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0005c90 con 0x7fd8b00b0610 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.910+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0007300 con 0x7fd8b00b0610 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.910+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b009bbd0 con 0x7fd8b00b0610 2023-10-07T09:41:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.910+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00c65c0 con 0x7fd8b00b0610 2023-10-07T09:41:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.911+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b00718b0 con 0x7fd8b00b0610 2023-10-07T09:41:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.912+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fd8a00076b0 con 0x7fd8b00b0610 2023-10-07T09:41:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.913+0000 7fd896ffd700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd89c067130 0x7fd89c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.913+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fd8a0083ef0 con 0x7fd8b00b0610 2023-10-07T09:41:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.913+0000 7fd8ae7fc700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd89c067130 0x7fd89c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.914+0000 7fd8ae7fc700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd89c067130 0x7fd89c0695e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0070ea0 tx=0x7fd8a4017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:26.917+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8a0050d40 con 0x7fd8b00b0610 2023-10-07T09:41:27.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.146+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8b0068d80 con 0x7fd89c067130 2023-10-07T09:41:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.156+0000 7fd896ffd700 1 -- 172.21.15.29:0/2924253383 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fd8b0068d80 con 0x7fd89c067130 2023-10-07T09:41:27.157 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:27.157 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:48", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd89c067130 msgr2=0x7fd89c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd89c067130 0x7fd89c0695e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0070ea0 tx=0x7fd8a4017000 comp rx=0 tx=0).stop 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 msgr2=0x7fd8b0156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 0x7fd8b0156fa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c5b0 tx=0x7fd8a0001070 comp rx=0 tx=0).stop 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 shutdown_connections 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd89c067130 0x7fd89c0695e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00ced00 0x7fd8b00cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00b0610 0x7fd8b0156fa0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.158+0000 7fd8b57ff700 1 --2- 172.21.15.29:0/2924253383 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b00c8a20 0x7fd8b009b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.159+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 >> 172.21.15.29:0/2924253383 conn(0x7fd8b0063a10 msgr2=0x7fd8b0158d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.159+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 shutdown_connections 2023-10-07T09:41:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.159+0000 7fd8b57ff700 1 -- 172.21.15.29:0/2924253383 wait complete. 2023-10-07T09:41:27.177 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:48", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:27.177 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:48', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:27.177 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:27.177 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:27.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:27.177 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-07T09:41:27.226 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:27.228 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:27.229 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:27.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir2 2023-10-07T09:41:27.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.429+0000 7f1f915ff700 1 Processor -- start 2023-10-07T09:41:27.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.429+0000 7f1f915ff700 1 -- start start 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.430+0000 7f1f915ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1f8c0b2a20 0x7f1f8c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f915ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0cc940 con 0x7f1f8c0cb6f0 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f915ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0b2df0 con 0x7f1f8c0cf2c0 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f915ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8c0cbd30 con 0x7f1f8c0b2a20 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f8a7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1f8c0cb6f0 0x7f1f8c0cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51806/0 (socket says 172.21.15.29:51806) 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f8a7fc700 1 -- 172.21.15.29:0/2427754381 learned_addr learned my addr 172.21.15.29:0/2427754381 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f8affd700 1 --2- 172.21.15.29:0/2427754381 >> v2:172.21.15.144:3300/0 conn(0x7f1f8c0b2a20 0x7f1f8c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588338731 0 0) 0x7f1f8c0cc940 con 0x7f1f8c0cb6f0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f74003570 con 0x7f1f8c0cb6f0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545223744 0 0) 0x7f1f8c0b2df0 con 0x7f1f8c0cf2c0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.431+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8c0cc940 con 0x7f1f8c0cf2c0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170523251 0 0) 0x7f1f74003570 con 0x7f1f8c0cb6f0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 >> v2:172.21.15.144:3300/0 conn(0x7f1f8c0b2a20 msgr2=0x7f1f8c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 --2- 172.21.15.29:0/2427754381 >> v2:172.21.15.144:3300/0 conn(0x7f1f8c0b2a20 0x7f1f8c0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 >> v1:172.21.15.144:6789/0 conn(0x7f1f8c0cf2c0 legacy=0x7f1f8c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0cc010 con 0x7f1f8c0cb6f0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f1f80001470 con 0x7f1f8c0cb6f0 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 >> v1:172.21.15.29:6789/0 conn(0x7f1f8c0cb6f0 legacy=0x7f1f8c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f74003bf0 0x7f1f74003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f74004810 0x7f1f74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f740051c0 0x7f1f7400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0cbd30 con 0x7f1f740051c0 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7400c0b0 con 0x7f1f74003bf0 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7400c270 con 0x7f1f74004810 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8b7fe700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f740051c0 0x7f1f7400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8affd700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f74003bf0 0x7f1f74003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8a7fc700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f74004810 0x7f1f74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8b7fe700 1 -- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f74004810 msgr2=0x7f1f74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8b7fe700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f74004810 0x7f1f74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8b7fe700 1 -- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f74003bf0 msgr2=0x7f1f74003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8b7fe700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f74003bf0 0x7f1f74003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.432+0000 7f1f8b7fe700 1 -- 172.21.15.29:0/2427754381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74003570 con 0x7f1f740051c0 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f8b7fe700 1 -- 172.21.15.29:0/2427754381 reap_dead start 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f8b7fe700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f740051c0 0x7f1f7400ba60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1f840020f0 tx=0x7f1f840017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6afa92c928781 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400c8b0 con 0x7f1f740051c0 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8400ca10 con 0x7f1f740051c0 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f89ffb700 1 -- 172.21.15.29:0/2427754381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400cd90 con 0x7f1f740051c0 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f915ff700 1 -- 172.21.15.29:0/2427754381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f740051c0 msgr2=0x7f1f7400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.433+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f740051c0 0x7f1f7400ba60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1f840020f0 tx=0x7f1f840017a0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.434+0000 7f1f915ff700 1 -- 172.21.15.29:0/2427754381 shutdown_connections 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.434+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2427754381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f740051c0 0x7f1f7400ba60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.434+0000 7f1f915ff700 1 -- 172.21.15.29:0/2427754381 >> 172.21.15.29:0/2427754381 conn(0x7f1f8c099820 msgr2=0x7f1f8c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.434+0000 7f1f915ff700 1 -- 172.21.15.29:0/2427754381 shutdown_connections 2023-10-07T09:41:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.434+0000 7f1f915ff700 1 -- 172.21.15.29:0/2427754381 wait complete. 2023-10-07T09:41:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.436+0000 7f1f915ff700 1 Processor -- start 2023-10-07T09:41:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.436+0000 7f1f915ff700 1 -- start start 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f915ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f740051c0 0x7f1f8c0b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f915ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 0x7f1f8c14dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f915ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8c16dcc0 0x7f1f8c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f915ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c1565a0 con 0x7f1f8c16d850 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f915ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0c9590 con 0x7f1f8c16dcc0 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f915ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0c9810 con 0x7f1f740051c0 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f8a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 0x7f1f8c14dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f8a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 0x7f1f8c14dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49280/0 (socket says 172.21.15.29:49280) 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.437+0000 7f1f8a7fc700 1 -- 172.21.15.29:0/2574069664 learned_addr learned my addr 172.21.15.29:0/2574069664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8affd700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f740051c0 0x7f1f8c0b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8b7fe700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8c16dcc0 0x7f1f8c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8a7fc700 1 -- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f740051c0 msgr2=0x7f1f8c0b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8a7fc700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f740051c0 0x7f1f8c0b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8a7fc700 1 -- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8c16dcc0 msgr2=0x7f1f8c14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8a7fc700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8c16dcc0 0x7f1f8c14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8a7fc700 1 -- 172.21.15.29:0/2574069664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0c9af0 con 0x7f1f8c16d850 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8b7fe700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8c16dcc0 0x7f1f8c14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f8a7fc700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 0x7f1f8c14dbc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000e7b0 tx=0x7f1f80001de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80010490 con 0x7f1f8c16d850 2023-10-07T09:41:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f80010040 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80010860 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.438+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c0c9dd0 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.439+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c0b6050 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.440+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c156a10 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.441+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f1f80010bd0 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.442+0000 7f1f72ffd700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1f78067160 0x7f1f78069610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.442+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f1f80083c30 con 0x7f1f8c16d850 2023-10-07T09:41:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.442+0000 7f1f8affd700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1f78067160 0x7f1f78069610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.443+0000 7f1f8affd700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1f78067160 0x7f1f78069610 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0090f0 tx=0x7f1f7c009120 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.446+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f80050970 con 0x7f1f8c16d850 2023-10-07T09:41:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.673+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f8c0c9f60 con 0x7f1f78067160 2023-10-07T09:41:27.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.682+0000 7f1f72ffd700 1 -- 172.21.15.29:0/2574069664 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f1f8c0c9f60 con 0x7f1f78067160 2023-10-07T09:41:27.684 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:27.684 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:49", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.684+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1f78067160 msgr2=0x7f1f78069610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.684+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1f78067160 0x7f1f78069610 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0090f0 tx=0x7f1f7c009120 comp rx=0 tx=0).stop 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.684+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 msgr2=0x7f1f8c14dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.684+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 0x7f1f8c14dbc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000e7b0 tx=0x7f1f80001de0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 shutdown_connections 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f1f78067160 0x7f1f78069610 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8c16dcc0 0x7f1f8c14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f740051c0 0x7f1f8c0b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 --2- 172.21.15.29:0/2574069664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8c16d850 0x7f1f8c14dbc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 >> 172.21.15.29:0/2574069664 conn(0x7f1f8c099820 msgr2=0x7f1f8c14eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:27.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 shutdown_connections 2023-10-07T09:41:27.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.685+0000 7f1f915ff700 1 -- 172.21.15.29:0/2574069664 wait complete. 2023-10-07T09:41:27.703 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:49", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:27.704 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:49', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:27.704 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:27.704 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:27.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:27.704 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_01_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:27.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:27.754 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_01_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:27.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir3 2023-10-07T09:41:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.953+0000 7ff7cbfff700 1 Processor -- start 2023-10-07T09:41:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.954+0000 7ff7cbfff700 1 -- start start 2023-10-07T09:41:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.955+0000 7ff7cbfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7c40ced00 0x7ff7c40b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.955+0000 7ff7cbfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff7c40c8a20 0x7ff7c40cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7cbfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c40690f0 con 0x7ff7c40b0610 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7cbfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40cf2e0 con 0x7ff7c40c8a20 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7cbfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40687c0 con 0x7ff7c40ced00 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7c9d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff7c40b0610 0x7ff7c40c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51812/0 (socket says 172.21.15.29:51812) 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7c9d9b700 1 -- 172.21.15.29:0/3440136371 learned_addr learned my addr 172.21.15.29:0/3440136371 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7ca59c700 1 --2- 172.21.15.29:0/3440136371 >> v2:172.21.15.29:3300/0 conn(0x7ff7c40c8a20 0x7ff7c40cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7c959a700 1 --2- 172.21.15.29:0/3440136371 >> v2:172.21.15.144:3300/0 conn(0x7ff7c40ced00 0x7ff7c40b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7ca59c700 1 -- 172.21.15.29:0/3440136371 >> v2:172.21.15.144:3300/0 conn(0x7ff7c40ced00 msgr2=0x7ff7c40b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7ca59c700 1 --2- 172.21.15.29:0/3440136371 >> v2:172.21.15.144:3300/0 conn(0x7ff7c40ced00 0x7ff7c40b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7ca59c700 1 -- 172.21.15.29:0/3440136371 >> v1:172.21.15.29:6789/0 conn(0x7ff7c40b0610 legacy=0x7ff7c40c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7ca59c700 1 -- 172.21.15.29:0/3440136371 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c4068aa0 con 0x7ff7c40c8a20 2023-10-07T09:41:27.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.956+0000 7ff7c8d99700 1 -- 172.21.15.29:0/3440136371 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668602618 0 0) 0x7ff7c40690f0 con 0x7ff7c40b0610 2023-10-07T09:41:27.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.957+0000 7ff7ca59c700 1 --2- 172.21.15.29:0/3440136371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40c8a20 0x7ff7c40cc7e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40cc2f0 tx=0x7ff7bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b835b64ebbd3e9fb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.957+0000 7ff7c8d99700 1 -- 172.21.15.29:0/3440136371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc00e070 con 0x7ff7c40c8a20 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.957+0000 7ff7c8d99700 1 -- 172.21.15.29:0/3440136371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc00bc30 con 0x7ff7c40c8a20 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.957+0000 7ff7c8d99700 1 -- 172.21.15.29:0/3440136371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc012430 con 0x7ff7c40c8a20 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.957+0000 7ff7cbfff700 1 -- 172.21.15.29:0/3440136371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40c8a20 msgr2=0x7ff7c40cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.957+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/3440136371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40c8a20 0x7ff7c40cc7e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40cc2f0 tx=0x7ff7bc009780 comp rx=0 tx=0).stop 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.958+0000 7ff7cbfff700 1 -- 172.21.15.29:0/3440136371 shutdown_connections 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.958+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/3440136371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40c8a20 0x7ff7c40cc7e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.958+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/3440136371 >> v2:172.21.15.144:3300/0 conn(0x7ff7c40ced00 0x7ff7c40b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.958+0000 7ff7cbfff700 1 -- 172.21.15.29:0/3440136371 >> 172.21.15.29:0/3440136371 conn(0x7ff7c4063a10 msgr2=0x7ff7c40ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:27.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.958+0000 7ff7cbfff700 1 -- 172.21.15.29:0/3440136371 shutdown_connections 2023-10-07T09:41:27.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.958+0000 7ff7cbfff700 1 -- 172.21.15.29:0/3440136371 wait complete. 2023-10-07T09:41:27.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.960+0000 7ff7cbfff700 1 Processor -- start 2023-10-07T09:41:27.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.960+0000 7ff7cbfff700 1 -- start start 2023-10-07T09:41:27.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7cbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 0x7ff7c4158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7cbfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40c8a20 0x7ff7c41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7cbfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c40ced00 0x7ff7c41693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7cbfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40c7580 con 0x7ff7c40b0610 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7cbfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c4158d50 con 0x7ff7c40c8a20 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7cbfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40c6c50 con 0x7ff7c40ced00 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7c9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 0x7ff7c4158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7c9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 0x7ff7c4158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49298/0 (socket says 172.21.15.29:49298) 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.961+0000 7ff7c9d9b700 1 -- 172.21.15.29:0/1535601372 learned_addr learned my addr 172.21.15.29:0/1535601372 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c9d9b700 1 -- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c40ced00 msgr2=0x7ff7c41693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7ca59c700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c40ced00 0x7ff7c41693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c959a700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40c8a20 0x7ff7c41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c9d9b700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c40ced00 0x7ff7c41693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c9d9b700 1 -- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40c8a20 msgr2=0x7ff7c41542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c9d9b700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40c8a20 0x7ff7c41542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c9d9b700 1 -- 172.21.15.29:0/1535601372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc009a20 con 0x7ff7c40b0610 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7ca59c700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c40ced00 0x7ff7c41693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7c9d9b700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 0x7ff7c4158770 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000c5b0 tx=0x7ff7c0001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0005b30 con 0x7ff7c40b0610 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c0005c90 con 0x7ff7c40b0610 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c40c6f30 con 0x7ff7c40b0610 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.962+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0007300 con 0x7ff7c40b0610 2023-10-07T09:41:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.963+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c4154940 con 0x7ff7c40b0610 2023-10-07T09:41:27.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.964+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c40718b0 con 0x7ff7c40b0610 2023-10-07T09:41:27.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.964+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7ff7c001b3b0 con 0x7ff7c40b0610 2023-10-07T09:41:27.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.965+0000 7ff7b9ffb700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7a8067130 0x7ff7a80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:27.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.965+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ff7c0083f80 con 0x7ff7c40b0610 2023-10-07T09:41:27.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.966+0000 7ff7c959a700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7a8067130 0x7ff7a80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:27.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.966+0000 7ff7c959a700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7a8067130 0x7ff7a80695e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40b4220 tx=0x7ff7b4008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:27.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:27.969+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7c0050dd0 con 0x7ff7c40b0610 2023-10-07T09:41:28.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.198+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7c40c76c0 con 0x7ff7a8067130 2023-10-07T09:41:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.206+0000 7ff7b9ffb700 1 -- 172.21.15.29:0/1535601372 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7ff7b4009710 con 0x7ff7a8067130 2023-10-07T09:41:28.208 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:28.208 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:49", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7a8067130 msgr2=0x7ff7a80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7a8067130 0x7ff7a80695e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40b4220 tx=0x7ff7b4008040 comp rx=0 tx=0).stop 2023-10-07T09:41:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 msgr2=0x7ff7c4158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 0x7ff7c4158770 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000c5b0 tx=0x7ff7c0001070 comp rx=0 tx=0).stop 2023-10-07T09:41:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 shutdown_connections 2023-10-07T09:41:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7a8067130 0x7ff7a80695e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c40ced00 0x7ff7c41693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7c40b0610 0x7ff7c4158770 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 --2- 172.21.15.29:0/1535601372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40c8a20 0x7ff7c41542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 >> 172.21.15.29:0/1535601372 conn(0x7ff7c4063a10 msgr2=0x7ff7c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:28.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 shutdown_connections 2023-10-07T09:41:28.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.209+0000 7ff7cbfff700 1 -- 172.21.15.29:0/1535601372 wait complete. 2023-10-07T09:41:28.227 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:49", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:28.228 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:49', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:28.228 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:28.228 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:28.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:28.228 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-07T09:41:28.277 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:28.278 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:28.279 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:28.279 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:28.279 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:28.280 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:28.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir4 2023-10-07T09:41:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.479+0000 7f853dbff700 1 Processor -- start 2023-10-07T09:41:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.479+0000 7f853dbff700 1 -- start start 2023-10-07T09:41:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.480+0000 7f853dbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f85380cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.480+0000 7f853dbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85380cb6f0 0x7f85380cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.480+0000 7f853dbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380cc940 con 0x7f85380cf0f0 2023-10-07T09:41:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.480+0000 7f853dbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380cf530 con 0x7f85380cb6f0 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f853dbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380b2df0 con 0x7f85380b2a20 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8536ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85380cb6f0 0x7f85380cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8537fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f85380cf0f0 0x7f8538159ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51816/0 (socket says 172.21.15.29:51816) 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8537fff700 1 -- 172.21.15.29:0/3963029001 learned_addr learned my addr 172.21.15.29:0/3963029001 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f85377fe700 1 --2- 172.21.15.29:0/3963029001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f85380cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8536ffd700 1 -- 172.21.15.29:0/3963029001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 msgr2=0x7f85380cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8536ffd700 1 --2- 172.21.15.29:0/3963029001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f85380cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8536ffd700 1 -- 172.21.15.29:0/3963029001 >> v1:172.21.15.29:6789/0 conn(0x7f85380cf0f0 legacy=0x7f8538159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.481+0000 7f8536ffd700 1 -- 172.21.15.29:0/3963029001 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85380cbd30 con 0x7f85380cb6f0 2023-10-07T09:41:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f8536ffd700 1 --2- 172.21.15.29:0/3963029001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cb6f0 0x7f85380cebb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098e0 tx=0x7f852c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18d2ec3f1c041e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f85367fc700 1 -- 172.21.15.29:0/3963029001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c009e80 con 0x7f85380cb6f0 2023-10-07T09:41:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f85367fc700 1 -- 172.21.15.29:0/3963029001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00b860 con 0x7f85380cb6f0 2023-10-07T09:41:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f85367fc700 1 -- 172.21.15.29:0/3963029001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00bbe0 con 0x7f85380cb6f0 2023-10-07T09:41:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 -- 172.21.15.29:0/3963029001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cb6f0 msgr2=0x7f85380cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 --2- 172.21.15.29:0/3963029001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cb6f0 0x7f85380cebb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098e0 tx=0x7f852c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 -- 172.21.15.29:0/3963029001 shutdown_connections 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 --2- 172.21.15.29:0/3963029001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f85380cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 --2- 172.21.15.29:0/3963029001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cb6f0 0x7f85380cebb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 -- 172.21.15.29:0/3963029001 >> 172.21.15.29:0/3963029001 conn(0x7f8538099820 msgr2=0x7f8538063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.482+0000 7f853dbff700 1 -- 172.21.15.29:0/3963029001 shutdown_connections 2023-10-07T09:41:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.483+0000 7f853dbff700 1 -- 172.21.15.29:0/3963029001 wait complete. 2023-10-07T09:41:28.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.485+0000 7f853dbff700 1 Processor -- start 2023-10-07T09:41:28.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.485+0000 7f853dbff700 1 -- start start 2023-10-07T09:41:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f853dbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f8538157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f853dbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85380cb6f0 0x7f853809b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f853dbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 0x7f8538159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f853dbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380b5d40 con 0x7f85380cf0f0 2023-10-07T09:41:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f853dbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853809af30 con 0x7f85380cb6f0 2023-10-07T09:41:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f853dbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380b5f90 con 0x7f85380b2a20 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f8537fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 0x7f8538159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f8537fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 0x7f8538159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49318/0 (socket says 172.21.15.29:49318) 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.486+0000 7f8537fff700 1 -- 172.21.15.29:0/2317522483 learned_addr learned my addr 172.21.15.29:0/2317522483 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f85377fe700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f8538157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8536ffd700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85380cb6f0 0x7f853809b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8537fff700 1 -- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 msgr2=0x7f8538157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8537fff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f8538157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8537fff700 1 -- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85380cb6f0 msgr2=0x7f853809b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8537fff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85380cb6f0 0x7f853809b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8537fff700 1 -- 172.21.15.29:0/2317522483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c009420 con 0x7f85380cf0f0 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8536ffd700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85380cb6f0 0x7f853809b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:28.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f8537fff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 0x7f8538159fe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f853000cbf0 tx=0x7f85300019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:28.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853000a980 con 0x7f85380cf0f0 2023-10-07T09:41:28.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853000aae0 con 0x7f85380cf0f0 2023-10-07T09:41:28.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853809a8e0 con 0x7f85380cf0f0 2023-10-07T09:41:28.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.487+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853000ae60 con 0x7f85380cf0f0 2023-10-07T09:41:28.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.488+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853815a520 con 0x7f85380cf0f0 2023-10-07T09:41:28.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.489+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85380718b0 con 0x7f85380cf0f0 2023-10-07T09:41:28.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.490+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f8530007300 con 0x7f85380cf0f0 2023-10-07T09:41:28.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.491+0000 7f851f7fe700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8524067130 0x7f85240695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:28.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.491+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f8530084da0 con 0x7f85380cf0f0 2023-10-07T09:41:28.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.492+0000 7f85377fe700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8524067130 0x7f85240695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:28.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.492+0000 7f85377fe700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8524067130 0x7f85240695e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8528006aa0 tx=0x7f8528008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:28.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.495+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8530051c80 con 0x7f85380cf0f0 2023-10-07T09:41:28.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.722+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85380cce70 con 0x7f8524067130 2023-10-07T09:41:28.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.732+0000 7f851f7fe700 1 -- 172.21.15.29:0/2317522483 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f85380cce70 con 0x7f8524067130 2023-10-07T09:41:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:28.734 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:50", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8524067130 msgr2=0x7f85240695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8524067130 0x7f85240695e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8528006aa0 tx=0x7f8528008040 comp rx=0 tx=0).stop 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 msgr2=0x7f8538159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 0x7f8538159fe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f853000cbf0 tx=0x7f85300019c0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 shutdown_connections 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f8524067130 0x7f85240695e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85380cf0f0 0x7f8538159fe0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85380b2a20 0x7f8538157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 --2- 172.21.15.29:0/2317522483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85380cb6f0 0x7f853809b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 >> 172.21.15.29:0/2317522483 conn(0x7f8538099820 msgr2=0x7f85380c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 shutdown_connections 2023-10-07T09:41:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:28.735+0000 7f853dbff700 1 -- 172.21.15.29:0/2317522483 wait complete. 2023-10-07T09:41:28.753 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:50", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:28.754 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:50', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:28.754 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:28.754 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:28.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:28.754 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_01_UTC 2023-10-07T09:41:28.802 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:28.803 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:28.803 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:28.803 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:28.803 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:28.804 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_01_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:28.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir5 2023-10-07T09:41:29.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.003+0000 7fa5ff1ff700 1 Processor -- start 2023-10-07T09:41:29.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.003+0000 7fa5ff1ff700 1 -- start start 2023-10-07T09:41:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5ff1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5f80b2a20 0x7fa5f80cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5ff1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80cb6f0 0x7fa5f80cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5ff1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5f80cf0f0 0x7fa5f8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5ff1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f80cc940 con 0x7fa5f80b2a20 2023-10-07T09:41:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5ff1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f80cf530 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5ff1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80b2df0 con 0x7fa5f80cb6f0 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5fcf9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5f80b2a20 0x7fa5f80cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5fcf9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5f80b2a20 0x7fa5f80cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49320/0 (socket says 172.21.15.29:49320) 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5fcf9b700 1 -- 172.21.15.29:0/2172714846 learned_addr learned my addr 172.21.15.29:0/2172714846 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5f7fff700 1 --2- 172.21.15.29:0/2172714846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80cb6f0 0x7fa5f80cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.005+0000 7fa5fd79c700 1 --2- 172.21.15.29:0/2172714846 >> v2:172.21.15.144:3300/0 conn(0x7fa5f80cf0f0 0x7fa5f8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fcf9b700 1 -- 172.21.15.29:0/2172714846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80cb6f0 msgr2=0x7fa5f80cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fcf9b700 1 --2- 172.21.15.29:0/2172714846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80cb6f0 0x7fa5f80cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fcf9b700 1 -- 172.21.15.29:0/2172714846 >> v2:172.21.15.144:3300/0 conn(0x7fa5f80cf0f0 msgr2=0x7fa5f8159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fcf9b700 1 --2- 172.21.15.29:0/2172714846 >> v2:172.21.15.144:3300/0 conn(0x7fa5f80cf0f0 0x7fa5f8159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fcf9b700 1 -- 172.21.15.29:0/2172714846 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f80cbd30 con 0x7fa5f80b2a20 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fd79c700 1 --2- 172.21.15.29:0/2172714846 >> v2:172.21.15.144:3300/0 conn(0x7fa5f80cf0f0 0x7fa5f8159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5fcf9b700 1 --2- 172.21.15.29:0/2172714846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80b2a20 0x7fa5f80cb1b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa5f815ae20 tx=0x7fa5ec009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ff4cd8141f18b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5f77fe700 1 -- 172.21.15.29:0/2172714846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec005c10 con 0x7fa5f80b2a20 2023-10-07T09:41:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5f77fe700 1 -- 172.21.15.29:0/2172714846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec005d70 con 0x7fa5f80b2a20 2023-10-07T09:41:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.006+0000 7fa5f77fe700 1 -- 172.21.15.29:0/2172714846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec006260 con 0x7fa5f80b2a20 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2172714846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80b2a20 msgr2=0x7fa5f80cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2172714846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80b2a20 0x7fa5f80cb1b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa5f815ae20 tx=0x7fa5ec009220 comp rx=0 tx=0).stop 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2172714846 shutdown_connections 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2172714846 >> v2:172.21.15.144:3300/0 conn(0x7fa5f80cf0f0 0x7fa5f8159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2172714846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80b2a20 0x7fa5f80cb1b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2172714846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80cb6f0 0x7fa5f80cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2172714846 >> 172.21.15.29:0/2172714846 conn(0x7fa5f8099820 msgr2=0x7fa5f8063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2172714846 shutdown_connections 2023-10-07T09:41:29.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.007+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2172714846 wait complete. 2023-10-07T09:41:29.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.008+0000 7fa5ff1ff700 1 Processor -- start 2023-10-07T09:41:29.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.008+0000 7fa5ff1ff700 1 -- start start 2023-10-07T09:41:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.009+0000 7fa5ff1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80b2a20 0x7fa5f814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.009+0000 7fa5ff1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f80cb6f0 0x7fa5f80c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.009+0000 7fa5ff1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 0x7fa5f80c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.009+0000 7fa5ff1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80ca660 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.009+0000 7fa5ff1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f816d630 con 0x7fa5f80cb6f0 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.009+0000 7fa5ff1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80ca0a0 con 0x7fa5f80b2a20 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 0x7fa5f80c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 0x7fa5f80c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49330/0 (socket says 172.21.15.29:49330) 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 -- 172.21.15.29:0/2569073613 learned_addr learned my addr 172.21.15.29:0/2569073613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fcf9b700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80b2a20 0x7fa5f814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5f7fff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f80cb6f0 0x7fa5f80c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 -- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80b2a20 msgr2=0x7fa5f814e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80b2a20 0x7fa5f814e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 -- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f80cb6f0 msgr2=0x7fa5f80c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f80cb6f0 0x7fa5f80c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 -- 172.21.15.29:0/2569073613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009480 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5f7fff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f80cb6f0 0x7fa5f80c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.010+0000 7fa5fd79c700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 0x7fa5f80c9790 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000d100 tx=0x7fa5f00099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.011+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000dcf0 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.011+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f000f040 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.011+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f80ca380 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.011+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0015690 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.011+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f80c8270 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.012+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f816d120 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.013+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fa5f001e3b0 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.014+0000 7fa5f4ff9700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5dc067130 0x7fa5dc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.014+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fa5f0084f40 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.015+0000 7fa5fcf9b700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5dc067130 0x7fa5dc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.015+0000 7fa5fcf9b700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5dc067130 0x7fa5dc0695e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009260 tx=0x7fa5ec016000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:29.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.018+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5f816d120 con 0x7fa5f80cf0f0 2023-10-07T09:41:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.246+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5f816d120 con 0x7fa5dc067130 2023-10-07T09:41:29.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.256+0000 7fa5f4ff9700 1 -- 172.21.15.29:0/2569073613 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fa5f816d120 con 0x7fa5dc067130 2023-10-07T09:41:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:29.257 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:51", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5dc067130 msgr2=0x7fa5dc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5dc067130 0x7fa5dc0695e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009260 tx=0x7fa5ec016000 comp rx=0 tx=0).stop 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 msgr2=0x7fa5f80c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 0x7fa5f80c9790 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000d100 tx=0x7fa5f00099b0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 shutdown_connections 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa5dc067130 0x7fa5dc0695e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5f80cf0f0 0x7fa5f80c9790 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f80b2a20 0x7fa5f814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 --2- 172.21.15.29:0/2569073613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f80cb6f0 0x7fa5f80c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 >> 172.21.15.29:0/2569073613 conn(0x7fa5f8099820 msgr2=0x7fa5f8158e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.258+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 shutdown_connections 2023-10-07T09:41:29.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.259+0000 7fa5ff1ff700 1 -- 172.21.15.29:0/2569073613 wait complete. 2023-10-07T09:41:29.277 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:51", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:29.277 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:51', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:29.277 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:29.277 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:29.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:29.278 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:29.328 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:29.328 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:29.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir6 2023-10-07T09:41:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.527+0000 7fb1701ff700 1 Processor -- start 2023-10-07T09:41:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.527+0000 7fb1701ff700 1 -- start start 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb1701ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680b2a20 0x7fb1680cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb1701ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1680cf2c0 0x7fb1680c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb1701ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1680cc940 con 0x7fb1680cb6f0 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb1701ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680b2df0 con 0x7fb1680cf2c0 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb1701ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680cbd30 con 0x7fb1680b2a20 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb16e79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1680cf2c0 0x7fb1680c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb16e79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1680cf2c0 0x7fb1680c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49332/0 (socket says 172.21.15.29:49332) 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb16e79c700 1 -- 172.21.15.29:0/3223699356 learned_addr learned my addr 172.21.15.29:0/3223699356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.528+0000 7fb16d79a700 1 --1- 172.21.15.29:0/3223699356 >> v1:172.21.15.29:6789/0 conn(0x7fb1680cb6f0 0x7fb1680cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51824/0 (socket says 172.21.15.29:51824) 2023-10-07T09:41:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16df9b700 1 --2- 172.21.15.29:0/3223699356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680b2a20 0x7fb1680cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16e79c700 1 -- 172.21.15.29:0/3223699356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680b2a20 msgr2=0x7fb1680cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16e79c700 1 --2- 172.21.15.29:0/3223699356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680b2a20 0x7fb1680cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16e79c700 1 -- 172.21.15.29:0/3223699356 >> v1:172.21.15.29:6789/0 conn(0x7fb1680cb6f0 legacy=0x7fb1680cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16e79c700 1 -- 172.21.15.29:0/3223699356 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1680cc010 con 0x7fb1680cf2c0 2023-10-07T09:41:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16e79c700 1 --2- 172.21.15.29:0/3223699356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cf2c0 0x7fb1680c5400 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb160009f20 tx=0x7fb160009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d73fc563db870d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16cf99700 1 -- 172.21.15.29:0/3223699356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16000e070 con 0x7fb1680cf2c0 2023-10-07T09:41:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16cf99700 1 -- 172.21.15.29:0/3223699356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16000ba80 con 0x7fb1680cf2c0 2023-10-07T09:41:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.529+0000 7fb16cf99700 1 -- 172.21.15.29:0/3223699356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1600092e0 con 0x7fb1680cf2c0 2023-10-07T09:41:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 -- 172.21.15.29:0/3223699356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cf2c0 msgr2=0x7fb1680c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 --2- 172.21.15.29:0/3223699356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cf2c0 0x7fb1680c5400 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb160009f20 tx=0x7fb160009840 comp rx=0 tx=0).stop 2023-10-07T09:41:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 -- 172.21.15.29:0/3223699356 shutdown_connections 2023-10-07T09:41:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 --2- 172.21.15.29:0/3223699356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cf2c0 0x7fb1680c5400 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 --2- 172.21.15.29:0/3223699356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680b2a20 0x7fb1680cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 -- 172.21.15.29:0/3223699356 >> 172.21.15.29:0/3223699356 conn(0x7fb168099820 msgr2=0x7fb168063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 -- 172.21.15.29:0/3223699356 shutdown_connections 2023-10-07T09:41:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.530+0000 7fb1701ff700 1 -- 172.21.15.29:0/3223699356 wait complete. 2023-10-07T09:41:29.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.532+0000 7fb1701ff700 1 Processor -- start 2023-10-07T09:41:29.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.532+0000 7fb1701ff700 1 -- start start 2023-10-07T09:41:29.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb1701ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680b2a20 0x7fb16816d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb1701ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 0x7fb16816d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb1701ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680cf2c0 0x7fb168156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb1701ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb168157b80 con 0x7fb1680cb6f0 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb1701ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb168156f70 con 0x7fb1680b2a20 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb1701ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb168157250 con 0x7fb1680cf2c0 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 0x7fb16816d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 0x7fb16816d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49338/0 (socket says 172.21.15.29:49338) 2023-10-07T09:41:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 -- 172.21.15.29:0/464331683 learned_addr learned my addr 172.21.15.29:0/464331683 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16e79c700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680cf2c0 0x7fb168156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16df9b700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680b2a20 0x7fb16816d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 -- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680cf2c0 msgr2=0x7fb168156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680cf2c0 0x7fb168156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 -- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680b2a20 msgr2=0x7fb16816d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680b2a20 0x7fb16816d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.533+0000 7fb16d79a700 1 -- 172.21.15.29:0/464331683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160009ae0 con 0x7fb1680cb6f0 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.534+0000 7fb16d79a700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 0x7fb16816d860 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb15800c7c0 tx=0x7fb158007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.534+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158005a70 con 0x7fb1680cb6f0 2023-10-07T09:41:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.534+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb158005bd0 con 0x7fb1680cb6f0 2023-10-07T09:41:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.534+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb168157530 con 0x7fb1680cb6f0 2023-10-07T09:41:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.534+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1580070d0 con 0x7fb1680cb6f0 2023-10-07T09:41:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.534+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb168158010 con 0x7fb1680cb6f0 2023-10-07T09:41:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.535+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb168158230 con 0x7fb1680cb6f0 2023-10-07T09:41:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.536+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fb158007480 con 0x7fb1680cb6f0 2023-10-07T09:41:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.537+0000 7fb15dffb700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb14c067130 0x7fb14c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.537+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fb158083e60 con 0x7fb1680cb6f0 2023-10-07T09:41:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.537+0000 7fb16df9b700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb14c067130 0x7fb14c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.538+0000 7fb16df9b700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb14c067130 0x7fb14c0695e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb164006aa0 tx=0x7fb164008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.541+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb158050cb0 con 0x7fb1680cb6f0 2023-10-07T09:41:29.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.766+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1680718b0 con 0x7fb14c067130 2023-10-07T09:41:29.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.776+0000 7fb15dffb700 1 -- 172.21.15.29:0/464331683 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fb1680718b0 con 0x7fb14c067130 2023-10-07T09:41:29.777 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:51", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb14c067130 msgr2=0x7fb14c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb14c067130 0x7fb14c0695e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb164006aa0 tx=0x7fb164008040 comp rx=0 tx=0).stop 2023-10-07T09:41:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 msgr2=0x7fb16816d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 0x7fb16816d860 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb15800c7c0 tx=0x7fb158007f70 comp rx=0 tx=0).stop 2023-10-07T09:41:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 shutdown_connections 2023-10-07T09:41:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb14c067130 0x7fb14c0695e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680cf2c0 0x7fb168156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680b2a20 0x7fb16816d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 --2- 172.21.15.29:0/464331683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1680cb6f0 0x7fb16816d860 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 >> 172.21.15.29:0/464331683 conn(0x7fb168099820 msgr2=0x7fb1680ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 shutdown_connections 2023-10-07T09:41:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:29.779+0000 7fb1701ff700 1 -- 172.21.15.29:0/464331683 wait complete. 2023-10-07T09:41:29.797 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:51", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:29.797 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:51', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:29.797 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:29.798 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:29.798 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:29.798 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_01_UTC 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:29.846 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:29.847 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:29.847 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:29.847 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:29.847 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:29.848 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_01_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:29.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir7 2023-10-07T09:41:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.047+0000 7f6e949ff700 1 Processor -- start 2023-10-07T09:41:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.047+0000 7f6e949ff700 1 -- start start 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.048+0000 7f6e949ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6e9015a930 0x7f6e900b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.048+0000 7f6e949ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e9015ad00 0x7f6e900cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.048+0000 7f6e949ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e900c9ac0 con 0x7f6e900cc1c0 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.048+0000 7f6e949ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e900cdc20 con 0x7f6e9015ad00 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.048+0000 7f6e949ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e900c49e0 con 0x7f6e9015a930 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e9015ad00 0x7f6e900cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8ed9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6e900cc1c0 0x7f6e900c9f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51836/0 (socket says 172.21.15.29:51836) 2023-10-07T09:41:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8ed9d700 1 -- 172.21.15.29:0/1406818437 learned_addr learned my addr 172.21.15.29:0/1406818437 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8e59c700 1 --2- 172.21.15.29:0/1406818437 >> v2:172.21.15.144:3300/0 conn(0x7f6e9015a930 0x7f6e900b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8dd9b700 1 -- 172.21.15.29:0/1406818437 >> v2:172.21.15.144:3300/0 conn(0x7f6e9015a930 msgr2=0x7f6e900b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8dd9b700 1 --2- 172.21.15.29:0/1406818437 >> v2:172.21.15.144:3300/0 conn(0x7f6e9015a930 0x7f6e900b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8dd9b700 1 -- 172.21.15.29:0/1406818437 >> v1:172.21.15.29:6789/0 conn(0x7f6e900cc1c0 legacy=0x7f6e900c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8dd9b700 1 -- 172.21.15.29:0/1406818437 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900c9470 con 0x7f6e9015ad00 2023-10-07T09:41:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8d59a700 1 -- 172.21.15.29:0/1406818437 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831870998 0 0) 0x7f6e900c9ac0 con 0x7f6e900cc1c0 2023-10-07T09:41:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.049+0000 7f6e8dd9b700 1 --2- 172.21.15.29:0/1406818437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900cd140 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6e90065a30 tx=0x7f6e840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1d0a40a6a75b07 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e8d59a700 1 -- 172.21.15.29:0/1406818437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e84011630 con 0x7f6e9015ad00 2023-10-07T09:41:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e8d59a700 1 -- 172.21.15.29:0/1406818437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e84011790 con 0x7f6e9015ad00 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e8d59a700 1 -- 172.21.15.29:0/1406818437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e84011b30 con 0x7f6e9015ad00 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 -- 172.21.15.29:0/1406818437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 msgr2=0x7f6e900cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 --2- 172.21.15.29:0/1406818437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900cd140 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6e90065a30 tx=0x7f6e840091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 -- 172.21.15.29:0/1406818437 shutdown_connections 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 --2- 172.21.15.29:0/1406818437 >> v2:172.21.15.144:3300/0 conn(0x7f6e9015a930 0x7f6e900b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 --2- 172.21.15.29:0/1406818437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900cd140 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 -- 172.21.15.29:0/1406818437 >> 172.21.15.29:0/1406818437 conn(0x7f6e90063a10 msgr2=0x7f6e900ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.050+0000 7f6e949ff700 1 -- 172.21.15.29:0/1406818437 shutdown_connections 2023-10-07T09:41:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.051+0000 7f6e949ff700 1 -- 172.21.15.29:0/1406818437 wait complete. 2023-10-07T09:41:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.052+0000 7f6e949ff700 1 Processor -- start 2023-10-07T09:41:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.052+0000 7f6e949ff700 1 -- start start 2023-10-07T09:41:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e949ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e900cc1c0 0x7f6e90155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e949ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e9015a930 0x7f6e900c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e949ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e949ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90158cb0 con 0x7f6e9015ad00 2023-10-07T09:41:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e949ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90068e10 con 0x7f6e900cc1c0 2023-10-07T09:41:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e949ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900c81a0 con 0x7f6e9015a930 2023-10-07T09:41:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e8ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e8ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900c36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49368/0 (socket says 172.21.15.29:49368) 2023-10-07T09:41:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.053+0000 7f6e8ed9d700 1 -- 172.21.15.29:0/2921772229 learned_addr learned my addr 172.21.15.29:0/2921772229 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8dd9b700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e9015a930 0x7f6e900c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8e59c700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e900cc1c0 0x7f6e90155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8ed9d700 1 -- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e9015a930 msgr2=0x7f6e900c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8ed9d700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e9015a930 0x7f6e900c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8ed9d700 1 -- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e900cc1c0 msgr2=0x7f6e90155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8ed9d700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e900cc1c0 0x7f6e90155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8ed9d700 1 -- 172.21.15.29:0/2921772229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84009420 con 0x7f6e9015ad00 2023-10-07T09:41:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e8ed9d700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900c36c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800cc90 tx=0x7f6e88001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800cea0 con 0x7f6e9015ad00 2023-10-07T09:41:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8800ab70 con 0x7f6e9015ad00 2023-10-07T09:41:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880070f0 con 0x7f6e9015ad00 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e901589d0 con 0x7f6e9015ad00 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.054+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e90068aa0 con 0x7f6e9015ad00 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.055+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e900718b0 con 0x7f6e9015ad00 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.056+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f6e880074a0 con 0x7f6e9015ad00 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.057+0000 7f6e7e7fc700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6e70067130 0x7f6e700695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.058+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f6e88084f20 con 0x7f6e9015ad00 2023-10-07T09:41:30.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.058+0000 7f6e8e59c700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6e70067130 0x7f6e700695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.059+0000 7f6e8e59c700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6e70067130 0x7f6e700695e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009d40 tx=0x7f6e80008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:30.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.061+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e88051df0 con 0x7f6e9015ad00 2023-10-07T09:41:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.289+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e90068f50 con 0x7f6e70067130 2023-10-07T09:41:30.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.298+0000 7f6e7e7fc700 1 -- 172.21.15.29:0/2921772229 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f6e90068f50 con 0x7f6e70067130 2023-10-07T09:41:30.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:30.299 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:52", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.300+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6e70067130 msgr2=0x7f6e700695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.300+0000 7f6e949ff700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6e70067130 0x7f6e700695e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009d40 tx=0x7f6e80008040 comp rx=0 tx=0).stop 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.300+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 msgr2=0x7f6e900c36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.300+0000 7f6e949ff700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900c36c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800cc90 tx=0x7f6e88001a80 comp rx=0 tx=0).stop 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 shutdown_connections 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f6e70067130 0x7f6e700695e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e9015ad00 0x7f6e900c36c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e900cc1c0 0x7f6e90155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 --2- 172.21.15.29:0/2921772229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e9015a930 0x7f6e900c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 >> 172.21.15.29:0/2921772229 conn(0x7f6e90063a10 msgr2=0x7f6e900c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 shutdown_connections 2023-10-07T09:41:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.301+0000 7f6e949ff700 1 -- 172.21.15.29:0/2921772229 wait complete. 2023-10-07T09:41:30.321 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:52", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:30.321 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:52', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:30.321 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:30.321 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:30.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:30.322 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-07T09:41:30.372 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:30.372 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:30.372 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:30.372 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:30.372 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:30.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:30.375 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_00_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:30.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir8 2023-10-07T09:41:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.575+0000 7f65f6fff700 1 Processor -- start 2023-10-07T09:41:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.576+0000 7f65f6fff700 1 -- start start 2023-10-07T09:41:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f6fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65f00b2a20 0x7f65f00cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00cb6f0 0x7f65f00cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f6fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65f00cf0f0 0x7f65f0159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f6fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f00cc940 con 0x7f65f00b2a20 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f6fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f00cf530 con 0x7f65f00cf0f0 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00b2df0 con 0x7f65f00cb6f0 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f4d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65f00b2a20 0x7f65f00cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f4d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65f00b2a20 0x7f65f00cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49380/0 (socket says 172.21.15.29:49380) 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f4d9b700 1 -- 172.21.15.29:0/1021021964 learned_addr learned my addr 172.21.15.29:0/1021021964 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65f559c700 1 --2- 172.21.15.29:0/1021021964 >> v2:172.21.15.144:3300/0 conn(0x7f65f00cf0f0 0x7f65f0159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.577+0000 7f65effff700 1 --2- 172.21.15.29:0/1021021964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00cb6f0 0x7f65f00cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65f4d9b700 1 -- 172.21.15.29:0/1021021964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00cb6f0 msgr2=0x7f65f00cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65f4d9b700 1 --2- 172.21.15.29:0/1021021964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00cb6f0 0x7f65f00cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65f4d9b700 1 -- 172.21.15.29:0/1021021964 >> v2:172.21.15.144:3300/0 conn(0x7f65f00cf0f0 msgr2=0x7f65f0159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65f4d9b700 1 --2- 172.21.15.29:0/1021021964 >> v2:172.21.15.144:3300/0 conn(0x7f65f00cf0f0 0x7f65f0159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65f4d9b700 1 -- 172.21.15.29:0/1021021964 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00cbd30 con 0x7f65f00b2a20 2023-10-07T09:41:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65f4d9b700 1 --2- 172.21.15.29:0/1021021964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00b2a20 0x7f65f00cb1b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f65f015ae20 tx=0x7f65e4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a02d27e368852df server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65ef7fe700 1 -- 172.21.15.29:0/1021021964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4005c10 con 0x7f65f00b2a20 2023-10-07T09:41:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65ef7fe700 1 -- 172.21.15.29:0/1021021964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4005d70 con 0x7f65f00b2a20 2023-10-07T09:41:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.578+0000 7f65ef7fe700 1 -- 172.21.15.29:0/1021021964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e400b880 con 0x7f65f00b2a20 2023-10-07T09:41:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 -- 172.21.15.29:0/1021021964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00b2a20 msgr2=0x7f65f00cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 --2- 172.21.15.29:0/1021021964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00b2a20 0x7f65f00cb1b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f65f015ae20 tx=0x7f65e4009220 comp rx=0 tx=0).stop 2023-10-07T09:41:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 -- 172.21.15.29:0/1021021964 shutdown_connections 2023-10-07T09:41:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 --2- 172.21.15.29:0/1021021964 >> v2:172.21.15.144:3300/0 conn(0x7f65f00cf0f0 0x7f65f0159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 --2- 172.21.15.29:0/1021021964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00b2a20 0x7f65f00cb1b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 --2- 172.21.15.29:0/1021021964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00cb6f0 0x7f65f00cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 -- 172.21.15.29:0/1021021964 >> 172.21.15.29:0/1021021964 conn(0x7f65f0099820 msgr2=0x7f65f0063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 -- 172.21.15.29:0/1021021964 shutdown_connections 2023-10-07T09:41:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.579+0000 7f65f6fff700 1 -- 172.21.15.29:0/1021021964 wait complete. 2023-10-07T09:41:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.580+0000 7f65f6fff700 1 Processor -- start 2023-10-07T09:41:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.581+0000 7f65f6fff700 1 -- start start 2023-10-07T09:41:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.581+0000 7f65f6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00b2a20 0x7f65f014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.581+0000 7f65f6fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 0x7f65f014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f6fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f00cf0f0 0x7f65f016d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f6fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f014e730 con 0x7f65f00cb6f0 2023-10-07T09:41:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f6fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0156a20 con 0x7f65f00cf0f0 2023-10-07T09:41:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00b4060 con 0x7f65f00b2a20 2023-10-07T09:41:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 0x7f65f014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 0x7f65f014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49382/0 (socket says 172.21.15.29:49382) 2023-10-07T09:41:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 -- 172.21.15.29:0/3837957787 learned_addr learned my addr 172.21.15.29:0/3837957787 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f4d9b700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00b2a20 0x7f65f014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f559c700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f00cf0f0 0x7f65f016d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 -- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00b2a20 msgr2=0x7f65f014dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00b2a20 0x7f65f014dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 -- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f00cf0f0 msgr2=0x7f65f016d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f00cf0f0 0x7f65f016d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 -- 172.21.15.29:0/3837957787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e4009480 con 0x7f65f00cb6f0 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65f559c700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f00cf0f0 0x7f65f016d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.582+0000 7f65effff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 0x7f65f014e1f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00d890 tx=0x7f65dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.583+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00b600 con 0x7f65f00cb6f0 2023-10-07T09:41:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.583+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc00b760 con 0x7f65f00cb6f0 2023-10-07T09:41:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.583+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00bae0 con 0x7f65f00cb6f0 2023-10-07T09:41:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.583+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f00b4340 con 0x7f65f00cb6f0 2023-10-07T09:41:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.583+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f00b3d30 con 0x7f65f00cb6f0 2023-10-07T09:41:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.584+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00718b0 con 0x7f65f00cb6f0 2023-10-07T09:41:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.585+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f65dc01e460 con 0x7f65f00cb6f0 2023-10-07T09:41:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.586+0000 7f65ecff9700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f65d4067130 0x7f65d40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.586+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f65dc085030 con 0x7f65f00cb6f0 2023-10-07T09:41:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.586+0000 7f65f4d9b700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f65d4067130 0x7f65d40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.587+0000 7f65f4d9b700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f65d4067130 0x7f65d40695e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009260 tx=0x7f65e4016000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.590+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65dc051f20 con 0x7f65f00cb6f0 2023-10-07T09:41:30.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.819+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65f00cce70 con 0x7f65d4067130 2023-10-07T09:41:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.829+0000 7f65ecff9700 1 -- 172.21.15.29:0/3837957787 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f65f00cce70 con 0x7f65d4067130 2023-10-07T09:41:30.831 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:30.831 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:52", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:30.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f65d4067130 msgr2=0x7f65d40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f65d4067130 0x7f65d40695e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009260 tx=0x7f65e4016000 comp rx=0 tx=0).stop 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 msgr2=0x7f65f014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 0x7f65f014e1f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00d890 tx=0x7f65dc008040 comp rx=0 tx=0).stop 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 shutdown_connections 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f65d4067130 0x7f65d40695e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f00cf0f0 0x7f65f016d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65f00b2a20 0x7f65f014dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 --2- 172.21.15.29:0/3837957787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00cb6f0 0x7f65f014e1f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 >> 172.21.15.29:0/3837957787 conn(0x7f65f0099820 msgr2=0x7f65f015fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 shutdown_connections 2023-10-07T09:41:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:30.832+0000 7f65f6fff700 1 -- 172.21.15.29:0/3837957787 wait complete. 2023-10-07T09:41:30.851 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:52", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:30.851 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:52', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:30.851 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:30.851 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:30.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:30.851 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_01_UTC 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:30.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:30.901 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:30.901 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:30.901 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:30.901 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:30.903 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_31_00_UTC', 'scheduled-2023-10-07-09_32_00_UTC', 'scheduled-2023-10-07-09_33_00_UTC', 'scheduled-2023-10-07-09_34_01_UTC', 'scheduled-2023-10-07-09_35_00_UTC', 'scheduled-2023-10-07-09_36_00_UTC', 'scheduled-2023-10-07-09_37_00_UTC', 'scheduled-2023-10-07-09_38_00_UTC', 'scheduled-2023-10-07-09_39_00_UTC', 'scheduled-2023-10-07-09_40_00_UTC', 'scheduled-2023-10-07-09_41_00_UTC'] 2023-10-07T09:41:30.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir9 2023-10-07T09:41:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.102+0000 7fede6bff700 1 Processor -- start 2023-10-07T09:41:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.102+0000 7fede6bff700 1 -- start start 2023-10-07T09:41:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.103+0000 7fede6bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fede00cb8c0 0x7fede00cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.103+0000 7fede6bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede00cc940 con 0x7fede00b2a20 2023-10-07T09:41:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.103+0000 7fede6bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede00cf530 con 0x7fede00cf0f0 2023-10-07T09:41:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.103+0000 7fede6bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede00b2df0 con 0x7fede00cb8c0 2023-10-07T09:41:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.103+0000 7fede499b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fede00b2a20 0x7fede00cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51844/0 (socket says 172.21.15.29:51844) 2023-10-07T09:41:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.103+0000 7fede499b700 1 -- 172.21.15.29:0/2447445719 learned_addr learned my addr 172.21.15.29:0/2447445719 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddffff700 1 --2- 172.21.15.29:0/2447445719 >> v2:172.21.15.144:3300/0 conn(0x7fede00cb8c0 0x7fede00cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066491548 0 0) 0x7fede00cc940 con 0x7fede00b2a20 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd0003570 con 0x7fede00b2a20 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959395716 0 0) 0x7fede00cf530 con 0x7fede00cf0f0 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede00cc940 con 0x7fede00cf0f0 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699049078 0 0) 0x7fedd0003570 con 0x7fede00b2a20 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 >> v2:172.21.15.144:3300/0 conn(0x7fede00cb8c0 msgr2=0x7fede00cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 --2- 172.21.15.29:0/2447445719 >> v2:172.21.15.144:3300/0 conn(0x7fede00cb8c0 0x7fede00cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 >> v1:172.21.15.144:6789/0 conn(0x7fede00cf0f0 legacy=0x7fede00c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00cbd30 con 0x7fede00b2a20 2023-10-07T09:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fedd40015f0 con 0x7fede00b2a20 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 >> v1:172.21.15.29:6789/0 conn(0x7fede00b2a20 legacy=0x7fede00cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd0003c40 0x7fedd0004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fedd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.104+0000 7feddf7fe700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd0005210 0x7fedd000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede00b2df0 con 0x7fedd0004810 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd000c130 con 0x7fedd0003c40 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd000c2c0 con 0x7fedd0005210 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fedd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7fede499b700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd0003c40 0x7fedd0004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7fede519c700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd0005210 0x7fedd000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 -- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd0005210 msgr2=0x7fedd000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd0005210 0x7fedd000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 -- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd0003c40 msgr2=0x7fedd0004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd0003c40 0x7fedd0004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 -- 172.21.15.29:0/2447445719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0003570 con 0x7fedd0004810 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7fede519c700 1 -- 172.21.15.29:0/2447445719 reap_dead start 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.105+0000 7feddffff700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fedd0004c80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fede015ae40 tx=0x7fedcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78e937cc4b306bb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc00e710 con 0x7fedd0004810 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc00e870 con 0x7fedd0004810 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7feddf7fe700 1 -- 172.21.15.29:0/2447445719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc00ec30 con 0x7fedd0004810 2023-10-07T09:41:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 -- 172.21.15.29:0/2447445719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 msgr2=0x7fedd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fedd0004c80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fede015ae40 tx=0x7fedcc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 -- 172.21.15.29:0/2447445719 shutdown_connections 2023-10-07T09:41:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 --2- 172.21.15.29:0/2447445719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fedd0004c80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 -- 172.21.15.29:0/2447445719 >> 172.21.15.29:0/2447445719 conn(0x7fede0099820 msgr2=0x7fede0063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 -- 172.21.15.29:0/2447445719 shutdown_connections 2023-10-07T09:41:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.106+0000 7fede6bff700 1 -- 172.21.15.29:0/2447445719 wait complete. 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.108+0000 7fede6bff700 1 Processor -- start 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.108+0000 7fede6bff700 1 -- start start 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede6bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fede01698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fede009a4f0 0x7fede009a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede6bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede00b4340 0x7fede00b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede6bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede009aea0 con 0x7fedd0004810 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede6bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede00b5c40 con 0x7fede00b4340 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede00b5030 con 0x7fede009a4f0 2023-10-07T09:41:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede499b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fede01698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede499b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fede01698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49392/0 (socket says 172.21.15.29:49392) 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede499b700 1 -- 172.21.15.29:0/3277527818 learned_addr learned my addr 172.21.15.29:0/3277527818 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7feddffff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fede009a4f0 0x7fede009a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede519c700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede00b4340 0x7fede00b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.109+0000 7fede499b700 1 -- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fede009a4f0 msgr2=0x7fede009a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede499b700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fede009a4f0 0x7fede009a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede499b700 1 -- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede00b4340 msgr2=0x7fede00b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede499b700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede00b4340 0x7fede00b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede499b700 1 -- 172.21.15.29:0/3277527818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00b5310 con 0x7fedd0004810 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede499b700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fede01698c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fedd40012a0 tx=0x7fedd4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd40072e0 con 0x7fedd0004810 2023-10-07T09:41:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd4007440 con 0x7fedd0004810 2023-10-07T09:41:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd40057f0 con 0x7fedd0004810 2023-10-07T09:41:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede00b55f0 con 0x7fedd0004810 2023-10-07T09:41:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.110+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede0169e00 con 0x7fedd0004810 2023-10-07T09:41:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.111+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede00718b0 con 0x7fedd0004810 2023-10-07T09:41:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.112+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fedd4005bc0 con 0x7fedd0004810 2023-10-07T09:41:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.113+0000 7feddcff9700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fedc4067130 0x7fedc40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.113+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fedd4084b90 con 0x7fedd0004810 2023-10-07T09:41:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.114+0000 7feddffff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fedc4067130 0x7fedc40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.114+0000 7feddffff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fedc4067130 0x7fedc40695e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0091e0 tx=0x7fedcc01a000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.117+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fedd40519e0 con 0x7fedd0004810 2023-10-07T09:41:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.344+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fede00cc010 con 0x7fedc4067130 2023-10-07T09:41:31.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.354+0000 7feddcff9700 1 -- 172.21.15.29:0/3277527818 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fede00cc010 con 0x7fedc4067130 2023-10-07T09:41:31.355 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:31.355 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:53", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:31.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fedc4067130 msgr2=0x7fedc40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fedc4067130 0x7fedc40695e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0091e0 tx=0x7fedcc01a000 comp rx=0 tx=0).stop 2023-10-07T09:41:31.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 msgr2=0x7fede01698c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fede01698c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fedd40012a0 tx=0x7fedd4009040 comp rx=0 tx=0).stop 2023-10-07T09:41:31.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 shutdown_connections 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fedc4067130 0x7fedc40695e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede00b4340 0x7fede00b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd0004810 0x7fede01698c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 --2- 172.21.15.29:0/3277527818 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fede009a4f0 0x7fede009a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 >> 172.21.15.29:0/3277527818 conn(0x7fede0099820 msgr2=0x7fede0063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 shutdown_connections 2023-10-07T09:41:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.357+0000 7fede6bff700 1 -- 172.21.15.29:0/3277527818 wait complete. 2023-10-07T09:41:31.375 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:30:53", "first": "2023-10-07T09:31:00", "last": "2023-10-07T09:41:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-07T09:41:31.375 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:30:53', 'first': '2023-10-07T09:31:00', 'last': '2023-10-07T09:41:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-07T09:41:31.375 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T09:41:31.375 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T09:41:31.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir0 1M 2023-10-07T09:41:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.574+0000 7fc88c7ff700 1 Processor -- start 2023-10-07T09:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.575+0000 7fc88c7ff700 1 -- start start 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc8840cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88c7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8840cf0f0 0x7fc8840c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88c7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840cc940 con 0x7fc8840b2a20 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88c7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8840cf530 con 0x7fc8840cf0f0 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840b2df0 con 0x7fc8840cb8c0 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88a59b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc8840b2a20 0x7fc8840cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51850/0 (socket says 172.21.15.29:51850) 2023-10-07T09:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88a59b700 1 -- 172.21.15.29:0/3878731697 learned_addr learned my addr 172.21.15.29:0/3878731697 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:31.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc889d9a700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc8840cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc88ad9c700 1 --2- 172.21.15.29:0/3878731697 >> v2:172.21.15.144:3300/0 conn(0x7fc8840cf0f0 0x7fc8840c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.576+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859022831 0 0) 0x7fc8840cc940 con 0x7fc8840b2a20 2023-10-07T09:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc870003570 con 0x7fc8840b2a20 2023-10-07T09:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1512548767 0 0) 0x7fc870003570 con 0x7fc8840b2a20 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 msgr2=0x7fc8840cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc8840cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 >> v2:172.21.15.144:3300/0 conn(0x7fc8840cf0f0 msgr2=0x7fc8840c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 --2- 172.21.15.29:0/3878731697 >> v2:172.21.15.144:3300/0 conn(0x7fc8840cf0f0 0x7fc8840c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840cbd30 con 0x7fc8840b2a20 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fc8800015f0 con 0x7fc8840b2a20 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 >> v1:172.21.15.29:6789/0 conn(0x7fc8840b2a20 legacy=0x7fc8840cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc870003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc870003fe0 0x7fc870004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc870004d50 0x7fc8700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840cf530 con 0x7fc870003fe0 2023-10-07T09:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840b2df0 con 0x7fc870004d50 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87000ba90 con 0x7fc8840cb8c0 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc889d9a700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc870003fe0 0x7fc870004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc870003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.577+0000 7fc88ad9c700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc870004d50 0x7fc8700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc88a59b700 1 -- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc870004d50 msgr2=0x7fc8700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc870004d50 0x7fc8700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc88a59b700 1 -- 172.21.15.29:0/3878731697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc870003fe0 msgr2=0x7fc870004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc870003fe0 0x7fc870004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc88a59b700 1 -- 172.21.15.29:0/3878731697 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870003570 con 0x7fc8840cb8c0 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc889d9a700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc870003fe0 0x7fc870004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc870003a60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc88000c3c0 tx=0x7fc880007920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bfd9acabd86b92c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88000cc30 con 0x7fc8840cb8c0 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88000cd90 con 0x7fc8840cb8c0 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.578+0000 7fc889599700 1 -- 172.21.15.29:0/3878731697 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc880007320 con 0x7fc8840cb8c0 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 msgr2=0x7fc870003a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3878731697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8840cb8c0 0x7fc870003a60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc88000c3c0 tx=0x7fc880007920 comp rx=0 tx=0).stop 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88ad9c700 1 -- 172.21.15.29:0/3878731697 reap_dead start 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3878731697 shutdown_connections 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3878731697 >> 172.21.15.29:0/3878731697 conn(0x7fc884099820 msgr2=0x7fc884063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3878731697 shutdown_connections 2023-10-07T09:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.579+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3878731697 wait complete. 2023-10-07T09:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.580+0000 7fc88c7ff700 1 Processor -- start 2023-10-07T09:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.581+0000 7fc88c7ff700 1 -- start start 2023-10-07T09:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.581+0000 7fc88c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8840cb8c0 0x7fc884157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.581+0000 7fc88c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc884157890 0x7fc8840b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 0x7fc88416da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88414f890 con 0x7fc8840b38b0 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc884156ec0 con 0x7fc8840cb8c0 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8841562b0 con 0x7fc884157890 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 0x7fc88416da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 0x7fc88416da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49418/0 (socket says 172.21.15.29:49418) 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 -- 172.21.15.29:0/3430286673 learned_addr learned my addr 172.21.15.29:0/3430286673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc889d9a700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc884157890 0x7fc8840b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8840cb8c0 0x7fc884157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 -- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc884157890 msgr2=0x7fc8840b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc884157890 0x7fc8840b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 -- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8840cb8c0 msgr2=0x7fc884157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8840cb8c0 0x7fc884157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.582+0000 7fc88ad9c700 1 -- 172.21.15.29:0/3430286673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884156590 con 0x7fc8840b38b0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.583+0000 7fc88ad9c700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 0x7fc88416da90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc88415d250 tx=0x7fc87c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.583+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00e040 con 0x7fc8840b38b0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.583+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c00f040 con 0x7fc8840b38b0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.583+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc884156870 con 0x7fc8840b38b0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.583+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00fe20 con 0x7fc8840b38b0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.583+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88416dfd0 con 0x7fc8840b38b0 2023-10-07T09:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.584+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc884156d10 con 0x7fc8840b38b0 2023-10-07T09:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.585+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fc87c015640 con 0x7fc8840b38b0 2023-10-07T09:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.586+0000 7fc87a7fc700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc868067130 0x7fc8680695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.586+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fc87c085360 con 0x7fc8840b38b0 2023-10-07T09:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.586+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc868067130 0x7fc8680695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:31.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.587+0000 7fc88a59b700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc868067130 0x7fc8680695e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc880008fd0 tx=0x7fc880015000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:31.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.590+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc87c0520a0 con 0x7fc8840b38b0 2023-10-07T09:41:31.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.816+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc884157c60 con 0x7fc868067130 2023-10-07T09:41:31.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.898+0000 7fc87a7fc700 1 -- 172.21.15.29:0/3430286673 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc884157c60 con 0x7fc868067130 2023-10-07T09:41:31.899 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc868067130 msgr2=0x7fc8680695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc868067130 0x7fc8680695e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc880008fd0 tx=0x7fc880015000 comp rx=0 tx=0).stop 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 msgr2=0x7fc88416da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 0x7fc88416da90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc88415d250 tx=0x7fc87c00b040 comp rx=0 tx=0).stop 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 shutdown_connections 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fc868067130 0x7fc8680695e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8840b38b0 0x7fc88416da90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8840cb8c0 0x7fc884157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 --2- 172.21.15.29:0/3430286673 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc884157890 0x7fc8840b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.900+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 >> 172.21.15.29:0/3430286673 conn(0x7fc884099820 msgr2=0x7fc8840d05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.901+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 shutdown_connections 2023-10-07T09:41:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:31.901+0000 7fc88c7ff700 1 -- 172.21.15.29:0/3430286673 wait complete. 2023-10-07T09:41:31.919 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-07T09:41:31.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:31.919 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:31.967 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:31.968 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:31.968 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:31.969 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:31.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:31.969 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:32.025 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:32.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.025 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:32.081 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:32.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.081 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:32.137 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:32.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.138 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:32.193 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:32.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.194 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:32.249 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:32.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.249 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:32.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:32.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.305 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:32.361 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:32.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.361 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:32.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:32.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.414 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:32.467 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:32.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.468 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:32.522 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:32.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.522 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:32.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:32.577 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-07T09:41:32.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir1 1M 2023-10-07T09:41:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.829+0000 7eff181ff700 1 Processor -- start 2023-10-07T09:41:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.829+0000 7eff181ff700 1 -- start start 2023-10-07T09:41:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.830+0000 7eff181ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7eff10157650 0x7eff100ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.830+0000 7eff181ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff100ae910 0x7eff100b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.830+0000 7eff181ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff100ca430 con 0x7eff10157a20 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.830+0000 7eff181ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff100ca710 con 0x7eff10157650 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.830+0000 7eff181ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff100ca9f0 con 0x7eff100ae910 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7eff10157650 0x7eff100ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7eff10157650 0x7eff100ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49434/0 (socket says 172.21.15.29:49434) 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 -- 172.21.15.29:0/4294664068 learned_addr learned my addr 172.21.15.29:0/4294664068 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff1579a700 1 --2- 172.21.15.29:0/4294664068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff100ae910 0x7eff100b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 -- 172.21.15.29:0/4294664068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff100ae910 msgr2=0x7eff100b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 --2- 172.21.15.29:0/4294664068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff100ae910 0x7eff100b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 -- 172.21.15.29:0/4294664068 >> v1:172.21.15.144:6789/0 conn(0x7eff10157a20 legacy=0x7eff100cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 -- 172.21.15.29:0/4294664068 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff100cacd0 con 0x7eff10157650 2023-10-07T09:41:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.831+0000 7eff15f9b700 1 --2- 172.21.15.29:0/4294664068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157650 0x7eff100ae3d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0099b0 tx=0x7eff0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8508336126abd10a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff14f99700 1 -- 172.21.15.29:0/4294664068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00b860 con 0x7eff10157650 2023-10-07T09:41:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff14f99700 1 -- 172.21.15.29:0/4294664068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c00b9c0 con 0x7eff10157650 2023-10-07T09:41:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff14f99700 1 -- 172.21.15.29:0/4294664068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00bd80 con 0x7eff10157650 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 -- 172.21.15.29:0/4294664068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157650 msgr2=0x7eff100ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 --2- 172.21.15.29:0/4294664068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157650 0x7eff100ae3d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0099b0 tx=0x7eff0c009250 comp rx=0 tx=0).stop 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 -- 172.21.15.29:0/4294664068 shutdown_connections 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 --2- 172.21.15.29:0/4294664068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157650 0x7eff100ae3d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 --2- 172.21.15.29:0/4294664068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff100ae910 0x7eff100b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 -- 172.21.15.29:0/4294664068 >> 172.21.15.29:0/4294664068 conn(0x7eff10067610 msgr2=0x7eff10067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.832+0000 7eff181ff700 1 -- 172.21.15.29:0/4294664068 shutdown_connections 2023-10-07T09:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.833+0000 7eff181ff700 1 -- 172.21.15.29:0/4294664068 wait complete. 2023-10-07T09:41:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.834+0000 7eff181ff700 1 Processor -- start 2023-10-07T09:41:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.834+0000 7eff181ff700 1 -- start start 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff181ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff10157650 0x7eff101520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff181ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 0x7eff101536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff181ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff10152950 0x7eff10152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff181ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff100af940 con 0x7eff10157a20 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff181ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff100c7ff0 con 0x7eff10157650 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff181ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff10153330 con 0x7eff10152950 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff1579a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 0x7eff101536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff1579a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 0x7eff101536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49450/0 (socket says 172.21.15.29:49450) 2023-10-07T09:41:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff1579a700 1 -- 172.21.15.29:0/1714015793 learned_addr learned my addr 172.21.15.29:0/1714015793 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff15f9b700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff10157650 0x7eff101520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.835+0000 7eff1679c700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff10152950 0x7eff10152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff1579a700 1 -- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff10152950 msgr2=0x7eff10152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff1579a700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff10152950 0x7eff10152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff1579a700 1 -- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff10157650 msgr2=0x7eff101520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff1579a700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff10157650 0x7eff101520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff1579a700 1 -- 172.21.15.29:0/1714015793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c0094f0 con 0x7eff10157a20 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff15f9b700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff10157650 0x7eff101520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff1579a700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 0x7eff101536f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7eff0000d860 tx=0x7eff00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff000125f0 con 0x7eff10157a20 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff00012750 con 0x7eff10157a20 2023-10-07T09:41:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00012ad0 con 0x7eff10157a20 2023-10-07T09:41:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff101569a0 con 0x7eff10157a20 2023-10-07T09:41:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.836+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff100afb40 con 0x7eff10157a20 2023-10-07T09:41:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.837+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff10156e90 con 0x7eff10157a20 2023-10-07T09:41:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.838+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7eff0001e460 con 0x7eff10157a20 2023-10-07T09:41:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.839+0000 7eff05ffb700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efef4067130 0x7efef40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.839+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7eff00085090 con 0x7eff10157a20 2023-10-07T09:41:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.840+0000 7eff15f9b700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efef4067130 0x7efef40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.840+0000 7eff15f9b700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efef4067130 0x7efef40695e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0099b0 tx=0x7eff0c017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:32.843+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff00055970 con 0x7eff10157a20 2023-10-07T09:41:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.071+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7eff100cafb0 con 0x7efef4067130 2023-10-07T09:41:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.153+0000 7eff05ffb700 1 -- 172.21.15.29:0/1714015793 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7eff100cafb0 con 0x7efef4067130 2023-10-07T09:41:33.154 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-07T09:41:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.155+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efef4067130 msgr2=0x7efef40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efef4067130 0x7efef40695e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0099b0 tx=0x7eff0c017000 comp rx=0 tx=0).stop 2023-10-07T09:41:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 msgr2=0x7eff101536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 0x7eff101536f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7eff0000d860 tx=0x7eff00008040 comp rx=0 tx=0).stop 2023-10-07T09:41:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 shutdown_connections 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7efef4067130 0x7efef40695e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff10152950 0x7eff10152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff10157650 0x7eff101520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 --2- 172.21.15.29:0/1714015793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff10157a20 0x7eff101536f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 >> 172.21.15.29:0/1714015793 conn(0x7eff10067610 msgr2=0x7eff100cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 shutdown_connections 2023-10-07T09:41:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:33.156+0000 7eff181ff700 1 -- 172.21.15.29:0/1714015793 wait complete. 2023-10-07T09:41:33.174 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-07T09:41:33.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.174 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-07T09:41:33.222 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:33.222 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:33.222 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:33.222 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:33.222 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:33.223 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:33.223 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:33.223 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:33.223 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:33.223 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:33.223 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:33.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:33.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.224 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:33.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:33.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.281 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:33.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:33.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.337 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:33.390 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:33.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.391 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:33.446 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:33.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.446 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:33.501 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:33.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.501 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:33.556 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:33.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.557 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:33.611 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:33.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.611 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:33.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:33.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.665 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:33.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:33.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.719 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:33.774 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:33.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.774 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:33.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:33.828 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-07T09:41:33.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir2 1M 2023-10-07T09:41:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.076+0000 7ff892bff700 1 Processor -- start 2023-10-07T09:41:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.076+0000 7ff892bff700 1 -- start start 2023-10-07T09:41:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.077+0000 7ff892bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff88c0b2a20 0x7ff88c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.077+0000 7ff892bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 0x7ff88c0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.077+0000 7ff892bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0cc940 con 0x7ff88c0cf0f0 2023-10-07T09:41:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.077+0000 7ff892bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c0cf530 con 0x7ff88c0b2a20 2023-10-07T09:41:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.077+0000 7ff892bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0b2df0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.078+0000 7ff88bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 0x7ff88c0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.078+0000 7ff88bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 0x7ff88c0cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:45148/0 (socket says 172.21.15.29:45148) 2023-10-07T09:41:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.078+0000 7ff88bfff700 1 -- 172.21.15.29:0/1936381429 learned_addr learned my addr 172.21.15.29:0/1936381429 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.078+0000 7ff89099b700 1 --2- 172.21.15.29:0/1936381429 >> v2:172.21.15.144:3300/0 conn(0x7ff88c0b2a20 0x7ff88c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.078+0000 7ff88bfff700 1 -- 172.21.15.29:0/1936381429 >> v2:172.21.15.144:3300/0 conn(0x7ff88c0b2a20 msgr2=0x7ff88c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.078+0000 7ff88bfff700 1 --2- 172.21.15.29:0/1936381429 >> v2:172.21.15.144:3300/0 conn(0x7ff88c0b2a20 0x7ff88c0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88bfff700 1 -- 172.21.15.29:0/1936381429 >> v1:172.21.15.144:6789/0 conn(0x7ff88c0cf0f0 legacy=0x7ff88c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88bfff700 1 -- 172.21.15.29:0/1936381429 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0cbd30 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88b7fe700 1 -- 172.21.15.29:0/1936381429 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785455919 0 0) 0x7ff88c0cc940 con 0x7ff88c0cf0f0 2023-10-07T09:41:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88bfff700 1 --2- 172.21.15.29:0/1936381429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 0x7ff88c0cebb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8780099b0 tx=0x7ff878009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=140fcecf63664cb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88b7fe700 1 -- 172.21.15.29:0/1936381429 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878009ea0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88b7fe700 1 -- 172.21.15.29:0/1936381429 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87800b860 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.079+0000 7ff88b7fe700 1 -- 172.21.15.29:0/1936381429 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87800bc00 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 -- 172.21.15.29:0/1936381429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 msgr2=0x7ff88c0cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 --2- 172.21.15.29:0/1936381429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 0x7ff88c0cebb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8780099b0 tx=0x7ff878009250 comp rx=0 tx=0).stop 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 -- 172.21.15.29:0/1936381429 shutdown_connections 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 --2- 172.21.15.29:0/1936381429 >> v2:172.21.15.144:3300/0 conn(0x7ff88c0b2a20 0x7ff88c0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 --2- 172.21.15.29:0/1936381429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0cb6f0 0x7ff88c0cebb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 -- 172.21.15.29:0/1936381429 >> 172.21.15.29:0/1936381429 conn(0x7ff88c099820 msgr2=0x7ff88c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 -- 172.21.15.29:0/1936381429 shutdown_connections 2023-10-07T09:41:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.080+0000 7ff892bff700 1 -- 172.21.15.29:0/1936381429 wait complete. 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.081+0000 7ff892bff700 1 Processor -- start 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.082+0000 7ff892bff700 1 -- start start 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff892bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0b2a20 0x7ff88c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff892bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 0x7ff88c0c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff892bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff88c0cf0f0 0x7ff88c15e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff892bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c1568a0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff892bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c156b80 con 0x7ff88c0cf0f0 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff892bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c156e60 con 0x7ff88c0b2a20 2023-10-07T09:41:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 0x7ff88c0c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 0x7ff88c0c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59066/0 (socket says 172.21.15.29:59066) 2023-10-07T09:41:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 -- 172.21.15.29:0/612479291 learned_addr learned my addr 172.21.15.29:0/612479291 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff89099b700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0b2a20 0x7ff88c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff89119c700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff88c0cf0f0 0x7ff88c15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 -- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0b2a20 msgr2=0x7ff88c1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0b2a20 0x7ff88c1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 -- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff88c0cf0f0 msgr2=0x7ff88c15e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff88c0cf0f0 0x7ff88c15e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.083+0000 7ff88bfff700 1 -- 172.21.15.29:0/612479291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780094f0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.084+0000 7ff88bfff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 0x7ff88c0c9570 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b2f30 tx=0x7ff878017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.084+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87800bbe0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.084+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878011420 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.084+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8780117e0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.084+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c157140 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.084+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c1577e0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.085+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c0718b0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.086+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7ff878011c10 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.087+0000 7ff888ff9700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff870067130 0x7ff8700695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:34.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.087+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ff8780862c0 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.087+0000 7ff89099b700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff870067130 0x7ff8700695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.088+0000 7ff89099b700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff870067130 0x7ff8700695e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff880009c90 tx=0x7ff880008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.091+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff878053110 con 0x7ff88c0cb6f0 2023-10-07T09:41:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.323+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff88c156cf0 con 0x7ff870067130 2023-10-07T09:41:34.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.402+0000 7ff888ff9700 1 -- 172.21.15.29:0/612479291 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff88c156cf0 con 0x7ff870067130 2023-10-07T09:41:34.403 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-07T09:41:34.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff870067130 msgr2=0x7ff8700695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:34.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff870067130 0x7ff8700695e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff880009c90 tx=0x7ff880008040 comp rx=0 tx=0).stop 2023-10-07T09:41:34.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 msgr2=0x7ff88c0c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:34.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 0x7ff88c0c9570 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b2f30 tx=0x7ff878017000 comp rx=0 tx=0).stop 2023-10-07T09:41:34.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 shutdown_connections 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff870067130 0x7ff8700695e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff88c0cf0f0 0x7ff88c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff88c0b2a20 0x7ff88c1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 --2- 172.21.15.29:0/612479291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff88c0cb6f0 0x7ff88c0c9570 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 >> 172.21.15.29:0/612479291 conn(0x7ff88c099820 msgr2=0x7ff88c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 shutdown_connections 2023-10-07T09:41:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:34.405+0000 7ff892bff700 1 -- 172.21.15.29:0/612479291 wait complete. 2023-10-07T09:41:34.423 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-07T09:41:34.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.424 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:34.473 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:34.473 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:34.473 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:34.473 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:34.474 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:34.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.474 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:34.529 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:34.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.529 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:34.583 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:34.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.583 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:34.637 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:34.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.637 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:34.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:34.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.692 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:34.746 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:34.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.747 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:34.801 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:34.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.801 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:34.855 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:34.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.855 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:34.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:34.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.909 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:34.963 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:34.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:34.963 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:35.017 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:35.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.018 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:35.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.071 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-07T09:41:35.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir3 1M 2023-10-07T09:41:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.321+0000 7f11e41ff700 1 Processor -- start 2023-10-07T09:41:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.321+0000 7f11e41ff700 1 -- start start 2023-10-07T09:41:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.322+0000 7f11e41ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11dc0b2a20 0x7f11dc0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.322+0000 7f11e41ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11dc0cb6f0 0x7f11dc0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.322+0000 7f11e41ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0cc940 con 0x7f11dc0cf0f0 2023-10-07T09:41:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.322+0000 7f11e41ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11dc0cf4f0 con 0x7f11dc0cb6f0 2023-10-07T09:41:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.322+0000 7f11e41ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11dc0b2df0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11dc0cb6f0 0x7f11dc0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11dc0cb6f0 0x7f11dc0cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59068/0 (socket says 172.21.15.29:59068) 2023-10-07T09:41:35.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 -- 172.21.15.29:0/1525889565 learned_addr learned my addr 172.21.15.29:0/1525889565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:35.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e279c700 1 --1- 172.21.15.29:0/1525889565 >> v1:172.21.15.29:6789/0 conn(0x7f11dc0cf0f0 0x7f11dc159ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58504/0 (socket says 172.21.15.29:58504) 2023-10-07T09:41:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e1f9b700 1 --2- 172.21.15.29:0/1525889565 >> v2:172.21.15.144:3300/0 conn(0x7f11dc0b2a20 0x7f11dc0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 -- 172.21.15.29:0/1525889565 >> v2:172.21.15.144:3300/0 conn(0x7f11dc0b2a20 msgr2=0x7f11dc0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 --2- 172.21.15.29:0/1525889565 >> v2:172.21.15.144:3300/0 conn(0x7f11dc0b2a20 0x7f11dc0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 -- 172.21.15.29:0/1525889565 >> v1:172.21.15.29:6789/0 conn(0x7f11dc0cf0f0 legacy=0x7f11dc159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.323+0000 7f11e179a700 1 -- 172.21.15.29:0/1525889565 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0cbd30 con 0x7f11dc0cb6f0 2023-10-07T09:41:35.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.324+0000 7f11e179a700 1 --2- 172.21.15.29:0/1525889565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc0cebb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f11dc08ac80 tx=0x7f11cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ec725bacdf3b94 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.324+0000 7f11e0f99700 1 -- 172.21.15.29:0/1525889565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc00b860 con 0x7f11dc0cb6f0 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.324+0000 7f11e0f99700 1 -- 172.21.15.29:0/1525889565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc00b9c0 con 0x7f11dc0cb6f0 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.324+0000 7f11e0f99700 1 -- 172.21.15.29:0/1525889565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc00bd60 con 0x7f11dc0cb6f0 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 -- 172.21.15.29:0/1525889565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0cb6f0 msgr2=0x7f11dc0cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 --2- 172.21.15.29:0/1525889565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc0cebb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f11dc08ac80 tx=0x7f11cc009250 comp rx=0 tx=0).stop 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 -- 172.21.15.29:0/1525889565 shutdown_connections 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 --2- 172.21.15.29:0/1525889565 >> v2:172.21.15.144:3300/0 conn(0x7f11dc0b2a20 0x7f11dc0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 --2- 172.21.15.29:0/1525889565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc0cebb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 -- 172.21.15.29:0/1525889565 >> 172.21.15.29:0/1525889565 conn(0x7f11dc099820 msgr2=0x7f11dc063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 -- 172.21.15.29:0/1525889565 shutdown_connections 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.325+0000 7f11e41ff700 1 -- 172.21.15.29:0/1525889565 wait complete. 2023-10-07T09:41:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.326+0000 7f11e41ff700 1 Processor -- start 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.327+0000 7f11e41ff700 1 -- start start 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e41ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 0x7f11dc169f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e41ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e41ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11dc0cf0f0 0x7f11dc0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e41ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc150be0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e41ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc155990 con 0x7f11dc0cb6f0 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e41ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc0b3fd0 con 0x7f11dc0cf0f0 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 0x7f11dc169f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 0x7f11dc169f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59078/0 (socket says 172.21.15.29:59078) 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 -- 172.21.15.29:0/3950963781 learned_addr learned my addr 172.21.15.29:0/3950963781 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e179a700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e279c700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11dc0cf0f0 0x7f11dc0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 -- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11dc0cf0f0 msgr2=0x7f11dc0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11dc0cf0f0 0x7f11dc0b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 -- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11dc0cb6f0 msgr2=0x7f11dc155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.328+0000 7f11e1f9b700 1 -- 172.21.15.29:0/3950963781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0094f0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.329+0000 7f11e1f9b700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 0x7f11dc169f70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f11d800d810 tx=0x7f11d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.329+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d80125f0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.329+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8012750 con 0x7f11dc0b2a20 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.329+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8012af0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.329+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc15d280 con 0x7f11dc0b2a20 2023-10-07T09:41:35.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.329+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc155e90 con 0x7f11dc0b2a20 2023-10-07T09:41:35.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.330+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc15d5d0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.331+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f11d801e460 con 0x7f11dc0b2a20 2023-10-07T09:41:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.331+0000 7f11d1ffb700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f11c0067130 0x7f11c00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.332+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f11d8085070 con 0x7f11dc0b2a20 2023-10-07T09:41:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.332+0000 7f11e179a700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f11c0067130 0x7f11c00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.333+0000 7f11e179a700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f11c0067130 0x7f11c00695e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11dc08ac80 tx=0x7f11cc017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:35.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.336+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11d8051ec0 con 0x7f11dc0b2a20 2023-10-07T09:41:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.567+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f11dc15ac60 con 0x7f11c0067130 2023-10-07T09:41:35.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.648+0000 7f11d1ffb700 1 -- 172.21.15.29:0/3950963781 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f11dc15ac60 con 0x7f11c0067130 2023-10-07T09:41:35.649 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f11c0067130 msgr2=0x7f11c00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f11c0067130 0x7f11c00695e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11dc08ac80 tx=0x7f11cc017000 comp rx=0 tx=0).stop 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 msgr2=0x7f11dc169f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 0x7f11dc169f70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f11d800d810 tx=0x7f11d8008040 comp rx=0 tx=0).stop 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 shutdown_connections 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f11c0067130 0x7f11c00695e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11dc0cf0f0 0x7f11dc0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11dc0b2a20 0x7f11dc169f70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.650+0000 7f11e41ff700 1 --2- 172.21.15.29:0/3950963781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11dc0cb6f0 0x7f11dc155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.651+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 >> 172.21.15.29:0/3950963781 conn(0x7f11dc099820 msgr2=0x7f11dc063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.651+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 shutdown_connections 2023-10-07T09:41:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:35.651+0000 7f11e41ff700 1 -- 172.21.15.29:0/3950963781 wait complete. 2023-10-07T09:41:35.669 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-07T09:41:35.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.669 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-07T09:41:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:35.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_01_UTC 2023-10-07T09:41:35.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:35.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:35.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:35.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:35.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:35.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.719 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:35.775 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:35.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.776 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:35.830 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:35.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.831 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:35.886 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:35.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.886 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:35.940 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:35.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.941 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:35.996 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:35.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:35.996 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:36.050 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_37_01_UTC 2023-10-07T09:41:36.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.051 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_37_01_UTC) 2023-10-07T09:41:36.105 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:36.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.106 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:36.159 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:36.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.160 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:36.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:36.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.214 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:36.268 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:36.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.268 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:36.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.322 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-07T09:41:36.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir4 1M 2023-10-07T09:41:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.572+0000 7f2d4ebff700 1 Processor -- start 2023-10-07T09:41:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.572+0000 7f2d4ebff700 1 -- start start 2023-10-07T09:41:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.573+0000 7f2d4ebff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d480ced00 0x7f2d480b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.573+0000 7f2d4ebff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d480c8a20 0x7f2d480cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.573+0000 7f2d4ebff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d480690f0 con 0x7f2d480b0610 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.573+0000 7f2d4ebff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d480cf2e0 con 0x7f2d480c8a20 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.573+0000 7f2d4ebff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d480687c0 con 0x7f2d480ced00 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4c99b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2d480b0610 0x7f2d480c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58512/0 (socket says 172.21.15.29:58512) 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4c99b700 1 -- 172.21.15.29:0/2410875662 learned_addr learned my addr 172.21.15.29:0/2410875662 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4d19c700 1 --2- 172.21.15.29:0/2410875662 >> v2:172.21.15.29:3300/0 conn(0x7f2d480c8a20 0x7f2d480cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d47fff700 1 --2- 172.21.15.29:0/2410875662 >> v2:172.21.15.144:3300/0 conn(0x7f2d480ced00 0x7f2d480b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d477fe700 1 -- 172.21.15.29:0/2410875662 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749836390 0 0) 0x7f2d480690f0 con 0x7f2d480b0610 2023-10-07T09:41:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4d19c700 1 -- 172.21.15.29:0/2410875662 >> v2:172.21.15.144:3300/0 conn(0x7f2d480ced00 msgr2=0x7f2d480b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4d19c700 1 --2- 172.21.15.29:0/2410875662 >> v2:172.21.15.144:3300/0 conn(0x7f2d480ced00 0x7f2d480b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4d19c700 1 -- 172.21.15.29:0/2410875662 >> v1:172.21.15.29:6789/0 conn(0x7f2d480b0610 legacy=0x7f2d480c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4d19c700 1 -- 172.21.15.29:0/2410875662 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d48068aa0 con 0x7f2d480c8a20 2023-10-07T09:41:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.574+0000 7f2d4d19c700 1 --2- 172.21.15.29:0/2410875662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480c8a20 0x7f2d480cc7e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2d480cc2f0 tx=0x7f2d40009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d61458504b2f04c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d477fe700 1 -- 172.21.15.29:0/2410875662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000e070 con 0x7f2d480c8a20 2023-10-07T09:41:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d477fe700 1 -- 172.21.15.29:0/2410875662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4000bb60 con 0x7f2d480c8a20 2023-10-07T09:41:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d477fe700 1 -- 172.21.15.29:0/2410875662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40012430 con 0x7f2d480c8a20 2023-10-07T09:41:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 -- 172.21.15.29:0/2410875662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480c8a20 msgr2=0x7f2d480cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/2410875662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480c8a20 0x7f2d480cc7e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2d480cc2f0 tx=0x7f2d40009780 comp rx=0 tx=0).stop 2023-10-07T09:41:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 -- 172.21.15.29:0/2410875662 shutdown_connections 2023-10-07T09:41:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/2410875662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480c8a20 0x7f2d480cc7e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/2410875662 >> v2:172.21.15.144:3300/0 conn(0x7f2d480ced00 0x7f2d480b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 -- 172.21.15.29:0/2410875662 >> 172.21.15.29:0/2410875662 conn(0x7f2d48063a10 msgr2=0x7f2d480ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 -- 172.21.15.29:0/2410875662 shutdown_connections 2023-10-07T09:41:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.575+0000 7f2d4ebff700 1 -- 172.21.15.29:0/2410875662 wait complete. 2023-10-07T09:41:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.577+0000 7f2d4ebff700 1 Processor -- start 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.578+0000 7f2d4ebff700 1 -- start start 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4ebff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d480b0610 0x7f2d48158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4ebff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d480c8a20 0x7f2d481542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4ebff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 0x7f2d481693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4ebff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480c7580 con 0x7f2d480ced00 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4ebff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d48158d50 con 0x7f2d480c8a20 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4ebff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480c6c50 con 0x7f2d480b0610 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4d19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 0x7f2d481693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4d19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 0x7f2d481693b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59084/0 (socket says 172.21.15.29:59084) 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.579+0000 7f2d4d19c700 1 -- 172.21.15.29:0/1381481774 learned_addr learned my addr 172.21.15.29:0/1381481774 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d47fff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d480c8a20 0x7f2d481542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4c99b700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d480b0610 0x7f2d48158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4d19c700 1 -- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d480b0610 msgr2=0x7f2d48158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4d19c700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d480b0610 0x7f2d48158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4d19c700 1 -- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d480c8a20 msgr2=0x7f2d481542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4d19c700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d480c8a20 0x7f2d481542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4d19c700 1 -- 172.21.15.29:0/1381481774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40009a20 con 0x7f2d480ced00 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4c99b700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d480b0610 0x7f2d48158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4d19c700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 0x7f2d481693b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2d400099f0 tx=0x7f2d40019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40012e80 con 0x7f2d480ced00 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4001d630 con 0x7f2d480ced00 2023-10-07T09:41:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4001d9f0 con 0x7f2d480ced00 2023-10-07T09:41:36.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d480c6f30 con 0x7f2d480ced00 2023-10-07T09:41:36.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.580+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d48154940 con 0x7f2d480ced00 2023-10-07T09:41:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.581+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d48154b60 con 0x7f2d480ced00 2023-10-07T09:41:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.582+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f2d40024070 con 0x7f2d480ced00 2023-10-07T09:41:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.583+0000 7f2d44ff9700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2d2c067130 0x7f2d2c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.583+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f2d40087240 con 0x7f2d480ced00 2023-10-07T09:41:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.584+0000 7f2d4c99b700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2d2c067130 0x7f2d2c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.584+0000 7f2d4c99b700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2d2c067130 0x7f2d2c0695e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c001810 tx=0x7f2d3c009040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:36.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.587+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d40054090 con 0x7f2d480ced00 2023-10-07T09:41:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.820+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d480718b0 con 0x7f2d2c067130 2023-10-07T09:41:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.907+0000 7f2d44ff9700 1 -- 172.21.15.29:0/1381481774 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f2d480718b0 con 0x7f2d2c067130 2023-10-07T09:41:36.908 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.909+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2d2c067130 msgr2=0x7f2d2c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.909+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2d2c067130 0x7f2d2c0695e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c001810 tx=0x7f2d3c009040 comp rx=0 tx=0).stop 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 msgr2=0x7f2d481693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 0x7f2d481693b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2d400099f0 tx=0x7f2d40019000 comp rx=0 tx=0).stop 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 shutdown_connections 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f2d2c067130 0x7f2d2c0695e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d480ced00 0x7f2d481693b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d480b0610 0x7f2d48158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 --2- 172.21.15.29:0/1381481774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d480c8a20 0x7f2d481542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 >> 172.21.15.29:0/1381481774 conn(0x7f2d48063a10 msgr2=0x7f2d480b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 shutdown_connections 2023-10-07T09:41:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:36.910+0000 7f2d4ebff700 1 -- 172.21.15.29:0/1381481774 wait complete. 2023-10-07T09:41:36.928 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-07T09:41:36.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.929 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-07T09:41:36.977 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:36.978 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:36.979 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:36.980 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:36.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:36.980 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:37.036 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:37.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.036 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:37.090 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:37.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.091 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:37.146 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:37.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.146 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:37.201 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:37.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.201 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:37.255 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:37.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.256 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:37.310 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:37.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.310 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:37.413 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:37.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.413 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:37.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:37.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.545 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:37.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:37.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.643 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:37.702 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:37.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.703 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:37.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:37.805 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-07T09:41:37.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir5 1M 2023-10-07T09:41:38.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.109+0000 7ff7f17ff700 1 Processor -- start 2023-10-07T09:41:38.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.110+0000 7ff7f17ff700 1 -- start start 2023-10-07T09:41:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7f17ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7ec0ced00 0x7ff7ec0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7f17ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0c8a20 0x7ff7ec0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7f17ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ec0690f0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7f17ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ec0cf2e0 con 0x7ff7ec0ced00 2023-10-07T09:41:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7f17ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec0687c0 con 0x7ff7ec0c8a20 2023-10-07T09:41:38.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7eb7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0c8a20 0x7ff7ec0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7ea7fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7ec0ced00 0x7ff7ec0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7eaffd700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff7ec0b0610 0x7ff7ec0c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:41782/0 (socket says 172.21.15.29:41782) 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.111+0000 7ff7eaffd700 1 -- 172.21.15.29:0/3551225457 learned_addr learned my addr 172.21.15.29:0/3551225457 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7eb7fe700 1 -- 172.21.15.29:0/3551225457 >> v2:172.21.15.144:3300/0 conn(0x7ff7ec0ced00 msgr2=0x7ff7ec0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7eb7fe700 1 --2- 172.21.15.29:0/3551225457 >> v2:172.21.15.144:3300/0 conn(0x7ff7ec0ced00 0x7ff7ec0b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7eb7fe700 1 -- 172.21.15.29:0/3551225457 >> v1:172.21.15.144:6789/0 conn(0x7ff7ec0b0610 legacy=0x7ff7ec0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7eb7fe700 1 -- 172.21.15.29:0/3551225457 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec068aa0 con 0x7ff7ec0c8a20 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7e9ffb700 1 -- 172.21.15.29:0/3551225457 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517223467 0 0) 0x7ff7ec0690f0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7ea7fc700 1 --2- 172.21.15.29:0/3551225457 >> v2:172.21.15.144:3300/0 conn(0x7ff7ec0ced00 0x7ff7ec0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7eb7fe700 1 --2- 172.21.15.29:0/3551225457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0c8a20 0x7ff7ec0cc7e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009e60 tx=0x7ff7e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd11ff289cc9301b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.112+0000 7ff7e9ffb700 1 -- 172.21.15.29:0/3551225457 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e400e070 con 0x7ff7ec0c8a20 2023-10-07T09:41:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7e9ffb700 1 -- 172.21.15.29:0/3551225457 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e400bb60 con 0x7ff7ec0c8a20 2023-10-07T09:41:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7e9ffb700 1 -- 172.21.15.29:0/3551225457 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4012430 con 0x7ff7ec0c8a20 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 -- 172.21.15.29:0/3551225457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0c8a20 msgr2=0x7ff7ec0cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/3551225457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0c8a20 0x7ff7ec0cc7e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009e60 tx=0x7ff7e4009780 comp rx=0 tx=0).stop 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 -- 172.21.15.29:0/3551225457 shutdown_connections 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/3551225457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0c8a20 0x7ff7ec0cc7e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/3551225457 >> v2:172.21.15.144:3300/0 conn(0x7ff7ec0ced00 0x7ff7ec0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 -- 172.21.15.29:0/3551225457 >> 172.21.15.29:0/3551225457 conn(0x7ff7ec063a10 msgr2=0x7ff7ec0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 -- 172.21.15.29:0/3551225457 shutdown_connections 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.113+0000 7ff7f17ff700 1 -- 172.21.15.29:0/3551225457 wait complete. 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.115+0000 7ff7f17ff700 1 Processor -- start 2023-10-07T09:41:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.115+0000 7ff7f17ff700 1 -- start start 2023-10-07T09:41:38.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7f17ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0b0610 0x7ff7ec098c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:38.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7f17ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7ec0c8a20 0x7ff7ec14b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7f17ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 0x7ff7ec099c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7f17ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec0c8860 con 0x7ff7ec0ced00 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7f17ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec0c82a0 con 0x7ff7ec0c8a20 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7f17ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec0c8580 con 0x7ff7ec0b0610 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 0x7ff7ec099c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eaffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0b0610 0x7ff7ec098c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 0x7ff7ec099c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59086/0 (socket says 172.21.15.29:59086) 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eb7fe700 1 -- 172.21.15.29:0/523705433 learned_addr learned my addr 172.21.15.29:0/523705433 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7ea7fc700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7ec0c8a20 0x7ff7ec14b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eaffd700 1 -- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7ec0c8a20 msgr2=0x7ff7ec14b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eaffd700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7ec0c8a20 0x7ff7ec14b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eaffd700 1 -- 172.21.15.29:0/523705433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 msgr2=0x7ff7ec099c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eaffd700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 0x7ff7ec099c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eaffd700 1 -- 172.21.15.29:0/523705433 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0009420 con 0x7ff7ec0b0610 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.116+0000 7ff7eb7fe700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 0x7ff7ec099c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.117+0000 7ff7eaffd700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0b0610 0x7ff7ec098c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00c4f0 tx=0x7ff7dc007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.117+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc00cea0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.117+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc005b30 con 0x7ff7ec0b0610 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.117+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4009a20 con 0x7ff7ec0b0610 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.117+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0070d0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.117+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec0c58a0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.118+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec0718b0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.119+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7ff7dc0074c0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.120+0000 7ff7d2ffd700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7d8067130 0x7ff7d80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.120+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6691+0+0 (secure 0 0 0) 0x7ff7dc084070 con 0x7ff7ec0b0610 2023-10-07T09:41:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.120+0000 7ff7ea7fc700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7d8067130 0x7ff7d80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.121+0000 7ff7ea7fc700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7d8067130 0x7ff7d80695e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0cec10 tx=0x7ff7e00091a0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:38.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.125+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7dc050be0 con 0x7ff7ec0b0610 2023-10-07T09:41:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.375+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7ec068d80 con 0x7ff7d8067130 2023-10-07T09:41:38.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.477+0000 7ff7d2ffd700 1 -- 172.21.15.29:0/523705433 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff7ec068d80 con 0x7ff7d8067130 2023-10-07T09:41:38.478 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-07T09:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7d8067130 msgr2=0x7ff7d80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7d8067130 0x7ff7d80695e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0cec10 tx=0x7ff7e00091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0b0610 msgr2=0x7ff7ec098c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0b0610 0x7ff7ec098c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00c4f0 tx=0x7ff7dc007f50 comp rx=0 tx=0).stop 2023-10-07T09:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 shutdown_connections 2023-10-07T09:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff7d8067130 0x7ff7d80695e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7ec0ced00 0x7ff7ec099c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7ec0b0610 0x7ff7ec098c90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 --2- 172.21.15.29:0/523705433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7ec0c8a20 0x7ff7ec14b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.479+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 >> 172.21.15.29:0/523705433 conn(0x7ff7ec063a10 msgr2=0x7ff7ec069580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.480+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 shutdown_connections 2023-10-07T09:41:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:38.480+0000 7ff7f17ff700 1 -- 172.21.15.29:0/523705433 wait complete. 2023-10-07T09:41:38.498 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-07T09:41:38.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.498 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-07T09:41:38.547 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:38.547 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:38.547 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:38.547 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_01_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:38.548 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:38.549 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:38.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.550 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:38.607 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:38.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.608 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:38.662 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:38.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.662 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:38.717 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:38.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.717 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:38.779 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:38.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.779 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:38.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_36_01_UTC 2023-10-07T09:41:38.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.834 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_36_01_UTC) 2023-10-07T09:41:38.887 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:38.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.888 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:38.942 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:38.943 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.943 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:38.997 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:38.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:38.997 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:39.051 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:39.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.051 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:39.106 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:39.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.107 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:39.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.166 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-07T09:41:39.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir6 1M 2023-10-07T09:41:39.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.435+0000 7ff25efff700 1 Processor -- start 2023-10-07T09:41:39.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.435+0000 7ff25efff700 1 -- start start 2023-10-07T09:41:39.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.436+0000 7ff25efff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff2580cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:39.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.436+0000 7ff25efff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff25815a590 0x7ff25815aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:39.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.436+0000 7ff25efff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2580ad160 con 0x7ff2580accf0 2023-10-07T09:41:39.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.436+0000 7ff25efff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25815a140 con 0x7ff25815a590 2023-10-07T09:41:39.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.436+0000 7ff25efff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580adca0 con 0x7ff2580afeb0 2023-10-07T09:41:39.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25cd9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff2580cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:39.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25d59c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff25815a590 0x7ff25815aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:39.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25cd9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff2580cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:45196/0 (socket says 172.21.15.29:45196) 2023-10-07T09:41:39.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25cd9b700 1 -- 172.21.15.29:0/3463516014 learned_addr learned my addr 172.21.15.29:0/3463516014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:39.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff257fff700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff2580accf0 0x7ff2580aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:41788/0 (socket says 172.21.15.29:41788) 2023-10-07T09:41:39.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25d59c700 1 -- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 msgr2=0x7ff2580cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:39.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25d59c700 1 --2- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff2580cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25d59c700 1 -- 172.21.15.29:0/3463516014 >> v1:172.21.15.144:6789/0 conn(0x7ff2580accf0 legacy=0x7ff2580aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:39.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25d59c700 1 -- 172.21.15.29:0/3463516014 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580ad400 con 0x7ff25815a590 2023-10-07T09:41:39.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff2577fe700 1 -- 172.21.15.29:0/3463516014 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994785040 0 0) 0x7ff2580ad160 con 0x7ff2580accf0 2023-10-07T09:41:39.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.437+0000 7ff25cd9b700 1 --2- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff2580cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:39.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25d59c700 1 --2- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 0x7ff25815aa10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff250009e60 tx=0x7ff250009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a77877cc8e22ff96 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:39.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff2577fe700 1 -- 172.21.15.29:0/3463516014 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25000e070 con 0x7ff25815a590 2023-10-07T09:41:39.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff2577fe700 1 -- 172.21.15.29:0/3463516014 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25000bb60 con 0x7ff25815a590 2023-10-07T09:41:39.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff2577fe700 1 -- 172.21.15.29:0/3463516014 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250012430 con 0x7ff25815a590 2023-10-07T09:41:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 -- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 msgr2=0x7ff25815aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 --2- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 0x7ff25815aa10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff250009e60 tx=0x7ff250009780 comp rx=0 tx=0).stop 2023-10-07T09:41:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 -- 172.21.15.29:0/3463516014 shutdown_connections 2023-10-07T09:41:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 --2- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 0x7ff25815aa10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 --2- 172.21.15.29:0/3463516014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff2580cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 -- 172.21.15.29:0/3463516014 >> 172.21.15.29:0/3463516014 conn(0x7ff258067610 msgr2=0x7ff258067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.438+0000 7ff25efff700 1 -- 172.21.15.29:0/3463516014 shutdown_connections 2023-10-07T09:41:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.439+0000 7ff25efff700 1 -- 172.21.15.29:0/3463516014 wait complete. 2023-10-07T09:41:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.441+0000 7ff25efff700 1 Processor -- start 2023-10-07T09:41:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.441+0000 7ff25efff700 1 -- start start 2023-10-07T09:41:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.442+0000 7ff25efff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2580accf0 0x7ff2580c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.443+0000 7ff25efff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff258170f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.443+0000 7ff25efff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 0x7ff2581714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:39.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.443+0000 7ff25efff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580c1840 con 0x7ff2580accf0 2023-10-07T09:41:39.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.443+0000 7ff25efff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580c1ae0 con 0x7ff25815a590 2023-10-07T09:41:39.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.443+0000 7ff25efff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580c1dc0 con 0x7ff2580afeb0 2023-10-07T09:41:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.444+0000 7ff257fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff258170f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.444+0000 7ff257fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff258170f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:45200/0 (socket says 172.21.15.29:45200) 2023-10-07T09:41:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.444+0000 7ff257fff700 1 -- 172.21.15.29:0/1961267408 learned_addr learned my addr 172.21.15.29:0/1961267408 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:39.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.447+0000 7ff257fff700 1 -- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 msgr2=0x7ff2581714b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:41:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.447+0000 7ff257fff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 0x7ff2581714b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.447+0000 7ff257fff700 1 -- 172.21.15.29:0/1961267408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2580accf0 msgr2=0x7ff2580c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.447+0000 7ff257fff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2580accf0 0x7ff2580c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.447+0000 7ff257fff700 1 -- 172.21.15.29:0/1961267408 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0094f0 con 0x7ff2580afeb0 2023-10-07T09:41:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.447+0000 7ff257fff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff258170f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff24400c5b0 tx=0x7ff244001040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.448+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244005a70 con 0x7ff2580afeb0 2023-10-07T09:41:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.448+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244005bd0 con 0x7ff2580afeb0 2023-10-07T09:41:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.448+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2440070d0 con 0x7ff2580afeb0 2023-10-07T09:41:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.448+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff250009a20 con 0x7ff2580afeb0 2023-10-07T09:41:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.449+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2580c2380 con 0x7ff2580afeb0 2023-10-07T09:41:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.451+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2580c25a0 con 0x7ff2580afeb0 2023-10-07T09:41:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.453+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7ff244007480 con 0x7ff2580afeb0 2023-10-07T09:41:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.455+0000 7ff254ff9700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff23c067280 0x7ff23c069730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.455+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6435+0+0 (secure 0 0 0) 0x7ff244084120 con 0x7ff2580afeb0 2023-10-07T09:41:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.456+0000 7ff25cd9b700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff23c067280 0x7ff23c069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.458+0000 7ff25cd9b700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff23c067280 0x7ff23c069730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2581595a0 tx=0x7ff24c009250 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.463+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff244050e10 con 0x7ff2580afeb0 2023-10-07T09:41:39.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.695+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2580c1c50 con 0x7ff23c067280 2023-10-07T09:41:39.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.771+0000 7ff254ff9700 1 -- 172.21.15.29:0/1961267408 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff2580c1c50 con 0x7ff23c067280 2023-10-07T09:41:39.772 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.773+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff23c067280 msgr2=0x7ff23c069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.773+0000 7ff25efff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff23c067280 0x7ff23c069730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2581595a0 tx=0x7ff24c009250 comp rx=0 tx=0).stop 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.773+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 msgr2=0x7ff258170f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.773+0000 7ff25efff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff258170f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff24400c5b0 tx=0x7ff244001040 comp rx=0 tx=0).stop 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 shutdown_connections 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7ff23c067280 0x7ff23c069730 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25815a590 0x7ff2581714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2580accf0 0x7ff2580c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 --2- 172.21.15.29:0/1961267408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2580afeb0 0x7ff258170f70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:39.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 >> 172.21.15.29:0/1961267408 conn(0x7ff258067610 msgr2=0x7ff258155a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:39.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 shutdown_connections 2023-10-07T09:41:39.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:39.774+0000 7ff25efff700 1 -- 172.21.15.29:0/1961267408 wait complete. 2023-10-07T09:41:39.791 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-07T09:41:39.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.792 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-07T09:41:39.840 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:39.840 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:39.840 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:39.841 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:39.842 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:39.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.842 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:39.898 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:39.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.898 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:39.951 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:39.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:39.952 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:40.006 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:40.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.006 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:40.060 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:40.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.061 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:40.115 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:40.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.115 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:40.169 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:40.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.169 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:40.223 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:40.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.224 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:40.278 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:40.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.278 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:40.332 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:40.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.332 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:40.430 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:40.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.430 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:40.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:40.493 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-07T09:41:40.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir7 1M 2023-10-07T09:41:40.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.765+0000 7f5a831ff700 1 Processor -- start 2023-10-07T09:41:40.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.766+0000 7f5a831ff700 1 -- start start 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a831ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a7c0cb8c0 0x7f5a7c0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a831ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a7c0cf0f0 0x7f5a7c0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a831ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7c0cc940 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a831ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7c0cf530 con 0x7f5a7c0cf0f0 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a831ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7c0b2df0 con 0x7f5a7c0cb8c0 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a80f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5a7c0b2a20 0x7f5a7c0cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58528/0 (socket says 172.21.15.29:58528) 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a8179c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a7c0cf0f0 0x7f5a7c0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a80f9b700 1 -- 172.21.15.29:0/618304203 learned_addr learned my addr 172.21.15.29:0/618304203 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a7bfff700 1 --2- 172.21.15.29:0/618304203 >> v2:172.21.15.144:3300/0 conn(0x7f5a7c0cb8c0 0x7f5a7c0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a8179c700 1 -- 172.21.15.29:0/618304203 >> v2:172.21.15.144:3300/0 conn(0x7f5a7c0cb8c0 msgr2=0x7f5a7c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a8179c700 1 --2- 172.21.15.29:0/618304203 >> v2:172.21.15.144:3300/0 conn(0x7f5a7c0cb8c0 0x7f5a7c0cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a7b7fe700 1 -- 172.21.15.29:0/618304203 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300490542 0 0) 0x7f5a7c0cc940 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a8179c700 1 -- 172.21.15.29:0/618304203 >> v1:172.21.15.29:6789/0 conn(0x7f5a7c0b2a20 legacy=0x7f5a7c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a8179c700 1 -- 172.21.15.29:0/618304203 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c0cbd30 con 0x7f5a7c0cf0f0 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a7bfff700 1 --2- 172.21.15.29:0/618304203 >> v2:172.21.15.144:3300/0 conn(0x7f5a7c0cb8c0 0x7f5a7c0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.768+0000 7f5a8179c700 1 --2- 172.21.15.29:0/618304203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c5400 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0889b0 tx=0x7f5a74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b325f79e37585f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a7b7fe700 1 -- 172.21.15.29:0/618304203 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400e070 con 0x7f5a7c0cf0f0 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a7b7fe700 1 -- 172.21.15.29:0/618304203 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7400bc30 con 0x7f5a7c0cf0f0 2023-10-07T09:41:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a7b7fe700 1 -- 172.21.15.29:0/618304203 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74012430 con 0x7f5a7c0cf0f0 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 -- 172.21.15.29:0/618304203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 msgr2=0x7f5a7c0c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 --2- 172.21.15.29:0/618304203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c5400 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0889b0 tx=0x7f5a74009780 comp rx=0 tx=0).stop 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 -- 172.21.15.29:0/618304203 shutdown_connections 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 --2- 172.21.15.29:0/618304203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c5400 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 --2- 172.21.15.29:0/618304203 >> v2:172.21.15.144:3300/0 conn(0x7f5a7c0cb8c0 0x7f5a7c0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 -- 172.21.15.29:0/618304203 >> 172.21.15.29:0/618304203 conn(0x7f5a7c099820 msgr2=0x7f5a7c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 -- 172.21.15.29:0/618304203 shutdown_connections 2023-10-07T09:41:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.769+0000 7f5a831ff700 1 -- 172.21.15.29:0/618304203 wait complete. 2023-10-07T09:41:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.772+0000 7f5a831ff700 1 Processor -- start 2023-10-07T09:41:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.772+0000 7f5a831ff700 1 -- start start 2023-10-07T09:41:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 0x7f5a7c16cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a7c0cb8c0 0x7f5a7c0695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0c9820 con 0x7f5a7c0cf0f0 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c16d4c0 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c16d740 con 0x7f5a7c0cb8c0 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a8179c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 0x7f5a7c16cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a8179c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59110/0 (socket says 172.21.15.29:59110) 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 0x7f5a7c16cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:50762/0 (socket says 172.21.15.29:50762) 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a8179c700 1 -- 172.21.15.29:0/1550290635 learned_addr learned my addr 172.21.15.29:0/1550290635 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a7bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a7c0cb8c0 0x7f5a7c0695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 -- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a7c0cb8c0 msgr2=0x7f5a7c0695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a7c0cb8c0 0x7f5a7c0695d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 -- 172.21.15.29:0/1550290635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 msgr2=0x7f5a7c0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 -- 172.21.15.29:0/1550290635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74009a20 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a7bfff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a7c0cb8c0 0x7f5a7c0695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a80f9b700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 0x7f5a7c16cf80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000c7e0 tx=0x7f5a70007ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a700072e0 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a70007440 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c16da20 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.774+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a700057f0 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.775+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c0c94b0 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.775+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c0c9b00 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.776+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f5a70005c00 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.777+0000 7f5a78ff9700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5a60067130 0x7f5a600695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.777+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f5a70084070 con 0x7f5a7c0b2a20 2023-10-07T09:41:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.778+0000 7f5a7bfff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5a60067130 0x7f5a600695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.778+0000 7f5a7bfff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5a60067130 0x7f5a600695e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c15bea0 tx=0x7f5a68008040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:40.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:40.783+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a70050ca0 con 0x7f5a7c0b2a20 2023-10-07T09:41:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.012+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a7c0718b0 con 0x7f5a60067130 2023-10-07T09:41:41.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.091+0000 7f5a78ff9700 1 -- 172.21.15.29:0/1550290635 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5a7c0718b0 con 0x7f5a60067130 2023-10-07T09:41:41.091 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-07T09:41:41.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.093+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5a60067130 msgr2=0x7f5a600695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:41.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.093+0000 7f5a831ff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5a60067130 0x7f5a600695e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c15bea0 tx=0x7f5a68008040 comp rx=0 tx=0).stop 2023-10-07T09:41:41.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.093+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 msgr2=0x7f5a7c16cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:41.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.093+0000 7f5a831ff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 0x7f5a7c16cf80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000c7e0 tx=0x7f5a70007ba0 comp rx=0 tx=0).stop 2023-10-07T09:41:41.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 shutdown_connections 2023-10-07T09:41:41.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5a60067130 0x7f5a600695e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:41.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cf0f0 0x7f5a7c0c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:41.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a7c0b2a20 0x7f5a7c16cf80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:41.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 --2- 172.21.15.29:0/1550290635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a7c0cb8c0 0x7f5a7c0695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:41.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 >> 172.21.15.29:0/1550290635 conn(0x7f5a7c099820 msgr2=0x7f5a7c0c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:41.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 shutdown_connections 2023-10-07T09:41:41.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:41.094+0000 7f5a831ff700 1 -- 172.21.15.29:0/1550290635 wait complete. 2023-10-07T09:41:41.111 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-07T09:41:41.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.111 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_01_UTC 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:41.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:41.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:41.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:41.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:41.162 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:41.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.162 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:41.218 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:41.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.218 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:41.273 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:41.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.273 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:41.327 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:41.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.327 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:41.412 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_35_01_UTC 2023-10-07T09:41:41.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.412 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_35_01_UTC) 2023-10-07T09:41:41.470 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:41.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.470 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:41.524 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:41.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.524 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:41.578 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:41.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.578 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:41.632 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:41.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.633 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:41.712 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:41.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.712 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:41.767 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:41.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.767 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:41.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:41.821 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-07T09:41:41.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir8 1M 2023-10-07T09:41:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.070+0000 7f9e995ff700 1 Processor -- start 2023-10-07T09:41:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.070+0000 7f9e995ff700 1 -- start start 2023-10-07T09:41:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e995ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9e9415a730 0x7f9e940c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e995ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e940c1ca0 0x7f9e940ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e995ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e94067610 con 0x7f9e940caad0 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e995ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e940caf50 con 0x7f9e940c1ca0 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e995ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e940c2140 con 0x7f9e9415a730 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e927fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e940c1ca0 0x7f9e940ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e927fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e940c1ca0 0x7f9e940ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59124/0 (socket says 172.21.15.29:59124) 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e927fc700 1 -- 172.21.15.29:0/3122575459 learned_addr learned my addr 172.21.15.29:0/3122575459 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e937fe700 1 --1- 172.21.15.29:0/3122575459 >> v1:172.21.15.29:6789/0 conn(0x7f9e940caad0 0x7f9e9415d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58542/0 (socket says 172.21.15.29:58542) 2023-10-07T09:41:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.072+0000 7f9e92ffd700 1 --2- 172.21.15.29:0/3122575459 >> v2:172.21.15.144:3300/0 conn(0x7f9e9415a730 0x7f9e940c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e927fc700 1 -- 172.21.15.29:0/3122575459 >> v2:172.21.15.144:3300/0 conn(0x7f9e9415a730 msgr2=0x7f9e940c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e927fc700 1 --2- 172.21.15.29:0/3122575459 >> v2:172.21.15.144:3300/0 conn(0x7f9e9415a730 0x7f9e940c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e927fc700 1 -- 172.21.15.29:0/3122575459 >> v1:172.21.15.29:6789/0 conn(0x7f9e940caad0 legacy=0x7f9e9415d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e927fc700 1 -- 172.21.15.29:0/3122575459 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940c2420 con 0x7f9e940c1ca0 2023-10-07T09:41:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e927fc700 1 --2- 172.21.15.29:0/3122575459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940ca590 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9e9406f4b0 tx=0x7f9e88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19040e0c4e5cd06d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e91ffb700 1 -- 172.21.15.29:0/3122575459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88009e70 con 0x7f9e940c1ca0 2023-10-07T09:41:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e91ffb700 1 -- 172.21.15.29:0/3122575459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8800b860 con 0x7f9e940c1ca0 2023-10-07T09:41:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.073+0000 7f9e91ffb700 1 -- 172.21.15.29:0/3122575459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8800bc00 con 0x7f9e940c1ca0 2023-10-07T09:41:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 -- 172.21.15.29:0/3122575459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 msgr2=0x7f9e940ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 --2- 172.21.15.29:0/3122575459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940ca590 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9e9406f4b0 tx=0x7f9e88009250 comp rx=0 tx=0).stop 2023-10-07T09:41:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 -- 172.21.15.29:0/3122575459 shutdown_connections 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 --2- 172.21.15.29:0/3122575459 >> v2:172.21.15.144:3300/0 conn(0x7f9e9415a730 0x7f9e940c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 --2- 172.21.15.29:0/3122575459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940ca590 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 -- 172.21.15.29:0/3122575459 >> 172.21.15.29:0/3122575459 conn(0x7f9e940c9430 msgr2=0x7f9e940c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 -- 172.21.15.29:0/3122575459 shutdown_connections 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.074+0000 7f9e995ff700 1 -- 172.21.15.29:0/3122575459 wait complete. 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.076+0000 7f9e995ff700 1 Processor -- start 2023-10-07T09:41:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.076+0000 7f9e995ff700 1 -- start start 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e995ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e995ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e940caad0 0x7f9e94154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e995ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e9415a730 0x7f9e94169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e995ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9416a9b0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e995ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940b0f20 con 0x7f9e940caad0 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e995ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9416a6d0 con 0x7f9e9415a730 2023-10-07T09:41:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.077+0000 7f9e92ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59132/0 (socket says 172.21.15.29:59132) 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 -- 172.21.15.29:0/424180112 learned_addr learned my addr 172.21.15.29:0/424180112 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e927fc700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e940caad0 0x7f9e94154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e937fe700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e9415a730 0x7f9e94169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 -- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e9415a730 msgr2=0x7f9e94169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e9415a730 0x7f9e94169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 -- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e940caad0 msgr2=0x7f9e94154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e940caad0 0x7f9e94154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 -- 172.21.15.29:0/424180112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880094f0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e927fc700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e940caad0 0x7f9e94154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e92ffd700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940acc50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400ec80 tx=0x7f9e840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8400c9d0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8400cb30 con 0x7f9e940c1ca0 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.078+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e940b08d0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.079+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84013470 con 0x7f9e940c1ca0 2023-10-07T09:41:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.079+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e941549a0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.080+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e940b1200 con 0x7f9e940c1ca0 2023-10-07T09:41:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.080+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7f9e84013820 con 0x7f9e940c1ca0 2023-10-07T09:41:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.081+0000 7f9e7affd700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9e80067130 0x7f9e800695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.081+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9e84084fb0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.082+0000 7f9e927fc700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9e80067130 0x7f9e800695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.082+0000 7f9e927fc700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9e80067130 0x7f9e800695e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9e9406f4b0 tx=0x7f9e88017000 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:42.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.086+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e84051bf0 con 0x7f9e940c1ca0 2023-10-07T09:41:42.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.377+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e940b1060 con 0x7f9e80067130 2023-10-07T09:41:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.454+0000 7f9e7affd700 1 -- 172.21.15.29:0/424180112 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9e8800b9d0 con 0x7f9e80067130 2023-10-07T09:41:42.455 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-07T09:41:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9e80067130 msgr2=0x7f9e800695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9e80067130 0x7f9e800695e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9e9406f4b0 tx=0x7f9e88017000 comp rx=0 tx=0).stop 2023-10-07T09:41:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 msgr2=0x7f9e940acc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940acc50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400ec80 tx=0x7f9e840091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 shutdown_connections 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9e80067130 0x7f9e800695e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e9415a730 0x7f9e94169930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e940c1ca0 0x7f9e940acc50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 --2- 172.21.15.29:0/424180112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e940caad0 0x7f9e94154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 >> 172.21.15.29:0/424180112 conn(0x7f9e940c9430 msgr2=0x7f9e94153580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 shutdown_connections 2023-10-07T09:41:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:42.457+0000 7f9e995ff700 1 -- 172.21.15.29:0/424180112 wait complete. 2023-10-07T09:41:42.474 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-07T09:41:42.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.474 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-07T09:41:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:42.524 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:42.524 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:42.524 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:42.524 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:42.524 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:42.524 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:42.525 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:42.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.525 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:42.581 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:42.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.581 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:42.635 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:42.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.635 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:42.689 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_34_00_UTC 2023-10-07T09:41:42.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.690 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_34_00_UTC) 2023-10-07T09:41:42.744 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:42.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.744 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:42.798 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:42.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.799 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:42.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:42.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.853 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:42.909 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:42.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.910 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:42.964 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:42.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:42.964 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:43.019 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:43.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.019 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:43.077 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:43.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.077 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:43.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.165 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-07T09:41:43.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir9 1M 2023-10-07T09:41:43.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.425+0000 7fb2f51ff700 1 Processor -- start 2023-10-07T09:41:43.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.425+0000 7fb2f51ff700 1 -- start start 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.426+0000 7fb2f51ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b0610 0x7fb2f00ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.426+0000 7fb2f51ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2f00b09e0 0x7fb2f00c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.426+0000 7fb2f51ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f00690f0 con 0x7fb2f00c8a20 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.426+0000 7fb2f51ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f00c4740 con 0x7fb2f00b09e0 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.426+0000 7fb2f51ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00cf2e0 con 0x7fb2f00b0610 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2f00b09e0 0x7fb2f00c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2eed9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b0610 0x7fb2f00ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2f00b09e0 0x7fb2f00c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51082/0 (socket says 172.21.15.29:51082) 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 -- 172.21.15.29:0/595562832 learned_addr learned my addr 172.21.15.29:0/595562832 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 -- 172.21.15.29:0/595562832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b0610 msgr2=0x7fb2f00ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2edd9b700 1 -- 172.21.15.29:0/595562832 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173929364 0 0) 0x7fb2f00690f0 con 0x7fb2f00c8a20 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 --2- 172.21.15.29:0/595562832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b0610 0x7fb2f00ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 -- 172.21.15.29:0/595562832 >> v1:172.21.15.144:6789/0 conn(0x7fb2f00c8a20 legacy=0x7fb2f00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 -- 172.21.15.29:0/595562832 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f00687c0 con 0x7fb2f00b09e0 2023-10-07T09:41:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.427+0000 7fb2ee59c700 1 --2- 172.21.15.29:0/595562832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00b09e0 0x7fb2f00c4200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40099b0 tx=0x7fb2e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c986a4642723175 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:43.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2edd9b700 1 -- 172.21.15.29:0/595562832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4009ea0 con 0x7fb2f00b09e0 2023-10-07T09:41:43.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2edd9b700 1 -- 172.21.15.29:0/595562832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e400b860 con 0x7fb2f00b09e0 2023-10-07T09:41:43.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2edd9b700 1 -- 172.21.15.29:0/595562832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e400bc00 con 0x7fb2f00b09e0 2023-10-07T09:41:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 -- 172.21.15.29:0/595562832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00b09e0 msgr2=0x7fb2f00c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/595562832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00b09e0 0x7fb2f00c4200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40099b0 tx=0x7fb2e4009250 comp rx=0 tx=0).stop 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 -- 172.21.15.29:0/595562832 shutdown_connections 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/595562832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b0610 0x7fb2f00ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/595562832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00b09e0 0x7fb2f00c4200 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 -- 172.21.15.29:0/595562832 >> 172.21.15.29:0/595562832 conn(0x7fb2f0063a10 msgr2=0x7fb2f00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.428+0000 7fb2f51ff700 1 -- 172.21.15.29:0/595562832 shutdown_connections 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.429+0000 7fb2f51ff700 1 -- 172.21.15.29:0/595562832 wait complete. 2023-10-07T09:41:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.430+0000 7fb2f51ff700 1 Processor -- start 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.430+0000 7fb2f51ff700 1 -- start start 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2f51ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2f00b0610 0x7fb2f01517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2f51ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b09e0 0x7fb2f00c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2f51ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 0x7fb2f00c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2f51ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0152030 con 0x7fb2f00c8a20 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2f51ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00b1ab0 con 0x7fb2f00b0610 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2f51ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00c6410 con 0x7fb2f00b09e0 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2ef59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 0x7fb2f00c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2ef59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 0x7fb2f00c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51092/0 (socket says 172.21.15.29:51092) 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2ef59e700 1 -- 172.21.15.29:0/2253860976 learned_addr learned my addr 172.21.15.29:0/2253860976 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.431+0000 7fb2eed9d700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2f00b0610 0x7fb2f01517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ee59c700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b09e0 0x7fb2f00c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ef59e700 1 -- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b09e0 msgr2=0x7fb2f00c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ef59e700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b09e0 0x7fb2f00c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ef59e700 1 -- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2f00b0610 msgr2=0x7fb2f01517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ef59e700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2f00b0610 0x7fb2f01517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ef59e700 1 -- 172.21.15.29:0/2253860976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e40094f0 con 0x7fb2f00c8a20 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2ef59e700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 0x7fb2f00c3f50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800cb60 tx=0x7fb2e8001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80040d0 con 0x7fb2f00c8a20 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e800a7a0 con 0x7fb2f00c8a20 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800ab60 con 0x7fb2f00c8a20 2023-10-07T09:41:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.432+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f00b1180 con 0x7fb2f00c8a20 2023-10-07T09:41:43.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.433+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f00b1740 con 0x7fb2f00c8a20 2023-10-07T09:41:43.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.434+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f00b1c40 con 0x7fb2f00c8a20 2023-10-07T09:41:43.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.434+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fb2e8012420 con 0x7fb2f00c8a20 2023-10-07T09:41:43.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.435+0000 7fb2d6ffd700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb2dc067130 0x7fb2dc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:43.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.436+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6419+0+0 (secure 0 0 0) 0x7fb2e8084870 con 0x7fb2f00c8a20 2023-10-07T09:41:43.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.436+0000 7fb2eed9d700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb2dc067130 0x7fb2dc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:43.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.437+0000 7fb2eed9d700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb2dc067130 0x7fb2dc0695e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009d10 tx=0x7fb2e00091a0 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.440+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2e80514f0 con 0x7fb2f00c8a20 2023-10-07T09:41:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.667+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 --> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2f00718b0 con 0x7fb2dc067130 2023-10-07T09:41:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.741+0000 7fb2d6ffd700 1 -- 172.21.15.29:0/2253860976 <== mgr.5092 v2:172.21.15.144:6832/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb2f00718b0 con 0x7fb2dc067130 2023-10-07T09:41:43.741 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-07T09:41:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb2dc067130 msgr2=0x7fb2dc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb2dc067130 0x7fb2dc0695e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009d10 tx=0x7fb2e00091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 msgr2=0x7fb2f00c3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 0x7fb2f00c3f50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800cb60 tx=0x7fb2e8001770 comp rx=0 tx=0).stop 2023-10-07T09:41:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 shutdown_connections 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fb2dc067130 0x7fb2dc0695e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2f00c8a20 0x7fb2f00c3f50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2f00b0610 0x7fb2f01517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 --2- 172.21.15.29:0/2253860976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2f00b09e0 0x7fb2f00c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.743+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 >> 172.21.15.29:0/2253860976 conn(0x7fb2f0063a10 msgr2=0x7fb2f00cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.744+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 shutdown_connections 2023-10-07T09:41:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:43.744+0000 7fb2f51ff700 1 -- 172.21.15.29:0/2253860976 wait complete. 2023-10-07T09:41:43.762 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-07T09:41:43.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.762 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-07T09:41:43.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:43.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:43.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:43.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_34_01_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:43.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:43.813 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_31_00_UTC 2023-10-07T09:41:43.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.813 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_31_00_UTC) 2023-10-07T09:41:43.869 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_32_00_UTC 2023-10-07T09:41:43.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.870 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_32_00_UTC) 2023-10-07T09:41:43.925 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_33_00_UTC 2023-10-07T09:41:43.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.925 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_33_00_UTC) 2023-10-07T09:41:43.979 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_34_01_UTC 2023-10-07T09:41:43.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:43.979 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_34_01_UTC) 2023-10-07T09:41:44.032 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_35_00_UTC 2023-10-07T09:41:44.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.033 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_35_00_UTC) 2023-10-07T09:41:44.085 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_36_00_UTC 2023-10-07T09:41:44.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.086 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_36_00_UTC) 2023-10-07T09:41:44.189 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_37_00_UTC 2023-10-07T09:41:44.189 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.190 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_37_00_UTC) 2023-10-07T09:41:44.284 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_38_00_UTC 2023-10-07T09:41:44.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.284 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_38_00_UTC) 2023-10-07T09:41:44.339 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_39_00_UTC 2023-10-07T09:41:44.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.339 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_39_00_UTC) 2023-10-07T09:41:44.393 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_40_00_UTC 2023-10-07T09:41:44.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.393 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_40_00_UTC) 2023-10-07T09:41:44.446 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_41_00_UTC 2023-10-07T09:41:44.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.446 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-07-09_41_00_UTC) 2023-10-07T09:41:44.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:44.499 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-07T09:41:44.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T09:41:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.744+0000 7fe254bff700 1 Processor -- start 2023-10-07T09:41:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.745+0000 7fe254bff700 1 -- start start 2023-10-07T09:41:44.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe254bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe250156180 0x7fe2500afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:44.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe254bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2500ad4d0 con 0x7fe2500ae570 2023-10-07T09:41:44.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe254bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2500ad7b0 con 0x7fe2500b0520 2023-10-07T09:41:44.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe254bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2500ada90 con 0x7fe250156180 2023-10-07T09:41:44.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe24dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe2500ae570 0x7fe2500accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47140/0 (socket says 172.21.15.29:47140) 2023-10-07T09:41:44.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe24dd9b700 1 -- 172.21.15.29:0/4126350277 learned_addr learned my addr 172.21.15.29:0/4126350277 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:44.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.746+0000 7fe24e59c700 1 --2- 172.21.15.29:0/4126350277 >> v2:172.21.15.144:3300/0 conn(0x7fe250156180 0x7fe2500afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:44.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924554913 0 0) 0x7fe2500ad4d0 con 0x7fe2500ae570 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe238003570 con 0x7fe2500ae570 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421178145 0 0) 0x7fe2500ad7b0 con 0x7fe2500b0520 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2500ad4d0 con 0x7fe2500b0520 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24e59c700 1 -- 172.21.15.29:0/4126350277 >> v1:172.21.15.144:6789/0 conn(0x7fe2500b0520 legacy=0x7fe2500b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24e59c700 1 -- 172.21.15.29:0/4126350277 >> v1:172.21.15.29:6789/0 conn(0x7fe2500ae570 legacy=0x7fe2500accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24e59c700 1 -- 172.21.15.29:0/4126350277 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2500add70 con 0x7fe250156180 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24e59c700 1 --2- 172.21.15.29:0/4126350277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500afc00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe2400098e0 tx=0x7fe2400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c345f69bbf6e343d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.747+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240011630 con 0x7fe250156180 2023-10-07T09:41:44.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe240011790 con 0x7fe250156180 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe24d59a700 1 -- 172.21.15.29:0/4126350277 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240011b30 con 0x7fe250156180 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 -- 172.21.15.29:0/4126350277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 msgr2=0x7fe2500afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 --2- 172.21.15.29:0/4126350277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500afc00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe2400098e0 tx=0x7fe2400091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 -- 172.21.15.29:0/4126350277 shutdown_connections 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 --2- 172.21.15.29:0/4126350277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500afc00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 -- 172.21.15.29:0/4126350277 >> 172.21.15.29:0/4126350277 conn(0x7fe250067610 msgr2=0x7fe250067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 -- 172.21.15.29:0/4126350277 shutdown_connections 2023-10-07T09:41:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.748+0000 7fe254bff700 1 -- 172.21.15.29:0/4126350277 wait complete. 2023-10-07T09:41:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.750+0000 7fe254bff700 1 Processor -- start 2023-10-07T09:41:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.750+0000 7fe254bff700 1 -- start start 2023-10-07T09:41:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe254bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2500ae570 0x7fe250151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe254bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2500b0520 0x7fe2500c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe254bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe254bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500ca7d0 con 0x7fe2500b0520 2023-10-07T09:41:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe254bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500cc1e0 con 0x7fe250156180 2023-10-07T09:41:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe254bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500c9bc0 con 0x7fe2500ae570 2023-10-07T09:41:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2500b0520 0x7fe2500c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24e59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2500ae570 0x7fe250151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24ed9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2500b0520 0x7fe2500c9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51098/0 (socket says 172.21.15.29:51098) 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24dd9b700 1 -- 172.21.15.29:0/2069476512 learned_addr learned my addr 172.21.15.29:0/2069476512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24e59c700 1 -- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 msgr2=0x7fe2500c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24e59c700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24e59c700 1 -- 172.21.15.29:0/2069476512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2500b0520 msgr2=0x7fe2500c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24e59c700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2500b0520 0x7fe2500c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.751+0000 7fe24e59c700 1 -- 172.21.15.29:0/2069476512 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe240009420 con 0x7fe2500ae570 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.752+0000 7fe24e59c700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2500ae570 0x7fe250151e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe240016000 tx=0x7fe240017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.752+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24001f070 con 0x7fe2500ae570 2023-10-07T09:41:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.752+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe240000bc0 con 0x7fe2500ae570 2023-10-07T09:41:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.752+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2500c9ea0 con 0x7fe2500ae570 2023-10-07T09:41:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.752+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240025420 con 0x7fe2500ae570 2023-10-07T09:41:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.752+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2500ca460 con 0x7fe2500ae570 2023-10-07T09:41:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.753+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25004c3c0 con 0x7fe2500ae570 2023-10-07T09:41:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.754+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 12) v1 ==== 84928+0+0 (secure 0 0 0) 0x7fe240023070 con 0x7fe2500ae570 2023-10-07T09:41:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.755+0000 7fe23e7fc700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe230067130 0x7fe2300695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.755+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6515+0+0 (secure 0 0 0) 0x7fe240085a00 con 0x7fe2500ae570 2023-10-07T09:41:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.755+0000 7fe24dd9b700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe230067130 0x7fe2300695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.756+0000 7fe24dd9b700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe230067130 0x7fe2300695e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe2500ad230 tx=0x7fe244009040 comp rx=0 tx=0).ready entity=mgr.5092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:44.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:44.760+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe240052620 con 0x7fe2500ae570 2023-10-07T09:41:45.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.018+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe2500ae050 con 0x7fe2500ae570 2023-10-07T09:41:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.120+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7fe2400529f0 con 0x7fe2500ae570 2023-10-07T09:41:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.121+0000 7fe23e7fc700 1 -- 172.21.15.29:0/2069476512 <== mon.2 v2:172.21.15.144:3301/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v13) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe24007ba20 con 0x7fe2500ae570 2023-10-07T09:41:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.129+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe230067130 msgr2=0x7fe2300695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.129+0000 7fe254bff700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe230067130 0x7fe2300695e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe2500ad230 tx=0x7fe244009040 comp rx=0 tx=0).stop 2023-10-07T09:41:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.129+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2500ae570 msgr2=0x7fe250151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.129+0000 7fe254bff700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2500ae570 0x7fe250151e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe240016000 tx=0x7fe240017000 comp rx=0 tx=0).stop 2023-10-07T09:41:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.131+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 shutdown_connections 2023-10-07T09:41:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.131+0000 7fe254bff700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe230067130 0x7fe2300695e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.131+0000 7fe254bff700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe250156180 0x7fe2500c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.131+0000 7fe254bff700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2500ae570 0x7fe250151e80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.131+0000 7fe254bff700 1 --2- 172.21.15.29:0/2069476512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2500b0520 0x7fe2500c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.131+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 >> 172.21.15.29:0/2069476512 conn(0x7fe250067610 msgr2=0x7fe250152a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.133+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 shutdown_connections 2023-10-07T09:41:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.133+0000 7fe254bff700 1 -- 172.21.15.29:0/2069476512 wait complete. 2023-10-07T09:41:45.164 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:41:45.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:41:45.273 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:41:45.352 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:45.353+0000 7fdc026ea1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:41:45.353 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:45.352+0000 7fc3aa51e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:41:45.382 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:41:45.382 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:41:45.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:41:45.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T09:41:45.506 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:41:45.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:41:45.507 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:41:45.536 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:45.535+0000 7fc3aa51e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:41:45.553 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:45.553+0000 7fdc026ea1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:41:45.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:41:45.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.807+0000 7f26bbbff700 1 Processor -- start 2023-10-07T09:41:45.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.807+0000 7f26bbbff700 1 -- start start 2023-10-07T09:41:45.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26bbbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b4162060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:45.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26bbbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4158ad0 con 0x7f26b40ae8b0 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26bbbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b415b140 con 0x7f26b4157290 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26bbbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415b420 con 0x7f26b4158670 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26b919a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f26b40ae8b0 0x7f26b40b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47152/0 (socket says 172.21.15.29:47152) 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26b919a700 1 -- 172.21.15.29:0/2574466866 learned_addr learned my addr 172.21.15.29:0/2574466866 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26ba19c700 1 --2- 172.21.15.29:0/2574466866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b4162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744747991 0 0) 0x7f26b4158ad0 con 0x7f26b40ae8b0 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a0003570 con 0x7f26b40ae8b0 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390189793 0 0) 0x7f26b415b140 con 0x7f26b4157290 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.809+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b4158ad0 con 0x7f26b4157290 2023-10-07T09:41:45.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26ba19c700 1 -- 172.21.15.29:0/2574466866 >> v1:172.21.15.144:6789/0 conn(0x7f26b4157290 legacy=0x7f26b40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26ba19c700 1 -- 172.21.15.29:0/2574466866 >> v1:172.21.15.29:6789/0 conn(0x7f26b40ae8b0 legacy=0x7f26b40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26ba19c700 1 -- 172.21.15.29:0/2574466866 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b415b700 con 0x7f26b4158670 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26ba19c700 1 --2- 172.21.15.29:0/2574466866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b4162060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009e90 tx=0x7f26ac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4b92a44de29fb74 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00e070 con 0x7f26b4158670 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac00bb80 con 0x7f26b4158670 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.810+0000 7f26b8999700 1 -- 172.21.15.29:0/2574466866 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac012430 con 0x7f26b4158670 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 -- 172.21.15.29:0/2574466866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 msgr2=0x7f26b4162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 --2- 172.21.15.29:0/2574466866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b4162060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009e90 tx=0x7f26ac009780 comp rx=0 tx=0).stop 2023-10-07T09:41:45.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 -- 172.21.15.29:0/2574466866 shutdown_connections 2023-10-07T09:41:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 --2- 172.21.15.29:0/2574466866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b4162060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 -- 172.21.15.29:0/2574466866 >> 172.21.15.29:0/2574466866 conn(0x7f26b40680a0 msgr2=0x7f26b40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 -- 172.21.15.29:0/2574466866 shutdown_connections 2023-10-07T09:41:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.811+0000 7f26bbbff700 1 -- 172.21.15.29:0/2574466866 wait complete. 2023-10-07T09:41:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.813+0000 7f26bbbff700 1 Processor -- start 2023-10-07T09:41:45.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.814+0000 7f26bbbff700 1 -- start start 2023-10-07T09:41:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.815+0000 7f26bbbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4157290 0x7f26b4149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.815+0000 7f26bbbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b40c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.815+0000 7f26bbbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 0x7f26b40af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.815+0000 7f26bbbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40b1120 con 0x7f26b40c62d0 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.815+0000 7f26bbbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40c8660 con 0x7f26b4157290 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.815+0000 7f26bbbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4169960 con 0x7f26b4158670 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 0x7f26b40af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 0x7f26b40af0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51128/0 (socket says 172.21.15.29:51128) 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 -- 172.21.15.29:0/3596352809 learned_addr learned my addr 172.21.15.29:0/3596352809 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26b999b700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4157290 0x7f26b4149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26b919a700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b40c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 -- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 msgr2=0x7f26b40c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b40c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 -- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4157290 msgr2=0x7f26b4149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4157290 0x7f26b4149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 -- 172.21.15.29:0/3596352809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac009a20 con 0x7f26b40c62d0 2023-10-07T09:41:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26ba19c700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 0x7f26b40af0f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f26b4068dd0 tx=0x7f26ac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:45.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.816+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac01b420 con 0x7f26b40c62d0 2023-10-07T09:41:45.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.817+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac01b5a0 con 0x7f26b40c62d0 2023-10-07T09:41:45.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.817+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac01b960 con 0x7f26b40c62d0 2023-10-07T09:41:45.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.817+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b40c7dc0 con 0x7f26b40c62d0 2023-10-07T09:41:45.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.817+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b40c8400 con 0x7f26b40c62d0 2023-10-07T09:41:45.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.818+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b41697c0 con 0x7f26b40c62d0 2023-10-07T09:41:45.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.819+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f26ac01bd50 con 0x7f26b40c62d0 2023-10-07T09:41:45.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.820+0000 7f26a9ffb700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 0x7f2698069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:45.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.820+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6451+0+0 (secure 0 0 0) 0x7f26ac086590 con 0x7f26b40c62d0 2023-10-07T09:41:45.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.820+0000 7f26b999b700 1 -- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 msgr2=0x7f2698069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:45.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.820+0000 7f26b999b700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 0x7f2698069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:45.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:45.824+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26b41697c0 con 0x7f26b40c62d0 2023-10-07T09:41:45.849 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:45.848+0000 7fc3aa51e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:41:45.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:45.896+0000 7fdc026ea1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:41:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.021+0000 7f26b999b700 1 -- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 msgr2=0x7f2698069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.021+0000 7f26b999b700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 0x7f2698069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:46.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.079+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26b41697c0 con 0x7f26b40c62d0 2023-10-07T09:41:46.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.080+0000 7f26a9ffb700 1 -- 172.21.15.29:0/3596352809 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f26b41697c0 con 0x7f26b40c62d0 2023-10-07T09:41:46.080 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:46.081 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:30:25.453869+0000","modified":"2023-10-07T09:30:29.396464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6836/3596704883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3596704883},{"type":"v1","addr":"172.21.15.144:6836","nonce":3596704883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-07T09:41:46.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 msgr2=0x7f2698069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:46.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 0x7f2698069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 msgr2=0x7f26b40af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 0x7f26b40af0f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f26b4068dd0 tx=0x7f26ac019000 comp rx=0 tx=0).stop 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 shutdown_connections 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f26980670e0 0x7f2698069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b40c62d0 0x7f26b40af0f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4157290 0x7f26b4149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 --2- 172.21.15.29:0/3596352809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4158670 0x7f26b40c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 >> 172.21.15.29:0/3596352809 conn(0x7f26b40680a0 msgr2=0x7f26b414ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:46.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.082+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 shutdown_connections 2023-10-07T09:41:46.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.083+0000 7f26bbbff700 1 -- 172.21.15.29:0/3596352809 wait complete. 2023-10-07T09:41:46.085 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-07T09:41:46.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:41:46.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.320+0000 7f24023ff700 1 Processor -- start 2023-10-07T09:41:46.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.320+0000 7f24023ff700 1 -- start start 2023-10-07T09:41:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.323+0000 7f24023ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f23fc156980 0x7f23fc152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.323+0000 7f24023ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc0b0120 0x7f23fc153220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.323+0000 7f24023ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc157bd0 con 0x7f23fc0afd50 2023-10-07T09:41:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.323+0000 7f24023ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23fc0b2120 con 0x7f23fc156980 2023-10-07T09:41:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.323+0000 7f24023ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc156fc0 con 0x7f23fc0b0120 2023-10-07T09:41:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.324+0000 7f23fbfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f23fc0afd50 0x7f23fc0b1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47166/0 (socket says 172.21.15.29:47166) 2023-10-07T09:41:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.324+0000 7f23fbfff700 1 -- 172.21.15.29:0/2417437683 learned_addr learned my addr 172.21.15.29:0/2417437683 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.325+0000 7f23fb7fe700 1 --2- 172.21.15.29:0/2417437683 >> v2:172.21.15.144:3300/0 conn(0x7f23fc156980 0x7f23fc152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.325+0000 7f240099c700 1 --2- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc0b0120 0x7f23fc153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.326+0000 7f23fb7fe700 1 -- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc0b0120 msgr2=0x7f23fc153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.326+0000 7f23fb7fe700 1 --2- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc0b0120 0x7f23fc153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.326+0000 7f23fb7fe700 1 -- 172.21.15.29:0/2417437683 >> v1:172.21.15.29:6789/0 conn(0x7f23fc0afd50 legacy=0x7f23fc0b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.326+0000 7f23fb7fe700 1 -- 172.21.15.29:0/2417437683 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc1572a0 con 0x7f23fc156980 2023-10-07T09:41:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.326+0000 7f23faffd700 1 -- 172.21.15.29:0/2417437683 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352163863 0 0) 0x7f23fc157bd0 con 0x7f23fc0afd50 2023-10-07T09:41:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.326+0000 7f23fb7fe700 1 --2- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc156980 0x7f23fc152ce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23f000aa90 tx=0x7f23f000a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=142c8eb621e8b8f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.327+0000 7f23faffd700 1 -- 172.21.15.29:0/2417437683 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f000c970 con 0x7f23fc156980 2023-10-07T09:41:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.327+0000 7f23faffd700 1 -- 172.21.15.29:0/2417437683 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f000cad0 con 0x7f23fc156980 2023-10-07T09:41:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.327+0000 7f23faffd700 1 -- 172.21.15.29:0/2417437683 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f000ce70 con 0x7f23fc156980 2023-10-07T09:41:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.328+0000 7f24023ff700 1 -- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc156980 msgr2=0x7f23fc152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.328+0000 7f24023ff700 1 --2- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc156980 0x7f23fc152ce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23f000aa90 tx=0x7f23f000a330 comp rx=0 tx=0).stop 2023-10-07T09:41:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.328+0000 7f24023ff700 1 -- 172.21.15.29:0/2417437683 shutdown_connections 2023-10-07T09:41:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.328+0000 7f24023ff700 1 --2- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc0b0120 0x7f23fc153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.328+0000 7f24023ff700 1 --2- 172.21.15.29:0/2417437683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc156980 0x7f23fc152ce0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.328+0000 7f24023ff700 1 -- 172.21.15.29:0/2417437683 >> 172.21.15.29:0/2417437683 conn(0x7f23fc0680a0 msgr2=0x7f23fc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.329+0000 7f24023ff700 1 -- 172.21.15.29:0/2417437683 shutdown_connections 2023-10-07T09:41:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.329+0000 7f24023ff700 1 -- 172.21.15.29:0/2417437683 wait complete. 2023-10-07T09:41:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.332+0000 7f24023ff700 1 Processor -- start 2023-10-07T09:41:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.332+0000 7f24023ff700 1 -- start start 2023-10-07T09:41:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f24023ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 0x7f23fc145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f24023ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc0b0120 0x7f23fc0adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f24023ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc156980 0x7f23fc14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f24023ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc145580 con 0x7f23fc0afd50 2023-10-07T09:41:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f24023ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc147c70 con 0x7f23fc0b0120 2023-10-07T09:41:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f24023ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc147f50 con 0x7f23fc156980 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f23fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 0x7f23fc145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f23fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 0x7f23fc145040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51134/0 (socket says 172.21.15.29:51134) 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f23fb7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc0b0120 0x7f23fc0adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f23fbfff700 1 -- 172.21.15.29:0/3899825305 learned_addr learned my addr 172.21.15.29:0/3899825305 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.334+0000 7f240099c700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc156980 0x7f23fc14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fb7fe700 1 -- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc156980 msgr2=0x7f23fc14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fb7fe700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc156980 0x7f23fc14e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fb7fe700 1 -- 172.21.15.29:0/3899825305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 msgr2=0x7f23fc145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fb7fe700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 0x7f23fc145040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fb7fe700 1 -- 172.21.15.29:0/3899825305 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f000a5d0 con 0x7f23fc0b0120 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fbfff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 0x7f23fc145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23fb7fe700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc0b0120 0x7f23fc0adff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f23f0016000 tx=0x7f23f0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f001c070 con 0x7f23fc0b0120 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f0020410 con 0x7f23fc0b0120 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f00207f0 con 0x7f23fc0b0120 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc148230 con 0x7f23fc0b0120 2023-10-07T09:41:46.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.335+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc0ae530 con 0x7f23fc0b0120 2023-10-07T09:41:46.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.336+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc04c3c0 con 0x7f23fc0b0120 2023-10-07T09:41:46.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.337+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f23f000f380 con 0x7f23fc0b0120 2023-10-07T09:41:46.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.338+0000 7f23e3fff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 0x7f23e8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.338+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6451+0+0 (secure 0 0 0) 0x7f23f0085800 con 0x7f23fc0b0120 2023-10-07T09:41:46.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.338+0000 7f23fbfff700 1 -- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 msgr2=0x7f23e8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:46.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.339+0000 7f23fbfff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 0x7f23e8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:46.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.343+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23f0052400 con 0x7f23fc0b0120 2023-10-07T09:41:46.393 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:46.392+0000 7fc3aa51e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:41:46.490 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:46.490+0000 7fdc026ea1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:41:46.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.540+0000 7f23fbfff700 1 -- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 msgr2=0x7f23e8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:46.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.540+0000 7f23fbfff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 0x7f23e8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.577+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23fc0cc960 con 0x7f23fc0b0120 2023-10-07T09:41:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.578+0000 7f23e3fff700 1 -- 172.21.15.29:0/3899825305 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16583 (secure 0 0 0) 0x7f23f0055a20 con 0x7f23fc0b0120 2023-10-07T09:41:46.579 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:46.579 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":47,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:41:46.110248+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:30:23.373202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-07T09:30:24.433861+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3859209094":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/441038837":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/2779235645":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/465296683":"2023-10-08T09:30:47.178466+0000","172.21.15.144:6832/114714":"2023-10-08T09:30:47.178466+0000","172.21.15.144:6833/114714":"2023-10-08T09:30:47.178466+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":3},{"begin":9,"length":3},{"begin":14,"length":3},{"begin":19,"length":3},{"begin":24,"length":3},{"begin":29,"length":3},{"begin":34,"length":3},{"begin":39,"length":4},{"begin":45,"length":3},{"begin":50,"length":4},{"begin":56,"length":3},{"begin":61,"length":4},{"begin":67,"length":3},{"begin":73,"length":3},{"begin":78,"length":3},{"begin":84,"length":3},{"begin":89,"length":4},{"begin":95,"length":3},{"begin":100,"length":4},{"begin":105,"length":3},{"begin":111,"length":1},{"begin":155,"length":66}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":42,"length":1},{"begin":53,"length":1},{"begin":64,"length":1},{"begin":75,"length":1},{"begin":86,"length":1},{"begin":97,"length":1},{"begin":215,"length":6}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:41:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 msgr2=0x7f23e8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 0x7f23e8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc0b0120 msgr2=0x7f23fc0adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc0b0120 0x7f23fc0adff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f23f0016000 tx=0x7f23f0017000 comp rx=0 tx=0).stop 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 shutdown_connections 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f23e80670e0 0x7f23e8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23fc156980 0x7f23fc14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23fc0afd50 0x7f23fc145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 --2- 172.21.15.29:0/3899825305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23fc0b0120 0x7f23fc0adff0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 >> 172.21.15.29:0/3899825305 conn(0x7f23fc0680a0 msgr2=0x7f23fc0aeff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 shutdown_connections 2023-10-07T09:41:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.581+0000 7f24023ff700 1 -- 172.21.15.29:0/3899825305 wait complete. 2023-10-07T09:41:46.583 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:46.580+0000 7fc3aa51e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:41:46.600 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:30:23.373202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:30:24.433861+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 47, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:41:46.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:41:46.686 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:46.686+0000 7fdc026ea1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:41:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.804+0000 7f9b173ff700 1 Processor -- start 2023-10-07T09:41:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.804+0000 7f9b173ff700 1 -- start start 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b173ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b100ae570 0x7f9b100afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b173ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b100ad4d0 con 0x7f9b100b0520 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b173ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b100ad7b0 con 0x7f9b10156180 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b173ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b100ada90 con 0x7f9b100ae570 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b1499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b100ae570 0x7f9b100afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b1499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b100ae570 0x7f9b100afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51150/0 (socket says 172.21.15.29:51150) 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b1519b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f9b10156180 0x7f9b100accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:37082/0 (socket says 172.21.15.29:37082) 2023-10-07T09:41:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b1499a700 1 -- 172.21.15.29:0/3843149125 learned_addr learned my addr 172.21.15.29:0/3843149125 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b1499a700 1 -- 172.21.15.29:0/3843149125 >> v1:172.21.15.144:6789/0 conn(0x7f9b10156180 legacy=0x7f9b100accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.806+0000 7f9b07fff700 1 -- 172.21.15.29:0/3843149125 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304198119 0 0) 0x7f9b100ad7b0 con 0x7f9b10156180 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b1499a700 1 -- 172.21.15.29:0/3843149125 >> v1:172.21.15.29:6789/0 conn(0x7f9b100b0520 legacy=0x7f9b10156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b1499a700 1 -- 172.21.15.29:0/3843149125 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b100add70 con 0x7f9b100ae570 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b1499a700 1 --2- 172.21.15.29:0/3843149125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b100ae570 0x7f9b100afc00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000bc50 tx=0x7f9b0000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850d74e258b5c359 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b07fff700 1 -- 172.21.15.29:0/3843149125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0000be60 con 0x7f9b100ae570 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b07fff700 1 -- 172.21.15.29:0/3843149125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0000de00 con 0x7f9b100ae570 2023-10-07T09:41:46.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.807+0000 7f9b07fff700 1 -- 172.21.15.29:0/3843149125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b000196b0 con 0x7f9b100ae570 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 -- 172.21.15.29:0/3843149125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b100ae570 msgr2=0x7f9b100afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3843149125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b100ae570 0x7f9b100afc00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000bc50 tx=0x7f9b0000b4f0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 -- 172.21.15.29:0/3843149125 shutdown_connections 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3843149125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b100ae570 0x7f9b100afc00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 -- 172.21.15.29:0/3843149125 >> 172.21.15.29:0/3843149125 conn(0x7f9b10067610 msgr2=0x7f9b10067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 -- 172.21.15.29:0/3843149125 shutdown_connections 2023-10-07T09:41:46.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.808+0000 7f9b173ff700 1 -- 172.21.15.29:0/3843149125 wait complete. 2023-10-07T09:41:46.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.810+0000 7f9b173ff700 1 Processor -- start 2023-10-07T09:41:46.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.810+0000 7f9b173ff700 1 -- start start 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.811+0000 7f9b173ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 0x7f9b1014f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.811+0000 7f9b173ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b100b0520 0x7f9b101539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.811+0000 7f9b173ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b10156180 0x7f9b100c27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.811+0000 7f9b173ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10151000 con 0x7f9b10156180 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.811+0000 7f9b173ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1014ff60 con 0x7f9b100ae570 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.811+0000 7f9b173ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10150240 con 0x7f9b100b0520 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 0x7f9b1014f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1499a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b100b0520 0x7f9b101539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 0x7f9b1014f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:33112/0 (socket says 172.21.15.29:33112) 2023-10-07T09:41:46.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 -- 172.21.15.29:0/3328487503 learned_addr learned my addr 172.21.15.29:0/3328487503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:46.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1599c700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b10156180 0x7f9b100c27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 -- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b100b0520 msgr2=0x7f9b101539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b100b0520 0x7f9b101539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 -- 172.21.15.29:0/3328487503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b10156180 msgr2=0x7f9b100c27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b10156180 0x7f9b100c27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1519b700 1 -- 172.21.15.29:0/3328487503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0000b790 con 0x7f9b100ae570 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.812+0000 7f9b1499a700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b100b0520 0x7f9b101539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.813+0000 7f9b1519b700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 0x7f9b1014f6a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00c560 tx=0x7f9b0c007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.813+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c005a70 con 0x7f9b100ae570 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.813+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c005bd0 con 0x7f9b100ae570 2023-10-07T09:41:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.813+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b10150520 con 0x7f9b100ae570 2023-10-07T09:41:46.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.813+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0070d0 con 0x7f9b100ae570 2023-10-07T09:41:46.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.813+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b10150ae0 con 0x7f9b100ae570 2023-10-07T09:41:46.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.814+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1004c3c0 con 0x7f9b100ae570 2023-10-07T09:41:46.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.815+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f9b0c0074c0 con 0x7f9b100ae570 2023-10-07T09:41:46.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.817+0000 7f9b057fa700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 0x7f9af4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:46.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.817+0000 7f9b1499a700 1 -- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 msgr2=0x7f9af4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:46.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.817+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6451+0+0 (secure 0 0 0) 0x7f9b0c083fb0 con 0x7f9b100ae570 2023-10-07T09:41:46.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.817+0000 7f9b1499a700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 0x7f9af4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:46.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:46.820+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b0c050c10 con 0x7f9b100ae570 2023-10-07T09:41:46.999 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:46.998+0000 7fc3aa51e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:41:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.018+0000 7f9b1499a700 1 -- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 msgr2=0x7f9af4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.018+0000 7f9b1499a700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 0x7f9af4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.030+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9b100ccfb0 con 0x7f9b100ae570 2023-10-07T09:41:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.031+0000 7f9b057fa700 1 -- 172.21.15.29:0/3328487503 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9b0c017020 con 0x7f9b100ae570 2023-10-07T09:41:47.031 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:41:47.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 msgr2=0x7f9af4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 0x7f9af4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 msgr2=0x7f9b1014f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 0x7f9b1014f6a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00c560 tx=0x7f9b0c007f60 comp rx=0 tx=0).stop 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 shutdown_connections 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f9af40670e0 0x7f9af4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b10156180 0x7f9b100c27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b100ae570 0x7f9b1014f6a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 --2- 172.21.15.29:0/3328487503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b100b0520 0x7f9b101539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 >> 172.21.15.29:0/3328487503 conn(0x7f9b10067610 msgr2=0x7f9b10154ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 shutdown_connections 2023-10-07T09:41:47.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.033+0000 7f9b173ff700 1 -- 172.21.15.29:0/3328487503 wait complete. 2023-10-07T09:41:47.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:41:47.133 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:47.133+0000 7fdc026ea1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:41:47.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.266+0000 7f91263ff700 1 Processor -- start 2023-10-07T09:41:47.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.267+0000 7f91263ff700 1 -- start start 2023-10-07T09:41:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.268+0000 7f91263ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159750 0x7f91200d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.268+0000 7f91263ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91200ce8a0 0x7f91200c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.268+0000 7f91263ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91200cbf20 con 0x7f9120159b20 2023-10-07T09:41:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.268+0000 7f91263ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91200cc200 con 0x7f91200ce8a0 2023-10-07T09:41:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.268+0000 7f91263ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91200cc4e0 con 0x7f9120159750 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91200ce8a0 0x7f91200c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91200ce8a0 0x7f91200c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51160/0 (socket says 172.21.15.29:51160) 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 -- 172.21.15.29:0/110684437 learned_addr learned my addr 172.21.15.29:0/110684437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f911ffff700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159750 0x7f91200d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 -- 172.21.15.29:0/110684437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159750 msgr2=0x7f91200d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159750 0x7f91200d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 -- 172.21.15.29:0/110684437 >> v1:172.21.15.144:6789/0 conn(0x7f9120159b20 legacy=0x7f91200ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f911effd700 1 -- 172.21.15.29:0/110684437 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692520377 0 0) 0x7f91200cbf20 con 0x7f9120159b20 2023-10-07T09:41:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f912499c700 1 -- 172.21.15.29:0/110684437 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91200cc7c0 con 0x7f91200ce8a0 2023-10-07T09:41:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.269+0000 7f911ffff700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159750 0x7f91200d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f912499c700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91200ce8a0 0x7f91200c5460 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9118009f20 tx=0x7f9118009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f1b03aad0d8b82 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f911effd700 1 -- 172.21.15.29:0/110684437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911800e070 con 0x7f91200ce8a0 2023-10-07T09:41:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f911effd700 1 -- 172.21.15.29:0/110684437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911800ba80 con 0x7f91200ce8a0 2023-10-07T09:41:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f911effd700 1 -- 172.21.15.29:0/110684437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180092e0 con 0x7f91200ce8a0 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f91263ff700 1 -- 172.21.15.29:0/110684437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91200ce8a0 msgr2=0x7f91200c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f91263ff700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91200ce8a0 0x7f91200c5460 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9118009f20 tx=0x7f9118009840 comp rx=0 tx=0).stop 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f91263ff700 1 -- 172.21.15.29:0/110684437 shutdown_connections 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f91263ff700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91200ce8a0 0x7f91200c5460 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f91263ff700 1 --2- 172.21.15.29:0/110684437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159750 0x7f91200d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.270+0000 7f91263ff700 1 -- 172.21.15.29:0/110684437 >> 172.21.15.29:0/110684437 conn(0x7f9120063a10 msgr2=0x7f91200d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.271+0000 7f91263ff700 1 -- 172.21.15.29:0/110684437 shutdown_connections 2023-10-07T09:41:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.271+0000 7f91263ff700 1 -- 172.21.15.29:0/110684437 wait complete. 2023-10-07T09:41:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.273+0000 7f91263ff700 1 Processor -- start 2023-10-07T09:41:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.273+0000 7f91263ff700 1 -- start start 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f91263ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91200ce8a0 0x7f912014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f91263ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9120159750 0x7f9120152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f91263ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159b20 0x7f91200c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f91263ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91201577e0 con 0x7f9120159750 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f91263ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91201525d0 con 0x7f91200ce8a0 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f91263ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9120157490 con 0x7f9120159b20 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f911ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91200ce8a0 0x7f912014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f911f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9120159750 0x7f9120152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.274+0000 7f911ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91200ce8a0 0x7f912014c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:33118/0 (socket says 172.21.15.29:33118) 2023-10-07T09:41:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911ffff700 1 -- 172.21.15.29:0/1891746184 learned_addr learned my addr 172.21.15.29:0/1891746184 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f912499c700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159b20 0x7f91200c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911f7fe700 1 -- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159b20 msgr2=0x7f91200c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911f7fe700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159b20 0x7f91200c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911f7fe700 1 -- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91200ce8a0 msgr2=0x7f912014c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911f7fe700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91200ce8a0 0x7f912014c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911f7fe700 1 -- 172.21.15.29:0/1891746184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9118009ae0 con 0x7f9120159750 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f911f7fe700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9120159750 0x7f9120152a70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f911400c510 tx=0x7f9114007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400ce80 con 0x7f9120159750 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9114011590 con 0x7f9120159750 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114011950 con 0x7f9120159750 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.275+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91200c83f0 con 0x7f9120159750 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.276+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91200b37e0 con 0x7f9120159750 2023-10-07T09:41:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.277+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9120157990 con 0x7f9120159750 2023-10-07T09:41:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.278+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f9114011d40 con 0x7f9120159750 2023-10-07T09:41:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.279+0000 7f9107fff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 0x7f910c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.279+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f9114083ff0 con 0x7f9120159750 2023-10-07T09:41:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.279+0000 7f911ffff700 1 -- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 msgr2=0x7f910c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.279+0000 7f911ffff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 0x7f910c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.283+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9120157990 con 0x7f9120159750 2023-10-07T09:41:47.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.480+0000 7f911ffff700 1 -- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 msgr2=0x7f910c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:47.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.480+0000 7f911ffff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 0x7f910c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.500+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9120157990 con 0x7f9120159750 2023-10-07T09:41:47.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.501+0000 7f9107fff700 1 -- 172.21.15.29:0/1891746184 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9120157990 con 0x7f9120159750 2023-10-07T09:41:47.501 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:41:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 msgr2=0x7f910c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 0x7f910c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9120159750 msgr2=0x7f9120152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9120159750 0x7f9120152a70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f911400c510 tx=0x7f9114007ec0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 shutdown_connections 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f910c0670e0 0x7f910c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9120159b20 0x7f91200c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91200ce8a0 0x7f912014c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 --2- 172.21.15.29:0/1891746184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9120159750 0x7f9120152a70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 >> 172.21.15.29:0/1891746184 conn(0x7f9120063a10 msgr2=0x7f912014d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 shutdown_connections 2023-10-07T09:41:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.503+0000 7f91263ff700 1 -- 172.21.15.29:0/1891746184 wait complete. 2023-10-07T09:41:47.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:41:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.724+0000 7fd6c37ff700 1 Processor -- start 2023-10-07T09:41:47.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.725+0000 7fd6c37ff700 1 -- start start 2023-10-07T09:41:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.726+0000 7fd6c37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc153cc0 0x7fd6bc159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.726+0000 7fd6c37ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd6bc14cca0 0x7fd6bc159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.726+0000 7fd6c37ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc157790 con 0x7fd6bc14c8d0 2023-10-07T09:41:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.726+0000 7fd6c37ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6bc0b0950 con 0x7fd6bc14cca0 2023-10-07T09:41:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.726+0000 7fd6c37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc157580 con 0x7fd6bc153cc0 2023-10-07T09:41:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c159b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd6bc14c8d0 0x7fd6bc0afed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47170/0 (socket says 172.21.15.29:47170) 2023-10-07T09:41:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c159b700 1 -- 172.21.15.29:0/1881450438 learned_addr learned my addr 172.21.15.29:0/1881450438 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c1d9c700 1 --2- 172.21.15.29:0/1881450438 >> v2:172.21.15.144:3300/0 conn(0x7fd6bc14cca0 0x7fd6bc159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c0d9a700 1 --2- 172.21.15.29:0/1881450438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc153cc0 0x7fd6bc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6b3fff700 1 -- 172.21.15.29:0/1881450438 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213837680 0 0) 0x7fd6bc157790 con 0x7fd6bc14c8d0 2023-10-07T09:41:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6b3fff700 1 -- 172.21.15.29:0/1881450438 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a8003570 con 0x7fd6bc14c8d0 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c0d9a700 1 -- 172.21.15.29:0/1881450438 >> v2:172.21.15.144:3300/0 conn(0x7fd6bc14cca0 msgr2=0x7fd6bc159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c0d9a700 1 --2- 172.21.15.29:0/1881450438 >> v2:172.21.15.144:3300/0 conn(0x7fd6bc14cca0 0x7fd6bc159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c0d9a700 1 -- 172.21.15.29:0/1881450438 >> v1:172.21.15.29:6789/0 conn(0x7fd6bc14c8d0 legacy=0x7fd6bc0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c0d9a700 1 -- 172.21.15.29:0/1881450438 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc157a70 con 0x7fd6bc153cc0 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.727+0000 7fd6c1d9c700 1 --2- 172.21.15.29:0/1881450438 >> v2:172.21.15.144:3300/0 conn(0x7fd6bc14cca0 0x7fd6bc159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.728+0000 7fd6c0d9a700 1 --2- 172.21.15.29:0/1881450438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc153cc0 0x7fd6bc159680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc1576c0 tx=0x7fd6ac00a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e17be11793b01f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.728+0000 7fd6b3fff700 1 -- 172.21.15.29:0/1881450438 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00aea0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.728+0000 7fd6b3fff700 1 -- 172.21.15.29:0/1881450438 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac00c970 con 0x7fd6bc153cc0 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.728+0000 7fd6b3fff700 1 -- 172.21.15.29:0/1881450438 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00cd30 con 0x7fd6bc153cc0 2023-10-07T09:41:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 -- 172.21.15.29:0/1881450438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc153cc0 msgr2=0x7fd6bc159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/1881450438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc153cc0 0x7fd6bc159680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc1576c0 tx=0x7fd6ac00a330 comp rx=0 tx=0).stop 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 -- 172.21.15.29:0/1881450438 shutdown_connections 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/1881450438 >> v2:172.21.15.144:3300/0 conn(0x7fd6bc14cca0 0x7fd6bc159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/1881450438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc153cc0 0x7fd6bc159680 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 -- 172.21.15.29:0/1881450438 >> 172.21.15.29:0/1881450438 conn(0x7fd6bc067610 msgr2=0x7fd6bc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 -- 172.21.15.29:0/1881450438 shutdown_connections 2023-10-07T09:41:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.729+0000 7fd6c37ff700 1 -- 172.21.15.29:0/1881450438 wait complete. 2023-10-07T09:41:47.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.731+0000 7fd6c37ff700 1 Processor -- start 2023-10-07T09:41:47.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.731+0000 7fd6c37ff700 1 -- start start 2023-10-07T09:41:47.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.732+0000 7fd6c37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc14c8d0 0x7fd6bc14a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.732+0000 7fd6c37ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6bc14cca0 0x7fd6bc166b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c37ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 0x7fd6bc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c37ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc149ea0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c37ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc14b840 con 0x7fd6bc14cca0 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc14a870 con 0x7fd6bc14c8d0 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 0x7fd6bc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c159b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc14c8d0 0x7fd6bc14a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 0x7fd6bc155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51180/0 (socket says 172.21.15.29:51180) 2023-10-07T09:41:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 -- 172.21.15.29:0/3399196158 learned_addr learned my addr 172.21.15.29:0/3399196158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:47.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c0d9a700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6bc14cca0 0x7fd6bc166b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:47.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 -- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc14c8d0 msgr2=0x7fd6bc14a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc14c8d0 0x7fd6bc14a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 -- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6bc14cca0 msgr2=0x7fd6bc166b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6bc14cca0 0x7fd6bc166b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c1d9c700 1 -- 172.21.15.29:0/3399196158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b4009a20 con 0x7fd6bc153cc0 2023-10-07T09:41:47.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c159b700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc14c8d0 0x7fd6bc14a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:47.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.733+0000 7fd6c0d9a700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6bc14cca0 0x7fd6bc166b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:47.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.734+0000 7fd6c1d9c700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 0x7fd6bc155df0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009fc0 tx=0x7fd6b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.734+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b400cbd0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.734+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b400f040 con 0x7fd6bc153cc0 2023-10-07T09:41:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.734+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac00a5d0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.734+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4014480 con 0x7fd6bc153cc0 2023-10-07T09:41:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.734+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc16a600 con 0x7fd6bc153cc0 2023-10-07T09:41:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.735+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc04c3c0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.736+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7fd6b4003020 con 0x7fd6bc153cc0 2023-10-07T09:41:47.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.737+0000 7fd6b17fa700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 0x7fd6a0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:47.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.737+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fd6b4085770 con 0x7fd6bc153cc0 2023-10-07T09:41:47.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.737+0000 7fd6c159b700 1 -- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 msgr2=0x7fd6a0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:47.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.737+0000 7fd6c159b700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 0x7fd6a0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:47.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.741+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6b40521c0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.797 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:47.796+0000 7fc3aa51e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:41:47.884 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:47.883+0000 7fc3aa51e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:41:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.938+0000 7fd6c159b700 1 -- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 msgr2=0x7fd6a0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.938+0000 7fd6c159b700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 0x7fd6a0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:47.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.949+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6bc149fe0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.949+0000 7fd6b17fa700 1 -- 172.21.15.29:0/3399196158 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6b40557e0 con 0x7fd6bc153cc0 2023-10-07T09:41:47.950 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:41:47.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 msgr2=0x7fd6a0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 0x7fd6a0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 msgr2=0x7fd6bc155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 0x7fd6bc155df0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009fc0 tx=0x7fd6b400b040 comp rx=0 tx=0).stop 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 shutdown_connections 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fd6a00670e0 0x7fd6a0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6bc153cc0 0x7fd6bc155df0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6bc14c8d0 0x7fd6bc14a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 --2- 172.21.15.29:0/3399196158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6bc14cca0 0x7fd6bc166b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 >> 172.21.15.29:0/3399196158 conn(0x7fd6bc067610 msgr2=0x7fd6bc159610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 shutdown_connections 2023-10-07T09:41:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:47.952+0000 7fd6c37ff700 1 -- 172.21.15.29:0/3399196158 wait complete. 2023-10-07T09:41:47.961 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:47.962+0000 7fdc026ea1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:41:47.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:41:48.039 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.038+0000 7fc3aa51e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:41:48.059 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.060+0000 7fdc026ea1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:41:48.125 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.124+0000 7fc3aa51e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:41:48.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.167+0000 7fa6ea1ff700 1 Processor -- start 2023-10-07T09:41:48.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.168+0000 7fa6ea1ff700 1 -- start start 2023-10-07T09:41:48.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6ea1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 0x7fa6e415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6ea1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 0x7fa6e40af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6ea1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e414d790 con 0x7fa6e40ad370 2023-10-07T09:41:48.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6ea1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e415a880 con 0x7fa6e4159ec0 2023-10-07T09:41:48.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6ea1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40ae5d0 con 0x7fa6e40af400 2023-10-07T09:41:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6e3fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 0x7fa6e40af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6e2ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 0x7fa6e415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6e2ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 0x7fa6e415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:33144/0 (socket says 172.21.15.29:33144) 2023-10-07T09:41:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6e3fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 0x7fa6e40af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:60724/0 (socket says 172.21.15.29:60724) 2023-10-07T09:41:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.169+0000 7fa6e2ffd700 1 -- 172.21.15.29:0/3682220827 learned_addr learned my addr 172.21.15.29:0/3682220827 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e27fc700 1 -- 172.21.15.29:0/3682220827 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921640063 0 0) 0x7fa6e414d790 con 0x7fa6e40ad370 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e27fc700 1 -- 172.21.15.29:0/3682220827 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc003570 con 0x7fa6e40ad370 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e3fff700 1 -- 172.21.15.29:0/3682220827 >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 msgr2=0x7fa6e415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e3fff700 1 --2- 172.21.15.29:0/3682220827 >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 0x7fa6e415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e3fff700 1 -- 172.21.15.29:0/3682220827 >> v1:172.21.15.144:6789/0 conn(0x7fa6e40ad370 legacy=0x7fa6e40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e3fff700 1 -- 172.21.15.29:0/3682220827 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e40ad740 con 0x7fa6e40af400 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e2ffd700 1 --2- 172.21.15.29:0/3682220827 >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 0x7fa6e415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e3fff700 1 --2- 172.21.15.29:0/3682220827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 0x7fa6e40af840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00d0d0 tx=0x7fa6dc00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78a072c4f91e01a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e27fc700 1 -- 172.21.15.29:0/3682220827 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc010070 con 0x7fa6e40af400 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e27fc700 1 -- 172.21.15.29:0/3682220827 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc00b2e0 con 0x7fa6e40af400 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.170+0000 7fa6e27fc700 1 -- 172.21.15.29:0/3682220827 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00dcd0 con 0x7fa6e40af400 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/3682220827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 msgr2=0x7fa6e40af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/3682220827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 0x7fa6e40af840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00d0d0 tx=0x7fa6dc00b990 comp rx=0 tx=0).stop 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/3682220827 shutdown_connections 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/3682220827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e40af400 0x7fa6e40af840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/3682220827 >> v2:172.21.15.144:3300/0 conn(0x7fa6e4159ec0 0x7fa6e415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/3682220827 >> 172.21.15.29:0/3682220827 conn(0x7fa6e40680a0 msgr2=0x7fa6e40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/3682220827 shutdown_connections 2023-10-07T09:41:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.171+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/3682220827 wait complete. 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.173+0000 7fa6ea1ff700 1 Processor -- start 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.173+0000 7fa6ea1ff700 1 -- start start 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6ea1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 0x7fa6e40c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6ea1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e40af400 0x7fa6e40c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6ea1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e4159ec0 0x7fa6e40ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6ea1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40a5800 con 0x7fa6e40ad370 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6ea1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40caa90 con 0x7fa6e40af400 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6ea1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40c9e80 con 0x7fa6e4159ec0 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6e37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 0x7fa6e40c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6e37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 0x7fa6e40c92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51190/0 (socket says 172.21.15.29:51190) 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6e37fe700 1 -- 172.21.15.29:0/668582185 learned_addr learned my addr 172.21.15.29:0/668582185 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6e2ffd700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e40af400 0x7fa6e40c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.174+0000 7fa6e3fff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e4159ec0 0x7fa6e40ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e37fe700 1 -- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e4159ec0 msgr2=0x7fa6e40ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e37fe700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e4159ec0 0x7fa6e40ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e37fe700 1 -- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e40af400 msgr2=0x7fa6e40c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e37fe700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e40af400 0x7fa6e40c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e37fe700 1 -- 172.21.15.29:0/668582185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80094f0 con 0x7fa6e40ad370 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e2ffd700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e40af400 0x7fa6e40c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6e37fe700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 0x7fa6e40c92a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c790 tx=0x7fa6d4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4010510 con 0x7fa6e40ad370 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4010670 con 0x7fa6e40ad370 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6dc00bc30 con 0x7fa6e40ad370 2023-10-07T09:41:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4010a10 con 0x7fa6e40ad370 2023-10-07T09:41:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.175+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e40ca440 con 0x7fa6e40ad370 2023-10-07T09:41:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.177+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e404c3c0 con 0x7fa6e40ad370 2023-10-07T09:41:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.177+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7fa6d401b3b0 con 0x7fa6e40ad370 2023-10-07T09:41:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.178+0000 7fa6cb7fe700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.178+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fa6d4084580 con 0x7fa6e40ad370 2023-10-07T09:41:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.178+0000 7fa6e2ffd700 1 -- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 msgr2=0x7fa6d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.178+0000 7fa6e2ffd700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:48.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.182+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6d40510e0 con 0x7fa6e40ad370 2023-10-07T09:41:48.229 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.229+0000 7fdc026ea1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:41:48.320 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.320+0000 7fdc026ea1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:41:48.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.341+0000 7fc3aa51e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:41:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.379+0000 7fa6e2ffd700 1 -- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 msgr2=0x7fa6d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.379+0000 7fa6e2ffd700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:48.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.431+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e40adad0 con 0x7fa6e40ad370 2023-10-07T09:41:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.432+0000 7fa6cb7fe700 1 -- 172.21.15.29:0/668582185 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4600 (secure 0 0 0) 0x7fa6d4054700 con 0x7fa6e40ad370 2023-10-07T09:41:48.432 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:48.432 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:30:25.453869+0000","modified":"2023-10-07T09:30:29.396464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6836/3596704883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3596704883},{"type":"v1","addr":"172.21.15.144:6836","nonce":3596704883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-07T09:41:48.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 msgr2=0x7fa6d0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:48.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 msgr2=0x7fa6e40c92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 0x7fa6e40c92a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c790 tx=0x7fa6d4007e10 comp rx=0 tx=0).stop 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 shutdown_connections 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e4159ec0 0x7fa6e40ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e40ad370 0x7fa6e40c92a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 --2- 172.21.15.29:0/668582185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e40af400 0x7fa6e40c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 >> 172.21.15.29:0/668582185 conn(0x7fa6e40680a0 msgr2=0x7fa6e406ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 shutdown_connections 2023-10-07T09:41:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.434+0000 7fa6ea1ff700 1 -- 172.21.15.29:0/668582185 wait complete. 2023-10-07T09:41:48.436 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-07T09:41:48.436 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.436+0000 7fc3aa51e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:41:48.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:41:48.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.529+0000 7fc3aa51e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:41:48.547 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.547+0000 7fdc026ea1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:41:48.625 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.624+0000 7fc3aa51e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:41:48.648 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.648+0000 7fdc026ea1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:41:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.656+0000 7fe0da3ff700 1 Processor -- start 2023-10-07T09:41:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.657+0000 7fe0da3ff700 1 -- start start 2023-10-07T09:41:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0da3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d40ae8b0 0x7fe0d40b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0da3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe0d40b1840 0x7fe0d4157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0da3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40b1ca0 con 0x7fe0d4157290 2023-10-07T09:41:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0da3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d4158ad0 con 0x7fe0d40b1840 2023-10-07T09:41:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0da3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d415b140 con 0x7fe0d40ae8b0 2023-10-07T09:41:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0d899c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe0d40b1840 0x7fe0d4157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0d37fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d40ae8b0 0x7fe0d40b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0d3fff700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe0d4157290 0x7fe0d40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:37108/0 (socket says 172.21.15.29:37108) 2023-10-07T09:41:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.658+0000 7fe0d3fff700 1 -- 172.21.15.29:0/729921697 learned_addr learned my addr 172.21.15.29:0/729921697 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d2ffd700 1 -- 172.21.15.29:0/729921697 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788641497 0 0) 0x7fe0c4001110 con 0x7fe0d4157290 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d2ffd700 1 -- 172.21.15.29:0/729921697 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0bc003570 con 0x7fe0d4157290 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d899c700 1 -- 172.21.15.29:0/729921697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d40ae8b0 msgr2=0x7fe0d40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d899c700 1 --2- 172.21.15.29:0/729921697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d40ae8b0 0x7fe0d40b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d899c700 1 -- 172.21.15.29:0/729921697 >> v1:172.21.15.144:6789/0 conn(0x7fe0d4157290 legacy=0x7fe0d40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d899c700 1 -- 172.21.15.29:0/729921697 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d415b420 con 0x7fe0d40b1840 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d37fe700 1 --2- 172.21.15.29:0/729921697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d40ae8b0 0x7fe0d40b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d899c700 1 --2- 172.21.15.29:0/729921697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d4157660 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009e60 tx=0x7fe0cc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40549dd8653c7105 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d2ffd700 1 -- 172.21.15.29:0/729921697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00e070 con 0x7fe0d40b1840 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d2ffd700 1 -- 172.21.15.29:0/729921697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00bab0 con 0x7fe0d40b1840 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.659+0000 7fe0d2ffd700 1 -- 172.21.15.29:0/729921697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00be70 con 0x7fe0d40b1840 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 -- 172.21.15.29:0/729921697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 msgr2=0x7fe0d4157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/729921697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d4157660 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009e60 tx=0x7fe0cc009780 comp rx=0 tx=0).stop 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 -- 172.21.15.29:0/729921697 shutdown_connections 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/729921697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d4157660 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/729921697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d40ae8b0 0x7fe0d40b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 -- 172.21.15.29:0/729921697 >> 172.21.15.29:0/729921697 conn(0x7fe0d40680a0 msgr2=0x7fe0d40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 -- 172.21.15.29:0/729921697 shutdown_connections 2023-10-07T09:41:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.660+0000 7fe0da3ff700 1 -- 172.21.15.29:0/729921697 wait complete. 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.662+0000 7fe0da3ff700 1 Processor -- start 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.662+0000 7fe0da3ff700 1 -- start start 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0da3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0da3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0d4157290 0x7fe0d415e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0da3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d415ec20 0x7fe0d40b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0da3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d40c8960 con 0x7fe0d40b1840 2023-10-07T09:41:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0da3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d415de10 con 0x7fe0d4157290 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0da3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d415e2a0 con 0x7fe0d415ec20 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d40c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51198/0 (socket says 172.21.15.29:51198) 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 -- 172.21.15.29:0/4008854516 learned_addr learned my addr 172.21.15.29:0/4008854516 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d37fe700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0d4157290 0x7fe0d415e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d899c700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d415ec20 0x7fe0d40b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 -- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d415ec20 msgr2=0x7fe0d40b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d415ec20 0x7fe0d40b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 -- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0d4157290 msgr2=0x7fe0d415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0d4157290 0x7fe0d415e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.663+0000 7fe0d3fff700 1 -- 172.21.15.29:0/4008854516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c8009420 con 0x7fe0d40b1840 2023-10-07T09:41:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.664+0000 7fe0d3fff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d40c8420 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400c590 tx=0x7fe0c4007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.664+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4005b30 con 0x7fe0d40b1840 2023-10-07T09:41:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.664+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c4005cb0 con 0x7fe0d40b1840 2023-10-07T09:41:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.664+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4007320 con 0x7fe0d40b1840 2023-10-07T09:41:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.664+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc009a20 con 0x7fe0d40b1840 2023-10-07T09:41:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.664+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d415d4e0 con 0x7fe0d40b1840 2023-10-07T09:41:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.665+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d404c3c0 con 0x7fe0d40b1840 2023-10-07T09:41:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.666+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7fe0c401b3b0 con 0x7fe0d40b1840 2023-10-07T09:41:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.667+0000 7fe0bbfff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 0x7fe0c0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.667+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fe0c4083f60 con 0x7fe0d40b1840 2023-10-07T09:41:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.667+0000 7fe0d37fe700 1 -- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 msgr2=0x7fe0c0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.667+0000 7fe0d37fe700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 0x7fe0c0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:48.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.671+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe0c4050ac0 con 0x7fe0d40b1840 2023-10-07T09:41:48.750 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.749+0000 7fdc026ea1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:41:48.811 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.810+0000 7fc3aa51e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:41:48.849 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:48.848+0000 7fdc026ea1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:41:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.867+0000 7fe0d37fe700 1 -- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 msgr2=0x7fe0c0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.867+0000 7fe0d37fe700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 0x7fe0c0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:48.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.874+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0d40b1ca0 con 0x7fe0d40b1840 2023-10-07T09:41:48.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.876+0000 7fe0bbfff700 1 -- 172.21.15.29:0/4008854516 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16991 (secure 0 0 0) 0x7fe0d40b1ca0 con 0x7fe0d40b1840 2023-10-07T09:41:48.877 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:48.877 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:41:47.095737+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:30:23.373202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-07T09:30:24.433861+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3859209094":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/441038837":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/2779235645":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/465296683":"2023-10-08T09:30:47.178466+0000","172.21.15.144:6832/114714":"2023-10-08T09:30:47.178466+0000","172.21.15.144:6833/114714":"2023-10-08T09:30:47.178466+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":108,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":4,"length":2},{"begin":9,"length":2},{"begin":14,"length":2},{"begin":19,"length":2},{"begin":24,"length":2},{"begin":29,"length":2},{"begin":34,"length":2},{"begin":39,"length":2},{"begin":45,"length":2},{"begin":50,"length":3},{"begin":56,"length":2},{"begin":61,"length":2},{"begin":67,"length":2},{"begin":73,"length":1},{"begin":78,"length":2},{"begin":84,"length":1},{"begin":89,"length":2},{"begin":95,"length":1},{"begin":100,"length":3},{"begin":105,"length":2},{"begin":111,"length":1},{"begin":155,"length":40}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:41:48.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 msgr2=0x7fe0c0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:48.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 0x7fe0c0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 msgr2=0x7fe0d40c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:48.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d40c8420 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400c590 tx=0x7fe0c4007c70 comp rx=0 tx=0).stop 2023-10-07T09:41:48.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 shutdown_connections 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fe0c00670e0 0x7fe0c0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0d415ec20 0x7fe0d40b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d40b1840 0x7fe0d40c8420 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 --2- 172.21.15.29:0/4008854516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0d4157290 0x7fe0d415e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.878+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 >> 172.21.15.29:0/4008854516 conn(0x7fe0d40680a0 msgr2=0x7fe0d4152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.879+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 shutdown_connections 2023-10-07T09:41:48.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:48.879+0000 7fe0da3ff700 1 -- 172.21.15.29:0/4008854516 wait complete. 2023-10-07T09:41:48.898 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:41:48.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:41:48.908 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:48.907+0000 7fc3aa51e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:41:49.055 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:49.054+0000 7fdc026ea1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:41:49.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.096+0000 7f7b927ff700 1 Processor -- start 2023-10-07T09:41:49.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.096+0000 7f7b927ff700 1 -- start start 2023-10-07T09:41:49.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.097+0000 7f7b927ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b8c159b20 0x7f7b8c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.097+0000 7f7b927ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b8c0ce8a0 0x7f7b8c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b927ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8c0cbf20 con 0x7f7b8c159750 2023-10-07T09:41:49.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b927ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c0cc200 con 0x7f7b8c159b20 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b927ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c0cc4e0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b8c159b20 0x7f7b8c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b8c159b20 0x7f7b8c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51204/0 (socket says 172.21.15.29:51204) 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 -- 172.21.15.29:0/1685967955 learned_addr learned my addr 172.21.15.29:0/1685967955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b90d9c700 1 --2- 172.21.15.29:0/1685967955 >> v2:172.21.15.144:3300/0 conn(0x7f7b8c0ce8a0 0x7f7b8c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 -- 172.21.15.29:0/1685967955 >> v2:172.21.15.144:3300/0 conn(0x7f7b8c0ce8a0 msgr2=0x7f7b8c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 --2- 172.21.15.29:0/1685967955 >> v2:172.21.15.144:3300/0 conn(0x7f7b8c0ce8a0 0x7f7b8c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 -- 172.21.15.29:0/1685967955 >> v1:172.21.15.29:6789/0 conn(0x7f7b8c159750 legacy=0x7f7b8c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8b7fe700 1 -- 172.21.15.29:0/1685967955 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c0cc7c0 con 0x7f7b8c159b20 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b90d9c700 1 --2- 172.21.15.29:0/1685967955 >> v2:172.21.15.144:3300/0 conn(0x7f7b8c0ce8a0 0x7f7b8c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.098+0000 7f7b8affd700 1 -- 172.21.15.29:0/1685967955 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687689457 0 0) 0x7f7b8c0cbf20 con 0x7f7b8c159750 2023-10-07T09:41:49.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.099+0000 7f7b8b7fe700 1 --2- 172.21.15.29:0/1685967955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c159b20 0x7f7b8c0d10a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c15e510 tx=0x7f7b80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547e06ece9faa4c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.099+0000 7f7b8affd700 1 -- 172.21.15.29:0/1685967955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000b860 con 0x7f7b8c159b20 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.099+0000 7f7b8affd700 1 -- 172.21.15.29:0/1685967955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8000b9c0 con 0x7f7b8c159b20 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.099+0000 7f7b8affd700 1 -- 172.21.15.29:0/1685967955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000bd60 con 0x7f7b8c159b20 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.099+0000 7f7b927ff700 1 -- 172.21.15.29:0/1685967955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c159b20 msgr2=0x7f7b8c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.099+0000 7f7b927ff700 1 --2- 172.21.15.29:0/1685967955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c159b20 0x7f7b8c0d10a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c15e510 tx=0x7f7b80009250 comp rx=0 tx=0).stop 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.100+0000 7f7b927ff700 1 -- 172.21.15.29:0/1685967955 shutdown_connections 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.100+0000 7f7b927ff700 1 --2- 172.21.15.29:0/1685967955 >> v2:172.21.15.144:3300/0 conn(0x7f7b8c0ce8a0 0x7f7b8c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.100+0000 7f7b927ff700 1 --2- 172.21.15.29:0/1685967955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c159b20 0x7f7b8c0d10a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.100+0000 7f7b927ff700 1 -- 172.21.15.29:0/1685967955 >> 172.21.15.29:0/1685967955 conn(0x7f7b8c063a10 msgr2=0x7f7b8c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.100+0000 7f7b927ff700 1 -- 172.21.15.29:0/1685967955 shutdown_connections 2023-10-07T09:41:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.100+0000 7f7b927ff700 1 -- 172.21.15.29:0/1685967955 wait complete. 2023-10-07T09:41:49.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.102+0000 7f7b927ff700 1 Processor -- start 2023-10-07T09:41:49.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.102+0000 7f7b927ff700 1 -- start start 2023-10-07T09:41:49.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b927ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 0x7f7b8c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b927ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b8c159750 0x7f7b8c169b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b927ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c159b20 0x7f7b8c16a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b927ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c142dd0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b927ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0c8640 con 0x7f7b8c159b20 2023-10-07T09:41:49.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b927ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c15c4d0 con 0x7f7b8c159750 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b8bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 0x7f7b8c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b8bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 0x7f7b8c159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51210/0 (socket says 172.21.15.29:51210) 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b8bfff700 1 -- 172.21.15.29:0/3348010824 learned_addr learned my addr 172.21.15.29:0/3348010824 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b8b7fe700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b8c159750 0x7f7b8c169b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.104+0000 7f7b90d9c700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c159b20 0x7f7b8c16a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b8bfff700 1 -- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b8c159750 msgr2=0x7f7b8c169b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b8bfff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b8c159750 0x7f7b8c169b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b8bfff700 1 -- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c159b20 msgr2=0x7f7b8c16a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b8bfff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c159b20 0x7f7b8c16a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b8bfff700 1 -- 172.21.15.29:0/3348010824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b800094f0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b8bfff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 0x7f7b8c159f10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c810 tx=0x7f7b7c007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:49.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c00e4a0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c00e620 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c00e9e0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.105+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c15c770 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.106+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c15cd30 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.107+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c04c3c0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.108+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f7b7c00ed50 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.109+0000 7f7b73fff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 0x7f7b78069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.109+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f7b7c084290 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.109+0000 7f7b8b7fe700 1 -- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 msgr2=0x7f7b78069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:49.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.109+0000 7f7b8b7fe700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 0x7f7b78069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:49.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.112+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b7c050df0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.163 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:49.162+0000 7fdc026ea1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:41:49.247 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:49.245+0000 7fc3aa51e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:41:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.310+0000 7f7b8b7fe700 1 -- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 msgr2=0x7f7b78069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.310+0000 7f7b8b7fe700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 0x7f7b78069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:49.334 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:49.333+0000 7fc3aa51e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:41:49.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.363+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b8c0c5bf0 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.364+0000 7f7b73fff700 1 -- 172.21.15.29:0/3348010824 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f7b7c017090 con 0x7f7b8c0ce8a0 2023-10-07T09:41:49.365 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:49.365 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:30:25.453869+0000","modified":"2023-10-07T09:30:29.396464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6836/3596704883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3596704883},{"type":"v1","addr":"172.21.15.144:6836","nonce":3596704883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-07T09:41:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 msgr2=0x7f7b78069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 0x7f7b78069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 msgr2=0x7f7b8c159f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 0x7f7b8c159f10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c810 tx=0x7f7b7c007d60 comp rx=0 tx=0).stop 2023-10-07T09:41:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 shutdown_connections 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f7b780670e0 0x7f7b78069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c159b20 0x7f7b8c16a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b8c0ce8a0 0x7f7b8c159f10 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 --2- 172.21.15.29:0/3348010824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b8c159750 0x7f7b8c169b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 >> 172.21.15.29:0/3348010824 conn(0x7f7b8c063a10 msgr2=0x7f7b8c155180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 shutdown_connections 2023-10-07T09:41:49.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.367+0000 7f7b927ff700 1 -- 172.21.15.29:0/3348010824 wait complete. 2023-10-07T09:41:49.370 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-07T09:41:49.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:41:49.518 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:49.517+0000 7fdc026ea1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:41:49.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.586+0000 7f73ab3ff700 1 Processor -- start 2023-10-07T09:41:49.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.586+0000 7f73ab3ff700 1 -- start start 2023-10-07T09:41:49.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.587+0000 7f73ab3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a40ad370 0x7f73a40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.587+0000 7f73ab3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a414d790 con 0x7f73a4159ec0 2023-10-07T09:41:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.587+0000 7f73ab3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a40ae5d0 con 0x7f73a40b0410 2023-10-07T09:41:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.587+0000 7f73ab3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40ad740 con 0x7f73a40ad370 2023-10-07T09:41:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f73a899a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f73a4159ec0 0x7f73a41568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47186/0 (socket says 172.21.15.29:47186) 2023-10-07T09:41:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f73a899a700 1 -- 172.21.15.29:0/3154247548 learned_addr learned my addr 172.21.15.29:0/3154247548 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f73a919b700 1 --2- 172.21.15.29:0/3154247548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a40ad370 0x7f73a40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187715590 0 0) 0x7f73a414d790 con 0x7f73a4159ec0 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7390003570 con 0x7f73a4159ec0 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348025560 0 0) 0x7f73a40ae5d0 con 0x7f73a40b0410 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a414d790 con 0x7f73a40b0410 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f73a919b700 1 -- 172.21.15.29:0/3154247548 >> v1:172.21.15.144:6789/0 conn(0x7f73a40b0410 legacy=0x7f73a4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f73a919b700 1 -- 172.21.15.29:0/3154247548 >> v1:172.21.15.29:6789/0 conn(0x7f73a4159ec0 legacy=0x7f73a41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.588+0000 7f73a919b700 1 -- 172.21.15.29:0/3154247548 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40ccdb0 con 0x7f73a40ad370 2023-10-07T09:41:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.589+0000 7f73a919b700 1 --2- 172.21.15.29:0/3154247548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a40ad370 0x7f73a40cc870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009910 tx=0x7f73a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5582a3f891e82a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.589+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011630 con 0x7f73a40ad370 2023-10-07T09:41:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.589+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0011790 con 0x7f73a40ad370 2023-10-07T09:41:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.589+0000 7f739bfff700 1 -- 172.21.15.29:0/3154247548 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011b30 con 0x7f73a40ad370 2023-10-07T09:41:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3154247548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a40ad370 msgr2=0x7f73a40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3154247548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a40ad370 0x7f73a40cc870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009910 tx=0x7f73a00091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3154247548 shutdown_connections 2023-10-07T09:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3154247548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a40ad370 0x7f73a40cc870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3154247548 >> 172.21.15.29:0/3154247548 conn(0x7f73a40680a0 msgr2=0x7f73a40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3154247548 shutdown_connections 2023-10-07T09:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.590+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3154247548 wait complete. 2023-10-07T09:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.591+0000 7f73ab3ff700 1 Processor -- start 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.592+0000 7f73ab3ff700 1 -- start start 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73ab3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73a40ad370 0x7f73a40ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73ab3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 0x7f73a415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73ab3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a4159ec0 0x7f73a416c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73ab3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4155cf0 con 0x7f73a40b0410 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73ab3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4156b50 con 0x7f73a40ad370 2023-10-07T09:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73ab3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4155f40 con 0x7f73a4159ec0 2023-10-07T09:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 0x7f73a415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a919b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73a40ad370 0x7f73a40ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 0x7f73a415b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51220/0 (socket says 172.21.15.29:51220) 2023-10-07T09:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 -- 172.21.15.29:0/3125348289 learned_addr learned my addr 172.21.15.29:0/3125348289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a999c700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a4159ec0 0x7f73a416c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 -- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a4159ec0 msgr2=0x7f73a416c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a4159ec0 0x7f73a416c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 -- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73a40ad370 msgr2=0x7f73a40ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73a40ad370 0x7f73a40ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a899a700 1 -- 172.21.15.29:0/3125348289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0009420 con 0x7f73a40b0410 2023-10-07T09:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.593+0000 7f73a919b700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73a40ad370 0x7f73a40ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.594+0000 7f73a899a700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 0x7f73a415b5a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f739400c790 tx=0x7f7394007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.594+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394005b30 con 0x7f73a40b0410 2023-10-07T09:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.594+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7394005c90 con 0x7f73a40b0410 2023-10-07T09:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.594+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394007300 con 0x7f73a40b0410 2023-10-07T09:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.594+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a4156220 con 0x7f73a40b0410 2023-10-07T09:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.594+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a41567e0 con 0x7f73a40b0410 2023-10-07T09:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.595+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a4156cc0 con 0x7f73a40b0410 2023-10-07T09:41:49.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.596+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f739401b3b0 con 0x7f73a40b0410 2023-10-07T09:41:49.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.597+0000 7f73997fa700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 0x7f7388069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.597+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f73940841c0 con 0x7f73a40b0410 2023-10-07T09:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.597+0000 7f73a919b700 1 -- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 msgr2=0x7f7388069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.598+0000 7f73a919b700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 0x7f7388069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.601+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7394050d20 con 0x7f73a40b0410 2023-10-07T09:41:49.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:49.614+0000 7fdc026ea1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:41:49.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.799+0000 7f73a919b700 1 -- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 msgr2=0x7f7388069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:49.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.799+0000 7f73a919b700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 0x7f7388069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:49.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.854+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73a404c3c0 con 0x7f73a40b0410 2023-10-07T09:41:49.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.854+0000 7f73997fa700 1 -- 172.21.15.29:0/3125348289 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f7394054340 con 0x7f73a40b0410 2023-10-07T09:41:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:49.855 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:30:25.453869+0000","modified":"2023-10-07T09:30:29.396464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6836/3596704883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3596704883},{"type":"v1","addr":"172.21.15.144:6836","nonce":3596704883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-07T09:41:49.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 msgr2=0x7f7388069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:49.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 0x7f7388069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 msgr2=0x7f73a415b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:49.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 0x7f73a415b5a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f739400c790 tx=0x7f7394007ea0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 shutdown_connections 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f73880670e0 0x7f7388069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73a4159ec0 0x7f73a416c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73a40ad370 0x7f73a40ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 --2- 172.21.15.29:0/3125348289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a40b0410 0x7f73a415b5a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 >> 172.21.15.29:0/3125348289 conn(0x7f73a40680a0 msgr2=0x7f73a414f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 shutdown_connections 2023-10-07T09:41:49.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:49.857+0000 7f73ab3ff700 1 -- 172.21.15.29:0/3125348289 wait complete. 2023-10-07T09:41:49.860 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-07T09:41:49.864 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:49.863+0000 7fc3aa51e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:41:49.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:41:50.046 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:50.044+0000 7fc3aa51e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:41:50.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.071+0000 7faa1cbff700 1 Processor -- start 2023-10-07T09:41:50.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.072+0000 7faa1cbff700 1 -- start start 2023-10-07T09:41:50.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1cbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1cbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faa18159b20 0x7faa180ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1cbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa180ce6d0 0x7faa180ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1cbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180cbf20 con 0x7faa180ce6d0 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1cbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180cc200 con 0x7faa18159b20 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1cbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa180cc4e0 con 0x7faa18159750 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa16d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa180ce6d0 0x7faa180ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa16d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa180ce6d0 0x7faa180ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51236/0 (socket says 172.21.15.29:51236) 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa16d9d700 1 -- 172.21.15.29:0/3661809068 learned_addr learned my addr 172.21.15.29:0/3661809068 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa1659c700 1 --2- 172.21.15.29:0/3661809068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.073+0000 7faa15d9b700 1 --2- 172.21.15.29:0/3661809068 >> v2:172.21.15.144:3300/0 conn(0x7faa18159b20 0x7faa180ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa16d9d700 1 -- 172.21.15.29:0/3661809068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 msgr2=0x7faa180d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa16d9d700 1 --2- 172.21.15.29:0/3661809068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa16d9d700 1 -- 172.21.15.29:0/3661809068 >> v2:172.21.15.144:3300/0 conn(0x7faa18159b20 msgr2=0x7faa180ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa16d9d700 1 --2- 172.21.15.29:0/3661809068 >> v2:172.21.15.144:3300/0 conn(0x7faa18159b20 0x7faa180ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa16d9d700 1 -- 172.21.15.29:0/3661809068 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180cc7c0 con 0x7faa180ce6d0 2023-10-07T09:41:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa16d9d700 1 --2- 172.21.15.29:0/3661809068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa180ceb50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa18088830 tx=0x7faa10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8d21d6f71e87d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa1559a700 1 -- 172.21.15.29:0/3661809068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1000e070 con 0x7faa180ce6d0 2023-10-07T09:41:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa1559a700 1 -- 172.21.15.29:0/3661809068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1000b970 con 0x7faa180ce6d0 2023-10-07T09:41:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.074+0000 7faa1559a700 1 -- 172.21.15.29:0/3661809068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100092e0 con 0x7faa180ce6d0 2023-10-07T09:41:50.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 -- 172.21.15.29:0/3661809068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 msgr2=0x7faa180ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 --2- 172.21.15.29:0/3661809068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa180ceb50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa18088830 tx=0x7faa10009840 comp rx=0 tx=0).stop 2023-10-07T09:41:50.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 -- 172.21.15.29:0/3661809068 shutdown_connections 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 --2- 172.21.15.29:0/3661809068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa180ceb50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 --2- 172.21.15.29:0/3661809068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 --2- 172.21.15.29:0/3661809068 >> v2:172.21.15.144:3300/0 conn(0x7faa18159b20 0x7faa180ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 -- 172.21.15.29:0/3661809068 >> 172.21.15.29:0/3661809068 conn(0x7faa18063a10 msgr2=0x7faa180d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 -- 172.21.15.29:0/3661809068 shutdown_connections 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.075+0000 7faa1cbff700 1 -- 172.21.15.29:0/3661809068 wait complete. 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.077+0000 7faa1cbff700 1 Processor -- start 2023-10-07T09:41:50.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.077+0000 7faa1cbff700 1 -- start start 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1cbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa1814d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1cbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1cbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa18159b20 0x7faa180c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1cbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180b4350 con 0x7faa180ce6d0 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1cbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180b3740 con 0x7faa18159b20 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1cbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa180b3a20 con 0x7faa18159750 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa1814d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa1814d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51238/0 (socket says 172.21.15.29:51238) 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 -- 172.21.15.29:0/711194598 learned_addr learned my addr 172.21.15.29:0/711194598 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa15d9b700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa16d9d700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa18159b20 0x7faa180c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 -- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 msgr2=0x7faa180c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 -- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa18159b20 msgr2=0x7faa180c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa18159b20 0x7faa180c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa1659c700 1 -- 172.21.15.29:0/711194598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009ae0 con 0x7faa180ce6d0 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.078+0000 7faa16d9d700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa18159b20 0x7faa180c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.079+0000 7faa1659c700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa1814d340 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faa0800ec80 tx=0x7faa08009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.079+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0800c850 con 0x7faa180ce6d0 2023-10-07T09:41:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.079+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0800c9b0 con 0x7faa180ce6d0 2023-10-07T09:41:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.079+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0800cd70 con 0x7faa180ce6d0 2023-10-07T09:41:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.079+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa180b3d00 con 0x7faa180ce6d0 2023-10-07T09:41:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.079+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa180b3360 con 0x7faa180ce6d0 2023-10-07T09:41:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.080+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa18089d50 con 0x7faa180ce6d0 2023-10-07T09:41:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.081+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7faa08013680 con 0x7faa180ce6d0 2023-10-07T09:41:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.082+0000 7faa067fc700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 0x7fa9f8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.082+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7faa080852b0 con 0x7faa180ce6d0 2023-10-07T09:41:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.082+0000 7faa15d9b700 1 -- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 msgr2=0x7fa9f8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.082+0000 7faa15d9b700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 0x7fa9f8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:50.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.086+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa08051eb0 con 0x7faa180ce6d0 2023-10-07T09:41:50.196 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:50.195+0000 7fdc026ea1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:41:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.283+0000 7faa15d9b700 1 -- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 msgr2=0x7fa9f8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.283+0000 7faa15d9b700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 0x7fa9f8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:50.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.295+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa1804c3c0 con 0x7faa180ce6d0 2023-10-07T09:41:50.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.296+0000 7faa067fc700 1 -- 172.21.15.29:0/711194598 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16991 (secure 0 0 0) 0x7faa080554d0 con 0x7faa180ce6d0 2023-10-07T09:41:50.297 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:50.297 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:41:47.095737+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:30:23.373202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-07T09:30:24.433861+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3859209094":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/441038837":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/2779235645":"2023-10-08T09:30:47.178466+0000","172.21.15.144:0/465296683":"2023-10-08T09:30:47.178466+0000","172.21.15.144:6832/114714":"2023-10-08T09:30:47.178466+0000","172.21.15.144:6833/114714":"2023-10-08T09:30:47.178466+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":108,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":4,"length":2},{"begin":9,"length":2},{"begin":14,"length":2},{"begin":19,"length":2},{"begin":24,"length":2},{"begin":29,"length":2},{"begin":34,"length":2},{"begin":39,"length":2},{"begin":45,"length":2},{"begin":50,"length":3},{"begin":56,"length":2},{"begin":61,"length":2},{"begin":67,"length":2},{"begin":73,"length":1},{"begin":78,"length":2},{"begin":84,"length":1},{"begin":89,"length":2},{"begin":95,"length":1},{"begin":100,"length":3},{"begin":105,"length":2},{"begin":111,"length":1},{"begin":155,"length":40}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 msgr2=0x7fa9f8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 0x7fa9f8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 msgr2=0x7faa1814d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa1814d340 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faa0800ec80 tx=0x7faa08009120 comp rx=0 tx=0).stop 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 shutdown_connections 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7fa9f80670e0 0x7fa9f8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa18159b20 0x7faa180c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa180ce6d0 0x7faa1814d340 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 --2- 172.21.15.29:0/711194598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa18159750 0x7faa180c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.298+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 >> 172.21.15.29:0/711194598 conn(0x7faa18063a10 msgr2=0x7faa180d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.299+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 shutdown_connections 2023-10-07T09:41:50.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.299+0000 7faa1cbff700 1 -- 172.21.15.29:0/711194598 wait complete. 2023-10-07T09:41:50.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:41:50.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:50.393+0000 7fdc026ea1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:41:50.420 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:50.418+0000 7fc3aa51e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:41:50.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.521+0000 7f85a81ff700 1 Processor -- start 2023-10-07T09:41:50.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.521+0000 7f85a81ff700 1 -- start start 2023-10-07T09:41:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.522+0000 7f85a81ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85a0157290 0x7f85a00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a81ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a0158ad0 con 0x7f85a00b1a10 2023-10-07T09:41:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a81ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a015b140 con 0x7f85a00ae8b0 2023-10-07T09:41:50.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a81ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a015b420 con 0x7f85a0157290 2023-10-07T09:41:50.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a679c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f85a00b1a10 0x7f85a0157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47192/0 (socket says 172.21.15.29:47192) 2023-10-07T09:41:50.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a679c700 1 -- 172.21.15.29:0/1880936456 learned_addr learned my addr 172.21.15.29:0/1880936456 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:50.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a5f9b700 1 --2- 172.21.15.29:0/1880936456 >> v2:172.21.15.144:3300/0 conn(0x7f85a0157290 0x7f85a00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a5f9b700 1 -- 172.21.15.29:0/1880936456 >> v1:172.21.15.144:6789/0 conn(0x7f85a00ae8b0 legacy=0x7f85a00b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a5f9b700 1 -- 172.21.15.29:0/1880936456 >> v1:172.21.15.29:6789/0 conn(0x7f85a00b1a10 legacy=0x7f85a0157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a4f99700 1 -- 172.21.15.29:0/1880936456 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310413691 0 0) 0x7f85a0158ad0 con 0x7f85a00b1a10 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.523+0000 7f85a5f9b700 1 -- 172.21.15.29:0/1880936456 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a015b700 con 0x7f85a0157290 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.524+0000 7f85a5f9b700 1 --2- 172.21.15.29:0/1880936456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a0157290 0x7f85a00cc870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f859c0098e0 tx=0x7f859c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f459138836e4256 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.524+0000 7f85a4f99700 1 -- 172.21.15.29:0/1880936456 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c009e80 con 0x7f85a0157290 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.524+0000 7f85a4f99700 1 -- 172.21.15.29:0/1880936456 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c00b860 con 0x7f85a0157290 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.524+0000 7f85a4f99700 1 -- 172.21.15.29:0/1880936456 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00bbe0 con 0x7f85a0157290 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 -- 172.21.15.29:0/1880936456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a0157290 msgr2=0x7f85a00cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 --2- 172.21.15.29:0/1880936456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a0157290 0x7f85a00cc870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f859c0098e0 tx=0x7f859c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 -- 172.21.15.29:0/1880936456 shutdown_connections 2023-10-07T09:41:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 --2- 172.21.15.29:0/1880936456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a0157290 0x7f85a00cc870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 -- 172.21.15.29:0/1880936456 >> 172.21.15.29:0/1880936456 conn(0x7f85a00680a0 msgr2=0x7f85a00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 -- 172.21.15.29:0/1880936456 shutdown_connections 2023-10-07T09:41:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.525+0000 7f85a81ff700 1 -- 172.21.15.29:0/1880936456 wait complete. 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.527+0000 7f85a81ff700 1 Processor -- start 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.527+0000 7f85a81ff700 1 -- start start 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a81ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 0x7f85a006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a81ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85a0157290 0x7f85a01697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a81ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a006d4a0 0x7f85a015ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a81ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a014a920 con 0x7f85a006d4a0 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a81ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00b0e70 con 0x7f85a00b1a10 2023-10-07T09:41:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a81ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0156b90 con 0x7f85a0157290 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a006d4a0 0x7f85a015ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a5f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 0x7f85a006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a5f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 0x7f85a006ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:33212/0 (socket says 172.21.15.29:33212) 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a006d4a0 0x7f85a015ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51248/0 (socket says 172.21.15.29:51248) 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a5f9b700 1 -- 172.21.15.29:0/3334194707 learned_addr learned my addr 172.21.15.29:0/3334194707 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.528+0000 7f85a579a700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85a0157290 0x7f85a01697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a5f9b700 1 -- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85a0157290 msgr2=0x7f85a01697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a5f9b700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85a0157290 0x7f85a01697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a5f9b700 1 -- 172.21.15.29:0/3334194707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a006d4a0 msgr2=0x7f85a015ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a5f9b700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a006d4a0 0x7f85a015ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a5f9b700 1 -- 172.21.15.29:0/3334194707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c009420 con 0x7f85a00b1a10 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a5f9b700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 0x7f85a006ec40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f85a0068dd0 tx=0x7f859c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00bbe0 con 0x7f85a00b1a10 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c011420 con 0x7f85a00b1a10 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c0117e0 con 0x7f85a00b1a10 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a014a170 con 0x7f85a00b1a10 2023-10-07T09:41:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.529+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a014a640 con 0x7f85a00b1a10 2023-10-07T09:41:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.531+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a006f2a0 con 0x7f85a00b1a10 2023-10-07T09:41:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.531+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f859c011b90 con 0x7f85a00b1a10 2023-10-07T09:41:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.532+0000 7f8595ffb700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 0x7f8584069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.532+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f859c0861d0 con 0x7f85a00b1a10 2023-10-07T09:41:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.532+0000 7f85a579a700 1 -- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 msgr2=0x7f8584069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.532+0000 7f85a579a700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 0x7f8584069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:41:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.537+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f85a006f2a0 con 0x7f85a00b1a10 2023-10-07T09:41:50.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.733+0000 7f85a579a700 1 -- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 msgr2=0x7f8584069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/114714 2023-10-07T09:41:50.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.733+0000 7f85a579a700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 0x7f8584069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:41:50.778 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:50.777+0000 7fdc026ea1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:41:50.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.783+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f85a00b0fb0 con 0x7f85a00b1a10 2023-10-07T09:41:50.785 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:41:50.784+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:41:50.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.790+0000 7f8595ffb700 1 -- 172.21.15.29:0/3334194707 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f85a00b0fb0 con 0x7f85a00b1a10 2023-10-07T09:41:50.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:41:50.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 msgr2=0x7f8584069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:41:50.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 0x7f8584069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 msgr2=0x7f85a006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:50.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 0x7f85a006ec40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f85a0068dd0 tx=0x7f859c017000 comp rx=0 tx=0).stop 2023-10-07T09:41:50.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 shutdown_connections 2023-10-07T09:41:50.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f85840670e0 0x7f8584069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a006d4a0 0x7f85a015ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85a00b1a10 0x7f85a006ec40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 --2- 172.21.15.29:0/3334194707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85a0157290 0x7f85a01697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 >> 172.21.15.29:0/3334194707 conn(0x7f85a00680a0 msgr2=0x7f85a0159890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 shutdown_connections 2023-10-07T09:41:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:50.792+0000 7f85a81ff700 1 -- 172.21.15.29:0/3334194707 wait complete. 2023-10-07T09:41:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:41:50.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:41:50.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:41:50.861 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:50.860+0000 7fc3aa51e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:41:50.869 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:41:51.045 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.044+0000 7fc3aa51e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:41:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.054+0000 7f5c769ff700 1 Processor -- start 2023-10-07T09:41:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.054+0000 7f5c769ff700 1 -- start start 2023-10-07T09:41:51.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c769ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c7015a960 0x7f5c70158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c769ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c7015ad30 0x7f5c700ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c769ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700acb40 con 0x7f5c700ae6d0 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c769ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c700aeb50 con 0x7f5c7015ad30 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c769ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70158c60 con 0x7f5c7015a960 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c7015ad30 0x7f5c700ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c7015ad30 0x7f5c700ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51262/0 (socket says 172.21.15.29:51262) 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 -- 172.21.15.29:0/555951521 learned_addr learned my addr 172.21.15.29:0/555951521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6ffff700 1 --2- 172.21.15.29:0/555951521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c7015a960 0x7f5c70158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 -- 172.21.15.29:0/555951521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c7015a960 msgr2=0x7f5c70158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 --2- 172.21.15.29:0/555951521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c7015a960 0x7f5c70158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 -- 172.21.15.29:0/555951521 >> v1:172.21.15.144:6789/0 conn(0x7f5c700ae6d0 legacy=0x7f5c700b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.056+0000 7f5c6f7fe700 1 -- 172.21.15.29:0/555951521 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700abf30 con 0x7f5c7015ad30 2023-10-07T09:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.057+0000 7f5c6f7fe700 1 --2- 172.21.15.29:0/555951521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015ad30 0x7f5c700ae190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5c640099b0 tx=0x7f5c64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9ca35c62e15547 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.057+0000 7f5c6effd700 1 -- 172.21.15.29:0/555951521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64009ea0 con 0x7f5c7015ad30 2023-10-07T09:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.057+0000 7f5c6effd700 1 -- 172.21.15.29:0/555951521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6400b860 con 0x7f5c7015ad30 2023-10-07T09:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.057+0000 7f5c6effd700 1 -- 172.21.15.29:0/555951521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400bc00 con 0x7f5c7015ad30 2023-10-07T09:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.057+0000 7f5c769ff700 1 -- 172.21.15.29:0/555951521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015ad30 msgr2=0x7f5c700ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.057+0000 7f5c769ff700 1 --2- 172.21.15.29:0/555951521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015ad30 0x7f5c700ae190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5c640099b0 tx=0x7f5c64009250 comp rx=0 tx=0).stop 2023-10-07T09:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.058+0000 7f5c769ff700 1 -- 172.21.15.29:0/555951521 shutdown_connections 2023-10-07T09:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.058+0000 7f5c769ff700 1 --2- 172.21.15.29:0/555951521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c7015a960 0x7f5c70158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.058+0000 7f5c769ff700 1 --2- 172.21.15.29:0/555951521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015ad30 0x7f5c700ae190 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.058+0000 7f5c769ff700 1 -- 172.21.15.29:0/555951521 >> 172.21.15.29:0/555951521 conn(0x7f5c70067610 msgr2=0x7f5c70067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.058+0000 7f5c769ff700 1 -- 172.21.15.29:0/555951521 shutdown_connections 2023-10-07T09:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.058+0000 7f5c769ff700 1 -- 172.21.15.29:0/555951521 wait complete. 2023-10-07T09:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.059+0000 7f5c769ff700 1 Processor -- start 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.059+0000 7f5c769ff700 1 -- start start 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.060+0000 7f5c769ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 0x7f5c70168710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.060+0000 7f5c769ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015a960 0x7f5c7006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.060+0000 7f5c769ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c7015ad30 0x7f5c70148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.060+0000 7f5c769ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70168c50 con 0x7f5c7015a960 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.060+0000 7f5c769ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700ca2b0 con 0x7f5c7015ad30 2023-10-07T09:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.060+0000 7f5c769ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700cbd90 con 0x7f5c700ae6d0 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015a960 0x7f5c7006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 0x7f5c70168710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c74f9c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c7015ad30 0x7f5c70148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 0x7f5c70168710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:60786/0 (socket says 172.21.15.29:60786) 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 -- 172.21.15.29:0/1227392727 learned_addr learned my addr 172.21.15.29:0/1227392727 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 -- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c7015ad30 msgr2=0x7f5c70148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c7015ad30 0x7f5c70148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 -- 172.21.15.29:0/1227392727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015a960 msgr2=0x7f5c7006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015a960 0x7f5c7006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6ffff700 1 -- 172.21.15.29:0/1227392727 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640094f0 con 0x7f5c700ae6d0 2023-10-07T09:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c74f9c700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c7015ad30 0x7f5c70148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.061+0000 7f5c6f7fe700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015a960 0x7f5c7006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.062+0000 7f5c6ffff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 0x7f5c70168710 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000ec80 tx=0x7f5c600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.062+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60009690 con 0x7f5c700ae6d0 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.062+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6000c850 con 0x7f5c700ae6d0 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.062+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000cc10 con 0x7f5c700ae6d0 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.062+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c700ca500 con 0x7f5c700ae6d0 2023-10-07T09:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.062+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c700caac0 con 0x7f5c700ae6d0 2023-10-07T09:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.063+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7004c3c0 con 0x7f5c700ae6d0 2023-10-07T09:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.064+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 13) v1 ==== 84911+0+0 (secure 0 0 0) 0x7f5c60005a00 con 0x7f5c700ae6d0 2023-10-07T09:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c57fff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 0x7f5c5c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5c60084d50 con 0x7f5c700ae6d0 2023-10-07T09:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c6f7fe700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 0x7f5c5c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c6f7fe700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 0x7f5c5c069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello connection peer type does not match what peer advertises 16 != 2 2023-10-07T09:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c6f7fe700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 0x7f5c5c069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 msgr2=0x7f5c5c069590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-07T09:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.065+0000 7f5c57fff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 0x7f5c5c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.069+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c600555c0 con 0x7f5c700ae6d0 2023-10-07T09:41:51.230 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.229+0000 7fc3aa51e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:41:51.235 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:51.234+0000 7fdc026ea1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:41:51.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.319+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5c700ac210 con 0x7f5c700ae6d0 2023-10-07T09:41:51.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.336+0000 7fc3aa51e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:41:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.347+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mgrmap(e 14) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f5c60051f00 con 0x7f5c700ae6d0 2023-10-07T09:41:51.412 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc327b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.412 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc327b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc327b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc327b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc327b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc324b83700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc324b83700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc324b83700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc324b83700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.413 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:41:51.411+0000 7fc324b83700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:41:51.427 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:51.426+0000 7fdc026ea1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:41:51.618 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:51.617+0000 7fdc026ea1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:41:51.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:41:51.729+0000 7fdc026ea1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:41:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.792+0000 7f5c57fff700 1 -- 172.21.15.29:0/1227392727 <== mon.2 v2:172.21.15.144:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5c60055330 con 0x7f5c700ae6d0 2023-10-07T09:41:51.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 msgr2=0x7f5c70168710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 0x7f5c70168710 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000ec80 tx=0x7f5c600091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 shutdown_connections 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:6832/114714,v1:172.21.15.144:6833/114714] conn(0x7f5c5c0670e0 0x7f5c5c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c7015ad30 0x7f5c70148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5c700ae6d0 0x7f5c70168710 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 --2- 172.21.15.29:0/1227392727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c7015a960 0x7f5c7006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 >> 172.21.15.29:0/1227392727 conn(0x7f5c70067610 msgr2=0x7f5c7015b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 shutdown_connections 2023-10-07T09:41:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:51.794+0000 7f5c769ff700 1 -- 172.21.15.29:0/1227392727 wait complete. 2023-10-07T09:41:51.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:41:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.011+0000 7f1f083ff700 1 Processor -- start 2023-10-07T09:41:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.012+0000 7f1f083ff700 1 -- start start 2023-10-07T09:41:52.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f083ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f000cb8c0 0x7f1f000cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f083ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f000c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f083ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000cc940 con 0x7f1f000b2a20 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f083ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f000cf530 con 0x7f1f000cb8c0 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f083ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000b2df0 con 0x7f1f000cf0f0 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f0599a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f000cb8c0 0x7f1f000cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f0599a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f000cb8c0 0x7f1f000cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51284/0 (socket says 172.21.15.29:51284) 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f0599a700 1 -- 172.21.15.29:0/71836640 learned_addr learned my addr 172.21.15.29:0/71836640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.013+0000 7f1f0699c700 1 --2- 172.21.15.29:0/71836640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f000c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f0599a700 1 -- 172.21.15.29:0/71836640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 msgr2=0x7f1f000c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:52.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f0599a700 1 --2- 172.21.15.29:0/71836640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f000c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:52.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f0599a700 1 -- 172.21.15.29:0/71836640 >> v1:172.21.15.144:6789/0 conn(0x7f1f000b2a20 legacy=0x7f1f000cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:52.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f0599a700 1 -- 172.21.15.29:0/71836640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000cbd30 con 0x7f1f000cb8c0 2023-10-07T09:41:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f0599a700 1 --2- 172.21.15.29:0/71836640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f000cebb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00099b0 tx=0x7f1ef0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f4e426bf8d0125 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f05199700 1 -- 172.21.15.29:0/71836640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef000b860 con 0x7f1f000cb8c0 2023-10-07T09:41:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f05199700 1 -- 172.21.15.29:0/71836640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef000b9c0 con 0x7f1f000cb8c0 2023-10-07T09:41:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.014+0000 7f1f05199700 1 -- 172.21.15.29:0/71836640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef000bd60 con 0x7f1f000cb8c0 2023-10-07T09:41:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 -- 172.21.15.29:0/71836640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 msgr2=0x7f1f000cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 --2- 172.21.15.29:0/71836640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f000cebb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00099b0 tx=0x7f1ef0009250 comp rx=0 tx=0).stop 2023-10-07T09:41:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 -- 172.21.15.29:0/71836640 shutdown_connections 2023-10-07T09:41:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 --2- 172.21.15.29:0/71836640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f000c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 --2- 172.21.15.29:0/71836640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f000cebb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 -- 172.21.15.29:0/71836640 >> 172.21.15.29:0/71836640 conn(0x7f1f00099820 msgr2=0x7f1f00063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 -- 172.21.15.29:0/71836640 shutdown_connections 2023-10-07T09:41:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.015+0000 7f1f083ff700 1 -- 172.21.15.29:0/71836640 wait complete. 2023-10-07T09:41:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.016+0000 7f1f083ff700 1 Processor -- start 2023-10-07T09:41:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.017+0000 7f1f083ff700 1 -- start start 2023-10-07T09:41:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.017+0000 7f1f083ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f000b2a20 0x7f1f0016cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f083ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f00156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f083ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f0015be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f083ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0016dee0 con 0x7f1f000cb8c0 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f083ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f001571a0 con 0x7f1f000b2a20 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f083ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000b47a0 con 0x7f1f000cf0f0 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f00156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f00156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51292/0 (socket says 172.21.15.29:51292) 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 -- 172.21.15.29:0/4153521360 learned_addr learned my addr 172.21.15.29:0/4153521360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0619b700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f000b2a20 0x7f1f0016cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0699c700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f0015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 -- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 msgr2=0x7f1f0015be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f0015be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 -- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f000b2a20 msgr2=0x7f1f0016cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f000b2a20 0x7f1f0016cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.018+0000 7f1f0599a700 1 -- 172.21.15.29:0/4153521360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00094f0 con 0x7f1f000cb8c0 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.019+0000 7f1f0599a700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f00156c60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1f00064210 tx=0x7f1ef0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.019+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0011e00 con 0x7f1f000cb8c0 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.019+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef0024440 con 0x7f1f000cb8c0 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.019+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0024800 con 0x7f1f000cb8c0 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.019+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0016d640 con 0x7f1f000cb8c0 2023-10-07T09:41:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.019+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0016dc00 con 0x7f1f000cb8c0 2023-10-07T09:41:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.020+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f000718b0 con 0x7f1f000cb8c0 2023-10-07T09:41:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.021+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f1ef001a420 con 0x7f1f000cb8c0 2023-10-07T09:41:52.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.022+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f1ef0086290 con 0x7f1f000cb8c0 2023-10-07T09:41:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.026+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ef00533f0 con 0x7f1f000cb8c0 2023-10-07T09:41:52.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.234+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1f000cc010 con 0x7f1f000cb8c0 2023-10-07T09:41:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.350+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 15) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f1ef0052ea0 con 0x7f1f000cb8c0 2023-10-07T09:41:52.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.351+0000 7f1ef67fc700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ee406c980 0x7f1ee406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.353+0000 7f1f0619b700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ee406c980 0x7f1ee406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:52.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:52.356+0000 7f1f0619b700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ee406c980 0x7f1ee406ed60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1efc007fd0 tx=0x7f1efc007eb0 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:53.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.134+0000 7f1ef67fc700 1 -- 172.21.15.29:0/4153521360 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v51) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1ef005a0e0 con 0x7f1f000cb8c0 2023-10-07T09:41:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ee406c980 msgr2=0x7f1ee406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ee406c980 0x7f1ee406ed60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1efc007fd0 tx=0x7f1efc007eb0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 msgr2=0x7f1f00156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f00156c60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1f00064210 tx=0x7f1ef0017000 comp rx=0 tx=0).stop 2023-10-07T09:41:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 shutdown_connections 2023-10-07T09:41:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ee406c980 0x7f1ee406ed60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f000cf0f0 0x7f1f0015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f000b2a20 0x7f1f0016cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 --2- 172.21.15.29:0/4153521360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f000cb8c0 0x7f1f00156c60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.138+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 >> 172.21.15.29:0/4153521360 conn(0x7f1f00099820 msgr2=0x7f1f0014ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.139+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 shutdown_connections 2023-10-07T09:41:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.139+0000 7f1f083ff700 1 -- 172.21.15.29:0/4153521360 wait complete. 2023-10-07T09:41:53.141 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:41:53.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:41:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.365+0000 7f499c7ff700 1 Processor -- start 2023-10-07T09:41:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.365+0000 7f499c7ff700 1 -- start start 2023-10-07T09:41:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.366+0000 7f499c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 0x7f49940b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.366+0000 7f499c7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f49940c8a20 0x7f49940cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.366+0000 7f499c7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49940690f0 con 0x7f49940b0610 2023-10-07T09:41:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.366+0000 7f499c7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49940cf2e0 con 0x7f49940c8a20 2023-10-07T09:41:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.366+0000 7f499c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940687c0 con 0x7f49940ced00 2023-10-07T09:41:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f499a59b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f49940b0610 0x7f49940c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34084/0 (socket says 172.21.15.29:34084) 2023-10-07T09:41:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f499a59b700 1 -- 172.21.15.29:0/2609593459 learned_addr learned my addr 172.21.15.29:0/2609593459 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160709089 0 0) 0x7f49940690f0 con 0x7f49940b0610 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4980003570 con 0x7f49940b0610 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999d9a700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 0x7f49940b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f499ad9c700 1 --2- 172.21.15.29:0/2609593459 >> v2:172.21.15.144:3300/0 conn(0x7f49940c8a20 0x7f49940cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3499877236 0 0) 0x7f4980003570 con 0x7f49940b0610 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 msgr2=0x7f49940b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:53.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 0x7f49940b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 >> v2:172.21.15.144:3300/0 conn(0x7f49940c8a20 msgr2=0x7f49940cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:53.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 --2- 172.21.15.29:0/2609593459 >> v2:172.21.15.144:3300/0 conn(0x7f49940c8a20 0x7f49940cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994068aa0 con 0x7f49940b0610 2023-10-07T09:41:53.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f4990001470 con 0x7f49940b0610 2023-10-07T09:41:53.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 >> v1:172.21.15.29:6789/0 conn(0x7f49940b0610 legacy=0x7f49940c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:53.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4980003ae0 0x7f4980003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 0x7f4980004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.367+0000 7f4999599700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f49800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940cf2e0 con 0x7f4980004d50 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940687c0 con 0x7f4980003ae0 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498000bcc0 con 0x7f49940ced00 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f4999d9a700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 0x7f4980004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499a59b700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4980003ae0 0x7f4980003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f49800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 -- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 msgr2=0x7f4980004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940ced00 0x7f4980004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 -- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4980003ae0 msgr2=0x7f4980003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4980003ae0 0x7f4980003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 -- 172.21.15.29:0/2609593459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4980003570 con 0x7f4980004d50 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.368+0000 7f499ad9c700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f49800051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f498c007440 tx=0x7f498c009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9439200993ea82d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c009a40 con 0x7f4980004d50 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c007cf0 con 0x7f4980004d50 2023-10-07T09:41:53.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f4999599700 1 -- 172.21.15.29:0/2609593459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c014630 con 0x7f4980004d50 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499c7ff700 1 -- 172.21.15.29:0/2609593459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 msgr2=0x7f49800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499c7ff700 1 --2- 172.21.15.29:0/2609593459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f49800051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f498c007440 tx=0x7f498c009710 comp rx=0 tx=0).stop 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499ad9c700 1 -- 172.21.15.29:0/2609593459 reap_dead start 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499c7ff700 1 -- 172.21.15.29:0/2609593459 shutdown_connections 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499c7ff700 1 -- 172.21.15.29:0/2609593459 >> 172.21.15.29:0/2609593459 conn(0x7f4994063a10 msgr2=0x7f49940ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499c7ff700 1 -- 172.21.15.29:0/2609593459 shutdown_connections 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.369+0000 7f499c7ff700 1 -- 172.21.15.29:0/2609593459 wait complete. 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.371+0000 7f499c7ff700 1 Processor -- start 2023-10-07T09:41:53.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.371+0000 7f499c7ff700 1 -- start start 2023-10-07T09:41:53.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f499414b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f499409b020 0x7f499409b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940c3950 0x7f49940c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940c4330 con 0x7f4980004d50 2023-10-07T09:41:53.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499414b840 con 0x7f499409b020 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940b27e0 con 0x7f49940c3950 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499a59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f499414b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499a59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f499414b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46830/0 (socket says 172.21.15.29:46830) 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499a59b700 1 -- 172.21.15.29:0/1733602834 learned_addr learned my addr 172.21.15.29:0/1733602834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499ad9c700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940c3950 0x7f49940c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499a59b700 1 -- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940c3950 msgr2=0x7f49940c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.372+0000 7f499a59b700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940c3950 0x7f49940c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499a59b700 1 -- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f499409b020 msgr2=0x7f499409b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f4999d9a700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f499409b020 0x7f499409b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499a59b700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f499409b020 0x7f499409b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499a59b700 1 -- 172.21.15.29:0/1733602834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994098c00 con 0x7f4980004d50 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499ad9c700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940c3950 0x7f49940c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f4999d9a700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f499409b020 0x7f499409b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499a59b700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f499414b300 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4990007fd0 tx=0x7f4990001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990018030 con 0x7f4980004d50 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49900077d0 con 0x7f4980004d50 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499000fac0 con 0x7f4980004d50 2023-10-07T09:41:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499409bb90 con 0x7f4980004d50 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.373+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49940b2470 con 0x7f4980004d50 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.374+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49940b1ff0 con 0x7f4980004d50 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.375+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f499001b070 con 0x7f4980004d50 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.376+0000 7f498a7fc700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4978066d80 0x7f4978069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.376+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f4990019070 con 0x7f4980004d50 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.376+0000 7f4999d9a700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4978066d80 0x7f4978069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.377+0000 7f4999d9a700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4978066d80 0x7f4978069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4984006f00 tx=0x7f4984008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:53.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.380+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49940b1ff0 con 0x7f4980004d50 2023-10-07T09:41:53.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:53.583+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f49940b1ff0 con 0x7f4980004d50 2023-10-07T09:41:54.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.139+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4990084080 con 0x7f4980004d50 2023-10-07T09:41:54.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.358+0000 7f498a7fc700 1 -- 172.21.15.29:0/1733602834 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v52) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4990054da0 con 0x7f4980004d50 2023-10-07T09:41:54.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4978066d80 msgr2=0x7f4978069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4978066d80 0x7f4978069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4984006f00 tx=0x7f4984008040 comp rx=0 tx=0).stop 2023-10-07T09:41:54.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 msgr2=0x7f499414b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f499414b300 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4990007fd0 tx=0x7f4990001dc0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 shutdown_connections 2023-10-07T09:41:54.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4978066d80 0x7f4978069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49940c3950 0x7f49940c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4980004d50 0x7f499414b300 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 --2- 172.21.15.29:0/1733602834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f499409b020 0x7f499409b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.362+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 >> 172.21.15.29:0/1733602834 conn(0x7f4994063a10 msgr2=0x7f49940cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:54.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.363+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 shutdown_connections 2023-10-07T09:41:54.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.363+0000 7f499c7ff700 1 -- 172.21.15.29:0/1733602834 wait complete. 2023-10-07T09:41:54.369 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:41:54.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T09:41:54.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.594+0000 7f10709ff700 1 Processor -- start 2023-10-07T09:41:54.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.594+0000 7f10709ff700 1 -- start start 2023-10-07T09:41:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.595+0000 7f10709ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f106c159750 0x7f106c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.595+0000 7f10709ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 0x7f106c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.595+0000 7f10709ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c0cbf20 con 0x7f106c159b20 2023-10-07T09:41:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.595+0000 7f10709ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c0cc200 con 0x7f106c159750 2023-10-07T09:41:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.595+0000 7f10709ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0cc4e0 con 0x7f106c0ce8a0 2023-10-07T09:41:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.596+0000 7f106a59c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f106c159750 0x7f106c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.596+0000 7f106ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 0x7f106c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.596+0000 7f106a59c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f106c159750 0x7f106c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:47132/0 (socket says 172.21.15.29:47132) 2023-10-07T09:41:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.596+0000 7f106ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 0x7f106c0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53322/0 (socket says 172.21.15.29:53322) 2023-10-07T09:41:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.596+0000 7f106a59c700 1 -- 172.21.15.29:0/3478356732 learned_addr learned my addr 172.21.15.29:0/3478356732 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.596+0000 7f1069d9b700 1 --1- 172.21.15.29:0/3478356732 >> v1:172.21.15.144:6789/0 conn(0x7f106c159b20 0x7f106c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:52636/0 (socket says 172.21.15.29:52636) 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106a59c700 1 -- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 msgr2=0x7f106c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106a59c700 1 --2- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 0x7f106c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106a59c700 1 -- 172.21.15.29:0/3478356732 >> v1:172.21.15.144:6789/0 conn(0x7f106c159b20 legacy=0x7f106c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106a59c700 1 -- 172.21.15.29:0/3478356732 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0cc7c0 con 0x7f106c159750 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106ad9d700 1 --2- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 0x7f106c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106959a700 1 -- 172.21.15.29:0/3478356732 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293226531 0 0) 0x7f106c0cbf20 con 0x7f106c159b20 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106a59c700 1 --2- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c159750 0x7f106c0d10a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f105c009990 tx=0x7f105c009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5de1515aefc47c33 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106959a700 1 -- 172.21.15.29:0/3478356732 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c009e80 con 0x7f106c159750 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106959a700 1 -- 172.21.15.29:0/3478356732 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c005c10 con 0x7f106c159750 2023-10-07T09:41:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.597+0000 7f106959a700 1 -- 172.21.15.29:0/3478356732 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00b650 con 0x7f106c159750 2023-10-07T09:41:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 -- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c159750 msgr2=0x7f106c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 --2- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c159750 0x7f106c0d10a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f105c009990 tx=0x7f105c009220 comp rx=0 tx=0).stop 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 -- 172.21.15.29:0/3478356732 shutdown_connections 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 --2- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c0ce8a0 0x7f106c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 --2- 172.21.15.29:0/3478356732 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c159750 0x7f106c0d10a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 -- 172.21.15.29:0/3478356732 >> 172.21.15.29:0/3478356732 conn(0x7f106c063a10 msgr2=0x7f106c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 -- 172.21.15.29:0/3478356732 shutdown_connections 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.598+0000 7f10709ff700 1 -- 172.21.15.29:0/3478356732 wait complete. 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.599+0000 7f10709ff700 1 Processor -- start 2023-10-07T09:41:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 -- start start 2023-10-07T09:41:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c0ce8a0 0x7f106c149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c159750 0x7f106c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 0x7f106c169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b43c0 con 0x7f106c159b20 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c15c5b0 con 0x7f106c0ce8a0 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.600+0000 7f10709ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c15c890 con 0x7f106c159750 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 0x7f106c169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 0x7f106c169a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46844/0 (socket says 172.21.15.29:46844) 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 -- 172.21.15.29:0/2829063328 learned_addr learned my addr 172.21.15.29:0/2829063328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f1069d9b700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c159750 0x7f106c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106a59c700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c0ce8a0 0x7f106c149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 -- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c159750 msgr2=0x7f106c159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c159750 0x7f106c159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 -- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c0ce8a0 msgr2=0x7f106c149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c0ce8a0 0x7f106c149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106ad9d700 1 -- 172.21.15.29:0/2829063328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064009a20 con 0x7f106c159b20 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.601+0000 7f106a59c700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c0ce8a0 0x7f106c149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.602+0000 7f106ad9d700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 0x7f106c169a90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1064009fc0 tx=0x7f106400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.602+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064009d20 con 0x7f106c159b20 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.602+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400f040 con 0x7f106c159b20 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.602+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064014480 con 0x7f106c159b20 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.602+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c009480 con 0x7f106c159b20 2023-10-07T09:41:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.602+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c15ce50 con 0x7f106c159b20 2023-10-07T09:41:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.603+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c0b4530 con 0x7f106c159b20 2023-10-07T09:41:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.604+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1064003020 con 0x7f106c159b20 2023-10-07T09:41:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.605+0000 7f105a7fc700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f104c066f30 0x7f104c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.605+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f1064084f80 con 0x7f106c159b20 2023-10-07T09:41:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.606+0000 7f106a59c700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f104c066f30 0x7f104c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.607+0000 7f106a59c700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f104c066f30 0x7f104c0693e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f105c009260 tx=0x7f105c016000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:54.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.610+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10640520c0 con 0x7f106c159b20 2023-10-07T09:41:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.810+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f106c04c3c0 con 0x7f106c159b20 2023-10-07T09:41:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.814+0000 7f105a7fc700 1 -- 172.21.15.29:0/2829063328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7f10640556e0 con 0x7f106c159b20 2023-10-07T09:41:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f104c066f30 msgr2=0x7f104c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f104c066f30 0x7f104c0693e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f105c009260 tx=0x7f105c016000 comp rx=0 tx=0).stop 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 msgr2=0x7f106c169a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 0x7f106c169a90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1064009fc0 tx=0x7f106400b040 comp rx=0 tx=0).stop 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 shutdown_connections 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f104c066f30 0x7f104c0693e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f106c159b20 0x7f106c169a90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f106c0ce8a0 0x7f106c149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 --2- 172.21.15.29:0/2829063328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f106c159750 0x7f106c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 >> 172.21.15.29:0/2829063328 conn(0x7f106c063a10 msgr2=0x7f106c0c6280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.816+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 shutdown_connections 2023-10-07T09:41:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:54.817+0000 7f10709ff700 1 -- 172.21.15.29:0/2829063328 wait complete. 2023-10-07T09:41:54.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T09:41:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.095+0000 7fc9d11ff700 1 Processor -- start 2023-10-07T09:41:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.096+0000 7fc9d11ff700 1 -- start start 2023-10-07T09:41:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9d11ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc9cc15a960 0x7fc9cc158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9d11ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9cc0ae8a0 0x7fc9cc0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9d11ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc0acb40 con 0x7fc9cc15ad30 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9d11ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc158c60 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9d11ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc0abf30 con 0x7fc9cc15a960 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9cb59e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9cc0ae8a0 0x7fc9cc0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9cb59e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9cc0ae8a0 0x7fc9cc0b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46846/0 (socket says 172.21.15.29:46846) 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9cb59e700 1 -- 172.21.15.29:0/1892570647 learned_addr learned my addr 172.21.15.29:0/1892570647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.097+0000 7fc9cad9d700 1 --2- 172.21.15.29:0/1892570647 >> v2:172.21.15.144:3300/0 conn(0x7fc9cc15a960 0x7fc9cc158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9cb59e700 1 -- 172.21.15.29:0/1892570647 >> v2:172.21.15.144:3300/0 conn(0x7fc9cc15a960 msgr2=0x7fc9cc158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9cb59e700 1 --2- 172.21.15.29:0/1892570647 >> v2:172.21.15.144:3300/0 conn(0x7fc9cc15a960 0x7fc9cc158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9cb59e700 1 -- 172.21.15.29:0/1892570647 >> v1:172.21.15.144:6789/0 conn(0x7fc9cc15ad30 legacy=0x7fc9cc0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9cb59e700 1 -- 172.21.15.29:0/1892570647 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc0ac210 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9cb59e700 1 --2- 172.21.15.29:0/1892570647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc0b1290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0aecc0 tx=0x7fc9c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25314c9efa762c69 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9c9d9b700 1 -- 172.21.15.29:0/1892570647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400e070 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9c9d9b700 1 -- 172.21.15.29:0/1892570647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c400ba20 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.098+0000 7fc9c9d9b700 1 -- 172.21.15.29:0/1892570647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c40092e0 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 -- 172.21.15.29:0/1892570647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 msgr2=0x7fc9cc0b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/1892570647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc0b1290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0aecc0 tx=0x7fc9c4009840 comp rx=0 tx=0).stop 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 -- 172.21.15.29:0/1892570647 shutdown_connections 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/1892570647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc0b1290 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/1892570647 >> v2:172.21.15.144:3300/0 conn(0x7fc9cc15a960 0x7fc9cc158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 -- 172.21.15.29:0/1892570647 >> 172.21.15.29:0/1892570647 conn(0x7fc9cc067610 msgr2=0x7fc9cc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 -- 172.21.15.29:0/1892570647 shutdown_connections 2023-10-07T09:41:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.099+0000 7fc9d11ff700 1 -- 172.21.15.29:0/1892570647 wait complete. 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.101+0000 7fc9d11ff700 1 Processor -- start 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.101+0000 7fc9d11ff700 1 -- start start 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9d11ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc16ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9d11ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9cc15a960 0x7fc9cc15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9d11ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9cc15ad30 0x7fc9cc0afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9d11ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc160180 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9d11ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc15f570 con 0x7fc9cc15a960 2023-10-07T09:41:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9d11ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc15f850 con 0x7fc9cc15ad30 2023-10-07T09:41:55.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc16ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc16ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46852/0 (socket says 172.21.15.29:46852) 2023-10-07T09:41:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 -- 172.21.15.29:0/2510875823 learned_addr learned my addr 172.21.15.29:0/2510875823 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9ca59c700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9cc15a960 0x7fc9cc15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cb59e700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9cc15ad30 0x7fc9cc0afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 -- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9cc15ad30 msgr2=0x7fc9cc0afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9cc15ad30 0x7fc9cc0afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 -- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9cc15a960 msgr2=0x7fc9cc15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9cc15a960 0x7fc9cc15c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cad9d700 1 -- 172.21.15.29:0/2510875823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c4009ae0 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9ca59c700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9cc15a960 0x7fc9cc15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.102+0000 7fc9cb59e700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9cc15ad30 0x7fc9cc0afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.103+0000 7fc9cad9d700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc16ccb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00ec80 tx=0x7fc9bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.103+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00c9d0 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.103+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc00cb30 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.103+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc013470 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.103+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9cc15fb30 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.103+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9cc16d1f0 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.104+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9cc16a100 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.105+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc9bc013820 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.106+0000 7fc9b2ffd700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc9b8066f30 0x7fc9b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.106+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fc9bc0849f0 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.107+0000 7fc9ca59c700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc9b8066f30 0x7fc9b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.107+0000 7fc9ca59c700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc9b8066f30 0x7fc9b80693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0a5d30 tx=0x7fc9c0009040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.110+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9bc051b30 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.317+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc9cc04c3c0 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.321+0000 7fc9b2ffd700 1 -- 172.21.15.29:0/2510875823 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc9bc055150 con 0x7fc9cc0ae8a0 2023-10-07T09:41:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc9b8066f30 msgr2=0x7fc9b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc9b8066f30 0x7fc9b80693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0a5d30 tx=0x7fc9c0009040 comp rx=0 tx=0).stop 2023-10-07T09:41:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 msgr2=0x7fc9cc16ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc16ccb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00ec80 tx=0x7fc9bc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 shutdown_connections 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc9b8066f30 0x7fc9b80693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9cc15ad30 0x7fc9cc0afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9cc0ae8a0 0x7fc9cc16ccb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 --2- 172.21.15.29:0/2510875823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9cc15a960 0x7fc9cc15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 >> 172.21.15.29:0/2510875823 conn(0x7fc9cc067610 msgr2=0x7fc9cc1514a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 shutdown_connections 2023-10-07T09:41:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.324+0000 7fc9d11ff700 1 -- 172.21.15.29:0/2510875823 wait complete. 2023-10-07T09:41:55.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T09:41:55.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.540+0000 7fe77b1ff700 1 Processor -- start 2023-10-07T09:41:55.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.540+0000 7fe77b1ff700 1 -- start start 2023-10-07T09:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.541+0000 7fe77b1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe774153cc0 0x7fe7740b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.541+0000 7fe77b1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe77415ade0 0x7fe7740b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.541+0000 7fe77b1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe774154090 con 0x7fe7740b0000 2023-10-07T09:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.541+0000 7fe77b1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe774159f60 con 0x7fe774153cc0 2023-10-07T09:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.541+0000 7fe77b1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77415a240 con 0x7fe77415ade0 2023-10-07T09:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe774153cc0 0x7fe7740b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe774153cc0 0x7fe7740b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46868/0 (socket says 172.21.15.29:46868) 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 -- 172.21.15.29:0/955676491 learned_addr learned my addr 172.21.15.29:0/955676491 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe77979c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe7740b0000 0x7fe77415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34096/0 (socket says 172.21.15.29:34096) 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe773fff700 1 --2- 172.21.15.29:0/955676491 >> v2:172.21.15.144:3300/0 conn(0x7fe77415ade0 0x7fe7740b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 -- 172.21.15.29:0/955676491 >> v2:172.21.15.144:3300/0 conn(0x7fe77415ade0 msgr2=0x7fe7740b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 --2- 172.21.15.29:0/955676491 >> v2:172.21.15.144:3300/0 conn(0x7fe77415ade0 0x7fe7740b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 -- 172.21.15.29:0/955676491 >> v1:172.21.15.29:6789/0 conn(0x7fe7740b0000 legacy=0x7fe77415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 -- 172.21.15.29:0/955676491 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77415a520 con 0x7fe774153cc0 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe773fff700 1 --2- 172.21.15.29:0/955676491 >> v2:172.21.15.144:3300/0 conn(0x7fe77415ade0 0x7fe7740b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe7737fe700 1 -- 172.21.15.29:0/955676491 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443609796 0 0) 0x7fe774154090 con 0x7fe7740b0000 2023-10-07T09:41:55.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.542+0000 7fe778f9b700 1 --2- 172.21.15.29:0/955676491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774153cc0 0x7fe7740b17c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe7740add60 tx=0x7fe768009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33db9a105d55084 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe7737fe700 1 -- 172.21.15.29:0/955676491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768005c10 con 0x7fe774153cc0 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe7737fe700 1 -- 172.21.15.29:0/955676491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768005d70 con 0x7fe774153cc0 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe7737fe700 1 -- 172.21.15.29:0/955676491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800b880 con 0x7fe774153cc0 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 -- 172.21.15.29:0/955676491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774153cc0 msgr2=0x7fe7740b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/955676491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774153cc0 0x7fe7740b17c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe7740add60 tx=0x7fe768009220 comp rx=0 tx=0).stop 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 -- 172.21.15.29:0/955676491 shutdown_connections 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/955676491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774153cc0 0x7fe7740b17c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/955676491 >> v2:172.21.15.144:3300/0 conn(0x7fe77415ade0 0x7fe7740b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 -- 172.21.15.29:0/955676491 >> 172.21.15.29:0/955676491 conn(0x7fe774067610 msgr2=0x7fe774067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 -- 172.21.15.29:0/955676491 shutdown_connections 2023-10-07T09:41:55.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.543+0000 7fe77b1ff700 1 -- 172.21.15.29:0/955676491 wait complete. 2023-10-07T09:41:55.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.545+0000 7fe77b1ff700 1 Processor -- start 2023-10-07T09:41:55.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.546+0000 7fe77b1ff700 1 -- start start 2023-10-07T09:41:55.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77b1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7740b0000 0x7fe7740c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77b1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe774153cc0 0x7fe77413fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77b1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 0x7fe774156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77b1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7741585f0 con 0x7fe77415ade0 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77b1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7741579e0 con 0x7fe7740b0000 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77b1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe774157cc0 con 0x7fe774153cc0 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77979c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 0x7fe774156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77979c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 0x7fe774156eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46882/0 (socket says 172.21.15.29:46882) 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe77979c700 1 -- 172.21.15.29:0/657107698 learned_addr learned my addr 172.21.15.29:0/657107698 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe778f9b700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7740b0000 0x7fe7740c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.547+0000 7fe773fff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe774153cc0 0x7fe77413fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77979c700 1 -- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe774153cc0 msgr2=0x7fe77413fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77979c700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe774153cc0 0x7fe77413fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77979c700 1 -- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7740b0000 msgr2=0x7fe7740c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77979c700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7740b0000 0x7fe7740c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77979c700 1 -- 172.21.15.29:0/657107698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009480 con 0x7fe77415ade0 2023-10-07T09:41:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe773fff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe774153cc0 0x7fe77413fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77979c700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 0x7fe774156eb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00cd50 tx=0x7fe76c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c005df0 con 0x7fe77415ade0 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c00a960 con 0x7fe77415ade0 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe774157fa0 con 0x7fe77415ade0 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c00ad20 con 0x7fe77415ade0 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.548+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe774152430 con 0x7fe77415ade0 2023-10-07T09:41:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.549+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77404c3c0 con 0x7fe77415ade0 2023-10-07T09:41:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.550+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe76c007300 con 0x7fe77415ade0 2023-10-07T09:41:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.551+0000 7fe770ff9700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe758066f30 0x7fe7580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.551+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fe76c084810 con 0x7fe77415ade0 2023-10-07T09:41:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.551+0000 7fe778f9b700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe758066f30 0x7fe7580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.552+0000 7fe778f9b700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe758066f30 0x7fe7580693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe768009260 tx=0x7fe768016000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.555+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe76c0519d0 con 0x7fe77415ade0 2023-10-07T09:41:55.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.758+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe774158730 con 0x7fe77415ade0 2023-10-07T09:41:55.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.762+0000 7fe770ff9700 1 -- 172.21.15.29:0/657107698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe76c054ff0 con 0x7fe77415ade0 2023-10-07T09:41:55.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe758066f30 msgr2=0x7fe7580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe758066f30 0x7fe7580693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe768009260 tx=0x7fe768016000 comp rx=0 tx=0).stop 2023-10-07T09:41:55.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 msgr2=0x7fe774156eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 0x7fe774156eb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00cd50 tx=0x7fe76c001b10 comp rx=0 tx=0).stop 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 shutdown_connections 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe758066f30 0x7fe7580693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77415ade0 0x7fe774156eb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7740b0000 0x7fe7740c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 --2- 172.21.15.29:0/657107698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe774153cc0 0x7fe77413fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 >> 172.21.15.29:0/657107698 conn(0x7fe774067610 msgr2=0x7fe7740b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 shutdown_connections 2023-10-07T09:41:55.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.764+0000 7fe77b1ff700 1 -- 172.21.15.29:0/657107698 wait complete. 2023-10-07T09:41:55.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-07T09:41:55.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.980+0000 7f1ec0dff700 1 Processor -- start 2023-10-07T09:41:55.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.980+0000 7f1ec0dff700 1 -- start start 2023-10-07T09:41:55.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1ec0dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ebc157290 0x7f1ebc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1ec0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0ae8b0 0x7f1ebc0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1ec0dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b1840 0x7f1ebc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1ec0dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc0b1ca0 con 0x7f1ebc157290 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1ec0dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc158ad0 con 0x7f1ebc0b1840 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1ec0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc15b140 con 0x7f1ebc0ae8b0 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1eba59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ebc157290 0x7f1ebc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1eba59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ebc157290 0x7f1ebc0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46884/0 (socket says 172.21.15.29:46884) 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.981+0000 7f1eba59c700 1 -- 172.21.15.29:0/4142948025 learned_addr learned my addr 172.21.15.29:0/4142948025 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1ebad9d700 1 --2- 172.21.15.29:0/4142948025 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b1840 0x7f1ebc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eb9d9b700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0ae8b0 0x7f1ebc0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eba59c700 1 -- 172.21.15.29:0/4142948025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0ae8b0 msgr2=0x7f1ebc0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0ae8b0 0x7f1ebc0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eba59c700 1 -- 172.21.15.29:0/4142948025 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b1840 msgr2=0x7f1ebc0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4142948025 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b1840 0x7f1ebc0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eba59c700 1 -- 172.21.15.29:0/4142948025 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc15b420 con 0x7f1ebc157290 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1ebad9d700 1 --2- 172.21.15.29:0/4142948025 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b1840 0x7f1ebc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eb9d9b700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0ae8b0 0x7f1ebc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc157290 0x7f1ebc0cc870 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0b2220 tx=0x7f1eac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c85f0eec92eae3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eb959a700 1 -- 172.21.15.29:0/4142948025 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00b860 con 0x7f1ebc157290 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eb959a700 1 -- 172.21.15.29:0/4142948025 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00b9c0 con 0x7f1ebc157290 2023-10-07T09:41:55.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.982+0000 7f1eb959a700 1 -- 172.21.15.29:0/4142948025 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00bd60 con 0x7f1ebc157290 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4142948025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc157290 msgr2=0x7f1ebc0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc157290 0x7f1ebc0cc870 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0b2220 tx=0x7f1eac009250 comp rx=0 tx=0).stop 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4142948025 shutdown_connections 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4142948025 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b1840 0x7f1ebc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc157290 0x7f1ebc0cc870 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4142948025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0ae8b0 0x7f1ebc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4142948025 >> 172.21.15.29:0/4142948025 conn(0x7f1ebc0680a0 msgr2=0x7f1ebc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4142948025 shutdown_connections 2023-10-07T09:41:55.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.983+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4142948025 wait complete. 2023-10-07T09:41:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.985+0000 7f1ec0dff700 1 Processor -- start 2023-10-07T09:41:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.985+0000 7f1ec0dff700 1 -- start start 2023-10-07T09:41:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ec0dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0b1840 0x7f1ebc0c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ec0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc157290 0x7f1ebc1678d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ec0dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 0x7f1ebc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ec0dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc1535c0 con 0x7f1ebc0c1980 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ec0dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c21d0 con 0x7f1ebc0b1840 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ec0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc153810 con 0x7f1ebc157290 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 0x7f1ebc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 0x7f1ebc155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46898/0 (socket says 172.21.15.29:46898) 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 -- 172.21.15.29:0/4248075531 learned_addr learned my addr 172.21.15.29:0/4248075531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0b1840 0x7f1ebc0c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 -- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc157290 msgr2=0x7f1ebc1678d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1eb9d9b700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc157290 0x7f1ebc1678d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc157290 0x7f1ebc1678d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 -- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0b1840 msgr2=0x7f1ebc0c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0b1840 0x7f1ebc0c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1ebad9d700 1 -- 172.21.15.29:0/4248075531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0094f0 con 0x7f1ebc0c1980 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1eb9d9b700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc157290 0x7f1ebc1678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.986+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0b1840 0x7f1ebc0c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.987+0000 7f1ebad9d700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 0x7f1ebc155ba0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d0d0 tx=0x7f1eb40099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.987+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb400dd70 con 0x7f1ebc0c1980 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.987+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb400f040 con 0x7f1ebc0c1980 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.987+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4015670 con 0x7f1ebc0c1980 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.987+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc153af0 con 0x7f1ebc0c1980 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.987+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc167e10 con 0x7f1ebc0c1980 2023-10-07T09:41:55.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.988+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc04c3c0 con 0x7f1ebc0c1980 2023-10-07T09:41:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.989+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1eb401e3b0 con 0x7f1ebc0c1980 2023-10-07T09:41:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.990+0000 7f1eaa7fc700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1e9c066f30 0x7f1e9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.990+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f1eb4084c10 con 0x7f1ebc0c1980 2023-10-07T09:41:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.990+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1e9c066f30 0x7f1e9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:55.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.991+0000 7f1eba59c700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1e9c066f30 0x7f1e9c0693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0b2220 tx=0x7f1eac017000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:55.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:55.994+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eb4051d50 con 0x7f1ebc0c1980 2023-10-07T09:41:56.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:56.277+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f1ebc15b700 con 0x7f1ebc0c1980 2023-10-07T09:41:57.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.153+0000 7f1eaa7fc700 1 -- 172.21.15.29:0/4248075531 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v453) v1 ==== 158+0+0 (secure 0 0 0) 0x7f1eb401a0e0 con 0x7f1ebc0c1980 2023-10-07T09:41:57.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1e9c066f30 msgr2=0x7f1e9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:57.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1e9c066f30 0x7f1e9c0693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0b2220 tx=0x7f1eac017000 comp rx=0 tx=0).stop 2023-10-07T09:41:57.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 msgr2=0x7f1ebc155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:57.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 0x7f1ebc155ba0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d0d0 tx=0x7f1eb40099d0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 shutdown_connections 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1e9c066f30 0x7f1e9c0693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0c1980 0x7f1ebc155ba0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0b1840 0x7f1ebc0c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 --2- 172.21.15.29:0/4248075531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc157290 0x7f1ebc1678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 >> 172.21.15.29:0/4248075531 conn(0x7f1ebc0680a0 msgr2=0x7f1ebc0ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 shutdown_connections 2023-10-07T09:41:57.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.156+0000 7f1ec0dff700 1 -- 172.21.15.29:0/4248075531 wait complete. 2023-10-07T09:41:57.177 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-07T09:41:57.178 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-07T09:41:57.178 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:41:57.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-07T09:41:57.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.378+0000 7f06c8dff700 1 Processor -- start 2023-10-07T09:41:57.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.379+0000 7f06c8dff700 1 -- start start 2023-10-07T09:41:57.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c8dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159750 0x7f06c40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c8dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f06c4159b20 0x7f06c40ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c8dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06c40ce6d0 0x7f06c40ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c8dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c40cbf20 con 0x7f06c40ce6d0 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c8dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c40cc200 con 0x7f06c4159b20 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c8dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40cc4e0 con 0x7f06c4159750 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c2d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06c40ce6d0 0x7f06c40ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c2d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06c40ce6d0 0x7f06c40ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46914/0 (socket says 172.21.15.29:46914) 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c2d9d700 1 -- 172.21.15.29:0/4268243424 learned_addr learned my addr 172.21.15.29:0/4268243424 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c259c700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159750 0x7f06c40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.380+0000 7f06c1d9b700 1 --2- 172.21.15.29:0/4268243424 >> v2:172.21.15.144:3300/0 conn(0x7f06c4159b20 0x7f06c40ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c2d9d700 1 -- 172.21.15.29:0/4268243424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159750 msgr2=0x7f06c40d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c2d9d700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159750 0x7f06c40d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c2d9d700 1 -- 172.21.15.29:0/4268243424 >> v2:172.21.15.144:3300/0 conn(0x7f06c4159b20 msgr2=0x7f06c40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c2d9d700 1 --2- 172.21.15.29:0/4268243424 >> v2:172.21.15.144:3300/0 conn(0x7f06c4159b20 0x7f06c40ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c2d9d700 1 -- 172.21.15.29:0/4268243424 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c40cc7c0 con 0x7f06c40ce6d0 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c259c700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159750 0x7f06c40d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c2d9d700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c40ce6d0 0x7f06c40ceb50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f06c4088830 tx=0x7f06bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275916af765d08e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c159a700 1 -- 172.21.15.29:0/4268243424 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc00e070 con 0x7f06c40ce6d0 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c159a700 1 -- 172.21.15.29:0/4268243424 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06bc00ba20 con 0x7f06c40ce6d0 2023-10-07T09:41:57.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.381+0000 7f06c159a700 1 -- 172.21.15.29:0/4268243424 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc0092e0 con 0x7f06c40ce6d0 2023-10-07T09:41:57.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 -- 172.21.15.29:0/4268243424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c40ce6d0 msgr2=0x7f06c40ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c40ce6d0 0x7f06c40ceb50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f06c4088830 tx=0x7f06bc009840 comp rx=0 tx=0).stop 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 -- 172.21.15.29:0/4268243424 shutdown_connections 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c40ce6d0 0x7f06c40ceb50 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 --2- 172.21.15.29:0/4268243424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159750 0x7f06c40d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 --2- 172.21.15.29:0/4268243424 >> v2:172.21.15.144:3300/0 conn(0x7f06c4159b20 0x7f06c40ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 -- 172.21.15.29:0/4268243424 >> 172.21.15.29:0/4268243424 conn(0x7f06c4063a10 msgr2=0x7f06c40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 -- 172.21.15.29:0/4268243424 shutdown_connections 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.382+0000 7f06c8dff700 1 -- 172.21.15.29:0/4268243424 wait complete. 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.383+0000 7f06c8dff700 1 Processor -- start 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.384+0000 7f06c8dff700 1 -- start start 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.384+0000 7f06c8dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c40ce6d0 0x7f06c414aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c8dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 0x7f06c40c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c8dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159b20 0x7f06c414b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c8dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c414c880 con 0x7f06c4159750 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c8dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c414bfe0 con 0x7f06c40ce6d0 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c8dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c414c2c0 con 0x7f06c4159b20 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 0x7f06c40c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 0x7f06c40c8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46928/0 (socket says 172.21.15.29:46928) 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 -- 172.21.15.29:0/2811302121 learned_addr learned my addr 172.21.15.29:0/2811302121 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c259c700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c40ce6d0 0x7f06c414aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c2d9d700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159b20 0x7f06c414b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 -- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159b20 msgr2=0x7f06c414b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159b20 0x7f06c414b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 -- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c40ce6d0 msgr2=0x7f06c414aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c40ce6d0 0x7f06c414aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c1d9b700 1 -- 172.21.15.29:0/2811302121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc009ae0 con 0x7f06c4159750 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.385+0000 7f06c2d9d700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159b20 0x7f06c414b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:41:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.386+0000 7f06c1d9b700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 0x7f06c40c8b60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f06b800e930 tx=0x7f06b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.386+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8013680 con 0x7f06c4159750 2023-10-07T09:41:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.386+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b80137e0 con 0x7f06c4159750 2023-10-07T09:41:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.386+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c414c5a0 con 0x7f06c4159750 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.386+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8013b80 con 0x7f06c4159750 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.386+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c414bcf0 con 0x7f06c4159750 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.387+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c406f0b0 con 0x7f06c4159750 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.388+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f06b800c460 con 0x7f06c4159750 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.389+0000 7f06b27fc700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f06a4066f30 0x7f06a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.389+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f06b8084890 con 0x7f06c4159750 2023-10-07T09:41:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.390+0000 7f06c259c700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f06a4066f30 0x7f06a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.390+0000 7f06c259c700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f06a4066f30 0x7f06a40693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06c41590c0 tx=0x7f06b4008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.393+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06b80519d0 con 0x7f06c4159750 2023-10-07T09:41:57.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:57.678+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f06c404c3c0 con 0x7f06c4159750 2023-10-07T09:41:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.160+0000 7f06b27fc700 1 -- 172.21.15.29:0/2811302121 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v454) v1 ==== 157+0+0 (secure 0 0 0) 0x7f06b80170e0 con 0x7f06c4159750 2023-10-07T09:41:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f06a4066f30 msgr2=0x7f06a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f06a4066f30 0x7f06a40693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06c41590c0 tx=0x7f06b4008040 comp rx=0 tx=0).stop 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 msgr2=0x7f06c40c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 0x7f06c40c8b60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f06b800e930 tx=0x7f06b80091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 shutdown_connections 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f06a4066f30 0x7f06a40693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06c4159b20 0x7f06c414b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c40ce6d0 0x7f06c414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 --2- 172.21.15.29:0/2811302121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06c4159750 0x7f06c40c8b60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.162+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 >> 172.21.15.29:0/2811302121 conn(0x7f06c4063a10 msgr2=0x7f06c40c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.163+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 shutdown_connections 2023-10-07T09:41:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.163+0000 7f06c8dff700 1 -- 172.21.15.29:0/2811302121 wait complete. 2023-10-07T09:41:58.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:41:58.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.385+0000 7f9d909ff700 1 Processor -- start 2023-10-07T09:41:58.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.385+0000 7f9d909ff700 1 -- start start 2023-10-07T09:41:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.386+0000 7f9d909ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d8c0ae8a0 0x7f9d8c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.386+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 0x7f9d8c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.386+0000 7f9d909ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0acb40 con 0x7f9d8c15a960 2023-10-07T09:41:58.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.386+0000 7f9d909ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c158c60 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.386+0000 7f9d909ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0abf30 con 0x7f9d8c15ad30 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d8ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 0x7f9d8c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d89d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d8c0ae8a0 0x7f9d8c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d8ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 0x7f9d8c0b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53394/0 (socket says 172.21.15.29:53394) 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d8ad9d700 1 -- 172.21.15.29:0/792001660 learned_addr learned my addr 172.21.15.29:0/792001660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d8959a700 1 -- 172.21.15.29:0/792001660 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857640069 0 0) 0x7f9d8c0acb40 con 0x7f9d8c15a960 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d8959a700 1 -- 172.21.15.29:0/792001660 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d74003570 con 0x7f9d8c15a960 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d89d9b700 1 -- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 msgr2=0x7f9d8c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d89d9b700 1 --2- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 0x7f9d8c0b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d89d9b700 1 -- 172.21.15.29:0/792001660 >> v1:172.21.15.144:6789/0 conn(0x7f9d8c15a960 legacy=0x7f9d8c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d89d9b700 1 -- 172.21.15.29:0/792001660 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0ac210 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.387+0000 7f9d8ad9d700 1 --2- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 0x7f9d8c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.388+0000 7f9d89d9b700 1 --2- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c158e20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9d800098e0 tx=0x7f9d800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b46a45d808787eb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.388+0000 7f9d8959a700 1 -- 172.21.15.29:0/792001660 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80011630 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.388+0000 7f9d8959a700 1 -- 172.21.15.29:0/792001660 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d80011790 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.388+0000 7f9d8959a700 1 -- 172.21.15.29:0/792001660 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80011b30 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 -- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c0ae8a0 msgr2=0x7f9d8c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 --2- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c158e20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9d800098e0 tx=0x7f9d800091a0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 -- 172.21.15.29:0/792001660 shutdown_connections 2023-10-07T09:41:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 --2- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15ad30 0x7f9d8c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 --2- 172.21.15.29:0/792001660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c158e20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 -- 172.21.15.29:0/792001660 >> 172.21.15.29:0/792001660 conn(0x7f9d8c067610 msgr2=0x7f9d8c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 -- 172.21.15.29:0/792001660 shutdown_connections 2023-10-07T09:41:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.389+0000 7f9d909ff700 1 -- 172.21.15.29:0/792001660 wait complete. 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.390+0000 7f9d909ff700 1 Processor -- start 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.390+0000 7f9d909ff700 1 -- start start 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.391+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.391+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15a960 0x7f9d8c0af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.391+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c15ad30 0x7f9d8c0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.391+0000 7f9d909ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0af4f0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.391+0000 7f9d909ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c14eb60 con 0x7f9d8c15ad30 2023-10-07T09:41:58.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.391+0000 7f9d909ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0a69b0 con 0x7f9d8c15a960 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c14e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46932/0 (socket says 172.21.15.29:46932) 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 -- 172.21.15.29:0/206973957 learned_addr learned my addr 172.21.15.29:0/206973957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d89d9b700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15a960 0x7f9d8c0af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8ad9d700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c15ad30 0x7f9d8c0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 -- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15a960 msgr2=0x7f9d8c0af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15a960 0x7f9d8c0af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 -- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c15ad30 msgr2=0x7f9d8c0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c15ad30 0x7f9d8c0afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 -- 172.21.15.29:0/206973957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d84009ae0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.392+0000 7f9d8a59c700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c14e620 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c7c0 tx=0x7f9d7c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.393+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c010510 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.393+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c010670 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.393+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c010a10 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.393+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d80009420 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.393+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c0b02f0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.394+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c04c3c0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.395+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9d7c01b3b0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.396+0000 7f9d7a7fc700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d6c066f30 0x7f9d6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.396+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f9d7c083ec0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.396+0000 7f9d89d9b700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d6c066f30 0x7f9d6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.397+0000 7f9d89d9b700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d6c066f30 0x7f9d6c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9d800091e0 tx=0x7f9d80017000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.400+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d7c051000 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.600+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9d8c0abbe0 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.604+0000 7f9d7a7fc700 1 -- 172.21.15.29:0/206973957 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d7c017020 con 0x7f9d8c0ae8a0 2023-10-07T09:41:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d6c066f30 msgr2=0x7f9d6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d6c066f30 0x7f9d6c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9d800091e0 tx=0x7f9d80017000 comp rx=0 tx=0).stop 2023-10-07T09:41:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 msgr2=0x7f9d8c14e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c14e620 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c7c0 tx=0x7f9d7c007e10 comp rx=0 tx=0).stop 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 shutdown_connections 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d6c066f30 0x7f9d6c0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8c15ad30 0x7f9d8c0afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d8c0ae8a0 0x7f9d8c14e620 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 --2- 172.21.15.29:0/206973957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8c15a960 0x7f9d8c0af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 >> 172.21.15.29:0/206973957 conn(0x7f9d8c067610 msgr2=0x7f9d8c14f380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 shutdown_connections 2023-10-07T09:41:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:58.607+0000 7f9d909ff700 1 -- 172.21.15.29:0/206973957 wait complete. 2023-10-07T09:41:58.625 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:41:58.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:41:58.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:41:58.832 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:41:58.833 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:41:58.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:41:59.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.027+0000 7f9de5dff700 1 Processor -- start 2023-10-07T09:41:59.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.027+0000 7f9de5dff700 1 -- start start 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9de5dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9de0159ec0 0x7f9de015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9de5dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9de00af400 0x7f9de00af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9de5dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de014d790 con 0x7f9de00ad370 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9de5dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de015a880 con 0x7f9de0159ec0 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9de5dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de00ae5d0 con 0x7f9de00af400 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9de0159ec0 0x7f9de015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9de0159ec0 0x7f9de015a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46942/0 (socket says 172.21.15.29:46942) 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 -- 172.21.15.29:0/1464639102 learned_addr learned my addr 172.21.15.29:0/1464639102 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddffff700 1 --2- 172.21.15.29:0/1464639102 >> v2:172.21.15.144:3300/0 conn(0x7f9de00af400 0x7f9de00af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 -- 172.21.15.29:0/1464639102 >> v2:172.21.15.144:3300/0 conn(0x7f9de00af400 msgr2=0x7f9de00af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 --2- 172.21.15.29:0/1464639102 >> v2:172.21.15.144:3300/0 conn(0x7f9de00af400 0x7f9de00af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 -- 172.21.15.29:0/1464639102 >> v1:172.21.15.144:6789/0 conn(0x7f9de00ad370 legacy=0x7f9de00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddeffd700 1 -- 172.21.15.29:0/1464639102 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00ad740 con 0x7f9de0159ec0 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.029+0000 7f9ddffff700 1 --2- 172.21.15.29:0/1464639102 >> v2:172.21.15.144:3300/0 conn(0x7f9de00af400 0x7f9de00af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9ddeffd700 1 --2- 172.21.15.29:0/1464639102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de0159ec0 0x7f9de015a340 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9de00b1270 tx=0x7f9dd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6f78a6c0f6c795 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9dde7fc700 1 -- 172.21.15.29:0/1464639102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd400b820 con 0x7f9de0159ec0 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9dde7fc700 1 -- 172.21.15.29:0/1464639102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd400b980 con 0x7f9de0159ec0 2023-10-07T09:41:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9dde7fc700 1 -- 172.21.15.29:0/1464639102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd400bd20 con 0x7f9de0159ec0 2023-10-07T09:41:59.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9de5dff700 1 -- 172.21.15.29:0/1464639102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de0159ec0 msgr2=0x7f9de015a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:59.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9de5dff700 1 --2- 172.21.15.29:0/1464639102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de0159ec0 0x7f9de015a340 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9de00b1270 tx=0x7f9dd4009250 comp rx=0 tx=0).stop 2023-10-07T09:41:59.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9de5dff700 1 -- 172.21.15.29:0/1464639102 shutdown_connections 2023-10-07T09:41:59.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9de5dff700 1 --2- 172.21.15.29:0/1464639102 >> v2:172.21.15.144:3300/0 conn(0x7f9de00af400 0x7f9de00af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9de5dff700 1 --2- 172.21.15.29:0/1464639102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de0159ec0 0x7f9de015a340 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.030+0000 7f9de5dff700 1 -- 172.21.15.29:0/1464639102 >> 172.21.15.29:0/1464639102 conn(0x7f9de00680a0 msgr2=0x7f9de00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:59.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.031+0000 7f9de5dff700 1 -- 172.21.15.29:0/1464639102 shutdown_connections 2023-10-07T09:41:59.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.031+0000 7f9de5dff700 1 -- 172.21.15.29:0/1464639102 wait complete. 2023-10-07T09:41:59.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.032+0000 7f9de5dff700 1 Processor -- start 2023-10-07T09:41:59.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.032+0000 7f9de5dff700 1 -- start start 2023-10-07T09:41:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9de5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de00ad370 0x7f9de016a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9de5dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 0x7f9de00c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9de5dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de0159ec0 0x7f9de00c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9de5dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de016ab50 con 0x7f9de00af400 2023-10-07T09:41:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9de5dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00c3350 con 0x7f9de0159ec0 2023-10-07T09:41:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9de5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00c2740 con 0x7f9de00ad370 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9ddeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 0x7f9de00c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9ddeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 0x7f9de00c7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46958/0 (socket says 172.21.15.29:46958) 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.033+0000 7f9ddeffd700 1 -- 172.21.15.29:0/642823826 learned_addr learned my addr 172.21.15.29:0/642823826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddffff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de0159ec0 0x7f9de00c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddf7fe700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de00ad370 0x7f9de016a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddeffd700 1 -- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de00ad370 msgr2=0x7f9de016a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddeffd700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de00ad370 0x7f9de016a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddeffd700 1 -- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de0159ec0 msgr2=0x7f9de00c37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddeffd700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de0159ec0 0x7f9de00c37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddeffd700 1 -- 172.21.15.29:0/642823826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40094f0 con 0x7f9de00af400 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddf7fe700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de00ad370 0x7f9de016a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:41:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9ddeffd700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 0x7f9de00c7b60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40030c0 tx=0x7f9dd4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4011e60 con 0x7f9de00af400 2023-10-07T09:41:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd4020070 con 0x7f9de00af400 2023-10-07T09:41:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd401b6a0 con 0x7f9de00af400 2023-10-07T09:41:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.034+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de00c2a20 con 0x7f9de00af400 2023-10-07T09:41:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.035+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de00c2fe0 con 0x7f9de00af400 2023-10-07T09:41:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.036+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de00b1350 con 0x7f9de00af400 2023-10-07T09:41:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.037+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9dd4022030 con 0x7f9de00af400 2023-10-07T09:41:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.038+0000 7f9dc77fe700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9dcc066f30 0x7f9dcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:41:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.038+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f9dd4086560 con 0x7f9de00af400 2023-10-07T09:41:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.038+0000 7f9ddf7fe700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9dcc066f30 0x7f9dcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:41:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.039+0000 7f9ddf7fe700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9dcc066f30 0x7f9dcc0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9de00cbc40 tx=0x7f9dd0009040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:41:59.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.042+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9dd4053740 con 0x7f9de00af400 2023-10-07T09:41:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.289+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9de00b1350 con 0x7f9de00af400 2023-10-07T09:41:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.290+0000 7f9dc77fe700 1 -- 172.21.15.29:0/642823826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f9dd4056d60 con 0x7f9de00af400 2023-10-07T09:41:59.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:41:59.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4736,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2128221113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2128221113},{"type":"v1","addr":"172.21.15.144:6837","nonce":2128221113}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4953,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1394397990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1394397990},{"type":"v1","addr":"172.21.15.29:6834","nonce":1394397990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/145327644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":145327644},{"type":"v1","addr":"172.21.15.29:6835","nonce":145327644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4058608785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4058608785},{"type":"v1","addr":"172.21.15.144:6833","nonce":4058608785}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[]} 2023-10-07T09:41:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9dcc066f30 msgr2=0x7f9dcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9dcc066f30 0x7f9dcc0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9de00cbc40 tx=0x7f9dd0009040 comp rx=0 tx=0).stop 2023-10-07T09:41:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 msgr2=0x7f9de00c7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:41:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 0x7f9de00c7b60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40030c0 tx=0x7f9dd4016000 comp rx=0 tx=0).stop 2023-10-07T09:41:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 shutdown_connections 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9dcc066f30 0x7f9dcc0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de0159ec0 0x7f9de00c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de00ad370 0x7f9de016a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 --2- 172.21.15.29:0/642823826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de00af400 0x7f9de00c7b60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 >> 172.21.15.29:0/642823826 conn(0x7f9de00680a0 msgr2=0x7f9de014f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 shutdown_connections 2023-10-07T09:41:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:41:59.292+0000 7f9de5dff700 1 -- 172.21.15.29:0/642823826 wait complete. 2023-10-07T09:41:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-07T09:41:59.311 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:41:59.311 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:41:59.311 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:41:59.311 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:41:59.312 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:41:59.312 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:41:59.312 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:41:59.312 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:41:59.312 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:41:59.312 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:41:59.313 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:41:59.313 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:41:59.313 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:41:59.313 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:41:59.313 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:41:59.313 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:41:59.325 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:41:59.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:41:59.326 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:41:59.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:41:59.334 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:41:59.334 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:41:59.369 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:41:59.370 DEBUG:teuthology.parallel:result is None 2023-10-07T09:41:59.372 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:41:59.372 DEBUG:teuthology.parallel:result is None 2023-10-07T09:41:59.379 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:41:59.379 DEBUG:teuthology.parallel:result is None 2023-10-07T09:41:59.512 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:41:59.512 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:41:59.514 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T09:41:59.513+0000 7f18743fd700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-07T09:41:59.522 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:42:05.335 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:42:05.335 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:42:05.340 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:42:05.340 DEBUG:teuthology.parallel:result is None 2023-10-07T09:42:05.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:42:05.485 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:42:05.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.548+0000 7fbc7e3ff700 1 Processor -- start 2023-10-07T09:42:05.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.548+0000 7fbc7e3ff700 1 -- start start 2023-10-07T09:42:05.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc7e3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc7815ade0 0x7fbc780b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:05.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc7e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc780b0540 0x7fbc7815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:05.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc7e3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780b09b0 con 0x7fbc78153cc0 2023-10-07T09:42:05.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc7e3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc78154090 con 0x7fbc7815ade0 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc7e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78159f60 con 0x7fbc780b0540 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc777fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc7815ade0 0x7fbc780b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc777fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc7815ade0 0x7fbc780b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39086/0 (socket says 172.21.15.29:39086) 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc777fe700 1 -- 172.21.15.29:0/405282750 learned_addr learned my addr 172.21.15.29:0/405282750 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc77fff700 1 --1- 172.21.15.29:0/405282750 >> v1:172.21.15.144:6789/0 conn(0x7fbc78153cc0 0x7fbc780b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:54634/0 (socket says 172.21.15.29:54634) 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.549+0000 7fbc7c99c700 1 --2- 172.21.15.29:0/405282750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc780b0540 0x7fbc7815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc777fe700 1 -- 172.21.15.29:0/405282750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc780b0540 msgr2=0x7fbc7815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc777fe700 1 --2- 172.21.15.29:0/405282750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc780b0540 0x7fbc7815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc76ffd700 1 -- 172.21.15.29:0/405282750 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191648156 0 0) 0x7fbc780b09b0 con 0x7fbc78153cc0 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc777fe700 1 -- 172.21.15.29:0/405282750 >> v1:172.21.15.144:6789/0 conn(0x7fbc78153cc0 legacy=0x7fbc780b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc777fe700 1 -- 172.21.15.29:0/405282750 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7815a240 con 0x7fbc7815ade0 2023-10-07T09:42:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc777fe700 1 --2- 172.21.15.29:0/405282750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7815ade0 0x7fbc780b0000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0099b0 tx=0x7fbc6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a91e0da546fff3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc76ffd700 1 -- 172.21.15.29:0/405282750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c009ea0 con 0x7fbc7815ade0 2023-10-07T09:42:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc76ffd700 1 -- 172.21.15.29:0/405282750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00b860 con 0x7fbc7815ade0 2023-10-07T09:42:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.550+0000 7fbc76ffd700 1 -- 172.21.15.29:0/405282750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00bc00 con 0x7fbc7815ade0 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/405282750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7815ade0 msgr2=0x7fbc780b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/405282750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7815ade0 0x7fbc780b0000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0099b0 tx=0x7fbc6c009250 comp rx=0 tx=0).stop 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/405282750 shutdown_connections 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/405282750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc780b0540 0x7fbc7815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/405282750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7815ade0 0x7fbc780b0000 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/405282750 >> 172.21.15.29:0/405282750 conn(0x7fbc78067610 msgr2=0x7fbc78067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/405282750 shutdown_connections 2023-10-07T09:42:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.551+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/405282750 wait complete. 2023-10-07T09:42:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.552+0000 7fbc7e3ff700 1 Processor -- start 2023-10-07T09:42:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.553+0000 7fbc7e3ff700 1 -- start start 2023-10-07T09:42:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7e3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc780b0540 0x7fbc78156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7e3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 0x7fbc78155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc7815ade0 0x7fbc78158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7e3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781591d0 con 0x7fbc78153cc0 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7e3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781567e0 con 0x7fbc780b0540 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78155bd0 con 0x7fbc7815ade0 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 0x7fbc78155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 0x7fbc78155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39092/0 (socket says 172.21.15.29:39092) 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 -- 172.21.15.29:0/2921022655 learned_addr learned my addr 172.21.15.29:0/2921022655 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc77fff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc780b0540 0x7fbc78156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc7c99c700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc7815ade0 0x7fbc78158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 -- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc7815ade0 msgr2=0x7fbc78158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc7815ade0 0x7fbc78158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 -- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc780b0540 msgr2=0x7fbc78156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc780b0540 0x7fbc78156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.554+0000 7fbc777fe700 1 -- 172.21.15.29:0/2921022655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c0094f0 con 0x7fbc78153cc0 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.555+0000 7fbc777fe700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 0x7fbc78155510 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbc780b0be0 tx=0x7fbc6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.555+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c000d10 con 0x7fbc78153cc0 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.555+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00bbe0 con 0x7fbc78153cc0 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.555+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c011690 con 0x7fbc78153cc0 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.555+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc78155eb0 con 0x7fbc78153cc0 2023-10-07T09:42:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.555+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc78156470 con 0x7fbc78153cc0 2023-10-07T09:42:05.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.556+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc78156950 con 0x7fbc78153cc0 2023-10-07T09:42:05.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.557+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbc6c011ac0 con 0x7fbc78153cc0 2023-10-07T09:42:05.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.558+0000 7fbc5ffff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc64066f30 0x7fbc640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:05.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.558+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fbc6c085ef0 con 0x7fbc78153cc0 2023-10-07T09:42:05.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.558+0000 7fbc77fff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc64066f30 0x7fbc640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:05.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.559+0000 7fbc77fff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc64066f30 0x7fbc640693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbc68001230 tx=0x7fbc68007d20 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:05.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.562+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc6c053030 con 0x7fbc78153cc0 2023-10-07T09:42:05.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-10-07T09:42:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:05.781+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbc7804c3c0 con 0x7fbc78153cc0 2023-10-07T09:42:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.191+0000 7fbc5ffff700 1 -- 172.21.15.29:0/2921022655 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbc6c015080 con 0x7fbc78153cc0 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc64066f30 msgr2=0x7fbc640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc64066f30 0x7fbc640693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbc68001230 tx=0x7fbc68007d20 comp rx=0 tx=0).stop 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 msgr2=0x7fbc78155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 0x7fbc78155510 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbc780b0be0 tx=0x7fbc6c017000 comp rx=0 tx=0).stop 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 shutdown_connections 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc64066f30 0x7fbc640693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc7815ade0 0x7fbc78158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc780b0540 0x7fbc78156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 --2- 172.21.15.29:0/2921022655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78153cc0 0x7fbc78155510 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 >> 172.21.15.29:0/2921022655 conn(0x7fbc78067610 msgr2=0x7fbc780aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 shutdown_connections 2023-10-07T09:42:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.194+0000 7fbc7e3ff700 1 -- 172.21.15.29:0/2921022655 wait complete. 2023-10-07T09:42:07.197 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:42:07.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:42:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.417+0000 7faf797ff700 1 Processor -- start 2023-10-07T09:42:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.417+0000 7faf797ff700 1 -- start start 2023-10-07T09:42:07.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.418+0000 7faf797ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf740ad370 0x7faf740cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.418+0000 7faf797ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7414d790 con 0x7faf74159ec0 2023-10-07T09:42:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.418+0000 7faf797ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf740ae5d0 con 0x7faf740b0410 2023-10-07T09:42:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.418+0000 7faf797ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf740ad740 con 0x7faf740ad370 2023-10-07T09:42:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf727fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faf74159ec0 0x7faf741568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59044/0 (socket says 172.21.15.29:59044) 2023-10-07T09:42:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf727fc700 1 -- 172.21.15.29:0/490721859 learned_addr learned my addr 172.21.15.29:0/490721859 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf72ffd700 1 --2- 172.21.15.29:0/490721859 >> v2:172.21.15.144:3300/0 conn(0x7faf740ad370 0x7faf740cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380881051 0 0) 0x7faf7414d790 con 0x7faf74159ec0 2023-10-07T09:42:07.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c003570 con 0x7faf74159ec0 2023-10-07T09:42:07.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146714679 0 0) 0x7faf740ae5d0 con 0x7faf740b0410 2023-10-07T09:42:07.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf7414d790 con 0x7faf740b0410 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf72ffd700 1 -- 172.21.15.29:0/490721859 >> v1:172.21.15.144:6789/0 conn(0x7faf740b0410 legacy=0x7faf74156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2066924400 0 0) 0x7faf5c003570 con 0x7faf74159ec0 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf72ffd700 1 -- 172.21.15.29:0/490721859 >> v1:172.21.15.29:6789/0 conn(0x7faf74159ec0 legacy=0x7faf741568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.419+0000 7faf72ffd700 1 -- 172.21.15.29:0/490721859 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf740ccdb0 con 0x7faf740ad370 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.420+0000 7faf72ffd700 1 --2- 172.21.15.29:0/490721859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740ad370 0x7faf740cc870 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faf64009910 tx=0x7faf640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb1e06af141cdc6c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.420+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68001470 con 0x7faf740ad370 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.420+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf64011420 con 0x7faf740ad370 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.420+0000 7faf71ffb700 1 -- 172.21.15.29:0/490721859 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640117c0 con 0x7faf740ad370 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 -- 172.21.15.29:0/490721859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740ad370 msgr2=0x7faf740cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 --2- 172.21.15.29:0/490721859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740ad370 0x7faf740cc870 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faf64009910 tx=0x7faf640091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 -- 172.21.15.29:0/490721859 shutdown_connections 2023-10-07T09:42:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 --2- 172.21.15.29:0/490721859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740ad370 0x7faf740cc870 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 -- 172.21.15.29:0/490721859 >> 172.21.15.29:0/490721859 conn(0x7faf740680a0 msgr2=0x7faf740cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 -- 172.21.15.29:0/490721859 shutdown_connections 2023-10-07T09:42:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.421+0000 7faf797ff700 1 -- 172.21.15.29:0/490721859 wait complete. 2023-10-07T09:42:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.422+0000 7faf797ff700 1 Processor -- start 2023-10-07T09:42:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 -- start start 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 0x7faf74150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740b0410 0x7faf7414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf74159ec0 0x7faf740c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7414f540 con 0x7faf740ad370 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf740c26f0 con 0x7faf740b0410 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.423+0000 7faf797ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7414f770 con 0x7faf74159ec0 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 0x7faf74150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 0x7faf74150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39106/0 (socket says 172.21.15.29:39106) 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 -- 172.21.15.29:0/1499915777 learned_addr learned my addr 172.21.15.29:0/1499915777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf727fc700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740b0410 0x7faf7414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf737fe700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf74159ec0 0x7faf740c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 -- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf74159ec0 msgr2=0x7faf740c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf74159ec0 0x7faf740c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 -- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740b0410 msgr2=0x7faf7414f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740b0410 0x7faf7414f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 -- 172.21.15.29:0/1499915777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf64009420 con 0x7faf740ad370 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf727fc700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740b0410 0x7faf7414f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.424+0000 7faf72ffd700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 0x7faf74150810 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faf6400a010 tx=0x7faf64016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.425+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6401bc80 con 0x7faf740ad370 2023-10-07T09:42:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.425+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6401bde0 con 0x7faf740ad370 2023-10-07T09:42:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.425+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6401c680 con 0x7faf740ad370 2023-10-07T09:42:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.425+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7414fa50 con 0x7faf740ad370 2023-10-07T09:42:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.425+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf74150010 con 0x7faf740ad370 2023-10-07T09:42:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.426+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7404c3c0 con 0x7faf740ad370 2023-10-07T09:42:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.427+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faf6401ca70 con 0x7faf740ad370 2023-10-07T09:42:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.428+0000 7faf5affd700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf60066f30 0x7faf600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.428+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5201+0+0 (secure 0 0 0) 0x7faf640856c0 con 0x7faf740ad370 2023-10-07T09:42:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.428+0000 7faf727fc700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf60066f30 0x7faf600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.429+0000 7faf727fc700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf60066f30 0x7faf600693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faf68000ff0 tx=0x7faf680011a0 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:07.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.432+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf64052a80 con 0x7faf740ad370 2023-10-07T09:42:07.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.714+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faf740acee0 con 0x7faf740ad370 2023-10-07T09:42:07.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.714+0000 7faf5affd700 1 -- 172.21.15.29:0/1499915777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7faf640560a0 con 0x7faf740ad370 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:42:07.716 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:42:07.719 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:42:07.721 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.722 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:42:07.724 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:42:07.725 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:42:07.728 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:42:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.730 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:07.731 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:42:07.732 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:42:07.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf60066f30 msgr2=0x7faf600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf60066f30 0x7faf600693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faf68000ff0 tx=0x7faf680011a0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 msgr2=0x7faf74150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 0x7faf74150810 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faf6400a010 tx=0x7faf64016000 comp rx=0 tx=0).stop 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 shutdown_connections 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf60066f30 0x7faf600693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf74159ec0 0x7faf740c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf740ad370 0x7faf74150810 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 --2- 172.21.15.29:0/1499915777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf740b0410 0x7faf7414f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.716+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 >> 172.21.15.29:0/1499915777 conn(0x7faf740680a0 msgr2=0x7faf74157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.717+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 shutdown_connections 2023-10-07T09:42:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.717+0000 7faf797ff700 1 -- 172.21.15.29:0/1499915777 wait complete. 2023-10-07T09:42:07.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:42:07.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.930+0000 7fcc11dff700 1 Processor -- start 2023-10-07T09:42:07.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.930+0000 7fcc11dff700 1 -- start start 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc11dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc11dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c0b0410 0x7fcc0c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc11dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c14d790 con 0x7fcc0c159ec0 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc11dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0c0ae5d0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc11dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c15a850 con 0x7fcc0c0ad370 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc0bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c0b0410 0x7fcc0c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.931+0000 7fcc0bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c0b0410 0x7fcc0c15a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39118/0 (socket says 172.21.15.29:39118) 2023-10-07T09:42:07.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0bfff700 1 -- 172.21.15.29:0/697265804 learned_addr learned my addr 172.21.15.29:0/697265804 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:07.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0b7fe700 1 --2- 172.21.15.29:0/697265804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0bfff700 1 -- 172.21.15.29:0/697265804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 msgr2=0x7fcc0c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:07.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0bfff700 1 --2- 172.21.15.29:0/697265804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0bfff700 1 -- 172.21.15.29:0/697265804 >> v1:172.21.15.144:6789/0 conn(0x7fcc0c159ec0 legacy=0x7fcc0c1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0bfff700 1 -- 172.21.15.29:0/697265804 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c0ad740 con 0x7fcc0c0b0410 2023-10-07T09:42:07.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0bfff700 1 --2- 172.21.15.29:0/697265804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c15a310 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009f20 tx=0x7fcc04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d82446a90ada9a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0a7fc700 1 -- 172.21.15.29:0/697265804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400e070 con 0x7fcc0c0b0410 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0a7fc700 1 -- 172.21.15.29:0/697265804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0400ba80 con 0x7fcc0c0b0410 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.932+0000 7fcc0a7fc700 1 -- 172.21.15.29:0/697265804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040092e0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 -- 172.21.15.29:0/697265804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 msgr2=0x7fcc0c15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 --2- 172.21.15.29:0/697265804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c15a310 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009f20 tx=0x7fcc04009840 comp rx=0 tx=0).stop 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 -- 172.21.15.29:0/697265804 shutdown_connections 2023-10-07T09:42:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 --2- 172.21.15.29:0/697265804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c15a310 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 --2- 172.21.15.29:0/697265804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 -- 172.21.15.29:0/697265804 >> 172.21.15.29:0/697265804 conn(0x7fcc0c0680a0 msgr2=0x7fcc0c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:07.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 -- 172.21.15.29:0/697265804 shutdown_connections 2023-10-07T09:42:07.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.933+0000 7fcc11dff700 1 -- 172.21.15.29:0/697265804 wait complete. 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.935+0000 7fcc11dff700 1 Processor -- start 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.935+0000 7fcc11dff700 1 -- start start 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc11dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc11dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c14ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc11dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc0c159ec0 0x7fcc0c15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc11dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c06da10 con 0x7fcc0c0b0410 2023-10-07T09:42:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc11dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c06dcf0 con 0x7fcc0c159ec0 2023-10-07T09:42:07.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc11dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c06dfd0 con 0x7fcc0c0ad370 2023-10-07T09:42:07.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c14ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c14ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39126/0 (socket says 172.21.15.29:39126) 2023-10-07T09:42:07.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 -- 172.21.15.29:0/472839232 learned_addr learned my addr 172.21.15.29:0/472839232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:07.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0bfff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc0c159ec0 0x7fcc0c15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0b7fe700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 -- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 msgr2=0x7fcc0c06eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c06eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 -- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc0c159ec0 msgr2=0x7fcc0c15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:07.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc0c159ec0 0x7fcc0c15bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:07.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.936+0000 7fcc0affd700 1 -- 172.21.15.29:0/472839232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc04009ae0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.937+0000 7fcc0affd700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c14ada0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c590 tx=0x7fcc0000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.937+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00011590 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.937+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00011710 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.937+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00011ad0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.937+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c06e2b0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.937+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c15c530 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.938+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c04c3c0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.939+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc0001b3b0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.940+0000 7fcbf37fe700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcbf8066f30 0x7fcbf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.940+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fcc000837b0 con 0x7fcc0c0b0410 2023-10-07T09:42:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.941+0000 7fcc0b7fe700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcbf8066f30 0x7fcbf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.941+0000 7fcc0b7fe700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcbf8066f30 0x7fcbf80693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc006fd0 tx=0x7fcbfc008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:07.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:07.944+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc00050c10 con 0x7fcc0c0b0410 2023-10-07T09:42:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.150+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc0c06e750 con 0x7fcc0c0b0410 2023-10-07T09:42:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.151+0000 7fcbf37fe700 1 -- 172.21.15.29:0/472839232 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+12757 (secure 0 0 0) 0x7fcc00017020 con 0x7fcc0c0b0410 2023-10-07T09:42:08.152 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:08.153 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:42:07.187729+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:42:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcbf8066f30 msgr2=0x7fcbf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcbf8066f30 0x7fcbf80693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc006fd0 tx=0x7fcbfc008040 comp rx=0 tx=0).stop 2023-10-07T09:42:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 msgr2=0x7fcc0c14ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c14ada0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c590 tx=0x7fcc0000c8a0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 shutdown_connections 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcbf8066f30 0x7fcbf80693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc0c159ec0 0x7fcc0c15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc0c0ad370 0x7fcc0c06eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 --2- 172.21.15.29:0/472839232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0b0410 0x7fcc0c14ada0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 >> 172.21.15.29:0/472839232 conn(0x7fcc0c0680a0 msgr2=0x7fcc0c0aadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 shutdown_connections 2023-10-07T09:42:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.154+0000 7fcc11dff700 1 -- 172.21.15.29:0/472839232 wait complete. 2023-10-07T09:42:08.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:42:08.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:42:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.371+0000 7f5fe3dff700 1 Processor -- start 2023-10-07T09:42:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.371+0000 7f5fe3dff700 1 -- start start 2023-10-07T09:42:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.372+0000 7f5fe3dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc15a960 0x7f5fdc158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.372+0000 7f5fe3dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0acb40 con 0x7f5fdc0ae8a0 2023-10-07T09:42:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe3dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0abf30 con 0x7f5fdc15ad30 2023-10-07T09:42:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe3dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc158c60 con 0x7f5fdc15a960 2023-10-07T09:42:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe239c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5fdc0ae8a0 0x7f5fdc0b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59050/0 (socket says 172.21.15.29:59050) 2023-10-07T09:42:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe239c700 1 -- 172.21.15.29:0/340697115 learned_addr learned my addr 172.21.15.29:0/340697115 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe1b9b700 1 --2- 172.21.15.29:0/340697115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc15a960 0x7f5fdc158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864522488 0 0) 0x7f5fdc0acb40 con 0x7f5fdc0ae8a0 2023-10-07T09:42:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.373+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc8003570 con 0x7f5fdc0ae8a0 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522434356 0 0) 0x7f5fdc0abf30 con 0x7f5fdc15ad30 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fdc0acb40 con 0x7f5fdc15ad30 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe1b9b700 1 -- 172.21.15.29:0/340697115 >> v1:172.21.15.144:6789/0 conn(0x7f5fdc15ad30 legacy=0x7f5fdc0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe1b9b700 1 -- 172.21.15.29:0/340697115 >> v1:172.21.15.29:6789/0 conn(0x7f5fdc0ae8a0 legacy=0x7f5fdc0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2076267855 0 0) 0x7f5fc8003570 con 0x7f5fdc0ae8a0 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe1b9b700 1 -- 172.21.15.29:0/340697115 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc0ac210 con 0x7f5fdc15a960 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe1b9b700 1 --2- 172.21.15.29:0/340697115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc15a960 0x7f5fdc158e20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009910 tx=0x7f5fd80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d10053a99501a1d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40017f0 con 0x7f5fdc15a960 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd8011420 con 0x7f5fdc15a960 2023-10-07T09:42:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.374+0000 7f5fe0b99700 1 -- 172.21.15.29:0/340697115 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80117c0 con 0x7f5fdc15a960 2023-10-07T09:42:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 -- 172.21.15.29:0/340697115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc15a960 msgr2=0x7f5fdc158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/340697115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc15a960 0x7f5fdc158e20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009910 tx=0x7f5fd80091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 -- 172.21.15.29:0/340697115 shutdown_connections 2023-10-07T09:42:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/340697115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc15a960 0x7f5fdc158e20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 -- 172.21.15.29:0/340697115 >> 172.21.15.29:0/340697115 conn(0x7f5fdc067610 msgr2=0x7f5fdc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 -- 172.21.15.29:0/340697115 shutdown_connections 2023-10-07T09:42:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.375+0000 7f5fe3dff700 1 -- 172.21.15.29:0/340697115 wait complete. 2023-10-07T09:42:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.376+0000 7f5fe3dff700 1 Processor -- start 2023-10-07T09:42:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 -- start start 2023-10-07T09:42:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc0ae8a0 0x7f5fdc1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fdc15a960 0x7f5fdc15dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 0x7f5fdc0afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc15cfb0 con 0x7f5fdc15ad30 2023-10-07T09:42:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc156f30 con 0x7f5fdc15a960 2023-10-07T09:42:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.377+0000 7f5fe3dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc15d830 con 0x7f5fdc0ae8a0 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 0x7f5fdc0afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 0x7f5fdc0afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39134/0 (socket says 172.21.15.29:39134) 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 -- 172.21.15.29:0/1802345758 learned_addr learned my addr 172.21.15.29:0/1802345758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe1b9b700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc0ae8a0 0x7f5fdc1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe139a700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fdc15a960 0x7f5fdc15dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 -- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc0ae8a0 msgr2=0x7f5fdc1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc0ae8a0 0x7f5fdc1547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 -- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fdc15a960 msgr2=0x7f5fdc15dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fdc15a960 0x7f5fdc15dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.378+0000 7f5fe239c700 1 -- 172.21.15.29:0/1802345758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8009420 con 0x7f5fdc15ad30 2023-10-07T09:42:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.379+0000 7f5fe239c700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 0x7f5fdc0afd90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400cbe0 tx=0x7f5fd4007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.379+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4004dc0 con 0x7f5fdc15ad30 2023-10-07T09:42:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.379+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd400a600 con 0x7f5fdc15ad30 2023-10-07T09:42:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.379+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400a9c0 con 0x7f5fdc15ad30 2023-10-07T09:42:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.379+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc156c50 con 0x7f5fdc15ad30 2023-10-07T09:42:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.379+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc15d4c0 con 0x7f5fdc15ad30 2023-10-07T09:42:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.380+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc04c3c0 con 0x7f5fdc15ad30 2023-10-07T09:42:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.381+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5fd4007150 con 0x7f5fdc15ad30 2023-10-07T09:42:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.382+0000 7f5fd1ffb700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5fc0066f30 0x7f5fc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.382+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f5fd4083de0 con 0x7f5fdc15ad30 2023-10-07T09:42:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.383+0000 7f5fe1b9b700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5fc0066f30 0x7f5fc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.384+0000 7f5fe1b9b700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5fc0066f30 0x7f5fc00693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009860 tx=0x7f5fd8016000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.386+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fd4051220 con 0x7f5fdc15ad30 2023-10-07T09:42:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.591+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fdc15d970 con 0x7f5fdc15ad30 2023-10-07T09:42:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.591+0000 7f5fd1ffb700 1 -- 172.21.15.29:0/1802345758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fd4018020 con 0x7f5fdc15ad30 2023-10-07T09:42:08.593 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:42:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5fc0066f30 msgr2=0x7f5fc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5fc0066f30 0x7f5fc00693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009860 tx=0x7f5fd8016000 comp rx=0 tx=0).stop 2023-10-07T09:42:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 msgr2=0x7f5fdc0afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 0x7f5fdc0afd90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400cbe0 tx=0x7f5fd4007cc0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 shutdown_connections 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5fc0066f30 0x7f5fc00693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fdc15ad30 0x7f5fdc0afd90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5fdc0ae8a0 0x7f5fdc1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 --2- 172.21.15.29:0/1802345758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fdc15a960 0x7f5fdc15dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 >> 172.21.15.29:0/1802345758 conn(0x7f5fdc067610 msgr2=0x7f5fdc159690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 shutdown_connections 2023-10-07T09:42:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.594+0000 7f5fe3dff700 1 -- 172.21.15.29:0/1802345758 wait complete. 2023-10-07T09:42:08.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:42:08.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.806+0000 7fd3211ff700 1 Processor -- start 2023-10-07T09:42:08.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.806+0000 7fd3211ff700 1 -- start start 2023-10-07T09:42:08.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.807+0000 7fd3211ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c159ec0 0x7fd31c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.807+0000 7fd3211ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd31c0af400 0x7fd31c0af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.807+0000 7fd3211ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31c14d790 con 0x7fd31c0ad370 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.807+0000 7fd3211ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c0ae5d0 con 0x7fd31c0af400 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.807+0000 7fd3211ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c15a850 con 0x7fd31c159ec0 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd31c0af400 0x7fd31c0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd31c0af400 0x7fd31c0af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39138/0 (socket says 172.21.15.29:39138) 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 -- 172.21.15.29:0/2821038764 learned_addr learned my addr 172.21.15.29:0/2821038764 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31a59c700 1 --2- 172.21.15.29:0/2821038764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c159ec0 0x7fd31c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 -- 172.21.15.29:0/2821038764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c159ec0 msgr2=0x7fd31c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 --2- 172.21.15.29:0/2821038764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c159ec0 0x7fd31c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 -- 172.21.15.29:0/2821038764 >> v1:172.21.15.144:6789/0 conn(0x7fd31c0ad370 legacy=0x7fd31c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 -- 172.21.15.29:0/2821038764 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c0ad740 con 0x7fd31c0af400 2023-10-07T09:42:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.808+0000 7fd31b59e700 1 --2- 172.21.15.29:0/2821038764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0af400 0x7fd31c0af810 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd314009f20 tx=0x7fd314009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6375b91a7dc1bf6d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd319d9b700 1 -- 172.21.15.29:0/2821038764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3140092e0 con 0x7fd31c0af400 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd319d9b700 1 -- 172.21.15.29:0/2821038764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31400b9d0 con 0x7fd31c0af400 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd319d9b700 1 -- 172.21.15.29:0/2821038764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31400bd90 con 0x7fd31c0af400 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 -- 172.21.15.29:0/2821038764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0af400 msgr2=0x7fd31c0af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 --2- 172.21.15.29:0/2821038764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0af400 0x7fd31c0af810 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd314009f20 tx=0x7fd314009840 comp rx=0 tx=0).stop 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 -- 172.21.15.29:0/2821038764 shutdown_connections 2023-10-07T09:42:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 --2- 172.21.15.29:0/2821038764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0af400 0x7fd31c0af810 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 --2- 172.21.15.29:0/2821038764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c159ec0 0x7fd31c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 -- 172.21.15.29:0/2821038764 >> 172.21.15.29:0/2821038764 conn(0x7fd31c0680a0 msgr2=0x7fd31c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.809+0000 7fd3211ff700 1 -- 172.21.15.29:0/2821038764 shutdown_connections 2023-10-07T09:42:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.810+0000 7fd3211ff700 1 -- 172.21.15.29:0/2821038764 wait complete. 2023-10-07T09:42:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.811+0000 7fd3211ff700 1 Processor -- start 2023-10-07T09:42:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.811+0000 7fd3211ff700 1 -- start start 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd3211ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 0x7fd31c14bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd3211ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c0af400 0x7fd31c0c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd3211ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd31c159ec0 0x7fd31c16c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd3211ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0c94c0 con 0x7fd31c0ad370 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd3211ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0c9790 con 0x7fd31c159ec0 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd3211ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0c9a70 con 0x7fd31c0af400 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd31ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 0x7fd31c14bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd31ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 0x7fd31c14bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39142/0 (socket says 172.21.15.29:39142) 2023-10-07T09:42:08.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.812+0000 7fd31ad9d700 1 -- 172.21.15.29:0/990156439 learned_addr learned my addr 172.21.15.29:0/990156439 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31a59c700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c0af400 0x7fd31c0c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31b59e700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd31c159ec0 0x7fd31c16c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31ad9d700 1 -- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c0af400 msgr2=0x7fd31c0c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31ad9d700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c0af400 0x7fd31c0c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31ad9d700 1 -- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd31c159ec0 msgr2=0x7fd31c16c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31ad9d700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd31c159ec0 0x7fd31c16c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31ad9d700 1 -- 172.21.15.29:0/990156439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd314009ae0 con 0x7fd31c0ad370 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd31ad9d700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 0x7fd31c14bae0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00c590 tx=0x7fd30c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011590 con 0x7fd31c0ad370 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c0116f0 con 0x7fd31c0ad370 2023-10-07T09:42:08.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011ab0 con 0x7fd31c0ad370 2023-10-07T09:42:08.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.813+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c0c9d50 con 0x7fd31c0ad370 2023-10-07T09:42:08.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.814+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c0ca310 con 0x7fd31c0ad370 2023-10-07T09:42:08.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.815+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c04c3c0 con 0x7fd31c0ad370 2023-10-07T09:42:08.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.816+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd30c01b3b0 con 0x7fd31c0ad370 2023-10-07T09:42:08.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.817+0000 7fd302ffd700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd308066f60 0x7fd308069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:08.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.817+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fd30c083990 con 0x7fd31c0ad370 2023-10-07T09:42:08.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.817+0000 7fd31a59c700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd308066f60 0x7fd308069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:08.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.818+0000 7fd31a59c700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd308066f60 0x7fd308069410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd31c149720 tx=0x7fd310008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:08.820+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd30c050d50 con 0x7fd31c0ad370 2023-10-07T09:42:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.025+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd31c0ccfe0 con 0x7fd31c0ad370 2023-10-07T09:42:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.025+0000 7fd302ffd700 1 -- 172.21.15.29:0/990156439 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd30c017020 con 0x7fd31c0ad370 2023-10-07T09:42:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:09.027 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:42:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:42:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:42:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:09.027 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd308066f60 msgr2=0x7fd308069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd308066f60 0x7fd308069410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd31c149720 tx=0x7fd310008040 comp rx=0 tx=0).stop 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 msgr2=0x7fd31c14bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 0x7fd31c14bae0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00c590 tx=0x7fd30c00c950 comp rx=0 tx=0).stop 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 shutdown_connections 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd308066f60 0x7fd308069410 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd31c159ec0 0x7fd31c16c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd31c0ad370 0x7fd31c14bae0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 --2- 172.21.15.29:0/990156439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31c0af400 0x7fd31c0c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 >> 172.21.15.29:0/990156439 conn(0x7fd31c0680a0 msgr2=0x7fd31c157fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 shutdown_connections 2023-10-07T09:42:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.028+0000 7fd3211ff700 1 -- 172.21.15.29:0/990156439 wait complete. 2023-10-07T09:42:09.046 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:42:09.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:42:09.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.246+0000 7fc892bff700 1 Processor -- start 2023-10-07T09:42:09.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.246+0000 7fc892bff700 1 -- start start 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.247+0000 7fc892bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc88c0ced00 0x7fc88c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.247+0000 7fc892bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc88c0c8a20 0x7fc88c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.247+0000 7fc892bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c0690f0 con 0x7fc88c0b0610 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.247+0000 7fc892bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c0cf2e0 con 0x7fc88c0ced00 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.247+0000 7fc892bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c0687c0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.247+0000 7fc88bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc88c0ced00 0x7fc88c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc88c0ced00 0x7fc88c0b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39144/0 (socket says 172.21.15.29:39144) 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 -- 172.21.15.29:0/339279063 learned_addr learned my addr 172.21.15.29:0/339279063 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc89119c700 1 --2- 172.21.15.29:0/339279063 >> v2:172.21.15.144:3300/0 conn(0x7fc88c0c8a20 0x7fc88c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 -- 172.21.15.29:0/339279063 >> v2:172.21.15.144:3300/0 conn(0x7fc88c0c8a20 msgr2=0x7fc88c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:09.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 --2- 172.21.15.29:0/339279063 >> v2:172.21.15.144:3300/0 conn(0x7fc88c0c8a20 0x7fc88c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 -- 172.21.15.29:0/339279063 >> v1:172.21.15.144:6789/0 conn(0x7fc88c0b0610 legacy=0x7fc88c0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:09.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 -- 172.21.15.29:0/339279063 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c068aa0 con 0x7fc88c0ced00 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88bfff700 1 --2- 172.21.15.29:0/339279063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0ced00 0x7fc88c0b09e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc88c070ea0 tx=0x7fc878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7b8df2615b8302 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.248+0000 7fc88b7fe700 1 -- 172.21.15.29:0/339279063 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87800b860 con 0x7fc88c0ced00 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc88b7fe700 1 -- 172.21.15.29:0/339279063 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87800b9c0 con 0x7fc88c0ced00 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc88b7fe700 1 -- 172.21.15.29:0/339279063 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87800bd60 con 0x7fc88c0ced00 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 -- 172.21.15.29:0/339279063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0ced00 msgr2=0x7fc88c0b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 --2- 172.21.15.29:0/339279063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0ced00 0x7fc88c0b09e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc88c070ea0 tx=0x7fc878009250 comp rx=0 tx=0).stop 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 -- 172.21.15.29:0/339279063 shutdown_connections 2023-10-07T09:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 --2- 172.21.15.29:0/339279063 >> v2:172.21.15.144:3300/0 conn(0x7fc88c0c8a20 0x7fc88c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 --2- 172.21.15.29:0/339279063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0ced00 0x7fc88c0b09e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 -- 172.21.15.29:0/339279063 >> 172.21.15.29:0/339279063 conn(0x7fc88c063a10 msgr2=0x7fc88c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 -- 172.21.15.29:0/339279063 shutdown_connections 2023-10-07T09:42:09.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.249+0000 7fc892bff700 1 -- 172.21.15.29:0/339279063 wait complete. 2023-10-07T09:42:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.251+0000 7fc892bff700 1 Processor -- start 2023-10-07T09:42:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.251+0000 7fc892bff700 1 -- start start 2023-10-07T09:42:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc892bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc88c0b0610 0x7fc88c0b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc892bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 0x7fc88c169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc892bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc88c0ced00 0x7fc88c1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc892bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c69f0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc892bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c5de0 con 0x7fc88c0b0610 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc892bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c60c0 con 0x7fc88c0ced00 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 0x7fc88c169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 0x7fc88c169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39158/0 (socket says 172.21.15.29:39158) 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 -- 172.21.15.29:0/3379134422 learned_addr learned my addr 172.21.15.29:0/3379134422 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc89099b700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc88c0b0610 0x7fc88c0b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc89119c700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc88c0ced00 0x7fc88c1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 -- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc88c0ced00 msgr2=0x7fc88c1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc88c0ced00 0x7fc88c1580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 -- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc88c0b0610 msgr2=0x7fc88c0b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc88c0b0610 0x7fc88c0b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.252+0000 7fc88bfff700 1 -- 172.21.15.29:0/3379134422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8780094f0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.253+0000 7fc88bfff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 0x7fc88c169960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06ea20 tx=0x7fc878017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.253+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878011e40 con 0x7fc88c0c8a20 2023-10-07T09:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.253+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87801a420 con 0x7fc88c0c8a20 2023-10-07T09:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.253+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87801a7c0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.253+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c0c63a0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.253+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c0b40e0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.254+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c0c6840 con 0x7fc88c0c8a20 2023-10-07T09:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.255+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc87801abb0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.256+0000 7fc888ff9700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc870066f30 0x7fc8700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.256+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fc878085da0 con 0x7fc88c0c8a20 2023-10-07T09:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.256+0000 7fc89099b700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc870066f30 0x7fc8700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:09.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.257+0000 7fc89099b700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc870066f30 0x7fc8700693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8800039e0 tx=0x7fc880007ec0 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:09.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.260+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc878053160 con 0x7fc88c0c8a20 2023-10-07T09:42:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:09.467+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc88c0c95d0 con 0x7fc88c0c8a20 2023-10-07T09:42:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.211+0000 7fc888ff9700 1 -- 172.21.15.29:0/3379134422 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v58) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc87801ae60 con 0x7fc88c0c8a20 2023-10-07T09:42:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.216+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc870066f30 msgr2=0x7fc8700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.216+0000 7fc892bff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc870066f30 0x7fc8700693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8800039e0 tx=0x7fc880007ec0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.216+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 msgr2=0x7fc88c169960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.216+0000 7fc892bff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 0x7fc88c169960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06ea20 tx=0x7fc878017000 comp rx=0 tx=0).stop 2023-10-07T09:42:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 shutdown_connections 2023-10-07T09:42:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc870066f30 0x7fc8700693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc88c0ced00 0x7fc88c1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc88c0b0610 0x7fc88c0b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 --2- 172.21.15.29:0/3379134422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c8a20 0x7fc88c169960 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 >> 172.21.15.29:0/3379134422 conn(0x7fc88c063a10 msgr2=0x7fc88c0c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 shutdown_connections 2023-10-07T09:42:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.217+0000 7fc892bff700 1 -- 172.21.15.29:0/3379134422 wait complete. 2023-10-07T09:42:10.221 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:42:10.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:42:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.443+0000 7f9ccdfff700 1 Processor -- start 2023-10-07T09:42:10.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.443+0000 7f9ccdfff700 1 -- start start 2023-10-07T09:42:10.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.444+0000 7f9ccdfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cc80d0010 0x7f9cc80b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:10.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.444+0000 7f9ccdfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc80c62d0 con 0x7f9cc80cb800 2023-10-07T09:42:10.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.444+0000 7f9ccdfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc8159e10 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.444+0000 7f9ccdfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc80d0410 con 0x7f9cc80d0010 2023-10-07T09:42:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cc80d0010 0x7f9cc80b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cc80d0010 0x7f9cc80b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39168/0 (socket says 172.21.15.29:39168) 2023-10-07T09:42:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 -- 172.21.15.29:0/4098676874 learned_addr learned my addr 172.21.15.29:0/4098676874 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 -- 172.21.15.29:0/4098676874 >> v1:172.21.15.144:6789/0 conn(0x7f9cc80cfbb0 legacy=0x7f9cc80c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 -- 172.21.15.29:0/4098676874 >> v1:172.21.15.29:6789/0 conn(0x7f9cc80cb800 legacy=0x7f9cc80cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 -- 172.21.15.29:0/4098676874 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc80c56c0 con 0x7f9cc80d0010 2023-10-07T09:42:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc67fc700 1 -- 172.21.15.29:0/4098676874 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672749809 0 0) 0x7f9cc80c62d0 con 0x7f9cc80cb800 2023-10-07T09:42:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.445+0000 7f9cc7fff700 1 --2- 172.21.15.29:0/4098676874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80d0010 0x7f9cc80b3340 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009f20 tx=0x7f9cc0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cda8e894e28838 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9cc67fc700 1 -- 172.21.15.29:0/4098676874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc000e070 con 0x7f9cc80d0010 2023-10-07T09:42:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9cc67fc700 1 -- 172.21.15.29:0/4098676874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc000bad0 con 0x7f9cc80d0010 2023-10-07T09:42:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9cc67fc700 1 -- 172.21.15.29:0/4098676874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc00092e0 con 0x7f9cc80d0010 2023-10-07T09:42:10.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4098676874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80d0010 msgr2=0x7f9cc80b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:10.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4098676874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80d0010 0x7f9cc80b3340 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009f20 tx=0x7f9cc0009840 comp rx=0 tx=0).stop 2023-10-07T09:42:10.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4098676874 shutdown_connections 2023-10-07T09:42:10.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4098676874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80d0010 0x7f9cc80b3340 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4098676874 >> 172.21.15.29:0/4098676874 conn(0x7f9cc8063a10 msgr2=0x7f9cc80d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4098676874 shutdown_connections 2023-10-07T09:42:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.446+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4098676874 wait complete. 2023-10-07T09:42:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.448+0000 7f9ccdfff700 1 Processor -- start 2023-10-07T09:42:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.448+0000 7f9ccdfff700 1 -- start start 2023-10-07T09:42:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9ccdfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cc80cb800 0x7f9cc8149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:10.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9ccdfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 0x7f9cc809ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:10.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9ccdfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cc80d0010 0x7f9cc80c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:10.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9ccdfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c83e0 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9ccdfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c77d0 con 0x7f9cc80d0010 2023-10-07T09:42:10.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9ccdfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c7ab0 con 0x7f9cc80cb800 2023-10-07T09:42:10.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9cc6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 0x7f9cc809ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:10.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9cc6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 0x7f9cc809ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39172/0 (socket says 172.21.15.29:39172) 2023-10-07T09:42:10.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.449+0000 7f9cc6ffd700 1 -- 172.21.15.29:0/4174681780 learned_addr learned my addr 172.21.15.29:0/4174681780 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:10.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc7fff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cc80d0010 0x7f9cc80c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:10.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc77fe700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cc80cb800 0x7f9cc8149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:10.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc6ffd700 1 -- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cc80cb800 msgr2=0x7f9cc8149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:10.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc6ffd700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cc80cb800 0x7f9cc8149140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc6ffd700 1 -- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cc80d0010 msgr2=0x7f9cc80c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:10.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc6ffd700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cc80d0010 0x7f9cc80c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:10.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc6ffd700 1 -- 172.21.15.29:0/4174681780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc0009ae0 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc7fff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cc80d0010 0x7f9cc80c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:10.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc77fe700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cc80cb800 0x7f9cc8149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:10.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9cc6ffd700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 0x7f9cc809ab90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c5d0 tx=0x7f9cbc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:10.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc011590 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc011710 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc011ad0 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc80c7d90 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.450+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc809b0d0 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.451+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc80718b0 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.452+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9cbc01c460 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.453+0000 7f9caf7fe700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9cb4066f30 0x7f9cb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:10.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.453+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5571+0+0 (secure 0 0 0) 0x7f9cbc084350 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.454+0000 7f9cc77fe700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9cb4066f30 0x7f9cb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:10.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.454+0000 7f9cc77fe700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9cb4066f30 0x7f9cb40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80018b0 tx=0x7f9cb8009040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:10.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.458+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cbc0515a0 con 0x7f9cc80cfbb0 2023-10-07T09:42:10.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:10.661+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9cc80c8520 con 0x7f9cc80cfbb0 2023-10-07T09:42:11.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.217+0000 7f9caf7fe700 1 -- 172.21.15.29:0/4174681780 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v59) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9cbc054bc0 con 0x7f9cc80cfbb0 2023-10-07T09:42:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.225+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9cb4066f30 msgr2=0x7f9cb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.225+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9cb4066f30 0x7f9cb40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80018b0 tx=0x7f9cb8009040 comp rx=0 tx=0).stop 2023-10-07T09:42:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.225+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 msgr2=0x7f9cc809ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.225+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 0x7f9cc809ab90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c5d0 tx=0x7f9cbc007f60 comp rx=0 tx=0).stop 2023-10-07T09:42:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.227+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 shutdown_connections 2023-10-07T09:42:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.227+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9cb4066f30 0x7f9cb40693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.227+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cc80d0010 0x7f9cc80c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.227+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cc80cb800 0x7f9cc8149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.227+0000 7f9ccdfff700 1 --2- 172.21.15.29:0/4174681780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80cfbb0 0x7f9cc809ab90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.227+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 >> 172.21.15.29:0/4174681780 conn(0x7f9cc8063a10 msgr2=0x7f9cc8068440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.231+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 shutdown_connections 2023-10-07T09:42:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.232+0000 7f9ccdfff700 1 -- 172.21.15.29:0/4174681780 wait complete. 2023-10-07T09:42:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:42:11.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:42:11.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.459+0000 7fc7e0dff700 1 Processor -- start 2023-10-07T09:42:11.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.459+0000 7fc7e0dff700 1 -- start start 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.460+0000 7fc7e0dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7dc0b0c30 0x7fc7dc0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.460+0000 7fc7e0dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0ad4d0 con 0x7fc7dc0ae570 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.460+0000 7fc7e0dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0ad7b0 con 0x7fc7dc156180 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.460+0000 7fc7e0dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc0ada90 con 0x7fc7dc0b0c30 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7dc0b0c30 0x7fc7dc0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7dc0b0c30 0x7fc7dc0afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39184/0 (socket says 172.21.15.29:39184) 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 -- 172.21.15.29:0/1093658822 learned_addr learned my addr 172.21.15.29:0/1093658822 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:11.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7d9d9b700 1 --1- 172.21.15.29:0/1093658822 >> v1:172.21.15.29:6789/0 conn(0x7fc7dc0ae570 0x7fc7dc0b0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59060/0 (socket says 172.21.15.29:59060) 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 -- 172.21.15.29:0/1093658822 >> v1:172.21.15.144:6789/0 conn(0x7fc7dc156180 legacy=0x7fc7dc0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 -- 172.21.15.29:0/1093658822 >> v1:172.21.15.29:6789/0 conn(0x7fc7dc0ae570 legacy=0x7fc7dc0b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 -- 172.21.15.29:0/1093658822 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0add70 con 0x7fc7dc0b0c30 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.461+0000 7fc7dad9d700 1 --2- 172.21.15.29:0/1093658822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc0b0c30 0x7fc7dc0afc00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009f20 tx=0x7fc7d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c29b8145d430826 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7d959a700 1 -- 172.21.15.29:0/1093658822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400e070 con 0x7fc7dc0b0c30 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7d959a700 1 -- 172.21.15.29:0/1093658822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d400ba80 con 0x7fc7dc0b0c30 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7d959a700 1 -- 172.21.15.29:0/1093658822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40092e0 con 0x7fc7dc0b0c30 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7e0dff700 1 -- 172.21.15.29:0/1093658822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc0b0c30 msgr2=0x7fc7dc0afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/1093658822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc0b0c30 0x7fc7dc0afc00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009f20 tx=0x7fc7d4009840 comp rx=0 tx=0).stop 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7e0dff700 1 -- 172.21.15.29:0/1093658822 shutdown_connections 2023-10-07T09:42:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/1093658822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc0b0c30 0x7fc7dc0afc00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7e0dff700 1 -- 172.21.15.29:0/1093658822 >> 172.21.15.29:0/1093658822 conn(0x7fc7dc067610 msgr2=0x7fc7dc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.462+0000 7fc7e0dff700 1 -- 172.21.15.29:0/1093658822 shutdown_connections 2023-10-07T09:42:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.463+0000 7fc7e0dff700 1 -- 172.21.15.29:0/1093658822 wait complete. 2023-10-07T09:42:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.464+0000 7fc7e0dff700 1 Processor -- start 2023-10-07T09:42:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.464+0000 7fc7e0dff700 1 -- start start 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7e0dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7dc0ae570 0x7fc7dc154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7e0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7dc0b0c30 0x7fc7dc15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7e0dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 0x7fc7dc15df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7e0dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc15ede0 con 0x7fc7dc156180 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7e0dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc0cb700 con 0x7fc7dc0ae570 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7e0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc0cb9e0 con 0x7fc7dc0b0c30 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7dad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 0x7fc7dc15df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7dad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 0x7fc7dc15df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39194/0 (socket says 172.21.15.29:39194) 2023-10-07T09:42:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7dad9d700 1 -- 172.21.15.29:0/2126601490 learned_addr learned my addr 172.21.15.29:0/2126601490 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7d9d9b700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7dc0b0c30 0x7fc7dc15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.465+0000 7fc7da59c700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7dc0ae570 0x7fc7dc154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7dad9d700 1 -- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7dc0b0c30 msgr2=0x7fc7dc15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7dad9d700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7dc0b0c30 0x7fc7dc15d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7dad9d700 1 -- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7dc0ae570 msgr2=0x7fc7dc154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7dad9d700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7dc0ae570 0x7fc7dc154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7dad9d700 1 -- 172.21.15.29:0/2126601490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4009ae0 con 0x7fc7dc156180 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7d9d9b700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7dc0b0c30 0x7fc7dc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:42:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7dad9d700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 0x7fc7dc15df90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4012000 tx=0x7fc7d4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:11.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4008dd0 con 0x7fc7dc156180 2023-10-07T09:42:11.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d4026460 con 0x7fc7dc156180 2023-10-07T09:42:11.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4026820 con 0x7fc7dc156180 2023-10-07T09:42:11.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.466+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc0cbcc0 con 0x7fc7dc156180 2023-10-07T09:42:11.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.467+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc15aed0 con 0x7fc7dc156180 2023-10-07T09:42:11.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.468+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc15b0b0 con 0x7fc7dc156180 2023-10-07T09:42:11.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.469+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc7d4003040 con 0x7fc7dc156180 2023-10-07T09:42:11.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.470+0000 7fc7ca7fc700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7bc066f30 0x7fc7bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:11.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.470+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5953+0+0 (secure 0 0 0) 0x7fc7d4086260 con 0x7fc7dc156180 2023-10-07T09:42:11.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.470+0000 7fc7da59c700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7bc066f30 0x7fc7bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:11.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.471+0000 7fc7da59c700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7bc066f30 0x7fc7bc0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc003a10 tx=0x7fc7cc007ec0 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:11.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.475+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7d40533e0 con 0x7fc7dc156180 2023-10-07T09:42:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:11.720+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc7dc04c3c0 con 0x7fc7dc156180 2023-10-07T09:42:12.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.235+0000 7fc7ca7fc700 1 -- 172.21.15.29:0/2126601490 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc7d4008f30 con 0x7fc7dc156180 2023-10-07T09:42:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.241+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7bc066f30 msgr2=0x7fc7bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.241+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7bc066f30 0x7fc7bc0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc003a10 tx=0x7fc7cc007ec0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.241+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 msgr2=0x7fc7dc15df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.241+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 0x7fc7dc15df90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4012000 tx=0x7fc7d4018000 comp rx=0 tx=0).stop 2023-10-07T09:42:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.242+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 shutdown_connections 2023-10-07T09:42:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.242+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7bc066f30 0x7fc7bc0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.242+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7dc156180 0x7fc7dc15df90 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.242+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7dc0ae570 0x7fc7dc154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.242+0000 7fc7e0dff700 1 --2- 172.21.15.29:0/2126601490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7dc0b0c30 0x7fc7dc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.242+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 >> 172.21.15.29:0/2126601490 conn(0x7fc7dc067610 msgr2=0x7fc7dc0aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.243+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 shutdown_connections 2023-10-07T09:42:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.243+0000 7fc7e0dff700 1 -- 172.21.15.29:0/2126601490 wait complete. 2023-10-07T09:42:12.247 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 6 and data pool 7 2023-10-07T09:42:12.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:42:12.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2023-10-07T09:42:12.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.463+0000 7f7fb2bff700 1 Processor -- start 2023-10-07T09:42:12.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.463+0000 7f7fb2bff700 1 -- start start 2023-10-07T09:42:12.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.464+0000 7f7fb2bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fac153cc0 0x7f7fac0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.464+0000 7f7fb2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.464+0000 7f7fb2bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac154090 con 0x7f7fac0b0000 2023-10-07T09:42:12.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.464+0000 7f7fb2bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac159f60 con 0x7f7fac153cc0 2023-10-07T09:42:12.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.464+0000 7f7fb2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac15a240 con 0x7f7fac15ade0 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fac153cc0 0x7f7fac0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fac153cc0 0x7f7fac0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39208/0 (socket says 172.21.15.29:39208) 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 -- 172.21.15.29:0/4013478645 learned_addr learned my addr 172.21.15.29:0/4013478645 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fabfff700 1 --2- 172.21.15.29:0/4013478645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 -- 172.21.15.29:0/4013478645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 msgr2=0x7f7fac0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 --2- 172.21.15.29:0/4013478645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 -- 172.21.15.29:0/4013478645 >> v1:172.21.15.144:6789/0 conn(0x7f7fac0b0000 legacy=0x7f7fac15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 -- 172.21.15.29:0/4013478645 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac15a520 con 0x7f7fac153cc0 2023-10-07T09:42:12.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.465+0000 7f7fb099b700 1 --2- 172.21.15.29:0/4013478645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac153cc0 0x7f7fac0b17c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009990 tx=0x7f7fa0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b3c8e7b1c92e1d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fab7fe700 1 -- 172.21.15.29:0/4013478645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0005c10 con 0x7f7fac153cc0 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fab7fe700 1 -- 172.21.15.29:0/4013478645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa0005d70 con 0x7f7fac153cc0 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fab7fe700 1 -- 172.21.15.29:0/4013478645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa000b880 con 0x7f7fac153cc0 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 -- 172.21.15.29:0/4013478645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac153cc0 msgr2=0x7f7fac0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/4013478645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac153cc0 0x7f7fac0b17c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009990 tx=0x7f7fa0009220 comp rx=0 tx=0).stop 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 -- 172.21.15.29:0/4013478645 shutdown_connections 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/4013478645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac153cc0 0x7f7fac0b17c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/4013478645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 -- 172.21.15.29:0/4013478645 >> 172.21.15.29:0/4013478645 conn(0x7f7fac067610 msgr2=0x7f7fac067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:12.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 -- 172.21.15.29:0/4013478645 shutdown_connections 2023-10-07T09:42:12.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.466+0000 7f7fb2bff700 1 -- 172.21.15.29:0/4013478645 wait complete. 2023-10-07T09:42:12.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.468+0000 7f7fb2bff700 1 Processor -- start 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.468+0000 7f7fb2bff700 1 -- start start 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb2bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 0x7f7fac145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb2bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fac153cc0 0x7f7fac155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb2bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac156b60 con 0x7f7fac0b0000 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb2bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1565a0 con 0x7f7fac153cc0 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac156880 con 0x7f7fac15ade0 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 0x7f7fac145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 0x7f7fac145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39212/0 (socket says 172.21.15.29:39212) 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb099b700 1 -- 172.21.15.29:0/738799985 learned_addr learned my addr 172.21.15.29:0/738799985 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fabfff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fac153cc0 0x7f7fac155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.469+0000 7f7fb119c700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb099b700 1 -- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 msgr2=0x7f7fac0b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb099b700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb099b700 1 -- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fac153cc0 msgr2=0x7f7fac155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb099b700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fac153cc0 0x7f7fac155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb099b700 1 -- 172.21.15.29:0/738799985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa0009480 con 0x7f7fac0b0000 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb099b700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 0x7f7fac145da0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0005d40 tx=0x7f7fa0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0006b60 con 0x7f7fac0b0000 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa00118c0 con 0x7f7fac0b0000 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac156ed0 con 0x7f7fac0b0000 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0011c40 con 0x7f7fac0b0000 2023-10-07T09:42:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.470+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac1462e0 con 0x7f7fac0b0000 2023-10-07T09:42:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.471+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac04c3c0 con 0x7f7fac0b0000 2023-10-07T09:42:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.472+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7fa001b640 con 0x7f7fac0b0000 2023-10-07T09:42:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.473+0000 7f7fa8ff9700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7f90066f30 0x7f7f900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.473+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7fa0085fa0 con 0x7f7fac0b0000 2023-10-07T09:42:12.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.474+0000 7f7fabfff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7f90066f30 0x7f7f900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.474+0000 7f7fabfff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7f90066f30 0x7f7f900693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0c7a30 tx=0x7f7f98008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:12.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.478+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7fa0053010 con 0x7f7fac0b0000 2023-10-07T09:42:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.683+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fac15a800 con 0x7f7fac0b0000 2023-10-07T09:42:12.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.684+0000 7f7fa8ff9700 1 -- 172.21.15.29:0/738799985 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+15986 (secure 0 0 0) 0x7f7fa002a020 con 0x7f7fac0b0000 2023-10-07T09:42:12.686 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:12.686 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:42:11.721715+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:42:09.508279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-07T09:42:10.701497+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:42:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7f90066f30 msgr2=0x7f7f900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7f90066f30 0x7f7f900693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0c7a30 tx=0x7f7f98008040 comp rx=0 tx=0).stop 2023-10-07T09:42:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 msgr2=0x7f7fac145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 0x7f7fac145da0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0005d40 tx=0x7f7fa0016000 comp rx=0 tx=0).stop 2023-10-07T09:42:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 shutdown_connections 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7f90066f30 0x7f7f900693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fac15ade0 0x7f7fac0b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fac0b0000 0x7f7fac145da0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 --2- 172.21.15.29:0/738799985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fac153cc0 0x7f7fac155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 >> 172.21.15.29:0/738799985 conn(0x7f7fac067610 msgr2=0x7f7fac15ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 shutdown_connections 2023-10-07T09:42:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.687+0000 7f7fb2bff700 1 -- 172.21.15.29:0/738799985 wait complete. 2023-10-07T09:42:12.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:42:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.900+0000 7f3c839ff700 1 Processor -- start 2023-10-07T09:42:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.900+0000 7f3c839ff700 1 -- start start 2023-10-07T09:42:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c839ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c839ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c839ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c06f4e0 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c839ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c06e8d0 con 0x7f3c7c0b0f50 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c839ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c06ebb0 con 0x7f3c7c159550 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c80f9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c81f9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3c7c0b2bb0 0x7f3c7c15d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37044/0 (socket says 172.21.15.29:37044) 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c81f9c700 1 -- 172.21.15.29:0/2419857105 learned_addr learned my addr 172.21.15.29:0/2419857105 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c8179b700 1 --2- 172.21.15.29:0/2419857105 >> v2:172.21.15.144:3300/0 conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c80f9a700 1 -- 172.21.15.29:0/2419857105 >> v2:172.21.15.144:3300/0 conn(0x7f3c7c159550 msgr2=0x7f3c7c0b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c80f9a700 1 --2- 172.21.15.29:0/2419857105 >> v2:172.21.15.144:3300/0 conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c80f9a700 1 -- 172.21.15.29:0/2419857105 >> v1:172.21.15.29:6789/0 conn(0x7f3c7c0b2bb0 legacy=0x7f3c7c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c80f9a700 1 -- 172.21.15.29:0/2419857105 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c06ee90 con 0x7f3c7c0b0f50 2023-10-07T09:42:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.902+0000 7f3c73fff700 1 -- 172.21.15.29:0/2419857105 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237064315 0 0) 0x7f3c7c06f4e0 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.903+0000 7f3c80f9a700 1 --2- 172.21.15.29:0/2419857105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c076b60 tx=0x7f3c6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abff35014fffade server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.903+0000 7f3c73fff700 1 -- 172.21.15.29:0/2419857105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c011630 con 0x7f3c7c0b0f50 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.903+0000 7f3c73fff700 1 -- 172.21.15.29:0/2419857105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c011790 con 0x7f3c7c0b0f50 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.903+0000 7f3c73fff700 1 -- 172.21.15.29:0/2419857105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c011b30 con 0x7f3c7c0b0f50 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 -- 172.21.15.29:0/2419857105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 msgr2=0x7f3c7c0b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 --2- 172.21.15.29:0/2419857105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c076b60 tx=0x7f3c6c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 -- 172.21.15.29:0/2419857105 shutdown_connections 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 --2- 172.21.15.29:0/2419857105 >> v2:172.21.15.144:3300/0 conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 --2- 172.21.15.29:0/2419857105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 -- 172.21.15.29:0/2419857105 >> 172.21.15.29:0/2419857105 conn(0x7f3c7c069190 msgr2=0x7f3c7c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 -- 172.21.15.29:0/2419857105 shutdown_connections 2023-10-07T09:42:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.904+0000 7f3c839ff700 1 -- 172.21.15.29:0/2419857105 wait complete. 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.905+0000 7f3c839ff700 1 Processor -- start 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.905+0000 7f3c839ff700 1 -- start start 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.906+0000 7f3c839ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c13daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.906+0000 7f3c839ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c15d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.906+0000 7f3c839ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c13f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.906+0000 7f3c839ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c15eb00 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.906+0000 7f3c839ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c15ef90 con 0x7f3c7c0b0f50 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.906+0000 7f3c839ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c15def0 con 0x7f3c7c159550 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c15d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c15d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58954/0 (socket says 172.21.15.29:58954) 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 -- 172.21.15.29:0/4103821586 learned_addr learned my addr 172.21.15.29:0/4103821586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c8179b700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c13daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c81f9c700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c13f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 -- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 msgr2=0x7f3c7c13f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c13f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 -- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 msgr2=0x7f3c7c13daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c13daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 -- 172.21.15.29:0/4103821586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c009420 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.907+0000 7f3c80f9a700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c15d4a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c06c520 tx=0x7f3c6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.908+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c000d10 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.908+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c011900 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.908+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c15e1d0 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.908+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c025420 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.908+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c15e790 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.909+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c15ec70 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.910+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3c6c023070 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.911+0000 7f3c717fa700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c60066f30 0x7f3c600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.911+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3c6c085b40 con 0x7f3c7c0b2bb0 2023-10-07T09:42:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.912+0000 7f3c8179b700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c60066f30 0x7f3c600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.912+0000 7f3c8179b700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c60066f30 0x7f3c600693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009d10 tx=0x7f3c780091a0 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:12.915+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c6c052c50 con 0x7f3c7c0b2bb0 2023-10-07T09:42:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.117+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c7c13e030 con 0x7f3c7c0b2bb0 2023-10-07T09:42:13.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.117+0000 7f3c717fa700 1 -- 172.21.15.29:0/4103821586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+15973 (secure 0 0 0) 0x7f3c6c056270 con 0x7f3c7c0b2bb0 2023-10-07T09:42:13.119 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:13.120 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":61,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:42:13.105568+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:42:09.508279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-07T09:42:10.701497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:42:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c60066f30 msgr2=0x7f3c600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c60066f30 0x7f3c600693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009d10 tx=0x7f3c780091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 msgr2=0x7f3c7c15d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c15d4a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c06c520 tx=0x7f3c6c017000 comp rx=0 tx=0).stop 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 shutdown_connections 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c60066f30 0x7f3c600693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c13f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c13daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 --2- 172.21.15.29:0/4103821586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c15d4a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 >> 172.21.15.29:0/4103821586 conn(0x7f3c7c069190 msgr2=0x7f3c7c0aa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 shutdown_connections 2023-10-07T09:42:13.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.120+0000 7f3c839ff700 1 -- 172.21.15.29:0/4103821586 wait complete. 2023-10-07T09:42:13.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:42:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.340+0000 7f2eeffff700 1 Processor -- start 2023-10-07T09:42:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.340+0000 7f2eeffff700 1 -- start start 2023-10-07T09:42:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.341+0000 7f2eeffff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ee8156180 0x7f2ee80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.341+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.341+0000 7f2eeffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee80b0ec0 con 0x7f2ee80ae570 2023-10-07T09:42:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.341+0000 7f2eeffff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee80ad4d0 con 0x7f2ee8156180 2023-10-07T09:42:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.341+0000 7f2eeffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80ad7b0 con 0x7f2ee80b0520 2023-10-07T09:42:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eed59a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2ee80ae570 0x7f2ee80accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37058/0 (socket says 172.21.15.29:37058) 2023-10-07T09:42:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eed59a700 1 -- 172.21.15.29:0/1185732962 learned_addr learned my addr 172.21.15.29:0/1185732962 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eee59c700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eedd9b700 1 --2- 172.21.15.29:0/1185732962 >> v2:172.21.15.144:3300/0 conn(0x7f2ee8156180 0x7f2ee80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eecd99700 1 -- 172.21.15.29:0/1185732962 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938088378 0 0) 0x7f2ed8001110 con 0x7f2ee80ae570 2023-10-07T09:42:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eecd99700 1 -- 172.21.15.29:0/1185732962 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed4003570 con 0x7f2ee80ae570 2023-10-07T09:42:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eedd9b700 1 -- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 msgr2=0x7f2ee80b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eedd9b700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eedd9b700 1 -- 172.21.15.29:0/1185732962 >> v1:172.21.15.29:6789/0 conn(0x7f2ee80ae570 legacy=0x7f2ee80accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:13.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eedd9b700 1 -- 172.21.15.29:0/1185732962 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80ada90 con 0x7f2ee8156180 2023-10-07T09:42:13.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.342+0000 7f2eee59c700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:13.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.343+0000 7f2eedd9b700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80afc00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40098e0 tx=0x7f2ee40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2bb82ac9e14cfe3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.343+0000 7f2eecd99700 1 -- 172.21.15.29:0/1185732962 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4011630 con 0x7f2ee8156180 2023-10-07T09:42:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.343+0000 7f2eecd99700 1 -- 172.21.15.29:0/1185732962 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4011790 con 0x7f2ee8156180 2023-10-07T09:42:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.343+0000 7f2eecd99700 1 -- 172.21.15.29:0/1185732962 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4011b30 con 0x7f2ee8156180 2023-10-07T09:42:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.343+0000 7f2eeffff700 1 -- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 msgr2=0x7f2ee80afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.343+0000 7f2eeffff700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80afc00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40098e0 tx=0x7f2ee40091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.344+0000 7f2eeffff700 1 -- 172.21.15.29:0/1185732962 shutdown_connections 2023-10-07T09:42:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.344+0000 7f2eeffff700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.344+0000 7f2eeffff700 1 --2- 172.21.15.29:0/1185732962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80afc00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.344+0000 7f2eeffff700 1 -- 172.21.15.29:0/1185732962 >> 172.21.15.29:0/1185732962 conn(0x7f2ee8067610 msgr2=0x7f2ee8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.344+0000 7f2eeffff700 1 -- 172.21.15.29:0/1185732962 shutdown_connections 2023-10-07T09:42:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.344+0000 7f2eeffff700 1 -- 172.21.15.29:0/1185732962 wait complete. 2023-10-07T09:42:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.345+0000 7f2eeffff700 1 Processor -- start 2023-10-07T09:42:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.345+0000 7f2eeffff700 1 -- start start 2023-10-07T09:42:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 0x7f2ee8150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eeffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80c9150 con 0x7f2ee80ae570 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eeffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80c9430 con 0x7f2ee8156180 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eeffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80c9710 con 0x7f2ee80b0520 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.346+0000 7f2eedd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 0x7f2ee8150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 0x7f2ee8150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58966/0 (socket says 172.21.15.29:58966) 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 -- 172.21.15.29:0/549002386 learned_addr learned my addr 172.21.15.29:0/549002386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eee59c700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eed59a700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 -- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 msgr2=0x7f2ee80acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80acc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 -- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 msgr2=0x7f2ee80ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 -- 172.21.15.29:0/549002386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee0009a20 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eedd9b700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 0x7f2ee8150f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80641e0 tx=0x7f2ee4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.347+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4011b10 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.348+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee40253f0 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.348+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40257b0 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.348+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee4009420 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.348+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee80c9cd0 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.349+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee80b1e00 con 0x7f2ee80ae570 2023-10-07T09:42:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.350+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ee4025ba0 con 0x7f2ee80ae570 2023-10-07T09:42:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.350+0000 7f2eddffb700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2ecc066f30 0x7f2ecc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.351+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2ee4085e10 con 0x7f2ee80ae570 2023-10-07T09:42:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.351+0000 7f2eed59a700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2ecc066f30 0x7f2ecc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.352+0000 7f2eed59a700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2ecc066f30 0x7f2ecc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80cabd0 tx=0x7f2ed8007bf0 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.355+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ee4089030 con 0x7f2ee80ae570 2023-10-07T09:42:13.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:13.603+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2ee804c3c0 con 0x7f2ee80ae570 2023-10-07T09:42:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.247+0000 7f2eddffb700 1 -- 172.21.15.29:0/549002386 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2ee4052f40 con 0x7f2ee80ae570 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2ecc066f30 msgr2=0x7f2ecc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2ecc066f30 0x7f2ecc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80cabd0 tx=0x7f2ed8007bf0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 msgr2=0x7f2ee8150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 0x7f2ee8150f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80641e0 tx=0x7f2ee4017000 comp rx=0 tx=0).stop 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 shutdown_connections 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2ecc066f30 0x7f2ecc0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ee8156180 0x7f2ee80ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee80ae570 0x7f2ee8150f00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 --2- 172.21.15.29:0/549002386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ee80b0520 0x7f2ee80acc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.250+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 >> 172.21.15.29:0/549002386 conn(0x7f2ee8067610 msgr2=0x7f2ee8157330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.251+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 shutdown_connections 2023-10-07T09:42:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.251+0000 7f2eeffff700 1 -- 172.21.15.29:0/549002386 wait complete. 2023-10-07T09:42:14.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:42:14.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.472+0000 7fa45d7ff700 1 Processor -- start 2023-10-07T09:42:14.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.472+0000 7fa45d7ff700 1 -- start start 2023-10-07T09:42:14.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.473+0000 7fa45d7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa4580afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:14.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.473+0000 7fa45d7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 0x7fa4580accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:14.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa45d7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580ad230 con 0x7fa4580b0520 2023-10-07T09:42:14.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa45d7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4580ad4d0 con 0x7fa4580ae570 2023-10-07T09:42:14.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa45d7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580ad7b0 con 0x7fa458156180 2023-10-07T09:42:14.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa4567fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 0x7fa4580accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:14.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa456ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa4580afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:14.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa4567fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 0x7fa4580accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:49068/0 (socket says 172.21.15.29:49068) 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa456ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa4580afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:54924/0 (socket says 172.21.15.29:54924) 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa4567fc700 1 -- 172.21.15.29:0/3555204953 learned_addr learned my addr 172.21.15.29:0/3555204953 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.474+0000 7fa4577fe700 1 --1- 172.21.15.29:0/3555204953 >> v1:172.21.15.144:6789/0 conn(0x7fa4580b0520 0x7fa458159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:34136/0 (socket says 172.21.15.29:34136) 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa456ffd700 1 -- 172.21.15.29:0/3555204953 >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 msgr2=0x7fa4580accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa456ffd700 1 --2- 172.21.15.29:0/3555204953 >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 0x7fa4580accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa456ffd700 1 -- 172.21.15.29:0/3555204953 >> v1:172.21.15.144:6789/0 conn(0x7fa4580b0520 legacy=0x7fa458159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa456ffd700 1 -- 172.21.15.29:0/3555204953 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4580ada90 con 0x7fa458156180 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa4567fc700 1 --2- 172.21.15.29:0/3555204953 >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 0x7fa4580accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa456ffd700 1 --2- 172.21.15.29:0/3555204953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa4580afc00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa4480098e0 tx=0x7fa4480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=493785c31380c511 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa455ffb700 1 -- 172.21.15.29:0/3555204953 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800b640 con 0x7fa458156180 2023-10-07T09:42:14.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa455ffb700 1 -- 172.21.15.29:0/3555204953 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44800b7a0 con 0x7fa458156180 2023-10-07T09:42:14.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.475+0000 7fa455ffb700 1 -- 172.21.15.29:0/3555204953 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800bb40 con 0x7fa458156180 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 -- 172.21.15.29:0/3555204953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 msgr2=0x7fa4580afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/3555204953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa4580afc00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa4480098e0 tx=0x7fa4480091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 -- 172.21.15.29:0/3555204953 shutdown_connections 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/3555204953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa4580afc00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/3555204953 >> v2:172.21.15.144:3300/0 conn(0x7fa4580ae570 0x7fa4580accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 -- 172.21.15.29:0/3555204953 >> 172.21.15.29:0/3555204953 conn(0x7fa458067610 msgr2=0x7fa458067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 -- 172.21.15.29:0/3555204953 shutdown_connections 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.476+0000 7fa45d7ff700 1 -- 172.21.15.29:0/3555204953 wait complete. 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.478+0000 7fa45d7ff700 1 Processor -- start 2023-10-07T09:42:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.478+0000 7fa45d7ff700 1 -- start start 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa45d7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4580ae570 0x7fa45814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa45d7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 0x7fa4581402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa45d7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa458144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa45d7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45814d5b0 con 0x7fa4580b0520 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa45d7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45814c9a0 con 0x7fa4580ae570 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa45d7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45814cc80 con 0x7fa458156180 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa4567fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 0x7fa4581402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa4567fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 0x7fa4581402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58980/0 (socket says 172.21.15.29:58980) 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa4577fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa458144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa456ffd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4580ae570 0x7fa45814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.479+0000 7fa4567fc700 1 -- 172.21.15.29:0/2058409632 learned_addr learned my addr 172.21.15.29:0/2058409632 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa456ffd700 1 -- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 msgr2=0x7fa458144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa456ffd700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa458144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa456ffd700 1 -- 172.21.15.29:0/2058409632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 msgr2=0x7fa4581402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa456ffd700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 0x7fa4581402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa456ffd700 1 -- 172.21.15.29:0/2058409632 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0094f0 con 0x7fa4580ae570 2023-10-07T09:42:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa4567fc700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 0x7fa4581402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa456ffd700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4580ae570 0x7fa45814c3c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa448016000 tx=0x7fa448017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44801cd80 con 0x7fa4580ae570 2023-10-07T09:42:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44801a450 con 0x7fa4580ae570 2023-10-07T09:42:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa448009420 con 0x7fa4580ae570 2023-10-07T09:42:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.480+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44801a850 con 0x7fa4580ae570 2023-10-07T09:42:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.481+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45814d240 con 0x7fa4580ae570 2023-10-07T09:42:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.482+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45804c3c0 con 0x7fa4580ae570 2023-10-07T09:42:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.482+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa44801ac80 con 0x7fa4580ae570 2023-10-07T09:42:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.483+0000 7fa43effd700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa444066f60 0x7fa444069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:14.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.483+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa448085f70 con 0x7fa4580ae570 2023-10-07T09:42:14.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.484+0000 7fa4567fc700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa444066f60 0x7fa444069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:14.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.484+0000 7fa4567fc700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa444066f60 0x7fa444069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cb720 tx=0x7fa44c009250 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.489+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4480530a0 con 0x7fa4580ae570 2023-10-07T09:42:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:14.738+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa4580add70 con 0x7fa4580ae570 2023-10-07T09:42:15.177 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-07T09:42:15.176+0000 7fe7329f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:42:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.254+0000 7fa43effd700 1 -- 172.21.15.29:0/2058409632 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa4480566c0 con 0x7fa4580ae570 2023-10-07T09:42:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa444066f60 msgr2=0x7fa444069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa444066f60 0x7fa444069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cb720 tx=0x7fa44c009250 comp rx=0 tx=0).stop 2023-10-07T09:42:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4580ae570 msgr2=0x7fa45814c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4580ae570 0x7fa45814c3c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa448016000 tx=0x7fa448017000 comp rx=0 tx=0).stop 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 shutdown_connections 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa444066f60 0x7fa444069410 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa458156180 0x7fa458144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4580ae570 0x7fa45814c3c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 --2- 172.21.15.29:0/2058409632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4580b0520 0x7fa4581402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 >> 172.21.15.29:0/2058409632 conn(0x7fa458067610 msgr2=0x7fa4581584f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 shutdown_connections 2023-10-07T09:42:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.257+0000 7fa45d7ff700 1 -- 172.21.15.29:0/2058409632 wait complete. 2023-10-07T09:42:15.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:42:15.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.477+0000 7f7cb13ff700 1 Processor -- start 2023-10-07T09:42:15.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.477+0000 7f7cb13ff700 1 -- start start 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7cb13ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cac157290 0x7f7cac0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7cb13ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac0ae8b0 0x7f7cac0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7cb13ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac0b1ca0 con 0x7f7cac0b1840 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7cb13ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cac158ad0 con 0x7f7cac157290 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7cb13ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac15b140 con 0x7f7cac0ae8b0 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cac157290 0x7f7cac0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cac157290 0x7f7cac0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58994/0 (socket says 172.21.15.29:58994) 2023-10-07T09:42:15.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 -- 172.21.15.29:0/2562830917 learned_addr learned my addr 172.21.15.29:0/2562830917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:15.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7cab7fe700 1 --1- 172.21.15.29:0/2562830917 >> v1:172.21.15.29:6789/0 conn(0x7f7cac0b1840 0x7f7cac0ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37070/0 (socket says 172.21.15.29:37070) 2023-10-07T09:42:15.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caa7fc700 1 --2- 172.21.15.29:0/2562830917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac0ae8b0 0x7f7cac0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 -- 172.21.15.29:0/2562830917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac0ae8b0 msgr2=0x7f7cac0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 --2- 172.21.15.29:0/2562830917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac0ae8b0 0x7f7cac0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 -- 172.21.15.29:0/2562830917 >> v1:172.21.15.29:6789/0 conn(0x7f7cac0b1840 legacy=0x7f7cac0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7ca9ffb700 1 -- 172.21.15.29:0/2562830917 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103328347 0 0) 0x7f7ca4001490 con 0x7f7cac0b1840 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.479+0000 7f7caaffd700 1 -- 172.21.15.29:0/2562830917 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac15b420 con 0x7f7cac157290 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.480+0000 7f7caaffd700 1 --2- 172.21.15.29:0/2562830917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac157290 0x7f7cac0cc870 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0099b0 tx=0x7f7c9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26343ba3c5d7c1e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.480+0000 7f7ca9ffb700 1 -- 172.21.15.29:0/2562830917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c00b860 con 0x7f7cac157290 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.480+0000 7f7ca9ffb700 1 -- 172.21.15.29:0/2562830917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c00b9c0 con 0x7f7cac157290 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.480+0000 7f7ca9ffb700 1 -- 172.21.15.29:0/2562830917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c00bd60 con 0x7f7cac157290 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.480+0000 7f7cb13ff700 1 -- 172.21.15.29:0/2562830917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac157290 msgr2=0x7f7cac0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.480+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/2562830917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac157290 0x7f7cac0cc870 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0099b0 tx=0x7f7c9c009250 comp rx=0 tx=0).stop 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.481+0000 7f7cb13ff700 1 -- 172.21.15.29:0/2562830917 shutdown_connections 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.481+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/2562830917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac157290 0x7f7cac0cc870 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.481+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/2562830917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac0ae8b0 0x7f7cac0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.481+0000 7f7cb13ff700 1 -- 172.21.15.29:0/2562830917 >> 172.21.15.29:0/2562830917 conn(0x7f7cac0680a0 msgr2=0x7f7cac0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:15.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.481+0000 7f7cb13ff700 1 -- 172.21.15.29:0/2562830917 shutdown_connections 2023-10-07T09:42:15.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.481+0000 7f7cb13ff700 1 -- 172.21.15.29:0/2562830917 wait complete. 2023-10-07T09:42:15.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.482+0000 7f7cb13ff700 1 Processor -- start 2023-10-07T09:42:15.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.482+0000 7f7cb13ff700 1 -- start start 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7cb13ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 0x7f7cac150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7cb13ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cac157290 0x7f7cac153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7cb13ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac1540a0 0x7f7cac0c1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7cb13ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac152870 con 0x7f7cac0b1840 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7cb13ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac155a70 con 0x7f7cac157290 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7cb13ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1505a0 con 0x7f7cac1540a0 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.483+0000 7f7caaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 0x7f7cac150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 0x7f7cac150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58998/0 (socket says 172.21.15.29:58998) 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 -- 172.21.15.29:0/735402451 learned_addr learned my addr 172.21.15.29:0/735402451 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 -- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac1540a0 msgr2=0x7f7cac0c1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac1540a0 0x7f7cac0c1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 -- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cac157290 msgr2=0x7f7cac153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cac157290 0x7f7cac153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 -- 172.21.15.29:0/735402451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c0094f0 con 0x7f7cac0b1840 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7caaffd700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 0x7f7cac150060 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009fd0 tx=0x7f7c9c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c024440 con 0x7f7cac0b1840 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c0245a0 con 0x7f7cac0b1840 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac152ac0 con 0x7f7cac0b1840 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.484+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c024940 con 0x7f7cac0b1840 2023-10-07T09:42:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.485+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac153080 con 0x7f7cac0b1840 2023-10-07T09:42:15.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.486+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac06a3c0 con 0x7f7cac0b1840 2023-10-07T09:42:15.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.487+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c9c01a420 con 0x7f7cac0b1840 2023-10-07T09:42:15.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.487+0000 7f7c92ffd700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7c98066f30 0x7f7c980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.488+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7c9c086460 con 0x7f7cac0b1840 2023-10-07T09:42:15.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.488+0000 7f7caa7fc700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7c98066f30 0x7f7c980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.489+0000 7f7caa7fc700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7c98066f30 0x7f7c980693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0c22c0 tx=0x7f7ca0008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.491+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c9c053570 con 0x7f7cac0b1840 2023-10-07T09:42:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.734+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cac04c3c0 con 0x7f7cac0b1840 2023-10-07T09:42:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.735+0000 7f7c92ffd700 1 -- 172.21.15.29:0/735402451 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f7c9c056b90 con 0x7f7cac0b1840 2023-10-07T09:42:15.736 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:15.736 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:42:11.722170+0000","modified":"2023-10-07T09:42:15.247175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6834/2769082805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2769082805},{"type":"v1","addr":"172.21.15.29:6834","nonce":2769082805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7c98066f30 msgr2=0x7f7c980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7c98066f30 0x7f7c980693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0c22c0 tx=0x7f7ca0008040 comp rx=0 tx=0).stop 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 msgr2=0x7f7cac150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 0x7f7cac150060 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009fd0 tx=0x7f7c9c017000 comp rx=0 tx=0).stop 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 shutdown_connections 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7c98066f30 0x7f7c980693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cac1540a0 0x7f7cac0c1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cac0b1840 0x7f7cac150060 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 --2- 172.21.15.29:0/735402451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cac157290 0x7f7cac153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.737+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 >> 172.21.15.29:0/735402451 conn(0x7f7cac0680a0 msgr2=0x7f7cac1508c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.738+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 shutdown_connections 2023-10-07T09:42:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.738+0000 7f7cb13ff700 1 -- 172.21.15.29:0/735402451 wait complete. 2023-10-07T09:42:15.740 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-07T09:42:15.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:42:15.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.952+0000 7f63ca9ff700 1 Processor -- start 2023-10-07T09:42:15.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.952+0000 7f63ca9ff700 1 -- start start 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63ca9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63c4159ec0 0x7f63c415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63ca9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c414d790 con 0x7f63c40ad370 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63ca9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40ae5d0 con 0x7f63c40af400 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63ca9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c415a880 con 0x7f63c4159ec0 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63c3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f63c40ad370 0x7f63c40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37082/0 (socket says 172.21.15.29:37082) 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63c3fff700 1 -- 172.21.15.29:0/2641068012 learned_addr learned my addr 172.21.15.29:0/2641068012 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63c37fe700 1 --2- 172.21.15.29:0/2641068012 >> v2:172.21.15.144:3300/0 conn(0x7f63c4159ec0 0x7f63c415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470605655 0 0) 0x7f63c414d790 con 0x7f63c40ad370 2023-10-07T09:42:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.954+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003570 con 0x7f63c40ad370 2023-10-07T09:42:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209342871 0 0) 0x7f63c40ae5d0 con 0x7f63c40af400 2023-10-07T09:42:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c414d790 con 0x7f63c40af400 2023-10-07T09:42:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584961389 0 0) 0x7f63ac003570 con 0x7f63c40ad370 2023-10-07T09:42:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c37fe700 1 -- 172.21.15.29:0/2641068012 >> v1:172.21.15.144:6789/0 conn(0x7f63c40af400 legacy=0x7f63c40af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c37fe700 1 -- 172.21.15.29:0/2641068012 >> v1:172.21.15.29:6789/0 conn(0x7f63c40ad370 legacy=0x7f63c40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c37fe700 1 -- 172.21.15.29:0/2641068012 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40ad740 con 0x7f63c4159ec0 2023-10-07T09:42:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c37fe700 1 --2- 172.21.15.29:0/2641068012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c415a340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098e0 tx=0x7f63b80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd42cd5824671888 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4001470 con 0x7f63c4159ec0 2023-10-07T09:42:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8011420 con 0x7f63c4159ec0 2023-10-07T09:42:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.955+0000 7f63c2ffd700 1 -- 172.21.15.29:0/2641068012 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80117c0 con 0x7f63c4159ec0 2023-10-07T09:42:15.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 -- 172.21.15.29:0/2641068012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 msgr2=0x7f63c415a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/2641068012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c415a340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098e0 tx=0x7f63b80091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 -- 172.21.15.29:0/2641068012 shutdown_connections 2023-10-07T09:42:15.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/2641068012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c415a340 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 -- 172.21.15.29:0/2641068012 >> 172.21.15.29:0/2641068012 conn(0x7f63c40680a0 msgr2=0x7f63c40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:15.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 -- 172.21.15.29:0/2641068012 shutdown_connections 2023-10-07T09:42:15.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.956+0000 7f63ca9ff700 1 -- 172.21.15.29:0/2641068012 wait complete. 2023-10-07T09:42:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.958+0000 7f63ca9ff700 1 Processor -- start 2023-10-07T09:42:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.958+0000 7f63ca9ff700 1 -- start start 2023-10-07T09:42:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63ca9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 0x7f63c414fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63ca9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40af400 0x7f63c4150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63ca9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c41692d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63ca9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4150900 con 0x7f63c40ad370 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63ca9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c416af00 con 0x7f63c4159ec0 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63ca9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4141130 con 0x7f63c40af400 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 0x7f63c414fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 0x7f63c414fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59000/0 (socket says 172.21.15.29:59000) 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 -- 172.21.15.29:0/3511232920 learned_addr learned my addr 172.21.15.29:0/3511232920 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c37fe700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40af400 0x7f63c4150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c8f9c700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c41692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 -- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40af400 msgr2=0x7f63c4150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40af400 0x7f63c4150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 -- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 msgr2=0x7f63c41692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c41692d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.959+0000 7f63c3fff700 1 -- 172.21.15.29:0/3511232920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8009420 con 0x7f63c40ad370 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.960+0000 7f63c3fff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 0x7f63c414fb30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f63b400cba0 tx=0x7f63b4001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.960+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007e90 con 0x7f63c40ad370 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.960+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4007360 con 0x7f63c40ad370 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.960+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4004830 con 0x7f63c40ad370 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.960+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c416a5d0 con 0x7f63c40ad370 2023-10-07T09:42:15.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.960+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c416ab90 con 0x7f63c40ad370 2023-10-07T09:42:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.961+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c404c3c0 con 0x7f63c40ad370 2023-10-07T09:42:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.962+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f63b4004ba0 con 0x7f63c40ad370 2023-10-07T09:42:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.963+0000 7f63abfff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.963+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f63b4083bd0 con 0x7f63c40ad370 2023-10-07T09:42:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.964+0000 7f63c37fe700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.965+0000 7f63c37fe700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63b80093f0 tx=0x7f63b8016000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:15.968+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63b4050cc0 con 0x7f63c40ad370 2023-10-07T09:42:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.171+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c40b0dd0 con 0x7f63c40ad370 2023-10-07T09:42:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.172+0000 7f63abfff700 1 -- 172.21.15.29:0/3511232920 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+15973 (secure 0 0 0) 0x7f63b4018020 con 0x7f63c40ad370 2023-10-07T09:42:16.173 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:16.173 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":61,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:42:13.105568+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:42:09.508279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-07T09:42:10.701497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 msgr2=0x7f63b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63b80093f0 tx=0x7f63b8016000 comp rx=0 tx=0).stop 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 msgr2=0x7f63c414fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 0x7f63c414fb30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f63b400cba0 tx=0x7f63b4001b50 comp rx=0 tx=0).stop 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 shutdown_connections 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c4159ec0 0x7f63c41692d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40ad370 0x7f63c414fb30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 --2- 172.21.15.29:0/3511232920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c40af400 0x7f63c4150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 >> 172.21.15.29:0/3511232920 conn(0x7f63c40680a0 msgr2=0x7f63c4069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 shutdown_connections 2023-10-07T09:42:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.175+0000 7f63ca9ff700 1 -- 172.21.15.29:0/3511232920 wait complete. 2023-10-07T09:42:16.194 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:42:16.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:42:16.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.389+0000 7f50eb7ff700 1 Processor -- start 2023-10-07T09:42:16.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.389+0000 7f50eb7ff700 1 -- start start 2023-10-07T09:42:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50eb7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50e40ad220 0x7f50e4159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50eb7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 0x7f50e40ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50eb7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e415af00 con 0x7f50e4068620 2023-10-07T09:42:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50eb7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e40ac9d0 con 0x7f50e40ad220 2023-10-07T09:42:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50eb7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40ada40 con 0x7f50e40ad5f0 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50e8d9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 0x7f50e40ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50e959b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50e40ad220 0x7f50e4159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 0x7f50e40ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:54954/0 (socket says 172.21.15.29:54954) 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 -- 172.21.15.29:0/2834071629 learned_addr learned my addr 172.21.15.29:0/2834071629 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.391+0000 7f50e9d9c700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f50e4068620 0x7f50e415d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:34154/0 (socket says 172.21.15.29:34154) 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50dbfff700 1 -- 172.21.15.29:0/2834071629 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570670425 0 0) 0x7f50e415af00 con 0x7f50e4068620 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50dbfff700 1 -- 172.21.15.29:0/2834071629 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0003570 con 0x7f50e4068620 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 -- 172.21.15.29:0/2834071629 >> v2:172.21.15.144:3300/0 conn(0x7f50e40ad220 msgr2=0x7f50e4159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 --2- 172.21.15.29:0/2834071629 >> v2:172.21.15.144:3300/0 conn(0x7f50e40ad220 0x7f50e4159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 -- 172.21.15.29:0/2834071629 >> v1:172.21.15.144:6789/0 conn(0x7f50e4068620 legacy=0x7f50e415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 -- 172.21.15.29:0/2834071629 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e415a250 con 0x7f50e40ad5f0 2023-10-07T09:42:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e959b700 1 --2- 172.21.15.29:0/2834071629 >> v2:172.21.15.144:3300/0 conn(0x7f50e40ad220 0x7f50e4159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.392+0000 7f50e8d9a700 1 --2- 172.21.15.29:0/2834071629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 0x7f50e40ac490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50d40099b0 tx=0x7f50d4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147e3dce061e9e38 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50dbfff700 1 -- 172.21.15.29:0/2834071629 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400b680 con 0x7f50e40ad5f0 2023-10-07T09:42:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50dbfff700 1 -- 172.21.15.29:0/2834071629 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d400b7e0 con 0x7f50e40ad5f0 2023-10-07T09:42:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50dbfff700 1 -- 172.21.15.29:0/2834071629 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400bb60 con 0x7f50e40ad5f0 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 -- 172.21.15.29:0/2834071629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 msgr2=0x7f50e40ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/2834071629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 0x7f50e40ac490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50d40099b0 tx=0x7f50d4009250 comp rx=0 tx=0).stop 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 -- 172.21.15.29:0/2834071629 shutdown_connections 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/2834071629 >> v2:172.21.15.144:3300/0 conn(0x7f50e40ad220 0x7f50e4159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/2834071629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e40ad5f0 0x7f50e40ac490 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 -- 172.21.15.29:0/2834071629 >> 172.21.15.29:0/2834071629 conn(0x7f50e406dd00 msgr2=0x7f50e406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.393+0000 7f50eb7ff700 1 -- 172.21.15.29:0/2834071629 shutdown_connections 2023-10-07T09:42:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.394+0000 7f50eb7ff700 1 -- 172.21.15.29:0/2834071629 wait complete. 2023-10-07T09:42:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.395+0000 7f50eb7ff700 1 Processor -- start 2023-10-07T09:42:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.395+0000 7f50eb7ff700 1 -- start start 2023-10-07T09:42:16.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50eb7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e4068620 0x7f50e40cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50eb7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 0x7f50e40c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50eb7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50e40ad5f0 0x7f50e4169980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50eb7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40b0190 con 0x7f50e40ad220 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50eb7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40c9db0 con 0x7f50e40ad5f0 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50eb7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40c9b20 con 0x7f50e4068620 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50e8d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 0x7f50e40c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50e8d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 0x7f50e40c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59008/0 (socket says 172.21.15.29:59008) 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.396+0000 7f50e8d9a700 1 -- 172.21.15.29:0/3813658584 learned_addr learned my addr 172.21.15.29:0/3813658584 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e9d9c700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50e40ad5f0 0x7f50e4169980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e959b700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e4068620 0x7f50e40cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e8d9a700 1 -- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e4068620 msgr2=0x7f50e40cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e8d9a700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e4068620 0x7f50e40cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e8d9a700 1 -- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50e40ad5f0 msgr2=0x7f50e4169980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e8d9a700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50e40ad5f0 0x7f50e4169980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e8d9a700 1 -- 172.21.15.29:0/3813658584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e0009420 con 0x7f50e40ad220 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e959b700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e4068620 0x7f50e40cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e9d9c700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50e40ad5f0 0x7f50e4169980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50e8d9a700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 0x7f50e40c95e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f50e40adb80 tx=0x7f50d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400b950 con 0x7f50e40ad220 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d400bad0 con 0x7f50e40ad220 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d401a680 con 0x7f50e40ad220 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.397+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d40094f0 con 0x7f50e40ad220 2023-10-07T09:42:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.398+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e40afb40 con 0x7f50e40ad220 2023-10-07T09:42:16.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.399+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e40b0300 con 0x7f50e40ad220 2023-10-07T09:42:16.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.400+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f50d401aa30 con 0x7f50e40ad220 2023-10-07T09:42:16.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.400+0000 7f50d97fa700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50c8066f30 0x7f50c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.400+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f50d40862b0 con 0x7f50e40ad220 2023-10-07T09:42:16.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.401+0000 7f50e959b700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50c8066f30 0x7f50c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.402+0000 7f50e959b700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50c8066f30 0x7f50c80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50e00093f0 tx=0x7f50e0008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.404+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50d40533c0 con 0x7f50e40ad220 2023-10-07T09:42:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.603+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50e404c3c0 con 0x7f50c8066f30 2023-10-07T09:42:16.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.614+0000 7f50d97fa700 1 -- 172.21.15.29:0/3813658584 <== mgr.5195 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288935 (secure 0 0 0) 0x7f50e404c3c0 con 0x7f50c8066f30 2023-10-07T09:42:16.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50c8066f30 msgr2=0x7f50c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50c8066f30 0x7f50c80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50e00093f0 tx=0x7f50e0008040 comp rx=0 tx=0).stop 2023-10-07T09:42:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 msgr2=0x7f50e40c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 0x7f50e40c95e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f50e40adb80 tx=0x7f50d4017000 comp rx=0 tx=0).stop 2023-10-07T09:42:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 shutdown_connections 2023-10-07T09:42:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50c8066f30 0x7f50c80693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50e40ad5f0 0x7f50e4169980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50e4068620 0x7f50e40cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 --2- 172.21.15.29:0/3813658584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e40ad220 0x7f50e40c95e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 >> 172.21.15.29:0/3813658584 conn(0x7f50e406dd00 msgr2=0x7f50e406ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 shutdown_connections 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.618+0000 7f50eb7ff700 1 -- 172.21.15.29:0/3813658584 wait complete. 2023-10-07T09:42:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:42:16.637 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-07T09:42:15.363255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2429980,"kb_used_data":6620,"kb_used_omap":0,"kb_used_meta":2423296,"kb_avail":752544740,"statfs":{"total":773094113280,"available":770605813760,"internally_reserved":0,"allocated":6778880,"data_stored":3727613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2481455104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.151188"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.124772+0000","last_change":"2023-10-07T09:42:12.124772+0000","last_active":"2023-10-07T09:42:12.124772+0000","last_peered":"2023-10-07T09:42:12.124772+0000","last_clean":"2023-10-07T09:42:12.124772+0000","last_became_active":"2023-10-07T09:42:12.124478+0000","last_became_peered":"2023-10-07T09:42:12.124478+0000","last_unstale":"2023-10-07T09:42:12.124772+0000","last_undegraded":"2023-10-07T09:42:12.124772+0000","last_fullsized":"2023-10-07T09:42:12.124772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:52:24.831499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126460+0000","last_change":"2023-10-07T09:42:12.126460+0000","last_active":"2023-10-07T09:42:12.126460+0000","last_peered":"2023-10-07T09:42:12.126460+0000","last_clean":"2023-10-07T09:42:12.126460+0000","last_became_active":"2023-10-07T09:42:12.126045+0000","last_became_peered":"2023-10-07T09:42:12.126045+0000","last_unstale":"2023-10-07T09:42:12.126460+0000","last_undegraded":"2023-10-07T09:42:12.126460+0000","last_fullsized":"2023-10-07T09:42:12.126460+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:05.782312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121617+0000","last_change":"2023-10-07T09:42:12.121617+0000","last_active":"2023-10-07T09:42:12.121617+0000","last_peered":"2023-10-07T09:42:12.121617+0000","last_clean":"2023-10-07T09:42:12.121617+0000","last_became_active":"2023-10-07T09:42:12.121245+0000","last_became_peered":"2023-10-07T09:42:12.121245+0000","last_unstale":"2023-10-07T09:42:12.121617+0000","last_undegraded":"2023-10-07T09:42:12.121617+0000","last_fullsized":"2023-10-07T09:42:12.121617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:14:00.035126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.130770+0000","last_change":"2023-10-07T09:42:12.130770+0000","last_active":"2023-10-07T09:42:12.130770+0000","last_peered":"2023-10-07T09:42:12.130770+0000","last_clean":"2023-10-07T09:42:12.130770+0000","last_became_active":"2023-10-07T09:42:12.130439+0000","last_became_peered":"2023-10-07T09:42:12.130439+0000","last_unstale":"2023-10-07T09:42:12.130770+0000","last_undegraded":"2023-10-07T09:42:12.130770+0000","last_fullsized":"2023-10-07T09:42:12.130770+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:57.864645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120436+0000","last_change":"2023-10-07T09:42:12.120436+0000","last_active":"2023-10-07T09:42:12.120436+0000","last_peered":"2023-10-07T09:42:12.120436+0000","last_clean":"2023-10-07T09:42:12.120436+0000","last_became_active":"2023-10-07T09:42:12.118858+0000","last_became_peered":"2023-10-07T09:42:12.118858+0000","last_unstale":"2023-10-07T09:42:12.120436+0000","last_undegraded":"2023-10-07T09:42:12.120436+0000","last_fullsized":"2023-10-07T09:42:12.120436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:05:13.537403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126860+0000","last_change":"2023-10-07T09:42:12.126860+0000","last_active":"2023-10-07T09:42:12.126860+0000","last_peered":"2023-10-07T09:42:12.126860+0000","last_clean":"2023-10-07T09:42:12.126860+0000","last_became_active":"2023-10-07T09:42:12.126574+0000","last_became_peered":"2023-10-07T09:42:12.126574+0000","last_unstale":"2023-10-07T09:42:12.126860+0000","last_undegraded":"2023-10-07T09:42:12.126860+0000","last_fullsized":"2023-10-07T09:42:12.126860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:33:31.178389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123128+0000","last_change":"2023-10-07T09:42:12.123128+0000","last_active":"2023-10-07T09:42:12.123128+0000","last_peered":"2023-10-07T09:42:12.123128+0000","last_clean":"2023-10-07T09:42:12.123128+0000","last_became_active":"2023-10-07T09:42:12.122816+0000","last_became_peered":"2023-10-07T09:42:12.122816+0000","last_unstale":"2023-10-07T09:42:12.123128+0000","last_undegraded":"2023-10-07T09:42:12.123128+0000","last_fullsized":"2023-10-07T09:42:12.123128+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:52:23.811737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131352+0000","last_change":"2023-10-07T09:42:12.131352+0000","last_active":"2023-10-07T09:42:12.131352+0000","last_peered":"2023-10-07T09:42:12.131352+0000","last_clean":"2023-10-07T09:42:12.131352+0000","last_became_active":"2023-10-07T09:42:12.131013+0000","last_became_peered":"2023-10-07T09:42:12.131013+0000","last_unstale":"2023-10-07T09:42:12.131352+0000","last_undegraded":"2023-10-07T09:42:12.131352+0000","last_fullsized":"2023-10-07T09:42:12.131352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:03.066571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120918+0000","last_change":"2023-10-07T09:42:12.120918+0000","last_active":"2023-10-07T09:42:12.120918+0000","last_peered":"2023-10-07T09:42:12.120918+0000","last_clean":"2023-10-07T09:42:12.120918+0000","last_became_active":"2023-10-07T09:42:12.120599+0000","last_became_peered":"2023-10-07T09:42:12.120599+0000","last_unstale":"2023-10-07T09:42:12.120918+0000","last_undegraded":"2023-10-07T09:42:12.120918+0000","last_fullsized":"2023-10-07T09:42:12.120918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:33.129125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.130815+0000","last_change":"2023-10-07T09:42:12.130815+0000","last_active":"2023-10-07T09:42:12.130815+0000","last_peered":"2023-10-07T09:42:12.130815+0000","last_clean":"2023-10-07T09:42:12.130815+0000","last_became_active":"2023-10-07T09:42:12.130446+0000","last_became_peered":"2023-10-07T09:42:12.130446+0000","last_unstale":"2023-10-07T09:42:12.130815+0000","last_undegraded":"2023-10-07T09:42:12.130815+0000","last_fullsized":"2023-10-07T09:42:12.130815+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:40.103291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120847+0000","last_change":"2023-10-07T09:42:12.120847+0000","last_active":"2023-10-07T09:42:12.120847+0000","last_peered":"2023-10-07T09:42:12.120847+0000","last_clean":"2023-10-07T09:42:12.120847+0000","last_became_active":"2023-10-07T09:42:12.120564+0000","last_became_peered":"2023-10-07T09:42:12.120564+0000","last_unstale":"2023-10-07T09:42:12.120847+0000","last_undegraded":"2023-10-07T09:42:12.120847+0000","last_fullsized":"2023-10-07T09:42:12.120847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:05.074867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120828+0000","last_change":"2023-10-07T09:42:12.120828+0000","last_active":"2023-10-07T09:42:12.120828+0000","last_peered":"2023-10-07T09:42:12.120828+0000","last_clean":"2023-10-07T09:42:12.120828+0000","last_became_active":"2023-10-07T09:42:12.120518+0000","last_became_peered":"2023-10-07T09:42:12.120518+0000","last_unstale":"2023-10-07T09:42:12.120828+0000","last_undegraded":"2023-10-07T09:42:12.120828+0000","last_fullsized":"2023-10-07T09:42:12.120828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:31:39.819015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132156+0000","last_change":"2023-10-07T09:42:12.132156+0000","last_active":"2023-10-07T09:42:12.132156+0000","last_peered":"2023-10-07T09:42:12.132156+0000","last_clean":"2023-10-07T09:42:12.132156+0000","last_became_active":"2023-10-07T09:42:12.131825+0000","last_became_peered":"2023-10-07T09:42:12.131825+0000","last_unstale":"2023-10-07T09:42:12.132156+0000","last_undegraded":"2023-10-07T09:42:12.132156+0000","last_fullsized":"2023-10-07T09:42:12.132156+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:13:17.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120211+0000","last_change":"2023-10-07T09:42:12.120211+0000","last_active":"2023-10-07T09:42:12.120211+0000","last_peered":"2023-10-07T09:42:12.120211+0000","last_clean":"2023-10-07T09:42:12.120211+0000","last_became_active":"2023-10-07T09:42:12.119878+0000","last_became_peered":"2023-10-07T09:42:12.119878+0000","last_unstale":"2023-10-07T09:42:12.120211+0000","last_undegraded":"2023-10-07T09:42:12.120211+0000","last_fullsized":"2023-10-07T09:42:12.120211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:51.992198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120556+0000","last_change":"2023-10-07T09:42:12.120556+0000","last_active":"2023-10-07T09:42:12.120556+0000","last_peered":"2023-10-07T09:42:12.120556+0000","last_clean":"2023-10-07T09:42:12.120556+0000","last_became_active":"2023-10-07T09:42:12.120290+0000","last_became_peered":"2023-10-07T09:42:12.120290+0000","last_unstale":"2023-10-07T09:42:12.120556+0000","last_undegraded":"2023-10-07T09:42:12.120556+0000","last_fullsized":"2023-10-07T09:42:12.120556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:50.901171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131719+0000","last_change":"2023-10-07T09:42:12.131719+0000","last_active":"2023-10-07T09:42:12.131719+0000","last_peered":"2023-10-07T09:42:12.131719+0000","last_clean":"2023-10-07T09:42:12.131719+0000","last_became_active":"2023-10-07T09:42:12.131252+0000","last_became_peered":"2023-10-07T09:42:12.131252+0000","last_unstale":"2023-10-07T09:42:12.131719+0000","last_undegraded":"2023-10-07T09:42:12.131719+0000","last_fullsized":"2023-10-07T09:42:12.131719+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:44.145951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.119571+0000","last_change":"2023-10-07T09:42:12.119571+0000","last_active":"2023-10-07T09:42:12.119571+0000","last_peered":"2023-10-07T09:42:12.119571+0000","last_clean":"2023-10-07T09:42:12.119571+0000","last_became_active":"2023-10-07T09:42:12.119315+0000","last_became_peered":"2023-10-07T09:42:12.119315+0000","last_unstale":"2023-10-07T09:42:12.119571+0000","last_undegraded":"2023-10-07T09:42:12.119571+0000","last_fullsized":"2023-10-07T09:42:12.119571+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:00:41.736832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127297+0000","last_change":"2023-10-07T09:42:12.127297+0000","last_active":"2023-10-07T09:42:12.127297+0000","last_peered":"2023-10-07T09:42:12.127297+0000","last_clean":"2023-10-07T09:42:12.127297+0000","last_became_active":"2023-10-07T09:42:12.126870+0000","last_became_peered":"2023-10-07T09:42:12.126870+0000","last_unstale":"2023-10-07T09:42:12.127297+0000","last_undegraded":"2023-10-07T09:42:12.127297+0000","last_fullsized":"2023-10-07T09:42:12.127297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:31:11.738882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131934+0000","last_change":"2023-10-07T09:42:12.131934+0000","last_active":"2023-10-07T09:42:12.131934+0000","last_peered":"2023-10-07T09:42:12.131934+0000","last_clean":"2023-10-07T09:42:12.131934+0000","last_became_active":"2023-10-07T09:42:12.131643+0000","last_became_peered":"2023-10-07T09:42:12.131643+0000","last_unstale":"2023-10-07T09:42:12.131934+0000","last_undegraded":"2023-10-07T09:42:12.131934+0000","last_fullsized":"2023-10-07T09:42:12.131934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:51:20.622847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126673+0000","last_change":"2023-10-07T09:42:12.126673+0000","last_active":"2023-10-07T09:42:12.126673+0000","last_peered":"2023-10-07T09:42:12.126673+0000","last_clean":"2023-10-07T09:42:12.126673+0000","last_became_active":"2023-10-07T09:42:12.126322+0000","last_became_peered":"2023-10-07T09:42:12.126322+0000","last_unstale":"2023-10-07T09:42:12.126673+0000","last_undegraded":"2023-10-07T09:42:12.126673+0000","last_fullsized":"2023-10-07T09:42:12.126673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:08.038437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131590+0000","last_change":"2023-10-07T09:42:12.131590+0000","last_active":"2023-10-07T09:42:12.131590+0000","last_peered":"2023-10-07T09:42:12.131590+0000","last_clean":"2023-10-07T09:42:12.131590+0000","last_became_active":"2023-10-07T09:42:12.131287+0000","last_became_peered":"2023-10-07T09:42:12.131287+0000","last_unstale":"2023-10-07T09:42:12.131590+0000","last_undegraded":"2023-10-07T09:42:12.131590+0000","last_fullsized":"2023-10-07T09:42:12.131590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:58:16.359660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129051+0000","last_change":"2023-10-07T09:42:12.129051+0000","last_active":"2023-10-07T09:42:12.129051+0000","last_peered":"2023-10-07T09:42:12.129051+0000","last_clean":"2023-10-07T09:42:12.129051+0000","last_became_active":"2023-10-07T09:42:12.128749+0000","last_became_peered":"2023-10-07T09:42:12.128749+0000","last_unstale":"2023-10-07T09:42:12.129051+0000","last_undegraded":"2023-10-07T09:42:12.129051+0000","last_fullsized":"2023-10-07T09:42:12.129051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:27.390563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127630+0000","last_change":"2023-10-07T09:42:12.127630+0000","last_active":"2023-10-07T09:42:12.127630+0000","last_peered":"2023-10-07T09:42:12.127630+0000","last_clean":"2023-10-07T09:42:12.127630+0000","last_became_active":"2023-10-07T09:42:12.127263+0000","last_became_peered":"2023-10-07T09:42:12.127263+0000","last_unstale":"2023-10-07T09:42:12.127630+0000","last_undegraded":"2023-10-07T09:42:12.127630+0000","last_fullsized":"2023-10-07T09:42:12.127630+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:49.566442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.128842+0000","last_change":"2023-10-07T09:42:12.128842+0000","last_active":"2023-10-07T09:42:12.128842+0000","last_peered":"2023-10-07T09:42:12.128842+0000","last_clean":"2023-10-07T09:42:12.128842+0000","last_became_active":"2023-10-07T09:42:12.128551+0000","last_became_peered":"2023-10-07T09:42:12.128551+0000","last_unstale":"2023-10-07T09:42:12.128842+0000","last_undegraded":"2023-10-07T09:42:12.128842+0000","last_fullsized":"2023-10-07T09:42:12.128842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:22.056475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132283+0000","last_change":"2023-10-07T09:42:12.132283+0000","last_active":"2023-10-07T09:42:12.132283+0000","last_peered":"2023-10-07T09:42:12.132283+0000","last_clean":"2023-10-07T09:42:12.132283+0000","last_became_active":"2023-10-07T09:42:12.131048+0000","last_became_peered":"2023-10-07T09:42:12.131048+0000","last_unstale":"2023-10-07T09:42:12.132283+0000","last_undegraded":"2023-10-07T09:42:12.132283+0000","last_fullsized":"2023-10-07T09:42:12.132283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:13:29.579840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132151+0000","last_change":"2023-10-07T09:42:12.132151+0000","last_active":"2023-10-07T09:42:12.132151+0000","last_peered":"2023-10-07T09:42:12.132151+0000","last_clean":"2023-10-07T09:42:12.132151+0000","last_became_active":"2023-10-07T09:42:12.131872+0000","last_became_peered":"2023-10-07T09:42:12.131872+0000","last_unstale":"2023-10-07T09:42:12.132151+0000","last_undegraded":"2023-10-07T09:42:12.132151+0000","last_fullsized":"2023-10-07T09:42:12.132151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:11:41.208779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.117072+0000","last_change":"2023-10-07T09:42:12.117072+0000","last_active":"2023-10-07T09:42:12.117072+0000","last_peered":"2023-10-07T09:42:12.117072+0000","last_clean":"2023-10-07T09:42:12.117072+0000","last_became_active":"2023-10-07T09:42:12.116727+0000","last_became_peered":"2023-10-07T09:42:12.116727+0000","last_unstale":"2023-10-07T09:42:12.117072+0000","last_undegraded":"2023-10-07T09:42:12.117072+0000","last_fullsized":"2023-10-07T09:42:12.117072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:54.073522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121966+0000","last_change":"2023-10-07T09:42:12.121966+0000","last_active":"2023-10-07T09:42:12.121966+0000","last_peered":"2023-10-07T09:42:12.121966+0000","last_clean":"2023-10-07T09:42:12.121966+0000","last_became_active":"2023-10-07T09:42:12.120878+0000","last_became_peered":"2023-10-07T09:42:12.120878+0000","last_unstale":"2023-10-07T09:42:12.121966+0000","last_undegraded":"2023-10-07T09:42:12.121966+0000","last_fullsized":"2023-10-07T09:42:12.121966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:34:26.900282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132072+0000","last_change":"2023-10-07T09:42:12.132072+0000","last_active":"2023-10-07T09:42:12.132072+0000","last_peered":"2023-10-07T09:42:12.132072+0000","last_clean":"2023-10-07T09:42:12.132072+0000","last_became_active":"2023-10-07T09:42:12.131735+0000","last_became_peered":"2023-10-07T09:42:12.131735+0000","last_unstale":"2023-10-07T09:42:12.132072+0000","last_undegraded":"2023-10-07T09:42:12.132072+0000","last_fullsized":"2023-10-07T09:42:12.132072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:11:41.208779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123904+0000","last_change":"2023-10-07T09:42:12.123904+0000","last_active":"2023-10-07T09:42:12.123904+0000","last_peered":"2023-10-07T09:42:12.123904+0000","last_clean":"2023-10-07T09:42:12.123904+0000","last_became_active":"2023-10-07T09:42:12.123638+0000","last_became_peered":"2023-10-07T09:42:12.123638+0000","last_unstale":"2023-10-07T09:42:12.123904+0000","last_undegraded":"2023-10-07T09:42:12.123904+0000","last_fullsized":"2023-10-07T09:42:12.123904+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:21:48.056713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123704+0000","last_change":"2023-10-07T09:42:12.123704+0000","last_active":"2023-10-07T09:42:12.123704+0000","last_peered":"2023-10-07T09:42:12.123704+0000","last_clean":"2023-10-07T09:42:12.123704+0000","last_became_active":"2023-10-07T09:42:12.123461+0000","last_became_peered":"2023-10-07T09:42:12.123461+0000","last_unstale":"2023-10-07T09:42:12.123704+0000","last_undegraded":"2023-10-07T09:42:12.123704+0000","last_fullsized":"2023-10-07T09:42:12.123704+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:05:13.537403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.128726+0000","last_change":"2023-10-07T09:42:12.128726+0000","last_active":"2023-10-07T09:42:12.128726+0000","last_peered":"2023-10-07T09:42:12.128726+0000","last_clean":"2023-10-07T09:42:12.128726+0000","last_became_active":"2023-10-07T09:42:12.128486+0000","last_became_peered":"2023-10-07T09:42:12.128486+0000","last_unstale":"2023-10-07T09:42:12.128726+0000","last_undegraded":"2023-10-07T09:42:12.128726+0000","last_fullsized":"2023-10-07T09:42:12.128726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:35.256308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.118348+0000","last_change":"2023-10-07T09:42:12.118348+0000","last_active":"2023-10-07T09:42:12.118348+0000","last_peered":"2023-10-07T09:42:12.118348+0000","last_clean":"2023-10-07T09:42:12.118348+0000","last_became_active":"2023-10-07T09:42:12.118076+0000","last_became_peered":"2023-10-07T09:42:12.118076+0000","last_unstale":"2023-10-07T09:42:12.118348+0000","last_undegraded":"2023-10-07T09:42:12.118348+0000","last_fullsized":"2023-10-07T09:42:12.118348+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:30.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"60'4","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.248161+0000","last_change":"2023-10-07T09:42:11.231473+0000","last_active":"2023-10-07T09:42:12.248161+0000","last_peered":"2023-10-07T09:42:12.248161+0000","last_clean":"2023-10-07T09:42:12.248161+0000","last_became_active":"2023-10-07T09:42:11.231207+0000","last_became_peered":"2023-10-07T09:42:11.231207+0000","last_unstale":"2023-10-07T09:42:12.248161+0000","last_undegraded":"2023-10-07T09:42:12.248161+0000","last_fullsized":"2023-10-07T09:42:12.248161+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:39.097152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132558+0000","last_change":"2023-10-07T09:42:12.132558+0000","last_active":"2023-10-07T09:42:12.132558+0000","last_peered":"2023-10-07T09:42:12.132558+0000","last_clean":"2023-10-07T09:42:12.132558+0000","last_became_active":"2023-10-07T09:42:12.130893+0000","last_became_peered":"2023-10-07T09:42:12.130893+0000","last_unstale":"2023-10-07T09:42:12.132558+0000","last_undegraded":"2023-10-07T09:42:12.132558+0000","last_fullsized":"2023-10-07T09:42:12.132558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:38:02.193963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.111536+0000","last_change":"2023-10-07T09:42:11.235972+0000","last_active":"2023-10-07T09:42:12.111536+0000","last_peered":"2023-10-07T09:42:12.111536+0000","last_clean":"2023-10-07T09:42:12.111536+0000","last_became_active":"2023-10-07T09:42:11.235710+0000","last_became_peered":"2023-10-07T09:42:11.235710+0000","last_unstale":"2023-10-07T09:42:12.111536+0000","last_undegraded":"2023-10-07T09:42:12.111536+0000","last_fullsized":"2023-10-07T09:42:12.111536+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:26.384424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.117347+0000","last_change":"2023-10-07T09:42:12.117347+0000","last_active":"2023-10-07T09:42:12.117347+0000","last_peered":"2023-10-07T09:42:12.117347+0000","last_clean":"2023-10-07T09:42:12.117347+0000","last_became_active":"2023-10-07T09:42:12.117054+0000","last_became_peered":"2023-10-07T09:42:12.117054+0000","last_unstale":"2023-10-07T09:42:12.117347+0000","last_undegraded":"2023-10-07T09:42:12.117347+0000","last_fullsized":"2023-10-07T09:42:12.117347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:50.901171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.245803+0000","last_change":"2023-10-07T09:42:11.234456+0000","last_active":"2023-10-07T09:42:12.245803+0000","last_peered":"2023-10-07T09:42:12.245803+0000","last_clean":"2023-10-07T09:42:12.245803+0000","last_became_active":"2023-10-07T09:42:11.234183+0000","last_became_peered":"2023-10-07T09:42:11.234183+0000","last_unstale":"2023-10-07T09:42:12.245803+0000","last_undegraded":"2023-10-07T09:42:12.245803+0000","last_fullsized":"2023-10-07T09:42:12.245803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:43.139812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129991+0000","last_change":"2023-10-07T09:42:12.129991+0000","last_active":"2023-10-07T09:42:12.129991+0000","last_peered":"2023-10-07T09:42:12.129991+0000","last_clean":"2023-10-07T09:42:12.129991+0000","last_became_active":"2023-10-07T09:42:12.129660+0000","last_became_peered":"2023-10-07T09:42:12.129660+0000","last_unstale":"2023-10-07T09:42:12.129991+0000","last_undegraded":"2023-10-07T09:42:12.129991+0000","last_fullsized":"2023-10-07T09:42:12.129991+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:05.074867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.245575+0000","last_change":"2023-10-07T09:42:11.244387+0000","last_active":"2023-10-07T09:42:12.245575+0000","last_peered":"2023-10-07T09:42:12.245575+0000","last_clean":"2023-10-07T09:42:12.245575+0000","last_became_active":"2023-10-07T09:42:11.244024+0000","last_became_peered":"2023-10-07T09:42:11.244024+0000","last_unstale":"2023-10-07T09:42:12.245575+0000","last_undegraded":"2023-10-07T09:42:12.245575+0000","last_fullsized":"2023-10-07T09:42:12.245575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:37.056219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.128392+0000","last_change":"2023-10-07T09:42:12.128392+0000","last_active":"2023-10-07T09:42:12.128392+0000","last_peered":"2023-10-07T09:42:12.128392+0000","last_clean":"2023-10-07T09:42:12.128392+0000","last_became_active":"2023-10-07T09:42:12.128121+0000","last_became_peered":"2023-10-07T09:42:12.128121+0000","last_unstale":"2023-10-07T09:42:12.128392+0000","last_undegraded":"2023-10-07T09:42:12.128392+0000","last_fullsized":"2023-10-07T09:42:12.128392+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:07.515132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.112554+0000","last_change":"2023-10-07T09:42:11.238834+0000","last_active":"2023-10-07T09:42:12.112554+0000","last_peered":"2023-10-07T09:42:12.112554+0000","last_clean":"2023-10-07T09:42:12.112554+0000","last_became_active":"2023-10-07T09:42:11.238558+0000","last_became_peered":"2023-10-07T09:42:11.238558+0000","last_unstale":"2023-10-07T09:42:12.112554+0000","last_undegraded":"2023-10-07T09:42:12.112554+0000","last_fullsized":"2023-10-07T09:42:12.112554+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:02.060432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.125116+0000","last_change":"2023-10-07T09:42:12.125116+0000","last_active":"2023-10-07T09:42:12.125116+0000","last_peered":"2023-10-07T09:42:12.125116+0000","last_clean":"2023-10-07T09:42:12.125116+0000","last_became_active":"2023-10-07T09:42:12.121195+0000","last_became_peered":"2023-10-07T09:42:12.121195+0000","last_unstale":"2023-10-07T09:42:12.125116+0000","last_undegraded":"2023-10-07T09:42:12.125116+0000","last_fullsized":"2023-10-07T09:42:12.125116+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:28.035281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.110744+0000","last_change":"2023-10-07T09:42:11.237502+0000","last_active":"2023-10-07T09:42:12.110744+0000","last_peered":"2023-10-07T09:42:12.110744+0000","last_clean":"2023-10-07T09:42:12.110744+0000","last_became_active":"2023-10-07T09:42:11.237223+0000","last_became_peered":"2023-10-07T09:42:11.237223+0000","last_unstale":"2023-10-07T09:42:12.110744+0000","last_undegraded":"2023-10-07T09:42:12.110744+0000","last_fullsized":"2023-10-07T09:42:12.110744+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:13:28.573701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.130988+0000","last_change":"2023-10-07T09:42:12.130988+0000","last_active":"2023-10-07T09:42:12.130988+0000","last_peered":"2023-10-07T09:42:12.130988+0000","last_clean":"2023-10-07T09:42:12.130988+0000","last_became_active":"2023-10-07T09:42:12.130643+0000","last_became_peered":"2023-10-07T09:42:12.130643+0000","last_unstale":"2023-10-07T09:42:12.130988+0000","last_undegraded":"2023-10-07T09:42:12.130988+0000","last_fullsized":"2023-10-07T09:42:12.130988+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:13:29.579840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.242663+0000","last_change":"2023-10-07T09:42:11.233655+0000","last_active":"2023-10-07T09:42:11.242663+0000","last_peered":"2023-10-07T09:42:11.242663+0000","last_clean":"2023-10-07T09:42:11.242663+0000","last_became_active":"2023-10-07T09:42:11.233406+0000","last_became_peered":"2023-10-07T09:42:11.233406+0000","last_unstale":"2023-10-07T09:42:11.242663+0000","last_undegraded":"2023-10-07T09:42:11.242663+0000","last_fullsized":"2023-10-07T09:42:11.242663+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:56.926207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129703+0000","last_change":"2023-10-07T09:42:12.129703+0000","last_active":"2023-10-07T09:42:12.129703+0000","last_peered":"2023-10-07T09:42:12.129703+0000","last_clean":"2023-10-07T09:42:12.129703+0000","last_became_active":"2023-10-07T09:42:12.129452+0000","last_became_peered":"2023-10-07T09:42:12.129452+0000","last_unstale":"2023-10-07T09:42:12.129703+0000","last_undegraded":"2023-10-07T09:42:12.129703+0000","last_fullsized":"2023-10-07T09:42:12.129703+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:15.488935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.238439+0000","last_change":"2023-10-07T09:42:11.236126+0000","last_active":"2023-10-07T09:42:11.238439+0000","last_peered":"2023-10-07T09:42:11.238439+0000","last_clean":"2023-10-07T09:42:11.238439+0000","last_became_active":"2023-10-07T09:42:11.235828+0000","last_became_peered":"2023-10-07T09:42:11.235828+0000","last_unstale":"2023-10-07T09:42:11.238439+0000","last_undegraded":"2023-10-07T09:42:11.238439+0000","last_fullsized":"2023-10-07T09:42:11.238439+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:39:45.612026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127111+0000","last_change":"2023-10-07T09:42:12.127111+0000","last_active":"2023-10-07T09:42:12.127111+0000","last_peered":"2023-10-07T09:42:12.127111+0000","last_clean":"2023-10-07T09:42:12.127111+0000","last_became_active":"2023-10-07T09:42:12.126525+0000","last_became_peered":"2023-10-07T09:42:12.126525+0000","last_unstale":"2023-10-07T09:42:12.127111+0000","last_undegraded":"2023-10-07T09:42:12.127111+0000","last_fullsized":"2023-10-07T09:42:12.127111+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:14:03.442618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.240274+0000","last_change":"2023-10-07T09:42:11.237252+0000","last_active":"2023-10-07T09:42:11.240274+0000","last_peered":"2023-10-07T09:42:11.240274+0000","last_clean":"2023-10-07T09:42:11.240274+0000","last_became_active":"2023-10-07T09:42:11.236946+0000","last_became_peered":"2023-10-07T09:42:11.236946+0000","last_unstale":"2023-10-07T09:42:11.240274+0000","last_undegraded":"2023-10-07T09:42:11.240274+0000","last_fullsized":"2023-10-07T09:42:11.240274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:21.050337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123936+0000","last_change":"2023-10-07T09:42:12.123936+0000","last_active":"2023-10-07T09:42:12.123936+0000","last_peered":"2023-10-07T09:42:12.123936+0000","last_clean":"2023-10-07T09:42:12.123936+0000","last_became_active":"2023-10-07T09:42:12.123615+0000","last_became_peered":"2023-10-07T09:42:12.123615+0000","last_unstale":"2023-10-07T09:42:12.123936+0000","last_undegraded":"2023-10-07T09:42:12.123936+0000","last_fullsized":"2023-10-07T09:42:12.123936+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:40:35.941030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.239756+0000","last_change":"2023-10-07T09:42:11.238098+0000","last_active":"2023-10-07T09:42:11.239756+0000","last_peered":"2023-10-07T09:42:11.239756+0000","last_clean":"2023-10-07T09:42:11.239756+0000","last_became_active":"2023-10-07T09:42:11.237800+0000","last_became_peered":"2023-10-07T09:42:11.237800+0000","last_unstale":"2023-10-07T09:42:11.239756+0000","last_undegraded":"2023-10-07T09:42:11.239756+0000","last_fullsized":"2023-10-07T09:42:11.239756+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:06:46.247772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.122886+0000","last_change":"2023-10-07T09:42:12.122886+0000","last_active":"2023-10-07T09:42:12.122886+0000","last_peered":"2023-10-07T09:42:12.122886+0000","last_clean":"2023-10-07T09:42:12.122886+0000","last_became_active":"2023-10-07T09:42:12.122128+0000","last_became_peered":"2023-10-07T09:42:12.122128+0000","last_unstale":"2023-10-07T09:42:12.122886+0000","last_undegraded":"2023-10-07T09:42:12.122886+0000","last_fullsized":"2023-10-07T09:42:12.122886+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:27:30.802150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"60'3","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.248088+0000","last_change":"2023-10-07T09:42:11.242023+0000","last_active":"2023-10-07T09:42:12.248088+0000","last_peered":"2023-10-07T09:42:12.248088+0000","last_clean":"2023-10-07T09:42:12.248088+0000","last_became_active":"2023-10-07T09:42:11.241674+0000","last_became_peered":"2023-10-07T09:42:11.241674+0000","last_unstale":"2023-10-07T09:42:12.248088+0000","last_undegraded":"2023-10-07T09:42:12.248088+0000","last_fullsized":"2023-10-07T09:42:12.248088+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:31:10.175821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126290+0000","last_change":"2023-10-07T09:42:12.126290+0000","last_active":"2023-10-07T09:42:12.126290+0000","last_peered":"2023-10-07T09:42:12.126290+0000","last_clean":"2023-10-07T09:42:12.126290+0000","last_became_active":"2023-10-07T09:42:12.125998+0000","last_became_peered":"2023-10-07T09:42:12.125998+0000","last_unstale":"2023-10-07T09:42:12.126290+0000","last_undegraded":"2023-10-07T09:42:12.126290+0000","last_fullsized":"2023-10-07T09:42:12.126290+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:58:11.376756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"60'6","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.246534+0000","last_change":"2023-10-07T09:42:11.241562+0000","last_active":"2023-10-07T09:42:12.246534+0000","last_peered":"2023-10-07T09:42:12.246534+0000","last_clean":"2023-10-07T09:42:12.246534+0000","last_became_active":"2023-10-07T09:42:11.241297+0000","last_became_peered":"2023-10-07T09:42:11.241297+0000","last_unstale":"2023-10-07T09:42:12.246534+0000","last_undegraded":"2023-10-07T09:42:12.246534+0000","last_fullsized":"2023-10-07T09:42:12.246534+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:22:43.173960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131813+0000","last_change":"2023-10-07T09:42:12.131813+0000","last_active":"2023-10-07T09:42:12.131813+0000","last_peered":"2023-10-07T09:42:12.131813+0000","last_clean":"2023-10-07T09:42:12.131813+0000","last_became_active":"2023-10-07T09:42:12.131547+0000","last_became_peered":"2023-10-07T09:42:12.131547+0000","last_unstale":"2023-10-07T09:42:12.131813+0000","last_undegraded":"2023-10-07T09:42:12.131813+0000","last_fullsized":"2023-10-07T09:42:12.131813+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:32.328778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.246869+0000","last_change":"2023-10-07T09:42:11.229919+0000","last_active":"2023-10-07T09:42:12.246869+0000","last_peered":"2023-10-07T09:42:12.246869+0000","last_clean":"2023-10-07T09:42:12.246869+0000","last_became_active":"2023-10-07T09:42:11.229655+0000","last_became_peered":"2023-10-07T09:42:11.229655+0000","last_unstale":"2023-10-07T09:42:12.246869+0000","last_undegraded":"2023-10-07T09:42:12.246869+0000","last_fullsized":"2023-10-07T09:42:12.246869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:29.091141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127547+0000","last_change":"2023-10-07T09:42:12.127547+0000","last_active":"2023-10-07T09:42:12.127547+0000","last_peered":"2023-10-07T09:42:12.127547+0000","last_clean":"2023-10-07T09:42:12.127547+0000","last_became_active":"2023-10-07T09:42:12.125853+0000","last_became_peered":"2023-10-07T09:42:12.125853+0000","last_unstale":"2023-10-07T09:42:12.127547+0000","last_undegraded":"2023-10-07T09:42:12.127547+0000","last_fullsized":"2023-10-07T09:42:12.127547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:07.515132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.242466+0000","last_change":"2023-10-07T09:42:11.240995+0000","last_active":"2023-10-07T09:42:11.242466+0000","last_peered":"2023-10-07T09:42:11.242466+0000","last_clean":"2023-10-07T09:42:11.242466+0000","last_became_active":"2023-10-07T09:42:11.240746+0000","last_became_peered":"2023-10-07T09:42:11.240746+0000","last_unstale":"2023-10-07T09:42:11.242466+0000","last_undegraded":"2023-10-07T09:42:11.242466+0000","last_fullsized":"2023-10-07T09:42:11.242466+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:57.052047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121867+0000","last_change":"2023-10-07T09:42:12.121867+0000","last_active":"2023-10-07T09:42:12.121867+0000","last_peered":"2023-10-07T09:42:12.121867+0000","last_clean":"2023-10-07T09:42:12.121867+0000","last_became_active":"2023-10-07T09:42:12.121607+0000","last_became_peered":"2023-10-07T09:42:12.121607+0000","last_unstale":"2023-10-07T09:42:12.121867+0000","last_undegraded":"2023-10-07T09:42:12.121867+0000","last_fullsized":"2023-10-07T09:42:12.121867+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:34:26.900282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.231572+0000","last_change":"2023-10-07T09:42:11.229317+0000","last_active":"2023-10-07T09:42:11.231572+0000","last_peered":"2023-10-07T09:42:11.231572+0000","last_clean":"2023-10-07T09:42:11.231572+0000","last_became_active":"2023-10-07T09:42:11.228770+0000","last_became_peered":"2023-10-07T09:42:11.228770+0000","last_unstale":"2023-10-07T09:42:11.231572+0000","last_undegraded":"2023-10-07T09:42:11.231572+0000","last_fullsized":"2023-10-07T09:42:11.231572+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:32.122986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129281+0000","last_change":"2023-10-07T09:42:12.129281+0000","last_active":"2023-10-07T09:42:12.129281+0000","last_peered":"2023-10-07T09:42:12.129281+0000","last_clean":"2023-10-07T09:42:12.129281+0000","last_became_active":"2023-10-07T09:42:12.128785+0000","last_became_peered":"2023-10-07T09:42:12.128785+0000","last_unstale":"2023-10-07T09:42:12.129281+0000","last_undegraded":"2023-10-07T09:42:12.129281+0000","last_fullsized":"2023-10-07T09:42:12.129281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:05:58.578759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"60'8","reported_seq":17,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.248913+0000","last_change":"2023-10-07T09:42:11.244077+0000","last_active":"2023-10-07T09:42:12.248913+0000","last_peered":"2023-10-07T09:42:12.248913+0000","last_clean":"2023-10-07T09:42:12.248913+0000","last_became_active":"2023-10-07T09:42:11.243779+0000","last_became_peered":"2023-10-07T09:42:11.243779+0000","last_unstale":"2023-10-07T09:42:12.248913+0000","last_undegraded":"2023-10-07T09:42:12.248913+0000","last_fullsized":"2023-10-07T09:42:12.248913+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:37:05.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123775+0000","last_change":"2023-10-07T09:42:12.123775+0000","last_active":"2023-10-07T09:42:12.123775+0000","last_peered":"2023-10-07T09:42:12.123775+0000","last_clean":"2023-10-07T09:42:12.123775+0000","last_became_active":"2023-10-07T09:42:12.123503+0000","last_became_peered":"2023-10-07T09:42:12.123503+0000","last_unstale":"2023-10-07T09:42:12.123775+0000","last_undegraded":"2023-10-07T09:42:12.123775+0000","last_fullsized":"2023-10-07T09:42:12.123775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:26:27.535362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.116761+0000","last_change":"2023-10-07T09:42:11.237191+0000","last_active":"2023-10-07T09:42:12.116761+0000","last_peered":"2023-10-07T09:42:12.116761+0000","last_clean":"2023-10-07T09:42:12.116761+0000","last_became_active":"2023-10-07T09:42:11.236876+0000","last_became_peered":"2023-10-07T09:42:11.236876+0000","last_unstale":"2023-10-07T09:42:12.116761+0000","last_undegraded":"2023-10-07T09:42:12.116761+0000","last_fullsized":"2023-10-07T09:42:12.116761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:35.007910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.117385+0000","last_change":"2023-10-07T09:42:12.117385+0000","last_active":"2023-10-07T09:42:12.117385+0000","last_peered":"2023-10-07T09:42:12.117385+0000","last_clean":"2023-10-07T09:42:12.117385+0000","last_became_active":"2023-10-07T09:42:12.117061+0000","last_became_peered":"2023-10-07T09:42:12.117061+0000","last_unstale":"2023-10-07T09:42:12.117385+0000","last_undegraded":"2023-10-07T09:42:12.117385+0000","last_fullsized":"2023-10-07T09:42:12.117385+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:44.145951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.246268+0000","last_change":"2023-10-07T09:42:11.234033+0000","last_active":"2023-10-07T09:42:12.246268+0000","last_peered":"2023-10-07T09:42:12.246268+0000","last_clean":"2023-10-07T09:42:12.246268+0000","last_became_active":"2023-10-07T09:42:11.233737+0000","last_became_peered":"2023-10-07T09:42:11.233737+0000","last_unstale":"2023-10-07T09:42:12.246268+0000","last_undegraded":"2023-10-07T09:42:12.246268+0000","last_fullsized":"2023-10-07T09:42:12.246268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:48.560303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132849+0000","last_change":"2023-10-07T09:42:12.132849+0000","last_active":"2023-10-07T09:42:12.132849+0000","last_peered":"2023-10-07T09:42:12.132849+0000","last_clean":"2023-10-07T09:42:12.132849+0000","last_became_active":"2023-10-07T09:42:12.132176+0000","last_became_peered":"2023-10-07T09:42:12.132176+0000","last_unstale":"2023-10-07T09:42:12.132849+0000","last_undegraded":"2023-10-07T09:42:12.132849+0000","last_fullsized":"2023-10-07T09:42:12.132849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:08.388551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.111524+0000","last_change":"2023-10-07T09:42:11.241129+0000","last_active":"2023-10-07T09:42:12.111524+0000","last_peered":"2023-10-07T09:42:12.111524+0000","last_clean":"2023-10-07T09:42:12.111524+0000","last_became_active":"2023-10-07T09:42:11.240788+0000","last_became_peered":"2023-10-07T09:42:11.240788+0000","last_unstale":"2023-10-07T09:42:12.111524+0000","last_undegraded":"2023-10-07T09:42:12.111524+0000","last_fullsized":"2023-10-07T09:42:12.111524+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:27.029142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131349+0000","last_change":"2023-10-07T09:42:12.131349+0000","last_active":"2023-10-07T09:42:12.131349+0000","last_peered":"2023-10-07T09:42:12.131349+0000","last_clean":"2023-10-07T09:42:12.131349+0000","last_became_active":"2023-10-07T09:42:12.131002+0000","last_became_peered":"2023-10-07T09:42:12.131002+0000","last_unstale":"2023-10-07T09:42:12.131349+0000","last_undegraded":"2023-10-07T09:42:12.131349+0000","last_fullsized":"2023-10-07T09:42:12.131349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:08.038437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.234450+0000","last_change":"2023-10-07T09:42:11.231480+0000","last_active":"2023-10-07T09:42:11.234450+0000","last_peered":"2023-10-07T09:42:11.234450+0000","last_clean":"2023-10-07T09:42:11.234450+0000","last_became_active":"2023-10-07T09:42:11.231208+0000","last_became_peered":"2023-10-07T09:42:11.231208+0000","last_unstale":"2023-10-07T09:42:11.234450+0000","last_undegraded":"2023-10-07T09:42:11.234450+0000","last_fullsized":"2023-10-07T09:42:11.234450+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:04.068728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121547+0000","last_change":"2023-10-07T09:42:12.121547+0000","last_active":"2023-10-07T09:42:12.121547+0000","last_peered":"2023-10-07T09:42:12.121547+0000","last_clean":"2023-10-07T09:42:12.121547+0000","last_became_active":"2023-10-07T09:42:12.121264+0000","last_became_peered":"2023-10-07T09:42:12.121264+0000","last_unstale":"2023-10-07T09:42:12.121547+0000","last_undegraded":"2023-10-07T09:42:12.121547+0000","last_fullsized":"2023-10-07T09:42:12.121547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:14:22.949422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.244936+0000","last_change":"2023-10-07T09:42:11.238654+0000","last_active":"2023-10-07T09:42:12.244936+0000","last_peered":"2023-10-07T09:42:12.244936+0000","last_clean":"2023-10-07T09:42:12.244936+0000","last_became_active":"2023-10-07T09:42:11.238372+0000","last_became_peered":"2023-10-07T09:42:11.238372+0000","last_unstale":"2023-10-07T09:42:12.244936+0000","last_undegraded":"2023-10-07T09:42:12.244936+0000","last_fullsized":"2023-10-07T09:42:12.244936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:50.986059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121532+0000","last_change":"2023-10-07T09:42:12.121532+0000","last_active":"2023-10-07T09:42:12.121532+0000","last_peered":"2023-10-07T09:42:12.121532+0000","last_clean":"2023-10-07T09:42:12.121532+0000","last_became_active":"2023-10-07T09:42:12.121265+0000","last_became_peered":"2023-10-07T09:42:12.121265+0000","last_unstale":"2023-10-07T09:42:12.121532+0000","last_undegraded":"2023-10-07T09:42:12.121532+0000","last_fullsized":"2023-10-07T09:42:12.121532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:33.129125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243645+0000","last_change":"2023-10-07T09:42:11.241017+0000","last_active":"2023-10-07T09:42:11.243645+0000","last_peered":"2023-10-07T09:42:11.243645+0000","last_clean":"2023-10-07T09:42:11.243645+0000","last_became_active":"2023-10-07T09:42:11.240733+0000","last_became_peered":"2023-10-07T09:42:11.240733+0000","last_unstale":"2023-10-07T09:42:11.243645+0000","last_undegraded":"2023-10-07T09:42:11.243645+0000","last_fullsized":"2023-10-07T09:42:11.243645+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:56.858507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.125039+0000","last_change":"2023-10-07T09:42:12.125039+0000","last_active":"2023-10-07T09:42:12.125039+0000","last_peered":"2023-10-07T09:42:12.125039+0000","last_clean":"2023-10-07T09:42:12.125039+0000","last_became_active":"2023-10-07T09:42:12.124786+0000","last_became_peered":"2023-10-07T09:42:12.124786+0000","last_unstale":"2023-10-07T09:42:12.125039+0000","last_undegraded":"2023-10-07T09:42:12.125039+0000","last_fullsized":"2023-10-07T09:42:12.125039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:08.388551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.239876+0000","last_change":"2023-10-07T09:42:11.238082+0000","last_active":"2023-10-07T09:42:11.239876+0000","last_peered":"2023-10-07T09:42:11.239876+0000","last_clean":"2023-10-07T09:42:11.239876+0000","last_became_active":"2023-10-07T09:42:11.237722+0000","last_became_peered":"2023-10-07T09:42:11.237722+0000","last_unstale":"2023-10-07T09:42:11.239876+0000","last_undegraded":"2023-10-07T09:42:11.239876+0000","last_fullsized":"2023-10-07T09:42:11.239876+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:35.007910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.119775+0000","last_change":"2023-10-07T09:42:12.119775+0000","last_active":"2023-10-07T09:42:12.119775+0000","last_peered":"2023-10-07T09:42:12.119775+0000","last_clean":"2023-10-07T09:42:12.119775+0000","last_became_active":"2023-10-07T09:42:12.119428+0000","last_became_peered":"2023-10-07T09:42:12.119428+0000","last_unstale":"2023-10-07T09:42:12.119775+0000","last_undegraded":"2023-10-07T09:42:12.119775+0000","last_fullsized":"2023-10-07T09:42:12.119775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:30.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243981+0000","last_change":"2023-10-07T09:42:11.241886+0000","last_active":"2023-10-07T09:42:11.243981+0000","last_peered":"2023-10-07T09:42:11.243981+0000","last_clean":"2023-10-07T09:42:11.243981+0000","last_became_active":"2023-10-07T09:42:11.241351+0000","last_became_peered":"2023-10-07T09:42:11.241351+0000","last_unstale":"2023-10-07T09:42:11.243981+0000","last_undegraded":"2023-10-07T09:42:11.243981+0000","last_fullsized":"2023-10-07T09:42:11.243981+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:29.975640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.133020+0000","last_change":"2023-10-07T09:42:12.133020+0000","last_active":"2023-10-07T09:42:12.133020+0000","last_peered":"2023-10-07T09:42:12.133020+0000","last_clean":"2023-10-07T09:42:12.133020+0000","last_became_active":"2023-10-07T09:42:12.132725+0000","last_became_peered":"2023-10-07T09:42:12.132725+0000","last_unstale":"2023-10-07T09:42:12.133020+0000","last_undegraded":"2023-10-07T09:42:12.133020+0000","last_fullsized":"2023-10-07T09:42:12.133020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:27.390563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.238249+0000","last_change":"2023-10-07T09:42:11.236763+0000","last_active":"2023-10-07T09:42:11.238249+0000","last_peered":"2023-10-07T09:42:11.238249+0000","last_clean":"2023-10-07T09:42:11.238249+0000","last_became_active":"2023-10-07T09:42:11.236456+0000","last_became_peered":"2023-10-07T09:42:11.236456+0000","last_unstale":"2023-10-07T09:42:11.238249+0000","last_undegraded":"2023-10-07T09:42:11.238249+0000","last_fullsized":"2023-10-07T09:42:11.238249+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:17:01.213995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120292+0000","last_change":"2023-10-07T09:42:12.120292+0000","last_active":"2023-10-07T09:42:12.120292+0000","last_peered":"2023-10-07T09:42:12.120292+0000","last_clean":"2023-10-07T09:42:12.120292+0000","last_became_active":"2023-10-07T09:42:12.119970+0000","last_became_peered":"2023-10-07T09:42:12.119970+0000","last_unstale":"2023-10-07T09:42:12.120292+0000","last_undegraded":"2023-10-07T09:42:12.120292+0000","last_fullsized":"2023-10-07T09:42:12.120292+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:17:02.220133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243884+0000","last_change":"2023-10-07T09:42:11.241971+0000","last_active":"2023-10-07T09:42:11.243884+0000","last_peered":"2023-10-07T09:42:11.243884+0000","last_clean":"2023-10-07T09:42:11.243884+0000","last_became_active":"2023-10-07T09:42:11.241669+0000","last_became_peered":"2023-10-07T09:42:11.241669+0000","last_unstale":"2023-10-07T09:42:11.243884+0000","last_undegraded":"2023-10-07T09:42:11.243884+0000","last_fullsized":"2023-10-07T09:42:11.243884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:06.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123539+0000","last_change":"2023-10-07T09:42:12.123539+0000","last_active":"2023-10-07T09:42:12.123539+0000","last_peered":"2023-10-07T09:42:12.123539+0000","last_clean":"2023-10-07T09:42:12.123539+0000","last_became_active":"2023-10-07T09:42:12.122756+0000","last_became_peered":"2023-10-07T09:42:12.122756+0000","last_unstale":"2023-10-07T09:42:12.123539+0000","last_undegraded":"2023-10-07T09:42:12.123539+0000","last_fullsized":"2023-10-07T09:42:12.123539+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:14:48.484141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.245026+0000","last_change":"2023-10-07T09:42:11.244235+0000","last_active":"2023-10-07T09:42:11.245026+0000","last_peered":"2023-10-07T09:42:11.245026+0000","last_clean":"2023-10-07T09:42:11.245026+0000","last_became_active":"2023-10-07T09:42:11.243833+0000","last_became_peered":"2023-10-07T09:42:11.243833+0000","last_unstale":"2023-10-07T09:42:11.245026+0000","last_undegraded":"2023-10-07T09:42:11.245026+0000","last_fullsized":"2023-10-07T09:42:11.245026+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:21.050337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.119487+0000","last_change":"2023-10-07T09:42:12.119487+0000","last_active":"2023-10-07T09:42:12.119487+0000","last_peered":"2023-10-07T09:42:12.119487+0000","last_clean":"2023-10-07T09:42:12.119487+0000","last_became_active":"2023-10-07T09:42:12.119195+0000","last_became_peered":"2023-10-07T09:42:12.119195+0000","last_unstale":"2023-10-07T09:42:12.119487+0000","last_undegraded":"2023-10-07T09:42:12.119487+0000","last_fullsized":"2023-10-07T09:42:12.119487+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:05.782312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243948+0000","last_change":"2023-10-07T09:42:11.241981+0000","last_active":"2023-10-07T09:42:11.243948+0000","last_peered":"2023-10-07T09:42:11.243948+0000","last_clean":"2023-10-07T09:42:11.243948+0000","last_became_active":"2023-10-07T09:42:11.241639+0000","last_became_peered":"2023-10-07T09:42:11.241639+0000","last_unstale":"2023-10-07T09:42:11.243948+0000","last_undegraded":"2023-10-07T09:42:11.243948+0000","last_fullsized":"2023-10-07T09:42:11.243948+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:14.416930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123049+0000","last_change":"2023-10-07T09:42:12.123049+0000","last_active":"2023-10-07T09:42:12.123049+0000","last_peered":"2023-10-07T09:42:12.123049+0000","last_clean":"2023-10-07T09:42:12.123049+0000","last_became_active":"2023-10-07T09:42:12.122731+0000","last_became_peered":"2023-10-07T09:42:12.122731+0000","last_unstale":"2023-10-07T09:42:12.123049+0000","last_undegraded":"2023-10-07T09:42:12.123049+0000","last_fullsized":"2023-10-07T09:42:12.123049+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:31:39.819015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.240040+0000","last_change":"2023-10-07T09:42:11.234988+0000","last_active":"2023-10-07T09:42:11.240040+0000","last_peered":"2023-10-07T09:42:11.240040+0000","last_clean":"2023-10-07T09:42:11.240040+0000","last_became_active":"2023-10-07T09:42:11.228153+0000","last_became_peered":"2023-10-07T09:42:11.228153+0000","last_unstale":"2023-10-07T09:42:11.240040+0000","last_undegraded":"2023-10-07T09:42:11.240040+0000","last_fullsized":"2023-10-07T09:42:11.240040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:43.139812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120650+0000","last_change":"2023-10-07T09:42:12.120650+0000","last_active":"2023-10-07T09:42:12.120650+0000","last_peered":"2023-10-07T09:42:12.120650+0000","last_clean":"2023-10-07T09:42:12.120650+0000","last_became_active":"2023-10-07T09:42:12.120358+0000","last_became_peered":"2023-10-07T09:42:12.120358+0000","last_unstale":"2023-10-07T09:42:12.120650+0000","last_undegraded":"2023-10-07T09:42:12.120650+0000","last_fullsized":"2023-10-07T09:42:12.120650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:14:22.949422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.233933+0000","last_change":"2023-10-07T09:42:11.232753+0000","last_active":"2023-10-07T09:42:11.233933+0000","last_peered":"2023-10-07T09:42:11.233933+0000","last_clean":"2023-10-07T09:42:11.233933+0000","last_became_active":"2023-10-07T09:42:11.232482+0000","last_became_peered":"2023-10-07T09:42:11.232482+0000","last_unstale":"2023-10-07T09:42:11.233933+0000","last_undegraded":"2023-10-07T09:42:11.233933+0000","last_fullsized":"2023-10-07T09:42:11.233933+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:48.560303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.118362+0000","last_change":"2023-10-07T09:42:12.118362+0000","last_active":"2023-10-07T09:42:12.118362+0000","last_peered":"2023-10-07T09:42:12.118362+0000","last_clean":"2023-10-07T09:42:12.118362+0000","last_became_active":"2023-10-07T09:42:12.118076+0000","last_became_peered":"2023-10-07T09:42:12.118076+0000","last_unstale":"2023-10-07T09:42:12.118362+0000","last_undegraded":"2023-10-07T09:42:12.118362+0000","last_fullsized":"2023-10-07T09:42:12.118362+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:28.035281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.235021+0000","last_change":"2023-10-07T09:42:11.229814+0000","last_active":"2023-10-07T09:42:11.235021+0000","last_peered":"2023-10-07T09:42:11.235021+0000","last_clean":"2023-10-07T09:42:11.235021+0000","last_became_active":"2023-10-07T09:42:11.229537+0000","last_became_peered":"2023-10-07T09:42:11.229537+0000","last_unstale":"2023-10-07T09:42:11.235021+0000","last_undegraded":"2023-10-07T09:42:11.235021+0000","last_fullsized":"2023-10-07T09:42:11.235021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:00:19.787456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127165+0000","last_change":"2023-10-07T09:42:12.127165+0000","last_active":"2023-10-07T09:42:12.127165+0000","last_peered":"2023-10-07T09:42:12.127165+0000","last_clean":"2023-10-07T09:42:12.127165+0000","last_became_active":"2023-10-07T09:42:12.126859+0000","last_became_peered":"2023-10-07T09:42:12.126859+0000","last_unstale":"2023-10-07T09:42:12.127165+0000","last_undegraded":"2023-10-07T09:42:12.127165+0000","last_fullsized":"2023-10-07T09:42:12.127165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:57.932346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.244797+0000","last_change":"2023-10-07T09:42:11.237671+0000","last_active":"2023-10-07T09:42:12.244797+0000","last_peered":"2023-10-07T09:42:12.244797+0000","last_clean":"2023-10-07T09:42:12.244797+0000","last_became_active":"2023-10-07T09:42:11.237419+0000","last_became_peered":"2023-10-07T09:42:11.237419+0000","last_unstale":"2023-10-07T09:42:12.244797+0000","last_undegraded":"2023-10-07T09:42:12.244797+0000","last_fullsized":"2023-10-07T09:42:12.244797+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:06.508994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"50'139","reported_seq":273,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.233842+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:42:11.233842+0000","last_peered":"2023-10-07T09:42:11.233842+0000","last_clean":"2023-10-07T09:42:11.233842+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:42:11.233842+0000","last_undegraded":"2023-10-07T09:42:11.233842+0000","last_fullsized":"2023-10-07T09:42:11.233842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302640,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069200,"statfs":{"total":96636764160,"available":96326860800,"internally_reserved":0,"allocated":1286144,"data_stored":905976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.70599999999999996},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.32300000000000001},"max":{"1min":0.876,"5min":0.998,"15min":1.026},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.73299999999999998,"15min":0.69599999999999995},"min":{"1min":0.56000000000000005,"5min":0.374,"15min":0.374},"max":{"1min":0.93899999999999995,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.88}]},{"osd":1,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.51600000000000001,"5min":0.45200000000000001,"15min":0.34599999999999997},"max":{"1min":0.875,"5min":1.014,"15min":1.157},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.72399999999999998},"min":{"1min":0.49099999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.998,"5min":1.002,"15min":1.002},"last":0.75700000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76300000000000001,"15min":0.73799999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40699999999999997},"max":{"1min":0.93500000000000005,"5min":0.97699999999999998,"15min":1.0289999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.74099999999999999,"15min":0.72499999999999998},"min":{"1min":0.58199999999999996,"5min":0.52100000000000002,"15min":0.439},"max":{"1min":0.86299999999999999,"5min":1.071,"15min":1.071},"last":0.83099999999999996}]},{"osd":3,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.73099999999999998},"min":{"1min":0.55200000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91600000000000004,"5min":0.96899999999999997,"15min":1.04},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.753,"15min":0.76100000000000001},"min":{"1min":0.57299999999999995,"5min":0.53000000000000003,"15min":0.371},"max":{"1min":0.90200000000000002,"5min":1.006,"15min":1.3440000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59199999999999997,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"max":{"1min":0.85499999999999998,"5min":0.88700000000000001,"15min":1.992},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.40200000000000002},"max":{"1min":0.84199999999999997,"5min":0.92900000000000005,"15min":1.6240000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63600000000000001,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.0900000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.70799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90600000000000003,"5min":0.94899999999999995,"15min":1.1399999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.56299999999999994,"5min":0.499,"15min":0.46400000000000002},"max":{"1min":0.89300000000000002,"5min":0.97999999999999998,"15min":1.0289999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.71799999999999997,"15min":0.73799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.96899999999999997,"15min":1.107},"last":0.67800000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509599,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.71899999999999997,"15min":0.76100000000000001},"min":{"1min":0.51200000000000001,"5min":0.47499999999999998,"15min":0.39300000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":2.984},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.51700000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85899999999999999,"5min":0.94599999999999995,"15min":1.6000000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72499999999999998,"15min":0.73599999999999999},"min":{"1min":0.52400000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93400000000000005,"5min":1.0149999999999999,"15min":2.4529999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73499999999999999,"15min":0.73899999999999999},"min":{"1min":0.56899999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.99199999999999999,"15min":1.754},"last":0.65800000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73299999999999998,"15min":0.81000000000000005},"min":{"1min":0.59499999999999997,"5min":0.45700000000000002,"15min":0.44400000000000001},"max":{"1min":0.88,"5min":1.002,"15min":7.867},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76800000000000002,"15min":0.84399999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92900000000000005,"5min":0.99199999999999999,"15min":7.7629999999999999},"last":0.68899999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76800000000000002,"15min":0.85599999999999998},"min":{"1min":0.54600000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.86799999999999999,"5min":0.999,"15min":8.0399999999999991},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.752,"15min":0.79600000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.435},"max":{"1min":0.93500000000000005,"5min":0.98099999999999998,"15min":2.7949999999999999},"last":0.78900000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61699999999999999,"15min":0.63500000000000001},"min":{"1min":0.46100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80400000000000005,"5min":1.01,"15min":1.296},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.67600000000000005},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.91100000000000003,"5min":0.92500000000000004,"15min":1.105},"last":0.76700000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91200000000000003,"5min":1.0509999999999999,"15min":1.4119999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.52200000000000002,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":1.0449999999999999,"15min":1.1779999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67700000000000005,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97499999999999998,"5min":1.0149999999999999,"15min":1.5049999999999999},"last":0.751},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":0.77900000000000003},"min":{"1min":0.59499999999999997,"5min":0.51700000000000002,"15min":0.48899999999999999},"max":{"1min":0.95799999999999996,"5min":1.002,"15min":1.3220000000000001},"last":0.80500000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302076,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069764,"statfs":{"total":96636764160,"available":96327438336,"internally_reserved":0,"allocated":708608,"data_stored":322789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65600000000000003,"5min":0.69699999999999995,"15min":1.585},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.29299999999999998},"max":{"1min":0.61899999999999999,"5min":0.84199999999999997,"15min":1.6080000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.60899999999999999,"5min":0.79600000000000004,"15min":1.468},"last":0.5},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77800000000000002,"5min":0.82499999999999996,"15min":1.5389999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59399999999999997,"15min":0.61299999999999999},"min":{"1min":0.437,"5min":0.377,"15min":0.34699999999999998},"max":{"1min":0.67200000000000004,"5min":0.78700000000000003,"15min":1.663},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":0.81399999999999995,"15min":1.1140000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.78000000000000003,"15min":0.78300000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.89400000000000002,"5min":1.2629999999999999,"15min":1.4259999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.54100000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":1.1120000000000001,"15min":1.766},"last":0.67200000000000004}]},{"osd":5,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.82699999999999996,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.54400000000000004},"max":{"1min":0.98199999999999998,"5min":1.177,"15min":1.3320000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.78900000000000003,"15min":0.79400000000000004},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.54400000000000004},"max":{"1min":0.92700000000000005,"5min":1.127,"15min":1.5149999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":0.83399999999999996},"min":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.57099999999999995},"max":{"1min":0.92000000000000004,"5min":1.091,"15min":1.627},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.83099999999999996,"15min":0.83299999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94799999999999995,"5min":1.1080000000000001,"15min":1.3700000000000001},"last":0.73199999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.83899999999999997,"15min":0.85199999999999998},"min":{"1min":0.622,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.90600000000000003,"5min":1.2050000000000001,"15min":1.706},"last":0.878},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.86599999999999999,"15min":0.85799999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.58399999999999996},"max":{"1min":0.98399999999999999,"5min":1.103,"15min":1.554},"last":0.65500000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542301,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302652,"kb_used_data":1268,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069188,"statfs":{"total":96636764160,"available":96326848512,"internally_reserved":0,"allocated":1298432,"data_stored":913087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.313,"15min":0.30299999999999999},"max":{"1min":0.63,"5min":0.74399999999999999,"15min":2.726},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":2.552},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.32800000000000001},"max":{"1min":0.66700000000000004,"5min":1.101,"15min":2.4910000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.50700000000000001,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":3.629},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"max":{"1min":0.69099999999999995,"5min":1.0009999999999999,"15min":2.6909999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.62,"15min":0.63800000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40200000000000002},"max":{"1min":0.80900000000000005,"5min":0.89500000000000002,"15min":2.5209999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77100000000000002,"15min":0.79100000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.45900000000000002},"max":{"1min":1.1850000000000001,"5min":1.232,"15min":2.637},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.79400000000000004,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"max":{"1min":0.95999999999999996,"5min":1.117,"15min":2.4180000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.79800000000000004,"15min":0.84899999999999998},"min":{"1min":0.52400000000000002,"5min":0.51300000000000001,"15min":0.501},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":3.8199999999999998},"last":0.874},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.83299999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55700000000000005},"max":{"1min":1.0920000000000001,"5min":1.194,"15min":2.6000000000000001},"last":0.69099999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.66500000000000004,"5min":0.60699999999999998,"15min":0.60499999999999998},"max":{"1min":1.129,"5min":1.129,"15min":2.3740000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.83099999999999996,"15min":0.877},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53600000000000003},"max":{"1min":1.103,"5min":1.103,"15min":3.855},"last":0.82499999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.84899999999999998,"15min":0.85899999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.56200000000000006},"max":{"1min":1.167,"5min":1.167,"15min":2.4550000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.84299999999999997,"15min":0.89800000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0609999999999999,"5min":1.177,"15min":3.778},"last":0.71699999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306156,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065684,"statfs":{"total":96636764160,"available":96323260416,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47399999999999998,"15min":0.45600000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.81000000000000005,"5min":1.258,"15min":1.258},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53600000000000003,"15min":0.497},"min":{"1min":0.314,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.72599999999999998,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.55200000000000005},"min":{"1min":0.39300000000000002,"5min":0.38100000000000001,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.45100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81799999999999995,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.94599999999999995,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.432},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61699999999999999,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81399999999999995,"5min":1.052,"15min":1.052},"last":0.38800000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:41:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.76000000000000001},"min":{"1min":0.48199999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0369999999999999,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.82199999999999995,"15min":0.79000000000000004},"min":{"1min":0.54400000000000004,"5min":0.49399999999999999,"15min":0.42099999999999999},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85499999999999998,"15min":0.81100000000000005},"min":{"1min":0.622,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80500000000000005,"15min":0.78900000000000003},"min":{"1min":0.56499999999999995,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.93899999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:41:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.86099999999999999,"15min":0.82399999999999995},"min":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.82899999999999996},"min":{"1min":0.60299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.306,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.872}]},{"osd":7,"last update":"Sat Oct 7 09:41:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.83499999999999996},"min":{"1min":0.60699999999999998,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85999999999999999,"15min":0.82299999999999995},"min":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54200000000000004},"max":{"1min":1.3520000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.75700000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306180,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065660,"statfs":{"total":96636764160,"available":96323235840,"internally_reserved":0,"allocated":716800,"data_stored":323329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45900000000000002,"15min":0.47599999999999998},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.25600000000000001},"max":{"1min":0.61399999999999999,"5min":0.77200000000000002,"15min":3.3980000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.64200000000000002,"5min":0.73199999999999998,"15min":3.274},"last":0.379}]},{"osd":1,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.53000000000000003},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.61099999999999999,"5min":0.67600000000000005,"15min":3.1299999999999999},"last":0.441},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.47299999999999998,"5min":0.42499999999999999,"15min":0.38200000000000001},"max":{"1min":0.70599999999999996,"5min":0.84099999999999997,"15min":3.4500000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":0.68600000000000005,"5min":0.86599999999999999,"15min":3.2309999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.79800000000000004,"15min":3.0910000000000002},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56699999999999995,"5min":0.51600000000000001,"15min":0.44900000000000001},"max":{"1min":1.0389999999999999,"5min":1.145,"15min":1.518},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78200000000000003,"15min":0.78700000000000003},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.48099999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.3799999999999999},"last":0.90100000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.59099999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.3240000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.50600000000000001},"max":{"1min":0.95599999999999996,"5min":1.1899999999999999,"15min":1.5189999999999999},"last":0.87}]},{"osd":6,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80500000000000005,"15min":0.80600000000000005},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0049999999999999,"5min":1.125,"15min":1.413},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.81899999999999995,"15min":0.82299999999999995},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.51000000000000001},"max":{"1min":1.1459999999999999,"5min":1.173,"15min":1.5840000000000001},"last":0.93500000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.82199999999999995,"15min":0.81699999999999995},"min":{"1min":0.59899999999999998,"5min":0.57299999999999995,"15min":0.54700000000000004},"max":{"1min":1.0980000000000001,"5min":1.163,"15min":1.45},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83399999999999996,"15min":0.83699999999999997},"min":{"1min":0.63100000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0660000000000001,"5min":1.1439999999999999,"15min":1.5940000000000001},"last":0.80000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509599,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306156,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065684,"statfs":{"total":96636764160,"available":96323260416,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71799999999999997,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.39600000000000002,"15min":0.30399999999999999},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.72699999999999998,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":2.3420000000000001},"last":0.80300000000000005}]},{"osd":1,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.745,"15min":0.70199999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.379},"max":{"1min":0.96599999999999997,"5min":1.04,"15min":1.054},"last":0.62},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.752,"15min":0.71599999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99099999999999999,"5min":1.046,"15min":1.2749999999999999},"last":0.748}]},{"osd":2,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76600000000000001,"15min":0.75900000000000001},"min":{"1min":0.64700000000000002,"5min":0.48799999999999999,"15min":0.432},"max":{"1min":0.97399999999999998,"5min":1.006,"15min":2.3239999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74399999999999999,"15min":0.70699999999999996},"min":{"1min":0.56799999999999995,"5min":0.497,"15min":0.40000000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.71599999999999997},"min":{"1min":0.53800000000000003,"5min":0.52300000000000002,"15min":0.496},"max":{"1min":0.96899999999999997,"5min":0.98699999999999999,"15min":1.032},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.745},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.496},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.75700000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.625,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.625,"15min":0.64500000000000002},"min":{"1min":0.49199999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.004},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.624,"15min":0.64500000000000002},"min":{"1min":0.44400000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"max":{"1min":0.93200000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.71699999999999997},"min":{"1min":0.51400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0049999999999999,"5min":1.024,"15min":1.0589999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73999999999999999,"15min":0.753},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.0209999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.71299999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.75900000000000001,"15min":0.74399999999999999},"min":{"1min":0.49199999999999999,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.91900000000000004,"5min":0.96799999999999997,"15min":1.1759999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.78900000000000003},"min":{"1min":0.53000000000000003,"5min":0.436,"15min":0.436},"max":{"1min":0.91500000000000004,"5min":1.103,"15min":1.151},"last":0.89900000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.73999999999999999,"15min":0.72299999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":0.91800000000000004,"5min":1.02,"15min":1.3049999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.78900000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":0.88300000000000001,"5min":0.98199999999999998,"15min":6.9720000000000004},"last":0.84899999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.78200000000000003},"min":{"1min":0.53500000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92600000000000005,"5min":0.97899999999999998,"15min":5.9299999999999997},"last":0.88},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.52200000000000002,"5min":0.47499999999999998,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":1.036,"15min":1.704},"last":0.96699999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.48899999999999999,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":1,"15min":1.899},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.749,"15min":0.81899999999999995},"min":{"1min":0.622,"5min":0.51200000000000001,"15min":0.50600000000000001},"max":{"1min":1.016,"5min":1.016,"15min":6.0620000000000003},"last":0.82299999999999995}]},{"osd":4,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.374},"max":{"1min":0.85199999999999998,"5min":0.92100000000000004,"15min":0.97199999999999998},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63500000000000001,"15min":0.63200000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.39400000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.73699999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.39800000000000002,"5min":0.38900000000000001,"15min":0.33900000000000002},"max":{"1min":0.88,"5min":0.998,"15min":0.998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.70699999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.93500000000000005,"5min":0.98999999999999999,"15min":1.032},"last":0.76500000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70099999999999996,"15min":0.70499999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.0069999999999999,"15min":1.0189999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77100000000000002,"15min":0.76300000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.96099999999999997,"5min":1.0529999999999999,"15min":1.0649999999999999},"last":0.94499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:42:16.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:42:16.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.871+0000 7ff9d31ff700 1 Processor -- start 2023-10-07T09:42:16.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.871+0000 7ff9d31ff700 1 -- start start 2023-10-07T09:42:16.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9d31ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff9cc159750 0x7ff9cc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9d31ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9cc159b20 0x7ff9cc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9d31ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc0ce6d0 0x7ff9cc0ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9d31ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9cc0cbf20 con 0x7ff9cc159b20 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9d31ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9cc0cc200 con 0x7ff9cc159750 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9d31ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0cc4e0 con 0x7ff9cc0ce6d0 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.872+0000 7ff9cbfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9cc159b20 0x7ff9cc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9cc159b20 0x7ff9cc0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59024/0 (socket says 172.21.15.29:59024) 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 -- 172.21.15.29:0/1683051393 learned_addr learned my addr 172.21.15.29:0/1683051393 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9d179c700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc0ce6d0 0x7ff9cc0ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 -- 172.21.15.29:0/1683051393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc0ce6d0 msgr2=0x7ff9cc0ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:16.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9d0f9b700 1 --2- 172.21.15.29:0/1683051393 >> v2:172.21.15.144:3300/0 conn(0x7ff9cc159750 0x7ff9cc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc0ce6d0 0x7ff9cc0ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 -- 172.21.15.29:0/1683051393 >> v2:172.21.15.144:3300/0 conn(0x7ff9cc159750 msgr2=0x7ff9cc0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 --2- 172.21.15.29:0/1683051393 >> v2:172.21.15.144:3300/0 conn(0x7ff9cc159750 0x7ff9cc0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 -- 172.21.15.29:0/1683051393 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc0cc7c0 con 0x7ff9cc159b20 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9d179c700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc0ce6d0 0x7ff9cc0ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9d0f9b700 1 --2- 172.21.15.29:0/1683051393 >> v2:172.21.15.144:3300/0 conn(0x7ff9cc159750 0x7ff9cc0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cbfff700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc0ce190 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc089840 tx=0x7ff9b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d042c0ae5beab2f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.873+0000 7ff9cb7fe700 1 -- 172.21.15.29:0/1683051393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b800b860 con 0x7ff9cc159b20 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9cb7fe700 1 -- 172.21.15.29:0/1683051393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b800b9c0 con 0x7ff9cc159b20 2023-10-07T09:42:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9cb7fe700 1 -- 172.21.15.29:0/1683051393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b800bd60 con 0x7ff9cc159b20 2023-10-07T09:42:16.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1683051393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 msgr2=0x7ff9cc0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc0ce190 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc089840 tx=0x7ff9b8009250 comp rx=0 tx=0).stop 2023-10-07T09:42:16.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1683051393 shutdown_connections 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1683051393 >> v2:172.21.15.144:3300/0 conn(0x7ff9cc159750 0x7ff9cc0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc0ce6d0 0x7ff9cc0ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1683051393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc0ce190 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1683051393 >> 172.21.15.29:0/1683051393 conn(0x7ff9cc063a10 msgr2=0x7ff9cc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1683051393 shutdown_connections 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.874+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1683051393 wait complete. 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.876+0000 7ff9d31ff700 1 Processor -- start 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.876+0000 7ff9d31ff700 1 -- start start 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d31ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9cc0ce6d0 0x7ff9cc14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d31ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc159750 0x7ff9cc1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d31ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d31ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc1523e0 con 0x7ff9cc159b20 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d31ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc151580 con 0x7ff9cc0ce6d0 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d31ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc1529e0 con 0x7ff9cc159750 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d179c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d179c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59040/0 (socket says 172.21.15.29:59040) 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d179c700 1 -- 172.21.15.29:0/1695067341 learned_addr learned my addr 172.21.15.29:0/1695067341 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9cbfff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc159750 0x7ff9cc1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d0f9b700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9cc0ce6d0 0x7ff9cc14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d179c700 1 -- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc159750 msgr2=0x7ff9cc1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d179c700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc159750 0x7ff9cc1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.877+0000 7ff9d179c700 1 -- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9cc0ce6d0 msgr2=0x7ff9cc14a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9d179c700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9cc0ce6d0 0x7ff9cc14a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9d179c700 1 -- 172.21.15.29:0/1695067341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b80094f0 con 0x7ff9cc159b20 2023-10-07T09:42:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9d179c700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc158050 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400cf60 tx=0x7ff9c4007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400ae40 con 0x7ff9cc159b20 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4015440 con 0x7ff9cc159b20 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc1517d0 con 0x7ff9cc159b20 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4015800 con 0x7ff9cc159b20 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.878+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9cc151d90 con 0x7ff9cc159b20 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.879+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9cc04c3c0 con 0x7ff9cc159b20 2023-10-07T09:42:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.880+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff9c400d440 con 0x7ff9cc159b20 2023-10-07T09:42:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.881+0000 7ff9c8ff9700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff9b0066f30 0x7ff9b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.881+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff9c4085320 con 0x7ff9cc159b20 2023-10-07T09:42:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.882+0000 7ff9d0f9b700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff9b0066f30 0x7ff9b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.883+0000 7ff9d0f9b700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff9b0066f30 0x7ff9b00693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000ad40 tx=0x7ff9c0008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:16.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:16.885+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9c4052390 con 0x7ff9cc159b20 2023-10-07T09:42:17.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.085+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9cc064040 con 0x7ff9b0066f30 2023-10-07T09:42:17.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.094+0000 7ff9c8ff9700 1 -- 172.21.15.29:0/1695067341 <== mgr.5195 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288935 (secure 0 0 0) 0x7ff9cc064040 con 0x7ff9b0066f30 2023-10-07T09:42:17.099 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:17.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff9b0066f30 msgr2=0x7ff9b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff9b0066f30 0x7ff9b00693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000ad40 tx=0x7ff9c0008040 comp rx=0 tx=0).stop 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 msgr2=0x7ff9cc158050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc158050 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400cf60 tx=0x7ff9c4007870 comp rx=0 tx=0).stop 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 shutdown_connections 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff9b0066f30 0x7ff9b00693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9cc159b20 0x7ff9cc158050 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9cc0ce6d0 0x7ff9cc14a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 --2- 172.21.15.29:0/1695067341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9cc159750 0x7ff9cc1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 >> 172.21.15.29:0/1695067341 conn(0x7ff9cc063a10 msgr2=0x7ff9cc0cde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 shutdown_connections 2023-10-07T09:42:17.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.099+0000 7ff9d31ff700 1 -- 172.21.15.29:0/1695067341 wait complete. 2023-10-07T09:42:17.103 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:42:17.118 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-07T09:42:15.363255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2429980,"kb_used_data":6620,"kb_used_omap":0,"kb_used_meta":2423296,"kb_avail":752544740,"statfs":{"total":773094113280,"available":770605813760,"internally_reserved":0,"allocated":6778880,"data_stored":3727613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2481455104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.151188"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.124772+0000","last_change":"2023-10-07T09:42:12.124772+0000","last_active":"2023-10-07T09:42:12.124772+0000","last_peered":"2023-10-07T09:42:12.124772+0000","last_clean":"2023-10-07T09:42:12.124772+0000","last_became_active":"2023-10-07T09:42:12.124478+0000","last_became_peered":"2023-10-07T09:42:12.124478+0000","last_unstale":"2023-10-07T09:42:12.124772+0000","last_undegraded":"2023-10-07T09:42:12.124772+0000","last_fullsized":"2023-10-07T09:42:12.124772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:52:24.831499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126460+0000","last_change":"2023-10-07T09:42:12.126460+0000","last_active":"2023-10-07T09:42:12.126460+0000","last_peered":"2023-10-07T09:42:12.126460+0000","last_clean":"2023-10-07T09:42:12.126460+0000","last_became_active":"2023-10-07T09:42:12.126045+0000","last_became_peered":"2023-10-07T09:42:12.126045+0000","last_unstale":"2023-10-07T09:42:12.126460+0000","last_undegraded":"2023-10-07T09:42:12.126460+0000","last_fullsized":"2023-10-07T09:42:12.126460+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:05.782312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121617+0000","last_change":"2023-10-07T09:42:12.121617+0000","last_active":"2023-10-07T09:42:12.121617+0000","last_peered":"2023-10-07T09:42:12.121617+0000","last_clean":"2023-10-07T09:42:12.121617+0000","last_became_active":"2023-10-07T09:42:12.121245+0000","last_became_peered":"2023-10-07T09:42:12.121245+0000","last_unstale":"2023-10-07T09:42:12.121617+0000","last_undegraded":"2023-10-07T09:42:12.121617+0000","last_fullsized":"2023-10-07T09:42:12.121617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:14:00.035126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.130770+0000","last_change":"2023-10-07T09:42:12.130770+0000","last_active":"2023-10-07T09:42:12.130770+0000","last_peered":"2023-10-07T09:42:12.130770+0000","last_clean":"2023-10-07T09:42:12.130770+0000","last_became_active":"2023-10-07T09:42:12.130439+0000","last_became_peered":"2023-10-07T09:42:12.130439+0000","last_unstale":"2023-10-07T09:42:12.130770+0000","last_undegraded":"2023-10-07T09:42:12.130770+0000","last_fullsized":"2023-10-07T09:42:12.130770+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:57.864645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120436+0000","last_change":"2023-10-07T09:42:12.120436+0000","last_active":"2023-10-07T09:42:12.120436+0000","last_peered":"2023-10-07T09:42:12.120436+0000","last_clean":"2023-10-07T09:42:12.120436+0000","last_became_active":"2023-10-07T09:42:12.118858+0000","last_became_peered":"2023-10-07T09:42:12.118858+0000","last_unstale":"2023-10-07T09:42:12.120436+0000","last_undegraded":"2023-10-07T09:42:12.120436+0000","last_fullsized":"2023-10-07T09:42:12.120436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:05:13.537403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126860+0000","last_change":"2023-10-07T09:42:12.126860+0000","last_active":"2023-10-07T09:42:12.126860+0000","last_peered":"2023-10-07T09:42:12.126860+0000","last_clean":"2023-10-07T09:42:12.126860+0000","last_became_active":"2023-10-07T09:42:12.126574+0000","last_became_peered":"2023-10-07T09:42:12.126574+0000","last_unstale":"2023-10-07T09:42:12.126860+0000","last_undegraded":"2023-10-07T09:42:12.126860+0000","last_fullsized":"2023-10-07T09:42:12.126860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:33:31.178389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123128+0000","last_change":"2023-10-07T09:42:12.123128+0000","last_active":"2023-10-07T09:42:12.123128+0000","last_peered":"2023-10-07T09:42:12.123128+0000","last_clean":"2023-10-07T09:42:12.123128+0000","last_became_active":"2023-10-07T09:42:12.122816+0000","last_became_peered":"2023-10-07T09:42:12.122816+0000","last_unstale":"2023-10-07T09:42:12.123128+0000","last_undegraded":"2023-10-07T09:42:12.123128+0000","last_fullsized":"2023-10-07T09:42:12.123128+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:52:23.811737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131352+0000","last_change":"2023-10-07T09:42:12.131352+0000","last_active":"2023-10-07T09:42:12.131352+0000","last_peered":"2023-10-07T09:42:12.131352+0000","last_clean":"2023-10-07T09:42:12.131352+0000","last_became_active":"2023-10-07T09:42:12.131013+0000","last_became_peered":"2023-10-07T09:42:12.131013+0000","last_unstale":"2023-10-07T09:42:12.131352+0000","last_undegraded":"2023-10-07T09:42:12.131352+0000","last_fullsized":"2023-10-07T09:42:12.131352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:03.066571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120918+0000","last_change":"2023-10-07T09:42:12.120918+0000","last_active":"2023-10-07T09:42:12.120918+0000","last_peered":"2023-10-07T09:42:12.120918+0000","last_clean":"2023-10-07T09:42:12.120918+0000","last_became_active":"2023-10-07T09:42:12.120599+0000","last_became_peered":"2023-10-07T09:42:12.120599+0000","last_unstale":"2023-10-07T09:42:12.120918+0000","last_undegraded":"2023-10-07T09:42:12.120918+0000","last_fullsized":"2023-10-07T09:42:12.120918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:33.129125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.130815+0000","last_change":"2023-10-07T09:42:12.130815+0000","last_active":"2023-10-07T09:42:12.130815+0000","last_peered":"2023-10-07T09:42:12.130815+0000","last_clean":"2023-10-07T09:42:12.130815+0000","last_became_active":"2023-10-07T09:42:12.130446+0000","last_became_peered":"2023-10-07T09:42:12.130446+0000","last_unstale":"2023-10-07T09:42:12.130815+0000","last_undegraded":"2023-10-07T09:42:12.130815+0000","last_fullsized":"2023-10-07T09:42:12.130815+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:40.103291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120847+0000","last_change":"2023-10-07T09:42:12.120847+0000","last_active":"2023-10-07T09:42:12.120847+0000","last_peered":"2023-10-07T09:42:12.120847+0000","last_clean":"2023-10-07T09:42:12.120847+0000","last_became_active":"2023-10-07T09:42:12.120564+0000","last_became_peered":"2023-10-07T09:42:12.120564+0000","last_unstale":"2023-10-07T09:42:12.120847+0000","last_undegraded":"2023-10-07T09:42:12.120847+0000","last_fullsized":"2023-10-07T09:42:12.120847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:05.074867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120828+0000","last_change":"2023-10-07T09:42:12.120828+0000","last_active":"2023-10-07T09:42:12.120828+0000","last_peered":"2023-10-07T09:42:12.120828+0000","last_clean":"2023-10-07T09:42:12.120828+0000","last_became_active":"2023-10-07T09:42:12.120518+0000","last_became_peered":"2023-10-07T09:42:12.120518+0000","last_unstale":"2023-10-07T09:42:12.120828+0000","last_undegraded":"2023-10-07T09:42:12.120828+0000","last_fullsized":"2023-10-07T09:42:12.120828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:31:39.819015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132156+0000","last_change":"2023-10-07T09:42:12.132156+0000","last_active":"2023-10-07T09:42:12.132156+0000","last_peered":"2023-10-07T09:42:12.132156+0000","last_clean":"2023-10-07T09:42:12.132156+0000","last_became_active":"2023-10-07T09:42:12.131825+0000","last_became_peered":"2023-10-07T09:42:12.131825+0000","last_unstale":"2023-10-07T09:42:12.132156+0000","last_undegraded":"2023-10-07T09:42:12.132156+0000","last_fullsized":"2023-10-07T09:42:12.132156+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:13:17.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120211+0000","last_change":"2023-10-07T09:42:12.120211+0000","last_active":"2023-10-07T09:42:12.120211+0000","last_peered":"2023-10-07T09:42:12.120211+0000","last_clean":"2023-10-07T09:42:12.120211+0000","last_became_active":"2023-10-07T09:42:12.119878+0000","last_became_peered":"2023-10-07T09:42:12.119878+0000","last_unstale":"2023-10-07T09:42:12.120211+0000","last_undegraded":"2023-10-07T09:42:12.120211+0000","last_fullsized":"2023-10-07T09:42:12.120211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:51.992198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120556+0000","last_change":"2023-10-07T09:42:12.120556+0000","last_active":"2023-10-07T09:42:12.120556+0000","last_peered":"2023-10-07T09:42:12.120556+0000","last_clean":"2023-10-07T09:42:12.120556+0000","last_became_active":"2023-10-07T09:42:12.120290+0000","last_became_peered":"2023-10-07T09:42:12.120290+0000","last_unstale":"2023-10-07T09:42:12.120556+0000","last_undegraded":"2023-10-07T09:42:12.120556+0000","last_fullsized":"2023-10-07T09:42:12.120556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:50.901171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131719+0000","last_change":"2023-10-07T09:42:12.131719+0000","last_active":"2023-10-07T09:42:12.131719+0000","last_peered":"2023-10-07T09:42:12.131719+0000","last_clean":"2023-10-07T09:42:12.131719+0000","last_became_active":"2023-10-07T09:42:12.131252+0000","last_became_peered":"2023-10-07T09:42:12.131252+0000","last_unstale":"2023-10-07T09:42:12.131719+0000","last_undegraded":"2023-10-07T09:42:12.131719+0000","last_fullsized":"2023-10-07T09:42:12.131719+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:44.145951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.119571+0000","last_change":"2023-10-07T09:42:12.119571+0000","last_active":"2023-10-07T09:42:12.119571+0000","last_peered":"2023-10-07T09:42:12.119571+0000","last_clean":"2023-10-07T09:42:12.119571+0000","last_became_active":"2023-10-07T09:42:12.119315+0000","last_became_peered":"2023-10-07T09:42:12.119315+0000","last_unstale":"2023-10-07T09:42:12.119571+0000","last_undegraded":"2023-10-07T09:42:12.119571+0000","last_fullsized":"2023-10-07T09:42:12.119571+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:00:41.736832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127297+0000","last_change":"2023-10-07T09:42:12.127297+0000","last_active":"2023-10-07T09:42:12.127297+0000","last_peered":"2023-10-07T09:42:12.127297+0000","last_clean":"2023-10-07T09:42:12.127297+0000","last_became_active":"2023-10-07T09:42:12.126870+0000","last_became_peered":"2023-10-07T09:42:12.126870+0000","last_unstale":"2023-10-07T09:42:12.127297+0000","last_undegraded":"2023-10-07T09:42:12.127297+0000","last_fullsized":"2023-10-07T09:42:12.127297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:31:11.738882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131934+0000","last_change":"2023-10-07T09:42:12.131934+0000","last_active":"2023-10-07T09:42:12.131934+0000","last_peered":"2023-10-07T09:42:12.131934+0000","last_clean":"2023-10-07T09:42:12.131934+0000","last_became_active":"2023-10-07T09:42:12.131643+0000","last_became_peered":"2023-10-07T09:42:12.131643+0000","last_unstale":"2023-10-07T09:42:12.131934+0000","last_undegraded":"2023-10-07T09:42:12.131934+0000","last_fullsized":"2023-10-07T09:42:12.131934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:51:20.622847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126673+0000","last_change":"2023-10-07T09:42:12.126673+0000","last_active":"2023-10-07T09:42:12.126673+0000","last_peered":"2023-10-07T09:42:12.126673+0000","last_clean":"2023-10-07T09:42:12.126673+0000","last_became_active":"2023-10-07T09:42:12.126322+0000","last_became_peered":"2023-10-07T09:42:12.126322+0000","last_unstale":"2023-10-07T09:42:12.126673+0000","last_undegraded":"2023-10-07T09:42:12.126673+0000","last_fullsized":"2023-10-07T09:42:12.126673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:08.038437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131590+0000","last_change":"2023-10-07T09:42:12.131590+0000","last_active":"2023-10-07T09:42:12.131590+0000","last_peered":"2023-10-07T09:42:12.131590+0000","last_clean":"2023-10-07T09:42:12.131590+0000","last_became_active":"2023-10-07T09:42:12.131287+0000","last_became_peered":"2023-10-07T09:42:12.131287+0000","last_unstale":"2023-10-07T09:42:12.131590+0000","last_undegraded":"2023-10-07T09:42:12.131590+0000","last_fullsized":"2023-10-07T09:42:12.131590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:58:16.359660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129051+0000","last_change":"2023-10-07T09:42:12.129051+0000","last_active":"2023-10-07T09:42:12.129051+0000","last_peered":"2023-10-07T09:42:12.129051+0000","last_clean":"2023-10-07T09:42:12.129051+0000","last_became_active":"2023-10-07T09:42:12.128749+0000","last_became_peered":"2023-10-07T09:42:12.128749+0000","last_unstale":"2023-10-07T09:42:12.129051+0000","last_undegraded":"2023-10-07T09:42:12.129051+0000","last_fullsized":"2023-10-07T09:42:12.129051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:27.390563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127630+0000","last_change":"2023-10-07T09:42:12.127630+0000","last_active":"2023-10-07T09:42:12.127630+0000","last_peered":"2023-10-07T09:42:12.127630+0000","last_clean":"2023-10-07T09:42:12.127630+0000","last_became_active":"2023-10-07T09:42:12.127263+0000","last_became_peered":"2023-10-07T09:42:12.127263+0000","last_unstale":"2023-10-07T09:42:12.127630+0000","last_undegraded":"2023-10-07T09:42:12.127630+0000","last_fullsized":"2023-10-07T09:42:12.127630+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:49.566442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.128842+0000","last_change":"2023-10-07T09:42:12.128842+0000","last_active":"2023-10-07T09:42:12.128842+0000","last_peered":"2023-10-07T09:42:12.128842+0000","last_clean":"2023-10-07T09:42:12.128842+0000","last_became_active":"2023-10-07T09:42:12.128551+0000","last_became_peered":"2023-10-07T09:42:12.128551+0000","last_unstale":"2023-10-07T09:42:12.128842+0000","last_undegraded":"2023-10-07T09:42:12.128842+0000","last_fullsized":"2023-10-07T09:42:12.128842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:22.056475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132283+0000","last_change":"2023-10-07T09:42:12.132283+0000","last_active":"2023-10-07T09:42:12.132283+0000","last_peered":"2023-10-07T09:42:12.132283+0000","last_clean":"2023-10-07T09:42:12.132283+0000","last_became_active":"2023-10-07T09:42:12.131048+0000","last_became_peered":"2023-10-07T09:42:12.131048+0000","last_unstale":"2023-10-07T09:42:12.132283+0000","last_undegraded":"2023-10-07T09:42:12.132283+0000","last_fullsized":"2023-10-07T09:42:12.132283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:13:29.579840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132151+0000","last_change":"2023-10-07T09:42:12.132151+0000","last_active":"2023-10-07T09:42:12.132151+0000","last_peered":"2023-10-07T09:42:12.132151+0000","last_clean":"2023-10-07T09:42:12.132151+0000","last_became_active":"2023-10-07T09:42:12.131872+0000","last_became_peered":"2023-10-07T09:42:12.131872+0000","last_unstale":"2023-10-07T09:42:12.132151+0000","last_undegraded":"2023-10-07T09:42:12.132151+0000","last_fullsized":"2023-10-07T09:42:12.132151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:11:41.208779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.117072+0000","last_change":"2023-10-07T09:42:12.117072+0000","last_active":"2023-10-07T09:42:12.117072+0000","last_peered":"2023-10-07T09:42:12.117072+0000","last_clean":"2023-10-07T09:42:12.117072+0000","last_became_active":"2023-10-07T09:42:12.116727+0000","last_became_peered":"2023-10-07T09:42:12.116727+0000","last_unstale":"2023-10-07T09:42:12.117072+0000","last_undegraded":"2023-10-07T09:42:12.117072+0000","last_fullsized":"2023-10-07T09:42:12.117072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:54.073522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121966+0000","last_change":"2023-10-07T09:42:12.121966+0000","last_active":"2023-10-07T09:42:12.121966+0000","last_peered":"2023-10-07T09:42:12.121966+0000","last_clean":"2023-10-07T09:42:12.121966+0000","last_became_active":"2023-10-07T09:42:12.120878+0000","last_became_peered":"2023-10-07T09:42:12.120878+0000","last_unstale":"2023-10-07T09:42:12.121966+0000","last_undegraded":"2023-10-07T09:42:12.121966+0000","last_fullsized":"2023-10-07T09:42:12.121966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:34:26.900282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132072+0000","last_change":"2023-10-07T09:42:12.132072+0000","last_active":"2023-10-07T09:42:12.132072+0000","last_peered":"2023-10-07T09:42:12.132072+0000","last_clean":"2023-10-07T09:42:12.132072+0000","last_became_active":"2023-10-07T09:42:12.131735+0000","last_became_peered":"2023-10-07T09:42:12.131735+0000","last_unstale":"2023-10-07T09:42:12.132072+0000","last_undegraded":"2023-10-07T09:42:12.132072+0000","last_fullsized":"2023-10-07T09:42:12.132072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:11:41.208779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123904+0000","last_change":"2023-10-07T09:42:12.123904+0000","last_active":"2023-10-07T09:42:12.123904+0000","last_peered":"2023-10-07T09:42:12.123904+0000","last_clean":"2023-10-07T09:42:12.123904+0000","last_became_active":"2023-10-07T09:42:12.123638+0000","last_became_peered":"2023-10-07T09:42:12.123638+0000","last_unstale":"2023-10-07T09:42:12.123904+0000","last_undegraded":"2023-10-07T09:42:12.123904+0000","last_fullsized":"2023-10-07T09:42:12.123904+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:21:48.056713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123704+0000","last_change":"2023-10-07T09:42:12.123704+0000","last_active":"2023-10-07T09:42:12.123704+0000","last_peered":"2023-10-07T09:42:12.123704+0000","last_clean":"2023-10-07T09:42:12.123704+0000","last_became_active":"2023-10-07T09:42:12.123461+0000","last_became_peered":"2023-10-07T09:42:12.123461+0000","last_unstale":"2023-10-07T09:42:12.123704+0000","last_undegraded":"2023-10-07T09:42:12.123704+0000","last_fullsized":"2023-10-07T09:42:12.123704+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:05:13.537403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.128726+0000","last_change":"2023-10-07T09:42:12.128726+0000","last_active":"2023-10-07T09:42:12.128726+0000","last_peered":"2023-10-07T09:42:12.128726+0000","last_clean":"2023-10-07T09:42:12.128726+0000","last_became_active":"2023-10-07T09:42:12.128486+0000","last_became_peered":"2023-10-07T09:42:12.128486+0000","last_unstale":"2023-10-07T09:42:12.128726+0000","last_undegraded":"2023-10-07T09:42:12.128726+0000","last_fullsized":"2023-10-07T09:42:12.128726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:05:35.256308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.118348+0000","last_change":"2023-10-07T09:42:12.118348+0000","last_active":"2023-10-07T09:42:12.118348+0000","last_peered":"2023-10-07T09:42:12.118348+0000","last_clean":"2023-10-07T09:42:12.118348+0000","last_became_active":"2023-10-07T09:42:12.118076+0000","last_became_peered":"2023-10-07T09:42:12.118076+0000","last_unstale":"2023-10-07T09:42:12.118348+0000","last_undegraded":"2023-10-07T09:42:12.118348+0000","last_fullsized":"2023-10-07T09:42:12.118348+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:30.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"60'4","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.248161+0000","last_change":"2023-10-07T09:42:11.231473+0000","last_active":"2023-10-07T09:42:12.248161+0000","last_peered":"2023-10-07T09:42:12.248161+0000","last_clean":"2023-10-07T09:42:12.248161+0000","last_became_active":"2023-10-07T09:42:11.231207+0000","last_became_peered":"2023-10-07T09:42:11.231207+0000","last_unstale":"2023-10-07T09:42:12.248161+0000","last_undegraded":"2023-10-07T09:42:12.248161+0000","last_fullsized":"2023-10-07T09:42:12.248161+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:39.097152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132558+0000","last_change":"2023-10-07T09:42:12.132558+0000","last_active":"2023-10-07T09:42:12.132558+0000","last_peered":"2023-10-07T09:42:12.132558+0000","last_clean":"2023-10-07T09:42:12.132558+0000","last_became_active":"2023-10-07T09:42:12.130893+0000","last_became_peered":"2023-10-07T09:42:12.130893+0000","last_unstale":"2023-10-07T09:42:12.132558+0000","last_undegraded":"2023-10-07T09:42:12.132558+0000","last_fullsized":"2023-10-07T09:42:12.132558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:38:02.193963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.111536+0000","last_change":"2023-10-07T09:42:11.235972+0000","last_active":"2023-10-07T09:42:12.111536+0000","last_peered":"2023-10-07T09:42:12.111536+0000","last_clean":"2023-10-07T09:42:12.111536+0000","last_became_active":"2023-10-07T09:42:11.235710+0000","last_became_peered":"2023-10-07T09:42:11.235710+0000","last_unstale":"2023-10-07T09:42:12.111536+0000","last_undegraded":"2023-10-07T09:42:12.111536+0000","last_fullsized":"2023-10-07T09:42:12.111536+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:26.384424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.117347+0000","last_change":"2023-10-07T09:42:12.117347+0000","last_active":"2023-10-07T09:42:12.117347+0000","last_peered":"2023-10-07T09:42:12.117347+0000","last_clean":"2023-10-07T09:42:12.117347+0000","last_became_active":"2023-10-07T09:42:12.117054+0000","last_became_peered":"2023-10-07T09:42:12.117054+0000","last_unstale":"2023-10-07T09:42:12.117347+0000","last_undegraded":"2023-10-07T09:42:12.117347+0000","last_fullsized":"2023-10-07T09:42:12.117347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:50.901171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.245803+0000","last_change":"2023-10-07T09:42:11.234456+0000","last_active":"2023-10-07T09:42:12.245803+0000","last_peered":"2023-10-07T09:42:12.245803+0000","last_clean":"2023-10-07T09:42:12.245803+0000","last_became_active":"2023-10-07T09:42:11.234183+0000","last_became_peered":"2023-10-07T09:42:11.234183+0000","last_unstale":"2023-10-07T09:42:12.245803+0000","last_undegraded":"2023-10-07T09:42:12.245803+0000","last_fullsized":"2023-10-07T09:42:12.245803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:43.139812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129991+0000","last_change":"2023-10-07T09:42:12.129991+0000","last_active":"2023-10-07T09:42:12.129991+0000","last_peered":"2023-10-07T09:42:12.129991+0000","last_clean":"2023-10-07T09:42:12.129991+0000","last_became_active":"2023-10-07T09:42:12.129660+0000","last_became_peered":"2023-10-07T09:42:12.129660+0000","last_unstale":"2023-10-07T09:42:12.129991+0000","last_undegraded":"2023-10-07T09:42:12.129991+0000","last_fullsized":"2023-10-07T09:42:12.129991+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:05.074867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.245575+0000","last_change":"2023-10-07T09:42:11.244387+0000","last_active":"2023-10-07T09:42:12.245575+0000","last_peered":"2023-10-07T09:42:12.245575+0000","last_clean":"2023-10-07T09:42:12.245575+0000","last_became_active":"2023-10-07T09:42:11.244024+0000","last_became_peered":"2023-10-07T09:42:11.244024+0000","last_unstale":"2023-10-07T09:42:12.245575+0000","last_undegraded":"2023-10-07T09:42:12.245575+0000","last_fullsized":"2023-10-07T09:42:12.245575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:37.056219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.128392+0000","last_change":"2023-10-07T09:42:12.128392+0000","last_active":"2023-10-07T09:42:12.128392+0000","last_peered":"2023-10-07T09:42:12.128392+0000","last_clean":"2023-10-07T09:42:12.128392+0000","last_became_active":"2023-10-07T09:42:12.128121+0000","last_became_peered":"2023-10-07T09:42:12.128121+0000","last_unstale":"2023-10-07T09:42:12.128392+0000","last_undegraded":"2023-10-07T09:42:12.128392+0000","last_fullsized":"2023-10-07T09:42:12.128392+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:07.515132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.112554+0000","last_change":"2023-10-07T09:42:11.238834+0000","last_active":"2023-10-07T09:42:12.112554+0000","last_peered":"2023-10-07T09:42:12.112554+0000","last_clean":"2023-10-07T09:42:12.112554+0000","last_became_active":"2023-10-07T09:42:11.238558+0000","last_became_peered":"2023-10-07T09:42:11.238558+0000","last_unstale":"2023-10-07T09:42:12.112554+0000","last_undegraded":"2023-10-07T09:42:12.112554+0000","last_fullsized":"2023-10-07T09:42:12.112554+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:02.060432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.125116+0000","last_change":"2023-10-07T09:42:12.125116+0000","last_active":"2023-10-07T09:42:12.125116+0000","last_peered":"2023-10-07T09:42:12.125116+0000","last_clean":"2023-10-07T09:42:12.125116+0000","last_became_active":"2023-10-07T09:42:12.121195+0000","last_became_peered":"2023-10-07T09:42:12.121195+0000","last_unstale":"2023-10-07T09:42:12.125116+0000","last_undegraded":"2023-10-07T09:42:12.125116+0000","last_fullsized":"2023-10-07T09:42:12.125116+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:28.035281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.110744+0000","last_change":"2023-10-07T09:42:11.237502+0000","last_active":"2023-10-07T09:42:12.110744+0000","last_peered":"2023-10-07T09:42:12.110744+0000","last_clean":"2023-10-07T09:42:12.110744+0000","last_became_active":"2023-10-07T09:42:11.237223+0000","last_became_peered":"2023-10-07T09:42:11.237223+0000","last_unstale":"2023-10-07T09:42:12.110744+0000","last_undegraded":"2023-10-07T09:42:12.110744+0000","last_fullsized":"2023-10-07T09:42:12.110744+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:13:28.573701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.130988+0000","last_change":"2023-10-07T09:42:12.130988+0000","last_active":"2023-10-07T09:42:12.130988+0000","last_peered":"2023-10-07T09:42:12.130988+0000","last_clean":"2023-10-07T09:42:12.130988+0000","last_became_active":"2023-10-07T09:42:12.130643+0000","last_became_peered":"2023-10-07T09:42:12.130643+0000","last_unstale":"2023-10-07T09:42:12.130988+0000","last_undegraded":"2023-10-07T09:42:12.130988+0000","last_fullsized":"2023-10-07T09:42:12.130988+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:13:29.579840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.242663+0000","last_change":"2023-10-07T09:42:11.233655+0000","last_active":"2023-10-07T09:42:11.242663+0000","last_peered":"2023-10-07T09:42:11.242663+0000","last_clean":"2023-10-07T09:42:11.242663+0000","last_became_active":"2023-10-07T09:42:11.233406+0000","last_became_peered":"2023-10-07T09:42:11.233406+0000","last_unstale":"2023-10-07T09:42:11.242663+0000","last_undegraded":"2023-10-07T09:42:11.242663+0000","last_fullsized":"2023-10-07T09:42:11.242663+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:56.926207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129703+0000","last_change":"2023-10-07T09:42:12.129703+0000","last_active":"2023-10-07T09:42:12.129703+0000","last_peered":"2023-10-07T09:42:12.129703+0000","last_clean":"2023-10-07T09:42:12.129703+0000","last_became_active":"2023-10-07T09:42:12.129452+0000","last_became_peered":"2023-10-07T09:42:12.129452+0000","last_unstale":"2023-10-07T09:42:12.129703+0000","last_undegraded":"2023-10-07T09:42:12.129703+0000","last_fullsized":"2023-10-07T09:42:12.129703+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:15.488935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.238439+0000","last_change":"2023-10-07T09:42:11.236126+0000","last_active":"2023-10-07T09:42:11.238439+0000","last_peered":"2023-10-07T09:42:11.238439+0000","last_clean":"2023-10-07T09:42:11.238439+0000","last_became_active":"2023-10-07T09:42:11.235828+0000","last_became_peered":"2023-10-07T09:42:11.235828+0000","last_unstale":"2023-10-07T09:42:11.238439+0000","last_undegraded":"2023-10-07T09:42:11.238439+0000","last_fullsized":"2023-10-07T09:42:11.238439+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:39:45.612026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127111+0000","last_change":"2023-10-07T09:42:12.127111+0000","last_active":"2023-10-07T09:42:12.127111+0000","last_peered":"2023-10-07T09:42:12.127111+0000","last_clean":"2023-10-07T09:42:12.127111+0000","last_became_active":"2023-10-07T09:42:12.126525+0000","last_became_peered":"2023-10-07T09:42:12.126525+0000","last_unstale":"2023-10-07T09:42:12.127111+0000","last_undegraded":"2023-10-07T09:42:12.127111+0000","last_fullsized":"2023-10-07T09:42:12.127111+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:14:03.442618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.240274+0000","last_change":"2023-10-07T09:42:11.237252+0000","last_active":"2023-10-07T09:42:11.240274+0000","last_peered":"2023-10-07T09:42:11.240274+0000","last_clean":"2023-10-07T09:42:11.240274+0000","last_became_active":"2023-10-07T09:42:11.236946+0000","last_became_peered":"2023-10-07T09:42:11.236946+0000","last_unstale":"2023-10-07T09:42:11.240274+0000","last_undegraded":"2023-10-07T09:42:11.240274+0000","last_fullsized":"2023-10-07T09:42:11.240274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:21.050337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123936+0000","last_change":"2023-10-07T09:42:12.123936+0000","last_active":"2023-10-07T09:42:12.123936+0000","last_peered":"2023-10-07T09:42:12.123936+0000","last_clean":"2023-10-07T09:42:12.123936+0000","last_became_active":"2023-10-07T09:42:12.123615+0000","last_became_peered":"2023-10-07T09:42:12.123615+0000","last_unstale":"2023-10-07T09:42:12.123936+0000","last_undegraded":"2023-10-07T09:42:12.123936+0000","last_fullsized":"2023-10-07T09:42:12.123936+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:40:35.941030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.239756+0000","last_change":"2023-10-07T09:42:11.238098+0000","last_active":"2023-10-07T09:42:11.239756+0000","last_peered":"2023-10-07T09:42:11.239756+0000","last_clean":"2023-10-07T09:42:11.239756+0000","last_became_active":"2023-10-07T09:42:11.237800+0000","last_became_peered":"2023-10-07T09:42:11.237800+0000","last_unstale":"2023-10-07T09:42:11.239756+0000","last_undegraded":"2023-10-07T09:42:11.239756+0000","last_fullsized":"2023-10-07T09:42:11.239756+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:06:46.247772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.122886+0000","last_change":"2023-10-07T09:42:12.122886+0000","last_active":"2023-10-07T09:42:12.122886+0000","last_peered":"2023-10-07T09:42:12.122886+0000","last_clean":"2023-10-07T09:42:12.122886+0000","last_became_active":"2023-10-07T09:42:12.122128+0000","last_became_peered":"2023-10-07T09:42:12.122128+0000","last_unstale":"2023-10-07T09:42:12.122886+0000","last_undegraded":"2023-10-07T09:42:12.122886+0000","last_fullsized":"2023-10-07T09:42:12.122886+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:27:30.802150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"60'3","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.248088+0000","last_change":"2023-10-07T09:42:11.242023+0000","last_active":"2023-10-07T09:42:12.248088+0000","last_peered":"2023-10-07T09:42:12.248088+0000","last_clean":"2023-10-07T09:42:12.248088+0000","last_became_active":"2023-10-07T09:42:11.241674+0000","last_became_peered":"2023-10-07T09:42:11.241674+0000","last_unstale":"2023-10-07T09:42:12.248088+0000","last_undegraded":"2023-10-07T09:42:12.248088+0000","last_fullsized":"2023-10-07T09:42:12.248088+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:31:10.175821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.126290+0000","last_change":"2023-10-07T09:42:12.126290+0000","last_active":"2023-10-07T09:42:12.126290+0000","last_peered":"2023-10-07T09:42:12.126290+0000","last_clean":"2023-10-07T09:42:12.126290+0000","last_became_active":"2023-10-07T09:42:12.125998+0000","last_became_peered":"2023-10-07T09:42:12.125998+0000","last_unstale":"2023-10-07T09:42:12.126290+0000","last_undegraded":"2023-10-07T09:42:12.126290+0000","last_fullsized":"2023-10-07T09:42:12.126290+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:58:11.376756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"60'6","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.246534+0000","last_change":"2023-10-07T09:42:11.241562+0000","last_active":"2023-10-07T09:42:12.246534+0000","last_peered":"2023-10-07T09:42:12.246534+0000","last_clean":"2023-10-07T09:42:12.246534+0000","last_became_active":"2023-10-07T09:42:11.241297+0000","last_became_peered":"2023-10-07T09:42:11.241297+0000","last_unstale":"2023-10-07T09:42:12.246534+0000","last_undegraded":"2023-10-07T09:42:12.246534+0000","last_fullsized":"2023-10-07T09:42:12.246534+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:22:43.173960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131813+0000","last_change":"2023-10-07T09:42:12.131813+0000","last_active":"2023-10-07T09:42:12.131813+0000","last_peered":"2023-10-07T09:42:12.131813+0000","last_clean":"2023-10-07T09:42:12.131813+0000","last_became_active":"2023-10-07T09:42:12.131547+0000","last_became_peered":"2023-10-07T09:42:12.131547+0000","last_unstale":"2023-10-07T09:42:12.131813+0000","last_undegraded":"2023-10-07T09:42:12.131813+0000","last_fullsized":"2023-10-07T09:42:12.131813+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:32.328778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.246869+0000","last_change":"2023-10-07T09:42:11.229919+0000","last_active":"2023-10-07T09:42:12.246869+0000","last_peered":"2023-10-07T09:42:12.246869+0000","last_clean":"2023-10-07T09:42:12.246869+0000","last_became_active":"2023-10-07T09:42:11.229655+0000","last_became_peered":"2023-10-07T09:42:11.229655+0000","last_unstale":"2023-10-07T09:42:12.246869+0000","last_undegraded":"2023-10-07T09:42:12.246869+0000","last_fullsized":"2023-10-07T09:42:12.246869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:29.091141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127547+0000","last_change":"2023-10-07T09:42:12.127547+0000","last_active":"2023-10-07T09:42:12.127547+0000","last_peered":"2023-10-07T09:42:12.127547+0000","last_clean":"2023-10-07T09:42:12.127547+0000","last_became_active":"2023-10-07T09:42:12.125853+0000","last_became_peered":"2023-10-07T09:42:12.125853+0000","last_unstale":"2023-10-07T09:42:12.127547+0000","last_undegraded":"2023-10-07T09:42:12.127547+0000","last_fullsized":"2023-10-07T09:42:12.127547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:07.515132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.242466+0000","last_change":"2023-10-07T09:42:11.240995+0000","last_active":"2023-10-07T09:42:11.242466+0000","last_peered":"2023-10-07T09:42:11.242466+0000","last_clean":"2023-10-07T09:42:11.242466+0000","last_became_active":"2023-10-07T09:42:11.240746+0000","last_became_peered":"2023-10-07T09:42:11.240746+0000","last_unstale":"2023-10-07T09:42:11.242466+0000","last_undegraded":"2023-10-07T09:42:11.242466+0000","last_fullsized":"2023-10-07T09:42:11.242466+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:22:57.052047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121867+0000","last_change":"2023-10-07T09:42:12.121867+0000","last_active":"2023-10-07T09:42:12.121867+0000","last_peered":"2023-10-07T09:42:12.121867+0000","last_clean":"2023-10-07T09:42:12.121867+0000","last_became_active":"2023-10-07T09:42:12.121607+0000","last_became_peered":"2023-10-07T09:42:12.121607+0000","last_unstale":"2023-10-07T09:42:12.121867+0000","last_undegraded":"2023-10-07T09:42:12.121867+0000","last_fullsized":"2023-10-07T09:42:12.121867+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:34:26.900282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.231572+0000","last_change":"2023-10-07T09:42:11.229317+0000","last_active":"2023-10-07T09:42:11.231572+0000","last_peered":"2023-10-07T09:42:11.231572+0000","last_clean":"2023-10-07T09:42:11.231572+0000","last_became_active":"2023-10-07T09:42:11.228770+0000","last_became_peered":"2023-10-07T09:42:11.228770+0000","last_unstale":"2023-10-07T09:42:11.231572+0000","last_undegraded":"2023-10-07T09:42:11.231572+0000","last_fullsized":"2023-10-07T09:42:11.231572+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:32.122986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.129281+0000","last_change":"2023-10-07T09:42:12.129281+0000","last_active":"2023-10-07T09:42:12.129281+0000","last_peered":"2023-10-07T09:42:12.129281+0000","last_clean":"2023-10-07T09:42:12.129281+0000","last_became_active":"2023-10-07T09:42:12.128785+0000","last_became_peered":"2023-10-07T09:42:12.128785+0000","last_unstale":"2023-10-07T09:42:12.129281+0000","last_undegraded":"2023-10-07T09:42:12.129281+0000","last_fullsized":"2023-10-07T09:42:12.129281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:05:58.578759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"60'8","reported_seq":17,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.248913+0000","last_change":"2023-10-07T09:42:11.244077+0000","last_active":"2023-10-07T09:42:12.248913+0000","last_peered":"2023-10-07T09:42:12.248913+0000","last_clean":"2023-10-07T09:42:12.248913+0000","last_became_active":"2023-10-07T09:42:11.243779+0000","last_became_peered":"2023-10-07T09:42:11.243779+0000","last_unstale":"2023-10-07T09:42:12.248913+0000","last_undegraded":"2023-10-07T09:42:12.248913+0000","last_fullsized":"2023-10-07T09:42:12.248913+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:37:05.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123775+0000","last_change":"2023-10-07T09:42:12.123775+0000","last_active":"2023-10-07T09:42:12.123775+0000","last_peered":"2023-10-07T09:42:12.123775+0000","last_clean":"2023-10-07T09:42:12.123775+0000","last_became_active":"2023-10-07T09:42:12.123503+0000","last_became_peered":"2023-10-07T09:42:12.123503+0000","last_unstale":"2023-10-07T09:42:12.123775+0000","last_undegraded":"2023-10-07T09:42:12.123775+0000","last_fullsized":"2023-10-07T09:42:12.123775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:26:27.535362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.116761+0000","last_change":"2023-10-07T09:42:11.237191+0000","last_active":"2023-10-07T09:42:12.116761+0000","last_peered":"2023-10-07T09:42:12.116761+0000","last_clean":"2023-10-07T09:42:12.116761+0000","last_became_active":"2023-10-07T09:42:11.236876+0000","last_became_peered":"2023-10-07T09:42:11.236876+0000","last_unstale":"2023-10-07T09:42:12.116761+0000","last_undegraded":"2023-10-07T09:42:12.116761+0000","last_fullsized":"2023-10-07T09:42:12.116761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:35.007910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.117385+0000","last_change":"2023-10-07T09:42:12.117385+0000","last_active":"2023-10-07T09:42:12.117385+0000","last_peered":"2023-10-07T09:42:12.117385+0000","last_clean":"2023-10-07T09:42:12.117385+0000","last_became_active":"2023-10-07T09:42:12.117061+0000","last_became_peered":"2023-10-07T09:42:12.117061+0000","last_unstale":"2023-10-07T09:42:12.117385+0000","last_undegraded":"2023-10-07T09:42:12.117385+0000","last_fullsized":"2023-10-07T09:42:12.117385+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:44.145951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.246268+0000","last_change":"2023-10-07T09:42:11.234033+0000","last_active":"2023-10-07T09:42:12.246268+0000","last_peered":"2023-10-07T09:42:12.246268+0000","last_clean":"2023-10-07T09:42:12.246268+0000","last_became_active":"2023-10-07T09:42:11.233737+0000","last_became_peered":"2023-10-07T09:42:11.233737+0000","last_unstale":"2023-10-07T09:42:12.246268+0000","last_undegraded":"2023-10-07T09:42:12.246268+0000","last_fullsized":"2023-10-07T09:42:12.246268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:48.560303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.132849+0000","last_change":"2023-10-07T09:42:12.132849+0000","last_active":"2023-10-07T09:42:12.132849+0000","last_peered":"2023-10-07T09:42:12.132849+0000","last_clean":"2023-10-07T09:42:12.132849+0000","last_became_active":"2023-10-07T09:42:12.132176+0000","last_became_peered":"2023-10-07T09:42:12.132176+0000","last_unstale":"2023-10-07T09:42:12.132849+0000","last_undegraded":"2023-10-07T09:42:12.132849+0000","last_fullsized":"2023-10-07T09:42:12.132849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:08.388551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"60'1","reported_seq":10,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.111524+0000","last_change":"2023-10-07T09:42:11.241129+0000","last_active":"2023-10-07T09:42:12.111524+0000","last_peered":"2023-10-07T09:42:12.111524+0000","last_clean":"2023-10-07T09:42:12.111524+0000","last_became_active":"2023-10-07T09:42:11.240788+0000","last_became_peered":"2023-10-07T09:42:11.240788+0000","last_unstale":"2023-10-07T09:42:12.111524+0000","last_undegraded":"2023-10-07T09:42:12.111524+0000","last_fullsized":"2023-10-07T09:42:12.111524+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:27.029142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.131349+0000","last_change":"2023-10-07T09:42:12.131349+0000","last_active":"2023-10-07T09:42:12.131349+0000","last_peered":"2023-10-07T09:42:12.131349+0000","last_clean":"2023-10-07T09:42:12.131349+0000","last_became_active":"2023-10-07T09:42:12.131002+0000","last_became_peered":"2023-10-07T09:42:12.131002+0000","last_unstale":"2023-10-07T09:42:12.131349+0000","last_undegraded":"2023-10-07T09:42:12.131349+0000","last_fullsized":"2023-10-07T09:42:12.131349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:07:08.038437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.234450+0000","last_change":"2023-10-07T09:42:11.231480+0000","last_active":"2023-10-07T09:42:11.234450+0000","last_peered":"2023-10-07T09:42:11.234450+0000","last_clean":"2023-10-07T09:42:11.234450+0000","last_became_active":"2023-10-07T09:42:11.231208+0000","last_became_peered":"2023-10-07T09:42:11.231208+0000","last_unstale":"2023-10-07T09:42:11.234450+0000","last_undegraded":"2023-10-07T09:42:11.234450+0000","last_fullsized":"2023-10-07T09:42:11.234450+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:04.068728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121547+0000","last_change":"2023-10-07T09:42:12.121547+0000","last_active":"2023-10-07T09:42:12.121547+0000","last_peered":"2023-10-07T09:42:12.121547+0000","last_clean":"2023-10-07T09:42:12.121547+0000","last_became_active":"2023-10-07T09:42:12.121264+0000","last_became_peered":"2023-10-07T09:42:12.121264+0000","last_unstale":"2023-10-07T09:42:12.121547+0000","last_undegraded":"2023-10-07T09:42:12.121547+0000","last_fullsized":"2023-10-07T09:42:12.121547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:14:22.949422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.244936+0000","last_change":"2023-10-07T09:42:11.238654+0000","last_active":"2023-10-07T09:42:12.244936+0000","last_peered":"2023-10-07T09:42:12.244936+0000","last_clean":"2023-10-07T09:42:12.244936+0000","last_became_active":"2023-10-07T09:42:11.238372+0000","last_became_peered":"2023-10-07T09:42:11.238372+0000","last_unstale":"2023-10-07T09:42:12.244936+0000","last_undegraded":"2023-10-07T09:42:12.244936+0000","last_fullsized":"2023-10-07T09:42:12.244936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:50.986059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.121532+0000","last_change":"2023-10-07T09:42:12.121532+0000","last_active":"2023-10-07T09:42:12.121532+0000","last_peered":"2023-10-07T09:42:12.121532+0000","last_clean":"2023-10-07T09:42:12.121532+0000","last_became_active":"2023-10-07T09:42:12.121265+0000","last_became_peered":"2023-10-07T09:42:12.121265+0000","last_unstale":"2023-10-07T09:42:12.121532+0000","last_undegraded":"2023-10-07T09:42:12.121532+0000","last_fullsized":"2023-10-07T09:42:12.121532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:33.129125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243645+0000","last_change":"2023-10-07T09:42:11.241017+0000","last_active":"2023-10-07T09:42:11.243645+0000","last_peered":"2023-10-07T09:42:11.243645+0000","last_clean":"2023-10-07T09:42:11.243645+0000","last_became_active":"2023-10-07T09:42:11.240733+0000","last_became_peered":"2023-10-07T09:42:11.240733+0000","last_unstale":"2023-10-07T09:42:11.243645+0000","last_undegraded":"2023-10-07T09:42:11.243645+0000","last_fullsized":"2023-10-07T09:42:11.243645+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:56.858507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.125039+0000","last_change":"2023-10-07T09:42:12.125039+0000","last_active":"2023-10-07T09:42:12.125039+0000","last_peered":"2023-10-07T09:42:12.125039+0000","last_clean":"2023-10-07T09:42:12.125039+0000","last_became_active":"2023-10-07T09:42:12.124786+0000","last_became_peered":"2023-10-07T09:42:12.124786+0000","last_unstale":"2023-10-07T09:42:12.125039+0000","last_undegraded":"2023-10-07T09:42:12.125039+0000","last_fullsized":"2023-10-07T09:42:12.125039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:49:08.388551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.239876+0000","last_change":"2023-10-07T09:42:11.238082+0000","last_active":"2023-10-07T09:42:11.239876+0000","last_peered":"2023-10-07T09:42:11.239876+0000","last_clean":"2023-10-07T09:42:11.239876+0000","last_became_active":"2023-10-07T09:42:11.237722+0000","last_became_peered":"2023-10-07T09:42:11.237722+0000","last_unstale":"2023-10-07T09:42:11.239876+0000","last_undegraded":"2023-10-07T09:42:11.239876+0000","last_fullsized":"2023-10-07T09:42:11.239876+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:35.007910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.119775+0000","last_change":"2023-10-07T09:42:12.119775+0000","last_active":"2023-10-07T09:42:12.119775+0000","last_peered":"2023-10-07T09:42:12.119775+0000","last_clean":"2023-10-07T09:42:12.119775+0000","last_became_active":"2023-10-07T09:42:12.119428+0000","last_became_peered":"2023-10-07T09:42:12.119428+0000","last_unstale":"2023-10-07T09:42:12.119775+0000","last_undegraded":"2023-10-07T09:42:12.119775+0000","last_fullsized":"2023-10-07T09:42:12.119775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:30.097280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243981+0000","last_change":"2023-10-07T09:42:11.241886+0000","last_active":"2023-10-07T09:42:11.243981+0000","last_peered":"2023-10-07T09:42:11.243981+0000","last_clean":"2023-10-07T09:42:11.243981+0000","last_became_active":"2023-10-07T09:42:11.241351+0000","last_became_peered":"2023-10-07T09:42:11.241351+0000","last_unstale":"2023-10-07T09:42:11.243981+0000","last_undegraded":"2023-10-07T09:42:11.243981+0000","last_fullsized":"2023-10-07T09:42:11.243981+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:29.975640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.133020+0000","last_change":"2023-10-07T09:42:12.133020+0000","last_active":"2023-10-07T09:42:12.133020+0000","last_peered":"2023-10-07T09:42:12.133020+0000","last_clean":"2023-10-07T09:42:12.133020+0000","last_became_active":"2023-10-07T09:42:12.132725+0000","last_became_peered":"2023-10-07T09:42:12.132725+0000","last_unstale":"2023-10-07T09:42:12.133020+0000","last_undegraded":"2023-10-07T09:42:12.133020+0000","last_fullsized":"2023-10-07T09:42:12.133020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:27.390563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.238249+0000","last_change":"2023-10-07T09:42:11.236763+0000","last_active":"2023-10-07T09:42:11.238249+0000","last_peered":"2023-10-07T09:42:11.238249+0000","last_clean":"2023-10-07T09:42:11.238249+0000","last_became_active":"2023-10-07T09:42:11.236456+0000","last_became_peered":"2023-10-07T09:42:11.236456+0000","last_unstale":"2023-10-07T09:42:11.238249+0000","last_undegraded":"2023-10-07T09:42:11.238249+0000","last_fullsized":"2023-10-07T09:42:11.238249+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:17:01.213995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120292+0000","last_change":"2023-10-07T09:42:12.120292+0000","last_active":"2023-10-07T09:42:12.120292+0000","last_peered":"2023-10-07T09:42:12.120292+0000","last_clean":"2023-10-07T09:42:12.120292+0000","last_became_active":"2023-10-07T09:42:12.119970+0000","last_became_peered":"2023-10-07T09:42:12.119970+0000","last_unstale":"2023-10-07T09:42:12.120292+0000","last_undegraded":"2023-10-07T09:42:12.120292+0000","last_fullsized":"2023-10-07T09:42:12.120292+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:17:02.220133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243884+0000","last_change":"2023-10-07T09:42:11.241971+0000","last_active":"2023-10-07T09:42:11.243884+0000","last_peered":"2023-10-07T09:42:11.243884+0000","last_clean":"2023-10-07T09:42:11.243884+0000","last_became_active":"2023-10-07T09:42:11.241669+0000","last_became_peered":"2023-10-07T09:42:11.241669+0000","last_unstale":"2023-10-07T09:42:11.243884+0000","last_undegraded":"2023-10-07T09:42:11.243884+0000","last_fullsized":"2023-10-07T09:42:11.243884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:06.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123539+0000","last_change":"2023-10-07T09:42:12.123539+0000","last_active":"2023-10-07T09:42:12.123539+0000","last_peered":"2023-10-07T09:42:12.123539+0000","last_clean":"2023-10-07T09:42:12.123539+0000","last_became_active":"2023-10-07T09:42:12.122756+0000","last_became_peered":"2023-10-07T09:42:12.122756+0000","last_unstale":"2023-10-07T09:42:12.123539+0000","last_undegraded":"2023-10-07T09:42:12.123539+0000","last_fullsized":"2023-10-07T09:42:12.123539+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:14:48.484141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.245026+0000","last_change":"2023-10-07T09:42:11.244235+0000","last_active":"2023-10-07T09:42:11.245026+0000","last_peered":"2023-10-07T09:42:11.245026+0000","last_clean":"2023-10-07T09:42:11.245026+0000","last_became_active":"2023-10-07T09:42:11.243833+0000","last_became_peered":"2023-10-07T09:42:11.243833+0000","last_unstale":"2023-10-07T09:42:11.245026+0000","last_undegraded":"2023-10-07T09:42:11.245026+0000","last_fullsized":"2023-10-07T09:42:11.245026+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:21.050337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.119487+0000","last_change":"2023-10-07T09:42:12.119487+0000","last_active":"2023-10-07T09:42:12.119487+0000","last_peered":"2023-10-07T09:42:12.119487+0000","last_clean":"2023-10-07T09:42:12.119487+0000","last_became_active":"2023-10-07T09:42:12.119195+0000","last_became_peered":"2023-10-07T09:42:12.119195+0000","last_unstale":"2023-10-07T09:42:12.119487+0000","last_undegraded":"2023-10-07T09:42:12.119487+0000","last_fullsized":"2023-10-07T09:42:12.119487+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:05.782312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.243948+0000","last_change":"2023-10-07T09:42:11.241981+0000","last_active":"2023-10-07T09:42:11.243948+0000","last_peered":"2023-10-07T09:42:11.243948+0000","last_clean":"2023-10-07T09:42:11.243948+0000","last_became_active":"2023-10-07T09:42:11.241639+0000","last_became_peered":"2023-10-07T09:42:11.241639+0000","last_unstale":"2023-10-07T09:42:11.243948+0000","last_undegraded":"2023-10-07T09:42:11.243948+0000","last_fullsized":"2023-10-07T09:42:11.243948+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:14.416930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.123049+0000","last_change":"2023-10-07T09:42:12.123049+0000","last_active":"2023-10-07T09:42:12.123049+0000","last_peered":"2023-10-07T09:42:12.123049+0000","last_clean":"2023-10-07T09:42:12.123049+0000","last_became_active":"2023-10-07T09:42:12.122731+0000","last_became_peered":"2023-10-07T09:42:12.122731+0000","last_unstale":"2023-10-07T09:42:12.123049+0000","last_undegraded":"2023-10-07T09:42:12.123049+0000","last_fullsized":"2023-10-07T09:42:12.123049+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:31:39.819015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.240040+0000","last_change":"2023-10-07T09:42:11.234988+0000","last_active":"2023-10-07T09:42:11.240040+0000","last_peered":"2023-10-07T09:42:11.240040+0000","last_clean":"2023-10-07T09:42:11.240040+0000","last_became_active":"2023-10-07T09:42:11.228153+0000","last_became_peered":"2023-10-07T09:42:11.228153+0000","last_unstale":"2023-10-07T09:42:11.240040+0000","last_undegraded":"2023-10-07T09:42:11.240040+0000","last_fullsized":"2023-10-07T09:42:11.240040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:51:43.139812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.120650+0000","last_change":"2023-10-07T09:42:12.120650+0000","last_active":"2023-10-07T09:42:12.120650+0000","last_peered":"2023-10-07T09:42:12.120650+0000","last_clean":"2023-10-07T09:42:12.120650+0000","last_became_active":"2023-10-07T09:42:12.120358+0000","last_became_peered":"2023-10-07T09:42:12.120358+0000","last_unstale":"2023-10-07T09:42:12.120650+0000","last_undegraded":"2023-10-07T09:42:12.120650+0000","last_fullsized":"2023-10-07T09:42:12.120650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:14:22.949422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.233933+0000","last_change":"2023-10-07T09:42:11.232753+0000","last_active":"2023-10-07T09:42:11.233933+0000","last_peered":"2023-10-07T09:42:11.233933+0000","last_clean":"2023-10-07T09:42:11.233933+0000","last_became_active":"2023-10-07T09:42:11.232482+0000","last_became_peered":"2023-10-07T09:42:11.232482+0000","last_unstale":"2023-10-07T09:42:11.233933+0000","last_undegraded":"2023-10-07T09:42:11.233933+0000","last_fullsized":"2023-10-07T09:42:11.233933+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:51:48.560303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.118362+0000","last_change":"2023-10-07T09:42:12.118362+0000","last_active":"2023-10-07T09:42:12.118362+0000","last_peered":"2023-10-07T09:42:12.118362+0000","last_clean":"2023-10-07T09:42:12.118362+0000","last_became_active":"2023-10-07T09:42:12.118076+0000","last_became_peered":"2023-10-07T09:42:12.118076+0000","last_unstale":"2023-10-07T09:42:12.118362+0000","last_undegraded":"2023-10-07T09:42:12.118362+0000","last_fullsized":"2023-10-07T09:42:12.118362+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:28.035281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.235021+0000","last_change":"2023-10-07T09:42:11.229814+0000","last_active":"2023-10-07T09:42:11.235021+0000","last_peered":"2023-10-07T09:42:11.235021+0000","last_clean":"2023-10-07T09:42:11.235021+0000","last_became_active":"2023-10-07T09:42:11.229537+0000","last_became_peered":"2023-10-07T09:42:11.229537+0000","last_unstale":"2023-10-07T09:42:11.235021+0000","last_undegraded":"2023-10-07T09:42:11.235021+0000","last_fullsized":"2023-10-07T09:42:11.235021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:00:19.787456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":8,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.127165+0000","last_change":"2023-10-07T09:42:12.127165+0000","last_active":"2023-10-07T09:42:12.127165+0000","last_peered":"2023-10-07T09:42:12.127165+0000","last_clean":"2023-10-07T09:42:12.127165+0000","last_became_active":"2023-10-07T09:42:12.126859+0000","last_became_peered":"2023-10-07T09:42:12.126859+0000","last_unstale":"2023-10-07T09:42:12.127165+0000","last_undegraded":"2023-10-07T09:42:12.127165+0000","last_fullsized":"2023-10-07T09:42:12.127165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:11.210808+0000","last_clean_scrub_stamp":"2023-10-07T09:42:11.210808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:57.932346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"60'2","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-07T09:42:12.244797+0000","last_change":"2023-10-07T09:42:11.237671+0000","last_active":"2023-10-07T09:42:12.244797+0000","last_peered":"2023-10-07T09:42:12.244797+0000","last_clean":"2023-10-07T09:42:12.244797+0000","last_became_active":"2023-10-07T09:42:11.237419+0000","last_became_peered":"2023-10-07T09:42:11.237419+0000","last_unstale":"2023-10-07T09:42:12.244797+0000","last_undegraded":"2023-10-07T09:42:12.244797+0000","last_fullsized":"2023-10-07T09:42:12.244797+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:42:10.204669+0000","last_clean_scrub_stamp":"2023-10-07T09:42:10.204669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:01:06.508994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"50'139","reported_seq":273,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-10-07T09:42:11.233842+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:42:11.233842+0000","last_peered":"2023-10-07T09:42:11.233842+0000","last_clean":"2023-10-07T09:42:11.233842+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:42:11.233842+0000","last_undegraded":"2023-10-07T09:42:11.233842+0000","last_fullsized":"2023-10-07T09:42:11.233842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302640,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069200,"statfs":{"total":96636764160,"available":96326860800,"internally_reserved":0,"allocated":1286144,"data_stored":905976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.70599999999999996},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.32300000000000001},"max":{"1min":0.876,"5min":0.998,"15min":1.026},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.73299999999999998,"15min":0.69599999999999995},"min":{"1min":0.56000000000000005,"5min":0.374,"15min":0.374},"max":{"1min":0.93899999999999995,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.88}]},{"osd":1,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.51600000000000001,"5min":0.45200000000000001,"15min":0.34599999999999997},"max":{"1min":0.875,"5min":1.014,"15min":1.157},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.72399999999999998},"min":{"1min":0.49099999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.998,"5min":1.002,"15min":1.002},"last":0.75700000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76300000000000001,"15min":0.73799999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40699999999999997},"max":{"1min":0.93500000000000005,"5min":0.97699999999999998,"15min":1.0289999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.74099999999999999,"15min":0.72499999999999998},"min":{"1min":0.58199999999999996,"5min":0.52100000000000002,"15min":0.439},"max":{"1min":0.86299999999999999,"5min":1.071,"15min":1.071},"last":0.83099999999999996}]},{"osd":3,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.73099999999999998},"min":{"1min":0.55200000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91600000000000004,"5min":0.96899999999999997,"15min":1.04},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.753,"15min":0.76100000000000001},"min":{"1min":0.57299999999999995,"5min":0.53000000000000003,"15min":0.371},"max":{"1min":0.90200000000000002,"5min":1.006,"15min":1.3440000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59199999999999997,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"max":{"1min":0.85499999999999998,"5min":0.88700000000000001,"15min":1.992},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.40200000000000002},"max":{"1min":0.84199999999999997,"5min":0.92900000000000005,"15min":1.6240000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63600000000000001,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.0900000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.70799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90600000000000003,"5min":0.94899999999999995,"15min":1.1399999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.56299999999999994,"5min":0.499,"15min":0.46400000000000002},"max":{"1min":0.89300000000000002,"5min":0.97999999999999998,"15min":1.0289999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.71799999999999997,"15min":0.73799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.96899999999999997,"15min":1.107},"last":0.67800000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509599,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.71899999999999997,"15min":0.76100000000000001},"min":{"1min":0.51200000000000001,"5min":0.47499999999999998,"15min":0.39300000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":2.984},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.70299999999999996,"15min":0.69899999999999995},"min":{"1min":0.51700000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85899999999999999,"5min":0.94599999999999995,"15min":1.6000000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72499999999999998,"15min":0.73599999999999999},"min":{"1min":0.52400000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93400000000000005,"5min":1.0149999999999999,"15min":2.4529999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73499999999999999,"15min":0.73899999999999999},"min":{"1min":0.56899999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.99199999999999999,"15min":1.754},"last":0.65800000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73299999999999998,"15min":0.81000000000000005},"min":{"1min":0.59499999999999997,"5min":0.45700000000000002,"15min":0.44400000000000001},"max":{"1min":0.88,"5min":1.002,"15min":7.867},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76800000000000002,"15min":0.84399999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92900000000000005,"5min":0.99199999999999999,"15min":7.7629999999999999},"last":0.68899999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76800000000000002,"15min":0.85599999999999998},"min":{"1min":0.54600000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.86799999999999999,"5min":0.999,"15min":8.0399999999999991},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.752,"15min":0.79600000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.435},"max":{"1min":0.93500000000000005,"5min":0.98099999999999998,"15min":2.7949999999999999},"last":0.78900000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61699999999999999,"15min":0.63500000000000001},"min":{"1min":0.46100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80400000000000005,"5min":1.01,"15min":1.296},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.67600000000000005},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.91100000000000003,"5min":0.92500000000000004,"15min":1.105},"last":0.76700000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91200000000000003,"5min":1.0509999999999999,"15min":1.4119999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.52200000000000002,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":1.0449999999999999,"15min":1.1779999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Sat Oct 7 09:41:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67700000000000005,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97499999999999998,"5min":1.0149999999999999,"15min":1.5049999999999999},"last":0.751},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":0.77900000000000003},"min":{"1min":0.59499999999999997,"5min":0.51700000000000002,"15min":0.48899999999999999},"max":{"1min":0.95799999999999996,"5min":1.002,"15min":1.3220000000000001},"last":0.80500000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302076,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069764,"statfs":{"total":96636764160,"available":96327438336,"internally_reserved":0,"allocated":708608,"data_stored":322789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65600000000000003,"5min":0.69699999999999995,"15min":1.585},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.29299999999999998},"max":{"1min":0.61899999999999999,"5min":0.84199999999999997,"15min":1.6080000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.60899999999999999,"5min":0.79600000000000004,"15min":1.468},"last":0.5},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77800000000000002,"5min":0.82499999999999996,"15min":1.5389999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59399999999999997,"15min":0.61299999999999999},"min":{"1min":0.437,"5min":0.377,"15min":0.34699999999999998},"max":{"1min":0.67200000000000004,"5min":0.78700000000000003,"15min":1.663},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":0.81399999999999995,"15min":1.1140000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.78000000000000003,"15min":0.78300000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.89400000000000002,"5min":1.2629999999999999,"15min":1.4259999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.54100000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":1.1120000000000001,"15min":1.766},"last":0.67200000000000004}]},{"osd":5,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.82699999999999996,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.54400000000000004},"max":{"1min":0.98199999999999998,"5min":1.177,"15min":1.3320000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.78900000000000003,"15min":0.79400000000000004},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.54400000000000004},"max":{"1min":0.92700000000000005,"5min":1.127,"15min":1.5149999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":0.83399999999999996},"min":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.57099999999999995},"max":{"1min":0.92000000000000004,"5min":1.091,"15min":1.627},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.83099999999999996,"15min":0.83299999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94799999999999995,"5min":1.1080000000000001,"15min":1.3700000000000001},"last":0.73199999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.83899999999999997,"15min":0.85199999999999998},"min":{"1min":0.622,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.90600000000000003,"5min":1.2050000000000001,"15min":1.706},"last":0.878},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.86599999999999999,"15min":0.85799999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.58399999999999996},"max":{"1min":0.98399999999999999,"5min":1.103,"15min":1.554},"last":0.65500000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542301,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302652,"kb_used_data":1268,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069188,"statfs":{"total":96636764160,"available":96326848512,"internally_reserved":0,"allocated":1298432,"data_stored":913087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.313,"15min":0.30299999999999999},"max":{"1min":0.63,"5min":0.74399999999999999,"15min":2.726},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":2.552},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.32800000000000001},"max":{"1min":0.66700000000000004,"5min":1.101,"15min":2.4910000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.50700000000000001,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":3.629},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"max":{"1min":0.69099999999999995,"5min":1.0009999999999999,"15min":2.6909999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.62,"15min":0.63800000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40200000000000002},"max":{"1min":0.80900000000000005,"5min":0.89500000000000002,"15min":2.5209999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77100000000000002,"15min":0.79100000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.45900000000000002},"max":{"1min":1.1850000000000001,"5min":1.232,"15min":2.637},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.79400000000000004,"15min":0.82299999999999995},"min":{"1min":0.58899999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"max":{"1min":0.95999999999999996,"5min":1.117,"15min":2.4180000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.79800000000000004,"15min":0.84899999999999998},"min":{"1min":0.52400000000000002,"5min":0.51300000000000001,"15min":0.501},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":3.8199999999999998},"last":0.874},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.83299999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55700000000000005},"max":{"1min":1.0920000000000001,"5min":1.194,"15min":2.6000000000000001},"last":0.69099999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.66500000000000004,"5min":0.60699999999999998,"15min":0.60499999999999998},"max":{"1min":1.129,"5min":1.129,"15min":2.3740000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.83099999999999996,"15min":0.877},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53600000000000003},"max":{"1min":1.103,"5min":1.103,"15min":3.855},"last":0.82499999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.84899999999999998,"15min":0.85899999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.56200000000000006},"max":{"1min":1.167,"5min":1.167,"15min":2.4550000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.84299999999999997,"15min":0.89800000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0609999999999999,"5min":1.177,"15min":3.778},"last":0.71699999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306156,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065684,"statfs":{"total":96636764160,"available":96323260416,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47399999999999998,"15min":0.45600000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.81000000000000005,"5min":1.258,"15min":1.258},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53600000000000003,"15min":0.497},"min":{"1min":0.314,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.72599999999999998,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.55200000000000005},"min":{"1min":0.39300000000000002,"5min":0.38100000000000001,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.45100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81799999999999995,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.94599999999999995,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.432},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61699999999999999,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81399999999999995,"5min":1.052,"15min":1.052},"last":0.38800000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:41:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.76000000000000001},"min":{"1min":0.48199999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0369999999999999,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.82199999999999995,"15min":0.79000000000000004},"min":{"1min":0.54400000000000004,"5min":0.49399999999999999,"15min":0.42099999999999999},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85499999999999998,"15min":0.81100000000000005},"min":{"1min":0.622,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80500000000000005,"15min":0.78900000000000003},"min":{"1min":0.56499999999999995,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.93899999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:41:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.86099999999999999,"15min":0.82399999999999995},"min":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.82899999999999996},"min":{"1min":0.60299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.306,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.872}]},{"osd":7,"last update":"Sat Oct 7 09:41:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.83499999999999996},"min":{"1min":0.60699999999999998,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85999999999999999,"15min":0.82299999999999995},"min":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54200000000000004},"max":{"1min":1.3520000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.75700000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306180,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065660,"statfs":{"total":96636764160,"available":96323235840,"internally_reserved":0,"allocated":716800,"data_stored":323329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45900000000000002,"15min":0.47599999999999998},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.25600000000000001},"max":{"1min":0.61399999999999999,"5min":0.77200000000000002,"15min":3.3980000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.64200000000000002,"5min":0.73199999999999998,"15min":3.274},"last":0.379}]},{"osd":1,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.53000000000000003},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.61099999999999999,"5min":0.67600000000000005,"15min":3.1299999999999999},"last":0.441},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.47299999999999998,"5min":0.42499999999999999,"15min":0.38200000000000001},"max":{"1min":0.70599999999999996,"5min":0.84099999999999997,"15min":3.4500000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":0.68600000000000005,"5min":0.86599999999999999,"15min":3.2309999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.79800000000000004,"15min":3.0910000000000002},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56699999999999995,"5min":0.51600000000000001,"15min":0.44900000000000001},"max":{"1min":1.0389999999999999,"5min":1.145,"15min":1.518},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78200000000000003,"15min":0.78700000000000003},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.48099999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.3799999999999999},"last":0.90100000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.59099999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.3240000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.50600000000000001},"max":{"1min":0.95599999999999996,"5min":1.1899999999999999,"15min":1.5189999999999999},"last":0.87}]},{"osd":6,"last update":"Sat Oct 7 09:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80500000000000005,"15min":0.80600000000000005},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0049999999999999,"5min":1.125,"15min":1.413},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.81899999999999995,"15min":0.82299999999999995},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.51000000000000001},"max":{"1min":1.1459999999999999,"5min":1.173,"15min":1.5840000000000001},"last":0.93500000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.82199999999999995,"15min":0.81699999999999995},"min":{"1min":0.59899999999999998,"5min":0.57299999999999995,"15min":0.54700000000000004},"max":{"1min":1.0980000000000001,"5min":1.163,"15min":1.45},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83399999999999996,"15min":0.83699999999999997},"min":{"1min":0.63100000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0660000000000001,"5min":1.1439999999999999,"15min":1.5940000000000001},"last":0.80000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509599,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306156,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065684,"statfs":{"total":96636764160,"available":96323260416,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.71799999999999997,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.39600000000000002,"15min":0.30399999999999999},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.72699999999999998,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":2.3420000000000001},"last":0.80300000000000005}]},{"osd":1,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.745,"15min":0.70199999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.379},"max":{"1min":0.96599999999999997,"5min":1.04,"15min":1.054},"last":0.62},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.752,"15min":0.71599999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99099999999999999,"5min":1.046,"15min":1.2749999999999999},"last":0.748}]},{"osd":2,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76600000000000001,"15min":0.75900000000000001},"min":{"1min":0.64700000000000002,"5min":0.48799999999999999,"15min":0.432},"max":{"1min":0.97399999999999998,"5min":1.006,"15min":2.3239999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74399999999999999,"15min":0.70699999999999996},"min":{"1min":0.56799999999999995,"5min":0.497,"15min":0.40000000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.71599999999999997},"min":{"1min":0.53800000000000003,"5min":0.52300000000000002,"15min":0.496},"max":{"1min":0.96899999999999997,"5min":0.98699999999999999,"15min":1.032},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.745},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.496},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.75700000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.625,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.625,"15min":0.64500000000000002},"min":{"1min":0.49199999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.004},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.624,"15min":0.64500000000000002},"min":{"1min":0.44400000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"max":{"1min":0.93200000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.71699999999999997},"min":{"1min":0.51400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0049999999999999,"5min":1.024,"15min":1.0589999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73999999999999999,"15min":0.753},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.0209999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.71299999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":692224,"data_stored":315608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.75900000000000001,"15min":0.74399999999999999},"min":{"1min":0.49199999999999999,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.91900000000000004,"5min":0.96799999999999997,"15min":1.1759999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.78900000000000003},"min":{"1min":0.53000000000000003,"5min":0.436,"15min":0.436},"max":{"1min":0.91500000000000004,"5min":1.103,"15min":1.151},"last":0.89900000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.73999999999999999,"15min":0.72299999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":0.91800000000000004,"5min":1.02,"15min":1.3049999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.78900000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":0.88300000000000001,"5min":0.98199999999999998,"15min":6.9720000000000004},"last":0.84899999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.78200000000000003},"min":{"1min":0.53500000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92600000000000005,"5min":0.97899999999999998,"15min":5.9299999999999997},"last":0.88},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.52200000000000002,"5min":0.47499999999999998,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":1.036,"15min":1.704},"last":0.96699999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.48899999999999999,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":1,"15min":1.899},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.749,"15min":0.81899999999999995},"min":{"1min":0.622,"5min":0.51200000000000001,"15min":0.50600000000000001},"max":{"1min":1.016,"5min":1.016,"15min":6.0620000000000003},"last":0.82299999999999995}]},{"osd":4,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.374},"max":{"1min":0.85199999999999998,"5min":0.92100000000000004,"15min":0.97199999999999998},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63500000000000001,"15min":0.63200000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.39400000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.73699999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.39800000000000002,"5min":0.38900000000000001,"15min":0.33900000000000002},"max":{"1min":0.88,"5min":0.998,"15min":0.998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.70699999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.93500000000000005,"5min":0.98999999999999999,"15min":1.032},"last":0.76500000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:41:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70099999999999996,"15min":0.70499999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.0069999999999999,"15min":1.0189999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77100000000000002,"15min":0.76300000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.96099999999999997,"5min":1.0529999999999999,"15min":1.0649999999999999},"last":0.94499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:42:17.124 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:42:17.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:42:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.360+0000 7f614d5ff700 1 Processor -- start 2023-10-07T09:42:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.361+0000 7f614d5ff700 1 -- start start 2023-10-07T09:42:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f614d5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 0x7f61480cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f614d5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f61480cfbb0 0x7f61480d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f614d5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480c62d0 con 0x7f61480c6760 2023-10-07T09:42:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f614d5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6148159e10 con 0x7f61480cfbb0 2023-10-07T09:42:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f614d5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61480cc110 con 0x7f61480cb800 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f6146ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 0x7f61480cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f61467fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f61480cfbb0 0x7f61480d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f61467fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f61480cfbb0 0x7f61480d0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:49150/0 (socket says 172.21.15.29:49150) 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f61467fc700 1 -- 172.21.15.29:0/2077757888 learned_addr learned my addr 172.21.15.29:0/2077757888 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.362+0000 7f6146ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 0x7f61480cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:55006/0 (socket says 172.21.15.29:55006) 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.363+0000 7f61467fc700 1 -- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 msgr2=0x7f61480cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.363+0000 7f61467fc700 1 --2- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 0x7f61480cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.363+0000 7f61467fc700 1 -- 172.21.15.29:0/2077757888 >> v1:172.21.15.144:6789/0 conn(0x7f61480c6760 legacy=0x7f61480b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.363+0000 7f61467fc700 1 -- 172.21.15.29:0/2077757888 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61480c56c0 con 0x7f61480cfbb0 2023-10-07T09:42:17.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.363+0000 7f6146ffd700 1 --2- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 0x7f61480cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.363+0000 7f61467fc700 1 --2- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cfbb0 0x7f61480d0000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f613c0099b0 tx=0x7f613c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6376a45cf668c827 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f6145ffb700 1 -- 172.21.15.29:0/2077757888 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c009ea0 con 0x7f61480cfbb0 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f6145ffb700 1 -- 172.21.15.29:0/2077757888 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c00b8f0 con 0x7f61480cfbb0 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f6145ffb700 1 -- 172.21.15.29:0/2077757888 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00bc90 con 0x7f61480cfbb0 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 -- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cfbb0 msgr2=0x7f61480d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 --2- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cfbb0 0x7f61480d0000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f613c0099b0 tx=0x7f613c009250 comp rx=0 tx=0).stop 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 -- 172.21.15.29:0/2077757888 shutdown_connections 2023-10-07T09:42:17.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 --2- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cb800 0x7f61480cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 --2- 172.21.15.29:0/2077757888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cfbb0 0x7f61480d0000 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 -- 172.21.15.29:0/2077757888 >> 172.21.15.29:0/2077757888 conn(0x7f6148063a10 msgr2=0x7f61480d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 -- 172.21.15.29:0/2077757888 shutdown_connections 2023-10-07T09:42:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.364+0000 7f614d5ff700 1 -- 172.21.15.29:0/2077757888 wait complete. 2023-10-07T09:42:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.366+0000 7f614d5ff700 1 Processor -- start 2023-10-07T09:42:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.366+0000 7f614d5ff700 1 -- start start 2023-10-07T09:42:17.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f614d5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 0x7f61480b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f614d5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cb800 0x7f614816dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f614d5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cfbb0 0x7f6148155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f614d5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61480ca9f0 con 0x7f61480c6760 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f614d5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61480c7f90 con 0x7f61480cb800 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f614d5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6148155fa0 con 0x7f61480cfbb0 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f6146ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 0x7f61480b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f6146ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 0x7f61480b5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59056/0 (socket says 172.21.15.29:59056) 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f6146ffd700 1 -- 172.21.15.29:0/4019831334 learned_addr learned my addr 172.21.15.29:0/4019831334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.367+0000 7f61467fc700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cb800 0x7f614816dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f61477fe700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cfbb0 0x7f6148155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f6146ffd700 1 -- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cfbb0 msgr2=0x7f6148155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f6146ffd700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cfbb0 0x7f6148155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f6146ffd700 1 -- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cb800 msgr2=0x7f614816dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f6146ffd700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cb800 0x7f614816dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f6146ffd700 1 -- 172.21.15.29:0/4019831334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380093c0 con 0x7f61480c6760 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f61477fe700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cfbb0 0x7f6148155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f6146ffd700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 0x7f61480b5f50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f613800ec80 tx=0x7f6138009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138005a00 con 0x7f61480c6760 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138005b60 con 0x7f61480c6760 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c0094f0 con 0x7f61480c6760 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613800c9d0 con 0x7f61480c6760 2023-10-07T09:42:17.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.368+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61480c7910 con 0x7f61480c6760 2023-10-07T09:42:17.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.369+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61480c8550 con 0x7f61480c6760 2023-10-07T09:42:17.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.370+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f613800cd80 con 0x7f61480c6760 2023-10-07T09:42:17.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.371+0000 7f612effd700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6134066f30 0x7f61340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.371+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6138084d10 con 0x7f61480c6760 2023-10-07T09:42:17.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.372+0000 7f61467fc700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6134066f30 0x7f61340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.372+0000 7f61467fc700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6134066f30 0x7f61340693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f613c0099b0 tx=0x7f613c017000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.375+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61480c8550 con 0x7f61480c6760 2023-10-07T09:42:17.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.659+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f61480c8550 con 0x7f61480c6760 2023-10-07T09:42:17.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.663+0000 7f612effd700 1 -- 172.21.15.29:0/4019831334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f61480c8550 con 0x7f61480c6760 2023-10-07T09:42:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6134066f30 msgr2=0x7f61340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6134066f30 0x7f61340693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f613c0099b0 tx=0x7f613c017000 comp rx=0 tx=0).stop 2023-10-07T09:42:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 msgr2=0x7f61480b5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 0x7f61480b5f50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f613800ec80 tx=0x7f6138009120 comp rx=0 tx=0).stop 2023-10-07T09:42:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 shutdown_connections 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6134066f30 0x7f61340693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61480cfbb0 0x7f6148155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61480c6760 0x7f61480b5f50 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 --2- 172.21.15.29:0/4019831334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61480cb800 0x7f614816dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 >> 172.21.15.29:0/4019831334 conn(0x7f6148063a10 msgr2=0x7f61480caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 shutdown_connections 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.666+0000 7f614d5ff700 1 -- 172.21.15.29:0/4019831334 wait complete. 2023-10-07T09:42:17.668 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:42:17.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:42:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.879+0000 7fc1633ff700 1 Processor -- start 2023-10-07T09:42:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.879+0000 7fc1633ff700 1 -- start start 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc1633ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c0b04b0 0x7fc15c156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc1633ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc15c1537c0 0x7fc15c157240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc1633ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15c159670 con 0x7fc15c1533f0 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc1633ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15c15a680 con 0x7fc15c1537c0 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc1633ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0b2070 con 0x7fc15c0b04b0 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc16119b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc15c1533f0 0x7fc15c0afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37094/0 (socket says 172.21.15.29:37094) 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.880+0000 7fc16119b700 1 -- 172.21.15.29:0/1886548371 learned_addr learned my addr 172.21.15.29:0/1886548371 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16199c700 1 --2- 172.21.15.29:0/1886548371 >> v2:172.21.15.144:3300/0 conn(0x7fc15c1537c0 0x7fc15c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16099a700 1 --2- 172.21.15.29:0/1886548371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c0b04b0 0x7fc15c156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc153fff700 1 -- 172.21.15.29:0/1886548371 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148318048 0 0) 0x7fc15c159670 con 0x7fc15c1533f0 2023-10-07T09:42:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc153fff700 1 -- 172.21.15.29:0/1886548371 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc148003570 con 0x7fc15c1533f0 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc153fff700 1 -- 172.21.15.29:0/1886548371 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325641660 0 0) 0x7fc148003570 con 0x7fc15c1533f0 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16099a700 1 -- 172.21.15.29:0/1886548371 >> v2:172.21.15.144:3300/0 conn(0x7fc15c1537c0 msgr2=0x7fc15c157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16099a700 1 --2- 172.21.15.29:0/1886548371 >> v2:172.21.15.144:3300/0 conn(0x7fc15c1537c0 0x7fc15c157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16099a700 1 -- 172.21.15.29:0/1886548371 >> v1:172.21.15.29:6789/0 conn(0x7fc15c1533f0 legacy=0x7fc15c0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16099a700 1 -- 172.21.15.29:0/1886548371 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c15a1f0 con 0x7fc15c0b04b0 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.881+0000 7fc16199c700 1 --2- 172.21.15.29:0/1886548371 >> v2:172.21.15.144:3300/0 conn(0x7fc15c1537c0 0x7fc15c157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc16099a700 1 --2- 172.21.15.29:0/1886548371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c0b04b0 0x7fc15c156a90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0098e0 tx=0x7fc14c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0ed0a3cf0cff63 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc153fff700 1 -- 172.21.15.29:0/1886548371 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158001470 con 0x7fc15c0b04b0 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc153fff700 1 -- 172.21.15.29:0/1886548371 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c011420 con 0x7fc15c0b04b0 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc153fff700 1 -- 172.21.15.29:0/1886548371 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c0117a0 con 0x7fc15c0b04b0 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc1633ff700 1 -- 172.21.15.29:0/1886548371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c0b04b0 msgr2=0x7fc15c156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc1633ff700 1 --2- 172.21.15.29:0/1886548371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c0b04b0 0x7fc15c156a90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0098e0 tx=0x7fc14c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc1633ff700 1 -- 172.21.15.29:0/1886548371 shutdown_connections 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc1633ff700 1 --2- 172.21.15.29:0/1886548371 >> v2:172.21.15.144:3300/0 conn(0x7fc15c1537c0 0x7fc15c157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.882+0000 7fc1633ff700 1 --2- 172.21.15.29:0/1886548371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c0b04b0 0x7fc15c156a90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.883+0000 7fc1633ff700 1 -- 172.21.15.29:0/1886548371 >> 172.21.15.29:0/1886548371 conn(0x7fc15c0680a0 msgr2=0x7fc15c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.883+0000 7fc1633ff700 1 -- 172.21.15.29:0/1886548371 shutdown_connections 2023-10-07T09:42:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.883+0000 7fc1633ff700 1 -- 172.21.15.29:0/1886548371 wait complete. 2023-10-07T09:42:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.884+0000 7fc1633ff700 1 Processor -- start 2023-10-07T09:42:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.884+0000 7fc1633ff700 1 -- start start 2023-10-07T09:42:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc1633ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 0x7fc15c143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc1633ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c1533f0 0x7fc15c1518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc1633ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15c1537c0 0x7fc15c14dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc1633ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c14b180 con 0x7fc15c0b04b0 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc1633ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c14b3d0 con 0x7fc15c1537c0 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc1633ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c14b6b0 con 0x7fc15c1533f0 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc16119b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 0x7fc15c143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc16119b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 0x7fc15c143e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59070/0 (socket says 172.21.15.29:59070) 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.885+0000 7fc16119b700 1 -- 172.21.15.29:0/2994261231 learned_addr learned my addr 172.21.15.29:0/2994261231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc16119b700 1 -- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c1533f0 msgr2=0x7fc15c1518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc16119b700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c1533f0 0x7fc15c1518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc16119b700 1 -- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15c1537c0 msgr2=0x7fc15c14dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc16119b700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15c1537c0 0x7fc15c14dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc16119b700 1 -- 172.21.15.29:0/2994261231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc154009a20 con 0x7fc15c0b04b0 2023-10-07T09:42:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc16119b700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 0x7fc15c143e30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc158001660 tx=0x7fc158001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158001d10 con 0x7fc15c0b04b0 2023-10-07T09:42:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158007360 con 0x7fc15c0b04b0 2023-10-07T09:42:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c009420 con 0x7fc15c0b04b0 2023-10-07T09:42:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158004830 con 0x7fc15c0b04b0 2023-10-07T09:42:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.886+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c14bc70 con 0x7fc15c0b04b0 2023-10-07T09:42:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.887+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c14be90 con 0x7fc15c0b04b0 2023-10-07T09:42:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.888+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc158004c20 con 0x7fc15c0b04b0 2023-10-07T09:42:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.889+0000 7fc1517fa700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc140066f30 0x7fc1400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.889+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc158083fa0 con 0x7fc15c0b04b0 2023-10-07T09:42:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.890+0000 7fc16099a700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc140066f30 0x7fc1400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.890+0000 7fc16099a700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc140066f30 0x7fc1400693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc15c1558c0 tx=0x7fc14c016000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:17.893+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc158051090 con 0x7fc15c0b04b0 2023-10-07T09:42:18.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.142+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc15c04c3c0 con 0x7fc15c0b04b0 2023-10-07T09:42:18.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.142+0000 7fc1517fa700 1 -- 172.21.15.29:0/2994261231 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4600 (secure 0 0 0) 0x7fc158018020 con 0x7fc15c0b04b0 2023-10-07T09:42:18.144 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:18.144 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:42:11.722170+0000","modified":"2023-10-07T09:42:15.247175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6834/2769082805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2769082805},{"type":"v1","addr":"172.21.15.29:6834","nonce":2769082805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-10-07T09:42:18.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc140066f30 msgr2=0x7fc1400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc140066f30 0x7fc1400693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc15c1558c0 tx=0x7fc14c016000 comp rx=0 tx=0).stop 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 msgr2=0x7fc15c143e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 0x7fc15c143e30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc158001660 tx=0x7fc158001a80 comp rx=0 tx=0).stop 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 shutdown_connections 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc140066f30 0x7fc1400693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15c1537c0 0x7fc15c14dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc15c0b04b0 0x7fc15c143e30 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 --2- 172.21.15.29:0/2994261231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15c1533f0 0x7fc15c1518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 >> 172.21.15.29:0/2994261231 conn(0x7fc15c0680a0 msgr2=0x7fc15c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 shutdown_connections 2023-10-07T09:42:18.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.145+0000 7fc1633ff700 1 -- 172.21.15.29:0/2994261231 wait complete. 2023-10-07T09:42:18.148 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-07T09:42:18.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:42:11.722170+0000', 'modified': '2023-10-07T09:42:15.247175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5775': {'gid': 5775, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.29:6834/2769082805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 2769082805}, {'type': 'v1', 'addr': '172.21.15.29:6834', 'nonce': 2769082805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:42:18.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:42:18.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-07T09:42:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.766+0000 7fc4ca1ff700 1 Processor -- start 2023-10-07T09:42:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.766+0000 7fc4ca1ff700 1 -- start start 2023-10-07T09:42:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.767+0000 7fc4ca1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4c415ade0 0x7fc4c40b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.767+0000 7fc4ca1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c4154090 con 0x7fc4c4153cc0 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.767+0000 7fc4ca1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c4159f60 con 0x7fc4c40b0540 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.767+0000 7fc4ca1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c40b09e0 con 0x7fc4c415ade0 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4c415ade0 0x7fc4c40b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c37fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc4c4153cc0 0x7fc4c40b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37096/0 (socket says 172.21.15.29:37096) 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4c415ade0 0x7fc4c40b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59086/0 (socket says 172.21.15.29:59086) 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c37fe700 1 -- 172.21.15.29:0/3138526118 learned_addr learned my addr 172.21.15.29:0/3138526118 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/3138526118 >> v1:172.21.15.144:6789/0 conn(0x7fc4c40b0540 legacy=0x7fc4c415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/3138526118 >> v1:172.21.15.29:6789/0 conn(0x7fc4c4153cc0 legacy=0x7fc4c40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/3138526118 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c415a240 con 0x7fc4c415ade0 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c27fc700 1 -- 172.21.15.29:0/3138526118 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360606894 0 0) 0x7fc4c4154090 con 0x7fc4c4153cc0 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.768+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/3138526118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c415ade0 0x7fc4c40b0000 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80098e0 tx=0x7fc4b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e4906d09fc4e46 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4c27fc700 1 -- 172.21.15.29:0/3138526118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8009e80 con 0x7fc4c415ade0 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4c27fc700 1 -- 172.21.15.29:0/3138526118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b800b820 con 0x7fc4c415ade0 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4c27fc700 1 -- 172.21.15.29:0/3138526118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800bba0 con 0x7fc4c415ade0 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/3138526118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c415ade0 msgr2=0x7fc4c40b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/3138526118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c415ade0 0x7fc4c40b0000 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80098e0 tx=0x7fc4b80091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/3138526118 shutdown_connections 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/3138526118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c415ade0 0x7fc4c40b0000 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/3138526118 >> 172.21.15.29:0/3138526118 conn(0x7fc4c4067610 msgr2=0x7fc4c4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/3138526118 shutdown_connections 2023-10-07T09:42:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.769+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/3138526118 wait complete. 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.771+0000 7fc4ca1ff700 1 Processor -- start 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.771+0000 7fc4ca1ff700 1 -- start start 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4ca1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 0x7fc4c4145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4ca1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c4153cc0 0x7fc4c4156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4ca1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c415ade0 0x7fc4c415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4ca1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c4146450 con 0x7fc4c40b0540 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4ca1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c4157f10 con 0x7fc4c415ade0 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4ca1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c4157300 con 0x7fc4c4153cc0 2023-10-07T09:42:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 0x7fc4c4145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 0x7fc4c4145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59088/0 (socket says 172.21.15.29:59088) 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4c37fe700 1 -- 172.21.15.29:0/1163190981 learned_addr learned my addr 172.21.15.29:0/1163190981 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c415ade0 0x7fc4c415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.772+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c4153cc0 0x7fc4c4156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4c37fe700 1 -- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c4153cc0 msgr2=0x7fc4c4156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c4153cc0 0x7fc4c4156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4c37fe700 1 -- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c415ade0 msgr2=0x7fc4c415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c415ade0 0x7fc4c415e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4c37fe700 1 -- 172.21.15.29:0/1163190981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8009420 con 0x7fc4c40b0540 2023-10-07T09:42:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 0x7fc4c4145f10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400c810 tx=0x7fc4b4007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4ab7fe700 1 -- 172.21.15.29:0/1163190981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4001ea0 con 0x7fc4c40b0540 2023-10-07T09:42:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4ab7fe700 1 -- 172.21.15.29:0/1163190981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b40072e0 con 0x7fc4c40b0540 2023-10-07T09:42:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c41575e0 con 0x7fc4c40b0540 2023-10-07T09:42:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4ab7fe700 1 -- 172.21.15.29:0/1163190981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40076a0 con 0x7fc4c40b0540 2023-10-07T09:42:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.773+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c4157ba0 con 0x7fc4c40b0540 2023-10-07T09:42:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.775+0000 7fc4ab7fe700 1 -- 172.21.15.29:0/1163190981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc4b4005a00 con 0x7fc4c40b0540 2023-10-07T09:42:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.776+0000 7fc4ab7fe700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc4b0066f30 0x7fc4b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.776+0000 7fc4ab7fe700 1 -- 172.21.15.29:0/1163190981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc4b4083ae0 con 0x7fc4c40b0540 2023-10-07T09:42:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.777+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc4b0066f30 0x7fc4b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.777+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc4b0066f30 0x7fc4b00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4c415b200 tx=0x7fc4b8016000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.778+0000 7fc4d89abb80 1 Processor -- start 2023-10-07T09:42:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.779+0000 7fc4d89abb80 1 -- start start 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.779+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6ff7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.779+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba7000680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.779+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba7000bc0 0x558ba7001010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6eca310 con 0x558ba6ff7a40 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6f07130 con 0x558ba6fdd530 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ba6f04af0 con 0x558ba7000bc0 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6ff7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6ff7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59102/0 (socket says 172.21.15.29:59102) 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 -- 172.21.15.29:0/132312613 learned_addr learned my addr 172.21.15.29:0/132312613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba7000680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba7000bc0 0x558ba7001010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 -- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba7000bc0 msgr2=0x558ba7001010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba7000bc0 0x558ba7001010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 -- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 msgr2=0x558ba7000680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba7000680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 -- 172.21.15.29:0/132312613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ba6df30a0 con 0x558ba6ff7a40 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.780+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6ff7e10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4081f50 tx=0x7fc4b4081e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392aa398d80b532a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4a8ff9700 1 -- 172.21.15.29:0/132312613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4054fb0 con 0x558ba6ff7a40 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4a8ff9700 1 -- 172.21.15.29:0/132312613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b40548b0 con 0x558ba6ff7a40 2023-10-07T09:42:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4a8ff9700 1 -- 172.21.15.29:0/132312613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4057b00 con 0x558ba6ff7a40 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 -- 172.21.15.29:0/132312613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 msgr2=0x558ba6ff7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6ff7e10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4081f50 tx=0x7fc4b4081e00 comp rx=0 tx=0).stop 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 -- 172.21.15.29:0/132312613 shutdown_connections 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba7000bc0 0x558ba7001010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6ff7e10 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/132312613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba7000680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 -- 172.21.15.29:0/132312613 >> 172.21.15.29:0/132312613 conn(0x558ba6ffae70 msgr2=0x558ba6ffb240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.781+0000 7fc4d89abb80 1 -- 172.21.15.29:0/132312613 shutdown_connections 2023-10-07T09:42:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.782+0000 7fc4d89abb80 1 -- 172.21.15.29:0/132312613 wait complete. 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.782+0000 7fc4d89abb80 1 -- start start 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.782+0000 7fc4d89abb80 1 Processor -- start 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.783+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6fdd530 0x558ba6fd63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.783+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6fd9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.783+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fdf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.783+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6f008e0 con 0x558ba6ff7a40 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.783+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6eac980 con 0x558ba7000bc0 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.783+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ba6eac620 con 0x558ba6fdd530 2023-10-07T09:42:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4d89abb80 10 client.0 _ll_register_callbacks cb 0x558ba6d0fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6fd9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6fd9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59104/0 (socket says 172.21.15.29:59104) 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/2142882089 learned_addr learned my addr 172.21.15.29:0/2142882089 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6fdd530 0x558ba6fd63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fdf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6fdd530 msgr2=0x558ba6fd63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6fdd530 0x558ba6fd63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 msgr2=0x558ba6fdf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fdf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 -- 172.21.15.29:0/2142882089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c4157ba0 con 0x558ba6ff7a40 2023-10-07T09:42:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fdf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:42:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6fd9160 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8002000 tx=0x7fc4b8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df4d72692df9711 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.784+0000 7fc4a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:42:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/2142882089 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8030070 con 0x558ba6ff7a40 2023-10-07T09:42:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/2142882089 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc4b8029420 con 0x558ba6ff7a40 2023-10-07T09:42:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ba6fa5560 con 0x558ba6ff7a40 2023-10-07T09:42:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/2142882089 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4b8021070 con 0x558ba6ff7a40 2023-10-07T09:42:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 10 client.5916 fetch_fsmap learned FSMap version 25 2023-10-07T09:42:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x558ba6ecf3c0 con 0x558ba6ff7a40 2023-10-07T09:42:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/2142882089 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558ba6fa5560 con 0x558ba6ff7a40 2023-10-07T09:42:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/2142882089 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc4b80297f0 con 0x558ba6ff7a40 2023-10-07T09:42:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 10 client.5916 fetch_fsmap finished waiting for FSMap version 25 2023-10-07T09:42:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 10 client.5916 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T09:42:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 20 client.5916 populate_metadata read hostname 'smithi029' 2023-10-07T09:42:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba7007330 0x558ba7007700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 4 client.5916 mds_command: new command op to 5775 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:42:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.785+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 --> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ba6f820e0 con 0x558ba7007330 2023-10-07T09:42:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.786+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba7007330 0x558ba7007700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.786+0000 7fc4a1ffb700 10 client.5916 ms_handle_connect on v2:172.21.15.29:6832/2769082805 2023-10-07T09:42:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.786+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba7007330 0x558ba7007700 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c157cc1bd282ec server_cookie=82824ba466611ad1 in_seq=0 out_seq=0 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/2142882089 <== mds.0 v2:172.21.15.29:6832/2769082805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x558ba6f820e0 con 0x558ba7007330 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4a1ffb700 10 client.5916 handle_command_reply: tid=0 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 client.5916 shutdown 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 msgr2=0x558ba6fd9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6fd9160 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8002000 tx=0x7fc4b8021000 comp rx=0 tx=0).stop 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 shutdown_connections 2023-10-07T09:42:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba7007330 0x558ba7007700 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fdf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6fdd530 0x558ba6fd63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4a1ffb700 0 client.5916 ms_handle_reset on v2:172.21.15.29:6832/2769082805 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/2142882089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6ff7a40 0x558ba6fd9160 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 >> 172.21.15.29:0/2142882089 conn(0x558ba6ffae70 msgr2=0x558ba6fd8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 shutdown_connections 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.787+0000 7fc4d89abb80 1 -- 172.21.15.29:0/2142882089 wait complete. 2023-10-07T09:42:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.788+0000 7fc4d89abb80 20 client.5916 trim_cache size 0 max 16384 2023-10-07T09:42:18.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.805+0000 7fc4d89abb80 1 Processor -- start 2023-10-07T09:42:18.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.805+0000 7fc4d89abb80 1 -- start start 2023-10-07T09:42:18.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 0x558ba6fdec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fb0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fb12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6d981e0 con 0x558ba6fdd530 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6fdcc80 con 0x558ba7000bc0 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ba6fb3970 con 0x558ba6ff7a40 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 0x558ba6fdec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 0x558ba6fdec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59112/0 (socket says 172.21.15.29:59112) 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 -- 172.21.15.29:0/38643768 learned_addr learned my addr 172.21.15.29:0/38643768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fb12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fb0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 -- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 msgr2=0x558ba6fb0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fb0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 -- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 msgr2=0x558ba6fb12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fb12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.806+0000 7fc4c37fe700 1 -- 172.21.15.29:0/38643768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ba6d671d0 con 0x558ba6fdd530 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 0x558ba6fdec70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400c7e0 tx=0x7fc4b407e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72936e4f90124f40 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4a97fa700 1 -- 172.21.15.29:0/38643768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b405b850 con 0x558ba6fdd530 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4a97fa700 1 -- 172.21.15.29:0/38643768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4057b00 con 0x558ba6fdd530 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4a97fa700 1 -- 172.21.15.29:0/38643768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40057f0 con 0x558ba6fdd530 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4d89abb80 1 -- 172.21.15.29:0/38643768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 msgr2=0x558ba6fdec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 0x558ba6fdec70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400c7e0 tx=0x7fc4b407e040 comp rx=0 tx=0).stop 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4d89abb80 1 -- 172.21.15.29:0/38643768 shutdown_connections 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.807+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba7000bc0 0x558ba6fb12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba6fdd530 0x558ba6fdec70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/38643768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fb0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 -- 172.21.15.29:0/38643768 >> 172.21.15.29:0/38643768 conn(0x558ba6ffae70 msgr2=0x558ba6fae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 -- 172.21.15.29:0/38643768 shutdown_connections 2023-10-07T09:42:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 -- 172.21.15.29:0/38643768 wait complete. 2023-10-07T09:42:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 -- start start 2023-10-07T09:42:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.808+0000 7fc4d89abb80 1 Processor -- start 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba705c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fd2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 0x558ba6fa67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba705caa0 con 0x558ba7000bc0 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ba6fdecb0 con 0x558ba6fdd530 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ba6fffcc0 con 0x558ba6ff7a40 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.809+0000 7fc4d89abb80 10 client.0 _ll_register_callbacks cb 0x558ba6d0fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 0x558ba6fa67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 0x558ba6fa67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59128/0 (socket says 172.21.15.29:59128) 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 -- 172.21.15.29:0/1276856635 learned_addr learned my addr 172.21.15.29:0/1276856635 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c2ffd700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fd2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba705c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 -- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 msgr2=0x558ba6fd2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fd2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 -- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 msgr2=0x558ba705c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba705c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 -- 172.21.15.29:0/1276856635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ba6df30a0 con 0x558ba7000bc0 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4c3fff700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 0x558ba6fa67d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc016a50 tx=0x7fc4bc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d038b1ba3358f3d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.810+0000 7fc4a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:42:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/1276856635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc006db0 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4c4155fd0 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/1276856635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc4bc017500 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/1276856635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4bc0178d0 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 10 client.5922 fetch_fsmap learned FSMap version 25 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x558ba6d40d60 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/1276856635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4155fd0 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/1276856635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc4bc017a30 con 0x558ba7000bc0 2023-10-07T09:42:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 10 client.5922 fetch_fsmap finished waiting for FSMap version 25 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 10 client.5922 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 20 client.5922 populate_metadata read hostname 'smithi029' 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba6fa6d10 0x558ba6fdb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 4 client.5922 mds_command: new command op to 5775 tid=0[{"prefix": "status"}] 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.811+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 --> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fc4b40548b0 con 0x558ba6fa6d10 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba6fa6d10 0x558ba6fdb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4c37fe700 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba6fa6d10 0x558ba6fdb4f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264a6a2fa5d9aace server_cookie=f2a02a3717ea23e8 in_seq=0 out_seq=0 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4a1ffb700 10 client.5922 ms_handle_connect on v2:172.21.15.29:6832/2769082805 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4a1ffb700 1 -- 172.21.15.29:0/1276856635 <== mds.0 v2:172.21.15.29:6832/2769082805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7fc4b40548b0 con 0x558ba6fa6d10 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4a1ffb700 10 client.5922 handle_command_reply: tid=0 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 client.5922 shutdown 2023-10-07T09:42:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 msgr2=0x558ba6fa67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 0x558ba6fa67d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc016a50 tx=0x7fc4bc009040 comp rx=0 tx=0).stop 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 shutdown_connections 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ba7000bc0 0x558ba6fa67d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.29:6832/2769082805,v1:172.21.15.29:6834/2769082805] conn(0x558ba6fa6d10 0x558ba6fdb4f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ba6fdd530 0x558ba705c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4d89abb80 1 --2- 172.21.15.29:0/1276856635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558ba6ff7a40 0x558ba6fd2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.812+0000 7fc4a1ffb700 0 client.5922 ms_handle_reset on v2:172.21.15.29:6832/2769082805 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 >> 172.21.15.29:0/1276856635 conn(0x558ba6ffae70 msgr2=0x558ba6fb2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 shutdown_connections 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4d89abb80 1 -- 172.21.15.29:0/1276856635 wait complete. 2023-10-07T09:42:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4d89abb80 20 client.5922 trim_cache size 0 max 16384 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 5775, 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5791753499999999, 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 25, 2023-10-07T09:42:18.821 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 61, 2023-10-07T09:42:18.822 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 61, 2023-10-07T09:42:18.822 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 19.29608511 2023-10-07T09:42:18.822 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc4b0066f30 msgr2=0x7fc4b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc4b0066f30 0x7fc4b00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4c415b200 tx=0x7fc4b8016000 comp rx=0 tx=0).stop 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 msgr2=0x7fc4c4145f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 0x7fc4c4145f10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400c810 tx=0x7fc4b4007c80 comp rx=0 tx=0).stop 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 shutdown_connections 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc4b0066f30 0x7fc4b00693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c415ade0 0x7fc4c415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c40b0540 0x7fc4c4145f10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 --2- 172.21.15.29:0/1163190981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c4153cc0 0x7fc4c4156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.813+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 >> 172.21.15.29:0/1163190981 conn(0x7fc4c4067610 msgr2=0x7fc4c4152c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.814+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 shutdown_connections 2023-10-07T09:42:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:18.814+0000 7fc4ca1ff700 1 -- 172.21.15.29:0/1163190981 wait complete. 2023-10-07T09:42:18.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:42:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.024+0000 7fbfda9ff700 1 Processor -- start 2023-10-07T09:42:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.024+0000 7fbfda9ff700 1 -- start start 2023-10-07T09:42:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfda9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbfd4157290 0x7fbfd40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfda9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd40b1a10 0x7fbfd4157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfda9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd4158ad0 con 0x7fbfd40ae8b0 2023-10-07T09:42:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfda9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd415b140 con 0x7fbfd4157290 2023-10-07T09:42:19.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfda9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd415b420 con 0x7fbfd40b1a10 2023-10-07T09:42:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfd8f9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd40b1a10 0x7fbfd4157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfd3fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbfd4157290 0x7fbfd40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:19.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfd37fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fbfd40ae8b0 0x7fbfd40b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:34188/0 (socket says 172.21.15.29:34188) 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.026+0000 7fbfd37fe700 1 -- 172.21.15.29:0/1103417847 learned_addr learned my addr 172.21.15.29:0/1103417847 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd2ffd700 1 -- 172.21.15.29:0/1103417847 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627775697 0 0) 0x7fbfd4158ad0 con 0x7fbfd40ae8b0 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1103417847 >> v2:172.21.15.144:3300/0 conn(0x7fbfd4157290 msgr2=0x7fbfd40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd8f9c700 1 --2- 172.21.15.29:0/1103417847 >> v2:172.21.15.144:3300/0 conn(0x7fbfd4157290 0x7fbfd40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1103417847 >> v1:172.21.15.144:6789/0 conn(0x7fbfd40ae8b0 legacy=0x7fbfd40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1103417847 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd415b700 con 0x7fbfd40b1a10 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd8f9c700 1 --2- 172.21.15.29:0/1103417847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd40b1a10 0x7fbfd4157660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009e60 tx=0x7fbfcc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476cd1d954d80396 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.027+0000 7fbfd2ffd700 1 -- 172.21.15.29:0/1103417847 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc00e070 con 0x7fbfd40b1a10 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfd2ffd700 1 -- 172.21.15.29:0/1103417847 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc00bb60 con 0x7fbfd40b1a10 2023-10-07T09:42:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfd2ffd700 1 -- 172.21.15.29:0/1103417847 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc012430 con 0x7fbfd40b1a10 2023-10-07T09:42:19.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1103417847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd40b1a10 msgr2=0x7fbfd4157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1103417847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd40b1a10 0x7fbfd4157660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009e60 tx=0x7fbfcc009780 comp rx=0 tx=0).stop 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1103417847 shutdown_connections 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1103417847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd40b1a10 0x7fbfd4157660 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1103417847 >> v2:172.21.15.144:3300/0 conn(0x7fbfd4157290 0x7fbfd40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1103417847 >> 172.21.15.29:0/1103417847 conn(0x7fbfd40680a0 msgr2=0x7fbfd40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1103417847 shutdown_connections 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.028+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1103417847 wait complete. 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.030+0000 7fbfda9ff700 1 Processor -- start 2023-10-07T09:42:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.030+0000 7fbfda9ff700 1 -- start start 2023-10-07T09:42:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfda9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd40b1a10 0x7fbfd4169970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfda9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd4157290 0x7fbfd4141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfda9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 0x7fbfd4159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfda9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4169eb0 con 0x7fbfd40c6790 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfda9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4142120 con 0x7fbfd40b1a10 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfda9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40c56f0 con 0x7fbfd4157290 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 0x7fbfd4159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 0x7fbfd4159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59142/0 (socket says 172.21.15.29:59142) 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1092956702 learned_addr learned my addr 172.21.15.29:0/1092956702 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd3fff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd40b1a10 0x7fbfd4169970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd37fe700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd4157290 0x7fbfd4141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd4157290 msgr2=0x7fbfd4141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd4157290 0x7fbfd4141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd40b1a10 msgr2=0x7fbfd4169970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd40b1a10 0x7fbfd4169970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.031+0000 7fbfd8f9c700 1 -- 172.21.15.29:0/1092956702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc009a20 con 0x7fbfd40c6790 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.032+0000 7fbfd8f9c700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 0x7fbfd4159750 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbfd415b560 tx=0x7fbfcc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.032+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc027470 con 0x7fbfd40c6790 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.032+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc0275d0 con 0x7fbfd40c6790 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.032+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd40c59d0 con 0x7fbfd40c6790 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.032+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc027990 con 0x7fbfd40c6790 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.032+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd40c5f90 con 0x7fbfd40c6790 2023-10-07T09:42:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.033+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd404c3c0 con 0x7fbfd40c6790 2023-10-07T09:42:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.034+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbfcc024070 con 0x7fbfd40c6790 2023-10-07T09:42:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.035+0000 7fbfbbfff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbfc0066f30 0x7fbfc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.035+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbfcc086e70 con 0x7fbfd40c6790 2023-10-07T09:42:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.036+0000 7fbfd3fff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbfc0066f30 0x7fbfc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.036+0000 7fbfd3fff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbfc0066f30 0x7fbfc00693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4155100 tx=0x7fbfc4008040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.039+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfcc053f80 con 0x7fbfd40c6790 2023-10-07T09:42:19.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.293+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbfd415b9e0 con 0x7fbfd40c6790 2023-10-07T09:42:19.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.293+0000 7fbfbbfff700 1 -- 172.21.15.29:0/1092956702 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbfcc0575a0 con 0x7fbfd40c6790 2023-10-07T09:42:19.295 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:42:19.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbfc0066f30 msgr2=0x7fbfc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:19.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbfc0066f30 0x7fbfc00693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4155100 tx=0x7fbfc4008040 comp rx=0 tx=0).stop 2023-10-07T09:42:19.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 msgr2=0x7fbfd4159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:19.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 0x7fbfd4159750 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbfd415b560 tx=0x7fbfcc019000 comp rx=0 tx=0).stop 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 shutdown_connections 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbfc0066f30 0x7fbfc00693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40c6790 0x7fbfd4159750 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd40b1a10 0x7fbfd4169970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 --2- 172.21.15.29:0/1092956702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfd4157290 0x7fbfd4141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 >> 172.21.15.29:0/1092956702 conn(0x7fbfd40680a0 msgr2=0x7fbfd40b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 shutdown_connections 2023-10-07T09:42:19.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:19.296+0000 7fbfda9ff700 1 -- 172.21.15.29:0/1092956702 wait complete. 2023-10-07T09:42:19.314 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:42:19.314 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:42:19.315 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:42:19.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:19.315 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:42:19.363 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:42:19.364 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:42:19.365 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41489sec preferred_lft 41489sec 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.366 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:42:19.367 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:42:19.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:19.368 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:42:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:42:19.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:19.439 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:42:19.489 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:42:19.489 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:42:19.489 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:42:19.489 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:42:19.489 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:42:19.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:19.490 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:42:19.491 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:42:19.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:42:19.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:42:19.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:42:19.491 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:42:19.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:19.663 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:42:19.709 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:42:19.709 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:42:19.710 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:42:19.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:19.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:42:19.781 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:42:19.781 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:42:19.828 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:42:19.828 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:42:19.828 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:42:19.914 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:42:19.914 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:42:19.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:42:20.054 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:42:20.054 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:42:20.054 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:42:20.069 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:42:20.069 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:42:20.069 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:42:20.070 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:42:20.070 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:42:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:42:20.098 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:42:20.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:42:20.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:42:20.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:42:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.368+0000 7ff9283ff700 1 Processor -- start 2023-10-07T09:42:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.368+0000 7ff9283ff700 1 -- start start 2023-10-07T09:42:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.369+0000 7ff9283ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff9200d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.369+0000 7ff9283ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9200cbf20 con 0x7ff920159750 2023-10-07T09:42:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff9283ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9200cc200 con 0x7ff9200ce8a0 2023-10-07T09:42:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff9283ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200cc4e0 con 0x7ff920159b20 2023-10-07T09:42:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff92619b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff920159750 0x7ff9200ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37106/0 (socket says 172.21.15.29:37106) 2023-10-07T09:42:20.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff92619b700 1 -- 172.21.15.29:0/51889103 learned_addr learned my addr 172.21.15.29:0/51889103 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff92599a700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff9200d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677991290 0 0) 0x7ff9200cbf20 con 0x7ff920159750 2023-10-07T09:42:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff90c003570 con 0x7ff920159750 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508548046 0 0) 0x7ff9200cc200 con 0x7ff9200ce8a0 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9200cbf20 con 0x7ff9200ce8a0 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.370+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2631968606 0 0) 0x7ff90c003570 con 0x7ff920159750 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 msgr2=0x7ff9200d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff9200d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 >> v1:172.21.15.144:6789/0 conn(0x7ff9200ce8a0 legacy=0x7ff9200c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9200cc7c0 con 0x7ff920159750 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7ff91c0015f0 con 0x7ff920159750 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92599a700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff9200d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 >> v1:172.21.15.29:6789/0 conn(0x7ff920159750 legacy=0x7ff9200ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff90c003c20 0x7ff90c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff90c004810 0x7ff90c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff90c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200cc4e0 con 0x7ff90c004810 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c00bb10 con 0x7ff90c003c20 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c00bcd0 con 0x7ff920159b20 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92599a700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff90c004810 0x7ff90c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92619b700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff90c003c20 0x7ff90c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92699c700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff90c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92599a700 1 -- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 msgr2=0x7ff90c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92599a700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff920159b20 0x7ff90c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92599a700 1 -- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff90c003c20 msgr2=0x7ff90c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.371+0000 7ff92599a700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff90c003c20 0x7ff90c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff92599a700 1 -- 172.21.15.29:0/51889103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff910009420 con 0x7ff90c004810 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff92599a700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff90c004810 0x7ff90c004c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff910009df0 tx=0x7ff9100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373766f4cfb3ddc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910009740 con 0x7ff90c004810 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91000e920 con 0x7ff90c004810 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff925199700 1 -- 172.21.15.29:0/51889103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91000ece0 con 0x7ff90c004810 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff9283ff700 1 -- 172.21.15.29:0/51889103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff90c004810 msgr2=0x7ff90c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.372+0000 7ff9283ff700 1 --2- 172.21.15.29:0/51889103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff90c004810 0x7ff90c004c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff910009df0 tx=0x7ff9100091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.373+0000 7ff92699c700 1 -- 172.21.15.29:0/51889103 reap_dead start 2023-10-07T09:42:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.373+0000 7ff9283ff700 1 -- 172.21.15.29:0/51889103 shutdown_connections 2023-10-07T09:42:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.373+0000 7ff9283ff700 1 -- 172.21.15.29:0/51889103 >> 172.21.15.29:0/51889103 conn(0x7ff920063a10 msgr2=0x7ff9200d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.373+0000 7ff9283ff700 1 -- 172.21.15.29:0/51889103 shutdown_connections 2023-10-07T09:42:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.373+0000 7ff9283ff700 1 -- 172.21.15.29:0/51889103 wait complete. 2023-10-07T09:42:20.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.374+0000 7ff9283ff700 1 Processor -- start 2023-10-07T09:42:20.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.374+0000 7ff9283ff700 1 -- start start 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff9283ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff90c004810 0x7ff920159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff9283ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 0x7ff9200c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff9283ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9201546e0 0x7ff9200b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff9283ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200c9aa0 con 0x7ff9200ca6e0 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff9283ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200b3dc0 con 0x7ff9201546e0 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff9283ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200b3490 con 0x7ff90c004810 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff92599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 0x7ff9200c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff92599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 0x7ff9200c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59164/0 (socket says 172.21.15.29:59164) 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.375+0000 7ff92599a700 1 -- 172.21.15.29:0/2429086220 learned_addr learned my addr 172.21.15.29:0/2429086220 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92619b700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff90c004810 0x7ff920159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92699c700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9201546e0 0x7ff9200b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92599a700 1 -- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff90c004810 msgr2=0x7ff920159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92599a700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff90c004810 0x7ff920159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92599a700 1 -- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9201546e0 msgr2=0x7ff9200b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92599a700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9201546e0 0x7ff9200b54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92599a700 1 -- 172.21.15.29:0/2429086220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9200b3770 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff92599a700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 0x7ff9200c9190 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff910006ed0 tx=0x7ff91001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910022070 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.376+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff910013c90 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.377+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9200b3a50 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.377+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91001c6a0 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.377+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff920154b50 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.378+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9200b3f50 con 0x7ff9200ca6e0 2023-10-07T09:42:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.379+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff910026070 con 0x7ff9200ca6e0 2023-10-07T09:42:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.379+0000 7ff9167fc700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff904066f60 0x7ff904069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.379+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff910085f40 con 0x7ff9200ca6e0 2023-10-07T09:42:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.380+0000 7ff92619b700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff904066f60 0x7ff904069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.381+0000 7ff92619b700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff904066f60 0x7ff904069410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff91c000e10 tx=0x7ff91c009040 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.384+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff910053050 con 0x7ff9200ca6e0 2023-10-07T09:42:20.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.614+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9200c9aa0 con 0x7ff904066f60 2023-10-07T09:42:20.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.617+0000 7ff9167fc700 1 -- 172.21.15.29:0/2429086220 <== mgr.5195 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff9200c9aa0 con 0x7ff904066f60 2023-10-07T09:42:20.617 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:42:20.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:42:20.618 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:42:20.618 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:42:20.618 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff904066f60 msgr2=0x7ff904069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff904066f60 0x7ff904069410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff91c000e10 tx=0x7ff91c009040 comp rx=0 tx=0).stop 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 msgr2=0x7ff9200c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 0x7ff9200c9190 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff910006ed0 tx=0x7ff91001a000 comp rx=0 tx=0).stop 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 shutdown_connections 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff904066f60 0x7ff904069410 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9201546e0 0x7ff9200b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff90c004810 0x7ff920159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.618+0000 7ff9283ff700 1 --2- 172.21.15.29:0/2429086220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9200ca6e0 0x7ff9200c9190 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.619+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 >> 172.21.15.29:0/2429086220 conn(0x7ff920063a10 msgr2=0x7ff920156640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.619+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 shutdown_connections 2023-10-07T09:42:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.619+0000 7ff9283ff700 1 -- 172.21.15.29:0/2429086220 wait complete. 2023-10-07T09:42:20.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:42:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.832+0000 7f332c5ff700 1 Processor -- start 2023-10-07T09:42:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.832+0000 7f332c5ff700 1 -- start start 2023-10-07T09:42:20.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.834+0000 7f332c5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f332414c8d0 0x7f33240afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.834+0000 7f332c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324159d90 0x7f332415fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.834+0000 7f332c5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3324157790 con 0x7f3324153cc0 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.834+0000 7f332c5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33240b0950 con 0x7f332414c8d0 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.834+0000 7f332c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3324157580 con 0x7f3324159d90 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.834+0000 7f332a39b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f332414c8d0 0x7f33240afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f332414c8d0 0x7f33240afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59174/0 (socket says 172.21.15.29:59174) 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 -- 172.21.15.29:0/1684871660 learned_addr learned my addr 172.21.15.29:0/1684871660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f3329b9a700 1 --1- 172.21.15.29:0/1684871660 >> v1:172.21.15.29:6789/0 conn(0x7f3324153cc0 0x7f3324159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37110/0 (socket says 172.21.15.29:37110) 2023-10-07T09:42:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332ab9c700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324159d90 0x7f332415fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 -- 172.21.15.29:0/1684871660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324159d90 msgr2=0x7f332415fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324159d90 0x7f332415fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 -- 172.21.15.29:0/1684871660 >> v1:172.21.15.29:6789/0 conn(0x7f3324153cc0 legacy=0x7f3324159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 -- 172.21.15.29:0/1684871660 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3324157a70 con 0x7f332414c8d0 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332ab9c700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324159d90 0x7f332415fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f3329399700 1 -- 172.21.15.29:0/1684871660 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010606673 0 0) 0x7f3324157790 con 0x7f3324153cc0 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.835+0000 7f332a39b700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f332414c8d0 0x7f33240afed0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f33200099b0 tx=0x7f3320009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a26a9dc581b58e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f3329399700 1 -- 172.21.15.29:0/1684871660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332000b860 con 0x7f332414c8d0 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f3329399700 1 -- 172.21.15.29:0/1684871660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332000b9c0 con 0x7f332414c8d0 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f3329399700 1 -- 172.21.15.29:0/1684871660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332000bd60 con 0x7f332414c8d0 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 -- 172.21.15.29:0/1684871660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f332414c8d0 msgr2=0x7f33240afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f332414c8d0 0x7f33240afed0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f33200099b0 tx=0x7f3320009250 comp rx=0 tx=0).stop 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 -- 172.21.15.29:0/1684871660 shutdown_connections 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324159d90 0x7f332415fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 --2- 172.21.15.29:0/1684871660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f332414c8d0 0x7f33240afed0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 -- 172.21.15.29:0/1684871660 >> 172.21.15.29:0/1684871660 conn(0x7f3324067610 msgr2=0x7f3324067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 -- 172.21.15.29:0/1684871660 shutdown_connections 2023-10-07T09:42:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.836+0000 7f332c5ff700 1 -- 172.21.15.29:0/1684871660 wait complete. 2023-10-07T09:42:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.838+0000 7f332c5ff700 1 Processor -- start 2023-10-07T09:42:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.838+0000 7f332c5ff700 1 -- start start 2023-10-07T09:42:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332c5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332414c8d0 0x7f332414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324153cc0 0x7f33241531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332c5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 0x7f33241697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332c5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241511c0 con 0x7f3324159d90 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332c5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241508b0 con 0x7f332414c8d0 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3324150b70 con 0x7f3324153cc0 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 0x7f33241697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 0x7f33241697c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59184/0 (socket says 172.21.15.29:59184) 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 -- 172.21.15.29:0/2051979157 learned_addr learned my addr 172.21.15.29:0/2051979157 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332a39b700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332414c8d0 0x7f332414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f3329b9a700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324153cc0 0x7f33241531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 -- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324153cc0 msgr2=0x7f33241531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324153cc0 0x7f33241531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 -- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332414c8d0 msgr2=0x7f332414ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332414c8d0 0x7f332414ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.839+0000 7f332ab9c700 1 -- 172.21.15.29:0/2051979157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200094f0 con 0x7f3324159d90 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.840+0000 7f332ab9c700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 0x7f33241697c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f331c007f50 tx=0x7f331c00c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.840+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c00f040 con 0x7f3324159d90 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.840+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c00a2e0 con 0x7f3324159d90 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.840+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c00dc20 con 0x7f3324159d90 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.840+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3324150e50 con 0x7f3324159d90 2023-10-07T09:42:20.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.840+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332414f400 con 0x7f3324159d90 2023-10-07T09:42:20.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.841+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332404c3c0 con 0x7f3324159d90 2023-10-07T09:42:20.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.842+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f331c008450 con 0x7f3324159d90 2023-10-07T09:42:20.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.843+0000 7f331a7fc700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3308066f30 0x7f33080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:20.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.843+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f331c084b20 con 0x7f3324159d90 2023-10-07T09:42:20.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.844+0000 7f332a39b700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3308066f30 0x7f33080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:20.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.844+0000 7f332a39b700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3308066f30 0x7f33080693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33200099b0 tx=0x7f3320017000 comp rx=0 tx=0).ready entity=mgr.5195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:20.847+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f331c051b90 con 0x7f3324159d90 2023-10-07T09:42:20.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:42:20.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:42:21.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.103+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f33240ccd00 con 0x7f3324159d90 2023-10-07T09:42:21.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.124+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 17) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f331c051f60 con 0x7f3324159d90 2023-10-07T09:42:21.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.125+0000 7f331a7fc700 1 -- 172.21.15.29:0/2051979157 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7f331c055620 con 0x7f3324159d90 2023-10-07T09:42:21.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3308066f30 msgr2=0x7f33080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3308066f30 0x7f33080693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33200099b0 tx=0x7f3320017000 comp rx=0 tx=0).stop 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 msgr2=0x7f33241697c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 0x7f33241697c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f331c007f50 tx=0x7f331c00c040 comp rx=0 tx=0).stop 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 shutdown_connections 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3308066f30 0x7f33080693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3324159d90 0x7f33241697c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332414c8d0 0x7f332414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 --2- 172.21.15.29:0/2051979157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3324153cc0 0x7f33241531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 >> 172.21.15.29:0/2051979157 conn(0x7f3324067610 msgr2=0x7f33241545b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:21.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 shutdown_connections 2023-10-07T09:42:21.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.128+0000 7f332c5ff700 1 -- 172.21.15.29:0/2051979157 wait complete. 2023-10-07T09:42:21.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:42:21.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:21.335+0000 7f7059a801c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:42:21.337 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:21.336+0000 7fd94b1291c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:42:21.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.352+0000 7fe2b7dff700 1 Processor -- start 2023-10-07T09:42:21.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.352+0000 7fe2b7dff700 1 -- start start 2023-10-07T09:42:21.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.353+0000 7fe2b7dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b0159750 0x7fe2b00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.353+0000 7fe2b7dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe2b0159b20 0x7fe2b00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.353+0000 7fe2b7dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b00cbf20 con 0x7fe2b00ce6d0 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.353+0000 7fe2b7dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b00cc200 con 0x7fe2b0159b20 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.353+0000 7fe2b7dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00cc4e0 con 0x7fe2b0159750 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe2b0159b20 0x7fe2b00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe2b0159b20 0x7fe2b00ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:49292/0 (socket says 172.21.15.29:49292) 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 -- 172.21.15.29:0/2833671192 learned_addr learned my addr 172.21.15.29:0/2833671192 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:21.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b5b9b700 1 --2- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b0159750 0x7fe2b00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 -- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b0159750 msgr2=0x7fe2b00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 --2- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b0159750 0x7fe2b00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 -- 172.21.15.29:0/2833671192 >> v1:172.21.15.144:6789/0 conn(0x7fe2b00ce6d0 legacy=0x7fe2b015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 -- 172.21.15.29:0/2833671192 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00cc7c0 con 0x7fe2b0159b20 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.354+0000 7fe2b539a700 1 --2- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00ce190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00099b0 tx=0x7fe2a0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62da223f3295d0da server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b4b99700 1 -- 172.21.15.29:0/2833671192 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0009ea0 con 0x7fe2b0159b20 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b4b99700 1 -- 172.21.15.29:0/2833671192 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000b860 con 0x7fe2b0159b20 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b4b99700 1 -- 172.21.15.29:0/2833671192 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000bc00 con 0x7fe2b0159b20 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 msgr2=0x7fe2b00ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00ce190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00099b0 tx=0x7fe2a0009250 comp rx=0 tx=0).stop 2023-10-07T09:42:21.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2833671192 shutdown_connections 2023-10-07T09:42:21.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b0159750 0x7fe2b00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2833671192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00ce190 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2833671192 >> 172.21.15.29:0/2833671192 conn(0x7fe2b0063a10 msgr2=0x7fe2b00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:21.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2833671192 shutdown_connections 2023-10-07T09:42:21.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.355+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2833671192 wait complete. 2023-10-07T09:42:21.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.357+0000 7fe2b7dff700 1 Processor -- start 2023-10-07T09:42:21.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.357+0000 7fe2b7dff700 1 -- start start 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b7dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b00ce6d0 0x7fe2b01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b7dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 0x7fe2b00c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b7dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b7dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00b6380 con 0x7fe2b0159750 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b7dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00c7810 con 0x7fe2b0159b20 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b7dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00c7af0 con 0x7fe2b00ce6d0 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b539a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 0x7fe2b00c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b539a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 0x7fe2b00c85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59208/0 (socket says 172.21.15.29:59208) 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b539a700 1 -- 172.21.15.29:0/2155048329 learned_addr learned my addr 172.21.15.29:0/2155048329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b5b9b700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b00ce6d0 0x7fe2b01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.358+0000 7fe2b639c700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b539a700 1 -- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b00ce6d0 msgr2=0x7fe2b01526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b539a700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b00ce6d0 0x7fe2b01526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b539a700 1 -- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 msgr2=0x7fe2b00c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b539a700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b539a700 1 -- 172.21.15.29:0/2155048329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00094f0 con 0x7fe2b0159750 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b539a700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 0x7fe2b00c85d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0088a40 tx=0x7fe2a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0000d10 con 0x7fe2b0159750 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000bbe0 con 0x7fe2b0159750 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0011670 con 0x7fe2b0159750 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.359+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b00c7dd0 con 0x7fe2b0159750 2023-10-07T09:42:21.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.360+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b014b3b0 con 0x7fe2b0159750 2023-10-07T09:42:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.361+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b004c3c0 con 0x7fe2b0159750 2023-10-07T09:42:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.361+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fe2a0011aa0 con 0x7fe2b0159750 2023-10-07T09:42:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.362+0000 7fe2a5ffb700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 0x7fe294069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.362+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe2a0085f50 con 0x7fe2b0159750 2023-10-07T09:42:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.363+0000 7fe2b5b9b700 1 -- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 msgr2=0x7fe294069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.363+0000 7fe2b5b9b700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 0x7fe294069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:42:21.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.367+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2a0052fc0 con 0x7fe2b0159750 2023-10-07T09:42:21.515 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:21.514+0000 7f7059a801c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:42:21.545 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:21.544+0000 7fd94b1291c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:42:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.564+0000 7fe2b5b9b700 1 -- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 msgr2=0x7fe294069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.564+0000 7fe2b5b9b700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 0x7fe294069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:42:21.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.607+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe2b0064040 con 0x7fe2b0159750 2023-10-07T09:42:21.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.611+0000 7fe2a5ffb700 1 -- 172.21.15.29:0/2155048329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe2a00565e0 con 0x7fe2b0159750 2023-10-07T09:42:21.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 msgr2=0x7fe294069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:42:21.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 0x7fe294069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 msgr2=0x7fe2b00c85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 0x7fe2b00c85d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0088a40 tx=0x7fe2a0017000 comp rx=0 tx=0).stop 2023-10-07T09:42:21.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 shutdown_connections 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe294066f80 0x7fe294069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b0159b20 0x7fe2b00c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b00ce6d0 0x7fe2b01526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 --2- 172.21.15.29:0/2155048329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2b0159750 0x7fe2b00c85d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 >> 172.21.15.29:0/2155048329 conn(0x7fe2b0063a10 msgr2=0x7fe2b00ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 shutdown_connections 2023-10-07T09:42:21.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.615+0000 7fe2b7dff700 1 -- 172.21.15.29:0/2155048329 wait complete. 2023-10-07T09:42:21.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:42:21.827 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:21.826+0000 7f7059a801c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:42:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.848+0000 7fed3fdff700 1 Processor -- start 2023-10-07T09:42:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.848+0000 7fed3fdff700 1 -- start start 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.849+0000 7fed3fdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed38158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.849+0000 7fed3fdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed380ae8a0 0x7fed380b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.849+0000 7fed3fdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed380acb40 con 0x7fed3815ad30 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.849+0000 7fed3fdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed38158c60 con 0x7fed380ae8a0 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.849+0000 7fed3fdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed380abf30 con 0x7fed3815a960 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3d39a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed3815ad30 0x7fed380ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37124/0 (socket says 172.21.15.29:37124) 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3d39a700 1 -- 172.21.15.29:0/1497688389 learned_addr learned my addr 172.21.15.29:0/1497688389 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3e39c700 1 --2- 172.21.15.29:0/1497688389 >> v2:172.21.15.144:3300/0 conn(0x7fed380ae8a0 0x7fed380b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3db9b700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed38158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3cb99700 1 -- 172.21.15.29:0/1497688389 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610773832 0 0) 0x7fed380acb40 con 0x7fed3815ad30 2023-10-07T09:42:21.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3cb99700 1 -- 172.21.15.29:0/1497688389 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24003570 con 0x7fed3815ad30 2023-10-07T09:42:21.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3e39c700 1 -- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 msgr2=0x7fed38158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:21.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3e39c700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed38158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3e39c700 1 -- 172.21.15.29:0/1497688389 >> v1:172.21.15.29:6789/0 conn(0x7fed3815ad30 legacy=0x7fed380ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3e39c700 1 -- 172.21.15.29:0/1497688389 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed380ac210 con 0x7fed380ae8a0 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.850+0000 7fed3db9b700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed38158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.851+0000 7fed3e39c700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed380ae8a0 0x7fed380b1290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fed30009e60 tx=0x7fed30009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4784e6a5afb2f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.851+0000 7fed3cb99700 1 -- 172.21.15.29:0/1497688389 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3000e070 con 0x7fed380ae8a0 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.851+0000 7fed3cb99700 1 -- 172.21.15.29:0/1497688389 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3000bab0 con 0x7fed380ae8a0 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.851+0000 7fed3cb99700 1 -- 172.21.15.29:0/1497688389 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3000be70 con 0x7fed380ae8a0 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 -- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed380ae8a0 msgr2=0x7fed380b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed380ae8a0 0x7fed380b1290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fed30009e60 tx=0x7fed30009780 comp rx=0 tx=0).stop 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 -- 172.21.15.29:0/1497688389 shutdown_connections 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed380ae8a0 0x7fed380b1290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 --2- 172.21.15.29:0/1497688389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed38158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 -- 172.21.15.29:0/1497688389 >> 172.21.15.29:0/1497688389 conn(0x7fed38067610 msgr2=0x7fed38067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:21.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 -- 172.21.15.29:0/1497688389 shutdown_connections 2023-10-07T09:42:21.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.852+0000 7fed3fdff700 1 -- 172.21.15.29:0/1497688389 wait complete. 2023-10-07T09:42:21.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.853+0000 7fed3fdff700 1 Processor -- start 2023-10-07T09:42:21.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.854+0000 7fed3fdff700 1 -- start start 2023-10-07T09:42:21.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.854+0000 7fed3fdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 0x7fed3816d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3fdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed3814dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3fdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3815ad30 0x7fed38069b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3fdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3816d9f0 con 0x7fed380ae8a0 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3fdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed38157200 con 0x7fed3815ad30 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3fdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed380af080 con 0x7fed3815a960 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 0x7fed3816d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 0x7fed3816d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59212/0 (socket says 172.21.15.29:59212) 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 -- 172.21.15.29:0/2448313233 learned_addr learned my addr 172.21.15.29:0/2448313233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3e39c700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3815ad30 0x7fed38069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3d39a700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed3814dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 -- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 msgr2=0x7fed3814dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed3814dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 -- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3815ad30 msgr2=0x7fed38069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:21.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3815ad30 0x7fed38069b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:21.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.855+0000 7fed3db9b700 1 -- 172.21.15.29:0/2448313233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34009420 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.856+0000 7fed3db9b700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 0x7fed3816d3e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fed3400ecb0 tx=0x7fed340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.856+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3400c920 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.856+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3400ca80 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.856+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3400ce00 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.856+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30009a20 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.856+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed38156d70 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.857+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed38156ad0 con 0x7fed380ae8a0 2023-10-07T09:42:21.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.858+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fed34013680 con 0x7fed380ae8a0 2023-10-07T09:42:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.859+0000 7fed2dffb700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 0x7fed1c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.859+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fed34084990 con 0x7fed380ae8a0 2023-10-07T09:42:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.860+0000 7fed3d39a700 1 -- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 msgr2=0x7fed1c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.860+0000 7fed3d39a700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 0x7fed1c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:42:21.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:21.864+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed34051aa0 con 0x7fed380ae8a0 2023-10-07T09:42:21.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:21.895+0000 7fd94b1291c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:42:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.061+0000 7fed3d39a700 1 -- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 msgr2=0x7fed1c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.061+0000 7fed3d39a700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 0x7fed1c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:42:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.065+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fed380ac4f0 con 0x7fed380ae8a0 2023-10-07T09:42:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.069+0000 7fed2dffb700 1 -- 172.21.15.29:0/2448313233 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fed34084020 con 0x7fed380ae8a0 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 msgr2=0x7fed1c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 0x7fed1c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 msgr2=0x7fed3816d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 0x7fed3816d3e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fed3400ecb0 tx=0x7fed340091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 shutdown_connections 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed1c066f80 0x7fed1c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3815ad30 0x7fed38069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380ae8a0 0x7fed3816d3e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 --2- 172.21.15.29:0/2448313233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed3815a960 0x7fed3814dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.071+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 >> 172.21.15.29:0/2448313233 conn(0x7fed38067610 msgr2=0x7fed3814f870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.072+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 shutdown_connections 2023-10-07T09:42:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.072+0000 7fed3fdff700 1 -- 172.21.15.29:0/2448313233 wait complete. 2023-10-07T09:42:22.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:22.091 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T09:42:22.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-10-07T09:42:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.349+0000 7f93129ff700 1 Processor -- start 2023-10-07T09:42:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.349+0000 7f93129ff700 1 -- start start 2023-10-07T09:42:22.351 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:22.350+0000 7f7059a801c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:42:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f93129ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f930c0cf2c0 0x7f930c164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f93129ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c0cc940 con 0x7f930c0cb8c0 2023-10-07T09:42:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f93129ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c0b2df0 con 0x7f930c0b2a20 2023-10-07T09:42:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f93129ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c0cbd30 con 0x7f930c0cf2c0 2023-10-07T09:42:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f9310f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f930c0cf2c0 0x7f930c164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f9310f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f930c0cf2c0 0x7f930c164330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59226/0 (socket says 172.21.15.29:59226) 2023-10-07T09:42:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f930b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f930c0cb8c0 0x7f930c0cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37138/0 (socket says 172.21.15.29:37138) 2023-10-07T09:42:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f9310f9c700 1 -- 172.21.15.29:0/2417522690 learned_addr learned my addr 172.21.15.29:0/2417522690 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.351+0000 7f930bfff700 1 --1- 172.21.15.29:0/2417522690 >> v1:172.21.15.144:6789/0 conn(0x7f930c0b2a20 0x7f930c0cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:34210/0 (socket says 172.21.15.29:34210) 2023-10-07T09:42:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f9310f9c700 1 -- 172.21.15.29:0/2417522690 >> v1:172.21.15.144:6789/0 conn(0x7f930c0b2a20 legacy=0x7f930c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f9310f9c700 1 -- 172.21.15.29:0/2417522690 >> v1:172.21.15.29:6789/0 conn(0x7f930c0cb8c0 legacy=0x7f930c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f9310f9c700 1 -- 172.21.15.29:0/2417522690 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0cc010 con 0x7f930c0cf2c0 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f930affd700 1 -- 172.21.15.29:0/2417522690 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609740161 0 0) 0x7f930c0cc940 con 0x7f930c0cb8c0 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f9310f9c700 1 --2- 172.21.15.29:0/2417522690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0cf2c0 0x7f930c164330 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9304009f20 tx=0x7f9304009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a4ba78ef0a8939 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f930affd700 1 -- 172.21.15.29:0/2417522690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930400e070 con 0x7f930c0cf2c0 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f930affd700 1 -- 172.21.15.29:0/2417522690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930400ba80 con 0x7f930c0cf2c0 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.352+0000 7f930affd700 1 -- 172.21.15.29:0/2417522690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93040092e0 con 0x7f930c0cf2c0 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 -- 172.21.15.29:0/2417522690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0cf2c0 msgr2=0x7f930c164330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 --2- 172.21.15.29:0/2417522690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0cf2c0 0x7f930c164330 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9304009f20 tx=0x7f9304009840 comp rx=0 tx=0).stop 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 -- 172.21.15.29:0/2417522690 shutdown_connections 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 --2- 172.21.15.29:0/2417522690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0cf2c0 0x7f930c164330 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 -- 172.21.15.29:0/2417522690 >> 172.21.15.29:0/2417522690 conn(0x7f930c099820 msgr2=0x7f930c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 -- 172.21.15.29:0/2417522690 shutdown_connections 2023-10-07T09:42:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.353+0000 7f93129ff700 1 -- 172.21.15.29:0/2417522690 wait complete. 2023-10-07T09:42:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.355+0000 7f93129ff700 1 Processor -- start 2023-10-07T09:42:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.355+0000 7f93129ff700 1 -- start start 2023-10-07T09:42:22.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f93129ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 0x7f930c0b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:22.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f93129ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f930c0cb8c0 0x7f930c0c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f93129ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f930c0cf2c0 0x7f930c0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f93129ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c15cf20 con 0x7f930c0b2a20 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f93129ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0b5660 con 0x7f930c0cb8c0 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f93129ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0b3730 con 0x7f930c0cf2c0 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f930bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 0x7f930c0b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f930bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 0x7f930c0b5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59236/0 (socket says 172.21.15.29:59236) 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f930bfff700 1 -- 172.21.15.29:0/3869800283 learned_addr learned my addr 172.21.15.29:0/3869800283 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f930c0cb8c0 0x7f930c0c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.356+0000 7f9310f9c700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f930c0cf2c0 0x7f930c0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f930bfff700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f930c0cf2c0 msgr2=0x7f930c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f930bfff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f930c0cf2c0 0x7f930c0ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f930bfff700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f930c0cb8c0 msgr2=0x7f930c0c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f930bfff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f930c0cb8c0 0x7f930c0c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f930bfff700 1 -- 172.21.15.29:0/3869800283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9304009ae0 con 0x7f930c0b2a20 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f930bfff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 0x7f930c0b5120 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00c790 tx=0x7f92fc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc005b90 con 0x7f930c0b2a20 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc005cf0 con 0x7f930c0b2a20 2023-10-07T09:42:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c0b5890 con 0x7f930c0b2a20 2023-10-07T09:42:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc005ea0 con 0x7f930c0b2a20 2023-10-07T09:42:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.357+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c0b6130 con 0x7f930c0b2a20 2023-10-07T09:42:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.359+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c0b3930 con 0x7f930c0b2a20 2023-10-07T09:42:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.359+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f92fc0075f0 con 0x7f930c0b2a20 2023-10-07T09:42:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.360+0000 7f92f3fff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.360+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f92fc083cb0 con 0x7f930c0b2a20 2023-10-07T09:42:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.360+0000 7f930b7fe700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.360+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:42:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.365+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92fc050d20 con 0x7f930c0b2a20 2023-10-07T09:42:22.462 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:22.461+0000 7fd94b1291c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:42:22.521 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:22.520+0000 7f7059a801c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:42:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.561+0000 7f930b7fe700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.561+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:42:22.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.593+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f930c0b5c80 con 0x7f92f8066f80 2023-10-07T09:42:22.638 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:22.637+0000 7fd94b1291c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:42:22.927 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:22.926+0000 7f7059a801c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:42:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.962+0000 7f930b7fe700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:22.962+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:42:23.064 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:23.064+0000 7fd94b1291c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:42:23.691 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:23.690+0000 7f7059a801c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:42:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:23.763+0000 7f930b7fe700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:23.763+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:42:23.776 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:23.775+0000 7f7059a801c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:42:23.874 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:23.873+0000 7fd94b1291c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:42:23.929 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:23.927+0000 7f7059a801c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:42:23.964 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:23.963+0000 7fd94b1291c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:42:24.015 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.014+0000 7f7059a801c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:42:24.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.123+0000 7fd94b1291c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:42:24.215 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.214+0000 7fd94b1291c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:42:24.231 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.230+0000 7f7059a801c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:42:24.325 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.325+0000 7f7059a801c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:42:24.420 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.419+0000 7f7059a801c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:42:24.433 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.432+0000 7fd94b1291c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:42:24.514 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.512+0000 7f7059a801c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:42:24.532 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.531+0000 7fd94b1291c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:42:24.630 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.629+0000 7fd94b1291c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:42:24.699 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.698+0000 7f7059a801c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:42:24.728 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.727+0000 7fd94b1291c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:42:24.794 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:24.793+0000 7f7059a801c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:42:24.921 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:24.920+0000 7fd94b1291c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:42:25.021 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:25.020+0000 7fd94b1291c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:42:25.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:25.130+0000 7f7059a801c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:42:25.219 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:25.218+0000 7f7059a801c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:42:25.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:25.365+0000 7f930b7fe700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:42:25.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:25.365+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:42:25.367 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:25.366+0000 7fd94b1291c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:42:25.458 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:25.457+0000 7fd94b1291c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:42:25.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:25.759+0000 7f7059a801c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:42:25.945 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:25.941+0000 7f7059a801c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:42:26.069 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:26.015+0000 7fd94b1291c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:42:26.206 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:26.205+0000 7fd94b1291c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:42:26.317 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:26.316+0000 7f7059a801c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:42:26.585 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:26.584+0000 7fd94b1291c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:42:26.738 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:26.737+0000 7f7059a801c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:42:26.916 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:26.915+0000 7f7059a801c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:42:27.019 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:27.018+0000 7fd94b1291c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:42:27.101 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.100+0000 7f7059a801c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:42:27.209 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:27.208+0000 7fd94b1291c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:42:27.210 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.209+0000 7f7059a801c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:42:27.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:27.222+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 18) v1 ==== 84468+0+0 (secure 0 0 0) 0x7f92fc0510f0 con 0x7f930c0b2a20 2023-10-07T09:42:27.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:27.222+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:42:27.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:27.222+0000 7f92f3fff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:27.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.340+0000 7f6fcc7f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.340+0000 7f6fcc7f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.340+0000 7f6fcc7f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.340+0000 7f6fcc7f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.340+0000 7f6fcc7f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.341+0000 7f6fc97ee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.341+0000 7f6fc97ee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.341+0000 7f6fc97ee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.341+0000 7f6fc97ee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:42:27.341+0000 7f6fc97ee700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:42:27.421 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:27.420+0000 7fd94b1291c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:42:27.532 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:42:27.531+0000 7fd94b1291c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:42:28.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.226+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 19) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f92fc0704a0 con 0x7f930c0b2a20 2023-10-07T09:42:28.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.227+0000 7f92f3fff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:28.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.227+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f930c0b5c80 con 0x7f92f8066f80 2023-10-07T09:42:28.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.228+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:28.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.231+0000 7f930b7fe700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f930c0ca760 tx=0x7f9300007d30 comp rx=0 tx=0).ready entity=mgr.5465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.303+0000 7f92f3fff700 1 -- 172.21.15.29:0/3869800283 <== mgr.5465 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f930c0b5c80 con 0x7f92f8066f80 2023-10-07T09:42:28.305 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-07T09:42:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 msgr2=0x7f92f8069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f930c0ca760 tx=0x7f9300007d30 comp rx=0 tx=0).stop 2023-10-07T09:42:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 msgr2=0x7f930c0b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 0x7f930c0b5120 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00c790 tx=0x7f92fc007f60 comp rx=0 tx=0).stop 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 shutdown_connections 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92f8066f80 0x7f92f8069430 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f930c0cf2c0 0x7f930c0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c0b2a20 0x7f930c0b5120 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 --2- 172.21.15.29:0/3869800283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f930c0cb8c0 0x7f930c0c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 >> 172.21.15.29:0/3869800283 conn(0x7f930c099820 msgr2=0x7f930c157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 shutdown_connections 2023-10-07T09:42:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.306+0000 7f93129ff700 1 -- 172.21.15.29:0/3869800283 wait complete. 2023-10-07T09:42:28.326 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-07T09:42:28.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 2M 2023-10-07T09:42:28.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.526+0000 7ff902dff700 1 Processor -- start 2023-10-07T09:42:28.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.526+0000 7ff902dff700 1 -- start start 2023-10-07T09:42:28.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.527+0000 7ff902dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8fc159b40 0x7ff8fc0ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:28.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.527+0000 7ff902dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0c8c70 con 0x7ff8fc1596d0 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.527+0000 7ff902dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0c8f50 con 0x7ff8fc1559f0 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.527+0000 7ff902dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc0c98f0 con 0x7ff8fc159b40 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8fc159b40 0x7ff8fc0ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8fc159b40 0x7ff8fc0ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47348/0 (socket says 172.21.15.29:47348) 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 -- 172.21.15.29:0/784351956 learned_addr learned my addr 172.21.15.29:0/784351956 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 -- 172.21.15.29:0/784351956 >> v1:172.21.15.144:6789/0 conn(0x7ff8fc1559f0 legacy=0x7ff8fc0b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 -- 172.21.15.29:0/784351956 >> v1:172.21.15.29:6789/0 conn(0x7ff8fc1596d0 legacy=0x7ff8fc0c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 -- 172.21.15.29:0/784351956 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0c9230 con 0x7ff8fc159b40 2023-10-07T09:42:28.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.528+0000 7ff90139c700 1 --2- 172.21.15.29:0/784351956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ca420 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009f20 tx=0x7ff8f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f9b26f227fb992 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:28.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff8fb7fe700 1 -- 172.21.15.29:0/784351956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f40092e0 con 0x7ff8fc159b40 2023-10-07T09:42:28.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff8fb7fe700 1 -- 172.21.15.29:0/784351956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f400ba80 con 0x7ff8fc159b40 2023-10-07T09:42:28.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff8fb7fe700 1 -- 172.21.15.29:0/784351956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f400be20 con 0x7ff8fc159b40 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff902dff700 1 -- 172.21.15.29:0/784351956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc159b40 msgr2=0x7ff8fc0ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff902dff700 1 --2- 172.21.15.29:0/784351956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ca420 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009f20 tx=0x7ff8f4009840 comp rx=0 tx=0).stop 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff902dff700 1 -- 172.21.15.29:0/784351956 shutdown_connections 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff902dff700 1 --2- 172.21.15.29:0/784351956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ca420 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.529+0000 7ff902dff700 1 -- 172.21.15.29:0/784351956 >> 172.21.15.29:0/784351956 conn(0x7ff8fc067610 msgr2=0x7ff8fc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.530+0000 7ff902dff700 1 -- 172.21.15.29:0/784351956 shutdown_connections 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.530+0000 7ff902dff700 1 -- 172.21.15.29:0/784351956 wait complete. 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.531+0000 7ff902dff700 1 Processor -- start 2023-10-07T09:42:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.531+0000 7ff902dff700 1 -- start start 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff902dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 0x7ff8fc13fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff902dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8fc1596d0 0x7ff8fc0adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff902dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff902dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc154280 con 0x7ff8fc1559f0 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff902dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc15abf0 con 0x7ff8fc159b40 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff902dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc1607c0 con 0x7ff8fc1596d0 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff900b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 0x7ff8fc13fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff900b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 0x7ff8fc13fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47364/0 (socket says 172.21.15.29:47364) 2023-10-07T09:42:28.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.532+0000 7ff900b9b700 1 -- 172.21.15.29:0/4115596335 learned_addr learned my addr 172.21.15.29:0/4115596335 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff90139c700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff900b9b700 1 -- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8fc1596d0 msgr2=0x7ff8fc0adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff8fbfff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8fc1596d0 0x7ff8fc0adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff900b9b700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8fc1596d0 0x7ff8fc0adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff900b9b700 1 -- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8fc159b40 msgr2=0x7ff8fc0ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff900b9b700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff900b9b700 1 -- 172.21.15.29:0/4115596335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4009ae0 con 0x7ff8fc1559f0 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff90139c700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff8fbfff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8fc1596d0 0x7ff8fc0adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:42:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff900b9b700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 0x7ff8fc13fe60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000c590 tx=0x7ff8f000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:28.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0011590 con 0x7ff8fc1559f0 2023-10-07T09:42:28.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f00116f0 con 0x7ff8fc1559f0 2023-10-07T09:42:28.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0011ab0 con 0x7ff8fc1559f0 2023-10-07T09:42:28.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc160200 con 0x7ff8fc1559f0 2023-10-07T09:42:28.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.533+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc15a880 con 0x7ff8fc1559f0 2023-10-07T09:42:28.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.534+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc0718b0 con 0x7ff8fc1559f0 2023-10-07T09:42:28.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.536+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 84595+0+0 (secure 0 0 0) 0x7ff8f001b3b0 con 0x7ff8fc1559f0 2023-10-07T09:42:28.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.537+0000 7ff8f8ff9700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8e0066dd0 0x7ff8e0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:28.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.537+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6311+0+0 (secure 0 0 0) 0x7ff8f0083b20 con 0x7ff8fc1559f0 2023-10-07T09:42:28.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.538+0000 7ff8fbfff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8e0066dd0 0x7ff8e0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:28.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.538+0000 7ff8fbfff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8e0066dd0 0x7ff8e0069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc158e90 tx=0x7ff8e8009040 comp rx=0 tx=0).ready entity=mgr.5465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:28.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.542+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8f0050d10 con 0x7ff8fc1559f0 2023-10-07T09:42:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.767+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "2M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8fc0c9510 con 0x7ff8e0066dd0 2023-10-07T09:42:28.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.829+0000 7ff8f8ff9700 1 -- 172.21.15.29:0/4115596335 <== mgr.5465 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff8fc0c9510 con 0x7ff8e0066dd0 2023-10-07T09:42:28.830 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-07T09:42:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8e0066dd0 msgr2=0x7ff8e0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8e0066dd0 0x7ff8e0069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc158e90 tx=0x7ff8e8009040 comp rx=0 tx=0).stop 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 msgr2=0x7ff8fc13fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 0x7ff8fc13fe60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000c590 tx=0x7ff8f000c950 comp rx=0 tx=0).stop 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 shutdown_connections 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8e0066dd0 0x7ff8e0069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8fc159b40 0x7ff8fc0ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8fc1559f0 0x7ff8fc13fe60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 --2- 172.21.15.29:0/4115596335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8fc1596d0 0x7ff8fc0adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 >> 172.21.15.29:0/4115596335 conn(0x7ff8fc067610 msgr2=0x7ff8fc06d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 shutdown_connections 2023-10-07T09:42:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:28.831+0000 7ff902dff700 1 -- 172.21.15.29:0/4115596335 wait complete. 2023-10-07T09:42:28.849 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-07T09:42:28.850 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-07-09_43_00 in ~32s... 2023-10-07T09:42:28.850 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-07-09_44_00 in ~92s... 2023-10-07T09:42:28.850 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M', '2M'], expected_retention: [] 2023-10-07T09:42:28.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-10-07T09:42:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.048+0000 7f7827fff700 1 Processor -- start 2023-10-07T09:42:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.048+0000 7f7827fff700 1 -- start start 2023-10-07T09:42:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7827fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 0x7f78200cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7827fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 0x7f782015aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7827fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78200ad160 con 0x7f78200accf0 2023-10-07T09:42:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7827fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782015a140 con 0x7f782015a590 2023-10-07T09:42:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7827fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78200adca0 con 0x7f78200afeb0 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f782659c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 0x7f782015aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7825d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 0x7f78200cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f782559a700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f78200accf0 0x7f78200aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:42214/0 (socket says 172.21.15.29:42214) 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f782659c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 0x7f782015aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:58584/0 (socket says 172.21.15.29:58584) 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f7825d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 0x7f78200cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:48974/0 (socket says 172.21.15.29:48974) 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.050+0000 7f782559a700 1 -- 172.21.15.29:0/4169933864 learned_addr learned my addr 172.21.15.29:0/4169933864 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:42:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7824d99700 1 -- 172.21.15.29:0/4169933864 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148693814 0 0) 0x7f78200ad160 con 0x7f78200accf0 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7824d99700 1 -- 172.21.15.29:0/4169933864 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f780c003570 con 0x7f78200accf0 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7825d9b700 1 -- 172.21.15.29:0/4169933864 >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 msgr2=0x7f782015aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7825d9b700 1 --2- 172.21.15.29:0/4169933864 >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 0x7f782015aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7825d9b700 1 -- 172.21.15.29:0/4169933864 >> v1:172.21.15.144:6789/0 conn(0x7f78200accf0 legacy=0x7f78200aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7825d9b700 1 -- 172.21.15.29:0/4169933864 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78200ad400 con 0x7f78200afeb0 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f782659c700 1 --2- 172.21.15.29:0/4169933864 >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 0x7f782015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.051+0000 7f7825d9b700 1 --2- 172.21.15.29:0/4169933864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 0x7f78200cc810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f781c0098e0 tx=0x7f781c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ad0a537c4dd3069 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7824d99700 1 -- 172.21.15.29:0/4169933864 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c011630 con 0x7f78200afeb0 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7824d99700 1 -- 172.21.15.29:0/4169933864 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c011790 con 0x7f78200afeb0 2023-10-07T09:42:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7824d99700 1 -- 172.21.15.29:0/4169933864 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c011b30 con 0x7f78200afeb0 2023-10-07T09:42:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7827fff700 1 -- 172.21.15.29:0/4169933864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 msgr2=0x7f78200cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7827fff700 1 --2- 172.21.15.29:0/4169933864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 0x7f78200cc810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f781c0098e0 tx=0x7f781c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7827fff700 1 -- 172.21.15.29:0/4169933864 shutdown_connections 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7827fff700 1 --2- 172.21.15.29:0/4169933864 >> v2:172.21.15.144:3300/0 conn(0x7f782015a590 0x7f782015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7827fff700 1 --2- 172.21.15.29:0/4169933864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78200afeb0 0x7f78200cc810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.052+0000 7f7827fff700 1 -- 172.21.15.29:0/4169933864 >> 172.21.15.29:0/4169933864 conn(0x7f7820067610 msgr2=0x7f7820067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.053+0000 7f7827fff700 1 -- 172.21.15.29:0/4169933864 shutdown_connections 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.053+0000 7f7827fff700 1 -- 172.21.15.29:0/4169933864 wait complete. 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.054+0000 7f7827fff700 1 Processor -- start 2023-10-07T09:42:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.054+0000 7f7827fff700 1 -- start start 2023-10-07T09:42:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f7827fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78200accf0 0x7f7820148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f7827fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 0x7f782015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f7827fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f782015a590 0x7f782016a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f7827fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78200c9b50 con 0x7f78200afeb0 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f7827fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782016ab50 con 0x7f78200accf0 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f7827fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78200c9220 con 0x7f782015a590 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f782559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 0x7f782015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f782559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 0x7f782015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47374/0 (socket says 172.21.15.29:47374) 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.055+0000 7f782559a700 1 -- 172.21.15.29:0/2852908887 learned_addr learned my addr 172.21.15.29:0/2852908887 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f7825d9b700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78200accf0 0x7f7820148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782659c700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f782015a590 0x7f782016a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782559a700 1 -- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f782015a590 msgr2=0x7f782016a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782559a700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f782015a590 0x7f782016a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782559a700 1 -- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78200accf0 msgr2=0x7f7820148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782559a700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78200accf0 0x7f7820148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782559a700 1 -- 172.21.15.29:0/2852908887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781800bc30 con 0x7f78200afeb0 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782659c700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f782015a590 0x7f782016a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f782559a700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 0x7f782015e680 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f781000c5c0 tx=0x7f7810007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810005b30 con 0x7f78200afeb0 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7810005cb0 con 0x7f78200afeb0 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c009420 con 0x7f78200afeb0 2023-10-07T09:42:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.056+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810007320 con 0x7f78200afeb0 2023-10-07T09:42:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.057+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78200c97e0 con 0x7f78200afeb0 2023-10-07T09:42:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.058+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78200c9d00 con 0x7f78200afeb0 2023-10-07T09:42:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.059+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f781001b3b0 con 0x7f78200afeb0 2023-10-07T09:42:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.059+0000 7f7815ffb700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066dd0 0x7f7804069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:42:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.059+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f7810083a50 con 0x7f78200afeb0 2023-10-07T09:42:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.060+0000 7f7825d9b700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066dd0 0x7f7804069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:42:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.061+0000 7f7825d9b700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066dd0 0x7f7804069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f781c0091e0 tx=0x7f781c017000 comp rx=0 tx=0).ready entity=mgr.5465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:42:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.064+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78200c9d00 con 0x7f78200afeb0 2023-10-07T09:42:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.232+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 20) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f7810051010 con 0x7f78200afeb0 2023-10-07T09:42:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.296+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f78200c9970 con 0x7f7804066dd0 2023-10-07T09:42:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.305+0000 7f7815ffb700 1 -- 172.21.15.29:0/2852908887 <== mgr.5465 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f78200c9970 con 0x7f7804066dd0 2023-10-07T09:42:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:42:29.306 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-10-07T09:42:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066dd0 msgr2=0x7f7804069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066dd0 0x7f7804069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f781c0091e0 tx=0x7f781c017000 comp rx=0 tx=0).stop 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 msgr2=0x7f782015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 0x7f782015e680 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f781000c5c0 tx=0x7f7810007e00 comp rx=0 tx=0).stop 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 shutdown_connections 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066dd0 0x7f7804069280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f782015a590 0x7f782016a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78200accf0 0x7f7820148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 --2- 172.21.15.29:0/2852908887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200afeb0 0x7f782015e680 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 >> 172.21.15.29:0/2852908887 conn(0x7f7820067610 msgr2=0x7f7820157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 shutdown_connections 2023-10-07T09:42:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:42:29.307+0000 7f7827fff700 1 -- 172.21.15.29:0/2852908887 wait complete. 2023-10-07T09:42:29.326 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-10-07T09:42:29.326 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M', '2M'], 'retention': [{}, {}]} 2023-10-07T09:42:29.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:29.327 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:29.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:29.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:29.375 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:30.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:30.376 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:30.430 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:30.430 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:30.430 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:31.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:31.431 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:31.485 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:31.486 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:31.486 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:32.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:32.487 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:32.541 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:32.541 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:32.541 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:33.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:33.543 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:33.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:33.596 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:33.596 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:34.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:34.597 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:34.651 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:34.651 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:34.651 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:35.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:35.653 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:35.707 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:35.707 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:35.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:36.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:36.709 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:36.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:36.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:36.764 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:37.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:37.765 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:37.819 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:37.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:37.819 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:38.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:38.820 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:38.875 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:38.875 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:38.875 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:39.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:39.876 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:39.930 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:39.930 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:39.930 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:40.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:40.931 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:40.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:40.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:40.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:41.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:41.987 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:42.041 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:42.041 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:42.041 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:43.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:43.042 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:43.096 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:43.096 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:43.096 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:44.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:44.097 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:44.154 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:44.154 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:44.154 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:45.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:45.155 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:45.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:45.209 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:45.209 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:46.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:46.210 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:46.264 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:46.264 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:46.264 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:47.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:47.266 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:47.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:47.320 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:47.320 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:48.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:48.321 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:48.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:48.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:48.375 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:49.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:49.377 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:49.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:49.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:49.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:50.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:50.572 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:50.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:50.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:50.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:51.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:42:51.041 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:42:51.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:51.627 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:51.682 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:51.682 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:51.682 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:52.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:52.684 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:52.738 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:52.738 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:52.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:53.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:53.740 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:53.793 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:53.793 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:53.793 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:54.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:54.794 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:54.849 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:54.849 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:54.849 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:55.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:55.850 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:55.904 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:55.904 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:55.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:56.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:56.931 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:56.985 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:56.985 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:56.985 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:57.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:57.987 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:58.040 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:58.040 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:58.041 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:42:59.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:42:59.041 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:42:59.095 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:42:59.095 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:42:59.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:00.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:00.096 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:00.153 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:43:00.154 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:00.154 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:00.844 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:43:00.842+0000 7f6fbcfd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:43:00.844 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:43:00.842+0000 7f6fbcfd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:43:00.844 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:43:00.842+0000 7f6fbcfd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:43:00.844 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:43:00.842+0000 7f6fbcfd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:43:00.844 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:43:00.842+0000 7f6fbcfd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:43:01.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:01.155 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:01.208 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:01.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:01.209 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:01.209 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:01.209 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-07-09_43_00_UTC'] 2023-10-07T09:43:01.210 DEBUG:tasks.cephfs.test_snap_schedules:exec=1696671748.8500118, now = 1696671781.2100754, timo = 32 2023-10-07T09:43:02.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:02.210 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:02.262 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:02.263 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:02.263 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:02.263 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:03.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:03.264 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:03.316 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:03.317 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:03.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:03.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:04.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:04.318 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:04.371 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:04.372 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:04.372 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:04.372 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:05.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:05.373 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:05.426 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:05.427 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:05.427 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:05.427 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:06.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:06.428 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:06.480 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:06.481 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:06.482 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:06.482 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:07.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:07.483 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:07.541 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:07.542 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:07.542 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:07.542 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:08.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:08.543 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:08.596 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:08.597 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:08.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:08.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:09.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:09.599 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:09.652 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:09.654 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:09.654 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:09.654 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:10.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:10.656 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:10.708 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:10.709 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:10.709 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:10.710 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:11.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:11.711 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:11.764 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:11.765 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:11.765 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:11.765 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:12.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:12.767 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:12.819 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:12.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:12.820 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:12.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:13.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:13.821 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:13.874 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:13.876 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:13.876 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:13.876 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:14.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:14.877 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:14.929 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:14.931 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:14.931 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:14.931 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:15.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:15.932 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:15.985 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:15.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:15.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:15.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:16.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:16.987 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:17.040 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:17.041 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:17.041 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:17.041 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:18.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:18.042 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:18.095 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:18.096 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:18.096 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:18.096 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:19.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:19.097 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:19.150 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:19.151 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:19.151 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:19.151 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:20.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:20.152 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:20.206 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:20.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:20.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:20.207 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:21.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:43:21.179 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:43:21.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:21.208 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:21.271 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:21.272 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:21.272 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:21.272 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:22.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:22.273 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:22.326 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:22.327 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:22.327 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:22.327 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:23.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:23.329 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:23.382 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:23.383 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:23.383 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:23.383 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:24.384 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:24.384 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:24.436 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:24.437 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:24.437 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:24.438 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:25.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:25.439 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:25.491 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:25.492 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:25.492 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:25.493 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:26.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:26.494 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:26.546 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:26.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:26.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:26.548 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:27.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:27.548 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:27.601 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:27.602 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:27.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:27.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:28.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:28.604 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:28.658 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:28.659 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:28.660 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:28.660 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:29.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:29.660 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:29.713 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:29.714 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:29.714 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:29.714 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:30.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:30.715 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:30.768 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:30.769 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:30.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:30.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:31.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:31.771 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:31.824 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:31.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:31.825 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:31.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:32.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:32.978 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:33.031 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:33.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:33.033 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:33.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:34.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:34.034 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:34.087 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:34.088 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:34.088 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:34.088 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:35.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:35.089 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:35.142 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:35.143 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:35.143 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:35.143 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:36.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:36.145 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:36.197 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:36.199 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:36.199 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:36.199 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:37.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:37.200 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:37.253 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:37.255 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:37.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:37.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:38.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:38.256 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:38.309 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:38.310 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:38.310 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:38.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:39.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:39.311 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:39.364 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:39.365 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:39.365 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:39.365 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:40.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:40.366 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:40.418 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:40.420 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:40.420 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:40.420 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:41.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:41.421 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:41.474 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:41.475 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:41.475 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:41.475 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:42.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:42.477 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:42.529 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:42.530 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:42.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:42.531 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:43.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:43.532 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:43.585 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:43.586 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:43.586 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:43.587 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:44.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:44.587 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:44.640 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:44.641 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:44.641 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:44.641 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:45.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:45.643 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:45.696 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:45.697 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:45.697 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:45.697 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:46.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:46.699 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:46.751 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:46.752 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:46.752 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:46.752 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:47.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:47.753 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:47.806 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:47.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:47.807 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:47.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:48.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:48.809 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:48.862 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:48.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:48.863 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:48.863 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:49.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:49.865 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:49.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:49.918 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:49.919 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:49.919 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:50.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:50.920 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:50.972 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:50.974 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:50.974 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:50.974 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:51.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:43:51.291 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:43:51.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:51.975 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:52.028 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:52.029 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:52.029 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:52.029 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:53.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:53.030 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:53.083 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:53.084 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:53.084 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:53.084 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:54.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:54.085 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:54.138 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:54.139 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:54.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:54.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:55.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:55.140 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:55.193 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:55.194 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:55.194 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:55.194 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:56.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:56.196 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:56.249 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:56.250 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:56.250 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:56.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:57.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:57.252 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:57.304 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:57.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:57.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:57.305 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:58.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:58.307 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:58.359 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:58.360 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:58.361 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:58.361 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:43:59.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:43:59.362 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:43:59.415 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:43:59.416 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:43:59.416 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:43:59.416 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:44:00.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:00.417 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:44:00.470 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:44:00.471 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC'} 2023-10-07T09:44:00.471 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:44:00.471 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:44:01.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:01.473 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:44:01.526 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:44:01.526 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_44_00_UTC 2023-10-07T09:44:01.527 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_43_00_UTC', 'scheduled-2023-10-07-09_44_00_UTC'} 2023-10-07T09:44:01.527 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-07-09_44_00_UTC'} 2023-10-07T09:44:01.528 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:44:01.528 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-07-09_44_00_UTC'] 2023-10-07T09:44:01.528 DEBUG:tasks.cephfs.test_snap_schedules:exec=1696671748.8500118, now = 1696671841.5284882, timo = 92 2023-10-07T09:44:02.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-10-07T09:44:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.740+0000 7fbcd03ff700 1 Processor -- start 2023-10-07T09:44:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.740+0000 7fbcd03ff700 1 -- start start 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.741+0000 7fbcd03ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbcc813fd30 0x7fbcc806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.741+0000 7fbcd03ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc80c1d50 con 0x7fbcc813f960 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.741+0000 7fbcd03ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc80c0e60 con 0x7fbcc806f3b0 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.741+0000 7fbcd03ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc806f7f0 con 0x7fbcc813fd30 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.741+0000 7fbccd99a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbcc813fd30 0x7fbcc806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.741+0000 7fbcce19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbcc813f960 0x7fbcc80c6700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34302/0 (socket says 172.21.15.29:34302) 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbcce19b700 1 -- 172.21.15.29:0/3266273347 learned_addr learned my addr 172.21.15.29:0/3266273347 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbccd99a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbcc813fd30 0x7fbcc806ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57346/0 (socket says 172.21.15.29:57346) 2023-10-07T09:44:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbccd99a700 1 -- 172.21.15.29:0/3266273347 >> v1:172.21.15.144:6789/0 conn(0x7fbcc806f3b0 legacy=0x7fbcc8159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbccd99a700 1 -- 172.21.15.29:0/3266273347 >> v1:172.21.15.29:6789/0 conn(0x7fbcc813f960 legacy=0x7fbcc80c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbccd99a700 1 -- 172.21.15.29:0/3266273347 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc80c1140 con 0x7fbcc813fd30 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbccd99a700 1 --2- 172.21.15.29:0/3266273347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813fd30 0x7fbcc806ee70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80098e0 tx=0x7fbcb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178e7565b5417b4d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.742+0000 7fbccd199700 1 -- 172.21.15.29:0/3266273347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8009e80 con 0x7fbcc813fd30 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbccd199700 1 -- 172.21.15.29:0/3266273347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb800b760 con 0x7fbcc813fd30 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbccd199700 1 -- 172.21.15.29:0/3266273347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800bb00 con 0x7fbcc813fd30 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 -- 172.21.15.29:0/3266273347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813fd30 msgr2=0x7fbcc806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/3266273347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813fd30 0x7fbcc806ee70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80098e0 tx=0x7fbcb80091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 -- 172.21.15.29:0/3266273347 shutdown_connections 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/3266273347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813fd30 0x7fbcc806ee70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 -- 172.21.15.29:0/3266273347 >> 172.21.15.29:0/3266273347 conn(0x7fbcc80cc0c0 msgr2=0x7fbcc80cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 -- 172.21.15.29:0/3266273347 shutdown_connections 2023-10-07T09:44:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.743+0000 7fbcd03ff700 1 -- 172.21.15.29:0/3266273347 wait complete. 2023-10-07T09:44:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.745+0000 7fbcd03ff700 1 Processor -- start 2023-10-07T09:44:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.745+0000 7fbcd03ff700 1 -- start start 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcd03ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcc806f3b0 0x7fbcc816a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcd03ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 0x7fbcc8159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcd03ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc813fd30 0x7fbcc80cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcd03ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc816e3d0 con 0x7fbcc813f960 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcd03ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc816d500 con 0x7fbcc813fd30 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcd03ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc816aeb0 con 0x7fbcc806f3b0 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbccd99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 0x7fbcc8159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbccd99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 0x7fbcc8159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57358/0 (socket says 172.21.15.29:57358) 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbccd99a700 1 -- 172.21.15.29:0/407836743 learned_addr learned my addr 172.21.15.29:0/407836743 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcce99c700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc813fd30 0x7fbcc80cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.746+0000 7fbcce19b700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcc806f3b0 0x7fbcc816a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbccd99a700 1 -- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcc806f3b0 msgr2=0x7fbcc816a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbccd99a700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcc806f3b0 0x7fbcc816a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbccd99a700 1 -- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc813fd30 msgr2=0x7fbcc80cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbccd99a700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc813fd30 0x7fbcc80cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbccd99a700 1 -- 172.21.15.29:0/407836743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb8009420 con 0x7fbcc813f960 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbcce19b700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcc806f3b0 0x7fbcc816a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbccd99a700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 0x7fbcc8159d00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800b3c0 tx=0x7fbcb8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800beb0 con 0x7fbcc813f960 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb8020070 con 0x7fbcc813f960 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8011e60 con 0x7fbcc813f960 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc816d750 con 0x7fbcc813f960 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.747+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc816dd10 con 0x7fbcc813f960 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.748+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc804c3c0 con 0x7fbcc813f960 2023-10-07T09:44:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.749+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) v1 ==== 84871+0+0 (secure 0 0 0) 0x7fbcb8022030 con 0x7fbcc813f960 2023-10-07T09:44:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.750+0000 7fbcbe7fc700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbcac0670a0 0x7fbcac069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.750+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6369+0+0 (secure 0 0 0) 0x7fbcb8086770 con 0x7fbcc813f960 2023-10-07T09:44:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.751+0000 7fbcce19b700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbcac0670a0 0x7fbcac069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.751+0000 7fbcce19b700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbcac0670a0 0x7fbcac069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40058d0 tx=0x7fbcc4009040 comp rx=0 tx=0).ready entity=mgr.5465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:02.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.755+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcb8053590 con 0x7fbcc813f960 2023-10-07T09:44:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:02.982+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcc813d340 con 0x7fbcac0670a0 2023-10-07T09:44:03.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.051+0000 7fbcbe7fc700 1 -- 172.21.15.29:0/407836743 <== mgr.5465 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbcc813d340 con 0x7fbcac0670a0 2023-10-07T09:44:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-07T09:44:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbcac0670a0 msgr2=0x7fbcac069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbcac0670a0 0x7fbcac069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40058d0 tx=0x7fbcc4009040 comp rx=0 tx=0).stop 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 msgr2=0x7fbcc8159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 0x7fbcc8159d00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800b3c0 tx=0x7fbcb8016000 comp rx=0 tx=0).stop 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 shutdown_connections 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbcac0670a0 0x7fbcac069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc813fd30 0x7fbcc80cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcc806f3b0 0x7fbcc816a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 --2- 172.21.15.29:0/407836743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcc813f960 0x7fbcc8159d00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 >> 172.21.15.29:0/407836743 conn(0x7fbcc80cc0c0 msgr2=0x7fbcc80b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 shutdown_connections 2023-10-07T09:44:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.054+0000 7fbcd03ff700 1 -- 172.21.15.29:0/407836743 wait complete. 2023-10-07T09:44:03.072 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-07T09:44:03.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:03.072 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:44:03.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:44:03.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_44_00_UTC 2023-10-07T09:44:03.121 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_43_00_UTC 2023-10-07T09:44:03.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:03.121 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_43_00_UTC) 2023-10-07T09:44:03.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_44_00_UTC 2023-10-07T09:44:03.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:03.214 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_44_00_UTC) 2023-10-07T09:44:03.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:03.267 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T09:44:03.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T09:44:03.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.512+0000 7f916fbff700 1 Processor -- start 2023-10-07T09:44:03.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.512+0000 7f916fbff700 1 -- start start 2023-10-07T09:44:03.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916fbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f916814c8d0 0x7f91680afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:03.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916fbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:03.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916fbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9168157790 con 0x7f91680b0410 2023-10-07T09:44:03.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916fbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91680b0950 con 0x7f916814c8d0 2023-10-07T09:44:03.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916fbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9168157580 con 0x7f9168153cc0 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916d99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f916814c8d0 0x7f91680afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916e19c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91680b0410 0x7f9168158550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46530/0 (socket says 172.21.15.29:46530) 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916e19c700 1 -- 172.21.15.29:0/2327309436 learned_addr learned my addr 172.21.15.29:0/2327309436 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.514+0000 7f916d19a700 1 --2- 172.21.15.29:0/2327309436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916d99b700 1 -- 172.21.15.29:0/2327309436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 msgr2=0x7f9168159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916d99b700 1 --2- 172.21.15.29:0/2327309436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916d99b700 1 -- 172.21.15.29:0/2327309436 >> v1:172.21.15.29:6789/0 conn(0x7f91680b0410 legacy=0x7f9168158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916d99b700 1 -- 172.21.15.29:0/2327309436 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168157a70 con 0x7f916814c8d0 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916c999700 1 -- 172.21.15.29:0/2327309436 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268260708 0 0) 0x7f9168157790 con 0x7f91680b0410 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916d99b700 1 --2- 172.21.15.29:0/2327309436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916814c8d0 0x7f91680afed0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f91640098e0 tx=0x7f91640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64efd5cae7bc1bd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:03.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916c999700 1 -- 172.21.15.29:0/2327309436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164009e80 con 0x7f916814c8d0 2023-10-07T09:44:03.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916c999700 1 -- 172.21.15.29:0/2327309436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916400b860 con 0x7f916814c8d0 2023-10-07T09:44:03.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.515+0000 7f916c999700 1 -- 172.21.15.29:0/2327309436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916400bbe0 con 0x7f916814c8d0 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 -- 172.21.15.29:0/2327309436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916814c8d0 msgr2=0x7f91680afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 --2- 172.21.15.29:0/2327309436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916814c8d0 0x7f91680afed0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f91640098e0 tx=0x7f91640091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 -- 172.21.15.29:0/2327309436 shutdown_connections 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 --2- 172.21.15.29:0/2327309436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916814c8d0 0x7f91680afed0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 --2- 172.21.15.29:0/2327309436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 -- 172.21.15.29:0/2327309436 >> 172.21.15.29:0/2327309436 conn(0x7f9168067610 msgr2=0x7f9168067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 -- 172.21.15.29:0/2327309436 shutdown_connections 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.516+0000 7f916fbff700 1 -- 172.21.15.29:0/2327309436 wait complete. 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.517+0000 7f916fbff700 1 Processor -- start 2023-10-07T09:44:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.517+0000 7f916fbff700 1 -- start start 2023-10-07T09:44:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916fbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 0x7f916814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916fbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916814c8d0 0x7f9168152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916fbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916fbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91681530e0 con 0x7f91680b0410 2023-10-07T09:44:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916fbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916814ddb0 con 0x7f916814c8d0 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916fbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916814d1a0 con 0x7f9168153cc0 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.518+0000 7f916d99b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 0x7f916814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 0x7f916814e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47118/0 (socket says 172.21.15.29:47118) 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 -- 172.21.15.29:0/1059399379 learned_addr learned my addr 172.21.15.29:0/1059399379 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d19a700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916814c8d0 0x7f9168152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916e19c700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 -- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 msgr2=0x7f9168152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 -- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916814c8d0 msgr2=0x7f9168152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916814c8d0 0x7f9168152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 -- 172.21.15.29:0/1059399379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164009420 con 0x7f91680b0410 2023-10-07T09:44:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f916d99b700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 0x7f916814e240 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f91680641e0 tx=0x7f9164017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916401f070 con 0x7f91680b0410 2023-10-07T09:44:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9164000c90 con 0x7f91680b0410 2023-10-07T09:44:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.519+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164025420 con 0x7f91680b0410 2023-10-07T09:44:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.520+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916814d480 con 0x7f91680b0410 2023-10-07T09:44:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.520+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916814da40 con 0x7f91680b0410 2023-10-07T09:44:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.521+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916804c3c0 con 0x7f91680b0410 2023-10-07T09:44:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.521+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f9164023070 con 0x7f91680b0410 2023-10-07T09:44:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.522+0000 7f915dffb700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f914c0670a0 0x7f914c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.523+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6369+0+0 (secure 0 0 0) 0x7f9164086070 con 0x7f91680b0410 2023-10-07T09:44:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.523+0000 7f916d19a700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f914c0670a0 0x7f914c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.524+0000 7f916d19a700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f914c0670a0 0x7f914c069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9168154ed0 tx=0x7f9158008040 comp rx=0 tx=0).ready entity=mgr.5465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:03.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.527+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91640b75a0 con 0x7f91680b0410 2023-10-07T09:44:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:03.787+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f916814e090 con 0x7f91680b0410 2023-10-07T09:44:04.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.198+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f9164053260 con 0x7f91680b0410 2023-10-07T09:44:04.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.199+0000 7f915dffb700 1 -- 172.21.15.29:0/1059399379 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v23) v1 ==== 93+0+0 (secure 0 0 0) 0x7f9164052e90 con 0x7f91680b0410 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.202+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f914c0670a0 msgr2=0x7f914c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.202+0000 7f916fbff700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f914c0670a0 0x7f914c069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9168154ed0 tx=0x7f9158008040 comp rx=0 tx=0).stop 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.202+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 msgr2=0x7f916814e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.202+0000 7f916fbff700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 0x7f916814e240 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f91680641e0 tx=0x7f9164017000 comp rx=0 tx=0).stop 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.202+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 shutdown_connections 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f914c0670a0 0x7f914c069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9168153cc0 0x7f9168152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91680b0410 0x7f916814e240 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 --2- 172.21.15.29:0/1059399379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916814c8d0 0x7f9168152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 >> 172.21.15.29:0/1059399379 conn(0x7f9168067610 msgr2=0x7f916806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:04.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 shutdown_connections 2023-10-07T09:44:04.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.203+0000 7f916fbff700 1 -- 172.21.15.29:0/1059399379 wait complete. 2023-10-07T09:44:04.223 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:44:04.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:04.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:04.413 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:04.412+0000 7f9c3d3881c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:44:04.421 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:04.420+0000 7fa55e89d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:44:04.435 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:44:04.435 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:44:04.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:04.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T09:44:04.542 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:44:04.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:04.543 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:44:04.600 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:04.599+0000 7fa55e89d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:44:04.615 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:04.614+0000 7f9c3d3881c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:44:04.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.820+0000 7f1066dff700 1 Processor -- start 2023-10-07T09:44:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.820+0000 7f1066dff700 1 -- start start 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.821+0000 7f1066dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1060159750 0x7f10600d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.821+0000 7f1066dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10600ce8a0 0x7f10600c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.821+0000 7f1066dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10600cbf20 con 0x7f1060159b20 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.821+0000 7f1066dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10600cc200 con 0x7f10600ce8a0 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.821+0000 7f1066dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10600cc4e0 con 0x7f1060159750 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f106539c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10600ce8a0 0x7f10600c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f105ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1060159b20 0x7f10600ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46536/0 (socket says 172.21.15.29:46536) 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f105ffff700 1 -- 172.21.15.29:0/1879487600 learned_addr learned my addr 172.21.15.29:0/1879487600 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f1064b9b700 1 --2- 172.21.15.29:0/1879487600 >> v2:172.21.15.144:3300/0 conn(0x7f1060159750 0x7f10600d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f105f7fe700 1 -- 172.21.15.29:0/1879487600 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744060284 0 0) 0x7f10600cbf20 con 0x7f1060159b20 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f106539c700 1 -- 172.21.15.29:0/1879487600 >> v2:172.21.15.144:3300/0 conn(0x7f1060159750 msgr2=0x7f10600d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f106539c700 1 --2- 172.21.15.29:0/1879487600 >> v2:172.21.15.144:3300/0 conn(0x7f1060159750 0x7f10600d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f106539c700 1 -- 172.21.15.29:0/1879487600 >> v1:172.21.15.29:6789/0 conn(0x7f1060159b20 legacy=0x7f10600ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f106539c700 1 -- 172.21.15.29:0/1879487600 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600cc7c0 con 0x7f10600ce8a0 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.822+0000 7f106539c700 1 --2- 172.21.15.29:0/1879487600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10600ce8a0 0x7f10600c5460 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1060089840 tx=0x7f1058009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58940f895ec13bb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f105f7fe700 1 -- 172.21.15.29:0/1879487600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800e070 con 0x7f10600ce8a0 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f105f7fe700 1 -- 172.21.15.29:0/1879487600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800bb60 con 0x7f10600ce8a0 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f105f7fe700 1 -- 172.21.15.29:0/1879487600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058012430 con 0x7f10600ce8a0 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 -- 172.21.15.29:0/1879487600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10600ce8a0 msgr2=0x7f10600c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 --2- 172.21.15.29:0/1879487600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10600ce8a0 0x7f10600c5460 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1060089840 tx=0x7f1058009780 comp rx=0 tx=0).stop 2023-10-07T09:44:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 -- 172.21.15.29:0/1879487600 shutdown_connections 2023-10-07T09:44:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 --2- 172.21.15.29:0/1879487600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10600ce8a0 0x7f10600c5460 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 --2- 172.21.15.29:0/1879487600 >> v2:172.21.15.144:3300/0 conn(0x7f1060159750 0x7f10600d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 -- 172.21.15.29:0/1879487600 >> 172.21.15.29:0/1879487600 conn(0x7f1060063a10 msgr2=0x7f10600d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.823+0000 7f1066dff700 1 -- 172.21.15.29:0/1879487600 shutdown_connections 2023-10-07T09:44:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.824+0000 7f1066dff700 1 -- 172.21.15.29:0/1879487600 wait complete. 2023-10-07T09:44:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.825+0000 7f1066dff700 1 Processor -- start 2023-10-07T09:44:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.825+0000 7f1066dff700 1 -- start start 2023-10-07T09:44:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1066dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10600ce8a0 0x7f106014e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1066dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1060159750 0x7f10600b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1066dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 0x7f1060152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:04.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1066dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106014e560 con 0x7f1060159b20 2023-10-07T09:44:04.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1066dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600b5470 con 0x7f1060159750 2023-10-07T09:44:04.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1066dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10600b5750 con 0x7f10600ce8a0 2023-10-07T09:44:04.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f106539c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 0x7f1060152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:04.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f106539c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 0x7f1060152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47146/0 (socket says 172.21.15.29:47146) 2023-10-07T09:44:04.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f106539c700 1 -- 172.21.15.29:0/3311395154 learned_addr learned my addr 172.21.15.29:0/3311395154 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f1064b9b700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10600ce8a0 0x7f106014e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.826+0000 7f105ffff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1060159750 0x7f10600b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f106539c700 1 -- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10600ce8a0 msgr2=0x7f106014e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f106539c700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10600ce8a0 0x7f106014e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f106539c700 1 -- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1060159750 msgr2=0x7f10600b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f106539c700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1060159750 0x7f10600b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f106539c700 1 -- 172.21.15.29:0/3311395154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058009a20 con 0x7f1060159b20 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f106539c700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 0x7f1060152fe0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1058009fd0 tx=0x7f1058019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058027470 con 0x7f1060159b20 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10580275f0 con 0x7f1060159b20 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10600b5a30 con 0x7f1060159b20 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10580279b0 con 0x7f1060159b20 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.827+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10600b5160 con 0x7f1060159b20 2023-10-07T09:44:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.828+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1060159fa0 con 0x7f1060159b20 2023-10-07T09:44:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.829+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f1058024070 con 0x7f1060159b20 2023-10-07T09:44:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.830+0000 7f105cff9700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 0x7f1044069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.830+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6369+0+0 (secure 0 0 0) 0x7f1058087320 con 0x7f1060159b20 2023-10-07T09:44:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.830+0000 7f1064b9b700 1 -- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 msgr2=0x7f1044069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.830+0000 7f1064b9b700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 0x7f1044069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:04.838+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1060159fa0 con 0x7f1060159b20 2023-10-07T09:44:04.919 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:04.912+0000 7fa55e89d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:44:04.962 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:04.961+0000 7f9c3d3881c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:44:05.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.031+0000 7f1064b9b700 1 -- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 msgr2=0x7f1044069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:05.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.031+0000 7f1064b9b700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 0x7f1044069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:05.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.089+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1060159fa0 con 0x7f1060159b20 2023-10-07T09:44:05.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.090+0000 7f105cff9700 1 -- 172.21.15.29:0/3311395154 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f1060159fa0 con 0x7f1060159b20 2023-10-07T09:44:05.091 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:05.092 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:42:11.722170+0000","modified":"2023-10-07T09:42:15.247175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6834/2769082805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2769082805},{"type":"v1","addr":"172.21.15.29:6834","nonce":2769082805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-10-07T09:44:05.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 msgr2=0x7f1044069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:05.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 0x7f1044069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 msgr2=0x7f1060152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 0x7f1060152fe0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1058009fd0 tx=0x7f1058019000 comp rx=0 tx=0).stop 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 shutdown_connections 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1044067050 0x7f1044069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1060159b20 0x7f1060152fe0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10600ce8a0 0x7f106014e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 --2- 172.21.15.29:0/3311395154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1060159750 0x7f10600b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 >> 172.21.15.29:0/3311395154 conn(0x7f1060063a10 msgr2=0x7f1060069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 shutdown_connections 2023-10-07T09:44:05.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.092+0000 7f1066dff700 1 -- 172.21.15.29:0/3311395154 wait complete. 2023-10-07T09:44:05.096 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-07T09:44:05.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.320+0000 7effbcdff700 1 Processor -- start 2023-10-07T09:44:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.320+0000 7effbcdff700 1 -- start start 2023-10-07T09:44:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effbcdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effbcdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7effb80ae8b0 0x7effb80b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effbcdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb8158ad0 con 0x7effb80b1840 2023-10-07T09:44:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effbcdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb80b1cc0 con 0x7effb80ae8b0 2023-10-07T09:44:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effbcdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb815b140 con 0x7effb8157290 2023-10-07T09:44:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effb659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effb5d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7effb80ae8b0 0x7effb80b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effb659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb80cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:41128/0 (socket says 172.21.15.29:41128) 2023-10-07T09:44:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.322+0000 7effb659c700 1 -- 172.21.15.29:0/4125821362 learned_addr learned my addr 172.21.15.29:0/4125821362 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb659c700 1 -- 172.21.15.29:0/4125821362 >> v2:172.21.15.144:3300/0 conn(0x7effb80ae8b0 msgr2=0x7effb80b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb659c700 1 --2- 172.21.15.29:0/4125821362 >> v2:172.21.15.144:3300/0 conn(0x7effb80ae8b0 0x7effb80b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb659c700 1 -- 172.21.15.29:0/4125821362 >> v1:172.21.15.144:6789/0 conn(0x7effb80b1840 legacy=0x7effb80ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb659c700 1 -- 172.21.15.29:0/4125821362 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb815b420 con 0x7effb8157290 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb5d9b700 1 --2- 172.21.15.29:0/4125821362 >> v2:172.21.15.144:3300/0 conn(0x7effb80ae8b0 0x7effb80b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb659c700 1 --2- 172.21.15.29:0/4125821362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb80cc870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7effa80099b0 tx=0x7effa8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a319668d257402 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb559a700 1 -- 172.21.15.29:0/4125821362 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8009ea0 con 0x7effb8157290 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb559a700 1 -- 172.21.15.29:0/4125821362 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa800b860 con 0x7effb8157290 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.323+0000 7effb559a700 1 -- 172.21.15.29:0/4125821362 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa800bc00 con 0x7effb8157290 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 -- 172.21.15.29:0/4125821362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 msgr2=0x7effb80cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 --2- 172.21.15.29:0/4125821362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb80cc870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7effa80099b0 tx=0x7effa8009250 comp rx=0 tx=0).stop 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 -- 172.21.15.29:0/4125821362 shutdown_connections 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 --2- 172.21.15.29:0/4125821362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb80cc870 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 --2- 172.21.15.29:0/4125821362 >> v2:172.21.15.144:3300/0 conn(0x7effb80ae8b0 0x7effb80b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 -- 172.21.15.29:0/4125821362 >> 172.21.15.29:0/4125821362 conn(0x7effb80680a0 msgr2=0x7effb80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 -- 172.21.15.29:0/4125821362 shutdown_connections 2023-10-07T09:44:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.324+0000 7effbcdff700 1 -- 172.21.15.29:0/4125821362 wait complete. 2023-10-07T09:44:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.326+0000 7effbcdff700 1 Processor -- start 2023-10-07T09:44:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.326+0000 7effbcdff700 1 -- start start 2023-10-07T09:44:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effbcdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80b1840 0x7effb806e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effbcdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effbcdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effb815a4f0 0x7effb814fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effbcdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb814ba80 con 0x7effb815a4f0 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effbcdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb806eb70 con 0x7effb80b1840 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effbcdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb806ee50 con 0x7effb8157290 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effb6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effb815a4f0 0x7effb814fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effb5d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effb659c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80b1840 0x7effb806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effb5d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:41134/0 (socket says 172.21.15.29:41134) 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.327+0000 7effb5d9b700 1 -- 172.21.15.29:0/518622199 learned_addr learned my addr 172.21.15.29:0/518622199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effb5d9b700 1 -- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80b1840 msgr2=0x7effb806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effb5d9b700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80b1840 0x7effb806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effb5d9b700 1 -- 172.21.15.29:0/518622199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effb815a4f0 msgr2=0x7effb814fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effb5d9b700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effb815a4f0 0x7effb814fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effb5d9b700 1 -- 172.21.15.29:0/518622199 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac009420 con 0x7effb8157290 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effb5d9b700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb8159fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7effac00ec80 tx=0x7effac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00c920 con 0x7effb8157290 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac00ca80 con 0x7effb8157290 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00ce00 con 0x7effb8157290 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa80094f0 con 0x7effb8157290 2023-10-07T09:44:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.328+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb806e740 con 0x7effb8157290 2023-10-07T09:44:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.330+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb814b870 con 0x7effb8157290 2023-10-07T09:44:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.330+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7effac013680 con 0x7effb8157290 2023-10-07T09:44:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.331+0000 7effa67fc700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 0x7eff98069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.332+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7effac084d10 con 0x7effb8157290 2023-10-07T09:44:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.332+0000 7effb659c700 1 -- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 msgr2=0x7eff98069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.332+0000 7effb659c700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 0x7eff98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.336+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effb814b870 con 0x7effb8157290 2023-10-07T09:44:05.442 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:05.441+0000 7fa55e89d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:44:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.533+0000 7effb659c700 1 -- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 msgr2=0x7eff98069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.533+0000 7effb659c700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 0x7eff98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:05.546 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:05.545+0000 7f9c3d3881c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:44:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.569+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effb814b870 con 0x7effb8157290 2023-10-07T09:44:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.570+0000 7effa67fc700 1 -- 172.21.15.29:0/518622199 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16553 (secure 0 0 0) 0x7effb814b870 con 0x7effb8157290 2023-10-07T09:44:05.572 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:05.573 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":65,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:05.195304+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:42:09.508279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-07T09:42:10.701497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":4}]},{"pgid":"7.2d","mappings":[{"from":0,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1583565526":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/3005988804":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/2771895322":"2023-10-08T09:42:27.213405+0000","172.21.15.144:6836/114714":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/2633526602":"2023-10-08T09:42:27.213405+0000","172.21.15.144:6834/114714":"2023-10-08T09:42:27.213405+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 msgr2=0x7eff98069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 0x7eff98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 msgr2=0x7effb8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb8159fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7effac00ec80 tx=0x7effac0091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 shutdown_connections 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7eff98067050 0x7eff98069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effb815a4f0 0x7effb814fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80b1840 0x7effb806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 --2- 172.21.15.29:0/518622199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb8157290 0x7effb8159fb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 >> 172.21.15.29:0/518622199 conn(0x7effb80680a0 msgr2=0x7effb80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 shutdown_connections 2023-10-07T09:44:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.573+0000 7effbcdff700 1 -- 172.21.15.29:0/518622199 wait complete. 2023-10-07T09:44:05.593 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:42:09.508279+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:42:10.701497+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:44:05.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:44:05.613 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:05.612+0000 7fa55e89d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:44:05.735 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:05.734+0000 7f9c3d3881c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:44:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.805+0000 7f132abff700 1 Processor -- start 2023-10-07T09:44:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.805+0000 7f132abff700 1 -- start start 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.806+0000 7f132abff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f132415da10 0x7f1324069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.806+0000 7f132abff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1324069400 con 0x7f1324159300 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.806+0000 7f132abff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1324069dd0 con 0x7f13240b0770 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.806+0000 7f132abff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132415de60 con 0x7f132415da10 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.806+0000 7f132919c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f132415da10 0x7f1324069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132919c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f132415da10 0x7f1324069890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47166/0 (socket says 172.21.15.29:47166) 2023-10-07T09:44:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132919c700 1 -- 172.21.15.29:0/3138413462 learned_addr learned my addr 172.21.15.29:0/3138413462 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132899b700 1 --1- 172.21.15.29:0/3138413462 >> v1:172.21.15.29:6789/0 conn(0x7f1324159300 0x7f13240b0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46540/0 (socket says 172.21.15.29:46540) 2023-10-07T09:44:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132919c700 1 -- 172.21.15.29:0/3138413462 >> v1:172.21.15.144:6789/0 conn(0x7f13240b0770 legacy=0x7f132415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132919c700 1 -- 172.21.15.29:0/3138413462 >> v1:172.21.15.29:6789/0 conn(0x7f1324159300 legacy=0x7f13240b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132919c700 1 -- 172.21.15.29:0/3138413462 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1324068b10 con 0x7f132415da10 2023-10-07T09:44:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f13237fe700 1 -- 172.21.15.29:0/3138413462 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248995058 0 0) 0x7f1324069400 con 0x7f1324159300 2023-10-07T09:44:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.807+0000 7f132919c700 1 --2- 172.21.15.29:0/3138413462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f1324069890 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f131c009f20 tx=0x7f131c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23900339f948448b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.808+0000 7f13237fe700 1 -- 172.21.15.29:0/3138413462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c00e070 con 0x7f132415da10 2023-10-07T09:44:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.808+0000 7f13237fe700 1 -- 172.21.15.29:0/3138413462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c00ba80 con 0x7f132415da10 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.808+0000 7f13237fe700 1 -- 172.21.15.29:0/3138413462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c0092e0 con 0x7f132415da10 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.808+0000 7f132abff700 1 -- 172.21.15.29:0/3138413462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 msgr2=0x7f1324069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.808+0000 7f132abff700 1 --2- 172.21.15.29:0/3138413462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f1324069890 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f131c009f20 tx=0x7f131c009840 comp rx=0 tx=0).stop 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.809+0000 7f132abff700 1 -- 172.21.15.29:0/3138413462 shutdown_connections 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.809+0000 7f132abff700 1 --2- 172.21.15.29:0/3138413462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f1324069890 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.809+0000 7f132abff700 1 -- 172.21.15.29:0/3138413462 >> 172.21.15.29:0/3138413462 conn(0x7f1324067610 msgr2=0x7f1324067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.809+0000 7f132abff700 1 -- 172.21.15.29:0/3138413462 shutdown_connections 2023-10-07T09:44:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.809+0000 7f132abff700 1 -- 172.21.15.29:0/3138413462 wait complete. 2023-10-07T09:44:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.811+0000 7f132abff700 1 Processor -- start 2023-10-07T09:44:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.811+0000 7f132abff700 1 -- start start 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13240b0770 0x7f13240ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1324159300 0x7f1324155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f13240cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240c76e0 con 0x7f132415da10 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240ca790 con 0x7f1324159300 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1324155680 con 0x7f13240b0770 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f13240cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f13240cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47180/0 (socket says 172.21.15.29:47180) 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132919c700 1 -- 172.21.15.29:0/147050682 learned_addr learned my addr 172.21.15.29:0/147050682 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f132899b700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13240b0770 0x7f13240ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.813+0000 7f1323fff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1324159300 0x7f1324155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132919c700 1 -- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13240b0770 msgr2=0x7f13240ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132919c700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13240b0770 0x7f13240ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132919c700 1 -- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1324159300 msgr2=0x7f1324155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132919c700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1324159300 0x7f1324155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132919c700 1 -- 172.21.15.29:0/147050682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c009ae0 con 0x7f132415da10 2023-10-07T09:44:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132919c700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f13240cc4a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f132406b300 tx=0x7f131c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c007d10 con 0x7f132415da10 2023-10-07T09:44:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c007e90 con 0x7f132415da10 2023-10-07T09:44:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13240caa30 con 0x7f132415da10 2023-10-07T09:44:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.814+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c0256b0 con 0x7f132415da10 2023-10-07T09:44:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.815+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13240caff0 con 0x7f132415da10 2023-10-07T09:44:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.816+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132404c3c0 con 0x7f132415da10 2023-10-07T09:44:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.817+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f131c01c410 con 0x7f132415da10 2023-10-07T09:44:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.817+0000 7f1320ff9700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 0x7f1308069580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.817+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f131c0864f0 con 0x7f132415da10 2023-10-07T09:44:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.818+0000 7f132899b700 1 -- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 msgr2=0x7f1308069580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.818+0000 7f132899b700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 0x7f1308069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:05.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:05.822+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f131c0531d0 con 0x7f132415da10 2023-10-07T09:44:06.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.019+0000 7f132899b700 1 -- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 msgr2=0x7f1308069580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:06.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.019+0000 7f132899b700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 0x7f1308069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:06.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.025+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1324068de0 con 0x7f132415da10 2023-10-07T09:44:06.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.026+0000 7f1320ff9700 1 -- 172.21.15.29:0/147050682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f131c0567f0 con 0x7f132415da10 2023-10-07T09:44:06.027 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 msgr2=0x7f1308069580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 0x7f1308069580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 msgr2=0x7f13240cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f13240cc4a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f132406b300 tx=0x7f131c018000 comp rx=0 tx=0).stop 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 shutdown_connections 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f13080670d0 0x7f1308069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132415da10 0x7f13240cc4a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13240b0770 0x7f13240ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 --2- 172.21.15.29:0/147050682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1324159300 0x7f1324155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 >> 172.21.15.29:0/147050682 conn(0x7f1324067610 msgr2=0x7f13240679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:06.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.028+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 shutdown_connections 2023-10-07T09:44:06.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.029+0000 7f132abff700 1 -- 172.21.15.29:0/147050682 wait complete. 2023-10-07T09:44:06.031 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:06.030+0000 7fa55e89d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:44:06.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:44:06.173 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:06.172+0000 7f9c3d3881c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:44:06.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.249+0000 7f50539ff700 1 Processor -- start 2023-10-07T09:44:06.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.249+0000 7f50539ff700 1 -- start start 2023-10-07T09:44:06.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f50539ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f504c0b0c30 0x7f504c0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f50539ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c0ad4d0 con 0x7f504c0ae570 2023-10-07T09:44:06.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f50539ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c0ad7b0 con 0x7f504c156180 2023-10-07T09:44:06.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f50539ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c0ada90 con 0x7f504c0b0c30 2023-10-07T09:44:06.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f5051f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f504c0b0c30 0x7f504c0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f5051f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f504c0b0c30 0x7f504c0afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47190/0 (socket says 172.21.15.29:47190) 2023-10-07T09:44:06.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f5050f9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f504c0ae570 0x7f504c0b0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46544/0 (socket says 172.21.15.29:46544) 2023-10-07T09:44:06.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.251+0000 7f5051f9c700 1 -- 172.21.15.29:0/3725767882 learned_addr learned my addr 172.21.15.29:0/3725767882 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:06.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5051f9c700 1 -- 172.21.15.29:0/3725767882 >> v1:172.21.15.144:6789/0 conn(0x7f504c156180 legacy=0x7f504c0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5051f9c700 1 -- 172.21.15.29:0/3725767882 >> v1:172.21.15.29:6789/0 conn(0x7f504c0ae570 legacy=0x7f504c0b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5043fff700 1 -- 172.21.15.29:0/3725767882 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556498138 0 0) 0x7f504c0ad7b0 con 0x7f504c156180 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5051f9c700 1 -- 172.21.15.29:0/3725767882 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c0add70 con 0x7f504c0b0c30 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5051f9c700 1 --2- 172.21.15.29:0/3725767882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0afc00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f504400d1a0 tx=0x7f504400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3895d2f3067c887 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5043fff700 1 -- 172.21.15.29:0/3725767882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044010070 con 0x7f504c0b0c30 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5043fff700 1 -- 172.21.15.29:0/3725767882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504400b0d0 con 0x7f504c0b0c30 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.252+0000 7f5043fff700 1 -- 172.21.15.29:0/3725767882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504400b490 con 0x7f504c0b0c30 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 -- 172.21.15.29:0/3725767882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 msgr2=0x7f504c0afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 --2- 172.21.15.29:0/3725767882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0afc00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f504400d1a0 tx=0x7f504400ba50 comp rx=0 tx=0).stop 2023-10-07T09:44:06.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 -- 172.21.15.29:0/3725767882 shutdown_connections 2023-10-07T09:44:06.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 --2- 172.21.15.29:0/3725767882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0afc00 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 -- 172.21.15.29:0/3725767882 >> 172.21.15.29:0/3725767882 conn(0x7f504c067610 msgr2=0x7f504c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:06.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 -- 172.21.15.29:0/3725767882 shutdown_connections 2023-10-07T09:44:06.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.253+0000 7f50539ff700 1 -- 172.21.15.29:0/3725767882 wait complete. 2023-10-07T09:44:06.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.255+0000 7f50539ff700 1 Processor -- start 2023-10-07T09:44:06.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.256+0000 7f50539ff700 1 -- start start 2023-10-07T09:44:06.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f50539ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f504c0ae570 0x7f504c143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f50539ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f50539ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504c156180 0x7f504c1586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f50539ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c1453e0 con 0x7f504c0b0c30 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f50539ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c1456c0 con 0x7f504c0ae570 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f50539ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c1459a0 con 0x7f504c156180 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f5050f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f505179b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f504c0ae570 0x7f504c143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f5050f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47204/0 (socket says 172.21.15.29:47204) 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f5050f9a700 1 -- 172.21.15.29:0/1745542117 learned_addr learned my addr 172.21.15.29:0/1745542117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.257+0000 7f5051f9c700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504c156180 0x7f504c1586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f505179b700 1 -- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504c156180 msgr2=0x7f504c1586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f505179b700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504c156180 0x7f504c1586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f505179b700 1 -- 172.21.15.29:0/1745542117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 msgr2=0x7f504c0c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f505179b700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f505179b700 1 -- 172.21.15.29:0/1745542117 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504400bcf0 con 0x7f504c0ae570 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f5051f9c700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504c156180 0x7f504c1586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f5050f9a700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f505179b700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f504c0ae570 0x7f504c143ad0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f504800c4c0 tx=0x7f5048007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:06.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504800ce30 con 0x7f504c0ae570 2023-10-07T09:44:06.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048011590 con 0x7f504c0ae570 2023-10-07T09:44:06.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048011950 con 0x7f504c0ae570 2023-10-07T09:44:06.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.258+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c145c80 con 0x7f504c0ae570 2023-10-07T09:44:06.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.259+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504c144010 con 0x7f504c0ae570 2023-10-07T09:44:06.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.260+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504c04c3c0 con 0x7f504c0ae570 2023-10-07T09:44:06.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.261+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f5048011d40 con 0x7f504c0ae570 2023-10-07T09:44:06.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.261+0000 7f50417fa700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 0x7f5030069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.261+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f5048083cf0 con 0x7f504c0ae570 2023-10-07T09:44:06.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.262+0000 7f5050f9a700 1 -- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 msgr2=0x7f5030069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:06.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.262+0000 7f5050f9a700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 0x7f5030069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:06.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.266+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5048050ae0 con 0x7f504c0ae570 2023-10-07T09:44:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.463+0000 7f5050f9a700 1 -- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 msgr2=0x7f5030069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.463+0000 7f5050f9a700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 0x7f5030069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.471+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f504c0ae050 con 0x7f504c0ae570 2023-10-07T09:44:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.471+0000 7f50417fa700 1 -- 172.21.15.29:0/1745542117 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v65) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5048017020 con 0x7f504c0ae570 2023-10-07T09:44:06.472 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:44:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 msgr2=0x7f5030069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:06.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 0x7f5030069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f504c0ae570 msgr2=0x7f504c143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f504c0ae570 0x7f504c143ad0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f504800c4c0 tx=0x7f5048007ec0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 shutdown_connections 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5030067050 0x7f5030069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504c156180 0x7f504c1586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f504c0ae570 0x7f504c143ad0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 --2- 172.21.15.29:0/1745542117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f504c0b0c30 0x7f504c0c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 >> 172.21.15.29:0/1745542117 conn(0x7f504c067610 msgr2=0x7f504c152b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 shutdown_connections 2023-10-07T09:44:06.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.475+0000 7f50539ff700 1 -- 172.21.15.29:0/1745542117 wait complete. 2023-10-07T09:44:06.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:44:06.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.700+0000 7f47809ff700 1 Processor -- start 2023-10-07T09:44:06.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.700+0000 7f47809ff700 1 -- start start 2023-10-07T09:44:06.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f47809ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c1596e0 0x7f477c0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f47809ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f477c0cc630 0x7f477c0acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f47809ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f477c0ad440 0x7f477c15d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f47809ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477c0aa8e0 con 0x7f477c0ad440 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f47809ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477c0ccaa0 con 0x7f477c0cc630 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f47809ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c159ab0 con 0x7f477c1596e0 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f477ad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f477c0ad440 0x7f477c15d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f477ad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f477c0ad440 0x7f477c15d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47206/0 (socket says 172.21.15.29:47206) 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f477ad9d700 1 -- 172.21.15.29:0/2194693044 learned_addr learned my addr 172.21.15.29:0/2194693044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f4779d9b700 1 --2- 172.21.15.29:0/2194693044 >> v2:172.21.15.144:3300/0 conn(0x7f477c0cc630 0x7f477c0acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.702+0000 7f477a59c700 1 --2- 172.21.15.29:0/2194693044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c1596e0 0x7f477c0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477ad9d700 1 -- 172.21.15.29:0/2194693044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c1596e0 msgr2=0x7f477c0cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477ad9d700 1 --2- 172.21.15.29:0/2194693044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c1596e0 0x7f477c0cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477ad9d700 1 -- 172.21.15.29:0/2194693044 >> v2:172.21.15.144:3300/0 conn(0x7f477c0cc630 msgr2=0x7f477c0acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477ad9d700 1 --2- 172.21.15.29:0/2194693044 >> v2:172.21.15.144:3300/0 conn(0x7f477c0cc630 0x7f477c0acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477ad9d700 1 -- 172.21.15.29:0/2194693044 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c0a98d0 con 0x7f477c0ad440 2023-10-07T09:44:06.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f4779d9b700 1 --2- 172.21.15.29:0/2194693044 >> v2:172.21.15.144:3300/0 conn(0x7f477c0cc630 0x7f477c0acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:06.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477ad9d700 1 --2- 172.21.15.29:0/2194693044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c0ad440 0x7f477c15d1e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f477c06d130 tx=0x7f4774009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de302f8ee24760bc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477959a700 1 -- 172.21.15.29:0/2194693044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477400e070 con 0x7f477c0ad440 2023-10-07T09:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.703+0000 7f477959a700 1 -- 172.21.15.29:0/2194693044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477400b970 con 0x7f477c0ad440 2023-10-07T09:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f477959a700 1 -- 172.21.15.29:0/2194693044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740092e0 con 0x7f477c0ad440 2023-10-07T09:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 -- 172.21.15.29:0/2194693044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c0ad440 msgr2=0x7f477c15d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 --2- 172.21.15.29:0/2194693044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c0ad440 0x7f477c15d1e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f477c06d130 tx=0x7f4774009840 comp rx=0 tx=0).stop 2023-10-07T09:44:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 -- 172.21.15.29:0/2194693044 shutdown_connections 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 --2- 172.21.15.29:0/2194693044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c0ad440 0x7f477c15d1e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 --2- 172.21.15.29:0/2194693044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c1596e0 0x7f477c0cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 --2- 172.21.15.29:0/2194693044 >> v2:172.21.15.144:3300/0 conn(0x7f477c0cc630 0x7f477c0acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 -- 172.21.15.29:0/2194693044 >> 172.21.15.29:0/2194693044 conn(0x7f477c067610 msgr2=0x7f477c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 -- 172.21.15.29:0/2194693044 shutdown_connections 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.704+0000 7f47809ff700 1 -- 172.21.15.29:0/2194693044 wait complete. 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.706+0000 7f47809ff700 1 Processor -- start 2023-10-07T09:44:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.706+0000 7f47809ff700 1 -- start start 2023-10-07T09:44:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f47809ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f477c0ad440 0x7f477c16a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f47809ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c0cc630 0x7f477c169070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f47809ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 0x7f477c0abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f47809ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c16db30 con 0x7f477c1596e0 2023-10-07T09:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f47809ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c16a7a0 con 0x7f477c0ad440 2023-10-07T09:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f47809ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1551a0 con 0x7f477c0cc630 2023-10-07T09:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f477ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 0x7f477c0abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f477ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 0x7f477c0abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47210/0 (socket says 172.21.15.29:47210) 2023-10-07T09:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f477a59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f477c0ad440 0x7f477c16a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f477ad9d700 1 -- 172.21.15.29:0/2832873872 learned_addr learned my addr 172.21.15.29:0/2832873872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.707+0000 7f4779d9b700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c0cc630 0x7f477c169070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f477ad9d700 1 -- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c0cc630 msgr2=0x7f477c169070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f477ad9d700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c0cc630 0x7f477c169070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f477ad9d700 1 -- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f477c0ad440 msgr2=0x7f477c16a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f477ad9d700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f477c0ad440 0x7f477c16a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f477ad9d700 1 -- 172.21.15.29:0/2832873872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770009420 con 0x7f477c1596e0 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f477ad9d700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 0x7f477c0abde0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f477c0641e0 tx=0x7f4774018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774008ad0 con 0x7f477c1596e0 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774008c50 con 0x7f477c1596e0 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774025460 con 0x7f477c1596e0 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.708+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4774009ae0 con 0x7f477c1596e0 2023-10-07T09:44:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.709+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c16d190 con 0x7f477c1596e0 2023-10-07T09:44:06.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.710+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477c16dc70 con 0x7f477c1596e0 2023-10-07T09:44:06.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.711+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f4774003040 con 0x7f477c1596e0 2023-10-07T09:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.712+0000 7f476a7fc700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 0x7f475c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.712+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f4774085f20 con 0x7f477c1596e0 2023-10-07T09:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.712+0000 7f477a59c700 1 -- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 msgr2=0x7f475c069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.712+0000 7f477a59c700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 0x7f475c069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:06.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.716+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f477c16dc70 con 0x7f477c1596e0 2023-10-07T09:44:06.806 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:06.805+0000 7fa55e89d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:44:06.893 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:06.892+0000 7fa55e89d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:44:06.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.913+0000 7f477a59c700 1 -- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 msgr2=0x7f475c069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:06.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.913+0000 7f477a59c700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 0x7f475c069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.919+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f477c16dc70 con 0x7f477c1596e0 2023-10-07T09:44:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.919+0000 7f476a7fc700 1 -- 172.21.15.29:0/2832873872 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v65) v1 ==== 100+0+11 (secure 0 0 0) 0x7f477c16dc70 con 0x7f477c1596e0 2023-10-07T09:44:06.920 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:44:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 msgr2=0x7f475c069530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 0x7f475c069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 msgr2=0x7f477c0abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 0x7f477c0abde0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f477c0641e0 tx=0x7f4774018000 comp rx=0 tx=0).stop 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 shutdown_connections 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f475c067080 0x7f475c069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f477c1596e0 0x7f477c0abde0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f477c0ad440 0x7f477c16a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 --2- 172.21.15.29:0/2832873872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f477c0cc630 0x7f477c169070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 >> 172.21.15.29:0/2832873872 conn(0x7f477c067610 msgr2=0x7f477c0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 shutdown_connections 2023-10-07T09:44:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:06.922+0000 7f47809ff700 1 -- 172.21.15.29:0/2832873872 wait complete. 2023-10-07T09:44:06.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:07.003 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.002+0000 7f9c3d3881c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:44:07.046 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.045+0000 7fa55e89d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:44:07.096 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.095+0000 7f9c3d3881c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:44:07.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.131+0000 7fa55e89d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:44:07.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.136+0000 7fd6df5ff700 1 Processor -- start 2023-10-07T09:44:07.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.136+0000 7fd6df5ff700 1 -- start start 2023-10-07T09:44:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6df5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd6d80b2670 0x7fd6d80b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6df5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6d80a98d0 0x7fd6d80a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6df5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d806f4e0 con 0x7fd6d8159550 2023-10-07T09:44:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6df5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d80b1120 con 0x7fd6d80a98d0 2023-10-07T09:44:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6df5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d806e8d0 con 0x7fd6d80b2670 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6ddb9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6d80a98d0 0x7fd6d80a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6dcb9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd6d80b2670 0x7fd6d80b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6dd39b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd6d8159550 0x7fd6d80b0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:50212/0 (socket says 172.21.15.29:50212) 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.139+0000 7fd6dd39b700 1 -- 172.21.15.29:0/445717128 learned_addr learned my addr 172.21.15.29:0/445717128 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6dcb9a700 1 -- 172.21.15.29:0/445717128 >> v2:172.21.15.29:3300/0 conn(0x7fd6d80a98d0 msgr2=0x7fd6d80a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6dcb9a700 1 --2- 172.21.15.29:0/445717128 >> v2:172.21.15.29:3300/0 conn(0x7fd6d80a98d0 0x7fd6d80a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6dcb9a700 1 -- 172.21.15.29:0/445717128 >> v1:172.21.15.144:6789/0 conn(0x7fd6d8159550 legacy=0x7fd6d80b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6dcb9a700 1 -- 172.21.15.29:0/445717128 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d806ebb0 con 0x7fd6d80b2670 2023-10-07T09:44:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6cffff700 1 -- 172.21.15.29:0/445717128 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212290652 0 0) 0x7fd6d806f4e0 con 0x7fd6d8159550 2023-10-07T09:44:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6dcb9a700 1 --2- 172.21.15.29:0/445717128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80b2ae0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd6d80ce380 tx=0x7fd6c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cd067a1f94740a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6cffff700 1 -- 172.21.15.29:0/445717128 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8011630 con 0x7fd6d80b2670 2023-10-07T09:44:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6cffff700 1 -- 172.21.15.29:0/445717128 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c8011790 con 0x7fd6d80b2670 2023-10-07T09:44:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.140+0000 7fd6cffff700 1 -- 172.21.15.29:0/445717128 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8011b30 con 0x7fd6d80b2670 2023-10-07T09:44:07.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 -- 172.21.15.29:0/445717128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 msgr2=0x7fd6d80b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/445717128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80b2ae0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd6d80ce380 tx=0x7fd6c80091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 -- 172.21.15.29:0/445717128 shutdown_connections 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/445717128 >> v2:172.21.15.29:3300/0 conn(0x7fd6d80a98d0 0x7fd6d80a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/445717128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80b2ae0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 -- 172.21.15.29:0/445717128 >> 172.21.15.29:0/445717128 conn(0x7fd6d8069190 msgr2=0x7fd6d806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 -- 172.21.15.29:0/445717128 shutdown_connections 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.141+0000 7fd6df5ff700 1 -- 172.21.15.29:0/445717128 wait complete. 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.143+0000 7fd6df5ff700 1 Processor -- start 2023-10-07T09:44:07.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.143+0000 7fd6df5ff700 1 -- start start 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.144+0000 7fd6df5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 0x7fd6d80aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.144+0000 7fd6df5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.144+0000 7fd6df5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6d8159550 0x7fd6d8158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.144+0000 7fd6df5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8154ce0 con 0x7fd6d80a98d0 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.144+0000 7fd6df5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8154fc0 con 0x7fd6d80b2670 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.144+0000 7fd6df5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81552a0 con 0x7fd6d8159550 2023-10-07T09:44:07.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 0x7fd6d80aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 0x7fd6d80aac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47228/0 (socket says 172.21.15.29:47228) 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 -- 172.21.15.29:0/3337687041 learned_addr learned my addr 172.21.15.29:0/3337687041 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dcb9a700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6ddb9c700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6d8159550 0x7fd6d8158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 -- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6d8159550 msgr2=0x7fd6d8158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6d8159550 0x7fd6d8158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 -- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 msgr2=0x7fd6d80ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 -- 172.21.15.29:0/3337687041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8009420 con 0x7fd6d80a98d0 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.145+0000 7fd6dd39b700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 0x7fd6d80aac70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c7e0 tx=0x7fd6d4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.146+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4005b30 con 0x7fd6d80a98d0 2023-10-07T09:44:07.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.146+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4005c90 con 0x7fd6d80a98d0 2023-10-07T09:44:07.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.146+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d8155580 con 0x7fd6d80a98d0 2023-10-07T09:44:07.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.146+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4007300 con 0x7fd6d80a98d0 2023-10-07T09:44:07.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.146+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d8140d40 con 0x7fd6d80a98d0 2023-10-07T09:44:07.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.147+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d804c3c0 con 0x7fd6d80a98d0 2023-10-07T09:44:07.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.148+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fd6d401b3b0 con 0x7fd6d80a98d0 2023-10-07T09:44:07.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.149+0000 7fd6cd7fa700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 0x7fd6bc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.149+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fd6d4083f40 con 0x7fd6d80a98d0 2023-10-07T09:44:07.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.149+0000 7fd6dcb9a700 1 -- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 msgr2=0x7fd6bc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:07.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.149+0000 7fd6dcb9a700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 0x7fd6bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:07.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.153+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6d4050d30 con 0x7fd6d80a98d0 2023-10-07T09:44:07.298 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.297+0000 7f9c3d3881c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:44:07.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.350+0000 7fd6dcb9a700 1 -- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 msgr2=0x7fd6bc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:07.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.350+0000 7fd6dcb9a700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 0x7fd6bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:07.352 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.351+0000 7fa55e89d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:44:07.388 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.387+0000 7f9c3d3881c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:44:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.446+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6d8154e20 con 0x7fd6d80a98d0 2023-10-07T09:44:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.446+0000 7fd6cd7fa700 1 -- 172.21.15.29:0/3337687041 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4600 (secure 0 0 0) 0x7fd6d8154e20 con 0x7fd6d80a98d0 2023-10-07T09:44:07.447 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:07.448 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:42:11.722170+0000","modified":"2023-10-07T09:42:15.247175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6834/2769082805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2769082805},{"type":"v1","addr":"172.21.15.29:6834","nonce":2769082805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-10-07T09:44:07.448 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.447+0000 7fa55e89d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:44:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 msgr2=0x7fd6bc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 0x7fd6bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 msgr2=0x7fd6d80aac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 0x7fd6d80aac70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c7e0 tx=0x7fd6d4007df0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 shutdown_connections 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6bc067050 0x7fd6bc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6d8159550 0x7fd6d8158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6d80a98d0 0x7fd6d80aac70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 --2- 172.21.15.29:0/3337687041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d80b2670 0x7fd6d80ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 >> 172.21.15.29:0/3337687041 conn(0x7fd6d8069190 msgr2=0x7fd6d815f800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 shutdown_connections 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.449+0000 7fd6df5ff700 1 -- 172.21.15.29:0/3337687041 wait complete. 2023-10-07T09:44:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-07T09:44:07.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:07.543 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.542+0000 7fa55e89d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:44:07.611 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.610+0000 7f9c3d3881c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:44:07.637 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.636+0000 7fa55e89d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:44:07.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.669+0000 7fcfb21ff700 1 Processor -- start 2023-10-07T09:44:07.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.669+0000 7fcfb21ff700 1 -- start start 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.671+0000 7fcfb21ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfac0ad220 0x7fcfac159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.671+0000 7fcfb21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac068620 0x7fcfac068a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.671+0000 7fcfb21ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac15af00 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.671+0000 7fcfb21ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac068fa0 con 0x7fcfac0ad220 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.671+0000 7fcfb21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0ada40 con 0x7fcfac068620 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfac0ad220 0x7fcfac159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfac0ad220 0x7fcfac159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47230/0 (socket says 172.21.15.29:47230) 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 -- 172.21.15.29:0/536353518 learned_addr learned my addr 172.21.15.29:0/536353518 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:07.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfabfff700 1 --2- 172.21.15.29:0/536353518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac068620 0x7fcfac068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 -- 172.21.15.29:0/536353518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac068620 msgr2=0x7fcfac068a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 --2- 172.21.15.29:0/536353518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac068620 0x7fcfac068a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 -- 172.21.15.29:0/536353518 >> v1:172.21.15.144:6789/0 conn(0x7fcfac0ad5f0 legacy=0x7fcfac0ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 -- 172.21.15.29:0/536353518 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac15a250 con 0x7fcfac0ad220 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.672+0000 7fcfab7fe700 1 --2- 172.21.15.29:0/536353518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad220 0x7fcfac159c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0099b0 tx=0x7fcf9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fc75ba893b31f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfaa7fc700 1 -- 172.21.15.29:0/536353518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c009ea0 con 0x7fcfac0ad220 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfaa7fc700 1 -- 172.21.15.29:0/536353518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c00b860 con 0x7fcfac0ad220 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfaa7fc700 1 -- 172.21.15.29:0/536353518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00bc00 con 0x7fcfac0ad220 2023-10-07T09:44:07.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfb21ff700 1 -- 172.21.15.29:0/536353518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad220 msgr2=0x7fcfac159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/536353518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad220 0x7fcfac159c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0099b0 tx=0x7fcf9c009250 comp rx=0 tx=0).stop 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfb21ff700 1 -- 172.21.15.29:0/536353518 shutdown_connections 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/536353518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac068620 0x7fcfac068a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.673+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/536353518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad220 0x7fcfac159c00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.674+0000 7fcfb21ff700 1 -- 172.21.15.29:0/536353518 >> 172.21.15.29:0/536353518 conn(0x7fcfac06dd00 msgr2=0x7fcfac06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.674+0000 7fcfb21ff700 1 -- 172.21.15.29:0/536353518 shutdown_connections 2023-10-07T09:44:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.674+0000 7fcfb21ff700 1 -- 172.21.15.29:0/536353518 wait complete. 2023-10-07T09:44:07.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.675+0000 7fcfb21ff700 1 Processor -- start 2023-10-07T09:44:07.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.675+0000 7fcfb21ff700 1 -- start start 2023-10-07T09:44:07.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.676+0000 7fcfb21ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfac068620 0x7fcfac169a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.676+0000 7fcfb21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac0ad220 0x7fcfac150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.676+0000 7fcfb21ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 0x7fcfac0cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.676+0000 7fcfb21ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0c98c0 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.676+0000 7fcfb21ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0cc610 con 0x7fcfac068620 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.676+0000 7fcfb21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac169f80 con 0x7fcfac0ad220 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 0x7fcfac0cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 0x7fcfac0cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47242/0 (socket says 172.21.15.29:47242) 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 -- 172.21.15.29:0/2719257704 learned_addr learned my addr 172.21.15.29:0/2719257704 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfaaffd700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac0ad220 0x7fcfac150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfab7fe700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfac068620 0x7fcfac169a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 -- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac0ad220 msgr2=0x7fcfac150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac0ad220 0x7fcfac150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 -- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfac068620 msgr2=0x7fcfac169a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfac068620 0x7fcfac169a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 -- 172.21.15.29:0/2719257704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0094f0 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.677+0000 7fcfabfff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 0x7fcfac0cbd10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4009df0 tx=0x7fcfa400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.678+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400caa0 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.678+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa400f040 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.678+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400ce70 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.678+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac0c8f90 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.678+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac0c9550 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.679+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac0c9a70 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.680+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fcfa4014690 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.681+0000 7fcf937fe700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 0x7fcf98069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.681+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fcfa40855b0 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.681+0000 7fcfab7fe700 1 -- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 msgr2=0x7fcf98069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.681+0000 7fcfab7fe700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 0x7fcf98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:07.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.688+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfac0c9a70 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.719 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.718+0000 7f9c3d3881c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:44:07.818 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.818+0000 7f9c3d3881c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:44:07.825 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.824+0000 7fa55e89d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:44:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.882+0000 7fcfab7fe700 1 -- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 msgr2=0x7fcf98069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.882+0000 7fcfab7fe700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 0x7fcf98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:07.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.892+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfac064dd0 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.893+0000 7fcf937fe700 1 -- 172.21.15.29:0/2719257704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16553 (secure 0 0 0) 0x7fcfa4055dc0 con 0x7fcfac0ad5f0 2023-10-07T09:44:07.895 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:07.895 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":65,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:05.195304+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:42:09.508279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-07T09:42:10.701497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":4}]},{"pgid":"7.2d","mappings":[{"from":0,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1583565526":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/3005988804":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/2771895322":"2023-10-08T09:42:27.213405+0000","172.21.15.144:6836/114714":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/2633526602":"2023-10-08T09:42:27.213405+0000","172.21.15.144:6834/114714":"2023-10-08T09:42:27.213405+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:07.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 msgr2=0x7fcf98069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:07.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 0x7fcf98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 msgr2=0x7fcfac0cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:07.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 0x7fcfac0cbd10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4009df0 tx=0x7fcfa400b040 comp rx=0 tx=0).stop 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 shutdown_connections 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcf98067050 0x7fcf98069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfac0ad5f0 0x7fcfac0cbd10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfac068620 0x7fcfac169a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 --2- 172.21.15.29:0/2719257704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfac0ad220 0x7fcfac150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 >> 172.21.15.29:0/2719257704 conn(0x7fcfac06dd00 msgr2=0x7fcfac0ab0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 shutdown_connections 2023-10-07T09:44:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:07.896+0000 7fcfb21ff700 1 -- 172.21.15.29:0/2719257704 wait complete. 2023-10-07T09:44:07.915 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:44:07.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:07.920 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:07.919+0000 7f9c3d3881c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:44:07.922 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:07.921+0000 7fa55e89d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:44:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.113+0000 7f0b98fff700 1 Processor -- start 2023-10-07T09:44:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.113+0000 7f0b98fff700 1 -- start start 2023-10-07T09:44:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b98fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 0x7f0b940d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b98fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b940ce8a0 0x7f0b940c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b98fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b940cbf20 con 0x7f0b94159b20 2023-10-07T09:44:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b98fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b940cc200 con 0x7f0b940ce8a0 2023-10-07T09:44:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b98fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940cc4e0 con 0x7f0b94159750 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b92d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b940ce8a0 0x7f0b940c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.114+0000 7f0b9259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 0x7f0b940d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b9259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 0x7f0b940d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:41204/0 (socket says 172.21.15.29:41204) 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b92d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b940ce8a0 0x7f0b940c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:58542/0 (socket says 172.21.15.29:58542) 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b9259c700 1 -- 172.21.15.29:0/3907317315 learned_addr learned my addr 172.21.15.29:0/3907317315 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b91d9b700 1 --1- 172.21.15.29:0/3907317315 >> v1:172.21.15.144:6789/0 conn(0x7f0b94159b20 0x7f0b940ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:50232/0 (socket says 172.21.15.29:50232) 2023-10-07T09:44:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b9159a700 1 -- 172.21.15.29:0/3907317315 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906927512 0 0) 0x7f0b940cbf20 con 0x7f0b94159b20 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b9159a700 1 -- 172.21.15.29:0/3907317315 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c003570 con 0x7f0b94159b20 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b92d9d700 1 -- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 msgr2=0x7f0b940d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b92d9d700 1 --2- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 0x7f0b940d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b92d9d700 1 -- 172.21.15.29:0/3907317315 >> v1:172.21.15.144:6789/0 conn(0x7f0b94159b20 legacy=0x7f0b940ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b92d9d700 1 -- 172.21.15.29:0/3907317315 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b940cc7c0 con 0x7f0b940ce8a0 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b9259c700 1 --2- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 0x7f0b940d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.115+0000 7f0b92d9d700 1 --2- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b940ce8a0 0x7f0b940c5460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009e60 tx=0x7f0b8c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80268d03454f9adc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b9159a700 1 -- 172.21.15.29:0/3907317315 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00e070 con 0x7f0b940ce8a0 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b9159a700 1 -- 172.21.15.29:0/3907317315 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c00bab0 con 0x7f0b940ce8a0 2023-10-07T09:44:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b9159a700 1 -- 172.21.15.29:0/3907317315 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00be70 con 0x7f0b940ce8a0 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 -- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b940ce8a0 msgr2=0x7f0b940c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 --2- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b940ce8a0 0x7f0b940c5460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009e60 tx=0x7f0b8c009780 comp rx=0 tx=0).stop 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 -- 172.21.15.29:0/3907317315 shutdown_connections 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 --2- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b940ce8a0 0x7f0b940c5460 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 --2- 172.21.15.29:0/3907317315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159750 0x7f0b940d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 -- 172.21.15.29:0/3907317315 >> 172.21.15.29:0/3907317315 conn(0x7f0b94063a10 msgr2=0x7f0b940d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 -- 172.21.15.29:0/3907317315 shutdown_connections 2023-10-07T09:44:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.116+0000 7f0b98fff700 1 -- 172.21.15.29:0/3907317315 wait complete. 2023-10-07T09:44:08.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.118+0000 7f0b98fff700 1 Processor -- start 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.118+0000 7f0b98fff700 1 -- start start 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b98fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 0x7f0b941535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b98fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b94159750 0x7f0b941518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b98fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159b20 0x7f0b94169540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b98fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94153ca0 con 0x7f0b940ce8a0 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b98fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94153f80 con 0x7f0b94159750 2023-10-07T09:44:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b98fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94169d90 con 0x7f0b94159b20 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b9259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 0x7f0b941535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b9259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 0x7f0b941535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47258/0 (socket says 172.21.15.29:47258) 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b9259c700 1 -- 172.21.15.29:0/826195605 learned_addr learned my addr 172.21.15.29:0/826195605 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b92d9d700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159b20 0x7f0b94169540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.119+0000 7f0b91d9b700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b94159750 0x7f0b941518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b9259c700 1 -- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159b20 msgr2=0x7f0b94169540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b9259c700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159b20 0x7f0b94169540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b9259c700 1 -- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b94159750 msgr2=0x7f0b941518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b9259c700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b94159750 0x7f0b941518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b9259c700 1 -- 172.21.15.29:0/826195605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84009480 con 0x7f0b940ce8a0 2023-10-07T09:44:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b91d9b700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b94159750 0x7f0b941518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b9259c700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 0x7f0b941535f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400ec80 tx=0x7f0b84009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84005a00 con 0x7f0b940ce8a0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b84005b60 con 0x7f0b940ce8a0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c009a20 con 0x7f0b940ce8a0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8400c8a0 con 0x7f0b940ce8a0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.120+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9416a350 con 0x7f0b940ce8a0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.122+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b94154260 con 0x7f0b940ce8a0 2023-10-07T09:44:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.122+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f0b8400cc50 con 0x7f0b940ce8a0 2023-10-07T09:44:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.124+0000 7f0b827fc700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 0x7f0b74069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.124+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f0b84084ec0 con 0x7f0b940ce8a0 2023-10-07T09:44:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.124+0000 7f0b91d9b700 1 -- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 msgr2=0x7f0b74069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.124+0000 7f0b91d9b700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 0x7f0b74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:08.128 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:08.127+0000 7f9c3d3881c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:44:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.127+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b84051d50 con 0x7f0b940ce8a0 2023-10-07T09:44:08.230 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:08.229+0000 7f9c3d3881c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:44:08.263 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:08.262+0000 7fa55e89d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:44:08.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.325+0000 7f0b91d9b700 1 -- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 msgr2=0x7f0b74069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:08.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.325+0000 7f0b91d9b700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 0x7f0b74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:08.350 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:08.349+0000 7fa55e89d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.379+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b9404c3c0 con 0x7f0b940ce8a0 2023-10-07T09:44:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.379+0000 7f0b827fc700 1 -- 172.21.15.29:0/826195605 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4600 (secure 0 0 0) 0x7f0b84055370 con 0x7f0b940ce8a0 2023-10-07T09:44:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:08.381 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:42:11.722170+0000","modified":"2023-10-07T09:42:15.247175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6834/2769082805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2769082805},{"type":"v1","addr":"172.21.15.29:6834","nonce":2769082805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-10-07T09:44:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 msgr2=0x7f0b74069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 0x7f0b74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 msgr2=0x7f0b941535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 0x7f0b941535f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400ec80 tx=0x7f0b84009220 comp rx=0 tx=0).stop 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 shutdown_connections 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b74067050 0x7f0b74069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b94159b20 0x7f0b94169540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b940ce8a0 0x7f0b941535f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 --2- 172.21.15.29:0/826195605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b94159750 0x7f0b941518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 >> 172.21.15.29:0/826195605 conn(0x7f0b94063a10 msgr2=0x7f0b940c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 shutdown_connections 2023-10-07T09:44:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.382+0000 7f0b98fff700 1 -- 172.21.15.29:0/826195605 wait complete. 2023-10-07T09:44:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-07T09:44:08.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:08.584 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:08.583+0000 7f9c3d3881c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:44:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.605+0000 7fbcf01ff700 1 Processor -- start 2023-10-07T09:44:08.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.606+0000 7fbcf01ff700 1 -- start start 2023-10-07T09:44:08.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcf01ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbce81533f0 0x7fbce8156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcf01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81537c0 0x7fbce80afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcf01ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbce80af6d0 0x7fbce80b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcf01ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce815a680 con 0x7fbce80af6d0 2023-10-07T09:44:08.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcf01ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce80b2070 con 0x7fbce81533f0 2023-10-07T09:44:08.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcf01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8159670 con 0x7fbce81537c0 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcee79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbce80af6d0 0x7fbce80b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcee79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbce80af6d0 0x7fbce80b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47262/0 (socket says 172.21.15.29:47262) 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcee79c700 1 -- 172.21.15.29:0/2031794422 learned_addr learned my addr 172.21.15.29:0/2031794422 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbced79a700 1 --2- 172.21.15.29:0/2031794422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81537c0 0x7fbce80afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.608+0000 7fbcedf9b700 1 --2- 172.21.15.29:0/2031794422 >> v2:172.21.15.144:3300/0 conn(0x7fbce81533f0 0x7fbce8156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcee79c700 1 -- 172.21.15.29:0/2031794422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81537c0 msgr2=0x7fbce80afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcee79c700 1 --2- 172.21.15.29:0/2031794422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81537c0 0x7fbce80afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcee79c700 1 -- 172.21.15.29:0/2031794422 >> v2:172.21.15.144:3300/0 conn(0x7fbce81533f0 msgr2=0x7fbce8156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcee79c700 1 --2- 172.21.15.29:0/2031794422 >> v2:172.21.15.144:3300/0 conn(0x7fbce81533f0 0x7fbce8156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcee79c700 1 -- 172.21.15.29:0/2031794422 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce815a1f0 con 0x7fbce80af6d0 2023-10-07T09:44:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcee79c700 1 --2- 172.21.15.29:0/2031794422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce80b02e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbce80b2240 tx=0x7fbce0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db6aa7a300a6774 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:08.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcecf99700 1 -- 172.21.15.29:0/2031794422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce000e070 con 0x7fbce80af6d0 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcecf99700 1 -- 172.21.15.29:0/2031794422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce000b970 con 0x7fbce80af6d0 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.609+0000 7fbcecf99700 1 -- 172.21.15.29:0/2031794422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce00092e0 con 0x7fbce80af6d0 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 -- 172.21.15.29:0/2031794422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 msgr2=0x7fbce80b02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/2031794422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce80b02e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbce80b2240 tx=0x7fbce0009840 comp rx=0 tx=0).stop 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 -- 172.21.15.29:0/2031794422 shutdown_connections 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/2031794422 >> v2:172.21.15.144:3300/0 conn(0x7fbce81533f0 0x7fbce8156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/2031794422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce80b02e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/2031794422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81537c0 0x7fbce80afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 -- 172.21.15.29:0/2031794422 >> 172.21.15.29:0/2031794422 conn(0x7fbce80680a0 msgr2=0x7fbce80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 -- 172.21.15.29:0/2031794422 shutdown_connections 2023-10-07T09:44:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.610+0000 7fbcf01ff700 1 -- 172.21.15.29:0/2031794422 wait complete. 2023-10-07T09:44:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.613+0000 7fbcf01ff700 1 Processor -- start 2023-10-07T09:44:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.613+0000 7fbcf01ff700 1 -- start start 2023-10-07T09:44:08.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcf01ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce816a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcf01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81533f0 0x7fbce80a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcf01ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbce81537c0 0x7fbce816a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcf01ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce80a9090 con 0x7fbce80af6d0 2023-10-07T09:44:08.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcf01ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce80ad2c0 con 0x7fbce81537c0 2023-10-07T09:44:08.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcf01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce816b390 con 0x7fbce81533f0 2023-10-07T09:44:08.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce816a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce816a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47266/0 (socket says 172.21.15.29:47266) 2023-10-07T09:44:08.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 -- 172.21.15.29:0/1197700526 learned_addr learned my addr 172.21.15.29:0/1197700526 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:08.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbced79a700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81533f0 0x7fbce80a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcee79c700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbce81537c0 0x7fbce816a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:08.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 -- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81533f0 msgr2=0x7fbce80a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81533f0 0x7fbce80a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 -- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbce81537c0 msgr2=0x7fbce816a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbce81537c0 0x7fbce816a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.615+0000 7fbcedf9b700 1 -- 172.21.15.29:0/1197700526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce0009ae0 con 0x7fbce80af6d0 2023-10-07T09:44:08.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.616+0000 7fbcedf9b700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce816a210 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbce400ec80 tx=0x7fbce40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:08.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.616+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce400c850 con 0x7fbce80af6d0 2023-10-07T09:44:08.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.616+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce400c9b0 con 0x7fbce80af6d0 2023-10-07T09:44:08.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.616+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce400cd50 con 0x7fbce80af6d0 2023-10-07T09:44:08.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.616+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce816b670 con 0x7fbce80af6d0 2023-10-07T09:44:08.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.616+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce80acbd0 con 0x7fbce80af6d0 2023-10-07T09:44:08.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.617+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce804c3c0 con 0x7fbce80af6d0 2023-10-07T09:44:08.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.618+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fbce4013680 con 0x7fbce80af6d0 2023-10-07T09:44:08.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.620+0000 7fbcddffb700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 0x7fbccc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:08.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.620+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fbce4084ca0 con 0x7fbce80af6d0 2023-10-07T09:44:08.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.620+0000 7fbced79a700 1 -- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 msgr2=0x7fbccc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:08.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.620+0000 7fbced79a700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 0x7fbccc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:08.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.623+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbce4051a90 con 0x7fbce80af6d0 2023-10-07T09:44:08.682 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:08.681+0000 7f9c3d3881c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:08.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.821+0000 7fbced79a700 1 -- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 msgr2=0x7fbccc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:08.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.821+0000 7fbced79a700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 0x7fbccc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.869+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbce806f660 con 0x7fbce80af6d0 2023-10-07T09:44:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.870+0000 7fbcddffb700 1 -- 172.21.15.29:0/1197700526 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4600 (secure 0 0 0) 0x7fbce40550b0 con 0x7fbce80af6d0 2023-10-07T09:44:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:08.871 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:42:11.722170+0000","modified":"2023-10-07T09:42:15.247175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6834/2769082805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2769082805},{"type":"v1","addr":"172.21.15.29:6834","nonce":2769082805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-10-07T09:44:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 msgr2=0x7fbccc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 0x7fbccc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 msgr2=0x7fbce816a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce816a210 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbce400ec80 tx=0x7fbce40091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 shutdown_connections 2023-10-07T09:44:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbccc067050 0x7fbccc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbce81537c0 0x7fbce816a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce80af6d0 0x7fbce816a210 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 --2- 172.21.15.29:0/1197700526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbce81533f0 0x7fbce80a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 >> 172.21.15.29:0/1197700526 conn(0x7fbce80680a0 msgr2=0x7fbce80cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 shutdown_connections 2023-10-07T09:44:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:08.872+0000 7fbcf01ff700 1 -- 172.21.15.29:0/1197700526 wait complete. 2023-10-07T09:44:08.876 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-07T09:44:08.887 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:08.886+0000 7fa55e89d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:44:08.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:09.072 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:09.071+0000 7fa55e89d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:44:09.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.098+0000 7f4c0afff700 1 Processor -- start 2023-10-07T09:44:09.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.098+0000 7f4c0afff700 1 -- start start 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.099+0000 7f4c0afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.099+0000 7f4c0afff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c040b0980 con 0x7f4c040b0540 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.099+0000 7f4c0afff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c04154090 con 0x7f4c04153cc0 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.099+0000 7f4c0afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c04159f60 con 0x7f4c0415ade0 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c0959c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4c040b0540 0x7f4c0415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46548/0 (socket says 172.21.15.29:46548) 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c0959c700 1 -- 172.21.15.29:0/681511307 learned_addr learned my addr 172.21.15.29:0/681511307 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c03fff700 1 --2- 172.21.15.29:0/681511307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243810417 0 0) 0x7f4c040b0980 con 0x7f4c040b0540 2023-10-07T09:44:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf4003570 con 0x7f4c040b0540 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942140901 0 0) 0x7f4c04154090 con 0x7f4c04153cc0 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c040b0980 con 0x7f4c04153cc0 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c03fff700 1 -- 172.21.15.29:0/681511307 >> v1:172.21.15.144:6789/0 conn(0x7f4c04153cc0 legacy=0x7f4c040b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c03fff700 1 -- 172.21.15.29:0/681511307 >> v1:172.21.15.29:6789/0 conn(0x7f4c040b0540 legacy=0x7f4c0415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.100+0000 7f4c03fff700 1 -- 172.21.15.29:0/681511307 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0415a240 con 0x7f4c0415ade0 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.101+0000 7f4c03fff700 1 --2- 172.21.15.29:0/681511307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040b0000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009910 tx=0x7f4bf00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e4001c77f6c0809 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.101+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0011630 con 0x7f4c0415ade0 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.101+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf0011790 con 0x7f4c0415ade0 2023-10-07T09:44:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.101+0000 7f4c037fe700 1 -- 172.21.15.29:0/681511307 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0011b30 con 0x7f4c0415ade0 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.101+0000 7f4c0afff700 1 -- 172.21.15.29:0/681511307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 msgr2=0x7f4c040b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.101+0000 7f4c0afff700 1 --2- 172.21.15.29:0/681511307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040b0000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009910 tx=0x7f4bf00091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.102+0000 7f4c0afff700 1 -- 172.21.15.29:0/681511307 shutdown_connections 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.102+0000 7f4c0afff700 1 --2- 172.21.15.29:0/681511307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040b0000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.102+0000 7f4c0afff700 1 -- 172.21.15.29:0/681511307 >> 172.21.15.29:0/681511307 conn(0x7f4c04067610 msgr2=0x7f4c04067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.102+0000 7f4c0afff700 1 -- 172.21.15.29:0/681511307 shutdown_connections 2023-10-07T09:44:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.102+0000 7f4c0afff700 1 -- 172.21.15.29:0/681511307 wait complete. 2023-10-07T09:44:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.103+0000 7f4c0afff700 1 Processor -- start 2023-10-07T09:44:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.103+0000 7f4c0afff700 1 -- start start 2023-10-07T09:44:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.104+0000 7f4c0afff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 0x7f4c0416d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.104+0000 7f4c0afff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04153cc0 0x7f4c04144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.104+0000 7f4c0afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.104+0000 7f4c0afff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0416de20 con 0x7f4c040b0540 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.104+0000 7f4c0afff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c04155bc0 con 0x7f4c04153cc0 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.104+0000 7f4c0afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c04156180 con 0x7f4c0415ade0 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 0x7f4c0416d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 0x7f4c0416d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47274/0 (socket says 172.21.15.29:47274) 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 -- 172.21.15.29:0/4225978035 learned_addr learned my addr 172.21.15.29:0/4225978035 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c03fff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04153cc0 0x7f4c04144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c0959c700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 -- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 msgr2=0x7f4c040aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 -- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04153cc0 msgr2=0x7f4c04144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04153cc0 0x7f4c04144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 -- 172.21.15.29:0/4225978035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf0009420 con 0x7f4c040b0540 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c03fff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04153cc0 0x7f4c04144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.105+0000 7f4c08d9b700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 0x7f4c0416d8e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800c7c0 tx=0x7f4bf8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.106+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8001530 con 0x7f4c040b0540 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.106+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf80016b0 con 0x7f4c040b0540 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.106+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8001a70 con 0x7f4c040b0540 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.106+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c040c7750 con 0x7f4c040b0540 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.106+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c04155e10 con 0x7f4c040b0540 2023-10-07T09:44:09.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.107+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0404c3c0 con 0x7f4c040b0540 2023-10-07T09:44:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.108+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f4bf801b3b0 con 0x7f4c040b0540 2023-10-07T09:44:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.109+0000 7f4c00ff9700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 0x7f4be8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.109+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f4bf8084200 con 0x7f4c040b0540 2023-10-07T09:44:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.109+0000 7f4c03fff700 1 -- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 msgr2=0x7f4be8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.109+0000 7f4c03fff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 0x7f4be8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.117+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bf8050ff0 con 0x7f4c040b0540 2023-10-07T09:44:09.266 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:09.265+0000 7f9c3d3881c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:44:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.310+0000 7f4c03fff700 1 -- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 msgr2=0x7f4be8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.310+0000 7f4c03fff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 0x7f4be8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:09.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.322+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c0415a520 con 0x7f4c040b0540 2023-10-07T09:44:09.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.323+0000 7f4c00ff9700 1 -- 172.21.15.29:0/4225978035 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16553 (secure 0 0 0) 0x7f4bf8054610 con 0x7f4c040b0540 2023-10-07T09:44:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:09.325 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":65,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:05.195304+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:42:09.508279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-07T09:42:10.701497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":4}]},{"pgid":"7.2d","mappings":[{"from":0,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1583565526":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/3005988804":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/2771895322":"2023-10-08T09:42:27.213405+0000","172.21.15.144:6836/114714":"2023-10-08T09:42:27.213405+0000","172.21.15.144:0/2633526602":"2023-10-08T09:42:27.213405+0000","172.21.15.144:6834/114714":"2023-10-08T09:42:27.213405+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 msgr2=0x7f4be8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 0x7f4be8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 msgr2=0x7f4c0416d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 0x7f4c0416d8e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800c7c0 tx=0x7f4bf8007e00 comp rx=0 tx=0).stop 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 shutdown_connections 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4be8067050 0x7f4be8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415ade0 0x7f4c040aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c040b0540 0x7f4c0416d8e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 --2- 172.21.15.29:0/4225978035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04153cc0 0x7f4c04144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 >> 172.21.15.29:0/4225978035 conn(0x7f4c04067610 msgr2=0x7f4c04152f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 shutdown_connections 2023-10-07T09:44:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.326+0000 7f4c0afff700 1 -- 172.21.15.29:0/4225978035 wait complete. 2023-10-07T09:44:09.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:44:09.450 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:09.449+0000 7fa55e89d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:09.462 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:09.461+0000 7f9c3d3881c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:44:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.547+0000 7f72977ff700 1 Processor -- start 2023-10-07T09:44:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.548+0000 7f72977ff700 1 -- start start 2023-10-07T09:44:09.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f72977ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72900afd20 0x7f72901596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f72977ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f72900acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f72977ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900b0dc0 con 0x7f7290159320 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f72977ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72900cd0d0 con 0x7f72900afd20 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f72977ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b01b0 con 0x7f72900b1250 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f7294d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72900afd20 0x7f72901596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f729559b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7290159320 0x7f72900cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46552/0 (socket says 172.21.15.29:46552) 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f7294d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72900afd20 0x7f72901596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47276/0 (socket says 172.21.15.29:47276) 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f729559b700 1 -- 172.21.15.29:0/3605563793 learned_addr learned my addr 172.21.15.29:0/3605563793 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.549+0000 7f7295d9c700 1 --2- 172.21.15.29:0/3605563793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f72900acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7294d9a700 1 -- 172.21.15.29:0/3605563793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 msgr2=0x7f72900acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7294d9a700 1 --2- 172.21.15.29:0/3605563793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f72900acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7294d9a700 1 -- 172.21.15.29:0/3605563793 >> v1:172.21.15.29:6789/0 conn(0x7f7290159320 legacy=0x7f72900cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7294d9a700 1 -- 172.21.15.29:0/3605563793 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900b0490 con 0x7f72900afd20 2023-10-07T09:44:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7287fff700 1 -- 172.21.15.29:0/3605563793 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623410236 0 0) 0x7f72900b0dc0 con 0x7f7290159320 2023-10-07T09:44:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7294d9a700 1 --2- 172.21.15.29:0/3605563793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f72901596f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f72800098e0 tx=0x7f72800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1181857856e92a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7287fff700 1 -- 172.21.15.29:0/3605563793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728000b6d0 con 0x7f72900afd20 2023-10-07T09:44:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7287fff700 1 -- 172.21.15.29:0/3605563793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728000b830 con 0x7f72900afd20 2023-10-07T09:44:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.550+0000 7f7287fff700 1 -- 172.21.15.29:0/3605563793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728000bbd0 con 0x7f72900afd20 2023-10-07T09:44:09.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 -- 172.21.15.29:0/3605563793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 msgr2=0x7f72901596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 --2- 172.21.15.29:0/3605563793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f72901596f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f72800098e0 tx=0x7f72800091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 -- 172.21.15.29:0/3605563793 shutdown_connections 2023-10-07T09:44:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 --2- 172.21.15.29:0/3605563793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f72900acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 --2- 172.21.15.29:0/3605563793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f72901596f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 -- 172.21.15.29:0/3605563793 >> 172.21.15.29:0/3605563793 conn(0x7f72900680a0 msgr2=0x7f72900cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.551+0000 7f72977ff700 1 -- 172.21.15.29:0/3605563793 shutdown_connections 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.552+0000 7f72977ff700 1 -- 172.21.15.29:0/3605563793 wait complete. 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.553+0000 7f72977ff700 1 Processor -- start 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.554+0000 7f72977ff700 1 -- start start 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f72977ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f729016d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f72977ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f7290158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f72977ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7290159320 0x7f7290168a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f72977ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900ab390 con 0x7f72900afd20 2023-10-07T09:44:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f72977ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729016da80 con 0x7f7290159320 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f72977ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900c1530 con 0x7f72900b1250 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f729559b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f729016d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f729559b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f729016d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47282/0 (socket says 172.21.15.29:47282) 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f729559b700 1 -- 172.21.15.29:0/510446393 learned_addr learned my addr 172.21.15.29:0/510446393 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.555+0000 7f7294d9a700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f7290158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7295d9c700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7290159320 0x7f7290168a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7294d9a700 1 -- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7290159320 msgr2=0x7f7290168a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7294d9a700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7290159320 0x7f7290168a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7294d9a700 1 -- 172.21.15.29:0/510446393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 msgr2=0x7f729016d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7294d9a700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f729016d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7294d9a700 1 -- 172.21.15.29:0/510446393 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280009420 con 0x7f72900b1250 2023-10-07T09:44:09.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f729559b700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f729016d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:09.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.556+0000 7f7294d9a700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f7290158210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7280016000 tx=0x7f7280017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:09.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.557+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280000d10 con 0x7f72900b1250 2023-10-07T09:44:09.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.557+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728000bbb0 con 0x7f72900b1250 2023-10-07T09:44:09.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.557+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280011670 con 0x7f72900b1250 2023-10-07T09:44:09.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.557+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900ab650 con 0x7f72900b1250 2023-10-07T09:44:09.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.557+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72900abc10 con 0x7f72900b1250 2023-10-07T09:44:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.558+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729004c3c0 con 0x7f72900b1250 2023-10-07T09:44:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.559+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f7280011a20 con 0x7f72900b1250 2023-10-07T09:44:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.560+0000 7f72857fa700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 0x7f7274069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.560+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7280085f30 con 0x7f72900b1250 2023-10-07T09:44:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.560+0000 7f729559b700 1 -- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 msgr2=0x7f7274069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.560+0000 7f729559b700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 0x7f7274069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:09.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.564+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7280052d20 con 0x7f72900b1250 2023-10-07T09:44:09.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.761+0000 7f729559b700 1 -- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 msgr2=0x7f7274069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:09.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.761+0000 7f729559b700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 0x7f7274069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:09.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.814+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f729006f7c0 con 0x7f72900b1250 2023-10-07T09:44:09.816 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:44:09.815+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:44:09.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.820+0000 7f72857fa700 1 -- 172.21.15.29:0/510446393 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7280056340 con 0x7f72900b1250 2023-10-07T09:44:09.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 msgr2=0x7f7274069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:09.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 0x7f7274069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 msgr2=0x7f7290158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f7290158210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7280016000 tx=0x7f7280017000 comp rx=0 tx=0).stop 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 shutdown_connections 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7274067050 0x7f7274069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7290159320 0x7f7290168a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72900afd20 0x7f729016d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 --2- 172.21.15.29:0/510446393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72900b1250 0x7f7290158210 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 >> 172.21.15.29:0/510446393 conn(0x7f72900680a0 msgr2=0x7f729006e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 shutdown_connections 2023-10-07T09:44:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:09.823+0000 7f72977ff700 1 -- 172.21.15.29:0/510446393 wait complete. 2023-10-07T09:44:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:44:09.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:44:09.851 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:09.850+0000 7f9c3d3881c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:09.876 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:09.875+0000 7fa55e89d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:44:09.884 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:44:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.045+0000 7f91eafff700 1 Processor -- start 2023-10-07T09:44:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.045+0000 7f91eafff700 1 -- start start 2023-10-07T09:44:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91eafff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91eafff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f91e40ce8a0 0x7f91e40c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91eafff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e40cbf20 con 0x7f91e4159b20 2023-10-07T09:44:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91eafff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e40cc200 con 0x7f91e40ce8a0 2023-10-07T09:44:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91eafff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40cc4e0 con 0x7f91e4159750 2023-10-07T09:44:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91e3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91e4159b20 0x7f91e40ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46578/0 (socket says 172.21.15.29:46578) 2023-10-07T09:44:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91e3fff700 1 -- 172.21.15.29:0/2713436554 learned_addr learned my addr 172.21.15.29:0/2713436554 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91e959c700 1 --2- 172.21.15.29:0/2713436554 >> v2:172.21.15.144:3300/0 conn(0x7f91e40ce8a0 0x7f91e40c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91e8d9b700 1 --2- 172.21.15.29:0/2713436554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:10.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.047+0000 7f91e37fe700 1 -- 172.21.15.29:0/2713436554 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488847884 0 0) 0x7f91e40cbf20 con 0x7f91e4159b20 2023-10-07T09:44:10.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e37fe700 1 -- 172.21.15.29:0/2713436554 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d4003570 con 0x7f91e4159b20 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e8d9b700 1 -- 172.21.15.29:0/2713436554 >> v2:172.21.15.144:3300/0 conn(0x7f91e40ce8a0 msgr2=0x7f91e40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e8d9b700 1 --2- 172.21.15.29:0/2713436554 >> v2:172.21.15.144:3300/0 conn(0x7f91e40ce8a0 0x7f91e40c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e8d9b700 1 -- 172.21.15.29:0/2713436554 >> v1:172.21.15.29:6789/0 conn(0x7f91e4159b20 legacy=0x7f91e40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e8d9b700 1 -- 172.21.15.29:0/2713436554 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40cc7c0 con 0x7f91e4159750 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e959c700 1 --2- 172.21.15.29:0/2713436554 >> v2:172.21.15.144:3300/0 conn(0x7f91e40ce8a0 0x7f91e40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e8d9b700 1 --2- 172.21.15.29:0/2713436554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e40d10a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91d80098d0 tx=0x7f91d8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a489ccc493a07444 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e37fe700 1 -- 172.21.15.29:0/2713436554 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8005c10 con 0x7f91e4159750 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e37fe700 1 -- 172.21.15.29:0/2713436554 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d8005d70 con 0x7f91e4159750 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.048+0000 7f91e37fe700 1 -- 172.21.15.29:0/2713436554 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8006260 con 0x7f91e4159750 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 -- 172.21.15.29:0/2713436554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 msgr2=0x7f91e40d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 --2- 172.21.15.29:0/2713436554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e40d10a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91d80098d0 tx=0x7f91d8009120 comp rx=0 tx=0).stop 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 -- 172.21.15.29:0/2713436554 shutdown_connections 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 --2- 172.21.15.29:0/2713436554 >> v2:172.21.15.144:3300/0 conn(0x7f91e40ce8a0 0x7f91e40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 --2- 172.21.15.29:0/2713436554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e40d10a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 -- 172.21.15.29:0/2713436554 >> 172.21.15.29:0/2713436554 conn(0x7f91e4063a10 msgr2=0x7f91e40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 -- 172.21.15.29:0/2713436554 shutdown_connections 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.049+0000 7f91eafff700 1 -- 172.21.15.29:0/2713436554 wait complete. 2023-10-07T09:44:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.051+0000 7f91eafff700 1 Processor -- start 2023-10-07T09:44:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.052+0000 7f91eafff700 1 -- start start 2023-10-07T09:44:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91eafff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 0x7f91e41585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91eafff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e4158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91eafff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e4159b20 0x7f91e414a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91eafff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e41580e0 con 0x7f91e40ce8a0 2023-10-07T09:44:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91eafff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e41574d0 con 0x7f91e4159b20 2023-10-07T09:44:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91eafff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e41577b0 con 0x7f91e4159750 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91e8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 0x7f91e41585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91e8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 0x7f91e41585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47302/0 (socket says 172.21.15.29:47302) 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91e8d9b700 1 -- 172.21.15.29:0/707149477 learned_addr learned my addr 172.21.15.29:0/707149477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91e959c700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e4159b20 0x7f91e414a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.053+0000 7f91e3fff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e4158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e8d9b700 1 -- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 msgr2=0x7f91e4158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e8d9b700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e4158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e8d9b700 1 -- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e4159b20 msgr2=0x7f91e414a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e8d9b700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e4159b20 0x7f91e414a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e8d9b700 1 -- 172.21.15.29:0/707149477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc009a20 con 0x7f91e40ce8a0 2023-10-07T09:44:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e959c700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e4159b20 0x7f91e414a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e8d9b700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 0x7f91e41585f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009ec0 tx=0x7f91d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d801d910 con 0x7f91e40ce8a0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d801da90 con 0x7f91e40ce8a0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d801de50 con 0x7f91e40ce8a0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d80093c0 con 0x7f91e40ce8a0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.054+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e4157d70 con 0x7f91e40ce8a0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.056+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e404c3c0 con 0x7f91e40ce8a0 2023-10-07T09:44:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.056+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f91d801b690 con 0x7f91e40ce8a0 2023-10-07T09:44:10.058 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.057+0000 7fa55e89d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.057+0000 7f91e0ff9700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 0x7f91c8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.057+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f91d80866d0 con 0x7f91e40ce8a0 2023-10-07T09:44:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.057+0000 7f91e3fff700 1 -- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 msgr2=0x7f91c8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.057+0000 7f91e3fff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 0x7f91c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:10.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.062+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91d8053580 con 0x7f91e40ce8a0 2023-10-07T09:44:10.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.243+0000 7fa55e89d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:44:10.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.258+0000 7f91e3fff700 1 -- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 msgr2=0x7f91c8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:10.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.258+0000 7f91e3fff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 0x7f91c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:10.306 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:10.305+0000 7f9c3d3881c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:44:10.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.313+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f91e40892e0 con 0x7f91e40ce8a0 2023-10-07T09:44:10.353 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.352+0000 7fa55e89d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.362+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 24) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f91d8053950 con 0x7f91e40ce8a0 2023-10-07T09:44:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.363+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 msgr2=0x7f91c8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:10.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.363+0000 7f91e0ff9700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 0x7f91c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.437 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.435+0000 7fa4daf07700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.437 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.435+0000 7fa4daf07700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.437 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.435+0000 7fa4daf07700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.435+0000 7fa4daf07700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.435+0000 7fa4daf07700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.436+0000 7fa4d8702700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.436+0000 7fa4d8702700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.436+0000 7fa4d8702700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.436+0000 7fa4d8702700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:10.436+0000 7fa4d8702700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:10.501 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:10.500+0000 7f9c3d3881c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:10.692 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:10.691+0000 7f9c3d3881c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:44:10.805 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:10.804+0000 7f9c3d3881c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.820+0000 7f91e0ff9700 1 -- 172.21.15.29:0/707149477 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f91d806ece0 con 0x7f91e40ce8a0 2023-10-07T09:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 msgr2=0x7f91e41585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 0x7f91e41585f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009ec0 tx=0x7f91d8016000 comp rx=0 tx=0).stop 2023-10-07T09:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 shutdown_connections 2023-10-07T09:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f91c8067050 0x7f91c8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e4159b20 0x7f91e414a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91e40ce8a0 0x7f91e41585f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 --2- 172.21.15.29:0/707149477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91e4159750 0x7f91e4158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 >> 172.21.15.29:0/707149477 conn(0x7f91e4063a10 msgr2=0x7f91e40705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 shutdown_connections 2023-10-07T09:44:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:10.823+0000 7f91eafff700 1 -- 172.21.15.29:0/707149477 wait complete. 2023-10-07T09:44:10.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:44:11.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.082+0000 7f88b73ff700 1 Processor -- start 2023-10-07T09:44:11.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.082+0000 7f88b73ff700 1 -- start start 2023-10-07T09:44:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b73ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b01559f0 0x7f88b00b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b73ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88b00b0660 0x7f88b01596d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b73ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f88b0159c10 0x7f88b00adb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b73ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b00c98f0 con 0x7f88b00b0660 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b73ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b015a090 con 0x7f88b0159c10 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b73ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00c8c70 con 0x7f88b01559f0 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88b00b0660 0x7f88b01596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88b00b0660 0x7f88b01596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47314/0 (socket says 172.21.15.29:47314) 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 -- 172.21.15.29:0/1827402336 learned_addr learned my addr 172.21.15.29:0/1827402336 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b519b700 1 --2- 172.21.15.29:0/1827402336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b01559f0 0x7f88b00b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b599c700 1 --2- 172.21.15.29:0/1827402336 >> v2:172.21.15.144:3300/0 conn(0x7f88b0159c10 0x7f88b00adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 -- 172.21.15.29:0/1827402336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b01559f0 msgr2=0x7f88b00b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 --2- 172.21.15.29:0/1827402336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b01559f0 0x7f88b00b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 -- 172.21.15.29:0/1827402336 >> v2:172.21.15.144:3300/0 conn(0x7f88b0159c10 msgr2=0x7f88b00adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 --2- 172.21.15.29:0/1827402336 >> v2:172.21.15.144:3300/0 conn(0x7f88b0159c10 0x7f88b00adb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b499a700 1 -- 172.21.15.29:0/1827402336 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00c8f50 con 0x7f88b00b0660 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.084+0000 7f88b599c700 1 --2- 172.21.15.29:0/1827402336 >> v2:172.21.15.144:3300/0 conn(0x7f88b0159c10 0x7f88b00adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.085+0000 7f88b499a700 1 --2- 172.21.15.29:0/1827402336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b00b0660 0x7f88b01596d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f88b00adaa0 tx=0x7f88a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6adf183b0e0d2ac server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:11.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.085+0000 7f88a7fff700 1 -- 172.21.15.29:0/1827402336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0009ea0 con 0x7f88b00b0660 2023-10-07T09:44:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.085+0000 7f88a7fff700 1 -- 172.21.15.29:0/1827402336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a000b820 con 0x7f88b00b0660 2023-10-07T09:44:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.085+0000 7f88a7fff700 1 -- 172.21.15.29:0/1827402336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a000bbe0 con 0x7f88b00b0660 2023-10-07T09:44:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.085+0000 7f88b73ff700 1 -- 172.21.15.29:0/1827402336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b00b0660 msgr2=0x7f88b01596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.085+0000 7f88b73ff700 1 --2- 172.21.15.29:0/1827402336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b00b0660 0x7f88b01596d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f88b00adaa0 tx=0x7f88a0009250 comp rx=0 tx=0).stop 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 -- 172.21.15.29:0/1827402336 shutdown_connections 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 --2- 172.21.15.29:0/1827402336 >> v2:172.21.15.144:3300/0 conn(0x7f88b0159c10 0x7f88b00adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 --2- 172.21.15.29:0/1827402336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b01559f0 0x7f88b00b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 --2- 172.21.15.29:0/1827402336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b00b0660 0x7f88b01596d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 -- 172.21.15.29:0/1827402336 >> 172.21.15.29:0/1827402336 conn(0x7f88b0067610 msgr2=0x7f88b0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 -- 172.21.15.29:0/1827402336 shutdown_connections 2023-10-07T09:44:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.086+0000 7f88b73ff700 1 -- 172.21.15.29:0/1827402336 wait complete. 2023-10-07T09:44:11.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.088+0000 7f88b73ff700 1 Processor -- start 2023-10-07T09:44:11.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.088+0000 7f88b73ff700 1 -- start start 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.089+0000 7f88b73ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b00b0660 0x7f88b01517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.089+0000 7f88b73ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 0x7f88b006e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.089+0000 7f88b73ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b0159c10 0x7f88b006edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.089+0000 7f88b73ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00a62b0 con 0x7f88b01559f0 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.089+0000 7f88b73ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00c7dc0 con 0x7f88b00b0660 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.089+0000 7f88b73ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00c71b0 con 0x7f88b0159c10 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 0x7f88b006e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 0x7f88b006e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47324/0 (socket says 172.21.15.29:47324) 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 -- 172.21.15.29:0/2876778626 learned_addr learned my addr 172.21.15.29:0/2876778626 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b519b700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b00b0660 0x7f88b01517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b599c700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b0159c10 0x7f88b006edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 -- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b0159c10 msgr2=0x7f88b006edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b0159c10 0x7f88b006edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 -- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b00b0660 msgr2=0x7f88b01517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b00b0660 0x7f88b01517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 -- 172.21.15.29:0/2876778626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00094f0 con 0x7f88b01559f0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.090+0000 7f88b499a700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 0x7f88b006e8b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f88b0064190 tx=0x7f88a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.091+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0002f30 con 0x7f88b01559f0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.091+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a0000bc0 con 0x7f88b01559f0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.091+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a000bbc0 con 0x7f88b01559f0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.091+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b00c7490 con 0x7f88b01559f0 2023-10-07T09:44:11.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.091+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b00c7a50 con 0x7f88b01559f0 2023-10-07T09:44:11.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.092+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b00718b0 con 0x7f88b01559f0 2023-10-07T09:44:11.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.093+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f88a0023070 con 0x7f88b01559f0 2023-10-07T09:44:11.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.094+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6741+0+0 (secure 0 0 0) 0x7f88a00856f0 con 0x7f88b01559f0 2023-10-07T09:44:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.099+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88a0052820 con 0x7f88b01559f0 2023-10-07T09:44:11.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.303+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f88b00c7f00 con 0x7f88b01559f0 2023-10-07T09:44:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.366+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 25) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f88a00522d0 con 0x7f88b01559f0 2023-10-07T09:44:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.366+0000 7f88a57fa700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f889406ca50 0x7f889406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.367+0000 7f88b519b700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f889406ca50 0x7f889406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:11.373+0000 7f88b519b700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f889406ca50 0x7f889406ee30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009db0 tx=0x7f88ac0091a0 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.215+0000 7f88a57fa700 1 -- 172.21.15.29:0/2876778626 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) v1 ==== 181+0+0 (secure 0 0 0) 0x7f88a0026780 con 0x7f88b01559f0 2023-10-07T09:44:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f889406ca50 msgr2=0x7f889406ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f889406ca50 0x7f889406ee30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009db0 tx=0x7f88ac0091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 msgr2=0x7f88b006e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 0x7f88b006e8b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f88b0064190 tx=0x7f88a0017000 comp rx=0 tx=0).stop 2023-10-07T09:44:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 shutdown_connections 2023-10-07T09:44:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f889406ca50 0x7f889406ee30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88b0159c10 0x7f88b006edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88b00b0660 0x7f88b01517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 --2- 172.21.15.29:0/2876778626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88b01559f0 0x7f88b006e8b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.220+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 >> 172.21.15.29:0/2876778626 conn(0x7f88b0067610 msgr2=0x7f88b006d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.221+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 shutdown_connections 2023-10-07T09:44:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.221+0000 7f88b73ff700 1 -- 172.21.15.29:0/2876778626 wait complete. 2023-10-07T09:44:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:44:12.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:44:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.447+0000 7f857f7ff700 1 Processor -- start 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.447+0000 7f857f7ff700 1 -- start start 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857f7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85780b2a20 0x7f85780cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857f7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f85780cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857f7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85780cc940 con 0x7f85780cf0f0 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857f7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85780cf530 con 0x7f85780b2a20 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857f7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85780b2df0 con 0x7f85780cb6f0 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85780b2a20 0x7f85780cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85780b2a20 0x7f85780cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47328/0 (socket says 172.21.15.29:47328) 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 -- 172.21.15.29:0/1309458222 learned_addr learned my addr 172.21.15.29:0/1309458222 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857cd9a700 1 --2- 172.21.15.29:0/1309458222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f85780cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 -- 172.21.15.29:0/1309458222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 msgr2=0x7f85780cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 --2- 172.21.15.29:0/1309458222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f85780cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 -- 172.21.15.29:0/1309458222 >> v1:172.21.15.144:6789/0 conn(0x7f85780cf0f0 legacy=0x7f8578159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.449+0000 7f857d59b700 1 -- 172.21.15.29:0/1309458222 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780cbd30 con 0x7f85780b2a20 2023-10-07T09:44:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.450+0000 7f857d59b700 1 --2- 172.21.15.29:0/1309458222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780b2a20 0x7f85780cb1b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8574009990 tx=0x7f8574009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ee8def919a05c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.450+0000 7f856ffff700 1 -- 172.21.15.29:0/1309458222 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8574005c10 con 0x7f85780b2a20 2023-10-07T09:44:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.450+0000 7f856ffff700 1 -- 172.21.15.29:0/1309458222 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8574005d70 con 0x7f85780b2a20 2023-10-07T09:44:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.450+0000 7f856ffff700 1 -- 172.21.15.29:0/1309458222 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857400b840 con 0x7f85780b2a20 2023-10-07T09:44:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.450+0000 7f857f7ff700 1 -- 172.21.15.29:0/1309458222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780b2a20 msgr2=0x7f85780cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.450+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1309458222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780b2a20 0x7f85780cb1b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8574009990 tx=0x7f8574009220 comp rx=0 tx=0).stop 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.451+0000 7f857f7ff700 1 -- 172.21.15.29:0/1309458222 shutdown_connections 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.451+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1309458222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780b2a20 0x7f85780cb1b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.451+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1309458222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f85780cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.451+0000 7f857f7ff700 1 -- 172.21.15.29:0/1309458222 >> 172.21.15.29:0/1309458222 conn(0x7f8578099820 msgr2=0x7f8578063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.451+0000 7f857f7ff700 1 -- 172.21.15.29:0/1309458222 shutdown_connections 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.451+0000 7f857f7ff700 1 -- 172.21.15.29:0/1309458222 wait complete. 2023-10-07T09:44:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.452+0000 7f857f7ff700 1 Processor -- start 2023-10-07T09:44:12.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.452+0000 7f857f7ff700 1 -- start start 2023-10-07T09:44:12.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857f7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85780b2a20 0x7f857816a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:12.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857f7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f857809c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857f7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 0x7f857814d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857f7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857809bd20 con 0x7f85780cf0f0 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857f7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857816a890 con 0x7f85780b2a20 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857f7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857809c750 con 0x7f85780cb6f0 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857dd9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 0x7f857814d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857dd9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 0x7f857814d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47344/0 (socket says 172.21.15.29:47344) 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857dd9c700 1 -- 172.21.15.29:0/1959312886 learned_addr learned my addr 172.21.15.29:0/1959312886 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.453+0000 7f857cd9a700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f857809c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857d59b700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85780b2a20 0x7f857816a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857dd9c700 1 -- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 msgr2=0x7f857809c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857dd9c700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f857809c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857dd9c700 1 -- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85780b2a20 msgr2=0x7f857816a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857dd9c700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85780b2a20 0x7f857816a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857dd9c700 1 -- 172.21.15.29:0/1959312886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8574009480 con 0x7f85780cf0f0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857d59b700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85780b2a20 0x7f857816a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857dd9c700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 0x7f857814d140 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f857000c930 tx=0x7f85700026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570012630 con 0x7f85780cf0f0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85700127d0 con 0x7f85780cf0f0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570012b90 con 0x7f85780cf0f0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.454+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857809d590 con 0x7f85780cf0f0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.455+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857809cf40 con 0x7f85780cf0f0 2023-10-07T09:44:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.456+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857809d720 con 0x7f85780cf0f0 2023-10-07T09:44:12.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.456+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f857000a3b0 con 0x7f85780cf0f0 2023-10-07T09:44:12.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.457+0000 7f856d7fa700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f855c066d80 0x7f855c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:12.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.457+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f8570084140 con 0x7f85780cf0f0 2023-10-07T09:44:12.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.458+0000 7f857d59b700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f855c066d80 0x7f855c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.459+0000 7f857d59b700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f855c066d80 0x7f855c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f85740030c0 tx=0x7f8574015000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:12.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.462+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8570051410 con 0x7f85780cf0f0 2023-10-07T09:44:12.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:12.673+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f85780718b0 con 0x7f85780cf0f0 2023-10-07T09:44:13.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.221+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f85700517e0 con 0x7f85780cf0f0 2023-10-07T09:44:13.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.376+0000 7f856d7fa700 1 -- 172.21.15.29:0/1959312886 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8570054ea0 con 0x7f85780cf0f0 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f855c066d80 msgr2=0x7f855c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f855c066d80 0x7f855c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f85740030c0 tx=0x7f8574015000 comp rx=0 tx=0).stop 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 msgr2=0x7f857814d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 0x7f857814d140 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f857000c930 tx=0x7f85700026e0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 shutdown_connections 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f855c066d80 0x7f855c069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85780cf0f0 0x7f857814d140 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85780b2a20 0x7f857816a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.380+0000 7f857f7ff700 1 --2- 172.21.15.29:0/1959312886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85780cb6f0 0x7f857809c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.381+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 >> 172.21.15.29:0/1959312886 conn(0x7f8578099820 msgr2=0x7f85780c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.381+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 shutdown_connections 2023-10-07T09:44:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.381+0000 7f857f7ff700 1 -- 172.21.15.29:0/1959312886 wait complete. 2023-10-07T09:44:13.384 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:44:13.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T09:44:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.605+0000 7f8c423ff700 1 Processor -- start 2023-10-07T09:44:13.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.605+0000 7f8c423ff700 1 -- start start 2023-10-07T09:44:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c423ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 0x7f8c3c0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c423ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8c3c159d90 0x7f8c3c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c423ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3c157790 con 0x7f8c3c153cc0 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c423ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c3c0b0950 con 0x7f8c3c159d90 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c423ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c157580 con 0x7f8c3c14c8d0 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c3b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8c3c153cc0 0x7f8c3c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39358/0 (socket says 172.21.15.29:39358) 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c3b7fe700 1 -- 172.21.15.29:0/3596937768 learned_addr learned my addr 172.21.15.29:0/3596937768 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 0x7f8c3c0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c4099c700 1 --2- 172.21.15.29:0/3596937768 >> v2:172.21.15.144:3300/0 conn(0x7f8c3c159d90 0x7f8c3c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.607+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215377549 0 0) 0x7f8c3c157790 con 0x7f8c3c153cc0 2023-10-07T09:44:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c24003570 con 0x7f8c3c153cc0 2023-10-07T09:44:13.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297675358 0 0) 0x7f8c24003570 con 0x7f8c3c153cc0 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 msgr2=0x7f8c3c0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 0x7f8c3c0afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 >> v2:172.21.15.144:3300/0 conn(0x7f8c3c159d90 msgr2=0x7f8c3c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 --2- 172.21.15.29:0/3596937768 >> v2:172.21.15.144:3300/0 conn(0x7f8c3c159d90 0x7f8c3c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c157a70 con 0x7f8c3c153cc0 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f8c30001470 con 0x7f8c3c153cc0 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 >> v1:172.21.15.29:6789/0 conn(0x7f8c3c153cc0 legacy=0x7f8c3c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c24003ae0 0x7f8c24003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24004810 0x7f8c24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 0x7f8c240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c157580 con 0x7f8c24003ae0 2023-10-07T09:44:13.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2400dca0 con 0x7f8c24004810 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2400de60 con 0x7f8c3c14c8d0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c24003ae0 0x7f8c24003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c3b7fe700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24004810 0x7f8c24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.608+0000 7f8c4099c700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 0x7f8c240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3bfff700 1 -- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 msgr2=0x7f8c240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c14c8d0 0x7f8c240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3bfff700 1 -- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24004810 msgr2=0x7f8c24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24004810 0x7f8c24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3bfff700 1 -- 172.21.15.29:0/3596937768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24003570 con 0x7f8c24003ae0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c24003ae0 0x7f8c24003ef0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009c00 tx=0x7f8c2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40eed14afa4cb4f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c00e710 con 0x7f8c24003ae0 2023-10-07T09:44:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c00e870 con 0x7f8c24003ae0 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.609+0000 7f8c3affd700 1 -- 172.21.15.29:0/3596937768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c00ec30 con 0x7f8c24003ae0 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c423ff700 1 -- 172.21.15.29:0/3596937768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c24003ae0 msgr2=0x7f8c24003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c423ff700 1 --2- 172.21.15.29:0/3596937768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c24003ae0 0x7f8c24003ef0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009c00 tx=0x7f8c2c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c4099c700 1 -- 172.21.15.29:0/3596937768 reap_dead start 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c423ff700 1 -- 172.21.15.29:0/3596937768 shutdown_connections 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c423ff700 1 -- 172.21.15.29:0/3596937768 >> 172.21.15.29:0/3596937768 conn(0x7f8c3c067610 msgr2=0x7f8c3c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c423ff700 1 -- 172.21.15.29:0/3596937768 shutdown_connections 2023-10-07T09:44:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.610+0000 7f8c423ff700 1 -- 172.21.15.29:0/3596937768 wait complete. 2023-10-07T09:44:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.611+0000 7f8c423ff700 1 Processor -- start 2023-10-07T09:44:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.611+0000 7f8c423ff700 1 -- start start 2023-10-07T09:44:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.612+0000 7f8c423ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24003ae0 0x7f8c3c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.612+0000 7f8c423ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 0x7f8c3c169ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.612+0000 7f8c423ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c16a010 0x7f8c3c14e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.612+0000 7f8c423ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c14c230 con 0x7f8c3c0c5850 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.612+0000 7f8c423ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c14dc70 con 0x7f8c24003ae0 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.612+0000 7f8c423ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c14df50 con 0x7f8c3c16a010 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 0x7f8c3c169ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 0x7f8c3c169ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57450/0 (socket says 172.21.15.29:57450) 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 -- 172.21.15.29:0/33153986 learned_addr learned my addr 172.21.15.29:0/33153986 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c4099c700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c16a010 0x7f8c3c14e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24003ae0 0x7f8c3c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 -- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c16a010 msgr2=0x7f8c3c14e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c16a010 0x7f8c3c14e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 -- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24003ae0 msgr2=0x7f8c3c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24003ae0 0x7f8c3c0c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 -- 172.21.15.29:0/33153986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c0c6000 con 0x7f8c3c0c5850 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.613+0000 7f8c3b7fe700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 0x7f8c3c169ad0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000ebf0 tx=0x7f8c30000d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.614+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3000c760 con 0x7f8c3c0c5850 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.614+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3000c8e0 con 0x7f8c3c0c5850 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.614+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3000cca0 con 0x7f8c3c0c5850 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.614+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3c0c62e0 con 0x7f8c3c0c5850 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.614+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3c0c5cc0 con 0x7f8c3c0c5850 2023-10-07T09:44:13.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.615+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3c04c3c0 con 0x7f8c3c0c5850 2023-10-07T09:44:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.616+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8c300133e0 con 0x7f8c3c0c5850 2023-10-07T09:44:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.617+0000 7f8c23fff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c28066f30 0x7f8c280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.617+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f8c30084170 con 0x7f8c3c0c5850 2023-10-07T09:44:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.617+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c28066f30 0x7f8c280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.618+0000 7f8c3bfff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c28066f30 0x7f8c280693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c156d10 tx=0x7f8c2c01a000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:13.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.621+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c30051340 con 0x7f8c3c0c5850 2023-10-07T09:44:13.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.819+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8c3c14e090 con 0x7f8c3c0c5850 2023-10-07T09:44:13.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.823+0000 7f8c23fff700 1 -- 172.21.15.29:0/33153986 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8c30054960 con 0x7f8c3c0c5850 2023-10-07T09:44:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c28066f30 msgr2=0x7f8c280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c28066f30 0x7f8c280693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c156d10 tx=0x7f8c2c01a000 comp rx=0 tx=0).stop 2023-10-07T09:44:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 msgr2=0x7f8c3c169ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 0x7f8c3c169ad0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000ebf0 tx=0x7f8c30000d00 comp rx=0 tx=0).stop 2023-10-07T09:44:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 shutdown_connections 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c28066f30 0x7f8c280693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c3c16a010 0x7f8c3c14e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c24003ae0 0x7f8c3c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 --2- 172.21.15.29:0/33153986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c3c0c5850 0x7f8c3c169ad0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.825+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 >> 172.21.15.29:0/33153986 conn(0x7f8c3c067610 msgr2=0x7f8c3c154d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.826+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 shutdown_connections 2023-10-07T09:44:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:13.826+0000 7f8c423ff700 1 -- 172.21.15.29:0/33153986 wait complete. 2023-10-07T09:44:13.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T09:44:14.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.040+0000 7f5f84fff700 1 Processor -- start 2023-10-07T09:44:14.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.040+0000 7f5f84fff700 1 -- start start 2023-10-07T09:44:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f84fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f80159550 0x7f5f80069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f84fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f800c9080 0x7f5f800b0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f84fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f800af190 con 0x7f5f800b0770 2023-10-07T09:44:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f84fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f800aef80 con 0x7f5f800c9080 2023-10-07T09:44:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f84fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f800af470 con 0x7f5f80159550 2023-10-07T09:44:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f7dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f800c9080 0x7f5f800b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f7dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f800c9080 0x7f5f800b0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57460/0 (socket says 172.21.15.29:57460) 2023-10-07T09:44:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f7dd9b700 1 -- 172.21.15.29:0/678593390 learned_addr learned my addr 172.21.15.29:0/678593390 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f7e59c700 1 --2- 172.21.15.29:0/678593390 >> v2:172.21.15.144:3300/0 conn(0x7f5f80159550 0x7f5f80069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f7dd9b700 1 -- 172.21.15.29:0/678593390 >> v2:172.21.15.144:3300/0 conn(0x7f5f80159550 msgr2=0x7f5f80069dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.042+0000 7f5f7dd9b700 1 --2- 172.21.15.29:0/678593390 >> v2:172.21.15.144:3300/0 conn(0x7f5f80159550 0x7f5f80069dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.043+0000 7f5f7dd9b700 1 -- 172.21.15.29:0/678593390 >> v1:172.21.15.144:6789/0 conn(0x7f5f800b0770 legacy=0x7f5f800accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.043+0000 7f5f7dd9b700 1 -- 172.21.15.29:0/678593390 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f800af750 con 0x7f5f800c9080 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.043+0000 7f5f7dd9b700 1 --2- 172.21.15.29:0/678593390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f800c9080 0x7f5f800b0230 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5f800664d0 tx=0x7f5f74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6c97f5bd28eb19 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.043+0000 7f5f7d59a700 1 -- 172.21.15.29:0/678593390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400b860 con 0x7f5f800c9080 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.043+0000 7f5f7d59a700 1 -- 172.21.15.29:0/678593390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7400b9c0 con 0x7f5f800c9080 2023-10-07T09:44:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.043+0000 7f5f7d59a700 1 -- 172.21.15.29:0/678593390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400bd60 con 0x7f5f800c9080 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 -- 172.21.15.29:0/678593390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f800c9080 msgr2=0x7f5f800b0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 --2- 172.21.15.29:0/678593390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f800c9080 0x7f5f800b0230 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5f800664d0 tx=0x7f5f74009250 comp rx=0 tx=0).stop 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 -- 172.21.15.29:0/678593390 shutdown_connections 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 --2- 172.21.15.29:0/678593390 >> v2:172.21.15.144:3300/0 conn(0x7f5f80159550 0x7f5f80069dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 --2- 172.21.15.29:0/678593390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f800c9080 0x7f5f800b0230 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 -- 172.21.15.29:0/678593390 >> 172.21.15.29:0/678593390 conn(0x7f5f80068090 msgr2=0x7f5f800696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 -- 172.21.15.29:0/678593390 shutdown_connections 2023-10-07T09:44:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.044+0000 7f5f84fff700 1 -- 172.21.15.29:0/678593390 wait complete. 2023-10-07T09:44:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.046+0000 7f5f84fff700 1 Processor -- start 2023-10-07T09:44:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.046+0000 7f5f84fff700 1 -- start start 2023-10-07T09:44:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.047+0000 7f5f84fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f800b0770 0x7f5f80154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.047+0000 7f5f84fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f800c9080 0x7f5f800aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.047+0000 7f5f84fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 0x7f5f80156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.047+0000 7f5f84fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f80155a10 con 0x7f5f80159550 2023-10-07T09:44:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.047+0000 7f5f84fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800adcb0 con 0x7f5f800c9080 2023-10-07T09:44:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.047+0000 7f5f84fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f80155c60 con 0x7f5f800b0770 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 0x7f5f80156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 0x7f5f80156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57470/0 (socket says 172.21.15.29:57470) 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 -- 172.21.15.29:0/2866384986 learned_addr learned my addr 172.21.15.29:0/2866384986 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7e59c700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f800b0770 0x7f5f80154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7dd9b700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f800c9080 0x7f5f800aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 -- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f800b0770 msgr2=0x7f5f80154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f800b0770 0x7f5f80154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 -- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f800c9080 msgr2=0x7f5f800aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f800c9080 0x7f5f800aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 -- 172.21.15.29:0/2866384986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740094f0 con 0x7f5f80159550 2023-10-07T09:44:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.048+0000 7f5f7ed9d700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 0x7f5f80156d00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c840 tx=0x7f5f78001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.049+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78012630 con 0x7f5f80159550 2023-10-07T09:44:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.049+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f780127b0 con 0x7f5f80159550 2023-10-07T09:44:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.049+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f80155f40 con 0x7f5f80159550 2023-10-07T09:44:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.049+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78012b70 con 0x7f5f80159550 2023-10-07T09:44:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.049+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f80156580 con 0x7f5f80159550 2023-10-07T09:44:14.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.050+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8004c3c0 con 0x7f5f80159550 2023-10-07T09:44:14.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.051+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5f7801a3a0 con 0x7f5f80159550 2023-10-07T09:44:14.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.051+0000 7f5f6e7fc700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f60066f30 0x7f5f600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.052+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f5f780847b0 con 0x7f5f80159550 2023-10-07T09:44:14.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.052+0000 7f5f7e59c700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f60066f30 0x7f5f600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.053+0000 7f5f7e59c700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f60066f30 0x7f5f600693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006fd0 tx=0x7f5f70008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.056+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f78051900 con 0x7f5f80159550 2023-10-07T09:44:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.272+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5f800ae9f0 con 0x7f5f80159550 2023-10-07T09:44:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.276+0000 7f5f6e7fc700 1 -- 172.21.15.29:0/2866384986 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5f78054f20 con 0x7f5f80159550 2023-10-07T09:44:14.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f60066f30 msgr2=0x7f5f600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f60066f30 0x7f5f600693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006fd0 tx=0x7f5f70008040 comp rx=0 tx=0).stop 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 msgr2=0x7f5f80156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 0x7f5f80156d00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c840 tx=0x7f5f78001670 comp rx=0 tx=0).stop 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 shutdown_connections 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f60066f30 0x7f5f600693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f80159550 0x7f5f80156d00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f800b0770 0x7f5f80154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 --2- 172.21.15.29:0/2866384986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f800c9080 0x7f5f800aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 >> 172.21.15.29:0/2866384986 conn(0x7f5f80068090 msgr2=0x7f5f80069d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 shutdown_connections 2023-10-07T09:44:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.278+0000 7f5f84fff700 1 -- 172.21.15.29:0/2866384986 wait complete. 2023-10-07T09:44:14.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T09:44:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.494+0000 7f6b657ff700 1 Processor -- start 2023-10-07T09:44:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.494+0000 7f6b657ff700 1 -- start start 2023-10-07T09:44:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b657ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b60156180 0x7f6b600afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b657ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b600ae570 0x7f6b600accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b657ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600b0520 0x7f6b600b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b657ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b600b0ea0 con 0x7f6b600ae570 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b657ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b600ad230 con 0x7f6b60156180 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b657ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b600ad4d0 con 0x7f6b600b0520 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b5e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b600ae570 0x7f6b600accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b5e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b600ae570 0x7f6b600accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57486/0 (socket says 172.21.15.29:57486) 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b5e7fc700 1 -- 172.21.15.29:0/4101476737 learned_addr learned my addr 172.21.15.29:0/4101476737 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b5f7fe700 1 --2- 172.21.15.29:0/4101476737 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600b0520 0x7f6b600b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.496+0000 7f6b5effd700 1 --2- 172.21.15.29:0/4101476737 >> v2:172.21.15.144:3300/0 conn(0x7f6b60156180 0x7f6b600afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5e7fc700 1 -- 172.21.15.29:0/4101476737 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600b0520 msgr2=0x7f6b600b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5e7fc700 1 --2- 172.21.15.29:0/4101476737 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600b0520 0x7f6b600b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5e7fc700 1 -- 172.21.15.29:0/4101476737 >> v2:172.21.15.144:3300/0 conn(0x7f6b60156180 msgr2=0x7f6b600afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5e7fc700 1 --2- 172.21.15.29:0/4101476737 >> v2:172.21.15.144:3300/0 conn(0x7f6b60156180 0x7f6b600afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5e7fc700 1 -- 172.21.15.29:0/4101476737 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b600ad7b0 con 0x7f6b600ae570 2023-10-07T09:44:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5e7fc700 1 --2- 172.21.15.29:0/4101476737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b600ae570 0x7f6b600accf0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6b600b2250 tx=0x7f6b54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572180fc54b1de76 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5dffb700 1 -- 172.21.15.29:0/4101476737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400b860 con 0x7f6b600ae570 2023-10-07T09:44:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5dffb700 1 -- 172.21.15.29:0/4101476737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5400b9c0 con 0x7f6b600ae570 2023-10-07T09:44:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.497+0000 7f6b5dffb700 1 -- 172.21.15.29:0/4101476737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400bd60 con 0x7f6b600ae570 2023-10-07T09:44:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 -- 172.21.15.29:0/4101476737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b600ae570 msgr2=0x7f6b600accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 --2- 172.21.15.29:0/4101476737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b600ae570 0x7f6b600accf0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6b600b2250 tx=0x7f6b54009250 comp rx=0 tx=0).stop 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 -- 172.21.15.29:0/4101476737 shutdown_connections 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 --2- 172.21.15.29:0/4101476737 >> v2:172.21.15.144:3300/0 conn(0x7f6b60156180 0x7f6b600afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 --2- 172.21.15.29:0/4101476737 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600b0520 0x7f6b600b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 --2- 172.21.15.29:0/4101476737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b600ae570 0x7f6b600accf0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 -- 172.21.15.29:0/4101476737 >> 172.21.15.29:0/4101476737 conn(0x7f6b60067610 msgr2=0x7f6b60067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 -- 172.21.15.29:0/4101476737 shutdown_connections 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.498+0000 7f6b657ff700 1 -- 172.21.15.29:0/4101476737 wait complete. 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.499+0000 7f6b657ff700 1 Processor -- start 2023-10-07T09:44:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.499+0000 7f6b657ff700 1 -- start start 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.500+0000 7f6b657ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600ae570 0x7f6b600c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.500+0000 7f6b657ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b600b0520 0x7f6b60169a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.500+0000 7f6b657ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 0x7f6b600af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.500+0000 7f6b657ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b600c79d0 con 0x7f6b60156180 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.500+0000 7f6b657ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b600acf60 con 0x7f6b600b0520 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.500+0000 7f6b657ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6016ae10 con 0x7f6b600ae570 2023-10-07T09:44:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 0x7f6b600af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 0x7f6b600af1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57500/0 (socket says 172.21.15.29:57500) 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 -- 172.21.15.29:0/2200440635 learned_addr learned my addr 172.21.15.29:0/2200440635 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5e7fc700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b600b0520 0x7f6b60169a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5effd700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600ae570 0x7f6b600c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 -- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600ae570 msgr2=0x7f6b600c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600ae570 0x7f6b600c5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 -- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b600b0520 msgr2=0x7f6b60169a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b600b0520 0x7f6b60169a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 -- 172.21.15.29:0/2200440635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b540094f0 con 0x7f6b60156180 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b5f7fe700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 0x7f6b600af1d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6b58007f20 tx=0x7f6b58009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.501+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5800eb30 con 0x7f6b60156180 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.502+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5800ecb0 con 0x7f6b60156180 2023-10-07T09:44:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.502+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b60169f60 con 0x7f6b60156180 2023-10-07T09:44:14.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.502+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58014670 con 0x7f6b60156180 2023-10-07T09:44:14.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.502+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6016a4d0 con 0x7f6b60156180 2023-10-07T09:44:14.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.503+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6004c3c0 con 0x7f6b60156180 2023-10-07T09:44:14.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.504+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6b580149e0 con 0x7f6b60156180 2023-10-07T09:44:14.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.504+0000 7f6b46ffd700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6b4c066f30 0x7f6b4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.505+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f6b58084c20 con 0x7f6b60156180 2023-10-07T09:44:14.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.505+0000 7f6b5effd700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6b4c066f30 0x7f6b4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.506+0000 7f6b5effd700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6b4c066f30 0x7f6b4c0693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b50006fd0 tx=0x7f6b50008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.509+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b58051d70 con 0x7f6b60156180 2023-10-07T09:44:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.707+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f6b600ccfb0 con 0x7f6b60156180 2023-10-07T09:44:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.711+0000 7f6b46ffd700 1 -- 172.21.15.29:0/2200440635 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6b58055390 con 0x7f6b60156180 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6b4c066f30 msgr2=0x7f6b4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6b4c066f30 0x7f6b4c0693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b50006fd0 tx=0x7f6b50008040 comp rx=0 tx=0).stop 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 msgr2=0x7f6b600af1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 0x7f6b600af1d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6b58007f20 tx=0x7f6b58009860 comp rx=0 tx=0).stop 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 shutdown_connections 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6b4c066f30 0x7f6b4c0693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b60156180 0x7f6b600af1d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b600ae570 0x7f6b600c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 --2- 172.21.15.29:0/2200440635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b600b0520 0x7f6b60169a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 >> 172.21.15.29:0/2200440635 conn(0x7f6b60067610 msgr2=0x7f6b60159180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 shutdown_connections 2023-10-07T09:44:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.714+0000 7f6b657ff700 1 -- 172.21.15.29:0/2200440635 wait complete. 2023-10-07T09:44:14.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-07T09:44:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.927+0000 7feda4dff700 1 Processor -- start 2023-10-07T09:44:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.927+0000 7feda4dff700 1 -- start start 2023-10-07T09:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7feda4dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feda0157290 0x7feda00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7feda4dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0158ad0 con 0x7feda00b1a10 2023-10-07T09:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7feda4dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda015b140 con 0x7feda00ae8b0 2023-10-07T09:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7feda4dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda015b420 con 0x7feda0157290 2023-10-07T09:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7fed9ed9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feda00b1a10 0x7feda0157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39364/0 (socket says 172.21.15.29:39364) 2023-10-07T09:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2321601958 learned_addr learned my addr 172.21.15.29:0/2321601958 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.929+0000 7fed9e59c700 1 --2- 172.21.15.29:0/2321601958 >> v2:172.21.15.144:3300/0 conn(0x7feda0157290 0x7feda00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593292510 0 0) 0x7feda0158ad0 con 0x7feda00b1a10 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed88003570 con 0x7feda00b1a10 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763045257 0 0) 0x7feda015b140 con 0x7feda00ae8b0 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0158ad0 con 0x7feda00ae8b0 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4047359907 0 0) 0x7fed88003570 con 0x7feda00b1a10 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 >> v2:172.21.15.144:3300/0 conn(0x7feda0157290 msgr2=0x7feda00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 --2- 172.21.15.29:0/2321601958 >> v2:172.21.15.144:3300/0 conn(0x7feda0157290 0x7feda00cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 >> v1:172.21.15.144:6789/0 conn(0x7feda00ae8b0 legacy=0x7feda00b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda015b700 con 0x7feda00b1a10 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fed980017f0 con 0x7feda00b1a10 2023-10-07T09:44:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9e59c700 1 --2- 172.21.15.29:0/2321601958 >> v2:172.21.15.144:3300/0 conn(0x7feda0157290 0x7feda00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 >> v1:172.21.15.29:6789/0 conn(0x7feda00b1a10 legacy=0x7feda0157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88003c40 0x7fed88004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed88004810 0x7fed88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed88005210 0x7fed8800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda015b420 con 0x7fed88005210 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8800c1b0 con 0x7fed88004810 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8800c370 con 0x7fed88003c40 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed88005210 0x7fed8800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9e59c700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88003c40 0x7fed88004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.930+0000 7fed9dd9b700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed88004810 0x7fed88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88003c40 msgr2=0x7fed88004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88003c40 0x7fed88004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed88004810 msgr2=0x7fed88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed88004810 0x7fed88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2321601958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88003570 con 0x7fed88005210 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2321601958 reap_dead start 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed88005210 0x7fed8800ba60 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7feda00cc060 tx=0x7fed980020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d280e3b1cdc94fde server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980070d0 con 0x7fed88005210 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed98007250 con 0x7fed88005210 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.931+0000 7fed9d59a700 1 -- 172.21.15.29:0/2321601958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed98007610 con 0x7fed88005210 2023-10-07T09:44:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 -- 172.21.15.29:0/2321601958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed88005210 msgr2=0x7fed8800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed88005210 0x7fed8800ba60 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7feda00cc060 tx=0x7fed980020c0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 -- 172.21.15.29:0/2321601958 shutdown_connections 2023-10-07T09:44:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 --2- 172.21.15.29:0/2321601958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed88005210 0x7fed8800ba60 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 -- 172.21.15.29:0/2321601958 >> 172.21.15.29:0/2321601958 conn(0x7feda00680a0 msgr2=0x7feda00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 -- 172.21.15.29:0/2321601958 shutdown_connections 2023-10-07T09:44:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.932+0000 7feda4dff700 1 -- 172.21.15.29:0/2321601958 wait complete. 2023-10-07T09:44:14.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.934+0000 7feda4dff700 1 Processor -- start 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.934+0000 7feda4dff700 1 -- start start 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.935+0000 7feda4dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88005210 0x7feda014ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7feda4dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feda014ad10 0x7feda00af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7feda4dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 0x7feda0154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7feda4dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01535d0 con 0x7feda00af630 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7feda4dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda00afb40 con 0x7feda014ad10 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7feda4dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda00afe20 con 0x7fed88005210 2023-10-07T09:44:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 0x7feda0154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9e59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88005210 0x7feda014ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9dd9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feda014ad10 0x7feda00af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 0x7feda0154c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57510/0 (socket says 172.21.15.29:57510) 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9e59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88005210 0x7feda014ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:37748/0 (socket says 172.21.15.29:37748) 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2467738244 learned_addr learned my addr 172.21.15.29:0/2467738244 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88005210 msgr2=0x7feda014ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88005210 0x7feda014ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feda014ad10 msgr2=0x7feda00af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feda014ad10 0x7feda00af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.936+0000 7fed9ed9d700 1 -- 172.21.15.29:0/2467738244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00b0100 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.937+0000 7fed9ed9d700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 0x7feda0154c90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fed98002100 tx=0x7fed98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.937+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980016f0 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.937+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed98001850 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.937+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda01537e0 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.937+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980073c0 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.937+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda014b180 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.938+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda014b370 con 0x7feda00af630 2023-10-07T09:44:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.939+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fed98004bb0 con 0x7feda00af630 2023-10-07T09:44:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.940+0000 7fed8e7fc700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed80066f30 0x7fed800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.940+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fed980853f0 con 0x7feda00af630 2023-10-07T09:44:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.940+0000 7fed9e59c700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed80066f30 0x7fed800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.941+0000 7fed9e59c700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed80066f30 0x7fed800693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fed90006f00 tx=0x7fed900091a0 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:14.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:14.944+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed98052540 con 0x7feda00af630 2023-10-07T09:44:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:15.227+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7feda00b02f0 con 0x7feda00af630 2023-10-07T09:44:16.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.232+0000 7fed8e7fc700 1 -- 172.21.15.29:0/2467738244 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v540) v1 ==== 154+0+0 (secure 0 0 0) 0x7fed98004e60 con 0x7feda00af630 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.235+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed80066f30 msgr2=0x7fed800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.235+0000 7feda4dff700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed80066f30 0x7fed800693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fed90006f00 tx=0x7fed900091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.235+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 msgr2=0x7feda0154c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.235+0000 7feda4dff700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 0x7feda0154c90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fed98002100 tx=0x7fed98016000 comp rx=0 tx=0).stop 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 shutdown_connections 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fed80066f30 0x7fed800693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda00af630 0x7feda0154c90 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed88005210 0x7feda014ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 --2- 172.21.15.29:0/2467738244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feda014ad10 0x7feda00af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 >> 172.21.15.29:0/2467738244 conn(0x7feda00680a0 msgr2=0x7feda01518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:16.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 shutdown_connections 2023-10-07T09:44:16.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.236+0000 7feda4dff700 1 -- 172.21.15.29:0/2467738244 wait complete. 2023-10-07T09:44:16.257 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-07T09:44:16.257 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-07T09:44:16.257 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:44:16.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-07T09:44:16.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.459+0000 7f790ffff700 1 Processor -- start 2023-10-07T09:44:16.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.459+0000 7f790ffff700 1 -- start start 2023-10-07T09:44:16.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.460+0000 7f790ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7908159b20 0x7f79080d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.460+0000 7f790ffff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79080ce8a0 0x7f79080c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.460+0000 7f790ffff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79080cbf20 con 0x7f7908159750 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.460+0000 7f790ffff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79080cc200 con 0x7f7908159b20 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.460+0000 7f790ffff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79080cc4e0 con 0x7f79080ce8a0 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7908159b20 0x7f79080d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7908159b20 0x7f79080d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57516/0 (socket says 172.21.15.29:57516) 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 -- 172.21.15.29:0/1587234261 learned_addr learned my addr 172.21.15.29:0/1587234261 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 -- 172.21.15.29:0/1587234261 >> v2:172.21.15.144:3300/0 conn(0x7f79080ce8a0 msgr2=0x7f79080c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790e59c700 1 --2- 172.21.15.29:0/1587234261 >> v2:172.21.15.144:3300/0 conn(0x7f79080ce8a0 0x7f79080c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 --2- 172.21.15.29:0/1587234261 >> v2:172.21.15.144:3300/0 conn(0x7f79080ce8a0 0x7f79080c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 -- 172.21.15.29:0/1587234261 >> v1:172.21.15.144:6789/0 conn(0x7f7908159750 legacy=0x7f79080ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:16.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 -- 172.21.15.29:0/1587234261 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79080cc7c0 con 0x7f7908159b20 2023-10-07T09:44:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790e59c700 1 --2- 172.21.15.29:0/1587234261 >> v2:172.21.15.144:3300/0 conn(0x7f79080ce8a0 0x7f79080c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.461+0000 7f790d59a700 1 --2- 172.21.15.29:0/1587234261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f79080d10a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f790815e510 tx=0x7f78f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd02d32a14e39f73 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790cd99700 1 -- 172.21.15.29:0/1587234261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f800b820 con 0x7f7908159b20 2023-10-07T09:44:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790cd99700 1 -- 172.21.15.29:0/1587234261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f800b980 con 0x7f7908159b20 2023-10-07T09:44:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790cd99700 1 -- 172.21.15.29:0/1587234261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f800bd20 con 0x7f7908159b20 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790ffff700 1 -- 172.21.15.29:0/1587234261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 msgr2=0x7f79080d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790ffff700 1 --2- 172.21.15.29:0/1587234261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f79080d10a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f790815e510 tx=0x7f78f8009250 comp rx=0 tx=0).stop 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790ffff700 1 -- 172.21.15.29:0/1587234261 shutdown_connections 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790ffff700 1 --2- 172.21.15.29:0/1587234261 >> v2:172.21.15.144:3300/0 conn(0x7f79080ce8a0 0x7f79080c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790ffff700 1 --2- 172.21.15.29:0/1587234261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f79080d10a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.462+0000 7f790ffff700 1 -- 172.21.15.29:0/1587234261 >> 172.21.15.29:0/1587234261 conn(0x7f7908063a10 msgr2=0x7f79080d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.463+0000 7f790ffff700 1 -- 172.21.15.29:0/1587234261 shutdown_connections 2023-10-07T09:44:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.463+0000 7f790ffff700 1 -- 172.21.15.29:0/1587234261 wait complete. 2023-10-07T09:44:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.464+0000 7f790ffff700 1 Processor -- start 2023-10-07T09:44:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.464+0000 7f790ffff700 1 -- start start 2023-10-07T09:44:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79080ce8a0 0x7f79080b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7908159750 0x7f79080b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f790815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79081536b0 con 0x7f7908159b20 2023-10-07T09:44:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790815e700 con 0x7f79080ce8a0 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790815e9e0 con 0x7f7908159750 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f790815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f790815f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57524/0 (socket says 172.21.15.29:57524) 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790e59c700 1 -- 172.21.15.29:0/4016313946 learned_addr learned my addr 172.21.15.29:0/4016313946 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790dd9b700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79080ce8a0 0x7f79080b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.465+0000 7f790d59a700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7908159750 0x7f79080b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790e59c700 1 -- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7908159750 msgr2=0x7f79080b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790e59c700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7908159750 0x7f79080b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790e59c700 1 -- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79080ce8a0 msgr2=0x7f79080b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790e59c700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79080ce8a0 0x7f79080b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790e59c700 1 -- 172.21.15.29:0/4016313946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80094f0 con 0x7f7908159b20 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790d59a700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7908159750 0x7f79080b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790e59c700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f790815f4c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f790000a380 tx=0x7f7900007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900015440 con 0x7f7908159b20 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79000155a0 con 0x7f7908159b20 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790815ecc0 con 0x7f7908159b20 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900015960 con 0x7f7908159b20 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.466+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79080c7670 con 0x7f7908159b20 2023-10-07T09:44:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.467+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790815e840 con 0x7f7908159b20 2023-10-07T09:44:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.468+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f790000d440 con 0x7f7908159b20 2023-10-07T09:44:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.469+0000 7f78fdffb700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f78ec066f30 0x7f78ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.469+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f79000851e0 con 0x7f7908159b20 2023-10-07T09:44:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.470+0000 7f790dd9b700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f78ec066f30 0x7f78ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.470+0000 7f790dd9b700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f78ec066f30 0x7f78ec0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7904003a70 tx=0x7f7904007ef0 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:16.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.473+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f790815e840 con 0x7f7908159b20 2023-10-07T09:44:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:16.755+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f790815e840 con 0x7f7908159b20 2023-10-07T09:44:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.236+0000 7f78fdffb700 1 -- 172.21.15.29:0/4016313946 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v541) v1 ==== 159+0+0 (secure 0 0 0) 0x7f790815e840 con 0x7f7908159b20 2023-10-07T09:44:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f78ec066f30 msgr2=0x7f78ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f78ec066f30 0x7f78ec0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7904003a70 tx=0x7f7904007ef0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 msgr2=0x7f790815f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f790815f4c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f790000a380 tx=0x7f7900007900 comp rx=0 tx=0).stop 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 shutdown_connections 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f78ec066f30 0x7f78ec0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908159b20 0x7f790815f4c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79080ce8a0 0x7f79080b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 --2- 172.21.15.29:0/4016313946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7908159750 0x7f79080b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 >> 172.21.15.29:0/4016313946 conn(0x7f7908063a10 msgr2=0x7f79080c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 shutdown_connections 2023-10-07T09:44:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.239+0000 7f790ffff700 1 -- 172.21.15.29:0/4016313946 wait complete. 2023-10-07T09:44:17.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:44:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.456+0000 7f2d1a1ff700 1 Processor -- start 2023-10-07T09:44:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.457+0000 7f2d1a1ff700 1 -- start start 2023-10-07T09:44:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d1a1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d14153cc0 0x7f2d140b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d1a1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d1415ade0 0x7f2d140b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d1a1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d14154090 con 0x7f2d140b0000 2023-10-07T09:44:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d1a1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d14159f60 con 0x7f2d1415ade0 2023-10-07T09:44:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d1a1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1415a240 con 0x7f2d14153cc0 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d12ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d1415ade0 0x7f2d140b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d12ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d1415ade0 0x7f2d140b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57526/0 (socket says 172.21.15.29:57526) 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d13fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2d140b0000 0x7f2d1415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39376/0 (socket says 172.21.15.29:39376) 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d12ffd700 1 -- 172.21.15.29:0/1765360814 learned_addr learned my addr 172.21.15.29:0/1765360814 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.458+0000 7f2d137fe700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d14153cc0 0x7f2d140b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d12ffd700 1 -- 172.21.15.29:0/1765360814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d14153cc0 msgr2=0x7f2d140b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d12ffd700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d14153cc0 0x7f2d140b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d12ffd700 1 -- 172.21.15.29:0/1765360814 >> v1:172.21.15.29:6789/0 conn(0x7f2d140b0000 legacy=0x7f2d1415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d12ffd700 1 -- 172.21.15.29:0/1765360814 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1415a520 con 0x7f2d1415ade0 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d137fe700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d14153cc0 0x7f2d140b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d127fc700 1 -- 172.21.15.29:0/1765360814 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188241862 0 0) 0x7f2d14154090 con 0x7f2d140b0000 2023-10-07T09:44:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d12ffd700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d1415ade0 0x7f2d140b1d00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2d080099b0 tx=0x7f2d08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b753e25b6b170a30 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d127fc700 1 -- 172.21.15.29:0/1765360814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800b860 con 0x7f2d1415ade0 2023-10-07T09:44:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d127fc700 1 -- 172.21.15.29:0/1765360814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0800b9c0 con 0x7f2d1415ade0 2023-10-07T09:44:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.459+0000 7f2d127fc700 1 -- 172.21.15.29:0/1765360814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800bd60 con 0x7f2d1415ade0 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/1765360814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d1415ade0 msgr2=0x7f2d140b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d1415ade0 0x7f2d140b1d00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2d080099b0 tx=0x7f2d08009250 comp rx=0 tx=0).stop 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/1765360814 shutdown_connections 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d14153cc0 0x7f2d140b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/1765360814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d1415ade0 0x7f2d140b1d00 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/1765360814 >> 172.21.15.29:0/1765360814 conn(0x7f2d14067610 msgr2=0x7f2d14067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/1765360814 shutdown_connections 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.460+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/1765360814 wait complete. 2023-10-07T09:44:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.461+0000 7f2d1a1ff700 1 Processor -- start 2023-10-07T09:44:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.461+0000 7f2d1a1ff700 1 -- start start 2023-10-07T09:44:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.462+0000 7f2d1a1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d140b0000 0x7f2d14141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.462+0000 7f2d1a1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 0x7f2d14155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.462+0000 7f2d1a1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d1415ade0 0x7f2d141560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.462+0000 7f2d1a1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1415c540 con 0x7f2d14153cc0 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.462+0000 7f2d1a1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14157380 con 0x7f2d1415ade0 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.462+0000 7f2d1a1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14156770 con 0x7f2d140b0000 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 0x7f2d14155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 0x7f2d14155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57534/0 (socket says 172.21.15.29:57534) 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 -- 172.21.15.29:0/2451879762 learned_addr learned my addr 172.21.15.29:0/2451879762 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d137fe700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d140b0000 0x7f2d14141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d13fff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d1415ade0 0x7f2d141560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 -- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d140b0000 msgr2=0x7f2d14141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d140b0000 0x7f2d14141d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 -- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d1415ade0 msgr2=0x7f2d141560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d1415ade0 0x7f2d141560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 -- 172.21.15.29:0/2451879762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080094f0 con 0x7f2d14153cc0 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d13fff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d1415ade0 0x7f2d141560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.463+0000 7f2d12ffd700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 0x7f2d14155b90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2d080090b0 tx=0x7f2d08017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.464+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08011e20 con 0x7f2d14153cc0 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.464+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0801f070 con 0x7f2d14153cc0 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.464+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d080266d0 con 0x7f2d14153cc0 2023-10-07T09:44:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.464+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d14156a50 con 0x7f2d14153cc0 2023-10-07T09:44:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.464+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d14157010 con 0x7f2d14153cc0 2023-10-07T09:44:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.465+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d14157510 con 0x7f2d14153cc0 2023-10-07T09:44:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.466+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2d08026ac0 con 0x7f2d14153cc0 2023-10-07T09:44:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.466+0000 7f2cfb7fe700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d00066f60 0x7f2d00069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.466+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f2d08021070 con 0x7f2d14153cc0 2023-10-07T09:44:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.467+0000 7f2d137fe700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d00066f60 0x7f2d00069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.468+0000 7f2d137fe700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d00066f60 0x7f2d00069410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2d04006ea0 tx=0x7f2d04008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:17.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.471+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d08052f30 con 0x7f2d14153cc0 2023-10-07T09:44:17.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.676+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2d1404c3c0 con 0x7f2d14153cc0 2023-10-07T09:44:17.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.680+0000 7f2cfb7fe700 1 -- 172.21.15.29:0/2451879762 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d080150c0 con 0x7f2d14153cc0 2023-10-07T09:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d00066f60 msgr2=0x7f2d00069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d00066f60 0x7f2d00069410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2d04006ea0 tx=0x7f2d04008040 comp rx=0 tx=0).stop 2023-10-07T09:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 msgr2=0x7f2d14155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 0x7f2d14155b90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2d080090b0 tx=0x7f2d08017000 comp rx=0 tx=0).stop 2023-10-07T09:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 shutdown_connections 2023-10-07T09:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d00066f60 0x7f2d00069410 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d1415ade0 0x7f2d141560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d140b0000 0x7f2d14141d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 --2- 172.21.15.29:0/2451879762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d14153cc0 0x7f2d14155b90 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:17.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 >> 172.21.15.29:0/2451879762 conn(0x7f2d14067610 msgr2=0x7f2d140b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:17.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 shutdown_connections 2023-10-07T09:44:17.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:17.682+0000 7f2d1a1ff700 1 -- 172.21.15.29:0/2451879762 wait complete. 2023-10-07T09:44:17.701 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:44:17.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:17.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:17.912 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:44:17.912 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:44:17.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:18.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.107+0000 7fe1a95ff700 1 Processor -- start 2023-10-07T09:44:18.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.107+0000 7fe1a95ff700 1 -- start start 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.108+0000 7fe1a95ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a40ad220 0x7fe1a4159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a95ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1a4068620 0x7fe1a4068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a95ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a415af00 con 0x7fe1a40ad5f0 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a95ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a4068f80 con 0x7fe1a40ad220 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a95ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a40ada40 con 0x7fe1a4068620 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a40ad220 0x7fe1a4159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a40ad220 0x7fe1a4159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57538/0 (socket says 172.21.15.29:57538) 2023-10-07T09:44:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/293321939 learned_addr learned my addr 172.21.15.29:0/293321939 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a37fe700 1 --2- 172.21.15.29:0/293321939 >> v2:172.21.15.144:3300/0 conn(0x7fe1a4068620 0x7fe1a4068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/293321939 >> v2:172.21.15.144:3300/0 conn(0x7fe1a4068620 msgr2=0x7fe1a4068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 --2- 172.21.15.29:0/293321939 >> v2:172.21.15.144:3300/0 conn(0x7fe1a4068620 0x7fe1a4068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/293321939 >> v1:172.21.15.29:6789/0 conn(0x7fe1a40ad5f0 legacy=0x7fe1a40ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/293321939 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a415a250 con 0x7fe1a40ad220 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.109+0000 7fe1a1ffb700 1 -- 172.21.15.29:0/293321939 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038205582 0 0) 0x7fe1a415af00 con 0x7fe1a40ad5f0 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.110+0000 7fe1a2ffd700 1 --2- 172.21.15.29:0/293321939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ad220 0x7fe1a4159c00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4154240 tx=0x7fe194009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e84489ee39cc76 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.110+0000 7fe1a1ffb700 1 -- 172.21.15.29:0/293321939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400b860 con 0x7fe1a40ad220 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.110+0000 7fe1a1ffb700 1 -- 172.21.15.29:0/293321939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19400b9c0 con 0x7fe1a40ad220 2023-10-07T09:44:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.110+0000 7fe1a1ffb700 1 -- 172.21.15.29:0/293321939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400bd60 con 0x7fe1a40ad220 2023-10-07T09:44:18.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.110+0000 7fe1a95ff700 1 -- 172.21.15.29:0/293321939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ad220 msgr2=0x7fe1a4159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.110+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/293321939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ad220 0x7fe1a4159c00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4154240 tx=0x7fe194009250 comp rx=0 tx=0).stop 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.111+0000 7fe1a95ff700 1 -- 172.21.15.29:0/293321939 shutdown_connections 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.111+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/293321939 >> v2:172.21.15.144:3300/0 conn(0x7fe1a4068620 0x7fe1a4068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.111+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/293321939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ad220 0x7fe1a4159c00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.111+0000 7fe1a95ff700 1 -- 172.21.15.29:0/293321939 >> 172.21.15.29:0/293321939 conn(0x7fe1a406dd00 msgr2=0x7fe1a406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.111+0000 7fe1a95ff700 1 -- 172.21.15.29:0/293321939 shutdown_connections 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.111+0000 7fe1a95ff700 1 -- 172.21.15.29:0/293321939 wait complete. 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.112+0000 7fe1a95ff700 1 Processor -- start 2023-10-07T09:44:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.112+0000 7fe1a95ff700 1 -- start start 2023-10-07T09:44:18.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a95ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 0x7fe1a4145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a95ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a40ad220 0x7fe1a40cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a95ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a40ad5f0 0x7fe1a40cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a95ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4152b10 con 0x7fe1a4068620 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a95ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4155ad0 con 0x7fe1a40ad220 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a95ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4154ec0 con 0x7fe1a40ad5f0 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 0x7fe1a4145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 0x7fe1a4145480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57554/0 (socket says 172.21.15.29:57554) 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.113+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/3680263511 learned_addr learned my addr 172.21.15.29:0/3680263511 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a37fe700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a40ad5f0 0x7fe1a40cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a27fc700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a40ad220 0x7fe1a40cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a40ad5f0 msgr2=0x7fe1a40cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a2ffd700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a40ad5f0 0x7fe1a40cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a40ad220 msgr2=0x7fe1a40cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a2ffd700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a40ad220 0x7fe1a40cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a2ffd700 1 -- 172.21.15.29:0/3680263511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940094f0 con 0x7fe1a4068620 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a27fc700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a40ad220 0x7fe1a40cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a2ffd700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 0x7fe1a4145480 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe194006f50 tx=0x7fe194017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194024440 con 0x7fe1a4068620 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1940245e0 con 0x7fe1a4068620 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a41551a0 con 0x7fe1a4068620 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.114+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1940249a0 con 0x7fe1a4068620 2023-10-07T09:44:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.115+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a4155760 con 0x7fe1a4068620 2023-10-07T09:44:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.116+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a404c3c0 con 0x7fe1a4068620 2023-10-07T09:44:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.117+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe194024d50 con 0x7fe1a4068620 2023-10-07T09:44:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.117+0000 7fe18affd700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe190066f30 0x7fe1900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.117+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fe1940862c0 con 0x7fe1a4068620 2023-10-07T09:44:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.118+0000 7fe1a27fc700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe190066f30 0x7fe1900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.118+0000 7fe1a27fc700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe190066f30 0x7fe1900693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe1980011e0 tx=0x7fe198009040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.122+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1940534b0 con 0x7fe1a4068620 2023-10-07T09:44:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.369+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1a415a530 con 0x7fe1a4068620 2023-10-07T09:44:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.370+0000 7fe18affd700 1 -- 172.21.15.29:0/3680263511 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fe194015de0 con 0x7fe1a4068620 2023-10-07T09:44:18.371 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:18.371 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/2316830499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2316830499},{"type":"v1","addr":"172.21.15.144:6833","nonce":2316830499}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5351,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1390257435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1390257435},{"type":"v1","addr":"172.21.15.144:6837","nonce":1390257435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/883244570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":883244570},{"type":"v1","addr":"172.21.15.29:6835","nonce":883244570}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6147,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/864355265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":864355265},{"type":"v1","addr":"172.21.15.29:6834","nonce":864355265}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2023-10-07T09:44:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe190066f30 msgr2=0x7fe1900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe190066f30 0x7fe1900693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe1980011e0 tx=0x7fe198009040 comp rx=0 tx=0).stop 2023-10-07T09:44:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 msgr2=0x7fe1a4145480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 0x7fe1a4145480 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe194006f50 tx=0x7fe194017000 comp rx=0 tx=0).stop 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 shutdown_connections 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe190066f30 0x7fe1900693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a40ad5f0 0x7fe1a40cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a4068620 0x7fe1a4145480 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 --2- 172.21.15.29:0/3680263511 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a40ad220 0x7fe1a40cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 >> 172.21.15.29:0/3680263511 conn(0x7fe1a406dd00 msgr2=0x7fe1a40b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 shutdown_connections 2023-10-07T09:44:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.372+0000 7fe1a95ff700 1 -- 172.21.15.29:0/3680263511 wait complete. 2023-10-07T09:44:18.375 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2023-10-07T09:44:18.391 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:44:18.392 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:44:18.392 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:44:18.392 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:18.392 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:44:18.392 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:44:18.393 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:44:18.393 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:18.393 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:44:18.393 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:44:18.393 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:44:18.393 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:18.393 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:44:18.394 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:44:18.394 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:44:18.394 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:18.404 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:44:18.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:44:18.407 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:44:18.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:44:18.410 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:44:18.410 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:44:18.411 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:44:18.411 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:44:18.451 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:44:18.451 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:18.454 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:44:18.455 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:18.455 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:44:18.455 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:18.458 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:44:18.458 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:18.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:44:18.601 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:44:18.602 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:44:18.602 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:44:18.603 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:44:18.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.667+0000 7fd2e09ff700 1 Processor -- start 2023-10-07T09:44:18.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.667+0000 7fd2e09ff700 1 -- start start 2023-10-07T09:44:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2e09ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2dc157290 0x7fd2dc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2e09ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd2dc0ae8b0 0x7fd2dc0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2e09ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc0b1840 0x7fd2dc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2e09ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2dc0b1cc0 con 0x7fd2dc157290 2023-10-07T09:44:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2e09ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2dc158ad0 con 0x7fd2dc0ae8b0 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2e09ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc15b140 con 0x7fd2dc0b1840 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2da59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2dc157290 0x7fd2dc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2da59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2dc157290 0x7fd2dc0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57586/0 (socket says 172.21.15.29:57586) 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2da59c700 1 -- 172.21.15.29:0/3123076221 learned_addr learned my addr 172.21.15.29:0/3123076221 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2d9d9b700 1 --2- 172.21.15.29:0/3123076221 >> v2:172.21.15.144:3300/0 conn(0x7fd2dc0ae8b0 0x7fd2dc0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.669+0000 7fd2dad9d700 1 --2- 172.21.15.29:0/3123076221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc0b1840 0x7fd2dc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2da59c700 1 -- 172.21.15.29:0/3123076221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc0b1840 msgr2=0x7fd2dc0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2da59c700 1 --2- 172.21.15.29:0/3123076221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc0b1840 0x7fd2dc0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2da59c700 1 -- 172.21.15.29:0/3123076221 >> v2:172.21.15.144:3300/0 conn(0x7fd2dc0ae8b0 msgr2=0x7fd2dc0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2da59c700 1 --2- 172.21.15.29:0/3123076221 >> v2:172.21.15.144:3300/0 conn(0x7fd2dc0ae8b0 0x7fd2dc0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2da59c700 1 -- 172.21.15.29:0/3123076221 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc15b420 con 0x7fd2dc157290 2023-10-07T09:44:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2d9d9b700 1 --2- 172.21.15.29:0/3123076221 >> v2:172.21.15.144:3300/0 conn(0x7fd2dc0ae8b0 0x7fd2dc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:18.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2da59c700 1 --2- 172.21.15.29:0/3123076221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc0cc870 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0c8d90 tx=0x7fd2cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51240c49e6b6c538 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:18.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2d959a700 1 -- 172.21.15.29:0/3123076221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc00b860 con 0x7fd2dc157290 2023-10-07T09:44:18.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2d959a700 1 -- 172.21.15.29:0/3123076221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc00b9c0 con 0x7fd2dc157290 2023-10-07T09:44:18.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.670+0000 7fd2d959a700 1 -- 172.21.15.29:0/3123076221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc00bd60 con 0x7fd2dc157290 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 -- 172.21.15.29:0/3123076221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 msgr2=0x7fd2dc0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/3123076221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc0cc870 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0c8d90 tx=0x7fd2cc009250 comp rx=0 tx=0).stop 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 -- 172.21.15.29:0/3123076221 shutdown_connections 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/3123076221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc0b1840 0x7fd2dc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/3123076221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc0cc870 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/3123076221 >> v2:172.21.15.144:3300/0 conn(0x7fd2dc0ae8b0 0x7fd2dc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 -- 172.21.15.29:0/3123076221 >> 172.21.15.29:0/3123076221 conn(0x7fd2dc0680a0 msgr2=0x7fd2dc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 -- 172.21.15.29:0/3123076221 shutdown_connections 2023-10-07T09:44:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.671+0000 7fd2e09ff700 1 -- 172.21.15.29:0/3123076221 wait complete. 2023-10-07T09:44:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.672+0000 7fd2e09ff700 1 Processor -- start 2023-10-07T09:44:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.673+0000 7fd2e09ff700 1 -- start start 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.673+0000 7fd2e09ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2dc0b1840 0x7fd2dc14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.673+0000 7fd2e09ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.673+0000 7fd2e09ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc153de0 0x7fd2dc0c2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2e09ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc150290 con 0x7fd2dc157290 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2e09ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc154380 con 0x7fd2dc0b1840 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2e09ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc154660 con 0x7fd2dc153de0 2023-10-07T09:44:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57598/0 (socket says 172.21.15.29:57598) 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 -- 172.21.15.29:0/302984032 learned_addr learned my addr 172.21.15.29:0/302984032 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2da59c700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2dc0b1840 0x7fd2dc14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2dad9d700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc153de0 0x7fd2dc0c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 -- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc153de0 msgr2=0x7fd2dc0c2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc153de0 0x7fd2dc0c2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 -- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2dc0b1840 msgr2=0x7fd2dc14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2dc0b1840 0x7fd2dc14fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 -- 172.21.15.29:0/302984032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc0094f0 con 0x7fd2dc157290 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.674+0000 7fd2d9d9b700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc156500 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000e8d0 tx=0x7fd2d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.675+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0013410 con 0x7fd2dc157290 2023-10-07T09:44:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.675+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0013570 con 0x7fd2dc157290 2023-10-07T09:44:18.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.675+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0013910 con 0x7fd2dc157290 2023-10-07T09:44:18.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.675+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2dc154940 con 0x7fd2dc157290 2023-10-07T09:44:18.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.675+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2dc1673a0 con 0x7fd2dc157290 2023-10-07T09:44:18.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.676+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2dc167610 con 0x7fd2dc157290 2023-10-07T09:44:18.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.677+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd2d0013d00 con 0x7fd2dc157290 2023-10-07T09:44:18.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.678+0000 7fd2ca7fc700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2bc06b340 0x7fd2bc06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:18.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.678+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fd2d00843e0 con 0x7fd2dc157290 2023-10-07T09:44:18.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.678+0000 7fd2da59c700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2bc06b340 0x7fd2bc06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:18.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.679+0000 7fd2da59c700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2bc06b340 0x7fd2bc06d7f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0c8d90 tx=0x7fd2cc017000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:18.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.682+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2dc167610 con 0x7fd2dc157290 2023-10-07T09:44:18.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:18.900+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd2dc167610 con 0x7fd2dc157290 2023-10-07T09:44:19.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.250+0000 7fd2ca7fc700 1 -- 172.21.15.29:0/302984032 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v71) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd2dc167610 con 0x7fd2dc157290 2023-10-07T09:44:19.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2bc06b340 msgr2=0x7fd2bc06d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2bc06b340 0x7fd2bc06d7f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0c8d90 tx=0x7fd2cc017000 comp rx=0 tx=0).stop 2023-10-07T09:44:19.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 msgr2=0x7fd2dc156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc156500 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000e8d0 tx=0x7fd2d0008040 comp rx=0 tx=0).stop 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 shutdown_connections 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2bc06b340 0x7fd2bc06d7f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2dc153de0 0x7fd2dc0c2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2dc0b1840 0x7fd2dc14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 --2- 172.21.15.29:0/302984032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2dc157290 0x7fd2dc156500 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 >> 172.21.15.29:0/302984032 conn(0x7fd2dc0680a0 msgr2=0x7fd2dc159500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 shutdown_connections 2023-10-07T09:44:19.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.253+0000 7fd2e09ff700 1 -- 172.21.15.29:0/302984032 wait complete. 2023-10-07T09:44:19.256 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:44:19.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:44:19.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.470+0000 7f01e41ff700 1 Processor -- start 2023-10-07T09:44:19.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.470+0000 7f01e41ff700 1 -- start start 2023-10-07T09:44:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.471+0000 7f01e41ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01dc0ae8b0 0x7f01dc0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.471+0000 7f01e41ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc0b1840 0x7f01dc157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.471+0000 7f01e41ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc158ad0 con 0x7f01dc157290 2023-10-07T09:44:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.471+0000 7f01e41ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01dc0b1cc0 con 0x7f01dc0ae8b0 2023-10-07T09:44:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.471+0000 7f01e41ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc15b140 con 0x7f01dc0b1840 2023-10-07T09:44:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01dc0ae8b0 0x7f01dc0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01dc0ae8b0 0x7f01dc0b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57612/0 (socket says 172.21.15.29:57612) 2023-10-07T09:44:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 -- 172.21.15.29:0/1002469693 learned_addr learned my addr 172.21.15.29:0/1002469693 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e1f9b700 1 --1- 172.21.15.29:0/1002469693 >> v1:172.21.15.29:6789/0 conn(0x7f01dc157290 0x7f01dc0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39414/0 (socket says 172.21.15.29:39414) 2023-10-07T09:44:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e279c700 1 --2- 172.21.15.29:0/1002469693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc0b1840 0x7f01dc157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 -- 172.21.15.29:0/1002469693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc0b1840 msgr2=0x7f01dc157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 --2- 172.21.15.29:0/1002469693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc0b1840 0x7f01dc157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 -- 172.21.15.29:0/1002469693 >> v1:172.21.15.29:6789/0 conn(0x7f01dc157290 legacy=0x7f01dc0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 -- 172.21.15.29:0/1002469693 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc15b420 con 0x7f01dc0ae8b0 2023-10-07T09:44:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.472+0000 7f01e179a700 1 --2- 172.21.15.29:0/1002469693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc0ae8b0 0x7f01dc0b1300 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0099b0 tx=0x7f01cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e116836f98e133 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e0f99700 1 -- 172.21.15.29:0/1002469693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc00b860 con 0x7f01dc0ae8b0 2023-10-07T09:44:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e0f99700 1 -- 172.21.15.29:0/1002469693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc00b9c0 con 0x7f01dc0ae8b0 2023-10-07T09:44:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e0f99700 1 -- 172.21.15.29:0/1002469693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc00bd60 con 0x7f01dc0ae8b0 2023-10-07T09:44:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 -- 172.21.15.29:0/1002469693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc0ae8b0 msgr2=0x7f01dc0b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 --2- 172.21.15.29:0/1002469693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc0ae8b0 0x7f01dc0b1300 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0099b0 tx=0x7f01cc009250 comp rx=0 tx=0).stop 2023-10-07T09:44:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 -- 172.21.15.29:0/1002469693 shutdown_connections 2023-10-07T09:44:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 --2- 172.21.15.29:0/1002469693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc0b1840 0x7f01dc157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 --2- 172.21.15.29:0/1002469693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc0ae8b0 0x7f01dc0b1300 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 -- 172.21.15.29:0/1002469693 >> 172.21.15.29:0/1002469693 conn(0x7f01dc0680a0 msgr2=0x7f01dc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 -- 172.21.15.29:0/1002469693 shutdown_connections 2023-10-07T09:44:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.473+0000 7f01e41ff700 1 -- 172.21.15.29:0/1002469693 wait complete. 2023-10-07T09:44:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.475+0000 7f01e41ff700 1 Processor -- start 2023-10-07T09:44:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.475+0000 7f01e41ff700 1 -- start start 2023-10-07T09:44:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e41ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01dc0b1840 0x7f01dc1506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e41ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc157290 0x7f01dc0c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e41ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 0x7f01dc16d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e41ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc0af270 con 0x7f01dc150bf0 2023-10-07T09:44:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e41ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc0af550 con 0x7f01dc0b1840 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e41ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc0af830 con 0x7f01dc157290 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 0x7f01dc16d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 0x7f01dc16d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57620/0 (socket says 172.21.15.29:57620) 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e279c700 1 -- 172.21.15.29:0/3632621138 learned_addr learned my addr 172.21.15.29:0/3632621138 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.476+0000 7f01e179a700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc157290 0x7f01dc0c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e1f9b700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01dc0b1840 0x7f01dc1506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e279c700 1 -- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc157290 msgr2=0x7f01dc0c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e279c700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc157290 0x7f01dc0c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e279c700 1 -- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01dc0b1840 msgr2=0x7f01dc1506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e279c700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01dc0b1840 0x7f01dc1506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e279c700 1 -- 172.21.15.29:0/3632621138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc0094f0 con 0x7f01dc150bf0 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e1f9b700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01dc0b1840 0x7f01dc1506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e279c700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 0x7f01dc16d5e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f01d4009fa0 tx=0x7f01d4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d400ec70 con 0x7f01dc150bf0 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d400edd0 con 0x7f01dc150bf0 2023-10-07T09:44:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4014670 con 0x7f01dc150bf0 2023-10-07T09:44:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.477+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01dc0afb10 con 0x7f01dc150bf0 2023-10-07T09:44:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.478+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01dc0c7f70 con 0x7f01dc150bf0 2023-10-07T09:44:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.479+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01dc0c8190 con 0x7f01dc150bf0 2023-10-07T09:44:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.479+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f01d40149e0 con 0x7f01dc150bf0 2023-10-07T09:44:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.480+0000 7f01d1ffb700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f01c0066f30 0x7f01c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.480+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f01d4084b30 con 0x7f01dc150bf0 2023-10-07T09:44:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.481+0000 7f01e1f9b700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f01c0066f30 0x7f01c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.482+0000 7f01e1f9b700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f01c0066f30 0x7f01c00693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f01d8007fd0 tx=0x7f01d8007e10 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.484+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01d4051ee0 con 0x7f01dc150bf0 2023-10-07T09:44:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.770+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f01dc0af3b0 con 0x7f01dc150bf0 2023-10-07T09:44:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.771+0000 7f01d1ffb700 1 -- 172.21.15.29:0/3632621138 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f01dc0af3b0 con 0x7f01dc150bf0 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.772 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.773 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:19.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:19.776 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.777 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:44:19.777 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:44:19.777 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.778 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.778 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:44:19.778 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:44:19.779 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.779 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.779 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.779 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:44:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:44:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.782 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.782 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:44:19.783 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:44:19.783 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.783 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.783 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.783 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:44:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:44:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:44:19.788 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.789 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:19.790 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:19.791 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:19.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:44:19.793 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f01c0066f30 msgr2=0x7f01c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f01c0066f30 0x7f01c00693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f01d8007fd0 tx=0x7f01d8007e10 comp rx=0 tx=0).stop 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 msgr2=0x7f01dc16d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 0x7f01dc16d5e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f01d4009fa0 tx=0x7f01d4009900 comp rx=0 tx=0).stop 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 shutdown_connections 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f01c0066f30 0x7f01c00693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01dc150bf0 0x7f01dc16d5e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01dc0b1840 0x7f01dc1506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 --2- 172.21.15.29:0/3632621138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01dc157290 0x7f01dc0c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 >> 172.21.15.29:0/3632621138 conn(0x7f01dc0680a0 msgr2=0x7f01dc158c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 shutdown_connections 2023-10-07T09:44:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.773+0000 7f01e41ff700 1 -- 172.21.15.29:0/3632621138 wait complete. 2023-10-07T09:44:19.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.990+0000 7fb08e7ff700 1 Processor -- start 2023-10-07T09:44:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.991+0000 7fb08e7ff700 1 -- start start 2023-10-07T09:44:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb08e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb0880cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb08e7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb0880b1a10 0x7fb088157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb08e7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088158ad0 con 0x7fb0880ae8b0 2023-10-07T09:44:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb08e7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb08815b140 con 0x7fb0880b1a10 2023-10-07T09:44:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb08e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08815b420 con 0x7fb088157290 2023-10-07T09:44:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb0877fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb0880ae8b0 0x7fb0880b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39426/0 (socket says 172.21.15.29:39426) 2023-10-07T09:44:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb0877fe700 1 -- 172.21.15.29:0/814729557 learned_addr learned my addr 172.21.15.29:0/814729557 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb087fff700 1 --2- 172.21.15.29:0/814729557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb0880cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.992+0000 7fb08cd9c700 1 --2- 172.21.15.29:0/814729557 >> v2:172.21.15.144:3300/0 conn(0x7fb0880b1a10 0x7fb088157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb086ffd700 1 -- 172.21.15.29:0/814729557 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126363973 0 0) 0x7fb088158ad0 con 0x7fb0880ae8b0 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb086ffd700 1 -- 172.21.15.29:0/814729557 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb070003570 con 0x7fb0880ae8b0 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb086ffd700 1 -- 172.21.15.29:0/814729557 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2404887873 0 0) 0x7fb070003570 con 0x7fb0880ae8b0 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb087fff700 1 -- 172.21.15.29:0/814729557 >> v2:172.21.15.144:3300/0 conn(0x7fb0880b1a10 msgr2=0x7fb088157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb087fff700 1 --2- 172.21.15.29:0/814729557 >> v2:172.21.15.144:3300/0 conn(0x7fb0880b1a10 0x7fb088157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb087fff700 1 -- 172.21.15.29:0/814729557 >> v1:172.21.15.29:6789/0 conn(0x7fb0880ae8b0 legacy=0x7fb0880b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb087fff700 1 -- 172.21.15.29:0/814729557 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08815b700 con 0x7fb088157290 2023-10-07T09:44:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.993+0000 7fb087fff700 1 --2- 172.21.15.29:0/814729557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb0880cc870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0780098e0 tx=0x7fb0780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405ade709a1926a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb086ffd700 1 -- 172.21.15.29:0/814729557 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c001470 con 0x7fb088157290 2023-10-07T09:44:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb086ffd700 1 -- 172.21.15.29:0/814729557 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb078011420 con 0x7fb088157290 2023-10-07T09:44:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb086ffd700 1 -- 172.21.15.29:0/814729557 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0780117a0 con 0x7fb088157290 2023-10-07T09:44:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 -- 172.21.15.29:0/814729557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 msgr2=0x7fb0880cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/814729557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb0880cc870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0780098e0 tx=0x7fb0780091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 -- 172.21.15.29:0/814729557 shutdown_connections 2023-10-07T09:44:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/814729557 >> v2:172.21.15.144:3300/0 conn(0x7fb0880b1a10 0x7fb088157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/814729557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb0880cc870 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 -- 172.21.15.29:0/814729557 >> 172.21.15.29:0/814729557 conn(0x7fb0880680a0 msgr2=0x7fb0880cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 -- 172.21.15.29:0/814729557 shutdown_connections 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.994+0000 7fb08e7ff700 1 -- 172.21.15.29:0/814729557 wait complete. 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.996+0000 7fb08e7ff700 1 Processor -- start 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.997+0000 7fb08e7ff700 1 -- start start 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08e7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 0x7fb08806efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb08814a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08e7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0881676c0 0x7fb088167b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08e7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb088155730 con 0x7fb0880b1a10 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08e7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08806f5a0 con 0x7fb0881676c0 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08806f880 con 0x7fb088157290 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb087fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 0x7fb08806efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb087fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 0x7fb08806efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57630/0 (socket says 172.21.15.29:57630) 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb087fff700 1 -- 172.21.15.29:0/4000221887 learned_addr learned my addr 172.21.15.29:0/4000221887 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb0877fe700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb08814a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.998+0000 7fb08cd9c700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0881676c0 0x7fb088167b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb087fff700 1 -- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 msgr2=0x7fb08814a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb087fff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb08814a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb087fff700 1 -- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0881676c0 msgr2=0x7fb088167b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb087fff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0881676c0 0x7fb088167b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb087fff700 1 -- 172.21.15.29:0/4000221887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078009420 con 0x7fb0880b1a10 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb08cd9c700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0881676c0 0x7fb088167b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb087fff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 0x7fb08806efc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb088070f30 tx=0x7fb078016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07801caf0 con 0x7fb0880b1a10 2023-10-07T09:44:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07801cc70 con 0x7fb0880b1a10 2023-10-07T09:44:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07801a450 con 0x7fb0880b1a10 2023-10-07T09:44:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08815c420 con 0x7fb0880b1a10 2023-10-07T09:44:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:19.999+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08815c9e0 con 0x7fb0880b1a10 2023-10-07T09:44:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.000+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08804c3c0 con 0x7fb0880b1a10 2023-10-07T09:44:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.001+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb07801a7e0 con 0x7fb0880b1a10 2023-10-07T09:44:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.002+0000 7fb06ffff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb074066f30 0x7fb0740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.002+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fb078085c10 con 0x7fb0880b1a10 2023-10-07T09:44:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.003+0000 7fb0877fe700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb074066f30 0x7fb0740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.003+0000 7fb0877fe700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb074066f30 0x7fb0740693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0880b1670 tx=0x7fb07c001a10 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.006+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb078053070 con 0x7fb0880b1a10 2023-10-07T09:44:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.209+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb08815b9e0 con 0x7fb0880b1a10 2023-10-07T09:44:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.210+0000 7fb06ffff700 1 -- 172.21.15.29:0/4000221887 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+12801 (secure 0 0 0) 0x7fb078029090 con 0x7fb0880b1a10 2023-10-07T09:44:20.211 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:20.211 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":73,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:19.255948+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb074066f30 msgr2=0x7fb0740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb074066f30 0x7fb0740693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0880b1670 tx=0x7fb07c001a10 comp rx=0 tx=0).stop 2023-10-07T09:44:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 msgr2=0x7fb08806efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 0x7fb08806efc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb088070f30 tx=0x7fb078016000 comp rx=0 tx=0).stop 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 shutdown_connections 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb074066f30 0x7fb0740693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0881676c0 0x7fb088167b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0880b1a10 0x7fb08806efc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 --2- 172.21.15.29:0/4000221887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb088157290 0x7fb08814a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 >> 172.21.15.29:0/4000221887 conn(0x7fb0880680a0 msgr2=0x7fb088151040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 shutdown_connections 2023-10-07T09:44:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.212+0000 7fb08e7ff700 1 -- 172.21.15.29:0/4000221887 wait complete. 2023-10-07T09:44:20.231 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:44:20.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:44:20.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.430+0000 7fd9acfff700 1 Processor -- start 2023-10-07T09:44:20.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.430+0000 7fd9acfff700 1 -- start start 2023-10-07T09:44:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.431+0000 7fd9acfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9a80b0710 0x7fd9a815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.431+0000 7fd9acfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a8154090 con 0x7fd9a8153cc0 2023-10-07T09:44:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.431+0000 7fd9acfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a8159f60 con 0x7fd9a815ade0 2023-10-07T09:44:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.431+0000 7fd9acfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a815a240 con 0x7fd9a80b0710 2023-10-07T09:44:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9a8153cc0 0x7fd9a80b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39442/0 (socket says 172.21.15.29:39442) 2023-10-07T09:44:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a659c700 1 -- 172.21.15.29:0/1784823457 learned_addr learned my addr 172.21.15.29:0/1784823457 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:20.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a6d9d700 1 --2- 172.21.15.29:0/1784823457 >> v2:172.21.15.144:3300/0 conn(0x7fd9a80b0710 0x7fd9a815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063407521 0 0) 0x7fd9a8154090 con 0x7fd9a8153cc0 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990003570 con 0x7fd9a8153cc0 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 652833083 0 0) 0x7fd990003570 con 0x7fd9a8153cc0 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 >> v2:172.21.15.144:3300/0 conn(0x7fd9a80b0710 msgr2=0x7fd9a815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a559a700 1 --2- 172.21.15.29:0/1784823457 >> v2:172.21.15.144:3300/0 conn(0x7fd9a80b0710 0x7fd9a815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.432+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 >> v1:172.21.15.144:6789/0 conn(0x7fd9a815ade0 legacy=0x7fd9a80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a815a520 con 0x7fd9a8153cc0 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fd9980015f0 con 0x7fd9a8153cc0 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 >> v1:172.21.15.29:6789/0 conn(0x7fd9a8153cc0 legacy=0x7fd9a80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990003a60 0x7fd990003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd990004810 0x7fd990004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9900051c0 0x7fd99000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a815a240 con 0x7fd990004810 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000c0d0 con 0x7fd9900051c0 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000c290 con 0x7fd990003a60 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a6d9d700 1 --2- 172.21.15.29:0/1784823457 >> v2:172.21.15.144:3300/0 conn(0x7fd9a80b0710 0x7fd9a815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd990004810 0x7fd990004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a659c700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990003a60 0x7fd990003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a6d9d700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9900051c0 0x7fd99000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990003a60 msgr2=0x7fd990003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990003a60 0x7fd990003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9900051c0 msgr2=0x7fd99000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9900051c0 0x7fd99000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1784823457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd990003570 con 0x7fd990004810 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a6d9d700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9900051c0 0x7fd99000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.433+0000 7fd9a6d9d700 1 -- 172.21.15.29:0/1784823457 reap_dead start 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.434+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd990004810 0x7fd990004c80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd9a815b200 tx=0x7fd99c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb3fe39e522f2c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.434+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c012420 con 0x7fd990004810 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.434+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c0125a0 con 0x7fd990004810 2023-10-07T09:44:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.434+0000 7fd9a559a700 1 -- 172.21.15.29:0/1784823457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c012960 con 0x7fd990004810 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.434+0000 7fd9acfff700 1 -- 172.21.15.29:0/1784823457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd990004810 msgr2=0x7fd990004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.434+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd990004810 0x7fd990004c80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd9a815b200 tx=0x7fd99c007d30 comp rx=0 tx=0).stop 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.435+0000 7fd9acfff700 1 -- 172.21.15.29:0/1784823457 shutdown_connections 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.435+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1784823457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd990004810 0x7fd990004c80 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.435+0000 7fd9acfff700 1 -- 172.21.15.29:0/1784823457 >> 172.21.15.29:0/1784823457 conn(0x7fd9a8067610 msgr2=0x7fd9a8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.435+0000 7fd9acfff700 1 -- 172.21.15.29:0/1784823457 shutdown_connections 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.435+0000 7fd9acfff700 1 -- 172.21.15.29:0/1784823457 wait complete. 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.436+0000 7fd9acfff700 1 Processor -- start 2023-10-07T09:44:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.436+0000 7fd9acfff700 1 -- start start 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.437+0000 7fd9acfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990004810 0x7fd9a813e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.437+0000 7fd9acfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 0x7fd9a81577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.437+0000 7fd9acfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9a8157d30 0x7fd9a80b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.437+0000 7fd9acfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a81512e0 con 0x7fd9a8159360 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.437+0000 7fd9acfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a8158ed0 con 0x7fd9a8157d30 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.437+0000 7fd9acfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a81582c0 con 0x7fd990004810 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 0x7fd9a81577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 0x7fd9a81577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57636/0 (socket says 172.21.15.29:57636) 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1142746304 learned_addr learned my addr 172.21.15.29:0/1142746304 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a659c700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990004810 0x7fd9a813e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a6d9d700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9a8157d30 0x7fd9a80b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990004810 msgr2=0x7fd9a813e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990004810 0x7fd9a813e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9a8157d30 msgr2=0x7fd9a80b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9a8157d30 0x7fd9a80b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 -- 172.21.15.29:0/1142746304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a81585a0 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.438+0000 7fd9a5d9b700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 0x7fd9a81577f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd99000c0a0 tx=0x7fd99c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.439+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c012420 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.439+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c012cd0 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.439+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a8158880 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.439+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c012940 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.439+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a8156680 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.440+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a8158d20 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.441+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd99c019070 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.442+0000 7fd9967fc700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd988066f30 0x7fd9880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.442+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fd99c085860 con 0x7fd9a8159360 2023-10-07T09:44:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.442+0000 7fd9a659c700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd988066f30 0x7fd9880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.443+0000 7fd9a659c700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd988066f30 0x7fd9880693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd998000c00 tx=0x7fd998009040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.446+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd99c052d30 con 0x7fd9a8159360 2023-10-07T09:44:20.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.642+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9a806c630 con 0x7fd9a8159360 2023-10-07T09:44:20.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.643+0000 7fd9967fc700 1 -- 172.21.15.29:0/1142746304 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd99c022020 con 0x7fd9a8159360 2023-10-07T09:44:20.645 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:44:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd988066f30 msgr2=0x7fd9880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd988066f30 0x7fd9880693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd998000c00 tx=0x7fd998009040 comp rx=0 tx=0).stop 2023-10-07T09:44:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 msgr2=0x7fd9a81577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 0x7fd9a81577f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd99000c0a0 tx=0x7fd99c015000 comp rx=0 tx=0).stop 2023-10-07T09:44:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 shutdown_connections 2023-10-07T09:44:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd988066f30 0x7fd9880693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9a8157d30 0x7fd9a80b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd990004810 0x7fd9a813e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 --2- 172.21.15.29:0/1142746304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9a8159360 0x7fd9a81577f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 >> 172.21.15.29:0/1142746304 conn(0x7fd9a8067610 msgr2=0x7fd9a80af360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:20.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 shutdown_connections 2023-10-07T09:44:20.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.645+0000 7fd9acfff700 1 -- 172.21.15.29:0/1142746304 wait complete. 2023-10-07T09:44:20.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:44:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.858+0000 7f5102bff700 1 Processor -- start 2023-10-07T09:44:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.858+0000 7f5102bff700 1 -- start start 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.859+0000 7f5102bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50fc159750 0x7f50fc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.859+0000 7f5102bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50fc159b20 0x7f50fc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.859+0000 7f5102bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc0ce6d0 0x7f50fc0ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.859+0000 7f5102bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50fc0cbf20 con 0x7f50fc159750 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.859+0000 7f5102bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50fc0cc200 con 0x7f50fc159b20 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.859+0000 7f5102bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0cc4e0 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50fc159750 0x7f50fc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50fc159750 0x7f50fc0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57644/0 (socket says 172.21.15.29:57644) 2023-10-07T09:44:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 -- 172.21.15.29:0/1547582882 learned_addr learned my addr 172.21.15.29:0/1547582882 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 -- 172.21.15.29:0/1547582882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc0ce6d0 msgr2=0x7f50fc0ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510119c700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc0ce6d0 0x7f50fc0ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f50fbfff700 1 --2- 172.21.15.29:0/1547582882 >> v2:172.21.15.144:3300/0 conn(0x7f50fc159b20 0x7f50fc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc0ce6d0 0x7f50fc0ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 -- 172.21.15.29:0/1547582882 >> v2:172.21.15.144:3300/0 conn(0x7f50fc159b20 msgr2=0x7f50fc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 --2- 172.21.15.29:0/1547582882 >> v2:172.21.15.144:3300/0 conn(0x7f50fc159b20 0x7f50fc0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 -- 172.21.15.29:0/1547582882 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc0cc7c0 con 0x7f50fc159750 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510119c700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc0ce6d0 0x7f50fc0ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f50fbfff700 1 --2- 172.21.15.29:0/1547582882 >> v2:172.21.15.144:3300/0 conn(0x7f50fc159b20 0x7f50fc0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.860+0000 7f510099b700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc159750 0x7f50fc0d10a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f50fc088a70 tx=0x7f50f0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7953e62501f783 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f50fb7fe700 1 -- 172.21.15.29:0/1547582882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0005c10 con 0x7f50fc159750 2023-10-07T09:44:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f50fb7fe700 1 -- 172.21.15.29:0/1547582882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f0005d70 con 0x7f50fc159750 2023-10-07T09:44:20.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f50fb7fe700 1 -- 172.21.15.29:0/1547582882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f000b880 con 0x7f50fc159750 2023-10-07T09:44:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 -- 172.21.15.29:0/1547582882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc159750 msgr2=0x7f50fc0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc159750 0x7f50fc0d10a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f50fc088a70 tx=0x7f50f0009220 comp rx=0 tx=0).stop 2023-10-07T09:44:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 -- 172.21.15.29:0/1547582882 shutdown_connections 2023-10-07T09:44:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc0ce6d0 0x7f50fc0ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 --2- 172.21.15.29:0/1547582882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc159750 0x7f50fc0d10a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 --2- 172.21.15.29:0/1547582882 >> v2:172.21.15.144:3300/0 conn(0x7f50fc159b20 0x7f50fc0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 -- 172.21.15.29:0/1547582882 >> 172.21.15.29:0/1547582882 conn(0x7f50fc063a10 msgr2=0x7f50fc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 -- 172.21.15.29:0/1547582882 shutdown_connections 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.861+0000 7f5102bff700 1 -- 172.21.15.29:0/1547582882 wait complete. 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.863+0000 7f5102bff700 1 Processor -- start 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.863+0000 7f5102bff700 1 -- start start 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f5102bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 0x7f50fc147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f5102bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc159750 0x7f50fc0c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f5102bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50fc159b20 0x7f50fc0c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f5102bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc15c8f0 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f5102bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc15bfc0 con 0x7f50fc159b20 2023-10-07T09:44:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f5102bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc15c2a0 con 0x7f50fc159750 2023-10-07T09:44:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f510099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 0x7f50fc147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f510099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 0x7f50fc147f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57650/0 (socket says 172.21.15.29:57650) 2023-10-07T09:44:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f510099b700 1 -- 172.21.15.29:0/3008812407 learned_addr learned my addr 172.21.15.29:0/3008812407 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f50fbfff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc159750 0x7f50fc0c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.864+0000 7f510119c700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50fc159b20 0x7f50fc0c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f510099b700 1 -- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc159750 msgr2=0x7f50fc0c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f510099b700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc159750 0x7f50fc0c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f510099b700 1 -- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50fc159b20 msgr2=0x7f50fc0c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f510099b700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50fc159b20 0x7f50fc0c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f510099b700 1 -- 172.21.15.29:0/3008812407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f0009480 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f510099b700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 0x7f50fc147f30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b3c0 tx=0x7f50f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0011740 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f0020070 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0011ad0 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.865+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50fc15c580 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.866+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc148470 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.866+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50fc15cbd0 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.867+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f50f0022030 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.868+0000 7f50f8ff9700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50e0066f30 0x7f50e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:20.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.868+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f50f0085ed0 con 0x7f50fc0ce6d0 2023-10-07T09:44:20.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.869+0000 7f50fbfff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50e0066f30 0x7f50e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:20.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.869+0000 7f50fbfff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50e0066f30 0x7f50e00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50fc069910 tx=0x7f50e8008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:20.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:20.873+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50f0053280 con 0x7f50fc0ce6d0 2023-10-07T09:44:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.076+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f50fc04c3c0 con 0x7f50fc0ce6d0 2023-10-07T09:44:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.076+0000 7f50f8ff9700 1 -- 172.21.15.29:0/3008812407 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) v1 ==== 84+0+65 (secure 0 0 0) 0x7f50f00568a0 con 0x7f50fc0ce6d0 2023-10-07T09:44:21.077 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:21.078 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:44:21.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:21.078 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:44:21.078 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:44:21.078 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:21.078 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50e0066f30 msgr2=0x7f50e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50e0066f30 0x7f50e00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50fc069910 tx=0x7f50e8008040 comp rx=0 tx=0).stop 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 msgr2=0x7f50fc147f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 0x7f50fc147f30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b3c0 tx=0x7f50f0016000 comp rx=0 tx=0).stop 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 shutdown_connections 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50e0066f30 0x7f50e00693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50fc159b20 0x7f50fc0c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50fc0ce6d0 0x7f50fc147f30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 --2- 172.21.15.29:0/3008812407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50fc159750 0x7f50fc0c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 >> 172.21.15.29:0/3008812407 conn(0x7f50fc063a10 msgr2=0x7f50fc0ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 shutdown_connections 2023-10-07T09:44:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.079+0000 7f5102bff700 1 -- 172.21.15.29:0/3008812407 wait complete. 2023-10-07T09:44:21.097 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:44:21.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:44:21.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.297+0000 7fa1785ff700 1 Processor -- start 2023-10-07T09:44:21.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.297+0000 7fa1785ff700 1 -- start start 2023-10-07T09:44:21.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa1785ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700cc810 0x7fa1700accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:21.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa1785ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa17015a590 0x7fa17015a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:21.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa1785ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700ad400 con 0x7fa1700afeb0 2023-10-07T09:44:21.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa1785ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17015a140 con 0x7fa17015a590 2023-10-07T09:44:21.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa1785ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700adca0 con 0x7fa1700cc810 2023-10-07T09:44:21.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa17639b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa1700afeb0 0x7fa1700aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39454/0 (socket says 172.21.15.29:39454) 2023-10-07T09:44:21.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa17639b700 1 -- 172.21.15.29:0/2381644278 learned_addr learned my addr 172.21.15.29:0/2381644278 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:21.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa176b9c700 1 --2- 172.21.15.29:0/2381644278 >> v2:172.21.15.29:3300/0 conn(0x7fa17015a590 0x7fa17015a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:21.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa175b9a700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700cc810 0x7fa1700accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:21.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.299+0000 7fa175399700 1 -- 172.21.15.29:0/2381644278 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714029099 0 0) 0x7fa1700ad400 con 0x7fa1700afeb0 2023-10-07T09:44:21.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa175399700 1 -- 172.21.15.29:0/2381644278 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c003570 con 0x7fa1700afeb0 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa176b9c700 1 -- 172.21.15.29:0/2381644278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700cc810 msgr2=0x7fa1700accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa176b9c700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700cc810 0x7fa1700accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa176b9c700 1 -- 172.21.15.29:0/2381644278 >> v1:172.21.15.29:6789/0 conn(0x7fa1700afeb0 legacy=0x7fa1700aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa176b9c700 1 -- 172.21.15.29:0/2381644278 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1700ad6e0 con 0x7fa17015a590 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa175b9a700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700cc810 0x7fa1700accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa176b9c700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17015a590 0x7fa17015a9f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa168009e60 tx=0x7fa168009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7690d76c64492b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa175399700 1 -- 172.21.15.29:0/2381644278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800e070 con 0x7fa17015a590 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa175399700 1 -- 172.21.15.29:0/2381644278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16800bb60 con 0x7fa17015a590 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.300+0000 7fa175399700 1 -- 172.21.15.29:0/2381644278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168012430 con 0x7fa17015a590 2023-10-07T09:44:21.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 -- 172.21.15.29:0/2381644278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17015a590 msgr2=0x7fa17015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17015a590 0x7fa17015a9f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa168009e60 tx=0x7fa168009780 comp rx=0 tx=0).stop 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 -- 172.21.15.29:0/2381644278 shutdown_connections 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17015a590 0x7fa17015a9f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 --2- 172.21.15.29:0/2381644278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700cc810 0x7fa1700accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 -- 172.21.15.29:0/2381644278 >> 172.21.15.29:0/2381644278 conn(0x7fa170067610 msgr2=0x7fa170067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 -- 172.21.15.29:0/2381644278 shutdown_connections 2023-10-07T09:44:21.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.301+0000 7fa1785ff700 1 -- 172.21.15.29:0/2381644278 wait complete. 2023-10-07T09:44:21.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.303+0000 7fa1785ff700 1 Processor -- start 2023-10-07T09:44:21.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.303+0000 7fa1785ff700 1 -- start start 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.304+0000 7fa1785ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700afeb0 0x7fa17014f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.304+0000 7fa1785ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 0x7fa1700ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.304+0000 7fa1785ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa17015a590 0x7fa1700ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa1785ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700c9080 con 0x7fa1700cc810 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa1785ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700c92d0 con 0x7fa17015a590 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa1785ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700c95b0 con 0x7fa1700afeb0 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 0x7fa1700ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 0x7fa1700ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57676/0 (socket says 172.21.15.29:57676) 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 -- 172.21.15.29:0/3214894245 learned_addr learned my addr 172.21.15.29:0/3214894245 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa17639b700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700afeb0 0x7fa17014f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa176b9c700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa17015a590 0x7fa1700ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 -- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700afeb0 msgr2=0x7fa17014f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700afeb0 0x7fa17014f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 -- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa17015a590 msgr2=0x7fa1700ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa17015a590 0x7fa1700ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 -- 172.21.15.29:0/3214894245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168009a20 con 0x7fa1700cc810 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.305+0000 7fa175b9a700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 0x7fa1700ac230 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa16000e8d0 tx=0x7fa160008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.306+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000ee70 con 0x7fa1700cc810 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.306+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16000c5e0 con 0x7fa1700cc810 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.306+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000c980 con 0x7fa1700cc810 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.306+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1700c9890 con 0x7fa1700cc810 2023-10-07T09:44:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.306+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17014fe50 con 0x7fa1700cc810 2023-10-07T09:44:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.307+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1700ad180 con 0x7fa1700cc810 2023-10-07T09:44:21.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.308+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa160013410 con 0x7fa1700cc810 2023-10-07T09:44:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.309+0000 7fa1667fc700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa154066f30 0x7fa1540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.309+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fa1600841e0 con 0x7fa1700cc810 2023-10-07T09:44:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.310+0000 7fa17639b700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa154066f30 0x7fa1540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:21.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.310+0000 7fa17639b700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa154066f30 0x7fa1540693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0058d0 tx=0x7fa16c009040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:21.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.313+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa160051590 con 0x7fa1700cc810 2023-10-07T09:44:21.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:44:21.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:44:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:21.519+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa1700718b0 con 0x7fa1700cc810 2023-10-07T09:44:22.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.263+0000 7fa1667fc700 1 -- 172.21.15.29:0/3214894245 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa160054bb0 con 0x7fa1700cc810 2023-10-07T09:44:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.268+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa154066f30 msgr2=0x7fa1540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.268+0000 7fa1785ff700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa154066f30 0x7fa1540693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0058d0 tx=0x7fa16c009040 comp rx=0 tx=0).stop 2023-10-07T09:44:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.268+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 msgr2=0x7fa1700ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.268+0000 7fa1785ff700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 0x7fa1700ac230 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa16000e8d0 tx=0x7fa160008040 comp rx=0 tx=0).stop 2023-10-07T09:44:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.268+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 shutdown_connections 2023-10-07T09:44:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.269+0000 7fa1785ff700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa154066f30 0x7fa1540693e0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0058d0 tx=0x7fa16c009040 comp rx=0 tx=0).stop 2023-10-07T09:44:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.269+0000 7fa1785ff700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa17015a590 0x7fa1700ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.269+0000 7fa1785ff700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1700afeb0 0x7fa17014f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.269+0000 7fa1785ff700 1 --2- 172.21.15.29:0/3214894245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1700cc810 0x7fa1700ac230 secure :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa16000e8d0 tx=0x7fa160008040 comp rx=0 tx=0).stop 2023-10-07T09:44:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.269+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 >> 172.21.15.29:0/3214894245 conn(0x7fa170067610 msgr2=0x7fa1700aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.270+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 shutdown_connections 2023-10-07T09:44:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.271+0000 7fa1785ff700 1 -- 172.21.15.29:0/3214894245 wait complete. 2023-10-07T09:44:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:44:22.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:44:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.542+0000 7fe33c3ff700 1 Processor -- start 2023-10-07T09:44:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.542+0000 7fe33c3ff700 1 -- start start 2023-10-07T09:44:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33c3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe334155cb0 0x7fe334159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33c3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 0x7fe3340c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33c3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3340c97e0 con 0x7fe3341558e0 2023-10-07T09:44:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33c3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33415a0c0 con 0x7fe334155cb0 2023-10-07T09:44:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33c3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340c8bd0 con 0x7fe334159c40 2023-10-07T09:44:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33a19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe3341558e0 0x7fe3340b0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39464/0 (socket says 172.21.15.29:39464) 2023-10-07T09:44:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.543+0000 7fe33a19b700 1 -- 172.21.15.29:0/3858471963 learned_addr learned my addr 172.21.15.29:0/3858471963 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe33999a700 1 --2- 172.21.15.29:0/3858471963 >> v2:172.21.15.144:3300/0 conn(0x7fe334155cb0 0x7fe334159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe33a99c700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 0x7fe3340c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411564078 0 0) 0x7fe3340c97e0 con 0x7fe3341558e0 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe320003570 con 0x7fe3341558e0 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545137974 0 0) 0x7fe320003570 con 0x7fe3341558e0 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 msgr2=0x7fe3340c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 0x7fe3340c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 >> v2:172.21.15.144:3300/0 conn(0x7fe334155cb0 msgr2=0x7fe334159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 --2- 172.21.15.29:0/3858471963 >> v2:172.21.15.144:3300/0 conn(0x7fe334155cb0 0x7fe334159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340c8eb0 con 0x7fe3341558e0 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fe330001470 con 0x7fe3341558e0 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 >> v1:172.21.15.29:6789/0 conn(0x7fe3341558e0 legacy=0x7fe3340b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe320003a60 0x7fe320003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 0x7fe320004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320004d50 0x7fe3200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33415a0c0 con 0x7fe320003a60 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340c8bd0 con 0x7fe320004d50 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.544+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32000dc70 con 0x7fe334159c40 2023-10-07T09:44:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe320003a60 0x7fe320003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33999a700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 0x7fe320004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a99c700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320004d50 0x7fe3200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 -- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 msgr2=0x7fe320004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe334159c40 0x7fe320004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 -- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320004d50 msgr2=0x7fe3200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320004d50 0x7fe3200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 -- 172.21.15.29:0/3858471963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320003570 con 0x7fe320003a60 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.545+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe320003a60 0x7fe320003e70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe33000c3c0 tx=0x7fe330001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234a380fae24ca7a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33000cc30 con 0x7fe320003a60 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33000cd90 con 0x7fe320003a60 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe339199700 1 -- 172.21.15.29:0/3858471963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330005b50 con 0x7fe320003a60 2023-10-07T09:44:22.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3858471963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe320003a60 msgr2=0x7fe320003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3858471963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe320003a60 0x7fe320003e70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe33000c3c0 tx=0x7fe330001e50 comp rx=0 tx=0).stop 2023-10-07T09:44:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33a99c700 1 -- 172.21.15.29:0/3858471963 reap_dead start 2023-10-07T09:44:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3858471963 shutdown_connections 2023-10-07T09:44:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3858471963 >> 172.21.15.29:0/3858471963 conn(0x7fe334067610 msgr2=0x7fe334067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3858471963 shutdown_connections 2023-10-07T09:44:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.546+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3858471963 wait complete. 2023-10-07T09:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.548+0000 7fe33c3ff700 1 Processor -- start 2023-10-07T09:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.548+0000 7fe33c3ff700 1 -- start start 2023-10-07T09:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33c3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320003a60 0x7fe3340c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33c3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3340c3650 0x7fe334159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33c3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 0x7fe334150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33c3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340c25b0 con 0x7fe3341595d0 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33c3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340c2890 con 0x7fe320003a60 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33c3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340c2b70 con 0x7fe3340c3650 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33a99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 0x7fe334150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33a99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 0x7fe334150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57708/0 (socket says 172.21.15.29:57708) 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.549+0000 7fe33a99c700 1 -- 172.21.15.29:0/3684767246 learned_addr learned my addr 172.21.15.29:0/3684767246 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320003a60 0x7fe3340c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33999a700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3340c3650 0x7fe334159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a99c700 1 -- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3340c3650 msgr2=0x7fe334159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a99c700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3340c3650 0x7fe334159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a99c700 1 -- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320003a60 msgr2=0x7fe3340c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a99c700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320003a60 0x7fe3340c11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a99c700 1 -- 172.21.15.29:0/3684767246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340c2e50 con 0x7fe3341595d0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33a99c700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 0x7fe334150410 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe33406b990 tx=0x7fe32c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00e040 con 0x7fe3341595d0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c00f040 con 0x7fe3341595d0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3340c3130 con 0x7fe3341595d0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00fe20 con 0x7fe3341595d0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.550+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3340c18c0 con 0x7fe3341595d0 2023-10-07T09:44:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.551+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340718b0 con 0x7fe3341595d0 2023-10-07T09:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.553+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe32c015640 con 0x7fe3341595d0 2023-10-07T09:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.553+0000 7fe32a7fc700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe318066f30 0x7fe3180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.553+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5599+0+0 (secure 0 0 0) 0x7fe32c0858c0 con 0x7fe3341595d0 2023-10-07T09:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.554+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe318066f30 0x7fe3180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.554+0000 7fe33a19b700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe318066f30 0x7fe3180693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe330007cd0 tx=0x7fe330015000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.557+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe32c052b80 con 0x7fe3341595d0 2023-10-07T09:44:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:22.764+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe3340ccd90 con 0x7fe3341595d0 2023-10-07T09:44:23.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.273+0000 7fe32a7fc700 1 -- 172.21.15.29:0/3684767246 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v75) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe32c0561a0 con 0x7fe3341595d0 2023-10-07T09:44:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe318066f30 msgr2=0x7fe3180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe318066f30 0x7fe3180693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe330007cd0 tx=0x7fe330015000 comp rx=0 tx=0).stop 2023-10-07T09:44:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 msgr2=0x7fe334150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 0x7fe334150410 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe33406b990 tx=0x7fe32c00b040 comp rx=0 tx=0).stop 2023-10-07T09:44:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 shutdown_connections 2023-10-07T09:44:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe318066f30 0x7fe3180693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3341595d0 0x7fe334150410 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe320003a60 0x7fe3340c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 --2- 172.21.15.29:0/3684767246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3340c3650 0x7fe334159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.279+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 >> 172.21.15.29:0/3684767246 conn(0x7fe334067610 msgr2=0x7fe33406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:23.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.280+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 shutdown_connections 2023-10-07T09:44:23.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.281+0000 7fe33c3ff700 1 -- 172.21.15.29:0/3684767246 wait complete. 2023-10-07T09:44:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:44:23.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:44:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.509+0000 7f26281ff700 1 Processor -- start 2023-10-07T09:44:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.509+0000 7f26281ff700 1 -- start start 2023-10-07T09:44:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.510+0000 7f26281ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f26200ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f26281ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f26200cad90 0x7f26200c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f26281ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26200cfa00 0x7f26200cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f26281ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26200d03c0 con 0x7f26200cfa00 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f26281ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26200cf570 con 0x7f26200cad90 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f26281ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26200ce960 con 0x7f2620159620 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26200cfa00 0x7f26200cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26200cfa00 0x7f26200cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42522/0 (socket says 172.21.15.29:42522) 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 -- 172.21.15.29:0/2003826948 learned_addr learned my addr 172.21.15.29:0/2003826948 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262579a700 1 --2- 172.21.15.29:0/2003826948 >> v2:172.21.15.144:3300/0 conn(0x7f26200cad90 0x7f26200c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f2625f9b700 1 --2- 172.21.15.29:0/2003826948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f26200ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 -- 172.21.15.29:0/2003826948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 msgr2=0x7f26200ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 --2- 172.21.15.29:0/2003826948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f26200ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 -- 172.21.15.29:0/2003826948 >> v2:172.21.15.144:3300/0 conn(0x7f26200cad90 msgr2=0x7f26200c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.511+0000 7f262679c700 1 --2- 172.21.15.29:0/2003826948 >> v2:172.21.15.144:3300/0 conn(0x7f26200cad90 0x7f26200c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.512+0000 7f262679c700 1 -- 172.21.15.29:0/2003826948 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200cec40 con 0x7f26200cfa00 2023-10-07T09:44:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.512+0000 7f262679c700 1 --2- 172.21.15.29:0/2003826948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cfa00 0x7f26200cfe80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f26200893f0 tx=0x7f2618009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90cc23c2516efe server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.512+0000 7f2624f99700 1 -- 172.21.15.29:0/2003826948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261800e070 con 0x7f26200cfa00 2023-10-07T09:44:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.512+0000 7f2624f99700 1 -- 172.21.15.29:0/2003826948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261800b970 con 0x7f26200cfa00 2023-10-07T09:44:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.512+0000 7f2624f99700 1 -- 172.21.15.29:0/2003826948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26180092e0 con 0x7f26200cfa00 2023-10-07T09:44:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 -- 172.21.15.29:0/2003826948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cfa00 msgr2=0x7f26200cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 --2- 172.21.15.29:0/2003826948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cfa00 0x7f26200cfe80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f26200893f0 tx=0x7f2618009840 comp rx=0 tx=0).stop 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 -- 172.21.15.29:0/2003826948 shutdown_connections 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 --2- 172.21.15.29:0/2003826948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cfa00 0x7f26200cfe80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 --2- 172.21.15.29:0/2003826948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f26200ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 --2- 172.21.15.29:0/2003826948 >> v2:172.21.15.144:3300/0 conn(0x7f26200cad90 0x7f26200c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 -- 172.21.15.29:0/2003826948 >> 172.21.15.29:0/2003826948 conn(0x7f2620089f20 msgr2=0x7f2620099840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 -- 172.21.15.29:0/2003826948 shutdown_connections 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.513+0000 7f26281ff700 1 -- 172.21.15.29:0/2003826948 wait complete. 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.514+0000 7f26281ff700 1 Processor -- start 2023-10-07T09:44:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.514+0000 7f26281ff700 1 -- start start 2023-10-07T09:44:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.515+0000 7f26281ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 0x7f2620169f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.515+0000 7f26281ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26200cfa00 0x7f262009cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.515+0000 7f26281ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f262009cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.515+0000 7f26281ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620169510 con 0x7f26200cad90 2023-10-07T09:44:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.515+0000 7f26281ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26200c5e20 con 0x7f26200cfa00 2023-10-07T09:44:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.515+0000 7f26281ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26200c6100 con 0x7f2620159620 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 0x7f2620169f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 0x7f2620169f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42532/0 (socket says 172.21.15.29:42532) 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 -- 172.21.15.29:0/555036235 learned_addr learned my addr 172.21.15.29:0/555036235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 -- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 msgr2=0x7f262009cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f262009cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 -- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26200cfa00 msgr2=0x7f262009cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26200cfa00 0x7f262009cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 -- 172.21.15.29:0/555036235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2618009ae0 con 0x7f26200cad90 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2625f9b700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 0x7f2620169f80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f261c00ea20 tx=0x7f261c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.516+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c00c850 con 0x7f26200cad90 2023-10-07T09:44:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.517+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c00c9b0 con 0x7f26200cad90 2023-10-07T09:44:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.517+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26200c63e0 con 0x7f26200cad90 2023-10-07T09:44:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.517+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c00cd70 con 0x7f26200cad90 2023-10-07T09:44:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.517+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26200c69a0 con 0x7f26200cad90 2023-10-07T09:44:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.518+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262009a1b0 con 0x7f26200cad90 2023-10-07T09:44:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.519+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f261c013680 con 0x7f26200cad90 2023-10-07T09:44:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.520+0000 7f2615ffb700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2604066f30 0x7f26040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.520+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f261c084ca0 con 0x7f26200cad90 2023-10-07T09:44:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.520+0000 7f262579a700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2604066f30 0x7f26040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.521+0000 7f262579a700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2604066f30 0x7f26040693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2620158af0 tx=0x7f26100091a0 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.525+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f262009a1b0 con 0x7f26200cad90 2023-10-07T09:44:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:23.773+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f262009a1b0 con 0x7f26200cad90 2023-10-07T09:44:24.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.277+0000 7f2615ffb700 1 -- 172.21.15.29:0/555036235 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f261c0210c0 con 0x7f26200cad90 2023-10-07T09:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.293+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2604066f30 msgr2=0x7f26040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.293+0000 7f26281ff700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2604066f30 0x7f26040693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2620158af0 tx=0x7f26100091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.293+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 msgr2=0x7f2620169f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.293+0000 7f26281ff700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 0x7f2620169f80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f261c00ea20 tx=0x7f261c009120 comp rx=0 tx=0).stop 2023-10-07T09:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.294+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 shutdown_connections 2023-10-07T09:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.294+0000 7f26281ff700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2604066f30 0x7f26040693e0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2620158af0 tx=0x7f26100091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.294+0000 7f26281ff700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2620159620 0x7f262009cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.294+0000 7f26281ff700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26200cad90 0x7f2620169f80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.294+0000 7f26281ff700 1 --2- 172.21.15.29:0/555036235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26200cfa00 0x7f262009cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.294+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 >> 172.21.15.29:0/555036235 conn(0x7f2620089f20 msgr2=0x7f26200ccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:24.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.301+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 shutdown_connections 2023-10-07T09:44:24.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.303+0000 7f26281ff700 1 -- 172.21.15.29:0/555036235 wait complete. 2023-10-07T09:44:24.311 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 8 and data pool 9 2023-10-07T09:44:24.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:24.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.525+0000 7fd4851ff700 1 Processor -- start 2023-10-07T09:44:24.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.525+0000 7fd4851ff700 1 -- start start 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd4851ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd480153cc0 0x7fd4800b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd4851ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd48015ade0 0x7fd4800b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd4851ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd4800b0000 0x7fd4800b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd4851ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4800b0980 con 0x7fd480153cc0 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd4851ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd480154090 con 0x7fd4800b0000 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd4851ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd480159f60 con 0x7fd48015ade0 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd47ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd480153cc0 0x7fd4800b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd47ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd480153cc0 0x7fd4800b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42536/0 (socket says 172.21.15.29:42536) 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd47ed9d700 1 -- 172.21.15.29:0/1428817663 learned_addr learned my addr 172.21.15.29:0/1428817663 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd47e59c700 1 --2- 172.21.15.29:0/1428817663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd48015ade0 0x7fd4800b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd47f59e700 1 --2- 172.21.15.29:0/1428817663 >> v2:172.21.15.144:3300/0 conn(0x7fd4800b0000 0x7fd4800b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.527+0000 7fd47ed9d700 1 -- 172.21.15.29:0/1428817663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd48015ade0 msgr2=0x7fd4800b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47ed9d700 1 --2- 172.21.15.29:0/1428817663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd48015ade0 0x7fd4800b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47ed9d700 1 -- 172.21.15.29:0/1428817663 >> v2:172.21.15.144:3300/0 conn(0x7fd4800b0000 msgr2=0x7fd4800b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47ed9d700 1 --2- 172.21.15.29:0/1428817663 >> v2:172.21.15.144:3300/0 conn(0x7fd4800b0000 0x7fd4800b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47ed9d700 1 -- 172.21.15.29:0/1428817663 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48015a240 con 0x7fd480153cc0 2023-10-07T09:44:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47ed9d700 1 --2- 172.21.15.29:0/1428817663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd4800b17c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd480068230 tx=0x7fd470009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372cb0c3515fdee8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47dd9b700 1 -- 172.21.15.29:0/1428817663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470005c10 con 0x7fd480153cc0 2023-10-07T09:44:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47dd9b700 1 -- 172.21.15.29:0/1428817663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd470005d70 con 0x7fd480153cc0 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.528+0000 7fd47dd9b700 1 -- 172.21.15.29:0/1428817663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47000b880 con 0x7fd480153cc0 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 -- 172.21.15.29:0/1428817663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 msgr2=0x7fd4800b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 --2- 172.21.15.29:0/1428817663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd4800b17c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd480068230 tx=0x7fd470009220 comp rx=0 tx=0).stop 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 -- 172.21.15.29:0/1428817663 shutdown_connections 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 --2- 172.21.15.29:0/1428817663 >> v2:172.21.15.144:3300/0 conn(0x7fd4800b0000 0x7fd4800b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 --2- 172.21.15.29:0/1428817663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd4800b17c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 --2- 172.21.15.29:0/1428817663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd48015ade0 0x7fd4800b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 -- 172.21.15.29:0/1428817663 >> 172.21.15.29:0/1428817663 conn(0x7fd480067610 msgr2=0x7fd480067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 -- 172.21.15.29:0/1428817663 shutdown_connections 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.529+0000 7fd4851ff700 1 -- 172.21.15.29:0/1428817663 wait complete. 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.530+0000 7fd4851ff700 1 Processor -- start 2023-10-07T09:44:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.530+0000 7fd4851ff700 1 -- start start 2023-10-07T09:44:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.531+0000 7fd4851ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800b0000 0x7fd4801698e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.531+0000 7fd4851ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd48016c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.531+0000 7fd4851ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd48015ade0 0x7fd480152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.531+0000 7fd4851ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4801569a0 con 0x7fd480153cc0 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.531+0000 7fd4851ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd480157890 con 0x7fd48015ade0 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.531+0000 7fd4851ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd480157b70 con 0x7fd4800b0000 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd48016c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd48016c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42544/0 (socket says 172.21.15.29:42544) 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 -- 172.21.15.29:0/686834319 learned_addr learned my addr 172.21.15.29:0/686834319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47f59e700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd48015ade0 0x7fd480152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47ed9d700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800b0000 0x7fd4801698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 -- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800b0000 msgr2=0x7fd4801698e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800b0000 0x7fd4801698e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 -- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd48015ade0 msgr2=0x7fd480152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd48015ade0 0x7fd480152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 -- 172.21.15.29:0/686834319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd470009480 con 0x7fd480153cc0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47f59e700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd48015ade0 0x7fd480152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd47e59c700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd48016c700 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd47400d860 tx=0x7fd474008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.532+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd474009610 con 0x7fd480153cc0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.533+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd474009770 con 0x7fd480153cc0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.533+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd474009b30 con 0x7fd480153cc0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.533+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd480157e50 con 0x7fd480153cc0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.533+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd480152e70 con 0x7fd480153cc0 2023-10-07T09:44:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.534+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4801576f0 con 0x7fd480153cc0 2023-10-07T09:44:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.535+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd47401e460 con 0x7fd480153cc0 2023-10-07T09:44:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.536+0000 7fd466ffd700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd46c066f30 0x7fd46c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.536+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd474084d20 con 0x7fd480153cc0 2023-10-07T09:44:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.536+0000 7fd47ed9d700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd46c066f30 0x7fd46c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.537+0000 7fd47ed9d700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd46c066f30 0x7fd46c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd470009260 tx=0x7fd470016000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.540+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4801576f0 con 0x7fd480153cc0 2023-10-07T09:44:24.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.741+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4801576f0 con 0x7fd480153cc0 2023-10-07T09:44:24.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.742+0000 7fd466ffd700 1 -- 172.21.15.29:0/686834319 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16030 (secure 0 0 0) 0x7fd4801576f0 con 0x7fd480153cc0 2023-10-07T09:44:24.743 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:24.743 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":76,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:23.774954+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:21.559742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-07T09:44:22.805091+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd46c066f30 msgr2=0x7fd46c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd46c066f30 0x7fd46c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd470009260 tx=0x7fd470016000 comp rx=0 tx=0).stop 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 msgr2=0x7fd48016c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd48016c700 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd47400d860 tx=0x7fd474008040 comp rx=0 tx=0).stop 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 shutdown_connections 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd46c066f30 0x7fd46c0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd48015ade0 0x7fd480152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800b0000 0x7fd4801698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 --2- 172.21.15.29:0/686834319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd480153cc0 0x7fd48016c700 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 >> 172.21.15.29:0/686834319 conn(0x7fd480067610 msgr2=0x7fd48015eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 shutdown_connections 2023-10-07T09:44:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.744+0000 7fd4851ff700 1 -- 172.21.15.29:0/686834319 wait complete. 2023-10-07T09:44:24.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:24.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.958+0000 7f9fa31ff700 1 Processor -- start 2023-10-07T09:44:24.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.958+0000 7f9fa31ff700 1 -- start start 2023-10-07T09:44:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa31ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0ae8b0 0x7f9f9c0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa31ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0b1840 0x7f9f9c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa31ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c158ad0 con 0x7f9f9c157290 2023-10-07T09:44:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa31ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0b1cc0 con 0x7f9f9c0ae8b0 2023-10-07T09:44:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa31ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c15b140 con 0x7f9f9c0b1840 2023-10-07T09:44:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9f9bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0ae8b0 0x7f9f9c0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa0f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9f9c157290 0x7f9f9c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57972/0 (socket says 172.21.15.29:57972) 2023-10-07T09:44:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.959+0000 7f9fa0f9b700 1 -- 172.21.15.29:0/4167000339 learned_addr learned my addr 172.21.15.29:0/4167000339 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9fa179c700 1 --2- 172.21.15.29:0/4167000339 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0b1840 0x7f9f9c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9b7fe700 1 -- 172.21.15.29:0/4167000339 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177062320 0 0) 0x7f9f9c158ad0 con 0x7f9f9c157290 2023-10-07T09:44:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9bfff700 1 -- 172.21.15.29:0/4167000339 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0b1840 msgr2=0x7f9f9c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9bfff700 1 --2- 172.21.15.29:0/4167000339 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0b1840 0x7f9f9c157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9bfff700 1 -- 172.21.15.29:0/4167000339 >> v1:172.21.15.29:6789/0 conn(0x7f9f9c157290 legacy=0x7f9f9c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9bfff700 1 -- 172.21.15.29:0/4167000339 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c15b420 con 0x7f9f9c0ae8b0 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9bfff700 1 --2- 172.21.15.29:0/4167000339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ae8b0 0x7f9f9c0b1300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0cbff0 tx=0x7f9f880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57fa3e367c2cfe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9b7fe700 1 -- 172.21.15.29:0/4167000339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88011630 con 0x7f9f9c0ae8b0 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9b7fe700 1 -- 172.21.15.29:0/4167000339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f88011790 con 0x7f9f9c0ae8b0 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.960+0000 7f9f9b7fe700 1 -- 172.21.15.29:0/4167000339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88011b30 con 0x7f9f9c0ae8b0 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 -- 172.21.15.29:0/4167000339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ae8b0 msgr2=0x7f9f9c0b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/4167000339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ae8b0 0x7f9f9c0b1300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0cbff0 tx=0x7f9f880091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 -- 172.21.15.29:0/4167000339 shutdown_connections 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/4167000339 >> v2:172.21.15.144:3300/0 conn(0x7f9f9c0b1840 0x7f9f9c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/4167000339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ae8b0 0x7f9f9c0b1300 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 -- 172.21.15.29:0/4167000339 >> 172.21.15.29:0/4167000339 conn(0x7f9f9c0680a0 msgr2=0x7f9f9c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 -- 172.21.15.29:0/4167000339 shutdown_connections 2023-10-07T09:44:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.961+0000 7f9fa31ff700 1 -- 172.21.15.29:0/4167000339 wait complete. 2023-10-07T09:44:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.963+0000 7f9fa31ff700 1 Processor -- start 2023-10-07T09:44:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.963+0000 7f9fa31ff700 1 -- start start 2023-10-07T09:44:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa31ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b1840 0x7f9f9c1677a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa31ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c157290 0x7f9f9c0c1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa31ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 0x7f9f9c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa31ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c159220 con 0x7f9f9c1584c0 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa31ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c159ac0 con 0x7f9f9c157290 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa31ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0c2790 con 0x7f9f9c0b1840 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa179c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 0x7f9f9c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa179c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 0x7f9f9c06d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42564/0 (socket says 172.21.15.29:42564) 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa179c700 1 -- 172.21.15.29:0/961364931 learned_addr learned my addr 172.21.15.29:0/961364931 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa0f9b700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b1840 0x7f9f9c1677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9f9bfff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c157290 0x7f9f9c0c1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa179c700 1 -- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b1840 msgr2=0x7f9f9c1677a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.964+0000 7f9fa179c700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b1840 0x7f9f9c1677a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9fa179c700 1 -- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c157290 msgr2=0x7f9f9c0c1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9fa179c700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c157290 0x7f9f9c0c1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9fa179c700 1 -- 172.21.15.29:0/961364931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f88009420 con 0x7f9f9c1584c0 2023-10-07T09:44:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9fa179c700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 0x7f9f9c06d6f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009ec0 tx=0x7f9f9400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9400cb70 con 0x7f9f9c1584c0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9400f040 con 0x7f9f9c1584c0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94014480 con 0x7f9f9c1584c0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c159470 con 0x7f9f9c1584c0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.965+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c167ce0 con 0x7f9f9c1584c0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.966+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c04c3c0 con 0x7f9f9c1584c0 2023-10-07T09:44:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.967+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f94014830 con 0x7f9f9c1584c0 2023-10-07T09:44:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.968+0000 7f9f98ff9700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f80066f30 0x7f9f800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.968+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f9f94084ff0 con 0x7f9f9c1584c0 2023-10-07T09:44:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.969+0000 7f9fa0f9b700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f80066f30 0x7f9f800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:24.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.969+0000 7f9fa0f9b700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f80066f30 0x7f9f800693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f90007fd0 tx=0x7f9f90007d30 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:24.972+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f94052040 con 0x7f9f9c1584c0 2023-10-07T09:44:25.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.173+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f9c06f3b0 con 0x7f9f9c1584c0 2023-10-07T09:44:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.174+0000 7f9f98ff9700 1 -- 172.21.15.29:0/961364931 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16030 (secure 0 0 0) 0x7f9f94055660 con 0x7f9f9c1584c0 2023-10-07T09:44:25.175 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:25.175 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":76,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:23.774954+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:21.559742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-07T09:44:22.805091+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.176+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f80066f30 msgr2=0x7f9f800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.176+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f80066f30 0x7f9f800693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f90007fd0 tx=0x7f9f90007d30 comp rx=0 tx=0).stop 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.176+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 msgr2=0x7f9f9c06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.176+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 0x7f9f9c06d6f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009ec0 tx=0x7f9f9400b040 comp rx=0 tx=0).stop 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.176+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 shutdown_connections 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f80066f30 0x7f9f800693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c1584c0 0x7f9f9c06d6f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b1840 0x7f9f9c1677a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 --2- 172.21.15.29:0/961364931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c157290 0x7f9f9c0c1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 >> 172.21.15.29:0/961364931 conn(0x7f9f9c0680a0 msgr2=0x7f9f9c0ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 shutdown_connections 2023-10-07T09:44:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.177+0000 7f9fa31ff700 1 -- 172.21.15.29:0/961364931 wait complete. 2023-10-07T09:44:25.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:44:25.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.396+0000 7f4068bff700 1 Processor -- start 2023-10-07T09:44:25.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.396+0000 7f4068bff700 1 -- start start 2023-10-07T09:44:25.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.397+0000 7f4068bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4064067d20 0x7f406406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:25.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.397+0000 7f4068bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40640c78c0 con 0x7f406413ff70 2023-10-07T09:44:25.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.397+0000 7f4068bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406406fa50 con 0x7f406413fba0 2023-10-07T09:44:25.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.397+0000 7f4068bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406406c0f0 con 0x7f4064067d20 2023-10-07T09:44:25.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.397+0000 7f4061d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4064067d20 0x7f406406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:25.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f4062d9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f406413ff70 0x7f40640c7d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57980/0 (socket says 172.21.15.29:57980) 2023-10-07T09:44:25.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f4062d9d700 1 -- 172.21.15.29:0/2084423133 learned_addr learned my addr 172.21.15.29:0/2084423133 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:25.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f4061d9b700 1 -- 172.21.15.29:0/2084423133 >> v1:172.21.15.144:6789/0 conn(0x7f406413fba0 legacy=0x7f4064067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f4061d9b700 1 -- 172.21.15.29:0/2084423133 >> v1:172.21.15.29:6789/0 conn(0x7f406413ff70 legacy=0x7f40640c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f4061d9b700 1 -- 172.21.15.29:0/2084423133 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406406c3d0 con 0x7f4064067d20 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f406159a700 1 -- 172.21.15.29:0/2084423133 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292181067 0 0) 0x7f40640c78c0 con 0x7f406413ff70 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.398+0000 7f4061d9b700 1 --2- 172.21.15.29:0/2084423133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f406406ba10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f40580098d0 tx=0x7f40580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492e0739b501d4c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f406159a700 1 -- 172.21.15.29:0/2084423133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058009e70 con 0x7f4064067d20 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f406159a700 1 -- 172.21.15.29:0/2084423133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405800b8f0 con 0x7f4064067d20 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f406159a700 1 -- 172.21.15.29:0/2084423133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405800bc90 con 0x7f4064067d20 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 -- 172.21.15.29:0/2084423133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 msgr2=0x7f406406ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 --2- 172.21.15.29:0/2084423133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f406406ba10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f40580098d0 tx=0x7f40580091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 -- 172.21.15.29:0/2084423133 shutdown_connections 2023-10-07T09:44:25.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 --2- 172.21.15.29:0/2084423133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f406406ba10 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 -- 172.21.15.29:0/2084423133 >> 172.21.15.29:0/2084423133 conn(0x7f406406b1d0 msgr2=0x7f40640cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:25.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 -- 172.21.15.29:0/2084423133 shutdown_connections 2023-10-07T09:44:25.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.399+0000 7f4068bff700 1 -- 172.21.15.29:0/2084423133 wait complete. 2023-10-07T09:44:25.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.401+0000 7f4068bff700 1 Processor -- start 2023-10-07T09:44:25.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.401+0000 7f4068bff700 1 -- start start 2023-10-07T09:44:25.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4068bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f40640c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:25.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4068bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f406413fba0 0x7f4064158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4068bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f406413ff70 0x7f406413da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4068bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640c6fa0 con 0x7f4064067d20 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4068bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640c44d0 con 0x7f406413ff70 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4068bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40640c47b0 con 0x7f406413fba0 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f406259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f40640c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f406259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f40640c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42586/0 (socket says 172.21.15.29:42586) 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f406259c700 1 -- 172.21.15.29:0/1480876925 learned_addr learned my addr 172.21.15.29:0/1480876925 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4062d9d700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f406413ff70 0x7f406413da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.402+0000 7f4061d9b700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f406413fba0 0x7f4064158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f406259c700 1 -- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f406413fba0 msgr2=0x7f4064158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f406259c700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f406413fba0 0x7f4064158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f406259c700 1 -- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f406413ff70 msgr2=0x7f406413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f406259c700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f406413ff70 0x7f406413da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f406259c700 1 -- 172.21.15.29:0/1480876925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580093c0 con 0x7f4064067d20 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f4061d9b700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f406413fba0 0x7f4064158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f4062d9d700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f406413ff70 0x7f406413da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f406259c700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f40640c6a60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f405400c590 tx=0x7f405400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054011590 con 0x7f4064067d20 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40540116f0 con 0x7f4064067d20 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054011ab0 con 0x7f4064067d20 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40640c4a90 con 0x7f4064067d20 2023-10-07T09:44:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.403+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4064159110 con 0x7f4064067d20 2023-10-07T09:44:25.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.404+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406404c3c0 con 0x7f4064067d20 2023-10-07T09:44:25.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.405+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f405401b3b0 con 0x7f4064067d20 2023-10-07T09:44:25.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.406+0000 7f40527fc700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4044066f30 0x7f40440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:25.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.406+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f4054083d00 con 0x7f4064067d20 2023-10-07T09:44:25.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.407+0000 7f4061d9b700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4044066f30 0x7f40440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:25.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.407+0000 7f4061d9b700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4044066f30 0x7f40440693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4064068120 tx=0x7f4058017000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.410+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4054050d50 con 0x7f4064067d20 2023-10-07T09:44:25.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:25.660+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f406406c6b0 con 0x7f4064067d20 2023-10-07T09:44:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.300+0000 7f40527fc700 1 -- 172.21.15.29:0/1480876925 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4054017020 con 0x7f4064067d20 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4044066f30 msgr2=0x7f40440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4044066f30 0x7f40440693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4064068120 tx=0x7f4058017000 comp rx=0 tx=0).stop 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 msgr2=0x7f40640c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f40640c6a60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f405400c590 tx=0x7f405400c950 comp rx=0 tx=0).stop 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 shutdown_connections 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4044066f30 0x7f40440693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f406413ff70 0x7f406413da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4064067d20 0x7f40640c6a60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 --2- 172.21.15.29:0/1480876925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f406413fba0 0x7f4064158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.303+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 >> 172.21.15.29:0/1480876925 conn(0x7f406406b1d0 msgr2=0x7f4064067610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.304+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 shutdown_connections 2023-10-07T09:44:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.304+0000 7f4068bff700 1 -- 172.21.15.29:0/1480876925 wait complete. 2023-10-07T09:44:26.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:44:26.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.527+0000 7fb43a9ff700 1 Processor -- start 2023-10-07T09:44:26.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.528+0000 7fb43a9ff700 1 -- start start 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb43a9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb434159790 0x7fb43406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb43a9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb43406b3f0 0x7fb4340c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb43a9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43406a430 con 0x7fb43406a8c0 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb43a9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb434159b60 con 0x7fb43406b3f0 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb43a9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43406b810 con 0x7fb434159790 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb438f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb43406b3f0 0x7fb4340c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb42bfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb43406a8c0 0x7fb4340c6f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57994/0 (socket says 172.21.15.29:57994) 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.529+0000 7fb42bfff700 1 -- 172.21.15.29:0/2724209601 learned_addr learned my addr 172.21.15.29:0/2724209601 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb433fff700 1 --2- 172.21.15.29:0/2724209601 >> v2:172.21.15.144:3300/0 conn(0x7fb434159790 0x7fb43406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb438f9c700 1 -- 172.21.15.29:0/2724209601 >> v2:172.21.15.144:3300/0 conn(0x7fb434159790 msgr2=0x7fb43406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb438f9c700 1 --2- 172.21.15.29:0/2724209601 >> v2:172.21.15.144:3300/0 conn(0x7fb434159790 0x7fb43406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb438f9c700 1 -- 172.21.15.29:0/2724209601 >> v1:172.21.15.29:6789/0 conn(0x7fb43406a8c0 legacy=0x7fb4340c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb438f9c700 1 -- 172.21.15.29:0/2724209601 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb434069820 con 0x7fb43406b3f0 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb433fff700 1 --2- 172.21.15.29:0/2724209601 >> v2:172.21.15.144:3300/0 conn(0x7fb434159790 0x7fb43406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb4337fe700 1 -- 172.21.15.29:0/2724209601 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103951932 0 0) 0x7fb43406a430 con 0x7fb43406a8c0 2023-10-07T09:44:26.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb438f9c700 1 --2- 172.21.15.29:0/2724209601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb43406b3f0 0x7fb4340c7650 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb43406d6c0 tx=0x7fb42c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f1c55543e1499f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:26.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb4337fe700 1 -- 172.21.15.29:0/2724209601 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00e070 con 0x7fb43406b3f0 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb4337fe700 1 -- 172.21.15.29:0/2724209601 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00bc30 con 0x7fb43406b3f0 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.530+0000 7fb4337fe700 1 -- 172.21.15.29:0/2724209601 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c012430 con 0x7fb43406b3f0 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 -- 172.21.15.29:0/2724209601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb43406b3f0 msgr2=0x7fb4340c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/2724209601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb43406b3f0 0x7fb4340c7650 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb43406d6c0 tx=0x7fb42c009780 comp rx=0 tx=0).stop 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 -- 172.21.15.29:0/2724209601 shutdown_connections 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/2724209601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb43406b3f0 0x7fb4340c7650 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/2724209601 >> v2:172.21.15.144:3300/0 conn(0x7fb434159790 0x7fb43406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 -- 172.21.15.29:0/2724209601 >> 172.21.15.29:0/2724209601 conn(0x7fb4340c9410 msgr2=0x7fb4340c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 -- 172.21.15.29:0/2724209601 shutdown_connections 2023-10-07T09:44:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.531+0000 7fb43a9ff700 1 -- 172.21.15.29:0/2724209601 wait complete. 2023-10-07T09:44:26.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.533+0000 7fb43a9ff700 1 Processor -- start 2023-10-07T09:44:26.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.533+0000 7fb43a9ff700 1 -- start start 2023-10-07T09:44:26.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb43a9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43406b3f0 0x7fb434152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:26.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb43a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb434159790 0x7fb4340ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:26.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb43a9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 0x7fb4340a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb43a9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb434152670 con 0x7fb4340acb10 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb43a9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340acfb0 con 0x7fb43406b3f0 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb43a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340b1f70 con 0x7fb434159790 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 0x7fb4340a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 0x7fb4340a9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42604/0 (socket says 172.21.15.29:42604) 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 -- 172.21.15.29:0/1673313396 learned_addr learned my addr 172.21.15.29:0/1673313396 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb433fff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43406b3f0 0x7fb434152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb42bfff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb434159790 0x7fb4340ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 -- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb434159790 msgr2=0x7fb4340ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb434159790 0x7fb4340ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 -- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43406b3f0 msgr2=0x7fb434152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:26.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43406b3f0 0x7fb434152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:26.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.534+0000 7fb438f9c700 1 -- 172.21.15.29:0/1673313396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c009a20 con 0x7fb4340acb10 2023-10-07T09:44:26.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.535+0000 7fb438f9c700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 0x7fb4340a9ea0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb43406c760 tx=0x7fb42c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:26.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.535+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c012e60 con 0x7fb4340acb10 2023-10-07T09:44:26.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.535+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c01d630 con 0x7fb4340acb10 2023-10-07T09:44:26.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.535+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c01d9f0 con 0x7fb4340acb10 2023-10-07T09:44:26.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.535+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4340b16d0 con 0x7fb4340acb10 2023-10-07T09:44:26.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.535+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4340b1c90 con 0x7fb4340acb10 2023-10-07T09:44:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.536+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43404c3c0 con 0x7fb4340acb10 2023-10-07T09:44:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.537+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb42c024070 con 0x7fb4340acb10 2023-10-07T09:44:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.538+0000 7fb430ff9700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb414066f30 0x7fb4140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.538+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb42c086fe0 con 0x7fb4340acb10 2023-10-07T09:44:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.539+0000 7fb433fff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb414066f30 0x7fb4140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.539+0000 7fb433fff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb414066f30 0x7fb4140693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb424006f00 tx=0x7fb424008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:26.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.542+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb42c054050 con 0x7fb4340acb10 2023-10-07T09:44:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:26.790+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb434068980 con 0x7fb4340acb10 2023-10-07T09:44:27.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.306+0000 7fb430ff9700 1 -- 172.21.15.29:0/1673313396 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb42c057670 con 0x7fb4340acb10 2023-10-07T09:44:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb414066f30 msgr2=0x7fb4140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb414066f30 0x7fb4140693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb424006f00 tx=0x7fb424008040 comp rx=0 tx=0).stop 2023-10-07T09:44:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 msgr2=0x7fb4340a9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 0x7fb4340a9ea0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb43406c760 tx=0x7fb42c019000 comp rx=0 tx=0).stop 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 shutdown_connections 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb414066f30 0x7fb4140693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4340acb10 0x7fb4340a9ea0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43406b3f0 0x7fb434152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 --2- 172.21.15.29:0/1673313396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb434159790 0x7fb4340ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 >> 172.21.15.29:0/1673313396 conn(0x7fb4340c9410 msgr2=0x7fb434153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 shutdown_connections 2023-10-07T09:44:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.309+0000 7fb43a9ff700 1 -- 172.21.15.29:0/1673313396 wait complete. 2023-10-07T09:44:27.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.529+0000 7f46bb1ff700 1 Processor -- start 2023-10-07T09:44:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.529+0000 7f46bb1ff700 1 -- start start 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46bb1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f46b40ad220 0x7f46b4159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46bb1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b40ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46bb1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b415af00 con 0x7f46b4068620 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46bb1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b40ac9d0 con 0x7f46b40ad220 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46bb1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40ada40 con 0x7f46b40ad5f0 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46b979c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f46b4068620 0x7f46b415d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58010/0 (socket says 172.21.15.29:58010) 2023-10-07T09:44:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46b979c700 1 -- 172.21.15.29:0/4180098870 learned_addr learned my addr 172.21.15.29:0/4180098870 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46b8f9b700 1 --2- 172.21.15.29:0/4180098870 >> v2:172.21.15.144:3300/0 conn(0x7f46b40ad220 0x7f46b4159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46b3fff700 1 --2- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b40ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.531+0000 7f46b37fe700 1 -- 172.21.15.29:0/4180098870 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990027936 0 0) 0x7f46b415af00 con 0x7f46b4068620 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b37fe700 1 -- 172.21.15.29:0/4180098870 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a4003570 con 0x7f46b4068620 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b8f9b700 1 -- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 msgr2=0x7f46b40ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b8f9b700 1 --2- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b40ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b37fe700 1 -- 172.21.15.29:0/4180098870 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243167219 0 0) 0x7f46a4003570 con 0x7f46b4068620 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b8f9b700 1 -- 172.21.15.29:0/4180098870 >> v1:172.21.15.29:6789/0 conn(0x7f46b4068620 legacy=0x7f46b415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b8f9b700 1 -- 172.21.15.29:0/4180098870 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b415a250 con 0x7f46b40ad220 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b8f9b700 1 --2- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40ad220 0x7f46b4159c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f46a80098e0 tx=0x7f46a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd546b378d6b6cd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.532+0000 7f46b37fe700 1 -- 172.21.15.29:0/4180098870 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac0017f0 con 0x7f46b40ad220 2023-10-07T09:44:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46b37fe700 1 -- 172.21.15.29:0/4180098870 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a8011420 con 0x7f46b40ad220 2023-10-07T09:44:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46b37fe700 1 -- 172.21.15.29:0/4180098870 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a80117c0 con 0x7f46b40ad220 2023-10-07T09:44:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 -- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40ad220 msgr2=0x7f46b4159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40ad220 0x7f46b4159c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f46a80098e0 tx=0x7f46a80091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 -- 172.21.15.29:0/4180098870 shutdown_connections 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40ad220 0x7f46b4159c00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/4180098870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b40ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 -- 172.21.15.29:0/4180098870 >> 172.21.15.29:0/4180098870 conn(0x7f46b406dd00 msgr2=0x7f46b406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 -- 172.21.15.29:0/4180098870 shutdown_connections 2023-10-07T09:44:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.533+0000 7f46bb1ff700 1 -- 172.21.15.29:0/4180098870 wait complete. 2023-10-07T09:44:27.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.535+0000 7f46bb1ff700 1 Processor -- start 2023-10-07T09:44:27.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.536+0000 7f46bb1ff700 1 -- start start 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46bb1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b4068620 0x7f46b4142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46bb1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 0x7f46b413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46bb1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b406a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46bb1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b4142a70 con 0x7f46b40ad220 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46bb1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b4140090 con 0x7f46b4068620 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46bb1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b4141e60 con 0x7f46b40ad5f0 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 0x7f46b413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 0x7f46b413fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42616/0 (socket says 172.21.15.29:42616) 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46b3fff700 1 -- 172.21.15.29:0/2494915946 learned_addr learned my addr 172.21.15.29:0/2494915946 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46b8f9b700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b4068620 0x7f46b4142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.537+0000 7f46b979c700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b406a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b3fff700 1 -- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 msgr2=0x7f46b406a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b3fff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b406a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b3fff700 1 -- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b4068620 msgr2=0x7f46b4142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b3fff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b4068620 0x7f46b4142f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b3fff700 1 -- 172.21.15.29:0/2494915946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8009420 con 0x7f46b40ad220 2023-10-07T09:44:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b3fff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 0x7f46b413fb50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f46a000ec80 tx=0x7f46a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a000c850 con 0x7f46b40ad220 2023-10-07T09:44:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a000c9b0 con 0x7f46b40ad220 2023-10-07T09:44:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a000cd50 con 0x7f46b40ad220 2023-10-07T09:44:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b4142140 con 0x7f46b40ad220 2023-10-07T09:44:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.538+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b4142700 con 0x7f46b40ad220 2023-10-07T09:44:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.539+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b404c3c0 con 0x7f46b40ad220 2023-10-07T09:44:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.540+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f46a0013680 con 0x7f46b40ad220 2023-10-07T09:44:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.541+0000 7f46b0ff9700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4698066f30 0x7f46980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.541+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f46a00849b0 con 0x7f46b40ad220 2023-10-07T09:44:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.542+0000 7f46b8f9b700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4698066f30 0x7f46980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.542+0000 7f46b8f9b700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4698066f30 0x7f46980693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46a80093f0 tx=0x7f46a8016000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:27.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.546+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46a0051a00 con 0x7f46b40ad220 2023-10-07T09:44:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.792+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b4142bb0 con 0x7f46b40ad220 2023-10-07T09:44:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.793+0000 7f46b0ff9700 1 -- 172.21.15.29:0/2494915946 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f46a0055020 con 0x7f46b40ad220 2023-10-07T09:44:27.794 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:27.794 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:44:23.775309+0000","modified":"2023-10-07T09:44:27.299910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6276":{"gid":6276,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2418315778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2418315778},{"type":"v1","addr":"172.21.15.29:6835","nonce":2418315778}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.795+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4698066f30 msgr2=0x7f46980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.795+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4698066f30 0x7f46980693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46a80093f0 tx=0x7f46a8016000 comp rx=0 tx=0).stop 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.795+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 msgr2=0x7f46b413fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.795+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 0x7f46b413fb50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f46a000ec80 tx=0x7f46a00091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 shutdown_connections 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4698066f30 0x7f46980693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46b40ad5f0 0x7f46b406a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b4068620 0x7f46b4142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 --2- 172.21.15.29:0/2494915946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46b40ad220 0x7f46b413fb50 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 >> 172.21.15.29:0/2494915946 conn(0x7f46b406dd00 msgr2=0x7f46b40695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 shutdown_connections 2023-10-07T09:44:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:27.796+0000 7f46bb1ff700 1 -- 172.21.15.29:0/2494915946 wait complete. 2023-10-07T09:44:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-07T09:44:27.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.010+0000 7f42361ff700 1 Processor -- start 2023-10-07T09:44:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.010+0000 7f42361ff700 1 -- start start 2023-10-07T09:44:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f42361ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4230159750 0x7f42300d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f42361ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42300ce8a0 0x7f42300c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f42361ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300cbf20 con 0x7f4230159b20 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f42361ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42300cc200 con 0x7f42300ce8a0 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f42361ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42300cc4e0 con 0x7f4230159750 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f422ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42300ce8a0 0x7f42300c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.011+0000 7f422effd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4230159b20 0x7f42300ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58018/0 (socket says 172.21.15.29:58018) 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42300ce8a0 0x7f42300c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42620/0 (socket says 172.21.15.29:42620) 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422effd700 1 -- 172.21.15.29:0/1838954321 learned_addr learned my addr 172.21.15.29:0/1838954321 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422f7fe700 1 --2- 172.21.15.29:0/1838954321 >> v2:172.21.15.144:3300/0 conn(0x7f4230159750 0x7f42300d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422ffff700 1 -- 172.21.15.29:0/1838954321 >> v2:172.21.15.144:3300/0 conn(0x7f4230159750 msgr2=0x7f42300d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422ffff700 1 --2- 172.21.15.29:0/1838954321 >> v2:172.21.15.144:3300/0 conn(0x7f4230159750 0x7f42300d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422ffff700 1 -- 172.21.15.29:0/1838954321 >> v1:172.21.15.29:6789/0 conn(0x7f4230159b20 legacy=0x7f42300ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422ffff700 1 -- 172.21.15.29:0/1838954321 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300cc7c0 con 0x7f42300ce8a0 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422ffff700 1 --2- 172.21.15.29:0/1838954321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f42300c5460 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4230089840 tx=0x7f4228009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e32219421ad7586 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422e7fc700 1 -- 172.21.15.29:0/1838954321 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800e070 con 0x7f42300ce8a0 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422e7fc700 1 -- 172.21.15.29:0/1838954321 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800bc30 con 0x7f42300ce8a0 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.012+0000 7f422e7fc700 1 -- 172.21.15.29:0/1838954321 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228012430 con 0x7f42300ce8a0 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 -- 172.21.15.29:0/1838954321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 msgr2=0x7f42300c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 --2- 172.21.15.29:0/1838954321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f42300c5460 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4230089840 tx=0x7f4228009780 comp rx=0 tx=0).stop 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 -- 172.21.15.29:0/1838954321 shutdown_connections 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 --2- 172.21.15.29:0/1838954321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f42300c5460 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 --2- 172.21.15.29:0/1838954321 >> v2:172.21.15.144:3300/0 conn(0x7f4230159750 0x7f42300d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 -- 172.21.15.29:0/1838954321 >> 172.21.15.29:0/1838954321 conn(0x7f4230063a10 msgr2=0x7f42300d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 -- 172.21.15.29:0/1838954321 shutdown_connections 2023-10-07T09:44:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.013+0000 7f42361ff700 1 -- 172.21.15.29:0/1838954321 wait complete. 2023-10-07T09:44:28.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.015+0000 7f42361ff700 1 Processor -- start 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.015+0000 7f42361ff700 1 -- start start 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f42361ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f423015b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f42361ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230159750 0x7f423015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f42361ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230159b20 0x7f423015cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f42361ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423015c1f0 con 0x7f42300ce8a0 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f42361ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300b3ae0 con 0x7f4230159b20 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f42361ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423015bf10 con 0x7f4230159750 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f423015b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f423015b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42634/0 (socket says 172.21.15.29:42634) 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 -- 172.21.15.29:0/2001275031 learned_addr learned my addr 172.21.15.29:0/2001275031 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 -- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230159750 msgr2=0x7f423015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230159750 0x7f423015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 -- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230159b20 msgr2=0x7f423015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230159b20 0x7f423015cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.016+0000 7f422f7fe700 1 -- 172.21.15.29:0/2001275031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009a20 con 0x7f42300ce8a0 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.017+0000 7f422f7fe700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f423015b8e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f422000d810 tx=0x7f4220008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.017+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42200125f0 con 0x7f42300ce8a0 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.017+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4220012750 con 0x7f42300ce8a0 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.017+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42300b3490 con 0x7f42300ce8a0 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.017+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220012b10 con 0x7f42300ce8a0 2023-10-07T09:44:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.017+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423015d110 con 0x7f42300ce8a0 2023-10-07T09:44:28.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.018+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423004c3c0 con 0x7f42300ce8a0 2023-10-07T09:44:28.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.019+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f422001e460 con 0x7f42300ce8a0 2023-10-07T09:44:28.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.020+0000 7f42177fe700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f421c066f30 0x7f421c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.020+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f4220084de0 con 0x7f42300ce8a0 2023-10-07T09:44:28.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.021+0000 7f422effd700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f421c066f30 0x7f421c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.021+0000 7f422effd700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f421c066f30 0x7f421c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4224007fd0 tx=0x7f4224007b90 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.024+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4220051e30 con 0x7f42300ce8a0 2023-10-07T09:44:28.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.228+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42300b3dc0 con 0x7f42300ce8a0 2023-10-07T09:44:28.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.229+0000 7f42177fe700 1 -- 172.21.15.29:0/2001275031 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16017 (secure 0 0 0) 0x7f4220055450 con 0x7f42300ce8a0 2023-10-07T09:44:28.231 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:28.231 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:25.273836+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:21.559742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-07T09:44:22.805091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:28.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f421c066f30 msgr2=0x7f421c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f421c066f30 0x7f421c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4224007fd0 tx=0x7f4224007b90 comp rx=0 tx=0).stop 2023-10-07T09:44:28.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 msgr2=0x7f423015b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f423015b8e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f422000d810 tx=0x7f4220008040 comp rx=0 tx=0).stop 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 shutdown_connections 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f421c066f30 0x7f421c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230159b20 0x7f423015cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300ce8a0 0x7f423015b8e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 --2- 172.21.15.29:0/2001275031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230159750 0x7f423015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 >> 172.21.15.29:0/2001275031 conn(0x7f4230063a10 msgr2=0x7f42300704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 shutdown_connections 2023-10-07T09:44:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.232+0000 7f42361ff700 1 -- 172.21.15.29:0/2001275031 wait complete. 2023-10-07T09:44:28.251 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:44:28.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:44:28.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.447+0000 7f57443ff700 1 Processor -- start 2023-10-07T09:44:28.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.447+0000 7f57443ff700 1 -- start start 2023-10-07T09:44:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.448+0000 7f57443ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f573c159c00 0x7f573c0ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.448+0000 7f57443ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c15af00 con 0x7f573c068620 2023-10-07T09:44:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.448+0000 7f57443ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c15a250 con 0x7f573c0ad220 2023-10-07T09:44:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.448+0000 7f57443ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c15a050 con 0x7f573c159c00 2023-10-07T09:44:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.448+0000 7f574299c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f573c068620 0x7f573c068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58032/0 (socket says 172.21.15.29:58032) 2023-10-07T09:44:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.448+0000 7f574299c700 1 -- 172.21.15.29:0/1582463925 learned_addr learned my addr 172.21.15.29:0/1582463925 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f574199a700 1 --2- 172.21.15.29:0/1582463925 >> v2:172.21.15.144:3300/0 conn(0x7f573c159c00 0x7f573c0ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f5741199700 1 -- 172.21.15.29:0/1582463925 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195868633 0 0) 0x7f573c15af00 con 0x7f573c068620 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f5741199700 1 -- 172.21.15.29:0/1582463925 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728003570 con 0x7f573c068620 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f574199a700 1 -- 172.21.15.29:0/1582463925 >> v1:172.21.15.144:6789/0 conn(0x7f573c0ad220 legacy=0x7f573c0ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f574199a700 1 -- 172.21.15.29:0/1582463925 >> v1:172.21.15.29:6789/0 conn(0x7f573c068620 legacy=0x7f573c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f5741199700 1 -- 172.21.15.29:0/1582463925 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726923663 0 0) 0x7f5728003570 con 0x7f573c068620 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.449+0000 7f574199a700 1 -- 172.21.15.29:0/1582463925 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c15a530 con 0x7f573c159c00 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f574199a700 1 --2- 172.21.15.29:0/1582463925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c0ad5f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098e0 tx=0x7f572c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61e49a0946e8ce42 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f5741199700 1 -- 172.21.15.29:0/1582463925 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340017f0 con 0x7f573c159c00 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f5741199700 1 -- 172.21.15.29:0/1582463925 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c011420 con 0x7f573c159c00 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f5741199700 1 -- 172.21.15.29:0/1582463925 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0117c0 con 0x7f573c159c00 2023-10-07T09:44:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f57443ff700 1 -- 172.21.15.29:0/1582463925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 msgr2=0x7f573c0ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f57443ff700 1 --2- 172.21.15.29:0/1582463925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c0ad5f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098e0 tx=0x7f572c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f57443ff700 1 -- 172.21.15.29:0/1582463925 shutdown_connections 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f57443ff700 1 --2- 172.21.15.29:0/1582463925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c0ad5f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.450+0000 7f57443ff700 1 -- 172.21.15.29:0/1582463925 >> 172.21.15.29:0/1582463925 conn(0x7f573c06dd00 msgr2=0x7f573c06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.451+0000 7f57443ff700 1 -- 172.21.15.29:0/1582463925 shutdown_connections 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.451+0000 7f57443ff700 1 -- 172.21.15.29:0/1582463925 wait complete. 2023-10-07T09:44:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.453+0000 7f57443ff700 1 Processor -- start 2023-10-07T09:44:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.453+0000 7f57443ff700 1 -- start start 2023-10-07T09:44:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f57443ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 0x7f573c0cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f57443ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f573c0ad220 0x7f573c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f57443ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f57443ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0ca6b0 con 0x7f573c068620 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f57443ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c9d80 con 0x7f573c159c00 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f57443ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0ca060 con 0x7f573c0ad220 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f574219b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 0x7f573c0cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f574219b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 0x7f573c0cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42640/0 (socket says 172.21.15.29:42640) 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.454+0000 7f574219b700 1 -- 172.21.15.29:0/1384753141 learned_addr learned my addr 172.21.15.29:0/1384753141 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574199a700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f573c0ad220 0x7f573c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574299c700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574219b700 1 -- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f573c0ad220 msgr2=0x7f573c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574219b700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f573c0ad220 0x7f573c15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574219b700 1 -- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 msgr2=0x7f573c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574219b700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c15d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574219b700 1 -- 172.21.15.29:0/1384753141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c009420 con 0x7f573c068620 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f574219b700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 0x7f573c0cbe30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f573800c590 tx=0x7f573800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738011590 con 0x7f573c068620 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57380116f0 con 0x7f573c068620 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c0ca340 con 0x7f573c068620 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738011ab0 con 0x7f573c068620 2023-10-07T09:44:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.455+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c151060 con 0x7f573c068620 2023-10-07T09:44:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.456+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c0caa40 con 0x7f573c068620 2023-10-07T09:44:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.457+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f573801b3b0 con 0x7f573c068620 2023-10-07T09:44:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.458+0000 7f57327fc700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5720066f30 0x7f57200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.458+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5738083cf0 con 0x7f573c068620 2023-10-07T09:44:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.459+0000 7f574199a700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5720066f30 0x7f57200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.460+0000 7f574199a700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5720066f30 0x7f57200693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f572c0097b0 tx=0x7f572c016000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.463+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f573c0caa40 con 0x7f573c068620 2023-10-07T09:44:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.663+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f573c0ca7f0 con 0x7f5720066f30 2023-10-07T09:44:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.674+0000 7f57327fc700 1 -- 172.21.15.29:0/1384753141 <== mgr.5899 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288902 (secure 0 0 0) 0x7f572c000ce0 con 0x7f5720066f30 2023-10-07T09:44:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5720066f30 msgr2=0x7f57200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5720066f30 0x7f57200693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f572c0097b0 tx=0x7f572c016000 comp rx=0 tx=0).stop 2023-10-07T09:44:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 msgr2=0x7f573c0cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 0x7f573c0cbe30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f573800c590 tx=0x7f573800c950 comp rx=0 tx=0).stop 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 shutdown_connections 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5720066f30 0x7f57200693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f573c159c00 0x7f573c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f573c068620 0x7f573c0cbe30 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 --2- 172.21.15.29:0/1384753141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f573c0ad220 0x7f573c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 >> 172.21.15.29:0/1384753141 conn(0x7f573c06dd00 msgr2=0x7f573c152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 shutdown_connections 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.678+0000 7f57443ff700 1 -- 172.21.15.29:0/1384753141 wait complete. 2023-10-07T09:44:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:44:28.699 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:44:28.380985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2435844,"kb_used_data":8388,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752538876,"statfs":{"total":773094113280,"available":770599809024,"internally_reserved":0,"allocated":8589312,"data_stored":4630868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001170"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.313081+0000","last_change":"2023-10-07T09:44:24.310970+0000","last_active":"2023-10-07T09:44:24.313081+0000","last_peered":"2023-10-07T09:44:24.313081+0000","last_clean":"2023-10-07T09:44:24.313081+0000","last_became_active":"2023-10-07T09:44:24.310728+0000","last_became_peered":"2023-10-07T09:44:24.310728+0000","last_unstale":"2023-10-07T09:44:24.313081+0000","last_undegraded":"2023-10-07T09:44:24.313081+0000","last_fullsized":"2023-10-07T09:44:24.313081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:23.587424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.294355+0000","last_change":"2023-10-07T09:44:24.291115+0000","last_active":"2023-10-07T09:44:24.294355+0000","last_peered":"2023-10-07T09:44:24.294355+0000","last_clean":"2023-10-07T09:44:24.294355+0000","last_became_active":"2023-10-07T09:44:24.290853+0000","last_became_peered":"2023-10-07T09:44:24.290853+0000","last_unstale":"2023-10-07T09:44:24.294355+0000","last_undegraded":"2023-10-07T09:44:24.294355+0000","last_fullsized":"2023-10-07T09:44:24.294355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:21:10.094242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302301+0000","last_change":"2023-10-07T09:44:24.292751+0000","last_active":"2023-10-07T09:44:24.302301+0000","last_peered":"2023-10-07T09:44:24.302301+0000","last_clean":"2023-10-07T09:44:24.302301+0000","last_became_active":"2023-10-07T09:44:24.292426+0000","last_became_peered":"2023-10-07T09:44:24.292426+0000","last_unstale":"2023-10-07T09:44:24.302301+0000","last_undegraded":"2023-10-07T09:44:24.302301+0000","last_fullsized":"2023-10-07T09:44:24.302301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:35:09.290288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309215+0000","last_change":"2023-10-07T09:44:24.307582+0000","last_active":"2023-10-07T09:44:24.309215+0000","last_peered":"2023-10-07T09:44:24.309215+0000","last_clean":"2023-10-07T09:44:24.309215+0000","last_became_active":"2023-10-07T09:44:24.307167+0000","last_became_peered":"2023-10-07T09:44:24.307167+0000","last_unstale":"2023-10-07T09:44:24.309215+0000","last_undegraded":"2023-10-07T09:44:24.309215+0000","last_fullsized":"2023-10-07T09:44:24.309215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:01:24.933011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311192+0000","last_change":"2023-10-07T09:44:24.308931+0000","last_active":"2023-10-07T09:44:24.311192+0000","last_peered":"2023-10-07T09:44:24.311192+0000","last_clean":"2023-10-07T09:44:24.311192+0000","last_became_active":"2023-10-07T09:44:24.308628+0000","last_became_peered":"2023-10-07T09:44:24.308628+0000","last_unstale":"2023-10-07T09:44:24.311192+0000","last_undegraded":"2023-10-07T09:44:24.311192+0000","last_fullsized":"2023-10-07T09:44:24.311192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:40:50.626505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311378+0000","last_change":"2023-10-07T09:44:24.306453+0000","last_active":"2023-10-07T09:44:24.311378+0000","last_peered":"2023-10-07T09:44:24.311378+0000","last_clean":"2023-10-07T09:44:24.311378+0000","last_became_active":"2023-10-07T09:44:24.306185+0000","last_became_peered":"2023-10-07T09:44:24.306185+0000","last_unstale":"2023-10-07T09:44:24.311378+0000","last_undegraded":"2023-10-07T09:44:24.311378+0000","last_fullsized":"2023-10-07T09:44:24.311378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:45.216789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301545+0000","last_change":"2023-10-07T09:44:24.295414+0000","last_active":"2023-10-07T09:44:24.301545+0000","last_peered":"2023-10-07T09:44:24.301545+0000","last_clean":"2023-10-07T09:44:24.301545+0000","last_became_active":"2023-10-07T09:44:24.295058+0000","last_became_peered":"2023-10-07T09:44:24.295058+0000","last_unstale":"2023-10-07T09:44:24.301545+0000","last_undegraded":"2023-10-07T09:44:24.301545+0000","last_fullsized":"2023-10-07T09:44:24.301545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:02.073157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.297758+0000","last_change":"2023-10-07T09:44:24.292891+0000","last_active":"2023-10-07T09:44:24.297758+0000","last_peered":"2023-10-07T09:44:24.297758+0000","last_clean":"2023-10-07T09:44:24.297758+0000","last_became_active":"2023-10-07T09:44:24.292567+0000","last_became_peered":"2023-10-07T09:44:24.292567+0000","last_unstale":"2023-10-07T09:44:24.297758+0000","last_undegraded":"2023-10-07T09:44:24.297758+0000","last_fullsized":"2023-10-07T09:44:24.297758+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:42:15.921547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.305376+0000","last_change":"2023-10-07T09:44:24.304246+0000","last_active":"2023-10-07T09:44:24.305376+0000","last_peered":"2023-10-07T09:44:24.305376+0000","last_clean":"2023-10-07T09:44:24.305376+0000","last_became_active":"2023-10-07T09:44:24.303933+0000","last_became_peered":"2023-10-07T09:44:24.303933+0000","last_unstale":"2023-10-07T09:44:24.305376+0000","last_undegraded":"2023-10-07T09:44:24.305376+0000","last_fullsized":"2023-10-07T09:44:24.305376+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:01.660743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306302+0000","last_change":"2023-10-07T09:44:24.291982+0000","last_active":"2023-10-07T09:44:24.306302+0000","last_peered":"2023-10-07T09:44:24.306302+0000","last_clean":"2023-10-07T09:44:24.306302+0000","last_became_active":"2023-10-07T09:44:24.291630+0000","last_became_peered":"2023-10-07T09:44:24.291630+0000","last_unstale":"2023-10-07T09:44:24.306302+0000","last_undegraded":"2023-10-07T09:44:24.306302+0000","last_fullsized":"2023-10-07T09:44:24.306302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:44:38.869997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.305027+0000","last_change":"2023-10-07T09:44:24.293784+0000","last_active":"2023-10-07T09:44:24.305027+0000","last_peered":"2023-10-07T09:44:24.305027+0000","last_clean":"2023-10-07T09:44:24.305027+0000","last_became_active":"2023-10-07T09:44:24.293523+0000","last_became_peered":"2023-10-07T09:44:24.293523+0000","last_unstale":"2023-10-07T09:44:24.305027+0000","last_undegraded":"2023-10-07T09:44:24.305027+0000","last_fullsized":"2023-10-07T09:44:24.305027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:07.649574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.308055+0000","last_change":"2023-10-07T09:44:24.304149+0000","last_active":"2023-10-07T09:44:24.308055+0000","last_peered":"2023-10-07T09:44:24.308055+0000","last_clean":"2023-10-07T09:44:24.308055+0000","last_became_active":"2023-10-07T09:44:24.303708+0000","last_became_peered":"2023-10-07T09:44:24.303708+0000","last_unstale":"2023-10-07T09:44:24.308055+0000","last_undegraded":"2023-10-07T09:44:24.308055+0000","last_fullsized":"2023-10-07T09:44:24.308055+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:05.505574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303324+0000","last_change":"2023-10-07T09:44:24.301270+0000","last_active":"2023-10-07T09:44:24.303324+0000","last_peered":"2023-10-07T09:44:24.303324+0000","last_clean":"2023-10-07T09:44:24.303324+0000","last_became_active":"2023-10-07T09:44:24.300834+0000","last_became_peered":"2023-10-07T09:44:24.300834+0000","last_unstale":"2023-10-07T09:44:24.303324+0000","last_undegraded":"2023-10-07T09:44:24.303324+0000","last_fullsized":"2023-10-07T09:44:24.303324+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:50:13.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306351+0000","last_change":"2023-10-07T09:44:24.305487+0000","last_active":"2023-10-07T09:44:24.306351+0000","last_peered":"2023-10-07T09:44:24.306351+0000","last_clean":"2023-10-07T09:44:24.306351+0000","last_became_active":"2023-10-07T09:44:24.305161+0000","last_became_peered":"2023-10-07T09:44:24.305161+0000","last_unstale":"2023-10-07T09:44:24.306351+0000","last_undegraded":"2023-10-07T09:44:24.306351+0000","last_fullsized":"2023-10-07T09:44:24.306351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:01.336800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.295559+0000","last_change":"2023-10-07T09:44:24.293709+0000","last_active":"2023-10-07T09:44:24.295559+0000","last_peered":"2023-10-07T09:44:24.295559+0000","last_clean":"2023-10-07T09:44:24.295559+0000","last_became_active":"2023-10-07T09:44:24.293405+0000","last_became_peered":"2023-10-07T09:44:24.293405+0000","last_unstale":"2023-10-07T09:44:24.295559+0000","last_undegraded":"2023-10-07T09:44:24.295559+0000","last_fullsized":"2023-10-07T09:44:24.295559+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:49:53.054391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307302+0000","last_change":"2023-10-07T09:44:24.297405+0000","last_active":"2023-10-07T09:44:24.307302+0000","last_peered":"2023-10-07T09:44:24.307302+0000","last_clean":"2023-10-07T09:44:24.307302+0000","last_became_active":"2023-10-07T09:44:24.296841+0000","last_became_peered":"2023-10-07T09:44:24.296841+0000","last_unstale":"2023-10-07T09:44:24.307302+0000","last_undegraded":"2023-10-07T09:44:24.307302+0000","last_fullsized":"2023-10-07T09:44:24.307302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:49:53.054391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.292717+0000","last_change":"2023-10-07T09:44:24.286094+0000","last_active":"2023-10-07T09:44:24.292717+0000","last_peered":"2023-10-07T09:44:24.292717+0000","last_clean":"2023-10-07T09:44:24.292717+0000","last_became_active":"2023-10-07T09:44:24.285739+0000","last_became_peered":"2023-10-07T09:44:24.285739+0000","last_unstale":"2023-10-07T09:44:24.292717+0000","last_undegraded":"2023-10-07T09:44:24.292717+0000","last_fullsized":"2023-10-07T09:44:24.292717+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:52.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309212+0000","last_change":"2023-10-07T09:44:24.303692+0000","last_active":"2023-10-07T09:44:24.309212+0000","last_peered":"2023-10-07T09:44:24.309212+0000","last_clean":"2023-10-07T09:44:24.309212+0000","last_became_active":"2023-10-07T09:44:24.303407+0000","last_became_peered":"2023-10-07T09:44:24.303407+0000","last_unstale":"2023-10-07T09:44:24.309212+0000","last_undegraded":"2023-10-07T09:44:24.309212+0000","last_fullsized":"2023-10-07T09:44:24.309212+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:20:53.802952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312681+0000","last_change":"2023-10-07T09:44:24.310459+0000","last_active":"2023-10-07T09:44:24.312681+0000","last_peered":"2023-10-07T09:44:24.312681+0000","last_clean":"2023-10-07T09:44:24.312681+0000","last_became_active":"2023-10-07T09:44:24.310206+0000","last_became_peered":"2023-10-07T09:44:24.310206+0000","last_unstale":"2023-10-07T09:44:24.312681+0000","last_undegraded":"2023-10-07T09:44:24.312681+0000","last_fullsized":"2023-10-07T09:44:24.312681+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:30.263388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.308898+0000","last_change":"2023-10-07T09:44:24.300426+0000","last_active":"2023-10-07T09:44:24.308898+0000","last_peered":"2023-10-07T09:44:24.308898+0000","last_clean":"2023-10-07T09:44:24.308898+0000","last_became_active":"2023-10-07T09:44:24.300169+0000","last_became_peered":"2023-10-07T09:44:24.300169+0000","last_unstale":"2023-10-07T09:44:24.308898+0000","last_undegraded":"2023-10-07T09:44:24.308898+0000","last_fullsized":"2023-10-07T09:44:24.308898+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:38:12.002708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301559+0000","last_change":"2023-10-07T09:44:24.299002+0000","last_active":"2023-10-07T09:44:24.301559+0000","last_peered":"2023-10-07T09:44:24.301559+0000","last_clean":"2023-10-07T09:44:24.301559+0000","last_became_active":"2023-10-07T09:44:24.297952+0000","last_became_peered":"2023-10-07T09:44:24.297952+0000","last_unstale":"2023-10-07T09:44:24.301559+0000","last_undegraded":"2023-10-07T09:44:24.301559+0000","last_fullsized":"2023-10-07T09:44:24.301559+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:17:26.117441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312630+0000","last_change":"2023-10-07T09:44:24.310885+0000","last_active":"2023-10-07T09:44:24.312630+0000","last_peered":"2023-10-07T09:44:24.312630+0000","last_clean":"2023-10-07T09:44:24.312630+0000","last_became_active":"2023-10-07T09:44:24.310465+0000","last_became_peered":"2023-10-07T09:44:24.310465+0000","last_unstale":"2023-10-07T09:44:24.312630+0000","last_undegraded":"2023-10-07T09:44:24.312630+0000","last_fullsized":"2023-10-07T09:44:24.312630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:43.243147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301055+0000","last_change":"2023-10-07T09:44:24.297591+0000","last_active":"2023-10-07T09:44:24.301055+0000","last_peered":"2023-10-07T09:44:24.301055+0000","last_clean":"2023-10-07T09:44:24.301055+0000","last_became_active":"2023-10-07T09:44:24.297013+0000","last_became_peered":"2023-10-07T09:44:24.297013+0000","last_unstale":"2023-10-07T09:44:24.301055+0000","last_undegraded":"2023-10-07T09:44:24.301055+0000","last_fullsized":"2023-10-07T09:44:24.301055+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:58:16.034746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303391+0000","last_change":"2023-10-07T09:44:24.301329+0000","last_active":"2023-10-07T09:44:24.303391+0000","last_peered":"2023-10-07T09:44:24.303391+0000","last_clean":"2023-10-07T09:44:24.303391+0000","last_became_active":"2023-10-07T09:44:24.300891+0000","last_became_peered":"2023-10-07T09:44:24.300891+0000","last_unstale":"2023-10-07T09:44:24.303391+0000","last_undegraded":"2023-10-07T09:44:24.303391+0000","last_fullsized":"2023-10-07T09:44:24.303391+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:44:38.869997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309149+0000","last_change":"2023-10-07T09:44:24.307494+0000","last_active":"2023-10-07T09:44:24.309149+0000","last_peered":"2023-10-07T09:44:24.309149+0000","last_clean":"2023-10-07T09:44:24.309149+0000","last_became_active":"2023-10-07T09:44:24.307216+0000","last_became_peered":"2023-10-07T09:44:24.307216+0000","last_unstale":"2023-10-07T09:44:24.309149+0000","last_undegraded":"2023-10-07T09:44:24.309149+0000","last_fullsized":"2023-10-07T09:44:24.309149+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:07:19.074986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306174+0000","last_change":"2023-10-07T09:44:24.302355+0000","last_active":"2023-10-07T09:44:24.306174+0000","last_peered":"2023-10-07T09:44:24.306174+0000","last_clean":"2023-10-07T09:44:24.306174+0000","last_became_active":"2023-10-07T09:44:24.302038+0000","last_became_peered":"2023-10-07T09:44:24.302038+0000","last_unstale":"2023-10-07T09:44:24.306174+0000","last_undegraded":"2023-10-07T09:44:24.306174+0000","last_fullsized":"2023-10-07T09:44:24.306174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:58:25.295458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306082+0000","last_change":"2023-10-07T09:44:24.299160+0000","last_active":"2023-10-07T09:44:24.306082+0000","last_peered":"2023-10-07T09:44:24.306082+0000","last_clean":"2023-10-07T09:44:24.306082+0000","last_became_active":"2023-10-07T09:44:24.298619+0000","last_became_peered":"2023-10-07T09:44:24.298619+0000","last_unstale":"2023-10-07T09:44:24.306082+0000","last_undegraded":"2023-10-07T09:44:24.306082+0000","last_fullsized":"2023-10-07T09:44:24.306082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:15:41.028294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307882+0000","last_change":"2023-10-07T09:44:24.304159+0000","last_active":"2023-10-07T09:44:24.307882+0000","last_peered":"2023-10-07T09:44:24.307882+0000","last_clean":"2023-10-07T09:44:24.307882+0000","last_became_active":"2023-10-07T09:44:24.303825+0000","last_became_peered":"2023-10-07T09:44:24.303825+0000","last_unstale":"2023-10-07T09:44:24.307882+0000","last_undegraded":"2023-10-07T09:44:24.307882+0000","last_fullsized":"2023-10-07T09:44:24.307882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:31:14.028736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302427+0000","last_change":"2023-10-07T09:44:24.295412+0000","last_active":"2023-10-07T09:44:24.302427+0000","last_peered":"2023-10-07T09:44:24.302427+0000","last_clean":"2023-10-07T09:44:24.302427+0000","last_became_active":"2023-10-07T09:44:24.295145+0000","last_became_peered":"2023-10-07T09:44:24.295145+0000","last_unstale":"2023-10-07T09:44:24.302427+0000","last_undegraded":"2023-10-07T09:44:24.302427+0000","last_fullsized":"2023-10-07T09:44:24.302427+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:31.116678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302395+0000","last_change":"2023-10-07T09:44:24.293530+0000","last_active":"2023-10-07T09:44:24.302395+0000","last_peered":"2023-10-07T09:44:24.302395+0000","last_clean":"2023-10-07T09:44:24.302395+0000","last_became_active":"2023-10-07T09:44:24.290943+0000","last_became_peered":"2023-10-07T09:44:24.290943+0000","last_unstale":"2023-10-07T09:44:24.302395+0000","last_undegraded":"2023-10-07T09:44:24.302395+0000","last_fullsized":"2023-10-07T09:44:24.302395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:53:58.454122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312069+0000","last_change":"2023-10-07T09:44:24.309726+0000","last_active":"2023-10-07T09:44:24.312069+0000","last_peered":"2023-10-07T09:44:24.312069+0000","last_clean":"2023-10-07T09:44:24.312069+0000","last_became_active":"2023-10-07T09:44:24.309467+0000","last_became_peered":"2023-10-07T09:44:24.309467+0000","last_unstale":"2023-10-07T09:44:24.312069+0000","last_undegraded":"2023-10-07T09:44:24.312069+0000","last_fullsized":"2023-10-07T09:44:24.312069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:55:32.282248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307283+0000","last_change":"2023-10-07T09:44:24.297242+0000","last_active":"2023-10-07T09:44:24.307283+0000","last_peered":"2023-10-07T09:44:24.307283+0000","last_clean":"2023-10-07T09:44:24.307283+0000","last_became_active":"2023-10-07T09:44:24.296541+0000","last_became_peered":"2023-10-07T09:44:24.296541+0000","last_unstale":"2023-10-07T09:44:24.307283+0000","last_undegraded":"2023-10-07T09:44:24.307283+0000","last_fullsized":"2023-10-07T09:44:24.307283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:46.054505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311356+0000","last_change":"2023-10-07T09:44:24.305638+0000","last_active":"2023-10-07T09:44:24.311356+0000","last_peered":"2023-10-07T09:44:24.311356+0000","last_clean":"2023-10-07T09:44:24.311356+0000","last_became_active":"2023-10-07T09:44:24.305054+0000","last_became_peered":"2023-10-07T09:44:24.305054+0000","last_unstale":"2023-10-07T09:44:24.311356+0000","last_undegraded":"2023-10-07T09:44:24.311356+0000","last_fullsized":"2023-10-07T09:44:24.311356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:59.732037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"76'4","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302987+0000","last_change":"2023-10-07T09:44:23.291971+0000","last_active":"2023-10-07T09:44:24.302987+0000","last_peered":"2023-10-07T09:44:24.302987+0000","last_clean":"2023-10-07T09:44:24.302987+0000","last_became_active":"2023-10-07T09:44:23.291699+0000","last_became_peered":"2023-10-07T09:44:23.291699+0000","last_unstale":"2023-10-07T09:44:24.302987+0000","last_undegraded":"2023-10-07T09:44:24.302987+0000","last_fullsized":"2023-10-07T09:44:24.302987+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:16:51.707247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288177+0000","last_change":"2023-10-07T09:44:24.286677+0000","last_active":"2023-10-07T09:44:24.288177+0000","last_peered":"2023-10-07T09:44:24.288177+0000","last_clean":"2023-10-07T09:44:24.288177+0000","last_became_active":"2023-10-07T09:44:24.286418+0000","last_became_peered":"2023-10-07T09:44:24.286418+0000","last_unstale":"2023-10-07T09:44:24.288177+0000","last_undegraded":"2023-10-07T09:44:24.288177+0000","last_fullsized":"2023-10-07T09:44:24.288177+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:40.873462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303110+0000","last_change":"2023-10-07T09:44:23.295503+0000","last_active":"2023-10-07T09:44:24.303110+0000","last_peered":"2023-10-07T09:44:24.303110+0000","last_clean":"2023-10-07T09:44:24.303110+0000","last_became_active":"2023-10-07T09:44:23.295266+0000","last_became_peered":"2023-10-07T09:44:23.295266+0000","last_unstale":"2023-10-07T09:44:24.303110+0000","last_undegraded":"2023-10-07T09:44:24.303110+0000","last_fullsized":"2023-10-07T09:44:24.303110+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:00.328720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311921+0000","last_change":"2023-10-07T09:44:24.305846+0000","last_active":"2023-10-07T09:44:24.311921+0000","last_peered":"2023-10-07T09:44:24.311921+0000","last_clean":"2023-10-07T09:44:24.311921+0000","last_became_active":"2023-10-07T09:44:24.305533+0000","last_became_peered":"2023-10-07T09:44:24.305533+0000","last_unstale":"2023-10-07T09:44:24.311921+0000","last_undegraded":"2023-10-07T09:44:24.311921+0000","last_fullsized":"2023-10-07T09:44:24.311921+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:55:10.571566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306282+0000","last_change":"2023-10-07T09:44:23.283966+0000","last_active":"2023-10-07T09:44:24.306282+0000","last_peered":"2023-10-07T09:44:24.306282+0000","last_clean":"2023-10-07T09:44:24.306282+0000","last_became_active":"2023-10-07T09:44:23.283694+0000","last_became_peered":"2023-10-07T09:44:23.283694+0000","last_unstale":"2023-10-07T09:44:24.306282+0000","last_undegraded":"2023-10-07T09:44:24.306282+0000","last_fullsized":"2023-10-07T09:44:24.306282+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:19.056016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306117+0000","last_change":"2023-10-07T09:44:24.299366+0000","last_active":"2023-10-07T09:44:24.306117+0000","last_peered":"2023-10-07T09:44:24.306117+0000","last_clean":"2023-10-07T09:44:24.306117+0000","last_became_active":"2023-10-07T09:44:24.298974+0000","last_became_peered":"2023-10-07T09:44:24.298974+0000","last_unstale":"2023-10-07T09:44:24.306117+0000","last_undegraded":"2023-10-07T09:44:24.306117+0000","last_fullsized":"2023-10-07T09:44:24.306117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:01.660743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304228+0000","last_change":"2023-10-07T09:44:23.300755+0000","last_active":"2023-10-07T09:44:24.304228+0000","last_peered":"2023-10-07T09:44:24.304228+0000","last_clean":"2023-10-07T09:44:24.304228+0000","last_became_active":"2023-10-07T09:44:23.300478+0000","last_became_peered":"2023-10-07T09:44:23.300478+0000","last_unstale":"2023-10-07T09:44:24.304228+0000","last_undegraded":"2023-10-07T09:44:24.304228+0000","last_fullsized":"2023-10-07T09:44:24.304228+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:39.865381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312608+0000","last_change":"2023-10-07T09:44:24.310721+0000","last_active":"2023-10-07T09:44:24.312608+0000","last_peered":"2023-10-07T09:44:24.312608+0000","last_clean":"2023-10-07T09:44:24.312608+0000","last_became_active":"2023-10-07T09:44:24.310109+0000","last_became_peered":"2023-10-07T09:44:24.310109+0000","last_unstale":"2023-10-07T09:44:24.312608+0000","last_undegraded":"2023-10-07T09:44:24.312608+0000","last_fullsized":"2023-10-07T09:44:24.312608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:30:21.331223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304353+0000","last_change":"2023-10-07T09:44:23.295182+0000","last_active":"2023-10-07T09:44:24.304353+0000","last_peered":"2023-10-07T09:44:24.304353+0000","last_clean":"2023-10-07T09:44:24.304353+0000","last_became_active":"2023-10-07T09:44:23.294935+0000","last_became_peered":"2023-10-07T09:44:23.294935+0000","last_unstale":"2023-10-07T09:44:24.304353+0000","last_undegraded":"2023-10-07T09:44:24.304353+0000","last_fullsized":"2023-10-07T09:44:24.304353+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:06.641494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311565+0000","last_change":"2023-10-07T09:44:24.298842+0000","last_active":"2023-10-07T09:44:24.311565+0000","last_peered":"2023-10-07T09:44:24.311565+0000","last_clean":"2023-10-07T09:44:24.311565+0000","last_became_active":"2023-10-07T09:44:24.298596+0000","last_became_peered":"2023-10-07T09:44:24.298596+0000","last_unstale":"2023-10-07T09:44:24.311565+0000","last_undegraded":"2023-10-07T09:44:24.311565+0000","last_fullsized":"2023-10-07T09:44:24.311565+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:25.708170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302594+0000","last_change":"2023-10-07T09:44:23.292056+0000","last_active":"2023-10-07T09:44:24.302594+0000","last_peered":"2023-10-07T09:44:24.302594+0000","last_clean":"2023-10-07T09:44:24.302594+0000","last_became_active":"2023-10-07T09:44:23.291816+0000","last_became_peered":"2023-10-07T09:44:23.291816+0000","last_unstale":"2023-10-07T09:44:24.302594+0000","last_undegraded":"2023-10-07T09:44:24.302594+0000","last_fullsized":"2023-10-07T09:44:24.302594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:10:42.036332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.294493+0000","last_change":"2023-10-07T09:44:24.287269+0000","last_active":"2023-10-07T09:44:24.294493+0000","last_peered":"2023-10-07T09:44:24.294493+0000","last_clean":"2023-10-07T09:44:24.294493+0000","last_became_active":"2023-10-07T09:44:24.286960+0000","last_became_peered":"2023-10-07T09:44:24.286960+0000","last_unstale":"2023-10-07T09:44:24.294493+0000","last_undegraded":"2023-10-07T09:44:24.294493+0000","last_fullsized":"2023-10-07T09:44:24.294493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:46.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288941+0000","last_change":"2023-10-07T09:44:23.295396+0000","last_active":"2023-10-07T09:44:24.288941+0000","last_peered":"2023-10-07T09:44:24.288941+0000","last_clean":"2023-10-07T09:44:24.288941+0000","last_became_active":"2023-10-07T09:44:23.295096+0000","last_became_peered":"2023-10-07T09:44:23.295096+0000","last_unstale":"2023-10-07T09:44:24.288941+0000","last_undegraded":"2023-10-07T09:44:24.288941+0000","last_fullsized":"2023-10-07T09:44:24.288941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:45.046424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.292422+0000","last_change":"2023-10-07T09:44:24.286652+0000","last_active":"2023-10-07T09:44:24.292422+0000","last_peered":"2023-10-07T09:44:24.292422+0000","last_clean":"2023-10-07T09:44:24.292422+0000","last_became_active":"2023-10-07T09:44:24.286180+0000","last_became_peered":"2023-10-07T09:44:24.286180+0000","last_unstale":"2023-10-07T09:44:24.292422+0000","last_undegraded":"2023-10-07T09:44:24.292422+0000","last_fullsized":"2023-10-07T09:44:24.292422+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:29.612688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.286699+0000","last_change":"2023-10-07T09:44:23.294852+0000","last_active":"2023-10-07T09:44:24.286699+0000","last_peered":"2023-10-07T09:44:24.286699+0000","last_clean":"2023-10-07T09:44:24.286699+0000","last_became_active":"2023-10-07T09:44:23.294590+0000","last_became_peered":"2023-10-07T09:44:23.294590+0000","last_unstale":"2023-10-07T09:44:24.286699+0000","last_undegraded":"2023-10-07T09:44:24.286699+0000","last_fullsized":"2023-10-07T09:44:24.286699+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:44:37.861916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309172+0000","last_change":"2023-10-07T09:44:24.307592+0000","last_active":"2023-10-07T09:44:24.309172+0000","last_peered":"2023-10-07T09:44:24.309172+0000","last_clean":"2023-10-07T09:44:24.309172+0000","last_became_active":"2023-10-07T09:44:24.307107+0000","last_became_peered":"2023-10-07T09:44:24.307107+0000","last_unstale":"2023-10-07T09:44:24.309172+0000","last_undegraded":"2023-10-07T09:44:24.309172+0000","last_fullsized":"2023-10-07T09:44:24.309172+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:15:41.028294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.282373+0000","last_change":"2023-10-07T09:44:23.300332+0000","last_active":"2023-10-07T09:44:24.282373+0000","last_peered":"2023-10-07T09:44:24.282373+0000","last_clean":"2023-10-07T09:44:24.282373+0000","last_became_active":"2023-10-07T09:44:23.299952+0000","last_became_peered":"2023-10-07T09:44:23.299952+0000","last_unstale":"2023-10-07T09:44:24.282373+0000","last_undegraded":"2023-10-07T09:44:24.282373+0000","last_fullsized":"2023-10-07T09:44:24.282373+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:44:02.443489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.298888+0000","last_change":"2023-10-07T09:44:24.291861+0000","last_active":"2023-10-07T09:44:24.298888+0000","last_peered":"2023-10-07T09:44:24.298888+0000","last_clean":"2023-10-07T09:44:24.298888+0000","last_became_active":"2023-10-07T09:44:24.291548+0000","last_became_peered":"2023-10-07T09:44:24.291548+0000","last_unstale":"2023-10-07T09:44:24.298888+0000","last_undegraded":"2023-10-07T09:44:24.298888+0000","last_fullsized":"2023-10-07T09:44:24.298888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:55:32.790004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.278296+0000","last_change":"2023-10-07T09:44:23.290458+0000","last_active":"2023-10-07T09:44:24.278296+0000","last_peered":"2023-10-07T09:44:24.278296+0000","last_clean":"2023-10-07T09:44:24.278296+0000","last_became_active":"2023-10-07T09:44:23.290191+0000","last_became_peered":"2023-10-07T09:44:23.290191+0000","last_unstale":"2023-10-07T09:44:24.278296+0000","last_undegraded":"2023-10-07T09:44:24.278296+0000","last_fullsized":"2023-10-07T09:44:24.278296+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:49:52.046310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304354+0000","last_change":"2023-10-07T09:44:24.299451+0000","last_active":"2023-10-07T09:44:24.304354+0000","last_peered":"2023-10-07T09:44:24.304354+0000","last_clean":"2023-10-07T09:44:24.304354+0000","last_became_active":"2023-10-07T09:44:24.298999+0000","last_became_peered":"2023-10-07T09:44:24.298999+0000","last_unstale":"2023-10-07T09:44:24.304354+0000","last_undegraded":"2023-10-07T09:44:24.304354+0000","last_fullsized":"2023-10-07T09:44:24.304354+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:50:13.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"76'3","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.313810+0000","last_change":"2023-10-07T09:44:23.300451+0000","last_active":"2023-10-07T09:44:24.313810+0000","last_peered":"2023-10-07T09:44:24.313810+0000","last_clean":"2023-10-07T09:44:24.313810+0000","last_became_active":"2023-10-07T09:44:23.300143+0000","last_became_peered":"2023-10-07T09:44:23.300143+0000","last_unstale":"2023-10-07T09:44:24.313810+0000","last_undegraded":"2023-10-07T09:44:24.313810+0000","last_fullsized":"2023-10-07T09:44:24.313810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:22:46.170377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310089+0000","last_change":"2023-10-07T09:44:24.306425+0000","last_active":"2023-10-07T09:44:24.310089+0000","last_peered":"2023-10-07T09:44:24.310089+0000","last_clean":"2023-10-07T09:44:24.310089+0000","last_became_active":"2023-10-07T09:44:24.306128+0000","last_became_peered":"2023-10-07T09:44:24.306128+0000","last_unstale":"2023-10-07T09:44:24.310089+0000","last_undegraded":"2023-10-07T09:44:24.310089+0000","last_fullsized":"2023-10-07T09:44:24.310089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:21.761574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"76'6","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312841+0000","last_change":"2023-10-07T09:44:23.295870+0000","last_active":"2023-10-07T09:44:24.312841+0000","last_peered":"2023-10-07T09:44:24.312841+0000","last_clean":"2023-10-07T09:44:24.312841+0000","last_became_active":"2023-10-07T09:44:23.295590+0000","last_became_peered":"2023-10-07T09:44:23.295590+0000","last_unstale":"2023-10-07T09:44:24.312841+0000","last_undegraded":"2023-10-07T09:44:24.312841+0000","last_fullsized":"2023-10-07T09:44:24.312841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:11:31.060288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310295+0000","last_change":"2023-10-07T09:44:24.309035+0000","last_active":"2023-10-07T09:44:24.310295+0000","last_peered":"2023-10-07T09:44:24.310295+0000","last_clean":"2023-10-07T09:44:24.310295+0000","last_became_active":"2023-10-07T09:44:24.308726+0000","last_became_peered":"2023-10-07T09:44:24.308726+0000","last_unstale":"2023-10-07T09:44:24.310295+0000","last_undegraded":"2023-10-07T09:44:24.310295+0000","last_fullsized":"2023-10-07T09:44:24.310295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:38:42.467052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302771+0000","last_change":"2023-10-07T09:44:23.290550+0000","last_active":"2023-10-07T09:44:24.302771+0000","last_peered":"2023-10-07T09:44:24.302771+0000","last_clean":"2023-10-07T09:44:24.302771+0000","last_became_active":"2023-10-07T09:44:23.290291+0000","last_became_peered":"2023-10-07T09:44:23.290291+0000","last_unstale":"2023-10-07T09:44:24.302771+0000","last_undegraded":"2023-10-07T09:44:24.302771+0000","last_fullsized":"2023-10-07T09:44:24.302771+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:15:40.020214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.298660+0000","last_change":"2023-10-07T09:44:24.290277+0000","last_active":"2023-10-07T09:44:24.298660+0000","last_peered":"2023-10-07T09:44:24.298660+0000","last_clean":"2023-10-07T09:44:24.298660+0000","last_became_active":"2023-10-07T09:44:24.289943+0000","last_became_peered":"2023-10-07T09:44:24.289943+0000","last_unstale":"2023-10-07T09:44:24.298660+0000","last_undegraded":"2023-10-07T09:44:24.298660+0000","last_fullsized":"2023-10-07T09:44:24.298660+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:01:23.707605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.283202+0000","last_change":"2023-10-07T09:44:23.295504+0000","last_active":"2023-10-07T09:44:24.283202+0000","last_peered":"2023-10-07T09:44:24.283202+0000","last_clean":"2023-10-07T09:44:24.283202+0000","last_became_active":"2023-10-07T09:44:23.295248+0000","last_became_peered":"2023-10-07T09:44:23.295248+0000","last_unstale":"2023-10-07T09:44:24.283202+0000","last_undegraded":"2023-10-07T09:44:24.283202+0000","last_fullsized":"2023-10-07T09:44:24.283202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:06.098008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.295833+0000","last_change":"2023-10-07T09:44:24.294432+0000","last_active":"2023-10-07T09:44:24.295833+0000","last_peered":"2023-10-07T09:44:24.295833+0000","last_clean":"2023-10-07T09:44:24.295833+0000","last_became_active":"2023-10-07T09:44:24.294159+0000","last_became_peered":"2023-10-07T09:44:24.294159+0000","last_unstale":"2023-10-07T09:44:24.295833+0000","last_undegraded":"2023-10-07T09:44:24.295833+0000","last_fullsized":"2023-10-07T09:44:24.295833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:59.732037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288978+0000","last_change":"2023-10-07T09:44:23.300648+0000","last_active":"2023-10-07T09:44:24.288978+0000","last_peered":"2023-10-07T09:44:24.288978+0000","last_clean":"2023-10-07T09:44:24.288978+0000","last_became_active":"2023-10-07T09:44:23.300334+0000","last_became_peered":"2023-10-07T09:44:23.300334+0000","last_unstale":"2023-10-07T09:44:24.288978+0000","last_undegraded":"2023-10-07T09:44:24.288978+0000","last_fullsized":"2023-10-07T09:44:24.288978+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:39:32.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301038+0000","last_change":"2023-10-07T09:44:24.297316+0000","last_active":"2023-10-07T09:44:24.301038+0000","last_peered":"2023-10-07T09:44:24.301038+0000","last_clean":"2023-10-07T09:44:24.301038+0000","last_became_active":"2023-10-07T09:44:24.296659+0000","last_became_peered":"2023-10-07T09:44:24.296659+0000","last_unstale":"2023-10-07T09:44:24.301038+0000","last_undegraded":"2023-10-07T09:44:24.301038+0000","last_fullsized":"2023-10-07T09:44:24.301038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:48.950118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"76'8","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310808+0000","last_change":"2023-10-07T09:44:23.299658+0000","last_active":"2023-10-07T09:44:24.310808+0000","last_peered":"2023-10-07T09:44:24.310808+0000","last_clean":"2023-10-07T09:44:24.310808+0000","last_became_active":"2023-10-07T09:44:23.299354+0000","last_became_peered":"2023-10-07T09:44:23.299354+0000","last_unstale":"2023-10-07T09:44:24.310808+0000","last_undegraded":"2023-10-07T09:44:24.310808+0000","last_fullsized":"2023-10-07T09:44:24.310808+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:50.186900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304712+0000","last_change":"2023-10-07T09:44:24.302800+0000","last_active":"2023-10-07T09:44:24.304712+0000","last_peered":"2023-10-07T09:44:24.304712+0000","last_clean":"2023-10-07T09:44:24.304712+0000","last_became_active":"2023-10-07T09:44:24.302516+0000","last_became_peered":"2023-10-07T09:44:24.302516+0000","last_unstale":"2023-10-07T09:44:24.304712+0000","last_undegraded":"2023-10-07T09:44:24.304712+0000","last_fullsized":"2023-10-07T09:44:24.304712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:26.584370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303208+0000","last_change":"2023-10-07T09:44:23.291240+0000","last_active":"2023-10-07T09:44:24.303208+0000","last_peered":"2023-10-07T09:44:24.303208+0000","last_clean":"2023-10-07T09:44:24.303208+0000","last_became_active":"2023-10-07T09:44:23.290942+0000","last_became_peered":"2023-10-07T09:44:23.290942+0000","last_unstale":"2023-10-07T09:44:24.303208+0000","last_undegraded":"2023-10-07T09:44:24.303208+0000","last_fullsized":"2023-10-07T09:44:24.303208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:55:09.563486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309141+0000","last_change":"2023-10-07T09:44:24.306770+0000","last_active":"2023-10-07T09:44:24.309141+0000","last_peered":"2023-10-07T09:44:24.309141+0000","last_clean":"2023-10-07T09:44:24.309141+0000","last_became_active":"2023-10-07T09:44:24.306496+0000","last_became_peered":"2023-10-07T09:44:24.306496+0000","last_unstale":"2023-10-07T09:44:24.309141+0000","last_undegraded":"2023-10-07T09:44:24.309141+0000","last_fullsized":"2023-10-07T09:44:24.309141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:21.662987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302436+0000","last_change":"2023-10-07T09:44:23.290435+0000","last_active":"2023-10-07T09:44:24.302436+0000","last_peered":"2023-10-07T09:44:24.302436+0000","last_clean":"2023-10-07T09:44:24.302436+0000","last_became_active":"2023-10-07T09:44:23.290166+0000","last_became_peered":"2023-10-07T09:44:23.290166+0000","last_unstale":"2023-10-07T09:44:24.302436+0000","last_undegraded":"2023-10-07T09:44:24.302436+0000","last_fullsized":"2023-10-07T09:44:24.302436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:06.098008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301437+0000","last_change":"2023-10-07T09:44:24.296481+0000","last_active":"2023-10-07T09:44:24.301437+0000","last_peered":"2023-10-07T09:44:24.301437+0000","last_clean":"2023-10-07T09:44:24.301437+0000","last_became_active":"2023-10-07T09:44:24.296201+0000","last_became_peered":"2023-10-07T09:44:24.296201+0000","last_unstale":"2023-10-07T09:44:24.301437+0000","last_undegraded":"2023-10-07T09:44:24.301437+0000","last_fullsized":"2023-10-07T09:44:24.301437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:59:30.551232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303990+0000","last_change":"2023-10-07T09:44:23.302051+0000","last_active":"2023-10-07T09:44:24.303990+0000","last_peered":"2023-10-07T09:44:24.303990+0000","last_clean":"2023-10-07T09:44:24.303990+0000","last_became_active":"2023-10-07T09:44:23.301738+0000","last_became_peered":"2023-10-07T09:44:23.301738+0000","last_unstale":"2023-10-07T09:44:24.303990+0000","last_undegraded":"2023-10-07T09:44:24.303990+0000","last_fullsized":"2023-10-07T09:44:24.303990+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:16.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310108+0000","last_change":"2023-10-07T09:44:24.306446+0000","last_active":"2023-10-07T09:44:24.310108+0000","last_peered":"2023-10-07T09:44:24.310108+0000","last_clean":"2023-10-07T09:44:24.310108+0000","last_became_active":"2023-10-07T09:44:24.306168+0000","last_became_peered":"2023-10-07T09:44:24.306168+0000","last_unstale":"2023-10-07T09:44:24.310108+0000","last_undegraded":"2023-10-07T09:44:24.310108+0000","last_fullsized":"2023-10-07T09:44:24.310108+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:12:07.230920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.285266+0000","last_change":"2023-10-07T09:44:23.284086+0000","last_active":"2023-10-07T09:44:24.285266+0000","last_peered":"2023-10-07T09:44:24.285266+0000","last_clean":"2023-10-07T09:44:24.285266+0000","last_became_active":"2023-10-07T09:44:23.283818+0000","last_became_peered":"2023-10-07T09:44:23.283818+0000","last_unstale":"2023-10-07T09:44:24.285266+0000","last_undegraded":"2023-10-07T09:44:24.285266+0000","last_fullsized":"2023-10-07T09:44:24.285266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:36.458090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310277+0000","last_change":"2023-10-07T09:44:24.308886+0000","last_active":"2023-10-07T09:44:24.310277+0000","last_peered":"2023-10-07T09:44:24.310277+0000","last_clean":"2023-10-07T09:44:24.310277+0000","last_became_active":"2023-10-07T09:44:24.308593+0000","last_became_peered":"2023-10-07T09:44:24.308593+0000","last_unstale":"2023-10-07T09:44:24.310277+0000","last_undegraded":"2023-10-07T09:44:24.310277+0000","last_fullsized":"2023-10-07T09:44:24.310277+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:44:01.882036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302482+0000","last_change":"2023-10-07T09:44:23.293905+0000","last_active":"2023-10-07T09:44:24.302482+0000","last_peered":"2023-10-07T09:44:24.302482+0000","last_clean":"2023-10-07T09:44:24.302482+0000","last_became_active":"2023-10-07T09:44:23.293655+0000","last_became_peered":"2023-10-07T09:44:23.293655+0000","last_unstale":"2023-10-07T09:44:24.302482+0000","last_undegraded":"2023-10-07T09:44:24.302482+0000","last_fullsized":"2023-10-07T09:44:24.302482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:45.046424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304066+0000","last_change":"2023-10-07T09:44:24.296850+0000","last_active":"2023-10-07T09:44:24.304066+0000","last_peered":"2023-10-07T09:44:24.304066+0000","last_clean":"2023-10-07T09:44:24.304066+0000","last_became_active":"2023-10-07T09:44:24.296473+0000","last_became_peered":"2023-10-07T09:44:24.296473+0000","last_unstale":"2023-10-07T09:44:24.304066+0000","last_undegraded":"2023-10-07T09:44:24.304066+0000","last_fullsized":"2023-10-07T09:44:24.304066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:01.336800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.281685+0000","last_change":"2023-10-07T09:44:23.290769+0000","last_active":"2023-10-07T09:44:24.281685+0000","last_peered":"2023-10-07T09:44:24.281685+0000","last_clean":"2023-10-07T09:44:24.281685+0000","last_became_active":"2023-10-07T09:44:23.290526+0000","last_became_peered":"2023-10-07T09:44:23.290526+0000","last_unstale":"2023-10-07T09:44:24.281685+0000","last_undegraded":"2023-10-07T09:44:24.281685+0000","last_fullsized":"2023-10-07T09:44:24.281685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:54.972045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307173+0000","last_change":"2023-10-07T09:44:24.304696+0000","last_active":"2023-10-07T09:44:24.307173+0000","last_peered":"2023-10-07T09:44:24.307173+0000","last_clean":"2023-10-07T09:44:24.307173+0000","last_became_active":"2023-10-07T09:44:24.304419+0000","last_became_peered":"2023-10-07T09:44:24.304419+0000","last_unstale":"2023-10-07T09:44:24.307173+0000","last_undegraded":"2023-10-07T09:44:24.307173+0000","last_fullsized":"2023-10-07T09:44:24.307173+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:34:55.415732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.289712+0000","last_change":"2023-10-07T09:44:23.296038+0000","last_active":"2023-10-07T09:44:24.289712+0000","last_peered":"2023-10-07T09:44:24.289712+0000","last_clean":"2023-10-07T09:44:24.289712+0000","last_became_active":"2023-10-07T09:44:23.295415+0000","last_became_peered":"2023-10-07T09:44:23.295415+0000","last_unstale":"2023-10-07T09:44:24.289712+0000","last_undegraded":"2023-10-07T09:44:24.289712+0000","last_fullsized":"2023-10-07T09:44:24.289712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:44.208709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.308912+0000","last_change":"2023-10-07T09:44:24.306296+0000","last_active":"2023-10-07T09:44:24.308912+0000","last_peered":"2023-10-07T09:44:24.308912+0000","last_clean":"2023-10-07T09:44:24.308912+0000","last_became_active":"2023-10-07T09:44:24.305992+0000","last_became_peered":"2023-10-07T09:44:24.305992+0000","last_unstale":"2023-10-07T09:44:24.308912+0000","last_undegraded":"2023-10-07T09:44:24.308912+0000","last_fullsized":"2023-10-07T09:44:24.308912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:41:28.132376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304668+0000","last_change":"2023-10-07T09:44:23.293710+0000","last_active":"2023-10-07T09:44:24.304668+0000","last_peered":"2023-10-07T09:44:24.304668+0000","last_clean":"2023-10-07T09:44:24.304668+0000","last_became_active":"2023-10-07T09:44:23.293451+0000","last_became_peered":"2023-10-07T09:44:23.293451+0000","last_unstale":"2023-10-07T09:44:24.304668+0000","last_undegraded":"2023-10-07T09:44:24.304668+0000","last_fullsized":"2023-10-07T09:44:24.304668+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:36.458090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307328+0000","last_change":"2023-10-07T09:44:24.303571+0000","last_active":"2023-10-07T09:44:24.307328+0000","last_peered":"2023-10-07T09:44:24.307328+0000","last_clean":"2023-10-07T09:44:24.307328+0000","last_became_active":"2023-10-07T09:44:24.303322+0000","last_became_peered":"2023-10-07T09:44:24.303322+0000","last_unstale":"2023-10-07T09:44:24.307328+0000","last_undegraded":"2023-10-07T09:44:24.307328+0000","last_fullsized":"2023-10-07T09:44:24.307328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:33:54.384857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.287823+0000","last_change":"2023-10-07T09:44:23.289459+0000","last_active":"2023-10-07T09:44:24.287823+0000","last_peered":"2023-10-07T09:44:24.287823+0000","last_clean":"2023-10-07T09:44:24.287823+0000","last_became_active":"2023-10-07T09:44:23.289123+0000","last_became_peered":"2023-10-07T09:44:23.289123+0000","last_unstale":"2023-10-07T09:44:24.287823+0000","last_undegraded":"2023-10-07T09:44:24.287823+0000","last_fullsized":"2023-10-07T09:44:24.287823+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:04:05.083808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304336+0000","last_change":"2023-10-07T09:44:24.299282+0000","last_active":"2023-10-07T09:44:24.304336+0000","last_peered":"2023-10-07T09:44:24.304336+0000","last_clean":"2023-10-07T09:44:24.304336+0000","last_became_active":"2023-10-07T09:44:24.298577+0000","last_became_peered":"2023-10-07T09:44:24.298577+0000","last_unstale":"2023-10-07T09:44:24.304336+0000","last_undegraded":"2023-10-07T09:44:24.304336+0000","last_fullsized":"2023-10-07T09:44:24.304336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:46.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.283081+0000","last_change":"2023-10-07T09:44:23.298822+0000","last_active":"2023-10-07T09:44:24.283081+0000","last_peered":"2023-10-07T09:44:24.283081+0000","last_clean":"2023-10-07T09:44:24.283081+0000","last_became_active":"2023-10-07T09:44:23.298575+0000","last_became_peered":"2023-10-07T09:44:23.298575+0000","last_unstale":"2023-10-07T09:44:24.283081+0000","last_undegraded":"2023-10-07T09:44:24.283081+0000","last_fullsized":"2023-10-07T09:44:24.283081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:02:22.668133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306101+0000","last_change":"2023-10-07T09:44:24.299298+0000","last_active":"2023-10-07T09:44:24.306101+0000","last_peered":"2023-10-07T09:44:24.306101+0000","last_clean":"2023-10-07T09:44:24.306101+0000","last_became_active":"2023-10-07T09:44:24.298836+0000","last_became_peered":"2023-10-07T09:44:24.298836+0000","last_unstale":"2023-10-07T09:44:24.306101+0000","last_undegraded":"2023-10-07T09:44:24.306101+0000","last_fullsized":"2023-10-07T09:44:24.306101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:02.073157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288889+0000","last_change":"2023-10-07T09:44:23.296333+0000","last_active":"2023-10-07T09:44:24.288889+0000","last_peered":"2023-10-07T09:44:24.288889+0000","last_clean":"2023-10-07T09:44:24.288889+0000","last_became_active":"2023-10-07T09:44:23.295143+0000","last_became_peered":"2023-10-07T09:44:23.295143+0000","last_unstale":"2023-10-07T09:44:24.288889+0000","last_undegraded":"2023-10-07T09:44:24.288889+0000","last_fullsized":"2023-10-07T09:44:24.288889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:55:09.563486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.294625+0000","last_change":"2023-10-07T09:44:24.291219+0000","last_active":"2023-10-07T09:44:24.294625+0000","last_peered":"2023-10-07T09:44:24.294625+0000","last_clean":"2023-10-07T09:44:24.294625+0000","last_became_active":"2023-10-07T09:44:24.290947+0000","last_became_peered":"2023-10-07T09:44:24.290947+0000","last_unstale":"2023-10-07T09:44:24.294625+0000","last_undegraded":"2023-10-07T09:44:24.294625+0000","last_fullsized":"2023-10-07T09:44:24.294625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:37:56.396212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.280280+0000","last_change":"2023-10-07T09:44:23.290267+0000","last_active":"2023-10-07T09:44:24.280280+0000","last_peered":"2023-10-07T09:44:24.280280+0000","last_clean":"2023-10-07T09:44:24.280280+0000","last_became_active":"2023-10-07T09:44:23.289997+0000","last_became_peered":"2023-10-07T09:44:23.289997+0000","last_unstale":"2023-10-07T09:44:24.280280+0000","last_undegraded":"2023-10-07T09:44:24.280280+0000","last_fullsized":"2023-10-07T09:44:24.280280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:25:50.787707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.300996+0000","last_change":"2023-10-07T09:44:24.296915+0000","last_active":"2023-10-07T09:44:24.300996+0000","last_peered":"2023-10-07T09:44:24.300996+0000","last_clean":"2023-10-07T09:44:24.300996+0000","last_became_active":"2023-10-07T09:44:24.296585+0000","last_became_peered":"2023-10-07T09:44:24.296585+0000","last_unstale":"2023-10-07T09:44:24.300996+0000","last_undegraded":"2023-10-07T09:44:24.300996+0000","last_fullsized":"2023-10-07T09:44:24.300996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:23.453707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.286508+0000","last_change":"2023-10-07T09:44:23.296699+0000","last_active":"2023-10-07T09:44:24.286508+0000","last_peered":"2023-10-07T09:44:24.286508+0000","last_clean":"2023-10-07T09:44:24.286508+0000","last_became_active":"2023-10-07T09:44:23.296307+0000","last_became_peered":"2023-10-07T09:44:23.296307+0000","last_unstale":"2023-10-07T09:44:24.286508+0000","last_undegraded":"2023-10-07T09:44:24.286508+0000","last_fullsized":"2023-10-07T09:44:24.286508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:45.052881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303841+0000","last_change":"2023-10-07T09:44:24.292140+0000","last_active":"2023-10-07T09:44:24.303841+0000","last_peered":"2023-10-07T09:44:24.303841+0000","last_clean":"2023-10-07T09:44:24.303841+0000","last_became_active":"2023-10-07T09:44:24.290209+0000","last_became_peered":"2023-10-07T09:44:24.290209+0000","last_unstale":"2023-10-07T09:44:24.303841+0000","last_undegraded":"2023-10-07T09:44:24.303841+0000","last_fullsized":"2023-10-07T09:44:24.303841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:11:32.068369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.280148+0000","last_change":"2023-10-07T09:44:23.286889+0000","last_active":"2023-10-07T09:44:24.280148+0000","last_peered":"2023-10-07T09:44:24.280148+0000","last_clean":"2023-10-07T09:44:24.280148+0000","last_became_active":"2023-10-07T09:44:23.286605+0000","last_became_peered":"2023-10-07T09:44:23.286605+0000","last_unstale":"2023-10-07T09:44:24.280148+0000","last_undegraded":"2023-10-07T09:44:24.280148+0000","last_fullsized":"2023-10-07T09:44:24.280148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:15:38.709944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.292402+0000","last_change":"2023-10-07T09:44:24.286461+0000","last_active":"2023-10-07T09:44:24.292402+0000","last_peered":"2023-10-07T09:44:24.292402+0000","last_clean":"2023-10-07T09:44:24.292402+0000","last_became_active":"2023-10-07T09:44:24.285822+0000","last_became_peered":"2023-10-07T09:44:24.285822+0000","last_unstale":"2023-10-07T09:44:24.292402+0000","last_undegraded":"2023-10-07T09:44:24.292402+0000","last_fullsized":"2023-10-07T09:44:24.292402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:53:32.891702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.277691+0000","last_change":"2023-10-07T09:44:23.293473+0000","last_active":"2023-10-07T09:44:24.277691+0000","last_peered":"2023-10-07T09:44:24.277691+0000","last_clean":"2023-10-07T09:44:24.277691+0000","last_became_active":"2023-10-07T09:44:23.293179+0000","last_became_peered":"2023-10-07T09:44:23.293179+0000","last_unstale":"2023-10-07T09:44:24.277691+0000","last_undegraded":"2023-10-07T09:44:24.277691+0000","last_fullsized":"2023-10-07T09:44:24.277691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:44.208709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307974+0000","last_change":"2023-10-07T09:44:24.299614+0000","last_active":"2023-10-07T09:44:24.307974+0000","last_peered":"2023-10-07T09:44:24.307974+0000","last_clean":"2023-10-07T09:44:24.307974+0000","last_became_active":"2023-10-07T09:44:24.299110+0000","last_became_peered":"2023-10-07T09:44:24.299110+0000","last_unstale":"2023-10-07T09:44:24.307974+0000","last_undegraded":"2023-10-07T09:44:24.307974+0000","last_fullsized":"2023-10-07T09:44:24.307974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:57:22.311791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.283347+0000","last_change":"2023-10-07T09:44:23.297144+0000","last_active":"2023-10-07T09:44:24.283347+0000","last_peered":"2023-10-07T09:44:24.283347+0000","last_clean":"2023-10-07T09:44:24.283347+0000","last_became_active":"2023-10-07T09:44:23.296828+0000","last_became_peered":"2023-10-07T09:44:23.296828+0000","last_unstale":"2023-10-07T09:44:24.283347+0000","last_undegraded":"2023-10-07T09:44:24.283347+0000","last_fullsized":"2023-10-07T09:44:24.283347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:16:51.707247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"67'161","reported_seq":365,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.282861+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:44:24.282861+0000","last_peered":"2023-10-07T09:44:24.282861+0000","last_clean":"2023-10-07T09:44:24.282861+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:44:24.282861+0000","last_undegraded":"2023-10-07T09:44:24.282861+0000","last_fullsized":"2023-10-07T09:44:24.282861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476920,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302872,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068968,"statfs":{"total":96636764160,"available":96326623232,"internally_reserved":0,"allocated":1523712,"data_stored":1023717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.46100000000000002,"5min":0.374,"15min":0.32300000000000001},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":1.026},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71899999999999997,"15min":0.70799999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.374},"max":{"1min":0.91300000000000003,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.82299999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.63800000000000001,"5min":0.42599999999999999,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":1.014,"15min":1.157},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.74099999999999999,"15min":0.73599999999999999},"min":{"1min":0.50800000000000001,"5min":0.49099999999999999,"15min":0.45600000000000002},"max":{"1min":0.91200000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.40699999999999997},"max":{"1min":0.82999999999999996,"5min":1.206,"15min":1.206},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.56699999999999995,"5min":0.504,"15min":0.439},"max":{"1min":0.85399999999999998,"5min":1.071,"15min":1.071},"last":0.85299999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72099999999999997,"15min":0.73299999999999998},"min":{"1min":0.54800000000000004,"5min":0.48899999999999999,"15min":0.46400000000000002},"max":{"1min":0.92300000000000004,"5min":0.96899999999999997,"15min":1.04},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.75600000000000001,"15min":0.76600000000000001},"min":{"1min":0.58699999999999997,"5min":0.53000000000000003,"15min":0.371},"max":{"1min":0.95599999999999996,"5min":1.137,"15min":1.3440000000000001},"last":0.83199999999999996}]},{"osd":4,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.48199999999999998,"5min":0.39600000000000002,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.88700000000000001,"15min":1.992},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64700000000000002,"15min":0.64600000000000002},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":1.6240000000000001},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64700000000000002,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92100000000000004,"5min":1.27,"15min":1.27},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.50800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.72499999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.747},"min":{"1min":0.59499999999999997,"5min":0.51200000000000001,"15min":0.46400000000000002},"max":{"1min":0.89700000000000002,"5min":1.0109999999999999,"15min":1.0289999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":0.89100000000000001,"5min":1.0229999999999999,"15min":1.107},"last":0.76000000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509624,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302284,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069556,"statfs":{"total":96636764160,"available":96327225344,"internally_reserved":0,"allocated":921600,"data_stored":433189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.52200000000000002,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":0.83299999999999996,"5min":0.96099999999999997,"15min":2.984},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.49199999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.94599999999999995,"15min":1.6000000000000001},"last":0.74299999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.54600000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":2.4529999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.72999999999999998},"min":{"1min":0.54300000000000004,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92900000000000005,"5min":0.99199999999999999,"15min":1.754},"last":0.67500000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71699999999999997,"15min":0.77100000000000002},"min":{"1min":0.61399999999999999,"5min":0.45700000000000002,"15min":0.44400000000000001},"max":{"1min":0.92100000000000004,"5min":0.96699999999999997,"15min":7.867},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.749,"15min":0.79700000000000004},"min":{"1min":0.624,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":7.7629999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.751,"15min":0.80800000000000005},"min":{"1min":0.56000000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92600000000000005,"5min":0.999,"15min":8.0399999999999991},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73999999999999999,"15min":0.77600000000000002},"min":{"1min":0.58299999999999996,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":0.90100000000000002,"5min":0.98099999999999998,"15min":2.7949999999999999},"last":0.78700000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.627,"15min":0.63},"min":{"1min":0.46100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":1.296},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63700000000000001,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.95599999999999996,"5min":0.97899999999999998,"15min":1.105},"last":0.76600000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.4119999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71199999999999997,"15min":0.72199999999999998},"min":{"1min":0.49399999999999999,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":1.0089999999999999,"5min":1.0449999999999999,"15min":1.1779999999999999},"last":0.84699999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67900000000000005,"15min":0.70099999999999996},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.5049999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.75,"15min":0.77200000000000002},"min":{"1min":0.56399999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97999999999999998,"5min":0.995,"15min":1.3220000000000001},"last":0.90100000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542326,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302276,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069564,"statfs":{"total":96636764160,"available":96327233536,"internally_reserved":0,"allocated":913408,"data_stored":426615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.36099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":1.585},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.55000000000000004,"15min":0.55700000000000005},"min":{"1min":0.51800000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.79800000000000004,"5min":0.84199999999999997,"15min":1.6080000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.46500000000000002,"15min":0.48599999999999999},"min":{"1min":0.41699999999999998,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.621,"5min":0.621,"15min":1.468},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54500000000000004,"15min":0.56200000000000006},"min":{"1min":0.39100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":1.5389999999999999},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.60299999999999998},"min":{"1min":0.46300000000000002,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.73299999999999998,"5min":0.75900000000000001,"15min":1.663},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":1.1140000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.77100000000000002,"15min":0.78400000000000003},"min":{"1min":0.68500000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0189999999999999,"5min":1.115,"15min":1.4259999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.64100000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96499999999999997,"5min":1.044,"15min":1.766},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.81599999999999995,"15min":0.81699999999999995},"min":{"1min":0.72499999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0660000000000001,"5min":1.099,"15min":1.3320000000000001},"last":0.626},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.61499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.5149999999999999},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.79800000000000004,"15min":0.82499999999999996},"min":{"1min":0.63800000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0109999999999999,"5min":1.0549999999999999,"15min":1.627},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81100000000000005,"15min":0.82999999999999996},"min":{"1min":0.65000000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0700000000000001,"5min":1.1040000000000001,"15min":1.3700000000000001},"last":0.67300000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.68200000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0229999999999999,"5min":1.0900000000000001,"15min":1.706},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83299999999999996,"15min":0.85199999999999998},"min":{"1min":0.68799999999999994,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0409999999999999,"5min":1.137,"15min":1.554},"last":0.69899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542326,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302880,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068960,"statfs":{"total":96636764160,"available":96326615040,"internally_reserved":0,"allocated":1531904,"data_stored":1025647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49199999999999999,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66300000000000003,"5min":0.73099999999999998,"15min":2.726},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.59099999999999997},"min":{"1min":0.44700000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":2.552},"last":0.52800000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.32800000000000001},"max":{"1min":0.68400000000000005,"5min":1.101,"15min":2.4910000000000001},"last":0.621},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.53900000000000003},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":3.629},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53100000000000003,"15min":0.55600000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.33900000000000002},"max":{"1min":0.71399999999999997,"5min":1.0009999999999999,"15min":2.6909999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61199999999999999,"15min":0.626},"min":{"1min":0.45100000000000001,"5min":0.42199999999999999,"15min":0.40200000000000002},"max":{"1min":0.76700000000000002,"5min":0.89500000000000002,"15min":2.5209999999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.56000000000000005,"5min":0.497,"15min":0.45900000000000002},"max":{"1min":0.97799999999999998,"5min":1.232,"15min":2.637},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.117,"15min":2.4180000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78400000000000003,"15min":0.81699999999999995},"min":{"1min":0.623,"5min":0.504,"15min":0.501},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":3.8199999999999998},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.80200000000000005,"15min":0.81899999999999995},"min":{"1min":0.68500000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.03,"5min":1.194,"15min":2.6000000000000001},"last":0.69599999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81399999999999995,"15min":0.83099999999999996},"min":{"1min":0.69099999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0069999999999999,"5min":1.129,"15min":2.3740000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81899999999999995,"15min":0.84699999999999998},"min":{"1min":0.60799999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.206,"5min":1.206,"15min":3.855},"last":0.81499999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":0.84599999999999997},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.56200000000000006},"max":{"1min":1.0920000000000001,"5min":1.167,"15min":2.4550000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83199999999999996,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.238,"5min":1.238,"15min":3.778},"last":0.72899999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542327,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310484,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061356,"statfs":{"total":96636764160,"available":96318828544,"internally_reserved":0,"allocated":929792,"data_stored":433245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46200000000000002,"15min":0.45100000000000001},"min":{"1min":0.33900000000000002,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.63500000000000001,"5min":1.258,"15min":1.258},"last":0.314},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.376,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.62,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.55600000000000005},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.33200000000000002},"max":{"1min":0.71599999999999997,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.60299999999999998},"min":{"1min":0.53100000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.371,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.502},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.58499999999999996},"min":{"1min":0.46000000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.75,"5min":1.052,"15min":1.052},"last":0.45300000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.53400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0660000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81399999999999995,"15min":0.79900000000000004},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.42099999999999999},"max":{"1min":1.327,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.80800000000000005},"min":{"1min":0.58199999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.173,"5min":1.98,"15min":1.98},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79600000000000004,"15min":0.79000000000000004},"min":{"1min":0.59999999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.208,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.84299999999999997,"15min":0.82899999999999996},"min":{"1min":0.63100000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1379999999999999,"5min":1.931,"15min":1.931},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.84299999999999997,"15min":0.83099999999999996},"min":{"1min":0.66900000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2849999999999999,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85399999999999998,"15min":0.84099999999999997},"min":{"1min":0.63800000000000001,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.2450000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84399999999999997,"15min":0.82799999999999996},"min":{"1min":0.70499999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"max":{"1min":1.0620000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.78400000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542328,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306376,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065464,"statfs":{"total":96636764160,"available":96323035136,"internally_reserved":0,"allocated":917504,"data_stored":426581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.25600000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":3.3980000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.503,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.274},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":3.1299999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.78400000000000003,"5min":0.80100000000000005,"15min":3.4500000000000002},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.438,"5min":0.317,"15min":0.317},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":3.2309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.40600000000000003,"5min":0.39800000000000002,"15min":0.377},"max":{"1min":0.72199999999999998,"5min":0.79800000000000004,"15min":3.0910000000000002},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.52500000000000002,"5min":0.497,"15min":0.44900000000000001},"max":{"1min":0.94499999999999995,"5min":1.145,"15min":1.518},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.59899999999999998,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":1.103,"15min":1.3799999999999999},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.65300000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.3240000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.78700000000000003,"15min":0.79000000000000004},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.50600000000000001},"max":{"1min":1.0640000000000001,"5min":1.1899999999999999,"15min":1.5189999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.202,"5min":1.202,"15min":1.413},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.55700000000000005,"15min":0.51000000000000001},"max":{"1min":0.97999999999999998,"5min":1.1459999999999999,"15min":1.5840000000000001},"last":0.67800000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81899999999999995,"15min":0.82099999999999995},"min":{"1min":0.70699999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.143,"5min":1.163,"15min":1.45},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.872,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.746,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1040000000000001,"5min":1.1140000000000001,"15min":1.5940000000000001},"last":0.626}]}]},{"osd":4,"up_from":15,"seq":64424509624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306392,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065448,"statfs":{"total":96636764160,"available":96323018752,"internally_reserved":0,"allocated":933888,"data_stored":434725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.88100000000000001,"5min":1.069,"15min":1.069},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.71499999999999997,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":0.91300000000000003,"5min":1.0009999999999999,"15min":2.3420000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71599999999999997,"15min":0.70699999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.873,"5min":0.96599999999999997,"15min":1.054},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73799999999999999,"15min":0.72399999999999998},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.35299999999999998},"max":{"1min":0.98999999999999999,"5min":1.478,"15min":1.478},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.75900000000000001,"15min":0.752},"min":{"1min":0.55000000000000004,"5min":0.48799999999999999,"15min":0.432},"max":{"1min":0.95399999999999996,"5min":1.518,"15min":2.3239999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.52500000000000002,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.71799999999999997},"min":{"1min":0.55700000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.85599999999999998,"5min":0.98699999999999999,"15min":1.032},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.76200000000000001,"15min":0.75},"min":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.496},"max":{"1min":0.93200000000000005,"5min":1.036,"15min":1.036},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.627,"15min":0.61499999999999999},"min":{"1min":0.504,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.873,"5min":1.639,"15min":1.639},"last":0.41899999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.63900000000000001},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.98199999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.55700000000000005,"5min":0.46600000000000003,"15min":0.46300000000000002},"max":{"1min":0.93200000000000005,"5min":1.585,"15min":1.585},"last":0.72499999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.52800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.94799999999999995,"5min":1.024,"15min":1.0589999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.753},"min":{"1min":0.64200000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90100000000000002,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.75700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542326,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302280,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069560,"statfs":{"total":96636764160,"available":96327229440,"internally_reserved":0,"allocated":917504,"data_stored":427149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.76600000000000001,"15min":0.753},"min":{"1min":0.68000000000000005,"5min":0.49199999999999999,"15min":0.39900000000000002},"max":{"1min":1.01,"5min":1.218,"15min":1.218},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79700000000000004,"15min":0.79600000000000004},"min":{"1min":0.65800000000000003,"5min":0.53000000000000003,"15min":0.436},"max":{"1min":0.99199999999999999,"5min":1.294,"15min":1.294},"last":0.82099999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.73399999999999999,"15min":0.72899999999999998},"min":{"1min":0.63400000000000001,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":1.048,"5min":1.1339999999999999,"15min":1.3049999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.72199999999999998,"15min":0.76500000000000001},"min":{"1min":0.54700000000000004,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":1.0249999999999999,"5min":1.194,"15min":6.9720000000000004},"last":0.84999999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.748},"min":{"1min":0.629,"5min":0.47699999999999998,"15min":0.41099999999999998},"max":{"1min":0.94399999999999995,"5min":1.119,"15min":5.9299999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.74399999999999999,"15min":0.748},"min":{"1min":0.60199999999999998,"5min":0.48499999999999999,"15min":0.42299999999999999},"max":{"1min":0.93999999999999995,"5min":1.0049999999999999,"15min":1.704},"last":0.78800000000000003}]},{"osd":3,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.77300000000000002,"15min":0.76900000000000002},"min":{"1min":0.63100000000000001,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":1.0569999999999999,"5min":1.2589999999999999,"15min":1.899},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73899999999999999,"15min":0.78000000000000003},"min":{"1min":0.58499999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.02,"5min":1.036,"15min":6.0620000000000003},"last":0.94999999999999996}]},{"osd":4,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.874,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63,"15min":0.629},"min":{"1min":0.47699999999999998,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.621}]},{"osd":6,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64800000000000002,"15min":0.64200000000000002},"min":{"1min":0.44700000000000001,"5min":0.39700000000000002,"15min":0.33900000000000002},"max":{"1min":0.93300000000000005,"5min":0.998,"15min":0.998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.497,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.91600000000000004,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71299999999999997,"15min":0.70599999999999996},"min":{"1min":0.503,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77000000000000002,"15min":0.76700000000000002},"min":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.95799999999999996,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:44:28.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:44:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.934+0000 7f4733fff700 1 Processor -- start 2023-10-07T09:44:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.934+0000 7f4733fff700 1 -- start start 2023-10-07T09:44:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.935+0000 7f4733fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f472c0c6470 0x7f472c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.935+0000 7f4733fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0cefb0 con 0x7f472c159750 2023-10-07T09:44:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.935+0000 7f4733fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0cbf20 con 0x7f472c159b20 2023-10-07T09:44:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.935+0000 7f4733fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0cc200 con 0x7f472c0c6470 2023-10-07T09:44:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.935+0000 7f4731d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f472c159750 0x7f472c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58048/0 (socket says 172.21.15.29:58048) 2023-10-07T09:44:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.935+0000 7f4731d9b700 1 -- 172.21.15.29:0/1645246514 learned_addr learned my addr 172.21.15.29:0/1645246514 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f473259c700 1 --2- 172.21.15.29:0/1645246514 >> v2:172.21.15.144:3300/0 conn(0x7f472c0c6470 0x7f472c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831596354 0 0) 0x7f472c0cefb0 con 0x7f472c159750 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4718003570 con 0x7f472c159750 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040942865 0 0) 0x7f472c0cbf20 con 0x7f472c159b20 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f472c0cefb0 con 0x7f472c159b20 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f473259c700 1 -- 172.21.15.29:0/1645246514 >> v1:172.21.15.144:6789/0 conn(0x7f472c159b20 legacy=0x7f472c0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f473259c700 1 -- 172.21.15.29:0/1645246514 >> v1:172.21.15.29:6789/0 conn(0x7f472c159750 legacy=0x7f472c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f473259c700 1 -- 172.21.15.29:0/1645246514 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0cc4e0 con 0x7f472c0c6470 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.936+0000 7f473259c700 1 --2- 172.21.15.29:0/1645246514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c0d10a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e60 tx=0x7f4724009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cae46aff08cacb51 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472400e070 con 0x7f472c0c6470 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472400bab0 con 0x7f472c0c6470 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4730d99700 1 -- 172.21.15.29:0/1645246514 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472400be70 con 0x7f472c0c6470 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 -- 172.21.15.29:0/1645246514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 msgr2=0x7f472c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 --2- 172.21.15.29:0/1645246514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c0d10a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e60 tx=0x7f4724009780 comp rx=0 tx=0).stop 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 -- 172.21.15.29:0/1645246514 shutdown_connections 2023-10-07T09:44:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 --2- 172.21.15.29:0/1645246514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c0d10a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 -- 172.21.15.29:0/1645246514 >> 172.21.15.29:0/1645246514 conn(0x7f472c063a10 msgr2=0x7f472c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 -- 172.21.15.29:0/1645246514 shutdown_connections 2023-10-07T09:44:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.937+0000 7f4733fff700 1 -- 172.21.15.29:0/1645246514 wait complete. 2023-10-07T09:44:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.939+0000 7f4733fff700 1 Processor -- start 2023-10-07T09:44:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.939+0000 7f4733fff700 1 -- start start 2023-10-07T09:44:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4733fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c14a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4733fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 0x7f472c0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4733fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472c159b20 0x7f472c1557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4733fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b3fb0 con 0x7f472c159750 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4733fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c155310 con 0x7f472c0c6470 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4733fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c154740 con 0x7f472c159b20 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f473159a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 0x7f472c0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f473159a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 0x7f472c0ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42648/0 (socket says 172.21.15.29:42648) 2023-10-07T09:44:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f473159a700 1 -- 172.21.15.29:0/2221377012 learned_addr learned my addr 172.21.15.29:0/2221377012 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f473259c700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472c159b20 0x7f472c1557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.940+0000 7f4731d9b700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c14a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f473159a700 1 -- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472c159b20 msgr2=0x7f472c1557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f473159a700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472c159b20 0x7f472c1557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f473159a700 1 -- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 msgr2=0x7f472c14a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f473159a700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c14a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f473159a700 1 -- 172.21.15.29:0/2221377012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724009a20 con 0x7f472c159750 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f473159a700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 0x7f472c0ca000 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c7c0 tx=0x7f471c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c016070 con 0x7f472c159750 2023-10-07T09:44:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c011680 con 0x7f472c159750 2023-10-07T09:44:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c1549e0 con 0x7f472c159750 2023-10-07T09:44:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c011a20 con 0x7f472c159750 2023-10-07T09:44:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.941+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c154fa0 con 0x7f472c159750 2023-10-07T09:44:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.942+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c155480 con 0x7f472c159750 2023-10-07T09:44:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.943+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f471c012440 con 0x7f472c159750 2023-10-07T09:44:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.944+0000 7f4721ffb700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4710066f30 0x7f47100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.944+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f471c0845a0 con 0x7f472c159750 2023-10-07T09:44:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.944+0000 7f4731d9b700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4710066f30 0x7f47100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:28.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.945+0000 7f4731d9b700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4710066f30 0x7f47100693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4728001360 tx=0x7f4728009040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:28.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:28.948+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f471c0515f0 con 0x7f472c159750 2023-10-07T09:44:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.146+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f472c04c3c0 con 0x7f4710066f30 2023-10-07T09:44:29.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.155+0000 7f4721ffb700 1 -- 172.21.15.29:0/2221377012 <== mgr.5899 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288902 (secure 0 0 0) 0x7f472c04c3c0 con 0x7f4710066f30 2023-10-07T09:44:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:29.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.158+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4710066f30 msgr2=0x7f47100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.158+0000 7f4733fff700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4710066f30 0x7f47100693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4728001360 tx=0x7f4728009040 comp rx=0 tx=0).stop 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.158+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 msgr2=0x7f472c0ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.158+0000 7f4733fff700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 0x7f472c0ca000 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c7c0 tx=0x7f471c007dd0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 shutdown_connections 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4710066f30 0x7f47100693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472c159b20 0x7f472c1557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f472c0c6470 0x7f472c14a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 --2- 172.21.15.29:0/2221377012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472c159750 0x7f472c0ca000 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 >> 172.21.15.29:0/2221377012 conn(0x7f472c063a10 msgr2=0x7f472c157080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 shutdown_connections 2023-10-07T09:44:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.159+0000 7f4733fff700 1 -- 172.21.15.29:0/2221377012 wait complete. 2023-10-07T09:44:29.163 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:44:29.177 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:44:28.380985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2435844,"kb_used_data":8388,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752538876,"statfs":{"total":773094113280,"available":770599809024,"internally_reserved":0,"allocated":8589312,"data_stored":4630868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001170"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.313081+0000","last_change":"2023-10-07T09:44:24.310970+0000","last_active":"2023-10-07T09:44:24.313081+0000","last_peered":"2023-10-07T09:44:24.313081+0000","last_clean":"2023-10-07T09:44:24.313081+0000","last_became_active":"2023-10-07T09:44:24.310728+0000","last_became_peered":"2023-10-07T09:44:24.310728+0000","last_unstale":"2023-10-07T09:44:24.313081+0000","last_undegraded":"2023-10-07T09:44:24.313081+0000","last_fullsized":"2023-10-07T09:44:24.313081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:08:23.587424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.294355+0000","last_change":"2023-10-07T09:44:24.291115+0000","last_active":"2023-10-07T09:44:24.294355+0000","last_peered":"2023-10-07T09:44:24.294355+0000","last_clean":"2023-10-07T09:44:24.294355+0000","last_became_active":"2023-10-07T09:44:24.290853+0000","last_became_peered":"2023-10-07T09:44:24.290853+0000","last_unstale":"2023-10-07T09:44:24.294355+0000","last_undegraded":"2023-10-07T09:44:24.294355+0000","last_fullsized":"2023-10-07T09:44:24.294355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:21:10.094242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302301+0000","last_change":"2023-10-07T09:44:24.292751+0000","last_active":"2023-10-07T09:44:24.302301+0000","last_peered":"2023-10-07T09:44:24.302301+0000","last_clean":"2023-10-07T09:44:24.302301+0000","last_became_active":"2023-10-07T09:44:24.292426+0000","last_became_peered":"2023-10-07T09:44:24.292426+0000","last_unstale":"2023-10-07T09:44:24.302301+0000","last_undegraded":"2023-10-07T09:44:24.302301+0000","last_fullsized":"2023-10-07T09:44:24.302301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:35:09.290288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309215+0000","last_change":"2023-10-07T09:44:24.307582+0000","last_active":"2023-10-07T09:44:24.309215+0000","last_peered":"2023-10-07T09:44:24.309215+0000","last_clean":"2023-10-07T09:44:24.309215+0000","last_became_active":"2023-10-07T09:44:24.307167+0000","last_became_peered":"2023-10-07T09:44:24.307167+0000","last_unstale":"2023-10-07T09:44:24.309215+0000","last_undegraded":"2023-10-07T09:44:24.309215+0000","last_fullsized":"2023-10-07T09:44:24.309215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:01:24.933011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311192+0000","last_change":"2023-10-07T09:44:24.308931+0000","last_active":"2023-10-07T09:44:24.311192+0000","last_peered":"2023-10-07T09:44:24.311192+0000","last_clean":"2023-10-07T09:44:24.311192+0000","last_became_active":"2023-10-07T09:44:24.308628+0000","last_became_peered":"2023-10-07T09:44:24.308628+0000","last_unstale":"2023-10-07T09:44:24.311192+0000","last_undegraded":"2023-10-07T09:44:24.311192+0000","last_fullsized":"2023-10-07T09:44:24.311192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:40:50.626505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311378+0000","last_change":"2023-10-07T09:44:24.306453+0000","last_active":"2023-10-07T09:44:24.311378+0000","last_peered":"2023-10-07T09:44:24.311378+0000","last_clean":"2023-10-07T09:44:24.311378+0000","last_became_active":"2023-10-07T09:44:24.306185+0000","last_became_peered":"2023-10-07T09:44:24.306185+0000","last_unstale":"2023-10-07T09:44:24.311378+0000","last_undegraded":"2023-10-07T09:44:24.311378+0000","last_fullsized":"2023-10-07T09:44:24.311378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:45.216789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301545+0000","last_change":"2023-10-07T09:44:24.295414+0000","last_active":"2023-10-07T09:44:24.301545+0000","last_peered":"2023-10-07T09:44:24.301545+0000","last_clean":"2023-10-07T09:44:24.301545+0000","last_became_active":"2023-10-07T09:44:24.295058+0000","last_became_peered":"2023-10-07T09:44:24.295058+0000","last_unstale":"2023-10-07T09:44:24.301545+0000","last_undegraded":"2023-10-07T09:44:24.301545+0000","last_fullsized":"2023-10-07T09:44:24.301545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:02.073157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.297758+0000","last_change":"2023-10-07T09:44:24.292891+0000","last_active":"2023-10-07T09:44:24.297758+0000","last_peered":"2023-10-07T09:44:24.297758+0000","last_clean":"2023-10-07T09:44:24.297758+0000","last_became_active":"2023-10-07T09:44:24.292567+0000","last_became_peered":"2023-10-07T09:44:24.292567+0000","last_unstale":"2023-10-07T09:44:24.297758+0000","last_undegraded":"2023-10-07T09:44:24.297758+0000","last_fullsized":"2023-10-07T09:44:24.297758+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:42:15.921547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.305376+0000","last_change":"2023-10-07T09:44:24.304246+0000","last_active":"2023-10-07T09:44:24.305376+0000","last_peered":"2023-10-07T09:44:24.305376+0000","last_clean":"2023-10-07T09:44:24.305376+0000","last_became_active":"2023-10-07T09:44:24.303933+0000","last_became_peered":"2023-10-07T09:44:24.303933+0000","last_unstale":"2023-10-07T09:44:24.305376+0000","last_undegraded":"2023-10-07T09:44:24.305376+0000","last_fullsized":"2023-10-07T09:44:24.305376+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:01.660743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306302+0000","last_change":"2023-10-07T09:44:24.291982+0000","last_active":"2023-10-07T09:44:24.306302+0000","last_peered":"2023-10-07T09:44:24.306302+0000","last_clean":"2023-10-07T09:44:24.306302+0000","last_became_active":"2023-10-07T09:44:24.291630+0000","last_became_peered":"2023-10-07T09:44:24.291630+0000","last_unstale":"2023-10-07T09:44:24.306302+0000","last_undegraded":"2023-10-07T09:44:24.306302+0000","last_fullsized":"2023-10-07T09:44:24.306302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:44:38.869997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.305027+0000","last_change":"2023-10-07T09:44:24.293784+0000","last_active":"2023-10-07T09:44:24.305027+0000","last_peered":"2023-10-07T09:44:24.305027+0000","last_clean":"2023-10-07T09:44:24.305027+0000","last_became_active":"2023-10-07T09:44:24.293523+0000","last_became_peered":"2023-10-07T09:44:24.293523+0000","last_unstale":"2023-10-07T09:44:24.305027+0000","last_undegraded":"2023-10-07T09:44:24.305027+0000","last_fullsized":"2023-10-07T09:44:24.305027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:07.649574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.308055+0000","last_change":"2023-10-07T09:44:24.304149+0000","last_active":"2023-10-07T09:44:24.308055+0000","last_peered":"2023-10-07T09:44:24.308055+0000","last_clean":"2023-10-07T09:44:24.308055+0000","last_became_active":"2023-10-07T09:44:24.303708+0000","last_became_peered":"2023-10-07T09:44:24.303708+0000","last_unstale":"2023-10-07T09:44:24.308055+0000","last_undegraded":"2023-10-07T09:44:24.308055+0000","last_fullsized":"2023-10-07T09:44:24.308055+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:49:05.505574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303324+0000","last_change":"2023-10-07T09:44:24.301270+0000","last_active":"2023-10-07T09:44:24.303324+0000","last_peered":"2023-10-07T09:44:24.303324+0000","last_clean":"2023-10-07T09:44:24.303324+0000","last_became_active":"2023-10-07T09:44:24.300834+0000","last_became_peered":"2023-10-07T09:44:24.300834+0000","last_unstale":"2023-10-07T09:44:24.303324+0000","last_undegraded":"2023-10-07T09:44:24.303324+0000","last_fullsized":"2023-10-07T09:44:24.303324+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:50:13.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306351+0000","last_change":"2023-10-07T09:44:24.305487+0000","last_active":"2023-10-07T09:44:24.306351+0000","last_peered":"2023-10-07T09:44:24.306351+0000","last_clean":"2023-10-07T09:44:24.306351+0000","last_became_active":"2023-10-07T09:44:24.305161+0000","last_became_peered":"2023-10-07T09:44:24.305161+0000","last_unstale":"2023-10-07T09:44:24.306351+0000","last_undegraded":"2023-10-07T09:44:24.306351+0000","last_fullsized":"2023-10-07T09:44:24.306351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:01.336800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.295559+0000","last_change":"2023-10-07T09:44:24.293709+0000","last_active":"2023-10-07T09:44:24.295559+0000","last_peered":"2023-10-07T09:44:24.295559+0000","last_clean":"2023-10-07T09:44:24.295559+0000","last_became_active":"2023-10-07T09:44:24.293405+0000","last_became_peered":"2023-10-07T09:44:24.293405+0000","last_unstale":"2023-10-07T09:44:24.295559+0000","last_undegraded":"2023-10-07T09:44:24.295559+0000","last_fullsized":"2023-10-07T09:44:24.295559+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:49:53.054391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307302+0000","last_change":"2023-10-07T09:44:24.297405+0000","last_active":"2023-10-07T09:44:24.307302+0000","last_peered":"2023-10-07T09:44:24.307302+0000","last_clean":"2023-10-07T09:44:24.307302+0000","last_became_active":"2023-10-07T09:44:24.296841+0000","last_became_peered":"2023-10-07T09:44:24.296841+0000","last_unstale":"2023-10-07T09:44:24.307302+0000","last_undegraded":"2023-10-07T09:44:24.307302+0000","last_fullsized":"2023-10-07T09:44:24.307302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:49:53.054391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.292717+0000","last_change":"2023-10-07T09:44:24.286094+0000","last_active":"2023-10-07T09:44:24.292717+0000","last_peered":"2023-10-07T09:44:24.292717+0000","last_clean":"2023-10-07T09:44:24.292717+0000","last_became_active":"2023-10-07T09:44:24.285739+0000","last_became_peered":"2023-10-07T09:44:24.285739+0000","last_unstale":"2023-10-07T09:44:24.292717+0000","last_undegraded":"2023-10-07T09:44:24.292717+0000","last_fullsized":"2023-10-07T09:44:24.292717+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:37:52.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309212+0000","last_change":"2023-10-07T09:44:24.303692+0000","last_active":"2023-10-07T09:44:24.309212+0000","last_peered":"2023-10-07T09:44:24.309212+0000","last_clean":"2023-10-07T09:44:24.309212+0000","last_became_active":"2023-10-07T09:44:24.303407+0000","last_became_peered":"2023-10-07T09:44:24.303407+0000","last_unstale":"2023-10-07T09:44:24.309212+0000","last_undegraded":"2023-10-07T09:44:24.309212+0000","last_fullsized":"2023-10-07T09:44:24.309212+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:20:53.802952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312681+0000","last_change":"2023-10-07T09:44:24.310459+0000","last_active":"2023-10-07T09:44:24.312681+0000","last_peered":"2023-10-07T09:44:24.312681+0000","last_clean":"2023-10-07T09:44:24.312681+0000","last_became_active":"2023-10-07T09:44:24.310206+0000","last_became_peered":"2023-10-07T09:44:24.310206+0000","last_unstale":"2023-10-07T09:44:24.312681+0000","last_undegraded":"2023-10-07T09:44:24.312681+0000","last_fullsized":"2023-10-07T09:44:24.312681+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:30.263388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.308898+0000","last_change":"2023-10-07T09:44:24.300426+0000","last_active":"2023-10-07T09:44:24.308898+0000","last_peered":"2023-10-07T09:44:24.308898+0000","last_clean":"2023-10-07T09:44:24.308898+0000","last_became_active":"2023-10-07T09:44:24.300169+0000","last_became_peered":"2023-10-07T09:44:24.300169+0000","last_unstale":"2023-10-07T09:44:24.308898+0000","last_undegraded":"2023-10-07T09:44:24.308898+0000","last_fullsized":"2023-10-07T09:44:24.308898+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:38:12.002708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301559+0000","last_change":"2023-10-07T09:44:24.299002+0000","last_active":"2023-10-07T09:44:24.301559+0000","last_peered":"2023-10-07T09:44:24.301559+0000","last_clean":"2023-10-07T09:44:24.301559+0000","last_became_active":"2023-10-07T09:44:24.297952+0000","last_became_peered":"2023-10-07T09:44:24.297952+0000","last_unstale":"2023-10-07T09:44:24.301559+0000","last_undegraded":"2023-10-07T09:44:24.301559+0000","last_fullsized":"2023-10-07T09:44:24.301559+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:17:26.117441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312630+0000","last_change":"2023-10-07T09:44:24.310885+0000","last_active":"2023-10-07T09:44:24.312630+0000","last_peered":"2023-10-07T09:44:24.312630+0000","last_clean":"2023-10-07T09:44:24.312630+0000","last_became_active":"2023-10-07T09:44:24.310465+0000","last_became_peered":"2023-10-07T09:44:24.310465+0000","last_unstale":"2023-10-07T09:44:24.312630+0000","last_undegraded":"2023-10-07T09:44:24.312630+0000","last_fullsized":"2023-10-07T09:44:24.312630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:43.243147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301055+0000","last_change":"2023-10-07T09:44:24.297591+0000","last_active":"2023-10-07T09:44:24.301055+0000","last_peered":"2023-10-07T09:44:24.301055+0000","last_clean":"2023-10-07T09:44:24.301055+0000","last_became_active":"2023-10-07T09:44:24.297013+0000","last_became_peered":"2023-10-07T09:44:24.297013+0000","last_unstale":"2023-10-07T09:44:24.301055+0000","last_undegraded":"2023-10-07T09:44:24.301055+0000","last_fullsized":"2023-10-07T09:44:24.301055+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:58:16.034746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303391+0000","last_change":"2023-10-07T09:44:24.301329+0000","last_active":"2023-10-07T09:44:24.303391+0000","last_peered":"2023-10-07T09:44:24.303391+0000","last_clean":"2023-10-07T09:44:24.303391+0000","last_became_active":"2023-10-07T09:44:24.300891+0000","last_became_peered":"2023-10-07T09:44:24.300891+0000","last_unstale":"2023-10-07T09:44:24.303391+0000","last_undegraded":"2023-10-07T09:44:24.303391+0000","last_fullsized":"2023-10-07T09:44:24.303391+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:44:38.869997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309149+0000","last_change":"2023-10-07T09:44:24.307494+0000","last_active":"2023-10-07T09:44:24.309149+0000","last_peered":"2023-10-07T09:44:24.309149+0000","last_clean":"2023-10-07T09:44:24.309149+0000","last_became_active":"2023-10-07T09:44:24.307216+0000","last_became_peered":"2023-10-07T09:44:24.307216+0000","last_unstale":"2023-10-07T09:44:24.309149+0000","last_undegraded":"2023-10-07T09:44:24.309149+0000","last_fullsized":"2023-10-07T09:44:24.309149+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:07:19.074986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306174+0000","last_change":"2023-10-07T09:44:24.302355+0000","last_active":"2023-10-07T09:44:24.306174+0000","last_peered":"2023-10-07T09:44:24.306174+0000","last_clean":"2023-10-07T09:44:24.306174+0000","last_became_active":"2023-10-07T09:44:24.302038+0000","last_became_peered":"2023-10-07T09:44:24.302038+0000","last_unstale":"2023-10-07T09:44:24.306174+0000","last_undegraded":"2023-10-07T09:44:24.306174+0000","last_fullsized":"2023-10-07T09:44:24.306174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:58:25.295458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306082+0000","last_change":"2023-10-07T09:44:24.299160+0000","last_active":"2023-10-07T09:44:24.306082+0000","last_peered":"2023-10-07T09:44:24.306082+0000","last_clean":"2023-10-07T09:44:24.306082+0000","last_became_active":"2023-10-07T09:44:24.298619+0000","last_became_peered":"2023-10-07T09:44:24.298619+0000","last_unstale":"2023-10-07T09:44:24.306082+0000","last_undegraded":"2023-10-07T09:44:24.306082+0000","last_fullsized":"2023-10-07T09:44:24.306082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:15:41.028294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307882+0000","last_change":"2023-10-07T09:44:24.304159+0000","last_active":"2023-10-07T09:44:24.307882+0000","last_peered":"2023-10-07T09:44:24.307882+0000","last_clean":"2023-10-07T09:44:24.307882+0000","last_became_active":"2023-10-07T09:44:24.303825+0000","last_became_peered":"2023-10-07T09:44:24.303825+0000","last_unstale":"2023-10-07T09:44:24.307882+0000","last_undegraded":"2023-10-07T09:44:24.307882+0000","last_fullsized":"2023-10-07T09:44:24.307882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:31:14.028736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302427+0000","last_change":"2023-10-07T09:44:24.295412+0000","last_active":"2023-10-07T09:44:24.302427+0000","last_peered":"2023-10-07T09:44:24.302427+0000","last_clean":"2023-10-07T09:44:24.302427+0000","last_became_active":"2023-10-07T09:44:24.295145+0000","last_became_peered":"2023-10-07T09:44:24.295145+0000","last_unstale":"2023-10-07T09:44:24.302427+0000","last_undegraded":"2023-10-07T09:44:24.302427+0000","last_fullsized":"2023-10-07T09:44:24.302427+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:31.116678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302395+0000","last_change":"2023-10-07T09:44:24.293530+0000","last_active":"2023-10-07T09:44:24.302395+0000","last_peered":"2023-10-07T09:44:24.302395+0000","last_clean":"2023-10-07T09:44:24.302395+0000","last_became_active":"2023-10-07T09:44:24.290943+0000","last_became_peered":"2023-10-07T09:44:24.290943+0000","last_unstale":"2023-10-07T09:44:24.302395+0000","last_undegraded":"2023-10-07T09:44:24.302395+0000","last_fullsized":"2023-10-07T09:44:24.302395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:53:58.454122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312069+0000","last_change":"2023-10-07T09:44:24.309726+0000","last_active":"2023-10-07T09:44:24.312069+0000","last_peered":"2023-10-07T09:44:24.312069+0000","last_clean":"2023-10-07T09:44:24.312069+0000","last_became_active":"2023-10-07T09:44:24.309467+0000","last_became_peered":"2023-10-07T09:44:24.309467+0000","last_unstale":"2023-10-07T09:44:24.312069+0000","last_undegraded":"2023-10-07T09:44:24.312069+0000","last_fullsized":"2023-10-07T09:44:24.312069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:55:32.282248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307283+0000","last_change":"2023-10-07T09:44:24.297242+0000","last_active":"2023-10-07T09:44:24.307283+0000","last_peered":"2023-10-07T09:44:24.307283+0000","last_clean":"2023-10-07T09:44:24.307283+0000","last_became_active":"2023-10-07T09:44:24.296541+0000","last_became_peered":"2023-10-07T09:44:24.296541+0000","last_unstale":"2023-10-07T09:44:24.307283+0000","last_undegraded":"2023-10-07T09:44:24.307283+0000","last_fullsized":"2023-10-07T09:44:24.307283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:46.054505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311356+0000","last_change":"2023-10-07T09:44:24.305638+0000","last_active":"2023-10-07T09:44:24.311356+0000","last_peered":"2023-10-07T09:44:24.311356+0000","last_clean":"2023-10-07T09:44:24.311356+0000","last_became_active":"2023-10-07T09:44:24.305054+0000","last_became_peered":"2023-10-07T09:44:24.305054+0000","last_unstale":"2023-10-07T09:44:24.311356+0000","last_undegraded":"2023-10-07T09:44:24.311356+0000","last_fullsized":"2023-10-07T09:44:24.311356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:59.732037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"76'4","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302987+0000","last_change":"2023-10-07T09:44:23.291971+0000","last_active":"2023-10-07T09:44:24.302987+0000","last_peered":"2023-10-07T09:44:24.302987+0000","last_clean":"2023-10-07T09:44:24.302987+0000","last_became_active":"2023-10-07T09:44:23.291699+0000","last_became_peered":"2023-10-07T09:44:23.291699+0000","last_unstale":"2023-10-07T09:44:24.302987+0000","last_undegraded":"2023-10-07T09:44:24.302987+0000","last_fullsized":"2023-10-07T09:44:24.302987+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:16:51.707247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288177+0000","last_change":"2023-10-07T09:44:24.286677+0000","last_active":"2023-10-07T09:44:24.288177+0000","last_peered":"2023-10-07T09:44:24.288177+0000","last_clean":"2023-10-07T09:44:24.288177+0000","last_became_active":"2023-10-07T09:44:24.286418+0000","last_became_peered":"2023-10-07T09:44:24.286418+0000","last_unstale":"2023-10-07T09:44:24.288177+0000","last_undegraded":"2023-10-07T09:44:24.288177+0000","last_fullsized":"2023-10-07T09:44:24.288177+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:40.873462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303110+0000","last_change":"2023-10-07T09:44:23.295503+0000","last_active":"2023-10-07T09:44:24.303110+0000","last_peered":"2023-10-07T09:44:24.303110+0000","last_clean":"2023-10-07T09:44:24.303110+0000","last_became_active":"2023-10-07T09:44:23.295266+0000","last_became_peered":"2023-10-07T09:44:23.295266+0000","last_unstale":"2023-10-07T09:44:24.303110+0000","last_undegraded":"2023-10-07T09:44:24.303110+0000","last_fullsized":"2023-10-07T09:44:24.303110+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:00.328720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311921+0000","last_change":"2023-10-07T09:44:24.305846+0000","last_active":"2023-10-07T09:44:24.311921+0000","last_peered":"2023-10-07T09:44:24.311921+0000","last_clean":"2023-10-07T09:44:24.311921+0000","last_became_active":"2023-10-07T09:44:24.305533+0000","last_became_peered":"2023-10-07T09:44:24.305533+0000","last_unstale":"2023-10-07T09:44:24.311921+0000","last_undegraded":"2023-10-07T09:44:24.311921+0000","last_fullsized":"2023-10-07T09:44:24.311921+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:55:10.571566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306282+0000","last_change":"2023-10-07T09:44:23.283966+0000","last_active":"2023-10-07T09:44:24.306282+0000","last_peered":"2023-10-07T09:44:24.306282+0000","last_clean":"2023-10-07T09:44:24.306282+0000","last_became_active":"2023-10-07T09:44:23.283694+0000","last_became_peered":"2023-10-07T09:44:23.283694+0000","last_unstale":"2023-10-07T09:44:24.306282+0000","last_undegraded":"2023-10-07T09:44:24.306282+0000","last_fullsized":"2023-10-07T09:44:24.306282+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:04:19.056016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306117+0000","last_change":"2023-10-07T09:44:24.299366+0000","last_active":"2023-10-07T09:44:24.306117+0000","last_peered":"2023-10-07T09:44:24.306117+0000","last_clean":"2023-10-07T09:44:24.306117+0000","last_became_active":"2023-10-07T09:44:24.298974+0000","last_became_peered":"2023-10-07T09:44:24.298974+0000","last_unstale":"2023-10-07T09:44:24.306117+0000","last_undegraded":"2023-10-07T09:44:24.306117+0000","last_fullsized":"2023-10-07T09:44:24.306117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:01.660743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304228+0000","last_change":"2023-10-07T09:44:23.300755+0000","last_active":"2023-10-07T09:44:24.304228+0000","last_peered":"2023-10-07T09:44:24.304228+0000","last_clean":"2023-10-07T09:44:24.304228+0000","last_became_active":"2023-10-07T09:44:23.300478+0000","last_became_peered":"2023-10-07T09:44:23.300478+0000","last_unstale":"2023-10-07T09:44:24.304228+0000","last_undegraded":"2023-10-07T09:44:24.304228+0000","last_fullsized":"2023-10-07T09:44:24.304228+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:39.865381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312608+0000","last_change":"2023-10-07T09:44:24.310721+0000","last_active":"2023-10-07T09:44:24.312608+0000","last_peered":"2023-10-07T09:44:24.312608+0000","last_clean":"2023-10-07T09:44:24.312608+0000","last_became_active":"2023-10-07T09:44:24.310109+0000","last_became_peered":"2023-10-07T09:44:24.310109+0000","last_unstale":"2023-10-07T09:44:24.312608+0000","last_undegraded":"2023-10-07T09:44:24.312608+0000","last_fullsized":"2023-10-07T09:44:24.312608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:30:21.331223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304353+0000","last_change":"2023-10-07T09:44:23.295182+0000","last_active":"2023-10-07T09:44:24.304353+0000","last_peered":"2023-10-07T09:44:24.304353+0000","last_clean":"2023-10-07T09:44:24.304353+0000","last_became_active":"2023-10-07T09:44:23.294935+0000","last_became_peered":"2023-10-07T09:44:23.294935+0000","last_unstale":"2023-10-07T09:44:24.304353+0000","last_undegraded":"2023-10-07T09:44:24.304353+0000","last_fullsized":"2023-10-07T09:44:24.304353+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:06.641494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.311565+0000","last_change":"2023-10-07T09:44:24.298842+0000","last_active":"2023-10-07T09:44:24.311565+0000","last_peered":"2023-10-07T09:44:24.311565+0000","last_clean":"2023-10-07T09:44:24.311565+0000","last_became_active":"2023-10-07T09:44:24.298596+0000","last_became_peered":"2023-10-07T09:44:24.298596+0000","last_unstale":"2023-10-07T09:44:24.311565+0000","last_undegraded":"2023-10-07T09:44:24.311565+0000","last_fullsized":"2023-10-07T09:44:24.311565+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:16:25.708170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302594+0000","last_change":"2023-10-07T09:44:23.292056+0000","last_active":"2023-10-07T09:44:24.302594+0000","last_peered":"2023-10-07T09:44:24.302594+0000","last_clean":"2023-10-07T09:44:24.302594+0000","last_became_active":"2023-10-07T09:44:23.291816+0000","last_became_peered":"2023-10-07T09:44:23.291816+0000","last_unstale":"2023-10-07T09:44:24.302594+0000","last_undegraded":"2023-10-07T09:44:24.302594+0000","last_fullsized":"2023-10-07T09:44:24.302594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:10:42.036332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.294493+0000","last_change":"2023-10-07T09:44:24.287269+0000","last_active":"2023-10-07T09:44:24.294493+0000","last_peered":"2023-10-07T09:44:24.294493+0000","last_clean":"2023-10-07T09:44:24.294493+0000","last_became_active":"2023-10-07T09:44:24.286960+0000","last_became_peered":"2023-10-07T09:44:24.286960+0000","last_unstale":"2023-10-07T09:44:24.294493+0000","last_undegraded":"2023-10-07T09:44:24.294493+0000","last_fullsized":"2023-10-07T09:44:24.294493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:46.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288941+0000","last_change":"2023-10-07T09:44:23.295396+0000","last_active":"2023-10-07T09:44:24.288941+0000","last_peered":"2023-10-07T09:44:24.288941+0000","last_clean":"2023-10-07T09:44:24.288941+0000","last_became_active":"2023-10-07T09:44:23.295096+0000","last_became_peered":"2023-10-07T09:44:23.295096+0000","last_unstale":"2023-10-07T09:44:24.288941+0000","last_undegraded":"2023-10-07T09:44:24.288941+0000","last_fullsized":"2023-10-07T09:44:24.288941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:45.046424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.292422+0000","last_change":"2023-10-07T09:44:24.286652+0000","last_active":"2023-10-07T09:44:24.292422+0000","last_peered":"2023-10-07T09:44:24.292422+0000","last_clean":"2023-10-07T09:44:24.292422+0000","last_became_active":"2023-10-07T09:44:24.286180+0000","last_became_peered":"2023-10-07T09:44:24.286180+0000","last_unstale":"2023-10-07T09:44:24.292422+0000","last_undegraded":"2023-10-07T09:44:24.292422+0000","last_fullsized":"2023-10-07T09:44:24.292422+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:29.612688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.286699+0000","last_change":"2023-10-07T09:44:23.294852+0000","last_active":"2023-10-07T09:44:24.286699+0000","last_peered":"2023-10-07T09:44:24.286699+0000","last_clean":"2023-10-07T09:44:24.286699+0000","last_became_active":"2023-10-07T09:44:23.294590+0000","last_became_peered":"2023-10-07T09:44:23.294590+0000","last_unstale":"2023-10-07T09:44:24.286699+0000","last_undegraded":"2023-10-07T09:44:24.286699+0000","last_fullsized":"2023-10-07T09:44:24.286699+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:44:37.861916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309172+0000","last_change":"2023-10-07T09:44:24.307592+0000","last_active":"2023-10-07T09:44:24.309172+0000","last_peered":"2023-10-07T09:44:24.309172+0000","last_clean":"2023-10-07T09:44:24.309172+0000","last_became_active":"2023-10-07T09:44:24.307107+0000","last_became_peered":"2023-10-07T09:44:24.307107+0000","last_unstale":"2023-10-07T09:44:24.309172+0000","last_undegraded":"2023-10-07T09:44:24.309172+0000","last_fullsized":"2023-10-07T09:44:24.309172+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:15:41.028294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.282373+0000","last_change":"2023-10-07T09:44:23.300332+0000","last_active":"2023-10-07T09:44:24.282373+0000","last_peered":"2023-10-07T09:44:24.282373+0000","last_clean":"2023-10-07T09:44:24.282373+0000","last_became_active":"2023-10-07T09:44:23.299952+0000","last_became_peered":"2023-10-07T09:44:23.299952+0000","last_unstale":"2023-10-07T09:44:24.282373+0000","last_undegraded":"2023-10-07T09:44:24.282373+0000","last_fullsized":"2023-10-07T09:44:24.282373+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:44:02.443489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.298888+0000","last_change":"2023-10-07T09:44:24.291861+0000","last_active":"2023-10-07T09:44:24.298888+0000","last_peered":"2023-10-07T09:44:24.298888+0000","last_clean":"2023-10-07T09:44:24.298888+0000","last_became_active":"2023-10-07T09:44:24.291548+0000","last_became_peered":"2023-10-07T09:44:24.291548+0000","last_unstale":"2023-10-07T09:44:24.298888+0000","last_undegraded":"2023-10-07T09:44:24.298888+0000","last_fullsized":"2023-10-07T09:44:24.298888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:55:32.790004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.278296+0000","last_change":"2023-10-07T09:44:23.290458+0000","last_active":"2023-10-07T09:44:24.278296+0000","last_peered":"2023-10-07T09:44:24.278296+0000","last_clean":"2023-10-07T09:44:24.278296+0000","last_became_active":"2023-10-07T09:44:23.290191+0000","last_became_peered":"2023-10-07T09:44:23.290191+0000","last_unstale":"2023-10-07T09:44:24.278296+0000","last_undegraded":"2023-10-07T09:44:24.278296+0000","last_fullsized":"2023-10-07T09:44:24.278296+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:49:52.046310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304354+0000","last_change":"2023-10-07T09:44:24.299451+0000","last_active":"2023-10-07T09:44:24.304354+0000","last_peered":"2023-10-07T09:44:24.304354+0000","last_clean":"2023-10-07T09:44:24.304354+0000","last_became_active":"2023-10-07T09:44:24.298999+0000","last_became_peered":"2023-10-07T09:44:24.298999+0000","last_unstale":"2023-10-07T09:44:24.304354+0000","last_undegraded":"2023-10-07T09:44:24.304354+0000","last_fullsized":"2023-10-07T09:44:24.304354+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:50:13.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"76'3","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.313810+0000","last_change":"2023-10-07T09:44:23.300451+0000","last_active":"2023-10-07T09:44:24.313810+0000","last_peered":"2023-10-07T09:44:24.313810+0000","last_clean":"2023-10-07T09:44:24.313810+0000","last_became_active":"2023-10-07T09:44:23.300143+0000","last_became_peered":"2023-10-07T09:44:23.300143+0000","last_unstale":"2023-10-07T09:44:24.313810+0000","last_undegraded":"2023-10-07T09:44:24.313810+0000","last_fullsized":"2023-10-07T09:44:24.313810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:22:46.170377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310089+0000","last_change":"2023-10-07T09:44:24.306425+0000","last_active":"2023-10-07T09:44:24.310089+0000","last_peered":"2023-10-07T09:44:24.310089+0000","last_clean":"2023-10-07T09:44:24.310089+0000","last_became_active":"2023-10-07T09:44:24.306128+0000","last_became_peered":"2023-10-07T09:44:24.306128+0000","last_unstale":"2023-10-07T09:44:24.310089+0000","last_undegraded":"2023-10-07T09:44:24.310089+0000","last_fullsized":"2023-10-07T09:44:24.310089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:57:21.761574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"76'6","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.312841+0000","last_change":"2023-10-07T09:44:23.295870+0000","last_active":"2023-10-07T09:44:24.312841+0000","last_peered":"2023-10-07T09:44:24.312841+0000","last_clean":"2023-10-07T09:44:24.312841+0000","last_became_active":"2023-10-07T09:44:23.295590+0000","last_became_peered":"2023-10-07T09:44:23.295590+0000","last_unstale":"2023-10-07T09:44:24.312841+0000","last_undegraded":"2023-10-07T09:44:24.312841+0000","last_fullsized":"2023-10-07T09:44:24.312841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:11:31.060288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310295+0000","last_change":"2023-10-07T09:44:24.309035+0000","last_active":"2023-10-07T09:44:24.310295+0000","last_peered":"2023-10-07T09:44:24.310295+0000","last_clean":"2023-10-07T09:44:24.310295+0000","last_became_active":"2023-10-07T09:44:24.308726+0000","last_became_peered":"2023-10-07T09:44:24.308726+0000","last_unstale":"2023-10-07T09:44:24.310295+0000","last_undegraded":"2023-10-07T09:44:24.310295+0000","last_fullsized":"2023-10-07T09:44:24.310295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:38:42.467052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302771+0000","last_change":"2023-10-07T09:44:23.290550+0000","last_active":"2023-10-07T09:44:24.302771+0000","last_peered":"2023-10-07T09:44:24.302771+0000","last_clean":"2023-10-07T09:44:24.302771+0000","last_became_active":"2023-10-07T09:44:23.290291+0000","last_became_peered":"2023-10-07T09:44:23.290291+0000","last_unstale":"2023-10-07T09:44:24.302771+0000","last_undegraded":"2023-10-07T09:44:24.302771+0000","last_fullsized":"2023-10-07T09:44:24.302771+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:15:40.020214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.298660+0000","last_change":"2023-10-07T09:44:24.290277+0000","last_active":"2023-10-07T09:44:24.298660+0000","last_peered":"2023-10-07T09:44:24.298660+0000","last_clean":"2023-10-07T09:44:24.298660+0000","last_became_active":"2023-10-07T09:44:24.289943+0000","last_became_peered":"2023-10-07T09:44:24.289943+0000","last_unstale":"2023-10-07T09:44:24.298660+0000","last_undegraded":"2023-10-07T09:44:24.298660+0000","last_fullsized":"2023-10-07T09:44:24.298660+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:01:23.707605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.283202+0000","last_change":"2023-10-07T09:44:23.295504+0000","last_active":"2023-10-07T09:44:24.283202+0000","last_peered":"2023-10-07T09:44:24.283202+0000","last_clean":"2023-10-07T09:44:24.283202+0000","last_became_active":"2023-10-07T09:44:23.295248+0000","last_became_peered":"2023-10-07T09:44:23.295248+0000","last_unstale":"2023-10-07T09:44:24.283202+0000","last_undegraded":"2023-10-07T09:44:24.283202+0000","last_fullsized":"2023-10-07T09:44:24.283202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:06.098008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.295833+0000","last_change":"2023-10-07T09:44:24.294432+0000","last_active":"2023-10-07T09:44:24.295833+0000","last_peered":"2023-10-07T09:44:24.295833+0000","last_clean":"2023-10-07T09:44:24.295833+0000","last_became_active":"2023-10-07T09:44:24.294159+0000","last_became_peered":"2023-10-07T09:44:24.294159+0000","last_unstale":"2023-10-07T09:44:24.295833+0000","last_undegraded":"2023-10-07T09:44:24.295833+0000","last_fullsized":"2023-10-07T09:44:24.295833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:59.732037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288978+0000","last_change":"2023-10-07T09:44:23.300648+0000","last_active":"2023-10-07T09:44:24.288978+0000","last_peered":"2023-10-07T09:44:24.288978+0000","last_clean":"2023-10-07T09:44:24.288978+0000","last_became_active":"2023-10-07T09:44:23.300334+0000","last_became_peered":"2023-10-07T09:44:23.300334+0000","last_unstale":"2023-10-07T09:44:24.288978+0000","last_undegraded":"2023-10-07T09:44:24.288978+0000","last_fullsized":"2023-10-07T09:44:24.288978+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:39:32.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301038+0000","last_change":"2023-10-07T09:44:24.297316+0000","last_active":"2023-10-07T09:44:24.301038+0000","last_peered":"2023-10-07T09:44:24.301038+0000","last_clean":"2023-10-07T09:44:24.301038+0000","last_became_active":"2023-10-07T09:44:24.296659+0000","last_became_peered":"2023-10-07T09:44:24.296659+0000","last_unstale":"2023-10-07T09:44:24.301038+0000","last_undegraded":"2023-10-07T09:44:24.301038+0000","last_fullsized":"2023-10-07T09:44:24.301038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:48.950118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"76'8","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310808+0000","last_change":"2023-10-07T09:44:23.299658+0000","last_active":"2023-10-07T09:44:24.310808+0000","last_peered":"2023-10-07T09:44:24.310808+0000","last_clean":"2023-10-07T09:44:24.310808+0000","last_became_active":"2023-10-07T09:44:23.299354+0000","last_became_peered":"2023-10-07T09:44:23.299354+0000","last_unstale":"2023-10-07T09:44:24.310808+0000","last_undegraded":"2023-10-07T09:44:24.310808+0000","last_fullsized":"2023-10-07T09:44:24.310808+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:50.186900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304712+0000","last_change":"2023-10-07T09:44:24.302800+0000","last_active":"2023-10-07T09:44:24.304712+0000","last_peered":"2023-10-07T09:44:24.304712+0000","last_clean":"2023-10-07T09:44:24.304712+0000","last_became_active":"2023-10-07T09:44:24.302516+0000","last_became_peered":"2023-10-07T09:44:24.302516+0000","last_unstale":"2023-10-07T09:44:24.304712+0000","last_undegraded":"2023-10-07T09:44:24.304712+0000","last_fullsized":"2023-10-07T09:44:24.304712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:26.584370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303208+0000","last_change":"2023-10-07T09:44:23.291240+0000","last_active":"2023-10-07T09:44:24.303208+0000","last_peered":"2023-10-07T09:44:24.303208+0000","last_clean":"2023-10-07T09:44:24.303208+0000","last_became_active":"2023-10-07T09:44:23.290942+0000","last_became_peered":"2023-10-07T09:44:23.290942+0000","last_unstale":"2023-10-07T09:44:24.303208+0000","last_undegraded":"2023-10-07T09:44:24.303208+0000","last_fullsized":"2023-10-07T09:44:24.303208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:55:09.563486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.309141+0000","last_change":"2023-10-07T09:44:24.306770+0000","last_active":"2023-10-07T09:44:24.309141+0000","last_peered":"2023-10-07T09:44:24.309141+0000","last_clean":"2023-10-07T09:44:24.309141+0000","last_became_active":"2023-10-07T09:44:24.306496+0000","last_became_peered":"2023-10-07T09:44:24.306496+0000","last_unstale":"2023-10-07T09:44:24.309141+0000","last_undegraded":"2023-10-07T09:44:24.309141+0000","last_fullsized":"2023-10-07T09:44:24.309141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:21.662987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302436+0000","last_change":"2023-10-07T09:44:23.290435+0000","last_active":"2023-10-07T09:44:24.302436+0000","last_peered":"2023-10-07T09:44:24.302436+0000","last_clean":"2023-10-07T09:44:24.302436+0000","last_became_active":"2023-10-07T09:44:23.290166+0000","last_became_peered":"2023-10-07T09:44:23.290166+0000","last_unstale":"2023-10-07T09:44:24.302436+0000","last_undegraded":"2023-10-07T09:44:24.302436+0000","last_fullsized":"2023-10-07T09:44:24.302436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:23:06.098008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.301437+0000","last_change":"2023-10-07T09:44:24.296481+0000","last_active":"2023-10-07T09:44:24.301437+0000","last_peered":"2023-10-07T09:44:24.301437+0000","last_clean":"2023-10-07T09:44:24.301437+0000","last_became_active":"2023-10-07T09:44:24.296201+0000","last_became_peered":"2023-10-07T09:44:24.296201+0000","last_unstale":"2023-10-07T09:44:24.301437+0000","last_undegraded":"2023-10-07T09:44:24.301437+0000","last_fullsized":"2023-10-07T09:44:24.301437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:59:30.551232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"76'1","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303990+0000","last_change":"2023-10-07T09:44:23.302051+0000","last_active":"2023-10-07T09:44:24.303990+0000","last_peered":"2023-10-07T09:44:24.303990+0000","last_clean":"2023-10-07T09:44:24.303990+0000","last_became_active":"2023-10-07T09:44:23.301738+0000","last_became_peered":"2023-10-07T09:44:23.301738+0000","last_unstale":"2023-10-07T09:44:24.303990+0000","last_undegraded":"2023-10-07T09:44:24.303990+0000","last_fullsized":"2023-10-07T09:44:24.303990+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:16.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310108+0000","last_change":"2023-10-07T09:44:24.306446+0000","last_active":"2023-10-07T09:44:24.310108+0000","last_peered":"2023-10-07T09:44:24.310108+0000","last_clean":"2023-10-07T09:44:24.310108+0000","last_became_active":"2023-10-07T09:44:24.306168+0000","last_became_peered":"2023-10-07T09:44:24.306168+0000","last_unstale":"2023-10-07T09:44:24.310108+0000","last_undegraded":"2023-10-07T09:44:24.310108+0000","last_fullsized":"2023-10-07T09:44:24.310108+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:12:07.230920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.285266+0000","last_change":"2023-10-07T09:44:23.284086+0000","last_active":"2023-10-07T09:44:24.285266+0000","last_peered":"2023-10-07T09:44:24.285266+0000","last_clean":"2023-10-07T09:44:24.285266+0000","last_became_active":"2023-10-07T09:44:23.283818+0000","last_became_peered":"2023-10-07T09:44:23.283818+0000","last_unstale":"2023-10-07T09:44:24.285266+0000","last_undegraded":"2023-10-07T09:44:24.285266+0000","last_fullsized":"2023-10-07T09:44:24.285266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:36.458090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.310277+0000","last_change":"2023-10-07T09:44:24.308886+0000","last_active":"2023-10-07T09:44:24.310277+0000","last_peered":"2023-10-07T09:44:24.310277+0000","last_clean":"2023-10-07T09:44:24.310277+0000","last_became_active":"2023-10-07T09:44:24.308593+0000","last_became_peered":"2023-10-07T09:44:24.308593+0000","last_unstale":"2023-10-07T09:44:24.310277+0000","last_undegraded":"2023-10-07T09:44:24.310277+0000","last_fullsized":"2023-10-07T09:44:24.310277+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:44:01.882036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.302482+0000","last_change":"2023-10-07T09:44:23.293905+0000","last_active":"2023-10-07T09:44:24.302482+0000","last_peered":"2023-10-07T09:44:24.302482+0000","last_clean":"2023-10-07T09:44:24.302482+0000","last_became_active":"2023-10-07T09:44:23.293655+0000","last_became_peered":"2023-10-07T09:44:23.293655+0000","last_unstale":"2023-10-07T09:44:24.302482+0000","last_undegraded":"2023-10-07T09:44:24.302482+0000","last_fullsized":"2023-10-07T09:44:24.302482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:45.046424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304066+0000","last_change":"2023-10-07T09:44:24.296850+0000","last_active":"2023-10-07T09:44:24.304066+0000","last_peered":"2023-10-07T09:44:24.304066+0000","last_clean":"2023-10-07T09:44:24.304066+0000","last_became_active":"2023-10-07T09:44:24.296473+0000","last_became_peered":"2023-10-07T09:44:24.296473+0000","last_unstale":"2023-10-07T09:44:24.304066+0000","last_undegraded":"2023-10-07T09:44:24.304066+0000","last_fullsized":"2023-10-07T09:44:24.304066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:01.336800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.281685+0000","last_change":"2023-10-07T09:44:23.290769+0000","last_active":"2023-10-07T09:44:24.281685+0000","last_peered":"2023-10-07T09:44:24.281685+0000","last_clean":"2023-10-07T09:44:24.281685+0000","last_became_active":"2023-10-07T09:44:23.290526+0000","last_became_peered":"2023-10-07T09:44:23.290526+0000","last_unstale":"2023-10-07T09:44:24.281685+0000","last_undegraded":"2023-10-07T09:44:24.281685+0000","last_fullsized":"2023-10-07T09:44:24.281685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:54.972045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307173+0000","last_change":"2023-10-07T09:44:24.304696+0000","last_active":"2023-10-07T09:44:24.307173+0000","last_peered":"2023-10-07T09:44:24.307173+0000","last_clean":"2023-10-07T09:44:24.307173+0000","last_became_active":"2023-10-07T09:44:24.304419+0000","last_became_peered":"2023-10-07T09:44:24.304419+0000","last_unstale":"2023-10-07T09:44:24.307173+0000","last_undegraded":"2023-10-07T09:44:24.307173+0000","last_fullsized":"2023-10-07T09:44:24.307173+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:34:55.415732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.289712+0000","last_change":"2023-10-07T09:44:23.296038+0000","last_active":"2023-10-07T09:44:24.289712+0000","last_peered":"2023-10-07T09:44:24.289712+0000","last_clean":"2023-10-07T09:44:24.289712+0000","last_became_active":"2023-10-07T09:44:23.295415+0000","last_became_peered":"2023-10-07T09:44:23.295415+0000","last_unstale":"2023-10-07T09:44:24.289712+0000","last_undegraded":"2023-10-07T09:44:24.289712+0000","last_fullsized":"2023-10-07T09:44:24.289712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:44.208709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.308912+0000","last_change":"2023-10-07T09:44:24.306296+0000","last_active":"2023-10-07T09:44:24.308912+0000","last_peered":"2023-10-07T09:44:24.308912+0000","last_clean":"2023-10-07T09:44:24.308912+0000","last_became_active":"2023-10-07T09:44:24.305992+0000","last_became_peered":"2023-10-07T09:44:24.305992+0000","last_unstale":"2023-10-07T09:44:24.308912+0000","last_undegraded":"2023-10-07T09:44:24.308912+0000","last_fullsized":"2023-10-07T09:44:24.308912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:41:28.132376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"76'2","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304668+0000","last_change":"2023-10-07T09:44:23.293710+0000","last_active":"2023-10-07T09:44:24.304668+0000","last_peered":"2023-10-07T09:44:24.304668+0000","last_clean":"2023-10-07T09:44:24.304668+0000","last_became_active":"2023-10-07T09:44:23.293451+0000","last_became_peered":"2023-10-07T09:44:23.293451+0000","last_unstale":"2023-10-07T09:44:24.304668+0000","last_undegraded":"2023-10-07T09:44:24.304668+0000","last_fullsized":"2023-10-07T09:44:24.304668+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:12:36.458090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307328+0000","last_change":"2023-10-07T09:44:24.303571+0000","last_active":"2023-10-07T09:44:24.307328+0000","last_peered":"2023-10-07T09:44:24.307328+0000","last_clean":"2023-10-07T09:44:24.307328+0000","last_became_active":"2023-10-07T09:44:24.303322+0000","last_became_peered":"2023-10-07T09:44:24.303322+0000","last_unstale":"2023-10-07T09:44:24.307328+0000","last_undegraded":"2023-10-07T09:44:24.307328+0000","last_fullsized":"2023-10-07T09:44:24.307328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:33:54.384857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.287823+0000","last_change":"2023-10-07T09:44:23.289459+0000","last_active":"2023-10-07T09:44:24.287823+0000","last_peered":"2023-10-07T09:44:24.287823+0000","last_clean":"2023-10-07T09:44:24.287823+0000","last_became_active":"2023-10-07T09:44:23.289123+0000","last_became_peered":"2023-10-07T09:44:23.289123+0000","last_unstale":"2023-10-07T09:44:24.287823+0000","last_undegraded":"2023-10-07T09:44:24.287823+0000","last_fullsized":"2023-10-07T09:44:24.287823+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:04:05.083808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.304336+0000","last_change":"2023-10-07T09:44:24.299282+0000","last_active":"2023-10-07T09:44:24.304336+0000","last_peered":"2023-10-07T09:44:24.304336+0000","last_clean":"2023-10-07T09:44:24.304336+0000","last_became_active":"2023-10-07T09:44:24.298577+0000","last_became_peered":"2023-10-07T09:44:24.298577+0000","last_unstale":"2023-10-07T09:44:24.304336+0000","last_undegraded":"2023-10-07T09:44:24.304336+0000","last_fullsized":"2023-10-07T09:44:24.304336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:46.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.283081+0000","last_change":"2023-10-07T09:44:23.298822+0000","last_active":"2023-10-07T09:44:24.283081+0000","last_peered":"2023-10-07T09:44:24.283081+0000","last_clean":"2023-10-07T09:44:24.283081+0000","last_became_active":"2023-10-07T09:44:23.298575+0000","last_became_peered":"2023-10-07T09:44:23.298575+0000","last_unstale":"2023-10-07T09:44:24.283081+0000","last_undegraded":"2023-10-07T09:44:24.283081+0000","last_fullsized":"2023-10-07T09:44:24.283081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:02:22.668133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.306101+0000","last_change":"2023-10-07T09:44:24.299298+0000","last_active":"2023-10-07T09:44:24.306101+0000","last_peered":"2023-10-07T09:44:24.306101+0000","last_clean":"2023-10-07T09:44:24.306101+0000","last_became_active":"2023-10-07T09:44:24.298836+0000","last_became_peered":"2023-10-07T09:44:24.298836+0000","last_unstale":"2023-10-07T09:44:24.306101+0000","last_undegraded":"2023-10-07T09:44:24.306101+0000","last_fullsized":"2023-10-07T09:44:24.306101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:02.073157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.288889+0000","last_change":"2023-10-07T09:44:23.296333+0000","last_active":"2023-10-07T09:44:24.288889+0000","last_peered":"2023-10-07T09:44:24.288889+0000","last_clean":"2023-10-07T09:44:24.288889+0000","last_became_active":"2023-10-07T09:44:23.295143+0000","last_became_peered":"2023-10-07T09:44:23.295143+0000","last_unstale":"2023-10-07T09:44:24.288889+0000","last_undegraded":"2023-10-07T09:44:24.288889+0000","last_fullsized":"2023-10-07T09:44:24.288889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:55:09.563486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.294625+0000","last_change":"2023-10-07T09:44:24.291219+0000","last_active":"2023-10-07T09:44:24.294625+0000","last_peered":"2023-10-07T09:44:24.294625+0000","last_clean":"2023-10-07T09:44:24.294625+0000","last_became_active":"2023-10-07T09:44:24.290947+0000","last_became_peered":"2023-10-07T09:44:24.290947+0000","last_unstale":"2023-10-07T09:44:24.294625+0000","last_undegraded":"2023-10-07T09:44:24.294625+0000","last_fullsized":"2023-10-07T09:44:24.294625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:37:56.396212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.280280+0000","last_change":"2023-10-07T09:44:23.290267+0000","last_active":"2023-10-07T09:44:24.280280+0000","last_peered":"2023-10-07T09:44:24.280280+0000","last_clean":"2023-10-07T09:44:24.280280+0000","last_became_active":"2023-10-07T09:44:23.289997+0000","last_became_peered":"2023-10-07T09:44:23.289997+0000","last_unstale":"2023-10-07T09:44:24.280280+0000","last_undegraded":"2023-10-07T09:44:24.280280+0000","last_fullsized":"2023-10-07T09:44:24.280280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:25:50.787707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.300996+0000","last_change":"2023-10-07T09:44:24.296915+0000","last_active":"2023-10-07T09:44:24.300996+0000","last_peered":"2023-10-07T09:44:24.300996+0000","last_clean":"2023-10-07T09:44:24.300996+0000","last_became_active":"2023-10-07T09:44:24.296585+0000","last_became_peered":"2023-10-07T09:44:24.296585+0000","last_unstale":"2023-10-07T09:44:24.300996+0000","last_undegraded":"2023-10-07T09:44:24.300996+0000","last_fullsized":"2023-10-07T09:44:24.300996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:23.453707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.286508+0000","last_change":"2023-10-07T09:44:23.296699+0000","last_active":"2023-10-07T09:44:24.286508+0000","last_peered":"2023-10-07T09:44:24.286508+0000","last_clean":"2023-10-07T09:44:24.286508+0000","last_became_active":"2023-10-07T09:44:23.296307+0000","last_became_peered":"2023-10-07T09:44:23.296307+0000","last_unstale":"2023-10-07T09:44:24.286508+0000","last_undegraded":"2023-10-07T09:44:24.286508+0000","last_fullsized":"2023-10-07T09:44:24.286508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:45.052881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.303841+0000","last_change":"2023-10-07T09:44:24.292140+0000","last_active":"2023-10-07T09:44:24.303841+0000","last_peered":"2023-10-07T09:44:24.303841+0000","last_clean":"2023-10-07T09:44:24.303841+0000","last_became_active":"2023-10-07T09:44:24.290209+0000","last_became_peered":"2023-10-07T09:44:24.290209+0000","last_unstale":"2023-10-07T09:44:24.303841+0000","last_undegraded":"2023-10-07T09:44:24.303841+0000","last_fullsized":"2023-10-07T09:44:24.303841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:11:32.068369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.280148+0000","last_change":"2023-10-07T09:44:23.286889+0000","last_active":"2023-10-07T09:44:24.280148+0000","last_peered":"2023-10-07T09:44:24.280148+0000","last_clean":"2023-10-07T09:44:24.280148+0000","last_became_active":"2023-10-07T09:44:23.286605+0000","last_became_peered":"2023-10-07T09:44:23.286605+0000","last_unstale":"2023-10-07T09:44:24.280148+0000","last_undegraded":"2023-10-07T09:44:24.280148+0000","last_fullsized":"2023-10-07T09:44:24.280148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:15:38.709944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.292402+0000","last_change":"2023-10-07T09:44:24.286461+0000","last_active":"2023-10-07T09:44:24.292402+0000","last_peered":"2023-10-07T09:44:24.292402+0000","last_clean":"2023-10-07T09:44:24.292402+0000","last_became_active":"2023-10-07T09:44:24.285822+0000","last_became_peered":"2023-10-07T09:44:24.285822+0000","last_unstale":"2023-10-07T09:44:24.292402+0000","last_undegraded":"2023-10-07T09:44:24.292402+0000","last_fullsized":"2023-10-07T09:44:24.292402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:53:32.891702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.277691+0000","last_change":"2023-10-07T09:44:23.293473+0000","last_active":"2023-10-07T09:44:24.277691+0000","last_peered":"2023-10-07T09:44:24.277691+0000","last_clean":"2023-10-07T09:44:24.277691+0000","last_became_active":"2023-10-07T09:44:23.293179+0000","last_became_peered":"2023-10-07T09:44:23.293179+0000","last_unstale":"2023-10-07T09:44:24.277691+0000","last_undegraded":"2023-10-07T09:44:24.277691+0000","last_fullsized":"2023-10-07T09:44:24.277691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:44.208709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.307974+0000","last_change":"2023-10-07T09:44:24.299614+0000","last_active":"2023-10-07T09:44:24.307974+0000","last_peered":"2023-10-07T09:44:24.307974+0000","last_clean":"2023-10-07T09:44:24.307974+0000","last_became_active":"2023-10-07T09:44:24.299110+0000","last_became_peered":"2023-10-07T09:44:24.299110+0000","last_unstale":"2023-10-07T09:44:24.307974+0000","last_undegraded":"2023-10-07T09:44:24.307974+0000","last_fullsized":"2023-10-07T09:44:24.307974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:23.263482+0000","last_clean_scrub_stamp":"2023-10-07T09:44:23.263482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:57:22.311791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.283347+0000","last_change":"2023-10-07T09:44:23.297144+0000","last_active":"2023-10-07T09:44:24.283347+0000","last_peered":"2023-10-07T09:44:24.283347+0000","last_clean":"2023-10-07T09:44:24.283347+0000","last_became_active":"2023-10-07T09:44:23.296828+0000","last_became_peered":"2023-10-07T09:44:23.296828+0000","last_unstale":"2023-10-07T09:44:24.283347+0000","last_undegraded":"2023-10-07T09:44:24.283347+0000","last_fullsized":"2023-10-07T09:44:24.283347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:22.255401+0000","last_clean_scrub_stamp":"2023-10-07T09:44:22.255401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:16:51.707247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"67'161","reported_seq":365,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-10-07T09:44:24.282861+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:44:24.282861+0000","last_peered":"2023-10-07T09:44:24.282861+0000","last_clean":"2023-10-07T09:44:24.282861+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:44:24.282861+0000","last_undegraded":"2023-10-07T09:44:24.282861+0000","last_fullsized":"2023-10-07T09:44:24.282861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476920,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302872,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068968,"statfs":{"total":96636764160,"available":96326623232,"internally_reserved":0,"allocated":1523712,"data_stored":1023717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.46100000000000002,"5min":0.374,"15min":0.32300000000000001},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":1.026},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71899999999999997,"15min":0.70799999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.374},"max":{"1min":0.91300000000000003,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.82299999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.63800000000000001,"5min":0.42599999999999999,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":1.014,"15min":1.157},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.74099999999999999,"15min":0.73599999999999999},"min":{"1min":0.50800000000000001,"5min":0.49099999999999999,"15min":0.45600000000000002},"max":{"1min":0.91200000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.59099999999999997,"5min":0.49399999999999999,"15min":0.40699999999999997},"max":{"1min":0.82999999999999996,"5min":1.206,"15min":1.206},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.56699999999999995,"5min":0.504,"15min":0.439},"max":{"1min":0.85399999999999998,"5min":1.071,"15min":1.071},"last":0.85299999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72099999999999997,"15min":0.73299999999999998},"min":{"1min":0.54800000000000004,"5min":0.48899999999999999,"15min":0.46400000000000002},"max":{"1min":0.92300000000000004,"5min":0.96899999999999997,"15min":1.04},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.75600000000000001,"15min":0.76600000000000001},"min":{"1min":0.58699999999999997,"5min":0.53000000000000003,"15min":0.371},"max":{"1min":0.95599999999999996,"5min":1.137,"15min":1.3440000000000001},"last":0.83199999999999996}]},{"osd":4,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.48199999999999998,"5min":0.39600000000000002,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.88700000000000001,"15min":1.992},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64700000000000002,"15min":0.64600000000000002},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":1.6240000000000001},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64700000000000002,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92100000000000004,"5min":1.27,"15min":1.27},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.50800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.72499999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.747},"min":{"1min":0.59499999999999997,"5min":0.51200000000000001,"15min":0.46400000000000002},"max":{"1min":0.89700000000000002,"5min":1.0109999999999999,"15min":1.0289999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":0.89100000000000001,"5min":1.0229999999999999,"15min":1.107},"last":0.76000000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509624,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302284,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069556,"statfs":{"total":96636764160,"available":96327225344,"internally_reserved":0,"allocated":921600,"data_stored":433189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.52200000000000002,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":0.83299999999999996,"5min":0.96099999999999997,"15min":2.984},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67900000000000005,"15min":0.69599999999999995},"min":{"1min":0.49199999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.94599999999999995,"15min":1.6000000000000001},"last":0.74299999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.54600000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":2.4529999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.72999999999999998},"min":{"1min":0.54300000000000004,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92900000000000005,"5min":0.99199999999999999,"15min":1.754},"last":0.67500000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71699999999999997,"15min":0.77100000000000002},"min":{"1min":0.61399999999999999,"5min":0.45700000000000002,"15min":0.44400000000000001},"max":{"1min":0.92100000000000004,"5min":0.96699999999999997,"15min":7.867},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.749,"15min":0.79700000000000004},"min":{"1min":0.624,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":7.7629999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.751,"15min":0.80800000000000005},"min":{"1min":0.56000000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92600000000000005,"5min":0.999,"15min":8.0399999999999991},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73999999999999999,"15min":0.77600000000000002},"min":{"1min":0.58299999999999996,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":0.90100000000000002,"5min":0.98099999999999998,"15min":2.7949999999999999},"last":0.78700000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.627,"15min":0.63},"min":{"1min":0.46100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":1.296},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63700000000000001,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.95599999999999996,"5min":0.97899999999999998,"15min":1.105},"last":0.76600000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.4119999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71199999999999997,"15min":0.72199999999999998},"min":{"1min":0.49399999999999999,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":1.0089999999999999,"5min":1.0449999999999999,"15min":1.1779999999999999},"last":0.84699999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67900000000000005,"15min":0.70099999999999996},"min":{"1min":0.41799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.5049999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.75,"15min":0.77200000000000002},"min":{"1min":0.56399999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97999999999999998,"5min":0.995,"15min":1.3220000000000001},"last":0.90100000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542326,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302276,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069564,"statfs":{"total":96636764160,"available":96327233536,"internally_reserved":0,"allocated":913408,"data_stored":426615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.36099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":1.585},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.55000000000000004,"15min":0.55700000000000005},"min":{"1min":0.51800000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.79800000000000004,"5min":0.84199999999999997,"15min":1.6080000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.46500000000000002,"15min":0.48599999999999999},"min":{"1min":0.41699999999999998,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.621,"5min":0.621,"15min":1.468},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54500000000000004,"15min":0.56200000000000006},"min":{"1min":0.39100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":1.5389999999999999},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.60299999999999998},"min":{"1min":0.46300000000000002,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.73299999999999998,"5min":0.75900000000000001,"15min":1.663},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":1.1140000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.77100000000000002,"15min":0.78400000000000003},"min":{"1min":0.68500000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0189999999999999,"5min":1.115,"15min":1.4259999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.64100000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96499999999999997,"5min":1.044,"15min":1.766},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.81599999999999995,"15min":0.81699999999999995},"min":{"1min":0.72499999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0660000000000001,"5min":1.099,"15min":1.3320000000000001},"last":0.626},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.61499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.5149999999999999},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.79800000000000004,"15min":0.82499999999999996},"min":{"1min":0.63800000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0109999999999999,"5min":1.0549999999999999,"15min":1.627},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81100000000000005,"15min":0.82999999999999996},"min":{"1min":0.65000000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0700000000000001,"5min":1.1040000000000001,"15min":1.3700000000000001},"last":0.67300000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:43:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.68200000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0229999999999999,"5min":1.0900000000000001,"15min":1.706},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83299999999999996,"15min":0.85199999999999998},"min":{"1min":0.68799999999999994,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0409999999999999,"5min":1.137,"15min":1.554},"last":0.69899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542326,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302880,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068960,"statfs":{"total":96636764160,"available":96326615040,"internally_reserved":0,"allocated":1531904,"data_stored":1025647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49199999999999999,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66300000000000003,"5min":0.73099999999999998,"15min":2.726},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.59099999999999997},"min":{"1min":0.44700000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":2.552},"last":0.52800000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.32800000000000001},"max":{"1min":0.68400000000000005,"5min":1.101,"15min":2.4910000000000001},"last":0.621},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.53900000000000003},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":3.629},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53100000000000003,"15min":0.55600000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.33900000000000002},"max":{"1min":0.71399999999999997,"5min":1.0009999999999999,"15min":2.6909999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61199999999999999,"15min":0.626},"min":{"1min":0.45100000000000001,"5min":0.42199999999999999,"15min":0.40200000000000002},"max":{"1min":0.76700000000000002,"5min":0.89500000000000002,"15min":2.5209999999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.56000000000000005,"5min":0.497,"15min":0.45900000000000002},"max":{"1min":0.97799999999999998,"5min":1.232,"15min":2.637},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.117,"15min":2.4180000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78400000000000003,"15min":0.81699999999999995},"min":{"1min":0.623,"5min":0.504,"15min":0.501},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":3.8199999999999998},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.80200000000000005,"15min":0.81899999999999995},"min":{"1min":0.68500000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.03,"5min":1.194,"15min":2.6000000000000001},"last":0.69599999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81399999999999995,"15min":0.83099999999999996},"min":{"1min":0.69099999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0069999999999999,"5min":1.129,"15min":2.3740000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81899999999999995,"15min":0.84699999999999998},"min":{"1min":0.60799999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.206,"5min":1.206,"15min":3.855},"last":0.81499999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":0.84599999999999997},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.56200000000000006},"max":{"1min":1.0920000000000001,"5min":1.167,"15min":2.4550000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83199999999999996,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.238,"5min":1.238,"15min":3.778},"last":0.72899999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542327,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310484,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061356,"statfs":{"total":96636764160,"available":96318828544,"internally_reserved":0,"allocated":929792,"data_stored":433245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46200000000000002,"15min":0.45100000000000001},"min":{"1min":0.33900000000000002,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.63500000000000001,"5min":1.258,"15min":1.258},"last":0.314},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.376,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.62,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.55600000000000005},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.33200000000000002},"max":{"1min":0.71599999999999997,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.60299999999999998},"min":{"1min":0.53100000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.371,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.502},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.58499999999999996},"min":{"1min":0.46000000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.75,"5min":1.052,"15min":1.052},"last":0.45300000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.53400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0660000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81399999999999995,"15min":0.79900000000000004},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.42099999999999999},"max":{"1min":1.327,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.80800000000000005},"min":{"1min":0.58199999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.173,"5min":1.98,"15min":1.98},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79600000000000004,"15min":0.79000000000000004},"min":{"1min":0.59999999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.208,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.84299999999999997,"15min":0.82899999999999996},"min":{"1min":0.63100000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1379999999999999,"5min":1.931,"15min":1.931},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.84299999999999997,"15min":0.83099999999999996},"min":{"1min":0.66900000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2849999999999999,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85399999999999998,"15min":0.84099999999999997},"min":{"1min":0.63800000000000001,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.2450000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84399999999999997,"15min":0.82799999999999996},"min":{"1min":0.70499999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"max":{"1min":1.0620000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.78400000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542328,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306376,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065464,"statfs":{"total":96636764160,"available":96323035136,"internally_reserved":0,"allocated":917504,"data_stored":426581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.25600000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":3.3980000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.503,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.274},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":3.1299999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.78400000000000003,"5min":0.80100000000000005,"15min":3.4500000000000002},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.438,"5min":0.317,"15min":0.317},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":3.2309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.40600000000000003,"5min":0.39800000000000002,"15min":0.377},"max":{"1min":0.72199999999999998,"5min":0.79800000000000004,"15min":3.0910000000000002},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.52500000000000002,"5min":0.497,"15min":0.44900000000000001},"max":{"1min":0.94499999999999995,"5min":1.145,"15min":1.518},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.59899999999999998,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":1.103,"15min":1.3799999999999999},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.65300000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.3240000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.78700000000000003,"15min":0.79000000000000004},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.50600000000000001},"max":{"1min":1.0640000000000001,"5min":1.1899999999999999,"15min":1.5189999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.202,"5min":1.202,"15min":1.413},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.80400000000000005,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.55700000000000005,"15min":0.51000000000000001},"max":{"1min":0.97999999999999998,"5min":1.1459999999999999,"15min":1.5840000000000001},"last":0.67800000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:43:34 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81899999999999995,"15min":0.82099999999999995},"min":{"1min":0.70699999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.143,"5min":1.163,"15min":1.45},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.872,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.746,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1040000000000001,"5min":1.1140000000000001,"15min":1.5940000000000001},"last":0.626}]}]},{"osd":4,"up_from":15,"seq":64424509624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306392,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065448,"statfs":{"total":96636764160,"available":96323018752,"internally_reserved":0,"allocated":933888,"data_stored":434725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.88100000000000001,"5min":1.069,"15min":1.069},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.71499999999999997,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":0.91300000000000003,"5min":1.0009999999999999,"15min":2.3420000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71599999999999997,"15min":0.70699999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.873,"5min":0.96599999999999997,"15min":1.054},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73799999999999999,"15min":0.72399999999999998},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.35299999999999998},"max":{"1min":0.98999999999999999,"5min":1.478,"15min":1.478},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.75900000000000001,"15min":0.752},"min":{"1min":0.55000000000000004,"5min":0.48799999999999999,"15min":0.432},"max":{"1min":0.95399999999999996,"5min":1.518,"15min":2.3239999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.52500000000000002,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.71799999999999997},"min":{"1min":0.55700000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.85599999999999998,"5min":0.98699999999999999,"15min":1.032},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.76200000000000001,"15min":0.75},"min":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.496},"max":{"1min":0.93200000000000005,"5min":1.036,"15min":1.036},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.627,"15min":0.61499999999999999},"min":{"1min":0.504,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.873,"5min":1.639,"15min":1.639},"last":0.41899999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.63900000000000001},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.98199999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.55700000000000005,"5min":0.46600000000000003,"15min":0.46300000000000002},"max":{"1min":0.93200000000000005,"5min":1.585,"15min":1.585},"last":0.72499999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.52800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.94799999999999995,"5min":1.024,"15min":1.0589999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.753},"min":{"1min":0.64200000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90100000000000002,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.75700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542326,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302280,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069560,"statfs":{"total":96636764160,"available":96327229440,"internally_reserved":0,"allocated":917504,"data_stored":427149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.76600000000000001,"15min":0.753},"min":{"1min":0.68000000000000005,"5min":0.49199999999999999,"15min":0.39900000000000002},"max":{"1min":1.01,"5min":1.218,"15min":1.218},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79700000000000004,"15min":0.79600000000000004},"min":{"1min":0.65800000000000003,"5min":0.53000000000000003,"15min":0.436},"max":{"1min":0.99199999999999999,"5min":1.294,"15min":1.294},"last":0.82099999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.73399999999999999,"15min":0.72899999999999998},"min":{"1min":0.63400000000000001,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":1.048,"5min":1.1339999999999999,"15min":1.3049999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.72199999999999998,"15min":0.76500000000000001},"min":{"1min":0.54700000000000004,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":1.0249999999999999,"5min":1.194,"15min":6.9720000000000004},"last":0.84999999999999998}]},{"osd":2,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.748},"min":{"1min":0.629,"5min":0.47699999999999998,"15min":0.41099999999999998},"max":{"1min":0.94399999999999995,"5min":1.119,"15min":5.9299999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.74399999999999999,"15min":0.748},"min":{"1min":0.60199999999999998,"5min":0.48499999999999999,"15min":0.42299999999999999},"max":{"1min":0.93999999999999995,"5min":1.0049999999999999,"15min":1.704},"last":0.78800000000000003}]},{"osd":3,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.77300000000000002,"15min":0.76900000000000002},"min":{"1min":0.63100000000000001,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":1.0569999999999999,"5min":1.2589999999999999,"15min":1.899},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73899999999999999,"15min":0.78000000000000003},"min":{"1min":0.58499999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.02,"5min":1.036,"15min":6.0620000000000003},"last":0.94999999999999996}]},{"osd":4,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.874,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63,"15min":0.629},"min":{"1min":0.47699999999999998,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.621}]},{"osd":6,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64800000000000002,"15min":0.64200000000000002},"min":{"1min":0.44700000000000001,"5min":0.39700000000000002,"15min":0.33900000000000002},"max":{"1min":0.93300000000000005,"5min":0.998,"15min":0.998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.497,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.91600000000000004,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71299999999999997,"15min":0.70599999999999996},"min":{"1min":0.503,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77000000000000002,"15min":0.76700000000000002},"min":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.95799999999999996,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:44:29.185 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:44:29.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:44:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.422+0000 7f6c805ff700 1 Processor -- start 2023-10-07T09:44:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.422+0000 7f6c805ff700 1 -- start start 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c805ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6c78153e90 0x7f6c7815a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c805ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 0x7f6c7815ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c805ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7815ab80 con 0x7f6c78153ac0 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c805ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c78069d40 con 0x7f6c78153e90 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c805ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78068ca0 con 0x7f6c780b0690 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c7e39b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c78153ac0 0x7f6c780aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58060/0 (socket says 172.21.15.29:58060) 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.423+0000 7f6c7e39b700 1 -- 172.21.15.29:0/2392234094 learned_addr learned my addr 172.21.15.29:0/2392234094 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7eb9c700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 0x7f6c7815ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/2392234094 >> v2:172.21.15.144:3300/0 conn(0x7f6c78153e90 0x7f6c7815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061597825 0 0) 0x7f6c74001110 con 0x7f6c78153ac0 2023-10-07T09:44:29.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c64003570 con 0x7f6c78153ac0 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1350360509 0 0) 0x7f6c64003570 con 0x7f6c78153ac0 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 msgr2=0x7f6c7815ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 0x7f6c7815ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 >> v2:172.21.15.144:3300/0 conn(0x7f6c78153e90 msgr2=0x7f6c7815a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 --2- 172.21.15.29:0/2392234094 >> v2:172.21.15.144:3300/0 conn(0x7f6c78153e90 0x7f6c7815a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78068f80 con 0x7f6c78153ac0 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f6c74001600 con 0x7f6c78153ac0 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 >> v1:172.21.15.29:6789/0 conn(0x7f6c78153ac0 legacy=0x7f6c780aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c64003a60 0x7f6c64003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c64004810 0x7f6c64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 0x7f6c640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78069d40 con 0x7f6c64004810 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78068ca0 con 0x7f6c64003a60 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.424+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6400ba60 con 0x7f6c780b0690 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c64004810 0x7f6c64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7e39b700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c64003a60 0x7f6c64003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7eb9c700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 0x7f6c640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 -- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 msgr2=0x7f6c640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c780b0690 0x7f6c640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 -- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c64003a60 msgr2=0x7f6c64003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c64003a60 0x7f6c64003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 -- 172.21.15.29:0/2392234094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64003570 con 0x7f6c64004810 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c64004810 0x7f6c64004c80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009c00 tx=0x7f6c680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30365757bea3fdf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:29.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.425+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6800e710 con 0x7f6c64004810 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6800e870 con 0x7f6c64004810 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c7d399700 1 -- 172.21.15.29:0/2392234094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6800ec30 con 0x7f6c64004810 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c805ff700 1 -- 172.21.15.29:0/2392234094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c64004810 msgr2=0x7f6c64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c805ff700 1 --2- 172.21.15.29:0/2392234094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c64004810 0x7f6c64004c80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009c00 tx=0x7f6c680091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c7eb9c700 1 -- 172.21.15.29:0/2392234094 reap_dead start 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c805ff700 1 -- 172.21.15.29:0/2392234094 shutdown_connections 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c805ff700 1 -- 172.21.15.29:0/2392234094 >> 172.21.15.29:0/2392234094 conn(0x7f6c78067610 msgr2=0x7f6c78067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c805ff700 1 -- 172.21.15.29:0/2392234094 shutdown_connections 2023-10-07T09:44:29.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.426+0000 7f6c805ff700 1 -- 172.21.15.29:0/2392234094 wait complete. 2023-10-07T09:44:29.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.428+0000 7f6c805ff700 1 Processor -- start 2023-10-07T09:44:29.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.428+0000 7f6c805ff700 1 -- start start 2023-10-07T09:44:29.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c805ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c64004810 0x7f6c781440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c805ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 0x7f6c78156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c805ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c78156940 0x7f6c780c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c805ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c781520b0 con 0x7f6c78151880 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c805ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78144620 con 0x7f6c78156940 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c805ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78156db0 con 0x7f6c64004810 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 0x7f6c78156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 0x7f6c78156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42652/0 (socket says 172.21.15.29:42652) 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 -- 172.21.15.29:0/3810093154 learned_addr learned my addr 172.21.15.29:0/3810093154 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7e39b700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c64004810 0x7f6c781440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7eb9c700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c78156940 0x7f6c780c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 -- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c64004810 msgr2=0x7f6c781440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c64004810 0x7f6c781440e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 -- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c78156940 msgr2=0x7f6c780c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c78156940 0x7f6c780c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7db9a700 1 -- 172.21.15.29:0/3810093154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c780cb1f0 con 0x7f6c78151880 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.429+0000 7f6c7eb9c700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c78156940 0x7f6c780c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.430+0000 7f6c7db9a700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 0x7f6c78156400 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6c6801efa0 tx=0x7f6c6801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.430+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68023070 con 0x7f6c78151880 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.430+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6800e9e0 con 0x7f6c78151880 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.430+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6801e030 con 0x7f6c78151880 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.430+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c780cb4d0 con 0x7f6c78151880 2023-10-07T09:44:29.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.430+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c780cba90 con 0x7f6c78151880 2023-10-07T09:44:29.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.431+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c781599c0 con 0x7f6c78151880 2023-10-07T09:44:29.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.432+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6c68013470 con 0x7f6c78151880 2023-10-07T09:44:29.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.433+0000 7f6c6e7fc700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c5c066f30 0x7f6c5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.434+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6c68085d70 con 0x7f6c78151880 2023-10-07T09:44:29.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.434+0000 7f6c7e39b700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c5c066f30 0x7f6c5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.434+0000 7f6c7e39b700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c5c066f30 0x7f6c5c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004900 tx=0x7f6c74007a80 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.438+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c68052dc0 con 0x7f6c78151880 2023-10-07T09:44:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.719+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6c78159f60 con 0x7f6c78151880 2023-10-07T09:44:29.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.723+0000 7f6c6e7fc700 1 -- 172.21.15.29:0/3810093154 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6c68017020 con 0x7f6c78151880 2023-10-07T09:44:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c5c066f30 msgr2=0x7f6c5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c5c066f30 0x7f6c5c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004900 tx=0x7f6c74007a80 comp rx=0 tx=0).stop 2023-10-07T09:44:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 msgr2=0x7f6c78156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 0x7f6c78156400 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6c6801efa0 tx=0x7f6c6801a000 comp rx=0 tx=0).stop 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 shutdown_connections 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c5c066f30 0x7f6c5c0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c78156940 0x7f6c780c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c64004810 0x7f6c781440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 --2- 172.21.15.29:0/3810093154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c78151880 0x7f6c78156400 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 >> 172.21.15.29:0/3810093154 conn(0x7f6c78067610 msgr2=0x7f6c781583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 shutdown_connections 2023-10-07T09:44:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.726+0000 7f6c805ff700 1 -- 172.21.15.29:0/3810093154 wait complete. 2023-10-07T09:44:29.729 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:44:29.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:29.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.939+0000 7f4a49bff700 1 Processor -- start 2023-10-07T09:44:29.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.939+0000 7f4a49bff700 1 -- start start 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a49bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4a44159b20 0x7f4a440d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a49bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440cbf20 con 0x7f4a440ce8a0 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a49bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440cc200 con 0x7f4a44159750 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a49bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440cc4e0 con 0x7f4a44159b20 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a43fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4a440ce8a0 0x7f4a440c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58068/0 (socket says 172.21.15.29:58068) 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a43fff700 1 -- 172.21.15.29:0/2317196937 learned_addr learned my addr 172.21.15.29:0/2317196937 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:29.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a42ffd700 1 --2- 172.21.15.29:0/2317196937 >> v2:172.21.15.144:3300/0 conn(0x7f4a44159b20 0x7f4a440d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220242312 0 0) 0x7f4a440cbf20 con 0x7f4a440ce8a0 2023-10-07T09:44:29.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a2c003570 con 0x7f4a440ce8a0 2023-10-07T09:44:29.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.941+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913436380 0 0) 0x7f4a440cc200 con 0x7f4a44159750 2023-10-07T09:44:29.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a440cbf20 con 0x7f4a44159750 2023-10-07T09:44:29.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a42ffd700 1 -- 172.21.15.29:0/2317196937 >> v1:172.21.15.144:6789/0 conn(0x7f4a44159750 legacy=0x7f4a440ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a42ffd700 1 -- 172.21.15.29:0/2317196937 >> v1:172.21.15.29:6789/0 conn(0x7f4a440ce8a0 legacy=0x7f4a440c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a42ffd700 1 -- 172.21.15.29:0/2317196937 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440cc7c0 con 0x7f4a44159b20 2023-10-07T09:44:29.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a42ffd700 1 --2- 172.21.15.29:0/2317196937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159b20 0x7f4a440d10a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4a380098e0 tx=0x7f4a380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6721934d7e16eeef server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:29.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38011630 con 0x7f4a44159b20 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38011790 con 0x7f4a44159b20 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.942+0000 7f4a427fc700 1 -- 172.21.15.29:0/2317196937 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38011b30 con 0x7f4a44159b20 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 -- 172.21.15.29:0/2317196937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159b20 msgr2=0x7f4a440d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 --2- 172.21.15.29:0/2317196937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159b20 0x7f4a440d10a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4a380098e0 tx=0x7f4a380091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 -- 172.21.15.29:0/2317196937 shutdown_connections 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 --2- 172.21.15.29:0/2317196937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159b20 0x7f4a440d10a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 -- 172.21.15.29:0/2317196937 >> 172.21.15.29:0/2317196937 conn(0x7f4a44063a10 msgr2=0x7f4a440d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 -- 172.21.15.29:0/2317196937 shutdown_connections 2023-10-07T09:44:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.943+0000 7f4a49bff700 1 -- 172.21.15.29:0/2317196937 wait complete. 2023-10-07T09:44:29.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.945+0000 7f4a49bff700 1 Processor -- start 2023-10-07T09:44:29.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.945+0000 7f4a49bff700 1 -- start start 2023-10-07T09:44:29.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a49bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 0x7f4a44169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a49bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159750 0x7f4a440c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a49bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a44159b20 0x7f4a440b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a49bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440b3f70 con 0x7f4a440ce8a0 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a49bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440b3360 con 0x7f4a44159750 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a49bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440b3640 con 0x7f4a44159b20 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 0x7f4a44169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 0x7f4a44169d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42662/0 (socket says 172.21.15.29:42662) 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 -- 172.21.15.29:0/1321080164 learned_addr learned my addr 172.21.15.29:0/1321080164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a43fff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a44159b20 0x7f4a440b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a42ffd700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159750 0x7f4a440c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 -- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a44159b20 msgr2=0x7f4a440b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a44159b20 0x7f4a440b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 -- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159750 msgr2=0x7f4a440c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159750 0x7f4a440c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.946+0000 7f4a437fe700 1 -- 172.21.15.29:0/1321080164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a38009420 con 0x7f4a440ce8a0 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.947+0000 7f4a437fe700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 0x7f4a44169d30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c7c0 tx=0x7f4a34007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.947+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34011680 con 0x7f4a440ce8a0 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.947+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a340117e0 con 0x7f4a440ce8a0 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.947+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440b3920 con 0x7f4a440ce8a0 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.947+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34011ba0 con 0x7f4a440ce8a0 2023-10-07T09:44:29.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.947+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4416a270 con 0x7f4a440ce8a0 2023-10-07T09:44:29.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.948+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4404c3c0 con 0x7f4a440ce8a0 2023-10-07T09:44:29.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.949+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4a34012440 con 0x7f4a440ce8a0 2023-10-07T09:44:29.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.950+0000 7f4a2b7fe700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4a30066f30 0x7f4a300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:29.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.950+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f4a34084700 con 0x7f4a440ce8a0 2023-10-07T09:44:29.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.950+0000 7f4a42ffd700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4a30066f30 0x7f4a300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:29.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.951+0000 7f4a42ffd700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4a30066f30 0x7f4a300693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4a380093f0 tx=0x7f4a38017000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:29.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:29.954+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a340517f0 con 0x7f4a440ce8a0 2023-10-07T09:44:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.201+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a440cbb60 con 0x7f4a440ce8a0 2023-10-07T09:44:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.202+0000 7f4a2b7fe700 1 -- 172.21.15.29:0/1321080164 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f4a34054e10 con 0x7f4a440ce8a0 2023-10-07T09:44:30.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:30.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:44:23.775309+0000","modified":"2023-10-07T09:44:27.299910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6276":{"gid":6276,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2418315778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2418315778},{"type":"v1","addr":"172.21.15.29:6835","nonce":2418315778}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-10-07T09:44:30.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4a30066f30 msgr2=0x7f4a300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4a30066f30 0x7f4a300693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4a380093f0 tx=0x7f4a38017000 comp rx=0 tx=0).stop 2023-10-07T09:44:30.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 msgr2=0x7f4a44169d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 0x7f4a44169d30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c7c0 tx=0x7f4a34007dc0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 shutdown_connections 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4a30066f30 0x7f4a300693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a44159b20 0x7f4a440b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a440ce8a0 0x7f4a44169d30 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 --2- 172.21.15.29:0/1321080164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a44159750 0x7f4a440c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 >> 172.21.15.29:0/1321080164 conn(0x7f4a44063a10 msgr2=0x7f4a44155ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 shutdown_connections 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.205+0000 7f4a49bff700 1 -- 172.21.15.29:0/1321080164 wait complete. 2023-10-07T09:44:30.207 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-07T09:44:30.254 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-07T09:44:30.254+0000 7f8b485f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:44:30.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:44:23.775309+0000', 'modified': '2023-10-07T09:44:27.299910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6276}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6276': {'gid': 6276, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/2418315778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 2418315778}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2418315778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:44:30.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:44:30.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-07T09:44:30.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.824+0000 7f00565ff700 1 Processor -- start 2023-10-07T09:44:30.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.825+0000 7f00565ff700 1 -- start start 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.826+0000 7f00565ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0050159750 0x7f00500d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.826+0000 7f00565ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0050159b20 0x7f00500ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.826+0000 7f00565ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00500ce6d0 0x7f00500ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.826+0000 7f00565ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00500cbf20 con 0x7f0050159b20 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.826+0000 7f00565ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00500cc200 con 0x7f00500ce6d0 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.826+0000 7f00565ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00500cc4e0 con 0x7f0050159750 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0050159b20 0x7f00500ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0050159b20 0x7f00500ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42668/0 (socket says 172.21.15.29:42668) 2023-10-07T09:44:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 -- 172.21.15.29:0/577024419 learned_addr learned my addr 172.21.15.29:0/577024419 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f0054b9c700 1 --2- 172.21.15.29:0/577024419 >> v2:172.21.15.144:3300/0 conn(0x7f00500ce6d0 0x7f00500ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004ffff700 1 --2- 172.21.15.29:0/577024419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0050159750 0x7f00500d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 -- 172.21.15.29:0/577024419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0050159750 msgr2=0x7f00500d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 --2- 172.21.15.29:0/577024419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0050159750 0x7f00500d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 -- 172.21.15.29:0/577024419 >> v2:172.21.15.144:3300/0 conn(0x7f00500ce6d0 msgr2=0x7f00500ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 --2- 172.21.15.29:0/577024419 >> v2:172.21.15.144:3300/0 conn(0x7f00500ce6d0 0x7f00500ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.827+0000 7f004f7fe700 1 -- 172.21.15.29:0/577024419 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00500cc7c0 con 0x7f0050159b20 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.828+0000 7f004f7fe700 1 --2- 172.21.15.29:0/577024419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500ce190 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f00500d0d60 tx=0x7f0044009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0abe7dc498ab96 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.828+0000 7f004effd700 1 -- 172.21.15.29:0/577024419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004400b860 con 0x7f0050159b20 2023-10-07T09:44:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.828+0000 7f004effd700 1 -- 172.21.15.29:0/577024419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004400b9c0 con 0x7f0050159b20 2023-10-07T09:44:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.828+0000 7f004effd700 1 -- 172.21.15.29:0/577024419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004400bd60 con 0x7f0050159b20 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.828+0000 7f00565ff700 1 -- 172.21.15.29:0/577024419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 msgr2=0x7f00500ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.828+0000 7f00565ff700 1 --2- 172.21.15.29:0/577024419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500ce190 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f00500d0d60 tx=0x7f0044009250 comp rx=0 tx=0).stop 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 -- 172.21.15.29:0/577024419 shutdown_connections 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 --2- 172.21.15.29:0/577024419 >> v2:172.21.15.144:3300/0 conn(0x7f00500ce6d0 0x7f00500ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 --2- 172.21.15.29:0/577024419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0050159750 0x7f00500d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 --2- 172.21.15.29:0/577024419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500ce190 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 -- 172.21.15.29:0/577024419 >> 172.21.15.29:0/577024419 conn(0x7f0050063a10 msgr2=0x7f00500d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 -- 172.21.15.29:0/577024419 shutdown_connections 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.829+0000 7f00565ff700 1 -- 172.21.15.29:0/577024419 wait complete. 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.830+0000 7f00565ff700 1 Processor -- start 2023-10-07T09:44:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.830+0000 7f00565ff700 1 -- start start 2023-10-07T09:44:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.831+0000 7f00565ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00500ce6d0 0x7f0050158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.831+0000 7f00565ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0050159750 0x7f0050151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.831+0000 7f00565ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.831+0000 7f00565ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00500b3ac0 con 0x7f0050159b20 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.831+0000 7f00565ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0050154520 con 0x7f0050159750 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.831+0000 7f00565ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0050154240 con 0x7f00500ce6d0 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500b4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42680/0 (socket says 172.21.15.29:42680) 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 -- 172.21.15.29:0/1120312603 learned_addr learned my addr 172.21.15.29:0/1120312603 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f004ffff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00500ce6d0 0x7f0050158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f004f7fe700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0050159750 0x7f0050151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 -- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00500ce6d0 msgr2=0x7f0050158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00500ce6d0 0x7f0050158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 -- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0050159750 msgr2=0x7f0050151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0050159750 0x7f0050151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 -- 172.21.15.29:0/1120312603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00440094f0 con 0x7f0050159b20 2023-10-07T09:44:30.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.832+0000 7f0054b9c700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500b4c80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0048007f20 tx=0x7f0048009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.833+0000 7f0037fff700 1 -- 172.21.15.29:0/1120312603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004800eca0 con 0x7f0050159b20 2023-10-07T09:44:30.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.833+0000 7f0037fff700 1 -- 172.21.15.29:0/1120312603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004800ee20 con 0x7f0050159b20 2023-10-07T09:44:30.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.833+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00500b3470 con 0x7f0050159b20 2023-10-07T09:44:30.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.833+0000 7f0037fff700 1 -- 172.21.15.29:0/1120312603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0048014670 con 0x7f0050159b20 2023-10-07T09:44:30.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.833+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00501519c0 con 0x7f0050159b20 2023-10-07T09:44:30.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.836+0000 7f0037fff700 1 -- 172.21.15.29:0/1120312603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f00480149e0 con 0x7f0050159b20 2023-10-07T09:44:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.837+0000 7f0037fff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f003c066f30 0x7f003c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.837+0000 7f0037fff700 1 -- 172.21.15.29:0/1120312603 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f0048084e80 con 0x7f0050159b20 2023-10-07T09:44:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.837+0000 7f004ffff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f003c066f30 0x7f003c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.838+0000 7f004ffff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f003c066f30 0x7f003c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0040006aa0 tx=0x7f0040008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.839+0000 7f0064e1eb80 1 Processor -- start 2023-10-07T09:44:30.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.839+0000 7f0064e1eb80 1 -- start start 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.839+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 0x556ce25cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25d8bc0 0x556ce25d9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce24a2310 con 0x556ce25cfa40 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce24df130 con 0x556ce25b5530 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce24dcaf0 con 0x556ce25d8bc0 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f004ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 0x556ce25cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f004ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 0x556ce25cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42682/0 (socket says 172.21.15.29:42682) 2023-10-07T09:44:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f004ffff700 1 -- 172.21.15.29:0/708176744 learned_addr learned my addr 172.21.15.29:0/708176744 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:30.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f0054b9c700 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25d8bc0 0x556ce25d9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.840+0000 7f004f7fe700 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f004ffff700 1 -- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25d8bc0 msgr2=0x556ce25d9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f004ffff700 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25d8bc0 0x556ce25d9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f004ffff700 1 -- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 msgr2=0x556ce25d8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f004ffff700 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f004ffff700 1 -- 172.21.15.29:0/708176744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ce23cb0a0 con 0x556ce25cfa40 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f004ffff700 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 0x556ce25cfe10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f004000fa40 tx=0x7f0040016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c05819eab0782f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f00357fa700 1 -- 172.21.15.29:0/708176744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040014610 con 0x556ce25cfa40 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f00357fa700 1 -- 172.21.15.29:0/708176744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0040019070 con 0x556ce25cfa40 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.841+0000 7f00357fa700 1 -- 172.21.15.29:0/708176744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040014980 con 0x556ce25cfa40 2023-10-07T09:44:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 -- 172.21.15.29:0/708176744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 msgr2=0x556ce25cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 0x556ce25cfe10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f004000fa40 tx=0x7f0040016000 comp rx=0 tx=0).stop 2023-10-07T09:44:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 -- 172.21.15.29:0/708176744 shutdown_connections 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25d8bc0 0x556ce25d9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25cfa40 0x556ce25cfe10 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/708176744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 -- 172.21.15.29:0/708176744 >> 172.21.15.29:0/708176744 conn(0x556ce25d2e70 msgr2=0x556ce25d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 -- 172.21.15.29:0/708176744 shutdown_connections 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.842+0000 7f0064e1eb80 1 -- 172.21.15.29:0/708176744 wait complete. 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.843+0000 7f0064e1eb80 1 -- start start 2023-10-07T09:44:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.843+0000 7f0064e1eb80 1 Processor -- start 2023-10-07T09:44:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce25b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce24d88e0 con 0x556ce25d8bc0 2023-10-07T09:44:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce2484980 con 0x556ce25b5530 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce2484620 con 0x556ce25cfa40 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0064e1eb80 10 client.0 _ll_register_callbacks cb 0x556ce22e7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0054b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce25b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0054b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce25b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42684/0 (socket says 172.21.15.29:42684) 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f0054b9c700 1 -- 172.21.15.29:0/617439416 learned_addr learned my addr 172.21.15.29:0/617439416 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f004f7fe700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.845+0000 7f004ffff700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0054b9c700 1 -- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 msgr2=0x556ce25d6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0054b9c700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0054b9c700 1 -- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 msgr2=0x556ce25d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0054b9c700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0054b9c700 1 -- 172.21.15.29:0/617439416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004000e040 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0054b9c700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce25b34f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0048083210 tx=0x7f0048058040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a9217cd108f6c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f002e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f002e7fc700 1 -- 172.21.15.29:0/617439416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004805cb30 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f002e7fc700 1 -- 172.21.15.29:0/617439416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f0048012dc0 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ce257d560 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f002e7fc700 1 -- 172.21.15.29:0/617439416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0048055290 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0064e1eb80 10 client.6414 fetch_fsmap learned FSMap version 34 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.846+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x556ce24a73c0 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f002e7fc700 1 -- 172.21.15.29:0/617439416 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556ce257d560 con 0x556ce25d8bc0 2023-10-07T09:44:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f002e7fc700 1 -- 172.21.15.29:0/617439416 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 34) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f004806f030 con 0x556ce25d8bc0 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f0064e1eb80 10 client.6414 fetch_fsmap finished waiting for FSMap version 34 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f0064e1eb80 10 client.6414 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f0064e1eb80 20 client.6414 populate_metadata read hostname 'smithi029' 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce25b3a30 0x556ce25e2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f0064e1eb80 4 client.6414 mds_command: new command op to 6276 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 --> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ce255a0e0 con 0x556ce25b3a30 2023-10-07T09:44:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.847+0000 7f004ffff700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce25b3a30 0x556ce25e2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.848+0000 7f004ffff700 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce25b3a30 0x556ce25e2540 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a39a2c742f52ea89 server_cookie=2d5f23c74cd02002 in_seq=0 out_seq=0 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.848+0000 7f002e7fc700 10 client.6414 ms_handle_connect on v2:172.21.15.29:6833/2418315778 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.848+0000 7f002e7fc700 1 -- 172.21.15.29:0/617439416 <== mds.0 v2:172.21.15.29:6833/2418315778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x556ce255a0e0 con 0x556ce25b3a30 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.848+0000 7f002e7fc700 10 client.6414 handle_command_reply: tid=0 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 client.6414 shutdown 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 msgr2=0x556ce25b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce25b34f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0048083210 tx=0x7f0048058040 comp rx=0 tx=0).stop 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 shutdown_connections 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce25b3a30 0x556ce25e2540 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce25b34f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25d4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/617439416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f002e7fc700 0 client.6414 ms_handle_reset on v2:172.21.15.29:6833/2418315778 2023-10-07T09:44:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 >> 172.21.15.29:0/617439416 conn(0x556ce25d2e70 msgr2=0x556ce2585250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 shutdown_connections 2023-10-07T09:44:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 1 -- 172.21.15.29:0/617439416 wait complete. 2023-10-07T09:44:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.849+0000 7f0064e1eb80 20 client.6414 trim_cache size 0 max 16384 2023-10-07T09:44:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.867+0000 7f0064e1eb80 1 Processor -- start 2023-10-07T09:44:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.867+0000 7f0064e1eb80 1 -- start start 2023-10-07T09:44:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.868+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 0x556ce25d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.868+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.868+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25d8bc0 0x556ce25e8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.868+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce23701e0 con 0x556ce25b5530 2023-10-07T09:44:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.868+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce25b46d0 con 0x556ce25d8bc0 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.868+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce25d2d30 con 0x556ce25cfa40 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 0x556ce25d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 0x556ce25d27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42696/0 (socket says 172.21.15.29:42696) 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 -- 172.21.15.29:0/2853922504 learned_addr learned my addr 172.21.15.29:0/2853922504 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004f7fe700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f0054b9c700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25d8bc0 0x556ce25e8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 -- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 msgr2=0x556ce25d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 -- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25d8bc0 msgr2=0x556ce25e8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25d8bc0 0x556ce25e8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 -- 172.21.15.29:0/2853922504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ce233f1d0 con 0x556ce25b5530 2023-10-07T09:44:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004f7fe700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f0054b9c700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25d8bc0 0x556ce25e8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.869+0000 7f004ffff700 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 0x556ce25d27f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0040022040 tx=0x7f0040032250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdc38757e8e39c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0035ffb700 1 -- 172.21.15.29:0/2853922504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040039070 con 0x556ce25b5530 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0035ffb700 1 -- 172.21.15.29:0/2853922504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0040030040 con 0x556ce25b5530 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0035ffb700 1 -- 172.21.15.29:0/2853922504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00400149f0 con 0x556ce25b5530 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 -- 172.21.15.29:0/2853922504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 msgr2=0x556ce25d27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 0x556ce25d27f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0040022040 tx=0x7f0040032250 comp rx=0 tx=0).stop 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 -- 172.21.15.29:0/2853922504 shutdown_connections 2023-10-07T09:44:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25d8bc0 0x556ce25e8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25b5530 0x556ce25d27f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/2853922504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce25d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 -- 172.21.15.29:0/2853922504 >> 172.21.15.29:0/2853922504 conn(0x556ce25d2e70 msgr2=0x556ce25d4900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 -- 172.21.15.29:0/2853922504 shutdown_connections 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.870+0000 7f0064e1eb80 1 -- 172.21.15.29:0/2853922504 wait complete. 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.871+0000 7f0064e1eb80 1 -- start start 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.871+0000 7f0064e1eb80 1 Processor -- start 2023-10-07T09:44:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25e13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce257f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce257f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce26358c0 con 0x556ce25d8bc0 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce25af250 con 0x556ce25b5530 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce25e1930 con 0x556ce25cfa40 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0064e1eb80 10 client.0 _ll_register_callbacks cb 0x556ce22e7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:44:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0054b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce257f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0054b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce257f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42698/0 (socket says 172.21.15.29:42698) 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f0054b9c700 1 -- 172.21.15.29:0/3909485768 learned_addr learned my addr 172.21.15.29:0/3909485768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.872+0000 7f004ffff700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25e13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f004f7fe700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce257f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0054b9c700 1 -- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 msgr2=0x556ce257f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0054b9c700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce257f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0054b9c700 1 -- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 msgr2=0x556ce25e13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0054b9c700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25e13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0054b9c700 1 -- 172.21.15.29:0/3909485768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ce23cb0a0 con 0x556ce25d8bc0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0054b9c700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce257f740 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0048007ef0 tx=0x7f004806a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd57bfd5e28f008 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f002e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f002e7fc700 1 -- 172.21.15.29:0/3909485768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0048055290 con 0x556ce25d8bc0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0050152cc0 con 0x556ce25d8bc0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.873+0000 7f002e7fc700 1 -- 172.21.15.29:0/3909485768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f004805cb30 con 0x556ce25d8bc0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f002e7fc700 1 -- 172.21.15.29:0/3909485768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00480099c0 con 0x556ce25d8bc0 2023-10-07T09:44:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 10 client.6420 fetch_fsmap learned FSMap version 34 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x556ce2318d60 con 0x556ce25d8bc0 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f002e7fc700 1 -- 172.21.15.29:0/3909485768 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0048076020 con 0x556ce25d8bc0 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f002e7fc700 1 -- 172.21.15.29:0/3909485768 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 34) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f00480522a0 con 0x556ce25d8bc0 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 10 client.6420 fetch_fsmap finished waiting for FSMap version 34 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 10 client.6420 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 20 client.6420 populate_metadata read hostname 'smithi029' 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce2584e40 0x556ce2585210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 4 client.6420 mds_command: new command op to 6276 tid=0[{"prefix": "status"}] 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 --> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0040019070 con 0x556ce2584e40 2023-10-07T09:44:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.874+0000 7f004ffff700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce2584e40 0x556ce2585210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f002e7fc700 10 client.6420 ms_handle_connect on v2:172.21.15.29:6833/2418315778 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f004ffff700 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce2584e40 0x556ce2585210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a111c20ca5760ee server_cookie=362c563158ecf9ad in_seq=0 out_seq=0 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f002e7fc700 1 -- 172.21.15.29:0/3909485768 <== mds.0 v2:172.21.15.29:6833/2418315778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f0040019070 con 0x556ce2584e40 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f002e7fc700 10 client.6420 handle_command_reply: tid=0 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 client.6420 shutdown 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 msgr2=0x556ce257f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce257f740 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0048007ef0 tx=0x7f004806a000 comp rx=0 tx=0).stop 2023-10-07T09:44:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 shutdown_connections 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:6833/2418315778,v1:172.21.15.29:6835/2418315778] conn(0x556ce2584e40 0x556ce2585210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ce25d8bc0 0x556ce257f740 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556ce25b5530 0x556ce25e13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 --2- 172.21.15.29:0/3909485768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556ce25cfa40 0x556ce257f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 >> 172.21.15.29:0/3909485768 conn(0x556ce25d2e70 msgr2=0x556ce25d4900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f002e7fc700 0 client.6420 ms_handle_reset on v2:172.21.15.29:6833/2418315778 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 shutdown_connections 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.875+0000 7f0064e1eb80 1 -- 172.21.15.29:0/3909485768 wait complete. 2023-10-07T09:44:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f0064e1eb80 20 client.6420 trim_cache size 0 max 16384 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 6276, 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:44:30.888 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:44:30.889 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.6017004249999998, 2023-10-07T09:44:30.889 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 34, 2023-10-07T09:44:30.889 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 77, 2023-10-07T09:44:30.889 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 77, 2023-10-07T09:44:30.889 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.270152053 2023-10-07T09:44:30.889 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:44:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f003c066f30 msgr2=0x7f003c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f003c066f30 0x7f003c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0040006aa0 tx=0x7f0040008040 comp rx=0 tx=0).stop 2023-10-07T09:44:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 msgr2=0x7f00500b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500b4c80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0048007f20 tx=0x7f0048009860 comp rx=0 tx=0).stop 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 shutdown_connections 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f003c066f30 0x7f003c0693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0050159b20 0x7f00500b4c80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00500ce6d0 0x7f0050158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 --2- 172.21.15.29:0/1120312603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0050159750 0x7f0050151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 >> 172.21.15.29:0/1120312603 conn(0x7f0050063a10 msgr2=0x7f0050155af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 shutdown_connections 2023-10-07T09:44:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:30.876+0000 7f00565ff700 1 -- 172.21.15.29:0/1120312603 wait complete. 2023-10-07T09:44:30.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:44:31.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.088+0000 7fcc3c7ff700 1 Processor -- start 2023-10-07T09:44:31.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.089+0000 7fcc3c7ff700 1 -- start start 2023-10-07T09:44:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.090+0000 7fcc3c7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc340ad220 0x7fcc34159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.090+0000 7fcc3c7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3415af00 con 0x7fcc34068620 2023-10-07T09:44:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.090+0000 7fcc3c7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc340ada40 con 0x7fcc340ad5f0 2023-10-07T09:44:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.090+0000 7fcc3c7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3415a250 con 0x7fcc340ad220 2023-10-07T09:44:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.090+0000 7fcc3ad9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcc34068620 0x7fcc34068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58072/0 (socket says 172.21.15.29:58072) 2023-10-07T09:44:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.090+0000 7fcc3ad9c700 1 -- 172.21.15.29:0/2038043297 learned_addr learned my addr 172.21.15.29:0/2038043297 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:31.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc3a59b700 1 --2- 172.21.15.29:0/2038043297 >> v2:172.21.15.144:3300/0 conn(0x7fcc340ad220 0x7fcc34159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc39599700 1 -- 172.21.15.29:0/2038043297 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391703155 0 0) 0x7fcc3415af00 con 0x7fcc34068620 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc39599700 1 -- 172.21.15.29:0/2038043297 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc20003570 con 0x7fcc34068620 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc3a59b700 1 -- 172.21.15.29:0/2038043297 >> v1:172.21.15.144:6789/0 conn(0x7fcc340ad5f0 legacy=0x7fcc340ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc3a59b700 1 -- 172.21.15.29:0/2038043297 >> v1:172.21.15.29:6789/0 conn(0x7fcc34068620 legacy=0x7fcc34068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc39599700 1 -- 172.21.15.29:0/2038043297 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 954670325 0 0) 0x7fcc20003570 con 0x7fcc34068620 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc3a59b700 1 -- 172.21.15.29:0/2038043297 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3415a530 con 0x7fcc340ad220 2023-10-07T09:44:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.091+0000 7fcc3a59b700 1 --2- 172.21.15.29:0/2038043297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc34159c00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcc300098e0 tx=0x7fcc300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95c69427e93bffa server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:31.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.092+0000 7fcc39599700 1 -- 172.21.15.29:0/2038043297 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c0017f0 con 0x7fcc340ad220 2023-10-07T09:44:31.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.092+0000 7fcc39599700 1 -- 172.21.15.29:0/2038043297 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc30011420 con 0x7fcc340ad220 2023-10-07T09:44:31.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.092+0000 7fcc39599700 1 -- 172.21.15.29:0/2038043297 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300117c0 con 0x7fcc340ad220 2023-10-07T09:44:31.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.092+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/2038043297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 msgr2=0x7fcc34159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.092+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/2038043297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc34159c00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcc300098e0 tx=0x7fcc300091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.093+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/2038043297 shutdown_connections 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.093+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/2038043297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc34159c00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.093+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/2038043297 >> 172.21.15.29:0/2038043297 conn(0x7fcc3406dd00 msgr2=0x7fcc3406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.093+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/2038043297 shutdown_connections 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.093+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/2038043297 wait complete. 2023-10-07T09:44:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.094+0000 7fcc3c7ff700 1 Processor -- start 2023-10-07T09:44:31.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.094+0000 7fcc3c7ff700 1 -- start start 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.095+0000 7fcc3c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34068620 0x7fcc341502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.095+0000 7fcc3c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc340afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.095+0000 7fcc3c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 0x7fcc340aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.095+0000 7fcc3c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3414e430 con 0x7fcc340ad5f0 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.095+0000 7fcc3c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340ca410 con 0x7fcc340ad220 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.095+0000 7fcc3c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3416d740 con 0x7fcc34068620 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 0x7fcc340aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 0x7fcc340aaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42714/0 (socket says 172.21.15.29:42714) 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 -- 172.21.15.29:0/3887582839 learned_addr learned my addr 172.21.15.29:0/3887582839 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3a59b700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34068620 0x7fcc341502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc39d9a700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc340afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 -- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34068620 msgr2=0x7fcc341502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34068620 0x7fcc341502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 -- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 msgr2=0x7fcc340afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc340afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 -- 172.21.15.29:0/3887582839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009420 con 0x7fcc340ad5f0 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.096+0000 7fcc3ad9c700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 0x7fcc340aaeb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00cae0 tx=0x7fcc2c007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.097+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004dc0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.097+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2c00a5d0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.097+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3416da20 con 0x7fcc340ad5f0 2023-10-07T09:44:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.097+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c00a990 con 0x7fcc340ad5f0 2023-10-07T09:44:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.097+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3416dfe0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.098+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3414e5a0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.099+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc2c007150 con 0x7fcc340ad5f0 2023-10-07T09:44:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.100+0000 7fcc2a7fc700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc18066f30 0x7fcc180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.100+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fcc2c0840f0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.100+0000 7fcc3a59b700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc18066f30 0x7fcc180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.101+0000 7fcc3a59b700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc18066f30 0x7fcc180693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcc300091e0 tx=0x7fcc30016000 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.104+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc2c0511c0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.354+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcc3404c3c0 con 0x7fcc340ad5f0 2023-10-07T09:44:31.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.354+0000 7fcc2a7fc700 1 -- 172.21.15.29:0/3887582839 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcc2c018070 con 0x7fcc340ad5f0 2023-10-07T09:44:31.356 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:44:31.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc18066f30 msgr2=0x7fcc180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc18066f30 0x7fcc180693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcc300091e0 tx=0x7fcc30016000 comp rx=0 tx=0).stop 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 msgr2=0x7fcc340aaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 0x7fcc340aaeb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00cae0 tx=0x7fcc2c007ca0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 shutdown_connections 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc18066f30 0x7fcc180693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ad5f0 0x7fcc340aaeb0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34068620 0x7fcc341502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 --2- 172.21.15.29:0/3887582839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc340ad220 0x7fcc340afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 >> 172.21.15.29:0/3887582839 conn(0x7fcc3406dd00 msgr2=0x7fcc34151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 shutdown_connections 2023-10-07T09:44:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:31.357+0000 7fcc3c7ff700 1 -- 172.21.15.29:0/3887582839 wait complete. 2023-10-07T09:44:31.375 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:44:31.376 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:44:31.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:31.377 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:44:31.424 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:44:31.424 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:44:31.424 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:44:31.425 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.425 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:44:31.425 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.425 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:44:31.425 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:44:31.425 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:44:31.426 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41357sec preferred_lft 41357sec 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:44:31.427 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:44:31.428 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:44:31.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:31.429 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:44:31.498 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:44:31.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:31.501 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:44:31.550 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:44:31.551 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:44:31.551 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:44:31.551 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:44:31.551 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:44:31.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:31.552 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:44:31.552 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:44:31.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:44:31.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:44:31.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:44:31.553 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:44:31.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:31.724 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:44:31.771 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:44:31.772 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:44:31.772 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:44:31.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:31.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:44:31.844 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:44:31.844 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:44:31.890 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:44:31.890 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:44:31.891 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:44:31.936 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:44:31.936 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:44:31.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:44:32.076 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:44:32.076 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:44:32.076 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:44:32.090 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:44:32.091 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:44:32.091 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:44:32.091 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:44:32.091 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:44:32.111 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:44:32.112 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:44:32.118 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:44:32.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:32.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:44:32.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:44:32.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.388+0000 7f9e835ff700 1 Processor -- start 2023-10-07T09:44:32.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.388+0000 7f9e835ff700 1 -- start start 2023-10-07T09:44:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e835ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e7c0ad370 0x7f9e7c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e835ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c159ec0 0x7f9e7c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e835ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c14d790 con 0x7f9e7c1568c0 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e835ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7c0ae5d0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e835ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c15a850 con 0x7f9e7c159ec0 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e8139b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e7c0ad370 0x7f9e7c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e8139b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e7c0ad370 0x7f9e7c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42718/0 (socket says 172.21.15.29:42718) 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e8139b700 1 -- 172.21.15.29:0/3017472285 learned_addr learned my addr 172.21.15.29:0/3017472285 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e80b9a700 1 --2- 172.21.15.29:0/3017472285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c159ec0 0x7f9e7c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e8139b700 1 -- 172.21.15.29:0/3017472285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c159ec0 msgr2=0x7f9e7c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e8139b700 1 --2- 172.21.15.29:0/3017472285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c159ec0 0x7f9e7c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.390+0000 7f9e8139b700 1 -- 172.21.15.29:0/3017472285 >> v1:172.21.15.144:6789/0 conn(0x7f9e7c1568c0 legacy=0x7f9e7c156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e8139b700 1 -- 172.21.15.29:0/3017472285 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0ad740 con 0x7f9e7c0ad370 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e8139b700 1 --2- 172.21.15.29:0/3017472285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0cc870 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9e780099b0 tx=0x7f9e78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ee8937852bce3d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e73fff700 1 -- 172.21.15.29:0/3017472285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78009ea0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e73fff700 1 -- 172.21.15.29:0/3017472285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7800b860 con 0x7f9e7c0ad370 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e73fff700 1 -- 172.21.15.29:0/3017472285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800bc00 con 0x7f9e7c0ad370 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e835ff700 1 -- 172.21.15.29:0/3017472285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 msgr2=0x7f9e7c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.391+0000 7f9e835ff700 1 --2- 172.21.15.29:0/3017472285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0cc870 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9e780099b0 tx=0x7f9e78009250 comp rx=0 tx=0).stop 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.392+0000 7f9e835ff700 1 -- 172.21.15.29:0/3017472285 shutdown_connections 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.392+0000 7f9e835ff700 1 --2- 172.21.15.29:0/3017472285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0cc870 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.392+0000 7f9e835ff700 1 --2- 172.21.15.29:0/3017472285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c159ec0 0x7f9e7c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.392+0000 7f9e835ff700 1 -- 172.21.15.29:0/3017472285 >> 172.21.15.29:0/3017472285 conn(0x7f9e7c0680a0 msgr2=0x7f9e7c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.392+0000 7f9e835ff700 1 -- 172.21.15.29:0/3017472285 shutdown_connections 2023-10-07T09:44:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.392+0000 7f9e835ff700 1 -- 172.21.15.29:0/3017472285 wait complete. 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.393+0000 7f9e835ff700 1 Processor -- start 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.393+0000 7f9e835ff700 1 -- start start 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e835ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0c1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e835ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c1568c0 0x7f9e7c0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e835ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e7c159ec0 0x7f9e7c0caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e835ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c14c830 con 0x7f9e7c0ad370 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e835ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0ca080 con 0x7f9e7c159ec0 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e835ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0c2690 con 0x7f9e7c1568c0 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e8139b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0c1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e8139b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0c1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42732/0 (socket says 172.21.15.29:42732) 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.394+0000 7f9e8139b700 1 -- 172.21.15.29:0/1836511184 learned_addr learned my addr 172.21.15.29:0/1836511184 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e80b9a700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c1568c0 0x7f9e7c0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e8139b700 1 -- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c1568c0 msgr2=0x7f9e7c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e81b9c700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e7c159ec0 0x7f9e7c0caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e8139b700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c1568c0 0x7f9e7c0ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e8139b700 1 -- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e7c159ec0 msgr2=0x7f9e7c0caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e8139b700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e7c159ec0 0x7f9e7c0caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e8139b700 1 -- 172.21.15.29:0/1836511184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e780094f0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e80b9a700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c1568c0 0x7f9e7c0ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e81b9c700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e7c159ec0 0x7f9e7c0caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.395+0000 7f9e8139b700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0c1800 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c068dd0 tx=0x7f9e78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.396+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800bbe0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.396+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e78011420 con 0x7f9e7c0ad370 2023-10-07T09:44:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.396+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e780117c0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.396+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c0c1df0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.396+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c0c23b0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.397+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c0ca360 con 0x7f9e7c0ad370 2023-10-07T09:44:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.398+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9e78011bf0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.398+0000 7f9e717fa700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e60066f30 0x7f9e600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.399+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f9e78086090 con 0x7f9e7c0ad370 2023-10-07T09:44:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.399+0000 7f9e80b9a700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e60066f30 0x7f9e600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.400+0000 7f9e80b9a700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e60066f30 0x7f9e600693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c006f00 tx=0x7f9e6c008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.403+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e780530e0 con 0x7f9e7c0ad370 2023-10-07T09:44:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.634+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e7c04c3c0 con 0x7f9e60066f30 2023-10-07T09:44:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.636+0000 7f9e717fa700 1 -- 172.21.15.29:0/1836511184 <== mgr.5899 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9e7c04c3c0 con 0x7f9e60066f30 2023-10-07T09:44:32.637 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:44:32.638 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:32.638 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:44:32.638 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:32.638 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e60066f30 msgr2=0x7f9e600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e60066f30 0x7f9e600693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c006f00 tx=0x7f9e6c008040 comp rx=0 tx=0).stop 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 msgr2=0x7f9e7c0c1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0c1800 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c068dd0 tx=0x7f9e78017000 comp rx=0 tx=0).stop 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 shutdown_connections 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e60066f30 0x7f9e600693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.638+0000 7f9e835ff700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e7c159ec0 0x7f9e7c0caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.639+0000 7f9e835ff700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e7c0ad370 0x7f9e7c0c1800 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.639+0000 7f9e835ff700 1 --2- 172.21.15.29:0/1836511184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e7c1568c0 0x7f9e7c0ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.639+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 >> 172.21.15.29:0/1836511184 conn(0x7f9e7c0680a0 msgr2=0x7f9e7c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.639+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 shutdown_connections 2023-10-07T09:44:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.639+0000 7f9e835ff700 1 -- 172.21.15.29:0/1836511184 wait complete. 2023-10-07T09:44:32.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:44:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.856+0000 7fbdf33ff700 1 Processor -- start 2023-10-07T09:44:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.856+0000 7fbdf33ff700 1 -- start start 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf33ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbdec159750 0x7fbdec0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159b20 0x7fbdec0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf33ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbdec0ce6d0 0x7fbdec0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf33ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec0cbf20 con 0x7fbdec159750 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf33ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec0cc200 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0cc4e0 con 0x7fbdec159b20 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbdec159750 0x7fbdec0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbdec159750 0x7fbdec0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42748/0 (socket says 172.21.15.29:42748) 2023-10-07T09:44:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 -- 172.21.15.29:0/960499574 learned_addr learned my addr 172.21.15.29:0/960499574 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf099a700 1 --2- 172.21.15.29:0/960499574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159b20 0x7fbdec0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf199c700 1 --2- 172.21.15.29:0/960499574 >> v2:172.21.15.144:3300/0 conn(0x7fbdec0ce6d0 0x7fbdec0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 -- 172.21.15.29:0/960499574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159b20 msgr2=0x7fbdec0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 --2- 172.21.15.29:0/960499574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159b20 0x7fbdec0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 -- 172.21.15.29:0/960499574 >> v2:172.21.15.144:3300/0 conn(0x7fbdec0ce6d0 msgr2=0x7fbdec0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 --2- 172.21.15.29:0/960499574 >> v2:172.21.15.144:3300/0 conn(0x7fbdec0ce6d0 0x7fbdec0ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.858+0000 7fbdf119b700 1 -- 172.21.15.29:0/960499574 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0cc7c0 con 0x7fbdec159750 2023-10-07T09:44:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf119b700 1 --2- 172.21.15.29:0/960499574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec159750 0x7fbdec0d10a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0d0d60 tx=0x7fbde8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da973d33940c7c18 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbde3fff700 1 -- 172.21.15.29:0/960499574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8005c10 con 0x7fbdec159750 2023-10-07T09:44:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbde3fff700 1 -- 172.21.15.29:0/960499574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8005d70 con 0x7fbdec159750 2023-10-07T09:44:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbde3fff700 1 -- 172.21.15.29:0/960499574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde800b880 con 0x7fbdec159750 2023-10-07T09:44:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 -- 172.21.15.29:0/960499574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec159750 msgr2=0x7fbdec0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/960499574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec159750 0x7fbdec0d10a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0d0d60 tx=0x7fbde8009220 comp rx=0 tx=0).stop 2023-10-07T09:44:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 -- 172.21.15.29:0/960499574 shutdown_connections 2023-10-07T09:44:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/960499574 >> v2:172.21.15.144:3300/0 conn(0x7fbdec0ce6d0 0x7fbdec0ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/960499574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec159750 0x7fbdec0d10a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/960499574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159b20 0x7fbdec0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.859+0000 7fbdf33ff700 1 -- 172.21.15.29:0/960499574 >> 172.21.15.29:0/960499574 conn(0x7fbdec063a10 msgr2=0x7fbdec0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.860+0000 7fbdf33ff700 1 -- 172.21.15.29:0/960499574 shutdown_connections 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.860+0000 7fbdf33ff700 1 -- 172.21.15.29:0/960499574 wait complete. 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.861+0000 7fbdf33ff700 1 Processor -- start 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.861+0000 7fbdf33ff700 1 -- start start 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf33ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 0x7fbdec15a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159750 0x7fbdec142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf33ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdec159b20 0x7fbdec0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf33ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec15d280 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf33ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec15a810 con 0x7fbdec159b20 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0c85e0 con 0x7fbdec159750 2023-10-07T09:44:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf119b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 0x7fbdec15a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf119b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 0x7fbdec15a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42764/0 (socket says 172.21.15.29:42764) 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.862+0000 7fbdf119b700 1 -- 172.21.15.29:0/3369941970 learned_addr learned my addr 172.21.15.29:0/3369941970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf099a700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159750 0x7fbdec142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf199c700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdec159b20 0x7fbdec0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf119b700 1 -- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159750 msgr2=0x7fbdec142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf119b700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159750 0x7fbdec142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf119b700 1 -- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdec159b20 msgr2=0x7fbdec0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf119b700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdec159b20 0x7fbdec0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf119b700 1 -- 172.21.15.29:0/3369941970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009480 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf199c700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdec159b20 0x7fbdec0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf119b700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 0x7fbdec15a2d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbde800b3c0 tx=0x7fbde8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8011920 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8020070 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec15cfa0 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.863+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8011cb0 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.864+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec0c82f0 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.864+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec0c7df0 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.865+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbde8022030 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.866+0000 7fbde17fa700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbdd0066f30 0x7fbdd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.866+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fbde80863e0 con 0x7fbdec0ce6d0 2023-10-07T09:44:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.867+0000 7fbdf099a700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbdd0066f30 0x7fbdd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.868+0000 7fbdf099a700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbdd0066f30 0x7fbdd00693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbdec158bc0 tx=0x7fbddc008040 comp rx=0 tx=0).ready entity=mgr.5899 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:32.871+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbdec0c7df0 con 0x7fbdec0ce6d0 2023-10-07T09:44:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.129+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbdec0c7df0 con 0x7fbdec0ce6d0 2023-10-07T09:44:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.337+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 27) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fbde80860c0 con 0x7fbdec0ce6d0 2023-10-07T09:44:33.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.338+0000 7fbde17fa700 1 -- 172.21.15.29:0/3369941970 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v27) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbde8069940 con 0x7fbdec0ce6d0 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.340+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbdd0066f30 msgr2=0x7fbdd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.340+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbdd0066f30 0x7fbdd00693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbdec158bc0 tx=0x7fbddc008040 comp rx=0 tx=0).stop 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.340+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 msgr2=0x7fbdec15a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.340+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 0x7fbdec15a2d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbde800b3c0 tx=0x7fbde8016000 comp rx=0 tx=0).stop 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 shutdown_connections 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbdd0066f30 0x7fbdd00693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdec159b20 0x7fbdec0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec0ce6d0 0x7fbdec15a2d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 --2- 172.21.15.29:0/3369941970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdec159750 0x7fbdec142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 >> 172.21.15.29:0/3369941970 conn(0x7fbdec063a10 msgr2=0x7fbdec155330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 shutdown_connections 2023-10-07T09:44:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.341+0000 7fbdf33ff700 1 -- 172.21.15.29:0/3369941970 wait complete. 2023-10-07T09:44:33.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:44:33.548 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:33.548+0000 7f4ebec8d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:44:33.551 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:33.550+0000 7fa6d304b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:44:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.564+0000 7fa8d77ff700 1 Processor -- start 2023-10-07T09:44:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.564+0000 7fa8d77ff700 1 -- start start 2023-10-07T09:44:33.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.565+0000 7fa8d77ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa8d0154d30 0x7fa8d0158430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:33.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.565+0000 7fa8d77ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0069dd0 0x7fa8d00b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:33.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.565+0000 7fa8d77ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d00cae70 con 0x7fa8d0155100 2023-10-07T09:44:33.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.565+0000 7fa8d77ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d0159be0 con 0x7fa8d0154d30 2023-10-07T09:44:33.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.565+0000 7fa8d77ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0158970 con 0x7fa8d0069dd0 2023-10-07T09:44:33.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d5d9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8d0155100 0x7fa8d015b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45052/0 (socket says 172.21.15.29:45052) 2023-10-07T09:44:33.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d5d9c700 1 -- 172.21.15.29:0/1985788407 learned_addr learned my addr 172.21.15.29:0/1985788407 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:33.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d4d9a700 1 --2- 172.21.15.29:0/1985788407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0069dd0 0x7fa8d00b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:33.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d559b700 1 --2- 172.21.15.29:0/1985788407 >> v2:172.21.15.144:3300/0 conn(0x7fa8d0154d30 0x7fa8d0158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8c7fff700 1 -- 172.21.15.29:0/1985788407 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415865569 0 0) 0x7fa8d00cae70 con 0x7fa8d0155100 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8c7fff700 1 -- 172.21.15.29:0/1985788407 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc003570 con 0x7fa8d0155100 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d4d9a700 1 -- 172.21.15.29:0/1985788407 >> v2:172.21.15.144:3300/0 conn(0x7fa8d0154d30 msgr2=0x7fa8d0158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d4d9a700 1 --2- 172.21.15.29:0/1985788407 >> v2:172.21.15.144:3300/0 conn(0x7fa8d0154d30 0x7fa8d0158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d4d9a700 1 -- 172.21.15.29:0/1985788407 >> v1:172.21.15.29:6789/0 conn(0x7fa8d0155100 legacy=0x7fa8d015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.566+0000 7fa8d4d9a700 1 -- 172.21.15.29:0/1985788407 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d00ca260 con 0x7fa8d0069dd0 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.567+0000 7fa8d4d9a700 1 --2- 172.21.15.29:0/1985788407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0069dd0 0x7fa8d00b1290 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00098e0 tx=0x7fa8c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db4d485ff065072 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.567+0000 7fa8c7fff700 1 -- 172.21.15.29:0/1985788407 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0011630 con 0x7fa8d0069dd0 2023-10-07T09:44:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.567+0000 7fa8c7fff700 1 -- 172.21.15.29:0/1985788407 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0011790 con 0x7fa8d0069dd0 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.567+0000 7fa8c7fff700 1 -- 172.21.15.29:0/1985788407 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0011b30 con 0x7fa8d0069dd0 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.567+0000 7fa8d77ff700 1 -- 172.21.15.29:0/1985788407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0069dd0 msgr2=0x7fa8d00b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.567+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/1985788407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0069dd0 0x7fa8d00b1290 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00098e0 tx=0x7fa8c00091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.568+0000 7fa8d77ff700 1 -- 172.21.15.29:0/1985788407 shutdown_connections 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.568+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/1985788407 >> v2:172.21.15.144:3300/0 conn(0x7fa8d0154d30 0x7fa8d0158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.568+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/1985788407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0069dd0 0x7fa8d00b1290 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.568+0000 7fa8d77ff700 1 -- 172.21.15.29:0/1985788407 >> 172.21.15.29:0/1985788407 conn(0x7fa8d0068090 msgr2=0x7fa8d00696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.568+0000 7fa8d77ff700 1 -- 172.21.15.29:0/1985788407 shutdown_connections 2023-10-07T09:44:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.568+0000 7fa8d77ff700 1 -- 172.21.15.29:0/1985788407 wait complete. 2023-10-07T09:44:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.569+0000 7fa8d77ff700 1 Processor -- start 2023-10-07T09:44:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.569+0000 7fa8d77ff700 1 -- start start 2023-10-07T09:44:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.570+0000 7fa8d77ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 0x7fa8d014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.570+0000 7fa8d77ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0154d30 0x7fa8d015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.570+0000 7fa8d77ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8d0155100 0x7fa8d00cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.570+0000 7fa8d77ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d016ccb0 con 0x7fa8d0069dd0 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.570+0000 7fa8d77ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d015d8a0 con 0x7fa8d0155100 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.570+0000 7fa8d77ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d016cf00 con 0x7fa8d0154d30 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 0x7fa8d014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 0x7fa8d014ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42746/0 (socket says 172.21.15.29:42746) 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 -- 172.21.15.29:0/2328303083 learned_addr learned my addr 172.21.15.29:0/2328303083 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 -- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0154d30 msgr2=0x7fa8d015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0154d30 0x7fa8d015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 -- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8d0155100 msgr2=0x7fa8d00cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8d0155100 0x7fa8d00cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 -- 172.21.15.29:0/2328303083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009420 con 0x7fa8d0069dd0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.571+0000 7fa8d559b700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 0x7fa8d014ccb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00ec80 tx=0x7fa8cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.572+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc00c9d0 con 0x7fa8d0069dd0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.572+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc00cb30 con 0x7fa8d0069dd0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.572+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc013470 con 0x7fa8d0069dd0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.572+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d016d1e0 con 0x7fa8d0069dd0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.572+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d014d1f0 con 0x7fa8d0069dd0 2023-10-07T09:44:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.573+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d016d680 con 0x7fa8d0069dd0 2023-10-07T09:44:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.574+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fa8cc013820 con 0x7fa8d0069dd0 2023-10-07T09:44:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.575+0000 7fa8c57fa700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 0x7fa8b4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.575+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fa8cc084b20 con 0x7fa8d0069dd0 2023-10-07T09:44:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.575+0000 7fa8d4d9a700 1 -- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 msgr2=0x7fa8b4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.575+0000 7fa8d4d9a700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 0x7fa8b4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:33.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.580+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8cc051b70 con 0x7fa8d0069dd0 2023-10-07T09:44:33.726 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:33.726+0000 7f4ebec8d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:44:33.759 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:33.758+0000 7fa6d304b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:44:33.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.776+0000 7fa8d4d9a700 1 -- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 msgr2=0x7fa8b4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:33.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.776+0000 7fa8d4d9a700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 0x7fa8b4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:33.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.819+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fa8d016d830 con 0x7fa8d0069dd0 2023-10-07T09:44:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.823+0000 7fa8c57fa700 1 -- 172.21.15.29:0/2328303083 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa8cc055190 con 0x7fa8d0069dd0 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 msgr2=0x7fa8b4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 0x7fa8b4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 msgr2=0x7fa8d014ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 0x7fa8d014ccb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00ec80 tx=0x7fa8cc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 shutdown_connections 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa8b4066f80 0x7fa8b4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8d0155100 0x7fa8d00cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d0069dd0 0x7fa8d014ccb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 --2- 172.21.15.29:0/2328303083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8d0154d30 0x7fa8d015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 >> 172.21.15.29:0/2328303083 conn(0x7fa8d0068090 msgr2=0x7fa8d006d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 shutdown_connections 2023-10-07T09:44:33.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:33.826+0000 7fa8d77ff700 1 -- 172.21.15.29:0/2328303083 wait complete. 2023-10-07T09:44:33.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:44:34.038 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:34.038+0000 7f4ebec8d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:44:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.046+0000 7f1491bff700 1 Processor -- start 2023-10-07T09:44:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.047+0000 7f1491bff700 1 -- start start 2023-10-07T09:44:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.048+0000 7f1491bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f148c0b0770 0x7f148c069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.048+0000 7f1491bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c069400 con 0x7f148c159300 2023-10-07T09:44:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.048+0000 7f1491bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c069dd0 con 0x7f148c15d300 2023-10-07T09:44:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.048+0000 7f1491bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c15de60 con 0x7f148c0b0770 2023-10-07T09:44:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.048+0000 7f148b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f148c159300 0x7f148c0b0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45062/0 (socket says 172.21.15.29:45062) 2023-10-07T09:44:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.048+0000 7f148b7fe700 1 -- 172.21.15.29:0/3322508046 learned_addr learned my addr 172.21.15.29:0/3322508046 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148affd700 1 --2- 172.21.15.29:0/3322508046 >> v2:172.21.15.144:3300/0 conn(0x7f148c0b0770 0x7f148c069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724590138 0 0) 0x7f148c069400 con 0x7f148c159300 2023-10-07T09:44:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1474003570 con 0x7f148c159300 2023-10-07T09:44:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868808025 0 0) 0x7f148c069dd0 con 0x7f148c15d300 2023-10-07T09:44:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148c069400 con 0x7f148c15d300 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684231953 0 0) 0x7f1474003570 con 0x7f148c159300 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148affd700 1 -- 172.21.15.29:0/3322508046 >> v1:172.21.15.144:6789/0 conn(0x7f148c15d300 legacy=0x7f148c15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148affd700 1 -- 172.21.15.29:0/3322508046 >> v1:172.21.15.29:6789/0 conn(0x7f148c159300 legacy=0x7f148c0b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.049+0000 7f148affd700 1 -- 172.21.15.29:0/3322508046 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c068b10 con 0x7f148c0b0770 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.050+0000 7f148affd700 1 --2- 172.21.15.29:0/3322508046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c069890 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f14800098e0 tx=0x7f14800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=701546e7c970f247 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.050+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c0015f0 con 0x7f148c0b0770 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.050+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480011420 con 0x7f148c0b0770 2023-10-07T09:44:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.050+0000 7f148a7fc700 1 -- 172.21.15.29:0/3322508046 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14800117c0 con 0x7f148c0b0770 2023-10-07T09:44:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 -- 172.21.15.29:0/3322508046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 msgr2=0x7f148c069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 --2- 172.21.15.29:0/3322508046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c069890 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f14800098e0 tx=0x7f14800091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 -- 172.21.15.29:0/3322508046 shutdown_connections 2023-10-07T09:44:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 --2- 172.21.15.29:0/3322508046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c069890 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 -- 172.21.15.29:0/3322508046 >> 172.21.15.29:0/3322508046 conn(0x7f148c067610 msgr2=0x7f148c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 -- 172.21.15.29:0/3322508046 shutdown_connections 2023-10-07T09:44:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.051+0000 7f1491bff700 1 -- 172.21.15.29:0/3322508046 wait complete. 2023-10-07T09:44:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.052+0000 7f1491bff700 1 Processor -- start 2023-10-07T09:44:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.052+0000 7f1491bff700 1 -- start start 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.053+0000 7f1491bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.053+0000 7f1491bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148c159300 0x7f148c15c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.053+0000 7f1491bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 0x7f148c169700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.053+0000 7f1491bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c15ce00 con 0x7f148c15d300 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.053+0000 7f1491bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c15a0b0 con 0x7f148c0b0770 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.053+0000 7f1491bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c15aed0 con 0x7f148c159300 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 0x7f148c169700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148affd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148c159300 0x7f148c15c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 0x7f148c169700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42754/0 (socket says 172.21.15.29:42754) 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 -- 172.21.15.29:0/2615106369 learned_addr learned my addr 172.21.15.29:0/2615106369 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148b7fe700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 -- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148c159300 msgr2=0x7f148c15c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148c159300 0x7f148c15c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 -- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 msgr2=0x7f148c15c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c15c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.054+0000 7f148bfff700 1 -- 172.21.15.29:0/2615106369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1480009420 con 0x7f148c15d300 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.055+0000 7f148bfff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 0x7f148c169700 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f148400c860 tx=0x7f1484002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.055+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484002030 con 0x7f148c15d300 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.055+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14840021b0 con 0x7f148c15d300 2023-10-07T09:44:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.055+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484012630 con 0x7f148c15d300 2023-10-07T09:44:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.055+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c154570 con 0x7f148c15d300 2023-10-07T09:44:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.055+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c15a620 con 0x7f148c15d300 2023-10-07T09:44:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.056+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c15b060 con 0x7f148c15d300 2023-10-07T09:44:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.058+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f1484012a20 con 0x7f148c15d300 2023-10-07T09:44:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.059+0000 7f14737fe700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 0x7f1478069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.059+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f14840846c0 con 0x7f148c15d300 2023-10-07T09:44:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.059+0000 7f148b7fe700 1 -- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 msgr2=0x7f1478069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.059+0000 7f148b7fe700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 0x7f1478069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.064+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1484051710 con 0x7f148c15d300 2023-10-07T09:44:34.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:34.094+0000 7fa6d304b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:44:34.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.260+0000 7f148b7fe700 1 -- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 msgr2=0x7f1478069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:34.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.260+0000 7f148b7fe700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 0x7f1478069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:34.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.268+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f148c068db0 con 0x7f148c15d300 2023-10-07T09:44:34.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.272+0000 7f14737fe700 1 -- 172.21.15.29:0/2615106369 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1484054d30 con 0x7f148c15d300 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 msgr2=0x7f1478069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 0x7f1478069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 msgr2=0x7f148c169700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 0x7f148c169700 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f148400c860 tx=0x7f1484002700 comp rx=0 tx=0).stop 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 shutdown_connections 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1478066f80 0x7f1478069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f148c15d300 0x7f148c169700 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0b0770 0x7f148c15c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.274+0000 7f1491bff700 1 --2- 172.21.15.29:0/2615106369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148c159300 0x7f148c15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.275+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 >> 172.21.15.29:0/2615106369 conn(0x7f148c067610 msgr2=0x7f148c1563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.275+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 shutdown_connections 2023-10-07T09:44:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.275+0000 7f1491bff700 1 -- 172.21.15.29:0/2615106369 wait complete. 2023-10-07T09:44:34.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:34.294 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T09:44:34.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-10-07T09:44:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.549+0000 7f2e315ff700 1 Processor -- start 2023-10-07T09:44:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.549+0000 7f2e315ff700 1 -- start start 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.551+0000 7f2e315ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b0610 0x7f2e2c0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e315ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2e2c0c8a20 0x7f2e2c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e315ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0690f0 con 0x7f2e2c0b09e0 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e315ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cf2e0 con 0x7f2e2c0c8a20 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e315ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0687c0 con 0x7f2e2c0b0610 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e2a7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2e2c0b09e0 0x7f2e2c0c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45074/0 (socket says 172.21.15.29:45074) 2023-10-07T09:44:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e2a7fc700 1 -- 172.21.15.29:0/2270311838 learned_addr learned my addr 172.21.15.29:0/2270311838 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e2b7fe700 1 --2- 172.21.15.29:0/2270311838 >> v2:172.21.15.144:3300/0 conn(0x7f2e2c0c8a20 0x7f2e2c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e2affd700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b0610 0x7f2e2c0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e29ffb700 1 -- 172.21.15.29:0/2270311838 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094711762 0 0) 0x7f2e2c0690f0 con 0x7f2e2c0b09e0 2023-10-07T09:44:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.552+0000 7f2e29ffb700 1 -- 172.21.15.29:0/2270311838 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e14003570 con 0x7f2e2c0b09e0 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e2b7fe700 1 -- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b0610 msgr2=0x7f2e2c0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e2b7fe700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b0610 0x7f2e2c0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e2b7fe700 1 -- 172.21.15.29:0/2270311838 >> v1:172.21.15.29:6789/0 conn(0x7f2e2c0b09e0 legacy=0x7f2e2c0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e2b7fe700 1 -- 172.21.15.29:0/2270311838 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c068aa0 con 0x7f2e2c0c8a20 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e2affd700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b0610 0x7f2e2c0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e2b7fe700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 0x7f2e2c0cc7e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009e60 tx=0x7f2e24009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90de469a1e6396a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e29ffb700 1 -- 172.21.15.29:0/2270311838 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400e070 con 0x7f2e2c0c8a20 2023-10-07T09:44:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e29ffb700 1 -- 172.21.15.29:0/2270311838 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2400bab0 con 0x7f2e2c0c8a20 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.553+0000 7f2e29ffb700 1 -- 172.21.15.29:0/2270311838 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400be70 con 0x7f2e2c0c8a20 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 -- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 msgr2=0x7f2e2c0cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 0x7f2e2c0cc7e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009e60 tx=0x7f2e24009780 comp rx=0 tx=0).stop 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 -- 172.21.15.29:0/2270311838 shutdown_connections 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 0x7f2e2c0cc7e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 --2- 172.21.15.29:0/2270311838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b0610 0x7f2e2c0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 -- 172.21.15.29:0/2270311838 >> 172.21.15.29:0/2270311838 conn(0x7f2e2c063a10 msgr2=0x7f2e2c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 -- 172.21.15.29:0/2270311838 shutdown_connections 2023-10-07T09:44:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.554+0000 7f2e315ff700 1 -- 172.21.15.29:0/2270311838 wait complete. 2023-10-07T09:44:34.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:34.556+0000 7f4ebec8d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.555+0000 7f2e315ff700 1 Processor -- start 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.556+0000 7f2e315ff700 1 -- start start 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.556+0000 7f2e315ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e2c0b0610 0x7f2e2c152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.556+0000 7f2e315ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b09e0 0x7f2e2c145230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.556+0000 7f2e315ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 0x7f2e2c153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.556+0000 7f2e315ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c16cf40 con 0x7f2e2c0b0610 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e315ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c145770 con 0x7f2e2c0c8a20 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e315ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c1544a0 con 0x7f2e2c0b09e0 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e2c0b0610 0x7f2e2c152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e2c0b0610 0x7f2e2c152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42770/0 (socket says 172.21.15.29:42770) 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 -- 172.21.15.29:0/695736260 learned_addr learned my addr 172.21.15.29:0/695736260 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b09e0 0x7f2e2c145230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2b7fe700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 0x7f2e2c153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b09e0 msgr2=0x7f2e2c145230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e2c0b09e0 0x7f2e2c145230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 msgr2=0x7f2e2c153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2c0c8a20 0x7f2e2c153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.557+0000 7f2e2affd700 1 -- 172.21.15.29:0/695736260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c009420 con 0x7f2e2c0b0610 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.558+0000 7f2e2affd700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e2c0b0610 0x7f2e2c152300 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00ec80 tx=0x7f2e1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.558+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c00c850 con 0x7f2e2c0b0610 2023-10-07T09:44:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.558+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c00c9b0 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.558+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c00cd30 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.558+0000 7f2e315ff700 1 -- 172.21.15.29:0/695736260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24009a20 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.558+0000 7f2e315ff700 1 -- 172.21.15.29:0/695736260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c154a20 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.559+0000 7f2e315ff700 1 -- 172.21.15.29:0/695736260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c0718b0 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.560+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f2e1c013680 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.561+0000 7f2e12ffd700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.561+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f2e1c0849f0 con 0x7f2e2c0b0610 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.561+0000 7f2e2a7fc700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 msgr2=0x7f2e18069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.561+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.569+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e1c051a40 con 0x7f2e2c0b0610 2023-10-07T09:44:34.654 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:34.653+0000 7fa6d304b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:44:34.724 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:34.724+0000 7f4ebec8d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:44:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.762+0000 7f2e2a7fc700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 msgr2=0x7f2e18069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.762+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:34.801+0000 7f2e315ff700 1 -- 172.21.15.29:0/695736260 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e2c16d080 con 0x7f2e18066f80 2023-10-07T09:44:34.830 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:34.829+0000 7fa6d304b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:44:35.136 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:35.136+0000 7f4ebec8d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:44:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:35.163+0000 7f2e2a7fc700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 msgr2=0x7f2e18069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:35.163+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:44:35.255 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:35.254+0000 7fa6d304b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:44:35.905 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:35.905+0000 7f4ebec8d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:44:35.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:35.964+0000 7f2e2a7fc700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 msgr2=0x7f2e18069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:35.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:35.964+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:44:35.994 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:35.992+0000 7f4ebec8d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:44:36.056 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.055+0000 7fa6d304b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:44:36.144 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.144+0000 7f4ebec8d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:44:36.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.146+0000 7fa6d304b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:44:36.230 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.230+0000 7f4ebec8d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:44:36.307 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.306+0000 7fa6d304b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:44:36.397 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.395+0000 7fa6d304b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:44:36.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.446+0000 7f4ebec8d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:44:36.540 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.540+0000 7f4ebec8d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:44:36.627 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.626+0000 7fa6d304b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:44:36.634 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.634+0000 7f4ebec8d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:44:36.726 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.725+0000 7fa6d304b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:44:36.727 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.727+0000 7f4ebec8d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:44:36.823 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.822+0000 7fa6d304b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:44:36.913 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:36.913+0000 7f4ebec8d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:44:36.921 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:36.920+0000 7fa6d304b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:44:37.013 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:37.011+0000 7f4ebec8d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:44:37.114 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:37.113+0000 7fa6d304b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:44:37.215 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:37.214+0000 7fa6d304b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:44:37.351 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:37.351+0000 7f4ebec8d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:44:37.440 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:37.440+0000 7f4ebec8d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:37.563 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:37.562+0000 7fa6d304b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:44:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:37.566+0000 7f2e2a7fc700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 msgr2=0x7f2e18069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:37.566+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:44:37.654 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:37.654+0000 7fa6d304b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:37.976 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:37.976+0000 7f4ebec8d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:44:38.158 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:38.158+0000 7f4ebec8d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:44:38.217 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:38.217+0000 7fa6d304b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:44:38.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:38.408+0000 7fa6d304b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:44:38.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:38.530+0000 7f4ebec8d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:38.791 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:38.790+0000 7fa6d304b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:38.948 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:38.947+0000 7f4ebec8d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:44:39.126 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.125+0000 7f4ebec8d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:39.229 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:39.229+0000 7fa6d304b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:44:39.312 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.311+0000 7f4ebec8d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:44:39.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:39.417+0000 7fa6d304b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:39.420 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.419+0000 7f4ebec8d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:39.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:39.433+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 28) v1 ==== 84468+0+0 (secure 0 0 0) 0x7f2e1c086d50 con 0x7f2e2c0b0610 2023-10-07T09:44:39.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:39.433+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 msgr2=0x7f2e18069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:39.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:39.433+0000 7f2e12ffd700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e18069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:39.555 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.554+0000 7f4e2ea6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.556 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.554+0000 7f4e2ea6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.556 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.554+0000 7f4e2ea6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.556 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.554+0000 7f4e2ea6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.556 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.554+0000 7f4e2ea6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.557 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.556+0000 7f4e32a76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.557 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.556+0000 7f4e32a76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.557 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.556+0000 7f4e32a76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.557 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.556+0000 7f4e32a76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.557 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:39.556+0000 7f4e32a76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:39.622 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:39.622+0000 7fa6d304b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:44:39.734 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:39.734+0000 7fa6d304b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:40.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.437+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 29) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f2e1c051e10 con 0x7f2e2c0b0610 2023-10-07T09:44:40.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.438+0000 7f2e12ffd700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e180693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.438+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e18033920 con 0x7f2e18066f80 2023-10-07T09:44:40.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.441+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e180693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.443+0000 7f2e2a7fc700 1 --2- 172.21.15.29:0/695736260 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2e18066f80 0x7f2e180693b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0b02b0 tx=0x7f2e20009040 comp rx=0 tx=0).ready entity=mgr.6232 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:40.453 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:40.452+0000 7f4e3a285700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-07T09:44:40.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.453+0000 7f2e12ffd700 1 -- 172.21.15.29:0/695736260 <== mgr.6232 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f2e18033920 con 0x7f2e18066f80 2023-10-07T09:44:40.456 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-07T09:44:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-07T09:44:40.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:40.465 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T09:44:40.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T09:44:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.770+0000 7f90541ff700 1 Processor -- start 2023-10-07T09:44:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.770+0000 7f90541ff700 1 -- start start 2023-10-07T09:44:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.771+0000 7f90541ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f904c13fba0 0x7f904c06ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.771+0000 7f90541ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f904c067d20 0x7f904c0c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.771+0000 7f90541ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0c78c0 con 0x7f904c13ff70 2023-10-07T09:44:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.771+0000 7f90541ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c06fa50 con 0x7f904c067d20 2023-10-07T09:44:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.771+0000 7f90541ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c06c0f0 con 0x7f904c13fba0 2023-10-07T09:44:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f904c067d20 0x7f904c0c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f904c067d20 0x7f904c0c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42784/0 (socket says 172.21.15.29:42784) 2023-10-07T09:44:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 -- 172.21.15.29:0/3101231893 learned_addr learned my addr 172.21.15.29:0/3101231893 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f9051f9b700 1 --2- 172.21.15.29:0/3101231893 >> v2:172.21.15.144:3300/0 conn(0x7f904c13fba0 0x7f904c06ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 -- 172.21.15.29:0/3101231893 >> v2:172.21.15.144:3300/0 conn(0x7f904c13fba0 msgr2=0x7f904c06ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 --2- 172.21.15.29:0/3101231893 >> v2:172.21.15.144:3300/0 conn(0x7f904c13fba0 0x7f904c06ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 -- 172.21.15.29:0/3101231893 >> v1:172.21.15.144:6789/0 conn(0x7f904c13ff70 legacy=0x7f904c067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 -- 172.21.15.29:0/3101231893 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c06c3d0 con 0x7f904c067d20 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.772+0000 7f905279c700 1 --2- 172.21.15.29:0/3101231893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c0c7d50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f904c0cd1e0 tx=0x7f9044009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a41562a0619ca3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f9050f99700 1 -- 172.21.15.29:0/3101231893 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904400e070 con 0x7f904c067d20 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f9050f99700 1 -- 172.21.15.29:0/3101231893 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904400bab0 con 0x7f904c067d20 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f9050f99700 1 -- 172.21.15.29:0/3101231893 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440092e0 con 0x7f904c067d20 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 -- 172.21.15.29:0/3101231893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 msgr2=0x7f904c0c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 --2- 172.21.15.29:0/3101231893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c0c7d50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f904c0cd1e0 tx=0x7f9044009840 comp rx=0 tx=0).stop 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 -- 172.21.15.29:0/3101231893 shutdown_connections 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 --2- 172.21.15.29:0/3101231893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c0c7d50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 --2- 172.21.15.29:0/3101231893 >> v2:172.21.15.144:3300/0 conn(0x7f904c13fba0 0x7f904c06ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 -- 172.21.15.29:0/3101231893 >> 172.21.15.29:0/3101231893 conn(0x7f904c06b1d0 msgr2=0x7f904c0cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 -- 172.21.15.29:0/3101231893 shutdown_connections 2023-10-07T09:44:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.773+0000 7f90541ff700 1 -- 172.21.15.29:0/3101231893 wait complete. 2023-10-07T09:44:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.775+0000 7f90541ff700 1 Processor -- start 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.775+0000 7f90541ff700 1 -- start start 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f90541ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c14d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f90541ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f904c13fba0 0x7f904c0c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f90541ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c13ff70 0x7f904c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f90541ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c134500 con 0x7f904c067d20 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f90541ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c14b2c0 con 0x7f904c13ff70 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f90541ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c14a6b0 con 0x7f904c13fba0 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c14d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c14d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42796/0 (socket says 172.21.15.29:42796) 2023-10-07T09:44:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 -- 172.21.15.29:0/3006733236 learned_addr learned my addr 172.21.15.29:0/3006733236 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f905279c700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c13ff70 0x7f904c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f905179a700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f904c13fba0 0x7f904c0c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 -- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f904c13fba0 msgr2=0x7f904c0c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f904c13fba0 0x7f904c0c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 -- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c13ff70 msgr2=0x7f904c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c13ff70 0x7f904c0c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.776+0000 7f9051f9b700 1 -- 172.21.15.29:0/3006733236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9044009ae0 con 0x7f904c067d20 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.777+0000 7f9051f9b700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c14d010 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f904800ecb0 tx=0x7f90480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.777+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904800c9d0 con 0x7f904c067d20 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.777+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904800cb30 con 0x7f904c067d20 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.777+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904800ced0 con 0x7f904c067d20 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.777+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c14a990 con 0x7f904c067d20 2023-10-07T09:44:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.777+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c14af50 con 0x7f904c067d20 2023-10-07T09:44:40.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.778+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c14b5a0 con 0x7f904c067d20 2023-10-07T09:44:40.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.779+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f9048013680 con 0x7f904c067d20 2023-10-07T09:44:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.780+0000 7f9041ffb700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9030066dd0 0x7f9030069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.780+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f9048084950 con 0x7f904c067d20 2023-10-07T09:44:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.781+0000 7f905179a700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9030066dd0 0x7f9030069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.781+0000 7f905179a700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9030066dd0 0x7f9030069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f904c16d000 tx=0x7f903c009040 comp rx=0 tx=0).ready entity=mgr.6232 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:40.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:40.785+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9048051b20 con 0x7f904c067d20 2023-10-07T09:44:41.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.043+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f904c04c3c0 con 0x7f904c067d20 2023-10-07T09:44:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.489+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f9048051ef0 con 0x7f904c067d20 2023-10-07T09:44:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.490+0000 7f9041ffb700 1 -- 172.21.15.29:0/3006733236 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v30) v1 ==== 93+0+0 (secure 0 0 0) 0x7f9048051b20 con 0x7f904c067d20 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.493+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9030066dd0 msgr2=0x7f9030069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.493+0000 7f90541ff700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9030066dd0 0x7f9030069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f904c16d000 tx=0x7f903c009040 comp rx=0 tx=0).stop 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.493+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 msgr2=0x7f904c14d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.493+0000 7f90541ff700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c14d010 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f904800ecb0 tx=0x7f90480091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.493+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 shutdown_connections 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9030066dd0 0x7f9030069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c13ff70 0x7f904c0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f904c067d20 0x7f904c14d010 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 --2- 172.21.15.29:0/3006733236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f904c13fba0 0x7f904c0c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 >> 172.21.15.29:0/3006733236 conn(0x7f904c06b1d0 msgr2=0x7f904c13ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:41.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 shutdown_connections 2023-10-07T09:44:41.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:41.494+0000 7f90541ff700 1 -- 172.21.15.29:0/3006733236 wait complete. 2023-10-07T09:44:41.522 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:44:41.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:41.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:41.707 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:41.706+0000 7f28820a61c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:44:41.712 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:41.712+0000 7fc0ecb251c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:44:41.736 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:44:41.736 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:44:41.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:44:41.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T09:44:41.840 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:44:41.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:44:41.841 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:44:41.886 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:41.885+0000 7f28820a61c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:44:41.903 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:41.903+0000 7fc0ecb251c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:44:41.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.160+0000 7f3ae23ff700 1 Processor -- start 2023-10-07T09:44:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.160+0000 7f3ae23ff700 1 -- start start 2023-10-07T09:44:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3ae23ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3adc158670 0x7f3adc162060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3ae23ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc158ad0 con 0x7f3adc157290 2023-10-07T09:44:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3ae23ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc15b140 con 0x7f3adc0ae8b0 2023-10-07T09:44:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3ae23ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc15b420 con 0x7f3adc158670 2023-10-07T09:44:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3adbfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3adc157290 0x7f3adc0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45092/0 (socket says 172.21.15.29:45092) 2023-10-07T09:44:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3adbfff700 1 -- 172.21.15.29:0/3243222381 learned_addr learned my addr 172.21.15.29:0/3243222381 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.162+0000 7f3ae099c700 1 --2- 172.21.15.29:0/3243222381 >> v2:172.21.15.29:3300/0 conn(0x7f3adc158670 0x7f3adc162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.163+0000 7f3ae099c700 1 -- 172.21.15.29:0/3243222381 >> v1:172.21.15.144:6789/0 conn(0x7f3adc0ae8b0 legacy=0x7f3adc0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.163+0000 7f3ae099c700 1 -- 172.21.15.29:0/3243222381 >> v1:172.21.15.29:6789/0 conn(0x7f3adc157290 legacy=0x7f3adc0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.163+0000 7f3adaffd700 1 -- 172.21.15.29:0/3243222381 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764518555 0 0) 0x7f3adc15b140 con 0x7f3adc0ae8b0 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.163+0000 7f3ae099c700 1 -- 172.21.15.29:0/3243222381 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc15b700 con 0x7f3adc158670 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.163+0000 7f3ae099c700 1 --2- 172.21.15.29:0/3243222381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc158670 0x7f3adc162060 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009e60 tx=0x7f3ad4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ee3c33b7aeb66 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.163+0000 7f3adaffd700 1 -- 172.21.15.29:0/3243222381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400e070 con 0x7f3adc158670 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3adaffd700 1 -- 172.21.15.29:0/3243222381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad400bb30 con 0x7f3adc158670 2023-10-07T09:44:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3adaffd700 1 -- 172.21.15.29:0/3243222381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400bed0 con 0x7f3adc158670 2023-10-07T09:44:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3243222381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc158670 msgr2=0x7f3adc162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3243222381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc158670 0x7f3adc162060 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009e60 tx=0x7f3ad4009780 comp rx=0 tx=0).stop 2023-10-07T09:44:42.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3243222381 shutdown_connections 2023-10-07T09:44:42.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3243222381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc158670 0x7f3adc162060 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.164+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3243222381 >> 172.21.15.29:0/3243222381 conn(0x7f3adc0680a0 msgr2=0x7f3adc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:42.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.165+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3243222381 shutdown_connections 2023-10-07T09:44:42.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.165+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3243222381 wait complete. 2023-10-07T09:44:42.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.166+0000 7f3ae23ff700 1 Processor -- start 2023-10-07T09:44:42.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.166+0000 7f3ae23ff700 1 -- start start 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.167+0000 7f3ae23ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3adc157290 0x7f3adc0acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.167+0000 7f3ae23ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3adc158670 0x7f3adc15c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.167+0000 7f3ae23ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 0x7f3adc0c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.167+0000 7f3ae23ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc15d8b0 con 0x7f3adc15c7d0 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.167+0000 7f3ae23ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc15cd30 con 0x7f3adc157290 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.167+0000 7f3ae23ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc15d010 con 0x7f3adc158670 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3ae099c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 0x7f3adc0c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adbfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3adc157290 0x7f3adc0acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adb7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3adc158670 0x7f3adc15c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3ae099c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 0x7f3adc0c5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42824/0 (socket says 172.21.15.29:42824) 2023-10-07T09:44:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3ae099c700 1 -- 172.21.15.29:0/3890312616 learned_addr learned my addr 172.21.15.29:0/3890312616 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adb7fe700 1 -- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3adc157290 msgr2=0x7f3adc0acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adb7fe700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3adc157290 0x7f3adc0acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adb7fe700 1 -- 172.21.15.29:0/3890312616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 msgr2=0x7f3adc0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adb7fe700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 0x7f3adc0c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adb7fe700 1 -- 172.21.15.29:0/3890312616 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4009a20 con 0x7f3adc158670 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3adbfff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3adc157290 0x7f3adc0acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.168+0000 7f3ae099c700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 0x7f3adc0c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.169+0000 7f3adb7fe700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3adc158670 0x7f3adc15c290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000c510 tx=0x7f3ad0007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.169+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad000ce80 con 0x7f3adc158670 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.169+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad0011590 con 0x7f3adc158670 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.169+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc15d2f0 con 0x7f3adc158670 2023-10-07T09:44:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.169+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0011950 con 0x7f3adc158670 2023-10-07T09:44:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.169+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc0ad120 con 0x7f3adc158670 2023-10-07T09:44:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.170+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc0af180 con 0x7f3adc158670 2023-10-07T09:44:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.172+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f3ad0011d40 con 0x7f3adc158670 2023-10-07T09:44:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.173+0000 7f3ac3fff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 0x7f3ac8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.173+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f3ad0083ea0 con 0x7f3adc158670 2023-10-07T09:44:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.173+0000 7f3adbfff700 1 -- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 msgr2=0x7f3ac8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.173+0000 7f3adbfff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 0x7f3ac8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.178+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ad0050b70 con 0x7f3adc158670 2023-10-07T09:44:42.201 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:42.200+0000 7f28820a61c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:44:42.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:42.238+0000 7fc0ecb251c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:44:42.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.374+0000 7f3adbfff700 1 -- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 msgr2=0x7f3ac8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:42.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.374+0000 7f3adbfff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 0x7f3ac8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.432+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3adc15d150 con 0x7f3adc158670 2023-10-07T09:44:42.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.433+0000 7f3ac3fff700 1 -- 172.21.15.29:0/3890312616 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f3adc15d150 con 0x7f3adc158670 2023-10-07T09:44:42.433 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:42.433 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:44:23.775309+0000","modified":"2023-10-07T09:44:27.299910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6276":{"gid":6276,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2418315778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2418315778},{"type":"v1","addr":"172.21.15.29:6835","nonce":2418315778}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 msgr2=0x7f3ac8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 0x7f3ac8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3adc158670 msgr2=0x7f3adc15c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3adc158670 0x7f3adc15c290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000c510 tx=0x7f3ad0007ec0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 shutdown_connections 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ac80670e0 0x7f3ac8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3adc15c7d0 0x7f3adc0c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3adc157290 0x7f3adc0acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 --2- 172.21.15.29:0/3890312616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3adc158670 0x7f3adc15c290 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.435+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 >> 172.21.15.29:0/3890312616 conn(0x7f3adc0680a0 msgr2=0x7f3adc153650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:42.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.436+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 shutdown_connections 2023-10-07T09:44:42.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.436+0000 7f3ae23ff700 1 -- 172.21.15.29:0/3890312616 wait complete. 2023-10-07T09:44:42.437 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-07T09:44:42.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:42.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.661+0000 7f8244dff700 1 Processor -- start 2023-10-07T09:44:42.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.661+0000 7f8244dff700 1 -- start start 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f8244dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f82400b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f8244dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82400a98d0 0x7f82400a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f8244dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824006f4e0 con 0x7f8240159550 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f8244dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82400b1120 con 0x7f82400a98d0 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f8244dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824006e8d0 con 0x7f82400b2670 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f823ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82400a98d0 0x7f82400a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f823ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82400a98d0 0x7f82400a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42830/0 (socket says 172.21.15.29:42830) 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f823ed9d700 1 -- 172.21.15.29:0/873313734 learned_addr learned my addr 172.21.15.29:0/873313734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.663+0000 7f823dd9b700 1 --2- 172.21.15.29:0/873313734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f82400b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823ed9d700 1 -- 172.21.15.29:0/873313734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 msgr2=0x7f82400b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823ed9d700 1 --2- 172.21.15.29:0/873313734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f82400b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823ed9d700 1 -- 172.21.15.29:0/873313734 >> v1:172.21.15.144:6789/0 conn(0x7f8240159550 legacy=0x7f82400b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823ed9d700 1 -- 172.21.15.29:0/873313734 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824006ebb0 con 0x7f82400a98d0 2023-10-07T09:44:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823d59a700 1 -- 172.21.15.29:0/873313734 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167232129 0 0) 0x7f824006f4e0 con 0x7f8240159550 2023-10-07T09:44:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823ed9d700 1 --2- 172.21.15.29:0/873313734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f82400a9d50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8238009f20 tx=0x7f8238009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e6e6c4e4d9566b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823d59a700 1 -- 172.21.15.29:0/873313734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823800e070 con 0x7f82400a98d0 2023-10-07T09:44:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823d59a700 1 -- 172.21.15.29:0/873313734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823800b9d0 con 0x7f82400a98d0 2023-10-07T09:44:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.664+0000 7f823d59a700 1 -- 172.21.15.29:0/873313734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82380092e0 con 0x7f82400a98d0 2023-10-07T09:44:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 -- 172.21.15.29:0/873313734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 msgr2=0x7f82400a9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 --2- 172.21.15.29:0/873313734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f82400a9d50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8238009f20 tx=0x7f8238009840 comp rx=0 tx=0).stop 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 -- 172.21.15.29:0/873313734 shutdown_connections 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 --2- 172.21.15.29:0/873313734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f82400a9d50 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 --2- 172.21.15.29:0/873313734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f82400b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 -- 172.21.15.29:0/873313734 >> 172.21.15.29:0/873313734 conn(0x7f8240069190 msgr2=0x7f824006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 -- 172.21.15.29:0/873313734 shutdown_connections 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.665+0000 7f8244dff700 1 -- 172.21.15.29:0/873313734 wait complete. 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.666+0000 7f8244dff700 1 Processor -- start 2023-10-07T09:44:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.666+0000 7f8244dff700 1 -- start start 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.667+0000 7f8244dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f824015c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.667+0000 7f8244dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f8240155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.667+0000 7f8244dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8240159550 0x7f82401537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.667+0000 7f8244dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240155580 con 0x7f82400a98d0 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.667+0000 7f8244dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240154bb0 con 0x7f8240159550 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.667+0000 7f8244dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8240153d10 con 0x7f82400b2670 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f824015c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f824015c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42834/0 (socket says 172.21.15.29:42834) 2023-10-07T09:44:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 -- 172.21.15.29:0/3818103592 learned_addr learned my addr 172.21.15.29:0/3818103592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823ed9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8240159550 0x7f82401537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823dd9b700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f8240155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 -- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 msgr2=0x7f8240155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f8240155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 -- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8240159550 msgr2=0x7f82401537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8240159550 0x7f82401537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 -- 172.21.15.29:0/3818103592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238009ae0 con 0x7f82400a98d0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.668+0000 7f823e59c700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f824015c490 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f823000c7e0 tx=0x7f8230007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.669+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230011590 con 0x7f82400a98d0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.669+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230011710 con 0x7f82400a98d0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.669+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230011ad0 con 0x7f82400a98d0 2023-10-07T09:44:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.669+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8240154280 con 0x7f82400a98d0 2023-10-07T09:44:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.669+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8240154840 con 0x7f82400a98d0 2023-10-07T09:44:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.670+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824004c3c0 con 0x7f82400a98d0 2023-10-07T09:44:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.672+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f823001c460 con 0x7f82400a98d0 2023-10-07T09:44:42.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.673+0000 7f822e7fc700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 0x7f8220069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.673+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f8230084850 con 0x7f82400a98d0 2023-10-07T09:44:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.674+0000 7f823dd9b700 1 -- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 msgr2=0x7f8220069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.674+0000 7f823dd9b700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 0x7f8220069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.678+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8230051520 con 0x7f82400a98d0 2023-10-07T09:44:42.724 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:42.724+0000 7f28820a61c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:44:42.800 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:42.800+0000 7fc0ecb251c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:44:42.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.875+0000 7f823dd9b700 1 -- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 msgr2=0x7f8220069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:42.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.875+0000 7f823dd9b700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 0x7f8220069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:42.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.883+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82400683d0 con 0x7f82400a98d0 2023-10-07T09:44:42.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.884+0000 7f822e7fc700 1 -- 172.21.15.29:0/3818103592 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+16387 (secure 0 0 0) 0x7f823005a030 con 0x7f82400a98d0 2023-10-07T09:44:42.884 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:42.884 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":78,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:39.423204+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:21.559742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-07T09:44:22.805091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/807765698":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/797249383":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/894321282":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/29562660":"2023-10-08T09:44:39.423150+0000","172.21.15.144:6836/114714":"2023-10-08T09:44:39.423150+0000","172.21.15.144:6834/114714":"2023-10-08T09:44:39.423150+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 msgr2=0x7f8220069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 0x7f8220069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 msgr2=0x7f824015c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f824015c490 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f823000c7e0 tx=0x7f8230007df0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 shutdown_connections 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82200670e0 0x7f8220069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8240159550 0x7f82401537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82400a98d0 0x7f824015c490 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.886+0000 7f8244dff700 1 --2- 172.21.15.29:0/3818103592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82400b2670 0x7f8240155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:42.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.887+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 >> 172.21.15.29:0/3818103592 conn(0x7f8240069190 msgr2=0x7f82400a9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:42.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.887+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 shutdown_connections 2023-10-07T09:44:42.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:42.887+0000 7f8244dff700 1 -- 172.21.15.29:0/3818103592 wait complete. 2023-10-07T09:44:42.892 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:42.892+0000 7f28820a61c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:44:42.906 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:44:21.559742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:44:22.805091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:44:42.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:44:42.979 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:42.979+0000 7fc0ecb251c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:44:43.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.110+0000 7f5005bff700 1 Processor -- start 2023-10-07T09:44:43.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.110+0000 7f5005bff700 1 -- start start 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f5005bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50000ae570 0x7f50000afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f5005bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000ad4d0 con 0x7f50000b0520 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f5005bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000ad7b0 con 0x7f5000156180 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f5005bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50000ada90 con 0x7f50000ae570 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f4ffeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50000ae570 0x7f50000afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f4ffeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50000ae570 0x7f50000afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57290/0 (socket says 172.21.15.29:57290) 2023-10-07T09:44:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.112+0000 7f4ffeffd700 1 -- 172.21.15.29:0/3244704640 learned_addr learned my addr 172.21.15.29:0/3244704640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffeffd700 1 -- 172.21.15.29:0/3244704640 >> v1:172.21.15.144:6789/0 conn(0x7f5000156180 legacy=0x7f50000accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffeffd700 1 -- 172.21.15.29:0/3244704640 >> v1:172.21.15.29:6789/0 conn(0x7f50000b0520 legacy=0x7f5000156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffeffd700 1 -- 172.21.15.29:0/3244704640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50000add70 con 0x7f50000ae570 2023-10-07T09:44:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffe7fc700 1 -- 172.21.15.29:0/3244704640 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124046653 0 0) 0x7f50000ad7b0 con 0x7f5000156180 2023-10-07T09:44:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffeffd700 1 --2- 172.21.15.29:0/3244704640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000ae570 0x7f50000afc00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400bc50 tx=0x7f4ff400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e9b503452f718a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffe7fc700 1 -- 172.21.15.29:0/3244704640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400be60 con 0x7f50000ae570 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffe7fc700 1 -- 172.21.15.29:0/3244704640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff400de00 con 0x7f50000ae570 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.113+0000 7f4ffe7fc700 1 -- 172.21.15.29:0/3244704640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40196b0 con 0x7f50000ae570 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 -- 172.21.15.29:0/3244704640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000ae570 msgr2=0x7f50000afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 --2- 172.21.15.29:0/3244704640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000ae570 0x7f50000afc00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400bc50 tx=0x7f4ff400b4f0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 -- 172.21.15.29:0/3244704640 shutdown_connections 2023-10-07T09:44:43.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 --2- 172.21.15.29:0/3244704640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000ae570 0x7f50000afc00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 -- 172.21.15.29:0/3244704640 >> 172.21.15.29:0/3244704640 conn(0x7f5000067610 msgr2=0x7f5000067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 -- 172.21.15.29:0/3244704640 shutdown_connections 2023-10-07T09:44:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.114+0000 7f5005bff700 1 -- 172.21.15.29:0/3244704640 wait complete. 2023-10-07T09:44:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.116+0000 7f5005bff700 1 Processor -- start 2023-10-07T09:44:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.116+0000 7f5005bff700 1 -- start start 2023-10-07T09:44:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f5005bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50000ae570 0x7f50000c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f5005bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 0x7f50001576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f5005bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5000156180 0x7f5000157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f5005bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50001658f0 con 0x7f50000b0520 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f5005bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50000c7160 con 0x7f5000156180 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f5005bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50000c7440 con 0x7f50000ae570 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f4ffeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 0x7f50001576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f4ffeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 0x7f50001576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57296/0 (socket says 172.21.15.29:57296) 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f4fff7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50000ae570 0x7f50000c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f4ffeffd700 1 -- 172.21.15.29:0/600733859 learned_addr learned my addr 172.21.15.29:0/600733859 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.117+0000 7f4ffffff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5000156180 0x7f5000157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4ffeffd700 1 -- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50000ae570 msgr2=0x7f50000c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4ffeffd700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50000ae570 0x7f50000c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4ffeffd700 1 -- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5000156180 msgr2=0x7f5000157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4ffeffd700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5000156180 0x7f5000157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4ffeffd700 1 -- 172.21.15.29:0/600733859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff400b790 con 0x7f50000b0520 2023-10-07T09:44:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4ffeffd700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 0x7f50001576d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4006f50 tx=0x7f4ff401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4026070 con 0x7f50000b0520 2023-10-07T09:44:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff401e040 con 0x7f50000b0520 2023-10-07T09:44:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40109a0 con 0x7f50000b0520 2023-10-07T09:44:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50000c7720 con 0x7f50000b0520 2023-10-07T09:44:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.118+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50000c6e70 con 0x7f50000b0520 2023-10-07T09:44:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.119+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5000165a60 con 0x7f50000b0520 2023-10-07T09:44:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.120+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f4ff402b410 con 0x7f50000b0520 2023-10-07T09:44:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.121+0000 7f4fe77fe700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 0x7f4fec069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.121+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f4ff4014070 con 0x7f50000b0520 2023-10-07T09:44:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.121+0000 7f4fff7fe700 1 -- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 msgr2=0x7f4fec069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.121+0000 7f4fff7fe700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 0x7f4fec069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.129+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ff4053580 con 0x7f50000b0520 2023-10-07T09:44:43.304 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:43.303+0000 7f28820a61c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:44:43.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.322+0000 7f4fff7fe700 1 -- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 msgr2=0x7f4fec069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:43.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.322+0000 7f4fff7fe700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 0x7f4fec069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.331+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f500004c3c0 con 0x7f50000b0520 2023-10-07T09:44:43.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.331+0000 7f4fe77fe700 1 -- 172.21.15.29:0/600733859 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ff4056ba0 con 0x7f50000b0520 2023-10-07T09:44:43.331 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:44:43.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 msgr2=0x7f4fec069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:43.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 0x7f4fec069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 msgr2=0x7f50001576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 0x7f50001576d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4006f50 tx=0x7f4ff401b000 comp rx=0 tx=0).stop 2023-10-07T09:44:43.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 shutdown_connections 2023-10-07T09:44:43.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4fec0670e0 0x7f4fec069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5000156180 0x7f5000157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50000ae570 0x7f50000c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 --2- 172.21.15.29:0/600733859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50000b0520 0x7f50001576d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 >> 172.21.15.29:0/600733859 conn(0x7f5000067610 msgr2=0x7f50000acbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:43.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 shutdown_connections 2023-10-07T09:44:43.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.334+0000 7f5005bff700 1 -- 172.21.15.29:0/600733859 wait complete. 2023-10-07T09:44:43.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:44:43.419 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:43.419+0000 7fc0ecb251c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:44:43.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.556+0000 7fd58b1ff700 1 Processor -- start 2023-10-07T09:44:43.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.556+0000 7fd58b1ff700 1 -- start start 2023-10-07T09:44:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd58b1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd58413fba0 0x7fd58406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd58b1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58413ff70 0x7fd584067610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd58b1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5840c78c0 con 0x7fd5840cdc50 2023-10-07T09:44:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd58b1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58406fa50 con 0x7fd58413fba0 2023-10-07T09:44:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd58b1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58406c0f0 con 0x7fd58413ff70 2023-10-07T09:44:43.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd588f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd58413fba0 0x7fd58406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd588f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd58413fba0 0x7fd58406ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57302/0 (socket says 172.21.15.29:57302) 2023-10-07T09:44:43.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd588f9b700 1 -- 172.21.15.29:0/430440881 learned_addr learned my addr 172.21.15.29:0/430440881 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:43.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.558+0000 7fd583fff700 1 --2- 172.21.15.29:0/430440881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58413ff70 0x7fd584067610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd588f9b700 1 -- 172.21.15.29:0/430440881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58413ff70 msgr2=0x7fd584067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd588f9b700 1 --2- 172.21.15.29:0/430440881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58413ff70 0x7fd584067610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd588f9b700 1 -- 172.21.15.29:0/430440881 >> v1:172.21.15.29:6789/0 conn(0x7fd5840cdc50 legacy=0x7fd58406e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd588f9b700 1 -- 172.21.15.29:0/430440881 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58406c3d0 con 0x7fd58413fba0 2023-10-07T09:44:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd588f9b700 1 --2- 172.21.15.29:0/430440881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58406ba10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd5780099b0 tx=0x7fd578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d49cc88fb03524 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd5837fe700 1 -- 172.21.15.29:0/430440881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578009ea0 con 0x7fd58413fba0 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd5837fe700 1 -- 172.21.15.29:0/430440881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57800b8f0 con 0x7fd58413fba0 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.559+0000 7fd5837fe700 1 -- 172.21.15.29:0/430440881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57800bc90 con 0x7fd58413fba0 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 -- 172.21.15.29:0/430440881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 msgr2=0x7fd58406ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/430440881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58406ba10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd5780099b0 tx=0x7fd578009250 comp rx=0 tx=0).stop 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 -- 172.21.15.29:0/430440881 shutdown_connections 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/430440881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58406ba10 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/430440881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58413ff70 0x7fd584067610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 -- 172.21.15.29:0/430440881 >> 172.21.15.29:0/430440881 conn(0x7fd58406b1d0 msgr2=0x7fd5840cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 -- 172.21.15.29:0/430440881 shutdown_connections 2023-10-07T09:44:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.560+0000 7fd58b1ff700 1 -- 172.21.15.29:0/430440881 wait complete. 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.562+0000 7fd58b1ff700 1 Processor -- start 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.562+0000 7fd58b1ff700 1 -- start start 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58b1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 0x7fd58413e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58b1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58b1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd58413ff70 0x7fd5840b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58b1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58413ee50 con 0x7fd58413fba0 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58b1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58415aaf0 con 0x7fd58413ff70 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58b1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd584159f20 con 0x7fd5840cdc50 2023-10-07T09:44:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd588f9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 0x7fd58413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd583fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd588f9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 0x7fd58413e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:55898/0 (socket says 172.21.15.29:55898) 2023-10-07T09:44:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd588f9b700 1 -- 172.21.15.29:0/2233999373 learned_addr learned my addr 172.21.15.29:0/2233999373 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.563+0000 7fd58979c700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd58413ff70 0x7fd5840b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd588f9b700 1 -- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd58413ff70 msgr2=0x7fd5840b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd588f9b700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd58413ff70 0x7fd5840b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd588f9b700 1 -- 172.21.15.29:0/2233999373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 msgr2=0x7fd58415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd588f9b700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd588f9b700 1 -- 172.21.15.29:0/2233999373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780094f0 con 0x7fd5840cdc50 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd583fff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd588f9b700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 0x7fd58413e910 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd58406dd00 tx=0x7fd578017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57800bc70 con 0x7fd5840cdc50 2023-10-07T09:44:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578011420 con 0x7fd5840cdc50 2023-10-07T09:44:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5780117e0 con 0x7fd5840cdc50 2023-10-07T09:44:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58415a1c0 con 0x7fd5840cdc50 2023-10-07T09:44:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.564+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58415a780 con 0x7fd5840cdc50 2023-10-07T09:44:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.566+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58404c3c0 con 0x7fd5840cdc50 2023-10-07T09:44:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.566+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fd578011b90 con 0x7fd5840cdc50 2023-10-07T09:44:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.567+0000 7fd580ff9700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 0x7fd568069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.567+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fd578086350 con 0x7fd5840cdc50 2023-10-07T09:44:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.567+0000 7fd583fff700 1 -- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 msgr2=0x7fd568069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.567+0000 7fd583fff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 0x7fd568069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.571+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd578053020 con 0x7fd5840cdc50 2023-10-07T09:44:43.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.769+0000 7fd583fff700 1 -- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 msgr2=0x7fd568069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:43.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.769+0000 7fd583fff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 0x7fd568069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:43.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.776+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5840ccfc0 con 0x7fd5840cdc50 2023-10-07T09:44:43.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.776+0000 7fd580ff9700 1 -- 172.21.15.29:0/2233999373 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd578056640 con 0x7fd5840cdc50 2023-10-07T09:44:43.777 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:44:43.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 msgr2=0x7fd568069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:43.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 0x7fd568069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 msgr2=0x7fd58413e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 0x7fd58413e910 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd58406dd00 tx=0x7fd578017000 comp rx=0 tx=0).stop 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 shutdown_connections 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5680670e0 0x7fd568069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd58413ff70 0x7fd5840b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5840cdc50 0x7fd58413e910 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 --2- 172.21.15.29:0/2233999373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58413fba0 0x7fd58415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 >> 172.21.15.29:0/2233999373 conn(0x7fd58406b1d0 msgr2=0x7fd584067610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 shutdown_connections 2023-10-07T09:44:43.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.779+0000 7fd58b1ff700 1 -- 172.21.15.29:0/2233999373 wait complete. 2023-10-07T09:44:43.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.998+0000 7fd0ad5ff700 1 Processor -- start 2023-10-07T09:44:43.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:43.998+0000 7fd0ad5ff700 1 -- start start 2023-10-07T09:44:44.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0ad5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd0a80ae570 0x7fd0a80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0ad5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0a80b0520 0x7fd0a80b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0ad5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80b0ea0 con 0x7fd0a8156180 2023-10-07T09:44:44.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0ad5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a80ad4d0 con 0x7fd0a80b0520 2023-10-07T09:44:44.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0ad5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a80ad7b0 con 0x7fd0a80ae570 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0a80b0520 0x7fd0a80b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0a80b0520 0x7fd0a80b0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57316/0 (socket says 172.21.15.29:57316) 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a6ffd700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd0a8156180 0x7fd0a80accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:42926/0 (socket says 172.21.15.29:42926) 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 -- 172.21.15.29:0/4171468319 learned_addr learned my addr 172.21.15.29:0/4171468319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a67fc700 1 --2- 172.21.15.29:0/4171468319 >> v2:172.21.15.144:3300/0 conn(0x7fd0a80ae570 0x7fd0a80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 -- 172.21.15.29:0/4171468319 >> v2:172.21.15.144:3300/0 conn(0x7fd0a80ae570 msgr2=0x7fd0a80afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 --2- 172.21.15.29:0/4171468319 >> v2:172.21.15.144:3300/0 conn(0x7fd0a80ae570 0x7fd0a80afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 -- 172.21.15.29:0/4171468319 >> v1:172.21.15.144:6789/0 conn(0x7fd0a8156180 legacy=0x7fd0a80accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a77fe700 1 -- 172.21.15.29:0/4171468319 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80ada90 con 0x7fd0a80b0520 2023-10-07T09:44:44.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.001+0000 7fd0a5ffb700 1 -- 172.21.15.29:0/4171468319 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092849067 0 0) 0x7fd0a80b0ea0 con 0x7fd0a8156180 2023-10-07T09:44:44.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.002+0000 7fd0a77fe700 1 --2- 172.21.15.29:0/4171468319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80b0520 0x7fd0a80b0960 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80c8d10 tx=0x7fd0a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a5869917bc9bc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:44.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.002+0000 7fd0a5ffb700 1 -- 172.21.15.29:0/4171468319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000e070 con 0x7fd0a80b0520 2023-10-07T09:44:44.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.002+0000 7fd0a5ffb700 1 -- 172.21.15.29:0/4171468319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a000ba20 con 0x7fd0a80b0520 2023-10-07T09:44:44.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.002+0000 7fd0a5ffb700 1 -- 172.21.15.29:0/4171468319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00092e0 con 0x7fd0a80b0520 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/4171468319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80b0520 msgr2=0x7fd0a80b0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/4171468319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80b0520 0x7fd0a80b0960 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80c8d10 tx=0x7fd0a0009840 comp rx=0 tx=0).stop 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/4171468319 shutdown_connections 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/4171468319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80b0520 0x7fd0a80b0960 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/4171468319 >> v2:172.21.15.144:3300/0 conn(0x7fd0a80ae570 0x7fd0a80afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/4171468319 >> 172.21.15.29:0/4171468319 conn(0x7fd0a8067610 msgr2=0x7fd0a8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/4171468319 shutdown_connections 2023-10-07T09:44:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.003+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/4171468319 wait complete. 2023-10-07T09:44:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.005+0000 7fd0ad5ff700 1 Processor -- start 2023-10-07T09:44:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.005+0000 7fd0ad5ff700 1 -- start start 2023-10-07T09:44:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0ad5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 0x7fd0a8152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0ad5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0a80b0520 0x7fd0a8169850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0ad5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0a8156180 0x7fd0a8158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0ad5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80c5de0 con 0x7fd0a80ae570 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0ad5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80c54b0 con 0x7fd0a80b0520 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0ad5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80c5790 con 0x7fd0a8156180 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 0x7fd0a8152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 0x7fd0a8152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57326/0 (socket says 172.21.15.29:57326) 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 -- 172.21.15.29:0/2437279061 learned_addr learned my addr 172.21.15.29:0/2437279061 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a67fc700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0a80b0520 0x7fd0a8169850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a77fe700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0a8156180 0x7fd0a8158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 -- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0a8156180 msgr2=0x7fd0a8158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0a8156180 0x7fd0a8158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 -- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0a80b0520 msgr2=0x7fd0a8169850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0a80b0520 0x7fd0a8169850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.007+0000 7fd0a6ffd700 1 -- 172.21.15.29:0/2437279061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a0009ae0 con 0x7fd0a80ae570 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.008+0000 7fd0a6ffd700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 0x7fd0a8152bd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd09800c5b0 tx=0x7fd098001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.008+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098005a70 con 0x7fd0a80ae570 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.008+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd098005bd0 con 0x7fd0a80ae570 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.008+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a80c5a70 con 0x7fd0a80ae570 2023-10-07T09:44:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.008+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0980070d0 con 0x7fd0a80ae570 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.008+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a8169d90 con 0x7fd0a80ae570 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.009+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a8169fb0 con 0x7fd0a80ae570 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.010+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fd098007480 con 0x7fd0a80ae570 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.011+0000 7fd08effd700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 0x7fd094069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.011+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fd098083f80 con 0x7fd0a80ae570 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.011+0000 7fd0a67fc700 1 -- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 msgr2=0x7fd094069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.012+0000 7fd0a67fc700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 0x7fd094069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:44.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.015+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd098050c50 con 0x7fd0a80ae570 2023-10-07T09:44:44.082 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.081+0000 7f28820a61c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:44:44.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.167+0000 7f28820a61c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:44:44.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.213+0000 7fd0a67fc700 1 -- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 msgr2=0x7fd094069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:44.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.213+0000 7fd0a67fc700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 0x7fd094069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:44.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.219+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0a804c3c0 con 0x7fd0a80ae570 2023-10-07T09:44:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.219+0000 7fd08effd700 1 -- 172.21.15.29:0/2437279061 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd098017070 con 0x7fd0a80ae570 2023-10-07T09:44:44.219 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 msgr2=0x7fd094069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 0x7fd094069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 msgr2=0x7fd0a8152bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 0x7fd0a8152bd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd09800c5b0 tx=0x7fd098001040 comp rx=0 tx=0).stop 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 shutdown_connections 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd0940670e0 0x7fd094069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0a8156180 0x7fd0a8158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a80ae570 0x7fd0a8152bd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.221+0000 7fd0ad5ff700 1 --2- 172.21.15.29:0/2437279061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0a80b0520 0x7fd0a8169850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.222+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 >> 172.21.15.29:0/2437279061 conn(0x7fd0a8067610 msgr2=0x7fd0a8153780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:44.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.222+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 shutdown_connections 2023-10-07T09:44:44.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.222+0000 7fd0ad5ff700 1 -- 172.21.15.29:0/2437279061 wait complete. 2023-10-07T09:44:44.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:44.240 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:44.239+0000 7fc0ecb251c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:44:44.320 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.319+0000 7f28820a61c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:44:44.340 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:44.340+0000 7fc0ecb251c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:44:44.407 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.406+0000 7f28820a61c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:44:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.445+0000 7f34f87ff700 1 Processor -- start 2023-10-07T09:44:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.445+0000 7f34f87ff700 1 -- start start 2023-10-07T09:44:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f87ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f34f0153cc0 0x7f34f00b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f015ade0 0x7f34f00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f87ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f0154090 con 0x7f34f00b0000 2023-10-07T09:44:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f87ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f0159f60 con 0x7f34f0153cc0 2023-10-07T09:44:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f015a240 con 0x7f34f015ade0 2023-10-07T09:44:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f6d9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f34f00b0000 0x7f34f015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58634/0 (socket says 172.21.15.29:58634) 2023-10-07T09:44:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f6d9c700 1 -- 172.21.15.29:0/929992430 learned_addr learned my addr 172.21.15.29:0/929992430 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f659b700 1 --2- 172.21.15.29:0/929992430 >> v2:172.21.15.144:3300/0 conn(0x7f34f0153cc0 0x7f34f00b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.446+0000 7f34f5d9a700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f015ade0 0x7f34f00b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f659b700 1 -- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f015ade0 msgr2=0x7f34f00b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f5599700 1 -- 172.21.15.29:0/929992430 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844714306 0 0) 0x7f34f0154090 con 0x7f34f00b0000 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f659b700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f015ade0 0x7f34f00b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f659b700 1 -- 172.21.15.29:0/929992430 >> v1:172.21.15.29:6789/0 conn(0x7f34f00b0000 legacy=0x7f34f015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f659b700 1 -- 172.21.15.29:0/929992430 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f015a520 con 0x7f34f0153cc0 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f5d9a700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f015ade0 0x7f34f00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.447+0000 7f34f659b700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f00b17c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098d0 tx=0x7f34ec009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ad7eea4a18b831 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f5599700 1 -- 172.21.15.29:0/929992430 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec005c10 con 0x7f34f0153cc0 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f5599700 1 -- 172.21.15.29:0/929992430 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec005d70 con 0x7f34f0153cc0 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f5599700 1 -- 172.21.15.29:0/929992430 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec006280 con 0x7f34f0153cc0 2023-10-07T09:44:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 -- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 msgr2=0x7f34f00b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f00b17c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098d0 tx=0x7f34ec009120 comp rx=0 tx=0).stop 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 -- 172.21.15.29:0/929992430 shutdown_connections 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f00b17c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 --2- 172.21.15.29:0/929992430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f015ade0 0x7f34f00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 -- 172.21.15.29:0/929992430 >> 172.21.15.29:0/929992430 conn(0x7f34f0067610 msgr2=0x7f34f0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 -- 172.21.15.29:0/929992430 shutdown_connections 2023-10-07T09:44:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.448+0000 7f34f87ff700 1 -- 172.21.15.29:0/929992430 wait complete. 2023-10-07T09:44:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.450+0000 7f34f87ff700 1 Processor -- start 2023-10-07T09:44:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.450+0000 7f34f87ff700 1 -- start start 2023-10-07T09:44:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f00b0000 0x7f34f013e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f87ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f0169db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f87ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 0x7f34f00aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f87ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00c81a0 con 0x7f34f015ade0 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f87ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0157410 con 0x7f34f0153cc0 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0156e30 con 0x7f34f00b0000 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f6d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 0x7f34f00aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f6d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 0x7f34f00aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57342/0 (socket says 172.21.15.29:57342) 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f6d9c700 1 -- 172.21.15.29:0/4050632409 learned_addr learned my addr 172.21.15.29:0/4050632409 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f5d9a700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f0169db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.451+0000 7f34f659b700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f00b0000 0x7f34f013e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f6d9c700 1 -- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f00b0000 msgr2=0x7f34f013e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f6d9c700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f00b0000 0x7f34f013e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f6d9c700 1 -- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 msgr2=0x7f34f0169db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f6d9c700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f0169db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f6d9c700 1 -- 172.21.15.29:0/4050632409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e0009420 con 0x7f34f015ade0 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f6d9c700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 0x7f34f00aefa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f34e800ccc0 tx=0x7f34e8001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e800a8a0 con 0x7f34f015ade0 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e800aa00 con 0x7f34f015ade0 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e800adc0 con 0x7f34f015ade0 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.452+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ec0093c0 con 0x7f34f015ade0 2023-10-07T09:44:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.453+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f016aba0 con 0x7f34f015ade0 2023-10-07T09:44:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.454+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f01575c0 con 0x7f34f015ade0 2023-10-07T09:44:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.455+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f34e8007300 con 0x7f34f015ade0 2023-10-07T09:44:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.455+0000 7f34e67fc700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 0x7f34d4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.455+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f34e8084d90 con 0x7f34f015ade0 2023-10-07T09:44:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.456+0000 7f34f659b700 1 -- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 msgr2=0x7f34d4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.456+0000 7f34f659b700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 0x7f34d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:44.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.459+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34f01575c0 con 0x7f34f015ade0 2023-10-07T09:44:44.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:44.505+0000 7fc0ecb251c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:44:44.597 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:44.597+0000 7fc0ecb251c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:44:44.625 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.624+0000 7f28820a61c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:44:44.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.657+0000 7f34f659b700 1 -- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 msgr2=0x7f34d4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:44.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.657+0000 7f34f659b700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 0x7f34d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:44.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.713+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34f01575c0 con 0x7f34f015ade0 2023-10-07T09:44:44.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.714+0000 7f34e67fc700 1 -- 172.21.15.29:0/4050632409 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f34f01575c0 con 0x7f34f015ade0 2023-10-07T09:44:44.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:44.714 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:44:23.775309+0000","modified":"2023-10-07T09:44:27.299910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6276":{"gid":6276,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2418315778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2418315778},{"type":"v1","addr":"172.21.15.29:6835","nonce":2418315778}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.716+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 msgr2=0x7f34d4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.716+0000 7f34f87ff700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 0x7f34d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.716+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 msgr2=0x7f34f00aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.716+0000 7f34f87ff700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 0x7f34f00aefa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f34e800ccc0 tx=0x7f34e8001a80 comp rx=0 tx=0).stop 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.716+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 shutdown_connections 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.716+0000 7f34f87ff700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f34d40670e0 0x7f34d4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.717+0000 7f34f87ff700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34f015ade0 0x7f34f00aefa0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.717+0000 7f34f87ff700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34f00b0000 0x7f34f013e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.717+0000 7f34f87ff700 1 --2- 172.21.15.29:0/4050632409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34f0153cc0 0x7f34f0169db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.717+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 >> 172.21.15.29:0/4050632409 conn(0x7f34f0067610 msgr2=0x7f34f006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:44.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.717+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 shutdown_connections 2023-10-07T09:44:44.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.717+0000 7f34f87ff700 1 -- 172.21.15.29:0/4050632409 wait complete. 2023-10-07T09:44:44.718 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-07T09:44:44.719 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.718+0000 7f28820a61c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:44:44.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:44.814 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.813+0000 7f28820a61c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:44:44.821 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:44.821+0000 7fc0ecb251c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:44:44.909 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:44.908+0000 7f28820a61c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:44:44.922 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:44.922+0000 7fc0ecb251c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:44:44.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.935+0000 7f33927ff700 1 Processor -- start 2023-10-07T09:44:44.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.936+0000 7f33927ff700 1 -- start start 2023-10-07T09:44:44.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f33927ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 0x7f338c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f33927ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f338c159ec0 0x7f338c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f33927ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c14d790 con 0x7f338c1568c0 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f33927ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338c0ae5d0 con 0x7f338c159ec0 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f33927ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c15a850 con 0x7f338c0ad370 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f338bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 0x7f338c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f338b7fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f338c159ec0 0x7f338c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f338bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 0x7f338c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:55930/0 (socket says 172.21.15.29:55930) 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f338bfff700 1 -- 172.21.15.29:0/1606906643 learned_addr learned my addr 172.21.15.29:0/1606906643 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:44.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.937+0000 7f3390d9c700 1 --1- 172.21.15.29:0/1606906643 >> v1:172.21.15.144:6789/0 conn(0x7f338c1568c0 0x7f338c156d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:42936/0 (socket says 172.21.15.29:42936) 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338affd700 1 -- 172.21.15.29:0/1606906643 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081024681 0 0) 0x7f338c14d790 con 0x7f338c1568c0 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338affd700 1 -- 172.21.15.29:0/1606906643 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3374003570 con 0x7f338c1568c0 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338bfff700 1 -- 172.21.15.29:0/1606906643 >> v2:172.21.15.144:3300/0 conn(0x7f338c159ec0 msgr2=0x7f338c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338bfff700 1 --2- 172.21.15.29:0/1606906643 >> v2:172.21.15.144:3300/0 conn(0x7f338c159ec0 0x7f338c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338bfff700 1 -- 172.21.15.29:0/1606906643 >> v1:172.21.15.144:6789/0 conn(0x7f338c1568c0 legacy=0x7f338c156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338bfff700 1 -- 172.21.15.29:0/1606906643 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c0ad740 con 0x7f338c0ad370 2023-10-07T09:44:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338b7fe700 1 --2- 172.21.15.29:0/1606906643 >> v2:172.21.15.144:3300/0 conn(0x7f338c159ec0 0x7f338c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338bfff700 1 --2- 172.21.15.29:0/1606906643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 0x7f338c0cc870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f337c0099b0 tx=0x7f337c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d381995ff4f5df31 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.938+0000 7f338affd700 1 -- 172.21.15.29:0/1606906643 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00f6b0 con 0x7f338c0ad370 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f338affd700 1 -- 172.21.15.29:0/1606906643 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c00f810 con 0x7f338c0ad370 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f338affd700 1 -- 172.21.15.29:0/1606906643 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00fbb0 con 0x7f338c0ad370 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 -- 172.21.15.29:0/1606906643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 msgr2=0x7f338c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 --2- 172.21.15.29:0/1606906643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 0x7f338c0cc870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f337c0099b0 tx=0x7f337c009250 comp rx=0 tx=0).stop 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 -- 172.21.15.29:0/1606906643 shutdown_connections 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 --2- 172.21.15.29:0/1606906643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c0ad370 0x7f338c0cc870 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 --2- 172.21.15.29:0/1606906643 >> v2:172.21.15.144:3300/0 conn(0x7f338c159ec0 0x7f338c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 -- 172.21.15.29:0/1606906643 >> 172.21.15.29:0/1606906643 conn(0x7f338c0680a0 msgr2=0x7f338c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 -- 172.21.15.29:0/1606906643 shutdown_connections 2023-10-07T09:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.939+0000 7f33927ff700 1 -- 172.21.15.29:0/1606906643 wait complete. 2023-10-07T09:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.941+0000 7f33927ff700 1 Processor -- start 2023-10-07T09:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.942+0000 7f33927ff700 1 -- start start 2023-10-07T09:44:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f33927ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 0x7f338c150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f33927ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c1568c0 0x7f338c0a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f33927ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338c159ec0 0x7f338c14cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f33927ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c0ab850 con 0x7f338c0ad370 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f33927ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c14d240 con 0x7f338c159ec0 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f33927ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c16d020 con 0x7f338c1568c0 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f338bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 0x7f338c150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f338bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 0x7f338c150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57356/0 (socket says 172.21.15.29:57356) 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f338bfff700 1 -- 172.21.15.29:0/831665457 learned_addr learned my addr 172.21.15.29:0/831665457 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f338b7fe700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c1568c0 0x7f338c0a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.943+0000 7f3390d9c700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338c159ec0 0x7f338c14cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f338bfff700 1 -- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c1568c0 msgr2=0x7f338c0a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f338bfff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c1568c0 0x7f338c0a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f338bfff700 1 -- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338c159ec0 msgr2=0x7f338c14cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f338bfff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338c159ec0 0x7f338c14cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f338bfff700 1 -- 172.21.15.29:0/831665457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3380009420 con 0x7f338c0ad370 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f338bfff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 0x7f338c150210 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f337c0094c0 tx=0x7f337c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00f980 con 0x7f338c0ad370 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c00fae0 con 0x7f338c0ad370 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c0106c0 con 0x7f338c0ad370 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c0094f0 con 0x7f338c0ad370 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.944+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c16d5e0 con 0x7f338c0ad370 2023-10-07T09:44:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.946+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c04c3c0 con 0x7f338c0ad370 2023-10-07T09:44:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.946+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f337c02c430 con 0x7f338c0ad370 2023-10-07T09:44:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.947+0000 7f3373fff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 0x7f3378069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.947+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f337c014070 con 0x7f338c0ad370 2023-10-07T09:44:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.947+0000 7f338b7fe700 1 -- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 msgr2=0x7f3378069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.947+0000 7f338b7fe700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 0x7f3378069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:44.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:44.951+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f337c0530d0 con 0x7f338c0ad370 2023-10-07T09:44:45.024 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:45.024+0000 7fc0ecb251c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:44:45.098 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:45.097+0000 7f28820a61c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:44:45.123 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:45.123+0000 7fc0ecb251c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:44:45.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.148+0000 7f338b7fe700 1 -- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 msgr2=0x7f3378069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:45.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.148+0000 7f338b7fe700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 0x7f3378069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.154+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f338c157590 con 0x7f338c0ad370 2023-10-07T09:44:45.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.155+0000 7f3373fff700 1 -- 172.21.15.29:0/831665457 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+16387 (secure 0 0 0) 0x7f337c0566f0 con 0x7f338c0ad370 2023-10-07T09:44:45.156 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:45.156 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":78,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:39.423204+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:21.559742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-07T09:44:22.805091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/807765698":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/797249383":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/894321282":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/29562660":"2023-10-08T09:44:39.423150+0000","172.21.15.144:6836/114714":"2023-10-08T09:44:39.423150+0000","172.21.15.144:6834/114714":"2023-10-08T09:44:39.423150+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.157+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 msgr2=0x7f3378069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.157+0000 7f33927ff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 0x7f3378069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.157+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 msgr2=0x7f338c150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.157+0000 7f33927ff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 0x7f338c150210 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f337c0094c0 tx=0x7f337c019000 comp rx=0 tx=0).stop 2023-10-07T09:44:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 shutdown_connections 2023-10-07T09:44:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f33780670e0 0x7f3378069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338c159ec0 0x7f338c14cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f338c0ad370 0x7f338c150210 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 --2- 172.21.15.29:0/831665457 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f338c1568c0 0x7f338c0a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 >> 172.21.15.29:0/831665457 conn(0x7f338c0680a0 msgr2=0x7f338c0af400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 shutdown_connections 2023-10-07T09:44:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.158+0000 7f33927ff700 1 -- 172.21.15.29:0/831665457 wait complete. 2023-10-07T09:44:45.176 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:44:45.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:45.193 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:45.192+0000 7f28820a61c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:44:45.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:45.326+0000 7fc0ecb251c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:44:45.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.374+0000 7facf0fff700 1 Processor -- start 2023-10-07T09:44:45.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.374+0000 7facf0fff700 1 -- start start 2023-10-07T09:44:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facf0fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7facec159750 0x7facec0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facf0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159b20 0x7facec0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facf0fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec0cbf20 con 0x7facec0ce6d0 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facf0fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0cc200 con 0x7facec159750 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facf0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0cc4e0 con 0x7facec159b20 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facea59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7facec159750 0x7facec0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facea59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7facec159750 0x7facec0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57362/0 (socket says 172.21.15.29:57362) 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facea59c700 1 -- 172.21.15.29:0/2368984831 learned_addr learned my addr 172.21.15.29:0/2368984831 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7face9d9b700 1 --2- 172.21.15.29:0/2368984831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159b20 0x7facec0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facea59c700 1 -- 172.21.15.29:0/2368984831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159b20 msgr2=0x7facec0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facea59c700 1 --2- 172.21.15.29:0/2368984831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159b20 0x7facec0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.376+0000 7facea59c700 1 -- 172.21.15.29:0/2368984831 >> v1:172.21.15.144:6789/0 conn(0x7facec0ce6d0 legacy=0x7facec15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.377+0000 7facea59c700 1 -- 172.21.15.29:0/2368984831 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0cc7c0 con 0x7facec159750 2023-10-07T09:44:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.377+0000 7facea59c700 1 --2- 172.21.15.29:0/2368984831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec159750 0x7facec0d10a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7facdc0099a0 tx=0x7facdc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bd466361bea1a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.377+0000 7face959a700 1 -- 172.21.15.29:0/2368984831 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc009e90 con 0x7facec159750 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.377+0000 7face959a700 1 -- 172.21.15.29:0/2368984831 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc00b860 con 0x7facec159750 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.377+0000 7face959a700 1 -- 172.21.15.29:0/2368984831 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00bc20 con 0x7facec159750 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 -- 172.21.15.29:0/2368984831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec159750 msgr2=0x7facec0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 --2- 172.21.15.29:0/2368984831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec159750 0x7facec0d10a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7facdc0099a0 tx=0x7facdc009250 comp rx=0 tx=0).stop 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 -- 172.21.15.29:0/2368984831 shutdown_connections 2023-10-07T09:44:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 --2- 172.21.15.29:0/2368984831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec159750 0x7facec0d10a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 --2- 172.21.15.29:0/2368984831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159b20 0x7facec0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 -- 172.21.15.29:0/2368984831 >> 172.21.15.29:0/2368984831 conn(0x7facec063a10 msgr2=0x7facec0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 -- 172.21.15.29:0/2368984831 shutdown_connections 2023-10-07T09:44:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.378+0000 7facf0fff700 1 -- 172.21.15.29:0/2368984831 wait complete. 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.379+0000 7facf0fff700 1 Processor -- start 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.380+0000 7facf0fff700 1 -- start start 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.380+0000 7facf0fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 0x7facec144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.380+0000 7facf0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159750 0x7facec153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.380+0000 7facf0fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facec159b20 0x7facec158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facf0fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec0c8620 con 0x7facec0ce6d0 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facf0fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec154b10 con 0x7facec159b20 2023-10-07T09:44:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facf0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0c7a10 con 0x7facec159750 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 0x7facec144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 0x7facec144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57366/0 (socket says 172.21.15.29:57366) 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 -- 172.21.15.29:0/1550242928 learned_addr learned my addr 172.21.15.29:0/1550242928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7face9d9b700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159750 0x7facec153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facead9d700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facec159b20 0x7facec158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 -- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159750 msgr2=0x7facec153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159750 0x7facec153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 -- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facec159b20 msgr2=0x7facec158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facec159b20 0x7facec158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.381+0000 7facea59c700 1 -- 172.21.15.29:0/1550242928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009490 con 0x7facec0ce6d0 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.382+0000 7facea59c700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 0x7facec144f90 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7facdc009290 tx=0x7facdc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.382+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc01f070 con 0x7facec0ce6d0 2023-10-07T09:44:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.382+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc000bc0 con 0x7facec0ce6d0 2023-10-07T09:44:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.382+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec0c7cf0 con 0x7facec0ce6d0 2023-10-07T09:44:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.382+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00bc20 con 0x7facec0ce6d0 2023-10-07T09:44:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.382+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec0c82b0 con 0x7facec0ce6d0 2023-10-07T09:44:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.383+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec04c3c0 con 0x7facec0ce6d0 2023-10-07T09:44:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.384+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7facdc023070 con 0x7facec0ce6d0 2023-10-07T09:44:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.385+0000 7facda7fc700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 0x7faccc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.385+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7facdc085f10 con 0x7facec0ce6d0 2023-10-07T09:44:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.385+0000 7face9d9b700 1 -- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 msgr2=0x7faccc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.385+0000 7face9d9b700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 0x7faccc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.389+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facdc052be0 con 0x7facec0ce6d0 2023-10-07T09:44:45.432 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:45.432+0000 7fc0ecb251c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:44:45.533 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:45.532+0000 7f28820a61c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:44:45.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.586+0000 7face9d9b700 1 -- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 msgr2=0x7faccc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:45.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.586+0000 7face9d9b700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 0x7faccc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:45.620 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:45.620+0000 7f28820a61c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.639+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facec15e2f0 con 0x7facec0ce6d0 2023-10-07T09:44:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.639+0000 7facda7fc700 1 -- 172.21.15.29:0/1550242928 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4597 (secure 0 0 0) 0x7facdc015090 con 0x7facec0ce6d0 2023-10-07T09:44:45.639 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:45.640 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:44:23.775309+0000","modified":"2023-10-07T09:44:27.299910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6276":{"gid":6276,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2418315778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2418315778},{"type":"v1","addr":"172.21.15.29:6835","nonce":2418315778}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-10-07T09:44:45.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 msgr2=0x7faccc069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:45.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 0x7faccc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 msgr2=0x7facec144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 0x7facec144f90 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7facdc009290 tx=0x7facdc017000 comp rx=0 tx=0).stop 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 shutdown_connections 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faccc0670e0 0x7faccc069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facec159b20 0x7facec158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facec0ce6d0 0x7facec144f90 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 --2- 172.21.15.29:0/1550242928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facec159750 0x7facec153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 >> 172.21.15.29:0/1550242928 conn(0x7facec063a10 msgr2=0x7facec1550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 shutdown_connections 2023-10-07T09:44:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.642+0000 7facf0fff700 1 -- 172.21.15.29:0/1550242928 wait complete. 2023-10-07T09:44:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-07T09:44:45.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:45.784 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:45.784+0000 7fc0ecb251c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:44:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.860+0000 7fee609ff700 1 Processor -- start 2023-10-07T09:44:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.860+0000 7fee609ff700 1 -- start start 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee609ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c0ae8b0 0x7fee5c0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee609ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fee5c0b1840 0x7fee5c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee609ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0b1ca0 con 0x7fee5c157290 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee609ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c158ad0 con 0x7fee5c0b1840 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee609ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c15b140 con 0x7fee5c0ae8b0 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee5a59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fee5c157290 0x7fee5c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58638/0 (socket says 172.21.15.29:58638) 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee5a59c700 1 -- 172.21.15.29:0/4203460722 learned_addr learned my addr 172.21.15.29:0/4203460722 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee5ad9d700 1 --2- 172.21.15.29:0/4203460722 >> v2:172.21.15.144:3300/0 conn(0x7fee5c0b1840 0x7fee5c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.862+0000 7fee59d9b700 1 --2- 172.21.15.29:0/4203460722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c0ae8b0 0x7fee5c0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee5959a700 1 -- 172.21.15.29:0/4203460722 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263641067 0 0) 0x7fee4c001110 con 0x7fee5c157290 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee5959a700 1 -- 172.21.15.29:0/4203460722 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44003570 con 0x7fee5c157290 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee59d9b700 1 -- 172.21.15.29:0/4203460722 >> v2:172.21.15.144:3300/0 conn(0x7fee5c0b1840 msgr2=0x7fee5c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee59d9b700 1 --2- 172.21.15.29:0/4203460722 >> v2:172.21.15.144:3300/0 conn(0x7fee5c0b1840 0x7fee5c157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee59d9b700 1 -- 172.21.15.29:0/4203460722 >> v1:172.21.15.29:6789/0 conn(0x7fee5c157290 legacy=0x7fee5c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee59d9b700 1 -- 172.21.15.29:0/4203460722 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c15b420 con 0x7fee5c0ae8b0 2023-10-07T09:44:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee59d9b700 1 --2- 172.21.15.29:0/4203460722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c0ae8b0 0x7fee5c0b1300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee500098e0 tx=0x7fee500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcdfa2a088e0f96a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee5959a700 1 -- 172.21.15.29:0/4203460722 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50011630 con 0x7fee5c0ae8b0 2023-10-07T09:44:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee5959a700 1 -- 172.21.15.29:0/4203460722 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50011790 con 0x7fee5c0ae8b0 2023-10-07T09:44:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.863+0000 7fee5959a700 1 -- 172.21.15.29:0/4203460722 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50011b30 con 0x7fee5c0ae8b0 2023-10-07T09:44:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 -- 172.21.15.29:0/4203460722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c0ae8b0 msgr2=0x7fee5c0b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 --2- 172.21.15.29:0/4203460722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c0ae8b0 0x7fee5c0b1300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee500098e0 tx=0x7fee500091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 -- 172.21.15.29:0/4203460722 shutdown_connections 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 --2- 172.21.15.29:0/4203460722 >> v2:172.21.15.144:3300/0 conn(0x7fee5c0b1840 0x7fee5c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 --2- 172.21.15.29:0/4203460722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c0ae8b0 0x7fee5c0b1300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 -- 172.21.15.29:0/4203460722 >> 172.21.15.29:0/4203460722 conn(0x7fee5c0680a0 msgr2=0x7fee5c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 -- 172.21.15.29:0/4203460722 shutdown_connections 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.864+0000 7fee609ff700 1 -- 172.21.15.29:0/4203460722 wait complete. 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.866+0000 7fee609ff700 1 Processor -- start 2023-10-07T09:44:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.866+0000 7fee609ff700 1 -- start start 2023-10-07T09:44:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee609ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 0x7fee5c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee609ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee5c157290 0x7fee5c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee609ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 0x7fee5c155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee609ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c14edc0 con 0x7fee5c157290 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee609ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c06e980 con 0x7fee5c0b1840 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee609ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c06ec60 con 0x7fee5c14ae10 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee59d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee5c157290 0x7fee5c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee5a59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 0x7fee5c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee5ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 0x7fee5c155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee59d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee5c157290 0x7fee5c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57368/0 (socket says 172.21.15.29:57368) 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee59d9b700 1 -- 172.21.15.29:0/1224200186 learned_addr learned my addr 172.21.15.29:0/1224200186 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee5a59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 0x7fee5c06e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:34894/0 (socket says 172.21.15.29:34894) 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.867+0000 7fee5ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 0x7fee5c155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:55982/0 (socket says 172.21.15.29:55982) 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5a59c700 1 -- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 msgr2=0x7fee5c155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5a59c700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 0x7fee5c155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5a59c700 1 -- 172.21.15.29:0/1224200186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee5c157290 msgr2=0x7fee5c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5a59c700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee5c157290 0x7fee5c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5a59c700 1 -- 172.21.15.29:0/1224200186 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50009420 con 0x7fee5c0b1840 2023-10-07T09:44:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5ad9d700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 0x7fee5c155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee5a59c700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 0x7fee5c06e340 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0058d0 tx=0x7fee4c007bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c001d70 con 0x7fee5c0b1840 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c005b30 con 0x7fee5c0b1840 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0070d0 con 0x7fee5c0b1840 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c06ef40 con 0x7fee5c0b1840 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.868+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c0b1230 con 0x7fee5c0b1840 2023-10-07T09:44:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.869+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c04c3c0 con 0x7fee5c0b1840 2023-10-07T09:44:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.870+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fee4c007440 con 0x7fee5c0b1840 2023-10-07T09:44:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.871+0000 7fee4a7fc700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 0x7fee3c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.872+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fee4c083d60 con 0x7fee5c0b1840 2023-10-07T09:44:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.872+0000 7fee59d9b700 1 -- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 msgr2=0x7fee3c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:45.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.872+0000 7fee59d9b700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 0x7fee3c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:45.879 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:45.879+0000 7fc0ecb251c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:45.880+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee4c050a30 con 0x7fee5c0b1840 2023-10-07T09:44:46.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.073+0000 7fee59d9b700 1 -- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 msgr2=0x7fee3c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:46.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.073+0000 7fee59d9b700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 0x7fee3c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.128+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee5c0b1ca0 con 0x7fee5c0b1840 2023-10-07T09:44:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.129+0000 7fee4a7fc700 1 -- 172.21.15.29:0/1224200186 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fee5c0b1ca0 con 0x7fee5c0b1840 2023-10-07T09:44:46.129 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:46.129 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:44:23.775309+0000","modified":"2023-10-07T09:44:27.299910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6276":{"gid":6276,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2418315778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2418315778},{"type":"v1","addr":"172.21.15.29:6835","nonce":2418315778}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-10-07T09:44:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 msgr2=0x7fee3c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 0x7fee3c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 msgr2=0x7fee5c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 0x7fee5c06e340 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0058d0 tx=0x7fee4c007bf0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 shutdown_connections 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee3c0670e0 0x7fee3c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee5c14ae10 0x7fee5c155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee5c0b1840 0x7fee5c06e340 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 --2- 172.21.15.29:0/1224200186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee5c157290 0x7fee5c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 >> 172.21.15.29:0/1224200186 conn(0x7fee5c0680a0 msgr2=0x7fee5c158c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:46.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.131+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 shutdown_connections 2023-10-07T09:44:46.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.132+0000 7fee609ff700 1 -- 172.21.15.29:0/1224200186 wait complete. 2023-10-07T09:44:46.133 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-07T09:44:46.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:46.163 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:46.161+0000 7f28820a61c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:44:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.346+0000 7f4aff7ff700 1 Processor -- start 2023-10-07T09:44:46.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.346+0000 7f4aff7ff700 1 -- start start 2023-10-07T09:44:46.347 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:46.345+0000 7f28820a61c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aff7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4af8159550 0x7f4af80b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aff7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af806f4e0 con 0x7f4af80b0f50 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aff7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af806e8d0 con 0x7f4af80b2d80 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aff7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af806ebb0 con 0x7f4af8159550 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4afcd9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4af80b0f50 0x7f4af80b2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58652/0 (socket says 172.21.15.29:58652) 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4afcd9a700 1 -- 172.21.15.29:0/631477459 learned_addr learned my addr 172.21.15.29:0/631477459 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4afd59b700 1 --2- 172.21.15.29:0/631477459 >> v2:172.21.15.144:3300/0 conn(0x7f4af8159550 0x7f4af80b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318498756 0 0) 0x7f4af806f4e0 con 0x7f4af80b0f50 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae4003570 con 0x7f4af80b0f50 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.348+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236220528 0 0) 0x7f4af806e8d0 con 0x7f4af80b2d80 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af806f4e0 con 0x7f4af80b2d80 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4afd59b700 1 -- 172.21.15.29:0/631477459 >> v1:172.21.15.144:6789/0 conn(0x7f4af80b2d80 legacy=0x7f4af80a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4afd59b700 1 -- 172.21.15.29:0/631477459 >> v1:172.21.15.29:6789/0 conn(0x7f4af80b0f50 legacy=0x7f4af80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4afd59b700 1 -- 172.21.15.29:0/631477459 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af806ee90 con 0x7f4af8159550 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4afd59b700 1 --2- 172.21.15.29:0/631477459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af80b0a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4af40098e0 tx=0x7f4af40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a53a25048621b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011630 con 0x7f4af8159550 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af4011790 con 0x7f4af8159550 2023-10-07T09:44:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.349+0000 7f4aeffff700 1 -- 172.21.15.29:0/631477459 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011b30 con 0x7f4af8159550 2023-10-07T09:44:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 -- 172.21.15.29:0/631477459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 msgr2=0x7f4af80b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/631477459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af80b0a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4af40098e0 tx=0x7f4af40091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 -- 172.21.15.29:0/631477459 shutdown_connections 2023-10-07T09:44:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/631477459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af80b0a10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 -- 172.21.15.29:0/631477459 >> 172.21.15.29:0/631477459 conn(0x7f4af8069190 msgr2=0x7f4af806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 -- 172.21.15.29:0/631477459 shutdown_connections 2023-10-07T09:44:46.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.350+0000 7f4aff7ff700 1 -- 172.21.15.29:0/631477459 wait complete. 2023-10-07T09:44:46.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.352+0000 7f4aff7ff700 1 Processor -- start 2023-10-07T09:44:46.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.352+0000 7f4aff7ff700 1 -- start start 2023-10-07T09:44:46.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4aff7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0f50 0x7f4af8150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4aff7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 0x7f4af815ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4aff7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af81548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4aff7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af815c9d0 con 0x7f4af80b2d80 2023-10-07T09:44:46.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4aff7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8151080 con 0x7f4af8159550 2023-10-07T09:44:46.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4aff7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af815d3a0 con 0x7f4af80b0f50 2023-10-07T09:44:46.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4afcd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 0x7f4af815ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4afcd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 0x7f4af815ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57380/0 (socket says 172.21.15.29:57380) 2023-10-07T09:44:46.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4afcd9a700 1 -- 172.21.15.29:0/618735949 learned_addr learned my addr 172.21.15.29:0/618735949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:46.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4afd59b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0f50 0x7f4af8150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.353+0000 7f4afdd9c700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af81548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afcd9a700 1 -- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0f50 msgr2=0x7f4af8150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afcd9a700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0f50 0x7f4af8150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afcd9a700 1 -- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 msgr2=0x7f4af81548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afcd9a700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af81548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afcd9a700 1 -- 172.21.15.29:0/618735949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4009420 con 0x7f4af80b2d80 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afd59b700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0f50 0x7f4af8150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:46.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4afcd9a700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 0x7f4af815ce60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c790 tx=0x7f4ae8007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8005b30 con 0x7f4af80b2d80 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae8005cd0 con 0x7f4af80b2d80 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8007320 con 0x7f4af80b2d80 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af815bdc0 con 0x7f4af80b2d80 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.354+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af815c380 con 0x7f4af80b2d80 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.355+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af804c3c0 con 0x7f4af80b2d80 2023-10-07T09:44:46.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.356+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f4ae8012400 con 0x7f4af80b2d80 2023-10-07T09:44:46.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.357+0000 7f4aed7fa700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 0x7f4adc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.357+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f4ae8083f10 con 0x7f4af80b2d80 2023-10-07T09:44:46.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.357+0000 7f4afd59b700 1 -- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 msgr2=0x7f4adc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:46.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.357+0000 7f4afd59b700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 0x7f4adc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.361+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ae8050d30 con 0x7f4af80b2d80 2023-10-07T09:44:46.459 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:46.458+0000 7fc0ecb251c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.557+0000 7f4afd59b700 1 -- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 msgr2=0x7f4adc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.557+0000 7f4afd59b700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 0x7f4adc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.566+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4af815cb10 con 0x7f4af80b2d80 2023-10-07T09:44:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.567+0000 7f4aed7fa700 1 -- 172.21.15.29:0/618735949 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+16387 (secure 0 0 0) 0x7f4ae8017020 con 0x7f4af80b2d80 2023-10-07T09:44:46.568 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:46.568 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":78,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:39.423204+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:21.559742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-07T09:44:22.805091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/807765698":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/797249383":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/894321282":"2023-10-08T09:44:39.423150+0000","172.21.15.144:0/29562660":"2023-10-08T09:44:39.423150+0000","172.21.15.144:6836/114714":"2023-10-08T09:44:39.423150+0000","172.21.15.144:6834/114714":"2023-10-08T09:44:39.423150+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 msgr2=0x7f4adc069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 0x7f4adc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 msgr2=0x7f4af815ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 0x7f4af815ce60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c790 tx=0x7f4ae8007e80 comp rx=0 tx=0).stop 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 shutdown_connections 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4adc0670e0 0x7f4adc069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af8159550 0x7f4af81548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80b0f50 0x7f4af8150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 --2- 172.21.15.29:0/618735949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af80b2d80 0x7f4af815ce60 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 >> 172.21.15.29:0/618735949 conn(0x7f4af8069190 msgr2=0x7f4af8152910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 shutdown_connections 2023-10-07T09:44:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.569+0000 7f4aff7ff700 1 -- 172.21.15.29:0/618735949 wait complete. 2023-10-07T09:44:46.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:44:46.657 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:46.656+0000 7fc0ecb251c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:44:46.723 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:46.722+0000 7f28820a61c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:46.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.791+0000 7fea351ff700 1 Processor -- start 2023-10-07T09:44:46.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.791+0000 7fea351ff700 1 -- start start 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.792+0000 7fea351ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea30159750 0x7fea300d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea351ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fea30159b20 0x7fea300ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea351ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea300ce6d0 0x7fea300ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea351ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300cbf20 con 0x7fea30159750 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea351ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300cc200 con 0x7fea30159b20 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea351ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea300cc4e0 con 0x7fea300ce6d0 2023-10-07T09:44:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea30159750 0x7fea300d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea30159750 0x7fea300d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57392/0 (socket says 172.21.15.29:57392) 2023-10-07T09:44:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 -- 172.21.15.29:0/320250513 learned_addr learned my addr 172.21.15.29:0/320250513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2e59c700 1 --2- 172.21.15.29:0/320250513 >> v2:172.21.15.144:3300/0 conn(0x7fea30159b20 0x7fea300ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2f59e700 1 --2- 172.21.15.29:0/320250513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea300ce6d0 0x7fea300ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 -- 172.21.15.29:0/320250513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea300ce6d0 msgr2=0x7fea300ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/320250513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea300ce6d0 0x7fea300ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 -- 172.21.15.29:0/320250513 >> v2:172.21.15.144:3300/0 conn(0x7fea30159b20 msgr2=0x7fea300ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/320250513 >> v2:172.21.15.144:3300/0 conn(0x7fea30159b20 0x7fea300ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.793+0000 7fea2ed9d700 1 -- 172.21.15.29:0/320250513 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea300cc7c0 con 0x7fea30159750 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.794+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/320250513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159750 0x7fea300d10a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fea30088a70 tx=0x7fea20009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d257a637a769a66 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.794+0000 7fea2dd9b700 1 -- 172.21.15.29:0/320250513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20005c10 con 0x7fea30159750 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.794+0000 7fea2dd9b700 1 -- 172.21.15.29:0/320250513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea20005d70 con 0x7fea30159750 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.794+0000 7fea2dd9b700 1 -- 172.21.15.29:0/320250513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20006260 con 0x7fea30159750 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 -- 172.21.15.29:0/320250513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159750 msgr2=0x7fea300d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 --2- 172.21.15.29:0/320250513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159750 0x7fea300d10a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fea30088a70 tx=0x7fea20009220 comp rx=0 tx=0).stop 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 -- 172.21.15.29:0/320250513 shutdown_connections 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 --2- 172.21.15.29:0/320250513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea300ce6d0 0x7fea300ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 --2- 172.21.15.29:0/320250513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159750 0x7fea300d10a0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 --2- 172.21.15.29:0/320250513 >> v2:172.21.15.144:3300/0 conn(0x7fea30159b20 0x7fea300ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 -- 172.21.15.29:0/320250513 >> 172.21.15.29:0/320250513 conn(0x7fea30063a10 msgr2=0x7fea300d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 -- 172.21.15.29:0/320250513 shutdown_connections 2023-10-07T09:44:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.795+0000 7fea351ff700 1 -- 172.21.15.29:0/320250513 wait complete. 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.797+0000 7fea351ff700 1 Processor -- start 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.797+0000 7fea351ff700 1 -- start start 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea351ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea300ce6d0 0x7fea3016ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea351ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159750 0x7fea3015b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea351ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 0x7fea30153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea351ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3016dd80 con 0x7fea30159b20 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea351ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3015a140 con 0x7fea300ce6d0 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea351ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3016d4e0 con 0x7fea30159750 2023-10-07T09:44:46.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 0x7fea30153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 0x7fea30153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57394/0 (socket says 172.21.15.29:57394) 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 -- 172.21.15.29:0/3703141956 learned_addr learned my addr 172.21.15.29:0/3703141956 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea300ce6d0 0x7fea3016ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2e59c700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159750 0x7fea3015b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 -- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159750 msgr2=0x7fea3015b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159750 0x7fea3015b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 -- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea300ce6d0 msgr2=0x7fea3016ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea300ce6d0 0x7fea3016ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.798+0000 7fea2f59e700 1 -- 172.21.15.29:0/3703141956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20009480 con 0x7fea30159b20 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.799+0000 7fea2f59e700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 0x7fea30153f60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fea28009e90 tx=0x7fea2800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.799+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800cb70 con 0x7fea30159b20 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.799+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2800f040 con 0x7fea30159b20 2023-10-07T09:44:46.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.799+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28014480 con 0x7fea30159b20 2023-10-07T09:44:46.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.799+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3016d7c0 con 0x7fea30159b20 2023-10-07T09:44:46.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.799+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea301544a0 con 0x7fea30159b20 2023-10-07T09:44:46.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.800+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30159f10 con 0x7fea30159b20 2023-10-07T09:44:46.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.801+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fea28014830 con 0x7fea30159b20 2023-10-07T09:44:46.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.802+0000 7fea16ffd700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 0x7fea1c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.803+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fea28085660 con 0x7fea30159b20 2023-10-07T09:44:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.803+0000 7fea2ed9d700 1 -- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 msgr2=0x7fea1c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.803+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 0x7fea1c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:46.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:46.807+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea30159f10 con 0x7fea30159b20 2023-10-07T09:44:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.004+0000 7fea2ed9d700 1 -- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 msgr2=0x7fea1c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.004+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 0x7fea1c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:47.046 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:47.045+0000 7fc0ecb251c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:47.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.057+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fea30159f10 con 0x7fea30159b20 2023-10-07T09:44:47.058 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:44:47.058+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:44:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.062+0000 7fea16ffd700 1 -- 172.21.15.29:0/3703141956 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7fea30159f10 con 0x7fea30159b20 2023-10-07T09:44:47.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 msgr2=0x7fea1c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:47.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 0x7fea1c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 msgr2=0x7fea30153f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 0x7fea30153f60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fea28009e90 tx=0x7fea2800b040 comp rx=0 tx=0).stop 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 shutdown_connections 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea1c0670e0 0x7fea1c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30159b20 0x7fea30153f60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea300ce6d0 0x7fea3016ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 --2- 172.21.15.29:0/3703141956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159750 0x7fea3015b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 >> 172.21.15.29:0/3703141956 conn(0x7fea30063a10 msgr2=0x7fea300ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 shutdown_connections 2023-10-07T09:44:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.065+0000 7fea351ff700 1 -- 172.21.15.29:0/3703141956 wait complete. 2023-10-07T09:44:47.068 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:44:47.069 INFO:tasks.ceph.mds.a.smithi029.stderr: -2090> 2023-10-07T09:44:30.254+0000 7f8b485f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:44:47.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:44:47.110 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:44:47.144 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.143+0000 7f28820a61c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:44:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.288+0000 7f745c1ff700 1 Processor -- start 2023-10-07T09:44:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.288+0000 7f745c1ff700 1 -- start start 2023-10-07T09:44:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.289+0000 7f745c1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406a8c0 0x7f74540c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.289+0000 7f745c1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745406a430 con 0x7f745406b5c0 2023-10-07T09:44:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.289+0000 7f745c1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7454159b60 con 0x7f7454159790 2023-10-07T09:44:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.289+0000 7f745c1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7454069820 con 0x7f745406a8c0 2023-10-07T09:44:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f7459f9b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7454159790 0x7f745406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:42968/0 (socket says 172.21.15.29:42968) 2023-10-07T09:44:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f7459f9b700 1 -- 172.21.15.29:0/3897243196 learned_addr learned my addr 172.21.15.29:0/3897243196 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f745979a700 1 --2- 172.21.15.29:0/3897243196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406a8c0 0x7f74540c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f7458f99700 1 -- 172.21.15.29:0/3897243196 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817029536 0 0) 0x7f7454159b60 con 0x7f7454159790 2023-10-07T09:44:47.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f745979a700 1 -- 172.21.15.29:0/3897243196 >> v1:172.21.15.144:6789/0 conn(0x7f7454159790 legacy=0x7f745406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:47.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f745979a700 1 -- 172.21.15.29:0/3897243196 >> v1:172.21.15.29:6789/0 conn(0x7f745406b5c0 legacy=0x7f74540c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:47.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.290+0000 7f745979a700 1 -- 172.21.15.29:0/3897243196 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454069b00 con 0x7f745406a8c0 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.291+0000 7f745979a700 1 --2- 172.21.15.29:0/3897243196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406a8c0 0x7f74540c6f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7444009910 tx=0x7f74440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2a2ffd1cc09d141 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.291+0000 7f7458f99700 1 -- 172.21.15.29:0/3897243196 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744400b860 con 0x7f745406a8c0 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.291+0000 7f7458f99700 1 -- 172.21.15.29:0/3897243196 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744400b9c0 con 0x7f745406a8c0 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.291+0000 7f7458f99700 1 -- 172.21.15.29:0/3897243196 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744400bd60 con 0x7f745406a8c0 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 -- 172.21.15.29:0/3897243196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406a8c0 msgr2=0x7f74540c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 --2- 172.21.15.29:0/3897243196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406a8c0 0x7f74540c6f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7444009910 tx=0x7f74440091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 -- 172.21.15.29:0/3897243196 shutdown_connections 2023-10-07T09:44:47.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 --2- 172.21.15.29:0/3897243196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406a8c0 0x7f74540c6f40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 -- 172.21.15.29:0/3897243196 >> 172.21.15.29:0/3897243196 conn(0x7f74540c9410 msgr2=0x7f74540c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:47.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 -- 172.21.15.29:0/3897243196 shutdown_connections 2023-10-07T09:44:47.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.292+0000 7f745c1ff700 1 -- 172.21.15.29:0/3897243196 wait complete. 2023-10-07T09:44:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.294+0000 7f745c1ff700 1 Processor -- start 2023-10-07T09:44:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.294+0000 7f745c1ff700 1 -- start start 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.295+0000 7f745c1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 0x7f745415de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.295+0000 7f745c1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7454159790 0x7f745416de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.295+0000 7f745c1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74541521a0 0x7f74540afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.295+0000 7f745c1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745416e380 con 0x7f74541521a0 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.295+0000 7f745c1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74540aa210 con 0x7f7454159790 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.295+0000 7f745c1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745415e500 con 0x7f745406b5c0 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f745a79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74541521a0 0x7f74540afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 0x7f745415de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f745979a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7454159790 0x7f745416de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:47.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 0x7f745415de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:56040/0 (socket says 172.21.15.29:56040) 2023-10-07T09:44:47.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f745979a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7454159790 0x7f745416de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:34932/0 (socket says 172.21.15.29:34932) 2023-10-07T09:44:47.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 -- 172.21.15.29:0/970450441 learned_addr learned my addr 172.21.15.29:0/970450441 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 -- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7454159790 msgr2=0x7f745416de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7454159790 0x7f745416de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 -- 172.21.15.29:0/970450441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74541521a0 msgr2=0x7f74540afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74541521a0 0x7f74540afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.296+0000 7f7459f9b700 1 -- 172.21.15.29:0/970450441 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7444009420 con 0x7f745406b5c0 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.297+0000 7f7459f9b700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 0x7f745415de70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f745000c560 tx=0x7f7450001070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.297+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450005b30 con 0x7f745406b5c0 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.297+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7450005c90 con 0x7f745406b5c0 2023-10-07T09:44:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.297+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74540a98e0 con 0x7f745406b5c0 2023-10-07T09:44:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.297+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74500072e0 con 0x7f745406b5c0 2023-10-07T09:44:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.297+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74540a9ea0 con 0x7f745406b5c0 2023-10-07T09:44:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.298+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7454159020 con 0x7f745406b5c0 2023-10-07T09:44:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.299+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 30) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f745001b3b0 con 0x7f745406b5c0 2023-10-07T09:44:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.300+0000 7f7449ffb700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 0x7f7438069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.300+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7450083e10 con 0x7f745406b5c0 2023-10-07T09:44:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.300+0000 7f745979a700 1 -- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 msgr2=0x7f7438069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.300+0000 7f745979a700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 0x7f7438069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:44:47.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.305+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f745004b5a0 con 0x7f745406b5c0 2023-10-07T09:44:47.325 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.324+0000 7f28820a61c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.501+0000 7f745979a700 1 -- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 msgr2=0x7f7438069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:44:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.501+0000 7f745979a700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 0x7f7438069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:44:47.502 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:47.501+0000 7fc0ecb251c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:44:47.511 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.510+0000 7f28820a61c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:44:47.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.553+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f745404c3c0 con 0x7f745406b5c0 2023-10-07T09:44:47.620 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.619+0000 7f28820a61c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:47.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.631+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mgrmap(e 31) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f7450050f50 con 0x7f745406b5c0 2023-10-07T09:44:47.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.631+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 msgr2=0x7f7438069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:44:47.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:47.631+0000 7f7449ffb700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 0x7f7438069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:47.697 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:47.696+0000 7fc0ecb251c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:44:47.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.704+0000 7f27fcf04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.704+0000 7f27fcf04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.704+0000 7f27fcf04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.704+0000 7f27fcf04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.704+0000 7f27fcf04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.705+0000 7f27f86fb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.705+0000 7f27f86fb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.705+0000 7f27f86fb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.705+0000 7f27f86fb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:44:47.705+0000 7f27f86fb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:44:47.890 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:47.889+0000 7fc0ecb251c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:44:48.002 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:44:48.001+0000 7fc0ecb251c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:44:48.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.063+0000 7f7449ffb700 1 -- 172.21.15.29:0/970450441 <== mon.2 v2:172.21.15.144:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f745007f6e0 con 0x7f745406b5c0 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.065+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 msgr2=0x7f745415de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.065+0000 7f745c1ff700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 0x7f745415de70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f745000c560 tx=0x7f7450001070 comp rx=0 tx=0).stop 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 shutdown_connections 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f74380670e0 0x7f7438069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74541521a0 0x7f74540afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f745406b5c0 0x7f745415de70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 --2- 172.21.15.29:0/970450441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7454159790 0x7f745416de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 >> 172.21.15.29:0/970450441 conn(0x7f74540c9410 msgr2=0x7f745415b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 shutdown_connections 2023-10-07T09:44:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.066+0000 7f745c1ff700 1 -- 172.21.15.29:0/970450441 wait complete. 2023-10-07T09:44:48.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:44:48.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.326+0000 7f84779ff700 1 Processor -- start 2023-10-07T09:44:48.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.326+0000 7f84779ff700 1 -- start start 2023-10-07T09:44:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.327+0000 7f84779ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f84700ced00 0x7f84700b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.327+0000 7f84779ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84700c8a20 0x7f84700cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.327+0000 7f84779ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84700690f0 con 0x7f84700b0610 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f84779ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84700cf2e0 con 0x7f84700c8a20 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f84779ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84700687c0 con 0x7f84700ced00 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f847579b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84700b0610 0x7f84700c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58678/0 (socket says 172.21.15.29:58678) 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f847579b700 1 -- 172.21.15.29:0/428310232 learned_addr learned my addr 172.21.15.29:0/428310232 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8475f9c700 1 --2- 172.21.15.29:0/428310232 >> v2:172.21.15.29:3300/0 conn(0x7f84700c8a20 0x7f84700cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8474f9a700 1 --2- 172.21.15.29:0/428310232 >> v2:172.21.15.144:3300/0 conn(0x7f84700ced00 0x7f84700b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8467fff700 1 -- 172.21.15.29:0/428310232 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322599493 0 0) 0x7f84700690f0 con 0x7f84700b0610 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8467fff700 1 -- 172.21.15.29:0/428310232 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c003570 con 0x7f84700b0610 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8475f9c700 1 -- 172.21.15.29:0/428310232 >> v2:172.21.15.144:3300/0 conn(0x7f84700ced00 msgr2=0x7f84700b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8475f9c700 1 --2- 172.21.15.29:0/428310232 >> v2:172.21.15.144:3300/0 conn(0x7f84700ced00 0x7f84700b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8475f9c700 1 -- 172.21.15.29:0/428310232 >> v1:172.21.15.29:6789/0 conn(0x7f84700b0610 legacy=0x7f84700c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.328+0000 7f8475f9c700 1 -- 172.21.15.29:0/428310232 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8470068aa0 con 0x7f84700c8a20 2023-10-07T09:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.329+0000 7f8475f9c700 1 --2- 172.21.15.29:0/428310232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700c8a20 0x7f84700cc7e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f84700cc2f0 tx=0x7f8468009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18202c02efd29c44 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.329+0000 7f8467fff700 1 -- 172.21.15.29:0/428310232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846800e070 con 0x7f84700c8a20 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.329+0000 7f8467fff700 1 -- 172.21.15.29:0/428310232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846800bab0 con 0x7f84700c8a20 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.329+0000 7f8467fff700 1 -- 172.21.15.29:0/428310232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846800be70 con 0x7f84700c8a20 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.329+0000 7f84779ff700 1 -- 172.21.15.29:0/428310232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700c8a20 msgr2=0x7f84700cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.329+0000 7f84779ff700 1 --2- 172.21.15.29:0/428310232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700c8a20 0x7f84700cc7e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f84700cc2f0 tx=0x7f8468009780 comp rx=0 tx=0).stop 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.330+0000 7f84779ff700 1 -- 172.21.15.29:0/428310232 shutdown_connections 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.330+0000 7f84779ff700 1 --2- 172.21.15.29:0/428310232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700c8a20 0x7f84700cc7e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.330+0000 7f84779ff700 1 --2- 172.21.15.29:0/428310232 >> v2:172.21.15.144:3300/0 conn(0x7f84700ced00 0x7f84700b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.330+0000 7f84779ff700 1 -- 172.21.15.29:0/428310232 >> 172.21.15.29:0/428310232 conn(0x7f8470063a10 msgr2=0x7f84700ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.330+0000 7f84779ff700 1 -- 172.21.15.29:0/428310232 shutdown_connections 2023-10-07T09:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.330+0000 7f84779ff700 1 -- 172.21.15.29:0/428310232 wait complete. 2023-10-07T09:44:48.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.332+0000 7f84779ff700 1 Processor -- start 2023-10-07T09:44:48.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.332+0000 7f84779ff700 1 -- start start 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f84779ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84700b0610 0x7f84700cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f84779ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84700c8a20 0x7f847014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f84779ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 0x7f847015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f84779ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847016a0e0 con 0x7f84700ced00 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f84779ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84701697d0 con 0x7f84700c8a20 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f84779ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84700cb380 con 0x7f84700b0610 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.333+0000 7f8475f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 0x7f847015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 0x7f847015c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57430/0 (socket says 172.21.15.29:57430) 2023-10-07T09:44:48.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 -- 172.21.15.29:0/2205294433 learned_addr learned my addr 172.21.15.29:0/2205294433 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8474f9a700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84700c8a20 0x7f847014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f847579b700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84700b0610 0x7f84700cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 -- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84700b0610 msgr2=0x7f84700cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84700b0610 0x7f84700cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 -- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84700c8a20 msgr2=0x7f847014b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84700c8a20 0x7f847014b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 -- 172.21.15.29:0/2205294433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8468009a20 con 0x7f84700ced00 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8474f9a700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84700c8a20 0x7f847014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.334+0000 7f8475f9c700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 0x7f847015c090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f84680097c0 tx=0x7f8468019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.335+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468027470 con 0x7f84700ced00 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.335+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84680275f0 con 0x7f84700ced00 2023-10-07T09:44:48.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.335+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84680279b0 con 0x7f84700ced00 2023-10-07T09:44:48.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.335+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8470169a90 con 0x7f84700ced00 2023-10-07T09:44:48.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.335+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8470158b30 con 0x7f84700ced00 2023-10-07T09:44:48.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.336+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847006a4a0 con 0x7f84700ced00 2023-10-07T09:44:48.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.337+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 31) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f8468024070 con 0x7f84700ced00 2023-10-07T09:44:48.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.338+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f8468086f70 con 0x7f84700ced00 2023-10-07T09:44:48.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.342+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84680540b0 con 0x7f84700ced00 2023-10-07T09:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.547+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f84700718b0 con 0x7f84700ced00 2023-10-07T09:44:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.634+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 32) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f8468003040 con 0x7f84700ced00 2023-10-07T09:44:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.634+0000 7f84657fa700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f845406c9e0 0x7f845406edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.635+0000 7f847579b700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f845406c9e0 0x7f845406edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:48.637+0000 7f847579b700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f845406c9e0 0x7f845406edc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f847015cec0 tx=0x7f846c009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:49.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.502+0000 7f84657fa700 1 -- 172.21.15.29:0/2205294433 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v81) v1 ==== 181+0+0 (secure 0 0 0) 0x7f846806e7a0 con 0x7f84700ced00 2023-10-07T09:44:49.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.507+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f845406c9e0 msgr2=0x7f845406edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:49.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.507+0000 7f84779ff700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f845406c9e0 0x7f845406edc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f847015cec0 tx=0x7f846c009040 comp rx=0 tx=0).stop 2023-10-07T09:44:49.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.507+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 msgr2=0x7f847015c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:49.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.507+0000 7f84779ff700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 0x7f847015c090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f84680097c0 tx=0x7f8468019000 comp rx=0 tx=0).stop 2023-10-07T09:44:49.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 shutdown_connections 2023-10-07T09:44:49.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f845406c9e0 0x7f845406edc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84700ced00 0x7f847015c090 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84700b0610 0x7f84700cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 --2- 172.21.15.29:0/2205294433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84700c8a20 0x7f847014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 >> 172.21.15.29:0/2205294433 conn(0x7f8470063a10 msgr2=0x7f84700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:49.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 shutdown_connections 2023-10-07T09:44:49.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.508+0000 7f84779ff700 1 -- 172.21.15.29:0/2205294433 wait complete. 2023-10-07T09:44:49.511 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:44:49.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:44:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.735+0000 7f7b251ff700 1 Processor -- start 2023-10-07T09:44:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.735+0000 7f7b251ff700 1 -- start start 2023-10-07T09:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b251ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 0x7f7b200cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b251ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b200cf0f0 0x7f7b200c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b251ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b200cc940 con 0x7f7b200b2a20 2023-10-07T09:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b251ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b200cf530 con 0x7f7b200cf0f0 2023-10-07T09:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b251ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200b2df0 con 0x7f7b200cb8c0 2023-10-07T09:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b1f59e700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b200cf0f0 0x7f7b200c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.737+0000 7f7b1e59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 0x7f7b200cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1ed9d700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7b200b2a20 0x7f7b200cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:42974/0 (socket says 172.21.15.29:42974) 2023-10-07T09:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1f59e700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b200cf0f0 0x7f7b200c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:34970/0 (socket says 172.21.15.29:34970) 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1e59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 0x7f7b200cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:56052/0 (socket says 172.21.15.29:56052) 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1ed9d700 1 -- 172.21.15.29:0/1187486882 learned_addr learned my addr 172.21.15.29:0/1187486882 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1dd9b700 1 -- 172.21.15.29:0/1187486882 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729429069 0 0) 0x7f7b200cc940 con 0x7f7b200b2a20 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1dd9b700 1 -- 172.21.15.29:0/1187486882 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08003570 con 0x7f7b200b2a20 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1f59e700 1 -- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 msgr2=0x7f7b200cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1f59e700 1 --2- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 0x7f7b200cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1f59e700 1 -- 172.21.15.29:0/1187486882 >> v1:172.21.15.144:6789/0 conn(0x7f7b200b2a20 legacy=0x7f7b200cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1f59e700 1 -- 172.21.15.29:0/1187486882 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b200cbd30 con 0x7f7b200cf0f0 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1e59c700 1 --2- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 0x7f7b200cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.738+0000 7f7b1f59e700 1 --2- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200cf0f0 0x7f7b200c5400 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009e60 tx=0x7f7b18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a3e1216b9351759 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b1dd9b700 1 -- 172.21.15.29:0/1187486882 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800e070 con 0x7f7b200cf0f0 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b1dd9b700 1 -- 172.21.15.29:0/1187486882 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1800bab0 con 0x7f7b200cf0f0 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b1dd9b700 1 -- 172.21.15.29:0/1187486882 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800be70 con 0x7f7b200cf0f0 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 -- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200cf0f0 msgr2=0x7f7b200c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 --2- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200cf0f0 0x7f7b200c5400 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009e60 tx=0x7f7b18009780 comp rx=0 tx=0).stop 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 -- 172.21.15.29:0/1187486882 shutdown_connections 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 --2- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200cf0f0 0x7f7b200c5400 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 --2- 172.21.15.29:0/1187486882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cb8c0 0x7f7b200cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 -- 172.21.15.29:0/1187486882 >> 172.21.15.29:0/1187486882 conn(0x7f7b20099820 msgr2=0x7f7b20063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 -- 172.21.15.29:0/1187486882 shutdown_connections 2023-10-07T09:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.739+0000 7f7b251ff700 1 -- 172.21.15.29:0/1187486882 wait complete. 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.741+0000 7f7b251ff700 1 Processor -- start 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.741+0000 7f7b251ff700 1 -- start start 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b251ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200b2a20 0x7f7b2014fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b251ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 0x7f7b200c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b251ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cf0f0 0x7f7b200683b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b251ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20156170 con 0x7f7b200cb8c0 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b251ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200c8010 con 0x7f7b200b2a20 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b251ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200c7720 con 0x7f7b200cf0f0 2023-10-07T09:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b1e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 0x7f7b200c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b1e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 0x7f7b200c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57440/0 (socket says 172.21.15.29:57440) 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b1e59c700 1 -- 172.21.15.29:0/2787345282 learned_addr learned my addr 172.21.15.29:0/2787345282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.742+0000 7f7b1ed9d700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200b2a20 0x7f7b2014fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1f59e700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cf0f0 0x7f7b200683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1e59c700 1 -- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cf0f0 msgr2=0x7f7b200683b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1e59c700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cf0f0 0x7f7b200683b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1e59c700 1 -- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200b2a20 msgr2=0x7f7b2014fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1e59c700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200b2a20 0x7f7b2014fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1e59c700 1 -- 172.21.15.29:0/2787345282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b14009420 con 0x7f7b200cb8c0 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b1e59c700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 0x7f7b200c9980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400ec50 tx=0x7f7b140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1400c9d0 con 0x7f7b200cb8c0 2023-10-07T09:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1400cb30 con 0x7f7b200cb8c0 2023-10-07T09:44:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14013470 con 0x7f7b200cb8c0 2023-10-07T09:44:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b18009a20 con 0x7f7b200cb8c0 2023-10-07T09:44:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.743+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b200c7ca0 con 0x7f7b200cb8c0 2023-10-07T09:44:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.744+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b200c8180 con 0x7f7b200cb8c0 2023-10-07T09:44:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.745+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 32) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f7b140137e0 con 0x7f7b200cb8c0 2023-10-07T09:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.746+0000 7f7b06ffd700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b0c066d80 0x7f7b0c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.747+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f7b140848d0 con 0x7f7b200cb8c0 2023-10-07T09:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.747+0000 7f7b1ed9d700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b0c066d80 0x7f7b0c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.748+0000 7f7b1ed9d700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b0c066d80 0x7f7b0c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b10005920 tx=0x7f7b10009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:49.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.750+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b14051b00 con 0x7f7b200cb8c0 2023-10-07T09:44:49.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:49.953+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7b200718b0 con 0x7f7b200cb8c0 2023-10-07T09:44:50.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.508+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b14051ed0 con 0x7f7b200cb8c0 2023-10-07T09:44:50.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.640+0000 7f7b06ffd700 1 -- 172.21.15.29:0/2787345282 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v82) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7b14051b00 con 0x7f7b200cb8c0 2023-10-07T09:44:50.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b0c066d80 msgr2=0x7f7b0c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b0c066d80 0x7f7b0c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b10005920 tx=0x7f7b10009040 comp rx=0 tx=0).stop 2023-10-07T09:44:50.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 msgr2=0x7f7b200c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 0x7f7b200c9980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400ec50 tx=0x7f7b140091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 shutdown_connections 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b0c066d80 0x7f7b0c069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b200cf0f0 0x7f7b200683b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b200b2a20 0x7f7b2014fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 --2- 172.21.15.29:0/2787345282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b200cb8c0 0x7f7b200c9980 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 >> 172.21.15.29:0/2787345282 conn(0x7f7b20099820 msgr2=0x7f7b200cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:50.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 shutdown_connections 2023-10-07T09:44:50.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.645+0000 7f7b251ff700 1 -- 172.21.15.29:0/2787345282 wait complete. 2023-10-07T09:44:50.651 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:44:50.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T09:44:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.872+0000 7f79193ff700 1 Processor -- start 2023-10-07T09:44:50.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.873+0000 7f79193ff700 1 -- start start 2023-10-07T09:44:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f79193ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 0x7f79140b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f79193ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7914154090 con 0x7f791415ade0 2023-10-07T09:44:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f79193ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7914159f60 con 0x7f79140b0710 2023-10-07T09:44:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f79193ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791415a240 con 0x7f7914153cc0 2023-10-07T09:44:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f79127fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f791415ade0 0x7f79140b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58682/0 (socket says 172.21.15.29:58682) 2023-10-07T09:44:50.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f79127fc700 1 -- 172.21.15.29:0/3695232368 learned_addr learned my addr 172.21.15.29:0/3695232368 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f7912ffd700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 0x7f79140b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.874+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074481525 0 0) 0x7f7914154090 con 0x7f791415ade0 2023-10-07T09:44:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc003570 con 0x7f791415ade0 2023-10-07T09:44:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3929778219 0 0) 0x7f78fc003570 con 0x7f791415ade0 2023-10-07T09:44:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 msgr2=0x7f79140b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 0x7f79140b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 >> v1:172.21.15.144:6789/0 conn(0x7f79140b0710 legacy=0x7f791415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791415a520 con 0x7f791415ade0 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f7908001470 con 0x7f791415ade0 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 >> v1:172.21.15.29:6789/0 conn(0x7f791415ade0 legacy=0x7f79140b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 0x7f78fc003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004010 0x7f78fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78fc004d50 0x7f78fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791415a240 con 0x7f78fc004d50 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc00bbf0 con 0x7f78fc004010 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc00bdb0 con 0x7f7914153cc0 2023-10-07T09:44:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f79137fe700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78fc004d50 0x7f78fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.875+0000 7f7912ffd700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 0x7f78fc003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79127fc700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004010 0x7f78fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79137fe700 1 -- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 msgr2=0x7f78fc003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79137fe700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7914153cc0 0x7f78fc003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79137fe700 1 -- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004010 msgr2=0x7f78fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79137fe700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004010 0x7f78fc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79137fe700 1 -- 172.21.15.29:0/3695232368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc003570 con 0x7f78fc004d50 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f79137fe700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78fc004d50 0x7f78fc0051c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f79140b0b10 tx=0x7f790c001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7136ba223b12f033 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c001950 con 0x7f78fc004d50 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c004030 con 0x7f78fc004d50 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.876+0000 7f7911ffb700 1 -- 172.21.15.29:0/3695232368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c00cb00 con 0x7f78fc004d50 2023-10-07T09:44:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79193ff700 1 -- 172.21.15.29:0/3695232368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78fc004d50 msgr2=0x7f78fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79193ff700 1 --2- 172.21.15.29:0/3695232368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78fc004d50 0x7f78fc0051c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f79140b0b10 tx=0x7f790c001640 comp rx=0 tx=0).stop 2023-10-07T09:44:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79137fe700 1 -- 172.21.15.29:0/3695232368 reap_dead start 2023-10-07T09:44:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79193ff700 1 -- 172.21.15.29:0/3695232368 shutdown_connections 2023-10-07T09:44:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79193ff700 1 -- 172.21.15.29:0/3695232368 >> 172.21.15.29:0/3695232368 conn(0x7f7914067610 msgr2=0x7f7914067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79193ff700 1 -- 172.21.15.29:0/3695232368 shutdown_connections 2023-10-07T09:44:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.877+0000 7f79193ff700 1 -- 172.21.15.29:0/3695232368 wait complete. 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.879+0000 7f79193ff700 1 Processor -- start 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.879+0000 7f79193ff700 1 -- start start 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79193ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004d50 0x7f79140cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79193ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 0x7f791415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79193ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791415bbe0 0x7f791415c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79193ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914142bb0 con 0x7f791415b230 2023-10-07T09:44:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79193ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914143170 con 0x7f78fc004d50 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79193ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79140ce250 con 0x7f791415bbe0 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 0x7f791415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 0x7f791415b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57452/0 (socket says 172.21.15.29:57452) 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 -- 172.21.15.29:0/3643199678 learned_addr learned my addr 172.21.15.29:0/3643199678 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f7912ffd700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004d50 0x7f79140cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79137fe700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791415bbe0 0x7f791415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 -- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791415bbe0 msgr2=0x7f791415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791415bbe0 0x7f791415c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 -- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004d50 msgr2=0x7f79140cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004d50 0x7f79140cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:50.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.880+0000 7f79127fc700 1 -- 172.21.15.29:0/3643199678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791415c740 con 0x7f791415b230 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.881+0000 7f79127fc700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 0x7f791415b6a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f79080010e0 tx=0x7f7908001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.881+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908018030 con 0x7f791415b230 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.881+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790800f8d0 con 0x7f791415b230 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.881+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791415ca20 con 0x7f791415b230 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.881+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790800fc70 con 0x7f791415b230 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.881+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7914152e70 con 0x7f791415b230 2023-10-07T09:44:50.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.882+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791404c3c0 con 0x7f791415b230 2023-10-07T09:44:50.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.883+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f790801b070 con 0x7f791415b230 2023-10-07T09:44:50.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.884+0000 7f78faffd700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7900066f30 0x7f79000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:50.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.884+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f7908019070 con 0x7f791415b230 2023-10-07T09:44:50.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.884+0000 7f7912ffd700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7900066f30 0x7f79000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:50.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.885+0000 7f7912ffd700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7900066f30 0x7f79000693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7904006ea0 tx=0x7f7904008040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:50.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:50.888+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7908051760 con 0x7f791415b230 2023-10-07T09:44:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.087+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f79141432b0 con 0x7f791415b230 2023-10-07T09:44:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.091+0000 7f78faffd700 1 -- 172.21.15.29:0/3643199678 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7908054d80 con 0x7f791415b230 2023-10-07T09:44:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7900066f30 msgr2=0x7f79000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7900066f30 0x7f79000693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7904006ea0 tx=0x7f7904008040 comp rx=0 tx=0).stop 2023-10-07T09:44:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 msgr2=0x7f791415b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 0x7f791415b6a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f79080010e0 tx=0x7f7908001e50 comp rx=0 tx=0).stop 2023-10-07T09:44:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 shutdown_connections 2023-10-07T09:44:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7900066f30 0x7f79000693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791415bbe0 0x7f791415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc004d50 0x7f79140cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.093+0000 7f79193ff700 1 --2- 172.21.15.29:0/3643199678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791415b230 0x7f791415b6a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.094+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 >> 172.21.15.29:0/3643199678 conn(0x7f7914067610 msgr2=0x7f79140b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.094+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 shutdown_connections 2023-10-07T09:44:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.094+0000 7f79193ff700 1 -- 172.21.15.29:0/3643199678 wait complete. 2023-10-07T09:44:51.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T09:44:51.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.316+0000 7f73f05ff700 1 Processor -- start 2023-10-07T09:44:51.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.316+0000 7f73f05ff700 1 -- start start 2023-10-07T09:44:51.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.317+0000 7f73f05ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 0x7f73e80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.317+0000 7f73f05ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 0x7f73e80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.317+0000 7f73f05ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e80b0ec0 con 0x7f73e8156180 2023-10-07T09:44:51.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.317+0000 7f73f05ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e80ad4d0 con 0x7f73e80b0520 2023-10-07T09:44:51.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.317+0000 7f73f05ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e80ad7b0 con 0x7f73e80ae570 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73eeb9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 0x7f73e80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73edb9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 0x7f73e80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73eeb9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 0x7f73e80b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:34996/0 (socket says 172.21.15.29:34996) 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73edb9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 0x7f73e80afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:56114/0 (socket says 172.21.15.29:56114) 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73eeb9c700 1 -- 172.21.15.29:0/3438741551 learned_addr learned my addr 172.21.15.29:0/3438741551 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73ee39b700 1 --1- 172.21.15.29:0/3438741551 >> v1:172.21.15.144:6789/0 conn(0x7f73e8156180 0x7f73e80accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:42986/0 (socket says 172.21.15.29:42986) 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73edb9a700 1 -- 172.21.15.29:0/3438741551 >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 msgr2=0x7f73e80b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73edb9a700 1 --2- 172.21.15.29:0/3438741551 >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 0x7f73e80b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73edb9a700 1 -- 172.21.15.29:0/3438741551 >> v1:172.21.15.144:6789/0 conn(0x7f73e8156180 legacy=0x7f73e80accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73edb9a700 1 -- 172.21.15.29:0/3438741551 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e80ada90 con 0x7f73e80ae570 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.318+0000 7f73eeb9c700 1 --2- 172.21.15.29:0/3438741551 >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 0x7f73e80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.319+0000 7f73edb9a700 1 --2- 172.21.15.29:0/3438741551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 0x7f73e80afc00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f73d80098e0 tx=0x7f73d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb7330eaa173a96 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.319+0000 7f73ed399700 1 -- 172.21.15.29:0/3438741551 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d800b640 con 0x7f73e80ae570 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.319+0000 7f73ed399700 1 -- 172.21.15.29:0/3438741551 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d800b7a0 con 0x7f73e80ae570 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.319+0000 7f73ed399700 1 -- 172.21.15.29:0/3438741551 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d800bb40 con 0x7f73e80ae570 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.319+0000 7f73f05ff700 1 -- 172.21.15.29:0/3438741551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 msgr2=0x7f73e80afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.319+0000 7f73f05ff700 1 --2- 172.21.15.29:0/3438741551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 0x7f73e80afc00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f73d80098e0 tx=0x7f73d80091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.320+0000 7f73f05ff700 1 -- 172.21.15.29:0/3438741551 shutdown_connections 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.320+0000 7f73f05ff700 1 --2- 172.21.15.29:0/3438741551 >> v2:172.21.15.144:3300/0 conn(0x7f73e80b0520 0x7f73e80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.320+0000 7f73f05ff700 1 --2- 172.21.15.29:0/3438741551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80ae570 0x7f73e80afc00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.320+0000 7f73f05ff700 1 -- 172.21.15.29:0/3438741551 >> 172.21.15.29:0/3438741551 conn(0x7f73e8067610 msgr2=0x7f73e8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.320+0000 7f73f05ff700 1 -- 172.21.15.29:0/3438741551 shutdown_connections 2023-10-07T09:44:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.320+0000 7f73f05ff700 1 -- 172.21.15.29:0/3438741551 wait complete. 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.321+0000 7f73f05ff700 1 Processor -- start 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.321+0000 7f73f05ff700 1 -- start start 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.322+0000 7f73f05ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 0x7f73e816a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.322+0000 7f73f05ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80b0520 0x7f73e8144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.322+0000 7f73f05ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73e8156180 0x7f73e8148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.322+0000 7f73f05ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e8154ac0 con 0x7f73e80ae570 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.322+0000 7f73f05ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e81547e0 con 0x7f73e8156180 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.322+0000 7f73f05ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e80c9150 con 0x7f73e80b0520 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 0x7f73e816a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 0x7f73e816a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57454/0 (socket says 172.21.15.29:57454) 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 -- 172.21.15.29:0/691462513 learned_addr learned my addr 172.21.15.29:0/691462513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73eeb9c700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73e8156180 0x7f73e8148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73edb9a700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80b0520 0x7f73e8144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 -- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80b0520 msgr2=0x7f73e8144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80b0520 0x7f73e8144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 -- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73e8156180 msgr2=0x7f73e8148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73e8156180 0x7f73e8148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 -- 172.21.15.29:0/691462513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e0009ae0 con 0x7f73e80ae570 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.323+0000 7f73ee39b700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 0x7f73e816a7e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f73e400c560 tx=0x7f73e4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.324+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4005a70 con 0x7f73e80ae570 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.324+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e4005bd0 con 0x7f73e80ae570 2023-10-07T09:44:51.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.324+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e40070d0 con 0x7f73e80ae570 2023-10-07T09:44:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.324+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d8009420 con 0x7f73e80ae570 2023-10-07T09:44:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.324+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e80c9710 con 0x7f73e80ae570 2023-10-07T09:44:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.325+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e80c8fb0 con 0x7f73e80ae570 2023-10-07T09:44:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.326+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f73e4007480 con 0x7f73e80ae570 2023-10-07T09:44:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.327+0000 7f73de7fc700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73cc066f30 0x7f73cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.327+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f73e4083870 con 0x7f73e80ae570 2023-10-07T09:44:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.327+0000 7f73edb9a700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73cc066f30 0x7f73cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.328+0000 7f73edb9a700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73cc066f30 0x7f73cc0693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ae980 tx=0x7f73d8017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:51.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.330+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73e80c8fb0 con 0x7f73e80ae570 2023-10-07T09:44:51.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:44:51.518 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:44:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.534+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f73e80b0ec0 con 0x7f73e80ae570 2023-10-07T09:44:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.538+0000 7f73de7fc700 1 -- 172.21.15.29:0/691462513 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f73e80b0ec0 con 0x7f73e80ae570 2023-10-07T09:44:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73cc066f30 msgr2=0x7f73cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73cc066f30 0x7f73cc0693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ae980 tx=0x7f73d8017000 comp rx=0 tx=0).stop 2023-10-07T09:44:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 msgr2=0x7f73e816a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 0x7f73e816a7e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f73e400c560 tx=0x7f73e4007f70 comp rx=0 tx=0).stop 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 shutdown_connections 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73cc066f30 0x7f73cc0693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73e8156180 0x7f73e8148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e80ae570 0x7f73e816a7e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 --2- 172.21.15.29:0/691462513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73e80b0520 0x7f73e8144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 >> 172.21.15.29:0/691462513 conn(0x7f73e8067610 msgr2=0x7f73e81587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 shutdown_connections 2023-10-07T09:44:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.541+0000 7f73f05ff700 1 -- 172.21.15.29:0/691462513 wait complete. 2023-10-07T09:44:51.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T09:44:51.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.798+0000 7fd4e33ff700 1 Processor -- start 2023-10-07T09:44:51.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.799+0000 7fd4e33ff700 1 -- start start 2023-10-07T09:44:51.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e33ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4dc14c8d0 0x7fd4dc0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e33ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd4dc14cd70 0x7fd4dc159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e33ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc157790 con 0x7fd4dc153cc0 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e33ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0b0950 con 0x7fd4dc14c8d0 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e33ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc157580 con 0x7fd4dc14cd70 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e099a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4dc14c8d0 0x7fd4dc0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e099a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4dc14c8d0 0x7fd4dc0afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57458/0 (socket says 172.21.15.29:57458) 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.800+0000 7fd4e099a700 1 -- 172.21.15.29:0/1788829501 learned_addr learned my addr 172.21.15.29:0/1788829501 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e199c700 1 --2- 172.21.15.29:0/1788829501 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc14cd70 0x7fd4dc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e099a700 1 -- 172.21.15.29:0/1788829501 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc14cd70 msgr2=0x7fd4dc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e099a700 1 --2- 172.21.15.29:0/1788829501 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc14cd70 0x7fd4dc159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e099a700 1 -- 172.21.15.29:0/1788829501 >> v1:172.21.15.29:6789/0 conn(0x7fd4dc153cc0 legacy=0x7fd4dc0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e099a700 1 -- 172.21.15.29:0/1788829501 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc157a70 con 0x7fd4dc14c8d0 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4d3fff700 1 -- 172.21.15.29:0/1788829501 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812901542 0 0) 0x7fd4dc157790 con 0x7fd4dc153cc0 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e199c700 1 --2- 172.21.15.29:0/1788829501 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc14cd70 0x7fd4dc159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4e099a700 1 --2- 172.21.15.29:0/1788829501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc14c8d0 0x7fd4dc0afed0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc15a760 tx=0x7fd4cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280b81e376cba2a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4d3fff700 1 -- 172.21.15.29:0/1788829501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00b860 con 0x7fd4dc14c8d0 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4d3fff700 1 -- 172.21.15.29:0/1788829501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc00b9c0 con 0x7fd4dc14c8d0 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.801+0000 7fd4d3fff700 1 -- 172.21.15.29:0/1788829501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00bd60 con 0x7fd4dc14c8d0 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1788829501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc14c8d0 msgr2=0x7fd4dc0afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1788829501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc14c8d0 0x7fd4dc0afed0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc15a760 tx=0x7fd4cc009250 comp rx=0 tx=0).stop 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1788829501 shutdown_connections 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1788829501 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc14cd70 0x7fd4dc159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1788829501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc14c8d0 0x7fd4dc0afed0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1788829501 >> 172.21.15.29:0/1788829501 conn(0x7fd4dc067610 msgr2=0x7fd4dc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1788829501 shutdown_connections 2023-10-07T09:44:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.802+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1788829501 wait complete. 2023-10-07T09:44:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.804+0000 7fd4e33ff700 1 Processor -- start 2023-10-07T09:44:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.804+0000 7fd4e33ff700 1 -- start start 2023-10-07T09:44:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc14c8d0 0x7fd4dc147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e33ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc14cd70 0x7fd4dc14af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e33ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 0x7fd4dc14b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e33ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc16abf0 con 0x7fd4dc153cc0 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e33ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc16aed0 con 0x7fd4dc14cd70 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc16b1b0 con 0x7fd4dc14c8d0 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 0x7fd4dc14b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 0x7fd4dc14b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57472/0 (socket says 172.21.15.29:57472) 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 -- 172.21.15.29:0/1405823582 learned_addr learned my addr 172.21.15.29:0/1405823582 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e099a700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc14cd70 0x7fd4dc14af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 -- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc14c8d0 msgr2=0x7fd4dc147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e119b700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc14c8d0 0x7fd4dc147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc14c8d0 0x7fd4dc147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 -- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc14cd70 msgr2=0x7fd4dc14af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc14cd70 0x7fd4dc14af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e199c700 1 -- 172.21.15.29:0/1405823582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc0094f0 con 0x7fd4dc153cc0 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e099a700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc14cd70 0x7fd4dc14af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.806+0000 7fd4e119b700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc14c8d0 0x7fd4dc147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.807+0000 7fd4e199c700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 0x7fd4dc14b440 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400a380 tx=0x7fd4d4007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.807+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400ae10 con 0x7fd4dc153cc0 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.807+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d400d650 con 0x7fd4dc153cc0 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.807+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc16b490 con 0x7fd4dc153cc0 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.807+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400da10 con 0x7fd4dc153cc0 2023-10-07T09:44:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.807+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc14ea50 con 0x7fd4dc153cc0 2023-10-07T09:44:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.808+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc04c3c0 con 0x7fd4dc153cc0 2023-10-07T09:44:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.809+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd4d401e3b0 con 0x7fd4dc153cc0 2023-10-07T09:44:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.810+0000 7fd4d17fa700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd4c0066f30 0x7fd4c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.810+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7fd4d40850d0 con 0x7fd4dc153cc0 2023-10-07T09:44:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.810+0000 7fd4e119b700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd4c0066f30 0x7fd4c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.811+0000 7fd4e119b700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd4c0066f30 0x7fd4c00693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8001820 tx=0x7fd4d8009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:51.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:51.814+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4d40521f0 con 0x7fd4dc153cc0 2023-10-07T09:44:52.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.015+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd4dc16ad30 con 0x7fd4dc153cc0 2023-10-07T09:44:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.019+0000 7fd4d17fa700 1 -- 172.21.15.29:0/1405823582 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd4dc16ad30 con 0x7fd4dc153cc0 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd4c0066f30 msgr2=0x7fd4c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd4c0066f30 0x7fd4c00693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8001820 tx=0x7fd4d8009040 comp rx=0 tx=0).stop 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 msgr2=0x7fd4dc14b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 0x7fd4dc14b440 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400a380 tx=0x7fd4d4007900 comp rx=0 tx=0).stop 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 shutdown_connections 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd4c0066f30 0x7fd4c00693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4dc153cc0 0x7fd4dc14b440 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc14c8d0 0x7fd4dc147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 --2- 172.21.15.29:0/1405823582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc14cd70 0x7fd4dc14af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 >> 172.21.15.29:0/1405823582 conn(0x7fd4dc067610 msgr2=0x7fd4dc1595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 shutdown_connections 2023-10-07T09:44:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.022+0000 7fd4e33ff700 1 -- 172.21.15.29:0/1405823582 wait complete. 2023-10-07T09:44:52.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-07T09:44:52.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.236+0000 7f52929ff700 1 Processor -- start 2023-10-07T09:44:52.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.236+0000 7f52929ff700 1 -- start start 2023-10-07T09:44:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f52929ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f528c159c00 0x7f528c0ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f52929ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f528c068620 0x7f528c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f52929ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c15af00 con 0x7f528c0ad220 2023-10-07T09:44:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f52929ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528c15a050 con 0x7f528c068620 2023-10-07T09:44:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f52929ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528c068f80 con 0x7f528c159c00 2023-10-07T09:44:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f528c068620 0x7f528c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f528c068620 0x7f528c068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57480/0 (socket says 172.21.15.29:57480) 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 -- 172.21.15.29:0/596597327 learned_addr learned my addr 172.21.15.29:0/596597327 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f528b7fe700 1 --2- 172.21.15.29:0/596597327 >> v2:172.21.15.144:3300/0 conn(0x7f528c159c00 0x7f528c0ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 -- 172.21.15.29:0/596597327 >> v2:172.21.15.144:3300/0 conn(0x7f528c159c00 msgr2=0x7f528c0ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 --2- 172.21.15.29:0/596597327 >> v2:172.21.15.144:3300/0 conn(0x7f528c159c00 0x7f528c0ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 -- 172.21.15.29:0/596597327 >> v1:172.21.15.29:6789/0 conn(0x7f528c0ad220 legacy=0x7f528c0ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f5290f9c700 1 -- 172.21.15.29:0/596597327 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c15a250 con 0x7f528c068620 2023-10-07T09:44:52.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.238+0000 7f528b7fe700 1 --2- 172.21.15.29:0/596597327 >> v2:172.21.15.144:3300/0 conn(0x7f528c159c00 0x7f528c0ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:52.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.239+0000 7f5290f9c700 1 --2- 172.21.15.29:0/596597327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c068620 0x7f528c068a40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f528c06d000 tx=0x7f5284009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922085cde1e178fb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:52.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.239+0000 7f528affd700 1 -- 172.21.15.29:0/596597327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400e070 con 0x7f528c068620 2023-10-07T09:44:52.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.239+0000 7f528affd700 1 -- 172.21.15.29:0/596597327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528400ba50 con 0x7f528c068620 2023-10-07T09:44:52.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.239+0000 7f528affd700 1 -- 172.21.15.29:0/596597327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52840092e0 con 0x7f528c068620 2023-10-07T09:44:52.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.239+0000 7f52929ff700 1 -- 172.21.15.29:0/596597327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c068620 msgr2=0x7f528c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:52.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.239+0000 7f52929ff700 1 --2- 172.21.15.29:0/596597327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c068620 0x7f528c068a40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f528c06d000 tx=0x7f5284009840 comp rx=0 tx=0).stop 2023-10-07T09:44:52.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.240+0000 7f52929ff700 1 -- 172.21.15.29:0/596597327 shutdown_connections 2023-10-07T09:44:52.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.240+0000 7f52929ff700 1 --2- 172.21.15.29:0/596597327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c068620 0x7f528c068a40 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.240+0000 7f52929ff700 1 --2- 172.21.15.29:0/596597327 >> v2:172.21.15.144:3300/0 conn(0x7f528c159c00 0x7f528c0ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.240+0000 7f52929ff700 1 -- 172.21.15.29:0/596597327 >> 172.21.15.29:0/596597327 conn(0x7f528c06dd00 msgr2=0x7f528c06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.240+0000 7f52929ff700 1 -- 172.21.15.29:0/596597327 shutdown_connections 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.240+0000 7f52929ff700 1 -- 172.21.15.29:0/596597327 wait complete. 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.241+0000 7f52929ff700 1 Processor -- start 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.241+0000 7f52929ff700 1 -- start start 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f52929ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c068620 0x7f528c13e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f52929ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c0ad220 0x7f528c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f52929ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 0x7f528c0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:52.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f52929ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c16c0b0 con 0x7f528c159c00 2023-10-07T09:44:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f52929ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c16b4a0 con 0x7f528c068620 2023-10-07T09:44:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f52929ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c16b780 con 0x7f528c0ad220 2023-10-07T09:44:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f5290f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 0x7f528c0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f5290f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 0x7f528c0cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57496/0 (socket says 172.21.15.29:57496) 2023-10-07T09:44:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.242+0000 7f5290f9c700 1 -- 172.21.15.29:0/1954330567 learned_addr learned my addr 172.21.15.29:0/1954330567 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f528b7fe700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c0ad220 0x7f528c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f528bfff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c068620 0x7f528c13e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5290f9c700 1 -- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c0ad220 msgr2=0x7f528c0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5290f9c700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c0ad220 0x7f528c0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5290f9c700 1 -- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c068620 msgr2=0x7f528c13e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5290f9c700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c068620 0x7f528c13e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5290f9c700 1 -- 172.21.15.29:0/1954330567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5284009ae0 con 0x7f528c159c00 2023-10-07T09:44:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f528b7fe700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c0ad220 0x7f528c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:52.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5290f9c700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 0x7f528c0cb2c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f528c06d9f0 tx=0x7f5284018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:52.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284008dd0 con 0x7f528c159c00 2023-10-07T09:44:52.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284025460 con 0x7f528c159c00 2023-10-07T09:44:52.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c16ba60 con 0x7f528c159c00 2023-10-07T09:44:52.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.243+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284025820 con 0x7f528c159c00 2023-10-07T09:44:52.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.244+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c13ec80 con 0x7f528c159c00 2023-10-07T09:44:52.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.245+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528c16c260 con 0x7f528c159c00 2023-10-07T09:44:52.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.245+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5284003040 con 0x7f528c159c00 2023-10-07T09:44:52.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.246+0000 7f5273fff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5278066f30 0x7f52780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:52.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.246+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f52840863f0 con 0x7f528c159c00 2023-10-07T09:44:52.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.247+0000 7f528bfff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5278066f30 0x7f52780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:52.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.248+0000 7f528bfff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5278066f30 0x7f52780693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f528c06b540 tx=0x7f527c009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:52.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.251+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5284053510 con 0x7f528c159c00 2023-10-07T09:44:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:52.534+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f528c04c3c0 con 0x7f528c159c00 2023-10-07T09:44:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.522+0000 7f5273fff700 1 -- 172.21.15.29:0/1954330567 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v573) v1 ==== 156+0+0 (secure 0 0 0) 0x7f5284008f30 con 0x7f528c159c00 2023-10-07T09:44:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5278066f30 msgr2=0x7f52780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5278066f30 0x7f52780693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f528c06b540 tx=0x7f527c009040 comp rx=0 tx=0).stop 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 msgr2=0x7f528c0cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 0x7f528c0cb2c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f528c06d9f0 tx=0x7f5284018000 comp rx=0 tx=0).stop 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 shutdown_connections 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5278066f30 0x7f52780693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c159c00 0x7f528c0cb2c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c068620 0x7f528c13e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 --2- 172.21.15.29:0/1954330567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c0ad220 0x7f528c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 >> 172.21.15.29:0/1954330567 conn(0x7f528c06dd00 msgr2=0x7f528c0a9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 shutdown_connections 2023-10-07T09:44:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.525+0000 7f52929ff700 1 -- 172.21.15.29:0/1954330567 wait complete. 2023-10-07T09:44:53.546 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-07T09:44:53.547 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-07T09:44:53.547 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:44:53.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-07T09:44:53.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.746+0000 7f46271ff700 1 Processor -- start 2023-10-07T09:44:53.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.746+0000 7f46271ff700 1 -- start start 2023-10-07T09:44:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f46271ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46200ad220 0x7f4620159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f46271ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f46200ad5f0 0x7f46200ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f46271ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462015af00 con 0x7f4620068620 2023-10-07T09:44:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f46271ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46200ac9d0 con 0x7f46200ad220 2023-10-07T09:44:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f46271ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46200ada40 con 0x7f46200ad5f0 2023-10-07T09:44:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46200ad220 0x7f4620159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46200ad220 0x7f4620159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42810/0 (socket says 172.21.15.29:42810) 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 -- 172.21.15.29:0/3855169665 learned_addr learned my addr 172.21.15.29:0/3855169665 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f461ffff700 1 --2- 172.21.15.29:0/3855169665 >> v2:172.21.15.144:3300/0 conn(0x7f46200ad5f0 0x7f46200ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 -- 172.21.15.29:0/3855169665 >> v2:172.21.15.144:3300/0 conn(0x7f46200ad5f0 msgr2=0x7f46200ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 --2- 172.21.15.29:0/3855169665 >> v2:172.21.15.144:3300/0 conn(0x7f46200ad5f0 0x7f46200ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 -- 172.21.15.29:0/3855169665 >> v1:172.21.15.29:6789/0 conn(0x7f4620068620 legacy=0x7f4620068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.748+0000 7f4624f9b700 1 -- 172.21.15.29:0/3855169665 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462015a250 con 0x7f46200ad220 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.749+0000 7f461ffff700 1 --2- 172.21.15.29:0/3855169665 >> v2:172.21.15.144:3300/0 conn(0x7f46200ad5f0 0x7f46200ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.749+0000 7f4624f9b700 1 --2- 172.21.15.29:0/3855169665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f4620159c00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f462006d000 tx=0x7f4614009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1e5055e3dffbd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.749+0000 7f461f7fe700 1 -- 172.21.15.29:0/3855169665 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461400b860 con 0x7f46200ad220 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.749+0000 7f461f7fe700 1 -- 172.21.15.29:0/3855169665 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461400b9c0 con 0x7f46200ad220 2023-10-07T09:44:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.749+0000 7f461f7fe700 1 -- 172.21.15.29:0/3855169665 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461400bd60 con 0x7f46200ad220 2023-10-07T09:44:53.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 -- 172.21.15.29:0/3855169665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 msgr2=0x7f4620159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 --2- 172.21.15.29:0/3855169665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f4620159c00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f462006d000 tx=0x7f4614009250 comp rx=0 tx=0).stop 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 -- 172.21.15.29:0/3855169665 shutdown_connections 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 --2- 172.21.15.29:0/3855169665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f4620159c00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 --2- 172.21.15.29:0/3855169665 >> v2:172.21.15.144:3300/0 conn(0x7f46200ad5f0 0x7f46200ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 -- 172.21.15.29:0/3855169665 >> 172.21.15.29:0/3855169665 conn(0x7f462006dd00 msgr2=0x7f462006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 -- 172.21.15.29:0/3855169665 shutdown_connections 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.750+0000 7f46271ff700 1 -- 172.21.15.29:0/3855169665 wait complete. 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.751+0000 7f46271ff700 1 Processor -- start 2023-10-07T09:44:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.752+0000 7f46271ff700 1 -- start start 2023-10-07T09:44:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.752+0000 7f46271ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4620068620 0x7f46200cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f46271ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f462015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f46271ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ad5f0 0x7f462016cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f46271ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200cb6d0 con 0x7f46200ad220 2023-10-07T09:44:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f46271ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462015e3b0 con 0x7f4620068620 2023-10-07T09:44:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f46271ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462015f2d0 con 0x7f46200ad5f0 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f462015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f462015de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42822/0 (socket says 172.21.15.29:42822) 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 -- 172.21.15.29:0/1180270023 learned_addr learned my addr 172.21.15.29:0/1180270023 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f4624f9b700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4620068620 0x7f46200cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f462579c700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ad5f0 0x7f462016cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 -- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ad5f0 msgr2=0x7f462016cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ad5f0 0x7f462016cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 -- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4620068620 msgr2=0x7f46200cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4620068620 0x7f46200cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.753+0000 7f461ffff700 1 -- 172.21.15.29:0/1180270023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46140094f0 con 0x7f46200ad220 2023-10-07T09:44:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.754+0000 7f461ffff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f462015de70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f460c00c830 tx=0x7f460c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.754+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c011600 con 0x7f46200ad220 2023-10-07T09:44:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.754+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c011760 con 0x7f46200ad220 2023-10-07T09:44:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.754+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462015e9a0 con 0x7f46200ad220 2023-10-07T09:44:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.754+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c011b20 con 0x7f46200ad220 2023-10-07T09:44:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.754+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462015efe0 con 0x7f46200ad220 2023-10-07T09:44:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.755+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462015f460 con 0x7f46200ad220 2023-10-07T09:44:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.756+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f460c0193b0 con 0x7f46200ad220 2023-10-07T09:44:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.757+0000 7f461cff9700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4604066f30 0x7f46040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.757+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f460c0845e0 con 0x7f46200ad220 2023-10-07T09:44:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.757+0000 7f4624f9b700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4604066f30 0x7f46040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.758+0000 7f4624f9b700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4604066f30 0x7f46040693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f462006d000 tx=0x7f4614017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:53.761+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f460c051700 con 0x7f46200ad220 2023-10-07T09:44:54.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.041+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f462004c3c0 con 0x7f46200ad220 2023-10-07T09:44:54.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.526+0000 7f461cff9700 1 -- 172.21.15.29:0/1180270023 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v574) v1 ==== 169+0+0 (secure 0 0 0) 0x7f460c054d20 con 0x7f46200ad220 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4604066f30 msgr2=0x7f46040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4604066f30 0x7f46040693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f462006d000 tx=0x7f4614017000 comp rx=0 tx=0).stop 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 msgr2=0x7f462015de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f462015de70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f460c00c830 tx=0x7f460c008040 comp rx=0 tx=0).stop 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 shutdown_connections 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4604066f30 0x7f46040693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ad5f0 0x7f462016cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4620068620 0x7f46200cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 --2- 172.21.15.29:0/1180270023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46200ad220 0x7f462015de70 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 >> 172.21.15.29:0/1180270023 conn(0x7f462006dd00 msgr2=0x7f46200aa400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:54.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 shutdown_connections 2023-10-07T09:44:54.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.528+0000 7f46271ff700 1 -- 172.21.15.29:0/1180270023 wait complete. 2023-10-07T09:44:54.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:44:54.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.747+0000 7fd24c3ff700 1 Processor -- start 2023-10-07T09:44:54.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.747+0000 7fd24c3ff700 1 -- start start 2023-10-07T09:44:54.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.748+0000 7fd24c3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 0x7fd2440b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.748+0000 7fd24c3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440b0980 con 0x7fd2440b0540 2023-10-07T09:44:54.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.748+0000 7fd24c3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244154090 con 0x7fd244153cc0 2023-10-07T09:44:54.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.748+0000 7fd24c3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd244159f60 con 0x7fd24415ade0 2023-10-07T09:44:54.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd24a99c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd2440b0540 0x7fd24415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32840/0 (socket says 172.21.15.29:32840) 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd24a99c700 1 -- 172.21.15.29:0/2770433644 learned_addr learned my addr 172.21.15.29:0/2770433644 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd24999a700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 0x7fd2440b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477564467 0 0) 0x7fd2440b0980 con 0x7fd2440b0540 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230003570 con 0x7fd2440b0540 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118107323 0 0) 0x7fd244154090 con 0x7fd244153cc0 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2440b0980 con 0x7fd244153cc0 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793373300 0 0) 0x7fd230003570 con 0x7fd2440b0540 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 msgr2=0x7fd2440b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 0x7fd2440b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 >> v1:172.21.15.144:6789/0 conn(0x7fd244153cc0 legacy=0x7fd2440b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.749+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24415a240 con 0x7fd2440b0540 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fd23c0017f0 con 0x7fd2440b0540 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 >> v1:172.21.15.29:6789/0 conn(0x7fd2440b0540 legacy=0x7fd24415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 0x7fd230003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd230004180 0x7fd230004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd2300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244159f60 con 0x7fd230004d50 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000bba0 con 0x7fd230004180 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000bd60 con 0x7fd24415ade0 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a99c700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd2300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24999a700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd230004180 0x7fd230004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a19b700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 0x7fd230003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a99c700 1 -- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 msgr2=0x7fd230003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a99c700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd24415ade0 0x7fd230003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a99c700 1 -- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd230004180 msgr2=0x7fd230004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a99c700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd230004180 0x7fd230004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.750+0000 7fd24a99c700 1 -- 172.21.15.29:0/2770433644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230003570 con 0x7fd230004d50 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd24a99c700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd2300051c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00a8f0 tx=0x7fd23c002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e48d0bc8404ec4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0070d0 con 0x7fd230004d50 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c007250 con 0x7fd230004d50 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd249199700 1 -- 172.21.15.29:0/2770433644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c007610 con 0x7fd230004d50 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd24c3ff700 1 -- 172.21.15.29:0/2770433644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 msgr2=0x7fd2300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/2770433644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd2300051c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00a8f0 tx=0x7fd23c002190 comp rx=0 tx=0).stop 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd24a99c700 1 -- 172.21.15.29:0/2770433644 reap_dead start 2023-10-07T09:44:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.751+0000 7fd24c3ff700 1 -- 172.21.15.29:0/2770433644 shutdown_connections 2023-10-07T09:44:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.752+0000 7fd24c3ff700 1 -- 172.21.15.29:0/2770433644 >> 172.21.15.29:0/2770433644 conn(0x7fd244067610 msgr2=0x7fd244067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.752+0000 7fd24c3ff700 1 -- 172.21.15.29:0/2770433644 shutdown_connections 2023-10-07T09:44:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.752+0000 7fd24c3ff700 1 -- 172.21.15.29:0/2770433644 wait complete. 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.753+0000 7fd24c3ff700 1 Processor -- start 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.753+0000 7fd24c3ff700 1 -- start start 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24c3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd24415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24c3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd24415b770 0x7fd2441697b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24c3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd244169cf0 0x7fd24416a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24c3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24415c890 con 0x7fd230004d50 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24c3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24416a190 con 0x7fd24415b770 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24c3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24415bc80 con 0x7fd244169cf0 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24a19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd24415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24a19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd24415b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42846/0 (socket says 172.21.15.29:42846) 2023-10-07T09:44:54.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24a19b700 1 -- 172.21.15.29:0/1419375128 learned_addr learned my addr 172.21.15.29:0/1419375128 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24a99c700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd244169cf0 0x7fd24416a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.754+0000 7fd24999a700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd24415b770 0x7fd2441697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24a19b700 1 -- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd244169cf0 msgr2=0x7fd24416a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24a19b700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd244169cf0 0x7fd24416a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24a19b700 1 -- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd24415b770 msgr2=0x7fd2441697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24a19b700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd24415b770 0x7fd2441697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24a19b700 1 -- 172.21.15.29:0/1419375128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24415bf60 con 0x7fd230004d50 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24a19b700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd24415b230 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd24000e7b0 tx=0x7fd240009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240001560 con 0x7fd230004d50 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240010040 con 0x7fd230004d50 2023-10-07T09:44:54.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240001930 con 0x7fd230004d50 2023-10-07T09:44:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24415c240 con 0x7fd230004d50 2023-10-07T09:44:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.755+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd244152e90 con 0x7fd230004d50 2023-10-07T09:44:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.756+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24415c6e0 con 0x7fd230004d50 2023-10-07T09:44:54.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.757+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd240001ca0 con 0x7fd230004d50 2023-10-07T09:44:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.758+0000 7fd23a7fc700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd228066f30 0x7fd2280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.758+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7fd2400843d0 con 0x7fd230004d50 2023-10-07T09:44:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.759+0000 7fd24999a700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd228066f30 0x7fd2280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:54.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.759+0000 7fd24999a700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd228066f30 0x7fd2280693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd234006f00 tx=0x7fd2340091a0 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:54.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.762+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2400514f0 con 0x7fd230004d50 2023-10-07T09:44:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.962+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd24404c3c0 con 0x7fd230004d50 2023-10-07T09:44:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.966+0000 7fd23a7fc700 1 -- 172.21.15.29:0/1419375128 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd240054b10 con 0x7fd230004d50 2023-10-07T09:44:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd228066f30 msgr2=0x7fd2280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd228066f30 0x7fd2280693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd234006f00 tx=0x7fd2340091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 msgr2=0x7fd24415b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd24415b230 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd24000e7b0 tx=0x7fd240009040 comp rx=0 tx=0).stop 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 shutdown_connections 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd228066f30 0x7fd2280693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd244169cf0 0x7fd24416a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd230004d50 0x7fd24415b230 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 --2- 172.21.15.29:0/1419375128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd24415b770 0x7fd2441697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 >> 172.21.15.29:0/1419375128 conn(0x7fd244067610 msgr2=0x7fd2440b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 shutdown_connections 2023-10-07T09:44:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:54.969+0000 7fd24c3ff700 1 -- 172.21.15.29:0/1419375128 wait complete. 2023-10-07T09:44:54.988 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:44:54.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:55.092 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:44:55.199 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:44:55.200 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:44:55.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:44:55.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.394+0000 7f766bbff700 1 Processor -- start 2023-10-07T09:44:55.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.394+0000 7f766bbff700 1 -- start start 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766bbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766bbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76640ce8a0 0x7f76640c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766bbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640cbf20 con 0x7f7664159b20 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766bbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76640cc200 con 0x7f76640ce8a0 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766bbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76640cc4e0 con 0x7f7664159750 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766a19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76640ce8a0 0x7f76640c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766a19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76640ce8a0 0x7f76640c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42856/0 (socket says 172.21.15.29:42856) 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766a19c700 1 -- 172.21.15.29:0/2218293745 learned_addr learned my addr 172.21.15.29:0/2218293745 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766919a700 1 --1- 172.21.15.29:0/2218293745 >> v1:172.21.15.29:6789/0 conn(0x7f7664159b20 0x7f76640ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32852/0 (socket says 172.21.15.29:32852) 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.396+0000 7f766999b700 1 --2- 172.21.15.29:0/2218293745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f766a19c700 1 -- 172.21.15.29:0/2218293745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 msgr2=0x7f76640d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f766a19c700 1 --2- 172.21.15.29:0/2218293745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f766a19c700 1 -- 172.21.15.29:0/2218293745 >> v1:172.21.15.29:6789/0 conn(0x7f7664159b20 legacy=0x7f76640ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f766a19c700 1 -- 172.21.15.29:0/2218293745 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76640cc7c0 con 0x7f76640ce8a0 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f766a19c700 1 --2- 172.21.15.29:0/2218293745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f76640c5460 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f765c009f20 tx=0x7f765c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21375e246f2e47c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f7668999700 1 -- 172.21.15.29:0/2218293745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c00e070 con 0x7f76640ce8a0 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f7668999700 1 -- 172.21.15.29:0/2218293745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c00b9d0 con 0x7f76640ce8a0 2023-10-07T09:44:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.397+0000 7f7668999700 1 -- 172.21.15.29:0/2218293745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0092e0 con 0x7f76640ce8a0 2023-10-07T09:44:55.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 -- 172.21.15.29:0/2218293745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 msgr2=0x7f76640c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 --2- 172.21.15.29:0/2218293745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f76640c5460 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f765c009f20 tx=0x7f765c009840 comp rx=0 tx=0).stop 2023-10-07T09:44:55.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 -- 172.21.15.29:0/2218293745 shutdown_connections 2023-10-07T09:44:55.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 --2- 172.21.15.29:0/2218293745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f76640c5460 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 --2- 172.21.15.29:0/2218293745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 -- 172.21.15.29:0/2218293745 >> 172.21.15.29:0/2218293745 conn(0x7f7664063a10 msgr2=0x7f76640d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:55.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 -- 172.21.15.29:0/2218293745 shutdown_connections 2023-10-07T09:44:55.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.398+0000 7f766bbff700 1 -- 172.21.15.29:0/2218293745 wait complete. 2023-10-07T09:44:55.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.400+0000 7f766bbff700 1 Processor -- start 2023-10-07T09:44:55.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.400+0000 7f766bbff700 1 -- start start 2023-10-07T09:44:55.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.401+0000 7f766bbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f7664154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.401+0000 7f766bbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.401+0000 7f766bbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7664159b20 0x7f7664151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.401+0000 7f766bbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76640c84d0 con 0x7f76640ce8a0 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.401+0000 7f766bbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664154a50 con 0x7f7664159b20 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.401+0000 7f766bbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7664154d30 con 0x7f7664159750 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f7664154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f7664154410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42860/0 (socket says 172.21.15.29:42860) 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 -- 172.21.15.29:0/1293415095 learned_addr learned my addr 172.21.15.29:0/1293415095 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766a19c700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7664159b20 0x7f7664151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766919a700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 -- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 msgr2=0x7f76640b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 -- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7664159b20 msgr2=0x7f7664151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7664159b20 0x7f7664151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 -- 172.21.15.29:0/1293415095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c009ae0 con 0x7f76640ce8a0 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766919a700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766a19c700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7664159b20 0x7f7664151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.402+0000 7f766999b700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f7664154410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f766000ec80 tx=0x7f7660009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.403+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766000c9d0 con 0x7f76640ce8a0 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.403+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766000cb30 con 0x7f76640ce8a0 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.403+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660013470 con 0x7f76640ce8a0 2023-10-07T09:44:55.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.403+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664155010 con 0x7f76640ce8a0 2023-10-07T09:44:55.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.403+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76640b61a0 con 0x7f76640ce8a0 2023-10-07T09:44:55.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.404+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76640c8770 con 0x7f76640ce8a0 2023-10-07T09:44:55.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.405+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7660013860 con 0x7f76640ce8a0 2023-10-07T09:44:55.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.405+0000 7f7659ffb700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7648066f30 0x7f76480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.406+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f7660084cb0 con 0x7f76640ce8a0 2023-10-07T09:44:55.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.406+0000 7f766919a700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7648066f30 0x7f76480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.407+0000 7f766919a700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7648066f30 0x7f76480693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7654007fd0 tx=0x7f7654007e10 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:55.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.409+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7660051e70 con 0x7f76640ce8a0 2023-10-07T09:44:55.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.656+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76640cf460 con 0x7f76640ce8a0 2023-10-07T09:44:55.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.656+0000 7f7659ffb700 1 -- 172.21.15.29:0/1293415095 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f7660021de0 con 0x7f76640ce8a0 2023-10-07T09:44:55.658 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:55.658 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/502021114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":502021114},{"type":"v1","addr":"172.21.15.144:6835","nonce":502021114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1367566713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1367566713},{"type":"v1","addr":"172.21.15.29:6834","nonce":1367566713}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/415810855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":415810855},{"type":"v1","addr":"172.21.15.144:6837","nonce":415810855}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2644860977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2644860977},{"type":"v1","addr":"172.21.15.29:6835","nonce":2644860977}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[]} 2023-10-07T09:44:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.658+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7648066f30 msgr2=0x7f76480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.658+0000 7f766bbff700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7648066f30 0x7f76480693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7654007fd0 tx=0x7f7654007e10 comp rx=0 tx=0).stop 2023-10-07T09:44:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.658+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 msgr2=0x7f7664154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f7664154410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f766000ec80 tx=0x7f7660009120 comp rx=0 tx=0).stop 2023-10-07T09:44:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 shutdown_connections 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7648066f30 0x7f76480693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7664159b20 0x7f7664151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76640ce8a0 0x7f7664154410 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 --2- 172.21.15.29:0/1293415095 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7664159750 0x7f76640b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 >> 172.21.15.29:0/1293415095 conn(0x7f7664063a10 msgr2=0x7f7664152d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 shutdown_connections 2023-10-07T09:44:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.659+0000 7f766bbff700 1 -- 172.21.15.29:0/1293415095 wait complete. 2023-10-07T09:44:55.662 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 36 2023-10-07T09:44:55.677 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:44:55.678 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:44:55.678 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:44:55.678 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:55.678 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:44:55.678 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:44:55.678 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:44:55.679 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:55.679 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:44:55.679 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:44:55.679 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:44:55.679 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:55.679 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:44:55.679 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:44:55.680 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:44:55.680 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:44:55.691 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:44:55.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:44:55.692 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:44:55.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:44:55.695 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:44:55.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:44:55.696 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:44:55.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:44:55.736 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:44:55.737 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:55.739 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:44:55.739 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:55.740 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:44:55.740 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:55.744 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:44:55.744 DEBUG:teuthology.parallel:result is None 2023-10-07T09:44:55.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:44:55.884 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:44:55.884 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:44:55.888 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:44:55.888 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:44:55.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.953+0000 7efdc2bff700 1 Processor -- start 2023-10-07T09:44:55.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.953+0000 7efdc2bff700 1 -- start start 2023-10-07T09:44:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.954+0000 7efdc2bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efdbc0ad370 0x7efdbc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.954+0000 7efdc2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.954+0000 7efdc2bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc14d790 con 0x7efdbc1568c0 2023-10-07T09:44:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.954+0000 7efdc2bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdbc0ae5d0 con 0x7efdbc0ad370 2023-10-07T09:44:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.954+0000 7efdc2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc15a850 con 0x7efdbc159ec0 2023-10-07T09:44:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbbfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdc119c700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7efdbc1568c0 0x7efdbc156d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:59856/0 (socket says 172.21.15.29:59856) 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdc119c700 1 -- 172.21.15.29:0/3811333055 learned_addr learned my addr 172.21.15.29:0/3811333055 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdc099b700 1 --2- 172.21.15.29:0/3811333055 >> v2:172.21.15.144:3300/0 conn(0x7efdbc0ad370 0x7efdbc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbb7fe700 1 -- 172.21.15.29:0/3811333055 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18959408 0 0) 0x7efdbc14d790 con 0x7efdbc1568c0 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbb7fe700 1 -- 172.21.15.29:0/3811333055 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdac003570 con 0x7efdbc1568c0 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbbfff700 1 -- 172.21.15.29:0/3811333055 >> v2:172.21.15.144:3300/0 conn(0x7efdbc0ad370 msgr2=0x7efdbc0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbbfff700 1 --2- 172.21.15.29:0/3811333055 >> v2:172.21.15.144:3300/0 conn(0x7efdbc0ad370 0x7efdbc0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbbfff700 1 -- 172.21.15.29:0/3811333055 >> v1:172.21.15.144:6789/0 conn(0x7efdbc1568c0 legacy=0x7efdbc156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.955+0000 7efdbbfff700 1 -- 172.21.15.29:0/3811333055 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc0ad740 con 0x7efdbc159ec0 2023-10-07T09:44:55.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdbbfff700 1 --2- 172.21.15.29:0/3811333055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc15a310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efda80098e0 tx=0x7efda80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a160afb520048cc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:55.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdbb7fe700 1 -- 172.21.15.29:0/3811333055 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8011630 con 0x7efdbc159ec0 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdbb7fe700 1 -- 172.21.15.29:0/3811333055 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda8011790 con 0x7efdbc159ec0 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdbb7fe700 1 -- 172.21.15.29:0/3811333055 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8011b30 con 0x7efdbc159ec0 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdc2bff700 1 -- 172.21.15.29:0/3811333055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 msgr2=0x7efdbc15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3811333055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc15a310 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efda80098e0 tx=0x7efda80091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdc2bff700 1 -- 172.21.15.29:0/3811333055 shutdown_connections 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3811333055 >> v2:172.21.15.144:3300/0 conn(0x7efdbc0ad370 0x7efdbc0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3811333055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc15a310 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.956+0000 7efdc2bff700 1 -- 172.21.15.29:0/3811333055 >> 172.21.15.29:0/3811333055 conn(0x7efdbc0680a0 msgr2=0x7efdbc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.957+0000 7efdc2bff700 1 -- 172.21.15.29:0/3811333055 shutdown_connections 2023-10-07T09:44:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.957+0000 7efdc2bff700 1 -- 172.21.15.29:0/3811333055 wait complete. 2023-10-07T09:44:55.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.958+0000 7efdc2bff700 1 Processor -- start 2023-10-07T09:44:55.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.958+0000 7efdc2bff700 1 -- start start 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc2bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 0x7efdbc168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc2bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdbc1568c0 0x7efdbc158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc2bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc14eec0 con 0x7efdbc0ad370 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc2bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc14e340 con 0x7efdbc1568c0 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc14e620 con 0x7efdbc159ec0 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 0x7efdbc168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 0x7efdbc168fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42898/0 (socket says 172.21.15.29:42898) 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.959+0000 7efdc099b700 1 -- 172.21.15.29:0/3659734659 learned_addr learned my addr 172.21.15.29:0/3659734659 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdbbfff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdbc1568c0 0x7efdbc158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc119c700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc099b700 1 -- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 msgr2=0x7efdbc1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc099b700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc1590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc099b700 1 -- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdbc1568c0 msgr2=0x7efdbc158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc099b700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdbc1568c0 0x7efdbc158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc099b700 1 -- 172.21.15.29:0/3659734659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda8009420 con 0x7efdbc0ad370 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdbbfff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdbc1568c0 0x7efdbc158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc099b700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 0x7efdbc168fc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7efdb000ec80 tx=0x7efdb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000c9d0 con 0x7efdbc0ad370 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb000cb30 con 0x7efdbc0ad370 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0013470 con 0x7efdbc0ad370 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.960+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc14e900 con 0x7efdbc0ad370 2023-10-07T09:44:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.961+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc169500 con 0x7efdbc0ad370 2023-10-07T09:44:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.962+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdbc169a10 con 0x7efdbc0ad370 2023-10-07T09:44:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.962+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efdb0013820 con 0x7efdbc0ad370 2023-10-07T09:44:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.963+0000 7efdb8ff9700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efda0066f30 0x7efda00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.963+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5874+0+0 (secure 0 0 0) 0x7efdb0084b00 con 0x7efdbc0ad370 2023-10-07T09:44:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.964+0000 7efdbbfff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efda0066f30 0x7efda00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.964+0000 7efdbbfff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efda0066f30 0x7efda00693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efda80091e0 tx=0x7efda8017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:55.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:55.967+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efdb0086e60 con 0x7efdbc0ad370 2023-10-07T09:44:56.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.183+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efdbc04c3c0 con 0x7efdbc0ad370 2023-10-07T09:44:56.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.538+0000 7efdb8ff9700 1 -- 172.21.15.29:0/3659734659 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v84) v1 ==== 97+0+0 (secure 0 0 0) 0x7efdb0017080 con 0x7efdbc0ad370 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efda0066f30 msgr2=0x7efda00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efda0066f30 0x7efda00693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efda80091e0 tx=0x7efda8017000 comp rx=0 tx=0).stop 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 msgr2=0x7efdbc168fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 0x7efdbc168fc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7efdb000ec80 tx=0x7efdb00091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 shutdown_connections 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efda0066f30 0x7efda00693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdbc159ec0 0x7efdbc1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdbc0ad370 0x7efdbc168fc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 --2- 172.21.15.29:0/3659734659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdbc1568c0 0x7efdbc158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 >> 172.21.15.29:0/3659734659 conn(0x7efdbc0680a0 msgr2=0x7efdbc1560e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:56.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.541+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 shutdown_connections 2023-10-07T09:44:56.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.542+0000 7efdc2bff700 1 -- 172.21.15.29:0/3659734659 wait complete. 2023-10-07T09:44:56.544 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:44:56.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:44:56.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.759+0000 7faf3a9ff700 1 Processor -- start 2023-10-07T09:44:56.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.759+0000 7faf3a9ff700 1 -- start start 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.760+0000 7faf3a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34140f80 0x7faf3415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.760+0000 7faf3a9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf34067610 0x7faf340c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.760+0000 7faf3a9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf34068150 con 0x7faf34140bb0 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.760+0000 7faf3a9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3406ab10 con 0x7faf34067610 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.760+0000 7faf3a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf34067b90 con 0x7faf34140f80 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf38f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf34067610 0x7faf340c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf33fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faf34140bb0 0x7faf340affa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32868/0 (socket says 172.21.15.29:32868) 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf33fff700 1 -- 172.21.15.29:0/2385605388 learned_addr learned my addr 172.21.15.29:0/2385605388 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:56.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf337fe700 1 --2- 172.21.15.29:0/2385605388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34140f80 0x7faf3415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf32ffd700 1 -- 172.21.15.29:0/2385605388 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463201920 0 0) 0x7faf34068150 con 0x7faf34140bb0 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf32ffd700 1 -- 172.21.15.29:0/2385605388 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c003570 con 0x7faf34140bb0 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf38f9c700 1 -- 172.21.15.29:0/2385605388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34140f80 msgr2=0x7faf3415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf38f9c700 1 --2- 172.21.15.29:0/2385605388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34140f80 0x7faf3415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf38f9c700 1 -- 172.21.15.29:0/2385605388 >> v1:172.21.15.29:6789/0 conn(0x7faf34140bb0 legacy=0x7faf340affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.761+0000 7faf38f9c700 1 -- 172.21.15.29:0/2385605388 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf34067e70 con 0x7faf34067610 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf38f9c700 1 --2- 172.21.15.29:0/2385605388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34067610 0x7faf340c7820 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009e60 tx=0x7faf2c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3f88822d3bafa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf32ffd700 1 -- 172.21.15.29:0/2385605388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c00e070 con 0x7faf34067610 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf32ffd700 1 -- 172.21.15.29:0/2385605388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c00bab0 con 0x7faf34067610 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf32ffd700 1 -- 172.21.15.29:0/2385605388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c00be70 con 0x7faf34067610 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 -- 172.21.15.29:0/2385605388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34067610 msgr2=0x7faf340c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/2385605388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34067610 0x7faf340c7820 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009e60 tx=0x7faf2c009780 comp rx=0 tx=0).stop 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 -- 172.21.15.29:0/2385605388 shutdown_connections 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/2385605388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34067610 0x7faf340c7820 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/2385605388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34140f80 0x7faf3415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 -- 172.21.15.29:0/2385605388 >> 172.21.15.29:0/2385605388 conn(0x7faf340691f0 msgr2=0x7faf340cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.762+0000 7faf3a9ff700 1 -- 172.21.15.29:0/2385605388 shutdown_connections 2023-10-07T09:44:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.763+0000 7faf3a9ff700 1 -- 172.21.15.29:0/2385605388 wait complete. 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.764+0000 7faf3a9ff700 1 Processor -- start 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.764+0000 7faf3a9ff700 1 -- start start 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf3a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34067610 0x7faf3413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf3a9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 0x7faf3406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf3a9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf34140f80 0x7faf3406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf3a9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf34167380 con 0x7faf34140bb0 2023-10-07T09:44:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf3a9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340c4a90 con 0x7faf34140f80 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf3a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf34157f70 con 0x7faf34067610 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 0x7faf3406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 0x7faf3406e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42914/0 (socket says 172.21.15.29:42914) 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.765+0000 7faf337fe700 1 -- 172.21.15.29:0/1113016534 learned_addr learned my addr 172.21.15.29:0/1113016534 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf337fe700 1 -- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34067610 msgr2=0x7faf3413fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf337fe700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34067610 0x7faf3413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf337fe700 1 -- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf34140f80 msgr2=0x7faf3406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf337fe700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf34140f80 0x7faf3406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf337fe700 1 -- 172.21.15.29:0/1113016534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c009a20 con 0x7faf34140bb0 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf337fe700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 0x7faf3406e800 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7faf2800ec80 tx=0x7faf280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2800c9d0 con 0x7faf34140bb0 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2800cb30 con 0x7faf34140bb0 2023-10-07T09:44:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28013470 con 0x7faf34140bb0 2023-10-07T09:44:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf34158250 con 0x7faf34140bb0 2023-10-07T09:44:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.766+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf340c4720 con 0x7faf34140bb0 2023-10-07T09:44:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.767+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3404c3c0 con 0x7faf34140bb0 2023-10-07T09:44:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.768+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faf28013820 con 0x7faf34140bb0 2023-10-07T09:44:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.770+0000 7faf1bfff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf20066f30 0x7faf200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.770+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5229+0+0 (secure 0 0 0) 0x7faf28084760 con 0x7faf34140bb0 2023-10-07T09:44:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.770+0000 7faf33fff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf20066f30 0x7faf200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.771+0000 7faf33fff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf20066f30 0x7faf200693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faf240058d0 tx=0x7faf240079d0 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:56.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:56.774+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf28051b10 con 0x7faf34140bb0 2023-10-07T09:44:57.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.057+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faf340c4c00 con 0x7faf34140bb0 2023-10-07T09:44:57.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.058+0000 7faf1bfff700 1 -- 172.21.15.29:0/1113016534 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7faf28055130 con 0x7faf34140bb0 2023-10-07T09:44:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:57.059 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:44:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:44:57.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:44:57.065 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:44:57.066 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:44:57.068 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:44:57.069 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:44:57.070 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T09:44:57.071 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:44:57.072 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:44:57.073 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:44:57.074 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:44:57.075 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:44:57.076 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:44:57.077 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.078 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:57.078 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:44:57.078 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:44:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf20066f30 msgr2=0x7faf200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf20066f30 0x7faf200693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faf240058d0 tx=0x7faf240079d0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 msgr2=0x7faf3406e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 0x7faf3406e800 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7faf2800ec80 tx=0x7faf280091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 shutdown_connections 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf20066f30 0x7faf200693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf34140f80 0x7faf3406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf34067610 0x7faf3413fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 --2- 172.21.15.29:0/1113016534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf34140bb0 0x7faf3406e800 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 >> 172.21.15.29:0/1113016534 conn(0x7faf340691f0 msgr2=0x7faf340b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 shutdown_connections 2023-10-07T09:44:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.060+0000 7faf3a9ff700 1 -- 172.21.15.29:0/1113016534 wait complete. 2023-10-07T09:44:57.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:44:57.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.276+0000 7ffb9efff700 1 Processor -- start 2023-10-07T09:44:57.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.276+0000 7ffb9efff700 1 -- start start 2023-10-07T09:44:57.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.277+0000 7ffb9efff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb98159b20 0x7ffb980d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.277+0000 7ffb9efff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffb980ce8a0 0x7ffb980c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.277+0000 7ffb9efff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb980cbf20 con 0x7ffb98159750 2023-10-07T09:44:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.277+0000 7ffb9efff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb980cc200 con 0x7ffb980ce8a0 2023-10-07T09:44:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.277+0000 7ffb9efff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb980cc4e0 con 0x7ffb98159b20 2023-10-07T09:44:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb9cd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffb98159750 0x7ffb980ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32872/0 (socket says 172.21.15.29:32872) 2023-10-07T09:44:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb9cd9b700 1 -- 172.21.15.29:0/431502793 learned_addr learned my addr 172.21.15.29:0/431502793 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb9d59c700 1 --2- 172.21.15.29:0/431502793 >> v2:172.21.15.144:3300/0 conn(0x7ffb980ce8a0 0x7ffb980c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb97fff700 1 --2- 172.21.15.29:0/431502793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb98159b20 0x7ffb980d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb977fe700 1 -- 172.21.15.29:0/431502793 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156701695 0 0) 0x7ffb980cbf20 con 0x7ffb98159750 2023-10-07T09:44:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb977fe700 1 -- 172.21.15.29:0/431502793 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb88003570 con 0x7ffb98159750 2023-10-07T09:44:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb97fff700 1 -- 172.21.15.29:0/431502793 >> v2:172.21.15.144:3300/0 conn(0x7ffb980ce8a0 msgr2=0x7ffb980c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb97fff700 1 --2- 172.21.15.29:0/431502793 >> v2:172.21.15.144:3300/0 conn(0x7ffb980ce8a0 0x7ffb980c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb97fff700 1 -- 172.21.15.29:0/431502793 >> v1:172.21.15.29:6789/0 conn(0x7ffb98159750 legacy=0x7ffb980ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb97fff700 1 -- 172.21.15.29:0/431502793 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb980cc7c0 con 0x7ffb98159b20 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.278+0000 7ffb9d59c700 1 --2- 172.21.15.29:0/431502793 >> v2:172.21.15.144:3300/0 conn(0x7ffb980ce8a0 0x7ffb980c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.279+0000 7ffb97fff700 1 --2- 172.21.15.29:0/431502793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb98159b20 0x7ffb980d10a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ffb840098e0 tx=0x7ffb840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b37560f8f0a22bb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.279+0000 7ffb977fe700 1 -- 172.21.15.29:0/431502793 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84011630 con 0x7ffb98159b20 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.279+0000 7ffb977fe700 1 -- 172.21.15.29:0/431502793 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb84011790 con 0x7ffb98159b20 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.279+0000 7ffb977fe700 1 -- 172.21.15.29:0/431502793 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84011b30 con 0x7ffb98159b20 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 -- 172.21.15.29:0/431502793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb98159b20 msgr2=0x7ffb980d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 --2- 172.21.15.29:0/431502793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb98159b20 0x7ffb980d10a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ffb840098e0 tx=0x7ffb840091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 -- 172.21.15.29:0/431502793 shutdown_connections 2023-10-07T09:44:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 --2- 172.21.15.29:0/431502793 >> v2:172.21.15.144:3300/0 conn(0x7ffb980ce8a0 0x7ffb980c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 --2- 172.21.15.29:0/431502793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb98159b20 0x7ffb980d10a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 -- 172.21.15.29:0/431502793 >> 172.21.15.29:0/431502793 conn(0x7ffb98063a10 msgr2=0x7ffb980d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 -- 172.21.15.29:0/431502793 shutdown_connections 2023-10-07T09:44:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.280+0000 7ffb9efff700 1 -- 172.21.15.29:0/431502793 wait complete. 2023-10-07T09:44:57.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.281+0000 7ffb9efff700 1 Processor -- start 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.282+0000 7ffb9efff700 1 -- start start 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.282+0000 7ffb9efff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb980ce8a0 0x7ffb98145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.282+0000 7ffb9efff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb98159750 0x7ffb980b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9efff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 0x7ffb9815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9efff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb981455d0 con 0x7ffb98159b20 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9efff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9815a320 con 0x7ffb98159750 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9efff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb980b4670 con 0x7ffb980ce8a0 2023-10-07T09:44:57.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 0x7ffb9815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 0x7ffb9815c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42928/0 (socket says 172.21.15.29:42928) 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 -- 172.21.15.29:0/2956587857 learned_addr learned my addr 172.21.15.29:0/2956587857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9cd9b700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb980ce8a0 0x7ffb98145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb97fff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb98159750 0x7ffb980b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 -- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb980ce8a0 msgr2=0x7ffb98145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb980ce8a0 0x7ffb98145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 -- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb98159750 msgr2=0x7ffb980b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb98159750 0x7ffb980b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.283+0000 7ffb9d59c700 1 -- 172.21.15.29:0/2956587857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb90009a20 con 0x7ffb98159b20 2023-10-07T09:44:57.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.284+0000 7ffb9d59c700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 0x7ffb9815c1c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009f20 tx=0x7ffb9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:57.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.284+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000cb70 con 0x7ffb98159b20 2023-10-07T09:44:57.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.284+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9000f040 con 0x7ffb98159b20 2023-10-07T09:44:57.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.284+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90014480 con 0x7ffb98159b20 2023-10-07T09:44:57.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.284+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb84009420 con 0x7ffb98159b20 2023-10-07T09:44:57.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.284+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb980b4c30 con 0x7ffb98159b20 2023-10-07T09:44:57.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.285+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9815a490 con 0x7ffb98159b20 2023-10-07T09:44:57.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.286+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffb90003020 con 0x7ffb98159b20 2023-10-07T09:44:57.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.287+0000 7ffb94ff9700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffb7c066f30 0x7ffb7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.287+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5229+0+0 (secure 0 0 0) 0x7ffb90084c10 con 0x7ffb98159b20 2023-10-07T09:44:57.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.287+0000 7ffb9cd9b700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffb7c066f30 0x7ffb7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.288+0000 7ffb9cd9b700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffb7c066f30 0x7ffb7c0693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c001360 tx=0x7ffb8c009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:57.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.291+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb90051fc0 con 0x7ffb98159b20 2023-10-07T09:44:57.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.493+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb9804c3c0 con 0x7ffb98159b20 2023-10-07T09:44:57.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.494+0000 7ffb94ff9700 1 -- 172.21.15.29:0/2956587857 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+12801 (secure 0 0 0) 0x7ffb900555e0 con 0x7ffb98159b20 2023-10-07T09:44:57.495 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:57.495 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:44:56.544953+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:44:57.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffb7c066f30 msgr2=0x7ffb7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffb7c066f30 0x7ffb7c0693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c001360 tx=0x7ffb8c009040 comp rx=0 tx=0).stop 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 msgr2=0x7ffb9815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 0x7ffb9815c1c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009f20 tx=0x7ffb9000b040 comp rx=0 tx=0).stop 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 shutdown_connections 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffb7c066f30 0x7ffb7c0693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb98159b20 0x7ffb9815c1c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb980ce8a0 0x7ffb98145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 --2- 172.21.15.29:0/2956587857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb98159750 0x7ffb980b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 >> 172.21.15.29:0/2956587857 conn(0x7ffb98063a10 msgr2=0x7ffb980c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 shutdown_connections 2023-10-07T09:44:57.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.496+0000 7ffb9efff700 1 -- 172.21.15.29:0/2956587857 wait complete. 2023-10-07T09:44:57.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:44:57.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:44:57.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.715+0000 7f25057ff700 1 Processor -- start 2023-10-07T09:44:57.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.715+0000 7f25057ff700 1 -- start start 2023-10-07T09:44:57.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f25057ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 0x7f25000d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f25057ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000cbf20 con 0x7f2500159b20 2023-10-07T09:44:57.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f25057ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000cc200 con 0x7f25000ce8a0 2023-10-07T09:44:57.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f25057ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25000cc4e0 con 0x7f2500159750 2023-10-07T09:44:57.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f24fe7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2500159b20 0x7f25000ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32882/0 (socket says 172.21.15.29:32882) 2023-10-07T09:44:57.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f24fe7fc700 1 -- 172.21.15.29:0/896509916 learned_addr learned my addr 172.21.15.29:0/896509916 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f24feffd700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 0x7f25000d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.717+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854356028 0 0) 0x7f25000cbf20 con 0x7f2500159b20 2023-10-07T09:44:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e8003570 con 0x7f2500159b20 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319411996 0 0) 0x7f25000cc200 con 0x7f25000ce8a0 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25000cbf20 con 0x7f25000ce8a0 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2883939139 0 0) 0x7f24e8003570 con 0x7f2500159b20 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 msgr2=0x7f25000d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 0x7f25000d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 >> v1:172.21.15.144:6789/0 conn(0x7f25000ce8a0 legacy=0x7f25000c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000cc7c0 con 0x7f2500159b20 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f24f4001470 con 0x7f2500159b20 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 >> v1:172.21.15.29:6789/0 conn(0x7f2500159b20 legacy=0x7f25000ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 0x7f24e8003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24e8004180 0x7f24e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f24e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000cc4e0 con 0x7f24e8004d50 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e800bba0 con 0x7f24e8004180 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e800bd60 con 0x7f2500159750 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24ff7fe700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f24e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24feffd700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 0x7f24e8003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.718+0000 7f24fe7fc700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24e8004180 0x7f24e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24ff7fe700 1 -- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 msgr2=0x7f24e8003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24ff7fe700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2500159750 0x7f24e8003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24ff7fe700 1 -- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24e8004180 msgr2=0x7f24e8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24ff7fe700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24e8004180 0x7f24e8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24ff7fe700 1 -- 172.21.15.29:0/896509916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e8003570 con 0x7f24e8004d50 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24ff7fe700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f24e80051c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f24f8002120 tx=0x7f24f80017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288f3838032b9517 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8001ab0 con 0x7f24e8004d50 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f800c980 con 0x7f24e8004d50 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.719+0000 7f24fdffb700 1 -- 172.21.15.29:0/896509916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f800cd20 con 0x7f24e8004d50 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f25057ff700 1 -- 172.21.15.29:0/896509916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 msgr2=0x7f24e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f25057ff700 1 --2- 172.21.15.29:0/896509916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f24e80051c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f24f8002120 tx=0x7f24f80017a0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f24ff7fe700 1 -- 172.21.15.29:0/896509916 reap_dead start 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f25057ff700 1 -- 172.21.15.29:0/896509916 shutdown_connections 2023-10-07T09:44:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f25057ff700 1 -- 172.21.15.29:0/896509916 >> 172.21.15.29:0/896509916 conn(0x7f2500063a10 msgr2=0x7f25000d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:57.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f25057ff700 1 -- 172.21.15.29:0/896509916 shutdown_connections 2023-10-07T09:44:57.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.720+0000 7f25057ff700 1 -- 172.21.15.29:0/896509916 wait complete. 2023-10-07T09:44:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.721+0000 7f25057ff700 1 Processor -- start 2023-10-07T09:44:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.721+0000 7f25057ff700 1 -- start start 2023-10-07T09:44:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.722+0000 7f25057ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f2500141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.722+0000 7f25057ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250009c6d0 0x7f250009cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.722+0000 7f25057ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25000c5360 0x7f25000c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.722+0000 7f25057ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000c5d40 con 0x7f24e8004d50 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.722+0000 7f25057ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2500153520 con 0x7f250009c6d0 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.722+0000 7f25057ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250009dda0 con 0x7f25000c5360 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f2500141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f2500141960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42950/0 (socket says 172.21.15.29:42950) 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 -- 172.21.15.29:0/3047304065 learned_addr learned my addr 172.21.15.29:0/3047304065 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24fe7fc700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250009c6d0 0x7f250009cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24ff7fe700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25000c5360 0x7f25000c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 -- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25000c5360 msgr2=0x7f25000c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25000c5360 0x7f25000c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 -- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250009c6d0 msgr2=0x7f250009cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250009c6d0 0x7f250009cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 -- 172.21.15.29:0/3047304065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250009d190 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24ff7fe700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25000c5360 0x7f25000c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24feffd700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f2500141960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f24f0011b40 tx=0x7f24f0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0005c10 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.723+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0005d70 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.724+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f000fa90 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.724+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250009d470 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.724+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250009da30 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.725+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250004c3c0 con 0x7f24e8004d50 2023-10-07T09:44:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.726+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f24f0006050 con 0x7f24e8004d50 2023-10-07T09:44:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.727+0000 7f24e6ffd700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24ec066f30 0x7f24ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.727+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f24f0084d90 con 0x7f24e8004d50 2023-10-07T09:44:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.727+0000 7f24fe7fc700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24ec066f30 0x7f24ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.728+0000 7f24fe7fc700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24ec066f30 0x7f24ec0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f25000c65d0 tx=0x7f24f4001df0 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.730+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24f0052200 con 0x7f24e8004d50 2023-10-07T09:44:57.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.936+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2500064040 con 0x7f24e8004d50 2023-10-07T09:44:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.937+0000 7f24e6ffd700 1 -- 172.21.15.29:0/3047304065 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24f00210a0 con 0x7f24e8004d50 2023-10-07T09:44:57.938 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:44:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24ec066f30 msgr2=0x7f24ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24ec066f30 0x7f24ec0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f25000c65d0 tx=0x7f24f4001df0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 msgr2=0x7f2500141960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f2500141960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f24f0011b40 tx=0x7f24f0009120 comp rx=0 tx=0).stop 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 shutdown_connections 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24ec066f30 0x7f24ec0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25000c5360 0x7f25000c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e8004d50 0x7f2500141960 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 --2- 172.21.15.29:0/3047304065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250009c6d0 0x7f250009cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 >> 172.21.15.29:0/3047304065 conn(0x7f2500063a10 msgr2=0x7f2500068440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 shutdown_connections 2023-10-07T09:44:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:57.939+0000 7f25057ff700 1 -- 172.21.15.29:0/3047304065 wait complete. 2023-10-07T09:44:57.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:44:58.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.153+0000 7f61e45ff700 1 Processor -- start 2023-10-07T09:44:58.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.153+0000 7f61e45ff700 1 -- start start 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e45ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61dc0ad370 0x7f61dc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e45ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0b0410 0x7f61dc15a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e45ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc14d790 con 0x7f61dc159ec0 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e45ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc15a880 con 0x7f61dc0ad370 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e45ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0ae5d0 con 0x7f61dc0b0410 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e239b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61dc0ad370 0x7f61dc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e1b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61dc159ec0 0x7f61dc1568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32898/0 (socket says 172.21.15.29:32898) 2023-10-07T09:44:58.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e1b9a700 1 -- 172.21.15.29:0/2390308516 learned_addr learned my addr 172.21.15.29:0/2390308516 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e2b9c700 1 --2- 172.21.15.29:0/2390308516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0b0410 0x7f61dc15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e1399700 1 -- 172.21.15.29:0/2390308516 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326980279 0 0) 0x7f61dc14d790 con 0x7f61dc159ec0 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e239b700 1 -- 172.21.15.29:0/2390308516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0b0410 msgr2=0x7f61dc15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e239b700 1 --2- 172.21.15.29:0/2390308516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0b0410 0x7f61dc15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e239b700 1 -- 172.21.15.29:0/2390308516 >> v1:172.21.15.29:6789/0 conn(0x7f61dc159ec0 legacy=0x7f61dc1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.155+0000 7f61e239b700 1 -- 172.21.15.29:0/2390308516 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc0ad740 con 0x7f61dc0ad370 2023-10-07T09:44:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.156+0000 7f61e239b700 1 --2- 172.21.15.29:0/2390308516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0ad370 0x7f61dc0cc870 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009910 tx=0x7f61d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a619df0dc0a119f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.156+0000 7f61e1399700 1 -- 172.21.15.29:0/2390308516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d800b820 con 0x7f61dc0ad370 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.156+0000 7f61e1399700 1 -- 172.21.15.29:0/2390308516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d800b980 con 0x7f61dc0ad370 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.156+0000 7f61e1399700 1 -- 172.21.15.29:0/2390308516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d800bd20 con 0x7f61dc0ad370 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.156+0000 7f61e45ff700 1 -- 172.21.15.29:0/2390308516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0ad370 msgr2=0x7f61dc0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.156+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2390308516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0ad370 0x7f61dc0cc870 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009910 tx=0x7f61d80091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.157+0000 7f61e45ff700 1 -- 172.21.15.29:0/2390308516 shutdown_connections 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.157+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2390308516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0b0410 0x7f61dc15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.157+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2390308516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0ad370 0x7f61dc0cc870 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.157+0000 7f61e45ff700 1 -- 172.21.15.29:0/2390308516 >> 172.21.15.29:0/2390308516 conn(0x7f61dc0680a0 msgr2=0x7f61dc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.157+0000 7f61e45ff700 1 -- 172.21.15.29:0/2390308516 shutdown_connections 2023-10-07T09:44:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.157+0000 7f61e45ff700 1 -- 172.21.15.29:0/2390308516 wait complete. 2023-10-07T09:44:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.158+0000 7f61e45ff700 1 Processor -- start 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.158+0000 7f61e45ff700 1 -- start start 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.159+0000 7f61e45ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0ad370 0x7f61dc14bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.159+0000 7f61e45ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 0x7f61dc0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.159+0000 7f61e45ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61dc159ec0 0x7f61dc0cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.159+0000 7f61e45ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc16dad0 con 0x7f61dc0b0410 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.159+0000 7f61e45ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0c2a40 con 0x7f61dc159ec0 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.159+0000 7f61e45ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc16d1a0 con 0x7f61dc0ad370 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 0x7f61dc0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 0x7f61dc0cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42968/0 (socket says 172.21.15.29:42968) 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 -- 172.21.15.29:0/2385630501 learned_addr learned my addr 172.21.15.29:0/2385630501 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e239b700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0ad370 0x7f61dc14bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e2b9c700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61dc159ec0 0x7f61dc0cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 -- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0ad370 msgr2=0x7f61dc14bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0ad370 0x7f61dc14bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 -- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61dc159ec0 msgr2=0x7f61dc0cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61dc159ec0 0x7f61dc0cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 -- 172.21.15.29:0/2385630501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d8009420 con 0x7f61dc0b0410 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e2b9c700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61dc159ec0 0x7f61dc0cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.160+0000 7f61e1b9a700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 0x7f61dc0cb530 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00c5b0 tx=0x7f61cc001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.161+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc005b30 con 0x7f61dc0b0410 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.161+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc005c90 con 0x7f61dc0b0410 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.161+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc007300 con 0x7f61dc0b0410 2023-10-07T09:44:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.161+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc16d480 con 0x7f61dc0b0410 2023-10-07T09:44:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.161+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc14c460 con 0x7f61dc0b0410 2023-10-07T09:44:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.162+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc04c3c0 con 0x7f61dc0b0410 2023-10-07T09:44:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.163+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f61cc01b3b0 con 0x7f61dc0b0410 2023-10-07T09:44:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.163+0000 7f61d27fc700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f61c0066f30 0x7f61c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.164+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f61cc083920 con 0x7f61dc0b0410 2023-10-07T09:44:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.164+0000 7f61e239b700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f61c0066f30 0x7f61c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.164+0000 7f61e239b700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f61c0066f30 0x7f61c00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61d80091e0 tx=0x7f61d8017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.167+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61cc050cd0 con 0x7f61dc0b0410 2023-10-07T09:44:58.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.375+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f61dc06a480 con 0x7f61dc0b0410 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.376+0000 7f61d27fc700 1 -- 172.21.15.29:0/2385630501 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) v1 ==== 84+0+65 (secure 0 0 0) 0x7f61cc017020 con 0x7f61dc0b0410 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:44:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f61c0066f30 msgr2=0x7f61c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f61c0066f30 0x7f61c00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61d80091e0 tx=0x7f61d8017000 comp rx=0 tx=0).stop 2023-10-07T09:44:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 msgr2=0x7f61dc0cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 0x7f61dc0cb530 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00c5b0 tx=0x7f61cc001070 comp rx=0 tx=0).stop 2023-10-07T09:44:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 shutdown_connections 2023-10-07T09:44:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f61c0066f30 0x7f61c00693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61dc159ec0 0x7f61dc0cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61dc0ad370 0x7f61dc14bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 --2- 172.21.15.29:0/2385630501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61dc0b0410 0x7f61dc0cb530 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 >> 172.21.15.29:0/2385630501 conn(0x7f61dc0680a0 msgr2=0x7f61dc069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 shutdown_connections 2023-10-07T09:44:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.378+0000 7f61e45ff700 1 -- 172.21.15.29:0/2385630501 wait complete. 2023-10-07T09:44:58.397 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:44:58.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:44:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.596+0000 7efc323ff700 1 Processor -- start 2023-10-07T09:44:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.596+0000 7efc323ff700 1 -- start start 2023-10-07T09:44:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.597+0000 7efc323ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efc2c0cb8c0 0x7efc2c0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.597+0000 7efc323ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c0cc940 con 0x7efc2c0b2a20 2023-10-07T09:44:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.597+0000 7efc323ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c0cf530 con 0x7efc2c0cf0f0 2023-10-07T09:44:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.597+0000 7efc323ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc2c0b2df0 con 0x7efc2c0cb8c0 2023-10-07T09:44:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.597+0000 7efc2bfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc2c0b2a20 0x7efc2c0cb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32904/0 (socket says 172.21.15.29:32904) 2023-10-07T09:44:58.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.597+0000 7efc2bfff700 1 -- 172.21.15.29:0/1824839818 learned_addr learned my addr 172.21.15.29:0/1824839818 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:58.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2b7fe700 1 --2- 172.21.15.29:0/1824839818 >> v2:172.21.15.144:3300/0 conn(0x7efc2c0cb8c0 0x7efc2c0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711599688 0 0) 0x7efc2c0cc940 con 0x7efc2c0b2a20 2023-10-07T09:44:58.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc14003570 con 0x7efc2c0b2a20 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462314529 0 0) 0x7efc2c0cf530 con 0x7efc2c0cf0f0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c0cc940 con 0x7efc2c0cf0f0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2b7fe700 1 -- 172.21.15.29:0/1824839818 >> v1:172.21.15.144:6789/0 conn(0x7efc2c0cf0f0 legacy=0x7efc2c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2b7fe700 1 -- 172.21.15.29:0/1824839818 >> v1:172.21.15.29:6789/0 conn(0x7efc2c0b2a20 legacy=0x7efc2c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 873203265 0 0) 0x7efc14003570 con 0x7efc2c0b2a20 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2b7fe700 1 -- 172.21.15.29:0/1824839818 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c0cbd30 con 0x7efc2c0cb8c0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.598+0000 7efc2b7fe700 1 --2- 172.21.15.29:0/1824839818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c0cebb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efc200098e0 tx=0x7efc200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2752b84ab23a5f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c0015f0 con 0x7efc2c0cb8c0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc20011420 con 0x7efc2c0cb8c0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc2affd700 1 -- 172.21.15.29:0/1824839818 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc200117c0 con 0x7efc2c0cb8c0 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc323ff700 1 -- 172.21.15.29:0/1824839818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 msgr2=0x7efc2c0cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc323ff700 1 --2- 172.21.15.29:0/1824839818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c0cebb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efc200098e0 tx=0x7efc200091a0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc323ff700 1 -- 172.21.15.29:0/1824839818 shutdown_connections 2023-10-07T09:44:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc323ff700 1 --2- 172.21.15.29:0/1824839818 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c0cebb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc323ff700 1 -- 172.21.15.29:0/1824839818 >> 172.21.15.29:0/1824839818 conn(0x7efc2c099820 msgr2=0x7efc2c063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.599+0000 7efc323ff700 1 -- 172.21.15.29:0/1824839818 shutdown_connections 2023-10-07T09:44:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.600+0000 7efc323ff700 1 -- 172.21.15.29:0/1824839818 wait complete. 2023-10-07T09:44:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.602+0000 7efc323ff700 1 Processor -- start 2023-10-07T09:44:58.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.602+0000 7efc323ff700 1 -- start start 2023-10-07T09:44:58.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc323ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 0x7efc2c14e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc323ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc323ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc2c0cf0f0 0x7efc2c15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc323ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c16d530 con 0x7efc2c0b2a20 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc323ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c16d810 con 0x7efc2c0cb8c0 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc323ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c16daf0 con 0x7efc2c0cf0f0 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc2bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 0x7efc2c14e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc2bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 0x7efc2c14e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42978/0 (socket says 172.21.15.29:42978) 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc2bfff700 1 -- 172.21.15.29:0/264151550 learned_addr learned my addr 172.21.15.29:0/264151550 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc3099c700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc2c0cf0f0 0x7efc2c15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.603+0000 7efc2b7fe700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc2bfff700 1 -- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc2c0cf0f0 msgr2=0x7efc2c15c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc2bfff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc2c0cf0f0 0x7efc2c15c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc2bfff700 1 -- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 msgr2=0x7efc2c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc2bfff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc2bfff700 1 -- 172.21.15.29:0/264151550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc20009420 con 0x7efc2c0b2a20 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc2bfff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 0x7efc2c14e030 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00cb90 tx=0x7efc1c002bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c007300 con 0x7efc2c0b2a20 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1c0074a0 con 0x7efc2c0b2a20 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c005a40 con 0x7efc2c0b2a20 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2c16ddd0 con 0x7efc2c0b2a20 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.604+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c14e570 con 0x7efc2c0b2a20 2023-10-07T09:44:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.605+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2c14e760 con 0x7efc2c0b2a20 2023-10-07T09:44:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.606+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc1c00a590 con 0x7efc2c0b2a20 2023-10-07T09:44:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.607+0000 7efc13fff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efc18066f30 0x7efc180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.607+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5229+0+0 (secure 0 0 0) 0x7efc1c084290 con 0x7efc2c0b2a20 2023-10-07T09:44:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.607+0000 7efc2b7fe700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efc18066f30 0x7efc180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.608+0000 7efc2b7fe700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efc18066f30 0x7efc180693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc200098e0 tx=0x7efc20016000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.611+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc2c14e760 con 0x7efc2c0b2a20 2023-10-07T09:44:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:58.813+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efc2c14e760 con 0x7efc2c0b2a20 2023-10-07T09:44:59.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.553+0000 7efc13fff700 1 -- 172.21.15.29:0/264151550 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v87) v1 ==== 138+0+0 (secure 0 0 0) 0x7efc2c14e760 con 0x7efc2c0b2a20 2023-10-07T09:44:59.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efc18066f30 msgr2=0x7efc180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efc18066f30 0x7efc180693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc200098e0 tx=0x7efc20016000 comp rx=0 tx=0).stop 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 msgr2=0x7efc2c14e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 0x7efc2c14e030 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00cb90 tx=0x7efc1c002bb0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 shutdown_connections 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efc18066f30 0x7efc180693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc2c0cf0f0 0x7efc2c15c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc2c0b2a20 0x7efc2c14e030 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 --2- 172.21.15.29:0/264151550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc2c0cb8c0 0x7efc2c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 >> 172.21.15.29:0/264151550 conn(0x7efc2c099820 msgr2=0x7efc2c14ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 shutdown_connections 2023-10-07T09:44:59.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.557+0000 7efc323ff700 1 -- 172.21.15.29:0/264151550 wait complete. 2023-10-07T09:44:59.560 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:44:59.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:44:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.789+0000 7f3c865ff700 1 Processor -- start 2023-10-07T09:44:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.789+0000 7f3c865ff700 1 -- start start 2023-10-07T09:44:59.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.790+0000 7f3c865ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3c800d0010 0x7f3c800b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:59.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.790+0000 7f3c865ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800c62d0 con 0x7f3c800cfbb0 2023-10-07T09:44:59.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.790+0000 7f3c865ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80159e10 con 0x7f3c800cb800 2023-10-07T09:44:59.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.790+0000 7f3c865ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c800d0410 con 0x7f3c800d0010 2023-10-07T09:44:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.790+0000 7f3c7f7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3c800cfbb0 0x7f3c800c6760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32908/0 (socket says 172.21.15.29:32908) 2023-10-07T09:44:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.790+0000 7f3c7f7fe700 1 -- 172.21.15.29:0/846638631 learned_addr learned my addr 172.21.15.29:0/846638631 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:44:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c84b9c700 1 --2- 172.21.15.29:0/846638631 >> v2:172.21.15.144:3300/0 conn(0x7f3c800d0010 0x7f3c800b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172309399 0 0) 0x7f3c800c62d0 con 0x7f3c800cfbb0 2023-10-07T09:44:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c68003570 con 0x7f3c800cfbb0 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895436882 0 0) 0x7f3c80159e10 con 0x7f3c800cb800 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c800c62d0 con 0x7f3c800cb800 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c84b9c700 1 -- 172.21.15.29:0/846638631 >> v1:172.21.15.144:6789/0 conn(0x7f3c800cb800 legacy=0x7f3c800cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c84b9c700 1 -- 172.21.15.29:0/846638631 >> v1:172.21.15.29:6789/0 conn(0x7f3c800cfbb0 legacy=0x7f3c800c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866577698 0 0) 0x7f3c68003570 con 0x7f3c800cfbb0 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c84b9c700 1 -- 172.21.15.29:0/846638631 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c800c56c0 con 0x7f3c800d0010 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.791+0000 7f3c84b9c700 1 --2- 172.21.15.29:0/846638631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800d0010 0x7f3c800b3340 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e60 tx=0x7f3c78009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bc591e080c360b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800e070 con 0x7f3c800d0010 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7800bb40 con 0x7f3c800d0010 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c7effd700 1 -- 172.21.15.29:0/846638631 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78012430 con 0x7f3c800d0010 2023-10-07T09:44:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 -- 172.21.15.29:0/846638631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800d0010 msgr2=0x7f3c800b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 --2- 172.21.15.29:0/846638631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800d0010 0x7f3c800b3340 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e60 tx=0x7f3c78009780 comp rx=0 tx=0).stop 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 -- 172.21.15.29:0/846638631 shutdown_connections 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 --2- 172.21.15.29:0/846638631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800d0010 0x7f3c800b3340 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 -- 172.21.15.29:0/846638631 >> 172.21.15.29:0/846638631 conn(0x7f3c80063a10 msgr2=0x7f3c800d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 -- 172.21.15.29:0/846638631 shutdown_connections 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.792+0000 7f3c865ff700 1 -- 172.21.15.29:0/846638631 wait complete. 2023-10-07T09:44:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.794+0000 7f3c865ff700 1 Processor -- start 2023-10-07T09:44:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.794+0000 7f3c865ff700 1 -- start start 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c865ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800cb800 0x7f3c800b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c865ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 0x7f3c8014fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c865ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c800d0010 0x7f3c800c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c865ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80150290 con 0x7f3c800cfbb0 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c865ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800c7fc0 con 0x7f3c800cb800 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c865ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800c76d0 con 0x7f3c800d0010 2023-10-07T09:44:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.795+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 0x7f3c8014fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 0x7f3c8014fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42990/0 (socket says 172.21.15.29:42990) 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 -- 172.21.15.29:0/2203071086 learned_addr learned my addr 172.21.15.29:0/2203071086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7ffff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800cb800 0x7f3c800b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c84b9c700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c800d0010 0x7f3c800c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 -- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c800d0010 msgr2=0x7f3c800c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c800d0010 0x7f3c800c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 -- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800cb800 msgr2=0x7f3c800b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800cb800 0x7f3c800b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 -- 172.21.15.29:0/2203071086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78009a20 con 0x7f3c800cfbb0 2023-10-07T09:44:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c7f7fe700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 0x7f3c8014fd50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c7f0 tx=0x7f3c74007b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.796+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74007360 con 0x7f3c800cfbb0 2023-10-07T09:44:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.797+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c740074c0 con 0x7f3c800cfbb0 2023-10-07T09:44:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.797+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74004830 con 0x7f3c800cfbb0 2023-10-07T09:44:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.797+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c800c9e00 con 0x7f3c800cfbb0 2023-10-07T09:44:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.797+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c800c7c50 con 0x7f3c800cfbb0 2023-10-07T09:44:59.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.798+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c800c8150 con 0x7f3c800cfbb0 2023-10-07T09:44:59.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.800+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3c74004c40 con 0x7f3c800cfbb0 2023-10-07T09:44:59.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.801+0000 7f3c67fff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c6c066f30 0x7f3c6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:44:59.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.801+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5599+0+0 (secure 0 0 0) 0x7f3c74084be0 con 0x7f3c800cfbb0 2023-10-07T09:44:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.802+0000 7f3c7ffff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c6c066f30 0x7f3c6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:44:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.802+0000 7f3c7ffff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c6c066f30 0x7f3c6c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c70003ae0 tx=0x7f3c70007d40 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:44:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:44:59.805+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c74051f20 con 0x7f3c800cfbb0 2023-10-07T09:45:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.007+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3c800ca100 con 0x7f3c800cfbb0 2023-10-07T09:45:00.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.562+0000 7f3c67fff700 1 -- 172.21.15.29:0/2203071086 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v88) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3c74055540 con 0x7f3c800cfbb0 2023-10-07T09:45:00.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.567+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c6c066f30 msgr2=0x7f3c6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:00.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.567+0000 7f3c865ff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c6c066f30 0x7f3c6c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c70003ae0 tx=0x7f3c70007d40 comp rx=0 tx=0).stop 2023-10-07T09:45:00.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.567+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 msgr2=0x7f3c8014fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:00.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.567+0000 7f3c865ff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 0x7f3c8014fd50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c7f0 tx=0x7f3c74007b60 comp rx=0 tx=0).stop 2023-10-07T09:45:00.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.569+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 shutdown_connections 2023-10-07T09:45:00.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.569+0000 7f3c865ff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c6c066f30 0x7f3c6c0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.569+0000 7f3c865ff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c800d0010 0x7f3c800c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.569+0000 7f3c865ff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c800cb800 0x7f3c800b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.569+0000 7f3c865ff700 1 --2- 172.21.15.29:0/2203071086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800cfbb0 0x7f3c8014fd50 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.569+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 >> 172.21.15.29:0/2203071086 conn(0x7f3c80063a10 msgr2=0x7f3c80157490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:00.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.572+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 shutdown_connections 2023-10-07T09:45:00.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.572+0000 7f3c865ff700 1 -- 172.21.15.29:0/2203071086 wait complete. 2023-10-07T09:45:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:45:00.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:45:00.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.805+0000 7f7414fff700 1 Processor -- start 2023-10-07T09:45:00.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.805+0000 7f7414fff700 1 -- start start 2023-10-07T09:45:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.806+0000 7f7414fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7410156180 0x7f74100afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.806+0000 7f7414fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f74100b0520 0x7f74100b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.806+0000 7f7414fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100ad4d0 con 0x7f74100ae570 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.806+0000 7f7414fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74100b0ee0 con 0x7f74100b0520 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f7414fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74100ad7b0 con 0x7f7410156180 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f74100ae570 0x7f74100accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32918/0 (socket says 172.21.15.29:32918) 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740dd9b700 1 -- 172.21.15.29:0/2885875202 learned_addr learned my addr 172.21.15.29:0/2885875202 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740ed9d700 1 --2- 172.21.15.29:0/2885875202 >> v2:172.21.15.144:3300/0 conn(0x7f74100b0520 0x7f74100b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740e59c700 1 --2- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7410156180 0x7f74100afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740d59a700 1 -- 172.21.15.29:0/2885875202 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070655290 0 0) 0x7f74100ad4d0 con 0x7f74100ae570 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740d59a700 1 -- 172.21.15.29:0/2885875202 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f8003570 con 0x7f74100ae570 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740ed9d700 1 -- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7410156180 msgr2=0x7f74100afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740ed9d700 1 --2- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7410156180 0x7f74100afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.807+0000 7f740ed9d700 1 -- 172.21.15.29:0/2885875202 >> v1:172.21.15.29:6789/0 conn(0x7f74100ae570 legacy=0x7f74100accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.808+0000 7f740ed9d700 1 -- 172.21.15.29:0/2885875202 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74100ada90 con 0x7f74100b0520 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.808+0000 7f740ed9d700 1 --2- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74100b0520 0x7f74100b09a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7408009e60 tx=0x7f7408009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea05ea74acb3255c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.808+0000 7f740d59a700 1 -- 172.21.15.29:0/2885875202 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740800e070 con 0x7f74100b0520 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.808+0000 7f740d59a700 1 -- 172.21.15.29:0/2885875202 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740800bab0 con 0x7f74100b0520 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.808+0000 7f740d59a700 1 -- 172.21.15.29:0/2885875202 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740800be70 con 0x7f74100b0520 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 -- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74100b0520 msgr2=0x7f74100b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 --2- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74100b0520 0x7f74100b09a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7408009e60 tx=0x7f7408009780 comp rx=0 tx=0).stop 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 -- 172.21.15.29:0/2885875202 shutdown_connections 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 --2- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74100b0520 0x7f74100b09a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 --2- 172.21.15.29:0/2885875202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7410156180 0x7f74100afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 -- 172.21.15.29:0/2885875202 >> 172.21.15.29:0/2885875202 conn(0x7f7410067610 msgr2=0x7f7410067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 -- 172.21.15.29:0/2885875202 shutdown_connections 2023-10-07T09:45:00.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.809+0000 7f7414fff700 1 -- 172.21.15.29:0/2885875202 wait complete. 2023-10-07T09:45:00.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.810+0000 7f7414fff700 1 Processor -- start 2023-10-07T09:45:00.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.810+0000 7f7414fff700 1 -- start start 2023-10-07T09:45:00.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.811+0000 7f7414fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 0x7f7410152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.811+0000 7f7414fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74100b0520 0x7f74100cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.811+0000 7f7414fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7410156180 0x7f7410169a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.811+0000 7f7414fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7410153100 con 0x7f74100ae570 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.811+0000 7f7414fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74100cc340 con 0x7f7410156180 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.811+0000 7f7414fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741015db20 con 0x7f74100b0520 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 0x7f7410152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 0x7f7410152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42992/0 (socket says 172.21.15.29:42992) 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 -- 172.21.15.29:0/1191962164 learned_addr learned my addr 172.21.15.29:0/1191962164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740ed9d700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7410156180 0x7f7410169a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740dd9b700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74100b0520 0x7f74100cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 -- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74100b0520 msgr2=0x7f74100cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74100b0520 0x7f74100cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 -- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7410156180 msgr2=0x7f7410169a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7410156180 0x7f7410169a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 -- 172.21.15.29:0/1191962164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408009a20 con 0x7f74100ae570 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740dd9b700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74100b0520 0x7f74100cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.812+0000 7f740e59c700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 0x7f7410152bc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f740000e930 tx=0x7f74000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.813+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400013680 con 0x7f74100ae570 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.813+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000137e0 con 0x7f74100ae570 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.813+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741015de00 con 0x7f74100ae570 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.813+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400013b80 con 0x7f74100ae570 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.813+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7410169550 con 0x7f74100ae570 2023-10-07T09:45:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.814+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741004c3c0 con 0x7f74100ae570 2023-10-07T09:45:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.815+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f740000c460 con 0x7f74100ae570 2023-10-07T09:45:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.817+0000 7f73fe7fc700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73f0066f30 0x7f73f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.817+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f7400085060 con 0x7f74100ae570 2023-10-07T09:45:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.817+0000 7f740dd9b700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73f0066f30 0x7f73f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.818+0000 7f740dd9b700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73f0066f30 0x7f73f00693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f741015d580 tx=0x7f7404009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:00.820+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74000521a0 con 0x7f74100ae570 2023-10-07T09:45:01.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.069+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f74100add70 con 0x7f74100ae570 2023-10-07T09:45:01.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.565+0000 7f73fe7fc700 1 -- 172.21.15.29:0/1191962164 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) v1 ==== 174+0+0 (secure 0 0 0) 0x7f74000557c0 con 0x7f74100ae570 2023-10-07T09:45:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.574+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73f0066f30 msgr2=0x7f73f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.574+0000 7f7414fff700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73f0066f30 0x7f73f00693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f741015d580 tx=0x7f7404009040 comp rx=0 tx=0).stop 2023-10-07T09:45:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.574+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 msgr2=0x7f7410152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.574+0000 7f7414fff700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 0x7f7410152bc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f740000e930 tx=0x7f74000091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.575+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 shutdown_connections 2023-10-07T09:45:01.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.575+0000 7f7414fff700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f73f0066f30 0x7f73f00693e0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f741015d580 tx=0x7f7404009040 comp rx=0 tx=0).stop 2023-10-07T09:45:01.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.575+0000 7f7414fff700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7410156180 0x7f7410169a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.575+0000 7f7414fff700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74100ae570 0x7f7410152bc0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.575+0000 7f7414fff700 1 --2- 172.21.15.29:0/1191962164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74100b0520 0x7f74100cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.575+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 >> 172.21.15.29:0/1191962164 conn(0x7f7410067610 msgr2=0x7f74100aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:01.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.579+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 shutdown_connections 2023-10-07T09:45:01.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.580+0000 7f7414fff700 1 -- 172.21.15.29:0/1191962164 wait complete. 2023-10-07T09:45:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 10 and data pool 11 2023-10-07T09:45:01.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.809+0000 7f15fb7ff700 1 Processor -- start 2023-10-07T09:45:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.809+0000 7f15fb7ff700 1 -- start start 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.810+0000 7f15fb7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15f4153cc0 0x7f15f40b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.810+0000 7f15fb7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f4154090 con 0x7f15f415ade0 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.810+0000 7f15fb7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f4159f60 con 0x7f15f40b0710 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.810+0000 7f15fb7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f415a240 con 0x7f15f4153cc0 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15f4153cc0 0x7f15f40b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15f4153cc0 0x7f15f40b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43004/0 (socket says 172.21.15.29:43004) 2023-10-07T09:45:01.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 -- 172.21.15.29:0/2991082211 learned_addr learned my addr 172.21.15.29:0/2991082211 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f8d9a700 1 --1- 172.21.15.29:0/2991082211 >> v1:172.21.15.29:6789/0 conn(0x7f15f415ade0 0x7f15f40b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32934/0 (socket says 172.21.15.29:32934) 2023-10-07T09:45:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 -- 172.21.15.29:0/2991082211 >> v1:172.21.15.144:6789/0 conn(0x7f15f40b0710 legacy=0x7f15f415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 -- 172.21.15.29:0/2991082211 >> v1:172.21.15.29:6789/0 conn(0x7f15f415ade0 legacy=0x7f15f40b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:01.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 -- 172.21.15.29:0/2991082211 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f415a520 con 0x7f15f4153cc0 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.811+0000 7f15f959b700 1 --2- 172.21.15.29:0/2991082211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f4153cc0 0x7f15f40b17c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009990 tx=0x7f15f0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5304da497b46618 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15ebfff700 1 -- 172.21.15.29:0/2991082211 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0005c10 con 0x7f15f4153cc0 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15ebfff700 1 -- 172.21.15.29:0/2991082211 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0005d70 con 0x7f15f4153cc0 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15ebfff700 1 -- 172.21.15.29:0/2991082211 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f000b840 con 0x7f15f4153cc0 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2991082211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f4153cc0 msgr2=0x7f15f40b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2991082211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f4153cc0 0x7f15f40b17c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009990 tx=0x7f15f0009220 comp rx=0 tx=0).stop 2023-10-07T09:45:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2991082211 shutdown_connections 2023-10-07T09:45:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2991082211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f4153cc0 0x7f15f40b17c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.812+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2991082211 >> 172.21.15.29:0/2991082211 conn(0x7f15f4067610 msgr2=0x7f15f4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.813+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2991082211 shutdown_connections 2023-10-07T09:45:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.813+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2991082211 wait complete. 2023-10-07T09:45:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.815+0000 7f15fb7ff700 1 Processor -- start 2023-10-07T09:45:01.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.815+0000 7f15fb7ff700 1 -- start start 2023-10-07T09:45:01.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15fb7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 0x7f15f413fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15fb7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15f4153cc0 0x7f15f4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15fb7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15f415ade0 0x7f15f4156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15fb7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4159300 con 0x7f15f40b0710 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15fb7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4140380 con 0x7f15f4153cc0 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15fb7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f41550f0 con 0x7f15f415ade0 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15f959b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 0x7f15f413fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15f959b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 0x7f15f413fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43018/0 (socket says 172.21.15.29:43018) 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.816+0000 7f15f959b700 1 -- 172.21.15.29:0/2573617712 learned_addr learned my addr 172.21.15.29:0/2573617712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f8d9a700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15f4153cc0 0x7f15f4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f9d9c700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15f415ade0 0x7f15f4156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:01.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f959b700 1 -- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15f415ade0 msgr2=0x7f15f4156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f959b700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15f415ade0 0x7f15f4156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f959b700 1 -- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15f4153cc0 msgr2=0x7f15f4152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f959b700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15f4153cc0 0x7f15f4152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f959b700 1 -- 172.21.15.29:0/2573617712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0009480 con 0x7f15f40b0710 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f8d9a700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15f4153cc0 0x7f15f4152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15f959b700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 0x7f15f413fe40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009260 tx=0x7f15f0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.817+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0011cc0 con 0x7f15f40b0710 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.818+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0011e20 con 0x7f15f40b0710 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.818+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f001d680 con 0x7f15f40b0710 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.818+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f41553d0 con 0x7f15f40b0710 2023-10-07T09:45:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.818+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f4155990 con 0x7f15f40b0710 2023-10-07T09:45:01.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.819+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f404c3c0 con 0x7f15f40b0710 2023-10-07T09:45:01.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.820+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f15f001da70 con 0x7f15f40b0710 2023-10-07T09:45:01.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.820+0000 7f15e97fa700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f15d8066f30 0x7f15d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:01.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.821+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f15f0086960 con 0x7f15f40b0710 2023-10-07T09:45:01.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.821+0000 7f15f8d9a700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f15d8066f30 0x7f15d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:01.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.822+0000 7f15f8d9a700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f15d8066f30 0x7f15d80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f15e4007fd0 tx=0x7f15e4007e10 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:01.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:01.825+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15f0053a50 con 0x7f15f40b0710 2023-10-07T09:45:02.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.028+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15f415a800 con 0x7f15f40b0710 2023-10-07T09:45:02.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.029+0000 7f15e97fa700 1 -- 172.21.15.29:0/2573617712 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16033 (secure 0 0 0) 0x7f15f002a090 con 0x7f15f40b0710 2023-10-07T09:45:02.033 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:02.033 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:01.071009+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:58.854049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:00.048655+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f15d8066f30 msgr2=0x7f15d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f15d8066f30 0x7f15d80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f15e4007fd0 tx=0x7f15e4007e10 comp rx=0 tx=0).stop 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 msgr2=0x7f15f413fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 0x7f15f413fe40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009260 tx=0x7f15f0015000 comp rx=0 tx=0).stop 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 shutdown_connections 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f15d8066f30 0x7f15d80693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f15f415ade0 0x7f15f4156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15f40b0710 0x7f15f413fe40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 --2- 172.21.15.29:0/2573617712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15f4153cc0 0x7f15f4152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 >> 172.21.15.29:0/2573617712 conn(0x7f15f4067610 msgr2=0x7f15f4152e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 shutdown_connections 2023-10-07T09:45:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.031+0000 7f15fb7ff700 1 -- 172.21.15.29:0/2573617712 wait complete. 2023-10-07T09:45:02.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:02.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.246+0000 7ff5e0fff700 1 Processor -- start 2023-10-07T09:45:02.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.246+0000 7ff5e0fff700 1 -- start start 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.247+0000 7ff5e0fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5dc068d30 0x7ff5dc0ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.247+0000 7ff5e0fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc15af00 con 0x7ff5dc159c00 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.247+0000 7ff5e0fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc15a250 con 0x7ff5dc0ad220 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.247+0000 7ff5e0fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc15a050 con 0x7ff5dc068d30 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5dc068d30 0x7ff5dc0ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5dc068d30 0x7ff5dc0ad5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43026/0 (socket says 172.21.15.29:43026) 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 -- 172.21.15.29:0/502628495 learned_addr learned my addr 172.21.15.29:0/502628495 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5d9d9b700 1 --1- 172.21.15.29:0/502628495 >> v1:172.21.15.29:6789/0 conn(0x7ff5dc159c00 0x7ff5dc068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32938/0 (socket says 172.21.15.29:32938) 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 -- 172.21.15.29:0/502628495 >> v1:172.21.15.144:6789/0 conn(0x7ff5dc0ad220 legacy=0x7ff5dc0ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 -- 172.21.15.29:0/502628495 >> v1:172.21.15.29:6789/0 conn(0x7ff5dc159c00 legacy=0x7ff5dc068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 -- 172.21.15.29:0/502628495 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc15a530 con 0x7ff5dc068d30 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.248+0000 7ff5dad9d700 1 --2- 172.21.15.29:0/502628495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc068d30 0x7ff5dc0ad5f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009f20 tx=0x7ff5d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570c8d6095fa9f48 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5d959a700 1 -- 172.21.15.29:0/502628495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400e070 con 0x7ff5dc068d30 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5d959a700 1 -- 172.21.15.29:0/502628495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d400ba80 con 0x7ff5dc068d30 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5d959a700 1 -- 172.21.15.29:0/502628495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40092e0 con 0x7ff5dc068d30 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 -- 172.21.15.29:0/502628495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc068d30 msgr2=0x7ff5dc0ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/502628495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc068d30 0x7ff5dc0ad5f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009f20 tx=0x7ff5d4009840 comp rx=0 tx=0).stop 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 -- 172.21.15.29:0/502628495 shutdown_connections 2023-10-07T09:45:02.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/502628495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc068d30 0x7ff5dc0ad5f0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 -- 172.21.15.29:0/502628495 >> 172.21.15.29:0/502628495 conn(0x7ff5dc06dd00 msgr2=0x7ff5dc06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 -- 172.21.15.29:0/502628495 shutdown_connections 2023-10-07T09:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.249+0000 7ff5e0fff700 1 -- 172.21.15.29:0/502628495 wait complete. 2023-10-07T09:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.251+0000 7ff5e0fff700 1 Processor -- start 2023-10-07T09:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.252+0000 7ff5e0fff700 1 -- start start 2023-10-07T09:45:02.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5e0fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5dc068d30 0x7ff5dc151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5e0fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 0x7ff5dc15e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5e0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5dc159c00 0x7ff5dc0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5e0fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc152050 con 0x7ff5dc0ad220 2023-10-07T09:45:02.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5e0fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0b18c0 con 0x7ff5dc068d30 2023-10-07T09:45:02.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5e0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0c96b0 con 0x7ff5dc159c00 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5d9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 0x7ff5dc15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5d9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 0x7ff5dc15e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43036/0 (socket says 172.21.15.29:43036) 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5d9d9b700 1 -- 172.21.15.29:0/2905172641 learned_addr learned my addr 172.21.15.29:0/2905172641 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5da59c700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5dc068d30 0x7ff5dc151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.253+0000 7ff5dad9d700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5dc159c00 0x7ff5dc0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5d9d9b700 1 -- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5dc159c00 msgr2=0x7ff5dc0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5d9d9b700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5dc159c00 0x7ff5dc0b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5d9d9b700 1 -- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5dc068d30 msgr2=0x7ff5dc151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5d9d9b700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5dc068d30 0x7ff5dc151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5d9d9b700 1 -- 172.21.15.29:0/2905172641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d4009ae0 con 0x7ff5dc0ad220 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5d9d9b700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 0x7ff5dc15e810 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000c7c0 tx=0x7ff5d0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:02.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0005a70 con 0x7ff5dc0ad220 2023-10-07T09:45:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0005bd0 con 0x7ff5dc0ad220 2023-10-07T09:45:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00070d0 con 0x7ff5dc0ad220 2023-10-07T09:45:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc0c9950 con 0x7ff5dc0ad220 2023-10-07T09:45:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.254+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0b1550 con 0x7ff5dc0ad220 2023-10-07T09:45:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.255+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc0ad5f0 con 0x7ff5dc0ad220 2023-10-07T09:45:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.256+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff5d0007480 con 0x7ff5dc0ad220 2023-10-07T09:45:02.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.257+0000 7ff5ca7fc700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff5bc066f30 0x7ff5bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.258+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6081+0+0 (secure 0 0 0) 0x7ff5d0083b40 con 0x7ff5dc0ad220 2023-10-07T09:45:02.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.258+0000 7ff5da59c700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff5bc066f30 0x7ff5bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.258+0000 7ff5da59c700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff5bc066f30 0x7ff5bc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc003a80 tx=0x7ff5cc007ec0 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:02.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.262+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5d0050b90 con 0x7ff5dc0ad220 2023-10-07T09:45:02.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.464+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5dc04c3c0 con 0x7ff5dc0ad220 2023-10-07T09:45:02.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.465+0000 7ff5ca7fc700 1 -- 172.21.15.29:0/2905172641 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16033 (secure 0 0 0) 0x7ff5d0017020 con 0x7ff5dc0ad220 2023-10-07T09:45:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:02.466 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:01.071009+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:58.854049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:00.048655+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:02.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff5bc066f30 msgr2=0x7ff5bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff5bc066f30 0x7ff5bc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc003a80 tx=0x7ff5cc007ec0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 msgr2=0x7ff5dc15e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 0x7ff5dc15e810 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000c7c0 tx=0x7ff5d0007f70 comp rx=0 tx=0).stop 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 shutdown_connections 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff5bc066f30 0x7ff5bc0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5dc159c00 0x7ff5dc0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5dc068d30 0x7ff5dc151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 --2- 172.21.15.29:0/2905172641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5dc0ad220 0x7ff5dc15e810 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 >> 172.21.15.29:0/2905172641 conn(0x7ff5dc06dd00 msgr2=0x7ff5dc0695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 shutdown_connections 2023-10-07T09:45:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.467+0000 7ff5e0fff700 1 -- 172.21.15.29:0/2905172641 wait complete. 2023-10-07T09:45:02.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:45:02.541 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T09:45:02.540+0000 7ff735df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:45:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.685+0000 7f1de83ff700 1 Processor -- start 2023-10-07T09:45:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.685+0000 7f1de83ff700 1 -- start start 2023-10-07T09:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.686+0000 7f1de83ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1de0159750 0x7f1de00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.686+0000 7f1de83ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1de00ce8a0 0x7f1de00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.686+0000 7f1de83ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de00cbf20 con 0x7f1de0159b20 2023-10-07T09:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.686+0000 7f1de83ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00cc200 con 0x7f1de00ce8a0 2023-10-07T09:45:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.686+0000 7f1de83ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00cc4e0 con 0x7f1de0159750 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1de00ce8a0 0x7f1de00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1de00ce8a0 0x7f1de00c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43040/0 (socket says 172.21.15.29:43040) 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 -- 172.21.15.29:0/3135403625 learned_addr learned my addr 172.21.15.29:0/3135403625 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de619b700 1 --2- 172.21.15.29:0/3135403625 >> v2:172.21.15.144:3300/0 conn(0x7f1de0159750 0x7f1de00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 -- 172.21.15.29:0/3135403625 >> v2:172.21.15.144:3300/0 conn(0x7f1de0159750 msgr2=0x7f1de00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 --2- 172.21.15.29:0/3135403625 >> v2:172.21.15.144:3300/0 conn(0x7f1de0159750 0x7f1de00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 -- 172.21.15.29:0/3135403625 >> v1:172.21.15.144:6789/0 conn(0x7f1de0159b20 legacy=0x7f1de00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 -- 172.21.15.29:0/3135403625 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00cc7c0 con 0x7f1de00ce8a0 2023-10-07T09:45:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.687+0000 7f1de699c700 1 --2- 172.21.15.29:0/3135403625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de00ce8a0 0x7f1de00c5460 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1de0089840 tx=0x7f1dd8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facd1fcd76952d4c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de5199700 1 -- 172.21.15.29:0/3135403625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd800e070 con 0x7f1de00ce8a0 2023-10-07T09:45:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de5199700 1 -- 172.21.15.29:0/3135403625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd800ba10 con 0x7f1de00ce8a0 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de5199700 1 -- 172.21.15.29:0/3135403625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80092e0 con 0x7f1de00ce8a0 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 -- 172.21.15.29:0/3135403625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de00ce8a0 msgr2=0x7f1de00c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 --2- 172.21.15.29:0/3135403625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de00ce8a0 0x7f1de00c5460 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1de0089840 tx=0x7f1dd8009840 comp rx=0 tx=0).stop 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 -- 172.21.15.29:0/3135403625 shutdown_connections 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 --2- 172.21.15.29:0/3135403625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de00ce8a0 0x7f1de00c5460 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 --2- 172.21.15.29:0/3135403625 >> v2:172.21.15.144:3300/0 conn(0x7f1de0159750 0x7f1de00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 -- 172.21.15.29:0/3135403625 >> 172.21.15.29:0/3135403625 conn(0x7f1de0063a10 msgr2=0x7f1de00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 -- 172.21.15.29:0/3135403625 shutdown_connections 2023-10-07T09:45:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.688+0000 7f1de83ff700 1 -- 172.21.15.29:0/3135403625 wait complete. 2023-10-07T09:45:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.690+0000 7f1de83ff700 1 Processor -- start 2023-10-07T09:45:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.690+0000 7f1de83ff700 1 -- start start 2023-10-07T09:45:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de83ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00ce8a0 0x7f1de0155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de83ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de0159750 0x7f1de016a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de83ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 0x7f1de01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de83ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de0148600 con 0x7f1de0159b20 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de83ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00ca130 con 0x7f1de00ce8a0 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de83ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00b3770 con 0x7f1de0159750 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de699c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 0x7f1de01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de699c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 0x7f1de01480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43054/0 (socket says 172.21.15.29:43054) 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de699c700 1 -- 172.21.15.29:0/2174937455 learned_addr learned my addr 172.21.15.29:0/2174937455 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de619b700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00ce8a0 0x7f1de0155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.691+0000 7f1de599a700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de0159750 0x7f1de016a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de699c700 1 -- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de0159750 msgr2=0x7f1de016a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de699c700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de0159750 0x7f1de016a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de699c700 1 -- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00ce8a0 msgr2=0x7f1de0155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de699c700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00ce8a0 0x7f1de0155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de699c700 1 -- 172.21.15.29:0/2174937455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8009ae0 con 0x7f1de0159b20 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de619b700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00ce8a0 0x7f1de0155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de699c700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 0x7f1de01480c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009fd0 tx=0x7f1dd8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd800bee0 con 0x7f1de0159b20 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd80032f0 con 0x7f1de0159b20 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd801d420 con 0x7f1de0159b20 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de01558c0 con 0x7f1de0159b20 2023-10-07T09:45:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.692+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de00c9e50 con 0x7f1de0159b20 2023-10-07T09:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.693+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de00b3900 con 0x7f1de0159b20 2023-10-07T09:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.694+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1dd801d850 con 0x7f1de0159b20 2023-10-07T09:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.695+0000 7f1dd67fc700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1dc4066f30 0x7f1dc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.695+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1dd8085f40 con 0x7f1de0159b20 2023-10-07T09:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.696+0000 7f1de619b700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1dc4066f30 0x7f1dc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.696+0000 7f1de619b700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1dc4066f30 0x7f1dc40693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc006aa0 tx=0x7f1ddc008040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:02.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.699+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1dd8052f90 con 0x7f1de0159b20 2023-10-07T09:45:02.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:02.946+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1de004c3c0 con 0x7f1de0159b20 2023-10-07T09:45:03.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.584+0000 7f1dd67fc700 1 -- 172.21.15.29:0/2174937455 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1dd80565b0 con 0x7f1de0159b20 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1dc4066f30 msgr2=0x7f1dc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1dc4066f30 0x7f1dc40693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc006aa0 tx=0x7f1ddc008040 comp rx=0 tx=0).stop 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 msgr2=0x7f1de01480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 0x7f1de01480c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009fd0 tx=0x7f1dd8017000 comp rx=0 tx=0).stop 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 shutdown_connections 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1dc4066f30 0x7f1dc40693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de0159b20 0x7f1de01480c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00ce8a0 0x7f1de0155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 --2- 172.21.15.29:0/2174937455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de0159750 0x7f1de016a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 >> 172.21.15.29:0/2174937455 conn(0x7f1de0063a10 msgr2=0x7f1de00c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:03.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 shutdown_connections 2023-10-07T09:45:03.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.587+0000 7f1de83ff700 1 -- 172.21.15.29:0/2174937455 wait complete. 2023-10-07T09:45:03.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:45:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.811+0000 7feed25ff700 1 Processor -- start 2023-10-07T09:45:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.811+0000 7feed25ff700 1 -- start start 2023-10-07T09:45:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.812+0000 7feed25ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc06a8c0 0x7feecc0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.812+0000 7feed25ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feecc06a430 con 0x7feecc159790 2023-10-07T09:45:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feed25ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feecc159b60 con 0x7feecc06b5c0 2023-10-07T09:45:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feed25ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc069820 con 0x7feecc06a8c0 2023-10-07T09:45:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecbfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feecc159790 0x7feecc06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32812/0 (socket says 172.21.15.29:32812) 2023-10-07T09:45:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecbfff700 1 -- 172.21.15.29:0/16106720 learned_addr learned my addr 172.21.15.29:0/16106720 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecb7fe700 1 --2- 172.21.15.29:0/16106720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc06a8c0 0x7feecc0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887243527 0 0) 0x7feecc06a430 con 0x7feecc159790 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb4003570 con 0x7feecc159790 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908440078 0 0) 0x7feecc159b60 con 0x7feecc06b5c0 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.813+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc06a430 con 0x7feecc06b5c0 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177103858 0 0) 0x7feeb4003570 con 0x7feecc159790 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecb7fe700 1 -- 172.21.15.29:0/16106720 >> v1:172.21.15.144:6789/0 conn(0x7feecc06b5c0 legacy=0x7feecc0c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecb7fe700 1 -- 172.21.15.29:0/16106720 >> v1:172.21.15.29:6789/0 conn(0x7feecc159790 legacy=0x7feecc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecb7fe700 1 -- 172.21.15.29:0/16106720 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc069b00 con 0x7feecc06a8c0 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecb7fe700 1 --2- 172.21.15.29:0/16106720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc06a8c0 0x7feecc0c6f40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feec0009910 tx=0x7feec00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27f8d301e1e7b0e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc001470 con 0x7feecc06a8c0 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec0011420 con 0x7feecc06a8c0 2023-10-07T09:45:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.814+0000 7feecaffd700 1 -- 172.21.15.29:0/16106720 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec00117c0 con 0x7feecc06a8c0 2023-10-07T09:45:03.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 -- 172.21.15.29:0/16106720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc06a8c0 msgr2=0x7feecc0c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 --2- 172.21.15.29:0/16106720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc06a8c0 0x7feecc0c6f40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feec0009910 tx=0x7feec00091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 -- 172.21.15.29:0/16106720 shutdown_connections 2023-10-07T09:45:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 --2- 172.21.15.29:0/16106720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc06a8c0 0x7feecc0c6f40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 -- 172.21.15.29:0/16106720 >> 172.21.15.29:0/16106720 conn(0x7feecc0c9410 msgr2=0x7feecc0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 -- 172.21.15.29:0/16106720 shutdown_connections 2023-10-07T09:45:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.815+0000 7feed25ff700 1 -- 172.21.15.29:0/16106720 wait complete. 2023-10-07T09:45:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.816+0000 7feed25ff700 1 Processor -- start 2023-10-07T09:45:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 -- start start 2023-10-07T09:45:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 0x7feecc1512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feecc159790 0x7feecc0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc0ac3f0 0x7feecc15ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc0aceb0 con 0x7feecc06b5c0 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc15d4e0 con 0x7feecc159790 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.817+0000 7feed25ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc0afc90 con 0x7feecc0ac3f0 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 0x7feecc1512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 0x7feecc1512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60106/0 (socket says 172.21.15.29:60106) 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 -- 172.21.15.29:0/3549010638 learned_addr learned my addr 172.21.15.29:0/3549010638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feed0b9c700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc0ac3f0 0x7feecc15ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecb7fe700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feecc159790 0x7feecc0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 -- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc0ac3f0 msgr2=0x7feecc15ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc0ac3f0 0x7feecc15ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 -- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feecc159790 msgr2=0x7feecc0af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feecc159790 0x7feecc0af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.818+0000 7feecbfff700 1 -- 172.21.15.29:0/3549010638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec0009420 con 0x7feecc06b5c0 2023-10-07T09:45:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.819+0000 7feecbfff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 0x7feecc1512e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7feebc00c7f0 tx=0x7feebc001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.819+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc007360 con 0x7feecc06b5c0 2023-10-07T09:45:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.819+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc0074e0 con 0x7feecc06b5c0 2023-10-07T09:45:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.819+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc004830 con 0x7feecc06b5c0 2023-10-07T09:45:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.819+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feecc0ac8f0 con 0x7feecc06b5c0 2023-10-07T09:45:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.819+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feecc15d170 con 0x7feecc06b5c0 2023-10-07T09:45:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.820+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feecc04c3c0 con 0x7feecc06b5c0 2023-10-07T09:45:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.821+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feebc004be0 con 0x7feecc06b5c0 2023-10-07T09:45:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.822+0000 7feeb3fff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feeb8066f30 0x7feeb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.822+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7feebc084610 con 0x7feecc06b5c0 2023-10-07T09:45:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.822+0000 7feecb7fe700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feeb8066f30 0x7feeb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.823+0000 7feecb7fe700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feeb8066f30 0x7feeb80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feec00030c0 tx=0x7feec0016000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:03.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:03.826+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feebc0516e0 con 0x7feecc06b5c0 2023-10-07T09:45:04.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.072+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7feecc063640 con 0x7feecc06b5c0 2023-10-07T09:45:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.591+0000 7feeb3fff700 1 -- 172.21.15.29:0/3549010638 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7feebc054d00 con 0x7feecc06b5c0 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feeb8066f30 msgr2=0x7feeb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feeb8066f30 0x7feeb80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feec00030c0 tx=0x7feec0016000 comp rx=0 tx=0).stop 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 msgr2=0x7feecc1512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 0x7feecc1512e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7feebc00c7f0 tx=0x7feebc001a10 comp rx=0 tx=0).stop 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 shutdown_connections 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feeb8066f30 0x7feeb80693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feecc0ac3f0 0x7feecc15ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feecc06b5c0 0x7feecc1512e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 --2- 172.21.15.29:0/3549010638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feecc159790 0x7feecc0af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.594+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 >> 172.21.15.29:0/3549010638 conn(0x7feecc0c9410 msgr2=0x7feecc1553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.595+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 shutdown_connections 2023-10-07T09:45:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.595+0000 7feed25ff700 1 -- 172.21.15.29:0/3549010638 wait complete. 2023-10-07T09:45:04.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:04.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.815+0000 7f98ef7ff700 1 Processor -- start 2023-10-07T09:45:04.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.815+0000 7f98ef7ff700 1 -- start start 2023-10-07T09:45:04.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.816+0000 7f98ef7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:04.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.816+0000 7f98ef7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f98e80ae8b0 0x7f98e80b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:04.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ef7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e8158ad0 con 0x7f98e80b1840 2023-10-07T09:45:04.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ef7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e80b1cc0 con 0x7f98e80ae8b0 2023-10-07T09:45:04.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ef7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e815b140 con 0x7f98e8157290 2023-10-07T09:45:04.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ecd9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f98e80ae8b0 0x7f98e80b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:04.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ed59b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:04.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98edd9c700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f98e80b1840 0x7f98e80ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:56226/0 (socket says 172.21.15.29:56226) 2023-10-07T09:45:04.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98edd9c700 1 -- 172.21.15.29:0/1522547097 learned_addr learned my addr 172.21.15.29:0/1522547097 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:04.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ecd9a700 1 -- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 msgr2=0x7f98e80cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:04.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ecd9a700 1 --2- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.817+0000 7f98ecd9a700 1 -- 172.21.15.29:0/1522547097 >> v1:172.21.15.144:6789/0 conn(0x7f98e80b1840 legacy=0x7f98e80ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:04.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.818+0000 7f98ecd9a700 1 -- 172.21.15.29:0/1522547097 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e815b420 con 0x7f98e80ae8b0 2023-10-07T09:45:04.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.818+0000 7f98ecd9a700 1 --2- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80ae8b0 0x7f98e80b1300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f98d80098e0 tx=0x7f98d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cb90dd3bf934376 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.818+0000 7f98dffff700 1 -- 172.21.15.29:0/1522547097 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800b640 con 0x7f98e80ae8b0 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.818+0000 7f98dffff700 1 -- 172.21.15.29:0/1522547097 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d800b7a0 con 0x7f98e80ae8b0 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.818+0000 7f98dffff700 1 -- 172.21.15.29:0/1522547097 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800bb40 con 0x7f98e80ae8b0 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 -- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80ae8b0 msgr2=0x7f98e80b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80ae8b0 0x7f98e80b1300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f98d80098e0 tx=0x7f98d80091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 -- 172.21.15.29:0/1522547097 shutdown_connections 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/1522547097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80ae8b0 0x7f98e80b1300 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 -- 172.21.15.29:0/1522547097 >> 172.21.15.29:0/1522547097 conn(0x7f98e80680a0 msgr2=0x7f98e80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 -- 172.21.15.29:0/1522547097 shutdown_connections 2023-10-07T09:45:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.819+0000 7f98ef7ff700 1 -- 172.21.15.29:0/1522547097 wait complete. 2023-10-07T09:45:04.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.820+0000 7f98ef7ff700 1 Processor -- start 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.820+0000 7f98ef7ff700 1 -- start start 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ef7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 0x7f98e814fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ef7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ef7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80b0870 0x7f98e80c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ef7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80afea0 con 0x7f98e80b1840 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ef7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80af290 con 0x7f98e80b0870 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ef7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80af570 con 0x7f98e8157290 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.821+0000 7f98ed59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 0x7f98e814fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 0x7f98e814fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60112/0 (socket says 172.21.15.29:60112) 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 -- 172.21.15.29:0/2689005111 learned_addr learned my addr 172.21.15.29:0/2689005111 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:04.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ecd9a700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98edd9c700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80b0870 0x7f98e80c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 -- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 msgr2=0x7f98e80b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 -- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80b0870 msgr2=0x7f98e80c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80b0870 0x7f98e80c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 -- 172.21.15.29:0/2689005111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8009420 con 0x7f98e80b1840 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98ed59b700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 0x7f98e814fd40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f98e400e930 tx=0x7f98e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.822+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e400eed0 con 0x7f98e80b1840 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.823+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e400c850 con 0x7f98e80b1840 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.823+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e80af850 con 0x7f98e80b1840 2023-10-07T09:45:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.823+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e400cbf0 con 0x7f98e80b1840 2023-10-07T09:45:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.823+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e8150280 con 0x7f98e80b1840 2023-10-07T09:45:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.824+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e804c3c0 con 0x7f98e80b1840 2023-10-07T09:45:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.825+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f98e4013470 con 0x7f98e80b1840 2023-10-07T09:45:04.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.825+0000 7f98dd7fa700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98cc066f30 0x7f98cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.825+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f98e40849c0 con 0x7f98e80b1840 2023-10-07T09:45:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.826+0000 7f98ecd9a700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98cc066f30 0x7f98cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.827+0000 7f98ecd9a700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98cc066f30 0x7f98cc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98d80093f0 tx=0x7f98d8017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:04.829+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98e4051a10 con 0x7f98e80b1840 2023-10-07T09:45:05.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.076+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98e815b700 con 0x7f98e80b1840 2023-10-07T09:45:05.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.077+0000 7f98dd7fa700 1 -- 172.21.15.29:0/2689005111 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f98e4055030 con 0x7f98e80b1840 2023-10-07T09:45:05.078 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:05.078 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:01.071366+0000","modified":"2023-10-07T09:45:04.584641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2799196407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2799196407},{"type":"v1","addr":"172.21.15.29:6835","nonce":2799196407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98cc066f30 msgr2=0x7f98cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98cc066f30 0x7f98cc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98d80093f0 tx=0x7f98d8017000 comp rx=0 tx=0).stop 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 msgr2=0x7f98e814fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 0x7f98e814fd40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f98e400e930 tx=0x7f98e40091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 shutdown_connections 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98cc066f30 0x7f98cc0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98e80b0870 0x7f98e80c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e80b1840 0x7f98e814fd40 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 --2- 172.21.15.29:0/2689005111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98e8157290 0x7f98e80b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.079+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 >> 172.21.15.29:0/2689005111 conn(0x7f98e80680a0 msgr2=0x7f98e8069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:05.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.080+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 shutdown_connections 2023-10-07T09:45:05.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.080+0000 7f98ef7ff700 1 -- 172.21.15.29:0/2689005111 wait complete. 2023-10-07T09:45:05.082 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-07T09:45:05.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:05.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.295+0000 7f38955ff700 1 Processor -- start 2023-10-07T09:45:05.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.295+0000 7f38955ff700 1 -- start start 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f38955ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3890159c00 0x7f38900ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f38955ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3890068620 0x7f3890068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f38955ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3890068fc0 con 0x7f38900ad220 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f38955ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389015a050 con 0x7f3890068620 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f38955ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389015af00 con 0x7f3890159c00 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388effd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f38900ad220 0x7f38900ac490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32824/0 (socket says 172.21.15.29:32824) 2023-10-07T09:45:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388effd700 1 -- 172.21.15.29:0/416129631 learned_addr learned my addr 172.21.15.29:0/416129631 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388f7fe700 1 --2- 172.21.15.29:0/416129631 >> v2:172.21.15.29:3300/0 conn(0x7f3890068620 0x7f3890068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388e7fc700 1 --2- 172.21.15.29:0/416129631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3890159c00 0x7f38900ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388dffb700 1 -- 172.21.15.29:0/416129631 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237045767 0 0) 0x7f3880001110 con 0x7f38900ad220 2023-10-07T09:45:05.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388dffb700 1 -- 172.21.15.29:0/416129631 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3878003570 con 0x7f38900ad220 2023-10-07T09:45:05.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388f7fe700 1 -- 172.21.15.29:0/416129631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3890159c00 msgr2=0x7f38900ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.297+0000 7f388f7fe700 1 --2- 172.21.15.29:0/416129631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3890159c00 0x7f38900ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.298+0000 7f388f7fe700 1 -- 172.21.15.29:0/416129631 >> v1:172.21.15.29:6789/0 conn(0x7f38900ad220 legacy=0x7f38900ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.298+0000 7f388f7fe700 1 -- 172.21.15.29:0/416129631 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389015a250 con 0x7f3890068620 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.298+0000 7f388f7fe700 1 --2- 172.21.15.29:0/416129631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890068620 0x7f3890068a80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3888009e60 tx=0x7f3888009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3710dfd58ddf58 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.298+0000 7f388dffb700 1 -- 172.21.15.29:0/416129631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388800e070 con 0x7f3890068620 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.298+0000 7f388dffb700 1 -- 172.21.15.29:0/416129631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388800bb60 con 0x7f3890068620 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.298+0000 7f388dffb700 1 -- 172.21.15.29:0/416129631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888012430 con 0x7f3890068620 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 -- 172.21.15.29:0/416129631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890068620 msgr2=0x7f3890068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 --2- 172.21.15.29:0/416129631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890068620 0x7f3890068a80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3888009e60 tx=0x7f3888009780 comp rx=0 tx=0).stop 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 -- 172.21.15.29:0/416129631 shutdown_connections 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 --2- 172.21.15.29:0/416129631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890068620 0x7f3890068a80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 --2- 172.21.15.29:0/416129631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3890159c00 0x7f38900ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 -- 172.21.15.29:0/416129631 >> 172.21.15.29:0/416129631 conn(0x7f389006dd00 msgr2=0x7f389006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:05.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 -- 172.21.15.29:0/416129631 shutdown_connections 2023-10-07T09:45:05.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.299+0000 7f38955ff700 1 -- 172.21.15.29:0/416129631 wait complete. 2023-10-07T09:45:05.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.300+0000 7f38955ff700 1 Processor -- start 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.300+0000 7f38955ff700 1 -- start start 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.301+0000 7f38955ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3890068620 0x7f389013fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.301+0000 7f38955ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38900ad220 0x7f3890158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.301+0000 7f38955ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 0x7f38900c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.301+0000 7f38955ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3890153860 con 0x7f3890159c00 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.301+0000 7f38955ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38901403c0 con 0x7f3890068620 2023-10-07T09:45:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.301+0000 7f38955ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900c99e0 con 0x7f38900ad220 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 0x7f38900c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 0x7f38900c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60126/0 (socket says 172.21.15.29:60126) 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 -- 172.21.15.29:0/2471860983 learned_addr learned my addr 172.21.15.29:0/2471860983 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388effd700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3890068620 0x7f389013fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388e7fc700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38900ad220 0x7f3890158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 -- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38900ad220 msgr2=0x7f3890158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38900ad220 0x7f3890158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 -- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3890068620 msgr2=0x7f389013fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3890068620 0x7f389013fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 -- 172.21.15.29:0/2471860983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888009a20 con 0x7f3890159c00 2023-10-07T09:45:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.302+0000 7f388f7fe700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 0x7f38900c8e00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f389006d9f0 tx=0x7f3888019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.303+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888012e80 con 0x7f3890159c00 2023-10-07T09:45:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.303+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388801d630 con 0x7f3890159c00 2023-10-07T09:45:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.303+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388801d9f0 con 0x7f3890159c00 2023-10-07T09:45:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.303+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38900c9cc0 con 0x7f3890159c00 2023-10-07T09:45:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.303+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38900ca280 con 0x7f3890159c00 2023-10-07T09:45:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.304+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389004c3c0 con 0x7f3890159c00 2023-10-07T09:45:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.305+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3888024070 con 0x7f3890159c00 2023-10-07T09:45:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.306+0000 7f3876ffd700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f387c066f30 0x7f387c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.306+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3888086fe0 con 0x7f3890159c00 2023-10-07T09:45:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.307+0000 7f388effd700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f387c066f30 0x7f387c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.307+0000 7f388effd700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f387c066f30 0x7f387c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3880001270 tx=0x7f3880009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:05.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.310+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38880541c0 con 0x7f3890159c00 2023-10-07T09:45:05.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.511+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f389006ec80 con 0x7f3890159c00 2023-10-07T09:45:05.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.512+0000 7f3876ffd700 1 -- 172.21.15.29:0/2471860983 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16020 (secure 0 0 0) 0x7f38880577e0 con 0x7f3890159c00 2023-10-07T09:45:05.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:05.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":90,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:02.571829+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:58.854049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:00.048655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.514+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f387c066f30 msgr2=0x7f387c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.514+0000 7f38955ff700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f387c066f30 0x7f387c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3880001270 tx=0x7f3880009040 comp rx=0 tx=0).stop 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.514+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 msgr2=0x7f38900c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.514+0000 7f38955ff700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 0x7f38900c8e00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f389006d9f0 tx=0x7f3888019000 comp rx=0 tx=0).stop 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 shutdown_connections 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f387c066f30 0x7f387c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3890159c00 0x7f38900c8e00 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3890068620 0x7f389013fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 --2- 172.21.15.29:0/2471860983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38900ad220 0x7f3890158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 >> 172.21.15.29:0/2471860983 conn(0x7f389006dd00 msgr2=0x7f38900aa0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:05.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 shutdown_connections 2023-10-07T09:45:05.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.515+0000 7f38955ff700 1 -- 172.21.15.29:0/2471860983 wait complete. 2023-10-07T09:45:05.534 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:45:05.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:45:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.730+0000 7f57c03ff700 1 Processor -- start 2023-10-07T09:45:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.730+0000 7f57c03ff700 1 -- start start 2023-10-07T09:45:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57c03ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 0x7f57b80b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57c03ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57b80b0540 0x7f57b815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57c03ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80b0980 con 0x7f57b8153cc0 2023-10-07T09:45:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57c03ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b8154090 con 0x7f57b80b0540 2023-10-07T09:45:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57c03ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b8159f60 con 0x7f57b815ade0 2023-10-07T09:45:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57be19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f57b8153cc0 0x7f57b80b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32836/0 (socket says 172.21.15.29:32836) 2023-10-07T09:45:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.731+0000 7f57be19b700 1 -- 172.21.15.29:0/477415193 learned_addr learned my addr 172.21.15.29:0/477415193 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd99a700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 0x7f57b80b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57be99c700 1 --2- 172.21.15.29:0/477415193 >> v2:172.21.15.144:3300/0 conn(0x7f57b80b0540 0x7f57b815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292250329 0 0) 0x7f57b80b0980 con 0x7f57b8153cc0 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a4003570 con 0x7f57b8153cc0 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921431440 0 0) 0x7f57a4003570 con 0x7f57b8153cc0 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 msgr2=0x7f57b80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 0x7f57b80b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 >> v2:172.21.15.144:3300/0 conn(0x7f57b80b0540 msgr2=0x7f57b815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 --2- 172.21.15.29:0/477415193 >> v2:172.21.15.144:3300/0 conn(0x7f57b80b0540 0x7f57b815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b815a240 con 0x7f57b8153cc0 2023-10-07T09:45:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f57b40015f0 con 0x7f57b8153cc0 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 >> v1:172.21.15.29:6789/0 conn(0x7f57b8153cc0 legacy=0x7f57b80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57a4003a60 0x7f57a4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 0x7f57a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4004d50 0x7f57a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b8154090 con 0x7f57a4003a60 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b8159f60 con 0x7f57a4004d50 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.732+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a400ba60 con 0x7f57b815ade0 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57a4003a60 0x7f57a4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57bd99a700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 0x7f57a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be99c700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4004d50 0x7f57a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 -- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 msgr2=0x7f57a4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b815ade0 0x7f57a4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 -- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4004d50 msgr2=0x7f57a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4004d50 0x7f57a40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 -- 172.21.15.29:0/477415193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4003570 con 0x7f57a4003a60 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57be19b700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57a4003a60 0x7f57a4003e70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f57b400c3c0 tx=0x7f57b4007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24ea06d26d456f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b400cc30 con 0x7f57a4003a60 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b400cd90 con 0x7f57a4003a60 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.733+0000 7f57bd199700 1 -- 172.21.15.29:0/477415193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4007320 con 0x7f57a4003a60 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57c03ff700 1 -- 172.21.15.29:0/477415193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57a4003a60 msgr2=0x7f57a4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57c03ff700 1 --2- 172.21.15.29:0/477415193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57a4003a60 0x7f57a4003e70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f57b400c3c0 tx=0x7f57b4007920 comp rx=0 tx=0).stop 2023-10-07T09:45:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57be99c700 1 -- 172.21.15.29:0/477415193 reap_dead start 2023-10-07T09:45:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57c03ff700 1 -- 172.21.15.29:0/477415193 shutdown_connections 2023-10-07T09:45:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57c03ff700 1 -- 172.21.15.29:0/477415193 >> 172.21.15.29:0/477415193 conn(0x7f57b8067610 msgr2=0x7f57b8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57c03ff700 1 -- 172.21.15.29:0/477415193 shutdown_connections 2023-10-07T09:45:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.734+0000 7f57c03ff700 1 -- 172.21.15.29:0/477415193 wait complete. 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.735+0000 7f57c03ff700 1 Processor -- start 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.736+0000 7f57c03ff700 1 -- start start 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.736+0000 7f57c03ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4003a60 0x7f57b8152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.736+0000 7f57c03ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 0x7f57b815fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57c03ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b80ce830 0x7f57b815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57c03ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80c75b0 con 0x7f57b8169840 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57c03ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80c7890 con 0x7f57a4003a60 2023-10-07T09:45:05.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57c03ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80c8280 con 0x7f57b80ce830 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 0x7f57b815fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 0x7f57b815fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60150/0 (socket says 172.21.15.29:60150) 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 -- 172.21.15.29:0/2563960277 learned_addr learned my addr 172.21.15.29:0/2563960277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57be99c700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b80ce830 0x7f57b815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57be19b700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4003a60 0x7f57b8152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 -- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b80ce830 msgr2=0x7f57b815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b80ce830 0x7f57b815ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 -- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4003a60 msgr2=0x7f57b8152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4003a60 0x7f57b8152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.737+0000 7f57bd99a700 1 -- 172.21.15.29:0/2563960277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80c8560 con 0x7f57b8169840 2023-10-07T09:45:05.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.738+0000 7f57bd99a700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 0x7f57b815fa80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f57a8011d40 tx=0x7f57a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.738+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a800f990 con 0x7f57b8169840 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.738+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a800faf0 con 0x7f57b8169840 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.738+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a800feb0 con 0x7f57b8169840 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.738+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b80c7f80 con 0x7f57b8169840 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.738+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b8152840 con 0x7f57b8169840 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.739+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b804c3c0 con 0x7f57b8169840 2023-10-07T09:45:05.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.740+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f57a8014650 con 0x7f57b8169840 2023-10-07T09:45:05.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.741+0000 7f57ae7fc700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f579c066f30 0x7f579c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:05.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.741+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f57a8018030 con 0x7f57b8169840 2023-10-07T09:45:05.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.741+0000 7f57be19b700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f579c066f30 0x7f579c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:05.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.742+0000 7f57be19b700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f579c066f30 0x7f579c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f57b4008fd0 tx=0x7f57b4015000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:05.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.745+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57a8052490 con 0x7f57b8169840 2023-10-07T09:45:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.942+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57b80c7a00 con 0x7f579c066f30 2023-10-07T09:45:05.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.953+0000 7f57ae7fc700 1 -- 172.21.15.29:0/2563960277 <== mgr.5885 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288964 (secure 0 0 0) 0x7f57b80c7a00 con 0x7f579c066f30 2023-10-07T09:45:05.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f579c066f30 msgr2=0x7f579c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f579c066f30 0x7f579c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f57b4008fd0 tx=0x7f57b4015000 comp rx=0 tx=0).stop 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 msgr2=0x7f57b815fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 0x7f57b815fa80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f57a8011d40 tx=0x7f57a80091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 shutdown_connections 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f579c066f30 0x7f579c0693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57b80ce830 0x7f57b815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a4003a60 0x7f57b8152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 --2- 172.21.15.29:0/2563960277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57b8169840 0x7f57b815fa80 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 >> 172.21.15.29:0/2563960277 conn(0x7f57b8067610 msgr2=0x7f57b80add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:05.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 shutdown_connections 2023-10-07T09:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:05.957+0000 7f57c03ff700 1 -- 172.21.15.29:0/2563960277 wait complete. 2023-10-07T09:45:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:45:05.976 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:45:05.645785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437156,"kb_used_data":9700,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752537564,"statfs":{"total":773094113280,"available":770598465536,"internally_reserved":0,"allocated":9932800,"data_stored":5300372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001329"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.588542+0000","last_change":"2023-10-07T09:45:01.588542+0000","last_active":"2023-10-07T09:45:01.588542+0000","last_peered":"2023-10-07T09:45:01.588542+0000","last_clean":"2023-10-07T09:45:01.588542+0000","last_became_active":"2023-10-07T09:45:01.588296+0000","last_became_peered":"2023-10-07T09:45:01.588296+0000","last_unstale":"2023-10-07T09:45:01.588542+0000","last_undegraded":"2023-10-07T09:45:01.588542+0000","last_fullsized":"2023-10-07T09:45:01.588542+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:50:39.121700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.590255+0000","last_change":"2023-10-07T09:45:01.584130+0000","last_active":"2023-10-07T09:45:01.590255+0000","last_peered":"2023-10-07T09:45:01.590255+0000","last_clean":"2023-10-07T09:45:01.590255+0000","last_became_active":"2023-10-07T09:45:01.583177+0000","last_became_peered":"2023-10-07T09:45:01.583177+0000","last_unstale":"2023-10-07T09:45:01.590255+0000","last_undegraded":"2023-10-07T09:45:01.590255+0000","last_fullsized":"2023-10-07T09:45:01.590255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:44.328328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.580465+0000","last_change":"2023-10-07T09:45:01.580465+0000","last_active":"2023-10-07T09:45:01.580465+0000","last_peered":"2023-10-07T09:45:01.580465+0000","last_clean":"2023-10-07T09:45:01.580465+0000","last_became_active":"2023-10-07T09:45:01.577079+0000","last_became_peered":"2023-10-07T09:45:01.577079+0000","last_unstale":"2023-10-07T09:45:01.580465+0000","last_undegraded":"2023-10-07T09:45:01.580465+0000","last_fullsized":"2023-10-07T09:45:01.580465+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:33:21.238319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596032+0000","last_change":"2023-10-07T09:45:01.593788+0000","last_active":"2023-10-07T09:45:01.596032+0000","last_peered":"2023-10-07T09:45:01.596032+0000","last_clean":"2023-10-07T09:45:01.596032+0000","last_became_active":"2023-10-07T09:45:01.593113+0000","last_became_peered":"2023-10-07T09:45:01.593113+0000","last_unstale":"2023-10-07T09:45:01.596032+0000","last_undegraded":"2023-10-07T09:45:01.596032+0000","last_fullsized":"2023-10-07T09:45:01.596032+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:19:01.805409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585874+0000","last_change":"2023-10-07T09:45:01.582201+0000","last_active":"2023-10-07T09:45:01.585874+0000","last_peered":"2023-10-07T09:45:01.585874+0000","last_clean":"2023-10-07T09:45:01.585874+0000","last_became_active":"2023-10-07T09:45:01.581890+0000","last_became_peered":"2023-10-07T09:45:01.581890+0000","last_unstale":"2023-10-07T09:45:01.585874+0000","last_undegraded":"2023-10-07T09:45:01.585874+0000","last_fullsized":"2023-10-07T09:45:01.585874+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:46.475578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592559+0000","last_change":"2023-10-07T09:45:01.584324+0000","last_active":"2023-10-07T09:45:01.592559+0000","last_peered":"2023-10-07T09:45:01.592559+0000","last_clean":"2023-10-07T09:45:01.592559+0000","last_became_active":"2023-10-07T09:45:01.583970+0000","last_became_peered":"2023-10-07T09:45:01.583970+0000","last_unstale":"2023-10-07T09:45:01.592559+0000","last_undegraded":"2023-10-07T09:45:01.592559+0000","last_fullsized":"2023-10-07T09:45:01.592559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:03.541472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594439+0000","last_change":"2023-10-07T09:45:01.588574+0000","last_active":"2023-10-07T09:45:01.594439+0000","last_peered":"2023-10-07T09:45:01.594439+0000","last_clean":"2023-10-07T09:45:01.594439+0000","last_became_active":"2023-10-07T09:45:01.587936+0000","last_became_peered":"2023-10-07T09:45:01.587936+0000","last_unstale":"2023-10-07T09:45:01.594439+0000","last_undegraded":"2023-10-07T09:45:01.594439+0000","last_fullsized":"2023-10-07T09:45:01.594439+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:41.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.586045+0000","last_change":"2023-10-07T09:45:01.581793+0000","last_active":"2023-10-07T09:45:01.586045+0000","last_peered":"2023-10-07T09:45:01.586045+0000","last_clean":"2023-10-07T09:45:01.586045+0000","last_became_active":"2023-10-07T09:45:01.581518+0000","last_became_peered":"2023-10-07T09:45:01.581518+0000","last_unstale":"2023-10-07T09:45:01.586045+0000","last_undegraded":"2023-10-07T09:45:01.586045+0000","last_fullsized":"2023-10-07T09:45:01.586045+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:12:49.510761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598787+0000","last_change":"2023-10-07T09:45:01.596603+0000","last_active":"2023-10-07T09:45:01.598787+0000","last_peered":"2023-10-07T09:45:01.598787+0000","last_clean":"2023-10-07T09:45:01.598787+0000","last_became_active":"2023-10-07T09:45:01.596321+0000","last_became_peered":"2023-10-07T09:45:01.596321+0000","last_unstale":"2023-10-07T09:45:01.598787+0000","last_undegraded":"2023-10-07T09:45:01.598787+0000","last_fullsized":"2023-10-07T09:45:01.598787+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:35.634346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598713+0000","last_change":"2023-10-07T09:45:01.588610+0000","last_active":"2023-10-07T09:45:01.598713+0000","last_peered":"2023-10-07T09:45:01.598713+0000","last_clean":"2023-10-07T09:45:01.598713+0000","last_became_active":"2023-10-07T09:45:01.588349+0000","last_became_peered":"2023-10-07T09:45:01.588349+0000","last_unstale":"2023-10-07T09:45:01.598713+0000","last_undegraded":"2023-10-07T09:45:01.598713+0000","last_fullsized":"2023-10-07T09:45:01.598713+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:20.855523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.590321+0000","last_change":"2023-10-07T09:45:01.590321+0000","last_active":"2023-10-07T09:45:01.590321+0000","last_peered":"2023-10-07T09:45:01.590321+0000","last_clean":"2023-10-07T09:45:01.590321+0000","last_became_active":"2023-10-07T09:45:01.589977+0000","last_became_peered":"2023-10-07T09:45:01.589977+0000","last_unstale":"2023-10-07T09:45:01.590321+0000","last_undegraded":"2023-10-07T09:45:01.590321+0000","last_fullsized":"2023-10-07T09:45:01.590321+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:52.391971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594678+0000","last_change":"2023-10-07T09:45:01.588451+0000","last_active":"2023-10-07T09:45:01.594678+0000","last_peered":"2023-10-07T09:45:01.594678+0000","last_clean":"2023-10-07T09:45:01.594678+0000","last_became_active":"2023-10-07T09:45:01.587860+0000","last_became_peered":"2023-10-07T09:45:01.587860+0000","last_unstale":"2023-10-07T09:45:01.594678+0000","last_undegraded":"2023-10-07T09:45:01.594678+0000","last_fullsized":"2023-10-07T09:45:01.594678+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:45.733391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598531+0000","last_change":"2023-10-07T09:45:01.592811+0000","last_active":"2023-10-07T09:45:01.598531+0000","last_peered":"2023-10-07T09:45:01.598531+0000","last_clean":"2023-10-07T09:45:01.598531+0000","last_became_active":"2023-10-07T09:45:01.592527+0000","last_became_peered":"2023-10-07T09:45:01.592527+0000","last_unstale":"2023-10-07T09:45:01.598531+0000","last_undegraded":"2023-10-07T09:45:01.598531+0000","last_fullsized":"2023-10-07T09:45:01.598531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:58:04.424056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595454+0000","last_change":"2023-10-07T09:45:01.590992+0000","last_active":"2023-10-07T09:45:01.595454+0000","last_peered":"2023-10-07T09:45:01.595454+0000","last_clean":"2023-10-07T09:45:01.595454+0000","last_became_active":"2023-10-07T09:45:01.589742+0000","last_became_peered":"2023-10-07T09:45:01.589742+0000","last_unstale":"2023-10-07T09:45:01.595454+0000","last_undegraded":"2023-10-07T09:45:01.595454+0000","last_fullsized":"2023-10-07T09:45:01.595454+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:07:47.154408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596261+0000","last_change":"2023-10-07T09:45:01.588043+0000","last_active":"2023-10-07T09:45:01.596261+0000","last_peered":"2023-10-07T09:45:01.596261+0000","last_clean":"2023-10-07T09:45:01.596261+0000","last_became_active":"2023-10-07T09:45:01.587673+0000","last_became_peered":"2023-10-07T09:45:01.587673+0000","last_unstale":"2023-10-07T09:45:01.596261+0000","last_undegraded":"2023-10-07T09:45:01.596261+0000","last_fullsized":"2023-10-07T09:45:01.596261+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:54:49.662078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598178+0000","last_change":"2023-10-07T09:45:01.592866+0000","last_active":"2023-10-07T09:45:01.598178+0000","last_peered":"2023-10-07T09:45:01.598178+0000","last_clean":"2023-10-07T09:45:01.598178+0000","last_became_active":"2023-10-07T09:45:01.592575+0000","last_became_peered":"2023-10-07T09:45:01.592575+0000","last_unstale":"2023-10-07T09:45:01.598178+0000","last_undegraded":"2023-10-07T09:45:01.598178+0000","last_fullsized":"2023-10-07T09:45:01.598178+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:08.408364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598769+0000","last_change":"2023-10-07T09:45:01.596449+0000","last_active":"2023-10-07T09:45:01.598769+0000","last_peered":"2023-10-07T09:45:01.598769+0000","last_clean":"2023-10-07T09:45:01.598769+0000","last_became_active":"2023-10-07T09:45:01.596097+0000","last_became_peered":"2023-10-07T09:45:01.596097+0000","last_unstale":"2023-10-07T09:45:01.598769+0000","last_undegraded":"2023-10-07T09:45:01.598769+0000","last_fullsized":"2023-10-07T09:45:01.598769+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:13:23.493619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594585+0000","last_change":"2023-10-07T09:45:01.590839+0000","last_active":"2023-10-07T09:45:01.594585+0000","last_peered":"2023-10-07T09:45:01.594585+0000","last_clean":"2023-10-07T09:45:01.594585+0000","last_became_active":"2023-10-07T09:45:01.590571+0000","last_became_peered":"2023-10-07T09:45:01.590571+0000","last_unstale":"2023-10-07T09:45:01.594585+0000","last_undegraded":"2023-10-07T09:45:01.594585+0000","last_fullsized":"2023-10-07T09:45:01.594585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:03.097063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597068+0000","last_change":"2023-10-07T09:45:01.590631+0000","last_active":"2023-10-07T09:45:01.597068+0000","last_peered":"2023-10-07T09:45:01.597068+0000","last_clean":"2023-10-07T09:45:01.597068+0000","last_became_active":"2023-10-07T09:45:01.590344+0000","last_became_peered":"2023-10-07T09:45:01.590344+0000","last_unstale":"2023-10-07T09:45:01.597068+0000","last_undegraded":"2023-10-07T09:45:01.597068+0000","last_fullsized":"2023-10-07T09:45:01.597068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:41.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597350+0000","last_change":"2023-10-07T09:45:01.588822+0000","last_active":"2023-10-07T09:45:01.597350+0000","last_peered":"2023-10-07T09:45:01.597350+0000","last_clean":"2023-10-07T09:45:01.597350+0000","last_became_active":"2023-10-07T09:45:01.588546+0000","last_became_peered":"2023-10-07T09:45:01.588546+0000","last_unstale":"2023-10-07T09:45:01.597350+0000","last_undegraded":"2023-10-07T09:45:01.597350+0000","last_fullsized":"2023-10-07T09:45:01.597350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:08:56.847110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594114+0000","last_change":"2023-10-07T09:45:01.580866+0000","last_active":"2023-10-07T09:45:01.594114+0000","last_peered":"2023-10-07T09:45:01.594114+0000","last_clean":"2023-10-07T09:45:01.594114+0000","last_became_active":"2023-10-07T09:45:01.580346+0000","last_became_peered":"2023-10-07T09:45:01.580346+0000","last_unstale":"2023-10-07T09:45:01.594114+0000","last_undegraded":"2023-10-07T09:45:01.594114+0000","last_fullsized":"2023-10-07T09:45:01.594114+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:08:28.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593305+0000","last_change":"2023-10-07T09:45:01.582763+0000","last_active":"2023-10-07T09:45:01.593305+0000","last_peered":"2023-10-07T09:45:01.593305+0000","last_clean":"2023-10-07T09:45:01.593305+0000","last_became_active":"2023-10-07T09:45:01.582512+0000","last_became_peered":"2023-10-07T09:45:01.582512+0000","last_unstale":"2023-10-07T09:45:01.593305+0000","last_undegraded":"2023-10-07T09:45:01.593305+0000","last_fullsized":"2023-10-07T09:45:01.593305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:44.328328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591368+0000","last_change":"2023-10-07T09:45:01.591368+0000","last_active":"2023-10-07T09:45:01.591368+0000","last_peered":"2023-10-07T09:45:01.591368+0000","last_clean":"2023-10-07T09:45:01.591368+0000","last_became_active":"2023-10-07T09:45:01.591086+0000","last_became_peered":"2023-10-07T09:45:01.591086+0000","last_unstale":"2023-10-07T09:45:01.591368+0000","last_undegraded":"2023-10-07T09:45:01.591368+0000","last_fullsized":"2023-10-07T09:45:01.591368+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:09:58.110670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592913+0000","last_change":"2023-10-07T09:45:01.592913+0000","last_active":"2023-10-07T09:45:01.592913+0000","last_peered":"2023-10-07T09:45:01.592913+0000","last_clean":"2023-10-07T09:45:01.592913+0000","last_became_active":"2023-10-07T09:45:01.592629+0000","last_became_peered":"2023-10-07T09:45:01.592629+0000","last_unstale":"2023-10-07T09:45:01.592913+0000","last_undegraded":"2023-10-07T09:45:01.592913+0000","last_fullsized":"2023-10-07T09:45:01.592913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:21:22.836441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593270+0000","last_change":"2023-10-07T09:45:01.580037+0000","last_active":"2023-10-07T09:45:01.593270+0000","last_peered":"2023-10-07T09:45:01.593270+0000","last_clean":"2023-10-07T09:45:01.593270+0000","last_became_active":"2023-10-07T09:45:01.579721+0000","last_became_peered":"2023-10-07T09:45:01.579721+0000","last_unstale":"2023-10-07T09:45:01.593270+0000","last_undegraded":"2023-10-07T09:45:01.593270+0000","last_fullsized":"2023-10-07T09:45:01.593270+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:58:04.424056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596049+0000","last_change":"2023-10-07T09:45:01.588995+0000","last_active":"2023-10-07T09:45:01.596049+0000","last_peered":"2023-10-07T09:45:01.596049+0000","last_clean":"2023-10-07T09:45:01.596049+0000","last_became_active":"2023-10-07T09:45:01.588345+0000","last_became_peered":"2023-10-07T09:45:01.588345+0000","last_unstale":"2023-10-07T09:45:01.596049+0000","last_undegraded":"2023-10-07T09:45:01.596049+0000","last_fullsized":"2023-10-07T09:45:01.596049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:53:49.463823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.581287+0000","last_change":"2023-10-07T09:45:01.581287+0000","last_active":"2023-10-07T09:45:01.581287+0000","last_peered":"2023-10-07T09:45:01.581287+0000","last_clean":"2023-10-07T09:45:01.581287+0000","last_became_active":"2023-10-07T09:45:01.581011+0000","last_became_peered":"2023-10-07T09:45:01.581011+0000","last_unstale":"2023-10-07T09:45:01.581287+0000","last_undegraded":"2023-10-07T09:45:01.581287+0000","last_fullsized":"2023-10-07T09:45:01.581287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:32.302603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598361+0000","last_change":"2023-10-07T09:45:01.587873+0000","last_active":"2023-10-07T09:45:01.598361+0000","last_peered":"2023-10-07T09:45:01.598361+0000","last_clean":"2023-10-07T09:45:01.598361+0000","last_became_active":"2023-10-07T09:45:01.587613+0000","last_became_peered":"2023-10-07T09:45:01.587613+0000","last_unstale":"2023-10-07T09:45:01.598361+0000","last_undegraded":"2023-10-07T09:45:01.598361+0000","last_fullsized":"2023-10-07T09:45:01.598361+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:03.541472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.581026+0000","last_change":"2023-10-07T09:45:01.581026+0000","last_active":"2023-10-07T09:45:01.581026+0000","last_peered":"2023-10-07T09:45:01.581026+0000","last_clean":"2023-10-07T09:45:01.581026+0000","last_became_active":"2023-10-07T09:45:01.580767+0000","last_became_peered":"2023-10-07T09:45:01.580767+0000","last_unstale":"2023-10-07T09:45:01.581026+0000","last_undegraded":"2023-10-07T09:45:01.581026+0000","last_fullsized":"2023-10-07T09:45:01.581026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:59.125204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593372+0000","last_change":"2023-10-07T09:45:01.588069+0000","last_active":"2023-10-07T09:45:01.593372+0000","last_peered":"2023-10-07T09:45:01.593372+0000","last_clean":"2023-10-07T09:45:01.593372+0000","last_became_active":"2023-10-07T09:45:01.587793+0000","last_became_peered":"2023-10-07T09:45:01.587793+0000","last_unstale":"2023-10-07T09:45:01.593372+0000","last_undegraded":"2023-10-07T09:45:01.593372+0000","last_fullsized":"2023-10-07T09:45:01.593372+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:39:27.825926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594462+0000","last_change":"2023-10-07T09:45:01.588759+0000","last_active":"2023-10-07T09:45:01.594462+0000","last_peered":"2023-10-07T09:45:01.594462+0000","last_clean":"2023-10-07T09:45:01.594462+0000","last_became_active":"2023-10-07T09:45:01.588233+0000","last_became_peered":"2023-10-07T09:45:01.588233+0000","last_unstale":"2023-10-07T09:45:01.594462+0000","last_undegraded":"2023-10-07T09:45:01.594462+0000","last_fullsized":"2023-10-07T09:45:01.594462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:53:39.408523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599671+0000","last_change":"2023-10-07T09:45:01.594754+0000","last_active":"2023-10-07T09:45:01.599671+0000","last_peered":"2023-10-07T09:45:01.599671+0000","last_clean":"2023-10-07T09:45:01.599671+0000","last_became_active":"2023-10-07T09:45:01.594218+0000","last_became_peered":"2023-10-07T09:45:01.594218+0000","last_unstale":"2023-10-07T09:45:01.599671+0000","last_undegraded":"2023-10-07T09:45:01.599671+0000","last_fullsized":"2023-10-07T09:45:01.599671+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:08.408364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592908+0000","last_change":"2023-10-07T09:45:01.592908+0000","last_active":"2023-10-07T09:45:01.592908+0000","last_peered":"2023-10-07T09:45:01.592908+0000","last_clean":"2023-10-07T09:45:01.592908+0000","last_became_active":"2023-10-07T09:45:01.592631+0000","last_became_peered":"2023-10-07T09:45:01.592631+0000","last_unstale":"2023-10-07T09:45:01.592908+0000","last_undegraded":"2023-10-07T09:45:01.592908+0000","last_fullsized":"2023-10-07T09:45:01.592908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:39:15.461534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"89'4","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598430+0000","last_change":"2023-10-07T09:45:00.576204+0000","last_active":"2023-10-07T09:45:01.598430+0000","last_peered":"2023-10-07T09:45:01.598430+0000","last_clean":"2023-10-07T09:45:01.598430+0000","last_became_active":"2023-10-07T09:45:00.575922+0000","last_became_peered":"2023-10-07T09:45:00.575922+0000","last_unstale":"2023-10-07T09:45:01.598430+0000","last_undegraded":"2023-10-07T09:45:01.598430+0000","last_fullsized":"2023-10-07T09:45:01.598430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:10.222273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596304+0000","last_change":"2023-10-07T09:45:01.587158+0000","last_active":"2023-10-07T09:45:01.596304+0000","last_peered":"2023-10-07T09:45:01.596304+0000","last_clean":"2023-10-07T09:45:01.596304+0000","last_became_active":"2023-10-07T09:45:01.586833+0000","last_became_peered":"2023-10-07T09:45:01.586833+0000","last_unstale":"2023-10-07T09:45:01.596304+0000","last_undegraded":"2023-10-07T09:45:01.596304+0000","last_fullsized":"2023-10-07T09:45:01.596304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:27:33.602389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.587592+0000","last_change":"2023-10-07T09:45:00.582161+0000","last_active":"2023-10-07T09:45:01.587592+0000","last_peered":"2023-10-07T09:45:01.587592+0000","last_clean":"2023-10-07T09:45:01.587592+0000","last_became_active":"2023-10-07T09:45:00.581861+0000","last_became_peered":"2023-10-07T09:45:00.581861+0000","last_unstale":"2023-10-07T09:45:01.587592+0000","last_undegraded":"2023-10-07T09:45:01.587592+0000","last_fullsized":"2023-10-07T09:45:01.587592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:53.508191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591363+0000","last_change":"2023-10-07T09:45:01.584530+0000","last_active":"2023-10-07T09:45:01.591363+0000","last_peered":"2023-10-07T09:45:01.591363+0000","last_clean":"2023-10-07T09:45:01.591363+0000","last_became_active":"2023-10-07T09:45:01.584195+0000","last_became_peered":"2023-10-07T09:45:01.584195+0000","last_unstale":"2023-10-07T09:45:01.591363+0000","last_undegraded":"2023-10-07T09:45:01.591363+0000","last_fullsized":"2023-10-07T09:45:01.591363+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:11:21.387986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600699+0000","last_change":"2023-10-07T09:45:00.583570+0000","last_active":"2023-10-07T09:45:01.600699+0000","last_peered":"2023-10-07T09:45:01.600699+0000","last_clean":"2023-10-07T09:45:01.600699+0000","last_became_active":"2023-10-07T09:45:00.581893+0000","last_became_peered":"2023-10-07T09:45:00.581893+0000","last_unstale":"2023-10-07T09:45:01.600699+0000","last_undegraded":"2023-10-07T09:45:01.600699+0000","last_fullsized":"2023-10-07T09:45:01.600699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:44.723902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.584327+0000","last_change":"2023-10-07T09:45:01.584327+0000","last_active":"2023-10-07T09:45:01.584327+0000","last_peered":"2023-10-07T09:45:01.584327+0000","last_clean":"2023-10-07T09:45:01.584327+0000","last_became_active":"2023-10-07T09:45:01.584066+0000","last_became_peered":"2023-10-07T09:45:01.584066+0000","last_unstale":"2023-10-07T09:45:01.584327+0000","last_undegraded":"2023-10-07T09:45:01.584327+0000","last_fullsized":"2023-10-07T09:45:01.584327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:27:37.268229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597244+0000","last_change":"2023-10-07T09:45:00.586700+0000","last_active":"2023-10-07T09:45:01.597244+0000","last_peered":"2023-10-07T09:45:01.597244+0000","last_clean":"2023-10-07T09:45:01.597244+0000","last_became_active":"2023-10-07T09:45:00.586375+0000","last_became_peered":"2023-10-07T09:45:00.586375+0000","last_unstale":"2023-10-07T09:45:01.597244+0000","last_undegraded":"2023-10-07T09:45:01.597244+0000","last_fullsized":"2023-10-07T09:45:01.597244+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:02.087574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592558+0000","last_change":"2023-10-07T09:45:01.586781+0000","last_active":"2023-10-07T09:45:01.592558+0000","last_peered":"2023-10-07T09:45:01.592558+0000","last_clean":"2023-10-07T09:45:01.592558+0000","last_became_active":"2023-10-07T09:45:01.586521+0000","last_became_peered":"2023-10-07T09:45:01.586521+0000","last_unstale":"2023-10-07T09:45:01.592558+0000","last_undegraded":"2023-10-07T09:45:01.592558+0000","last_fullsized":"2023-10-07T09:45:01.592558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:39:05.423766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"89'1","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.565353+0000","last_change":"2023-10-07T09:45:00.573052+0000","last_active":"2023-10-07T09:45:01.565353+0000","last_peered":"2023-10-07T09:45:01.565353+0000","last_clean":"2023-10-07T09:45:01.565353+0000","last_became_active":"2023-10-07T09:45:00.572773+0000","last_became_peered":"2023-10-07T09:45:00.572773+0000","last_unstale":"2023-10-07T09:45:01.565353+0000","last_undegraded":"2023-10-07T09:45:01.565353+0000","last_fullsized":"2023-10-07T09:45:01.565353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:53:48.454334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594135+0000","last_change":"2023-10-07T09:45:01.581016+0000","last_active":"2023-10-07T09:45:01.594135+0000","last_peered":"2023-10-07T09:45:01.594135+0000","last_clean":"2023-10-07T09:45:01.594135+0000","last_became_active":"2023-10-07T09:45:01.580643+0000","last_became_peered":"2023-10-07T09:45:01.580643+0000","last_unstale":"2023-10-07T09:45:01.594135+0000","last_undegraded":"2023-10-07T09:45:01.594135+0000","last_fullsized":"2023-10-07T09:45:01.594135+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:11:21.387986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600115+0000","last_change":"2023-10-07T09:45:00.585520+0000","last_active":"2023-10-07T09:45:01.600115+0000","last_peered":"2023-10-07T09:45:01.600115+0000","last_clean":"2023-10-07T09:45:01.600115+0000","last_became_active":"2023-10-07T09:45:00.585216+0000","last_became_peered":"2023-10-07T09:45:00.585216+0000","last_unstale":"2023-10-07T09:45:01.600115+0000","last_undegraded":"2023-10-07T09:45:01.600115+0000","last_fullsized":"2023-10-07T09:45:01.600115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:33.528525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585902+0000","last_change":"2023-10-07T09:45:01.579739+0000","last_active":"2023-10-07T09:45:01.585902+0000","last_peered":"2023-10-07T09:45:01.585902+0000","last_clean":"2023-10-07T09:45:01.585902+0000","last_became_active":"2023-10-07T09:45:01.579468+0000","last_became_peered":"2023-10-07T09:45:01.579468+0000","last_unstale":"2023-10-07T09:45:01.585902+0000","last_undegraded":"2023-10-07T09:45:01.585902+0000","last_fullsized":"2023-10-07T09:45:01.585902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:38.079014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.574903+0000","last_change":"2023-10-07T09:45:00.580674+0000","last_active":"2023-10-07T09:45:01.574903+0000","last_peered":"2023-10-07T09:45:01.574903+0000","last_clean":"2023-10-07T09:45:01.574903+0000","last_became_active":"2023-10-07T09:45:00.580399+0000","last_became_peered":"2023-10-07T09:45:00.580399+0000","last_unstale":"2023-10-07T09:45:01.574903+0000","last_undegraded":"2023-10-07T09:45:01.574903+0000","last_fullsized":"2023-10-07T09:45:01.574903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:07:46.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.587891+0000","last_change":"2023-10-07T09:45:01.580745+0000","last_active":"2023-10-07T09:45:01.587891+0000","last_peered":"2023-10-07T09:45:01.587891+0000","last_clean":"2023-10-07T09:45:01.587891+0000","last_became_active":"2023-10-07T09:45:01.580485+0000","last_became_peered":"2023-10-07T09:45:01.580485+0000","last_unstale":"2023-10-07T09:45:01.587891+0000","last_undegraded":"2023-10-07T09:45:01.587891+0000","last_fullsized":"2023-10-07T09:45:01.587891+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:41.565669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571932+0000","last_change":"2023-10-07T09:45:00.575690+0000","last_active":"2023-10-07T09:45:01.571932+0000","last_peered":"2023-10-07T09:45:01.571932+0000","last_clean":"2023-10-07T09:45:01.571932+0000","last_became_active":"2023-10-07T09:45:00.575385+0000","last_became_peered":"2023-10-07T09:45:00.575385+0000","last_unstale":"2023-10-07T09:45:01.571932+0000","last_undegraded":"2023-10-07T09:45:01.571932+0000","last_fullsized":"2023-10-07T09:45:01.571932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:38:28.683977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596071+0000","last_change":"2023-10-07T09:45:01.589160+0000","last_active":"2023-10-07T09:45:01.596071+0000","last_peered":"2023-10-07T09:45:01.596071+0000","last_clean":"2023-10-07T09:45:01.596071+0000","last_became_active":"2023-10-07T09:45:01.588698+0000","last_became_peered":"2023-10-07T09:45:01.588698+0000","last_unstale":"2023-10-07T09:45:01.596071+0000","last_undegraded":"2023-10-07T09:45:01.596071+0000","last_fullsized":"2023-10-07T09:45:01.596071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:50:55.009515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.568487+0000","last_change":"2023-10-07T09:45:00.571961+0000","last_active":"2023-10-07T09:45:01.568487+0000","last_peered":"2023-10-07T09:45:01.568487+0000","last_clean":"2023-10-07T09:45:01.568487+0000","last_became_active":"2023-10-07T09:45:00.571678+0000","last_became_peered":"2023-10-07T09:45:00.571678+0000","last_unstale":"2023-10-07T09:45:01.568487+0000","last_undegraded":"2023-10-07T09:45:01.568487+0000","last_fullsized":"2023-10-07T09:45:01.568487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:53.508191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593855+0000","last_change":"2023-10-07T09:45:01.593855+0000","last_active":"2023-10-07T09:45:01.593855+0000","last_peered":"2023-10-07T09:45:01.593855+0000","last_clean":"2023-10-07T09:45:01.593855+0000","last_became_active":"2023-10-07T09:45:01.593545+0000","last_became_peered":"2023-10-07T09:45:01.593545+0000","last_unstale":"2023-10-07T09:45:01.593855+0000","last_undegraded":"2023-10-07T09:45:01.593855+0000","last_fullsized":"2023-10-07T09:45:01.593855+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:02:32.151024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.570091+0000","last_change":"2023-10-07T09:45:00.586979+0000","last_active":"2023-10-07T09:45:01.570091+0000","last_peered":"2023-10-07T09:45:01.570091+0000","last_clean":"2023-10-07T09:45:01.570091+0000","last_became_active":"2023-10-07T09:45:00.586666+0000","last_became_peered":"2023-10-07T09:45:00.586666+0000","last_unstale":"2023-10-07T09:45:01.570091+0000","last_undegraded":"2023-10-07T09:45:01.570091+0000","last_fullsized":"2023-10-07T09:45:01.570091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:50:23.900305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591340+0000","last_change":"2023-10-07T09:45:01.584409+0000","last_active":"2023-10-07T09:45:01.591340+0000","last_peered":"2023-10-07T09:45:01.591340+0000","last_clean":"2023-10-07T09:45:01.591340+0000","last_became_active":"2023-10-07T09:45:01.583873+0000","last_became_peered":"2023-10-07T09:45:01.583873+0000","last_unstale":"2023-10-07T09:45:01.591340+0000","last_undegraded":"2023-10-07T09:45:01.591340+0000","last_fullsized":"2023-10-07T09:45:01.591340+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:20.855523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"89'3","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594335+0000","last_change":"2023-10-07T09:45:00.580821+0000","last_active":"2023-10-07T09:45:01.594335+0000","last_peered":"2023-10-07T09:45:01.594335+0000","last_clean":"2023-10-07T09:45:01.594335+0000","last_became_active":"2023-10-07T09:45:00.580554+0000","last_became_peered":"2023-10-07T09:45:00.580554+0000","last_unstale":"2023-10-07T09:45:01.594335+0000","last_undegraded":"2023-10-07T09:45:01.594335+0000","last_fullsized":"2023-10-07T09:45:01.594335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:39:04.414277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595647+0000","last_change":"2023-10-07T09:45:01.584412+0000","last_active":"2023-10-07T09:45:01.595647+0000","last_peered":"2023-10-07T09:45:01.595647+0000","last_clean":"2023-10-07T09:45:01.595647+0000","last_became_active":"2023-10-07T09:45:01.584125+0000","last_became_peered":"2023-10-07T09:45:01.584125+0000","last_unstale":"2023-10-07T09:45:01.595647+0000","last_undegraded":"2023-10-07T09:45:01.595647+0000","last_fullsized":"2023-10-07T09:45:01.595647+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:05:16.984744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"89'6","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600950+0000","last_change":"2023-10-07T09:45:00.581873+0000","last_active":"2023-10-07T09:45:01.600950+0000","last_peered":"2023-10-07T09:45:01.600950+0000","last_clean":"2023-10-07T09:45:01.600950+0000","last_became_active":"2023-10-07T09:45:00.581609+0000","last_became_peered":"2023-10-07T09:45:00.581609+0000","last_unstale":"2023-10-07T09:45:01.600950+0000","last_undegraded":"2023-10-07T09:45:01.600950+0000","last_fullsized":"2023-10-07T09:45:01.600950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:54:47.096630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.587985+0000","last_change":"2023-10-07T09:45:01.587985+0000","last_active":"2023-10-07T09:45:01.587985+0000","last_peered":"2023-10-07T09:45:01.587985+0000","last_clean":"2023-10-07T09:45:01.587985+0000","last_became_active":"2023-10-07T09:45:01.587733+0000","last_became_peered":"2023-10-07T09:45:01.587733+0000","last_unstale":"2023-10-07T09:45:01.587985+0000","last_undegraded":"2023-10-07T09:45:01.587985+0000","last_fullsized":"2023-10-07T09:45:01.587985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:22:24.046350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599119+0000","last_change":"2023-10-07T09:45:00.576696+0000","last_active":"2023-10-07T09:45:01.599119+0000","last_peered":"2023-10-07T09:45:01.599119+0000","last_clean":"2023-10-07T09:45:01.599119+0000","last_became_active":"2023-10-07T09:45:00.576403+0000","last_became_peered":"2023-10-07T09:45:00.576403+0000","last_unstale":"2023-10-07T09:45:01.599119+0000","last_undegraded":"2023-10-07T09:45:01.599119+0000","last_fullsized":"2023-10-07T09:45:01.599119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:18:04.726202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600151+0000","last_change":"2023-10-07T09:45:01.597092+0000","last_active":"2023-10-07T09:45:01.600151+0000","last_peered":"2023-10-07T09:45:01.600151+0000","last_clean":"2023-10-07T09:45:01.600151+0000","last_became_active":"2023-10-07T09:45:01.596616+0000","last_became_peered":"2023-10-07T09:45:01.596616+0000","last_unstale":"2023-10-07T09:45:01.600151+0000","last_undegraded":"2023-10-07T09:45:01.600151+0000","last_fullsized":"2023-10-07T09:45:01.600151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:59:54.880296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.570978+0000","last_change":"2023-10-07T09:45:00.586049+0000","last_active":"2023-10-07T09:45:01.570978+0000","last_peered":"2023-10-07T09:45:01.570978+0000","last_clean":"2023-10-07T09:45:01.570978+0000","last_became_active":"2023-10-07T09:45:00.585796+0000","last_became_peered":"2023-10-07T09:45:00.585796+0000","last_unstale":"2023-10-07T09:45:01.570978+0000","last_undegraded":"2023-10-07T09:45:01.570978+0000","last_fullsized":"2023-10-07T09:45:01.570978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:53.508191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598599+0000","last_change":"2023-10-07T09:45:01.593370+0000","last_active":"2023-10-07T09:45:01.598599+0000","last_peered":"2023-10-07T09:45:01.598599+0000","last_clean":"2023-10-07T09:45:01.598599+0000","last_became_active":"2023-10-07T09:45:01.593098+0000","last_became_peered":"2023-10-07T09:45:01.593098+0000","last_unstale":"2023-10-07T09:45:01.598599+0000","last_undegraded":"2023-10-07T09:45:01.598599+0000","last_fullsized":"2023-10-07T09:45:01.598599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:25:55.411251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.576615+0000","last_change":"2023-10-07T09:45:00.585097+0000","last_active":"2023-10-07T09:45:01.576615+0000","last_peered":"2023-10-07T09:45:01.576615+0000","last_clean":"2023-10-07T09:45:01.576615+0000","last_became_active":"2023-10-07T09:45:00.584783+0000","last_became_peered":"2023-10-07T09:45:00.584783+0000","last_unstale":"2023-10-07T09:45:01.576615+0000","last_undegraded":"2023-10-07T09:45:01.576615+0000","last_fullsized":"2023-10-07T09:45:01.576615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:09.915300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598700+0000","last_change":"2023-10-07T09:45:01.596654+0000","last_active":"2023-10-07T09:45:01.598700+0000","last_peered":"2023-10-07T09:45:01.598700+0000","last_clean":"2023-10-07T09:45:01.598700+0000","last_became_active":"2023-10-07T09:45:01.596415+0000","last_became_peered":"2023-10-07T09:45:01.596415+0000","last_unstale":"2023-10-07T09:45:01.598700+0000","last_undegraded":"2023-10-07T09:45:01.598700+0000","last_fullsized":"2023-10-07T09:45:01.598700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:54:48.106119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"89'8","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.602023+0000","last_change":"2023-10-07T09:45:00.576661+0000","last_active":"2023-10-07T09:45:01.602023+0000","last_peered":"2023-10-07T09:45:01.602023+0000","last_clean":"2023-10-07T09:45:01.602023+0000","last_became_active":"2023-10-07T09:45:00.576372+0000","last_became_peered":"2023-10-07T09:45:00.576372+0000","last_unstale":"2023-10-07T09:45:01.602023+0000","last_undegraded":"2023-10-07T09:45:01.602023+0000","last_fullsized":"2023-10-07T09:45:01.602023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:32.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593263+0000","last_change":"2023-10-07T09:45:01.585499+0000","last_active":"2023-10-07T09:45:01.593263+0000","last_peered":"2023-10-07T09:45:01.593263+0000","last_clean":"2023-10-07T09:45:01.593263+0000","last_became_active":"2023-10-07T09:45:01.585181+0000","last_became_peered":"2023-10-07T09:45:01.585181+0000","last_unstale":"2023-10-07T09:45:01.593263+0000","last_undegraded":"2023-10-07T09:45:01.593263+0000","last_fullsized":"2023-10-07T09:45:01.593263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:54:49.662078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599438+0000","last_change":"2023-10-07T09:45:00.583493+0000","last_active":"2023-10-07T09:45:01.599438+0000","last_peered":"2023-10-07T09:45:01.599438+0000","last_clean":"2023-10-07T09:45:01.599438+0000","last_became_active":"2023-10-07T09:45:00.583057+0000","last_became_peered":"2023-10-07T09:45:00.583057+0000","last_unstale":"2023-10-07T09:45:01.599438+0000","last_undegraded":"2023-10-07T09:45:01.599438+0000","last_fullsized":"2023-10-07T09:45:01.599438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:32.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598705+0000","last_change":"2023-10-07T09:45:01.598705+0000","last_active":"2023-10-07T09:45:01.598705+0000","last_peered":"2023-10-07T09:45:01.598705+0000","last_clean":"2023-10-07T09:45:01.598705+0000","last_became_active":"2023-10-07T09:45:01.598205+0000","last_became_peered":"2023-10-07T09:45:01.598205+0000","last_unstale":"2023-10-07T09:45:01.598705+0000","last_undegraded":"2023-10-07T09:45:01.598705+0000","last_fullsized":"2023-10-07T09:45:01.598705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:27:39.563996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.601181+0000","last_change":"2023-10-07T09:45:00.584038+0000","last_active":"2023-10-07T09:45:01.601181+0000","last_peered":"2023-10-07T09:45:01.601181+0000","last_clean":"2023-10-07T09:45:01.601181+0000","last_became_active":"2023-10-07T09:45:00.583785+0000","last_became_peered":"2023-10-07T09:45:00.583785+0000","last_unstale":"2023-10-07T09:45:01.601181+0000","last_undegraded":"2023-10-07T09:45:01.601181+0000","last_fullsized":"2023-10-07T09:45:01.601181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:02.087574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594661+0000","last_change":"2023-10-07T09:45:01.588280+0000","last_active":"2023-10-07T09:45:01.594661+0000","last_peered":"2023-10-07T09:45:01.594661+0000","last_clean":"2023-10-07T09:45:01.594661+0000","last_became_active":"2023-10-07T09:45:01.587537+0000","last_became_peered":"2023-10-07T09:45:01.587537+0000","last_unstale":"2023-10-07T09:45:01.594661+0000","last_undegraded":"2023-10-07T09:45:01.594661+0000","last_fullsized":"2023-10-07T09:45:01.594661+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:54.517679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599660+0000","last_change":"2023-10-07T09:45:00.580412+0000","last_active":"2023-10-07T09:45:01.599660+0000","last_peered":"2023-10-07T09:45:01.599660+0000","last_clean":"2023-10-07T09:45:01.599660+0000","last_became_active":"2023-10-07T09:45:00.580147+0000","last_became_peered":"2023-10-07T09:45:00.580147+0000","last_unstale":"2023-10-07T09:45:01.599660+0000","last_undegraded":"2023-10-07T09:45:01.599660+0000","last_fullsized":"2023-10-07T09:45:01.599660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:07:46.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595667+0000","last_change":"2023-10-07T09:45:01.592951+0000","last_active":"2023-10-07T09:45:01.595667+0000","last_peered":"2023-10-07T09:45:01.595667+0000","last_clean":"2023-10-07T09:45:01.595667+0000","last_became_active":"2023-10-07T09:45:01.592702+0000","last_became_peered":"2023-10-07T09:45:01.592702+0000","last_unstale":"2023-10-07T09:45:01.595667+0000","last_undegraded":"2023-10-07T09:45:01.595667+0000","last_fullsized":"2023-10-07T09:45:01.595667+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:18:05.735691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-07T09:45:00.585110+0000","last_change":"2023-10-07T09:45:00.580718+0000","last_active":"2023-10-07T09:45:00.585110+0000","last_peered":"2023-10-07T09:45:00.585110+0000","last_clean":"2023-10-07T09:45:00.585110+0000","last_became_active":"2023-10-07T09:45:00.580418+0000","last_became_peered":"2023-10-07T09:45:00.580418+0000","last_unstale":"2023-10-07T09:45:00.585110+0000","last_undegraded":"2023-10-07T09:45:00.585110+0000","last_fullsized":"2023-10-07T09:45:00.585110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:00:25.326228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.580272+0000","last_change":"2023-10-07T09:45:01.580272+0000","last_active":"2023-10-07T09:45:01.580272+0000","last_peered":"2023-10-07T09:45:01.580272+0000","last_clean":"2023-10-07T09:45:01.580272+0000","last_became_active":"2023-10-07T09:45:01.579520+0000","last_became_peered":"2023-10-07T09:45:01.579520+0000","last_unstale":"2023-10-07T09:45:01.580272+0000","last_undegraded":"2023-10-07T09:45:01.580272+0000","last_fullsized":"2023-10-07T09:45:01.580272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:22:26.957532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571084+0000","last_change":"2023-10-07T09:45:00.584903+0000","last_active":"2023-10-07T09:45:01.571084+0000","last_peered":"2023-10-07T09:45:01.571084+0000","last_clean":"2023-10-07T09:45:01.571084+0000","last_became_active":"2023-10-07T09:45:00.582723+0000","last_became_peered":"2023-10-07T09:45:00.582723+0000","last_unstale":"2023-10-07T09:45:01.571084+0000","last_undegraded":"2023-10-07T09:45:01.571084+0000","last_fullsized":"2023-10-07T09:45:01.571084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:05:26.598159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594634+0000","last_change":"2023-10-07T09:45:01.594634+0000","last_active":"2023-10-07T09:45:01.594634+0000","last_peered":"2023-10-07T09:45:01.594634+0000","last_clean":"2023-10-07T09:45:01.594634+0000","last_became_active":"2023-10-07T09:45:01.594332+0000","last_became_peered":"2023-10-07T09:45:01.594332+0000","last_unstale":"2023-10-07T09:45:01.594634+0000","last_undegraded":"2023-10-07T09:45:01.594634+0000","last_fullsized":"2023-10-07T09:45:01.594634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:11.518340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.576716+0000","last_change":"2023-10-07T09:45:00.582709+0000","last_active":"2023-10-07T09:45:01.576716+0000","last_peered":"2023-10-07T09:45:01.576716+0000","last_clean":"2023-10-07T09:45:01.576716+0000","last_became_active":"2023-10-07T09:45:00.582450+0000","last_became_peered":"2023-10-07T09:45:00.582450+0000","last_unstale":"2023-10-07T09:45:01.576716+0000","last_undegraded":"2023-10-07T09:45:01.576716+0000","last_fullsized":"2023-10-07T09:45:01.576716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:54:48.652589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585101+0000","last_change":"2023-10-07T09:45:01.585101+0000","last_active":"2023-10-07T09:45:01.585101+0000","last_peered":"2023-10-07T09:45:01.585101+0000","last_clean":"2023-10-07T09:45:01.585101+0000","last_became_active":"2023-10-07T09:45:01.584797+0000","last_became_peered":"2023-10-07T09:45:01.584797+0000","last_unstale":"2023-10-07T09:45:01.585101+0000","last_undegraded":"2023-10-07T09:45:01.585101+0000","last_fullsized":"2023-10-07T09:45:01.585101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:40.137616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.573290+0000","last_change":"2023-10-07T09:45:00.581931+0000","last_active":"2023-10-07T09:45:01.573290+0000","last_peered":"2023-10-07T09:45:01.573290+0000","last_clean":"2023-10-07T09:45:01.573290+0000","last_became_active":"2023-10-07T09:45:00.581669+0000","last_became_peered":"2023-10-07T09:45:00.581669+0000","last_unstale":"2023-10-07T09:45:01.573290+0000","last_undegraded":"2023-10-07T09:45:01.573290+0000","last_fullsized":"2023-10-07T09:45:01.573290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:44.723902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597300+0000","last_change":"2023-10-07T09:45:01.588188+0000","last_active":"2023-10-07T09:45:01.597300+0000","last_peered":"2023-10-07T09:45:01.597300+0000","last_clean":"2023-10-07T09:45:01.597300+0000","last_became_active":"2023-10-07T09:45:01.587895+0000","last_became_peered":"2023-10-07T09:45:01.587895+0000","last_unstale":"2023-10-07T09:45:01.597300+0000","last_undegraded":"2023-10-07T09:45:01.597300+0000","last_fullsized":"2023-10-07T09:45:01.597300+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:10.924789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599332+0000","last_change":"2023-10-07T09:45:00.583324+0000","last_active":"2023-10-07T09:45:01.599332+0000","last_peered":"2023-10-07T09:45:01.599332+0000","last_clean":"2023-10-07T09:45:01.599332+0000","last_became_active":"2023-10-07T09:45:00.583042+0000","last_became_peered":"2023-10-07T09:45:00.583042+0000","last_unstale":"2023-10-07T09:45:01.599332+0000","last_undegraded":"2023-10-07T09:45:01.599332+0000","last_fullsized":"2023-10-07T09:45:01.599332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:33:52.065837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593403+0000","last_change":"2023-10-07T09:45:01.588053+0000","last_active":"2023-10-07T09:45:01.593403+0000","last_peered":"2023-10-07T09:45:01.593403+0000","last_clean":"2023-10-07T09:45:01.593403+0000","last_became_active":"2023-10-07T09:45:01.587780+0000","last_became_peered":"2023-10-07T09:45:01.587780+0000","last_unstale":"2023-10-07T09:45:01.593403+0000","last_undegraded":"2023-10-07T09:45:01.593403+0000","last_fullsized":"2023-10-07T09:45:01.593403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:12:49.510761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.579772+0000","last_change":"2023-10-07T09:45:00.580701+0000","last_active":"2023-10-07T09:45:01.579772+0000","last_peered":"2023-10-07T09:45:01.579772+0000","last_clean":"2023-10-07T09:45:01.579772+0000","last_became_active":"2023-10-07T09:45:00.579652+0000","last_became_peered":"2023-10-07T09:45:00.579652+0000","last_unstale":"2023-10-07T09:45:01.579772+0000","last_undegraded":"2023-10-07T09:45:01.579772+0000","last_fullsized":"2023-10-07T09:45:01.579772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:40.556180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.586010+0000","last_change":"2023-10-07T09:45:01.586010+0000","last_active":"2023-10-07T09:45:01.586010+0000","last_peered":"2023-10-07T09:45:01.586010+0000","last_clean":"2023-10-07T09:45:01.586010+0000","last_became_active":"2023-10-07T09:45:01.585668+0000","last_became_peered":"2023-10-07T09:45:01.585668+0000","last_unstale":"2023-10-07T09:45:01.586010+0000","last_undegraded":"2023-10-07T09:45:01.586010+0000","last_fullsized":"2023-10-07T09:45:01.586010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:02.998961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571724+0000","last_change":"2023-10-07T09:45:00.581928+0000","last_active":"2023-10-07T09:45:01.571724+0000","last_peered":"2023-10-07T09:45:01.571724+0000","last_clean":"2023-10-07T09:45:01.571724+0000","last_became_active":"2023-10-07T09:45:00.581654+0000","last_became_peered":"2023-10-07T09:45:00.581654+0000","last_unstale":"2023-10-07T09:45:01.571724+0000","last_undegraded":"2023-10-07T09:45:01.571724+0000","last_fullsized":"2023-10-07T09:45:01.571724+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:42:52.201833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591508+0000","last_change":"2023-10-07T09:45:01.591508+0000","last_active":"2023-10-07T09:45:01.591508+0000","last_peered":"2023-10-07T09:45:01.591508+0000","last_clean":"2023-10-07T09:45:01.591508+0000","last_became_active":"2023-10-07T09:45:01.590577+0000","last_became_peered":"2023-10-07T09:45:01.590577+0000","last_unstale":"2023-10-07T09:45:01.591508+0000","last_undegraded":"2023-10-07T09:45:01.591508+0000","last_fullsized":"2023-10-07T09:45:01.591508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:16:49.958129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.573806+0000","last_change":"2023-10-07T09:45:00.585367+0000","last_active":"2023-10-07T09:45:01.573806+0000","last_peered":"2023-10-07T09:45:01.573806+0000","last_clean":"2023-10-07T09:45:01.573806+0000","last_became_active":"2023-10-07T09:45:00.584978+0000","last_became_peered":"2023-10-07T09:45:00.584978+0000","last_unstale":"2023-10-07T09:45:01.573806+0000","last_undegraded":"2023-10-07T09:45:01.573806+0000","last_fullsized":"2023-10-07T09:45:01.573806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:32.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595432+0000","last_change":"2023-10-07T09:45:01.586702+0000","last_active":"2023-10-07T09:45:01.595432+0000","last_peered":"2023-10-07T09:45:01.595432+0000","last_clean":"2023-10-07T09:45:01.595432+0000","last_became_active":"2023-10-07T09:45:01.586414+0000","last_became_peered":"2023-10-07T09:45:01.586414+0000","last_unstale":"2023-10-07T09:45:01.595432+0000","last_undegraded":"2023-10-07T09:45:01.595432+0000","last_fullsized":"2023-10-07T09:45:01.595432+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:47:25.094133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571496+0000","last_change":"2023-10-07T09:45:00.586571+0000","last_active":"2023-10-07T09:45:01.571496+0000","last_peered":"2023-10-07T09:45:01.571496+0000","last_clean":"2023-10-07T09:45:01.571496+0000","last_became_active":"2023-10-07T09:45:00.586267+0000","last_became_peered":"2023-10-07T09:45:00.586267+0000","last_unstale":"2023-10-07T09:45:01.571496+0000","last_undegraded":"2023-10-07T09:45:01.571496+0000","last_fullsized":"2023-10-07T09:45:01.571496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:19.523433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591995+0000","last_change":"2023-10-07T09:45:01.585383+0000","last_active":"2023-10-07T09:45:01.591995+0000","last_peered":"2023-10-07T09:45:01.591995+0000","last_clean":"2023-10-07T09:45:01.591995+0000","last_became_active":"2023-10-07T09:45:01.585125+0000","last_became_peered":"2023-10-07T09:45:01.585125+0000","last_unstale":"2023-10-07T09:45:01.591995+0000","last_undegraded":"2023-10-07T09:45:01.591995+0000","last_fullsized":"2023-10-07T09:45:01.591995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:58:04.424056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.574688+0000","last_change":"2023-10-07T09:45:00.580812+0000","last_active":"2023-10-07T09:45:01.574688+0000","last_peered":"2023-10-07T09:45:01.574688+0000","last_clean":"2023-10-07T09:45:01.574688+0000","last_became_active":"2023-10-07T09:45:00.580488+0000","last_became_peered":"2023-10-07T09:45:00.580488+0000","last_unstale":"2023-10-07T09:45:01.574688+0000","last_undegraded":"2023-10-07T09:45:01.574688+0000","last_fullsized":"2023-10-07T09:45:01.574688+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:37.069525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.590395+0000","last_change":"2023-10-07T09:45:01.586475+0000","last_active":"2023-10-07T09:45:01.590395+0000","last_peered":"2023-10-07T09:45:01.590395+0000","last_clean":"2023-10-07T09:45:01.590395+0000","last_became_active":"2023-10-07T09:45:01.586176+0000","last_became_peered":"2023-10-07T09:45:01.586176+0000","last_unstale":"2023-10-07T09:45:01.590395+0000","last_undegraded":"2023-10-07T09:45:01.590395+0000","last_fullsized":"2023-10-07T09:45:01.590395+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:33.956834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.567480+0000","last_change":"2023-10-07T09:45:00.579555+0000","last_active":"2023-10-07T09:45:01.567480+0000","last_peered":"2023-10-07T09:45:01.567480+0000","last_clean":"2023-10-07T09:45:01.567480+0000","last_became_active":"2023-10-07T09:45:00.579304+0000","last_became_peered":"2023-10-07T09:45:00.579304+0000","last_unstale":"2023-10-07T09:45:01.567480+0000","last_undegraded":"2023-10-07T09:45:01.567480+0000","last_fullsized":"2023-10-07T09:45:01.567480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:33:52.065837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599581+0000","last_change":"2023-10-07T09:45:01.593507+0000","last_active":"2023-10-07T09:45:01.599581+0000","last_peered":"2023-10-07T09:45:01.599581+0000","last_clean":"2023-10-07T09:45:01.599581+0000","last_became_active":"2023-10-07T09:45:01.593245+0000","last_became_peered":"2023-10-07T09:45:01.593245+0000","last_unstale":"2023-10-07T09:45:01.599581+0000","last_undegraded":"2023-10-07T09:45:01.599581+0000","last_fullsized":"2023-10-07T09:45:01.599581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:08:28.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.572385+0000","last_change":"2023-10-07T09:45:00.586526+0000","last_active":"2023-10-07T09:45:01.572385+0000","last_peered":"2023-10-07T09:45:01.572385+0000","last_clean":"2023-10-07T09:45:01.572385+0000","last_became_active":"2023-10-07T09:45:00.586217+0000","last_became_peered":"2023-10-07T09:45:00.586217+0000","last_unstale":"2023-10-07T09:45:01.572385+0000","last_undegraded":"2023-10-07T09:45:01.572385+0000","last_fullsized":"2023-10-07T09:45:01.572385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:44.723902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585426+0000","last_change":"2023-10-07T09:45:01.585426+0000","last_active":"2023-10-07T09:45:01.585426+0000","last_peered":"2023-10-07T09:45:01.585426+0000","last_clean":"2023-10-07T09:45:01.585426+0000","last_became_active":"2023-10-07T09:45:01.585020+0000","last_became_peered":"2023-10-07T09:45:01.585020+0000","last_unstale":"2023-10-07T09:45:01.585426+0000","last_undegraded":"2023-10-07T09:45:01.585426+0000","last_fullsized":"2023-10-07T09:45:01.585426+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:54:31.609831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599585+0000","last_change":"2023-10-07T09:45:00.584131+0000","last_active":"2023-10-07T09:45:01.599585+0000","last_peered":"2023-10-07T09:45:01.599585+0000","last_clean":"2023-10-07T09:45:01.599585+0000","last_became_active":"2023-10-07T09:45:00.583844+0000","last_became_peered":"2023-10-07T09:45:00.583844+0000","last_unstale":"2023-10-07T09:45:01.599585+0000","last_undegraded":"2023-10-07T09:45:01.599585+0000","last_fullsized":"2023-10-07T09:45:01.599585+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:19:00.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"80'183","reported_seq":454,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-07T09:45:00.566197+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:45:00.566197+0000","last_peered":"2023-10-07T09:45:00.566197+0000","last_clean":"2023-10-07T09:45:00.566197+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:45:00.566197+0000","last_undegraded":"2023-10-07T09:45:00.566197+0000","last_fullsized":"2023-10-07T09:45:00.566197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303024,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068816,"statfs":{"total":96636764160,"available":96326467584,"internally_reserved":0,"allocated":1679360,"data_stored":1101169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68799999999999994,"15min":0.71199999999999997},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.32300000000000001},"max":{"1min":0.95199999999999996,"5min":0.998,"15min":1.026},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.434,"5min":0.42999999999999999,"15min":0.374},"max":{"1min":0.94899999999999995,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66800000000000004}]},{"osd":1,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71999999999999997,"15min":0.72299999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":0.93300000000000005,"5min":1.014,"15min":1.157},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45600000000000002},"max":{"1min":0.875,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.40699999999999997},"max":{"1min":0.99199999999999999,"5min":1.206,"15min":1.206},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72399999999999998,"15min":0.73599999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.439},"max":{"1min":0.91200000000000003,"5min":1.071,"15min":1.071},"last":0.75700000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71599999999999997,"15min":0.73599999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88900000000000001,"5min":0.96899999999999997,"15min":1.04},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.76900000000000002},"min":{"1min":0.64300000000000002,"5min":0.53000000000000003,"15min":0.371},"max":{"1min":0.96099999999999997,"5min":1.137,"15min":1.3440000000000001},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39100000000000001},"max":{"1min":0.85499999999999998,"5min":0.88700000000000001,"15min":1.992},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.48699999999999999,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.6240000000000001},"last":0.89900000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64700000000000002,"15min":0.66000000000000003},"min":{"1min":0.42699999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85899999999999999,"5min":1.27,"15min":1.27},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.1399999999999999},"last":0.88100000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.46400000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71299999999999997,"15min":0.72399999999999998},"min":{"1min":0.52100000000000002,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.107},"last":0.95599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509630,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302444,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069396,"statfs":{"total":96636764160,"available":96327061504,"internally_reserved":0,"allocated":1085440,"data_stored":510305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52400000000000002,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":1.274,"5min":1.274,"15min":2.984},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.6000000000000001},"last":0.95699999999999996}]},{"osd":1,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.46300000000000002,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":1.169,"5min":1.169,"15min":2.4529999999999998},"last":1.042},{"interface":"front","average":{"1min":0.749,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.55400000000000005,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.7,"5min":1.7,"15min":1.754},"last":1.02}]},{"osd":2,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.75600000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.44400000000000001},"max":{"1min":1.482,"5min":1.482,"15min":7.867},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.75800000000000001,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.643,"5min":1.643,"15min":7.7629999999999999},"last":0.92400000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.747,"15min":0.78500000000000003},"min":{"1min":0.53200000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.458,"5min":1.458,"15min":8.0399999999999991},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.54800000000000004,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":2.7949999999999999},"last":1.006}]},{"osd":4,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.623,"15min":0.624},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.29899999999999999},"max":{"1min":0.80500000000000005,"5min":0.92400000000000004,"15min":1.296},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.371},"max":{"1min":0.84799999999999998,"5min":0.97899999999999998,"15min":1.105},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.41199999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.4119999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.51400000000000001,"5min":0.49399999999999999,"15min":0.47999999999999998},"max":{"1min":0.98999999999999999,"5min":1.0089999999999999,"15min":1.1779999999999999},"last":0.79100000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.39700000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.5049999999999999},"last":1.099},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.751,"15min":0.76600000000000001},"min":{"1min":0.59099999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91100000000000003,"5min":0.995,"15min":1.3220000000000001},"last":0.745}]}]},{"osd":1,"up_from":14,"seq":60129542332,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":1110016,"data_stored":519001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71799999999999997,"5min":0.746,"15min":1.585},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.372,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.65400000000000003,"5min":0.79800000000000004,"15min":1.6080000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.45400000000000001,"15min":0.47999999999999998},"min":{"1min":0.32600000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.55000000000000004,"5min":0.621,"15min":1.468},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68000000000000005,"5min":0.78400000000000003,"15min":1.5389999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.59699999999999998},"min":{"1min":0.42399999999999999,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.75900000000000001,"15min":1.663},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52500000000000002,"15min":0.54700000000000004},"min":{"1min":0.39100000000000001,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.751,"5min":0.80700000000000005,"15min":1.1140000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.75600000000000001,"15min":0.78200000000000003},"min":{"1min":0.54300000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73899999999999999,"15min":0.77500000000000002},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98399999999999999,"5min":1.032,"15min":1.766},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.96999999999999997,"5min":1.099,"15min":1.3320000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.749,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.93400000000000005,"5min":1.0680000000000001,"15min":1.5149999999999999},"last":0.61399999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.81599999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.95199999999999996,"5min":1.0109999999999999,"15min":1.627},"last":0.746},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.78300000000000003,"15min":0.82399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.3700000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.79600000000000004,"15min":0.83799999999999997},"min":{"1min":0.55400000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.706},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.79800000000000004,"15min":0.84399999999999997},"min":{"1min":0.53900000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0149999999999999,"5min":1.137,"15min":1.554},"last":0.64500000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542332,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303028,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068812,"statfs":{"total":96636764160,"available":96326463488,"internally_reserved":0,"allocated":1683456,"data_stored":1107245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48899999999999999,"15min":0.502},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65700000000000003,"5min":0.73099999999999998,"15min":2.726},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.311,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":0.84599999999999997,"15min":2.552},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53300000000000003,"15min":0.55200000000000005},"min":{"1min":0.41599999999999998,"5min":0.33800000000000002,"15min":0.32800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":2.4910000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.502,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.67100000000000004,"5min":0.80900000000000005,"15min":3.629},"last":0.45600000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.33900000000000002},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":2.6909999999999998},"last":0.432},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59899999999999998,"15min":0.61899999999999999},"min":{"1min":0.45800000000000002,"5min":0.42199999999999999,"15min":0.40200000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":2.5209999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.52500000000000002,"5min":0.497,"15min":0.45900000000000002},"max":{"1min":0.89700000000000002,"5min":1.232,"15min":2.637},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":2.4180000000000001},"last":0.86699999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.77600000000000002,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":3.8199999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.80900000000000005},"min":{"1min":0.55900000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91100000000000003,"5min":1.194,"15min":2.6000000000000001},"last":0.752}]},{"osd":6,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.98799999999999999,"5min":1.129,"15min":2.3740000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80200000000000005,"15min":0.82999999999999996},"min":{"1min":0.53300000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.95299999999999996,"5min":1.206,"15min":3.855},"last":0.88400000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81399999999999995,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.57499999999999996,"15min":0.56200000000000006},"max":{"1min":0.97599999999999998,"5min":1.167,"15min":2.4550000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81699999999999995,"15min":0.84899999999999998},"min":{"1min":0.56499999999999995,"5min":0.503,"15min":0.503},"max":{"1min":0.98999999999999999,"5min":1.238,"15min":3.778},"last":0.79100000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542333,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310652,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061188,"statfs":{"total":96636764160,"available":96318656512,"internally_reserved":0,"allocated":1101824,"data_stored":518413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.441,"15min":0.44900000000000001},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.63700000000000001,"5min":0.81000000000000005,"15min":1.258},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":0.80100000000000005,"15min":1.2290000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":1.1950000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":1.3200000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.501,"15min":0.51500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.94599999999999995,"15min":1.5820000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.69399999999999995,"5min":0.81399999999999995,"15min":1.052},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73399999999999999,"15min":0.75700000000000001},"min":{"1min":0.48099999999999998,"5min":0.46200000000000002,"15min":0.35699999999999998},"max":{"1min":0.81000000000000005,"5min":1.0660000000000001,"15min":1.4630000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93500000000000005,"5min":1.98,"15min":1.98},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.52500000000000002,"5min":0.52400000000000002,"15min":0.46100000000000002},"max":{"1min":0.88800000000000001,"5min":1.208,"15min":1.5169999999999999},"last":0.83099999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":0.877,"5min":1.931,"15min":1.931},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.59099999999999997,"5min":0.52800000000000002,"15min":0.47299999999999998},"max":{"1min":0.93700000000000006,"5min":1.306,"15min":1.8120000000000001},"last":0.92600000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.83899999999999997},"min":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.54200000000000004},"max":{"1min":0.89800000000000002,"5min":1.3520000000000001,"15min":1.3839999999999999},"last":0.877}]}]},{"osd":3,"up_from":14,"seq":60129542334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306540,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065300,"statfs":{"total":96636764160,"available":96322867200,"internally_reserved":0,"allocated":1085440,"data_stored":510325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":0.71599999999999997,"5min":0.74099999999999999,"15min":3.3980000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.35399999999999998,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":3.274},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32300000000000001},"max":{"1min":0.64900000000000002,"5min":0.73499999999999999,"15min":3.1299999999999999},"last":0.437},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.88,"5min":0.88,"15min":3.4500000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53300000000000003,"15min":0.55600000000000005},"min":{"1min":0.42699999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":3.2309999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56699999999999995,"15min":0.57399999999999995},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.377},"max":{"1min":0.70799999999999996,"5min":0.79800000000000004,"15min":3.0910000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.75600000000000001,"15min":0.77300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.44900000000000001},"max":{"1min":0.94999999999999996,"5min":1.0389999999999999,"15min":1.518},"last":0.628},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.78600000000000003},"min":{"1min":0.60199999999999998,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":1.099,"5min":1.103,"15min":1.3799999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.57399999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0629999999999999,"5min":1.246,"15min":1.3240000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78300000000000003,"15min":0.79200000000000004},"min":{"1min":0.60599999999999998,"5min":0.53200000000000003,"15min":0.50600000000000001},"max":{"1min":1.024,"5min":1.0640000000000001,"15min":1.5189999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.79400000000000004,"15min":0.80600000000000005},"min":{"1min":0.61899999999999999,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.123,"5min":1.202,"15min":1.413},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80000000000000004,"15min":0.82199999999999995},"min":{"1min":0.63400000000000001,"5min":0.58399999999999996,"15min":0.51000000000000001},"max":{"1min":1.046,"5min":1.1459999999999999,"15min":1.5840000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.58499999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.026,"5min":1.143,"15min":1.45},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0089999999999999,"5min":1.1040000000000001,"15min":1.5940000000000001},"last":0.73899999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509630,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306548,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065292,"statfs":{"total":96636764160,"available":96322859008,"internally_reserved":0,"allocated":1093632,"data_stored":516947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.86499999999999999,"5min":1.069,"15min":1.069},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.54900000000000004,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":2.3420000000000001},"last":0.66200000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.70899999999999996},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":1.054},"last":0.874},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.72499999999999998},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.35299999999999998},"max":{"1min":0.96199999999999997,"5min":1.478,"15min":1.478},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.75700000000000001,"15min":0.747},"min":{"1min":0.54000000000000004,"5min":0.48799999999999999,"15min":0.432},"max":{"1min":0.996,"5min":1.518,"15min":2.3239999999999998},"last":0.621},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70199999999999996,"15min":0.71199999999999997},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.84199999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84599999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.71299999999999997,"15min":0.71799999999999997},"min":{"1min":0.53200000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.80000000000000004,"5min":0.98699999999999999,"15min":1.032},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.747,"15min":0.751},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.496},"max":{"1min":0.96099999999999997,"5min":1.036,"15min":1.036},"last":0.76000000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.40799999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81000000000000005,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76900000000000002,"5min":1.639,"15min":1.639},"last":0.68300000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.81699999999999995,"5min":1.585,"15min":1.585},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.443},"max":{"1min":0.86899999999999999,"5min":1.024,"15min":1.0589999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.52700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88400000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.79600000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542332,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":1093632,"data_stored":516967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76300000000000001,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.49199999999999999,"15min":0.39900000000000002},"max":{"1min":0.94999999999999996,"5min":1.218,"15min":1.218},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.436},"max":{"1min":0.95499999999999996,"5min":1.294,"15min":1.294},"last":0.53400000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.505,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":1.1339999999999999,"15min":1.3049999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.73299999999999998,"15min":0.75600000000000001},"min":{"1min":0.58199999999999996,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":1.028,"5min":1.194,"15min":6.9720000000000004},"last":0.62}]},{"osd":2,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71399999999999997,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.119,"15min":5.9299999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.59899999999999998,"5min":0.48499999999999999,"15min":0.42299999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.704},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.60799999999999998,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":0.93799999999999994,"5min":1.2589999999999999,"15min":1.899},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51900000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94999999999999996,"5min":1.036,"15min":6.0620000000000003},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.83499999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.64000000000000001,"15min":0.63600000000000001},"min":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.97499999999999998,"15min":0.998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50900000000000001,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.81599999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.498}]},{"osd":7,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70699999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":0.83399999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76500000000000001},"min":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.89100000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:45:05.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:45:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.210+0000 7f767fdff700 1 Processor -- start 2023-10-07T09:45:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.210+0000 7f767fdff700 1 -- start start 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.211+0000 7f767fdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f76780b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.211+0000 7f767fdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f767815ade0 0x7f76780b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.211+0000 7f767fdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7678154090 con 0x7f76780b0000 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767fdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7678159f60 con 0x7f767815ade0 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767fdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767815a240 con 0x7f7678153cc0 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f767815ade0 0x7f76780b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f767815ade0 0x7f76780b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60154/0 (socket says 172.21.15.29:60154) 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 -- 172.21.15.29:0/152786633 learned_addr learned my addr 172.21.15.29:0/152786633 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767db9b700 1 --2- 172.21.15.29:0/152786633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f76780b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 -- 172.21.15.29:0/152786633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 msgr2=0x7f76780b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 --2- 172.21.15.29:0/152786633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f76780b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 -- 172.21.15.29:0/152786633 >> v1:172.21.15.144:6789/0 conn(0x7f76780b0000 legacy=0x7f767815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767d39a700 1 -- 172.21.15.29:0/152786633 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767815a520 con 0x7f767815ade0 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.212+0000 7f767cb99700 1 -- 172.21.15.29:0/152786633 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868675021 0 0) 0x7f7678154090 con 0x7f76780b0000 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.213+0000 7f767d39a700 1 --2- 172.21.15.29:0/152786633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815ade0 0x7f76780b1d00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f76680099b0 tx=0x7f7668009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e94e47887f45700 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.213+0000 7f767cb99700 1 -- 172.21.15.29:0/152786633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668009ea0 con 0x7f767815ade0 2023-10-07T09:45:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.213+0000 7f767cb99700 1 -- 172.21.15.29:0/152786633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766800b860 con 0x7f767815ade0 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.213+0000 7f767cb99700 1 -- 172.21.15.29:0/152786633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766800bc00 con 0x7f767815ade0 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.213+0000 7f767fdff700 1 -- 172.21.15.29:0/152786633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815ade0 msgr2=0x7f76780b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.213+0000 7f767fdff700 1 --2- 172.21.15.29:0/152786633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815ade0 0x7f76780b1d00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f76680099b0 tx=0x7f7668009250 comp rx=0 tx=0).stop 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.214+0000 7f767fdff700 1 -- 172.21.15.29:0/152786633 shutdown_connections 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.214+0000 7f767fdff700 1 --2- 172.21.15.29:0/152786633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f76780b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.214+0000 7f767fdff700 1 --2- 172.21.15.29:0/152786633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815ade0 0x7f76780b1d00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.214+0000 7f767fdff700 1 -- 172.21.15.29:0/152786633 >> 172.21.15.29:0/152786633 conn(0x7f7678067610 msgr2=0x7f7678067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.214+0000 7f767fdff700 1 -- 172.21.15.29:0/152786633 shutdown_connections 2023-10-07T09:45:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.214+0000 7f767fdff700 1 -- 172.21.15.29:0/152786633 wait complete. 2023-10-07T09:45:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.215+0000 7f767fdff700 1 Processor -- start 2023-10-07T09:45:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.215+0000 7f767fdff700 1 -- start start 2023-10-07T09:45:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767fdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 0x7f7678145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767fdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f7678157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767fdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f767815ade0 0x7f7678158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767fdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7678156e80 con 0x7f76780b0000 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767fdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76781464e0 con 0x7f767815ade0 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767fdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76780c7da0 con 0x7f7678153cc0 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767db9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 0x7f7678145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767db9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 0x7f7678145fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60170/0 (socket says 172.21.15.29:60170) 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.216+0000 7f767db9b700 1 -- 172.21.15.29:0/4119250832 learned_addr learned my addr 172.21.15.29:0/4119250832 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767d39a700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f7678157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767e39c700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f767815ade0 0x7f7678158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767db9b700 1 -- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 msgr2=0x7f7678157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767db9b700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f7678157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767db9b700 1 -- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f767815ade0 msgr2=0x7f7678158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767db9b700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f767815ade0 0x7f7678158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767db9b700 1 -- 172.21.15.29:0/4119250832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680094f0 con 0x7f76780b0000 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767db9b700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 0x7f7678145fa0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f767400ece0 tx=0x7f7674009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767400c850 con 0x7f76780b0000 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767400c9b0 con 0x7f76780b0000 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.217+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7678156ba0 con 0x7f76780b0000 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.218+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767400cd30 con 0x7f76780b0000 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.218+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76780c7a30 con 0x7f76780b0000 2023-10-07T09:45:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.219+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76780c8080 con 0x7f76780b0000 2023-10-07T09:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.220+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7674013680 con 0x7f76780b0000 2023-10-07T09:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.220+0000 7f766dffb700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f765c066f30 0x7f765c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.220+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f7674084d20 con 0x7f76780b0000 2023-10-07T09:45:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.221+0000 7f767d39a700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f765c066f30 0x7f765c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.221+0000 7f767d39a700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f765c066f30 0x7f765c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f76680099b0 tx=0x7f7668017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.224+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7674051d70 con 0x7f76780b0000 2023-10-07T09:45:06.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.422+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76780c75b0 con 0x7f765c066f30 2023-10-07T09:45:06.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.432+0000 7f766dffb700 1 -- 172.21.15.29:0/4119250832 <== mgr.5885 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288964 (secure 0 0 0) 0x7f76780c75b0 con 0x7f765c066f30 2023-10-07T09:45:06.437 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f765c066f30 msgr2=0x7f765c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f765c066f30 0x7f765c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f76680099b0 tx=0x7f7668017000 comp rx=0 tx=0).stop 2023-10-07T09:45:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 msgr2=0x7f7678145fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 0x7f7678145fa0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f767400ece0 tx=0x7f7674009120 comp rx=0 tx=0).stop 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 shutdown_connections 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f765c066f30 0x7f765c0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f767815ade0 0x7f7678158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76780b0000 0x7f7678145fa0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 --2- 172.21.15.29:0/4119250832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678153cc0 0x7f7678157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 >> 172.21.15.29:0/4119250832 conn(0x7f7678067610 msgr2=0x7f767815b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 shutdown_connections 2023-10-07T09:45:06.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.436+0000 7f767fdff700 1 -- 172.21.15.29:0/4119250832 wait complete. 2023-10-07T09:45:06.442 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:45:06.456 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:45:05.645785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437156,"kb_used_data":9700,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752537564,"statfs":{"total":773094113280,"available":770598465536,"internally_reserved":0,"allocated":9932800,"data_stored":5300372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001329"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.588542+0000","last_change":"2023-10-07T09:45:01.588542+0000","last_active":"2023-10-07T09:45:01.588542+0000","last_peered":"2023-10-07T09:45:01.588542+0000","last_clean":"2023-10-07T09:45:01.588542+0000","last_became_active":"2023-10-07T09:45:01.588296+0000","last_became_peered":"2023-10-07T09:45:01.588296+0000","last_unstale":"2023-10-07T09:45:01.588542+0000","last_undegraded":"2023-10-07T09:45:01.588542+0000","last_fullsized":"2023-10-07T09:45:01.588542+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:50:39.121700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.590255+0000","last_change":"2023-10-07T09:45:01.584130+0000","last_active":"2023-10-07T09:45:01.590255+0000","last_peered":"2023-10-07T09:45:01.590255+0000","last_clean":"2023-10-07T09:45:01.590255+0000","last_became_active":"2023-10-07T09:45:01.583177+0000","last_became_peered":"2023-10-07T09:45:01.583177+0000","last_unstale":"2023-10-07T09:45:01.590255+0000","last_undegraded":"2023-10-07T09:45:01.590255+0000","last_fullsized":"2023-10-07T09:45:01.590255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:44.328328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.580465+0000","last_change":"2023-10-07T09:45:01.580465+0000","last_active":"2023-10-07T09:45:01.580465+0000","last_peered":"2023-10-07T09:45:01.580465+0000","last_clean":"2023-10-07T09:45:01.580465+0000","last_became_active":"2023-10-07T09:45:01.577079+0000","last_became_peered":"2023-10-07T09:45:01.577079+0000","last_unstale":"2023-10-07T09:45:01.580465+0000","last_undegraded":"2023-10-07T09:45:01.580465+0000","last_fullsized":"2023-10-07T09:45:01.580465+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:33:21.238319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596032+0000","last_change":"2023-10-07T09:45:01.593788+0000","last_active":"2023-10-07T09:45:01.596032+0000","last_peered":"2023-10-07T09:45:01.596032+0000","last_clean":"2023-10-07T09:45:01.596032+0000","last_became_active":"2023-10-07T09:45:01.593113+0000","last_became_peered":"2023-10-07T09:45:01.593113+0000","last_unstale":"2023-10-07T09:45:01.596032+0000","last_undegraded":"2023-10-07T09:45:01.596032+0000","last_fullsized":"2023-10-07T09:45:01.596032+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:19:01.805409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585874+0000","last_change":"2023-10-07T09:45:01.582201+0000","last_active":"2023-10-07T09:45:01.585874+0000","last_peered":"2023-10-07T09:45:01.585874+0000","last_clean":"2023-10-07T09:45:01.585874+0000","last_became_active":"2023-10-07T09:45:01.581890+0000","last_became_peered":"2023-10-07T09:45:01.581890+0000","last_unstale":"2023-10-07T09:45:01.585874+0000","last_undegraded":"2023-10-07T09:45:01.585874+0000","last_fullsized":"2023-10-07T09:45:01.585874+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:47:46.475578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592559+0000","last_change":"2023-10-07T09:45:01.584324+0000","last_active":"2023-10-07T09:45:01.592559+0000","last_peered":"2023-10-07T09:45:01.592559+0000","last_clean":"2023-10-07T09:45:01.592559+0000","last_became_active":"2023-10-07T09:45:01.583970+0000","last_became_peered":"2023-10-07T09:45:01.583970+0000","last_unstale":"2023-10-07T09:45:01.592559+0000","last_undegraded":"2023-10-07T09:45:01.592559+0000","last_fullsized":"2023-10-07T09:45:01.592559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:03.541472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594439+0000","last_change":"2023-10-07T09:45:01.588574+0000","last_active":"2023-10-07T09:45:01.594439+0000","last_peered":"2023-10-07T09:45:01.594439+0000","last_clean":"2023-10-07T09:45:01.594439+0000","last_became_active":"2023-10-07T09:45:01.587936+0000","last_became_peered":"2023-10-07T09:45:01.587936+0000","last_unstale":"2023-10-07T09:45:01.594439+0000","last_undegraded":"2023-10-07T09:45:01.594439+0000","last_fullsized":"2023-10-07T09:45:01.594439+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:41.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.586045+0000","last_change":"2023-10-07T09:45:01.581793+0000","last_active":"2023-10-07T09:45:01.586045+0000","last_peered":"2023-10-07T09:45:01.586045+0000","last_clean":"2023-10-07T09:45:01.586045+0000","last_became_active":"2023-10-07T09:45:01.581518+0000","last_became_peered":"2023-10-07T09:45:01.581518+0000","last_unstale":"2023-10-07T09:45:01.586045+0000","last_undegraded":"2023-10-07T09:45:01.586045+0000","last_fullsized":"2023-10-07T09:45:01.586045+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:12:49.510761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598787+0000","last_change":"2023-10-07T09:45:01.596603+0000","last_active":"2023-10-07T09:45:01.598787+0000","last_peered":"2023-10-07T09:45:01.598787+0000","last_clean":"2023-10-07T09:45:01.598787+0000","last_became_active":"2023-10-07T09:45:01.596321+0000","last_became_peered":"2023-10-07T09:45:01.596321+0000","last_unstale":"2023-10-07T09:45:01.598787+0000","last_undegraded":"2023-10-07T09:45:01.598787+0000","last_fullsized":"2023-10-07T09:45:01.598787+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:35.634346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598713+0000","last_change":"2023-10-07T09:45:01.588610+0000","last_active":"2023-10-07T09:45:01.598713+0000","last_peered":"2023-10-07T09:45:01.598713+0000","last_clean":"2023-10-07T09:45:01.598713+0000","last_became_active":"2023-10-07T09:45:01.588349+0000","last_became_peered":"2023-10-07T09:45:01.588349+0000","last_unstale":"2023-10-07T09:45:01.598713+0000","last_undegraded":"2023-10-07T09:45:01.598713+0000","last_fullsized":"2023-10-07T09:45:01.598713+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:20.855523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.590321+0000","last_change":"2023-10-07T09:45:01.590321+0000","last_active":"2023-10-07T09:45:01.590321+0000","last_peered":"2023-10-07T09:45:01.590321+0000","last_clean":"2023-10-07T09:45:01.590321+0000","last_became_active":"2023-10-07T09:45:01.589977+0000","last_became_peered":"2023-10-07T09:45:01.589977+0000","last_unstale":"2023-10-07T09:45:01.590321+0000","last_undegraded":"2023-10-07T09:45:01.590321+0000","last_fullsized":"2023-10-07T09:45:01.590321+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:44:52.391971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594678+0000","last_change":"2023-10-07T09:45:01.588451+0000","last_active":"2023-10-07T09:45:01.594678+0000","last_peered":"2023-10-07T09:45:01.594678+0000","last_clean":"2023-10-07T09:45:01.594678+0000","last_became_active":"2023-10-07T09:45:01.587860+0000","last_became_peered":"2023-10-07T09:45:01.587860+0000","last_unstale":"2023-10-07T09:45:01.594678+0000","last_undegraded":"2023-10-07T09:45:01.594678+0000","last_fullsized":"2023-10-07T09:45:01.594678+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:45.733391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598531+0000","last_change":"2023-10-07T09:45:01.592811+0000","last_active":"2023-10-07T09:45:01.598531+0000","last_peered":"2023-10-07T09:45:01.598531+0000","last_clean":"2023-10-07T09:45:01.598531+0000","last_became_active":"2023-10-07T09:45:01.592527+0000","last_became_peered":"2023-10-07T09:45:01.592527+0000","last_unstale":"2023-10-07T09:45:01.598531+0000","last_undegraded":"2023-10-07T09:45:01.598531+0000","last_fullsized":"2023-10-07T09:45:01.598531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:58:04.424056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595454+0000","last_change":"2023-10-07T09:45:01.590992+0000","last_active":"2023-10-07T09:45:01.595454+0000","last_peered":"2023-10-07T09:45:01.595454+0000","last_clean":"2023-10-07T09:45:01.595454+0000","last_became_active":"2023-10-07T09:45:01.589742+0000","last_became_peered":"2023-10-07T09:45:01.589742+0000","last_unstale":"2023-10-07T09:45:01.595454+0000","last_undegraded":"2023-10-07T09:45:01.595454+0000","last_fullsized":"2023-10-07T09:45:01.595454+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:07:47.154408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596261+0000","last_change":"2023-10-07T09:45:01.588043+0000","last_active":"2023-10-07T09:45:01.596261+0000","last_peered":"2023-10-07T09:45:01.596261+0000","last_clean":"2023-10-07T09:45:01.596261+0000","last_became_active":"2023-10-07T09:45:01.587673+0000","last_became_peered":"2023-10-07T09:45:01.587673+0000","last_unstale":"2023-10-07T09:45:01.596261+0000","last_undegraded":"2023-10-07T09:45:01.596261+0000","last_fullsized":"2023-10-07T09:45:01.596261+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:54:49.662078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598178+0000","last_change":"2023-10-07T09:45:01.592866+0000","last_active":"2023-10-07T09:45:01.598178+0000","last_peered":"2023-10-07T09:45:01.598178+0000","last_clean":"2023-10-07T09:45:01.598178+0000","last_became_active":"2023-10-07T09:45:01.592575+0000","last_became_peered":"2023-10-07T09:45:01.592575+0000","last_unstale":"2023-10-07T09:45:01.598178+0000","last_undegraded":"2023-10-07T09:45:01.598178+0000","last_fullsized":"2023-10-07T09:45:01.598178+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:08.408364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598769+0000","last_change":"2023-10-07T09:45:01.596449+0000","last_active":"2023-10-07T09:45:01.598769+0000","last_peered":"2023-10-07T09:45:01.598769+0000","last_clean":"2023-10-07T09:45:01.598769+0000","last_became_active":"2023-10-07T09:45:01.596097+0000","last_became_peered":"2023-10-07T09:45:01.596097+0000","last_unstale":"2023-10-07T09:45:01.598769+0000","last_undegraded":"2023-10-07T09:45:01.598769+0000","last_fullsized":"2023-10-07T09:45:01.598769+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:13:23.493619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594585+0000","last_change":"2023-10-07T09:45:01.590839+0000","last_active":"2023-10-07T09:45:01.594585+0000","last_peered":"2023-10-07T09:45:01.594585+0000","last_clean":"2023-10-07T09:45:01.594585+0000","last_became_active":"2023-10-07T09:45:01.590571+0000","last_became_peered":"2023-10-07T09:45:01.590571+0000","last_unstale":"2023-10-07T09:45:01.594585+0000","last_undegraded":"2023-10-07T09:45:01.594585+0000","last_fullsized":"2023-10-07T09:45:01.594585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:03.097063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597068+0000","last_change":"2023-10-07T09:45:01.590631+0000","last_active":"2023-10-07T09:45:01.597068+0000","last_peered":"2023-10-07T09:45:01.597068+0000","last_clean":"2023-10-07T09:45:01.597068+0000","last_became_active":"2023-10-07T09:45:01.590344+0000","last_became_peered":"2023-10-07T09:45:01.590344+0000","last_unstale":"2023-10-07T09:45:01.597068+0000","last_undegraded":"2023-10-07T09:45:01.597068+0000","last_fullsized":"2023-10-07T09:45:01.597068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:41.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597350+0000","last_change":"2023-10-07T09:45:01.588822+0000","last_active":"2023-10-07T09:45:01.597350+0000","last_peered":"2023-10-07T09:45:01.597350+0000","last_clean":"2023-10-07T09:45:01.597350+0000","last_became_active":"2023-10-07T09:45:01.588546+0000","last_became_peered":"2023-10-07T09:45:01.588546+0000","last_unstale":"2023-10-07T09:45:01.597350+0000","last_undegraded":"2023-10-07T09:45:01.597350+0000","last_fullsized":"2023-10-07T09:45:01.597350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:08:56.847110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594114+0000","last_change":"2023-10-07T09:45:01.580866+0000","last_active":"2023-10-07T09:45:01.594114+0000","last_peered":"2023-10-07T09:45:01.594114+0000","last_clean":"2023-10-07T09:45:01.594114+0000","last_became_active":"2023-10-07T09:45:01.580346+0000","last_became_peered":"2023-10-07T09:45:01.580346+0000","last_unstale":"2023-10-07T09:45:01.594114+0000","last_undegraded":"2023-10-07T09:45:01.594114+0000","last_fullsized":"2023-10-07T09:45:01.594114+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:08:28.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593305+0000","last_change":"2023-10-07T09:45:01.582763+0000","last_active":"2023-10-07T09:45:01.593305+0000","last_peered":"2023-10-07T09:45:01.593305+0000","last_clean":"2023-10-07T09:45:01.593305+0000","last_became_active":"2023-10-07T09:45:01.582512+0000","last_became_peered":"2023-10-07T09:45:01.582512+0000","last_unstale":"2023-10-07T09:45:01.593305+0000","last_undegraded":"2023-10-07T09:45:01.593305+0000","last_fullsized":"2023-10-07T09:45:01.593305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:44.328328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591368+0000","last_change":"2023-10-07T09:45:01.591368+0000","last_active":"2023-10-07T09:45:01.591368+0000","last_peered":"2023-10-07T09:45:01.591368+0000","last_clean":"2023-10-07T09:45:01.591368+0000","last_became_active":"2023-10-07T09:45:01.591086+0000","last_became_peered":"2023-10-07T09:45:01.591086+0000","last_unstale":"2023-10-07T09:45:01.591368+0000","last_undegraded":"2023-10-07T09:45:01.591368+0000","last_fullsized":"2023-10-07T09:45:01.591368+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:09:58.110670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592913+0000","last_change":"2023-10-07T09:45:01.592913+0000","last_active":"2023-10-07T09:45:01.592913+0000","last_peered":"2023-10-07T09:45:01.592913+0000","last_clean":"2023-10-07T09:45:01.592913+0000","last_became_active":"2023-10-07T09:45:01.592629+0000","last_became_peered":"2023-10-07T09:45:01.592629+0000","last_unstale":"2023-10-07T09:45:01.592913+0000","last_undegraded":"2023-10-07T09:45:01.592913+0000","last_fullsized":"2023-10-07T09:45:01.592913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:21:22.836441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593270+0000","last_change":"2023-10-07T09:45:01.580037+0000","last_active":"2023-10-07T09:45:01.593270+0000","last_peered":"2023-10-07T09:45:01.593270+0000","last_clean":"2023-10-07T09:45:01.593270+0000","last_became_active":"2023-10-07T09:45:01.579721+0000","last_became_peered":"2023-10-07T09:45:01.579721+0000","last_unstale":"2023-10-07T09:45:01.593270+0000","last_undegraded":"2023-10-07T09:45:01.593270+0000","last_fullsized":"2023-10-07T09:45:01.593270+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:58:04.424056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596049+0000","last_change":"2023-10-07T09:45:01.588995+0000","last_active":"2023-10-07T09:45:01.596049+0000","last_peered":"2023-10-07T09:45:01.596049+0000","last_clean":"2023-10-07T09:45:01.596049+0000","last_became_active":"2023-10-07T09:45:01.588345+0000","last_became_peered":"2023-10-07T09:45:01.588345+0000","last_unstale":"2023-10-07T09:45:01.596049+0000","last_undegraded":"2023-10-07T09:45:01.596049+0000","last_fullsized":"2023-10-07T09:45:01.596049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:53:49.463823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.581287+0000","last_change":"2023-10-07T09:45:01.581287+0000","last_active":"2023-10-07T09:45:01.581287+0000","last_peered":"2023-10-07T09:45:01.581287+0000","last_clean":"2023-10-07T09:45:01.581287+0000","last_became_active":"2023-10-07T09:45:01.581011+0000","last_became_peered":"2023-10-07T09:45:01.581011+0000","last_unstale":"2023-10-07T09:45:01.581287+0000","last_undegraded":"2023-10-07T09:45:01.581287+0000","last_fullsized":"2023-10-07T09:45:01.581287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:32.302603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598361+0000","last_change":"2023-10-07T09:45:01.587873+0000","last_active":"2023-10-07T09:45:01.598361+0000","last_peered":"2023-10-07T09:45:01.598361+0000","last_clean":"2023-10-07T09:45:01.598361+0000","last_became_active":"2023-10-07T09:45:01.587613+0000","last_became_peered":"2023-10-07T09:45:01.587613+0000","last_unstale":"2023-10-07T09:45:01.598361+0000","last_undegraded":"2023-10-07T09:45:01.598361+0000","last_fullsized":"2023-10-07T09:45:01.598361+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:05:03.541472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.581026+0000","last_change":"2023-10-07T09:45:01.581026+0000","last_active":"2023-10-07T09:45:01.581026+0000","last_peered":"2023-10-07T09:45:01.581026+0000","last_clean":"2023-10-07T09:45:01.581026+0000","last_became_active":"2023-10-07T09:45:01.580767+0000","last_became_peered":"2023-10-07T09:45:01.580767+0000","last_unstale":"2023-10-07T09:45:01.581026+0000","last_undegraded":"2023-10-07T09:45:01.581026+0000","last_fullsized":"2023-10-07T09:45:01.581026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:59.125204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593372+0000","last_change":"2023-10-07T09:45:01.588069+0000","last_active":"2023-10-07T09:45:01.593372+0000","last_peered":"2023-10-07T09:45:01.593372+0000","last_clean":"2023-10-07T09:45:01.593372+0000","last_became_active":"2023-10-07T09:45:01.587793+0000","last_became_peered":"2023-10-07T09:45:01.587793+0000","last_unstale":"2023-10-07T09:45:01.593372+0000","last_undegraded":"2023-10-07T09:45:01.593372+0000","last_fullsized":"2023-10-07T09:45:01.593372+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:39:27.825926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594462+0000","last_change":"2023-10-07T09:45:01.588759+0000","last_active":"2023-10-07T09:45:01.594462+0000","last_peered":"2023-10-07T09:45:01.594462+0000","last_clean":"2023-10-07T09:45:01.594462+0000","last_became_active":"2023-10-07T09:45:01.588233+0000","last_became_peered":"2023-10-07T09:45:01.588233+0000","last_unstale":"2023-10-07T09:45:01.594462+0000","last_undegraded":"2023-10-07T09:45:01.594462+0000","last_fullsized":"2023-10-07T09:45:01.594462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:53:39.408523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599671+0000","last_change":"2023-10-07T09:45:01.594754+0000","last_active":"2023-10-07T09:45:01.599671+0000","last_peered":"2023-10-07T09:45:01.599671+0000","last_clean":"2023-10-07T09:45:01.599671+0000","last_became_active":"2023-10-07T09:45:01.594218+0000","last_became_peered":"2023-10-07T09:45:01.594218+0000","last_unstale":"2023-10-07T09:45:01.599671+0000","last_undegraded":"2023-10-07T09:45:01.599671+0000","last_fullsized":"2023-10-07T09:45:01.599671+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:08.408364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592908+0000","last_change":"2023-10-07T09:45:01.592908+0000","last_active":"2023-10-07T09:45:01.592908+0000","last_peered":"2023-10-07T09:45:01.592908+0000","last_clean":"2023-10-07T09:45:01.592908+0000","last_became_active":"2023-10-07T09:45:01.592631+0000","last_became_peered":"2023-10-07T09:45:01.592631+0000","last_unstale":"2023-10-07T09:45:01.592908+0000","last_undegraded":"2023-10-07T09:45:01.592908+0000","last_fullsized":"2023-10-07T09:45:01.592908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:39:15.461534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"89'4","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598430+0000","last_change":"2023-10-07T09:45:00.576204+0000","last_active":"2023-10-07T09:45:01.598430+0000","last_peered":"2023-10-07T09:45:01.598430+0000","last_clean":"2023-10-07T09:45:01.598430+0000","last_became_active":"2023-10-07T09:45:00.575922+0000","last_became_peered":"2023-10-07T09:45:00.575922+0000","last_unstale":"2023-10-07T09:45:01.598430+0000","last_undegraded":"2023-10-07T09:45:01.598430+0000","last_fullsized":"2023-10-07T09:45:01.598430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:10.222273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596304+0000","last_change":"2023-10-07T09:45:01.587158+0000","last_active":"2023-10-07T09:45:01.596304+0000","last_peered":"2023-10-07T09:45:01.596304+0000","last_clean":"2023-10-07T09:45:01.596304+0000","last_became_active":"2023-10-07T09:45:01.586833+0000","last_became_peered":"2023-10-07T09:45:01.586833+0000","last_unstale":"2023-10-07T09:45:01.596304+0000","last_undegraded":"2023-10-07T09:45:01.596304+0000","last_fullsized":"2023-10-07T09:45:01.596304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:27:33.602389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.587592+0000","last_change":"2023-10-07T09:45:00.582161+0000","last_active":"2023-10-07T09:45:01.587592+0000","last_peered":"2023-10-07T09:45:01.587592+0000","last_clean":"2023-10-07T09:45:01.587592+0000","last_became_active":"2023-10-07T09:45:00.581861+0000","last_became_peered":"2023-10-07T09:45:00.581861+0000","last_unstale":"2023-10-07T09:45:01.587592+0000","last_undegraded":"2023-10-07T09:45:01.587592+0000","last_fullsized":"2023-10-07T09:45:01.587592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:53.508191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591363+0000","last_change":"2023-10-07T09:45:01.584530+0000","last_active":"2023-10-07T09:45:01.591363+0000","last_peered":"2023-10-07T09:45:01.591363+0000","last_clean":"2023-10-07T09:45:01.591363+0000","last_became_active":"2023-10-07T09:45:01.584195+0000","last_became_peered":"2023-10-07T09:45:01.584195+0000","last_unstale":"2023-10-07T09:45:01.591363+0000","last_undegraded":"2023-10-07T09:45:01.591363+0000","last_fullsized":"2023-10-07T09:45:01.591363+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:11:21.387986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600699+0000","last_change":"2023-10-07T09:45:00.583570+0000","last_active":"2023-10-07T09:45:01.600699+0000","last_peered":"2023-10-07T09:45:01.600699+0000","last_clean":"2023-10-07T09:45:01.600699+0000","last_became_active":"2023-10-07T09:45:00.581893+0000","last_became_peered":"2023-10-07T09:45:00.581893+0000","last_unstale":"2023-10-07T09:45:01.600699+0000","last_undegraded":"2023-10-07T09:45:01.600699+0000","last_fullsized":"2023-10-07T09:45:01.600699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:44.723902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.584327+0000","last_change":"2023-10-07T09:45:01.584327+0000","last_active":"2023-10-07T09:45:01.584327+0000","last_peered":"2023-10-07T09:45:01.584327+0000","last_clean":"2023-10-07T09:45:01.584327+0000","last_became_active":"2023-10-07T09:45:01.584066+0000","last_became_peered":"2023-10-07T09:45:01.584066+0000","last_unstale":"2023-10-07T09:45:01.584327+0000","last_undegraded":"2023-10-07T09:45:01.584327+0000","last_fullsized":"2023-10-07T09:45:01.584327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:27:37.268229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597244+0000","last_change":"2023-10-07T09:45:00.586700+0000","last_active":"2023-10-07T09:45:01.597244+0000","last_peered":"2023-10-07T09:45:01.597244+0000","last_clean":"2023-10-07T09:45:01.597244+0000","last_became_active":"2023-10-07T09:45:00.586375+0000","last_became_peered":"2023-10-07T09:45:00.586375+0000","last_unstale":"2023-10-07T09:45:01.597244+0000","last_undegraded":"2023-10-07T09:45:01.597244+0000","last_fullsized":"2023-10-07T09:45:01.597244+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:02.087574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.592558+0000","last_change":"2023-10-07T09:45:01.586781+0000","last_active":"2023-10-07T09:45:01.592558+0000","last_peered":"2023-10-07T09:45:01.592558+0000","last_clean":"2023-10-07T09:45:01.592558+0000","last_became_active":"2023-10-07T09:45:01.586521+0000","last_became_peered":"2023-10-07T09:45:01.586521+0000","last_unstale":"2023-10-07T09:45:01.592558+0000","last_undegraded":"2023-10-07T09:45:01.592558+0000","last_fullsized":"2023-10-07T09:45:01.592558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:39:05.423766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"89'1","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.565353+0000","last_change":"2023-10-07T09:45:00.573052+0000","last_active":"2023-10-07T09:45:01.565353+0000","last_peered":"2023-10-07T09:45:01.565353+0000","last_clean":"2023-10-07T09:45:01.565353+0000","last_became_active":"2023-10-07T09:45:00.572773+0000","last_became_peered":"2023-10-07T09:45:00.572773+0000","last_unstale":"2023-10-07T09:45:01.565353+0000","last_undegraded":"2023-10-07T09:45:01.565353+0000","last_fullsized":"2023-10-07T09:45:01.565353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:53:48.454334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594135+0000","last_change":"2023-10-07T09:45:01.581016+0000","last_active":"2023-10-07T09:45:01.594135+0000","last_peered":"2023-10-07T09:45:01.594135+0000","last_clean":"2023-10-07T09:45:01.594135+0000","last_became_active":"2023-10-07T09:45:01.580643+0000","last_became_peered":"2023-10-07T09:45:01.580643+0000","last_unstale":"2023-10-07T09:45:01.594135+0000","last_undegraded":"2023-10-07T09:45:01.594135+0000","last_fullsized":"2023-10-07T09:45:01.594135+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:11:21.387986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600115+0000","last_change":"2023-10-07T09:45:00.585520+0000","last_active":"2023-10-07T09:45:01.600115+0000","last_peered":"2023-10-07T09:45:01.600115+0000","last_clean":"2023-10-07T09:45:01.600115+0000","last_became_active":"2023-10-07T09:45:00.585216+0000","last_became_peered":"2023-10-07T09:45:00.585216+0000","last_unstale":"2023-10-07T09:45:01.600115+0000","last_undegraded":"2023-10-07T09:45:01.600115+0000","last_fullsized":"2023-10-07T09:45:01.600115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:33.528525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585902+0000","last_change":"2023-10-07T09:45:01.579739+0000","last_active":"2023-10-07T09:45:01.585902+0000","last_peered":"2023-10-07T09:45:01.585902+0000","last_clean":"2023-10-07T09:45:01.585902+0000","last_became_active":"2023-10-07T09:45:01.579468+0000","last_became_peered":"2023-10-07T09:45:01.579468+0000","last_unstale":"2023-10-07T09:45:01.585902+0000","last_undegraded":"2023-10-07T09:45:01.585902+0000","last_fullsized":"2023-10-07T09:45:01.585902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:38.079014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.574903+0000","last_change":"2023-10-07T09:45:00.580674+0000","last_active":"2023-10-07T09:45:01.574903+0000","last_peered":"2023-10-07T09:45:01.574903+0000","last_clean":"2023-10-07T09:45:01.574903+0000","last_became_active":"2023-10-07T09:45:00.580399+0000","last_became_peered":"2023-10-07T09:45:00.580399+0000","last_unstale":"2023-10-07T09:45:01.574903+0000","last_undegraded":"2023-10-07T09:45:01.574903+0000","last_fullsized":"2023-10-07T09:45:01.574903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:07:46.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.587891+0000","last_change":"2023-10-07T09:45:01.580745+0000","last_active":"2023-10-07T09:45:01.587891+0000","last_peered":"2023-10-07T09:45:01.587891+0000","last_clean":"2023-10-07T09:45:01.587891+0000","last_became_active":"2023-10-07T09:45:01.580485+0000","last_became_peered":"2023-10-07T09:45:01.580485+0000","last_unstale":"2023-10-07T09:45:01.587891+0000","last_undegraded":"2023-10-07T09:45:01.587891+0000","last_fullsized":"2023-10-07T09:45:01.587891+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:41.565669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571932+0000","last_change":"2023-10-07T09:45:00.575690+0000","last_active":"2023-10-07T09:45:01.571932+0000","last_peered":"2023-10-07T09:45:01.571932+0000","last_clean":"2023-10-07T09:45:01.571932+0000","last_became_active":"2023-10-07T09:45:00.575385+0000","last_became_peered":"2023-10-07T09:45:00.575385+0000","last_unstale":"2023-10-07T09:45:01.571932+0000","last_undegraded":"2023-10-07T09:45:01.571932+0000","last_fullsized":"2023-10-07T09:45:01.571932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:38:28.683977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.596071+0000","last_change":"2023-10-07T09:45:01.589160+0000","last_active":"2023-10-07T09:45:01.596071+0000","last_peered":"2023-10-07T09:45:01.596071+0000","last_clean":"2023-10-07T09:45:01.596071+0000","last_became_active":"2023-10-07T09:45:01.588698+0000","last_became_peered":"2023-10-07T09:45:01.588698+0000","last_unstale":"2023-10-07T09:45:01.596071+0000","last_undegraded":"2023-10-07T09:45:01.596071+0000","last_fullsized":"2023-10-07T09:45:01.596071+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:50:55.009515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.568487+0000","last_change":"2023-10-07T09:45:00.571961+0000","last_active":"2023-10-07T09:45:01.568487+0000","last_peered":"2023-10-07T09:45:01.568487+0000","last_clean":"2023-10-07T09:45:01.568487+0000","last_became_active":"2023-10-07T09:45:00.571678+0000","last_became_peered":"2023-10-07T09:45:00.571678+0000","last_unstale":"2023-10-07T09:45:01.568487+0000","last_undegraded":"2023-10-07T09:45:01.568487+0000","last_fullsized":"2023-10-07T09:45:01.568487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:53.508191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593855+0000","last_change":"2023-10-07T09:45:01.593855+0000","last_active":"2023-10-07T09:45:01.593855+0000","last_peered":"2023-10-07T09:45:01.593855+0000","last_clean":"2023-10-07T09:45:01.593855+0000","last_became_active":"2023-10-07T09:45:01.593545+0000","last_became_peered":"2023-10-07T09:45:01.593545+0000","last_unstale":"2023-10-07T09:45:01.593855+0000","last_undegraded":"2023-10-07T09:45:01.593855+0000","last_fullsized":"2023-10-07T09:45:01.593855+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:02:32.151024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.570091+0000","last_change":"2023-10-07T09:45:00.586979+0000","last_active":"2023-10-07T09:45:01.570091+0000","last_peered":"2023-10-07T09:45:01.570091+0000","last_clean":"2023-10-07T09:45:01.570091+0000","last_became_active":"2023-10-07T09:45:00.586666+0000","last_became_peered":"2023-10-07T09:45:00.586666+0000","last_unstale":"2023-10-07T09:45:01.570091+0000","last_undegraded":"2023-10-07T09:45:01.570091+0000","last_fullsized":"2023-10-07T09:45:01.570091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:50:23.900305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591340+0000","last_change":"2023-10-07T09:45:01.584409+0000","last_active":"2023-10-07T09:45:01.591340+0000","last_peered":"2023-10-07T09:45:01.591340+0000","last_clean":"2023-10-07T09:45:01.591340+0000","last_became_active":"2023-10-07T09:45:01.583873+0000","last_became_peered":"2023-10-07T09:45:01.583873+0000","last_unstale":"2023-10-07T09:45:01.591340+0000","last_undegraded":"2023-10-07T09:45:01.591340+0000","last_fullsized":"2023-10-07T09:45:01.591340+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:20.855523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"89'3","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594335+0000","last_change":"2023-10-07T09:45:00.580821+0000","last_active":"2023-10-07T09:45:01.594335+0000","last_peered":"2023-10-07T09:45:01.594335+0000","last_clean":"2023-10-07T09:45:01.594335+0000","last_became_active":"2023-10-07T09:45:00.580554+0000","last_became_peered":"2023-10-07T09:45:00.580554+0000","last_unstale":"2023-10-07T09:45:01.594335+0000","last_undegraded":"2023-10-07T09:45:01.594335+0000","last_fullsized":"2023-10-07T09:45:01.594335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:39:04.414277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595647+0000","last_change":"2023-10-07T09:45:01.584412+0000","last_active":"2023-10-07T09:45:01.595647+0000","last_peered":"2023-10-07T09:45:01.595647+0000","last_clean":"2023-10-07T09:45:01.595647+0000","last_became_active":"2023-10-07T09:45:01.584125+0000","last_became_peered":"2023-10-07T09:45:01.584125+0000","last_unstale":"2023-10-07T09:45:01.595647+0000","last_undegraded":"2023-10-07T09:45:01.595647+0000","last_fullsized":"2023-10-07T09:45:01.595647+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:05:16.984744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"89'6","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600950+0000","last_change":"2023-10-07T09:45:00.581873+0000","last_active":"2023-10-07T09:45:01.600950+0000","last_peered":"2023-10-07T09:45:01.600950+0000","last_clean":"2023-10-07T09:45:01.600950+0000","last_became_active":"2023-10-07T09:45:00.581609+0000","last_became_peered":"2023-10-07T09:45:00.581609+0000","last_unstale":"2023-10-07T09:45:01.600950+0000","last_undegraded":"2023-10-07T09:45:01.600950+0000","last_fullsized":"2023-10-07T09:45:01.600950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:54:47.096630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.587985+0000","last_change":"2023-10-07T09:45:01.587985+0000","last_active":"2023-10-07T09:45:01.587985+0000","last_peered":"2023-10-07T09:45:01.587985+0000","last_clean":"2023-10-07T09:45:01.587985+0000","last_became_active":"2023-10-07T09:45:01.587733+0000","last_became_peered":"2023-10-07T09:45:01.587733+0000","last_unstale":"2023-10-07T09:45:01.587985+0000","last_undegraded":"2023-10-07T09:45:01.587985+0000","last_fullsized":"2023-10-07T09:45:01.587985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:22:24.046350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599119+0000","last_change":"2023-10-07T09:45:00.576696+0000","last_active":"2023-10-07T09:45:01.599119+0000","last_peered":"2023-10-07T09:45:01.599119+0000","last_clean":"2023-10-07T09:45:01.599119+0000","last_became_active":"2023-10-07T09:45:00.576403+0000","last_became_peered":"2023-10-07T09:45:00.576403+0000","last_unstale":"2023-10-07T09:45:01.599119+0000","last_undegraded":"2023-10-07T09:45:01.599119+0000","last_fullsized":"2023-10-07T09:45:01.599119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:18:04.726202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.600151+0000","last_change":"2023-10-07T09:45:01.597092+0000","last_active":"2023-10-07T09:45:01.600151+0000","last_peered":"2023-10-07T09:45:01.600151+0000","last_clean":"2023-10-07T09:45:01.600151+0000","last_became_active":"2023-10-07T09:45:01.596616+0000","last_became_peered":"2023-10-07T09:45:01.596616+0000","last_unstale":"2023-10-07T09:45:01.600151+0000","last_undegraded":"2023-10-07T09:45:01.600151+0000","last_fullsized":"2023-10-07T09:45:01.600151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:59:54.880296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.570978+0000","last_change":"2023-10-07T09:45:00.586049+0000","last_active":"2023-10-07T09:45:01.570978+0000","last_peered":"2023-10-07T09:45:01.570978+0000","last_clean":"2023-10-07T09:45:01.570978+0000","last_became_active":"2023-10-07T09:45:00.585796+0000","last_became_peered":"2023-10-07T09:45:00.585796+0000","last_unstale":"2023-10-07T09:45:01.570978+0000","last_undegraded":"2023-10-07T09:45:01.570978+0000","last_fullsized":"2023-10-07T09:45:01.570978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:53.508191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598599+0000","last_change":"2023-10-07T09:45:01.593370+0000","last_active":"2023-10-07T09:45:01.598599+0000","last_peered":"2023-10-07T09:45:01.598599+0000","last_clean":"2023-10-07T09:45:01.598599+0000","last_became_active":"2023-10-07T09:45:01.593098+0000","last_became_peered":"2023-10-07T09:45:01.593098+0000","last_unstale":"2023-10-07T09:45:01.598599+0000","last_undegraded":"2023-10-07T09:45:01.598599+0000","last_fullsized":"2023-10-07T09:45:01.598599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:25:55.411251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.576615+0000","last_change":"2023-10-07T09:45:00.585097+0000","last_active":"2023-10-07T09:45:01.576615+0000","last_peered":"2023-10-07T09:45:01.576615+0000","last_clean":"2023-10-07T09:45:01.576615+0000","last_became_active":"2023-10-07T09:45:00.584783+0000","last_became_peered":"2023-10-07T09:45:00.584783+0000","last_unstale":"2023-10-07T09:45:01.576615+0000","last_undegraded":"2023-10-07T09:45:01.576615+0000","last_fullsized":"2023-10-07T09:45:01.576615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:09.915300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598700+0000","last_change":"2023-10-07T09:45:01.596654+0000","last_active":"2023-10-07T09:45:01.598700+0000","last_peered":"2023-10-07T09:45:01.598700+0000","last_clean":"2023-10-07T09:45:01.598700+0000","last_became_active":"2023-10-07T09:45:01.596415+0000","last_became_peered":"2023-10-07T09:45:01.596415+0000","last_unstale":"2023-10-07T09:45:01.598700+0000","last_undegraded":"2023-10-07T09:45:01.598700+0000","last_fullsized":"2023-10-07T09:45:01.598700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:54:48.106119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"89'8","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.602023+0000","last_change":"2023-10-07T09:45:00.576661+0000","last_active":"2023-10-07T09:45:01.602023+0000","last_peered":"2023-10-07T09:45:01.602023+0000","last_clean":"2023-10-07T09:45:01.602023+0000","last_became_active":"2023-10-07T09:45:00.576372+0000","last_became_peered":"2023-10-07T09:45:00.576372+0000","last_unstale":"2023-10-07T09:45:01.602023+0000","last_undegraded":"2023-10-07T09:45:01.602023+0000","last_fullsized":"2023-10-07T09:45:01.602023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:32.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593263+0000","last_change":"2023-10-07T09:45:01.585499+0000","last_active":"2023-10-07T09:45:01.593263+0000","last_peered":"2023-10-07T09:45:01.593263+0000","last_clean":"2023-10-07T09:45:01.593263+0000","last_became_active":"2023-10-07T09:45:01.585181+0000","last_became_peered":"2023-10-07T09:45:01.585181+0000","last_unstale":"2023-10-07T09:45:01.593263+0000","last_undegraded":"2023-10-07T09:45:01.593263+0000","last_fullsized":"2023-10-07T09:45:01.593263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:54:49.662078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599438+0000","last_change":"2023-10-07T09:45:00.583493+0000","last_active":"2023-10-07T09:45:01.599438+0000","last_peered":"2023-10-07T09:45:01.599438+0000","last_clean":"2023-10-07T09:45:01.599438+0000","last_became_active":"2023-10-07T09:45:00.583057+0000","last_became_peered":"2023-10-07T09:45:00.583057+0000","last_unstale":"2023-10-07T09:45:01.599438+0000","last_undegraded":"2023-10-07T09:45:01.599438+0000","last_fullsized":"2023-10-07T09:45:01.599438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:32.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.598705+0000","last_change":"2023-10-07T09:45:01.598705+0000","last_active":"2023-10-07T09:45:01.598705+0000","last_peered":"2023-10-07T09:45:01.598705+0000","last_clean":"2023-10-07T09:45:01.598705+0000","last_became_active":"2023-10-07T09:45:01.598205+0000","last_became_peered":"2023-10-07T09:45:01.598205+0000","last_unstale":"2023-10-07T09:45:01.598705+0000","last_undegraded":"2023-10-07T09:45:01.598705+0000","last_fullsized":"2023-10-07T09:45:01.598705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:27:39.563996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.601181+0000","last_change":"2023-10-07T09:45:00.584038+0000","last_active":"2023-10-07T09:45:01.601181+0000","last_peered":"2023-10-07T09:45:01.601181+0000","last_clean":"2023-10-07T09:45:01.601181+0000","last_became_active":"2023-10-07T09:45:00.583785+0000","last_became_peered":"2023-10-07T09:45:00.583785+0000","last_unstale":"2023-10-07T09:45:01.601181+0000","last_undegraded":"2023-10-07T09:45:01.601181+0000","last_fullsized":"2023-10-07T09:45:01.601181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:02.087574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594661+0000","last_change":"2023-10-07T09:45:01.588280+0000","last_active":"2023-10-07T09:45:01.594661+0000","last_peered":"2023-10-07T09:45:01.594661+0000","last_clean":"2023-10-07T09:45:01.594661+0000","last_became_active":"2023-10-07T09:45:01.587537+0000","last_became_peered":"2023-10-07T09:45:01.587537+0000","last_unstale":"2023-10-07T09:45:01.594661+0000","last_undegraded":"2023-10-07T09:45:01.594661+0000","last_fullsized":"2023-10-07T09:45:01.594661+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:03:54.517679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599660+0000","last_change":"2023-10-07T09:45:00.580412+0000","last_active":"2023-10-07T09:45:01.599660+0000","last_peered":"2023-10-07T09:45:01.599660+0000","last_clean":"2023-10-07T09:45:01.599660+0000","last_became_active":"2023-10-07T09:45:00.580147+0000","last_became_peered":"2023-10-07T09:45:00.580147+0000","last_unstale":"2023-10-07T09:45:01.599660+0000","last_undegraded":"2023-10-07T09:45:01.599660+0000","last_fullsized":"2023-10-07T09:45:01.599660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:07:46.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595667+0000","last_change":"2023-10-07T09:45:01.592951+0000","last_active":"2023-10-07T09:45:01.595667+0000","last_peered":"2023-10-07T09:45:01.595667+0000","last_clean":"2023-10-07T09:45:01.595667+0000","last_became_active":"2023-10-07T09:45:01.592702+0000","last_became_peered":"2023-10-07T09:45:01.592702+0000","last_unstale":"2023-10-07T09:45:01.595667+0000","last_undegraded":"2023-10-07T09:45:01.595667+0000","last_fullsized":"2023-10-07T09:45:01.595667+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:18:05.735691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-07T09:45:00.585110+0000","last_change":"2023-10-07T09:45:00.580718+0000","last_active":"2023-10-07T09:45:00.585110+0000","last_peered":"2023-10-07T09:45:00.585110+0000","last_clean":"2023-10-07T09:45:00.585110+0000","last_became_active":"2023-10-07T09:45:00.580418+0000","last_became_peered":"2023-10-07T09:45:00.580418+0000","last_unstale":"2023-10-07T09:45:00.585110+0000","last_undegraded":"2023-10-07T09:45:00.585110+0000","last_fullsized":"2023-10-07T09:45:00.585110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:00:25.326228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.580272+0000","last_change":"2023-10-07T09:45:01.580272+0000","last_active":"2023-10-07T09:45:01.580272+0000","last_peered":"2023-10-07T09:45:01.580272+0000","last_clean":"2023-10-07T09:45:01.580272+0000","last_became_active":"2023-10-07T09:45:01.579520+0000","last_became_peered":"2023-10-07T09:45:01.579520+0000","last_unstale":"2023-10-07T09:45:01.580272+0000","last_undegraded":"2023-10-07T09:45:01.580272+0000","last_fullsized":"2023-10-07T09:45:01.580272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:22:26.957532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571084+0000","last_change":"2023-10-07T09:45:00.584903+0000","last_active":"2023-10-07T09:45:01.571084+0000","last_peered":"2023-10-07T09:45:01.571084+0000","last_clean":"2023-10-07T09:45:01.571084+0000","last_became_active":"2023-10-07T09:45:00.582723+0000","last_became_peered":"2023-10-07T09:45:00.582723+0000","last_unstale":"2023-10-07T09:45:01.571084+0000","last_undegraded":"2023-10-07T09:45:01.571084+0000","last_fullsized":"2023-10-07T09:45:01.571084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:05:26.598159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.594634+0000","last_change":"2023-10-07T09:45:01.594634+0000","last_active":"2023-10-07T09:45:01.594634+0000","last_peered":"2023-10-07T09:45:01.594634+0000","last_clean":"2023-10-07T09:45:01.594634+0000","last_became_active":"2023-10-07T09:45:01.594332+0000","last_became_peered":"2023-10-07T09:45:01.594332+0000","last_unstale":"2023-10-07T09:45:01.594634+0000","last_undegraded":"2023-10-07T09:45:01.594634+0000","last_fullsized":"2023-10-07T09:45:01.594634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:11.518340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.576716+0000","last_change":"2023-10-07T09:45:00.582709+0000","last_active":"2023-10-07T09:45:01.576716+0000","last_peered":"2023-10-07T09:45:01.576716+0000","last_clean":"2023-10-07T09:45:01.576716+0000","last_became_active":"2023-10-07T09:45:00.582450+0000","last_became_peered":"2023-10-07T09:45:00.582450+0000","last_unstale":"2023-10-07T09:45:01.576716+0000","last_undegraded":"2023-10-07T09:45:01.576716+0000","last_fullsized":"2023-10-07T09:45:01.576716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:54:48.652589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585101+0000","last_change":"2023-10-07T09:45:01.585101+0000","last_active":"2023-10-07T09:45:01.585101+0000","last_peered":"2023-10-07T09:45:01.585101+0000","last_clean":"2023-10-07T09:45:01.585101+0000","last_became_active":"2023-10-07T09:45:01.584797+0000","last_became_peered":"2023-10-07T09:45:01.584797+0000","last_unstale":"2023-10-07T09:45:01.585101+0000","last_undegraded":"2023-10-07T09:45:01.585101+0000","last_fullsized":"2023-10-07T09:45:01.585101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:45:40.137616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.573290+0000","last_change":"2023-10-07T09:45:00.581931+0000","last_active":"2023-10-07T09:45:01.573290+0000","last_peered":"2023-10-07T09:45:01.573290+0000","last_clean":"2023-10-07T09:45:01.573290+0000","last_became_active":"2023-10-07T09:45:00.581669+0000","last_became_peered":"2023-10-07T09:45:00.581669+0000","last_unstale":"2023-10-07T09:45:01.573290+0000","last_undegraded":"2023-10-07T09:45:01.573290+0000","last_fullsized":"2023-10-07T09:45:01.573290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:44.723902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.597300+0000","last_change":"2023-10-07T09:45:01.588188+0000","last_active":"2023-10-07T09:45:01.597300+0000","last_peered":"2023-10-07T09:45:01.597300+0000","last_clean":"2023-10-07T09:45:01.597300+0000","last_became_active":"2023-10-07T09:45:01.587895+0000","last_became_peered":"2023-10-07T09:45:01.587895+0000","last_unstale":"2023-10-07T09:45:01.597300+0000","last_undegraded":"2023-10-07T09:45:01.597300+0000","last_fullsized":"2023-10-07T09:45:01.597300+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:10.924789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599332+0000","last_change":"2023-10-07T09:45:00.583324+0000","last_active":"2023-10-07T09:45:01.599332+0000","last_peered":"2023-10-07T09:45:01.599332+0000","last_clean":"2023-10-07T09:45:01.599332+0000","last_became_active":"2023-10-07T09:45:00.583042+0000","last_became_peered":"2023-10-07T09:45:00.583042+0000","last_unstale":"2023-10-07T09:45:01.599332+0000","last_undegraded":"2023-10-07T09:45:01.599332+0000","last_fullsized":"2023-10-07T09:45:01.599332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:33:52.065837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.593403+0000","last_change":"2023-10-07T09:45:01.588053+0000","last_active":"2023-10-07T09:45:01.593403+0000","last_peered":"2023-10-07T09:45:01.593403+0000","last_clean":"2023-10-07T09:45:01.593403+0000","last_became_active":"2023-10-07T09:45:01.587780+0000","last_became_peered":"2023-10-07T09:45:01.587780+0000","last_unstale":"2023-10-07T09:45:01.593403+0000","last_undegraded":"2023-10-07T09:45:01.593403+0000","last_fullsized":"2023-10-07T09:45:01.593403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:12:49.510761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.579772+0000","last_change":"2023-10-07T09:45:00.580701+0000","last_active":"2023-10-07T09:45:01.579772+0000","last_peered":"2023-10-07T09:45:01.579772+0000","last_clean":"2023-10-07T09:45:01.579772+0000","last_became_active":"2023-10-07T09:45:00.579652+0000","last_became_peered":"2023-10-07T09:45:00.579652+0000","last_unstale":"2023-10-07T09:45:01.579772+0000","last_undegraded":"2023-10-07T09:45:01.579772+0000","last_fullsized":"2023-10-07T09:45:01.579772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:37:40.556180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.586010+0000","last_change":"2023-10-07T09:45:01.586010+0000","last_active":"2023-10-07T09:45:01.586010+0000","last_peered":"2023-10-07T09:45:01.586010+0000","last_clean":"2023-10-07T09:45:01.586010+0000","last_became_active":"2023-10-07T09:45:01.585668+0000","last_became_peered":"2023-10-07T09:45:01.585668+0000","last_unstale":"2023-10-07T09:45:01.586010+0000","last_undegraded":"2023-10-07T09:45:01.586010+0000","last_fullsized":"2023-10-07T09:45:01.586010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:02.998961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571724+0000","last_change":"2023-10-07T09:45:00.581928+0000","last_active":"2023-10-07T09:45:01.571724+0000","last_peered":"2023-10-07T09:45:01.571724+0000","last_clean":"2023-10-07T09:45:01.571724+0000","last_became_active":"2023-10-07T09:45:00.581654+0000","last_became_peered":"2023-10-07T09:45:00.581654+0000","last_unstale":"2023-10-07T09:45:01.571724+0000","last_undegraded":"2023-10-07T09:45:01.571724+0000","last_fullsized":"2023-10-07T09:45:01.571724+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:42:52.201833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591508+0000","last_change":"2023-10-07T09:45:01.591508+0000","last_active":"2023-10-07T09:45:01.591508+0000","last_peered":"2023-10-07T09:45:01.591508+0000","last_clean":"2023-10-07T09:45:01.591508+0000","last_became_active":"2023-10-07T09:45:01.590577+0000","last_became_peered":"2023-10-07T09:45:01.590577+0000","last_unstale":"2023-10-07T09:45:01.591508+0000","last_undegraded":"2023-10-07T09:45:01.591508+0000","last_fullsized":"2023-10-07T09:45:01.591508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:16:49.958129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.573806+0000","last_change":"2023-10-07T09:45:00.585367+0000","last_active":"2023-10-07T09:45:01.573806+0000","last_peered":"2023-10-07T09:45:01.573806+0000","last_clean":"2023-10-07T09:45:01.573806+0000","last_became_active":"2023-10-07T09:45:00.584978+0000","last_became_peered":"2023-10-07T09:45:00.584978+0000","last_unstale":"2023-10-07T09:45:01.573806+0000","last_undegraded":"2023-10-07T09:45:01.573806+0000","last_fullsized":"2023-10-07T09:45:01.573806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:32.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.595432+0000","last_change":"2023-10-07T09:45:01.586702+0000","last_active":"2023-10-07T09:45:01.595432+0000","last_peered":"2023-10-07T09:45:01.595432+0000","last_clean":"2023-10-07T09:45:01.595432+0000","last_became_active":"2023-10-07T09:45:01.586414+0000","last_became_peered":"2023-10-07T09:45:01.586414+0000","last_unstale":"2023-10-07T09:45:01.595432+0000","last_undegraded":"2023-10-07T09:45:01.595432+0000","last_fullsized":"2023-10-07T09:45:01.595432+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:47:25.094133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.571496+0000","last_change":"2023-10-07T09:45:00.586571+0000","last_active":"2023-10-07T09:45:01.571496+0000","last_peered":"2023-10-07T09:45:01.571496+0000","last_clean":"2023-10-07T09:45:01.571496+0000","last_became_active":"2023-10-07T09:45:00.586267+0000","last_became_peered":"2023-10-07T09:45:00.586267+0000","last_unstale":"2023-10-07T09:45:01.571496+0000","last_undegraded":"2023-10-07T09:45:01.571496+0000","last_fullsized":"2023-10-07T09:45:01.571496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:49:19.523433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.591995+0000","last_change":"2023-10-07T09:45:01.585383+0000","last_active":"2023-10-07T09:45:01.591995+0000","last_peered":"2023-10-07T09:45:01.591995+0000","last_clean":"2023-10-07T09:45:01.591995+0000","last_became_active":"2023-10-07T09:45:01.585125+0000","last_became_peered":"2023-10-07T09:45:01.585125+0000","last_unstale":"2023-10-07T09:45:01.591995+0000","last_undegraded":"2023-10-07T09:45:01.591995+0000","last_fullsized":"2023-10-07T09:45:01.591995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:58:04.424056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.574688+0000","last_change":"2023-10-07T09:45:00.580812+0000","last_active":"2023-10-07T09:45:01.574688+0000","last_peered":"2023-10-07T09:45:01.574688+0000","last_clean":"2023-10-07T09:45:01.574688+0000","last_became_active":"2023-10-07T09:45:00.580488+0000","last_became_peered":"2023-10-07T09:45:00.580488+0000","last_unstale":"2023-10-07T09:45:01.574688+0000","last_undegraded":"2023-10-07T09:45:01.574688+0000","last_fullsized":"2023-10-07T09:45:01.574688+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:37.069525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.590395+0000","last_change":"2023-10-07T09:45:01.586475+0000","last_active":"2023-10-07T09:45:01.590395+0000","last_peered":"2023-10-07T09:45:01.590395+0000","last_clean":"2023-10-07T09:45:01.590395+0000","last_became_active":"2023-10-07T09:45:01.586176+0000","last_became_peered":"2023-10-07T09:45:01.586176+0000","last_unstale":"2023-10-07T09:45:01.590395+0000","last_undegraded":"2023-10-07T09:45:01.590395+0000","last_fullsized":"2023-10-07T09:45:01.590395+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:33.956834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.567480+0000","last_change":"2023-10-07T09:45:00.579555+0000","last_active":"2023-10-07T09:45:01.567480+0000","last_peered":"2023-10-07T09:45:01.567480+0000","last_clean":"2023-10-07T09:45:01.567480+0000","last_became_active":"2023-10-07T09:45:00.579304+0000","last_became_peered":"2023-10-07T09:45:00.579304+0000","last_unstale":"2023-10-07T09:45:01.567480+0000","last_undegraded":"2023-10-07T09:45:01.567480+0000","last_fullsized":"2023-10-07T09:45:01.567480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:33:52.065837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599581+0000","last_change":"2023-10-07T09:45:01.593507+0000","last_active":"2023-10-07T09:45:01.599581+0000","last_peered":"2023-10-07T09:45:01.599581+0000","last_clean":"2023-10-07T09:45:01.599581+0000","last_became_active":"2023-10-07T09:45:01.593245+0000","last_became_peered":"2023-10-07T09:45:01.593245+0000","last_unstale":"2023-10-07T09:45:01.599581+0000","last_undegraded":"2023-10-07T09:45:01.599581+0000","last_fullsized":"2023-10-07T09:45:01.599581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:08:28.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.572385+0000","last_change":"2023-10-07T09:45:00.586526+0000","last_active":"2023-10-07T09:45:01.572385+0000","last_peered":"2023-10-07T09:45:01.572385+0000","last_clean":"2023-10-07T09:45:01.572385+0000","last_became_active":"2023-10-07T09:45:00.586217+0000","last_became_peered":"2023-10-07T09:45:00.586217+0000","last_unstale":"2023-10-07T09:45:01.572385+0000","last_undegraded":"2023-10-07T09:45:01.572385+0000","last_fullsized":"2023-10-07T09:45:01.572385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:31:44.723902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":8,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.585426+0000","last_change":"2023-10-07T09:45:01.585426+0000","last_active":"2023-10-07T09:45:01.585426+0000","last_peered":"2023-10-07T09:45:01.585426+0000","last_clean":"2023-10-07T09:45:01.585426+0000","last_became_active":"2023-10-07T09:45:01.585020+0000","last_became_peered":"2023-10-07T09:45:01.585020+0000","last_unstale":"2023-10-07T09:45:01.585426+0000","last_undegraded":"2023-10-07T09:45:01.585426+0000","last_fullsized":"2023-10-07T09:45:01.585426+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:00.553257+0000","last_clean_scrub_stamp":"2023-10-07T09:45:00.553257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:54:31.609831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-07T09:45:01.599585+0000","last_change":"2023-10-07T09:45:00.584131+0000","last_active":"2023-10-07T09:45:01.599585+0000","last_peered":"2023-10-07T09:45:01.599585+0000","last_clean":"2023-10-07T09:45:01.599585+0000","last_became_active":"2023-10-07T09:45:00.583844+0000","last_became_peered":"2023-10-07T09:45:00.583844+0000","last_unstale":"2023-10-07T09:45:01.599585+0000","last_undegraded":"2023-10-07T09:45:01.599585+0000","last_fullsized":"2023-10-07T09:45:01.599585+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:44:59.543768+0000","last_clean_scrub_stamp":"2023-10-07T09:44:59.543768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:19:00.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"80'183","reported_seq":454,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-07T09:45:00.566197+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:45:00.566197+0000","last_peered":"2023-10-07T09:45:00.566197+0000","last_clean":"2023-10-07T09:45:00.566197+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:45:00.566197+0000","last_undegraded":"2023-10-07T09:45:00.566197+0000","last_fullsized":"2023-10-07T09:45:00.566197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303024,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068816,"statfs":{"total":96636764160,"available":96326467584,"internally_reserved":0,"allocated":1679360,"data_stored":1101169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68799999999999994,"15min":0.71199999999999997},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.32300000000000001},"max":{"1min":0.95199999999999996,"5min":0.998,"15min":1.026},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.434,"5min":0.42999999999999999,"15min":0.374},"max":{"1min":0.94899999999999995,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66800000000000004}]},{"osd":1,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71999999999999997,"15min":0.72299999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":0.93300000000000005,"5min":1.014,"15min":1.157},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45600000000000002},"max":{"1min":0.875,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.40699999999999997},"max":{"1min":0.99199999999999999,"5min":1.206,"15min":1.206},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72399999999999998,"15min":0.73599999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.439},"max":{"1min":0.91200000000000003,"5min":1.071,"15min":1.071},"last":0.75700000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71599999999999997,"15min":0.73599999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88900000000000001,"5min":0.96899999999999997,"15min":1.04},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.76900000000000002},"min":{"1min":0.64300000000000002,"5min":0.53000000000000003,"15min":0.371},"max":{"1min":0.96099999999999997,"5min":1.137,"15min":1.3440000000000001},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39100000000000001},"max":{"1min":0.85499999999999998,"5min":0.88700000000000001,"15min":1.992},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.48699999999999999,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.6240000000000001},"last":0.89900000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64700000000000002,"15min":0.66000000000000003},"min":{"1min":0.42699999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85899999999999999,"5min":1.27,"15min":1.27},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68400000000000005,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.1399999999999999},"last":0.88100000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.46400000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71299999999999997,"15min":0.72399999999999998},"min":{"1min":0.52100000000000002,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.107},"last":0.95599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509630,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302444,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069396,"statfs":{"total":96636764160,"available":96327061504,"internally_reserved":0,"allocated":1085440,"data_stored":510305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52400000000000002,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":1.274,"5min":1.274,"15min":2.984},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.6000000000000001},"last":0.95699999999999996}]},{"osd":1,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.46300000000000002,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":1.169,"5min":1.169,"15min":2.4529999999999998},"last":1.042},{"interface":"front","average":{"1min":0.749,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.55400000000000005,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.7,"5min":1.7,"15min":1.754},"last":1.02}]},{"osd":2,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.75600000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.44400000000000001},"max":{"1min":1.482,"5min":1.482,"15min":7.867},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.75800000000000001,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.643,"5min":1.643,"15min":7.7629999999999999},"last":0.92400000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.747,"15min":0.78500000000000003},"min":{"1min":0.53200000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.458,"5min":1.458,"15min":8.0399999999999991},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.54800000000000004,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":2.7949999999999999},"last":1.006}]},{"osd":4,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.623,"15min":0.624},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.29899999999999999},"max":{"1min":0.80500000000000005,"5min":0.92400000000000004,"15min":1.296},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.371},"max":{"1min":0.84799999999999998,"5min":0.97899999999999998,"15min":1.105},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.41199999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.4119999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.51400000000000001,"5min":0.49399999999999999,"15min":0.47999999999999998},"max":{"1min":0.98999999999999999,"5min":1.0089999999999999,"15min":1.1779999999999999},"last":0.79100000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.39700000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.5049999999999999},"last":1.099},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.751,"15min":0.76600000000000001},"min":{"1min":0.59099999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91100000000000003,"5min":0.995,"15min":1.3220000000000001},"last":0.745}]}]},{"osd":1,"up_from":14,"seq":60129542332,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":1110016,"data_stored":519001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71799999999999997,"5min":0.746,"15min":1.585},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.372,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.65400000000000003,"5min":0.79800000000000004,"15min":1.6080000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.45400000000000001,"15min":0.47999999999999998},"min":{"1min":0.32600000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.55000000000000004,"5min":0.621,"15min":1.468},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68000000000000005,"5min":0.78400000000000003,"15min":1.5389999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.59699999999999998},"min":{"1min":0.42399999999999999,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.75900000000000001,"15min":1.663},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52500000000000002,"15min":0.54700000000000004},"min":{"1min":0.39100000000000001,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.751,"5min":0.80700000000000005,"15min":1.1140000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.75600000000000001,"15min":0.78200000000000003},"min":{"1min":0.54300000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73899999999999999,"15min":0.77500000000000002},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98399999999999999,"5min":1.032,"15min":1.766},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.96999999999999997,"5min":1.099,"15min":1.3320000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.749,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.93400000000000005,"5min":1.0680000000000001,"15min":1.5149999999999999},"last":0.61399999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.81599999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.95199999999999996,"5min":1.0109999999999999,"15min":1.627},"last":0.746},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.78300000000000003,"15min":0.82399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.3700000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.79600000000000004,"15min":0.83799999999999997},"min":{"1min":0.55400000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.706},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.79800000000000004,"15min":0.84399999999999997},"min":{"1min":0.53900000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0149999999999999,"5min":1.137,"15min":1.554},"last":0.64500000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542332,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303028,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068812,"statfs":{"total":96636764160,"available":96326463488,"internally_reserved":0,"allocated":1683456,"data_stored":1107245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48899999999999999,"15min":0.502},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65700000000000003,"5min":0.73099999999999998,"15min":2.726},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.311,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":0.84599999999999997,"15min":2.552},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53300000000000003,"15min":0.55200000000000005},"min":{"1min":0.41599999999999998,"5min":0.33800000000000002,"15min":0.32800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":2.4910000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.502,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.67100000000000004,"5min":0.80900000000000005,"15min":3.629},"last":0.45600000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.33900000000000002},"max":{"1min":0.73199999999999998,"5min":0.84599999999999997,"15min":2.6909999999999998},"last":0.432},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59899999999999998,"15min":0.61899999999999999},"min":{"1min":0.45800000000000002,"5min":0.42199999999999999,"15min":0.40200000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":2.5209999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.52500000000000002,"5min":0.497,"15min":0.45900000000000002},"max":{"1min":0.89700000000000002,"5min":1.232,"15min":2.637},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":2.4180000000000001},"last":0.86699999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.77600000000000002,"15min":0.80200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":3.8199999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.80900000000000005},"min":{"1min":0.55900000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91100000000000003,"5min":1.194,"15min":2.6000000000000001},"last":0.752}]},{"osd":6,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.98799999999999999,"5min":1.129,"15min":2.3740000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80200000000000005,"15min":0.82999999999999996},"min":{"1min":0.53300000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.95299999999999996,"5min":1.206,"15min":3.855},"last":0.88400000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81399999999999995,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.57499999999999996,"15min":0.56200000000000006},"max":{"1min":0.97599999999999998,"5min":1.167,"15min":2.4550000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81699999999999995,"15min":0.84899999999999998},"min":{"1min":0.56499999999999995,"5min":0.503,"15min":0.503},"max":{"1min":0.98999999999999999,"5min":1.238,"15min":3.778},"last":0.79100000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542333,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310652,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061188,"statfs":{"total":96636764160,"available":96318656512,"internally_reserved":0,"allocated":1101824,"data_stored":518413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.441,"15min":0.44900000000000001},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.63700000000000001,"5min":0.81000000000000005,"15min":1.258},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":0.80100000000000005,"15min":1.2290000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":1.1950000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":1.3200000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.501,"15min":0.51500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.94599999999999995,"15min":1.5820000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.69399999999999995,"5min":0.81399999999999995,"15min":1.052},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73399999999999999,"15min":0.75700000000000001},"min":{"1min":0.48099999999999998,"5min":0.46200000000000002,"15min":0.35699999999999998},"max":{"1min":0.81000000000000005,"5min":1.0660000000000001,"15min":1.4630000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93500000000000005,"5min":1.98,"15min":1.98},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.52500000000000002,"5min":0.52400000000000002,"15min":0.46100000000000002},"max":{"1min":0.88800000000000001,"5min":1.208,"15min":1.5169999999999999},"last":0.83099999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":0.877,"5min":1.931,"15min":1.931},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.59099999999999997,"5min":0.52800000000000002,"15min":0.47299999999999998},"max":{"1min":0.93700000000000006,"5min":1.306,"15min":1.8120000000000001},"last":0.92600000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.83899999999999997},"min":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.54200000000000004},"max":{"1min":0.89800000000000002,"5min":1.3520000000000001,"15min":1.3839999999999999},"last":0.877}]}]},{"osd":3,"up_from":14,"seq":60129542334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306540,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065300,"statfs":{"total":96636764160,"available":96322867200,"internally_reserved":0,"allocated":1085440,"data_stored":510325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":0.71599999999999997,"5min":0.74099999999999999,"15min":3.3980000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.35399999999999998,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":3.274},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32300000000000001},"max":{"1min":0.64900000000000002,"5min":0.73499999999999999,"15min":3.1299999999999999},"last":0.437},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.88,"5min":0.88,"15min":3.4500000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53300000000000003,"15min":0.55600000000000005},"min":{"1min":0.42699999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":3.2309999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56699999999999995,"15min":0.57399999999999995},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.377},"max":{"1min":0.70799999999999996,"5min":0.79800000000000004,"15min":3.0910000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.75600000000000001,"15min":0.77300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.44900000000000001},"max":{"1min":0.94999999999999996,"5min":1.0389999999999999,"15min":1.518},"last":0.628},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.78600000000000003},"min":{"1min":0.60199999999999998,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":1.099,"5min":1.103,"15min":1.3799999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.57399999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0629999999999999,"5min":1.246,"15min":1.3240000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78300000000000003,"15min":0.79200000000000004},"min":{"1min":0.60599999999999998,"5min":0.53200000000000003,"15min":0.50600000000000001},"max":{"1min":1.024,"5min":1.0640000000000001,"15min":1.5189999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.79400000000000004,"15min":0.80600000000000005},"min":{"1min":0.61899999999999999,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.123,"5min":1.202,"15min":1.413},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80000000000000004,"15min":0.82199999999999995},"min":{"1min":0.63400000000000001,"5min":0.58399999999999996,"15min":0.51000000000000001},"max":{"1min":1.046,"5min":1.1459999999999999,"15min":1.5840000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Sat Oct 7 09:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81399999999999995,"15min":0.82399999999999995},"min":{"1min":0.58499999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.026,"5min":1.143,"15min":1.45},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0089999999999999,"5min":1.1040000000000001,"15min":1.5940000000000001},"last":0.73899999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509630,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306548,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065292,"statfs":{"total":96636764160,"available":96322859008,"internally_reserved":0,"allocated":1093632,"data_stored":516947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.86499999999999999,"5min":1.069,"15min":1.069},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.54900000000000004,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":2.3420000000000001},"last":0.66200000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.70899999999999996},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":1.054},"last":0.874},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.72499999999999998},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.35299999999999998},"max":{"1min":0.96199999999999997,"5min":1.478,"15min":1.478},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.75700000000000001,"15min":0.747},"min":{"1min":0.54000000000000004,"5min":0.48799999999999999,"15min":0.432},"max":{"1min":0.996,"5min":1.518,"15min":2.3239999999999998},"last":0.621},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70199999999999996,"15min":0.71199999999999997},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.84199999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84599999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.71299999999999997,"15min":0.71799999999999997},"min":{"1min":0.53200000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.80000000000000004,"5min":0.98699999999999999,"15min":1.032},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.747,"15min":0.751},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.496},"max":{"1min":0.96099999999999997,"5min":1.036,"15min":1.036},"last":0.76000000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.40799999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81000000000000005,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76900000000000002,"5min":1.639,"15min":1.639},"last":0.68300000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.84499999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.81699999999999995,"5min":1.585,"15min":1.585},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.443},"max":{"1min":0.86899999999999999,"5min":1.024,"15min":1.0589999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.75800000000000001,"15min":0.75},"min":{"1min":0.52700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88400000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.79600000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542332,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":1093632,"data_stored":516967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76300000000000001,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.49199999999999999,"15min":0.39900000000000002},"max":{"1min":0.94999999999999996,"5min":1.218,"15min":1.218},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.436},"max":{"1min":0.95499999999999996,"5min":1.294,"15min":1.294},"last":0.53400000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.505,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":1.1339999999999999,"15min":1.3049999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.73299999999999998,"15min":0.75600000000000001},"min":{"1min":0.58199999999999996,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":1.028,"5min":1.194,"15min":6.9720000000000004},"last":0.62}]},{"osd":2,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71399999999999997,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.119,"15min":5.9299999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.59899999999999998,"5min":0.48499999999999999,"15min":0.42299999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.704},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.60799999999999998,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":0.93799999999999994,"5min":1.2589999999999999,"15min":1.899},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51900000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94999999999999996,"5min":1.036,"15min":6.0620000000000003},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.83499999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.64000000000000001,"15min":0.63600000000000001},"min":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.97499999999999998,"15min":0.998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50900000000000001,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.81599999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.498}]},{"osd":7,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70699999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":0.83399999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76500000000000001},"min":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.89100000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:45:06.462 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:45:06.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:45:06.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.700+0000 7fd2d4fff700 1 Processor -- start 2023-10-07T09:45:06.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.700+0000 7fd2d4fff700 1 -- start start 2023-10-07T09:45:06.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.701+0000 7fd2d4fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2d00d0010 0x7fd2d00b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.701+0000 7fd2d4fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d00c62d0 con 0x7fd2d00cfbb0 2023-10-07T09:45:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.701+0000 7fd2d4fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d0159e10 con 0x7fd2d00cb800 2023-10-07T09:45:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.701+0000 7fd2d4fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d00d0410 con 0x7fd2d00d0010 2023-10-07T09:45:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2d00d0010 0x7fd2d00b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2d00d0010 0x7fd2d00b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60176/0 (socket says 172.21.15.29:60176) 2023-10-07T09:45:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 -- 172.21.15.29:0/4138386410 learned_addr learned my addr 172.21.15.29:0/4138386410 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 -- 172.21.15.29:0/4138386410 >> v1:172.21.15.144:6789/0 conn(0x7fd2d00cb800 legacy=0x7fd2d00cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 -- 172.21.15.29:0/4138386410 >> v1:172.21.15.29:6789/0 conn(0x7fd2d00cfbb0 legacy=0x7fd2d00c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 -- 172.21.15.29:0/4138386410 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d00c56c0 con 0x7fd2d00d0010 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.702+0000 7fd2ced9d700 1 --2- 172.21.15.29:0/4138386410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00d0010 0x7fd2d00b3340 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009f20 tx=0x7fd2c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785636e3e7044d96 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2cd59a700 1 -- 172.21.15.29:0/4138386410 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c800e070 con 0x7fd2d00d0010 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2cd59a700 1 -- 172.21.15.29:0/4138386410 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c800bb10 con 0x7fd2d00d0010 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2cd59a700 1 -- 172.21.15.29:0/4138386410 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80092e0 con 0x7fd2d00d0010 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 -- 172.21.15.29:0/4138386410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00d0010 msgr2=0x7fd2d00b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/4138386410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00d0010 0x7fd2d00b3340 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009f20 tx=0x7fd2c8009840 comp rx=0 tx=0).stop 2023-10-07T09:45:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 -- 172.21.15.29:0/4138386410 shutdown_connections 2023-10-07T09:45:06.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/4138386410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00d0010 0x7fd2d00b3340 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 -- 172.21.15.29:0/4138386410 >> 172.21.15.29:0/4138386410 conn(0x7fd2d0063a10 msgr2=0x7fd2d00d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:06.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 -- 172.21.15.29:0/4138386410 shutdown_connections 2023-10-07T09:45:06.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.703+0000 7fd2d4fff700 1 -- 172.21.15.29:0/4138386410 wait complete. 2023-10-07T09:45:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.705+0000 7fd2d4fff700 1 Processor -- start 2023-10-07T09:45:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.705+0000 7fd2d4fff700 1 -- start start 2023-10-07T09:45:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2d4fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 0x7fd2d00cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2d4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2d00cfbb0 0x7fd2d016d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2d4fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2d00d0010 0x7fd2d016d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2d4fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00ca9c0 con 0x7fd2d00cb800 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2d4fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d016df70 con 0x7fd2d00d0010 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2d4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d016e250 con 0x7fd2d00cfbb0 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 0x7fd2d00cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 0x7fd2d00cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60190/0 (socket says 172.21.15.29:60190) 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 -- 172.21.15.29:0/3544967865 learned_addr learned my addr 172.21.15.29:0/3544967865 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 -- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2d00cfbb0 msgr2=0x7fd2d016d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2d00cfbb0 0x7fd2d016d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ced9d700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2d00d0010 0x7fd2d016d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 -- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2d00d0010 msgr2=0x7fd2d016d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2d00d0010 0x7fd2d016d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.706+0000 7fd2ce59c700 1 -- 172.21.15.29:0/3544967865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8009ae0 con 0x7fd2d00cb800 2023-10-07T09:45:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2ced9d700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2d00d0010 0x7fd2d016d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2ce59c700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 0x7fd2d00cadf0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c7c0 tx=0x7fd2c0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0011590 con 0x7fd2d00cb800 2023-10-07T09:45:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c0011710 con 0x7fd2d00cb800 2023-10-07T09:45:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0011ad0 con 0x7fd2d00cb800 2023-10-07T09:45:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d0150fe0 con 0x7fd2d00cb800 2023-10-07T09:45:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.707+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d01515a0 con 0x7fd2d00cb800 2023-10-07T09:45:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.708+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d0152c40 con 0x7fd2d00cb800 2023-10-07T09:45:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.709+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd2c001b3b0 con 0x7fd2d00cb800 2023-10-07T09:45:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.710+0000 7fd2be7fc700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2b0066f30 0x7fd2b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.710+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd2c0083c60 con 0x7fd2d00cb800 2023-10-07T09:45:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.711+0000 7fd2cdd9b700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2b0066f30 0x7fd2b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.711+0000 7fd2cdd9b700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2b0066f30 0x7fd2b00693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0153000 tx=0x7fd2c4007b40 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:06.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.714+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2c0050d60 con 0x7fd2d00cb800 2023-10-07T09:45:06.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:06.996+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd2d00718b0 con 0x7fd2d00cb800 2023-10-07T09:45:07.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.000+0000 7fd2be7fc700 1 -- 172.21.15.29:0/3544967865 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd2c0017070 con 0x7fd2d00cb800 2023-10-07T09:45:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2b0066f30 msgr2=0x7fd2b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2b0066f30 0x7fd2b00693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0153000 tx=0x7fd2c4007b40 comp rx=0 tx=0).stop 2023-10-07T09:45:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 msgr2=0x7fd2d00cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 0x7fd2d00cadf0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c7c0 tx=0x7fd2c0007f70 comp rx=0 tx=0).stop 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 shutdown_connections 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2b0066f30 0x7fd2b00693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2d00d0010 0x7fd2d016d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d00cb800 0x7fd2d00cadf0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 --2- 172.21.15.29:0/3544967865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2d00cfbb0 0x7fd2d016d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 >> 172.21.15.29:0/3544967865 conn(0x7fd2d0063a10 msgr2=0x7fd2d00b5330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 shutdown_connections 2023-10-07T09:45:07.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.003+0000 7fd2d4fff700 1 -- 172.21.15.29:0/3544967865 wait complete. 2023-10-07T09:45:07.006 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:45:07.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:07.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.218+0000 7f95003ff700 1 Processor -- start 2023-10-07T09:45:07.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.218+0000 7f95003ff700 1 -- start start 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.219+0000 7f95003ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94f80b0710 0x7f94f815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f95003ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f8154090 con 0x7f94f8153cc0 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f95003ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f8159f60 con 0x7f94f815ade0 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f95003ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f815a240 con 0x7f94f80b0710 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe99c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94f80b0710 0x7f94f815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe99c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94f80b0710 0x7f94f815c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60198/0 (socket says 172.21.15.29:60198) 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f94f8153cc0 0x7f94f80b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32852/0 (socket says 172.21.15.29:32852) 2023-10-07T09:45:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe99c700 1 -- 172.21.15.29:0/4252895184 learned_addr learned my addr 172.21.15.29:0/4252895184 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe99c700 1 -- 172.21.15.29:0/4252895184 >> v1:172.21.15.144:6789/0 conn(0x7f94f815ade0 legacy=0x7f94f80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe99c700 1 -- 172.21.15.29:0/4252895184 >> v1:172.21.15.29:6789/0 conn(0x7f94f8153cc0 legacy=0x7f94f80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.220+0000 7f94fe99c700 1 -- 172.21.15.29:0/4252895184 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f815a520 con 0x7f94f80b0710 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f94fe99c700 1 --2- 172.21.15.29:0/4252895184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f80b0710 0x7f94f815c240 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009f20 tx=0x7f94f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0243c73356bb83b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f94fd199700 1 -- 172.21.15.29:0/4252895184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f000e070 con 0x7f94f80b0710 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f94fd199700 1 -- 172.21.15.29:0/4252895184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f000ba80 con 0x7f94f80b0710 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f94fd199700 1 -- 172.21.15.29:0/4252895184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00092e0 con 0x7f94f80b0710 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f95003ff700 1 -- 172.21.15.29:0/4252895184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f80b0710 msgr2=0x7f94f815c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f95003ff700 1 --2- 172.21.15.29:0/4252895184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f80b0710 0x7f94f815c240 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009f20 tx=0x7f94f0009840 comp rx=0 tx=0).stop 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f95003ff700 1 -- 172.21.15.29:0/4252895184 shutdown_connections 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f95003ff700 1 --2- 172.21.15.29:0/4252895184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f80b0710 0x7f94f815c240 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.221+0000 7f95003ff700 1 -- 172.21.15.29:0/4252895184 >> 172.21.15.29:0/4252895184 conn(0x7f94f8067610 msgr2=0x7f94f8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.222+0000 7f95003ff700 1 -- 172.21.15.29:0/4252895184 shutdown_connections 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.222+0000 7f95003ff700 1 -- 172.21.15.29:0/4252895184 wait complete. 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.223+0000 7f95003ff700 1 Processor -- start 2023-10-07T09:45:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.223+0000 7f95003ff700 1 -- start start 2023-10-07T09:45:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f95003ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94f80b0710 0x7f94f8156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f95003ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94f8153cc0 0x7f94f80c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f95003ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 0x7f94f81586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f95003ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80c8860 con 0x7f94f815ade0 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f95003ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f8157610 con 0x7f94f8153cc0 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f95003ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f81578f0 con 0x7f94f80b0710 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f94fe99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 0x7f94f81586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f94fe99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 0x7f94f81586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60212/0 (socket says 172.21.15.29:60212) 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f94fe99c700 1 -- 172.21.15.29:0/3748655693 learned_addr learned my addr 172.21.15.29:0/3748655693 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f94fd99a700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94f8153cc0 0x7f94f80c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.224+0000 7f94fe19b700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94f80b0710 0x7f94f8156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94fe99c700 1 -- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94f80b0710 msgr2=0x7f94f8156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94fe99c700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94f80b0710 0x7f94f8156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94fe99c700 1 -- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94f8153cc0 msgr2=0x7f94f80c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94fe99c700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94f8153cc0 0x7f94f80c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94fe99c700 1 -- 172.21.15.29:0/3748655693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f0009ae0 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94fe99c700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 0x7f94f81586b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f94f0012000 tx=0x7f94f0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0008d20 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f0008e80 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00266b0 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.225+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f8157bd0 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.226+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f8158190 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.226+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f804c3c0 con 0x7f94f815ade0 2023-10-07T09:45:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.227+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f94f0003040 con 0x7f94f815ade0 2023-10-07T09:45:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.228+0000 7f94ee7fc700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f94dc066f30 0x7f94dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.228+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f94f0086770 con 0x7f94f815ade0 2023-10-07T09:45:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.229+0000 7f94fe19b700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f94dc066f30 0x7f94dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:07.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.229+0000 7f94fe19b700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f94dc066f30 0x7f94dc0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94f40029f0 tx=0x7f94f4009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:07.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.233+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f94f00537c0 con 0x7f94f815ade0 2023-10-07T09:45:07.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.480+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f815a800 con 0x7f94f815ade0 2023-10-07T09:45:07.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.481+0000 7f94ee7fc700 1 -- 172.21.15.29:0/3748655693 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f94f0056de0 con 0x7f94f815ade0 2023-10-07T09:45:07.482 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:07.482 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:01.071366+0000","modified":"2023-10-07T09:45:04.584641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2799196407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2799196407},{"type":"v1","addr":"172.21.15.29:6835","nonce":2799196407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.483+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f94dc066f30 msgr2=0x7f94dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.483+0000 7f95003ff700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f94dc066f30 0x7f94dc0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94f40029f0 tx=0x7f94f4009040 comp rx=0 tx=0).stop 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.483+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 msgr2=0x7f94f81586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.483+0000 7f95003ff700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 0x7f94f81586b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f94f0012000 tx=0x7f94f0018000 comp rx=0 tx=0).stop 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 shutdown_connections 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f94dc066f30 0x7f94dc0693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f815ade0 0x7f94f81586b0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94f80b0710 0x7f94f8156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 --2- 172.21.15.29:0/3748655693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94f8153cc0 0x7f94f80c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:07.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 >> 172.21.15.29:0/3748655693 conn(0x7f94f8067610 msgr2=0x7f94f8152b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:07.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 shutdown_connections 2023-10-07T09:45:07.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:07.484+0000 7f95003ff700 1 -- 172.21.15.29:0/3748655693 wait complete. 2023-10-07T09:45:07.486 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-07T09:45:07.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:45:01.071366+0000', 'modified': '2023-10-07T09:45:04.584641+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6738': {'gid': 6738, 'name': 'c', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/2799196407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 2799196407}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2799196407}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:45:07.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:45:07.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-07T09:45:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.105+0000 7f3de7fff700 1 Processor -- start 2023-10-07T09:45:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.105+0000 7f3de7fff700 1 -- start start 2023-10-07T09:45:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.106+0000 7f3de7fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3de00ad370 0x7f3de00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.106+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 0x7f3de015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de7fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de014d790 con 0x7f3de0159ec0 2023-10-07T09:45:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de7fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de015a880 con 0x7f3de00ad370 2023-10-07T09:45:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00ae5d0 con 0x7f3de00b0410 2023-10-07T09:45:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 0x7f3de015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 0x7f3de015a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:33368/0 (socket says 172.21.15.29:33368) 2023-10-07T09:45:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de659c700 1 -- 172.21.15.29:0/290046511 learned_addr learned my addr 172.21.15.29:0/290046511 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/290046511 >> v2:172.21.15.144:3300/0 conn(0x7f3de00ad370 0x7f3de00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.107+0000 7f3de4d99700 1 -- 172.21.15.29:0/290046511 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697113802 0 0) 0x7f3de014d790 con 0x7f3de0159ec0 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de659c700 1 -- 172.21.15.29:0/290046511 >> v2:172.21.15.144:3300/0 conn(0x7f3de00ad370 msgr2=0x7f3de00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de659c700 1 --2- 172.21.15.29:0/290046511 >> v2:172.21.15.144:3300/0 conn(0x7f3de00ad370 0x7f3de00cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de659c700 1 -- 172.21.15.29:0/290046511 >> v1:172.21.15.144:6789/0 conn(0x7f3de0159ec0 legacy=0x7f3de01568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de659c700 1 -- 172.21.15.29:0/290046511 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00ad740 con 0x7f3de00b0410 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/290046511 >> v2:172.21.15.144:3300/0 conn(0x7f3de00ad370 0x7f3de00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de659c700 1 --2- 172.21.15.29:0/290046511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 0x7f3de015a340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009f50 tx=0x7f3dd8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b52bcc0fc831988 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de4d99700 1 -- 172.21.15.29:0/290046511 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd800e070 con 0x7f3de00b0410 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de4d99700 1 -- 172.21.15.29:0/290046511 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd800b9c0 con 0x7f3de00b0410 2023-10-07T09:45:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.108+0000 7f3de4d99700 1 -- 172.21.15.29:0/290046511 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80092e0 con 0x7f3de00b0410 2023-10-07T09:45:08.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 -- 172.21.15.29:0/290046511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 msgr2=0x7f3de015a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 --2- 172.21.15.29:0/290046511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 0x7f3de015a340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009f50 tx=0x7f3dd8009840 comp rx=0 tx=0).stop 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 -- 172.21.15.29:0/290046511 shutdown_connections 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 --2- 172.21.15.29:0/290046511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00b0410 0x7f3de015a340 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 --2- 172.21.15.29:0/290046511 >> v2:172.21.15.144:3300/0 conn(0x7f3de00ad370 0x7f3de00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 -- 172.21.15.29:0/290046511 >> 172.21.15.29:0/290046511 conn(0x7f3de00680a0 msgr2=0x7f3de00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 -- 172.21.15.29:0/290046511 shutdown_connections 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.109+0000 7f3de7fff700 1 -- 172.21.15.29:0/290046511 wait complete. 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.110+0000 7f3de7fff700 1 Processor -- start 2023-10-07T09:45:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.110+0000 7f3de7fff700 1 -- start start 2023-10-07T09:45:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.111+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00ad370 0x7f3de00a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.111+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 0x7f3de014cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.111+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de0159ec0 0x7f3de00a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.111+0000 7f3de7fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a5f70 con 0x7f3de00b0410 2023-10-07T09:45:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.111+0000 7f3de7fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a5640 con 0x7f3de0159ec0 2023-10-07T09:45:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.111+0000 7f3de7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a5920 con 0x7f3de00ad370 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 0x7f3de014cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 0x7f3de014cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60214/0 (socket says 172.21.15.29:60214) 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 -- 172.21.15.29:0/3902462568 learned_addr learned my addr 172.21.15.29:0/3902462568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00ad370 0x7f3de00a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de659c700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de0159ec0 0x7f3de00a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 -- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00ad370 msgr2=0x7f3de00a6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00ad370 0x7f3de00a6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 -- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de0159ec0 msgr2=0x7f3de00a9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de0159ec0 0x7f3de00a9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 -- 172.21.15.29:0/3902462568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc009420 con 0x7f3de00b0410 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.112+0000 7f3de559a700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 0x7f3de014cc90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000c530 tx=0x7f3dd000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.113+0000 7f3dd5ffb700 1 -- 172.21.15.29:0/3902462568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011590 con 0x7f3de00b0410 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.113+0000 7f3dd5ffb700 1 -- 172.21.15.29:0/3902462568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd0011710 con 0x7f3de00b0410 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.113+0000 7f3dd5ffb700 1 -- 172.21.15.29:0/3902462568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011ad0 con 0x7f3de00b0410 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.113+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd8009ae0 con 0x7f3de00b0410 2023-10-07T09:45:08.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.113+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de00a6940 con 0x7f3de00b0410 2023-10-07T09:45:08.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.115+0000 7f3dd5ffb700 1 -- 172.21.15.29:0/3902462568 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3dd001b3b0 con 0x7f3de00b0410 2023-10-07T09:45:08.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.116+0000 7f3dd5ffb700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc4066f30 0x7f3dc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.116+0000 7f3dd5ffb700 1 -- 172.21.15.29:0/3902462568 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3dd0083be0 con 0x7f3de00b0410 2023-10-07T09:45:08.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.116+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc4066f30 0x7f3dc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.117+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc4066f30 0x7f3dc40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009f40 tx=0x7f3ddc0091a0 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.117+0000 7f3df668db80 1 Processor -- start 2023-10-07T09:45:08.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 -- start start 2023-10-07T09:45:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb8eef20 0x55ddcb91f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 0x55ddcb8ef2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb80c310 con 0x55ddcb91bf10 2023-10-07T09:45:08.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb849130 con 0x55ddcb8eef20 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.118+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb846af0 con 0x55ddcb91ef00 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 0x55ddcb8ef2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 0x55ddcb8ef2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60226/0 (socket says 172.21.15.29:60226) 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 -- 172.21.15.29:0/2724384261 learned_addr learned my addr 172.21.15.29:0/2724384261 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de659c700 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb8eef20 0x55ddcb91f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 -- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 msgr2=0x55ddcb91fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 -- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb8eef20 msgr2=0x55ddcb91f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb8eef20 0x55ddcb91f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 -- 172.21.15.29:0/2724384261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddcb7350a0 con 0x55ddcb91bf10 2023-10-07T09:45:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de659c700 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:08.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.119+0000 7f3de559a700 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 0x55ddcb8ef2f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0054880 tx=0x7f3dd0081e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb585ac4de2bb33 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3dc2ffd700 1 -- 172.21.15.29:0/2724384261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00548f0 con 0x55ddcb91bf10 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3dc2ffd700 1 -- 172.21.15.29:0/2724384261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd0068070 con 0x55ddcb91bf10 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3dc2ffd700 1 -- 172.21.15.29:0/2724384261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0057b40 con 0x55ddcb91bf10 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 -- 172.21.15.29:0/2724384261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 msgr2=0x55ddcb8ef2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 0x55ddcb8ef2f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0054880 tx=0x7f3dd0081e40 comp rx=0 tx=0).stop 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 -- 172.21.15.29:0/2724384261 shutdown_connections 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb8eef20 0x55ddcb91f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 --2- 172.21.15.29:0/2724384261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91bf10 0x55ddcb8ef2f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 -- 172.21.15.29:0/2724384261 >> 172.21.15.29:0/2724384261 conn(0x55ddcb93ce70 msgr2=0x55ddcb93d240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 -- 172.21.15.29:0/2724384261 shutdown_connections 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.120+0000 7f3df668db80 1 -- 172.21.15.29:0/2724384261 wait complete. 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.121+0000 7f3df668db80 1 -- start start 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.121+0000 7f3df668db80 1 Processor -- start 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb921e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb91ef00 0x55ddcb918580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 0x55ddcb918ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb8428e0 con 0x55ddcb922360 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb7ee980 con 0x55ddcb91ef00 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb7ee620 con 0x55ddcb8eef20 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.122+0000 7f3df668db80 10 client.0 _ll_register_callbacks cb 0x55ddcb651a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 0x55ddcb918ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 0x55ddcb918ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60238/0 (socket says 172.21.15.29:60238) 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 -- 172.21.15.29:0/3875718050 learned_addr learned my addr 172.21.15.29:0/3875718050 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de559a700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb91ef00 0x55ddcb918580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb921e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 -- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 msgr2=0x55ddcb921e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb921e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 -- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb91ef00 msgr2=0x55ddcb918580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb91ef00 0x55ddcb918580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 -- 172.21.15.29:0/3875718050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de00a6940 con 0x55ddcb922360 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb921e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3de659c700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 0x55ddcb918ac0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3dd801eb90 tx=0x7f3dd8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a7af14b112d089 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.123+0000 7f3dbbfff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:45:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3dbbfff700 1 -- 172.21.15.29:0/3875718050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8023070 con 0x55ddcb922360 2023-10-07T09:45:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ddcb8e7560 con 0x55ddcb922360 2023-10-07T09:45:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3dbbfff700 1 -- 172.21.15.29:0/3875718050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3dd80092e0 con 0x55ddcb922360 2023-10-07T09:45:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3dbbfff700 1 -- 172.21.15.29:0/3875718050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dd8027640 con 0x55ddcb922360 2023-10-07T09:45:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 10 client.6876 fetch_fsmap learned FSMap version 42 2023-10-07T09:45:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55ddcb8113c0 con 0x55ddcb922360 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3dbbfff700 1 -- 172.21.15.29:0/3875718050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ddcb8e7560 con 0x55ddcb922360 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3dbbfff700 1 -- 172.21.15.29:0/3875718050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 42) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f3dd800b050 con 0x55ddcb922360 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 10 client.6876 fetch_fsmap finished waiting for FSMap version 42 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 10 client.6876 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 20 client.6876 populate_metadata read hostname 'smithi029' 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb919f20 0x55ddcb95acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 4 client.6876 mds_command: new command op to 6738 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.124+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 --> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ddcb8c40e0 con 0x55ddcb919f20 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.125+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb919f20 0x55ddcb95acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.125+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb919f20 0x55ddcb95acd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8041f076511e3336 server_cookie=b00d6d3542c6b3be in_seq=0 out_seq=0 2023-10-07T09:45:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.125+0000 7f3dbbfff700 10 client.6876 ms_handle_connect on v2:172.21.15.29:6832/2799196407 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3dbbfff700 1 -- 172.21.15.29:0/3875718050 <== mds.0 v2:172.21.15.29:6832/2799196407 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55ddcb8c40e0 con 0x55ddcb919f20 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3dbbfff700 10 client.6876 handle_command_reply: tid=0 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 client.6876 shutdown 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 msgr2=0x55ddcb918ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 0x55ddcb918ac0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3dd801eb90 tx=0x7f3dd8018000 comp rx=0 tx=0).stop 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 shutdown_connections 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb919f20 0x55ddcb95acd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb922360 0x55ddcb918ac0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb921e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 --2- 172.21.15.29:0/3875718050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb91ef00 0x55ddcb918580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3dbbfff700 0 client.6876 ms_handle_reset on v2:172.21.15.29:6832/2799196407 2023-10-07T09:45:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.126+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 >> 172.21.15.29:0/3875718050 conn(0x55ddcb93ce70 msgr2=0x55ddcb93d240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.127+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 shutdown_connections 2023-10-07T09:45:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.127+0000 7f3df668db80 1 -- 172.21.15.29:0/3875718050 wait complete. 2023-10-07T09:45:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.127+0000 7f3df668db80 20 client.6876 trim_cache size 0 max 16384 2023-10-07T09:45:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.144+0000 7f3df668db80 1 Processor -- start 2023-10-07T09:45:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.144+0000 7f3df668db80 1 -- start start 2023-10-07T09:45:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.145+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb8e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.145+0000 7f3df668db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 0x55ddcb921200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.145+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb918580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.145+0000 7f3df668db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb6da1e0 con 0x55ddcb91ef00 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.145+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb91c390 con 0x55ddcb922360 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.145+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb8eff30 con 0x55ddcb8eef20 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 0x55ddcb921200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 0x55ddcb921200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60248/0 (socket says 172.21.15.29:60248) 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 -- 172.21.15.29:0/3426733978 learned_addr learned my addr 172.21.15.29:0/3426733978 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb8e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de659c700 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb918580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 -- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 msgr2=0x55ddcb8e99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb8e99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 -- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 msgr2=0x55ddcb918580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb918580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 -- 172.21.15.29:0/3426733978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddcb6a91d0 con 0x55ddcb91ef00 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3de559a700 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 0x55ddcb921200 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000cb90 tx=0x7f3dd0068000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6731a8a43da85d80 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3dc37fe700 1 -- 172.21.15.29:0/3426733978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0081070 con 0x55ddcb91ef00 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3dc37fe700 1 -- 172.21.15.29:0/3426733978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd00546c0 con 0x55ddcb91ef00 2023-10-07T09:45:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.146+0000 7f3dc37fe700 1 -- 172.21.15.29:0/3426733978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd005b890 con 0x55ddcb91ef00 2023-10-07T09:45:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 -- 172.21.15.29:0/3426733978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 msgr2=0x55ddcb921200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 0x55ddcb921200 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000cb90 tx=0x7f3dd0068000 comp rx=0 tx=0).stop 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 -- 172.21.15.29:0/3426733978 shutdown_connections 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb918580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb8eef20 0x55ddcb8e99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 --2- 172.21.15.29:0/3426733978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb91ef00 0x55ddcb921200 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 -- 172.21.15.29:0/3426733978 >> 172.21.15.29:0/3426733978 conn(0x55ddcb93ce70 msgr2=0x55ddcb8f4730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 -- 172.21.15.29:0/3426733978 shutdown_connections 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.147+0000 7f3df668db80 1 -- 172.21.15.29:0/3426733978 wait complete. 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.148+0000 7f3df668db80 1 -- start start 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.148+0000 7f3df668db80 1 Processor -- start 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 0x55ddcb94c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb91dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb93c880 con 0x55ddcb8eef20 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb91e1d0 con 0x55ddcb922360 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddcb8e9260 con 0x55ddcb91ef00 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3df668db80 10 client.0 _ll_register_callbacks cb 0x55ddcb651a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 0x55ddcb94c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 0x55ddcb94c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60262/0 (socket says 172.21.15.29:60262) 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 -- 172.21.15.29:0/444897085 learned_addr learned my addr 172.21.15.29:0/444897085 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 -- 172.21.15.29:0/444897085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 msgr2=0x55ddcb91d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 -- 172.21.15.29:0/444897085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 msgr2=0x55ddcb91dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.149+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb91dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3de5d9b700 1 -- 172.21.15.29:0/444897085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd006e040 con 0x55ddcb8eef20 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3de5d9b700 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 0x55ddcb94c290 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0119e0 tx=0x7f3ddc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6339a0d3abf665a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3dbbfff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3dbbfff700 1 -- 172.21.15.29:0/444897085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc031070 con 0x55ddcb8eef20 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3dbbfff700 1 -- 172.21.15.29:0/444897085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3ddc025020 con 0x55ddcb8eef20 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3de014d540 con 0x55ddcb8eef20 2023-10-07T09:45:08.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3dbbfff700 1 -- 172.21.15.29:0/444897085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ddc02a070 con 0x55ddcb8eef20 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3df668db80 10 client.6882 fetch_fsmap learned FSMap version 42 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.150+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55ddcb682d60 con 0x55ddcb8eef20 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3dbbfff700 1 -- 172.21.15.29:0/444897085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de014d540 con 0x55ddcb8eef20 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3dbbfff700 1 -- 172.21.15.29:0/444897085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 42) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f3ddc034020 con 0x55ddcb8eef20 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3df668db80 10 client.6882 fetch_fsmap finished waiting for FSMap version 42 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3df668db80 10 client.6882 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3df668db80 20 client.6882 populate_metadata read hostname 'smithi029' 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3df668db80 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb9183c0 0x55ddcb918790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3df668db80 4 client.6882 mds_command: new command op to 6738 tid=0[{"prefix": "status"}] 2023-10-07T09:45:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 --> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f3dd0068070 con 0x55ddcb9183c0 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3de559a700 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb9183c0 0x55ddcb918790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3de559a700 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb9183c0 0x55ddcb918790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6366a5f5c29a47ab server_cookie=29e56a9701dcc2ec in_seq=0 out_seq=0 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.151+0000 7f3dbbfff700 10 client.6882 ms_handle_connect on v2:172.21.15.29:6832/2799196407 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3dbbfff700 1 -- 172.21.15.29:0/444897085 <== mds.0 v2:172.21.15.29:6832/2799196407 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f3dd0068070 con 0x55ddcb9183c0 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3dbbfff700 10 client.6882 handle_command_reply: tid=0 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 client.6882 shutdown 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 msgr2=0x55ddcb94c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 0x55ddcb94c290 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0119e0 tx=0x7f3ddc02a250 comp rx=0 tx=0).stop 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 shutdown_connections 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ddcb922360 0x55ddcb91dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:6832/2799196407,v1:172.21.15.29:6835/2799196407] conn(0x55ddcb9183c0 0x55ddcb918790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ddcb8eef20 0x55ddcb94c290 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 --2- 172.21.15.29:0/444897085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ddcb91ef00 0x55ddcb91d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3dbbfff700 0 client.6882 ms_handle_reset on v2:172.21.15.29:6832/2799196407 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 >> 172.21.15.29:0/444897085 conn(0x55ddcb93ce70 msgr2=0x55ddcb93d240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 shutdown_connections 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 1 -- 172.21.15.29:0/444897085 wait complete. 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.152+0000 7f3df668db80 20 client.6882 trim_cache size 0 max 16384 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:45:08.159 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 6738, 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5902402909999998, 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 42, 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 90, 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 90, 2023-10-07T09:45:08.160 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.263184206 2023-10-07T09:45:08.161 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:45:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc4066f30 msgr2=0x7f3dc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc4066f30 0x7f3dc40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009f40 tx=0x7f3ddc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 msgr2=0x7f3de014cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 0x7f3de014cc90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000c530 tx=0x7f3dd000c840 comp rx=0 tx=0).stop 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 shutdown_connections 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc4066f30 0x7f3dc40693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de0159ec0 0x7f3de00a9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de00ad370 0x7f3de00a6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 --2- 172.21.15.29:0/3902462568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de00b0410 0x7f3de014cc90 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 >> 172.21.15.29:0/3902462568 conn(0x7f3de00680a0 msgr2=0x7f3de0158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 shutdown_connections 2023-10-07T09:45:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.153+0000 7f3de7fff700 1 -- 172.21.15.29:0/3902462568 wait complete. 2023-10-07T09:45:08.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:45:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.364+0000 7ff8fc9ff700 1 Processor -- start 2023-10-07T09:45:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.365+0000 7ff8fc9ff700 1 -- start start 2023-10-07T09:45:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8fc9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8f80ae8b0 0x7ff8f80b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8fc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8fc9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f8158ad0 con 0x7ff8f8157290 2023-10-07T09:45:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8fc9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f80b1cc0 con 0x7ff8f80ae8b0 2023-10-07T09:45:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8fc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f815b140 con 0x7ff8f80b1840 2023-10-07T09:45:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8f5d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8f80ae8b0 0x7ff8f80b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8f659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff8f8157290 0x7ff8f80cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32864/0 (socket says 172.21.15.29:32864) 2023-10-07T09:45:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8f5d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8f80ae8b0 0x7ff8f80b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60266/0 (socket says 172.21.15.29:60266) 2023-10-07T09:45:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.366+0000 7ff8f659c700 1 -- 172.21.15.29:0/1258696340 learned_addr learned my addr 172.21.15.29:0/1258696340 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f6d9d700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/1258696340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 msgr2=0x7ff8f8157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f5d9b700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/1258696340 >> v1:172.21.15.29:6789/0 conn(0x7ff8f8157290 legacy=0x7ff8f80cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/1258696340 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f815b420 con 0x7ff8f80ae8b0 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f559a700 1 -- 172.21.15.29:0/1258696340 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469619125 0 0) 0x7ff8f8158ad0 con 0x7ff8f8157290 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f6d9d700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f5d9b700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ae8b0 0x7ff8f80b1300 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0098e0 tx=0x7ff8ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad13e22534b0e6a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f559a700 1 -- 172.21.15.29:0/1258696340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec009e80 con 0x7ff8f80ae8b0 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f559a700 1 -- 172.21.15.29:0/1258696340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec00b860 con 0x7ff8f80ae8b0 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.367+0000 7ff8f559a700 1 -- 172.21.15.29:0/1258696340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec00bbe0 con 0x7ff8f80ae8b0 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/1258696340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ae8b0 msgr2=0x7ff8f80b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ae8b0 0x7ff8f80b1300 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0098e0 tx=0x7ff8ec0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/1258696340 shutdown_connections 2023-10-07T09:45:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/1258696340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ae8b0 0x7ff8f80b1300 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/1258696340 >> 172.21.15.29:0/1258696340 conn(0x7ff8f80680a0 msgr2=0x7ff8f80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/1258696340 shutdown_connections 2023-10-07T09:45:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.368+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/1258696340 wait complete. 2023-10-07T09:45:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.370+0000 7ff8fc9ff700 1 Processor -- start 2023-10-07T09:45:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.370+0000 7ff8fc9ff700 1 -- start start 2023-10-07T09:45:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8fc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8fc9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 0x7ff8f8141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8fc9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f8141bb0 0x7ff8f815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8fc9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f80c5590 con 0x7ff8f8157290 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8fc9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8157660 con 0x7ff8f8141bb0 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8fc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f81578b0 con 0x7ff8f80b1840 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 0x7ff8f8141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 0x7ff8f8141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60274/0 (socket says 172.21.15.29:60274) 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/423761358 learned_addr learned my addr 172.21.15.29:0/423761358 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f659c700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f6d9d700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f8141bb0 0x7ff8f815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 msgr2=0x7ff8f8155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f8141bb0 msgr2=0x7ff8f815df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f8141bb0 0x7ff8f815df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f5d9b700 1 -- 172.21.15.29:0/423761358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec009420 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f659c700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.371+0000 7ff8f6d9d700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f8141bb0 0x7ff8f815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.372+0000 7ff8f5d9b700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 0x7ff8f8141670 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8068dd0 tx=0x7ff8ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.372+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec00bbe0 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.372+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec011420 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.372+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8157b90 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.372+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0117e0 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.372+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f8158150 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.373+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f806f000 con 0x7ff8f8157290 2023-10-07T09:45:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.374+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff8ec011bd0 con 0x7ff8f8157290 2023-10-07T09:45:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.375+0000 7ff8e67fc700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8d8066f30 0x7ff8d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.375+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7ff8ec086190 con 0x7ff8f8157290 2023-10-07T09:45:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.375+0000 7ff8f659c700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8d8066f30 0x7ff8d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.376+0000 7ff8f659c700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8d8066f30 0x7ff8d80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8001840 tx=0x7ff8e8009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.379+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8ec053280 con 0x7ff8f8157290 2023-10-07T09:45:08.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.627+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff8f8158370 con 0x7ff8f8157290 2023-10-07T09:45:08.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.627+0000 7ff8e67fc700 1 -- 172.21.15.29:0/423761358 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff8ec0568a0 con 0x7ff8f8157290 2023-10-07T09:45:08.628 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:45:08.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8d8066f30 msgr2=0x7ff8d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8d8066f30 0x7ff8d80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8001840 tx=0x7ff8e8009040 comp rx=0 tx=0).stop 2023-10-07T09:45:08.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 msgr2=0x7ff8f8141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:08.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 0x7ff8f8141670 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8068dd0 tx=0x7ff8ec017000 comp rx=0 tx=0).stop 2023-10-07T09:45:08.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 shutdown_connections 2023-10-07T09:45:08.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff8d8066f30 0x7ff8d80693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f8141bb0 0x7ff8f815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f80b1840 0x7ff8f8155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 --2- 172.21.15.29:0/423761358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f8157290 0x7ff8f8141670 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 >> 172.21.15.29:0/423761358 conn(0x7ff8f80680a0 msgr2=0x7ff8f80b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 shutdown_connections 2023-10-07T09:45:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:08.630+0000 7ff8fc9ff700 1 -- 172.21.15.29:0/423761358 wait complete. 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:45:08.649 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:45:08.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:08.650 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:45:08.696 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:45:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41320sec preferred_lft 41320sec 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:45:08.698 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:45:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:08.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:08.701 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:45:08.809 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:45:08.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:08.812 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:45:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:45:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:45:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:45:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:45:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:45:08.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:08.863 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:45:08.863 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:45:08.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:45:08.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:45:08.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:45:08.864 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:45:09.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:09.037 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:45:09.083 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:45:09.084 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:45:09.085 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:45:09.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:09.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:45:09.157 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:45:09.157 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:09.204 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:45:09.204 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:45:09.204 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:09.290 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:45:09.290 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:45:09.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:09.431 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:09.432 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:45:09.432 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:45:09.446 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:09.446 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:45:09.446 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:45:09.446 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:45:09.446 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:45:09.467 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:09.467 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:45:09.468 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:45:09.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:45:09.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:45:09.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:45:09.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:45:09.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:45:09.473 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:45:09.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:09.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:09.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:45:09.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.745+0000 7f2f51bff700 1 Processor -- start 2023-10-07T09:45:09.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.745+0000 7f2f51bff700 1 -- start start 2023-10-07T09:45:09.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.746+0000 7f2f51bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2f4c159b20 0x7f2f4c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:09.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.746+0000 7f2f51bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:09.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.746+0000 7f2f51bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c0cbf20 con 0x7f2f4c159750 2023-10-07T09:45:09.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.746+0000 7f2f51bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4c0cc200 con 0x7f2f4c159b20 2023-10-07T09:45:09.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.746+0000 7f2f51bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c0cc4e0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4affd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2f4c159b20 0x7f2f4c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:33444/0 (socket says 172.21.15.29:33444) 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4bfff700 1 -- 172.21.15.29:0/271434161 learned_addr learned my addr 172.21.15.29:0/271434161 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4b7fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f2f4c159750 0x7f2f4c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:56276/0 (socket says 172.21.15.29:56276) 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4a7fc700 1 -- 172.21.15.29:0/271434161 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885471288 0 0) 0x7f2f4c0cbf20 con 0x7f2f4c159750 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4a7fc700 1 -- 172.21.15.29:0/271434161 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f34003570 con 0x7f2f4c159750 2023-10-07T09:45:09.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4affd700 1 -- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 msgr2=0x7f2f4c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4affd700 1 --2- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4affd700 1 -- 172.21.15.29:0/271434161 >> v1:172.21.15.144:6789/0 conn(0x7f2f4c159750 legacy=0x7f2f4c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4affd700 1 -- 172.21.15.29:0/271434161 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c0cc7c0 con 0x7f2f4c159b20 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.747+0000 7f2f4bfff700 1 --2- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f4affd700 1 --2- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159b20 0x7f2f4c0d10a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2f400098e0 tx=0x7f2f400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d1e514eee5f8f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f4a7fc700 1 -- 172.21.15.29:0/271434161 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40011630 con 0x7f2f4c159b20 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f4a7fc700 1 -- 172.21.15.29:0/271434161 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40011790 con 0x7f2f4c159b20 2023-10-07T09:45:09.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f4a7fc700 1 -- 172.21.15.29:0/271434161 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40011b30 con 0x7f2f4c159b20 2023-10-07T09:45:09.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f51bff700 1 -- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159b20 msgr2=0x7f2f4c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:09.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f51bff700 1 --2- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159b20 0x7f2f4c0d10a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2f400098e0 tx=0x7f2f400091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f51bff700 1 -- 172.21.15.29:0/271434161 shutdown_connections 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f51bff700 1 --2- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f51bff700 1 --2- 172.21.15.29:0/271434161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159b20 0x7f2f4c0d10a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.748+0000 7f2f51bff700 1 -- 172.21.15.29:0/271434161 >> 172.21.15.29:0/271434161 conn(0x7f2f4c063a10 msgr2=0x7f2f4c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.749+0000 7f2f51bff700 1 -- 172.21.15.29:0/271434161 shutdown_connections 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.749+0000 7f2f51bff700 1 -- 172.21.15.29:0/271434161 wait complete. 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.750+0000 7f2f51bff700 1 Processor -- start 2023-10-07T09:45:09.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.750+0000 7f2f51bff700 1 -- start start 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f51bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f51bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159750 0x7f2f4c068690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f51bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c159b20 0x7f2f4c153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f51bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c15b820 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f51bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c15b560 con 0x7f2f4c159750 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f51bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c0c8de0 con 0x7f2f4c159b20 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:09.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f4b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60284/0 (socket says 172.21.15.29:60284) 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.751+0000 7f2f4b7fe700 1 -- 172.21.15.29:0/2407523101 learned_addr learned my addr 172.21.15.29:0/2407523101 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4bfff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c159b20 0x7f2f4c153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4affd700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159750 0x7f2f4c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4b7fe700 1 -- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c159b20 msgr2=0x7f2f4c153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4b7fe700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c159b20 0x7f2f4c153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4b7fe700 1 -- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159750 msgr2=0x7f2f4c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4b7fe700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159750 0x7f2f4c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4b7fe700 1 -- 172.21.15.29:0/2407523101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f44009ae0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f4b7fe700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c9f80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00c790 tx=0x7f2f3c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c005ac0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c005c20 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0070d0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.752+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f40009420 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.753+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4c0c8b00 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.754+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4c04c3c0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.754+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f3c007480 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.756+0000 7f2f337fe700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2f38066f30 0x7f2f380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:09.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.756+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f2f3c083bd0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.756+0000 7f2f4affd700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2f38066f30 0x7f2f380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:09.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.757+0000 7f2f4affd700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2f38066f30 0x7f2f380693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2f400091e0 tx=0x7f2f40017000 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.760+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f3c04b5a0 con 0x7f2f4c0ce8a0 2023-10-07T09:45:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.987+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f4c064040 con 0x7f2f38066f30 2023-10-07T09:45:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.990+0000 7f2f337fe700 1 -- 172.21.15.29:0/2407523101 <== mgr.5885 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2f4c064040 con 0x7f2f38066f30 2023-10-07T09:45:09.990 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:45:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:45:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.991+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2f38066f30 msgr2=0x7f2f380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.991+0000 7f2f51bff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2f38066f30 0x7f2f380693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2f400091e0 tx=0x7f2f40017000 comp rx=0 tx=0).stop 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 msgr2=0x7f2f4c0c9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c9f80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00c790 tx=0x7f2f3c007df0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 shutdown_connections 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2f38066f30 0x7f2f380693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2f4c159b20 0x7f2f4c153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f4c0ce8a0 0x7f2f4c0c9f80 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 --2- 172.21.15.29:0/2407523101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f4c159750 0x7f2f4c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 >> 172.21.15.29:0/2407523101 conn(0x7f2f4c063a10 msgr2=0x7f2f4c0c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 shutdown_connections 2023-10-07T09:45:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:09.992+0000 7f2f51bff700 1 -- 172.21.15.29:0/2407523101 wait complete. 2023-10-07T09:45:10.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:45:10.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.207+0000 7f66f71ff700 1 Processor -- start 2023-10-07T09:45:10.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.208+0000 7f66f71ff700 1 -- start start 2023-10-07T09:45:10.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f71ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66f00b0710 0x7f66f015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f71ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f0154090 con 0x7f66f0153cc0 2023-10-07T09:45:10.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f71ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f0159f60 con 0x7f66f015ade0 2023-10-07T09:45:10.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f71ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f015a240 con 0x7f66f00b0710 2023-10-07T09:45:10.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f579c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66f00b0710 0x7f66f015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f4f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f66f0153cc0 0x7f66f00b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32880/0 (socket says 172.21.15.29:32880) 2023-10-07T09:45:10.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.209+0000 7f66f4f9b700 1 -- 172.21.15.29:0/1947058468 learned_addr learned my addr 172.21.15.29:0/1947058468 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66ef7fe700 1 -- 172.21.15.29:0/1947058468 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212022265 0 0) 0x7f66f0154090 con 0x7f66f0153cc0 2023-10-07T09:45:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66f579c700 1 -- 172.21.15.29:0/1947058468 >> v1:172.21.15.144:6789/0 conn(0x7f66f015ade0 legacy=0x7f66f00b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66f579c700 1 -- 172.21.15.29:0/1947058468 >> v1:172.21.15.29:6789/0 conn(0x7f66f0153cc0 legacy=0x7f66f00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66f579c700 1 -- 172.21.15.29:0/1947058468 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f015a520 con 0x7f66f00b0710 2023-10-07T09:45:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66f579c700 1 --2- 172.21.15.29:0/1947058468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00b0710 0x7f66f015c240 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009e60 tx=0x7f66e8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721b20645d6c87d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66ef7fe700 1 -- 172.21.15.29:0/1947058468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e800e070 con 0x7f66f00b0710 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66ef7fe700 1 -- 172.21.15.29:0/1947058468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e800bc30 con 0x7f66f00b0710 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.210+0000 7f66ef7fe700 1 -- 172.21.15.29:0/1947058468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8012430 con 0x7f66f00b0710 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 -- 172.21.15.29:0/1947058468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00b0710 msgr2=0x7f66f015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1947058468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00b0710 0x7f66f015c240 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009e60 tx=0x7f66e8009780 comp rx=0 tx=0).stop 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 -- 172.21.15.29:0/1947058468 shutdown_connections 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1947058468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00b0710 0x7f66f015c240 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 -- 172.21.15.29:0/1947058468 >> 172.21.15.29:0/1947058468 conn(0x7f66f0067610 msgr2=0x7f66f0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 -- 172.21.15.29:0/1947058468 shutdown_connections 2023-10-07T09:45:10.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.211+0000 7f66f71ff700 1 -- 172.21.15.29:0/1947058468 wait complete. 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.212+0000 7f66f71ff700 1 Processor -- start 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.213+0000 7f66f71ff700 1 -- start start 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.213+0000 7f66f71ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66f00b0710 0x7f66f015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.213+0000 7f66f71ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66f0153cc0 0x7f66f01583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.213+0000 7f66f71ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 0x7f66f015d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f71ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f015b770 con 0x7f66f015ade0 2023-10-07T09:45:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f71ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0158ba0 con 0x7f66f00b0710 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f71ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0155790 con 0x7f66f0153cc0 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 0x7f66f015d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 0x7f66f015d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60310/0 (socket says 172.21.15.29:60310) 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 -- 172.21.15.29:0/1576358760 learned_addr learned my addr 172.21.15.29:0/1576358760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f4f9b700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66f00b0710 0x7f66f015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66effff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66f0153cc0 0x7f66f01583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 -- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66f0153cc0 msgr2=0x7f66f01583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66f0153cc0 0x7f66f01583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 -- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66f00b0710 msgr2=0x7f66f015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66f00b0710 0x7f66f015b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.214+0000 7f66f579c700 1 -- 172.21.15.29:0/1576358760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e8009a20 con 0x7f66f015ade0 2023-10-07T09:45:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.215+0000 7f66f579c700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 0x7f66f015d220 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f66f00b0be0 tx=0x7f66e8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.215+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e801b630 con 0x7f66f015ade0 2023-10-07T09:45:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.215+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e801b790 con 0x7f66f015ade0 2023-10-07T09:45:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.215+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e801bb50 con 0x7f66f015ade0 2023-10-07T09:45:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.215+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f0158900 con 0x7f66f015ade0 2023-10-07T09:45:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.215+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f0155d10 con 0x7f66f015ade0 2023-10-07T09:45:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.216+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f004c3c0 con 0x7f66f015ade0 2023-10-07T09:45:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.217+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f66e801c400 con 0x7f66f015ade0 2023-10-07T09:45:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.218+0000 7f66ecff9700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f66d4066f30 0x7f66d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.218+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f66e8086650 con 0x7f66f015ade0 2023-10-07T09:45:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.218+0000 7f66f4f9b700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f66d4066f30 0x7f66d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.219+0000 7f66f4f9b700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f66d4066f30 0x7f66d40693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66e4001360 tx=0x7f66e4009040 comp rx=0 tx=0).ready entity=mgr.5885 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.222+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66e80536a0 con 0x7f66f015ade0 2023-10-07T09:45:10.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.475+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f66f015a800 con 0x7f66f015ade0 2023-10-07T09:45:10.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.618+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 34) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f66e8053a70 con 0x7f66f015ade0 2023-10-07T09:45:10.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.619+0000 7f66ecff9700 1 -- 172.21.15.29:0/1576358760 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v34) v1 ==== 92+0+0 (secure 0 0 0) 0x7f66e8057130 con 0x7f66f015ade0 2023-10-07T09:45:10.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f66d4066f30 msgr2=0x7f66d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f66d4066f30 0x7f66d40693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66e4001360 tx=0x7f66e4009040 comp rx=0 tx=0).stop 2023-10-07T09:45:10.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 msgr2=0x7f66f015d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 0x7f66f015d220 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f66f00b0be0 tx=0x7f66e8019000 comp rx=0 tx=0).stop 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 shutdown_connections 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f66d4066f30 0x7f66d40693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f015ade0 0x7f66f015d220 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66f00b0710 0x7f66f015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 --2- 172.21.15.29:0/1576358760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66f0153cc0 0x7f66f01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 >> 172.21.15.29:0/1576358760 conn(0x7f66f0067610 msgr2=0x7f66f00b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 shutdown_connections 2023-10-07T09:45:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.622+0000 7f66f71ff700 1 -- 172.21.15.29:0/1576358760 wait complete. 2023-10-07T09:45:10.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:45:10.829 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:10.828+0000 7f9bca6ff1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:45:10.832 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:10.831+0000 7f92086ff1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:45:10.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.844+0000 7f4dc75ff700 1 Processor -- start 2023-10-07T09:45:10.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.844+0000 7f4dc75ff700 1 -- start start 2023-10-07T09:45:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc75ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dc015a960 0x7f4dc0158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc75ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00acb40 con 0x7f4dc015ad30 2023-10-07T09:45:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc75ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00abf30 con 0x7f4dc00ae8a0 2023-10-07T09:45:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc75ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc0158c60 con 0x7f4dc015a960 2023-10-07T09:45:10.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc539b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dc015a960 0x7f4dc0158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc4b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4dc015ad30 0x7f4dc00ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32892/0 (socket says 172.21.15.29:32892) 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.846+0000 7f4dc4b9a700 1 -- 172.21.15.29:0/1147389985 learned_addr learned my addr 172.21.15.29:0/1147389985 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365283989 0 0) 0x7f4dc00acb40 con 0x7f4dc015ad30 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac003570 con 0x7f4dc015ad30 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638669799 0 0) 0x7f4dc00abf30 con 0x7f4dc00ae8a0 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc00acb40 con 0x7f4dc00ae8a0 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4dc539b700 1 -- 172.21.15.29:0/1147389985 >> v1:172.21.15.144:6789/0 conn(0x7f4dc00ae8a0 legacy=0x7f4dc00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4dc539b700 1 -- 172.21.15.29:0/1147389985 >> v1:172.21.15.29:6789/0 conn(0x7f4dc015ad30 legacy=0x7f4dc00ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4dc539b700 1 -- 172.21.15.29:0/1147389985 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc00ac210 con 0x7f4dc015a960 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4dc539b700 1 --2- 172.21.15.29:0/1147389985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015a960 0x7f4dc0158e20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0098e0 tx=0x7f4dbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187bb9a5e3c3f6c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc011630 con 0x7f4dc015a960 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.847+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc011790 con 0x7f4dc015a960 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4db7fff700 1 -- 172.21.15.29:0/1147389985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc011b30 con 0x7f4dc015a960 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1147389985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015a960 msgr2=0x7f4dc0158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1147389985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015a960 0x7f4dc0158e20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0098e0 tx=0x7f4dbc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1147389985 shutdown_connections 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1147389985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015a960 0x7f4dc0158e20 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1147389985 >> 172.21.15.29:0/1147389985 conn(0x7f4dc0067610 msgr2=0x7f4dc0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1147389985 shutdown_connections 2023-10-07T09:45:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.848+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1147389985 wait complete. 2023-10-07T09:45:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.850+0000 7f4dc75ff700 1 Processor -- start 2023-10-07T09:45:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.850+0000 7f4dc75ff700 1 -- start start 2023-10-07T09:45:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc75ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dc00ae8a0 0x7f4dc015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc75ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dc015a960 0x7f4dc013dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc75ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 0x7f4dc015e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc75ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc015e9a0 con 0x7f4dc015ad30 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc75ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc015dfd0 con 0x7f4dc015a960 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc75ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc015c540 con 0x7f4dc00ae8a0 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc5b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 0x7f4dc015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc539b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dc00ae8a0 0x7f4dc015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc5b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 0x7f4dc015e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60344/0 (socket says 172.21.15.29:60344) 2023-10-07T09:45:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc5b9c700 1 -- 172.21.15.29:0/1547916593 learned_addr learned my addr 172.21.15.29:0/1547916593 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.851+0000 7f4dc4b9a700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dc015a960 0x7f4dc013dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc5b9c700 1 -- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dc00ae8a0 msgr2=0x7f4dc015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc5b9c700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dc00ae8a0 0x7f4dc015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc5b9c700 1 -- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dc015a960 msgr2=0x7f4dc013dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc5b9c700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dc015a960 0x7f4dc013dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc5b9c700 1 -- 172.21.15.29:0/1547916593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009420 con 0x7f4dc015ad30 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc5b9c700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 0x7f4dc015e460 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4db800c910 tx=0x7f4db8002720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8012630 con 0x7f4dc015ad30 2023-10-07T09:45:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db80127b0 con 0x7f4dc015ad30 2023-10-07T09:45:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8012b70 con 0x7f4dc015ad30 2023-10-07T09:45:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc015d6a0 con 0x7f4dc015ad30 2023-10-07T09:45:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.852+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc015dc60 con 0x7f4dc015ad30 2023-10-07T09:45:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.854+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc004c3c0 con 0x7f4dc015ad30 2023-10-07T09:45:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.855+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f4db801b400 con 0x7f4dc015ad30 2023-10-07T09:45:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.855+0000 7f4db57fa700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 0x7f4da4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.856+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f4db8084990 con 0x7f4dc015ad30 2023-10-07T09:45:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.856+0000 7f4dc539b700 1 -- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 msgr2=0x7f4da4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.856+0000 7f4dc539b700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 0x7f4da4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:10.859+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4db8051a80 con 0x7f4dc015ad30 2023-10-07T09:45:11.005 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:11.004+0000 7f9bca6ff1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:45:11.022 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:11.021+0000 7f92086ff1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:45:11.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.057+0000 7f4dc539b700 1 -- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 msgr2=0x7f4da4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:11.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.057+0000 7f4dc539b700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 0x7f4da4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:11.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.103+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f4dc00ac4f0 con 0x7f4dc015ad30 2023-10-07T09:45:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.107+0000 7f4db57fa700 1 -- 172.21.15.29:0/1547916593 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4db80550a0 con 0x7f4dc015ad30 2023-10-07T09:45:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 msgr2=0x7f4da4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 0x7f4da4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 msgr2=0x7f4dc015e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 0x7f4dc015e460 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4db800c910 tx=0x7f4db8002720 comp rx=0 tx=0).stop 2023-10-07T09:45:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 shutdown_connections 2023-10-07T09:45:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4da4066f80 0x7f4da4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc015ad30 0x7f4dc015e460 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dc00ae8a0 0x7f4dc015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 --2- 172.21.15.29:0/1547916593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dc015a960 0x7f4dc013dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 >> 172.21.15.29:0/1547916593 conn(0x7f4dc0067610 msgr2=0x7f4dc0156f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 shutdown_connections 2023-10-07T09:45:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.110+0000 7f4dc75ff700 1 -- 172.21.15.29:0/1547916593 wait complete. 2023-10-07T09:45:11.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:45:11.320 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:11.318+0000 7f9bca6ff1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:45:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.330+0000 7f1a6a5ff700 1 Processor -- start 2023-10-07T09:45:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.330+0000 7f1a6a5ff700 1 -- start start 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.331+0000 7f1a6a5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1a64159b20 0x7f1a640d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.331+0000 7f1a6a5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a640cbf20 con 0x7f1a64159750 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.331+0000 7f1a6a5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a640cc200 con 0x7f1a640ce8a0 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.331+0000 7f1a6a5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a640cc4e0 con 0x7f1a64159b20 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a63fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a64159750 0x7f1a640ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32906/0 (socket says 172.21.15.29:32906) 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a63fff700 1 -- 172.21.15.29:0/2143416676 learned_addr learned my addr 172.21.15.29:0/2143416676 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a637fe700 1 --2- 172.21.15.29:0/2143416676 >> v2:172.21.15.144:3300/0 conn(0x7f1a64159b20 0x7f1a640d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656699761 0 0) 0x7f1a640cbf20 con 0x7f1a64159750 2023-10-07T09:45:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a4c003570 con 0x7f1a64159750 2023-10-07T09:45:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11907654 0 0) 0x7f1a640cc200 con 0x7f1a640ce8a0 2023-10-07T09:45:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a640cbf20 con 0x7f1a640ce8a0 2023-10-07T09:45:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407344356 0 0) 0x7f1a4c003570 con 0x7f1a64159750 2023-10-07T09:45:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a637fe700 1 -- 172.21.15.29:0/2143416676 >> v1:172.21.15.144:6789/0 conn(0x7f1a640ce8a0 legacy=0x7f1a640c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a637fe700 1 -- 172.21.15.29:0/2143416676 >> v1:172.21.15.29:6789/0 conn(0x7f1a64159750 legacy=0x7f1a640ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.332+0000 7f1a637fe700 1 -- 172.21.15.29:0/2143416676 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a640cc7c0 con 0x7f1a64159b20 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.333+0000 7f1a637fe700 1 --2- 172.21.15.29:0/2143416676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159b20 0x7f1a640d10a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1a580098e0 tx=0x7f1a580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5869058f4c356b44 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.333+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540015f0 con 0x7f1a64159b20 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.333+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a58011420 con 0x7f1a64159b20 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.333+0000 7f1a62ffd700 1 -- 172.21.15.29:0/2143416676 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a580117c0 con 0x7f1a64159b20 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/2143416676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159b20 msgr2=0x7f1a640d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/2143416676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159b20 0x7f1a640d10a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1a580098e0 tx=0x7f1a580091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/2143416676 shutdown_connections 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/2143416676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159b20 0x7f1a640d10a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/2143416676 >> 172.21.15.29:0/2143416676 conn(0x7f1a64063a10 msgr2=0x7f1a640d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/2143416676 shutdown_connections 2023-10-07T09:45:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.334+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/2143416676 wait complete. 2023-10-07T09:45:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.335+0000 7f1a6a5ff700 1 Processor -- start 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.336+0000 7f1a6a5ff700 1 -- start start 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a6a5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 0x7f1a64154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a6a5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159750 0x7f1a6416b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a6a5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a64159b20 0x7f1a640c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a6a5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6415a870 con 0x7f1a640ce8a0 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a6a5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64154d50 con 0x7f1a64159750 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a6a5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64159fb0 con 0x7f1a64159b20 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 0x7f1a64154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 0x7f1a64154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60358/0 (socket says 172.21.15.29:60358) 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 -- 172.21.15.29:0/3769968967 learned_addr learned my addr 172.21.15.29:0/3769968967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a637fe700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159750 0x7f1a6416b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a68b9c700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a64159b20 0x7f1a640c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 -- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a64159b20 msgr2=0x7f1a640c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a64159b20 0x7f1a640c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 -- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159750 msgr2=0x7f1a6416b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159750 0x7f1a6416b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.337+0000 7f1a63fff700 1 -- 172.21.15.29:0/3769968967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58009420 con 0x7f1a640ce8a0 2023-10-07T09:45:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.338+0000 7f1a63fff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 0x7f1a64154810 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400cb90 tx=0x7f1a54002b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.338+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400cea0 con 0x7f1a640ce8a0 2023-10-07T09:45:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.338+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54007e40 con 0x7f1a640ce8a0 2023-10-07T09:45:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.338+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6416b8d0 con 0x7f1a640ce8a0 2023-10-07T09:45:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.338+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54007530 con 0x7f1a640ce8a0 2023-10-07T09:45:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.338+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6415a500 con 0x7f1a640ce8a0 2023-10-07T09:45:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.339+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a640c7280 con 0x7f1a640ce8a0 2023-10-07T09:45:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.340+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f1a54005a00 con 0x7f1a640ce8a0 2023-10-07T09:45:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.341+0000 7f1a4bfff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 0x7f1a50069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.341+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1a54084390 con 0x7f1a640ce8a0 2023-10-07T09:45:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.341+0000 7f1a637fe700 1 -- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 msgr2=0x7f1a50069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.341+0000 7f1a637fe700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 0x7f1a50069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.345+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a54051460 con 0x7f1a640ce8a0 2023-10-07T09:45:11.363 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:11.361+0000 7f92086ff1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:45:11.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.542+0000 7f1a637fe700 1 -- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 msgr2=0x7f1a50069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:11.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.542+0000 7f1a637fe700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 0x7f1a50069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:11.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.546+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f1a6404c3c0 con 0x7f1a640ce8a0 2023-10-07T09:45:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.551+0000 7f1a4bfff700 1 -- 172.21.15.29:0/3769968967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1a54054a80 con 0x7f1a640ce8a0 2023-10-07T09:45:11.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 msgr2=0x7f1a50069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 0x7f1a50069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 msgr2=0x7f1a64154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 0x7f1a64154810 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400cb90 tx=0x7f1a54002b60 comp rx=0 tx=0).stop 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 shutdown_connections 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1a50066f80 0x7f1a50069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a64159b20 0x7f1a640c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a640ce8a0 0x7f1a64154810 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 --2- 172.21.15.29:0/3769968967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a64159750 0x7f1a6416b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 >> 172.21.15.29:0/3769968967 conn(0x7f1a64063a10 msgr2=0x7f1a640c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 shutdown_connections 2023-10-07T09:45:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.553+0000 7f1a6a5ff700 1 -- 172.21.15.29:0/3769968967 wait complete. 2023-10-07T09:45:11.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-10-07T09:45:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.779+0000 7fb184bff700 1 Processor -- start 2023-10-07T09:45:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.779+0000 7fb184bff700 1 -- start start 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.780+0000 7fb184bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800ced00 0x7fb1800b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.780+0000 7fb184bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1800c8a20 0x7fb1800cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.780+0000 7fb184bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800690f0 con 0x7fb1800b0610 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.780+0000 7fb184bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1800cf2e0 con 0x7fb1800c8a20 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.780+0000 7fb184bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800687c0 con 0x7fb1800ced00 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17e59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb1800b0610 0x7fb1800c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32910/0 (socket says 172.21.15.29:32910) 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17e59c700 1 -- 172.21.15.29:0/804466072 learned_addr learned my addr 172.21.15.29:0/804466072 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17ed9d700 1 --2- 172.21.15.29:0/804466072 >> v2:172.21.15.29:3300/0 conn(0x7fb1800c8a20 0x7fb1800cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/804466072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800ced00 0x7fb1800b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17d59a700 1 -- 172.21.15.29:0/804466072 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789044133 0 0) 0x7fb1800690f0 con 0x7fb1800b0610 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17d59a700 1 -- 172.21.15.29:0/804466072 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb168003570 con 0x7fb1800b0610 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17ed9d700 1 -- 172.21.15.29:0/804466072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800ced00 msgr2=0x7fb1800b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17ed9d700 1 --2- 172.21.15.29:0/804466072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800ced00 0x7fb1800b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17ed9d700 1 -- 172.21.15.29:0/804466072 >> v1:172.21.15.29:6789/0 conn(0x7fb1800b0610 legacy=0x7fb1800c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.781+0000 7fb17ed9d700 1 -- 172.21.15.29:0/804466072 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb180068aa0 con 0x7fb1800c8a20 2023-10-07T09:45:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb17ed9d700 1 --2- 172.21.15.29:0/804466072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800c8a20 0x7fb1800cc7e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb178009e60 tx=0x7fb178009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a99b3ac579258c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb17d59a700 1 -- 172.21.15.29:0/804466072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800e070 con 0x7fb1800c8a20 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb17d59a700 1 -- 172.21.15.29:0/804466072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17800bab0 con 0x7fb1800c8a20 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb17d59a700 1 -- 172.21.15.29:0/804466072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800be70 con 0x7fb1800c8a20 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb184bff700 1 -- 172.21.15.29:0/804466072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800c8a20 msgr2=0x7fb1800cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb184bff700 1 --2- 172.21.15.29:0/804466072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800c8a20 0x7fb1800cc7e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb178009e60 tx=0x7fb178009780 comp rx=0 tx=0).stop 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb184bff700 1 -- 172.21.15.29:0/804466072 shutdown_connections 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb184bff700 1 --2- 172.21.15.29:0/804466072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800c8a20 0x7fb1800cc7e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb184bff700 1 --2- 172.21.15.29:0/804466072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800ced00 0x7fb1800b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.782+0000 7fb184bff700 1 -- 172.21.15.29:0/804466072 >> 172.21.15.29:0/804466072 conn(0x7fb180063a10 msgr2=0x7fb1800ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.783+0000 7fb184bff700 1 -- 172.21.15.29:0/804466072 shutdown_connections 2023-10-07T09:45:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.783+0000 7fb184bff700 1 -- 172.21.15.29:0/804466072 wait complete. 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.784+0000 7fb184bff700 1 Processor -- start 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.784+0000 7fb184bff700 1 -- start start 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb184bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800b0610 0x7fb1801546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb184bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800c8a20 0x7fb1801526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb184bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1800ced00 0x7fb1800b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb184bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800b17a0 con 0x7fb1800b0610 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb184bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800c6e80 con 0x7fb1800ced00 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb184bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800c7160 con 0x7fb1800c8a20 2023-10-07T09:45:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb17e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800b0610 0x7fb1801546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb17e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800b0610 0x7fb1801546d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60390/0 (socket says 172.21.15.29:60390) 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb17e59c700 1 -- 172.21.15.29:0/2040234639 learned_addr learned my addr 172.21.15.29:0/2040234639 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb17ed9d700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1800ced00 0x7fb1800b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.785+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800c8a20 0x7fb1801526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb17e59c700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800c8a20 msgr2=0x7fb1801526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb17e59c700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1800c8a20 0x7fb1801526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb17e59c700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1800ced00 msgr2=0x7fb1800b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb17e59c700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1800ced00 0x7fb1800b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb17e59c700 1 -- 172.21.15.29:0/2040234639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178009a20 con 0x7fb1800b0610 2023-10-07T09:45:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb17e59c700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800b0610 0x7fb1801546d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb17000c590 tx=0x7fb170007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170005b30 con 0x7fb1800b0610 2023-10-07T09:45:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170005cb0 con 0x7fb1800b0610 2023-10-07T09:45:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb184bff700 1 -- 172.21.15.29:0/2040234639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1800b1a10 con 0x7fb1800b0610 2023-10-07T09:45:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.786+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170007320 con 0x7fb1800b0610 2023-10-07T09:45:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.787+0000 7fb184bff700 1 -- 172.21.15.29:0/2040234639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb180154c10 con 0x7fb1800b0610 2023-10-07T09:45:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.788+0000 7fb184bff700 1 -- 172.21.15.29:0/2040234639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1800b1c50 con 0x7fb1800b0610 2023-10-07T09:45:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.789+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fb17001b3b0 con 0x7fb1800b0610 2023-10-07T09:45:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.789+0000 7fb16e7fc700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.790+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb170083bc0 con 0x7fb1800b0610 2023-10-07T09:45:11.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.790+0000 7fb17dd9b700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 msgr2=0x7fb160069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:11.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.790+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.797+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1800b1c50 con 0x7fb1800b0610 2023-10-07T09:45:11.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:11.841+0000 7f9bca6ff1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:45:11.928 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:11.927+0000 7f92086ff1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:45:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.991+0000 7fb17dd9b700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 msgr2=0x7fb160069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:11.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:11.991+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:12.011 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:12.010+0000 7f9bca6ff1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:45:12.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:12.029+0000 7fb184bff700 1 -- 172.21.15.29:0/2040234639 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1800b1c50 con 0x7fb160066f80 2023-10-07T09:45:12.105 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:12.104+0000 7f92086ff1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:45:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:12.392+0000 7fb17dd9b700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 msgr2=0x7fb160069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:12.392+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:45:12.423 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:12.422+0000 7f9bca6ff1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:45:12.534 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:12.533+0000 7f92086ff1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:45:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:13.193+0000 7fb17dd9b700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 msgr2=0x7fb160069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:13.193+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:45:13.195 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.194+0000 7f9bca6ff1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:45:13.282 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.281+0000 7f9bca6ff1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:45:13.342 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:13.341+0000 7f92086ff1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:45:13.434 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:13.433+0000 7f92086ff1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:45:13.436 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.435+0000 7f9bca6ff1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:45:13.524 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.523+0000 7f9bca6ff1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:45:13.596 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:13.595+0000 7f92086ff1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:45:13.688 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:13.687+0000 7f92086ff1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:45:13.738 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.737+0000 7f9bca6ff1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:45:13.831 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.830+0000 7f9bca6ff1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:45:13.907 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:13.906+0000 7f92086ff1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:45:13.924 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:13.923+0000 7f9bca6ff1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:45:14.009 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.009+0000 7f92086ff1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:45:14.019 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:14.018+0000 7f9bca6ff1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:45:14.152 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.151+0000 7f92086ff1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:45:14.206 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:14.205+0000 7f9bca6ff1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:45:14.251 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.250+0000 7f92086ff1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:45:14.302 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:14.301+0000 7f9bca6ff1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:45:14.448 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.447+0000 7f92086ff1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:45:14.550 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.549+0000 7f92086ff1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:45:14.641 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:14.640+0000 7f9bca6ff1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:45:14.728 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:14.727+0000 7f9bca6ff1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:14.795+0000 7fb17dd9b700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 msgr2=0x7fb160069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:14.795+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:45:14.899 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.898+0000 7f92086ff1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:45:14.991 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:14.990+0000 7f92086ff1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:15.261 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:15.260+0000 7f9bca6ff1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:45:15.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:15.446+0000 7f9bca6ff1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:45:15.557 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:15.556+0000 7f92086ff1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:45:15.750 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:15.749+0000 7f92086ff1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:45:15.819 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:15.818+0000 7f9bca6ff1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:16.134 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:16.133+0000 7f92086ff1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:16.232 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.231+0000 7f9bca6ff1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:45:16.411 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.410+0000 7f9bca6ff1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:16.583 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:16.582+0000 7f92086ff1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:45:16.596 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.595+0000 7f9bca6ff1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:45:16.703 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.702+0000 7f9bca6ff1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:16.712+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 35) v1 ==== 84468+0+0 (secure 0 0 0) 0x7fb170051080 con 0x7fb1800b0610 2023-10-07T09:45:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:16.712+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 msgr2=0x7fb160069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:16.712+0000 7fb16e7fc700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:16.780 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:16.779+0000 7f92086ff1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:16.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3b4f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3b4f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3b4f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3b4f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3b4f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3ecf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.843 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3ecf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.843 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3ecf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.843 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3ecf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.843 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:16.840+0000 7f9b3ecf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:16.978 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:16.977+0000 7f92086ff1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:45:17.091 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:17.090+0000 7f92086ff1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.718+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 36) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fb170034c40 con 0x7fb1800b0610 2023-10-07T09:45:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.718+0000 7fb16e7fc700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.718+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1800b1c50 con 0x7fb160066f80 2023-10-07T09:45:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.725+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.730+0000 7fb17dd9b700 1 --2- 172.21.15.29:0/2040234639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb160066f80 0x7fb160069430 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb1800c78f0 tx=0x7fb1740091a0 comp rx=0 tx=0).ready entity=mgr.6155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:17.739 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:17.738+0000 7f9b45d05700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-07T09:45:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.738+0000 7fb16e7fc700 1 -- 172.21.15.29:0/2040234639 <== mgr.6155 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fb1800b1c50 con 0x7fb160066f80 2023-10-07T09:45:17.741 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-07T09:45:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-07T09:45:17.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T09:45:17.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.949+0000 7fcfce9ff700 1 Processor -- start 2023-10-07T09:45:17.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.949+0000 7fcfce9ff700 1 -- start start 2023-10-07T09:45:17.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfce9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcfc815a960 0x7fcfc8158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfce9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfc80ae8a0 0x7fcfc80b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfce9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc80acb40 con 0x7fcfc815ad30 2023-10-07T09:45:17.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfce9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc8158c60 con 0x7fcfc80ae8a0 2023-10-07T09:45:17.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfce9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc80abf30 con 0x7fcfc815a960 2023-10-07T09:45:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfc80ae8a0 0x7fcfc80b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfc80ae8a0 0x7fcfc80b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49200/0 (socket says 172.21.15.29:49200) 2023-10-07T09:45:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 -- 172.21.15.29:0/1634143628 learned_addr learned my addr 172.21.15.29:0/1634143628 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfc7fff700 1 --2- 172.21.15.29:0/1634143628 >> v2:172.21.15.144:3300/0 conn(0x7fcfc815a960 0x7fcfc8158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 -- 172.21.15.29:0/1634143628 >> v2:172.21.15.144:3300/0 conn(0x7fcfc815a960 msgr2=0x7fcfc8158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 --2- 172.21.15.29:0/1634143628 >> v2:172.21.15.144:3300/0 conn(0x7fcfc815a960 0x7fcfc8158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 -- 172.21.15.29:0/1634143628 >> v1:172.21.15.29:6789/0 conn(0x7fcfc815ad30 legacy=0x7fcfc80ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfccf9c700 1 -- 172.21.15.29:0/1634143628 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc80ac210 con 0x7fcfc80ae8a0 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.951+0000 7fcfc7fff700 1 --2- 172.21.15.29:0/1634143628 >> v2:172.21.15.144:3300/0 conn(0x7fcfc815a960 0x7fcfc8158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.952+0000 7fcfccf9c700 1 --2- 172.21.15.29:0/1634143628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc80b1290 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80aecc0 tx=0x7fcfc0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ce8e33a07c2ca7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.952+0000 7fcfc6ffd700 1 -- 172.21.15.29:0/1634143628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc000e070 con 0x7fcfc80ae8a0 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.952+0000 7fcfc6ffd700 1 -- 172.21.15.29:0/1634143628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc000ba50 con 0x7fcfc80ae8a0 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.952+0000 7fcfc6ffd700 1 -- 172.21.15.29:0/1634143628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00092e0 con 0x7fcfc80ae8a0 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.952+0000 7fcfce9ff700 1 -- 172.21.15.29:0/1634143628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc80ae8a0 msgr2=0x7fcfc80b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/1634143628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc80b1290 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80aecc0 tx=0x7fcfc0009840 comp rx=0 tx=0).stop 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 -- 172.21.15.29:0/1634143628 shutdown_connections 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/1634143628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc80b1290 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/1634143628 >> v2:172.21.15.144:3300/0 conn(0x7fcfc815a960 0x7fcfc8158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 -- 172.21.15.29:0/1634143628 >> 172.21.15.29:0/1634143628 conn(0x7fcfc8067610 msgr2=0x7fcfc8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 -- 172.21.15.29:0/1634143628 shutdown_connections 2023-10-07T09:45:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.953+0000 7fcfce9ff700 1 -- 172.21.15.29:0/1634143628 wait complete. 2023-10-07T09:45:17.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.954+0000 7fcfce9ff700 1 Processor -- start 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.954+0000 7fcfce9ff700 1 -- start start 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.955+0000 7fcfce9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.955+0000 7fcfce9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc815a960 0x7fcfc8150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.955+0000 7fcfce9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 0x7fcfc816d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.955+0000 7fcfce9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc815bcb0 con 0x7fcfc815ad30 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.955+0000 7fcfce9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc815b380 con 0x7fcfc80ae8a0 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.955+0000 7fcfce9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc815b660 con 0x7fcfc815a960 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 0x7fcfc816d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 0x7fcfc816d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49210/0 (socket says 172.21.15.29:49210) 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 -- 172.21.15.29:0/4229310159 learned_addr learned my addr 172.21.15.29:0/4229310159 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfc77fe700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc815a960 0x7fcfc8150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfc7fff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 -- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc815a960 msgr2=0x7fcfc8150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc815a960 0x7fcfc8150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 -- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc80ae8a0 msgr2=0x7fcfc815c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc815c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 -- 172.21.15.29:0/4229310159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0009ae0 con 0x7fcfc815ad30 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.956+0000 7fcfccf9c700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 0x7fcfc816d6d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0012000 tx=0x7fcfc0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.957+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0008e80 con 0x7fcfc815ad30 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.957+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc000be60 con 0x7fcfc815ad30 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.957+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00266b0 con 0x7fcfc815ad30 2023-10-07T09:45:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.957+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc815b940 con 0x7fcfc815ad30 2023-10-07T09:45:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.957+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc815d680 con 0x7fcfc815ad30 2023-10-07T09:45:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.958+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc804c3c0 con 0x7fcfc815ad30 2023-10-07T09:45:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.959+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fcfc0003040 con 0x7fcfc815ad30 2023-10-07T09:45:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.959+0000 7fcfaffff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcfb4066dd0 0x7fcfb4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.959+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fcfc00865b0 con 0x7fcfc815ad30 2023-10-07T09:45:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.960+0000 7fcfc7fff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcfb4066dd0 0x7fcfb4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.961+0000 7fcfc7fff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcfb4066dd0 0x7fcfb4069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8159a20 tx=0x7fcfb8006f90 comp rx=0 tx=0).ready entity=mgr.6155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:17.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:17.964+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfc0053780 con 0x7fcfc815ad30 2023-10-07T09:45:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.220+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fcfc80ac4f0 con 0x7fcfc815ad30 2023-10-07T09:45:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.774+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fcfc0053b50 con 0x7fcfc815ad30 2023-10-07T09:45:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.776+0000 7fcfaffff700 1 -- 172.21.15.29:0/4229310159 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v37) v1 ==== 93+0+0 (secure 0 0 0) 0x7fcfc0057210 con 0x7fcfc815ad30 2023-10-07T09:45:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.778+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcfb4066dd0 msgr2=0x7fcfb4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.778+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcfb4066dd0 0x7fcfb4069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8159a20 tx=0x7fcfb8006f90 comp rx=0 tx=0).stop 2023-10-07T09:45:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 msgr2=0x7fcfc816d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 0x7fcfc816d6d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0012000 tx=0x7fcfc0018000 comp rx=0 tx=0).stop 2023-10-07T09:45:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 shutdown_connections 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcfb4066dd0 0x7fcfb4069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc815ad30 0x7fcfc816d6d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc80ae8a0 0x7fcfc815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 --2- 172.21.15.29:0/4229310159 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc815a960 0x7fcfc8150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 >> 172.21.15.29:0/4229310159 conn(0x7fcfc8067610 msgr2=0x7fcfc80ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 shutdown_connections 2023-10-07T09:45:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:18.779+0000 7fcfce9ff700 1 -- 172.21.15.29:0/4229310159 wait complete. 2023-10-07T09:45:18.802 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:45:18.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:45:18.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:45:18.991 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:18.990+0000 7fdafd6a51c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:45:18.995 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:18.994+0000 7faf1252a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:45:19.027 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:45:19.027 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:45:19.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:45:19.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:19.138 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:45:19.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:19.138 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:45:19.172 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:19.171+0000 7faf1252a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:45:19.180 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:19.179+0000 7fdafd6a51c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:45:19.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:19.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.438+0000 7f7d2d5ff700 1 Processor -- start 2023-10-07T09:45:19.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.438+0000 7f7d2d5ff700 1 -- start start 2023-10-07T09:45:19.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.439+0000 7f7d2d5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280ad370 0x7f7d280cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.439+0000 7f7d2d5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7d280b0410 0x7f7d2815a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.439+0000 7f7d2d5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2814d790 con 0x7f7d28159ec0 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.439+0000 7f7d2d5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d280ae5d0 con 0x7f7d280b0410 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.439+0000 7f7d2d5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2815a850 con 0x7f7d280ad370 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d277fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7d280b0410 0x7f7d2815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d26ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280ad370 0x7f7d280cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d267fc700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7d28159ec0 0x7f7d281568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:45022/0 (socket says 172.21.15.29:45022) 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d267fc700 1 -- 172.21.15.29:0/588340280 learned_addr learned my addr 172.21.15.29:0/588340280 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:19.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d26ffd700 1 --2- 172.21.15.29:0/588340280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280ad370 0x7f7d280cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:46700/0 (socket says 172.21.15.29:46700) 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d25ffb700 1 -- 172.21.15.29:0/588340280 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803980644 0 0) 0x7f7d2814d790 con 0x7f7d28159ec0 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d277fe700 1 -- 172.21.15.29:0/588340280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280ad370 msgr2=0x7f7d280cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d277fe700 1 --2- 172.21.15.29:0/588340280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280ad370 0x7f7d280cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d277fe700 1 -- 172.21.15.29:0/588340280 >> v1:172.21.15.144:6789/0 conn(0x7f7d28159ec0 legacy=0x7f7d281568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d277fe700 1 -- 172.21.15.29:0/588340280 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280ad740 con 0x7f7d280b0410 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.440+0000 7f7d277fe700 1 --2- 172.21.15.29:0/588340280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d280b0410 0x7f7d2815a310 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d0d0 tx=0x7f7d2000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e3bda0fde0eae4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:19.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d25ffb700 1 -- 172.21.15.29:0/588340280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20010070 con 0x7f7d280b0410 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d25ffb700 1 -- 172.21.15.29:0/588340280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000b2e0 con 0x7f7d280b0410 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d25ffb700 1 -- 172.21.15.29:0/588340280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000dd80 con 0x7f7d280b0410 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/588340280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d280b0410 msgr2=0x7f7d2815a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/588340280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d280b0410 0x7f7d2815a310 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d0d0 tx=0x7f7d2000b990 comp rx=0 tx=0).stop 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/588340280 shutdown_connections 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/588340280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d280b0410 0x7f7d2815a310 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/588340280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280ad370 0x7f7d280cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/588340280 >> 172.21.15.29:0/588340280 conn(0x7f7d280680a0 msgr2=0x7f7d280cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.441+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/588340280 shutdown_connections 2023-10-07T09:45:19.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.442+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/588340280 wait complete. 2023-10-07T09:45:19.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.443+0000 7f7d2d5ff700 1 Processor -- start 2023-10-07T09:45:19.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.443+0000 7f7d2d5ff700 1 -- start start 2023-10-07T09:45:19.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d2d5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d280ad370 0x7f7d28156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d2d5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280b0410 0x7f7d28155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d2d5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 0x7f7d2814a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d2d5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28156270 con 0x7f7d28159ec0 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d2d5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281552e0 con 0x7f7d280ad370 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d2d5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281558a0 con 0x7f7d280b0410 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 0x7f7d2814a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 0x7f7d2814a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49238/0 (socket says 172.21.15.29:49238) 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d277fe700 1 -- 172.21.15.29:0/2735170126 learned_addr learned my addr 172.21.15.29:0/2735170126 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d26ffd700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d280ad370 0x7f7d28156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.444+0000 7f7d267fc700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280b0410 0x7f7d28155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d277fe700 1 -- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280b0410 msgr2=0x7f7d28155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d277fe700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280b0410 0x7f7d28155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d277fe700 1 -- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d280ad370 msgr2=0x7f7d28156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d277fe700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d280ad370 0x7f7d28156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d277fe700 1 -- 172.21.15.29:0/2735170126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2000bc30 con 0x7f7d28159ec0 2023-10-07T09:45:19.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d277fe700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 0x7f7d2814a2d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7d28068dd0 tx=0x7f7d2001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:19.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000b630 con 0x7f7d28159ec0 2023-10-07T09:45:19.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000deb0 con 0x7f7d28159ec0 2023-10-07T09:45:19.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2814a810 con 0x7f7d28159ec0 2023-10-07T09:45:19.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.445+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2001e670 con 0x7f7d28159ec0 2023-10-07T09:45:19.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.446+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d280cbd60 con 0x7f7d28159ec0 2023-10-07T09:45:19.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.447+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d28155a50 con 0x7f7d28159ec0 2023-10-07T09:45:19.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.447+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f7d20003040 con 0x7f7d28159ec0 2023-10-07T09:45:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.448+0000 7f7d0effd700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 0x7f7d14069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.448+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f7d20086740 con 0x7f7d28159ec0 2023-10-07T09:45:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.448+0000 7f7d26ffd700 1 -- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 msgr2=0x7f7d14069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.448+0000 7f7d26ffd700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 0x7f7d14069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:19.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.452+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d28155a50 con 0x7f7d28159ec0 2023-10-07T09:45:19.485 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:19.484+0000 7faf1252a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:45:19.529 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:19.528+0000 7fdafd6a51c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:45:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.649+0000 7f7d26ffd700 1 -- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 msgr2=0x7f7d14069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.649+0000 7f7d26ffd700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 0x7f7d14069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:19.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.720+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d2806f560 con 0x7f7d28159ec0 2023-10-07T09:45:19.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.721+0000 7f7d0effd700 1 -- 172.21.15.29:0/2735170126 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f7d20053300 con 0x7f7d28159ec0 2023-10-07T09:45:19.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:19.723 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:01.071366+0000","modified":"2023-10-07T09:45:04.584641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2799196407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2799196407},{"type":"v1","addr":"172.21.15.29:6835","nonce":2799196407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-10-07T09:45:19.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.723+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 msgr2=0x7f7d14069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:19.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 0x7f7d14069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 msgr2=0x7f7d2814a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 0x7f7d2814a2d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7d28068dd0 tx=0x7f7d2001a000 comp rx=0 tx=0).stop 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 shutdown_connections 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7d140670e0 0x7f7d14069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d28159ec0 0x7f7d2814a2d0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d280ad370 0x7f7d28156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 --2- 172.21.15.29:0/2735170126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d280b0410 0x7f7d28155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 >> 172.21.15.29:0/2735170126 conn(0x7f7d280680a0 msgr2=0x7f7d281507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 shutdown_connections 2023-10-07T09:45:19.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.724+0000 7f7d2d5ff700 1 -- 172.21.15.29:0/2735170126 wait complete. 2023-10-07T09:45:19.727 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-07T09:45:19.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.948+0000 7fb7c65ff700 1 Processor -- start 2023-10-07T09:45:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.948+0000 7fb7c65ff700 1 -- start start 2023-10-07T09:45:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c65ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7c0157290 0x7fb7c00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c65ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7c00b1a10 0x7fb7c0157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c65ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c0158ad0 con 0x7fb7c00ae8b0 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c65ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c015b140 con 0x7fb7c00b1a10 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c65ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c015b420 con 0x7fb7c0157290 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c4b9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7c00b1a10 0x7fb7c0157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c4b9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7c00b1a10 0x7fb7c0157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49244/0 (socket says 172.21.15.29:49244) 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7c4b9c700 1 -- 172.21.15.29:0/70789510 learned_addr learned my addr 172.21.15.29:0/70789510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7bf7fe700 1 --1- 172.21.15.29:0/70789510 >> v1:172.21.15.29:6789/0 conn(0x7fb7c00ae8b0 0x7fb7c00b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53286/0 (socket says 172.21.15.29:53286) 2023-10-07T09:45:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.950+0000 7fb7bffff700 1 --2- 172.21.15.29:0/70789510 >> v2:172.21.15.144:3300/0 conn(0x7fb7c0157290 0x7fb7c00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7c4b9c700 1 -- 172.21.15.29:0/70789510 >> v2:172.21.15.144:3300/0 conn(0x7fb7c0157290 msgr2=0x7fb7c00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7c4b9c700 1 --2- 172.21.15.29:0/70789510 >> v2:172.21.15.144:3300/0 conn(0x7fb7c0157290 0x7fb7c00cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7c4b9c700 1 -- 172.21.15.29:0/70789510 >> v1:172.21.15.29:6789/0 conn(0x7fb7c00ae8b0 legacy=0x7fb7c00b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7beffd700 1 -- 172.21.15.29:0/70789510 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934970541 0 0) 0x7fb7c0158ad0 con 0x7fb7c00ae8b0 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7c4b9c700 1 -- 172.21.15.29:0/70789510 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c015b700 con 0x7fb7c00b1a10 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7c4b9c700 1 --2- 172.21.15.29:0/70789510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0157660 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00685f0 tx=0x7fb7b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6b4a68741abf53 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7beffd700 1 -- 172.21.15.29:0/70789510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b800e070 con 0x7fb7c00b1a10 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7beffd700 1 -- 172.21.15.29:0/70789510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b800ba20 con 0x7fb7c00b1a10 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.951+0000 7fb7beffd700 1 -- 172.21.15.29:0/70789510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80092e0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 -- 172.21.15.29:0/70789510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 msgr2=0x7fb7c0157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/70789510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0157660 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00685f0 tx=0x7fb7b8009840 comp rx=0 tx=0).stop 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 -- 172.21.15.29:0/70789510 shutdown_connections 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/70789510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0157660 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/70789510 >> v2:172.21.15.144:3300/0 conn(0x7fb7c0157290 0x7fb7c00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 -- 172.21.15.29:0/70789510 >> 172.21.15.29:0/70789510 conn(0x7fb7c00680a0 msgr2=0x7fb7c00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 -- 172.21.15.29:0/70789510 shutdown_connections 2023-10-07T09:45:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.952+0000 7fb7c65ff700 1 -- 172.21.15.29:0/70789510 wait complete. 2023-10-07T09:45:19.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.954+0000 7fb7c65ff700 1 Processor -- start 2023-10-07T09:45:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.954+0000 7fb7c65ff700 1 -- start start 2023-10-07T09:45:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7c65ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7c65ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7c0157290 0x7fb7c014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7c65ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7c0150680 0x7fb7c00afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7c65ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00af7e0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7c65ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00c5380 con 0x7fb7c0157290 2023-10-07T09:45:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7c65ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00c5620 con 0x7fb7c0150680 2023-10-07T09:45:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.955+0000 7fb7bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49250/0 (socket says 172.21.15.29:49250) 2023-10-07T09:45:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 -- 172.21.15.29:0/3626475695 learned_addr learned my addr 172.21.15.29:0/3626475695 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:19.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7c4b9c700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7c0150680 0x7fb7c00afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bf7fe700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7c0157290 0x7fb7c014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:19.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 -- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7c0150680 msgr2=0x7fb7c00afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7c0150680 0x7fb7c00afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 -- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7c0157290 msgr2=0x7fb7c014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:19.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7c0157290 0x7fb7c014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:19.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 -- 172.21.15.29:0/3626475695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8009ae0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7bffff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0158620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000e950 tx=0x7fb7b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:19.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.956+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0013680 con 0x7fb7c00b1a10 2023-10-07T09:45:19.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.957+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b00137e0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.957+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0013ba0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.957+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c00c5900 con 0x7fb7c00b1a10 2023-10-07T09:45:19.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.957+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c00af470 con 0x7fb7c00b1a10 2023-10-07T09:45:19.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.958+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c004c3c0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.959+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fb7b00123d0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.960+0000 7fb7a7fff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 0x7fb7ac069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.960+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fb7b0084cf0 con 0x7fb7c00b1a10 2023-10-07T09:45:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.960+0000 7fb7bf7fe700 1 -- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 msgr2=0x7fb7ac069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.960+0000 7fb7bf7fe700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 0x7fb7ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:19.964+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7b00519c0 con 0x7fb7c00b1a10 2023-10-07T09:45:20.011 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:20.010+0000 7faf1252a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:45:20.113 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:20.112+0000 7fdafd6a51c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:45:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.161+0000 7fb7bf7fe700 1 -- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 msgr2=0x7fb7ac069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.161+0000 7fb7bf7fe700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 0x7fb7ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:20.179 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:20.178+0000 7faf1252a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:45:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.203+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c015b9e0 con 0x7fb7c00b1a10 2023-10-07T09:45:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.204+0000 7fb7a7fff700 1 -- 172.21.15.29:0/3626475695 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16394 (secure 0 0 0) 0x7fb7b0017090 con 0x7fb7c00b1a10 2023-10-07T09:45:20.206 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:20.206 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:16.705837+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:58.854049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:00.048655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3721675226":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/3991888806":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/1527687027":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/262634276":"2023-10-08T09:45:16.705779+0000","172.21.15.144:6836/114714":"2023-10-08T09:45:16.705779+0000","172.21.15.144:6834/114714":"2023-10-08T09:45:16.705779+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.206+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 msgr2=0x7fb7ac069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.206+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 0x7fb7ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 msgr2=0x7fb7c0158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0158620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000e950 tx=0x7fb7b00091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 shutdown_connections 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7ac0670e0 0x7fb7ac069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7c0150680 0x7fb7c00afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c00b1a10 0x7fb7c0158620 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 --2- 172.21.15.29:0/3626475695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7c0157290 0x7fb7c014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 >> 172.21.15.29:0/3626475695 conn(0x7fb7c00680a0 msgr2=0x7fb7c00ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 shutdown_connections 2023-10-07T09:45:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.207+0000 7fb7c65ff700 1 -- 172.21.15.29:0/3626475695 wait complete. 2023-10-07T09:45:20.226 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:44:58.854049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:45:00.048655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:45:20.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:45:20.295 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:20.293+0000 7fdafd6a51c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:45:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.431+0000 7f2dba5ff700 1 Processor -- start 2023-10-07T09:45:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.432+0000 7f2dba5ff700 1 -- start start 2023-10-07T09:45:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.433+0000 7f2dba5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2db415a960 0x7f2db4158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.433+0000 7f2dba5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2db415ad30 0x7f2db40ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.433+0000 7f2dba5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40acb40 con 0x7f2db40ae6d0 2023-10-07T09:45:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.433+0000 7f2dba5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db4158c60 con 0x7f2db415a960 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.433+0000 7f2dba5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db40abf30 con 0x7f2db415ad30 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2db415a960 0x7f2db4158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2db415a960 0x7f2db4158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49258/0 (socket says 172.21.15.29:49258) 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 -- 172.21.15.29:0/3767107005 learned_addr learned my addr 172.21.15.29:0/3767107005 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db37fe700 1 --2- 172.21.15.29:0/3767107005 >> v2:172.21.15.144:3300/0 conn(0x7f2db415ad30 0x7f2db40ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db8b9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2db40ae6d0 0x7f2db40b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53298/0 (socket says 172.21.15.29:53298) 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 -- 172.21.15.29:0/3767107005 >> v2:172.21.15.144:3300/0 conn(0x7f2db415ad30 msgr2=0x7f2db40ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 --2- 172.21.15.29:0/3767107005 >> v2:172.21.15.144:3300/0 conn(0x7f2db415ad30 0x7f2db40ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 -- 172.21.15.29:0/3767107005 >> v1:172.21.15.29:6789/0 conn(0x7f2db40ae6d0 legacy=0x7f2db40b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 -- 172.21.15.29:0/3767107005 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db40ac210 con 0x7f2db415a960 2023-10-07T09:45:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.434+0000 7f2db3fff700 1 --2- 172.21.15.29:0/3767107005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4158e20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2db406b910 tx=0x7f2da4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683a17315aaa7a51 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.435+0000 7f2db2ffd700 1 -- 172.21.15.29:0/3767107005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4009e70 con 0x7f2db415a960 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.435+0000 7f2db2ffd700 1 -- 172.21.15.29:0/3767107005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da400b860 con 0x7f2db415a960 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.435+0000 7f2db2ffd700 1 -- 172.21.15.29:0/3767107005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da400bc00 con 0x7f2db415a960 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.435+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3767107005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 msgr2=0x7f2db4158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.435+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3767107005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4158e20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2db406b910 tx=0x7f2da4009250 comp rx=0 tx=0).stop 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.436+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3767107005 shutdown_connections 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.436+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3767107005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4158e20 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.436+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3767107005 >> v2:172.21.15.144:3300/0 conn(0x7f2db415ad30 0x7f2db40ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.436+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3767107005 >> 172.21.15.29:0/3767107005 conn(0x7f2db4067610 msgr2=0x7f2db4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.436+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3767107005 shutdown_connections 2023-10-07T09:45:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.436+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3767107005 wait complete. 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.437+0000 7f2dba5ff700 1 Processor -- start 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.437+0000 7f2dba5ff700 1 -- start start 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.438+0000 7f2dba5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2db40ae6d0 0x7f2db4150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.438+0000 7f2dba5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.438+0000 7f2dba5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2db415ad30 0x7f2db4158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.438+0000 7f2dba5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40cbcd0 con 0x7f2db415a960 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.438+0000 7f2dba5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db415e5d0 con 0x7f2db415ad30 2023-10-07T09:45:20.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.438+0000 7f2dba5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db415cd70 con 0x7f2db40ae6d0 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49272/0 (socket says 172.21.15.29:49272) 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 -- 172.21.15.29:0/3699168831 learned_addr learned my addr 172.21.15.29:0/3699168831 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db8b9c700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2db415ad30 0x7f2db4158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db3fff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2db40ae6d0 0x7f2db4150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 -- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2db40ae6d0 msgr2=0x7f2db4150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2db40ae6d0 0x7f2db4150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 -- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2db415ad30 msgr2=0x7f2db4158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2db415ad30 0x7f2db4158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 -- 172.21.15.29:0/3699168831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da40094f0 con 0x7f2db415a960 2023-10-07T09:45:20.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.439+0000 7f2db37fe700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4157ae0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2da800e930 tx=0x7f2da80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:20.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.440+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8013680 con 0x7f2db415a960 2023-10-07T09:45:20.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.440+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da80137e0 con 0x7f2db415a960 2023-10-07T09:45:20.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.440+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8013b80 con 0x7f2db415a960 2023-10-07T09:45:20.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.440+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db4151280 con 0x7f2db415a960 2023-10-07T09:45:20.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.440+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db415c720 con 0x7f2db415a960 2023-10-07T09:45:20.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.441+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db415cf00 con 0x7f2db415a960 2023-10-07T09:45:20.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.444+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f2da80123d0 con 0x7f2db415a960 2023-10-07T09:45:20.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.444+0000 7f2d9bfff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 0x7f2da0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.444+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f2da8084b70 con 0x7f2db415a960 2023-10-07T09:45:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.445+0000 7f2db3fff700 1 -- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 msgr2=0x7f2da0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:20.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.445+0000 7f2db3fff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 0x7f2da0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.449+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2da8051840 con 0x7f2db415a960 2023-10-07T09:45:20.589 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:20.588+0000 7faf1252a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:45:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.646+0000 7f2db3fff700 1 -- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 msgr2=0x7f2da0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.646+0000 7f2db3fff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 0x7f2da0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:20.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.649+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2db404c3c0 con 0x7f2db415a960 2023-10-07T09:45:20.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.649+0000 7f2d9bfff700 1 -- 172.21.15.29:0/3699168831 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2da8054e60 con 0x7f2db415a960 2023-10-07T09:45:20.651 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:45:20.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 msgr2=0x7f2da0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:20.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 0x7f2da0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 msgr2=0x7f2db4157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4157ae0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2da800e930 tx=0x7f2da80091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 shutdown_connections 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2da00670e0 0x7f2da0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2db415ad30 0x7f2db4158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2db40ae6d0 0x7f2db4150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 --2- 172.21.15.29:0/3699168831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db415a960 0x7f2db4157ae0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 >> 172.21.15.29:0/3699168831 conn(0x7f2db4067610 msgr2=0x7f2db40c8460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:20.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 shutdown_connections 2023-10-07T09:45:20.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.652+0000 7f2dba5ff700 1 -- 172.21.15.29:0/3699168831 wait complete. 2023-10-07T09:45:20.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:45:20.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:20.729+0000 7fdafd6a51c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:45:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.874+0000 7f04d83ff700 1 Processor -- start 2023-10-07T09:45:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.874+0000 7f04d83ff700 1 -- start start 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d83ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d0158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d83ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d01578a0 con 0x7f04d00ce390 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d83ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00cc830 con 0x7f04d00b07c0 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d83ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d0157ce0 con 0x7f04d00b3e60 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d599a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f04d00ce390 0x7f04d00b00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53302/0 (socket says 172.21.15.29:53302) 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d599a700 1 -- 172.21.15.29:0/2681154568 learned_addr learned my addr 172.21.15.29:0/2681154568 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d619b700 1 --2- 172.21.15.29:0/2681154568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d0158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.876+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538591767 0 0) 0x7f04d01578a0 con 0x7f04d00ce390 2023-10-07T09:45:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc003570 con 0x7f04d00ce390 2023-10-07T09:45:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925730835 0 0) 0x7f04d00cc830 con 0x7f04d00b07c0 2023-10-07T09:45:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d01578a0 con 0x7f04d00b07c0 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d619b700 1 -- 172.21.15.29:0/2681154568 >> v1:172.21.15.144:6789/0 conn(0x7f04d00b07c0 legacy=0x7f04d00cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d619b700 1 -- 172.21.15.29:0/2681154568 >> v1:172.21.15.29:6789/0 conn(0x7f04d00ce390 legacy=0x7f04d00b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988478248 0 0) 0x7f04bc003570 con 0x7f04d00ce390 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d619b700 1 -- 172.21.15.29:0/2681154568 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00cee50 con 0x7f04d00b3e60 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d619b700 1 --2- 172.21.15.29:0/2681154568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d0158b40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009900 tx=0x7f04cc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28089f33fbf64d65 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.877+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c0001470 con 0x7f04d00b3e60 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc011420 con 0x7f04d00b3e60 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d5199700 1 -- 172.21.15.29:0/2681154568 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc0117c0 con 0x7f04d00b3e60 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 -- 172.21.15.29:0/2681154568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 msgr2=0x7f04d0158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 --2- 172.21.15.29:0/2681154568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d0158b40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009900 tx=0x7f04cc009120 comp rx=0 tx=0).stop 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 -- 172.21.15.29:0/2681154568 shutdown_connections 2023-10-07T09:45:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 --2- 172.21.15.29:0/2681154568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d0158b40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 -- 172.21.15.29:0/2681154568 >> 172.21.15.29:0/2681154568 conn(0x7f04d0067610 msgr2=0x7f04d0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:20.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 -- 172.21.15.29:0/2681154568 shutdown_connections 2023-10-07T09:45:20.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.878+0000 7f04d83ff700 1 -- 172.21.15.29:0/2681154568 wait complete. 2023-10-07T09:45:20.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.880+0000 7f04d83ff700 1 Processor -- start 2023-10-07T09:45:20.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.880+0000 7f04d83ff700 1 -- start start 2023-10-07T09:45:20.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d83ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 0x7f04d00b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d83ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d00af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d83ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04d00ce390 0x7f04d0158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d83ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d00b1bc0 con 0x7f04d00ce390 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d83ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d01588f0 con 0x7f04d00b07c0 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d83ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d00c7bc0 con 0x7f04d00b3e60 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d599a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d00af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d619b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 0x7f04d00b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d699c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04d00ce390 0x7f04d0158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d619b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 0x7f04d00b1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:49120/0 (socket says 172.21.15.29:49120) 2023-10-07T09:45:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.881+0000 7f04d619b700 1 -- 172.21.15.29:0/1597947330 learned_addr learned my addr 172.21.15.29:0/1597947330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04d619b700 1 -- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 msgr2=0x7f04d00af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04d619b700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d00af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04d619b700 1 -- 172.21.15.29:0/1597947330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04d00ce390 msgr2=0x7f04d0158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04d619b700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04d00ce390 0x7f04d0158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04d619b700 1 -- 172.21.15.29:0/1597947330 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0093c0 con 0x7f04d00b07c0 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04d619b700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 0x7f04d00b1680 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009160 tx=0x7f04cc016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc01cb30 con 0x7f04d00b07c0 2023-10-07T09:45:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc01ccd0 con 0x7f04d00b07c0 2023-10-07T09:45:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.882+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc01a6a0 con 0x7f04d00b07c0 2023-10-07T09:45:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.883+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d00af820 con 0x7f04d00b07c0 2023-10-07T09:45:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.883+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d00aceb0 con 0x7f04d00b07c0 2023-10-07T09:45:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.884+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d004c3c0 con 0x7f04d00b07c0 2023-10-07T09:45:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.885+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f04cc01aa50 con 0x7f04d00b07c0 2023-10-07T09:45:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.886+0000 7f04c67fc700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 0x7f04b4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.886+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f04cc086440 con 0x7f04d00b07c0 2023-10-07T09:45:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.886+0000 7f04d599a700 1 -- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 msgr2=0x7f04b4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.886+0000 7f04d599a700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 0x7f04b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:20.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:20.891+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04cc053110 con 0x7f04d00b07c0 2023-10-07T09:45:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.087+0000 7f04d599a700 1 -- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 msgr2=0x7f04b4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.087+0000 7f04d599a700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 0x7f04b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:21.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.091+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f04d00cf130 con 0x7f04d00b07c0 2023-10-07T09:45:21.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.092+0000 7f04c67fc700 1 -- 172.21.15.29:0/1597947330 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f04cc056730 con 0x7f04d00b07c0 2023-10-07T09:45:21.093 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:45:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 msgr2=0x7f04b4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 0x7f04b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 msgr2=0x7f04d00b1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 0x7f04d00b1680 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009160 tx=0x7f04cc016000 comp rx=0 tx=0).stop 2023-10-07T09:45:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 shutdown_connections 2023-10-07T09:45:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f04b40670e0 0x7f04b4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04d00ce390 0x7f04d0158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04d00b07c0 0x7f04d00b1680 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 --2- 172.21.15.29:0/1597947330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04d00b3e60 0x7f04d00af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.094+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 >> 172.21.15.29:0/1597947330 conn(0x7f04d0067610 msgr2=0x7f04d0156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.095+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 shutdown_connections 2023-10-07T09:45:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.095+0000 7f04d83ff700 1 -- 172.21.15.29:0/1597947330 wait complete. 2023-10-07T09:45:21.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:45:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.315+0000 7f09687ff700 1 Processor -- start 2023-10-07T09:45:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.316+0000 7f09687ff700 1 -- start start 2023-10-07T09:45:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f09687ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0960159b20 0x7f09600d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f09687ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09600ce8a0 0x7f09600c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f09687ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09600cbf20 con 0x7f0960159750 2023-10-07T09:45:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f09687ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09600cc200 con 0x7f0960159b20 2023-10-07T09:45:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f09687ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09600cc4e0 con 0x7f09600ce8a0 2023-10-07T09:45:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f0965d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0960159b20 0x7f09600d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f0965d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0960159b20 0x7f09600d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49284/0 (socket says 172.21.15.29:49284) 2023-10-07T09:45:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f0965d9a700 1 -- 172.21.15.29:0/2825177197 learned_addr learned my addr 172.21.15.29:0/2825177197 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f096659b700 1 --1- 172.21.15.29:0/2825177197 >> v1:172.21.15.29:6789/0 conn(0x7f0960159750 0x7f09600ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53306/0 (socket says 172.21.15.29:53306) 2023-10-07T09:45:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f0966d9c700 1 --2- 172.21.15.29:0/2825177197 >> v2:172.21.15.144:3300/0 conn(0x7f09600ce8a0 0x7f09600c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f0965d9a700 1 -- 172.21.15.29:0/2825177197 >> v2:172.21.15.144:3300/0 conn(0x7f09600ce8a0 msgr2=0x7f09600c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.318+0000 7f0965d9a700 1 --2- 172.21.15.29:0/2825177197 >> v2:172.21.15.144:3300/0 conn(0x7f09600ce8a0 0x7f09600c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965d9a700 1 -- 172.21.15.29:0/2825177197 >> v1:172.21.15.29:6789/0 conn(0x7f0960159750 legacy=0x7f09600ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965d9a700 1 -- 172.21.15.29:0/2825177197 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600cc7c0 con 0x7f0960159b20 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965599700 1 -- 172.21.15.29:0/2825177197 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007181562 0 0) 0x7f09600cbf20 con 0x7f0960159750 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965d9a700 1 --2- 172.21.15.29:0/2825177197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600d10a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f096015e510 tx=0x7f0950009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5e0140b9d41045 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965599700 1 -- 172.21.15.29:0/2825177197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950009e70 con 0x7f0960159b20 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965599700 1 -- 172.21.15.29:0/2825177197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095000b860 con 0x7f0960159b20 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.319+0000 7f0965599700 1 -- 172.21.15.29:0/2825177197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095000bc00 con 0x7f0960159b20 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 -- 172.21.15.29:0/2825177197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 msgr2=0x7f09600d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 --2- 172.21.15.29:0/2825177197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600d10a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f096015e510 tx=0x7f0950009250 comp rx=0 tx=0).stop 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 -- 172.21.15.29:0/2825177197 shutdown_connections 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 --2- 172.21.15.29:0/2825177197 >> v2:172.21.15.144:3300/0 conn(0x7f09600ce8a0 0x7f09600c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 --2- 172.21.15.29:0/2825177197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600d10a0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 -- 172.21.15.29:0/2825177197 >> 172.21.15.29:0/2825177197 conn(0x7f0960063a10 msgr2=0x7f09600d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 -- 172.21.15.29:0/2825177197 shutdown_connections 2023-10-07T09:45:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.320+0000 7f09687ff700 1 -- 172.21.15.29:0/2825177197 wait complete. 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.321+0000 7f09687ff700 1 Processor -- start 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.322+0000 7f09687ff700 1 -- start start 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f09687ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09600ce8a0 0x7f09600b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f09687ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0960159750 0x7f096013f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f09687ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f09687ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09601588a0 con 0x7f0960159b20 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f09687ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09600c7de0 con 0x7f0960159750 2023-10-07T09:45:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f09687ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09600c80c0 con 0x7f09600ce8a0 2023-10-07T09:45:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49286/0 (socket says 172.21.15.29:49286) 2023-10-07T09:45:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 -- 172.21.15.29:0/973436922 learned_addr learned my addr 172.21.15.29:0/973436922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f096659b700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09600ce8a0 0x7f09600b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0965d9a700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0960159750 0x7f096013f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 -- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09600ce8a0 msgr2=0x7f09600b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09600ce8a0 0x7f09600b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 -- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0960159750 msgr2=0x7f096013f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0960159750 0x7f096013f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.323+0000 7f0966d9c700 1 -- 172.21.15.29:0/973436922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500094f0 con 0x7f0960159b20 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.324+0000 7f0966d9c700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600b50f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0958009e90 tx=0x7f095800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.324+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095800cb70 con 0x7f0960159b20 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.324+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095800f040 con 0x7f0960159b20 2023-10-07T09:45:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.324+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09600c83a0 con 0x7f0960159b20 2023-10-07T09:45:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.324+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0958014480 con 0x7f0960159b20 2023-10-07T09:45:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.324+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096013fe60 con 0x7f0960159b20 2023-10-07T09:45:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.325+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09600c88c0 con 0x7f0960159b20 2023-10-07T09:45:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.326+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f0958014830 con 0x7f0960159b20 2023-10-07T09:45:21.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.327+0000 7f09567fc700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 0x7f0944069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.327+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f0958085650 con 0x7f0960159b20 2023-10-07T09:45:21.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.327+0000 7f096659b700 1 -- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 msgr2=0x7f0944069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:21.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.327+0000 7f096659b700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 0x7f0944069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.331+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09600c88c0 con 0x7f0960159b20 2023-10-07T09:45:21.368 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:21.367+0000 7faf1252a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:45:21.455 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:21.454+0000 7faf1252a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:45:21.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.528+0000 7f096659b700 1 -- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 msgr2=0x7f0944069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:21.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.528+0000 7f096659b700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 0x7f0944069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:21.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.534+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09600c88c0 con 0x7f0960159b20 2023-10-07T09:45:21.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.535+0000 7f09567fc700 1 -- 172.21.15.29:0/973436922 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09600c88c0 con 0x7f0960159b20 2023-10-07T09:45:21.536 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.537+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 msgr2=0x7f0944069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.537+0000 7f09687ff700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 0x7f0944069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.537+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 msgr2=0x7f09600b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.537+0000 7f09687ff700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600b50f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0958009e90 tx=0x7f095800b040 comp rx=0 tx=0).stop 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 shutdown_connections 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09440670e0 0x7f0944069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0960159b20 0x7f09600b50f0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09600ce8a0 0x7f09600b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 --2- 172.21.15.29:0/973436922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0960159750 0x7f096013f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 >> 172.21.15.29:0/973436922 conn(0x7f0960063a10 msgr2=0x7f0960154ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:21.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 shutdown_connections 2023-10-07T09:45:21.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.538+0000 7f09687ff700 1 -- 172.21.15.29:0/973436922 wait complete. 2023-10-07T09:45:21.554 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:21.553+0000 7fdafd6a51c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:45:21.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:21.608 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:21.607+0000 7faf1252a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:45:21.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:45:21.631 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:45:21.654 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:21.653+0000 7fdafd6a51c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:45:21.701 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:21.700+0000 7faf1252a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.756+0000 7f6c9d3ff700 1 Processor -- start 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.756+0000 7f6c9d3ff700 1 -- start start 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c9d3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c98157290 0x7f6c980cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c9d3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c98158ad0 con 0x7f6c980ae8b0 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c9d3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9815b140 con 0x7f6c980b1a10 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c9d3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9815b420 con 0x7f6c98157290 2023-10-07T09:45:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c96ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c98157290 0x7f6c980cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c96ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c98157290 0x7f6c980cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49296/0 (socket says 172.21.15.29:49296) 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c96ffd700 1 -- 172.21.15.29:0/3074578642 learned_addr learned my addr 172.21.15.29:0/3074578642 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c967fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c980ae8b0 0x7f6c980b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53322/0 (socket says 172.21.15.29:53322) 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c96ffd700 1 -- 172.21.15.29:0/3074578642 >> v1:172.21.15.144:6789/0 conn(0x7f6c980b1a10 legacy=0x7f6c98157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c96ffd700 1 -- 172.21.15.29:0/3074578642 >> v1:172.21.15.29:6789/0 conn(0x7f6c980ae8b0 legacy=0x7f6c980b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.758+0000 7f6c96ffd700 1 -- 172.21.15.29:0/3074578642 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9815b700 con 0x7f6c98157290 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.759+0000 7f6c96ffd700 1 --2- 172.21.15.29:0/3074578642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98157290 0x7f6c980cc870 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6c880099b0 tx=0x7f6c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f69e572ad0640289 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.759+0000 7f6c95ffb700 1 -- 172.21.15.29:0/3074578642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800b860 con 0x7f6c98157290 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.759+0000 7f6c95ffb700 1 -- 172.21.15.29:0/3074578642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8800b9c0 con 0x7f6c98157290 2023-10-07T09:45:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.759+0000 7f6c95ffb700 1 -- 172.21.15.29:0/3074578642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800bd60 con 0x7f6c98157290 2023-10-07T09:45:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/3074578642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98157290 msgr2=0x7f6c980cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/3074578642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98157290 0x7f6c980cc870 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6c880099b0 tx=0x7f6c88009250 comp rx=0 tx=0).stop 2023-10-07T09:45:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/3074578642 shutdown_connections 2023-10-07T09:45:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/3074578642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c98157290 0x7f6c980cc870 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/3074578642 >> 172.21.15.29:0/3074578642 conn(0x7f6c980680a0 msgr2=0x7f6c980cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/3074578642 shutdown_connections 2023-10-07T09:45:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.760+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/3074578642 wait complete. 2023-10-07T09:45:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.762+0000 7f6c9d3ff700 1 Processor -- start 2023-10-07T09:45:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.763+0000 7f6c9d3ff700 1 -- start start 2023-10-07T09:45:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c9d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 0x7f6c980c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c9d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c98157290 0x7f6c98150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c9d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c980c21b0 0x7f6c980c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c9d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980c30a0 con 0x7f6c980b1a10 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c9d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980c3380 con 0x7f6c980c21b0 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c9d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980c3660 con 0x7f6c98157290 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 0x7f6c980c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 0x7f6c980c2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49304/0 (socket says 172.21.15.29:49304) 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.764+0000 7f6c96ffd700 1 -- 172.21.15.29:0/2533274536 learned_addr learned my addr 172.21.15.29:0/2533274536 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c967fc700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c98157290 0x7f6c98150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c977fe700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c980c21b0 0x7f6c980c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c96ffd700 1 -- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c98157290 msgr2=0x7f6c98150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c96ffd700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c98157290 0x7f6c98150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c96ffd700 1 -- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c980c21b0 msgr2=0x7f6c980c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c96ffd700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c980c21b0 0x7f6c980c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c96ffd700 1 -- 172.21.15.29:0/2533274536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880094f0 con 0x7f6c980b1a10 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c96ffd700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 0x7f6c980c2730 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6c880094c0 tx=0x7f6c88017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88024440 con 0x7f6c980b1a10 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c880245a0 con 0x7f6c980b1a10 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.765+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c980c41b0 con 0x7f6c980b1a10 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.766+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88024940 con 0x7f6c980b1a10 2023-10-07T09:45:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.766+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c980c2c70 con 0x7f6c980b1a10 2023-10-07T09:45:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.767+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c980c2e60 con 0x7f6c980b1a10 2023-10-07T09:45:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.768+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f6c8801a420 con 0x7f6c980b1a10 2023-10-07T09:45:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.769+0000 7f6c7effd700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 0x7f6c84069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.769+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f6c88086a00 con 0x7f6c980b1a10 2023-10-07T09:45:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.769+0000 7f6c967fc700 1 -- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 msgr2=0x7f6c84069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.769+0000 7f6c967fc700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 0x7f6c84069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:21.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.773+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c980c2e60 con 0x7f6c980b1a10 2023-10-07T09:45:21.826 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:21.825+0000 7fdafd6a51c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:45:21.917 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:21.916+0000 7fdafd6a51c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:45:21.921 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:21.920+0000 7faf1252a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:45:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.970+0000 7f6c967fc700 1 -- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 msgr2=0x7f6c84069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:21.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:21.970+0000 7f6c967fc700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 0x7f6c84069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:22.016 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.015+0000 7faf1252a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:45:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.023+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c980c2e60 con 0x7f6c980b1a10 2023-10-07T09:45:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.024+0000 7f6c7effd700 1 -- 172.21.15.29:0/2533274536 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f6c980c2e60 con 0x7f6c980b1a10 2023-10-07T09:45:22.025 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:22.025 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:01.071366+0000","modified":"2023-10-07T09:45:04.584641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2799196407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2799196407},{"type":"v1","addr":"172.21.15.29:6835","nonce":2799196407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-10-07T09:45:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 msgr2=0x7f6c84069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 0x7f6c84069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 msgr2=0x7f6c980c2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 0x7f6c980c2730 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6c880094c0 tx=0x7f6c88017000 comp rx=0 tx=0).stop 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 shutdown_connections 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c840670e0 0x7f6c84069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c980c21b0 0x7f6c980c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c980b1a10 0x7f6c980c2730 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 --2- 172.21.15.29:0/2533274536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c98157290 0x7f6c98150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 >> 172.21.15.29:0/2533274536 conn(0x7f6c980680a0 msgr2=0x7f6c980ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.026+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 shutdown_connections 2023-10-07T09:45:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.027+0000 7f6c9d3ff700 1 -- 172.21.15.29:0/2533274536 wait complete. 2023-10-07T09:45:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-07T09:45:22.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:22.111 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.110+0000 7faf1252a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:45:22.140 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:22.139+0000 7fdafd6a51c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:45:22.207 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.206+0000 7faf1252a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:45:22.242 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:22.241+0000 7fdafd6a51c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:45:22.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.284+0000 7f3deb9ff700 1 Processor -- start 2023-10-07T09:45:22.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.284+0000 7f3deb9ff700 1 -- start start 2023-10-07T09:45:22.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3deb9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3deb9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de414d790 con 0x7f3de40b0410 2023-10-07T09:45:22.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3deb9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40ae5d0 con 0x7f3de4159ec0 2023-10-07T09:45:22.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3deb9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de40ad740 con 0x7f3de40ad370 2023-10-07T09:45:22.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3de979b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3de979b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49306/0 (socket says 172.21.15.29:49306) 2023-10-07T09:45:22.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3de979b700 1 -- 172.21.15.29:0/4161140537 learned_addr learned my addr 172.21.15.29:0/4161140537 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:22.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.286+0000 7f3de9f9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3de40b0410 0x7f3de4156fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53330/0 (socket says 172.21.15.29:53330) 2023-10-07T09:45:22.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3de979b700 1 -- 172.21.15.29:0/4161140537 >> v1:172.21.15.144:6789/0 conn(0x7f3de4159ec0 legacy=0x7f3de41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3de979b700 1 -- 172.21.15.29:0/4161140537 >> v1:172.21.15.29:6789/0 conn(0x7f3de40b0410 legacy=0x7f3de4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3de979b700 1 -- 172.21.15.29:0/4161140537 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40ccdb0 con 0x7f3de40ad370 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3ddbfff700 1 -- 172.21.15.29:0/4161140537 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778451052 0 0) 0x7f3de414d790 con 0x7f3de40b0410 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3de979b700 1 --2- 172.21.15.29:0/4161140537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40ad370 0x7f3de40cc870 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3de00099e0 tx=0x7f3de0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a8824305614fb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3ddbfff700 1 -- 172.21.15.29:0/4161140537 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de000b820 con 0x7f3de40ad370 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3ddbfff700 1 -- 172.21.15.29:0/4161140537 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de000b980 con 0x7f3de40ad370 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.287+0000 7f3ddbfff700 1 -- 172.21.15.29:0/4161140537 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de000bd20 con 0x7f3de40ad370 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 -- 172.21.15.29:0/4161140537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40ad370 msgr2=0x7f3de40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/4161140537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40ad370 0x7f3de40cc870 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3de00099e0 tx=0x7f3de0009250 comp rx=0 tx=0).stop 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 -- 172.21.15.29:0/4161140537 shutdown_connections 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/4161140537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 -- 172.21.15.29:0/4161140537 >> 172.21.15.29:0/4161140537 conn(0x7f3de40680a0 msgr2=0x7f3de40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:22.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 -- 172.21.15.29:0/4161140537 shutdown_connections 2023-10-07T09:45:22.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.288+0000 7f3deb9ff700 1 -- 172.21.15.29:0/4161140537 wait complete. 2023-10-07T09:45:22.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.290+0000 7f3deb9ff700 1 Processor -- start 2023-10-07T09:45:22.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.290+0000 7f3deb9ff700 1 -- start start 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3deb9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de40ad370 0x7f3de415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3deb9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 0x7f3de40cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3deb9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de4159ec0 0x7f3de415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3deb9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de415a880 con 0x7f3de40b0410 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3deb9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de415ab20 con 0x7f3de40ad370 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3deb9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de415ae00 con 0x7f3de4159ec0 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3de8f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 0x7f3de40cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3de8f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 0x7f3de40cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49310/0 (socket says 172.21.15.29:49310) 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3de8f9a700 1 -- 172.21.15.29:0/1867153993 learned_addr learned my addr 172.21.15.29:0/1867153993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3de979b700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de40ad370 0x7f3de415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.291+0000 7f3de9f9c700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de4159ec0 0x7f3de415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3de8f9a700 1 -- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de4159ec0 msgr2=0x7f3de415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3de8f9a700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de4159ec0 0x7f3de415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3de8f9a700 1 -- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de40ad370 msgr2=0x7f3de415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3de8f9a700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de40ad370 0x7f3de415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3de8f9a700 1 -- 172.21.15.29:0/1867153993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00094f0 con 0x7f3de40b0410 2023-10-07T09:45:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3de8f9a700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 0x7f3de40cbb60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400c4e0 tx=0x7f3dd400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4011590 con 0x7f3de40b0410 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd4011710 con 0x7f3de40b0410 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de415b0e0 con 0x7f3de40b0410 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4011ad0 con 0x7f3de40b0410 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.292+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de415b6a0 con 0x7f3de40b0410 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.293+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de415b8c0 con 0x7f3de40b0410 2023-10-07T09:45:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.294+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f3dd401b3b0 con 0x7f3de40b0410 2023-10-07T09:45:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.295+0000 7f3dd97fa700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 0x7f3dc8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.295+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f3dd4083f70 con 0x7f3de40b0410 2023-10-07T09:45:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.295+0000 7f3de979b700 1 -- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 msgr2=0x7f3dc8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.295+0000 7f3de979b700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 0x7f3dc8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.299+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3dd4050c40 con 0x7f3de40b0410 2023-10-07T09:45:22.344 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:22.343+0000 7fdafd6a51c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:45:22.398 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.396+0000 7faf1252a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:45:22.444 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:22.443+0000 7fdafd6a51c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:45:22.494 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.493+0000 7faf1252a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:45:22.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.496+0000 7f3de979b700 1 -- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 msgr2=0x7f3dc8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:22.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.496+0000 7f3de979b700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 0x7f3dc8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:22.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.502+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3de404c3c0 con 0x7f3de40b0410 2023-10-07T09:45:22.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.503+0000 7f3dd97fa700 1 -- 172.21.15.29:0/1867153993 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16394 (secure 0 0 0) 0x7f3dd4017070 con 0x7f3de40b0410 2023-10-07T09:45:22.505 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:22.505 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:16.705837+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:58.854049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:00.048655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3721675226":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/3991888806":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/1527687027":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/262634276":"2023-10-08T09:45:16.705779+0000","172.21.15.144:6836/114714":"2023-10-08T09:45:16.705779+0000","172.21.15.144:6834/114714":"2023-10-08T09:45:16.705779+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 msgr2=0x7f3dc8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 0x7f3dc8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 msgr2=0x7f3de40cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 0x7f3de40cbb60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400c4e0 tx=0x7f3dd400c7f0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 shutdown_connections 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3dc80670e0 0x7f3dc8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de4159ec0 0x7f3de415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de40ad370 0x7f3de415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 --2- 172.21.15.29:0/1867153993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de40b0410 0x7f3de40cbb60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 >> 172.21.15.29:0/1867153993 conn(0x7f3de40680a0 msgr2=0x7f3de40abc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 shutdown_connections 2023-10-07T09:45:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.506+0000 7f3deb9ff700 1 -- 172.21.15.29:0/1867153993 wait complete. 2023-10-07T09:45:22.527 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:45:22.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:22.646 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:22.645+0000 7fdafd6a51c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:45:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.727+0000 7fb23afff700 1 Processor -- start 2023-10-07T09:45:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.727+0000 7fb23afff700 1 -- start start 2023-10-07T09:45:22.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.728+0000 7fb23afff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2341595f0 0x7fb23415d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.728+0000 7fb23afff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340cf570 con 0x7fb23415dfe0 2023-10-07T09:45:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.728+0000 7fb23afff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340cf850 con 0x7fb2341599c0 2023-10-07T09:45:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.728+0000 7fb23afff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2340cfb30 con 0x7fb2341595f0 2023-10-07T09:45:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb23959c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb23415dfe0 0x7fb2340caba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53340/0 (socket says 172.21.15.29:53340) 2023-10-07T09:45:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb23959c700 1 -- 172.21.15.29:0/3998655215 learned_addr learned my addr 172.21.15.29:0/3998655215 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb238d9b700 1 --2- 172.21.15.29:0/3998655215 >> v2:172.21.15.144:3300/0 conn(0x7fb2341595f0 0x7fb23415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153214536 0 0) 0x7fb2340cf570 con 0x7fb23415dfe0 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb224003570 con 0x7fb23415dfe0 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820883923 0 0) 0x7fb2340cf850 con 0x7fb2341599c0 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2340cf570 con 0x7fb2341599c0 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb238d9b700 1 -- 172.21.15.29:0/3998655215 >> v1:172.21.15.144:6789/0 conn(0x7fb2341599c0 legacy=0x7fb23415d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb238d9b700 1 -- 172.21.15.29:0/3998655215 >> v1:172.21.15.29:6789/0 conn(0x7fb23415dfe0 legacy=0x7fb2340caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.729+0000 7fb238d9b700 1 -- 172.21.15.29:0/3998655215 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340cfe10 con 0x7fb2341595f0 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.730+0000 7fb238d9b700 1 --2- 172.21.15.29:0/3998655215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341595f0 0x7fb23415d390 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2280098d0 tx=0x7fb228009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f9a2767a4f651f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.730+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228005c10 con 0x7fb2341595f0 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.730+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb228005d70 con 0x7fb2341595f0 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.730+0000 7fb2337fe700 1 -- 172.21.15.29:0/3998655215 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228006280 con 0x7fb2341595f0 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 -- 172.21.15.29:0/3998655215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341595f0 msgr2=0x7fb23415d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 --2- 172.21.15.29:0/3998655215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341595f0 0x7fb23415d390 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2280098d0 tx=0x7fb228009120 comp rx=0 tx=0).stop 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 -- 172.21.15.29:0/3998655215 shutdown_connections 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 --2- 172.21.15.29:0/3998655215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341595f0 0x7fb23415d390 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 -- 172.21.15.29:0/3998655215 >> 172.21.15.29:0/3998655215 conn(0x7fb234063a10 msgr2=0x7fb2340d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 -- 172.21.15.29:0/3998655215 shutdown_connections 2023-10-07T09:45:22.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.731+0000 7fb23afff700 1 -- 172.21.15.29:0/3998655215 wait complete. 2023-10-07T09:45:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.732+0000 7fb23afff700 1 Processor -- start 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.732+0000 7fb23afff700 1 -- start start 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.733+0000 7fb23afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2341595f0 0x7fb23414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.733+0000 7fb23afff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341599c0 0x7fb23415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.733+0000 7fb23afff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 0x7fb2340c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.733+0000 7fb23afff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23415ae90 con 0x7fb23415dfe0 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.733+0000 7fb23afff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23415bcf0 con 0x7fb2341599c0 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.733+0000 7fb23afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23414b780 con 0x7fb2341595f0 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 0x7fb2340c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 0x7fb2340c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49314/0 (socket says 172.21.15.29:49314) 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 -- 172.21.15.29:0/3790880551 learned_addr learned my addr 172.21.15.29:0/3790880551 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb238d9b700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2341595f0 0x7fb23414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb233fff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341599c0 0x7fb23415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 -- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2341595f0 msgr2=0x7fb23414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2341595f0 0x7fb23414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 -- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341599c0 msgr2=0x7fb23415c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341599c0 0x7fb23415c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 -- 172.21.15.29:0/3790880551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2280093c0 con 0x7fb23415dfe0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.734+0000 7fb23959c700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 0x7fb2340c8170 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00cc50 tx=0x7fb22c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.735+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00a810 con 0x7fb23415dfe0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.735+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c00a970 con 0x7fb23415dfe0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.735+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00acf0 con 0x7fb23415dfe0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.735+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23415b0e0 con 0x7fb23415dfe0 2023-10-07T09:45:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.735+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23415b6a0 con 0x7fb23415dfe0 2023-10-07T09:45:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.736+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23415be60 con 0x7fb23415dfe0 2023-10-07T09:45:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.737+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fb22c007300 con 0x7fb23415dfe0 2023-10-07T09:45:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.738+0000 7fb230ff9700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 0x7fb218069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.738+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fb22c084e10 con 0x7fb23415dfe0 2023-10-07T09:45:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.738+0000 7fb238d9b700 1 -- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 msgr2=0x7fb218069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.738+0000 7fb238d9b700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 0x7fb218069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.746+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb22c088030 con 0x7fb23415dfe0 2023-10-07T09:45:22.756 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:22.755+0000 7fdafd6a51c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:45:22.836 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.835+0000 7faf1252a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:45:22.924 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:22.923+0000 7faf1252a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:22.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.939+0000 7fb238d9b700 1 -- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 msgr2=0x7fb218069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:22.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.939+0000 7fb238d9b700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 0x7fb218069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.993+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb23404c3c0 con 0x7fb23415dfe0 2023-10-07T09:45:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.994+0000 7fb230ff9700 1 -- 172.21.15.29:0/3790880551 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fb22c051ae0 con 0x7fb23415dfe0 2023-10-07T09:45:22.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:22.995 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:01.071366+0000","modified":"2023-10-07T09:45:04.584641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2799196407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2799196407},{"type":"v1","addr":"172.21.15.29:6835","nonce":2799196407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 msgr2=0x7fb218069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 0x7fb218069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 msgr2=0x7fb2340c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 0x7fb2340c8170 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00cc50 tx=0x7fb22c001a40 comp rx=0 tx=0).stop 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 shutdown_connections 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb2180670e0 0x7fb218069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23415dfe0 0x7fb2340c8170 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2341595f0 0x7fb23414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 --2- 172.21.15.29:0/3790880551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2341599c0 0x7fb23415c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.996+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 >> 172.21.15.29:0/3790880551 conn(0x7fb234063a10 msgr2=0x7fb234155190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.997+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 shutdown_connections 2023-10-07T09:45:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:22.997+0000 7fb23afff700 1 -- 172.21.15.29:0/3790880551 wait complete. 2023-10-07T09:45:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-07T09:45:23.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:23.111 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:23.110+0000 7fdafd6a51c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:45:23.203 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:23.202+0000 7fdafd6a51c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.214+0000 7f1441fff700 1 Processor -- start 2023-10-07T09:45:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.215+0000 7f1441fff700 1 -- start start 2023-10-07T09:45:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f1441fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c157300 0x7f143c0b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f1441fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f143c0cd330 0x7f143c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f1441fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c0cb330 con 0x7f143c1576d0 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f1441fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143c06f980 con 0x7f143c0cd330 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f1441fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143c0cbe90 con 0x7f143c157300 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f143bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f143c0cd330 0x7f143c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f143b7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c157300 0x7f143c0b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f143bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f143c0cd330 0x7f143c0b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46434/0 (socket says 172.21.15.29:46434) 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f143bfff700 1 -- 172.21.15.29:0/2939619860 learned_addr learned my addr 172.21.15.29:0/2939619860 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f143b7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c157300 0x7f143c0b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:49870/0 (socket says 172.21.15.29:49870) 2023-10-07T09:45:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.216+0000 7f143affd700 1 --1- 172.21.15.29:0/2939619860 >> v1:172.21.15.144:6789/0 conn(0x7f143c1576d0 0x7f143c067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:32786/0 (socket says 172.21.15.29:32786) 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143bfff700 1 -- 172.21.15.29:0/2939619860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c157300 msgr2=0x7f143c0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143bfff700 1 --2- 172.21.15.29:0/2939619860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c157300 0x7f143c0b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143bfff700 1 -- 172.21.15.29:0/2939619860 >> v1:172.21.15.144:6789/0 conn(0x7f143c1576d0 legacy=0x7f143c067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143bfff700 1 -- 172.21.15.29:0/2939619860 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c0cb560 con 0x7f143c0cd330 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143a7fc700 1 -- 172.21.15.29:0/2939619860 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092279532 0 0) 0x7f143c0cb330 con 0x7f143c1576d0 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143bfff700 1 --2- 172.21.15.29:0/2939619860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c0cd330 0x7f143c0b28c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1434009f20 tx=0x7f1434009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f554b1ad4173aa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143a7fc700 1 -- 172.21.15.29:0/2939619860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143400e070 con 0x7f143c0cd330 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143a7fc700 1 -- 172.21.15.29:0/2939619860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143400b9d0 con 0x7f143c0cd330 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.217+0000 7f143a7fc700 1 -- 172.21.15.29:0/2939619860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14340092e0 con 0x7f143c0cd330 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 -- 172.21.15.29:0/2939619860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c0cd330 msgr2=0x7f143c0b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 --2- 172.21.15.29:0/2939619860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c0cd330 0x7f143c0b28c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1434009f20 tx=0x7f1434009840 comp rx=0 tx=0).stop 2023-10-07T09:45:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 -- 172.21.15.29:0/2939619860 shutdown_connections 2023-10-07T09:45:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 --2- 172.21.15.29:0/2939619860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c0cd330 0x7f143c0b28c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 --2- 172.21.15.29:0/2939619860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c157300 0x7f143c0b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 -- 172.21.15.29:0/2939619860 >> 172.21.15.29:0/2939619860 conn(0x7f143c069190 msgr2=0x7f143c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 -- 172.21.15.29:0/2939619860 shutdown_connections 2023-10-07T09:45:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.218+0000 7f1441fff700 1 -- 172.21.15.29:0/2939619860 wait complete. 2023-10-07T09:45:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.220+0000 7f1441fff700 1 Processor -- start 2023-10-07T09:45:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.220+0000 7f1441fff700 1 -- start start 2023-10-07T09:45:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f1441fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 0x7f143c0a7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f1441fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c157300 0x7f143c0c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f1441fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c159730 0x7f143c16ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f1441fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c0c9e00 con 0x7f143c157300 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f1441fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c15b370 con 0x7f143c0cd330 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f1441fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143c15b650 con 0x7f143c159730 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f143b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 0x7f143c0a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f143affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c157300 0x7f143c0c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f143b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 0x7f143c0a7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52814/0 (socket says 172.21.15.29:52814) 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f143bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c159730 0x7f143c16ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.221+0000 7f143b7fe700 1 -- 172.21.15.29:0/1911615320 learned_addr learned my addr 172.21.15.29:0/1911615320 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143b7fe700 1 -- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c159730 msgr2=0x7f143c16ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143b7fe700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c159730 0x7f143c16ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143b7fe700 1 -- 172.21.15.29:0/1911615320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c157300 msgr2=0x7f143c0c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143b7fe700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c157300 0x7f143c0c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143b7fe700 1 -- 172.21.15.29:0/1911615320 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1434009ae0 con 0x7f143c0cd330 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143affd700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c157300 0x7f143c0c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f143b7fe700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 0x7f143c0a7880 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f142c00ec80 tx=0x7f142c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c00c850 con 0x7f143c0cd330 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c00c9d0 con 0x7f143c0cd330 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c00cd90 con 0x7f143c0cd330 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.222+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c15b930 con 0x7f143c0cd330 2023-10-07T09:45:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.223+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c15bf70 con 0x7f143c0cd330 2023-10-07T09:45:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.224+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c04c3c0 con 0x7f143c0cd330 2023-10-07T09:45:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.225+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f142c013680 con 0x7f143c0cd330 2023-10-07T09:45:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.225+0000 7f14237fe700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 0x7f1428069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.226+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f142c084da0 con 0x7f143c0cd330 2023-10-07T09:45:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.226+0000 7f143affd700 1 -- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 msgr2=0x7f1428069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.226+0000 7f143affd700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 0x7f1428069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:23.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.230+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f142c051a70 con 0x7f143c0cd330 2023-10-07T09:45:23.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.427+0000 7f143affd700 1 -- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 msgr2=0x7f1428069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:23.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.427+0000 7f143affd700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 0x7f1428069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:23.461 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:23.460+0000 7faf1252a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:45:23.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.477+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f143c0cb840 con 0x7f143c0cd330 2023-10-07T09:45:23.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.478+0000 7f14237fe700 1 -- 172.21.15.29:0/1911615320 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f142c055090 con 0x7f143c0cd330 2023-10-07T09:45:23.479 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:23.479 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:01.071366+0000","modified":"2023-10-07T09:45:04.584641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2799196407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2799196407},{"type":"v1","addr":"172.21.15.29:6835","nonce":2799196407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-10-07T09:45:23.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 msgr2=0x7f1428069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:23.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 0x7f1428069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 msgr2=0x7f143c0a7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 0x7f143c0a7880 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f142c00ec80 tx=0x7f142c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 shutdown_connections 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f14280670e0 0x7f1428069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f143c159730 0x7f143c16ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f143c0cd330 0x7f143c0a7880 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 --2- 172.21.15.29:0/1911615320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f143c157300 0x7f143c0c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 >> 172.21.15.29:0/1911615320 conn(0x7f143c069190 msgr2=0x7f143c152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 shutdown_connections 2023-10-07T09:45:23.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.480+0000 7f1441fff700 1 -- 172.21.15.29:0/1911615320 wait complete. 2023-10-07T09:45:23.483 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-07T09:45:23.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:23.645 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:23.644+0000 7faf1252a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:45:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.696+0000 7fd6f89ff700 1 Processor -- start 2023-10-07T09:45:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.696+0000 7fd6f89ff700 1 -- start start 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.697+0000 7fd6f89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f4159ec0 0x7fd6f415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.697+0000 7fd6f89ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f414d790 con 0x7fd6f40ad370 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.697+0000 7fd6f89ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40ae5d0 con 0x7fd6f40af400 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.697+0000 7fd6f89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f415a850 con 0x7fd6f4159ec0 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f259c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd6f40ad370 0x7fd6f40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33638/0 (socket says 172.21.15.29:33638) 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f259c700 1 -- 172.21.15.29:0/2086316534 learned_addr learned my addr 172.21.15.29:0/2086316534 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f1d9b700 1 --2- 172.21.15.29:0/2086316534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f4159ec0 0x7fd6f415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f159a700 1 -- 172.21.15.29:0/2086316534 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807571150 0 0) 0x7fd6f414d790 con 0x7fd6f40ad370 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f159a700 1 -- 172.21.15.29:0/2086316534 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6dc003570 con 0x7fd6f40ad370 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f159a700 1 -- 172.21.15.29:0/2086316534 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649340854 0 0) 0x7fd6f40ae5d0 con 0x7fd6f40af400 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f1d9b700 1 -- 172.21.15.29:0/2086316534 >> v1:172.21.15.144:6789/0 conn(0x7fd6f40af400 legacy=0x7fd6f40af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f1d9b700 1 -- 172.21.15.29:0/2086316534 >> v1:172.21.15.29:6789/0 conn(0x7fd6f40ad370 legacy=0x7fd6f40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.698+0000 7fd6f1d9b700 1 -- 172.21.15.29:0/2086316534 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f40ad740 con 0x7fd6f4159ec0 2023-10-07T09:45:23.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.699+0000 7fd6f1d9b700 1 --2- 172.21.15.29:0/2086316534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f4159ec0 0x7fd6f415a310 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009910 tx=0x7fd6e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb6417886326411 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.699+0000 7fd6f159a700 1 -- 172.21.15.29:0/2086316534 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e800b680 con 0x7fd6f4159ec0 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.699+0000 7fd6f159a700 1 -- 172.21.15.29:0/2086316534 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e800b7e0 con 0x7fd6f4159ec0 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.699+0000 7fd6f159a700 1 -- 172.21.15.29:0/2086316534 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e800bba0 con 0x7fd6f4159ec0 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.699+0000 7fd6f89ff700 1 -- 172.21.15.29:0/2086316534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f4159ec0 msgr2=0x7fd6f415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.699+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/2086316534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f4159ec0 0x7fd6f415a310 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009910 tx=0x7fd6e80091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.700+0000 7fd6f89ff700 1 -- 172.21.15.29:0/2086316534 shutdown_connections 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.700+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/2086316534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f4159ec0 0x7fd6f415a310 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.700+0000 7fd6f89ff700 1 -- 172.21.15.29:0/2086316534 >> 172.21.15.29:0/2086316534 conn(0x7fd6f40680a0 msgr2=0x7fd6f40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.700+0000 7fd6f89ff700 1 -- 172.21.15.29:0/2086316534 shutdown_connections 2023-10-07T09:45:23.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.700+0000 7fd6f89ff700 1 -- 172.21.15.29:0/2086316534 wait complete. 2023-10-07T09:45:23.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.701+0000 7fd6f89ff700 1 Processor -- start 2023-10-07T09:45:23.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.701+0000 7fd6f89ff700 1 -- start start 2023-10-07T09:45:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.702+0000 7fd6f89ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 0x7fd6f416a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.702+0000 7fd6f89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f40af400 0x7fd6f413fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.702+0000 7fd6f89ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6f4159ec0 0x7fd6f40a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.702+0000 7fd6f89ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40ab0b0 con 0x7fd6f4159ec0 2023-10-07T09:45:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.702+0000 7fd6f89ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f4169f40 con 0x7fd6f40ad370 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.702+0000 7fd6f89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f4140100 con 0x7fd6f40af400 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 0x7fd6f416a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6f4159ec0 0x7fd6f40a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f1d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f40af400 0x7fd6f413fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 0x7fd6f416a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52824/0 (socket says 172.21.15.29:52824) 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f259c700 1 -- 172.21.15.29:0/3403026589 learned_addr learned my addr 172.21.15.29:0/3403026589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f1d9b700 1 -- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 msgr2=0x7fd6f416a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f1d9b700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 0x7fd6f416a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f1d9b700 1 -- 172.21.15.29:0/3403026589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6f4159ec0 msgr2=0x7fd6f40a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f1d9b700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6f4159ec0 0x7fd6f40a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f1d9b700 1 -- 172.21.15.29:0/3403026589 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e8009420 con 0x7fd6f40af400 2023-10-07T09:45:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.703+0000 7fd6f259c700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 0x7fd6f416a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.704+0000 7fd6f1d9b700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f40af400 0x7fd6f413fbc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8016000 tx=0x7fd6e8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.704+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e800bb60 con 0x7fd6f40af400 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.704+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8000c40 con 0x7fd6f40af400 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.704+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e80276a0 con 0x7fd6f40af400 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.704+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f4169330 con 0x7fd6f40af400 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.704+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f41698f0 con 0x7fd6f40af400 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.705+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f416a0d0 con 0x7fd6f40af400 2023-10-07T09:45:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.706+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fd6e8027a10 con 0x7fd6f40af400 2023-10-07T09:45:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.707+0000 7fd6e27fc700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 0x7fd6d4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.707+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fd6e8086940 con 0x7fd6f40af400 2023-10-07T09:45:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.707+0000 7fd6f259c700 1 -- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 msgr2=0x7fd6d4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.707+0000 7fd6f259c700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 0x7fd6d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.712+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6e8053610 con 0x7fd6f40af400 2023-10-07T09:45:23.778 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:23.777+0000 7fdafd6a51c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:45:23.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.908+0000 7fd6f259c700 1 -- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 msgr2=0x7fd6d4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:23.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.908+0000 7fd6f259c700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 0x7fd6d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:23.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.917+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6f404c3c0 con 0x7fd6f40af400 2023-10-07T09:45:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.919+0000 7fd6e27fc700 1 -- 172.21.15.29:0/3403026589 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16394 (secure 0 0 0) 0x7fd6e8015090 con 0x7fd6f40af400 2023-10-07T09:45:23.920 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:23.920 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:16.705837+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:44:58.854049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:00.048655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3721675226":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/3991888806":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/1527687027":"2023-10-08T09:45:16.705779+0000","172.21.15.144:0/262634276":"2023-10-08T09:45:16.705779+0000","172.21.15.144:6836/114714":"2023-10-08T09:45:16.705779+0000","172.21.15.144:6834/114714":"2023-10-08T09:45:16.705779+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 msgr2=0x7fd6d4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 0x7fd6d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f40af400 msgr2=0x7fd6f413fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f40af400 0x7fd6f413fbc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8016000 tx=0x7fd6e8017000 comp rx=0 tx=0).stop 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 shutdown_connections 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd6d40670e0 0x7fd6d4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6f4159ec0 0x7fd6f40a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6f40ad370 0x7fd6f416a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 --2- 172.21.15.29:0/3403026589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6f40af400 0x7fd6f413fbc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 >> 172.21.15.29:0/3403026589 conn(0x7fd6f40680a0 msgr2=0x7fd6f40abc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:23.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 shutdown_connections 2023-10-07T09:45:23.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:23.921+0000 7fd6f89ff700 1 -- 172.21.15.29:0/3403026589 wait complete. 2023-10-07T09:45:23.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:45:23.974 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:23.973+0000 7fdafd6a51c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:45:24.019 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:24.018+0000 7faf1252a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:24.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.144+0000 7ff25bdff700 1 Processor -- start 2023-10-07T09:45:24.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.144+0000 7ff25bdff700 1 -- start start 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.145+0000 7ff25bdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff25415ade0 0x7ff2540b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.145+0000 7ff25bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2540b0540 0x7ff25415c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.145+0000 7ff25bdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2540b09b0 con 0x7ff254153cc0 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.145+0000 7ff25bdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff254154090 con 0x7ff25415ade0 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.145+0000 7ff25bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff254159f60 con 0x7ff2540b0540 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25939a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff25415ade0 0x7ff2540b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25a39c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2540b0540 0x7ff25415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff259b9b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff254153cc0 0x7ff2540b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:32808/0 (socket says 172.21.15.29:32808) 2023-10-07T09:45:24.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff259b9b700 1 -- 172.21.15.29:0/875261847 learned_addr learned my addr 172.21.15.29:0/875261847 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff258b99700 1 -- 172.21.15.29:0/875261847 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993406624 0 0) 0x7ff2540b09b0 con 0x7ff254153cc0 2023-10-07T09:45:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff258b99700 1 -- 172.21.15.29:0/875261847 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240003570 con 0x7ff254153cc0 2023-10-07T09:45:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25939a700 1 -- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2540b0540 msgr2=0x7ff25415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25939a700 1 --2- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2540b0540 0x7ff25415c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25939a700 1 -- 172.21.15.29:0/875261847 >> v1:172.21.15.144:6789/0 conn(0x7ff254153cc0 legacy=0x7ff2540b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25939a700 1 -- 172.21.15.29:0/875261847 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25415a240 con 0x7ff25415ade0 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.146+0000 7ff25a39c700 1 --2- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2540b0540 0x7ff25415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.147+0000 7ff25939a700 1 --2- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25415ade0 0x7ff2540b0000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff2440098e0 tx=0x7ff2440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad4e4c1b29ac9d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.147+0000 7ff258b99700 1 -- 172.21.15.29:0/875261847 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244011630 con 0x7ff25415ade0 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.147+0000 7ff258b99700 1 -- 172.21.15.29:0/875261847 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244011790 con 0x7ff25415ade0 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.147+0000 7ff258b99700 1 -- 172.21.15.29:0/875261847 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244011b30 con 0x7ff25415ade0 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 -- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25415ade0 msgr2=0x7ff2540b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 --2- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25415ade0 0x7ff2540b0000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff2440098e0 tx=0x7ff2440091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 -- 172.21.15.29:0/875261847 shutdown_connections 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 --2- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2540b0540 0x7ff25415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 --2- 172.21.15.29:0/875261847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff25415ade0 0x7ff2540b0000 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 -- 172.21.15.29:0/875261847 >> 172.21.15.29:0/875261847 conn(0x7ff254067610 msgr2=0x7ff254067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 -- 172.21.15.29:0/875261847 shutdown_connections 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.148+0000 7ff25bdff700 1 -- 172.21.15.29:0/875261847 wait complete. 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.149+0000 7ff25bdff700 1 Processor -- start 2023-10-07T09:45:24.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.149+0000 7ff25bdff700 1 -- start start 2023-10-07T09:45:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.150+0000 7ff25bdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2540b0540 0x7ff25416a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.150+0000 7ff25bdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 0x7ff254152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.150+0000 7ff25bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff25415ade0 0x7ff2540c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.150+0000 7ff25bdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25416a890 con 0x7ff2540b0540 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.150+0000 7ff25bdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540c7990 con 0x7ff254153cc0 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.150+0000 7ff25bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540c7c70 con 0x7ff25415ade0 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff259b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2540b0540 0x7ff25416a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 0x7ff254152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 0x7ff254152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52838/0 (socket says 172.21.15.29:52838) 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 -- 172.21.15.29:0/2473578092 learned_addr learned my addr 172.21.15.29:0/2473578092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25a39c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff25415ade0 0x7ff2540c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 -- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff25415ade0 msgr2=0x7ff2540c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff25415ade0 0x7ff2540c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 -- 172.21.15.29:0/2473578092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2540b0540 msgr2=0x7ff25416a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2540b0540 0x7ff25416a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 -- 172.21.15.29:0/2473578092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c009a20 con 0x7ff254153cc0 2023-10-07T09:45:24.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.151+0000 7ff25939a700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 0x7ff254152a60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff244013000 tx=0x7ff244017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:24.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.152+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244011b10 con 0x7ff254153cc0 2023-10-07T09:45:24.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.152+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2440253f0 con 0x7ff254153cc0 2023-10-07T09:45:24.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.152+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff244009420 con 0x7ff254153cc0 2023-10-07T09:45:24.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.152+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2440257b0 con 0x7ff254153cc0 2023-10-07T09:45:24.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.152+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2540c8230 con 0x7ff254153cc0 2023-10-07T09:45:24.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.153+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25404c3c0 con 0x7ff254153cc0 2023-10-07T09:45:24.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.154+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7ff244025ba0 con 0x7ff254153cc0 2023-10-07T09:45:24.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.155+0000 7ff249ffb700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 0x7ff238069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.155+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6339+0+0 (secure 0 0 0) 0x7ff244086250 con 0x7ff254153cc0 2023-10-07T09:45:24.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.155+0000 7ff259b9b700 1 -- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 msgr2=0x7ff238069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:24.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.155+0000 7ff259b9b700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 0x7ff238069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:24.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.164+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff244052fe0 con 0x7ff254153cc0 2023-10-07T09:45:24.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.356+0000 7ff259b9b700 1 -- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 msgr2=0x7ff238069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:24.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.356+0000 7ff259b9b700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 0x7ff238069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:24.363 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:24.362+0000 7fdafd6a51c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.412+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff25415a520 con 0x7ff254153cc0 2023-10-07T09:45:24.414 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:45:24.414+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:45:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.421+0000 7ff249ffb700 1 -- 172.21.15.29:0/2473578092 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff244056600 con 0x7ff254153cc0 2023-10-07T09:45:24.423 INFO:tasks.ceph.mds.c.smithi029.stderr: -1277> 2023-10-07T09:45:02.540+0000 7ff735df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:45:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 msgr2=0x7ff238069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 0x7ff238069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 msgr2=0x7ff254152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 0x7ff254152a60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff244013000 tx=0x7ff244017000 comp rx=0 tx=0).stop 2023-10-07T09:45:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 shutdown_connections 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff2380670e0 0x7ff238069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff25415ade0 0x7ff2540c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2540b0540 0x7ff25416a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 --2- 172.21.15.29:0/2473578092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff254153cc0 0x7ff254152a60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.423+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 >> 172.21.15.29:0/2473578092 conn(0x7ff254067610 msgr2=0x7ff25415b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.424+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 shutdown_connections 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.424+0000 7ff25bdff700 1 -- 172.21.15.29:0/2473578092 wait complete. 2023-10-07T09:45:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:45:24.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:45:24.452 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:24.451+0000 7faf1252a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:45:24.473 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:45:24.637 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:24.636+0000 7faf1252a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:24.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.684+0000 7f194ffff700 1 Processor -- start 2023-10-07T09:45:24.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.684+0000 7f194ffff700 1 -- start start 2023-10-07T09:45:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19481596e0 0x7f19480cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480cc630 0x7f19480acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194ffff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19480ad440 0x7f194815d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194ffff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480aa8e0 con 0x7f19481596e0 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194ffff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480ccaa0 con 0x7f19480ad440 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1948159ab0 con 0x7f19480cc630 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19481596e0 0x7f19480cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19481596e0 0x7f19480cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46476/0 (socket says 172.21.15.29:46476) 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194dd9b700 1 -- 172.21.15.29:0/1886559206 learned_addr learned my addr 172.21.15.29:0/1886559206 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194e59c700 1 --2- 172.21.15.29:0/1886559206 >> v2:172.21.15.144:3300/0 conn(0x7f19480ad440 0x7f194815d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.686+0000 7f194d59a700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480cc630 0x7f19480acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194dd9b700 1 -- 172.21.15.29:0/1886559206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480cc630 msgr2=0x7f19480acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194dd9b700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480cc630 0x7f19480acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194dd9b700 1 -- 172.21.15.29:0/1886559206 >> v2:172.21.15.144:3300/0 conn(0x7f19480ad440 msgr2=0x7f194815d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194dd9b700 1 --2- 172.21.15.29:0/1886559206 >> v2:172.21.15.144:3300/0 conn(0x7f19480ad440 0x7f194815d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194dd9b700 1 -- 172.21.15.29:0/1886559206 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480a98d0 con 0x7f19481596e0 2023-10-07T09:45:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194d59a700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480cc630 0x7f19480acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194dd9b700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19481596e0 0x7f19480cc0f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f194806cdc0 tx=0x7f1944009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba2b80a81385999 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194cd99700 1 -- 172.21.15.29:0/1886559206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944009e70 con 0x7f19481596e0 2023-10-07T09:45:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194cd99700 1 -- 172.21.15.29:0/1886559206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194400b860 con 0x7f19481596e0 2023-10-07T09:45:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.687+0000 7f194cd99700 1 -- 172.21.15.29:0/1886559206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194400bc00 con 0x7f19481596e0 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 -- 172.21.15.29:0/1886559206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19481596e0 msgr2=0x7f19480cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19481596e0 0x7f19480cc0f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f194806cdc0 tx=0x7f1944009250 comp rx=0 tx=0).stop 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 -- 172.21.15.29:0/1886559206 shutdown_connections 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 --2- 172.21.15.29:0/1886559206 >> v2:172.21.15.144:3300/0 conn(0x7f19480ad440 0x7f194815d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19481596e0 0x7f19480cc0f0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 --2- 172.21.15.29:0/1886559206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480cc630 0x7f19480acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 -- 172.21.15.29:0/1886559206 >> 172.21.15.29:0/1886559206 conn(0x7f1948067610 msgr2=0x7f1948067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 -- 172.21.15.29:0/1886559206 shutdown_connections 2023-10-07T09:45:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.688+0000 7f194ffff700 1 -- 172.21.15.29:0/1886559206 wait complete. 2023-10-07T09:45:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.689+0000 7f194ffff700 1 Processor -- start 2023-10-07T09:45:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 -- start start 2023-10-07T09:45:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480ad440 0x7f1948156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 0x7f19480ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19481596e0 0x7f1948169f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19481566c0 con 0x7f19480cc630 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948155ab0 con 0x7f19481596e0 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.690+0000 7f194ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1948155d90 con 0x7f19480ad440 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194d59a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 0x7f19480ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194dd9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480ad440 0x7f1948156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194d59a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 0x7f19480ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46486/0 (socket says 172.21.15.29:46486) 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194d59a700 1 -- 172.21.15.29:0/2911034850 learned_addr learned my addr 172.21.15.29:0/2911034850 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194e59c700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19481596e0 0x7f1948169f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194dd9b700 1 -- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19481596e0 msgr2=0x7f1948169f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194dd9b700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19481596e0 0x7f1948169f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194dd9b700 1 -- 172.21.15.29:0/2911034850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 msgr2=0x7f19480ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194dd9b700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 0x7f19480ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194dd9b700 1 -- 172.21.15.29:0/2911034850 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19440094f0 con 0x7f19480ad440 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.691+0000 7f194d59a700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 0x7f19480ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.692+0000 7f194dd9b700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480ad440 0x7f1948156b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f19480642c0 tx=0x7f1944017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.692+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194400bbe0 con 0x7f19480ad440 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.692+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194401f070 con 0x7f19480ad440 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.692+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1948156070 con 0x7f19480ad440 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.692+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19440266b0 con 0x7f19480ad440 2023-10-07T09:45:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.692+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19480cad70 con 0x7f19480ad440 2023-10-07T09:45:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.693+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19480cb020 con 0x7f19480ad440 2023-10-07T09:45:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.694+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 37) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f1944026aa0 con 0x7f19480ad440 2023-10-07T09:45:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.695+0000 7f193dffb700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 0x7f192c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.695+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f1944021070 con 0x7f19480ad440 2023-10-07T09:45:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.695+0000 7f194d59a700 1 -- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 msgr2=0x7f192c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.695+0000 7f194d59a700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 0x7f192c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:24.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.699+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f19480cb020 con 0x7f19480ad440 2023-10-07T09:45:24.827 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:24.826+0000 7faf1252a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:45:24.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:24.840+0000 7fdafd6a51c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:45:24.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.896+0000 7f194d59a700 1 -- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 msgr2=0x7f192c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:24.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.896+0000 7f194d59a700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 0x7f192c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:24.937 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:24.936+0000 7faf1252a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:24.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.948+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mgrmap(e 38) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f1944026d50 con 0x7f19480ad440 2023-10-07T09:45:24.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.948+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 msgr2=0x7f192c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:24.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.948+0000 7f193dffb700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 0x7f192c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:24.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:24.955+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19480cb020 con 0x7f19480ad440 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae8ebc7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae8ebc7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae8ebc7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae8ebc7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae8ebc7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae883ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.019+0000 7fae883ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.020+0000 7fae883ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.020+0000 7fae883ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.022 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:25.020+0000 7fae883ba700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:25.036 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:25.035+0000 7fdafd6a51c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:25.230 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:25.229+0000 7fdafd6a51c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:45:25.342 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:25.340+0000 7fdafd6a51c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.419+0000 7f193dffb700 1 -- 172.21.15.29:0/2911034850 <== mon.2 v2:172.21.15.144:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f19480cb020 con 0x7f19480ad440 2023-10-07T09:45:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480ad440 msgr2=0x7f1948156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480ad440 0x7f1948156b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f19480642c0 tx=0x7f1944017000 comp rx=0 tx=0).stop 2023-10-07T09:45:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 shutdown_connections 2023-10-07T09:45:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f192c0670e0 0x7f192c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19481596e0 0x7f1948169f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19480ad440 0x7f1948156b50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 --2- 172.21.15.29:0/2911034850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19480cc630 0x7f19480ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 >> 172.21.15.29:0/2911034850 conn(0x7f1948067610 msgr2=0x7f1948154cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 shutdown_connections 2023-10-07T09:45:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.422+0000 7f194ffff700 1 -- 172.21.15.29:0/2911034850 wait complete. 2023-10-07T09:45:25.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:45:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.641+0000 7f8c319ff700 1 Processor -- start 2023-10-07T09:45:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.641+0000 7f8c319ff700 1 -- start start 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.642+0000 7f8c319ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c2c0afeb0 0x7f8c2c0cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.642+0000 7f8c319ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c15a590 0x7f8c2c15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.642+0000 7f8c319ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2c0ad160 con 0x7f8c2c0accf0 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.642+0000 7f8c319ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2c15a140 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.642+0000 7f8c319ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c0adca0 con 0x7f8c2c15a590 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c2c0afeb0 0x7f8c2c0cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c2c0afeb0 0x7f8c2c0cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46498/0 (socket says 172.21.15.29:46498) 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 -- 172.21.15.29:0/1600125301 learned_addr learned my addr 172.21.15.29:0/1600125301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2b7fe700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c15a590 0x7f8c2c15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 -- 172.21.15.29:0/1600125301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c15a590 msgr2=0x7f8c2c15aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c15a590 0x7f8c2c15aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 -- 172.21.15.29:0/1600125301 >> v1:172.21.15.144:6789/0 conn(0x7f8c2c0accf0 legacy=0x7f8c2c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 -- 172.21.15.29:0/1600125301 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2c0ad400 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2b7fe700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c15a590 0x7f8c2c15aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.643+0000 7f8c2affd700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c0cc810 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0099b0 tx=0x7f8c1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fb2db88f015973 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c29ffb700 1 -- 172.21.15.29:0/1600125301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c00b820 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c29ffb700 1 -- 172.21.15.29:0/1600125301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1c00b980 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c29ffb700 1 -- 172.21.15.29:0/1600125301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c00bd20 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 -- 172.21.15.29:0/1600125301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 msgr2=0x7f8c2c0cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c0cc810 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0099b0 tx=0x7f8c1c009250 comp rx=0 tx=0).stop 2023-10-07T09:45:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 -- 172.21.15.29:0/1600125301 shutdown_connections 2023-10-07T09:45:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c15a590 0x7f8c2c15aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 --2- 172.21.15.29:0/1600125301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c0cc810 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 -- 172.21.15.29:0/1600125301 >> 172.21.15.29:0/1600125301 conn(0x7f8c2c067610 msgr2=0x7f8c2c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 -- 172.21.15.29:0/1600125301 shutdown_connections 2023-10-07T09:45:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.644+0000 7f8c319ff700 1 -- 172.21.15.29:0/1600125301 wait complete. 2023-10-07T09:45:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.646+0000 7f8c319ff700 1 Processor -- start 2023-10-07T09:45:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.646+0000 7f8c319ff700 1 -- start start 2023-10-07T09:45:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c319ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c0accf0 0x7f8c2c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c319ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c15d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c319ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c2c15a590 0x7f8c2c0cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c319ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c15db90 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c319ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c15dda0 con 0x7f8c2c15a590 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c319ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c15e080 con 0x7f8c2c0accf0 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c2a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c2a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c15d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46500/0 (socket says 172.21.15.29:46500) 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c2a7fc700 1 -- 172.21.15.29:0/746709192 learned_addr learned my addr 172.21.15.29:0/746709192 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c2affd700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c0accf0 0x7f8c2c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.647+0000 7f8c2b7fe700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c2c15a590 0x7f8c2c0cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c2a7fc700 1 -- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c0accf0 msgr2=0x7f8c2c15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c2a7fc700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c0accf0 0x7f8c2c15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c2a7fc700 1 -- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c2c15a590 msgr2=0x7f8c2c0cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c2a7fc700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c2c15a590 0x7f8c2c0cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c2a7fc700 1 -- 172.21.15.29:0/746709192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c0094f0 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c2a7fc700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c15d650 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000b5e0 tx=0x7f8c200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000f510 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2000f6b0 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2c0c95b0 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000fa70 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.648+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2c0cbf50 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.649+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2c0c9410 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.650+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f8c2001a400 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.651+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f8c20083f40 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.655+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c2c0c9410 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.860+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8c2c0c9410 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.950+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 39) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f8c20058d50 con 0x7f8c2c0afeb0 2023-10-07T09:45:25.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.950+0000 7f8c12ffd700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c1806c9e0 0x7f8c1806edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:25.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.952+0000 7f8c2affd700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c1806c9e0 0x7f8c1806edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:25.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:25.954+0000 7f8c2affd700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c1806c9e0 0x7f8c1806edc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0099b0 tx=0x7f8c1c016000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.789+0000 7f8c12ffd700 1 -- 172.21.15.29:0/746709192 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8c20081230 con 0x7f8c2c0afeb0 2023-10-07T09:45:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c1806c9e0 msgr2=0x7f8c1806edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c1806c9e0 0x7f8c1806edc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0099b0 tx=0x7f8c1c016000 comp rx=0 tx=0).stop 2023-10-07T09:45:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 msgr2=0x7f8c2c15d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c15d650 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000b5e0 tx=0x7f8c200090f0 comp rx=0 tx=0).stop 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 shutdown_connections 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c1806c9e0 0x7f8c1806edc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c2c15a590 0x7f8c2c0cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c2c0accf0 0x7f8c2c15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 --2- 172.21.15.29:0/746709192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c2c0afeb0 0x7f8c2c15d650 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.795+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 >> 172.21.15.29:0/746709192 conn(0x7f8c2c067610 msgr2=0x7f8c2c155980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.796+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 shutdown_connections 2023-10-07T09:45:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:26.796+0000 7f8c319ff700 1 -- 172.21.15.29:0/746709192 wait complete. 2023-10-07T09:45:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:45:26.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:45:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.052+0000 7fa75f1ff700 1 Processor -- start 2023-10-07T09:45:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.052+0000 7fa75f1ff700 1 -- start start 2023-10-07T09:45:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75f1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7580ced00 0x7fa7580b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75f1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580c8a20 0x7fa7580cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75f1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580690f0 con 0x7fa7580b0610 2023-10-07T09:45:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75f1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7580cf2e0 con 0x7fa7580ced00 2023-10-07T09:45:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75f1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580687c0 con 0x7fa7580c8a20 2023-10-07T09:45:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa757fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7580ced00 0x7fa7580b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:27.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa757fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7580ced00 0x7fa7580b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52868/0 (socket says 172.21.15.29:52868) 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa757fff700 1 -- 172.21.15.29:0/3998204003 learned_addr learned my addr 172.21.15.29:0/3998204003 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75d79c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580c8a20 0x7fa7580cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.054+0000 7fa75cf9b700 1 --1- 172.21.15.29:0/3998204003 >> v1:172.21.15.144:6789/0 conn(0x7fa7580b0610 0x7fa7580c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:32824/0 (socket says 172.21.15.29:32824) 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa757fff700 1 -- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580c8a20 msgr2=0x7fa7580cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa757fff700 1 --2- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580c8a20 0x7fa7580cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa757fff700 1 -- 172.21.15.29:0/3998204003 >> v1:172.21.15.144:6789/0 conn(0x7fa7580b0610 legacy=0x7fa7580c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa757fff700 1 -- 172.21.15.29:0/3998204003 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa758068aa0 con 0x7fa7580ced00 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa757fff700 1 --2- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ced00 0x7fa7580b09e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7440099b0 tx=0x7fa744009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d32b4b06b1eb6c79 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa7577fe700 1 -- 172.21.15.29:0/3998204003 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa744009ea0 con 0x7fa7580ced00 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa7577fe700 1 -- 172.21.15.29:0/3998204003 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74400b860 con 0x7fa7580ced00 2023-10-07T09:45:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.055+0000 7fa7577fe700 1 -- 172.21.15.29:0/3998204003 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74400bc00 con 0x7fa7580ced00 2023-10-07T09:45:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 -- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ced00 msgr2=0x7fa7580b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ced00 0x7fa7580b09e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7440099b0 tx=0x7fa744009250 comp rx=0 tx=0).stop 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 -- 172.21.15.29:0/3998204003 shutdown_connections 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580c8a20 0x7fa7580cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/3998204003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580ced00 0x7fa7580b09e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 -- 172.21.15.29:0/3998204003 >> 172.21.15.29:0/3998204003 conn(0x7fa758063a10 msgr2=0x7fa7580ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 -- 172.21.15.29:0/3998204003 shutdown_connections 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.056+0000 7fa75f1ff700 1 -- 172.21.15.29:0/3998204003 wait complete. 2023-10-07T09:45:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.058+0000 7fa75f1ff700 1 Processor -- start 2023-10-07T09:45:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.058+0000 7fa75f1ff700 1 -- start start 2023-10-07T09:45:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75f1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 0x7fa758157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75f1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580c8a20 0x7fa7580c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75f1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580ced00 0x7fa7580c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75f1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75806cef0 con 0x7fa7580b0610 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75f1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580c7130 con 0x7fa7580c8a20 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75f1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580c7410 con 0x7fa7580ced00 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 0x7fa758157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 0x7fa758157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46512/0 (socket says 172.21.15.29:46512) 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 -- 172.21.15.29:0/1763132896 learned_addr learned my addr 172.21.15.29:0/1763132896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa757fff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580c8a20 0x7fa7580c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75d79c700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580ced00 0x7fa7580c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 -- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580ced00 msgr2=0x7fa7580c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580ced00 0x7fa7580c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 -- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580c8a20 msgr2=0x7fa7580c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580c8a20 0x7fa7580c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.059+0000 7fa75cf9b700 1 -- 172.21.15.29:0/1763132896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440094f0 con 0x7fa7580b0610 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.060+0000 7fa75cf9b700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 0x7fa758157130 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00c560 tx=0x7fa74c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.060+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c005a70 con 0x7fa7580b0610 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.060+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c005bd0 con 0x7fa7580b0610 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.060+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c0070d0 con 0x7fa7580b0610 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.060+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7580c76f0 con 0x7fa7580b0610 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.060+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75809a2f0 con 0x7fa7580b0610 2023-10-07T09:45:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.061+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75809a150 con 0x7fa7580b0610 2023-10-07T09:45:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.062+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 39) v1 ==== 84578+0+0 (secure 0 0 0) 0x7fa74c007480 con 0x7fa7580b0610 2023-10-07T09:45:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.063+0000 7fa754ff9700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa73c066d80 0x7fa73c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.063+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fa74c083840 con 0x7fa7580b0610 2023-10-07T09:45:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.063+0000 7fa757fff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa73c066d80 0x7fa73c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.064+0000 7fa757fff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa73c066d80 0x7fa73c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa7580b0f20 tx=0x7fa744017000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.067+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa75809a150 con 0x7fa7580b0610 2023-10-07T09:45:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.271+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa75809a150 con 0x7fa7580b0610 2023-10-07T09:45:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:27.794+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa74c050e40 con 0x7fa7580b0610 2023-10-07T09:45:28.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.155+0000 7fa754ff9700 1 -- 172.21.15.29:0/1763132896 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa74c050a70 con 0x7fa7580b0610 2023-10-07T09:45:28.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa73c066d80 msgr2=0x7fa73c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa73c066d80 0x7fa73c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa7580b0f20 tx=0x7fa744017000 comp rx=0 tx=0).stop 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 msgr2=0x7fa758157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 0x7fa758157130 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00c560 tx=0x7fa74c007f70 comp rx=0 tx=0).stop 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 shutdown_connections 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa73c066d80 0x7fa73c069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7580ced00 0x7fa7580c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7580b0610 0x7fa758157130 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 --2- 172.21.15.29:0/1763132896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7580c8a20 0x7fa7580c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 >> 172.21.15.29:0/1763132896 conn(0x7fa758063a10 msgr2=0x7fa7580c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 shutdown_connections 2023-10-07T09:45:28.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.160+0000 7fa75f1ff700 1 -- 172.21.15.29:0/1763132896 wait complete. 2023-10-07T09:45:28.163 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:45:28.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T09:45:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.384+0000 7f47ff9ff700 1 Processor -- start 2023-10-07T09:45:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.384+0000 7f47ff9ff700 1 -- start start 2023-10-07T09:45:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.385+0000 7f47ff9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f8153cc0 0x7f47f80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.385+0000 7f47ff9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f47f80b0710 0x7f47f80b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.385+0000 7f47ff9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f8154090 con 0x7f47f815ade0 2023-10-07T09:45:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.385+0000 7f47ff9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f8159f60 con 0x7f47f80b0710 2023-10-07T09:45:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.385+0000 7f47ff9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f815a240 con 0x7f47f8153cc0 2023-10-07T09:45:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fcf9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f47f815ade0 0x7f47f80b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33658/0 (socket says 172.21.15.29:33658) 2023-10-07T09:45:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fcf9a700 1 -- 172.21.15.29:0/3312737087 learned_addr learned my addr 172.21.15.29:0/3312737087 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fd79b700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f8153cc0 0x7f47f80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fdf9c700 1 --2- 172.21.15.29:0/3312737087 >> v2:172.21.15.144:3300/0 conn(0x7f47f80b0710 0x7f47f80b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47effff700 1 -- 172.21.15.29:0/3312737087 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172375848 0 0) 0x7f47f8154090 con 0x7f47f815ade0 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47effff700 1 -- 172.21.15.29:0/3312737087 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e4003570 con 0x7f47f815ade0 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fdf9c700 1 -- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f8153cc0 msgr2=0x7f47f80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fdf9c700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f8153cc0 0x7f47f80b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fdf9c700 1 -- 172.21.15.29:0/3312737087 >> v1:172.21.15.29:6789/0 conn(0x7f47f815ade0 legacy=0x7f47f80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fdf9c700 1 -- 172.21.15.29:0/3312737087 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f815a520 con 0x7f47f80b0710 2023-10-07T09:45:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.386+0000 7f47fd79b700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f8153cc0 0x7f47f80b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.387+0000 7f47fdf9c700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f80b0710 0x7f47f80b1d00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009e60 tx=0x7f47f0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=518c350cd1ae5701 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.387+0000 7f47effff700 1 -- 172.21.15.29:0/3312737087 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f000e070 con 0x7f47f80b0710 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.387+0000 7f47effff700 1 -- 172.21.15.29:0/3312737087 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f000bab0 con 0x7f47f80b0710 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.387+0000 7f47effff700 1 -- 172.21.15.29:0/3312737087 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f000be70 con 0x7f47f80b0710 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.387+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f80b0710 msgr2=0x7f47f80b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.387+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f80b0710 0x7f47f80b1d00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009e60 tx=0x7f47f0009780 comp rx=0 tx=0).stop 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.388+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3312737087 shutdown_connections 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.388+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f80b0710 0x7f47f80b1d00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.388+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3312737087 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f8153cc0 0x7f47f80b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.388+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3312737087 >> 172.21.15.29:0/3312737087 conn(0x7f47f8067610 msgr2=0x7f47f8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.388+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3312737087 shutdown_connections 2023-10-07T09:45:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.388+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3312737087 wait complete. 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.389+0000 7f47ff9ff700 1 Processor -- start 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.389+0000 7f47ff9ff700 1 -- start start 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47ff9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 0x7f47f80c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47ff9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f8153cc0 0x7f47f815ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47ff9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f815ade0 0x7f47f81515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47ff9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f815e8c0 con 0x7f47f80b0710 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47ff9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f815dff0 con 0x7f47f8153cc0 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47ff9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8151af0 con 0x7f47f815ade0 2023-10-07T09:45:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47fd79b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 0x7f47f80c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47fd79b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 0x7f47f80c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46524/0 (socket says 172.21.15.29:46524) 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.390+0000 7f47fd79b700 1 -- 172.21.15.29:0/3657711949 learned_addr learned my addr 172.21.15.29:0/3657711949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fdf9c700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f815ade0 0x7f47f81515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fcf9a700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f8153cc0 0x7f47f815ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fd79b700 1 -- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f815ade0 msgr2=0x7f47f81515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fd79b700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f815ade0 0x7f47f81515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fd79b700 1 -- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f8153cc0 msgr2=0x7f47f815ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fd79b700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f8153cc0 0x7f47f815ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fd79b700 1 -- 172.21.15.29:0/3657711949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f40093c0 con 0x7f47f80b0710 2023-10-07T09:45:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fcf9a700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f8153cc0 0x7f47f815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47fd79b700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 0x7f47f80c75b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f47f400ecb0 tx=0x7f47f4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400ee60 con 0x7f47f80b0710 2023-10-07T09:45:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f400cbe0 con 0x7f47f80b0710 2023-10-07T09:45:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f0009a20 con 0x7f47f80b0710 2023-10-07T09:45:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.391+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4005a00 con 0x7f47f80b0710 2023-10-07T09:45:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.392+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f815e270 con 0x7f47f80b0710 2023-10-07T09:45:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.393+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f815eba0 con 0x7f47f80b0710 2023-10-07T09:45:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.393+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f47f4006030 con 0x7f47f80b0710 2023-10-07T09:45:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.394+0000 7f47ed7fa700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47dc066f30 0x7f47dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.394+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f47f4084c60 con 0x7f47f80b0710 2023-10-07T09:45:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.395+0000 7f47fcf9a700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47dc066f30 0x7f47dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.395+0000 7f47fcf9a700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47dc066f30 0x7f47dc0693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f47e8007fd0 tx=0x7f47e8009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:28.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.398+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47f4051d80 con 0x7f47f80b0710 2023-10-07T09:45:28.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.600+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f47f80b0dd0 con 0x7f47f80b0710 2023-10-07T09:45:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.604+0000 7f47ed7fa700 1 -- 172.21.15.29:0/3657711949 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f47f40553a0 con 0x7f47f80b0710 2023-10-07T09:45:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47dc066f30 msgr2=0x7f47dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47dc066f30 0x7f47dc0693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f47e8007fd0 tx=0x7f47e8009040 comp rx=0 tx=0).stop 2023-10-07T09:45:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 msgr2=0x7f47f80c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 0x7f47f80c75b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f47f400ecb0 tx=0x7f47f4009120 comp rx=0 tx=0).stop 2023-10-07T09:45:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 shutdown_connections 2023-10-07T09:45:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47dc066f30 0x7f47dc0693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f47f815ade0 0x7f47f81515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f80b0710 0x7f47f80c75b0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.606+0000 7f47ff9ff700 1 --2- 172.21.15.29:0/3657711949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f8153cc0 0x7f47f815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.607+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 >> 172.21.15.29:0/3657711949 conn(0x7f47f8067610 msgr2=0x7f47f80addc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.607+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 shutdown_connections 2023-10-07T09:45:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.607+0000 7f47ff9ff700 1 -- 172.21.15.29:0/3657711949 wait complete. 2023-10-07T09:45:28.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T09:45:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.822+0000 7f03897ff700 1 Processor -- start 2023-10-07T09:45:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.822+0000 7f03897ff700 1 -- start start 2023-10-07T09:45:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f03897ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0384159620 0x7f03840ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f03897ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840cf570 con 0x7f03840cad90 2023-10-07T09:45:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f03897ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840c7d00 con 0x7f03840cfa00 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f03897ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03840ce960 con 0x7f0384159620 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f0382ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0384159620 0x7f03840ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f0382ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0384159620 0x7f03840ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46532/0 (socket says 172.21.15.29:46532) 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f0382ffd700 1 -- 172.21.15.29:0/2517110864 learned_addr learned my addr 172.21.15.29:0/2517110864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f03827fc700 1 --1- 172.21.15.29:0/2517110864 >> v1:172.21.15.29:6789/0 conn(0x7f03840cad90 0x7f03840c75f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33672/0 (socket says 172.21.15.29:33672) 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f0382ffd700 1 -- 172.21.15.29:0/2517110864 >> v1:172.21.15.144:6789/0 conn(0x7f03840cfa00 legacy=0x7f03840cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f0382ffd700 1 -- 172.21.15.29:0/2517110864 >> v1:172.21.15.29:6789/0 conn(0x7f03840cad90 legacy=0x7f03840c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.824+0000 7f0382ffd700 1 -- 172.21.15.29:0/2517110864 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03840cec40 con 0x7f0384159620 2023-10-07T09:45:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.825+0000 7f0382ffd700 1 --2- 172.21.15.29:0/2517110864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0384159620 0x7f03840ce110 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f03740099a0 tx=0x7f0374009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef9ef9719dc9961 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.825+0000 7f0381ffb700 1 -- 172.21.15.29:0/2517110864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400b820 con 0x7f0384159620 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.825+0000 7f0381ffb700 1 -- 172.21.15.29:0/2517110864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037400b980 con 0x7f0384159620 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.825+0000 7f0381ffb700 1 -- 172.21.15.29:0/2517110864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400bd20 con 0x7f0384159620 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 -- 172.21.15.29:0/2517110864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0384159620 msgr2=0x7f03840ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 --2- 172.21.15.29:0/2517110864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0384159620 0x7f03840ce110 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f03740099a0 tx=0x7f0374009250 comp rx=0 tx=0).stop 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 -- 172.21.15.29:0/2517110864 shutdown_connections 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 --2- 172.21.15.29:0/2517110864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0384159620 0x7f03840ce110 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 -- 172.21.15.29:0/2517110864 >> 172.21.15.29:0/2517110864 conn(0x7f0384089f20 msgr2=0x7f0384099840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 -- 172.21.15.29:0/2517110864 shutdown_connections 2023-10-07T09:45:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.826+0000 7f03897ff700 1 -- 172.21.15.29:0/2517110864 wait complete. 2023-10-07T09:45:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.828+0000 7f03897ff700 1 Processor -- start 2023-10-07T09:45:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.828+0000 7f03897ff700 1 -- start start 2023-10-07T09:45:28.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f03897ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 0x7f0384155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f03897ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03840cfa00 0x7f03840c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f03897ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0384159620 0x7f03840c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f03897ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03840c49b0 con 0x7f03840cad90 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f03897ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03840c4c90 con 0x7f0384159620 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f03897ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03840c4f70 con 0x7f03840cfa00 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.829+0000 7f0382ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 0x7f0384155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 0x7f0384155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46542/0 (socket says 172.21.15.29:46542) 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 -- 172.21.15.29:0/2322889944 learned_addr learned my addr 172.21.15.29:0/2322889944 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f03827fc700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03840cfa00 0x7f03840c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f03837fe700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0384159620 0x7f03840c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 -- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03840cfa00 msgr2=0x7f03840c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03840cfa00 0x7f03840c4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 -- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0384159620 msgr2=0x7f03840c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0384159620 0x7f03840c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 -- 172.21.15.29:0/2322889944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374009490 con 0x7f03840cad90 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f0382ffd700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 0x7f0384155b60 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f03840680e0 tx=0x7f0374016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.830+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374011e60 con 0x7f03840cad90 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.831+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0374020070 con 0x7f03840cad90 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.831+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037401b6a0 con 0x7f03840cad90 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.831+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03840c5250 con 0x7f03840cad90 2023-10-07T09:45:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.831+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03840b3a90 con 0x7f03840cad90 2023-10-07T09:45:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.832+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038404c3c0 con 0x7f03840cad90 2023-10-07T09:45:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.833+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0374022030 con 0x7f03840cad90 2023-10-07T09:45:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.834+0000 7f036affd700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0370066f30 0x7f03700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.834+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f03740865f0 con 0x7f03840cad90 2023-10-07T09:45:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.834+0000 7f03827fc700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0370066f30 0x7f03700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.835+0000 7f03827fc700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0370066f30 0x7f03700693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378007d70 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:28.838+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0374053710 con 0x7f03840cad90 2023-10-07T09:45:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.037+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f038404c3c0 con 0x7f03840cad90 2023-10-07T09:45:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.041+0000 7f036affd700 1 -- 172.21.15.29:0/2322889944 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0374056d30 con 0x7f03840cad90 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.043+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0370066f30 msgr2=0x7f03700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.043+0000 7f03897ff700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0370066f30 0x7f03700693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378007d70 comp rx=0 tx=0).stop 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.043+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 msgr2=0x7f0384155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.043+0000 7f03897ff700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 0x7f0384155b60 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f03840680e0 tx=0x7f0374016000 comp rx=0 tx=0).stop 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 shutdown_connections 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0370066f30 0x7f03700693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0384159620 0x7f03840c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03840cad90 0x7f0384155b60 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 --2- 172.21.15.29:0/2322889944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03840cfa00 0x7f03840c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 >> 172.21.15.29:0/2322889944 conn(0x7f0384089f20 msgr2=0x7f03840c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:29.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 shutdown_connections 2023-10-07T09:45:29.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.044+0000 7f03897ff700 1 -- 172.21.15.29:0/2322889944 wait complete. 2023-10-07T09:45:29.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T09:45:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.261+0000 7fac003ff700 1 Processor -- start 2023-10-07T09:45:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.261+0000 7fac003ff700 1 -- start start 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fac003ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fabf8156180 0x7fabf80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fac003ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80ad4d0 con 0x7fabf80ae570 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fac003ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80ad7b0 con 0x7fabf80b0520 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fac003ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf80ada90 con 0x7fabf8156180 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fabfe19b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fabf8156180 0x7fabf80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fabfe19b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fabf8156180 0x7fabf80afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46550/0 (socket says 172.21.15.29:46550) 2023-10-07T09:45:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fabfe19b700 1 -- 172.21.15.29:0/4031014354 learned_addr learned my addr 172.21.15.29:0/4031014354 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fabfd99a700 1 --1- 172.21.15.29:0/4031014354 >> v1:172.21.15.29:6789/0 conn(0x7fabf80ae570 0x7fabf80accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33688/0 (socket says 172.21.15.29:33688) 2023-10-07T09:45:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.263+0000 7fabfe19b700 1 -- 172.21.15.29:0/4031014354 >> v1:172.21.15.144:6789/0 conn(0x7fabf80b0520 legacy=0x7fabf80b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.264+0000 7fabfe19b700 1 -- 172.21.15.29:0/4031014354 >> v1:172.21.15.29:6789/0 conn(0x7fabf80ae570 legacy=0x7fabf80accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.264+0000 7fabfe19b700 1 -- 172.21.15.29:0/4031014354 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80add70 con 0x7fabf8156180 2023-10-07T09:45:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.264+0000 7fabfe19b700 1 --2- 172.21.15.29:0/4031014354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf8156180 0x7fabf80afc00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fabf40099b0 tx=0x7fabf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f756354c9c3e98d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.264+0000 7fabfd199700 1 -- 172.21.15.29:0/4031014354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf400b820 con 0x7fabf8156180 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.264+0000 7fabfd199700 1 -- 172.21.15.29:0/4031014354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf400b980 con 0x7fabf8156180 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.264+0000 7fabfd199700 1 -- 172.21.15.29:0/4031014354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf400bd20 con 0x7fabf8156180 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 -- 172.21.15.29:0/4031014354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf8156180 msgr2=0x7fabf80afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 --2- 172.21.15.29:0/4031014354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf8156180 0x7fabf80afc00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fabf40099b0 tx=0x7fabf4009250 comp rx=0 tx=0).stop 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 -- 172.21.15.29:0/4031014354 shutdown_connections 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 --2- 172.21.15.29:0/4031014354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf8156180 0x7fabf80afc00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 -- 172.21.15.29:0/4031014354 >> 172.21.15.29:0/4031014354 conn(0x7fabf8067610 msgr2=0x7fabf8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 -- 172.21.15.29:0/4031014354 shutdown_connections 2023-10-07T09:45:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.265+0000 7fac003ff700 1 -- 172.21.15.29:0/4031014354 wait complete. 2023-10-07T09:45:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.266+0000 7fac003ff700 1 Processor -- start 2023-10-07T09:45:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.266+0000 7fac003ff700 1 -- start start 2023-10-07T09:45:29.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.267+0000 7fac003ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 0x7fabf80c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.267+0000 7fac003ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80b0520 0x7fabf80c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.267+0000 7fac003ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8156180 0x7fabf81520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.267+0000 7fac003ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80c7390 con 0x7fabf80ae570 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.267+0000 7fac003ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80c7d60 con 0x7fabf8156180 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.267+0000 7fac003ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80c6780 con 0x7fabf80b0520 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 0x7fabf80c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 0x7fabf80c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46558/0 (socket says 172.21.15.29:46558) 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 -- 172.21.15.29:0/3087515568 learned_addr learned my addr 172.21.15.29:0/3087515568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfd99a700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80b0520 0x7fabf80c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe99c700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8156180 0x7fabf81520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 -- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80b0520 msgr2=0x7fabf80c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80b0520 0x7fabf80c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 -- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8156180 msgr2=0x7fabf81520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8156180 0x7fabf81520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 -- 172.21.15.29:0/3087515568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf40094f0 con 0x7fabf80ae570 2023-10-07T09:45:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfd99a700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80b0520 0x7fabf80c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe99c700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8156180 0x7fabf81520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.268+0000 7fabfe19b700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 0x7fabf80c7820 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c010 tx=0x7fabf4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.269+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf4011e10 con 0x7fabf80ae570 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.269+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4020070 con 0x7fabf80ae570 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.269+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf80c6a60 con 0x7fabf80ae570 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.269+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf401b6a0 con 0x7fabf80ae570 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.269+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf80c7020 con 0x7fabf80ae570 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.270+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf804c3c0 con 0x7fabf80ae570 2023-10-07T09:45:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.271+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fabf4022030 con 0x7fabf80ae570 2023-10-07T09:45:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.271+0000 7fabee7fc700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabdc066f30 0x7fabdc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.271+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7fabf4086600 con 0x7fabf80ae570 2023-10-07T09:45:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.272+0000 7fabfd99a700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabdc066f30 0x7fabdc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.273+0000 7fabfd99a700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabdc066f30 0x7fabdc0693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fabf80a7c00 tx=0x7fabe8009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:29.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.276+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabf4053720 con 0x7fabf80ae570 2023-10-07T09:45:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.483+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fabf80c7670 con 0x7fabf80ae570 2023-10-07T09:45:29.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.487+0000 7fabee7fc700 1 -- 172.21.15.29:0/3087515568 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7fabf4056d40 con 0x7fabf80ae570 2023-10-07T09:45:29.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabdc066f30 msgr2=0x7fabdc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabdc066f30 0x7fabdc0693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fabf80a7c00 tx=0x7fabe8009040 comp rx=0 tx=0).stop 2023-10-07T09:45:29.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 msgr2=0x7fabf80c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 0x7fabf80c7820 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c010 tx=0x7fabf4016000 comp rx=0 tx=0).stop 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 shutdown_connections 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabdc066f30 0x7fabdc0693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8156180 0x7fabf81520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ae570 0x7fabf80c7820 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 --2- 172.21.15.29:0/3087515568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80b0520 0x7fabf80c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 >> 172.21.15.29:0/3087515568 conn(0x7fabf8067610 msgr2=0x7fabf8153000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 shutdown_connections 2023-10-07T09:45:29.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.490+0000 7fac003ff700 1 -- 172.21.15.29:0/3087515568 wait complete. 2023-10-07T09:45:29.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-07T09:45:29.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.703+0000 7f9f63fff700 1 Processor -- start 2023-10-07T09:45:29.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.704+0000 7f9f63fff700 1 -- start start 2023-10-07T09:45:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.705+0000 7f9f63fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f5c153cc0 0x7f9f5c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.705+0000 7f9f63fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 0x7f9f5c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.705+0000 7f9f63fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c154090 con 0x7f9f5c15ade0 2023-10-07T09:45:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.705+0000 7f9f63fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5c159f60 con 0x7f9f5c153cc0 2023-10-07T09:45:29.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.705+0000 7f9f63fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c15a240 con 0x7f9f5c0b0710 2023-10-07T09:45:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f5c153cc0 0x7f9f5c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f6259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 0x7f9f5c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f5c153cc0 0x7f9f5c0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52920/0 (socket says 172.21.15.29:52920) 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f6259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 0x7f9f5c0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:50030/0 (socket says 172.21.15.29:50030) 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 -- 172.21.15.29:0/2171357154 learned_addr learned my addr 172.21.15.29:0/2171357154 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 -- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 msgr2=0x7f9f5c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 --2- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 0x7f9f5c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 -- 172.21.15.29:0/2171357154 >> v1:172.21.15.144:6789/0 conn(0x7f9f5c15ade0 legacy=0x7f9f5c0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f61d9b700 1 -- 172.21.15.29:0/2171357154 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c15a520 con 0x7f9f5c153cc0 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.706+0000 7f9f6259c700 1 --2- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 0x7f9f5c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f61d9b700 1 --2- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c153cc0 0x7f9f5c0b17c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009990 tx=0x7f9f58009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=886d7878744cd3d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f60d99700 1 -- 172.21.15.29:0/2171357154 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58009e80 con 0x7f9f5c153cc0 2023-10-07T09:45:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f60d99700 1 -- 172.21.15.29:0/2171357154 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58005c10 con 0x7f9f5c153cc0 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f60d99700 1 -- 172.21.15.29:0/2171357154 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5800b650 con 0x7f9f5c153cc0 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f63fff700 1 -- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c153cc0 msgr2=0x7f9f5c0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f63fff700 1 --2- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c153cc0 0x7f9f5c0b17c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009990 tx=0x7f9f58009220 comp rx=0 tx=0).stop 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.707+0000 7f9f63fff700 1 -- 172.21.15.29:0/2171357154 shutdown_connections 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.708+0000 7f9f63fff700 1 --2- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c0b0710 0x7f9f5c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.708+0000 7f9f63fff700 1 --2- 172.21.15.29:0/2171357154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c153cc0 0x7f9f5c0b17c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.708+0000 7f9f63fff700 1 -- 172.21.15.29:0/2171357154 >> 172.21.15.29:0/2171357154 conn(0x7f9f5c067610 msgr2=0x7f9f5c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.708+0000 7f9f63fff700 1 -- 172.21.15.29:0/2171357154 shutdown_connections 2023-10-07T09:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.708+0000 7f9f63fff700 1 -- 172.21.15.29:0/2171357154 wait complete. 2023-10-07T09:45:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.709+0000 7f9f63fff700 1 Processor -- start 2023-10-07T09:45:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.709+0000 7f9f63fff700 1 -- start start 2023-10-07T09:45:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f63fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 0x7f9f5c142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f63fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c153cc0 0x7f9f5c155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f63fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c15ade0 0x7f9f5c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f63fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c1575f0 con 0x7f9f5c0b0710 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f63fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c1564c0 con 0x7f9f5c15ade0 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f63fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c1567a0 con 0x7f9f5c153cc0 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f61d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 0x7f9f5c142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f61d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 0x7f9f5c142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46568/0 (socket says 172.21.15.29:46568) 2023-10-07T09:45:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.710+0000 7f9f61d9b700 1 -- 172.21.15.29:0/1097987934 learned_addr learned my addr 172.21.15.29:0/1097987934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f6259c700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c15ade0 0x7f9f5c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f6159a700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c153cc0 0x7f9f5c155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f61d9b700 1 -- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c153cc0 msgr2=0x7f9f5c155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f61d9b700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c153cc0 0x7f9f5c155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f61d9b700 1 -- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c15ade0 msgr2=0x7f9f5c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f61d9b700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c15ade0 0x7f9f5c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f61d9b700 1 -- 172.21.15.29:0/1097987934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54009a20 con 0x7f9f5c0b0710 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f6159a700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c153cc0 0x7f9f5c155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f61d9b700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 0x7f9f5c142e00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800b3c0 tx=0x7f9f58016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58011ae0 con 0x7f9f5c0b0710 2023-10-07T09:45:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58020070 con 0x7f9f5c0b0710 2023-10-07T09:45:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58011e70 con 0x7f9f5c0b0710 2023-10-07T09:45:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.711+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f58009480 con 0x7f9f5c0b0710 2023-10-07T09:45:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.712+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c157280 con 0x7f9f5c0b0710 2023-10-07T09:45:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.713+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c04c3c0 con 0x7f9f5c0b0710 2023-10-07T09:45:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.713+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f58022030 con 0x7f9f5c0b0710 2023-10-07T09:45:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.714+0000 7f9f51ffb700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.714+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f9f580862f0 con 0x7f9f5c0b0710 2023-10-07T09:45:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.715+0000 7f9f6159a700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.715+0000 7f9f6159a700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f40066f30 0x7f9f400693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c06c910 tx=0x7f9f4c009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:29.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.718+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f58053410 con 0x7f9f5c0b0710 2023-10-07T09:45:30.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:29.999+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f9f5c15a800 con 0x7f9f5c0b0710 2023-10-07T09:45:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.811+0000 7f9f51ffb700 1 -- 172.21.15.29:0/1097987934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v606) v1 ==== 166+0+0 (secure 0 0 0) 0x7f9f58056a30 con 0x7f9f5c0b0710 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f40066f30 msgr2=0x7f9f400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f40066f30 0x7f9f400693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c06c910 tx=0x7f9f4c009040 comp rx=0 tx=0).stop 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 msgr2=0x7f9f5c142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 0x7f9f5c142e00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800b3c0 tx=0x7f9f58016000 comp rx=0 tx=0).stop 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 shutdown_connections 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f5c15ade0 0x7f9f5c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c0b0710 0x7f9f5c142e00 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 --2- 172.21.15.29:0/1097987934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f5c153cc0 0x7f9f5c155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.814+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 >> 172.21.15.29:0/1097987934 conn(0x7f9f5c067610 msgr2=0x7f9f5c0add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.815+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 shutdown_connections 2023-10-07T09:45:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:30.815+0000 7f9f63fff700 1 -- 172.21.15.29:0/1097987934 wait complete. 2023-10-07T09:45:30.836 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-07T09:45:30.836 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-07T09:45:30.836 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:45:30.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-07T09:45:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.035+0000 7f452dbff700 1 Processor -- start 2023-10-07T09:45:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.036+0000 7f452dbff700 1 -- start start 2023-10-07T09:45:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f452dbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280c8690 0x7f4528156820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f452dbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4528158dd0 0x7f4528157470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f452dbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45280adc50 con 0x7f45280ad720 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f452dbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45280adf30 con 0x7f4528158dd0 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f452dbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45280ae210 con 0x7f45280c8690 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f4527fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4528158dd0 0x7f4528157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f4527fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4528158dd0 0x7f4528157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46572/0 (socket says 172.21.15.29:46572) 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f4527fff700 1 -- 172.21.15.29:0/682057488 learned_addr learned my addr 172.21.15.29:0/682057488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f4526ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f45280ad720 0x7f4528156d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33696/0 (socket says 172.21.15.29:33696) 2023-10-07T09:45:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.037+0000 7f45277fe700 1 --2- 172.21.15.29:0/682057488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280c8690 0x7f4528156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f4527fff700 1 -- 172.21.15.29:0/682057488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280c8690 msgr2=0x7f4528156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f4527fff700 1 --2- 172.21.15.29:0/682057488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280c8690 0x7f4528156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f4527fff700 1 -- 172.21.15.29:0/682057488 >> v1:172.21.15.29:6789/0 conn(0x7f45280ad720 legacy=0x7f4528156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f4527fff700 1 -- 172.21.15.29:0/682057488 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45280ae4f0 con 0x7f4528158dd0 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f45267fc700 1 -- 172.21.15.29:0/682057488 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644383743 0 0) 0x7f45280adc50 con 0x7f45280ad720 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f4527fff700 1 --2- 172.21.15.29:0/682057488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528158dd0 0x7f4528157470 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f4520009f20 tx=0x7f4520009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2047bf5a110c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f45267fc700 1 -- 172.21.15.29:0/682057488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452000e070 con 0x7f4528158dd0 2023-10-07T09:45:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f45267fc700 1 -- 172.21.15.29:0/682057488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452000b9d0 con 0x7f4528158dd0 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.038+0000 7f45267fc700 1 -- 172.21.15.29:0/682057488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45200092e0 con 0x7f4528158dd0 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 -- 172.21.15.29:0/682057488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528158dd0 msgr2=0x7f4528157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 --2- 172.21.15.29:0/682057488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528158dd0 0x7f4528157470 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f4520009f20 tx=0x7f4520009840 comp rx=0 tx=0).stop 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 -- 172.21.15.29:0/682057488 shutdown_connections 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 --2- 172.21.15.29:0/682057488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528158dd0 0x7f4528157470 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 --2- 172.21.15.29:0/682057488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280c8690 0x7f4528156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 -- 172.21.15.29:0/682057488 >> 172.21.15.29:0/682057488 conn(0x7f45280680e0 msgr2=0x7f45280cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 -- 172.21.15.29:0/682057488 shutdown_connections 2023-10-07T09:45:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.039+0000 7f452dbff700 1 -- 172.21.15.29:0/682057488 wait complete. 2023-10-07T09:45:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.041+0000 7f452dbff700 1 Processor -- start 2023-10-07T09:45:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.041+0000 7f452dbff700 1 -- start start 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f452dbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280ad720 0x7f452814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f452dbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 0x7f452814ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f452dbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4528158dd0 0x7f45280af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f452dbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45280af400 con 0x7f45280c8690 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f452dbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452814a560 con 0x7f4528158dd0 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f452dbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45280cafc0 con 0x7f45280ad720 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 0x7f452814ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 0x7f452814ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46584/0 (socket says 172.21.15.29:46584) 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 -- 172.21.15.29:0/1474328609 learned_addr learned my addr 172.21.15.29:0/1474328609 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f45277fe700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280ad720 0x7f452814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4527fff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4528158dd0 0x7f45280af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 -- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280ad720 msgr2=0x7f452814a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280ad720 0x7f452814a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 -- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4528158dd0 msgr2=0x7f45280af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4528158dd0 0x7f45280af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.042+0000 7f4526ffd700 1 -- 172.21.15.29:0/1474328609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520009ae0 con 0x7f45280c8690 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.043+0000 7f4526ffd700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 0x7f452814ab60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c7e0 tx=0x7f451c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.043+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c005a70 con 0x7f45280c8690 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.043+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c005bd0 con 0x7f45280c8690 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.043+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c0070d0 con 0x7f45280c8690 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.043+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45280cb2a0 con 0x7f45280c8690 2023-10-07T09:45:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.043+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45280af090 con 0x7f45280c8690 2023-10-07T09:45:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.044+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452804c3c0 con 0x7f45280c8690 2023-10-07T09:45:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.045+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f451c007440 con 0x7f45280c8690 2023-10-07T09:45:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.046+0000 7f450f7fe700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4514066f30 0x7f45140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:31.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.046+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f451c083b50 con 0x7f45280c8690 2023-10-07T09:45:31.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.046+0000 7f45277fe700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4514066f30 0x7f45140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:31.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.047+0000 7f45277fe700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4514066f30 0x7f45140693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f45280ca6f0 tx=0x7f451800b430 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.050+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f451c050c70 con 0x7f45280c8690 2023-10-07T09:45:31.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.333+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f45280af540 con 0x7f45280c8690 2023-10-07T09:45:31.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.817+0000 7f450f7fe700 1 -- 172.21.15.29:0/1474328609 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v607) v1 ==== 186+0+0 (secure 0 0 0) 0x7f451c017070 con 0x7f45280c8690 2023-10-07T09:45:31.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.819+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4514066f30 msgr2=0x7f45140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:31.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.819+0000 7f452dbff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4514066f30 0x7f45140693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f45280ca6f0 tx=0x7f451800b430 comp rx=0 tx=0).stop 2023-10-07T09:45:31.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.819+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 msgr2=0x7f452814ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:31.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.819+0000 7f452dbff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 0x7f452814ab60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c7e0 tx=0x7f451c007e10 comp rx=0 tx=0).stop 2023-10-07T09:45:31.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 shutdown_connections 2023-10-07T09:45:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4514066f30 0x7f45140693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4528158dd0 0x7f45280af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45280ad720 0x7f452814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 --2- 172.21.15.29:0/1474328609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45280c8690 0x7f452814ab60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 >> 172.21.15.29:0/1474328609 conn(0x7f45280680e0 msgr2=0x7f452806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:31.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 shutdown_connections 2023-10-07T09:45:31.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:31.820+0000 7f452dbff700 1 -- 172.21.15.29:0/1474328609 wait complete. 2023-10-07T09:45:31.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:45:32.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.038+0000 7fe578bff700 1 Processor -- start 2023-10-07T09:45:32.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.038+0000 7fe578bff700 1 -- start start 2023-10-07T09:45:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.039+0000 7fe578bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159750 0x7fe5740d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.039+0000 7fe578bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe574159b20 0x7fe5740ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.039+0000 7fe578bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5740ce6d0 0x7fe5740ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.039+0000 7fe578bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740cbf20 con 0x7fe574159b20 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.039+0000 7fe578bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740cc200 con 0x7fe5740ce6d0 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.039+0000 7fe578bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740cc4e0 con 0x7fe574159750 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe574159b20 0x7fe5740ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe574159b20 0x7fe5740ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46588/0 (socket says 172.21.15.29:46588) 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 -- 172.21.15.29:0/2638152276 learned_addr learned my addr 172.21.15.29:0/2638152276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe57259c700 1 --2- 172.21.15.29:0/2638152276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159750 0x7fe5740d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe572d9d700 1 --2- 172.21.15.29:0/2638152276 >> v2:172.21.15.144:3300/0 conn(0x7fe5740ce6d0 0x7fe5740ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 -- 172.21.15.29:0/2638152276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159750 msgr2=0x7fe5740d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 --2- 172.21.15.29:0/2638152276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159750 0x7fe5740d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 -- 172.21.15.29:0/2638152276 >> v2:172.21.15.144:3300/0 conn(0x7fe5740ce6d0 msgr2=0x7fe5740ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 --2- 172.21.15.29:0/2638152276 >> v2:172.21.15.144:3300/0 conn(0x7fe5740ce6d0 0x7fe5740ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 -- 172.21.15.29:0/2638152276 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5740cc7c0 con 0x7fe574159b20 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe572d9d700 1 --2- 172.21.15.29:0/2638152276 >> v2:172.21.15.144:3300/0 conn(0x7fe5740ce6d0 0x7fe5740ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.040+0000 7fe571d9b700 1 --2- 172.21.15.29:0/2638152276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574159b20 0x7fe5740ce190 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe5740d0d60 tx=0x7fe568009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f325368653b496b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe57159a700 1 -- 172.21.15.29:0/2638152276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56800b860 con 0x7fe574159b20 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe57159a700 1 -- 172.21.15.29:0/2638152276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56800b9c0 con 0x7fe574159b20 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe57159a700 1 -- 172.21.15.29:0/2638152276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56800bd60 con 0x7fe574159b20 2023-10-07T09:45:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 -- 172.21.15.29:0/2638152276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574159b20 msgr2=0x7fe5740ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 --2- 172.21.15.29:0/2638152276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574159b20 0x7fe5740ce190 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe5740d0d60 tx=0x7fe568009250 comp rx=0 tx=0).stop 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 -- 172.21.15.29:0/2638152276 shutdown_connections 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 --2- 172.21.15.29:0/2638152276 >> v2:172.21.15.144:3300/0 conn(0x7fe5740ce6d0 0x7fe5740ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 --2- 172.21.15.29:0/2638152276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159750 0x7fe5740d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 --2- 172.21.15.29:0/2638152276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574159b20 0x7fe5740ce190 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 -- 172.21.15.29:0/2638152276 >> 172.21.15.29:0/2638152276 conn(0x7fe574063a10 msgr2=0x7fe5740d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.041+0000 7fe578bff700 1 -- 172.21.15.29:0/2638152276 shutdown_connections 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.042+0000 7fe578bff700 1 -- 172.21.15.29:0/2638152276 wait complete. 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.043+0000 7fe578bff700 1 Processor -- start 2023-10-07T09:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.043+0000 7fe578bff700 1 -- start start 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe578bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 0x7fe57414bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe578bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574159750 0x7fe574169be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe578bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159b20 0x7fe5740cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe578bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740b5760 con 0x7fe5740ce6d0 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe578bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57414c170 con 0x7fe574159750 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe578bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57416ab70 con 0x7fe574159b20 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe57259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 0x7fe57414bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe57259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 0x7fe57414bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46602/0 (socket says 172.21.15.29:46602) 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.044+0000 7fe57259c700 1 -- 172.21.15.29:0/1361770569 learned_addr learned my addr 172.21.15.29:0/1361770569 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe571d9b700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574159750 0x7fe574169be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe572d9d700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159b20 0x7fe5740cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe57259c700 1 -- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159b20 msgr2=0x7fe5740cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe57259c700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159b20 0x7fe5740cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe57259c700 1 -- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574159750 msgr2=0x7fe574169be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe57259c700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574159750 0x7fe574169be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe57259c700 1 -- 172.21.15.29:0/1361770569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680094f0 con 0x7fe5740ce6d0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe57259c700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 0x7fe57414bc30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe56400d840 tx=0x7fe564008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400b600 con 0x7fe5740ce6d0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56400b760 con 0x7fe5740ce6d0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400bb20 con 0x7fe5740ce6d0 2023-10-07T09:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57416a5b0 con 0x7fe5740ce6d0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.045+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57416d370 con 0x7fe5740ce6d0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.046+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57404c3c0 con 0x7fe5740ce6d0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.047+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe56401e460 con 0x7fe5740ce6d0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.049+0000 7fe5627fc700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe554066f30 0x7fe5540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.049+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7fe564084ca0 con 0x7fe5740ce6d0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.049+0000 7fe571d9b700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe554066f30 0x7fe5540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.050+0000 7fe571d9b700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe554066f30 0x7fe5540693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe5740d0d60 tx=0x7fe568017000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.052+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe564051e60 con 0x7fe5740ce6d0 2023-10-07T09:45:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.258+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe5740cef80 con 0x7fe5740ce6d0 2023-10-07T09:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.262+0000 7fe5627fc700 1 -- 172.21.15.29:0/1361770569 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe564055480 con 0x7fe5740ce6d0 2023-10-07T09:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe554066f30 msgr2=0x7fe5540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe554066f30 0x7fe5540693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe5740d0d60 tx=0x7fe568017000 comp rx=0 tx=0).stop 2023-10-07T09:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 msgr2=0x7fe57414bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 0x7fe57414bc30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe56400d840 tx=0x7fe564008040 comp rx=0 tx=0).stop 2023-10-07T09:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 shutdown_connections 2023-10-07T09:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe554066f30 0x7fe5540693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574159b20 0x7fe5740cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740ce6d0 0x7fe57414bc30 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 --2- 172.21.15.29:0/1361770569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574159750 0x7fe574169be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 >> 172.21.15.29:0/1361770569 conn(0x7fe574063a10 msgr2=0x7fe57414d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 shutdown_connections 2023-10-07T09:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.265+0000 7fe578bff700 1 -- 172.21.15.29:0/1361770569 wait complete. 2023-10-07T09:45:32.284 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:45:32.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:45:32.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:45:32.515 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:45:32.515 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:45:32.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.713+0000 7f0bd21ff700 1 Processor -- start 2023-10-07T09:45:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.714+0000 7f0bd21ff700 1 -- start start 2023-10-07T09:45:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bd21ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0bcc1533f0 0x7f0bcc156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bd21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 0x7f0bcc0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bd21ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc159670 con 0x7f0bcc0af6d0 2023-10-07T09:45:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bd21ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc15a680 con 0x7f0bcc1533f0 2023-10-07T09:45:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bd21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc0b2070 con 0x7f0bcc1537c0 2023-10-07T09:45:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bcaffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 0x7f0bcc0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bcaffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 0x7f0bcc0afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:50092/0 (socket says 172.21.15.29:50092) 2023-10-07T09:45:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bcaffd700 1 -- 172.21.15.29:0/897073911 learned_addr learned my addr 172.21.15.29:0/897073911 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.716+0000 7f0bcb7fe700 1 --2- 172.21.15.29:0/897073911 >> v2:172.21.15.144:3300/0 conn(0x7f0bcc1533f0 0x7f0bcc156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bcaffd700 1 -- 172.21.15.29:0/897073911 >> v2:172.21.15.144:3300/0 conn(0x7f0bcc1533f0 msgr2=0x7f0bcc156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bcaffd700 1 --2- 172.21.15.29:0/897073911 >> v2:172.21.15.144:3300/0 conn(0x7f0bcc1533f0 0x7f0bcc156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bcaffd700 1 -- 172.21.15.29:0/897073911 >> v1:172.21.15.144:6789/0 conn(0x7f0bcc0af6d0 legacy=0x7f0bcc0adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bcaffd700 1 -- 172.21.15.29:0/897073911 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc15a1f0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bcaffd700 1 --2- 172.21.15.29:0/897073911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 0x7f0bcc0afda0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00099b0 tx=0x7f0bc0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d29cda9025324626 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bca7fc700 1 -- 172.21.15.29:0/897073911 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0009ea0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bca7fc700 1 -- 172.21.15.29:0/897073911 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc000b860 con 0x7f0bcc1537c0 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.717+0000 7f0bca7fc700 1 -- 172.21.15.29:0/897073911 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc000bc20 con 0x7f0bcc1537c0 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 -- 172.21.15.29:0/897073911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 msgr2=0x7f0bcc0afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/897073911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 0x7f0bcc0afda0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00099b0 tx=0x7f0bc0009250 comp rx=0 tx=0).stop 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 -- 172.21.15.29:0/897073911 shutdown_connections 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/897073911 >> v2:172.21.15.144:3300/0 conn(0x7f0bcc1533f0 0x7f0bcc156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/897073911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1537c0 0x7f0bcc0afda0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 -- 172.21.15.29:0/897073911 >> 172.21.15.29:0/897073911 conn(0x7f0bcc0680a0 msgr2=0x7f0bcc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 -- 172.21.15.29:0/897073911 shutdown_connections 2023-10-07T09:45:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.718+0000 7f0bd21ff700 1 -- 172.21.15.29:0/897073911 wait complete. 2023-10-07T09:45:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.719+0000 7f0bd21ff700 1 Processor -- start 2023-10-07T09:45:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 -- start start 2023-10-07T09:45:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bcc0af6d0 0x7f0bcc14ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1533f0 0x7f0bcc15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 0x7f0bcc0c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc15c220 con 0x7f0bcc1537c0 2023-10-07T09:45:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc15b930 con 0x7f0bcc0af6d0 2023-10-07T09:45:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.720+0000 7f0bd21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc14b2d0 con 0x7f0bcc1533f0 2023-10-07T09:45:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 0x7f0bcc0c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 0x7f0bcc0c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46618/0 (socket says 172.21.15.29:46618) 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 -- 172.21.15.29:0/472089344 learned_addr learned my addr 172.21.15.29:0/472089344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcaffd700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1533f0 0x7f0bcc15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcb7fe700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bcc0af6d0 0x7f0bcc14ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 -- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1533f0 msgr2=0x7f0bcc15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1533f0 0x7f0bcc15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 -- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bcc0af6d0 msgr2=0x7f0bcc14ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bcc0af6d0 0x7f0bcc14ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.721+0000 7f0bcbfff700 1 -- 172.21.15.29:0/472089344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00094f0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.722+0000 7f0bcbfff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 0x7f0bcc0c5d00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c840 tx=0x7f0bc4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.722+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4012630 con 0x7f0bcc1537c0 2023-10-07T09:45:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.722+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc40127b0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.722+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc15bbd0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.722+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4012b70 con 0x7f0bcc1537c0 2023-10-07T09:45:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.722+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc160920 con 0x7f0bcc1537c0 2023-10-07T09:45:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.723+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc0b1e20 con 0x7f0bcc1537c0 2023-10-07T09:45:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.724+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0bc401a3a0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.725+0000 7f0bb37fe700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0bb8066f30 0x7f0bb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.725+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f0bc40847b0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.725+0000 7f0bcb7fe700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0bb8066f30 0x7f0bb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.726+0000 7f0bcb7fe700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0bb8066f30 0x7f0bb80693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc15d840 tx=0x7f0bbc0091a0 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.729+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0bc40518d0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.976+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bcc04c3c0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.977+0000 7f0bb37fe700 1 -- 172.21.15.29:0/472089344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f0bc4054ef0 con 0x7f0bcc1537c0 2023-10-07T09:45:32.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:32.978 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3851765971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3851765971},{"type":"v1","addr":"172.21.15.29:6834","nonce":3851765971}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6502,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2196864328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2196864328},{"type":"v1","addr":"172.21.15.144:6835","nonce":2196864328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2373398063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":2373398063},{"type":"v1","addr":"172.21.15.144:6837","nonce":2373398063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":7089,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/996342990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":996342990},{"type":"v1","addr":"172.21.15.29:6835","nonce":996342990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[]} 2023-10-07T09:45:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0bb8066f30 msgr2=0x7f0bb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0bb8066f30 0x7f0bb80693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc15d840 tx=0x7f0bbc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 msgr2=0x7f0bcc0c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 0x7f0bcc0c5d00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c840 tx=0x7f0bc4001670 comp rx=0 tx=0).stop 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 shutdown_connections 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0bb8066f30 0x7f0bb80693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc1537c0 0x7f0bcc0c5d00 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bcc0af6d0 0x7f0bcc14ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 --2- 172.21.15.29:0/472089344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bcc1533f0 0x7f0bcc15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 >> 172.21.15.29:0/472089344 conn(0x7f0bcc0680a0 msgr2=0x7f0bcc153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 shutdown_connections 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:32.979+0000 7f0bd21ff700 1 -- 172.21.15.29:0/472089344 wait complete. 2023-10-07T09:45:32.981 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 44 2023-10-07T09:45:32.998 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:45:32.998 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:45:32.998 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:45:32.998 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:45:32.998 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:45:32.998 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:45:32.999 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:45:32.999 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:45:32.999 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:45:32.999 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:45:32.999 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:45:32.999 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:45:32.999 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:45:33.000 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:45:33.000 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:45:33.000 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:45:33.010 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:45:33.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:45:33.012 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:45:33.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:45:33.015 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:45:33.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:45:33.016 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:45:33.016 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:45:33.056 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:45:33.056 DEBUG:teuthology.parallel:result is None 2023-10-07T09:45:33.059 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:45:33.060 DEBUG:teuthology.parallel:result is None 2023-10-07T09:45:33.060 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:45:33.060 DEBUG:teuthology.parallel:result is None 2023-10-07T09:45:33.063 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:45:33.063 DEBUG:teuthology.parallel:result is None 2023-10-07T09:45:33.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:45:33.203 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:45:33.206 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:45:33.206 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:45:33.207 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:45:33.208 INFO:tasks.ceph.mds.b.smithi144.stderr:2023-10-07T09:45:33.208+0000 7fa6b99fd700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-07T09:45:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.271+0000 7f301ddff700 1 Processor -- start 2023-10-07T09:45:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.272+0000 7f301ddff700 1 -- start start 2023-10-07T09:45:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f301ddff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f301815ade0 0x7f30180b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f301ddff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f30180b0540 0x7f301815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f301ddff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3018154090 con 0x7f3018153cc0 2023-10-07T09:45:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f301ddff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30180b09e0 con 0x7f301815ade0 2023-10-07T09:45:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f301ddff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3018159f60 con 0x7f30180b0540 2023-10-07T09:45:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f3016ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f301815ade0 0x7f30180b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f3016ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f301815ade0 0x7f30180b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59096/0 (socket says 172.21.15.29:59096) 2023-10-07T09:45:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.273+0000 7f3016ffd700 1 -- 172.21.15.29:0/1868656248 learned_addr learned my addr 172.21.15.29:0/1868656248 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3017fff700 1 --2- 172.21.15.29:0/1868656248 >> v2:172.21.15.144:3300/0 conn(0x7f30180b0540 0x7f301815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3016ffd700 1 -- 172.21.15.29:0/1868656248 >> v2:172.21.15.144:3300/0 conn(0x7f30180b0540 msgr2=0x7f301815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3016ffd700 1 --2- 172.21.15.29:0/1868656248 >> v2:172.21.15.144:3300/0 conn(0x7f30180b0540 0x7f301815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3016ffd700 1 -- 172.21.15.29:0/1868656248 >> v1:172.21.15.144:6789/0 conn(0x7f3018153cc0 legacy=0x7f30180b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3016ffd700 1 -- 172.21.15.29:0/1868656248 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301815a240 con 0x7f301815ade0 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3017fff700 1 --2- 172.21.15.29:0/1868656248 >> v2:172.21.15.144:3300/0 conn(0x7f30180b0540 0x7f301815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f3016ffd700 1 --2- 172.21.15.29:0/1868656248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301815ade0 0x7f30180b0000 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f301806c400 tx=0x7f300c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2697a4a74117b828 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f30167fc700 1 -- 172.21.15.29:0/1868656248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c00b820 con 0x7f301815ade0 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f30167fc700 1 -- 172.21.15.29:0/1868656248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c00b980 con 0x7f301815ade0 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.274+0000 7f30167fc700 1 -- 172.21.15.29:0/1868656248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c00bd20 con 0x7f301815ade0 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 -- 172.21.15.29:0/1868656248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301815ade0 msgr2=0x7f30180b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 --2- 172.21.15.29:0/1868656248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301815ade0 0x7f30180b0000 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f301806c400 tx=0x7f300c009250 comp rx=0 tx=0).stop 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 -- 172.21.15.29:0/1868656248 shutdown_connections 2023-10-07T09:45:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 --2- 172.21.15.29:0/1868656248 >> v2:172.21.15.144:3300/0 conn(0x7f30180b0540 0x7f301815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 --2- 172.21.15.29:0/1868656248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301815ade0 0x7f30180b0000 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 -- 172.21.15.29:0/1868656248 >> 172.21.15.29:0/1868656248 conn(0x7f3018067610 msgr2=0x7f3018067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 -- 172.21.15.29:0/1868656248 shutdown_connections 2023-10-07T09:45:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.275+0000 7f301ddff700 1 -- 172.21.15.29:0/1868656248 wait complete. 2023-10-07T09:45:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.276+0000 7f301ddff700 1 Processor -- start 2023-10-07T09:45:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.277+0000 7f301ddff700 1 -- start start 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f301ddff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30180b0540 0x7f301813e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f301ddff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 0x7f30180c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f301ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f301815ade0 0x7f30181585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f301ddff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3018158110 con 0x7f3018153cc0 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f301ddff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3018157500 con 0x7f30180b0540 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f301ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30181577e0 con 0x7f301815ade0 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 0x7f30180c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 0x7f30180c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59108/0 (socket says 172.21.15.29:59108) 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 -- 172.21.15.29:0/1783681173 learned_addr learned my addr 172.21.15.29:0/1783681173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f30177fe700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30180b0540 0x7f301813e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3017fff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f301815ade0 0x7f30181585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 -- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f301815ade0 msgr2=0x7f30181585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f301815ade0 0x7f30181585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 -- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30180b0540 msgr2=0x7f301813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30180b0540 0x7f301813e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.278+0000 7f3016ffd700 1 -- 172.21.15.29:0/1783681173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c0094f0 con 0x7f3018153cc0 2023-10-07T09:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.279+0000 7f3016ffd700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 0x7f30180c75b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f300c0030c0 tx=0x7f300c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.279+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c011d00 con 0x7f3018153cc0 2023-10-07T09:45:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.279+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c020070 con 0x7f3018153cc0 2023-10-07T09:45:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.279+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c01b6a0 con 0x7f3018153cc0 2023-10-07T09:45:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.279+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3018157ac0 con 0x7f3018153cc0 2023-10-07T09:45:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.279+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301813eef0 con 0x7f3018153cc0 2023-10-07T09:45:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.280+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301804c3c0 con 0x7f3018153cc0 2023-10-07T09:45:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.281+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f300c022030 con 0x7f3018153cc0 2023-10-07T09:45:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.282+0000 7f2fff7fe700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3004066f30 0x7f30040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.282+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f300c0868d0 con 0x7f3018153cc0 2023-10-07T09:45:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.283+0000 7f30177fe700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3004066f30 0x7f30040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.283+0000 7f30177fe700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3004066f30 0x7f30040693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3008003a70 tx=0x7f3008007ef0 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.286+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f300c053a70 con 0x7f3018153cc0 2023-10-07T09:45:33.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.509+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f301804c3c0 con 0x7f3018153cc0 2023-10-07T09:45:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.834+0000 7f2fff7fe700 1 -- 172.21.15.29:0/1783681173 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v97) v1 ==== 97+0+0 (secure 0 0 0) 0x7f300c057090 con 0x7f3018153cc0 2023-10-07T09:45:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3004066f30 msgr2=0x7f30040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3004066f30 0x7f30040693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3008003a70 tx=0x7f3008007ef0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 msgr2=0x7f30180c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 0x7f30180c75b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f300c0030c0 tx=0x7f300c016000 comp rx=0 tx=0).stop 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 shutdown_connections 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3004066f30 0x7f30040693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f301815ade0 0x7f30181585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30180b0540 0x7f301813e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 --2- 172.21.15.29:0/1783681173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3018153cc0 0x7f30180c75b0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 >> 172.21.15.29:0/1783681173 conn(0x7f3018067610 msgr2=0x7f3018152db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.837+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 shutdown_connections 2023-10-07T09:45:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:33.838+0000 7f301ddff700 1 -- 172.21.15.29:0/1783681173 wait complete. 2023-10-07T09:45:33.840 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:45:33.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:45:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.056+0000 7f42d19ff700 1 Processor -- start 2023-10-07T09:45:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.056+0000 7f42d19ff700 1 -- start start 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.057+0000 7f42d19ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f42cc157290 0x7f42cc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.057+0000 7f42d19ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42cc0b1a10 0x7f42cc157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.057+0000 7f42d19ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc158ad0 con 0x7f42cc0ae8b0 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.057+0000 7f42d19ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc15b140 con 0x7f42cc0b1a10 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.057+0000 7f42d19ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc15b420 con 0x7f42cc157290 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42cc0b1a10 0x7f42cc157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42cc0b1a10 0x7f42cc157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59120/0 (socket says 172.21.15.29:59120) 2023-10-07T09:45:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42ca7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42cc0ae8b0 0x7f42cc0b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39242/0 (socket says 172.21.15.29:39242) 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 -- 172.21.15.29:0/263583429 learned_addr learned my addr 172.21.15.29:0/263583429 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42caffd700 1 --2- 172.21.15.29:0/263583429 >> v2:172.21.15.144:3300/0 conn(0x7f42cc157290 0x7f42cc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 -- 172.21.15.29:0/263583429 >> v2:172.21.15.144:3300/0 conn(0x7f42cc157290 msgr2=0x7f42cc0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 --2- 172.21.15.29:0/263583429 >> v2:172.21.15.144:3300/0 conn(0x7f42cc157290 0x7f42cc0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 -- 172.21.15.29:0/263583429 >> v1:172.21.15.29:6789/0 conn(0x7f42cc0ae8b0 legacy=0x7f42cc0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 -- 172.21.15.29:0/263583429 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc15b700 con 0x7f42cc0b1a10 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.058+0000 7f42cb7fe700 1 --2- 172.21.15.29:0/263583429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc0b1a10 0x7f42cc157660 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0685f0 tx=0x7f42c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5687d780868c6c00 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42c9ffb700 1 -- 172.21.15.29:0/263583429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c400e070 con 0x7f42cc0b1a10 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42c9ffb700 1 -- 172.21.15.29:0/263583429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c400ba50 con 0x7f42cc0b1a10 2023-10-07T09:45:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42c9ffb700 1 -- 172.21.15.29:0/263583429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c40092e0 con 0x7f42cc0b1a10 2023-10-07T09:45:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 -- 172.21.15.29:0/263583429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc0b1a10 msgr2=0x7f42cc157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 --2- 172.21.15.29:0/263583429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc0b1a10 0x7f42cc157660 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0685f0 tx=0x7f42c4009840 comp rx=0 tx=0).stop 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 -- 172.21.15.29:0/263583429 shutdown_connections 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 --2- 172.21.15.29:0/263583429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc0b1a10 0x7f42cc157660 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 --2- 172.21.15.29:0/263583429 >> v2:172.21.15.144:3300/0 conn(0x7f42cc157290 0x7f42cc0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 -- 172.21.15.29:0/263583429 >> 172.21.15.29:0/263583429 conn(0x7f42cc0680a0 msgr2=0x7f42cc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.059+0000 7f42d19ff700 1 -- 172.21.15.29:0/263583429 shutdown_connections 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.060+0000 7f42d19ff700 1 -- 172.21.15.29:0/263583429 wait complete. 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.061+0000 7f42d19ff700 1 Processor -- start 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.061+0000 7f42d19ff700 1 -- start start 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42d19ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42cc0b1a10 0x7f42cc14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42d19ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42cc157290 0x7f42cc149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42d19ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 0x7f42cc14a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42d19ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc14dd70 con 0x7f42cc14bbe0 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42d19ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc15cea0 con 0x7f42cc0b1a10 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42d19ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc14d720 con 0x7f42cc157290 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 0x7f42cc14a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 0x7f42cc14a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59122/0 (socket says 172.21.15.29:59122) 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.062+0000 7f42cb7fe700 1 -- 172.21.15.29:0/2087241317 learned_addr learned my addr 172.21.15.29:0/2087241317 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42ca7fc700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42cc157290 0x7f42cc149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42caffd700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42cc0b1a10 0x7f42cc14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42cb7fe700 1 -- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42cc157290 msgr2=0x7f42cc149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42cb7fe700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42cc157290 0x7f42cc149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42cb7fe700 1 -- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42cc0b1a10 msgr2=0x7f42cc14e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42cb7fe700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42cc0b1a10 0x7f42cc14e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42cb7fe700 1 -- 172.21.15.29:0/2087241317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c4009ae0 con 0x7f42cc14bbe0 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42ca7fc700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42cc157290 0x7f42cc149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42caffd700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42cc0b1a10 0x7f42cc14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42cb7fe700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 0x7f42cc14a400 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f42c4012000 tx=0x7f42c4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c40094e0 con 0x7f42cc14bbe0 2023-10-07T09:45:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c4008e80 con 0x7f42cc14bbe0 2023-10-07T09:45:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c40266b0 con 0x7f42cc14bbe0 2023-10-07T09:45:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc14da00 con 0x7f42cc14bbe0 2023-10-07T09:45:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.063+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc14e740 con 0x7f42cc14bbe0 2023-10-07T09:45:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.064+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc14dee0 con 0x7f42cc14bbe0 2023-10-07T09:45:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.065+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f42c4003040 con 0x7f42cc14bbe0 2023-10-07T09:45:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.066+0000 7f42b2ffd700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f42b8066f30 0x7f42b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.066+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f42c4086340 con 0x7f42cc14bbe0 2023-10-07T09:45:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.067+0000 7f42caffd700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f42b8066f30 0x7f42b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.068+0000 7f42caffd700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f42b8066f30 0x7f42b80693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009990 tx=0x7f42bc008040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.070+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f42c40536f0 con 0x7f42cc14bbe0 2023-10-07T09:45:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.354+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f42cc14d520 con 0x7f42cc14bbe0 2023-10-07T09:45:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.354+0000 7f42b2ffd700 1 -- 172.21.15.29:0/2087241317 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f42c4009940 con 0x7f42cc14bbe0 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:45:34.356 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:45:34.357 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.359 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.361 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.362 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:45:34.363 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:45:34.366 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:45:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:45:34.369 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:45:34.371 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:45:34.372 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:45:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f42b8066f30 msgr2=0x7f42b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f42b8066f30 0x7f42b80693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009990 tx=0x7f42bc008040 comp rx=0 tx=0).stop 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 msgr2=0x7f42cc14a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 0x7f42cc14a400 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f42c4012000 tx=0x7f42c4018000 comp rx=0 tx=0).stop 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 shutdown_connections 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f42b8066f30 0x7f42b80693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42cc14bbe0 0x7f42cc14a400 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42cc0b1a10 0x7f42cc14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 --2- 172.21.15.29:0/2087241317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42cc157290 0x7f42cc149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 >> 172.21.15.29:0/2087241317 conn(0x7f42cc0680a0 msgr2=0x7f42cc1520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 shutdown_connections 2023-10-07T09:45:34.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.357+0000 7f42d19ff700 1 -- 172.21.15.29:0/2087241317 wait complete. 2023-10-07T09:45:34.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.570+0000 7fcc6adff700 1 Processor -- start 2023-10-07T09:45:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.570+0000 7fcc6adff700 1 -- start start 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc6adff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc640ad370 0x7fcc640cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc6adff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc64159ec0 0x7fcc6415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc6adff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6414d790 con 0x7fcc641568c0 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc6adff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640ae5d0 con 0x7fcc64159ec0 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc6adff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6415a850 con 0x7fcc640ad370 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc63fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc64159ec0 0x7fcc6415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc63fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc64159ec0 0x7fcc6415a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59132/0 (socket says 172.21.15.29:59132) 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.571+0000 7fcc63fff700 1 -- 172.21.15.29:0/134104723 learned_addr learned my addr 172.21.15.29:0/134104723 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc68b9b700 1 --2- 172.21.15.29:0/134104723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc640ad370 0x7fcc640cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc63fff700 1 -- 172.21.15.29:0/134104723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc640ad370 msgr2=0x7fcc640cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc63fff700 1 --2- 172.21.15.29:0/134104723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc640ad370 0x7fcc640cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc63fff700 1 -- 172.21.15.29:0/134104723 >> v1:172.21.15.144:6789/0 conn(0x7fcc641568c0 legacy=0x7fcc64156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc63fff700 1 -- 172.21.15.29:0/134104723 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640ad740 con 0x7fcc64159ec0 2023-10-07T09:45:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc63fff700 1 --2- 172.21.15.29:0/134104723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc64159ec0 0x7fcc6415a310 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fcc500099b0 tx=0x7fcc50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0b85662c98c1f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc637fe700 1 -- 172.21.15.29:0/134104723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000b860 con 0x7fcc64159ec0 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc637fe700 1 -- 172.21.15.29:0/134104723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5000b9c0 con 0x7fcc64159ec0 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.572+0000 7fcc637fe700 1 -- 172.21.15.29:0/134104723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000bd60 con 0x7fcc64159ec0 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 -- 172.21.15.29:0/134104723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc64159ec0 msgr2=0x7fcc6415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 --2- 172.21.15.29:0/134104723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc64159ec0 0x7fcc6415a310 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fcc500099b0 tx=0x7fcc50009250 comp rx=0 tx=0).stop 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 -- 172.21.15.29:0/134104723 shutdown_connections 2023-10-07T09:45:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 --2- 172.21.15.29:0/134104723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc640ad370 0x7fcc640cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 --2- 172.21.15.29:0/134104723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc64159ec0 0x7fcc6415a310 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 -- 172.21.15.29:0/134104723 >> 172.21.15.29:0/134104723 conn(0x7fcc640680a0 msgr2=0x7fcc640cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 -- 172.21.15.29:0/134104723 shutdown_connections 2023-10-07T09:45:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.573+0000 7fcc6adff700 1 -- 172.21.15.29:0/134104723 wait complete. 2023-10-07T09:45:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.574+0000 7fcc6adff700 1 Processor -- start 2023-10-07T09:45:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.575+0000 7fcc6adff700 1 -- start start 2023-10-07T09:45:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6adff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc640ad370 0x7fcc640c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6adff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 0x7fcc640cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6adff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc64159ec0 0x7fcc6414f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6adff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64169580 con 0x7fcc641568c0 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6adff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640cbce0 con 0x7fcc640ad370 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6adff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640cae20 con 0x7fcc64159ec0 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 0x7fcc640cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 0x7fcc640cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59148/0 (socket says 172.21.15.29:59148) 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 -- 172.21.15.29:0/3596359430 learned_addr learned my addr 172.21.15.29:0/3596359430 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc68b9b700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc640ad370 0x7fcc640c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc6939c700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc64159ec0 0x7fcc6414f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 -- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc64159ec0 msgr2=0x7fcc6414f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc64159ec0 0x7fcc6414f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 -- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc640ad370 msgr2=0x7fcc640c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc640ad370 0x7fcc640c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.576+0000 7fcc63fff700 1 -- 172.21.15.29:0/3596359430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500094f0 con 0x7fcc641568c0 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.577+0000 7fcc63fff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 0x7fcc640cb7a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009290 tx=0x7fcc50019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.577+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50019db0 con 0x7fcc641568c0 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.577+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50010470 con 0x7fcc641568c0 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.577+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc64169800 con 0x7fcc641568c0 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.577+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50010830 con 0x7fcc641568c0 2023-10-07T09:45:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.577+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc640ca8e0 con 0x7fcc641568c0 2023-10-07T09:45:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.578+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc64169990 con 0x7fcc641568c0 2023-10-07T09:45:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.579+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc5002c430 con 0x7fcc641568c0 2023-10-07T09:45:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.580+0000 7fcc60ff9700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc48066f30 0x7fcc480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.581+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fcc50014070 con 0x7fcc641568c0 2023-10-07T09:45:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.581+0000 7fcc68b9b700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc48066f30 0x7fcc480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.581+0000 7fcc68b9b700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc48066f30 0x7fcc480693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcc58006fd0 tx=0x7fcc58008040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.585+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc500531c0 con 0x7fcc641568c0 2023-10-07T09:45:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.786+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc6404c3c0 con 0x7fcc641568c0 2023-10-07T09:45:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.787+0000 7fcc60ff9700 1 -- 172.21.15.29:0/3596359430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+12802 (secure 0 0 0) 0x7fcc500567e0 con 0x7fcc641568c0 2023-10-07T09:45:34.788 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:34.788 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":99,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:33.844675+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc48066f30 msgr2=0x7fcc480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc48066f30 0x7fcc480693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcc58006fd0 tx=0x7fcc58008040 comp rx=0 tx=0).stop 2023-10-07T09:45:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 msgr2=0x7fcc640cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 0x7fcc640cb7a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009290 tx=0x7fcc50019000 comp rx=0 tx=0).stop 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 shutdown_connections 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc48066f30 0x7fcc480693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc64159ec0 0x7fcc6414f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc640ad370 0x7fcc640c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 --2- 172.21.15.29:0/3596359430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc641568c0 0x7fcc640cb7a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 >> 172.21.15.29:0/3596359430 conn(0x7fcc640680a0 msgr2=0x7fcc64155f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 shutdown_connections 2023-10-07T09:45:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:34.789+0000 7fcc6adff700 1 -- 172.21.15.29:0/3596359430 wait complete. 2023-10-07T09:45:34.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:45:34.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:45:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.006+0000 7f2c76fff700 1 Processor -- start 2023-10-07T09:45:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.006+0000 7f2c76fff700 1 -- start start 2023-10-07T09:45:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.007+0000 7f2c76fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c70153cc0 0x7f2c700b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.007+0000 7f2c76fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c7015ade0 0x7f2c700b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c76fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c70154090 con 0x7f2c700b0000 2023-10-07T09:45:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c76fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c70159f60 con 0x7f2c7015ade0 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c76fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7015a240 con 0x7f2c70153cc0 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c7015ade0 0x7f2c700b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c74d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c70153cc0 0x7f2c700b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c7015ade0 0x7f2c700b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59156/0 (socket says 172.21.15.29:59156) 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 -- 172.21.15.29:0/2417272143 learned_addr learned my addr 172.21.15.29:0/2417272143 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 -- 172.21.15.29:0/2417272143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c70153cc0 msgr2=0x7f2c700b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 --2- 172.21.15.29:0/2417272143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c70153cc0 0x7f2c700b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 -- 172.21.15.29:0/2417272143 >> v1:172.21.15.144:6789/0 conn(0x7f2c700b0000 legacy=0x7f2c7015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.008+0000 7f2c6ffff700 1 -- 172.21.15.29:0/2417272143 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7015a520 con 0x7f2c7015ade0 2023-10-07T09:45:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c6ffff700 1 --2- 172.21.15.29:0/2417272143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7015ade0 0x7f2c700b1d00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0099b0 tx=0x7f2c5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28781f1f2ae30dd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c6f7fe700 1 -- 172.21.15.29:0/2417272143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c009ea0 con 0x7f2c7015ade0 2023-10-07T09:45:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c6f7fe700 1 -- 172.21.15.29:0/2417272143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c00b860 con 0x7f2c7015ade0 2023-10-07T09:45:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c6f7fe700 1 -- 172.21.15.29:0/2417272143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00bc00 con 0x7f2c7015ade0 2023-10-07T09:45:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c76fff700 1 -- 172.21.15.29:0/2417272143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7015ade0 msgr2=0x7f2c700b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c76fff700 1 --2- 172.21.15.29:0/2417272143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7015ade0 0x7f2c700b1d00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0099b0 tx=0x7f2c5c009250 comp rx=0 tx=0).stop 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c76fff700 1 -- 172.21.15.29:0/2417272143 shutdown_connections 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c76fff700 1 --2- 172.21.15.29:0/2417272143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c70153cc0 0x7f2c700b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c76fff700 1 --2- 172.21.15.29:0/2417272143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7015ade0 0x7f2c700b1d00 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.009+0000 7f2c76fff700 1 -- 172.21.15.29:0/2417272143 >> 172.21.15.29:0/2417272143 conn(0x7f2c70067610 msgr2=0x7f2c70067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.010+0000 7f2c76fff700 1 -- 172.21.15.29:0/2417272143 shutdown_connections 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.010+0000 7f2c76fff700 1 -- 172.21.15.29:0/2417272143 wait complete. 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.011+0000 7f2c76fff700 1 Processor -- start 2023-10-07T09:45:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.011+0000 7f2c76fff700 1 -- start start 2023-10-07T09:45:35.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c76fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c700b0000 0x7f2c70156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c76fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 0x7f2c700ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c76fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7015ade0 0x7f2c70169770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c76fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70156d60 con 0x7f2c70153cc0 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c76fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70157040 con 0x7f2c700b0000 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c76fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70169ff0 con 0x7f2c7015ade0 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c6ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 0x7f2c700ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c6ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 0x7f2c700ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59170/0 (socket says 172.21.15.29:59170) 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.012+0000 7f2c6ffff700 1 -- 172.21.15.29:0/333624857 learned_addr learned my addr 172.21.15.29:0/333624857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c7559c700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7015ade0 0x7f2c70169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6ffff700 1 -- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7015ade0 msgr2=0x7f2c70169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c74d9b700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c700b0000 0x7f2c70156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6ffff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7015ade0 0x7f2c70169770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6ffff700 1 -- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c700b0000 msgr2=0x7f2c70156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6ffff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c700b0000 0x7f2c70156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6ffff700 1 -- 172.21.15.29:0/333624857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c0094f0 con 0x7f2c70153cc0 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c7559c700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7015ade0 0x7f2c70169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c74d9b700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c700b0000 0x7f2c70156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6ffff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 0x7f2c700ce440 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c006fd0 tx=0x7f2c5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00bbe0 con 0x7f2c70153cc0 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c011420 con 0x7f2c70153cc0 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7016a2d0 con 0x7f2c70153cc0 2023-10-07T09:45:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.013+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0117c0 con 0x7f2c70153cc0 2023-10-07T09:45:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.014+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c70169cb0 con 0x7f2c70153cc0 2023-10-07T09:45:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.015+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c701571d0 con 0x7f2c70153cc0 2023-10-07T09:45:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.015+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2c5c011bf0 con 0x7f2c70153cc0 2023-10-07T09:45:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.016+0000 7f2c6cff9700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c54066f30 0x7f2c540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.016+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f2c5c085bb0 con 0x7f2c70153cc0 2023-10-07T09:45:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.017+0000 7f2c74d9b700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c54066f30 0x7f2c540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.017+0000 7f2c74d9b700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c54066f30 0x7f2c540693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c64006ea0 tx=0x7f2c64008040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.020+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c5c053000 con 0x7f2c70153cc0 2023-10-07T09:45:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.223+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c7004c3c0 con 0x7f2c70153cc0 2023-10-07T09:45:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.223+0000 7f2c6cff9700 1 -- 172.21.15.29:0/333624857 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c5c056620 con 0x7f2c70153cc0 2023-10-07T09:45:35.225 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:45:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c54066f30 msgr2=0x7f2c540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c54066f30 0x7f2c540693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c64006ea0 tx=0x7f2c64008040 comp rx=0 tx=0).stop 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 msgr2=0x7f2c700ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 0x7f2c700ce440 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c006fd0 tx=0x7f2c5c017000 comp rx=0 tx=0).stop 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 shutdown_connections 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c54066f30 0x7f2c540693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7015ade0 0x7f2c70169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c700b0000 0x7f2c70156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 --2- 172.21.15.29:0/333624857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c70153cc0 0x7f2c700ce440 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 >> 172.21.15.29:0/333624857 conn(0x7f2c70067610 msgr2=0x7f2c7015b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 shutdown_connections 2023-10-07T09:45:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.226+0000 7f2c76fff700 1 -- 172.21.15.29:0/333624857 wait complete. 2023-10-07T09:45:35.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:45:35.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.438+0000 7f9f213ff700 1 Processor -- start 2023-10-07T09:45:35.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.439+0000 7f9f213ff700 1 -- start start 2023-10-07T09:45:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f213ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c1533f0 0x7f9f1c156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f213ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f1c1537c0 0x7f9f1c0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f213ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c159670 con 0x7f9f1c0af6d0 2023-10-07T09:45:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f213ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c15a680 con 0x7f9f1c1537c0 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f213ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0b2070 con 0x7f9f1c1533f0 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f1a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f1c1537c0 0x7f9f1c0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f1a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f1c1537c0 0x7f9f1c0afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59174/0 (socket says 172.21.15.29:59174) 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/3400947418 learned_addr learned my addr 172.21.15.29:0/3400947418 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f1affd700 1 --2- 172.21.15.29:0/3400947418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c1533f0 0x7f9f1c156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.440+0000 7f9f1b7fe700 1 --1- 172.21.15.29:0/3400947418 >> v1:172.21.15.29:6789/0 conn(0x7f9f1c0af6d0 0x7f9f1c0adc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39244/0 (socket says 172.21.15.29:39244) 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/3400947418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c1533f0 msgr2=0x7f9f1c156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f1a7fc700 1 --2- 172.21.15.29:0/3400947418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c1533f0 0x7f9f1c156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/3400947418 >> v1:172.21.15.29:6789/0 conn(0x7f9f1c0af6d0 legacy=0x7f9f1c0adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/3400947418 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c15a1f0 con 0x7f9f1c1537c0 2023-10-07T09:45:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f19ffb700 1 -- 172.21.15.29:0/3400947418 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725932075 0 0) 0x7f9f1c159670 con 0x7f9f1c0af6d0 2023-10-07T09:45:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f1a7fc700 1 --2- 172.21.15.29:0/3400947418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c0afda0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9f100099b0 tx=0x7f9f10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b749011156af760 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f19ffb700 1 -- 172.21.15.29:0/3400947418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000b860 con 0x7f9f1c1537c0 2023-10-07T09:45:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f19ffb700 1 -- 172.21.15.29:0/3400947418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1000b9c0 con 0x7f9f1c1537c0 2023-10-07T09:45:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.441+0000 7f9f19ffb700 1 -- 172.21.15.29:0/3400947418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000bd80 con 0x7f9f1c1537c0 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 -- 172.21.15.29:0/3400947418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1537c0 msgr2=0x7f9f1c0afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 --2- 172.21.15.29:0/3400947418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c0afda0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9f100099b0 tx=0x7f9f10009250 comp rx=0 tx=0).stop 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 -- 172.21.15.29:0/3400947418 shutdown_connections 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 --2- 172.21.15.29:0/3400947418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c1533f0 0x7f9f1c156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 --2- 172.21.15.29:0/3400947418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c0afda0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 -- 172.21.15.29:0/3400947418 >> 172.21.15.29:0/3400947418 conn(0x7f9f1c0680a0 msgr2=0x7f9f1c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 -- 172.21.15.29:0/3400947418 shutdown_connections 2023-10-07T09:45:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.442+0000 7f9f213ff700 1 -- 172.21.15.29:0/3400947418 wait complete. 2023-10-07T09:45:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.443+0000 7f9f213ff700 1 Processor -- start 2023-10-07T09:45:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.444+0000 7f9f213ff700 1 -- start start 2023-10-07T09:45:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f213ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c0af6d0 0x7f9f1c1515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f213ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 0x7f9f1c0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f213ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c14ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f213ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0c67e0 con 0x7f9f1c1533f0 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f213ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0c6ac0 con 0x7f9f1c1537c0 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f213ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0c6da0 con 0x7f9f1c0af6d0 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 0x7f9f1c0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 0x7f9f1c0c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59178/0 (socket says 172.21.15.29:59178) 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/1942020319 learned_addr learned my addr 172.21.15.29:0/1942020319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1affd700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c0af6d0 0x7f9f1c1515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1b7fe700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c14ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c0af6d0 msgr2=0x7f9f1c1515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c0af6d0 0x7f9f1c1515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c1537c0 msgr2=0x7f9f1c14ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c14ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.445+0000 7f9f1a7fc700 1 -- 172.21.15.29:0/1942020319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f100094f0 con 0x7f9f1c1533f0 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.446+0000 7f9f1a7fc700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 0x7f9f1c0c7880 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9f10006fd0 tx=0x7f9f10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.446+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10020070 con 0x7f9f1c1533f0 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.446+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10011d70 con 0x7f9f1c1533f0 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.446+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1001a650 con 0x7f9f1c1533f0 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.446+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c0c7080 con 0x7f9f1c1533f0 2023-10-07T09:45:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.446+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c14b220 con 0x7f9f1c1533f0 2023-10-07T09:45:35.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.447+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c0c75a0 con 0x7f9f1c1533f0 2023-10-07T09:45:35.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.448+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f1001aa80 con 0x7f9f1c1533f0 2023-10-07T09:45:35.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.449+0000 7f9f02ffd700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f08066f30 0x7f9f080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.449+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f9f10084e40 con 0x7f9f1c1533f0 2023-10-07T09:45:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.450+0000 7f9f1affd700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f08066f30 0x7f9f080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.450+0000 7f9f1affd700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f08066f30 0x7f9f080693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00beb0 tx=0x7f9f0c00f040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.453+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f1c0c75a0 con 0x7f9f1c1533f0 2023-10-07T09:45:35.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.659+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9f1c0c75a0 con 0x7f9f1c1533f0 2023-10-07T09:45:35.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.659+0000 7f9f02ffd700 1 -- 172.21.15.29:0/1942020319 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9f1c0c75a0 con 0x7f9f1c1533f0 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:35.661 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:45:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f08066f30 msgr2=0x7f9f080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f08066f30 0x7f9f080693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00beb0 tx=0x7f9f0c00f040 comp rx=0 tx=0).stop 2023-10-07T09:45:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 msgr2=0x7f9f1c0c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 0x7f9f1c0c7880 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9f10006fd0 tx=0x7f9f10017000 comp rx=0 tx=0).stop 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 shutdown_connections 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f08066f30 0x7f9f080693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c1537c0 0x7f9f1c14ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f1c0af6d0 0x7f9f1c1515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 --2- 172.21.15.29:0/1942020319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c1533f0 0x7f9f1c0c7880 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 >> 172.21.15.29:0/1942020319 conn(0x7f9f1c0680a0 msgr2=0x7f9f1c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 shutdown_connections 2023-10-07T09:45:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.662+0000 7f9f213ff700 1 -- 172.21.15.29:0/1942020319 wait complete. 2023-10-07T09:45:35.680 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:45:35.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:45:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.879+0000 7f24cf59e700 1 Processor -- start 2023-10-07T09:45:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.880+0000 7f24cf59e700 1 -- start start 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24cf59e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24d00b2a10 0x7f24d00b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24cf59e700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f24d00c6610 0x7f24d00c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24cf59e700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d00c6180 con 0x7f24d00c6fc0 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24cf59e700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d00b3930 con 0x7f24d00b2a10 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24cf59e700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d0159d60 con 0x7f24d00c6610 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24ce59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24d00b2a10 0x7f24d00b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24ce59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24d00b2a10 0x7f24d00b33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59186/0 (socket says 172.21.15.29:59186) 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24ce59c700 1 -- 172.21.15.29:0/966735985 learned_addr learned my addr 172.21.15.29:0/966735985 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.881+0000 7f24cdd9b700 1 --2- 172.21.15.29:0/966735985 >> v2:172.21.15.144:3300/0 conn(0x7f24d00c6610 0x7f24d00c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24ce59c700 1 -- 172.21.15.29:0/966735985 >> v2:172.21.15.144:3300/0 conn(0x7f24d00c6610 msgr2=0x7f24d00c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24ce59c700 1 --2- 172.21.15.29:0/966735985 >> v2:172.21.15.144:3300/0 conn(0x7f24d00c6610 0x7f24d00c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24ce59c700 1 -- 172.21.15.29:0/966735985 >> v1:172.21.15.144:6789/0 conn(0x7f24d00c6fc0 legacy=0x7f24d00ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24ce59c700 1 -- 172.21.15.29:0/966735985 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d0064040 con 0x7f24d00b2a10 2023-10-07T09:45:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24ce59c700 1 --2- 172.21.15.29:0/966735985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d00b33f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f24d00884d0 tx=0x7f24c0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c15d27fc908c4a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24cd59a700 1 -- 172.21.15.29:0/966735985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0005c10 con 0x7f24d00b2a10 2023-10-07T09:45:35.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24cd59a700 1 -- 172.21.15.29:0/966735985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c0005d70 con 0x7f24d00b2a10 2023-10-07T09:45:35.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.882+0000 7f24cd59a700 1 -- 172.21.15.29:0/966735985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000b840 con 0x7f24d00b2a10 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 -- 172.21.15.29:0/966735985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 msgr2=0x7f24d00b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 --2- 172.21.15.29:0/966735985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d00b33f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f24d00884d0 tx=0x7f24c0009220 comp rx=0 tx=0).stop 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 -- 172.21.15.29:0/966735985 shutdown_connections 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 --2- 172.21.15.29:0/966735985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d00b33f0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 --2- 172.21.15.29:0/966735985 >> v2:172.21.15.144:3300/0 conn(0x7f24d00c6610 0x7f24d00c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 -- 172.21.15.29:0/966735985 >> 172.21.15.29:0/966735985 conn(0x7f24d0063a10 msgr2=0x7f24d00d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 -- 172.21.15.29:0/966735985 shutdown_connections 2023-10-07T09:45:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.883+0000 7f24cf59e700 1 -- 172.21.15.29:0/966735985 wait complete. 2023-10-07T09:45:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.885+0000 7f24cf59e700 1 Processor -- start 2023-10-07T09:45:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.885+0000 7f24cf59e700 1 -- start start 2023-10-07T09:45:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.886+0000 7f24cf59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.886+0000 7f24cf59e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24d00c6610 0x7f24d00c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.886+0000 7f24cf59e700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24d00c6fc0 0x7f24d0069640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.886+0000 7f24cf59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d016a230 con 0x7f24d00b2a10 2023-10-07T09:45:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.886+0000 7f24cf59e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d016dd60 con 0x7f24d00c6610 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.886+0000 7f24cf59e700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d016e040 con 0x7f24d00c6fc0 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d0159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59200/0 (socket says 172.21.15.29:59200) 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 -- 172.21.15.29:0/3533536601 learned_addr learned my addr 172.21.15.29:0/3533536601 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24cdd9b700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24d00c6610 0x7f24d00c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ced9d700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24d00c6fc0 0x7f24d0069640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 -- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24d00c6fc0 msgr2=0x7f24d0069640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24d00c6fc0 0x7f24d0069640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 -- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24d00c6610 msgr2=0x7f24d00c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24d00c6610 0x7f24d00c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 -- 172.21.15.29:0/3533536601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c0009480 con 0x7f24d00b2a10 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ced9d700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24d00c6fc0 0x7f24d0069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.887+0000 7f24ce59c700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d0159f30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f24c00030c0 tx=0x7f24c0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.888+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0011ac0 con 0x7f24d00b2a10 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.888+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c0011c40 con 0x7f24d00b2a10 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.888+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d016e320 con 0x7f24d00b2a10 2023-10-07T09:45:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.888+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c001d430 con 0x7f24d00b2a10 2023-10-07T09:45:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.888+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d015a470 con 0x7f24d00b2a10 2023-10-07T09:45:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.889+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d004c3c0 con 0x7f24d00b2a10 2023-10-07T09:45:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.890+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f24c001d820 con 0x7f24d00b2a10 2023-10-07T09:45:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.891+0000 7f24be7fc700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24b0066f30 0x7f24b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.891+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f24c00866e0 con 0x7f24d00b2a10 2023-10-07T09:45:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.891+0000 7f24cdd9b700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24b0066f30 0x7f24b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.892+0000 7f24cdd9b700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24b0066f30 0x7f24b00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f24d009e300 tx=0x7f24c4008040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:35.895+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24c0053a90 con 0x7f24d00b2a10 2023-10-07T09:45:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.098+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f24d00c5570 con 0x7f24d00b2a10 2023-10-07T09:45:36.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.850+0000 7f24be7fc700 1 -- 172.21.15.29:0/3533536601 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) v1 ==== 138+0+0 (secure 0 0 0) 0x7f24c00570b0 con 0x7f24d00b2a10 2023-10-07T09:45:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24b0066f30 msgr2=0x7f24b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24b0066f30 0x7f24b00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f24d009e300 tx=0x7f24c4008040 comp rx=0 tx=0).stop 2023-10-07T09:45:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 msgr2=0x7f24d0159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d0159f30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f24c00030c0 tx=0x7f24c0015000 comp rx=0 tx=0).stop 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 shutdown_connections 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f24b0066f30 0x7f24b00693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f24d00c6fc0 0x7f24d0069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24d00b2a10 0x7f24d0159f30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 --2- 172.21.15.29:0/3533536601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24d00c6610 0x7f24d00c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.855+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 >> 172.21.15.29:0/3533536601 conn(0x7f24d0063a10 msgr2=0x7f24d0157b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.856+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 shutdown_connections 2023-10-07T09:45:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:36.856+0000 7f24cf59e700 1 -- 172.21.15.29:0/3533536601 wait complete. 2023-10-07T09:45:36.859 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:45:36.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:45:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.087+0000 7fa4a13ff700 1 Processor -- start 2023-10-07T09:45:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.087+0000 7fa4a13ff700 1 -- start start 2023-10-07T09:45:37.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa4a13ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa49c0cc570 0x7fa49c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:37.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa4a13ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c15adf0 con 0x7fa49c156540 2023-10-07T09:45:37.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa4a13ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0ad7c0 con 0x7fa49c156170 2023-10-07T09:45:37.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa4a13ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49c0b0000 con 0x7fa49c0cc570 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa49a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa49c0cc570 0x7fa49c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa49b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa49c156540 0x7fa49c15a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39252/0 (socket says 172.21.15.29:39252) 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa49b7fe700 1 -- 172.21.15.29:0/3745753457 learned_addr learned my addr 172.21.15.29:0/3745753457 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa49a7fc700 1 -- 172.21.15.29:0/3745753457 >> v1:172.21.15.144:6789/0 conn(0x7fa49c156170 legacy=0x7fa49c0cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa49a7fc700 1 -- 172.21.15.29:0/3745753457 >> v1:172.21.15.29:6789/0 conn(0x7fa49c156540 legacy=0x7fa49c15a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa499ffb700 1 -- 172.21.15.29:0/3745753457 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527986358 0 0) 0x7fa49c0ad7c0 con 0x7fa49c156170 2023-10-07T09:45:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.089+0000 7fa49a7fc700 1 -- 172.21.15.29:0/3745753457 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0ace90 con 0x7fa49c0cc570 2023-10-07T09:45:37.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.090+0000 7fa49a7fc700 1 --2- 172.21.15.29:0/3745753457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a0c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa4900098e0 tx=0x7fa4900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc4ae712021a338 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:37.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.090+0000 7fa499ffb700 1 -- 172.21.15.29:0/3745753457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490009e80 con 0x7fa49c0cc570 2023-10-07T09:45:37.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.090+0000 7fa499ffb700 1 -- 172.21.15.29:0/3745753457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49000b760 con 0x7fa49c0cc570 2023-10-07T09:45:37.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.090+0000 7fa499ffb700 1 -- 172.21.15.29:0/3745753457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49000bae0 con 0x7fa49c0cc570 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 -- 172.21.15.29:0/3745753457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 msgr2=0x7fa49c15a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/3745753457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a0c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa4900098e0 tx=0x7fa4900091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 -- 172.21.15.29:0/3745753457 shutdown_connections 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/3745753457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a0c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 -- 172.21.15.29:0/3745753457 >> 172.21.15.29:0/3745753457 conn(0x7fa49c067610 msgr2=0x7fa49c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 -- 172.21.15.29:0/3745753457 shutdown_connections 2023-10-07T09:45:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.091+0000 7fa4a13ff700 1 -- 172.21.15.29:0/3745753457 wait complete. 2023-10-07T09:45:37.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.092+0000 7fa4a13ff700 1 Processor -- start 2023-10-07T09:45:37.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.092+0000 7fa4a13ff700 1 -- start start 2023-10-07T09:45:37.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.093+0000 7fa4a13ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:37.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.093+0000 7fa4a13ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa49c156170 0x7fa49c169830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:37.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.093+0000 7fa4a13ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa49c156540 0x7fa49c158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:37.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.093+0000 7fa4a13ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c157b00 con 0x7fa49c0cc570 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.093+0000 7fa4a13ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c157de0 con 0x7fa49c156540 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.093+0000 7fa4a13ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c1580c0 con 0x7fa49c156170 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59212/0 (socket says 172.21.15.29:59212) 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 -- 172.21.15.29:0/1395777158 learned_addr learned my addr 172.21.15.29:0/1395777158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49b7fe700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa49c156540 0x7fa49c158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49a7fc700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa49c156170 0x7fa49c169830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 -- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa49c156170 msgr2=0x7fa49c169830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa49c156170 0x7fa49c169830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 -- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa49c156540 msgr2=0x7fa49c158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa49c156540 0x7fa49c158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 -- 172.21.15.29:0/1395777158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa490009420 con 0x7fa49c0cc570 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49b7fe700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa49c156540 0x7fa49c158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.094+0000 7fa49affd700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a3a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00c5e0 tx=0x7fa48c007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.095+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c011590 con 0x7fa49c0cc570 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.095+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c0116f0 con 0x7fa49c0cc570 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.095+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c1583a0 con 0x7fa49c0cc570 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.095+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c011ab0 con 0x7fa49c0cc570 2023-10-07T09:45:37.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.095+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c15a8e0 con 0x7fa49c0cc570 2023-10-07T09:45:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.096+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49c15aad0 con 0x7fa49c0cc570 2023-10-07T09:45:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.097+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa48c01b3b0 con 0x7fa49c0cc570 2023-10-07T09:45:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.098+0000 7fa482ffd700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa488066f30 0x7fa4880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.098+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5599+0+0 (secure 0 0 0) 0x7fa48c083f30 con 0x7fa49c0cc570 2023-10-07T09:45:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.098+0000 7fa49a7fc700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa488066f30 0x7fa4880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:37.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.099+0000 7fa49a7fc700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa488066f30 0x7fa4880693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0cc970 tx=0x7fa490016000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:37.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.102+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa48c0511f0 con 0x7fa49c0cc570 2023-10-07T09:45:37.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.304+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa49c0718b0 con 0x7fa49c0cc570 2023-10-07T09:45:37.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.862+0000 7fa482ffd700 1 -- 172.21.15.29:0/1395777158 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa48c017020 con 0x7fa49c0cc570 2023-10-07T09:45:37.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.868+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa488066f30 msgr2=0x7fa4880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.868+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa488066f30 0x7fa4880693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0cc970 tx=0x7fa490016000 comp rx=0 tx=0).stop 2023-10-07T09:45:37.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.868+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 msgr2=0x7fa49c15a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:37.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.868+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a3a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00c5e0 tx=0x7fa48c007e60 comp rx=0 tx=0).stop 2023-10-07T09:45:37.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.876+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 shutdown_connections 2023-10-07T09:45:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.876+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa488066f30 0x7fa4880693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.876+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa49c156540 0x7fa49c158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.876+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa49c0cc570 0x7fa49c15a3a0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.876+0000 7fa4a13ff700 1 --2- 172.21.15.29:0/1395777158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa49c156170 0x7fa49c169830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.876+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 >> 172.21.15.29:0/1395777158 conn(0x7fa49c067610 msgr2=0x7fa49c150810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.882+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 shutdown_connections 2023-10-07T09:45:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:37.883+0000 7fa4a13ff700 1 -- 172.21.15.29:0/1395777158 wait complete. 2023-10-07T09:45:37.888 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:45:37.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:45:38.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.109+0000 7fd5255ff700 1 Processor -- start 2023-10-07T09:45:38.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.110+0000 7fd5255ff700 1 -- start start 2023-10-07T09:45:38.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd5255ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd520153cc0 0x7fd5200b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd5255ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd5255ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5200b0000 0x7fd5200b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd5255ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5200b0980 con 0x7fd5200b0000 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd5255ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd520154090 con 0x7fd520153cc0 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd5255ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd520159f60 con 0x7fd52015ade0 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd51f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5200b0000 0x7fd5200b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.111+0000 7fd51f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5200b0000 0x7fd5200b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59216/0 (socket says 172.21.15.29:59216) 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 -- 172.21.15.29:0/2169619221 learned_addr learned my addr 172.21.15.29:0/2169619221 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51e7fc700 1 --2- 172.21.15.29:0/2169619221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51effd700 1 --2- 172.21.15.29:0/2169619221 >> v2:172.21.15.144:3300/0 conn(0x7fd520153cc0 0x7fd5200b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 -- 172.21.15.29:0/2169619221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 msgr2=0x7fd5200b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:38.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 --2- 172.21.15.29:0/2169619221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 -- 172.21.15.29:0/2169619221 >> v2:172.21.15.144:3300/0 conn(0x7fd520153cc0 msgr2=0x7fd5200b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 --2- 172.21.15.29:0/2169619221 >> v2:172.21.15.144:3300/0 conn(0x7fd520153cc0 0x7fd5200b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 -- 172.21.15.29:0/2169619221 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52015a240 con 0x7fd5200b0000 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51f7fe700 1 --2- 172.21.15.29:0/2169619221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5200b0000 0x7fd5200b0440 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd5200cf440 tx=0x7fd518009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc357a2044f7c3bc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51dffb700 1 -- 172.21.15.29:0/2169619221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51800e070 con 0x7fd5200b0000 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51dffb700 1 -- 172.21.15.29:0/2169619221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51800b970 con 0x7fd5200b0000 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.112+0000 7fd51dffb700 1 -- 172.21.15.29:0/2169619221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180092e0 con 0x7fd5200b0000 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 -- 172.21.15.29:0/2169619221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5200b0000 msgr2=0x7fd5200b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 --2- 172.21.15.29:0/2169619221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5200b0000 0x7fd5200b0440 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd5200cf440 tx=0x7fd518009840 comp rx=0 tx=0).stop 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 -- 172.21.15.29:0/2169619221 shutdown_connections 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 --2- 172.21.15.29:0/2169619221 >> v2:172.21.15.144:3300/0 conn(0x7fd520153cc0 0x7fd5200b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 --2- 172.21.15.29:0/2169619221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5200b0000 0x7fd5200b0440 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 --2- 172.21.15.29:0/2169619221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 -- 172.21.15.29:0/2169619221 >> 172.21.15.29:0/2169619221 conn(0x7fd520067610 msgr2=0x7fd520067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 -- 172.21.15.29:0/2169619221 shutdown_connections 2023-10-07T09:45:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.113+0000 7fd5255ff700 1 -- 172.21.15.29:0/2169619221 wait complete. 2023-10-07T09:45:38.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.115+0000 7fd5255ff700 1 Processor -- start 2023-10-07T09:45:38.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.115+0000 7fd5255ff700 1 -- start start 2023-10-07T09:45:38.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd5255ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5200b0000 0x7fd520153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd5255ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 0x7fd520155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd5255ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd5255ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201535b0 con 0x7fd520153cc0 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd5255ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd520155280 con 0x7fd5200b0000 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd5255ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd520157360 con 0x7fd52015ade0 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 0x7fd520155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 0x7fd520155710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59222/0 (socket says 172.21.15.29:59222) 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 -- 172.21.15.29:0/4178503872 learned_addr learned my addr 172.21.15.29:0/4178503872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51f7fe700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51effd700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5200b0000 0x7fd520153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 -- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 msgr2=0x7fd5200c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 -- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5200b0000 msgr2=0x7fd520153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5200b0000 0x7fd520153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51e7fc700 1 -- 172.21.15.29:0/4178503872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd518009ae0 con 0x7fd520153cc0 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.116+0000 7fd51f7fe700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.117+0000 7fd51e7fc700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 0x7fd520155710 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e930 tx=0x7fd5140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.117+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd514013680 con 0x7fd520153cc0 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.117+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5140137e0 con 0x7fd520153cc0 2023-10-07T09:45:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.117+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd520154670 con 0x7fd520153cc0 2023-10-07T09:45:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.117+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd514013b80 con 0x7fd520153cc0 2023-10-07T09:45:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.117+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd520154c30 con 0x7fd520153cc0 2023-10-07T09:45:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.118+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd520155410 con 0x7fd520153cc0 2023-10-07T09:45:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.119+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd51400c460 con 0x7fd520153cc0 2023-10-07T09:45:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.120+0000 7fd506ffd700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd50c066f30 0x7fd50c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.120+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5981+0+0 (secure 0 0 0) 0x7fd514084850 con 0x7fd520153cc0 2023-10-07T09:45:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.121+0000 7fd51effd700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd50c066f30 0x7fd50c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.121+0000 7fd51effd700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd50c066f30 0x7fd50c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd510009da0 tx=0x7fd510009120 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:38.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.125+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd514051910 con 0x7fd520153cc0 2023-10-07T09:45:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.376+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd52004c3c0 con 0x7fd520153cc0 2023-10-07T09:45:38.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.865+0000 7fd506ffd700 1 -- 172.21.15.29:0/4178503872 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd514054f30 con 0x7fd520153cc0 2023-10-07T09:45:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.872+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd50c066f30 msgr2=0x7fd50c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.872+0000 7fd5255ff700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd50c066f30 0x7fd50c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd510009da0 tx=0x7fd510009120 comp rx=0 tx=0).stop 2023-10-07T09:45:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.872+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 msgr2=0x7fd520155710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.872+0000 7fd5255ff700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 0x7fd520155710 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e930 tx=0x7fd5140091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.873+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 shutdown_connections 2023-10-07T09:45:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.873+0000 7fd5255ff700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd50c066f30 0x7fd50c0693e0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd510009da0 tx=0x7fd510009120 comp rx=0 tx=0).stop 2023-10-07T09:45:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.873+0000 7fd5255ff700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd52015ade0 0x7fd5200c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.873+0000 7fd5255ff700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5200b0000 0x7fd520153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.873+0000 7fd5255ff700 1 --2- 172.21.15.29:0/4178503872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd520153cc0 0x7fd520155710 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.873+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 >> 172.21.15.29:0/4178503872 conn(0x7fd520067610 msgr2=0x7fd5200add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.884+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 shutdown_connections 2023-10-07T09:45:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:38.886+0000 7fd5255ff700 1 -- 172.21.15.29:0/4178503872 wait complete. 2023-10-07T09:45:38.889 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 12 and data pool 13 2023-10-07T09:45:38.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.110+0000 7f3ede5ff700 1 Processor -- start 2023-10-07T09:45:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.110+0000 7f3ede5ff700 1 -- start start 2023-10-07T09:45:39.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.111+0000 7f3ede5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed80b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.111+0000 7f3ede5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ed80b0540 0x7f3ed815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.111+0000 7f3ede5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed80b0980 con 0x7f3ed8153cc0 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.111+0000 7f3ede5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed8154090 con 0x7f3ed80b0540 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.111+0000 7f3ede5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed8159f60 con 0x7f3ed815ade0 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.111+0000 7f3edcb9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ed80b0540 0x7f3ed815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ed80b0540 0x7f3ed815c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59238/0 (socket says 172.21.15.29:59238) 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 -- 172.21.15.29:0/3046588132 learned_addr learned my addr 172.21.15.29:0/3046588132 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3ed77fe700 1 --2- 172.21.15.29:0/3046588132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed80b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 -- 172.21.15.29:0/3046588132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 msgr2=0x7f3ed80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 --2- 172.21.15.29:0/3046588132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed80b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 -- 172.21.15.29:0/3046588132 >> v1:172.21.15.144:6789/0 conn(0x7f3ed8153cc0 legacy=0x7f3ed80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 -- 172.21.15.29:0/3046588132 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed815a240 con 0x7f3ed80b0540 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3ed6ffd700 1 -- 172.21.15.29:0/3046588132 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094818253 0 0) 0x7f3ed80b0980 con 0x7f3ed8153cc0 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3edcb9c700 1 --2- 172.21.15.29:0/3046588132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed815c240 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009f20 tx=0x7f3ed0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6495b049acb5b27 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.112+0000 7f3ed6ffd700 1 -- 172.21.15.29:0/3046588132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000e070 con 0x7f3ed80b0540 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ed6ffd700 1 -- 172.21.15.29:0/3046588132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed000b9d0 con 0x7f3ed80b0540 2023-10-07T09:45:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ed6ffd700 1 -- 172.21.15.29:0/3046588132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00092e0 con 0x7f3ed80b0540 2023-10-07T09:45:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3046588132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 msgr2=0x7f3ed815c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3046588132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed815c240 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009f20 tx=0x7f3ed0009840 comp rx=0 tx=0).stop 2023-10-07T09:45:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3046588132 shutdown_connections 2023-10-07T09:45:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3046588132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed815c240 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3046588132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed80b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3046588132 >> 172.21.15.29:0/3046588132 conn(0x7f3ed8067610 msgr2=0x7f3ed8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3046588132 shutdown_connections 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.113+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3046588132 wait complete. 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.115+0000 7f3ede5ff700 1 Processor -- start 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.115+0000 7f3ede5ff700 1 -- start start 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ede5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed8149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ede5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ed8153cc0 0x7f3ed81504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ede5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed81509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ede5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed816ce90 con 0x7f3ed80b0540 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ede5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed816da10 con 0x7f3ed8153cc0 2023-10-07T09:45:39.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ede5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81698f0 con 0x7f3ed815ade0 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ed7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed8149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ed7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed8149b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59248/0 (socket says 172.21.15.29:59248) 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ed7fff700 1 -- 172.21.15.29:0/3359123504 learned_addr learned my addr 172.21.15.29:0/3359123504 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3ed77fe700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ed8153cc0 0x7f3ed81504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.116+0000 7f3edcb9c700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed81509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ed7fff700 1 -- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 msgr2=0x7f3ed81509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ed7fff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed81509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ed7fff700 1 -- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ed8153cc0 msgr2=0x7f3ed81504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ed7fff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ed8153cc0 0x7f3ed81504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ed7fff700 1 -- 172.21.15.29:0/3359123504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0009ae0 con 0x7f3ed80b0540 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ed7fff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed8149b40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800c840 tx=0x7f3ec8007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011590 con 0x7f3ed80b0540 2023-10-07T09:45:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8011710 con 0x7f3ed80b0540 2023-10-07T09:45:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011ad0 con 0x7f3ed80b0540 2023-10-07T09:45:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed816d0e0 con 0x7f3ed80b0540 2023-10-07T09:45:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.117+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed816d6a0 con 0x7f3ed80b0540 2023-10-07T09:45:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.118+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed804c3c0 con 0x7f3ed80b0540 2023-10-07T09:45:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.119+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3ec801b3b0 con 0x7f3ed80b0540 2023-10-07T09:45:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.120+0000 7f3ebffff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ec4066f30 0x7f3ec40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.120+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3ec8083b80 con 0x7f3ed80b0540 2023-10-07T09:45:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.121+0000 7f3ed77fe700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ec4066f30 0x7f3ec40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.121+0000 7f3ed77fe700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ec4066f30 0x7f3ec40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ed815cfb0 tx=0x7f3ecc0091a0 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.124+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ec8050c80 con 0x7f3ed80b0540 2023-10-07T09:45:39.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.323+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ed815a520 con 0x7f3ed80b0540 2023-10-07T09:45:39.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.324+0000 7f3ebffff700 1 -- 172.21.15.29:0/3359123504 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16044 (secure 0 0 0) 0x7f3ec8017070 con 0x7f3ed80b0540 2023-10-07T09:45:39.324 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:39.324 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":102,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:38.376892+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:45:36.139381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:37.345605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:39.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ec4066f30 msgr2=0x7f3ec40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ec4066f30 0x7f3ec40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ed815cfb0 tx=0x7f3ecc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 msgr2=0x7f3ed8149b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed8149b40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800c840 tx=0x7f3ec8007dd0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 shutdown_connections 2023-10-07T09:45:39.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3ec4066f30 0x7f3ec40693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ed815ade0 0x7f3ed81509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ed80b0540 0x7f3ed8149b40 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 --2- 172.21.15.29:0/3359123504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ed8153cc0 0x7f3ed81504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 >> 172.21.15.29:0/3359123504 conn(0x7f3ed8067610 msgr2=0x7f3ed80b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:39.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 shutdown_connections 2023-10-07T09:45:39.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.326+0000 7f3ede5ff700 1 -- 172.21.15.29:0/3359123504 wait complete. 2023-10-07T09:45:39.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:39.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.540+0000 7fb7c15ff700 1 Processor -- start 2023-10-07T09:45:39.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.540+0000 7fb7c15ff700 1 -- start start 2023-10-07T09:45:39.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.541+0000 7fb7c15ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7bc159750 0x7fb7bc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.541+0000 7fb7c15ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7bc0ce8a0 0x7fb7bc0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.541+0000 7fb7c15ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0cbf20 con 0x7fb7bc159b20 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.541+0000 7fb7c15ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7bc0cc200 con 0x7fb7bc0ce8a0 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.541+0000 7fb7c15ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7bc0cc4e0 con 0x7fb7bc159750 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7bc0ce8a0 0x7fb7bc0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7baffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7bc159750 0x7fb7bc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7bc0ce8a0 0x7fb7bc0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59260/0 (socket says 172.21.15.29:59260) 2023-10-07T09:45:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3330521255 learned_addr learned my addr 172.21.15.29:0/3330521255 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3330521255 >> v2:172.21.15.144:3300/0 conn(0x7fb7bc159750 msgr2=0x7fb7bc0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 --2- 172.21.15.29:0/3330521255 >> v2:172.21.15.144:3300/0 conn(0x7fb7bc159750 0x7fb7bc0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3330521255 >> v1:172.21.15.144:6789/0 conn(0x7fb7bc159b20 legacy=0x7fb7bc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3330521255 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc0cc7c0 con 0x7fb7bc0ce8a0 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.542+0000 7fb7bb7fe700 1 --2- 172.21.15.29:0/3330521255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0ce8a0 0x7fb7bc0c5460 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc089840 tx=0x7fb7b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5b85c2d1e8d9ba server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7b9ffb700 1 -- 172.21.15.29:0/3330521255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400e070 con 0x7fb7bc0ce8a0 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7b9ffb700 1 -- 172.21.15.29:0/3330521255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b400ba20 con 0x7fb7bc0ce8a0 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7b9ffb700 1 -- 172.21.15.29:0/3330521255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40092e0 con 0x7fb7bc0ce8a0 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3330521255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0ce8a0 msgr2=0x7fb7bc0c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3330521255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0ce8a0 0x7fb7bc0c5460 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc089840 tx=0x7fb7b4009840 comp rx=0 tx=0).stop 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3330521255 shutdown_connections 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3330521255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0ce8a0 0x7fb7bc0c5460 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3330521255 >> v2:172.21.15.144:3300/0 conn(0x7fb7bc159750 0x7fb7bc0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3330521255 >> 172.21.15.29:0/3330521255 conn(0x7fb7bc063a10 msgr2=0x7fb7bc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3330521255 shutdown_connections 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.543+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3330521255 wait complete. 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.545+0000 7fb7c15ff700 1 Processor -- start 2023-10-07T09:45:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.545+0000 7fb7c15ff700 1 -- start start 2023-10-07T09:45:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7c15ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7bc0ce8a0 0x7fb7bc15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7c15ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7bc159750 0x7fb7bc15d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7c15ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 0x7fb7bc15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7c15ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc15c1f0 con 0x7fb7bc159b20 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7c15ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc0b3ae0 con 0x7fb7bc159750 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7c15ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc15bf10 con 0x7fb7bc0ce8a0 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7bb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 0x7fb7bc15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7bb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 0x7fb7bc15cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59264/0 (socket says 172.21.15.29:59264) 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3591546502 learned_addr learned my addr 172.21.15.29:0/3591546502 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7baffd700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7bc0ce8a0 0x7fb7bc15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.546+0000 7fb7ba7fc700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7bc159750 0x7fb7bc15d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7bc0ce8a0 msgr2=0x7fb7bc15b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7bb7fe700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7bc0ce8a0 0x7fb7bc15b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7bc159750 msgr2=0x7fb7bc15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7bb7fe700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7bc159750 0x7fb7bc15d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7bb7fe700 1 -- 172.21.15.29:0/3591546502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4009ae0 con 0x7fb7bc159b20 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7bb7fe700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 0x7fb7bc15cbd0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009880 tx=0x7fb7b4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4008a80 con 0x7fb7bc159b20 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b4008be0 con 0x7fb7bc159b20 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4026460 con 0x7fb7bc159b20 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc0b3490 con 0x7fb7bc159b20 2023-10-07T09:45:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.547+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc15d110 con 0x7fb7bc159b20 2023-10-07T09:45:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.549+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc0b3c50 con 0x7fb7bc159b20 2023-10-07T09:45:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.549+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb7b4003040 con 0x7fb7bc159b20 2023-10-07T09:45:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.551+0000 7fb7a2ffd700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7a8066f30 0x7fb7a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.551+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb7b4086560 con 0x7fb7bc159b20 2023-10-07T09:45:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.551+0000 7fb7baffd700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7a8066f30 0x7fb7a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.552+0000 7fb7baffd700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7a8066f30 0x7fb7a80693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009da0 tx=0x7fb7ac009120 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.555+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7b40535b0 con 0x7fb7bc159b20 2023-10-07T09:45:39.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.758+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7bc04c3c0 con 0x7fb7bc159b20 2023-10-07T09:45:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.759+0000 7fb7a2ffd700 1 -- 172.21.15.29:0/3591546502 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16044 (secure 0 0 0) 0x7fb7b4056bd0 con 0x7fb7bc159b20 2023-10-07T09:45:39.760 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:39.760 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":102,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:38.376892+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:45:36.139381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:37.345605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7a8066f30 msgr2=0x7fb7a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7a8066f30 0x7fb7a80693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009da0 tx=0x7fb7ac009120 comp rx=0 tx=0).stop 2023-10-07T09:45:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 msgr2=0x7fb7bc15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 0x7fb7bc15cbd0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009880 tx=0x7fb7b4018000 comp rx=0 tx=0).stop 2023-10-07T09:45:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 shutdown_connections 2023-10-07T09:45:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb7a8066f30 0x7fb7a80693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc159b20 0x7fb7bc15cbd0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7bc0ce8a0 0x7fb7bc15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 --2- 172.21.15.29:0/3591546502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7bc159750 0x7fb7bc15d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 >> 172.21.15.29:0/3591546502 conn(0x7fb7bc063a10 msgr2=0x7fb7bc0704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 shutdown_connections 2023-10-07T09:45:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.762+0000 7fb7c15ff700 1 -- 172.21.15.29:0/3591546502 wait complete. 2023-10-07T09:45:39.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:45:39.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.984+0000 7f9251dff700 1 Processor -- start 2023-10-07T09:45:39.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.984+0000 7f9251dff700 1 -- start start 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f9251dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f924c153cc0 0x7f924c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f9251dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f924c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f9251dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924c154090 con 0x7f924c15ade0 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f9251dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924c159f60 con 0x7f924c153cc0 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f9251dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c15a240 con 0x7f924c0b0710 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f924affd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f924c15ade0 0x7f924c0b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39266/0 (socket says 172.21.15.29:39266) 2023-10-07T09:45:39.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f924affd700 1 -- 172.21.15.29:0/2250291249 learned_addr learned my addr 172.21.15.29:0/2250291249 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:39.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f924b7fe700 1 --2- 172.21.15.29:0/2250291249 >> v2:172.21.15.144:3300/0 conn(0x7f924c153cc0 0x7f924c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973448343 0 0) 0x7f924c154090 con 0x7f924c15ade0 2023-10-07T09:45:39.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.986+0000 7f924bfff700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f924c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9234003570 con 0x7f924c15ade0 2023-10-07T09:45:39.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 463711381 0 0) 0x7f9234003570 con 0x7f924c15ade0 2023-10-07T09:45:39.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 msgr2=0x7f924c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f924c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 >> v2:172.21.15.144:3300/0 conn(0x7f924c153cc0 msgr2=0x7f924c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 --2- 172.21.15.29:0/2250291249 >> v2:172.21.15.144:3300/0 conn(0x7f924c153cc0 0x7f924c0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c15a520 con 0x7f924c15ade0 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f9240001470 con 0x7f924c15ade0 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 >> v1:172.21.15.29:6789/0 conn(0x7f924c15ade0 legacy=0x7f924c0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f9234003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f9234004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9234004d50 0x7f92340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c159f60 con 0x7f9234003fe0 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c15a240 con 0x7f9234004d50 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923400ba90 con 0x7f924c0b0710 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924affd700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f9234004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924bfff700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9234004d50 0x7f92340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.987+0000 7f924b7fe700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f9234003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924affd700 1 -- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 msgr2=0x7f9234003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924affd700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f9234003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924affd700 1 -- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9234004d50 msgr2=0x7f92340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924affd700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9234004d50 0x7f92340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924affd700 1 -- 172.21.15.29:0/2250291249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234003570 con 0x7f9234003fe0 2023-10-07T09:45:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924b7fe700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c0b0710 0x7f9234003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924affd700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f9234004810 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f924c15b200 tx=0x7f9240001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8dce9e31b8e233 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000cbd0 con 0x7f9234003fe0 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924000cd70 con 0x7f9234003fe0 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.988+0000 7f924a7fc700 1 -- 172.21.15.29:0/2250291249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240005b70 con 0x7f9234003fe0 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f9251dff700 1 -- 172.21.15.29:0/2250291249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 msgr2=0x7f9234004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f9251dff700 1 --2- 172.21.15.29:0/2250291249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f9234004810 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f924c15b200 tx=0x7f9240001e90 comp rx=0 tx=0).stop 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f924bfff700 1 -- 172.21.15.29:0/2250291249 reap_dead start 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f9251dff700 1 -- 172.21.15.29:0/2250291249 shutdown_connections 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f9251dff700 1 -- 172.21.15.29:0/2250291249 >> 172.21.15.29:0/2250291249 conn(0x7f924c067610 msgr2=0x7f924c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f9251dff700 1 -- 172.21.15.29:0/2250291249 shutdown_connections 2023-10-07T09:45:39.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.989+0000 7f9251dff700 1 -- 172.21.15.29:0/2250291249 wait complete. 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.990+0000 7f9251dff700 1 Processor -- start 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.990+0000 7f9251dff700 1 -- start start 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f9251dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f924c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f9251dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c1569e0 0x7f924c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f9251dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f924c144a80 0x7f924c144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f9251dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c169790 con 0x7f9234003fe0 2023-10-07T09:45:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f9251dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c145430 con 0x7f924c144a80 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f9251dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c156540 con 0x7f924c1569e0 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f924b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f924c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.991+0000 7f924b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f924c15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59280/0 (socket says 172.21.15.29:59280) 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 -- 172.21.15.29:0/1502544512 learned_addr learned my addr 172.21.15.29:0/1502544512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924affd700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c1569e0 0x7f924c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924bfff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f924c144a80 0x7f924c144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 -- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c1569e0 msgr2=0x7f924c156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c1569e0 0x7f924c156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 -- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f924c144a80 msgr2=0x7f924c144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f924c144a80 0x7f924c144ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 -- 172.21.15.29:0/1502544512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c157550 con 0x7f9234003fe0 2023-10-07T09:45:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924affd700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c1569e0 0x7f924c156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924bfff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f924c144a80 0x7f924c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f924b7fe700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f924c15e460 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f923c0118e0 tx=0x7f923c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c011e80 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c00f8e0 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924c157830 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.992+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c00fc80 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.993+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c157df0 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.994+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924c158010 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.994+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f923c015670 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.995+0000 7f92337fe700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9238066f30 0x7f92380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.995+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f923c084db0 con 0x7f9234003fe0 2023-10-07T09:45:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.996+0000 7f924affd700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9238066f30 0x7f92380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:39.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:39.997+0000 7f924affd700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9238066f30 0x7f92380693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f924c15b200 tx=0x7f9240015000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:40.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.000+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f923c055930 con 0x7f9234003fe0 2023-10-07T09:45:40.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.255+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f924c06c630 con 0x7f9234003fe0 2023-10-07T09:45:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.893+0000 7f92337fe700 1 -- 172.21.15.29:0/1502544512 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7f923c022020 con 0x7f9234003fe0 2023-10-07T09:45:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9238066f30 msgr2=0x7f92380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9238066f30 0x7f92380693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f924c15b200 tx=0x7f9240015000 comp rx=0 tx=0).stop 2023-10-07T09:45:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 msgr2=0x7f924c15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f924c15e460 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f923c0118e0 tx=0x7f923c008040 comp rx=0 tx=0).stop 2023-10-07T09:45:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 shutdown_connections 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9238066f30 0x7f92380693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f924c144a80 0x7f924c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9234003fe0 0x7f924c15e460 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 --2- 172.21.15.29:0/1502544512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f924c1569e0 0x7f924c156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 >> 172.21.15.29:0/1502544512 conn(0x7f924c067610 msgr2=0x7f924c152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 shutdown_connections 2023-10-07T09:45:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:40.897+0000 7f9251dff700 1 -- 172.21.15.29:0/1502544512 wait complete. 2023-10-07T09:45:40.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:45:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.121+0000 7f5a1d3ff700 1 Processor -- start 2023-10-07T09:45:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.122+0000 7f5a1d3ff700 1 -- start start 2023-10-07T09:45:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a1d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a18159750 0x7f5a180d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a1d3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a180ce8a0 0x7f5a180c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a1d3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180cbf20 con 0x7f5a18159b20 2023-10-07T09:45:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a1d3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a180cc200 con 0x7f5a180ce8a0 2023-10-07T09:45:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a1d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180cc4e0 con 0x7f5a18159750 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a167fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5a18159b20 0x7f5a180ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39276/0 (socket says 172.21.15.29:39276) 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a167fc700 1 -- 172.21.15.29:0/478460587 learned_addr learned my addr 172.21.15.29:0/478460587 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a177fe700 1 --2- 172.21.15.29:0/478460587 >> v2:172.21.15.144:3300/0 conn(0x7f5a180ce8a0 0x7f5a180c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.123+0000 7f5a16ffd700 1 --2- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a18159750 0x7f5a180d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a15ffb700 1 -- 172.21.15.29:0/478460587 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337173372 0 0) 0x7f5a180cbf20 con 0x7f5a18159b20 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a15ffb700 1 -- 172.21.15.29:0/478460587 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00003570 con 0x7f5a18159b20 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a177fe700 1 -- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a18159750 msgr2=0x7f5a180d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a177fe700 1 --2- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a18159750 0x7f5a180d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a177fe700 1 -- 172.21.15.29:0/478460587 >> v1:172.21.15.29:6789/0 conn(0x7f5a18159b20 legacy=0x7f5a180ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a177fe700 1 -- 172.21.15.29:0/478460587 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180cc7c0 con 0x7f5a180ce8a0 2023-10-07T09:45:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.124+0000 7f5a177fe700 1 --2- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a180ce8a0 0x7f5a180c5460 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009e60 tx=0x7f5a10009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acdc8928d1f25f5b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a15ffb700 1 -- 172.21.15.29:0/478460587 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000e070 con 0x7f5a180ce8a0 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a15ffb700 1 -- 172.21.15.29:0/478460587 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1000bab0 con 0x7f5a180ce8a0 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a15ffb700 1 -- 172.21.15.29:0/478460587 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000be70 con 0x7f5a180ce8a0 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a180ce8a0 msgr2=0x7f5a180c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a180ce8a0 0x7f5a180c5460 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009e60 tx=0x7f5a10009780 comp rx=0 tx=0).stop 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/478460587 shutdown_connections 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a180ce8a0 0x7f5a180c5460 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/478460587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a18159750 0x7f5a180d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/478460587 >> 172.21.15.29:0/478460587 conn(0x7f5a18063a10 msgr2=0x7f5a180d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/478460587 shutdown_connections 2023-10-07T09:45:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.125+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/478460587 wait complete. 2023-10-07T09:45:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.127+0000 7f5a1d3ff700 1 Processor -- start 2023-10-07T09:45:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.127+0000 7f5a1d3ff700 1 -- start start 2023-10-07T09:45:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a1d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a180ce8a0 0x7f5a181697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a1d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a18159750 0x7f5a1815e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a1d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 0x7f5a1816a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a1d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1815f850 con 0x7f5a18159b20 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a1d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b6300 con 0x7f5a18159750 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a1d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1815ec40 con 0x7f5a180ce8a0 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 0x7f5a1816a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 0x7f5a1816a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59284/0 (socket says 172.21.15.29:59284) 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a177fe700 1 -- 172.21.15.29:0/996706853 learned_addr learned my addr 172.21.15.29:0/996706853 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a16ffd700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a180ce8a0 0x7f5a181697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.128+0000 7f5a167fc700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a18159750 0x7f5a1815e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a177fe700 1 -- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a180ce8a0 msgr2=0x7f5a181697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a177fe700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a180ce8a0 0x7f5a181697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a177fe700 1 -- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a18159750 msgr2=0x7f5a1815e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a177fe700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a18159750 0x7f5a1815e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a177fe700 1 -- 172.21.15.29:0/996706853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10009a20 con 0x7f5a18159b20 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a167fc700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a18159750 0x7f5a1815e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a177fe700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 0x7f5a1816a120 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5a18088a40 tx=0x7f5a10019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10012e10 con 0x7f5a18159b20 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a10027470 con 0x7f5a18159b20 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10027830 con 0x7f5a18159b20 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.129+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1815ef20 con 0x7f5a18159b20 2023-10-07T09:45:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.130+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1815f4e0 con 0x7f5a18159b20 2023-10-07T09:45:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.130+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1804c3c0 con 0x7f5a18159b20 2023-10-07T09:45:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.131+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a10024070 con 0x7f5a18159b20 2023-10-07T09:45:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.132+0000 7f59feffd700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a04066f30 0x7f5a040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.132+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5a10087170 con 0x7f5a18159b20 2023-10-07T09:45:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.133+0000 7f5a16ffd700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a04066f30 0x7f5a040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.133+0000 7f5a16ffd700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a04066f30 0x7f5a040693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009d80 tx=0x7f5a08009120 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:41.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.136+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a100541c0 con 0x7f5a18159b20 2023-10-07T09:45:41.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.387+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5a180ca970 con 0x7f5a18159b20 2023-10-07T09:45:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.896+0000 7f59feffd700 1 -- 172.21.15.29:0/996706853 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5a100577e0 con 0x7f5a18159b20 2023-10-07T09:45:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a04066f30 msgr2=0x7f5a040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a04066f30 0x7f5a040693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009d80 tx=0x7f5a08009120 comp rx=0 tx=0).stop 2023-10-07T09:45:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 msgr2=0x7f5a1816a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 0x7f5a1816a120 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5a18088a40 tx=0x7f5a10019000 comp rx=0 tx=0).stop 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 shutdown_connections 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a04066f30 0x7f5a040693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a18159b20 0x7f5a1816a120 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a180ce8a0 0x7f5a181697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 --2- 172.21.15.29:0/996706853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a18159750 0x7f5a1815e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 >> 172.21.15.29:0/996706853 conn(0x7f5a18063a10 msgr2=0x7f5a180ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 shutdown_connections 2023-10-07T09:45:41.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:41.900+0000 7f5a1d3ff700 1 -- 172.21.15.29:0/996706853 wait complete. 2023-10-07T09:45:41.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:42.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.120+0000 7f75639ff700 1 Processor -- start 2023-10-07T09:45:42.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.120+0000 7f75639ff700 1 -- start start 2023-10-07T09:45:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.121+0000 7f75639ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f755c0b0710 0x7f755c15c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.121+0000 7f75639ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c154090 con 0x7f755c15ade0 2023-10-07T09:45:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.121+0000 7f75639ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c159f60 con 0x7f755c153cc0 2023-10-07T09:45:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.121+0000 7f75639ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755c15a240 con 0x7f755c0b0710 2023-10-07T09:45:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7560f9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f755c15ade0 0x7f755c0b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39278/0 (socket says 172.21.15.29:39278) 2023-10-07T09:45:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7560f9a700 1 -- 172.21.15.29:0/3672367934 learned_addr learned my addr 172.21.15.29:0/3672367934 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:42.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7561f9c700 1 --2- 172.21.15.29:0/3672367934 >> v2:172.21.15.144:3300/0 conn(0x7f755c0b0710 0x7f755c15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848853152 0 0) 0x7f755c154090 con 0x7f755c15ade0 2023-10-07T09:45:42.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548003570 con 0x7f755c15ade0 2023-10-07T09:45:42.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912439782 0 0) 0x7f755c159f60 con 0x7f755c153cc0 2023-10-07T09:45:42.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.122+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c154090 con 0x7f755c153cc0 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572834289 0 0) 0x7f7548003570 con 0x7f755c15ade0 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 >> v2:172.21.15.144:3300/0 conn(0x7f755c0b0710 msgr2=0x7f755c15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 --2- 172.21.15.29:0/3672367934 >> v2:172.21.15.144:3300/0 conn(0x7f755c0b0710 0x7f755c15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 >> v1:172.21.15.144:6789/0 conn(0x7f755c153cc0 legacy=0x7f755c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c15a520 con 0x7f755c15ade0 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f754c001470 con 0x7f755c15ade0 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 >> v1:172.21.15.29:6789/0 conn(0x7f755c15ade0 legacy=0x7f755c0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7548004ca0 0x7f7548005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75480037b0 0x7f7548003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75480041b0 0x7f754800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c15a240 con 0x7f75480041b0 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754800c130 con 0x7f75480037b0 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754800c2c0 con 0x7f7548004ca0 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7561f9c700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75480041b0 0x7f754800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f7560f9a700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75480037b0 0x7f7548003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.123+0000 7f756179b700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7548004ca0 0x7f7548005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 -- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7548004ca0 msgr2=0x7f7548005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7548004ca0 0x7f7548005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 -- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75480037b0 msgr2=0x7f7548003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75480037b0 0x7f7548003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 -- 172.21.15.29:0/3672367934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7548003570 con 0x7f75480041b0 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 -- 172.21.15.29:0/3672367934 reap_dead start 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7561f9c700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75480041b0 0x7f754800ba60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f755400e210 tx=0x7f755400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee48957b857d2ec1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554009ac0 con 0x7f75480041b0 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755400f040 con 0x7f75480041b0 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.124+0000 7f7553fff700 1 -- 172.21.15.29:0/3672367934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755400ed10 con 0x7f75480041b0 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 -- 172.21.15.29:0/3672367934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75480041b0 msgr2=0x7f754800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75480041b0 0x7f754800ba60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f755400e210 tx=0x7f755400b040 comp rx=0 tx=0).stop 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 -- 172.21.15.29:0/3672367934 shutdown_connections 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 --2- 172.21.15.29:0/3672367934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75480041b0 0x7f754800ba60 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 -- 172.21.15.29:0/3672367934 >> 172.21.15.29:0/3672367934 conn(0x7f755c067610 msgr2=0x7f755c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:42.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 -- 172.21.15.29:0/3672367934 shutdown_connections 2023-10-07T09:45:42.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.125+0000 7f75639ff700 1 -- 172.21.15.29:0/3672367934 wait complete. 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.126+0000 7f75639ff700 1 Processor -- start 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.126+0000 7f75639ff700 1 -- start start 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.127+0000 7f75639ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75480041b0 0x7f755c158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.127+0000 7f75639ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 0x7f755c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.127+0000 7f75639ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f755c0c8630 0x7f755c0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.127+0000 7f75639ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0c8fe0 con 0x7f755c154b00 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.127+0000 7f75639ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c157890 con 0x7f755c0c8630 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.127+0000 7f75639ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c155650 con 0x7f75480041b0 2023-10-07T09:45:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 0x7f755c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 0x7f755c154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59306/0 (socket says 172.21.15.29:59306) 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 -- 172.21.15.29:0/4174570650 learned_addr learned my addr 172.21.15.29:0/4174570650 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f756179b700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75480041b0 0x7f755c158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7561f9c700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f755c0c8630 0x7f755c0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 -- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75480041b0 msgr2=0x7f755c158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75480041b0 0x7f755c158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 -- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f755c0c8630 msgr2=0x7f755c0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f755c0c8630 0x7f755c0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 -- 172.21.15.29:0/4174570650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c155930 con 0x7f755c154b00 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f7560f9a700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 0x7f755c154f70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f754c000de0 tx=0x7f754c001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c011590 con 0x7f755c154b00 2023-10-07T09:45:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c0116f0 con 0x7f755c154b00 2023-10-07T09:45:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c155c10 con 0x7f755c154b00 2023-10-07T09:45:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.128+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c011ab0 con 0x7f755c154b00 2023-10-07T09:45:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.129+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c1561d0 con 0x7f755c154b00 2023-10-07T09:45:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.130+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c04c3c0 con 0x7f755c154b00 2023-10-07T09:45:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.130+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f754c01d460 con 0x7f755c154b00 2023-10-07T09:45:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.131+0000 7f75517fa700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7540066f30 0x7f75400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.132+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f754c084830 con 0x7f755c154b00 2023-10-07T09:45:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.132+0000 7f756179b700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7540066f30 0x7f75400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.133+0000 7f756179b700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7540066f30 0x7f75400693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7558007e90 tx=0x7f7558009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:42.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.135+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f754c051880 con 0x7f755c154b00 2023-10-07T09:45:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.384+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f755c15a800 con 0x7f755c154b00 2023-10-07T09:45:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.384+0000 7f75517fa700 1 -- 172.21.15.29:0/4174570650 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f754c054ea0 con 0x7f755c154b00 2023-10-07T09:45:42.384 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:42.384 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:38.377260+0000","modified":"2023-10-07T09:45:41.892365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3302133648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3302133648},{"type":"v1","addr":"172.21.15.29:6835","nonce":3302133648}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7540066f30 msgr2=0x7f75400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7540066f30 0x7f75400693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7558007e90 tx=0x7f7558009040 comp rx=0 tx=0).stop 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 msgr2=0x7f755c154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 0x7f755c154f70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f754c000de0 tx=0x7f754c001d40 comp rx=0 tx=0).stop 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 shutdown_connections 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7540066f30 0x7f75400693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f755c0c8630 0x7f755c0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75480041b0 0x7f755c158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 --2- 172.21.15.29:0/4174570650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c154b00 0x7f755c154f70 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.386+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 >> 172.21.15.29:0/4174570650 conn(0x7f755c067610 msgr2=0x7f755c0b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.387+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 shutdown_connections 2023-10-07T09:45:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.387+0000 7f75639ff700 1 -- 172.21.15.29:0/4174570650 wait complete. 2023-10-07T09:45:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-07T09:45:42.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:45:42.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.600+0000 7fb5a5dff700 1 Processor -- start 2023-10-07T09:45:42.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.600+0000 7fb5a5dff700 1 -- start start 2023-10-07T09:45:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.601+0000 7fb5a5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0158670 0x7fb5a0162060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.601+0000 7fb5a5dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0158ad0 con 0x7fb5a00ae8b0 2023-10-07T09:45:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.601+0000 7fb5a5dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a015b140 con 0x7fb5a0157290 2023-10-07T09:45:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.601+0000 7fb5a5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a015b420 con 0x7fb5a0158670 2023-10-07T09:45:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59effd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb5a00ae8b0 0x7fb5a00b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39288/0 (socket says 172.21.15.29:39288) 2023-10-07T09:45:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59effd700 1 -- 172.21.15.29:0/2769003386 learned_addr learned my addr 172.21.15.29:0/2769003386 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59ffff700 1 --2- 172.21.15.29:0/2769003386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0158670 0x7fb5a0162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704803012 0 0) 0x7fb5a0158ad0 con 0x7fb5a00ae8b0 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb588003570 con 0x7fb5a00ae8b0 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366111589 0 0) 0x7fb5a015b140 con 0x7fb5a0157290 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a0158ad0 con 0x7fb5a0157290 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59ffff700 1 -- 172.21.15.29:0/2769003386 >> v1:172.21.15.144:6789/0 conn(0x7fb5a0157290 legacy=0x7fb5a00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59ffff700 1 -- 172.21.15.29:0/2769003386 >> v1:172.21.15.29:6789/0 conn(0x7fb5a00ae8b0 legacy=0x7fb5a00b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.602+0000 7fb59ffff700 1 -- 172.21.15.29:0/2769003386 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a015b700 con 0x7fb5a0158670 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.603+0000 7fb59ffff700 1 --2- 172.21.15.29:0/2769003386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0158670 0x7fb5a0162060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb598009e90 tx=0x7fb598009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9c43efc9ef97e78 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.603+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800e070 con 0x7fb5a0158670 2023-10-07T09:45:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.603+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59800bb80 con 0x7fb5a0158670 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.603+0000 7fb59e7fc700 1 -- 172.21.15.29:0/2769003386 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598012430 con 0x7fb5a0158670 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 -- 172.21.15.29:0/2769003386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0158670 msgr2=0x7fb5a0162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/2769003386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0158670 0x7fb5a0162060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb598009e90 tx=0x7fb598009780 comp rx=0 tx=0).stop 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 -- 172.21.15.29:0/2769003386 shutdown_connections 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/2769003386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0158670 0x7fb5a0162060 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 -- 172.21.15.29:0/2769003386 >> 172.21.15.29:0/2769003386 conn(0x7fb5a00680a0 msgr2=0x7fb5a00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 -- 172.21.15.29:0/2769003386 shutdown_connections 2023-10-07T09:45:42.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.604+0000 7fb5a5dff700 1 -- 172.21.15.29:0/2769003386 wait complete. 2023-10-07T09:45:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.605+0000 7fb5a5dff700 1 Processor -- start 2023-10-07T09:45:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.605+0000 7fb5a5dff700 1 -- start start 2023-10-07T09:45:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.606+0000 7fb5a5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0157290 0x7fb5a0149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.606+0000 7fb5a5dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a0158670 0x7fb5a014a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.606+0000 7fb5a5dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 0x7fb5a006e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.606+0000 7fb5a5dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0153900 con 0x7fb5a014a760 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.606+0000 7fb5a5dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0149660 con 0x7fb5a0158670 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.606+0000 7fb5a5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a01535d0 con 0x7fb5a0157290 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 0x7fb5a006e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 0x7fb5a006e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59318/0 (socket says 172.21.15.29:59318) 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 -- 172.21.15.29:0/1901427397 learned_addr learned my addr 172.21.15.29:0/1901427397 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59f7fe700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0157290 0x7fb5a0149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59effd700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a0158670 0x7fb5a014a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 -- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0157290 msgr2=0x7fb5a0149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0157290 0x7fb5a0149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 -- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a0158670 msgr2=0x7fb5a014a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a0158670 0x7fb5a014a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 -- 172.21.15.29:0/1901427397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009a20 con 0x7fb5a014a760 2023-10-07T09:45:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59f7fe700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0157290 0x7fb5a0149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59effd700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a0158670 0x7fb5a014a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.607+0000 7fb59ffff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 0x7fb5a006e310 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0068dd0 tx=0x7fb598019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.608+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598012e00 con 0x7fb5a014a760 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.608+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59801b630 con 0x7fb5a014a760 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.608+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a014ac40 con 0x7fb5a014a760 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.608+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59801b9f0 con 0x7fb5a014a760 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.608+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a014b200 con 0x7fb5a014a760 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.609+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a0153ab0 con 0x7fb5a014a760 2023-10-07T09:45:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.610+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb59801c400 con 0x7fb5a014a760 2023-10-07T09:45:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.611+0000 7fb5877fe700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb58c066f30 0x7fb58c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.611+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb598086210 con 0x7fb5a014a760 2023-10-07T09:45:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.611+0000 7fb59f7fe700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb58c066f30 0x7fb58c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.612+0000 7fb59f7fe700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb58c066f30 0x7fb58c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb590003990 tx=0x7fb590009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:42.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.615+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5a0153ab0 con 0x7fb5a014a760 2023-10-07T09:45:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.819+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5a0153710 con 0x7fb5a014a760 2023-10-07T09:45:42.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.821+0000 7fb5877fe700 1 -- 172.21.15.29:0/1901427397 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+16031 (secure 0 0 0) 0x7fb598056da0 con 0x7fb5a014a760 2023-10-07T09:45:42.821 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:42.821 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":103,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:45:39.878380+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:45:36.139381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:37.345605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:45:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb58c066f30 msgr2=0x7fb58c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb58c066f30 0x7fb58c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb590003990 tx=0x7fb590009040 comp rx=0 tx=0).stop 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 msgr2=0x7fb5a006e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 0x7fb5a006e310 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0068dd0 tx=0x7fb598019000 comp rx=0 tx=0).stop 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 shutdown_connections 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb58c066f30 0x7fb58c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a014a760 0x7fb5a006e310 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a0157290 0x7fb5a0149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 --2- 172.21.15.29:0/1901427397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a0158670 0x7fb5a014a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 >> 172.21.15.29:0/1901427397 conn(0x7fb5a00680a0 msgr2=0x7fb5a0152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 shutdown_connections 2023-10-07T09:45:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:42.823+0000 7fb5a5dff700 1 -- 172.21.15.29:0/1901427397 wait complete. 2023-10-07T09:45:42.841 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:45:42.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:45:43.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.038+0000 7f38a85ff700 1 Processor -- start 2023-10-07T09:45:43.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.038+0000 7f38a85ff700 1 -- start start 2023-10-07T09:45:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a85ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f38a00ad370 0x7f38a00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a85ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f38a00b0410 0x7f38a015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a85ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a014d790 con 0x7f38a0159ec0 2023-10-07T09:45:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a85ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a015a880 con 0x7f38a00ad370 2023-10-07T09:45:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a85ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a00ae5d0 con 0x7f38a00b0410 2023-10-07T09:45:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a639b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f38a00ad370 0x7f38a00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a639b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f38a00ad370 0x7f38a00cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47194/0 (socket says 172.21.15.29:47194) 2023-10-07T09:45:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a639b700 1 -- 172.21.15.29:0/3382247352 learned_addr learned my addr 172.21.15.29:0/3382247352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.040+0000 7f38a6b9c700 1 --2- 172.21.15.29:0/3382247352 >> v2:172.21.15.144:3300/0 conn(0x7f38a00b0410 0x7f38a015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a639b700 1 -- 172.21.15.29:0/3382247352 >> v2:172.21.15.144:3300/0 conn(0x7f38a00b0410 msgr2=0x7f38a015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a639b700 1 --2- 172.21.15.29:0/3382247352 >> v2:172.21.15.144:3300/0 conn(0x7f38a00b0410 0x7f38a015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a639b700 1 -- 172.21.15.29:0/3382247352 >> v1:172.21.15.144:6789/0 conn(0x7f38a0159ec0 legacy=0x7f38a01568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a639b700 1 -- 172.21.15.29:0/3382247352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a00ad740 con 0x7f38a00ad370 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a639b700 1 --2- 172.21.15.29:0/3382247352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a00cc870 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f389c0099b0 tx=0x7f389c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d24bad0d83bbf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a5399700 1 -- 172.21.15.29:0/3382247352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c009ea0 con 0x7f38a00ad370 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a5399700 1 -- 172.21.15.29:0/3382247352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c00b820 con 0x7f38a00ad370 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.041+0000 7f38a5399700 1 -- 172.21.15.29:0/3382247352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c00bbc0 con 0x7f38a00ad370 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 -- 172.21.15.29:0/3382247352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 msgr2=0x7f38a00cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 --2- 172.21.15.29:0/3382247352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a00cc870 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f389c0099b0 tx=0x7f389c009250 comp rx=0 tx=0).stop 2023-10-07T09:45:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 -- 172.21.15.29:0/3382247352 shutdown_connections 2023-10-07T09:45:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 --2- 172.21.15.29:0/3382247352 >> v2:172.21.15.144:3300/0 conn(0x7f38a00b0410 0x7f38a015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 --2- 172.21.15.29:0/3382247352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a00cc870 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 -- 172.21.15.29:0/3382247352 >> 172.21.15.29:0/3382247352 conn(0x7f38a00680a0 msgr2=0x7f38a00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 -- 172.21.15.29:0/3382247352 shutdown_connections 2023-10-07T09:45:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.042+0000 7f38a85ff700 1 -- 172.21.15.29:0/3382247352 wait complete. 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.043+0000 7f38a85ff700 1 Processor -- start 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 -- start start 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a014c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38a00b0410 0x7f38a0167500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38a0159ec0 0x7f38a0154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a0155b00 con 0x7f38a00ad370 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a0153240 con 0x7f38a0159ec0 2023-10-07T09:45:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.044+0000 7f38a85ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a0155540 con 0x7f38a00b0410 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a014c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a014c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47210/0 (socket says 172.21.15.29:47210) 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 -- 172.21.15.29:0/1851175373 learned_addr learned my addr 172.21.15.29:0/1851175373 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a6b9c700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38a0159ec0 0x7f38a0154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a5b9a700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38a00b0410 0x7f38a0167500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 -- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38a00b0410 msgr2=0x7f38a0167500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38a00b0410 0x7f38a0167500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 -- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38a0159ec0 msgr2=0x7f38a0154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38a0159ec0 0x7f38a0154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 -- 172.21.15.29:0/1851175373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c0094f0 con 0x7f38a00ad370 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.045+0000 7f38a639b700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a014c0b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f389c0094c0 tx=0x7f389c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.046+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c00bba0 con 0x7f38a00ad370 2023-10-07T09:45:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.046+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c011420 con 0x7f38a00ad370 2023-10-07T09:45:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.046+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0117e0 con 0x7f38a00ad370 2023-10-07T09:45:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.046+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a0155820 con 0x7f38a00ad370 2023-10-07T09:45:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.046+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a0152790 con 0x7f38a00ad370 2023-10-07T09:45:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.047+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a004c3c0 con 0x7f38a00ad370 2023-10-07T09:45:43.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.048+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f389c011b90 con 0x7f38a00ad370 2023-10-07T09:45:43.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.049+0000 7f38967fc700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3884066f30 0x7f38840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.049+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f389c086010 con 0x7f38a00ad370 2023-10-07T09:45:43.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.049+0000 7f38a5b9a700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3884066f30 0x7f38840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.050+0000 7f38a5b9a700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3884066f30 0x7f38840693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38a016bc10 tx=0x7f3890009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.053+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f389c053100 con 0x7f38a00ad370 2023-10-07T09:45:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.252+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38a01571d0 con 0x7f3884066f30 2023-10-07T09:45:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.264+0000 7f38967fc700 1 -- 172.21.15.29:0/1851175373 <== mgr.6721 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289289 (secure 0 0 0) 0x7f38a01571d0 con 0x7f3884066f30 2023-10-07T09:45:43.268 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:43.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3884066f30 msgr2=0x7f38840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3884066f30 0x7f38840693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38a016bc10 tx=0x7f3890009040 comp rx=0 tx=0).stop 2023-10-07T09:45:43.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 msgr2=0x7f38a014c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a014c0b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f389c0094c0 tx=0x7f389c017000 comp rx=0 tx=0).stop 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 shutdown_connections 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3884066f30 0x7f38840693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38a0159ec0 0x7f38a0154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a00ad370 0x7f38a014c0b0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 --2- 172.21.15.29:0/1851175373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38a00b0410 0x7f38a0167500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 >> 172.21.15.29:0/1851175373 conn(0x7f38a00680a0 msgr2=0x7f38a014d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 shutdown_connections 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.268+0000 7f38a85ff700 1 -- 172.21.15.29:0/1851175373 wait complete. 2023-10-07T09:45:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:45:43.286 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:45:42.964130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2438492,"kb_used_data":11036,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536228,"statfs":{"total":773094113280,"available":770597097472,"internally_reserved":0,"allocated":11300864,"data_stored":5983204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001479"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894475+0000","last_change":"2023-10-07T09:45:38.886372+0000","last_active":"2023-10-07T09:45:38.894475+0000","last_peered":"2023-10-07T09:45:38.894475+0000","last_clean":"2023-10-07T09:45:38.894475+0000","last_became_active":"2023-10-07T09:45:38.885972+0000","last_became_peered":"2023-10-07T09:45:38.885972+0000","last_unstale":"2023-10-07T09:45:38.894475+0000","last_undegraded":"2023-10-07T09:45:38.894475+0000","last_fullsized":"2023-10-07T09:45:38.894475+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:37:12.898759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894753+0000","last_change":"2023-10-07T09:45:38.886441+0000","last_active":"2023-10-07T09:45:38.894753+0000","last_peered":"2023-10-07T09:45:38.894753+0000","last_clean":"2023-10-07T09:45:38.894753+0000","last_became_active":"2023-10-07T09:45:38.886052+0000","last_became_peered":"2023-10-07T09:45:38.886052+0000","last_unstale":"2023-10-07T09:45:38.894753+0000","last_undegraded":"2023-10-07T09:45:38.894753+0000","last_fullsized":"2023-10-07T09:45:38.894753+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:21.596245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884723+0000","last_change":"2023-10-07T09:45:38.883381+0000","last_active":"2023-10-07T09:45:38.884723+0000","last_peered":"2023-10-07T09:45:38.884723+0000","last_clean":"2023-10-07T09:45:38.884723+0000","last_became_active":"2023-10-07T09:45:38.883012+0000","last_became_peered":"2023-10-07T09:45:38.883012+0000","last_unstale":"2023-10-07T09:45:38.884723+0000","last_undegraded":"2023-10-07T09:45:38.884723+0000","last_fullsized":"2023-10-07T09:45:38.884723+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:02:48.025477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897639+0000","last_change":"2023-10-07T09:45:38.889307+0000","last_active":"2023-10-07T09:45:38.897639+0000","last_peered":"2023-10-07T09:45:38.897639+0000","last_clean":"2023-10-07T09:45:38.897639+0000","last_became_active":"2023-10-07T09:45:38.889007+0000","last_became_peered":"2023-10-07T09:45:38.889007+0000","last_unstale":"2023-10-07T09:45:38.897639+0000","last_undegraded":"2023-10-07T09:45:38.897639+0000","last_fullsized":"2023-10-07T09:45:38.897639+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:30:53.319015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893261+0000","last_change":"2023-10-07T09:45:38.887705+0000","last_active":"2023-10-07T09:45:38.893261+0000","last_peered":"2023-10-07T09:45:38.893261+0000","last_clean":"2023-10-07T09:45:38.893261+0000","last_became_active":"2023-10-07T09:45:38.887432+0000","last_became_peered":"2023-10-07T09:45:38.887432+0000","last_unstale":"2023-10-07T09:45:38.893261+0000","last_undegraded":"2023-10-07T09:45:38.893261+0000","last_fullsized":"2023-10-07T09:45:38.893261+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:11:03.897589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896424+0000","last_change":"2023-10-07T09:45:38.892949+0000","last_active":"2023-10-07T09:45:38.896424+0000","last_peered":"2023-10-07T09:45:38.896424+0000","last_clean":"2023-10-07T09:45:38.896424+0000","last_became_active":"2023-10-07T09:45:38.892694+0000","last_became_peered":"2023-10-07T09:45:38.892694+0000","last_unstale":"2023-10-07T09:45:38.896424+0000","last_undegraded":"2023-10-07T09:45:38.896424+0000","last_fullsized":"2023-10-07T09:45:38.896424+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:32:56.602781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894636+0000","last_change":"2023-10-07T09:45:38.881048+0000","last_active":"2023-10-07T09:45:38.894636+0000","last_peered":"2023-10-07T09:45:38.894636+0000","last_clean":"2023-10-07T09:45:38.894636+0000","last_became_active":"2023-10-07T09:45:38.880776+0000","last_became_peered":"2023-10-07T09:45:38.880776+0000","last_unstale":"2023-10-07T09:45:38.894636+0000","last_undegraded":"2023-10-07T09:45:38.894636+0000","last_fullsized":"2023-10-07T09:45:38.894636+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:34.751650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896482+0000","last_change":"2023-10-07T09:45:38.893493+0000","last_active":"2023-10-07T09:45:38.896482+0000","last_peered":"2023-10-07T09:45:38.896482+0000","last_clean":"2023-10-07T09:45:38.896482+0000","last_became_active":"2023-10-07T09:45:38.893216+0000","last_became_peered":"2023-10-07T09:45:38.893216+0000","last_unstale":"2023-10-07T09:45:38.896482+0000","last_undegraded":"2023-10-07T09:45:38.896482+0000","last_fullsized":"2023-10-07T09:45:38.896482+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:53.816169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892973+0000","last_change":"2023-10-07T09:45:38.886964+0000","last_active":"2023-10-07T09:45:38.892973+0000","last_peered":"2023-10-07T09:45:38.892973+0000","last_clean":"2023-10-07T09:45:38.892973+0000","last_became_active":"2023-10-07T09:45:38.886702+0000","last_became_peered":"2023-10-07T09:45:38.886702+0000","last_unstale":"2023-10-07T09:45:38.892973+0000","last_undegraded":"2023-10-07T09:45:38.892973+0000","last_fullsized":"2023-10-07T09:45:38.892973+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:43:40.383305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896269+0000","last_change":"2023-10-07T09:45:38.894801+0000","last_active":"2023-10-07T09:45:38.896269+0000","last_peered":"2023-10-07T09:45:38.896269+0000","last_clean":"2023-10-07T09:45:38.896269+0000","last_became_active":"2023-10-07T09:45:38.894529+0000","last_became_peered":"2023-10-07T09:45:38.894529+0000","last_unstale":"2023-10-07T09:45:38.896269+0000","last_undegraded":"2023-10-07T09:45:38.896269+0000","last_fullsized":"2023-10-07T09:45:38.896269+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:52:53.997248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.881812+0000","last_change":"2023-10-07T09:45:38.876623+0000","last_active":"2023-10-07T09:45:38.881812+0000","last_peered":"2023-10-07T09:45:38.881812+0000","last_clean":"2023-10-07T09:45:38.881812+0000","last_became_active":"2023-10-07T09:45:38.876354+0000","last_became_peered":"2023-10-07T09:45:38.876354+0000","last_unstale":"2023-10-07T09:45:38.881812+0000","last_undegraded":"2023-10-07T09:45:38.881812+0000","last_fullsized":"2023-10-07T09:45:38.881812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:33.411089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896434+0000","last_change":"2023-10-07T09:45:38.895306+0000","last_active":"2023-10-07T09:45:38.896434+0000","last_peered":"2023-10-07T09:45:38.896434+0000","last_clean":"2023-10-07T09:45:38.896434+0000","last_became_active":"2023-10-07T09:45:38.895003+0000","last_became_peered":"2023-10-07T09:45:38.895003+0000","last_unstale":"2023-10-07T09:45:38.896434+0000","last_undegraded":"2023-10-07T09:45:38.896434+0000","last_fullsized":"2023-10-07T09:45:38.896434+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:09:37.743358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.883939+0000","last_change":"2023-10-07T09:45:38.880791+0000","last_active":"2023-10-07T09:45:38.883939+0000","last_peered":"2023-10-07T09:45:38.883939+0000","last_clean":"2023-10-07T09:45:38.883939+0000","last_became_active":"2023-10-07T09:45:38.880403+0000","last_became_peered":"2023-10-07T09:45:38.880403+0000","last_unstale":"2023-10-07T09:45:38.883939+0000","last_undegraded":"2023-10-07T09:45:38.883939+0000","last_fullsized":"2023-10-07T09:45:38.883939+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:21:10.049308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897738+0000","last_change":"2023-10-07T09:45:38.891245+0000","last_active":"2023-10-07T09:45:38.897738+0000","last_peered":"2023-10-07T09:45:38.897738+0000","last_clean":"2023-10-07T09:45:38.897738+0000","last_became_active":"2023-10-07T09:45:38.890644+0000","last_became_peered":"2023-10-07T09:45:38.890644+0000","last_unstale":"2023-10-07T09:45:38.897738+0000","last_undegraded":"2023-10-07T09:45:38.897738+0000","last_fullsized":"2023-10-07T09:45:38.897738+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:20.367513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891216+0000","last_change":"2023-10-07T09:45:38.875458+0000","last_active":"2023-10-07T09:45:38.891216+0000","last_peered":"2023-10-07T09:45:38.891216+0000","last_clean":"2023-10-07T09:45:38.891216+0000","last_became_active":"2023-10-07T09:45:38.874814+0000","last_became_peered":"2023-10-07T09:45:38.874814+0000","last_unstale":"2023-10-07T09:45:38.891216+0000","last_undegraded":"2023-10-07T09:45:38.891216+0000","last_fullsized":"2023-10-07T09:45:38.891216+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:42.306675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.890965+0000","last_change":"2023-10-07T09:45:38.883256+0000","last_active":"2023-10-07T09:45:38.890965+0000","last_peered":"2023-10-07T09:45:38.890965+0000","last_clean":"2023-10-07T09:45:38.890965+0000","last_became_active":"2023-10-07T09:45:38.883020+0000","last_became_peered":"2023-10-07T09:45:38.883020+0000","last_unstale":"2023-10-07T09:45:38.890965+0000","last_undegraded":"2023-10-07T09:45:38.890965+0000","last_fullsized":"2023-10-07T09:45:38.890965+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:00:48.224115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892412+0000","last_change":"2023-10-07T09:45:38.890588+0000","last_active":"2023-10-07T09:45:38.892412+0000","last_peered":"2023-10-07T09:45:38.892412+0000","last_clean":"2023-10-07T09:45:38.892412+0000","last_became_active":"2023-10-07T09:45:38.890055+0000","last_became_peered":"2023-10-07T09:45:38.890055+0000","last_unstale":"2023-10-07T09:45:38.892412+0000","last_undegraded":"2023-10-07T09:45:38.892412+0000","last_fullsized":"2023-10-07T09:45:38.892412+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:32:56.602781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897773+0000","last_change":"2023-10-07T09:45:38.888562+0000","last_active":"2023-10-07T09:45:38.897773+0000","last_peered":"2023-10-07T09:45:38.897773+0000","last_clean":"2023-10-07T09:45:38.897773+0000","last_became_active":"2023-10-07T09:45:38.888254+0000","last_became_peered":"2023-10-07T09:45:38.888254+0000","last_unstale":"2023-10-07T09:45:38.897773+0000","last_undegraded":"2023-10-07T09:45:38.897773+0000","last_fullsized":"2023-10-07T09:45:38.897773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:29:40.247224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899870+0000","last_change":"2023-10-07T09:45:38.894148+0000","last_active":"2023-10-07T09:45:38.899870+0000","last_peered":"2023-10-07T09:45:38.899870+0000","last_clean":"2023-10-07T09:45:38.899870+0000","last_became_active":"2023-10-07T09:45:38.893861+0000","last_became_peered":"2023-10-07T09:45:38.893861+0000","last_unstale":"2023-10-07T09:45:38.899870+0000","last_undegraded":"2023-10-07T09:45:38.899870+0000","last_fullsized":"2023-10-07T09:45:38.899870+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:56:53.556431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.900882+0000","last_change":"2023-10-07T09:45:38.898861+0000","last_active":"2023-10-07T09:45:38.900882+0000","last_peered":"2023-10-07T09:45:38.900882+0000","last_clean":"2023-10-07T09:45:38.900882+0000","last_became_active":"2023-10-07T09:45:38.898578+0000","last_became_peered":"2023-10-07T09:45:38.898578+0000","last_unstale":"2023-10-07T09:45:38.900882+0000","last_undegraded":"2023-10-07T09:45:38.900882+0000","last_fullsized":"2023-10-07T09:45:38.900882+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:02:26.040494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899773+0000","last_change":"2023-10-07T09:45:38.898801+0000","last_active":"2023-10-07T09:45:38.899773+0000","last_peered":"2023-10-07T09:45:38.899773+0000","last_clean":"2023-10-07T09:45:38.899773+0000","last_became_active":"2023-10-07T09:45:38.898550+0000","last_became_peered":"2023-10-07T09:45:38.898550+0000","last_unstale":"2023-10-07T09:45:38.899773+0000","last_undegraded":"2023-10-07T09:45:38.899773+0000","last_fullsized":"2023-10-07T09:45:38.899773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:02:07.427536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894531+0000","last_change":"2023-10-07T09:45:38.885026+0000","last_active":"2023-10-07T09:45:38.894531+0000","last_peered":"2023-10-07T09:45:38.894531+0000","last_clean":"2023-10-07T09:45:38.894531+0000","last_became_active":"2023-10-07T09:45:38.884764+0000","last_became_peered":"2023-10-07T09:45:38.884764+0000","last_unstale":"2023-10-07T09:45:38.894531+0000","last_undegraded":"2023-10-07T09:45:38.894531+0000","last_fullsized":"2023-10-07T09:45:38.894531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:40:40.367333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893224+0000","last_change":"2023-10-07T09:45:38.885885+0000","last_active":"2023-10-07T09:45:38.893224+0000","last_peered":"2023-10-07T09:45:38.893224+0000","last_clean":"2023-10-07T09:45:38.893224+0000","last_became_active":"2023-10-07T09:45:38.885601+0000","last_became_peered":"2023-10-07T09:45:38.885601+0000","last_unstale":"2023-10-07T09:45:38.893224+0000","last_undegraded":"2023-10-07T09:45:38.893224+0000","last_fullsized":"2023-10-07T09:45:38.893224+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:21:11.460981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.900078+0000","last_change":"2023-10-07T09:45:38.898714+0000","last_active":"2023-10-07T09:45:38.900078+0000","last_peered":"2023-10-07T09:45:38.900078+0000","last_clean":"2023-10-07T09:45:38.900078+0000","last_became_active":"2023-10-07T09:45:38.898435+0000","last_became_peered":"2023-10-07T09:45:38.898435+0000","last_unstale":"2023-10-07T09:45:38.900078+0000","last_undegraded":"2023-10-07T09:45:38.900078+0000","last_fullsized":"2023-10-07T09:45:38.900078+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:04.408429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894788+0000","last_change":"2023-10-07T09:45:38.886751+0000","last_active":"2023-10-07T09:45:38.894788+0000","last_peered":"2023-10-07T09:45:38.894788+0000","last_clean":"2023-10-07T09:45:38.894788+0000","last_became_active":"2023-10-07T09:45:38.885802+0000","last_became_peered":"2023-10-07T09:45:38.885802+0000","last_unstale":"2023-10-07T09:45:38.894788+0000","last_undegraded":"2023-10-07T09:45:38.894788+0000","last_fullsized":"2023-10-07T09:45:38.894788+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:42.716793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884758+0000","last_change":"2023-10-07T09:45:38.883306+0000","last_active":"2023-10-07T09:45:38.884758+0000","last_peered":"2023-10-07T09:45:38.884758+0000","last_clean":"2023-10-07T09:45:38.884758+0000","last_became_active":"2023-10-07T09:45:38.883036+0000","last_became_peered":"2023-10-07T09:45:38.883036+0000","last_unstale":"2023-10-07T09:45:38.884758+0000","last_undegraded":"2023-10-07T09:45:38.884758+0000","last_fullsized":"2023-10-07T09:45:38.884758+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:33:10.749620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.880245+0000","last_change":"2023-10-07T09:45:38.878155+0000","last_active":"2023-10-07T09:45:38.880245+0000","last_peered":"2023-10-07T09:45:38.880245+0000","last_clean":"2023-10-07T09:45:38.880245+0000","last_became_active":"2023-10-07T09:45:38.877829+0000","last_became_peered":"2023-10-07T09:45:38.877829+0000","last_unstale":"2023-10-07T09:45:38.880245+0000","last_undegraded":"2023-10-07T09:45:38.880245+0000","last_fullsized":"2023-10-07T09:45:38.880245+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:18:33.071732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893252+0000","last_change":"2023-10-07T09:45:38.886496+0000","last_active":"2023-10-07T09:45:38.893252+0000","last_peered":"2023-10-07T09:45:38.893252+0000","last_clean":"2023-10-07T09:45:38.893252+0000","last_became_active":"2023-10-07T09:45:38.886131+0000","last_became_peered":"2023-10-07T09:45:38.886131+0000","last_unstale":"2023-10-07T09:45:38.893252+0000","last_undegraded":"2023-10-07T09:45:38.893252+0000","last_fullsized":"2023-10-07T09:45:38.893252+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:58.436253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899304+0000","last_change":"2023-10-07T09:45:38.894170+0000","last_active":"2023-10-07T09:45:38.899304+0000","last_peered":"2023-10-07T09:45:38.899304+0000","last_clean":"2023-10-07T09:45:38.899304+0000","last_became_active":"2023-10-07T09:45:38.893895+0000","last_became_peered":"2023-10-07T09:45:38.893895+0000","last_unstale":"2023-10-07T09:45:38.899304+0000","last_undegraded":"2023-10-07T09:45:38.899304+0000","last_fullsized":"2023-10-07T09:45:38.899304+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:06.060682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892939+0000","last_change":"2023-10-07T09:45:38.883079+0000","last_active":"2023-10-07T09:45:38.892939+0000","last_peered":"2023-10-07T09:45:38.892939+0000","last_clean":"2023-10-07T09:45:38.892939+0000","last_became_active":"2023-10-07T09:45:38.882533+0000","last_became_peered":"2023-10-07T09:45:38.882533+0000","last_unstale":"2023-10-07T09:45:38.892939+0000","last_undegraded":"2023-10-07T09:45:38.892939+0000","last_fullsized":"2023-10-07T09:45:38.892939+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:59:05.924980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891407+0000","last_change":"2023-10-07T09:45:38.884839+0000","last_active":"2023-10-07T09:45:38.891407+0000","last_peered":"2023-10-07T09:45:38.891407+0000","last_clean":"2023-10-07T09:45:38.891407+0000","last_became_active":"2023-10-07T09:45:38.884560+0000","last_became_peered":"2023-10-07T09:45:38.884560+0000","last_unstale":"2023-10-07T09:45:38.891407+0000","last_undegraded":"2023-10-07T09:45:38.891407+0000","last_fullsized":"2023-10-07T09:45:38.891407+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:07:45.181900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893194+0000","last_change":"2023-10-07T09:45:38.886581+0000","last_active":"2023-10-07T09:45:38.893194+0000","last_peered":"2023-10-07T09:45:38.893194+0000","last_clean":"2023-10-07T09:45:38.893194+0000","last_became_active":"2023-10-07T09:45:38.886045+0000","last_became_peered":"2023-10-07T09:45:38.886045+0000","last_unstale":"2023-10-07T09:45:38.893194+0000","last_undegraded":"2023-10-07T09:45:38.893194+0000","last_fullsized":"2023-10-07T09:45:38.893194+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:30.225686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891074+0000","last_change":"2023-10-07T09:45:38.882100+0000","last_active":"2023-10-07T09:45:38.891074+0000","last_peered":"2023-10-07T09:45:38.891074+0000","last_clean":"2023-10-07T09:45:38.891074+0000","last_became_active":"2023-10-07T09:45:38.881856+0000","last_became_peered":"2023-10-07T09:45:38.881856+0000","last_unstale":"2023-10-07T09:45:38.891074+0000","last_undegraded":"2023-10-07T09:45:38.891074+0000","last_fullsized":"2023-10-07T09:45:38.891074+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:30.225686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"102'4","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898506+0000","last_change":"2023-10-07T09:45:37.885464+0000","last_active":"2023-10-07T09:45:38.898506+0000","last_peered":"2023-10-07T09:45:38.898506+0000","last_clean":"2023-10-07T09:45:38.898506+0000","last_became_active":"2023-10-07T09:45:37.885148+0000","last_became_peered":"2023-10-07T09:45:37.885148+0000","last_unstale":"2023-10-07T09:45:38.898506+0000","last_undegraded":"2023-10-07T09:45:38.898506+0000","last_fullsized":"2023-10-07T09:45:38.898506+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:36:01.656048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897888+0000","last_change":"2023-10-07T09:45:38.897426+0000","last_active":"2023-10-07T09:45:38.897888+0000","last_peered":"2023-10-07T09:45:38.897888+0000","last_clean":"2023-10-07T09:45:38.897888+0000","last_became_active":"2023-10-07T09:45:38.897111+0000","last_became_peered":"2023-10-07T09:45:38.897111+0000","last_unstale":"2023-10-07T09:45:38.897888+0000","last_undegraded":"2023-10-07T09:45:38.897888+0000","last_fullsized":"2023-10-07T09:45:38.897888+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:10.381540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898125+0000","last_change":"2023-10-07T09:45:37.887295+0000","last_active":"2023-10-07T09:45:38.898125+0000","last_peered":"2023-10-07T09:45:38.898125+0000","last_clean":"2023-10-07T09:45:38.898125+0000","last_became_active":"2023-10-07T09:45:37.886437+0000","last_became_peered":"2023-10-07T09:45:37.886437+0000","last_unstale":"2023-10-07T09:45:38.898125+0000","last_undegraded":"2023-10-07T09:45:38.898125+0000","last_fullsized":"2023-10-07T09:45:38.898125+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:40:39.358901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893271+0000","last_change":"2023-10-07T09:45:38.886842+0000","last_active":"2023-10-07T09:45:38.893271+0000","last_peered":"2023-10-07T09:45:38.893271+0000","last_clean":"2023-10-07T09:45:38.893271+0000","last_became_active":"2023-10-07T09:45:38.886577+0000","last_became_peered":"2023-10-07T09:45:38.886577+0000","last_unstale":"2023-10-07T09:45:38.893271+0000","last_undegraded":"2023-10-07T09:45:38.893271+0000","last_fullsized":"2023-10-07T09:45:38.893271+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:42.306675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899824+0000","last_change":"2023-10-07T09:45:37.878828+0000","last_active":"2023-10-07T09:45:38.899824+0000","last_peered":"2023-10-07T09:45:38.899824+0000","last_clean":"2023-10-07T09:45:38.899824+0000","last_became_active":"2023-10-07T09:45:37.878563+0000","last_became_peered":"2023-10-07T09:45:37.878563+0000","last_unstale":"2023-10-07T09:45:38.899824+0000","last_undegraded":"2023-10-07T09:45:38.899824+0000","last_fullsized":"2023-10-07T09:45:38.899824+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:19:33.995655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884266+0000","last_change":"2023-10-07T09:45:38.882087+0000","last_active":"2023-10-07T09:45:38.884266+0000","last_peered":"2023-10-07T09:45:38.884266+0000","last_clean":"2023-10-07T09:45:38.884266+0000","last_became_active":"2023-10-07T09:45:38.881661+0000","last_became_peered":"2023-10-07T09:45:38.881661+0000","last_unstale":"2023-10-07T09:45:38.884266+0000","last_undegraded":"2023-10-07T09:45:38.884266+0000","last_fullsized":"2023-10-07T09:45:38.884266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:14.847690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899280+0000","last_change":"2023-10-07T09:45:37.877726+0000","last_active":"2023-10-07T09:45:38.899280+0000","last_peered":"2023-10-07T09:45:38.899280+0000","last_clean":"2023-10-07T09:45:38.899280+0000","last_became_active":"2023-10-07T09:45:37.877451+0000","last_became_peered":"2023-10-07T09:45:37.877451+0000","last_unstale":"2023-10-07T09:45:38.899280+0000","last_undegraded":"2023-10-07T09:45:38.899280+0000","last_fullsized":"2023-10-07T09:45:38.899280+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:59:04.916547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892919+0000","last_change":"2023-10-07T09:45:38.882805+0000","last_active":"2023-10-07T09:45:38.892919+0000","last_peered":"2023-10-07T09:45:38.892919+0000","last_clean":"2023-10-07T09:45:38.892919+0000","last_became_active":"2023-10-07T09:45:38.882106+0000","last_became_peered":"2023-10-07T09:45:38.882106+0000","last_unstale":"2023-10-07T09:45:38.892919+0000","last_undegraded":"2023-10-07T09:45:38.892919+0000","last_fullsized":"2023-10-07T09:45:38.892919+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:06:01.086981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895062+0000","last_change":"2023-10-07T09:45:37.882623+0000","last_active":"2023-10-07T09:45:38.895062+0000","last_peered":"2023-10-07T09:45:38.895062+0000","last_clean":"2023-10-07T09:45:38.895062+0000","last_became_active":"2023-10-07T09:45:37.882363+0000","last_became_peered":"2023-10-07T09:45:37.882363+0000","last_unstale":"2023-10-07T09:45:38.895062+0000","last_undegraded":"2023-10-07T09:45:38.895062+0000","last_fullsized":"2023-10-07T09:45:38.895062+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:32.402656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893241+0000","last_change":"2023-10-07T09:45:38.887642+0000","last_active":"2023-10-07T09:45:38.893241+0000","last_peered":"2023-10-07T09:45:38.893241+0000","last_clean":"2023-10-07T09:45:38.893241+0000","last_became_active":"2023-10-07T09:45:38.887359+0000","last_became_peered":"2023-10-07T09:45:38.887359+0000","last_unstale":"2023-10-07T09:45:38.893241+0000","last_undegraded":"2023-10-07T09:45:38.893241+0000","last_fullsized":"2023-10-07T09:45:38.893241+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:31:35.952352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898151+0000","last_change":"2023-10-07T09:45:37.879843+0000","last_active":"2023-10-07T09:45:38.898151+0000","last_peered":"2023-10-07T09:45:38.898151+0000","last_clean":"2023-10-07T09:45:38.898151+0000","last_became_active":"2023-10-07T09:45:37.879595+0000","last_became_peered":"2023-10-07T09:45:37.879595+0000","last_unstale":"2023-10-07T09:45:38.898151+0000","last_undegraded":"2023-10-07T09:45:38.898151+0000","last_fullsized":"2023-10-07T09:45:38.898151+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:04.097156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895900+0000","last_change":"2023-10-07T09:45:38.893518+0000","last_active":"2023-10-07T09:45:38.895900+0000","last_peered":"2023-10-07T09:45:38.895900+0000","last_clean":"2023-10-07T09:45:38.895900+0000","last_became_active":"2023-10-07T09:45:38.892643+0000","last_became_peered":"2023-10-07T09:45:38.892643+0000","last_unstale":"2023-10-07T09:45:38.895900+0000","last_undegraded":"2023-10-07T09:45:38.895900+0000","last_fullsized":"2023-10-07T09:45:38.895900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:54:52.797058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.871139+0000","last_change":"2023-10-07T09:45:37.887004+0000","last_active":"2023-10-07T09:45:38.871139+0000","last_peered":"2023-10-07T09:45:38.871139+0000","last_clean":"2023-10-07T09:45:38.871139+0000","last_became_active":"2023-10-07T09:45:37.886730+0000","last_became_peered":"2023-10-07T09:45:37.886730+0000","last_unstale":"2023-10-07T09:45:38.871139+0000","last_undegraded":"2023-10-07T09:45:38.871139+0000","last_fullsized":"2023-10-07T09:45:38.871139+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:25.332188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891235+0000","last_change":"2023-10-07T09:45:38.875634+0000","last_active":"2023-10-07T09:45:38.891235+0000","last_peered":"2023-10-07T09:45:38.891235+0000","last_clean":"2023-10-07T09:45:38.891235+0000","last_became_active":"2023-10-07T09:45:38.875208+0000","last_became_peered":"2023-10-07T09:45:38.875208+0000","last_unstale":"2023-10-07T09:45:38.891235+0000","last_undegraded":"2023-10-07T09:45:38.891235+0000","last_fullsized":"2023-10-07T09:45:38.891235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:30.225686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.868314+0000","last_change":"2023-10-07T09:45:37.879446+0000","last_active":"2023-10-07T09:45:38.868314+0000","last_peered":"2023-10-07T09:45:38.868314+0000","last_clean":"2023-10-07T09:45:38.868314+0000","last_became_active":"2023-10-07T09:45:37.879011+0000","last_became_peered":"2023-10-07T09:45:37.879011+0000","last_unstale":"2023-10-07T09:45:38.868314+0000","last_undegraded":"2023-10-07T09:45:38.868314+0000","last_fullsized":"2023-10-07T09:45:38.868314+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:02:25.032061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891452+0000","last_change":"2023-10-07T09:45:38.874236+0000","last_active":"2023-10-07T09:45:38.891452+0000","last_peered":"2023-10-07T09:45:38.891452+0000","last_clean":"2023-10-07T09:45:38.891452+0000","last_became_active":"2023-10-07T09:45:38.873880+0000","last_became_peered":"2023-10-07T09:45:38.873880+0000","last_unstale":"2023-10-07T09:45:38.891452+0000","last_undegraded":"2023-10-07T09:45:38.891452+0000","last_fullsized":"2023-10-07T09:45:38.891452+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:00:48.224115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.878765+0000","last_change":"2023-10-07T09:45:37.885958+0000","last_active":"2023-10-07T09:45:38.878765+0000","last_peered":"2023-10-07T09:45:38.878765+0000","last_clean":"2023-10-07T09:45:38.878765+0000","last_became_active":"2023-10-07T09:45:37.885675+0000","last_became_peered":"2023-10-07T09:45:37.885675+0000","last_unstale":"2023-10-07T09:45:38.878765+0000","last_undegraded":"2023-10-07T09:45:38.878765+0000","last_fullsized":"2023-10-07T09:45:38.878765+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:32.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895119+0000","last_change":"2023-10-07T09:45:38.892445+0000","last_active":"2023-10-07T09:45:38.895119+0000","last_peered":"2023-10-07T09:45:38.895119+0000","last_clean":"2023-10-07T09:45:38.895119+0000","last_became_active":"2023-10-07T09:45:38.892162+0000","last_became_peered":"2023-10-07T09:45:38.892162+0000","last_unstale":"2023-10-07T09:45:38.895119+0000","last_undegraded":"2023-10-07T09:45:38.895119+0000","last_fullsized":"2023-10-07T09:45:38.895119+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:55.429915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.877494+0000","last_change":"2023-10-07T09:45:37.885087+0000","last_active":"2023-10-07T09:45:38.877494+0000","last_peered":"2023-10-07T09:45:38.877494+0000","last_clean":"2023-10-07T09:45:38.877494+0000","last_became_active":"2023-10-07T09:45:37.884818+0000","last_became_peered":"2023-10-07T09:45:37.884818+0000","last_unstale":"2023-10-07T09:45:38.877494+0000","last_undegraded":"2023-10-07T09:45:38.877494+0000","last_fullsized":"2023-10-07T09:45:38.877494+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:39:00.446142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899327+0000","last_change":"2023-10-07T09:45:38.889341+0000","last_active":"2023-10-07T09:45:38.899327+0000","last_peered":"2023-10-07T09:45:38.899327+0000","last_clean":"2023-10-07T09:45:38.899327+0000","last_became_active":"2023-10-07T09:45:38.889046+0000","last_became_peered":"2023-10-07T09:45:38.889046+0000","last_unstale":"2023-10-07T09:45:38.899327+0000","last_undegraded":"2023-10-07T09:45:38.899327+0000","last_fullsized":"2023-10-07T09:45:38.899327+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:19:14.474121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"102'3","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899707+0000","last_change":"2023-10-07T09:45:37.884692+0000","last_active":"2023-10-07T09:45:38.899707+0000","last_peered":"2023-10-07T09:45:38.899707+0000","last_clean":"2023-10-07T09:45:38.899707+0000","last_became_active":"2023-10-07T09:45:37.884389+0000","last_became_peered":"2023-10-07T09:45:37.884389+0000","last_unstale":"2023-10-07T09:45:38.899707+0000","last_undegraded":"2023-10-07T09:45:38.899707+0000","last_fullsized":"2023-10-07T09:45:38.899707+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:27.004476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895657+0000","last_change":"2023-10-07T09:45:38.894234+0000","last_active":"2023-10-07T09:45:38.895657+0000","last_peered":"2023-10-07T09:45:38.895657+0000","last_clean":"2023-10-07T09:45:38.895657+0000","last_became_active":"2023-10-07T09:45:38.893840+0000","last_became_peered":"2023-10-07T09:45:38.893840+0000","last_unstale":"2023-10-07T09:45:38.895657+0000","last_undegraded":"2023-10-07T09:45:38.895657+0000","last_fullsized":"2023-10-07T09:45:38.895657+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:53.816169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"102'6","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.901846+0000","last_change":"2023-10-07T09:45:37.887177+0000","last_active":"2023-10-07T09:45:38.901846+0000","last_peered":"2023-10-07T09:45:38.901846+0000","last_clean":"2023-10-07T09:45:38.901846+0000","last_became_active":"2023-10-07T09:45:37.886808+0000","last_became_peered":"2023-10-07T09:45:37.886808+0000","last_unstale":"2023-10-07T09:45:38.901846+0000","last_undegraded":"2023-10-07T09:45:38.901846+0000","last_fullsized":"2023-10-07T09:45:38.901846+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:47.046035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899198+0000","last_change":"2023-10-07T09:45:38.885066+0000","last_active":"2023-10-07T09:45:38.899198+0000","last_peered":"2023-10-07T09:45:38.899198+0000","last_clean":"2023-10-07T09:45:38.899198+0000","last_became_active":"2023-10-07T09:45:38.883834+0000","last_became_peered":"2023-10-07T09:45:38.883834+0000","last_unstale":"2023-10-07T09:45:38.899198+0000","last_undegraded":"2023-10-07T09:45:38.899198+0000","last_fullsized":"2023-10-07T09:45:38.899198+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:48.054468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.900491+0000","last_change":"2023-10-07T09:45:37.877618+0000","last_active":"2023-10-07T09:45:38.900491+0000","last_peered":"2023-10-07T09:45:38.900491+0000","last_clean":"2023-10-07T09:45:38.900491+0000","last_became_active":"2023-10-07T09:45:37.877083+0000","last_became_peered":"2023-10-07T09:45:37.877083+0000","last_unstale":"2023-10-07T09:45:38.900491+0000","last_undegraded":"2023-10-07T09:45:38.900491+0000","last_fullsized":"2023-10-07T09:45:38.900491+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:30:52.310582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893372+0000","last_change":"2023-10-07T09:45:38.891324+0000","last_active":"2023-10-07T09:45:38.893372+0000","last_peered":"2023-10-07T09:45:38.893372+0000","last_clean":"2023-10-07T09:45:38.893372+0000","last_became_active":"2023-10-07T09:45:38.891010+0000","last_became_peered":"2023-10-07T09:45:38.891010+0000","last_unstale":"2023-10-07T09:45:38.893372+0000","last_undegraded":"2023-10-07T09:45:38.893372+0000","last_fullsized":"2023-10-07T09:45:38.893372+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:34:06.906408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.864862+0000","last_change":"2023-10-07T09:45:37.876112+0000","last_active":"2023-10-07T09:45:38.864862+0000","last_peered":"2023-10-07T09:45:38.864862+0000","last_clean":"2023-10-07T09:45:38.864862+0000","last_became_active":"2023-10-07T09:45:37.875835+0000","last_became_peered":"2023-10-07T09:45:37.875835+0000","last_unstale":"2023-10-07T09:45:38.864862+0000","last_undegraded":"2023-10-07T09:45:38.864862+0000","last_fullsized":"2023-10-07T09:45:38.864862+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:48:53.197259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892451+0000","last_change":"2023-10-07T09:45:38.882410+0000","last_active":"2023-10-07T09:45:38.892451+0000","last_peered":"2023-10-07T09:45:38.892451+0000","last_clean":"2023-10-07T09:45:38.892451+0000","last_became_active":"2023-10-07T09:45:38.882111+0000","last_became_peered":"2023-10-07T09:45:38.882111+0000","last_unstale":"2023-10-07T09:45:38.892451+0000","last_undegraded":"2023-10-07T09:45:38.892451+0000","last_fullsized":"2023-10-07T09:45:38.892451+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:01.160885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.873897+0000","last_change":"2023-10-07T09:45:37.885798+0000","last_active":"2023-10-07T09:45:38.873897+0000","last_peered":"2023-10-07T09:45:38.873897+0000","last_clean":"2023-10-07T09:45:38.873897+0000","last_became_active":"2023-10-07T09:45:37.885452+0000","last_became_peered":"2023-10-07T09:45:37.885452+0000","last_unstale":"2023-10-07T09:45:38.873897+0000","last_undegraded":"2023-10-07T09:45:38.873897+0000","last_fullsized":"2023-10-07T09:45:38.873897+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:39:00.446142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.882576+0000","last_change":"2023-10-07T09:45:38.880813+0000","last_active":"2023-10-07T09:45:38.882576+0000","last_peered":"2023-10-07T09:45:38.882576+0000","last_clean":"2023-10-07T09:45:38.882576+0000","last_became_active":"2023-10-07T09:45:38.880574+0000","last_became_peered":"2023-10-07T09:45:38.880574+0000","last_unstale":"2023-10-07T09:45:38.882576+0000","last_undegraded":"2023-10-07T09:45:38.882576+0000","last_fullsized":"2023-10-07T09:45:38.882576+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:05.105589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"102'8","reported_seq":18,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899486+0000","last_change":"2023-10-07T09:45:37.872059+0000","last_active":"2023-10-07T09:45:38.899486+0000","last_peered":"2023-10-07T09:45:38.899486+0000","last_clean":"2023-10-07T09:45:38.899486+0000","last_became_active":"2023-10-07T09:45:37.871763+0000","last_became_peered":"2023-10-07T09:45:37.871763+0000","last_unstale":"2023-10-07T09:45:38.899486+0000","last_undegraded":"2023-10-07T09:45:38.899486+0000","last_fullsized":"2023-10-07T09:45:38.899486+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:32.402656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.883967+0000","last_change":"2023-10-07T09:45:38.880669+0000","last_active":"2023-10-07T09:45:38.883967+0000","last_peered":"2023-10-07T09:45:38.883967+0000","last_clean":"2023-10-07T09:45:38.883967+0000","last_became_active":"2023-10-07T09:45:38.880378+0000","last_became_peered":"2023-10-07T09:45:38.880378+0000","last_unstale":"2023-10-07T09:45:38.883967+0000","last_undegraded":"2023-10-07T09:45:38.883967+0000","last_fullsized":"2023-10-07T09:45:38.883967+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:34.472605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896718+0000","last_change":"2023-10-07T09:45:37.888660+0000","last_active":"2023-10-07T09:45:38.896718+0000","last_peered":"2023-10-07T09:45:38.896718+0000","last_clean":"2023-10-07T09:45:38.896718+0000","last_became_active":"2023-10-07T09:45:37.888340+0000","last_became_peered":"2023-10-07T09:45:37.888340+0000","last_unstale":"2023-10-07T09:45:38.896718+0000","last_undegraded":"2023-10-07T09:45:38.896718+0000","last_fullsized":"2023-10-07T09:45:38.896718+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:19:13.465689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894494+0000","last_change":"2023-10-07T09:45:38.887611+0000","last_active":"2023-10-07T09:45:38.894494+0000","last_peered":"2023-10-07T09:45:38.894494+0000","last_clean":"2023-10-07T09:45:38.894494+0000","last_became_active":"2023-10-07T09:45:38.886371+0000","last_became_peered":"2023-10-07T09:45:38.886371+0000","last_unstale":"2023-10-07T09:45:38.894494+0000","last_undegraded":"2023-10-07T09:45:38.894494+0000","last_fullsized":"2023-10-07T09:45:38.894494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:34.771752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898817+0000","last_change":"2023-10-07T09:45:37.882528+0000","last_active":"2023-10-07T09:45:38.898817+0000","last_peered":"2023-10-07T09:45:38.898817+0000","last_clean":"2023-10-07T09:45:38.898817+0000","last_became_active":"2023-10-07T09:45:37.880457+0000","last_became_peered":"2023-10-07T09:45:37.880457+0000","last_unstale":"2023-10-07T09:45:38.898817+0000","last_undegraded":"2023-10-07T09:45:38.898817+0000","last_fullsized":"2023-10-07T09:45:38.898817+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:00.152453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891462+0000","last_change":"2023-10-07T09:45:38.888529+0000","last_active":"2023-10-07T09:45:38.891462+0000","last_peered":"2023-10-07T09:45:38.891462+0000","last_clean":"2023-10-07T09:45:38.891462+0000","last_became_active":"2023-10-07T09:45:38.888266+0000","last_became_peered":"2023-10-07T09:45:38.888266+0000","last_unstale":"2023-10-07T09:45:38.891462+0000","last_undegraded":"2023-10-07T09:45:38.891462+0000","last_fullsized":"2023-10-07T09:45:38.891462+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:55.429915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.872206+0000","last_change":"2023-10-07T09:45:37.885925+0000","last_active":"2023-10-07T09:45:38.872206+0000","last_peered":"2023-10-07T09:45:38.872206+0000","last_clean":"2023-10-07T09:45:38.872206+0000","last_became_active":"2023-10-07T09:45:37.885563+0000","last_became_peered":"2023-10-07T09:45:37.885563+0000","last_unstale":"2023-10-07T09:45:38.872206+0000","last_undegraded":"2023-10-07T09:45:38.872206+0000","last_fullsized":"2023-10-07T09:45:38.872206+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:05.052249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897716+0000","last_change":"2023-10-07T09:45:38.891051+0000","last_active":"2023-10-07T09:45:38.897716+0000","last_peered":"2023-10-07T09:45:38.897716+0000","last_clean":"2023-10-07T09:45:38.897716+0000","last_became_active":"2023-10-07T09:45:38.890531+0000","last_became_peered":"2023-10-07T09:45:38.890531+0000","last_unstale":"2023-10-07T09:45:38.897716+0000","last_undegraded":"2023-10-07T09:45:38.897716+0000","last_fullsized":"2023-10-07T09:45:38.897716+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:53:28.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898160+0000","last_change":"2023-10-07T09:45:37.878267+0000","last_active":"2023-10-07T09:45:38.898160+0000","last_peered":"2023-10-07T09:45:38.898160+0000","last_clean":"2023-10-07T09:45:38.898160+0000","last_became_active":"2023-10-07T09:45:37.877969+0000","last_became_peered":"2023-10-07T09:45:37.877969+0000","last_unstale":"2023-10-07T09:45:38.898160+0000","last_undegraded":"2023-10-07T09:45:38.898160+0000","last_fullsized":"2023-10-07T09:45:38.898160+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:43:39.374873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896804+0000","last_change":"2023-10-07T09:45:38.896056+0000","last_active":"2023-10-07T09:45:38.896804+0000","last_peered":"2023-10-07T09:45:38.896804+0000","last_clean":"2023-10-07T09:45:38.896804+0000","last_became_active":"2023-10-07T09:45:38.895776+0000","last_became_peered":"2023-10-07T09:45:38.895776+0000","last_unstale":"2023-10-07T09:45:38.896804+0000","last_undegraded":"2023-10-07T09:45:38.896804+0000","last_fullsized":"2023-10-07T09:45:38.896804+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:31:35.952352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.875443+0000","last_change":"2023-10-07T09:45:37.877915+0000","last_active":"2023-10-07T09:45:38.875443+0000","last_peered":"2023-10-07T09:45:38.875443+0000","last_clean":"2023-10-07T09:45:38.875443+0000","last_became_active":"2023-10-07T09:45:37.877644+0000","last_became_peered":"2023-10-07T09:45:37.877644+0000","last_unstale":"2023-10-07T09:45:38.875443+0000","last_undegraded":"2023-10-07T09:45:38.875443+0000","last_fullsized":"2023-10-07T09:45:38.875443+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:46:36.915586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894579+0000","last_change":"2023-10-07T09:45:38.883241+0000","last_active":"2023-10-07T09:45:38.894579+0000","last_peered":"2023-10-07T09:45:38.894579+0000","last_clean":"2023-10-07T09:45:38.894579+0000","last_became_active":"2023-10-07T09:45:38.882979+0000","last_became_peered":"2023-10-07T09:45:38.882979+0000","last_unstale":"2023-10-07T09:45:38.894579+0000","last_undegraded":"2023-10-07T09:45:38.894579+0000","last_fullsized":"2023-10-07T09:45:38.894579+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:27:48.098129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.874267+0000","last_change":"2023-10-07T09:45:37.888352+0000","last_active":"2023-10-07T09:45:38.874267+0000","last_peered":"2023-10-07T09:45:38.874267+0000","last_clean":"2023-10-07T09:45:38.874267+0000","last_became_active":"2023-10-07T09:45:37.888021+0000","last_became_peered":"2023-10-07T09:45:37.888021+0000","last_unstale":"2023-10-07T09:45:38.874267+0000","last_undegraded":"2023-10-07T09:45:38.874267+0000","last_fullsized":"2023-10-07T09:45:38.874267+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:04.192109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893165+0000","last_change":"2023-10-07T09:45:38.890077+0000","last_active":"2023-10-07T09:45:38.893165+0000","last_peered":"2023-10-07T09:45:38.893165+0000","last_clean":"2023-10-07T09:45:38.893165+0000","last_became_active":"2023-10-07T09:45:38.889775+0000","last_became_peered":"2023-10-07T09:45:38.889775+0000","last_unstale":"2023-10-07T09:45:38.893165+0000","last_undegraded":"2023-10-07T09:45:38.893165+0000","last_fullsized":"2023-10-07T09:45:38.893165+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:11:03.897589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.864644+0000","last_change":"2023-10-07T09:45:37.878192+0000","last_active":"2023-10-07T09:45:38.864644+0000","last_peered":"2023-10-07T09:45:38.864644+0000","last_clean":"2023-10-07T09:45:38.864644+0000","last_became_active":"2023-10-07T09:45:37.877887+0000","last_became_peered":"2023-10-07T09:45:37.877887+0000","last_unstale":"2023-10-07T09:45:38.864644+0000","last_undegraded":"2023-10-07T09:45:38.864644+0000","last_fullsized":"2023-10-07T09:45:38.864644+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:54.421482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.885491+0000","last_change":"2023-10-07T09:45:38.879605+0000","last_active":"2023-10-07T09:45:38.885491+0000","last_peered":"2023-10-07T09:45:38.885491+0000","last_clean":"2023-10-07T09:45:38.885491+0000","last_became_active":"2023-10-07T09:45:38.879326+0000","last_became_peered":"2023-10-07T09:45:38.879326+0000","last_unstale":"2023-10-07T09:45:38.885491+0000","last_undegraded":"2023-10-07T09:45:38.885491+0000","last_fullsized":"2023-10-07T09:45:38.885491+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:42.306675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894443+0000","last_change":"2023-10-07T09:45:37.880609+0000","last_active":"2023-10-07T09:45:38.894443+0000","last_peered":"2023-10-07T09:45:38.894443+0000","last_clean":"2023-10-07T09:45:38.894443+0000","last_became_active":"2023-10-07T09:45:37.880352+0000","last_became_peered":"2023-10-07T09:45:37.880352+0000","last_unstale":"2023-10-07T09:45:38.894443+0000","last_undegraded":"2023-10-07T09:45:38.894443+0000","last_fullsized":"2023-10-07T09:45:38.894443+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:53:27.563837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892025+0000","last_change":"2023-10-07T09:45:38.883893+0000","last_active":"2023-10-07T09:45:38.892025+0000","last_peered":"2023-10-07T09:45:38.892025+0000","last_clean":"2023-10-07T09:45:38.892025+0000","last_became_active":"2023-10-07T09:45:38.883586+0000","last_became_peered":"2023-10-07T09:45:38.883586+0000","last_unstale":"2023-10-07T09:45:38.892025+0000","last_undegraded":"2023-10-07T09:45:38.892025+0000","last_fullsized":"2023-10-07T09:45:38.892025+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:53:10.245641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.869560+0000","last_change":"2023-10-07T09:45:37.876479+0000","last_active":"2023-10-07T09:45:38.869560+0000","last_peered":"2023-10-07T09:45:38.869560+0000","last_clean":"2023-10-07T09:45:38.869560+0000","last_became_active":"2023-10-07T09:45:37.876203+0000","last_became_peered":"2023-10-07T09:45:37.876203+0000","last_unstale":"2023-10-07T09:45:38.869560+0000","last_undegraded":"2023-10-07T09:45:38.869560+0000","last_fullsized":"2023-10-07T09:45:38.869560+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:19.359081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895174+0000","last_change":"2023-10-07T09:45:38.888133+0000","last_active":"2023-10-07T09:45:38.895174+0000","last_peered":"2023-10-07T09:45:38.895174+0000","last_clean":"2023-10-07T09:45:38.895174+0000","last_became_active":"2023-10-07T09:45:38.887883+0000","last_became_peered":"2023-10-07T09:45:38.887883+0000","last_unstale":"2023-10-07T09:45:38.895174+0000","last_undegraded":"2023-10-07T09:45:38.895174+0000","last_fullsized":"2023-10-07T09:45:38.895174+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:17.765418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.875749+0000","last_change":"2023-10-07T09:45:37.885288+0000","last_active":"2023-10-07T09:45:38.875749+0000","last_peered":"2023-10-07T09:45:38.875749+0000","last_clean":"2023-10-07T09:45:38.875749+0000","last_became_active":"2023-10-07T09:45:37.884962+0000","last_became_peered":"2023-10-07T09:45:37.884962+0000","last_unstale":"2023-10-07T09:45:38.875749+0000","last_undegraded":"2023-10-07T09:45:38.875749+0000","last_fullsized":"2023-10-07T09:45:38.875749+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:41.708361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896310+0000","last_change":"2023-10-07T09:45:38.894498+0000","last_active":"2023-10-07T09:45:38.896310+0000","last_peered":"2023-10-07T09:45:38.896310+0000","last_clean":"2023-10-07T09:45:38.896310+0000","last_became_active":"2023-10-07T09:45:38.894225+0000","last_became_peered":"2023-10-07T09:45:38.894225+0000","last_unstale":"2023-10-07T09:45:38.896310+0000","last_undegraded":"2023-10-07T09:45:38.896310+0000","last_fullsized":"2023-10-07T09:45:38.896310+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:48:54.205691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.876843+0000","last_change":"2023-10-07T09:45:37.886728+0000","last_active":"2023-10-07T09:45:38.876843+0000","last_peered":"2023-10-07T09:45:38.876843+0000","last_clean":"2023-10-07T09:45:38.876843+0000","last_became_active":"2023-10-07T09:45:37.886403+0000","last_became_peered":"2023-10-07T09:45:37.886403+0000","last_unstale":"2023-10-07T09:45:38.876843+0000","last_undegraded":"2023-10-07T09:45:38.876843+0000","last_fullsized":"2023-10-07T09:45:38.876843+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:47.046035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896812+0000","last_change":"2023-10-07T09:45:38.896072+0000","last_active":"2023-10-07T09:45:38.896812+0000","last_peered":"2023-10-07T09:45:38.896812+0000","last_clean":"2023-10-07T09:45:38.896812+0000","last_became_active":"2023-10-07T09:45:38.895787+0000","last_became_peered":"2023-10-07T09:45:38.895787+0000","last_unstale":"2023-10-07T09:45:38.896812+0000","last_undegraded":"2023-10-07T09:45:38.896812+0000","last_fullsized":"2023-10-07T09:45:38.896812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:18:33.071732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.871825+0000","last_change":"2023-10-07T09:45:37.872060+0000","last_active":"2023-10-07T09:45:38.871825+0000","last_peered":"2023-10-07T09:45:38.871825+0000","last_clean":"2023-10-07T09:45:38.871825+0000","last_became_active":"2023-10-07T09:45:37.871777+0000","last_became_peered":"2023-10-07T09:45:37.871777+0000","last_unstale":"2023-10-07T09:45:38.871825+0000","last_undegraded":"2023-10-07T09:45:38.871825+0000","last_fullsized":"2023-10-07T09:45:38.871825+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:06:00.078548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899470+0000","last_change":"2023-10-07T09:45:38.887965+0000","last_active":"2023-10-07T09:45:38.899470+0000","last_peered":"2023-10-07T09:45:38.899470+0000","last_clean":"2023-10-07T09:45:38.899470+0000","last_became_active":"2023-10-07T09:45:38.884352+0000","last_became_peered":"2023-10-07T09:45:38.884352+0000","last_unstale":"2023-10-07T09:45:38.899470+0000","last_undegraded":"2023-10-07T09:45:38.899470+0000","last_fullsized":"2023-10-07T09:45:38.899470+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:39:01.454574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.868271+0000","last_change":"2023-10-07T09:45:37.879761+0000","last_active":"2023-10-07T09:45:38.868271+0000","last_peered":"2023-10-07T09:45:38.868271+0000","last_clean":"2023-10-07T09:45:38.868271+0000","last_became_active":"2023-10-07T09:45:37.879511+0000","last_became_peered":"2023-10-07T09:45:37.879511+0000","last_unstale":"2023-10-07T09:45:38.868271+0000","last_undegraded":"2023-10-07T09:45:38.868271+0000","last_fullsized":"2023-10-07T09:45:38.868271+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:00.152453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884763+0000","last_change":"2023-10-07T09:45:38.883144+0000","last_active":"2023-10-07T09:45:38.884763+0000","last_peered":"2023-10-07T09:45:38.884763+0000","last_clean":"2023-10-07T09:45:38.884763+0000","last_became_active":"2023-10-07T09:45:38.882879+0000","last_became_peered":"2023-10-07T09:45:38.882879+0000","last_unstale":"2023-10-07T09:45:38.884763+0000","last_undegraded":"2023-10-07T09:45:38.884763+0000","last_fullsized":"2023-10-07T09:45:38.884763+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:00.339232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.876176+0000","last_change":"2023-10-07T09:45:37.886356+0000","last_active":"2023-10-07T09:45:38.876176+0000","last_peered":"2023-10-07T09:45:38.876176+0000","last_clean":"2023-10-07T09:45:38.876176+0000","last_became_active":"2023-10-07T09:45:37.886059+0000","last_became_peered":"2023-10-07T09:45:37.886059+0000","last_unstale":"2023-10-07T09:45:38.876176+0000","last_undegraded":"2023-10-07T09:45:38.876176+0000","last_fullsized":"2023-10-07T09:45:38.876176+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:56:52.547998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894975+0000","last_change":"2023-10-07T09:45:38.888578+0000","last_active":"2023-10-07T09:45:38.894975+0000","last_peered":"2023-10-07T09:45:38.894975+0000","last_clean":"2023-10-07T09:45:38.894975+0000","last_became_active":"2023-10-07T09:45:38.888279+0000","last_became_peered":"2023-10-07T09:45:38.888279+0000","last_unstale":"2023-10-07T09:45:38.894975+0000","last_undegraded":"2023-10-07T09:45:38.894975+0000","last_fullsized":"2023-10-07T09:45:38.894975+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:18:02.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.868459+0000","last_change":"2023-10-07T09:45:37.882096+0000","last_active":"2023-10-07T09:45:38.868459+0000","last_peered":"2023-10-07T09:45:38.868459+0000","last_clean":"2023-10-07T09:45:38.868459+0000","last_became_active":"2023-10-07T09:45:37.881572+0000","last_became_peered":"2023-10-07T09:45:37.881572+0000","last_unstale":"2023-10-07T09:45:38.868459+0000","last_undegraded":"2023-10-07T09:45:38.868459+0000","last_fullsized":"2023-10-07T09:45:38.868459+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:20.587812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.882253+0000","last_change":"2023-10-07T09:45:38.879934+0000","last_active":"2023-10-07T09:45:38.882253+0000","last_peered":"2023-10-07T09:45:38.882253+0000","last_clean":"2023-10-07T09:45:38.882253+0000","last_became_active":"2023-10-07T09:45:38.879682+0000","last_became_peered":"2023-10-07T09:45:38.879682+0000","last_unstale":"2023-10-07T09:45:38.882253+0000","last_undegraded":"2023-10-07T09:45:38.882253+0000","last_fullsized":"2023-10-07T09:45:38.882253+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:08:29.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899609+0000","last_change":"2023-10-07T09:45:37.880127+0000","last_active":"2023-10-07T09:45:38.899609+0000","last_peered":"2023-10-07T09:45:38.899609+0000","last_clean":"2023-10-07T09:45:38.899609+0000","last_became_active":"2023-10-07T09:45:37.879853+0000","last_became_peered":"2023-10-07T09:45:37.879853+0000","last_unstale":"2023-10-07T09:45:38.899609+0000","last_undegraded":"2023-10-07T09:45:38.899609+0000","last_fullsized":"2023-10-07T09:45:38.899609+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:13:42.180823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"93'205","reported_seq":544,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.869721+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:45:38.869721+0000","last_peered":"2023-10-07T09:45:38.869721+0000","last_clean":"2023-10-07T09:45:38.869721+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:45:38.869721+0000","last_undegraded":"2023-10-07T09:45:38.869721+0000","last_fullsized":"2023-10-07T09:45:38.869721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476932,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303196,"kb_used_data":1812,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068644,"statfs":{"total":96636764160,"available":96326291456,"internally_reserved":0,"allocated":1855488,"data_stored":1191003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.71399999999999997},"min":{"1min":0.45300000000000001,"5min":0.374,"15min":0.32300000000000001},"max":{"1min":0.95399999999999996,"5min":0.96599999999999997,"15min":1.026},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.69099999999999995,"15min":0.71699999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":0.78900000000000003,"5min":0.94899999999999995,"15min":1.0349999999999999},"last":0.76600000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.47899999999999998,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":0.85099999999999998,"5min":0.95599999999999996,"15min":1.014},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.73599999999999999,"15min":0.74099999999999999},"min":{"1min":0.46600000000000003,"5min":0.45900000000000002,"15min":0.45600000000000002},"max":{"1min":0.88,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.78000000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.746},"min":{"1min":0.51100000000000001,"5min":0.41599999999999998,"15min":0.40699999999999997},"max":{"1min":0.89200000000000002,"5min":1.206,"15min":1.206},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71199999999999997,"15min":0.73699999999999999},"min":{"1min":0.5,"5min":0.47899999999999998,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":1.038,"15min":1.071},"last":0.81699999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69899999999999995,"15min":0.73599999999999999},"min":{"1min":0.48399999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":0.92300000000000004,"15min":1.01},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.76500000000000001,"15min":0.76600000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.371},"max":{"1min":0.98699999999999999,"5min":1.137,"15min":1.137},"last":0.67400000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61899999999999999,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39100000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.66900000000000004,"15min":0.65800000000000003},"min":{"1min":0.48599999999999999,"5min":0.39600000000000002,"15min":0.36899999999999999},"max":{"1min":0.92400000000000004,"5min":1.27,"15min":1.27},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.88100000000000001,"5min":1.0069999999999999,"15min":1.1399999999999999},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.751,"15min":0.745},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.72599999999999998,"15min":0.72299999999999998},"min":{"1min":0.59999999999999998,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":1.0109999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.745}]}]},{"osd":6,"up_from":15,"seq":64424509636,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302624,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069216,"statfs":{"total":96636764160,"available":96326877184,"internally_reserved":0,"allocated":1269760,"data_stored":601231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.69099999999999995,"15min":0.71199999999999997},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":1.0780000000000001,"5min":1.274,"15min":1.274},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.45200000000000001,"5min":0.40699999999999997,"15min":0.34999999999999998},"max":{"1min":0.95699999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.74299999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69699999999999995,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":1.042,"5min":1.169,"15min":1.169},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70599999999999996,"15min":0.72599999999999998},"min":{"1min":0.53300000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.02,"5min":1.7,"15min":1.7},"last":0.80000000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":0.98299999999999998,"5min":1.482,"15min":1.482},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.48899999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94499999999999995,"5min":1.643,"15min":1.643},"last":0.76200000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73599999999999999,"15min":0.75700000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":1.458,"15min":1.458},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72799999999999998,"15min":0.754},"min":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.72499999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.621,"15min":0.62},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.29899999999999999},"max":{"1min":0.83499999999999996,"5min":0.92400000000000004,"15min":1.01},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65700000000000003,"15min":0.65800000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.41199999999999998},"max":{"1min":0.94499999999999995,"5min":1.036,"15min":1.0509999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":1.0089999999999999,"15min":1.0449999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.39700000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.749,"15min":0.76100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.99099999999999999,"5min":0.995,"15min":1.01},"last":0.86299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542338,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302624,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069216,"statfs":{"total":96636764160,"available":96326877184,"internally_reserved":0,"allocated":1269760,"data_stored":600759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.34599999999999997,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.871},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.52500000000000002,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.59999999999999998,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45200000000000001,"15min":0.47699999999999998},"min":{"1min":0.32700000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.79600000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.35699999999999998,"15min":0.33600000000000002},"max":{"1min":0.61699999999999999,"5min":0.78400000000000003,"15min":1.028},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56599999999999995,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.97599999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.79100000000000004,"5min":0.79200000000000004,"15min":0.95099999999999996},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.77800000000000002},"min":{"1min":0.54600000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90700000000000003,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.72099999999999997,"15min":0.76600000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.46500000000000002},"max":{"1min":0.82299999999999995,"5min":1.032,"15min":1.196},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.77000000000000002,"15min":0.80600000000000005},"min":{"1min":0.57499999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.89300000000000002,"5min":1.099,"15min":1.2769999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.73599999999999999,"15min":0.78000000000000003},"min":{"1min":0.56000000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.85099999999999998,"5min":1.0680000000000001,"15min":1.3300000000000001},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.80800000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1,"5min":1.0109999999999999,"15min":1.131},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.755,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.87,"5min":1.1040000000000001,"15min":1.165},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.82899999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.2450000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.83599999999999997},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":1.137,"15min":1.385},"last":0.627}]}]},{"osd":0,"up_from":14,"seq":60129542338,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303196,"kb_used_data":1812,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068644,"statfs":{"total":96636764160,"available":96326291456,"internally_reserved":0,"allocated":1855488,"data_stored":1190967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.62,"5min":0.68400000000000005,"15min":0.745},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.55000000000000004,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.63900000000000001,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51700000000000002,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.63900000000000001,"5min":0.876,"15min":1.101},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.48799999999999999,"15min":0.503},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60999999999999999,"5min":0.80900000000000005,"15min":0.92200000000000004},"last":0.378}]},{"osd":3,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50600000000000001,"15min":0.53000000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.83799999999999997,"15min":1.0009999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.81100000000000005,"15min":0.92100000000000004},"last":0.503}]},{"osd":4,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.72799999999999998,"15min":0.755},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85899999999999999,"5min":1.1850000000000001,"15min":1.232},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77900000000000003,"15min":0.79600000000000004},"min":{"1min":0.52300000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0489999999999999,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.86599999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.753,"15min":0.77900000000000003},"min":{"1min":0.505,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.81100000000000005,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.79600000000000004},"min":{"1min":0.54600000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.94699999999999995,"5min":1.0920000000000001,"15min":1.194},"last":0.82799999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.78100000000000003,"15min":0.81000000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.84799999999999998,"5min":1.129,"15min":1.196},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77700000000000002,"15min":0.80800000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.206,"15min":1.206},"last":0.85299999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79100000000000004,"15min":0.82799999999999996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.97099999999999997,"5min":1.167,"15min":1.167},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.79000000000000004,"15min":0.82599999999999996},"min":{"1min":0.56200000000000006,"5min":0.503,"15min":0.503},"max":{"1min":0.90500000000000003,"5min":1.238,"15min":1.238},"last":0.79700000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542339,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310796,"kb_used_data":1220,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061044,"statfs":{"total":96636764160,"available":96318509056,"internally_reserved":0,"allocated":1249280,"data_stored":594455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.441,"15min":0.44900000000000001},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.63700000000000001,"5min":0.81000000000000005,"15min":1.258},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":0.80100000000000005,"15min":1.2290000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":1.1950000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":1.3200000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.501,"15min":0.51500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.94599999999999995,"15min":1.5820000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.69399999999999995,"5min":0.81399999999999995,"15min":1.052},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73399999999999999,"15min":0.75700000000000001},"min":{"1min":0.48099999999999998,"5min":0.46200000000000002,"15min":0.35699999999999998},"max":{"1min":0.81000000000000005,"5min":1.0660000000000001,"15min":1.4630000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93500000000000005,"5min":1.98,"15min":1.98},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.52500000000000002,"5min":0.52400000000000002,"15min":0.46100000000000002},"max":{"1min":0.88800000000000001,"5min":1.208,"15min":1.5169999999999999},"last":0.79000000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":0.877,"5min":1.931,"15min":1.931},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.59099999999999997,"5min":0.52800000000000002,"15min":0.47299999999999998},"max":{"1min":0.93700000000000006,"5min":1.306,"15min":1.8120000000000001},"last":0.77300000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.83899999999999997},"min":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.54200000000000004},"max":{"1min":0.89800000000000002,"5min":1.3520000000000001,"15min":1.3839999999999999},"last":0.80700000000000005}]}]},{"osd":3,"up_from":14,"seq":60129542340,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306720,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065120,"statfs":{"total":96636764160,"available":96322682880,"internally_reserved":0,"allocated":1269760,"data_stored":602101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45200000000000001,"15min":0.45700000000000002},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48299999999999998,"15min":0.50900000000000001},"min":{"1min":0.35399999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.42999999999999999,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.46600000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52300000000000002,"15min":0.54900000000000004},"min":{"1min":0.39900000000000002,"5min":0.317,"15min":0.317},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.437},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54500000000000004,"5min":0.48899999999999999,"15min":0.44900000000000001},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.754,"5min":0.77100000000000002,"15min":0.78400000000000003},"min":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.3009999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.52500000000000002},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78400000000000003,"15min":0.79400000000000004},"min":{"1min":0.56599999999999995,"5min":0.53200000000000003,"15min":0.50900000000000001},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.93600000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.56100000000000005,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80000000000000004,"15min":0.82099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.628,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":2.194,"5min":2.194,"15min":2.194},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.878,"5min":0.81999999999999995,"15min":0.83399999999999996},"min":{"1min":0.58099999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":0.96299999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306708,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065132,"statfs":{"total":96636764160,"available":96322695168,"internally_reserved":0,"allocated":1257472,"data_stored":600565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90000000000000002,"5min":1.069,"15min":1.069},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.69599999999999995,"15min":0.70699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84499999999999997,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.70999999999999996},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.874,"5min":0.96599999999999997,"15min":1.054},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.91900000000000004,"5min":1.478,"15min":1.478},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.74199999999999999,"15min":0.74299999999999999},"min":{"1min":0.55900000000000005,"5min":0.51100000000000001,"15min":0.432},"max":{"1min":1.0680000000000001,"5min":1.518,"15min":1.518},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68500000000000005,"15min":0.71399999999999997},"min":{"1min":0.53100000000000003,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.626}]},{"osd":3,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68999999999999995,"15min":0.71599999999999997},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86299999999999999,"5min":0.96899999999999997,"15min":1.032},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.752},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.036,"15min":1.036},"last":0.64000000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85999999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.627,"15min":0.626},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70799999999999996,"5min":1.639,"15min":1.639},"last":0.40799999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.83599999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69599999999999995,"15min":0.69099999999999995},"min":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94699999999999995,"5min":1.585,"15min":1.585},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69999999999999996},"min":{"1min":0.45900000000000002,"5min":0.45100000000000001,"15min":0.443},"max":{"1min":0.96299999999999997,"5min":1.0049999999999999,"15min":1.0589999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.755,"15min":0.747},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.45200000000000001},"max":{"1min":1.012,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.72799999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302628,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069212,"statfs":{"total":96636764160,"available":96326873088,"internally_reserved":0,"allocated":1273856,"data_stored":602123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76300000000000001,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.49199999999999999,"15min":0.39900000000000002},"max":{"1min":0.94999999999999996,"5min":1.218,"15min":1.218},"last":1.115},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.436},"max":{"1min":0.95499999999999996,"5min":1.294,"15min":1.294},"last":0.91200000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.505,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":1.1339999999999999,"15min":1.3049999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.73299999999999998,"15min":0.75600000000000001},"min":{"1min":0.58199999999999996,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":1.028,"5min":1.194,"15min":6.9720000000000004},"last":0.88300000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71399999999999997,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.119,"15min":5.9299999999999997},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.59899999999999998,"5min":0.48499999999999999,"15min":0.42299999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.704},"last":1.0369999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.60799999999999998,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":0.93799999999999994,"5min":1.2589999999999999,"15min":1.899},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51900000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94999999999999996,"5min":1.036,"15min":6.0620000000000003},"last":0.96299999999999997}]},{"osd":4,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.83499999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.64000000000000001,"15min":0.63600000000000001},"min":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.97499999999999998,"15min":0.998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50900000000000001,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.81599999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.81100000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70699999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":0.83399999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76500000000000001},"min":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.89100000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:45:43.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:45:43.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.524+0000 7f148ffff700 1 Processor -- start 2023-10-07T09:45:43.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.524+0000 7f148ffff700 1 -- start start 2023-10-07T09:45:43.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.525+0000 7f148ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f14880d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.525+0000 7f148ffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880cbf20 con 0x7f1488159750 2023-10-07T09:45:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.525+0000 7f148ffff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880cc200 con 0x7f14880ce8a0 2023-10-07T09:45:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.525+0000 7f148ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cc4e0 con 0x7f1488159b20 2023-10-07T09:45:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1488159750 0x7f14880ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53298/0 (socket says 172.21.15.29:53298) 2023-10-07T09:45:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148dd9b700 1 -- 172.21.15.29:0/3813792525 learned_addr learned my addr 172.21.15.29:0/3813792525 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148d59a700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f14880d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400310573 0 0) 0x7f14880cbf20 con 0x7f1488159750 2023-10-07T09:45:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1474003570 con 0x7f1488159750 2023-10-07T09:45:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205652435 0 0) 0x7f14880cc200 con 0x7f14880ce8a0 2023-10-07T09:45:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14880cbf20 con 0x7f14880ce8a0 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3399015250 0 0) 0x7f1474003570 con 0x7f1488159750 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 msgr2=0x7f14880d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f14880d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 >> v1:172.21.15.144:6789/0 conn(0x7f14880ce8a0 legacy=0x7f14880c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880cc7c0 con 0x7f1488159750 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f14840015f0 con 0x7f1488159750 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148d59a700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f14880d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.526+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 >> v1:172.21.15.29:6789/0 conn(0x7f1488159750 legacy=0x7f14880ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148cd99700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f1474003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148cd99700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f1474004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148cd99700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1474004d50 0x7f14740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cc4e0 con 0x7f14740041b0 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147400bbf0 con 0x7f1474004d50 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147400bdb0 con 0x7f1488159b20 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148d59a700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f1474004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148dd9b700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f1474003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148e59c700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1474004d50 0x7f14740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148d59a700 1 -- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 msgr2=0x7f1474003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148d59a700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1488159b20 0x7f1474003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148d59a700 1 -- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1474004d50 msgr2=0x7f14740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148d59a700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1474004d50 0x7f14740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.527+0000 7f148d59a700 1 -- 172.21.15.29:0/3813792525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478009420 con 0x7f14740041b0 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148d59a700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f1474004810 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1478009db0 tx=0x7f14780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7565034a1fe3920d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478009740 con 0x7f14740041b0 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147800e920 con 0x7f14740041b0 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148cd99700 1 -- 172.21.15.29:0/3813792525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800ece0 con 0x7f14740041b0 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148ffff700 1 -- 172.21.15.29:0/3813792525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 msgr2=0x7f1474004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148ffff700 1 --2- 172.21.15.29:0/3813792525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f1474004810 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1478009db0 tx=0x7f14780091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148e59c700 1 -- 172.21.15.29:0/3813792525 reap_dead start 2023-10-07T09:45:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148ffff700 1 -- 172.21.15.29:0/3813792525 shutdown_connections 2023-10-07T09:45:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.528+0000 7f148ffff700 1 -- 172.21.15.29:0/3813792525 >> 172.21.15.29:0/3813792525 conn(0x7f1488063a10 msgr2=0x7f14880d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.529+0000 7f148ffff700 1 -- 172.21.15.29:0/3813792525 shutdown_connections 2023-10-07T09:45:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.529+0000 7f148ffff700 1 -- 172.21.15.29:0/3813792525 wait complete. 2023-10-07T09:45:43.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.530+0000 7f148ffff700 1 Processor -- start 2023-10-07T09:45:43.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.530+0000 7f148ffff700 1 -- start start 2023-10-07T09:45:43.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f14881582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880c7670 0x7f14880c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1488146c50 0x7f14881470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880c8020 con 0x7f14740041b0 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880b36b0 con 0x7f1488146c50 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880c87f0 con 0x7f14880c7670 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f14881582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f14881582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47230/0 (socket says 172.21.15.29:47230) 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.531+0000 7f148dd9b700 1 -- 172.21.15.29:0/1224861220 learned_addr learned my addr 172.21.15.29:0/1224861220 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148dd9b700 1 -- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880c7670 msgr2=0x7f14880c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148dd9b700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880c7670 0x7f14880c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148dd9b700 1 -- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1488146c50 msgr2=0x7f14881470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148dd9b700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1488146c50 0x7f14881470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148dd9b700 1 -- 172.21.15.29:0/1224861220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1488147600 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148dd9b700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f14881582a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f148400fa00 tx=0x7f1484009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14840073b0 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484007510 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14880c8230 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484005a40 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.532+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14880c8b40 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.533+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148804c3c0 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.534+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f148400d640 con 0x7f14740041b0 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.535+0000 7f147dffb700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f146c066f30 0x7f146c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:43.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.535+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f14840849a0 con 0x7f14740041b0 2023-10-07T09:45:43.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.536+0000 7f148d59a700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f146c066f30 0x7f146c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:43.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.536+0000 7f148d59a700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f146c066f30 0x7f146c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14880c6bd0 tx=0x7f1478019000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:43.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.539+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f14840519f0 con 0x7f14740041b0 2023-10-07T09:45:43.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.736+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1488064040 con 0x7f146c066f30 2023-10-07T09:45:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.745+0000 7f147dffb700 1 -- 172.21.15.29:0/1224861220 <== mgr.6721 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289289 (secure 0 0 0) 0x7f1488064040 con 0x7f146c066f30 2023-10-07T09:45:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:43.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f146c066f30 msgr2=0x7f146c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f146c066f30 0x7f146c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14880c6bd0 tx=0x7f1478019000 comp rx=0 tx=0).stop 2023-10-07T09:45:43.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 msgr2=0x7f14881582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:43.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f14881582a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f148400fa00 tx=0x7f1484009040 comp rx=0 tx=0).stop 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 shutdown_connections 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f146c066f30 0x7f146c0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1488146c50 0x7f14881470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14740041b0 0x7f14881582a0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 --2- 172.21.15.29:0/1224861220 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880c7670 0x7f14880c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 >> 172.21.15.29:0/1224861220 conn(0x7f1488063a10 msgr2=0x7f1488153030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 shutdown_connections 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:43.749+0000 7f148ffff700 1 -- 172.21.15.29:0/1224861220 wait complete. 2023-10-07T09:45:43.752 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:45:43.767 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:45:42.964130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2438492,"kb_used_data":11036,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536228,"statfs":{"total":773094113280,"available":770597097472,"internally_reserved":0,"allocated":11300864,"data_stored":5983204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001479"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894475+0000","last_change":"2023-10-07T09:45:38.886372+0000","last_active":"2023-10-07T09:45:38.894475+0000","last_peered":"2023-10-07T09:45:38.894475+0000","last_clean":"2023-10-07T09:45:38.894475+0000","last_became_active":"2023-10-07T09:45:38.885972+0000","last_became_peered":"2023-10-07T09:45:38.885972+0000","last_unstale":"2023-10-07T09:45:38.894475+0000","last_undegraded":"2023-10-07T09:45:38.894475+0000","last_fullsized":"2023-10-07T09:45:38.894475+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:37:12.898759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894753+0000","last_change":"2023-10-07T09:45:38.886441+0000","last_active":"2023-10-07T09:45:38.894753+0000","last_peered":"2023-10-07T09:45:38.894753+0000","last_clean":"2023-10-07T09:45:38.894753+0000","last_became_active":"2023-10-07T09:45:38.886052+0000","last_became_peered":"2023-10-07T09:45:38.886052+0000","last_unstale":"2023-10-07T09:45:38.894753+0000","last_undegraded":"2023-10-07T09:45:38.894753+0000","last_fullsized":"2023-10-07T09:45:38.894753+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:21.596245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884723+0000","last_change":"2023-10-07T09:45:38.883381+0000","last_active":"2023-10-07T09:45:38.884723+0000","last_peered":"2023-10-07T09:45:38.884723+0000","last_clean":"2023-10-07T09:45:38.884723+0000","last_became_active":"2023-10-07T09:45:38.883012+0000","last_became_peered":"2023-10-07T09:45:38.883012+0000","last_unstale":"2023-10-07T09:45:38.884723+0000","last_undegraded":"2023-10-07T09:45:38.884723+0000","last_fullsized":"2023-10-07T09:45:38.884723+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:02:48.025477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897639+0000","last_change":"2023-10-07T09:45:38.889307+0000","last_active":"2023-10-07T09:45:38.897639+0000","last_peered":"2023-10-07T09:45:38.897639+0000","last_clean":"2023-10-07T09:45:38.897639+0000","last_became_active":"2023-10-07T09:45:38.889007+0000","last_became_peered":"2023-10-07T09:45:38.889007+0000","last_unstale":"2023-10-07T09:45:38.897639+0000","last_undegraded":"2023-10-07T09:45:38.897639+0000","last_fullsized":"2023-10-07T09:45:38.897639+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:30:53.319015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893261+0000","last_change":"2023-10-07T09:45:38.887705+0000","last_active":"2023-10-07T09:45:38.893261+0000","last_peered":"2023-10-07T09:45:38.893261+0000","last_clean":"2023-10-07T09:45:38.893261+0000","last_became_active":"2023-10-07T09:45:38.887432+0000","last_became_peered":"2023-10-07T09:45:38.887432+0000","last_unstale":"2023-10-07T09:45:38.893261+0000","last_undegraded":"2023-10-07T09:45:38.893261+0000","last_fullsized":"2023-10-07T09:45:38.893261+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:11:03.897589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896424+0000","last_change":"2023-10-07T09:45:38.892949+0000","last_active":"2023-10-07T09:45:38.896424+0000","last_peered":"2023-10-07T09:45:38.896424+0000","last_clean":"2023-10-07T09:45:38.896424+0000","last_became_active":"2023-10-07T09:45:38.892694+0000","last_became_peered":"2023-10-07T09:45:38.892694+0000","last_unstale":"2023-10-07T09:45:38.896424+0000","last_undegraded":"2023-10-07T09:45:38.896424+0000","last_fullsized":"2023-10-07T09:45:38.896424+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:32:56.602781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894636+0000","last_change":"2023-10-07T09:45:38.881048+0000","last_active":"2023-10-07T09:45:38.894636+0000","last_peered":"2023-10-07T09:45:38.894636+0000","last_clean":"2023-10-07T09:45:38.894636+0000","last_became_active":"2023-10-07T09:45:38.880776+0000","last_became_peered":"2023-10-07T09:45:38.880776+0000","last_unstale":"2023-10-07T09:45:38.894636+0000","last_undegraded":"2023-10-07T09:45:38.894636+0000","last_fullsized":"2023-10-07T09:45:38.894636+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:34.751650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896482+0000","last_change":"2023-10-07T09:45:38.893493+0000","last_active":"2023-10-07T09:45:38.896482+0000","last_peered":"2023-10-07T09:45:38.896482+0000","last_clean":"2023-10-07T09:45:38.896482+0000","last_became_active":"2023-10-07T09:45:38.893216+0000","last_became_peered":"2023-10-07T09:45:38.893216+0000","last_unstale":"2023-10-07T09:45:38.896482+0000","last_undegraded":"2023-10-07T09:45:38.896482+0000","last_fullsized":"2023-10-07T09:45:38.896482+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:53.816169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892973+0000","last_change":"2023-10-07T09:45:38.886964+0000","last_active":"2023-10-07T09:45:38.892973+0000","last_peered":"2023-10-07T09:45:38.892973+0000","last_clean":"2023-10-07T09:45:38.892973+0000","last_became_active":"2023-10-07T09:45:38.886702+0000","last_became_peered":"2023-10-07T09:45:38.886702+0000","last_unstale":"2023-10-07T09:45:38.892973+0000","last_undegraded":"2023-10-07T09:45:38.892973+0000","last_fullsized":"2023-10-07T09:45:38.892973+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:43:40.383305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896269+0000","last_change":"2023-10-07T09:45:38.894801+0000","last_active":"2023-10-07T09:45:38.896269+0000","last_peered":"2023-10-07T09:45:38.896269+0000","last_clean":"2023-10-07T09:45:38.896269+0000","last_became_active":"2023-10-07T09:45:38.894529+0000","last_became_peered":"2023-10-07T09:45:38.894529+0000","last_unstale":"2023-10-07T09:45:38.896269+0000","last_undegraded":"2023-10-07T09:45:38.896269+0000","last_fullsized":"2023-10-07T09:45:38.896269+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:52:53.997248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.881812+0000","last_change":"2023-10-07T09:45:38.876623+0000","last_active":"2023-10-07T09:45:38.881812+0000","last_peered":"2023-10-07T09:45:38.881812+0000","last_clean":"2023-10-07T09:45:38.881812+0000","last_became_active":"2023-10-07T09:45:38.876354+0000","last_became_peered":"2023-10-07T09:45:38.876354+0000","last_unstale":"2023-10-07T09:45:38.881812+0000","last_undegraded":"2023-10-07T09:45:38.881812+0000","last_fullsized":"2023-10-07T09:45:38.881812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:33.411089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896434+0000","last_change":"2023-10-07T09:45:38.895306+0000","last_active":"2023-10-07T09:45:38.896434+0000","last_peered":"2023-10-07T09:45:38.896434+0000","last_clean":"2023-10-07T09:45:38.896434+0000","last_became_active":"2023-10-07T09:45:38.895003+0000","last_became_peered":"2023-10-07T09:45:38.895003+0000","last_unstale":"2023-10-07T09:45:38.896434+0000","last_undegraded":"2023-10-07T09:45:38.896434+0000","last_fullsized":"2023-10-07T09:45:38.896434+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:09:37.743358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.883939+0000","last_change":"2023-10-07T09:45:38.880791+0000","last_active":"2023-10-07T09:45:38.883939+0000","last_peered":"2023-10-07T09:45:38.883939+0000","last_clean":"2023-10-07T09:45:38.883939+0000","last_became_active":"2023-10-07T09:45:38.880403+0000","last_became_peered":"2023-10-07T09:45:38.880403+0000","last_unstale":"2023-10-07T09:45:38.883939+0000","last_undegraded":"2023-10-07T09:45:38.883939+0000","last_fullsized":"2023-10-07T09:45:38.883939+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:21:10.049308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897738+0000","last_change":"2023-10-07T09:45:38.891245+0000","last_active":"2023-10-07T09:45:38.897738+0000","last_peered":"2023-10-07T09:45:38.897738+0000","last_clean":"2023-10-07T09:45:38.897738+0000","last_became_active":"2023-10-07T09:45:38.890644+0000","last_became_peered":"2023-10-07T09:45:38.890644+0000","last_unstale":"2023-10-07T09:45:38.897738+0000","last_undegraded":"2023-10-07T09:45:38.897738+0000","last_fullsized":"2023-10-07T09:45:38.897738+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:20.367513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891216+0000","last_change":"2023-10-07T09:45:38.875458+0000","last_active":"2023-10-07T09:45:38.891216+0000","last_peered":"2023-10-07T09:45:38.891216+0000","last_clean":"2023-10-07T09:45:38.891216+0000","last_became_active":"2023-10-07T09:45:38.874814+0000","last_became_peered":"2023-10-07T09:45:38.874814+0000","last_unstale":"2023-10-07T09:45:38.891216+0000","last_undegraded":"2023-10-07T09:45:38.891216+0000","last_fullsized":"2023-10-07T09:45:38.891216+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:42.306675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.890965+0000","last_change":"2023-10-07T09:45:38.883256+0000","last_active":"2023-10-07T09:45:38.890965+0000","last_peered":"2023-10-07T09:45:38.890965+0000","last_clean":"2023-10-07T09:45:38.890965+0000","last_became_active":"2023-10-07T09:45:38.883020+0000","last_became_peered":"2023-10-07T09:45:38.883020+0000","last_unstale":"2023-10-07T09:45:38.890965+0000","last_undegraded":"2023-10-07T09:45:38.890965+0000","last_fullsized":"2023-10-07T09:45:38.890965+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:00:48.224115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892412+0000","last_change":"2023-10-07T09:45:38.890588+0000","last_active":"2023-10-07T09:45:38.892412+0000","last_peered":"2023-10-07T09:45:38.892412+0000","last_clean":"2023-10-07T09:45:38.892412+0000","last_became_active":"2023-10-07T09:45:38.890055+0000","last_became_peered":"2023-10-07T09:45:38.890055+0000","last_unstale":"2023-10-07T09:45:38.892412+0000","last_undegraded":"2023-10-07T09:45:38.892412+0000","last_fullsized":"2023-10-07T09:45:38.892412+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:32:56.602781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897773+0000","last_change":"2023-10-07T09:45:38.888562+0000","last_active":"2023-10-07T09:45:38.897773+0000","last_peered":"2023-10-07T09:45:38.897773+0000","last_clean":"2023-10-07T09:45:38.897773+0000","last_became_active":"2023-10-07T09:45:38.888254+0000","last_became_peered":"2023-10-07T09:45:38.888254+0000","last_unstale":"2023-10-07T09:45:38.897773+0000","last_undegraded":"2023-10-07T09:45:38.897773+0000","last_fullsized":"2023-10-07T09:45:38.897773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:29:40.247224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899870+0000","last_change":"2023-10-07T09:45:38.894148+0000","last_active":"2023-10-07T09:45:38.899870+0000","last_peered":"2023-10-07T09:45:38.899870+0000","last_clean":"2023-10-07T09:45:38.899870+0000","last_became_active":"2023-10-07T09:45:38.893861+0000","last_became_peered":"2023-10-07T09:45:38.893861+0000","last_unstale":"2023-10-07T09:45:38.899870+0000","last_undegraded":"2023-10-07T09:45:38.899870+0000","last_fullsized":"2023-10-07T09:45:38.899870+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:56:53.556431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.900882+0000","last_change":"2023-10-07T09:45:38.898861+0000","last_active":"2023-10-07T09:45:38.900882+0000","last_peered":"2023-10-07T09:45:38.900882+0000","last_clean":"2023-10-07T09:45:38.900882+0000","last_became_active":"2023-10-07T09:45:38.898578+0000","last_became_peered":"2023-10-07T09:45:38.898578+0000","last_unstale":"2023-10-07T09:45:38.900882+0000","last_undegraded":"2023-10-07T09:45:38.900882+0000","last_fullsized":"2023-10-07T09:45:38.900882+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:02:26.040494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899773+0000","last_change":"2023-10-07T09:45:38.898801+0000","last_active":"2023-10-07T09:45:38.899773+0000","last_peered":"2023-10-07T09:45:38.899773+0000","last_clean":"2023-10-07T09:45:38.899773+0000","last_became_active":"2023-10-07T09:45:38.898550+0000","last_became_peered":"2023-10-07T09:45:38.898550+0000","last_unstale":"2023-10-07T09:45:38.899773+0000","last_undegraded":"2023-10-07T09:45:38.899773+0000","last_fullsized":"2023-10-07T09:45:38.899773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:02:07.427536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894531+0000","last_change":"2023-10-07T09:45:38.885026+0000","last_active":"2023-10-07T09:45:38.894531+0000","last_peered":"2023-10-07T09:45:38.894531+0000","last_clean":"2023-10-07T09:45:38.894531+0000","last_became_active":"2023-10-07T09:45:38.884764+0000","last_became_peered":"2023-10-07T09:45:38.884764+0000","last_unstale":"2023-10-07T09:45:38.894531+0000","last_undegraded":"2023-10-07T09:45:38.894531+0000","last_fullsized":"2023-10-07T09:45:38.894531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:40:40.367333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893224+0000","last_change":"2023-10-07T09:45:38.885885+0000","last_active":"2023-10-07T09:45:38.893224+0000","last_peered":"2023-10-07T09:45:38.893224+0000","last_clean":"2023-10-07T09:45:38.893224+0000","last_became_active":"2023-10-07T09:45:38.885601+0000","last_became_peered":"2023-10-07T09:45:38.885601+0000","last_unstale":"2023-10-07T09:45:38.893224+0000","last_undegraded":"2023-10-07T09:45:38.893224+0000","last_fullsized":"2023-10-07T09:45:38.893224+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:21:11.460981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.900078+0000","last_change":"2023-10-07T09:45:38.898714+0000","last_active":"2023-10-07T09:45:38.900078+0000","last_peered":"2023-10-07T09:45:38.900078+0000","last_clean":"2023-10-07T09:45:38.900078+0000","last_became_active":"2023-10-07T09:45:38.898435+0000","last_became_peered":"2023-10-07T09:45:38.898435+0000","last_unstale":"2023-10-07T09:45:38.900078+0000","last_undegraded":"2023-10-07T09:45:38.900078+0000","last_fullsized":"2023-10-07T09:45:38.900078+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:04.408429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894788+0000","last_change":"2023-10-07T09:45:38.886751+0000","last_active":"2023-10-07T09:45:38.894788+0000","last_peered":"2023-10-07T09:45:38.894788+0000","last_clean":"2023-10-07T09:45:38.894788+0000","last_became_active":"2023-10-07T09:45:38.885802+0000","last_became_peered":"2023-10-07T09:45:38.885802+0000","last_unstale":"2023-10-07T09:45:38.894788+0000","last_undegraded":"2023-10-07T09:45:38.894788+0000","last_fullsized":"2023-10-07T09:45:38.894788+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:42.716793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884758+0000","last_change":"2023-10-07T09:45:38.883306+0000","last_active":"2023-10-07T09:45:38.884758+0000","last_peered":"2023-10-07T09:45:38.884758+0000","last_clean":"2023-10-07T09:45:38.884758+0000","last_became_active":"2023-10-07T09:45:38.883036+0000","last_became_peered":"2023-10-07T09:45:38.883036+0000","last_unstale":"2023-10-07T09:45:38.884758+0000","last_undegraded":"2023-10-07T09:45:38.884758+0000","last_fullsized":"2023-10-07T09:45:38.884758+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:33:10.749620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.880245+0000","last_change":"2023-10-07T09:45:38.878155+0000","last_active":"2023-10-07T09:45:38.880245+0000","last_peered":"2023-10-07T09:45:38.880245+0000","last_clean":"2023-10-07T09:45:38.880245+0000","last_became_active":"2023-10-07T09:45:38.877829+0000","last_became_peered":"2023-10-07T09:45:38.877829+0000","last_unstale":"2023-10-07T09:45:38.880245+0000","last_undegraded":"2023-10-07T09:45:38.880245+0000","last_fullsized":"2023-10-07T09:45:38.880245+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:18:33.071732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893252+0000","last_change":"2023-10-07T09:45:38.886496+0000","last_active":"2023-10-07T09:45:38.893252+0000","last_peered":"2023-10-07T09:45:38.893252+0000","last_clean":"2023-10-07T09:45:38.893252+0000","last_became_active":"2023-10-07T09:45:38.886131+0000","last_became_peered":"2023-10-07T09:45:38.886131+0000","last_unstale":"2023-10-07T09:45:38.893252+0000","last_undegraded":"2023-10-07T09:45:38.893252+0000","last_fullsized":"2023-10-07T09:45:38.893252+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:58.436253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899304+0000","last_change":"2023-10-07T09:45:38.894170+0000","last_active":"2023-10-07T09:45:38.899304+0000","last_peered":"2023-10-07T09:45:38.899304+0000","last_clean":"2023-10-07T09:45:38.899304+0000","last_became_active":"2023-10-07T09:45:38.893895+0000","last_became_peered":"2023-10-07T09:45:38.893895+0000","last_unstale":"2023-10-07T09:45:38.899304+0000","last_undegraded":"2023-10-07T09:45:38.899304+0000","last_fullsized":"2023-10-07T09:45:38.899304+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:06.060682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892939+0000","last_change":"2023-10-07T09:45:38.883079+0000","last_active":"2023-10-07T09:45:38.892939+0000","last_peered":"2023-10-07T09:45:38.892939+0000","last_clean":"2023-10-07T09:45:38.892939+0000","last_became_active":"2023-10-07T09:45:38.882533+0000","last_became_peered":"2023-10-07T09:45:38.882533+0000","last_unstale":"2023-10-07T09:45:38.892939+0000","last_undegraded":"2023-10-07T09:45:38.892939+0000","last_fullsized":"2023-10-07T09:45:38.892939+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:59:05.924980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891407+0000","last_change":"2023-10-07T09:45:38.884839+0000","last_active":"2023-10-07T09:45:38.891407+0000","last_peered":"2023-10-07T09:45:38.891407+0000","last_clean":"2023-10-07T09:45:38.891407+0000","last_became_active":"2023-10-07T09:45:38.884560+0000","last_became_peered":"2023-10-07T09:45:38.884560+0000","last_unstale":"2023-10-07T09:45:38.891407+0000","last_undegraded":"2023-10-07T09:45:38.891407+0000","last_fullsized":"2023-10-07T09:45:38.891407+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:07:45.181900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893194+0000","last_change":"2023-10-07T09:45:38.886581+0000","last_active":"2023-10-07T09:45:38.893194+0000","last_peered":"2023-10-07T09:45:38.893194+0000","last_clean":"2023-10-07T09:45:38.893194+0000","last_became_active":"2023-10-07T09:45:38.886045+0000","last_became_peered":"2023-10-07T09:45:38.886045+0000","last_unstale":"2023-10-07T09:45:38.893194+0000","last_undegraded":"2023-10-07T09:45:38.893194+0000","last_fullsized":"2023-10-07T09:45:38.893194+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:30.225686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891074+0000","last_change":"2023-10-07T09:45:38.882100+0000","last_active":"2023-10-07T09:45:38.891074+0000","last_peered":"2023-10-07T09:45:38.891074+0000","last_clean":"2023-10-07T09:45:38.891074+0000","last_became_active":"2023-10-07T09:45:38.881856+0000","last_became_peered":"2023-10-07T09:45:38.881856+0000","last_unstale":"2023-10-07T09:45:38.891074+0000","last_undegraded":"2023-10-07T09:45:38.891074+0000","last_fullsized":"2023-10-07T09:45:38.891074+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:30.225686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"102'4","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898506+0000","last_change":"2023-10-07T09:45:37.885464+0000","last_active":"2023-10-07T09:45:38.898506+0000","last_peered":"2023-10-07T09:45:38.898506+0000","last_clean":"2023-10-07T09:45:38.898506+0000","last_became_active":"2023-10-07T09:45:37.885148+0000","last_became_peered":"2023-10-07T09:45:37.885148+0000","last_unstale":"2023-10-07T09:45:38.898506+0000","last_undegraded":"2023-10-07T09:45:38.898506+0000","last_fullsized":"2023-10-07T09:45:38.898506+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:36:01.656048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897888+0000","last_change":"2023-10-07T09:45:38.897426+0000","last_active":"2023-10-07T09:45:38.897888+0000","last_peered":"2023-10-07T09:45:38.897888+0000","last_clean":"2023-10-07T09:45:38.897888+0000","last_became_active":"2023-10-07T09:45:38.897111+0000","last_became_peered":"2023-10-07T09:45:38.897111+0000","last_unstale":"2023-10-07T09:45:38.897888+0000","last_undegraded":"2023-10-07T09:45:38.897888+0000","last_fullsized":"2023-10-07T09:45:38.897888+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:10.381540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898125+0000","last_change":"2023-10-07T09:45:37.887295+0000","last_active":"2023-10-07T09:45:38.898125+0000","last_peered":"2023-10-07T09:45:38.898125+0000","last_clean":"2023-10-07T09:45:38.898125+0000","last_became_active":"2023-10-07T09:45:37.886437+0000","last_became_peered":"2023-10-07T09:45:37.886437+0000","last_unstale":"2023-10-07T09:45:38.898125+0000","last_undegraded":"2023-10-07T09:45:38.898125+0000","last_fullsized":"2023-10-07T09:45:38.898125+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:40:39.358901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893271+0000","last_change":"2023-10-07T09:45:38.886842+0000","last_active":"2023-10-07T09:45:38.893271+0000","last_peered":"2023-10-07T09:45:38.893271+0000","last_clean":"2023-10-07T09:45:38.893271+0000","last_became_active":"2023-10-07T09:45:38.886577+0000","last_became_peered":"2023-10-07T09:45:38.886577+0000","last_unstale":"2023-10-07T09:45:38.893271+0000","last_undegraded":"2023-10-07T09:45:38.893271+0000","last_fullsized":"2023-10-07T09:45:38.893271+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:42.306675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899824+0000","last_change":"2023-10-07T09:45:37.878828+0000","last_active":"2023-10-07T09:45:38.899824+0000","last_peered":"2023-10-07T09:45:38.899824+0000","last_clean":"2023-10-07T09:45:38.899824+0000","last_became_active":"2023-10-07T09:45:37.878563+0000","last_became_peered":"2023-10-07T09:45:37.878563+0000","last_unstale":"2023-10-07T09:45:38.899824+0000","last_undegraded":"2023-10-07T09:45:38.899824+0000","last_fullsized":"2023-10-07T09:45:38.899824+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:19:33.995655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884266+0000","last_change":"2023-10-07T09:45:38.882087+0000","last_active":"2023-10-07T09:45:38.884266+0000","last_peered":"2023-10-07T09:45:38.884266+0000","last_clean":"2023-10-07T09:45:38.884266+0000","last_became_active":"2023-10-07T09:45:38.881661+0000","last_became_peered":"2023-10-07T09:45:38.881661+0000","last_unstale":"2023-10-07T09:45:38.884266+0000","last_undegraded":"2023-10-07T09:45:38.884266+0000","last_fullsized":"2023-10-07T09:45:38.884266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:14.847690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899280+0000","last_change":"2023-10-07T09:45:37.877726+0000","last_active":"2023-10-07T09:45:38.899280+0000","last_peered":"2023-10-07T09:45:38.899280+0000","last_clean":"2023-10-07T09:45:38.899280+0000","last_became_active":"2023-10-07T09:45:37.877451+0000","last_became_peered":"2023-10-07T09:45:37.877451+0000","last_unstale":"2023-10-07T09:45:38.899280+0000","last_undegraded":"2023-10-07T09:45:38.899280+0000","last_fullsized":"2023-10-07T09:45:38.899280+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:59:04.916547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892919+0000","last_change":"2023-10-07T09:45:38.882805+0000","last_active":"2023-10-07T09:45:38.892919+0000","last_peered":"2023-10-07T09:45:38.892919+0000","last_clean":"2023-10-07T09:45:38.892919+0000","last_became_active":"2023-10-07T09:45:38.882106+0000","last_became_peered":"2023-10-07T09:45:38.882106+0000","last_unstale":"2023-10-07T09:45:38.892919+0000","last_undegraded":"2023-10-07T09:45:38.892919+0000","last_fullsized":"2023-10-07T09:45:38.892919+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:06:01.086981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895062+0000","last_change":"2023-10-07T09:45:37.882623+0000","last_active":"2023-10-07T09:45:38.895062+0000","last_peered":"2023-10-07T09:45:38.895062+0000","last_clean":"2023-10-07T09:45:38.895062+0000","last_became_active":"2023-10-07T09:45:37.882363+0000","last_became_peered":"2023-10-07T09:45:37.882363+0000","last_unstale":"2023-10-07T09:45:38.895062+0000","last_undegraded":"2023-10-07T09:45:38.895062+0000","last_fullsized":"2023-10-07T09:45:38.895062+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:32.402656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893241+0000","last_change":"2023-10-07T09:45:38.887642+0000","last_active":"2023-10-07T09:45:38.893241+0000","last_peered":"2023-10-07T09:45:38.893241+0000","last_clean":"2023-10-07T09:45:38.893241+0000","last_became_active":"2023-10-07T09:45:38.887359+0000","last_became_peered":"2023-10-07T09:45:38.887359+0000","last_unstale":"2023-10-07T09:45:38.893241+0000","last_undegraded":"2023-10-07T09:45:38.893241+0000","last_fullsized":"2023-10-07T09:45:38.893241+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:31:35.952352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898151+0000","last_change":"2023-10-07T09:45:37.879843+0000","last_active":"2023-10-07T09:45:38.898151+0000","last_peered":"2023-10-07T09:45:38.898151+0000","last_clean":"2023-10-07T09:45:38.898151+0000","last_became_active":"2023-10-07T09:45:37.879595+0000","last_became_peered":"2023-10-07T09:45:37.879595+0000","last_unstale":"2023-10-07T09:45:38.898151+0000","last_undegraded":"2023-10-07T09:45:38.898151+0000","last_fullsized":"2023-10-07T09:45:38.898151+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:04.097156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895900+0000","last_change":"2023-10-07T09:45:38.893518+0000","last_active":"2023-10-07T09:45:38.895900+0000","last_peered":"2023-10-07T09:45:38.895900+0000","last_clean":"2023-10-07T09:45:38.895900+0000","last_became_active":"2023-10-07T09:45:38.892643+0000","last_became_peered":"2023-10-07T09:45:38.892643+0000","last_unstale":"2023-10-07T09:45:38.895900+0000","last_undegraded":"2023-10-07T09:45:38.895900+0000","last_fullsized":"2023-10-07T09:45:38.895900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:54:52.797058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.871139+0000","last_change":"2023-10-07T09:45:37.887004+0000","last_active":"2023-10-07T09:45:38.871139+0000","last_peered":"2023-10-07T09:45:38.871139+0000","last_clean":"2023-10-07T09:45:38.871139+0000","last_became_active":"2023-10-07T09:45:37.886730+0000","last_became_peered":"2023-10-07T09:45:37.886730+0000","last_unstale":"2023-10-07T09:45:38.871139+0000","last_undegraded":"2023-10-07T09:45:38.871139+0000","last_fullsized":"2023-10-07T09:45:38.871139+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:25.332188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891235+0000","last_change":"2023-10-07T09:45:38.875634+0000","last_active":"2023-10-07T09:45:38.891235+0000","last_peered":"2023-10-07T09:45:38.891235+0000","last_clean":"2023-10-07T09:45:38.891235+0000","last_became_active":"2023-10-07T09:45:38.875208+0000","last_became_peered":"2023-10-07T09:45:38.875208+0000","last_unstale":"2023-10-07T09:45:38.891235+0000","last_undegraded":"2023-10-07T09:45:38.891235+0000","last_fullsized":"2023-10-07T09:45:38.891235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:52:30.225686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.868314+0000","last_change":"2023-10-07T09:45:37.879446+0000","last_active":"2023-10-07T09:45:38.868314+0000","last_peered":"2023-10-07T09:45:38.868314+0000","last_clean":"2023-10-07T09:45:38.868314+0000","last_became_active":"2023-10-07T09:45:37.879011+0000","last_became_peered":"2023-10-07T09:45:37.879011+0000","last_unstale":"2023-10-07T09:45:38.868314+0000","last_undegraded":"2023-10-07T09:45:38.868314+0000","last_fullsized":"2023-10-07T09:45:38.868314+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:02:25.032061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891452+0000","last_change":"2023-10-07T09:45:38.874236+0000","last_active":"2023-10-07T09:45:38.891452+0000","last_peered":"2023-10-07T09:45:38.891452+0000","last_clean":"2023-10-07T09:45:38.891452+0000","last_became_active":"2023-10-07T09:45:38.873880+0000","last_became_peered":"2023-10-07T09:45:38.873880+0000","last_unstale":"2023-10-07T09:45:38.891452+0000","last_undegraded":"2023-10-07T09:45:38.891452+0000","last_fullsized":"2023-10-07T09:45:38.891452+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:00:48.224115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.878765+0000","last_change":"2023-10-07T09:45:37.885958+0000","last_active":"2023-10-07T09:45:38.878765+0000","last_peered":"2023-10-07T09:45:38.878765+0000","last_clean":"2023-10-07T09:45:38.878765+0000","last_became_active":"2023-10-07T09:45:37.885675+0000","last_became_peered":"2023-10-07T09:45:37.885675+0000","last_unstale":"2023-10-07T09:45:38.878765+0000","last_undegraded":"2023-10-07T09:45:38.878765+0000","last_fullsized":"2023-10-07T09:45:38.878765+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:32.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895119+0000","last_change":"2023-10-07T09:45:38.892445+0000","last_active":"2023-10-07T09:45:38.895119+0000","last_peered":"2023-10-07T09:45:38.895119+0000","last_clean":"2023-10-07T09:45:38.895119+0000","last_became_active":"2023-10-07T09:45:38.892162+0000","last_became_peered":"2023-10-07T09:45:38.892162+0000","last_unstale":"2023-10-07T09:45:38.895119+0000","last_undegraded":"2023-10-07T09:45:38.895119+0000","last_fullsized":"2023-10-07T09:45:38.895119+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:55.429915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.877494+0000","last_change":"2023-10-07T09:45:37.885087+0000","last_active":"2023-10-07T09:45:38.877494+0000","last_peered":"2023-10-07T09:45:38.877494+0000","last_clean":"2023-10-07T09:45:38.877494+0000","last_became_active":"2023-10-07T09:45:37.884818+0000","last_became_peered":"2023-10-07T09:45:37.884818+0000","last_unstale":"2023-10-07T09:45:38.877494+0000","last_undegraded":"2023-10-07T09:45:38.877494+0000","last_fullsized":"2023-10-07T09:45:38.877494+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:39:00.446142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899327+0000","last_change":"2023-10-07T09:45:38.889341+0000","last_active":"2023-10-07T09:45:38.899327+0000","last_peered":"2023-10-07T09:45:38.899327+0000","last_clean":"2023-10-07T09:45:38.899327+0000","last_became_active":"2023-10-07T09:45:38.889046+0000","last_became_peered":"2023-10-07T09:45:38.889046+0000","last_unstale":"2023-10-07T09:45:38.899327+0000","last_undegraded":"2023-10-07T09:45:38.899327+0000","last_fullsized":"2023-10-07T09:45:38.899327+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:19:14.474121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"102'3","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899707+0000","last_change":"2023-10-07T09:45:37.884692+0000","last_active":"2023-10-07T09:45:38.899707+0000","last_peered":"2023-10-07T09:45:38.899707+0000","last_clean":"2023-10-07T09:45:38.899707+0000","last_became_active":"2023-10-07T09:45:37.884389+0000","last_became_peered":"2023-10-07T09:45:37.884389+0000","last_unstale":"2023-10-07T09:45:38.899707+0000","last_undegraded":"2023-10-07T09:45:38.899707+0000","last_fullsized":"2023-10-07T09:45:38.899707+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:27.004476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895657+0000","last_change":"2023-10-07T09:45:38.894234+0000","last_active":"2023-10-07T09:45:38.895657+0000","last_peered":"2023-10-07T09:45:38.895657+0000","last_clean":"2023-10-07T09:45:38.895657+0000","last_became_active":"2023-10-07T09:45:38.893840+0000","last_became_peered":"2023-10-07T09:45:38.893840+0000","last_unstale":"2023-10-07T09:45:38.895657+0000","last_undegraded":"2023-10-07T09:45:38.895657+0000","last_fullsized":"2023-10-07T09:45:38.895657+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:15:53.816169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"102'6","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.901846+0000","last_change":"2023-10-07T09:45:37.887177+0000","last_active":"2023-10-07T09:45:38.901846+0000","last_peered":"2023-10-07T09:45:38.901846+0000","last_clean":"2023-10-07T09:45:38.901846+0000","last_became_active":"2023-10-07T09:45:37.886808+0000","last_became_peered":"2023-10-07T09:45:37.886808+0000","last_unstale":"2023-10-07T09:45:38.901846+0000","last_undegraded":"2023-10-07T09:45:38.901846+0000","last_fullsized":"2023-10-07T09:45:38.901846+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:47.046035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899198+0000","last_change":"2023-10-07T09:45:38.885066+0000","last_active":"2023-10-07T09:45:38.899198+0000","last_peered":"2023-10-07T09:45:38.899198+0000","last_clean":"2023-10-07T09:45:38.899198+0000","last_became_active":"2023-10-07T09:45:38.883834+0000","last_became_peered":"2023-10-07T09:45:38.883834+0000","last_unstale":"2023-10-07T09:45:38.899198+0000","last_undegraded":"2023-10-07T09:45:38.899198+0000","last_fullsized":"2023-10-07T09:45:38.899198+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:48.054468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.900491+0000","last_change":"2023-10-07T09:45:37.877618+0000","last_active":"2023-10-07T09:45:38.900491+0000","last_peered":"2023-10-07T09:45:38.900491+0000","last_clean":"2023-10-07T09:45:38.900491+0000","last_became_active":"2023-10-07T09:45:37.877083+0000","last_became_peered":"2023-10-07T09:45:37.877083+0000","last_unstale":"2023-10-07T09:45:38.900491+0000","last_undegraded":"2023-10-07T09:45:38.900491+0000","last_fullsized":"2023-10-07T09:45:38.900491+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:30:52.310582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893372+0000","last_change":"2023-10-07T09:45:38.891324+0000","last_active":"2023-10-07T09:45:38.893372+0000","last_peered":"2023-10-07T09:45:38.893372+0000","last_clean":"2023-10-07T09:45:38.893372+0000","last_became_active":"2023-10-07T09:45:38.891010+0000","last_became_peered":"2023-10-07T09:45:38.891010+0000","last_unstale":"2023-10-07T09:45:38.893372+0000","last_undegraded":"2023-10-07T09:45:38.893372+0000","last_fullsized":"2023-10-07T09:45:38.893372+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:34:06.906408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.864862+0000","last_change":"2023-10-07T09:45:37.876112+0000","last_active":"2023-10-07T09:45:38.864862+0000","last_peered":"2023-10-07T09:45:38.864862+0000","last_clean":"2023-10-07T09:45:38.864862+0000","last_became_active":"2023-10-07T09:45:37.875835+0000","last_became_peered":"2023-10-07T09:45:37.875835+0000","last_unstale":"2023-10-07T09:45:38.864862+0000","last_undegraded":"2023-10-07T09:45:38.864862+0000","last_fullsized":"2023-10-07T09:45:38.864862+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:48:53.197259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892451+0000","last_change":"2023-10-07T09:45:38.882410+0000","last_active":"2023-10-07T09:45:38.892451+0000","last_peered":"2023-10-07T09:45:38.892451+0000","last_clean":"2023-10-07T09:45:38.892451+0000","last_became_active":"2023-10-07T09:45:38.882111+0000","last_became_peered":"2023-10-07T09:45:38.882111+0000","last_unstale":"2023-10-07T09:45:38.892451+0000","last_undegraded":"2023-10-07T09:45:38.892451+0000","last_fullsized":"2023-10-07T09:45:38.892451+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:01.160885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.873897+0000","last_change":"2023-10-07T09:45:37.885798+0000","last_active":"2023-10-07T09:45:38.873897+0000","last_peered":"2023-10-07T09:45:38.873897+0000","last_clean":"2023-10-07T09:45:38.873897+0000","last_became_active":"2023-10-07T09:45:37.885452+0000","last_became_peered":"2023-10-07T09:45:37.885452+0000","last_unstale":"2023-10-07T09:45:38.873897+0000","last_undegraded":"2023-10-07T09:45:38.873897+0000","last_fullsized":"2023-10-07T09:45:38.873897+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:39:00.446142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.882576+0000","last_change":"2023-10-07T09:45:38.880813+0000","last_active":"2023-10-07T09:45:38.882576+0000","last_peered":"2023-10-07T09:45:38.882576+0000","last_clean":"2023-10-07T09:45:38.882576+0000","last_became_active":"2023-10-07T09:45:38.880574+0000","last_became_peered":"2023-10-07T09:45:38.880574+0000","last_unstale":"2023-10-07T09:45:38.882576+0000","last_undegraded":"2023-10-07T09:45:38.882576+0000","last_fullsized":"2023-10-07T09:45:38.882576+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:05.105589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"102'8","reported_seq":18,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899486+0000","last_change":"2023-10-07T09:45:37.872059+0000","last_active":"2023-10-07T09:45:38.899486+0000","last_peered":"2023-10-07T09:45:38.899486+0000","last_clean":"2023-10-07T09:45:38.899486+0000","last_became_active":"2023-10-07T09:45:37.871763+0000","last_became_peered":"2023-10-07T09:45:37.871763+0000","last_unstale":"2023-10-07T09:45:38.899486+0000","last_undegraded":"2023-10-07T09:45:38.899486+0000","last_fullsized":"2023-10-07T09:45:38.899486+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:32.402656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.883967+0000","last_change":"2023-10-07T09:45:38.880669+0000","last_active":"2023-10-07T09:45:38.883967+0000","last_peered":"2023-10-07T09:45:38.883967+0000","last_clean":"2023-10-07T09:45:38.883967+0000","last_became_active":"2023-10-07T09:45:38.880378+0000","last_became_peered":"2023-10-07T09:45:38.880378+0000","last_unstale":"2023-10-07T09:45:38.883967+0000","last_undegraded":"2023-10-07T09:45:38.883967+0000","last_fullsized":"2023-10-07T09:45:38.883967+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:34.472605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896718+0000","last_change":"2023-10-07T09:45:37.888660+0000","last_active":"2023-10-07T09:45:38.896718+0000","last_peered":"2023-10-07T09:45:38.896718+0000","last_clean":"2023-10-07T09:45:38.896718+0000","last_became_active":"2023-10-07T09:45:37.888340+0000","last_became_peered":"2023-10-07T09:45:37.888340+0000","last_unstale":"2023-10-07T09:45:38.896718+0000","last_undegraded":"2023-10-07T09:45:38.896718+0000","last_fullsized":"2023-10-07T09:45:38.896718+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:19:13.465689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894494+0000","last_change":"2023-10-07T09:45:38.887611+0000","last_active":"2023-10-07T09:45:38.894494+0000","last_peered":"2023-10-07T09:45:38.894494+0000","last_clean":"2023-10-07T09:45:38.894494+0000","last_became_active":"2023-10-07T09:45:38.886371+0000","last_became_peered":"2023-10-07T09:45:38.886371+0000","last_unstale":"2023-10-07T09:45:38.894494+0000","last_undegraded":"2023-10-07T09:45:38.894494+0000","last_fullsized":"2023-10-07T09:45:38.894494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:42:34.771752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898817+0000","last_change":"2023-10-07T09:45:37.882528+0000","last_active":"2023-10-07T09:45:38.898817+0000","last_peered":"2023-10-07T09:45:38.898817+0000","last_clean":"2023-10-07T09:45:38.898817+0000","last_became_active":"2023-10-07T09:45:37.880457+0000","last_became_peered":"2023-10-07T09:45:37.880457+0000","last_unstale":"2023-10-07T09:45:38.898817+0000","last_undegraded":"2023-10-07T09:45:38.898817+0000","last_fullsized":"2023-10-07T09:45:38.898817+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:00.152453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.891462+0000","last_change":"2023-10-07T09:45:38.888529+0000","last_active":"2023-10-07T09:45:38.891462+0000","last_peered":"2023-10-07T09:45:38.891462+0000","last_clean":"2023-10-07T09:45:38.891462+0000","last_became_active":"2023-10-07T09:45:38.888266+0000","last_became_peered":"2023-10-07T09:45:38.888266+0000","last_unstale":"2023-10-07T09:45:38.891462+0000","last_undegraded":"2023-10-07T09:45:38.891462+0000","last_fullsized":"2023-10-07T09:45:38.891462+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:55.429915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.872206+0000","last_change":"2023-10-07T09:45:37.885925+0000","last_active":"2023-10-07T09:45:38.872206+0000","last_peered":"2023-10-07T09:45:38.872206+0000","last_clean":"2023-10-07T09:45:38.872206+0000","last_became_active":"2023-10-07T09:45:37.885563+0000","last_became_peered":"2023-10-07T09:45:37.885563+0000","last_unstale":"2023-10-07T09:45:38.872206+0000","last_undegraded":"2023-10-07T09:45:38.872206+0000","last_fullsized":"2023-10-07T09:45:38.872206+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:05.052249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.897716+0000","last_change":"2023-10-07T09:45:38.891051+0000","last_active":"2023-10-07T09:45:38.897716+0000","last_peered":"2023-10-07T09:45:38.897716+0000","last_clean":"2023-10-07T09:45:38.897716+0000","last_became_active":"2023-10-07T09:45:38.890531+0000","last_became_peered":"2023-10-07T09:45:38.890531+0000","last_unstale":"2023-10-07T09:45:38.897716+0000","last_undegraded":"2023-10-07T09:45:38.897716+0000","last_fullsized":"2023-10-07T09:45:38.897716+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:53:28.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.898160+0000","last_change":"2023-10-07T09:45:37.878267+0000","last_active":"2023-10-07T09:45:38.898160+0000","last_peered":"2023-10-07T09:45:38.898160+0000","last_clean":"2023-10-07T09:45:38.898160+0000","last_became_active":"2023-10-07T09:45:37.877969+0000","last_became_peered":"2023-10-07T09:45:37.877969+0000","last_unstale":"2023-10-07T09:45:38.898160+0000","last_undegraded":"2023-10-07T09:45:38.898160+0000","last_fullsized":"2023-10-07T09:45:38.898160+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:43:39.374873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896804+0000","last_change":"2023-10-07T09:45:38.896056+0000","last_active":"2023-10-07T09:45:38.896804+0000","last_peered":"2023-10-07T09:45:38.896804+0000","last_clean":"2023-10-07T09:45:38.896804+0000","last_became_active":"2023-10-07T09:45:38.895776+0000","last_became_peered":"2023-10-07T09:45:38.895776+0000","last_unstale":"2023-10-07T09:45:38.896804+0000","last_undegraded":"2023-10-07T09:45:38.896804+0000","last_fullsized":"2023-10-07T09:45:38.896804+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:31:35.952352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.875443+0000","last_change":"2023-10-07T09:45:37.877915+0000","last_active":"2023-10-07T09:45:38.875443+0000","last_peered":"2023-10-07T09:45:38.875443+0000","last_clean":"2023-10-07T09:45:38.875443+0000","last_became_active":"2023-10-07T09:45:37.877644+0000","last_became_peered":"2023-10-07T09:45:37.877644+0000","last_unstale":"2023-10-07T09:45:38.875443+0000","last_undegraded":"2023-10-07T09:45:38.875443+0000","last_fullsized":"2023-10-07T09:45:38.875443+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:46:36.915586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894579+0000","last_change":"2023-10-07T09:45:38.883241+0000","last_active":"2023-10-07T09:45:38.894579+0000","last_peered":"2023-10-07T09:45:38.894579+0000","last_clean":"2023-10-07T09:45:38.894579+0000","last_became_active":"2023-10-07T09:45:38.882979+0000","last_became_peered":"2023-10-07T09:45:38.882979+0000","last_unstale":"2023-10-07T09:45:38.894579+0000","last_undegraded":"2023-10-07T09:45:38.894579+0000","last_fullsized":"2023-10-07T09:45:38.894579+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:27:48.098129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.874267+0000","last_change":"2023-10-07T09:45:37.888352+0000","last_active":"2023-10-07T09:45:38.874267+0000","last_peered":"2023-10-07T09:45:38.874267+0000","last_clean":"2023-10-07T09:45:38.874267+0000","last_became_active":"2023-10-07T09:45:37.888021+0000","last_became_peered":"2023-10-07T09:45:37.888021+0000","last_unstale":"2023-10-07T09:45:38.874267+0000","last_undegraded":"2023-10-07T09:45:38.874267+0000","last_fullsized":"2023-10-07T09:45:38.874267+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:04.192109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.893165+0000","last_change":"2023-10-07T09:45:38.890077+0000","last_active":"2023-10-07T09:45:38.893165+0000","last_peered":"2023-10-07T09:45:38.893165+0000","last_clean":"2023-10-07T09:45:38.893165+0000","last_became_active":"2023-10-07T09:45:38.889775+0000","last_became_peered":"2023-10-07T09:45:38.889775+0000","last_unstale":"2023-10-07T09:45:38.893165+0000","last_undegraded":"2023-10-07T09:45:38.893165+0000","last_fullsized":"2023-10-07T09:45:38.893165+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:11:03.897589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.864644+0000","last_change":"2023-10-07T09:45:37.878192+0000","last_active":"2023-10-07T09:45:38.864644+0000","last_peered":"2023-10-07T09:45:38.864644+0000","last_clean":"2023-10-07T09:45:38.864644+0000","last_became_active":"2023-10-07T09:45:37.877887+0000","last_became_peered":"2023-10-07T09:45:37.877887+0000","last_unstale":"2023-10-07T09:45:38.864644+0000","last_undegraded":"2023-10-07T09:45:38.864644+0000","last_fullsized":"2023-10-07T09:45:38.864644+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:54.421482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.885491+0000","last_change":"2023-10-07T09:45:38.879605+0000","last_active":"2023-10-07T09:45:38.885491+0000","last_peered":"2023-10-07T09:45:38.885491+0000","last_clean":"2023-10-07T09:45:38.885491+0000","last_became_active":"2023-10-07T09:45:38.879326+0000","last_became_peered":"2023-10-07T09:45:38.879326+0000","last_unstale":"2023-10-07T09:45:38.885491+0000","last_undegraded":"2023-10-07T09:45:38.885491+0000","last_fullsized":"2023-10-07T09:45:38.885491+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:42.306675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894443+0000","last_change":"2023-10-07T09:45:37.880609+0000","last_active":"2023-10-07T09:45:38.894443+0000","last_peered":"2023-10-07T09:45:38.894443+0000","last_clean":"2023-10-07T09:45:38.894443+0000","last_became_active":"2023-10-07T09:45:37.880352+0000","last_became_peered":"2023-10-07T09:45:37.880352+0000","last_unstale":"2023-10-07T09:45:38.894443+0000","last_undegraded":"2023-10-07T09:45:38.894443+0000","last_fullsized":"2023-10-07T09:45:38.894443+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:53:27.563837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.892025+0000","last_change":"2023-10-07T09:45:38.883893+0000","last_active":"2023-10-07T09:45:38.892025+0000","last_peered":"2023-10-07T09:45:38.892025+0000","last_clean":"2023-10-07T09:45:38.892025+0000","last_became_active":"2023-10-07T09:45:38.883586+0000","last_became_peered":"2023-10-07T09:45:38.883586+0000","last_unstale":"2023-10-07T09:45:38.892025+0000","last_undegraded":"2023-10-07T09:45:38.892025+0000","last_fullsized":"2023-10-07T09:45:38.892025+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:53:10.245641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.869560+0000","last_change":"2023-10-07T09:45:37.876479+0000","last_active":"2023-10-07T09:45:38.869560+0000","last_peered":"2023-10-07T09:45:38.869560+0000","last_clean":"2023-10-07T09:45:38.869560+0000","last_became_active":"2023-10-07T09:45:37.876203+0000","last_became_peered":"2023-10-07T09:45:37.876203+0000","last_unstale":"2023-10-07T09:45:38.869560+0000","last_undegraded":"2023-10-07T09:45:38.869560+0000","last_fullsized":"2023-10-07T09:45:38.869560+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:17:19.359081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.895174+0000","last_change":"2023-10-07T09:45:38.888133+0000","last_active":"2023-10-07T09:45:38.895174+0000","last_peered":"2023-10-07T09:45:38.895174+0000","last_clean":"2023-10-07T09:45:38.895174+0000","last_became_active":"2023-10-07T09:45:38.887883+0000","last_became_peered":"2023-10-07T09:45:38.887883+0000","last_unstale":"2023-10-07T09:45:38.895174+0000","last_undegraded":"2023-10-07T09:45:38.895174+0000","last_fullsized":"2023-10-07T09:45:38.895174+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:17.765418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.875749+0000","last_change":"2023-10-07T09:45:37.885288+0000","last_active":"2023-10-07T09:45:38.875749+0000","last_peered":"2023-10-07T09:45:38.875749+0000","last_clean":"2023-10-07T09:45:38.875749+0000","last_became_active":"2023-10-07T09:45:37.884962+0000","last_became_peered":"2023-10-07T09:45:37.884962+0000","last_unstale":"2023-10-07T09:45:38.875749+0000","last_undegraded":"2023-10-07T09:45:38.875749+0000","last_fullsized":"2023-10-07T09:45:38.875749+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:41.708361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896310+0000","last_change":"2023-10-07T09:45:38.894498+0000","last_active":"2023-10-07T09:45:38.896310+0000","last_peered":"2023-10-07T09:45:38.896310+0000","last_clean":"2023-10-07T09:45:38.896310+0000","last_became_active":"2023-10-07T09:45:38.894225+0000","last_became_peered":"2023-10-07T09:45:38.894225+0000","last_unstale":"2023-10-07T09:45:38.896310+0000","last_undegraded":"2023-10-07T09:45:38.896310+0000","last_fullsized":"2023-10-07T09:45:38.896310+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:48:54.205691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.876843+0000","last_change":"2023-10-07T09:45:37.886728+0000","last_active":"2023-10-07T09:45:38.876843+0000","last_peered":"2023-10-07T09:45:38.876843+0000","last_clean":"2023-10-07T09:45:38.876843+0000","last_became_active":"2023-10-07T09:45:37.886403+0000","last_became_peered":"2023-10-07T09:45:37.886403+0000","last_unstale":"2023-10-07T09:45:38.876843+0000","last_undegraded":"2023-10-07T09:45:38.876843+0000","last_fullsized":"2023-10-07T09:45:38.876843+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:47.046035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.896812+0000","last_change":"2023-10-07T09:45:38.896072+0000","last_active":"2023-10-07T09:45:38.896812+0000","last_peered":"2023-10-07T09:45:38.896812+0000","last_clean":"2023-10-07T09:45:38.896812+0000","last_became_active":"2023-10-07T09:45:38.895787+0000","last_became_peered":"2023-10-07T09:45:38.895787+0000","last_unstale":"2023-10-07T09:45:38.896812+0000","last_undegraded":"2023-10-07T09:45:38.896812+0000","last_fullsized":"2023-10-07T09:45:38.896812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:18:33.071732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.871825+0000","last_change":"2023-10-07T09:45:37.872060+0000","last_active":"2023-10-07T09:45:38.871825+0000","last_peered":"2023-10-07T09:45:38.871825+0000","last_clean":"2023-10-07T09:45:38.871825+0000","last_became_active":"2023-10-07T09:45:37.871777+0000","last_became_peered":"2023-10-07T09:45:37.871777+0000","last_unstale":"2023-10-07T09:45:38.871825+0000","last_undegraded":"2023-10-07T09:45:38.871825+0000","last_fullsized":"2023-10-07T09:45:38.871825+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:06:00.078548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899470+0000","last_change":"2023-10-07T09:45:38.887965+0000","last_active":"2023-10-07T09:45:38.899470+0000","last_peered":"2023-10-07T09:45:38.899470+0000","last_clean":"2023-10-07T09:45:38.899470+0000","last_became_active":"2023-10-07T09:45:38.884352+0000","last_became_peered":"2023-10-07T09:45:38.884352+0000","last_unstale":"2023-10-07T09:45:38.899470+0000","last_undegraded":"2023-10-07T09:45:38.899470+0000","last_fullsized":"2023-10-07T09:45:38.899470+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:39:01.454574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.868271+0000","last_change":"2023-10-07T09:45:37.879761+0000","last_active":"2023-10-07T09:45:38.868271+0000","last_peered":"2023-10-07T09:45:38.868271+0000","last_clean":"2023-10-07T09:45:38.868271+0000","last_became_active":"2023-10-07T09:45:37.879511+0000","last_became_peered":"2023-10-07T09:45:37.879511+0000","last_unstale":"2023-10-07T09:45:38.868271+0000","last_undegraded":"2023-10-07T09:45:38.868271+0000","last_fullsized":"2023-10-07T09:45:38.868271+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:07:00.152453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.884763+0000","last_change":"2023-10-07T09:45:38.883144+0000","last_active":"2023-10-07T09:45:38.884763+0000","last_peered":"2023-10-07T09:45:38.884763+0000","last_clean":"2023-10-07T09:45:38.884763+0000","last_became_active":"2023-10-07T09:45:38.882879+0000","last_became_peered":"2023-10-07T09:45:38.882879+0000","last_unstale":"2023-10-07T09:45:38.884763+0000","last_undegraded":"2023-10-07T09:45:38.884763+0000","last_fullsized":"2023-10-07T09:45:38.884763+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:00.339232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.876176+0000","last_change":"2023-10-07T09:45:37.886356+0000","last_active":"2023-10-07T09:45:38.876176+0000","last_peered":"2023-10-07T09:45:38.876176+0000","last_clean":"2023-10-07T09:45:38.876176+0000","last_became_active":"2023-10-07T09:45:37.886059+0000","last_became_peered":"2023-10-07T09:45:37.886059+0000","last_unstale":"2023-10-07T09:45:38.876176+0000","last_undegraded":"2023-10-07T09:45:38.876176+0000","last_fullsized":"2023-10-07T09:45:38.876176+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:56:52.547998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.894975+0000","last_change":"2023-10-07T09:45:38.888578+0000","last_active":"2023-10-07T09:45:38.894975+0000","last_peered":"2023-10-07T09:45:38.894975+0000","last_clean":"2023-10-07T09:45:38.894975+0000","last_became_active":"2023-10-07T09:45:38.888279+0000","last_became_peered":"2023-10-07T09:45:38.888279+0000","last_unstale":"2023-10-07T09:45:38.894975+0000","last_undegraded":"2023-10-07T09:45:38.894975+0000","last_fullsized":"2023-10-07T09:45:38.894975+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:18:02.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.868459+0000","last_change":"2023-10-07T09:45:37.882096+0000","last_active":"2023-10-07T09:45:38.868459+0000","last_peered":"2023-10-07T09:45:38.868459+0000","last_clean":"2023-10-07T09:45:38.868459+0000","last_became_active":"2023-10-07T09:45:37.881572+0000","last_became_peered":"2023-10-07T09:45:37.881572+0000","last_unstale":"2023-10-07T09:45:38.868459+0000","last_undegraded":"2023-10-07T09:45:38.868459+0000","last_fullsized":"2023-10-07T09:45:38.868459+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:02:20.587812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.882253+0000","last_change":"2023-10-07T09:45:38.879934+0000","last_active":"2023-10-07T09:45:38.882253+0000","last_peered":"2023-10-07T09:45:38.882253+0000","last_clean":"2023-10-07T09:45:38.882253+0000","last_became_active":"2023-10-07T09:45:38.879682+0000","last_became_peered":"2023-10-07T09:45:38.879682+0000","last_unstale":"2023-10-07T09:45:38.882253+0000","last_undegraded":"2023-10-07T09:45:38.882253+0000","last_fullsized":"2023-10-07T09:45:38.882253+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:37.850114+0000","last_clean_scrub_stamp":"2023-10-07T09:45:37.850114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:08:29.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.899609+0000","last_change":"2023-10-07T09:45:37.880127+0000","last_active":"2023-10-07T09:45:38.899609+0000","last_peered":"2023-10-07T09:45:38.899609+0000","last_clean":"2023-10-07T09:45:38.899609+0000","last_became_active":"2023-10-07T09:45:37.879853+0000","last_became_peered":"2023-10-07T09:45:37.879853+0000","last_unstale":"2023-10-07T09:45:38.899609+0000","last_undegraded":"2023-10-07T09:45:38.899609+0000","last_fullsized":"2023-10-07T09:45:38.899609+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:45:36.841682+0000","last_clean_scrub_stamp":"2023-10-07T09:45:36.841682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:13:42.180823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"93'205","reported_seq":544,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-07T09:45:38.869721+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:45:38.869721+0000","last_peered":"2023-10-07T09:45:38.869721+0000","last_clean":"2023-10-07T09:45:38.869721+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:45:38.869721+0000","last_undegraded":"2023-10-07T09:45:38.869721+0000","last_fullsized":"2023-10-07T09:45:38.869721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476932,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303196,"kb_used_data":1812,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068644,"statfs":{"total":96636764160,"available":96326291456,"internally_reserved":0,"allocated":1855488,"data_stored":1191003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.71399999999999997},"min":{"1min":0.45300000000000001,"5min":0.374,"15min":0.32300000000000001},"max":{"1min":0.95399999999999996,"5min":0.96599999999999997,"15min":1.026},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.69099999999999995,"15min":0.71699999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":0.78900000000000003,"5min":0.94899999999999995,"15min":1.0349999999999999},"last":0.76600000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.47899999999999998,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":0.85099999999999998,"5min":0.95599999999999996,"15min":1.014},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.73599999999999999,"15min":0.74099999999999999},"min":{"1min":0.46600000000000003,"5min":0.45900000000000002,"15min":0.45600000000000002},"max":{"1min":0.88,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.78000000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.746},"min":{"1min":0.51100000000000001,"5min":0.41599999999999998,"15min":0.40699999999999997},"max":{"1min":0.89200000000000002,"5min":1.206,"15min":1.206},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71199999999999997,"15min":0.73699999999999999},"min":{"1min":0.5,"5min":0.47899999999999998,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":1.038,"15min":1.071},"last":0.81699999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69899999999999995,"15min":0.73599999999999999},"min":{"1min":0.48399999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":0.92300000000000004,"15min":1.01},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.76500000000000001,"15min":0.76600000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.371},"max":{"1min":0.98699999999999999,"5min":1.137,"15min":1.137},"last":0.67400000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61899999999999999,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39100000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.66900000000000004,"15min":0.65800000000000003},"min":{"1min":0.48599999999999999,"5min":0.39600000000000002,"15min":0.36899999999999999},"max":{"1min":0.92400000000000004,"5min":1.27,"15min":1.27},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68999999999999995,"15min":0.69499999999999995},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.88100000000000001,"5min":1.0069999999999999,"15min":1.1399999999999999},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.751,"15min":0.745},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.72599999999999998,"15min":0.72299999999999998},"min":{"1min":0.59999999999999998,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":1.0109999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.745}]}]},{"osd":6,"up_from":15,"seq":64424509636,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302624,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069216,"statfs":{"total":96636764160,"available":96326877184,"internally_reserved":0,"allocated":1269760,"data_stored":601231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.69099999999999995,"15min":0.71199999999999997},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":1.0780000000000001,"5min":1.274,"15min":1.274},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.45200000000000001,"5min":0.40699999999999997,"15min":0.34999999999999998},"max":{"1min":0.95699999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.74299999999999999}]},{"osd":1,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69699999999999995,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":1.042,"5min":1.169,"15min":1.169},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70599999999999996,"15min":0.72599999999999998},"min":{"1min":0.53300000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.02,"5min":1.7,"15min":1.7},"last":0.80000000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":0.98299999999999998,"5min":1.482,"15min":1.482},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.48899999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94499999999999995,"5min":1.643,"15min":1.643},"last":0.76200000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73599999999999999,"15min":0.75700000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":1.458,"15min":1.458},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72799999999999998,"15min":0.754},"min":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.72499999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.621,"15min":0.62},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.29899999999999999},"max":{"1min":0.83499999999999996,"5min":0.92400000000000004,"15min":1.01},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65700000000000003,"15min":0.65800000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.41199999999999998},"max":{"1min":0.94499999999999995,"5min":1.036,"15min":1.0509999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":1.0089999999999999,"15min":1.0449999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.39700000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.749,"15min":0.76100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.99099999999999999,"5min":0.995,"15min":1.01},"last":0.86299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542338,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302624,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069216,"statfs":{"total":96636764160,"available":96326877184,"internally_reserved":0,"allocated":1269760,"data_stored":600759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.34599999999999997,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.871},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.52500000000000002,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.59999999999999998,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45200000000000001,"15min":0.47699999999999998},"min":{"1min":0.32700000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.79600000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.35699999999999998,"15min":0.33600000000000002},"max":{"1min":0.61699999999999999,"5min":0.78400000000000003,"15min":1.028},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56599999999999995,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.97599999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.79100000000000004,"5min":0.79200000000000004,"15min":0.95099999999999996},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.77800000000000002},"min":{"1min":0.54600000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90700000000000003,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.72099999999999997,"15min":0.76600000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.46500000000000002},"max":{"1min":0.82299999999999995,"5min":1.032,"15min":1.196},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.77000000000000002,"15min":0.80600000000000005},"min":{"1min":0.57499999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.89300000000000002,"5min":1.099,"15min":1.2769999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.73599999999999999,"15min":0.78000000000000003},"min":{"1min":0.56000000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.85099999999999998,"5min":1.0680000000000001,"15min":1.3300000000000001},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.80800000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1,"5min":1.0109999999999999,"15min":1.131},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.755,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.87,"5min":1.1040000000000001,"15min":1.165},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.82899999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.2450000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.83599999999999997},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":1.137,"15min":1.385},"last":0.627}]}]},{"osd":0,"up_from":14,"seq":60129542338,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303196,"kb_used_data":1812,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068644,"statfs":{"total":96636764160,"available":96326291456,"internally_reserved":0,"allocated":1855488,"data_stored":1190967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.62,"5min":0.68400000000000005,"15min":0.745},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.55000000000000004,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.63900000000000001,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51700000000000002,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.63900000000000001,"5min":0.876,"15min":1.101},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.48799999999999999,"15min":0.503},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60999999999999999,"5min":0.80900000000000005,"15min":0.92200000000000004},"last":0.378}]},{"osd":3,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50600000000000001,"15min":0.53000000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.83799999999999997,"15min":1.0009999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.81100000000000005,"15min":0.92100000000000004},"last":0.503}]},{"osd":4,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.72799999999999998,"15min":0.755},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85899999999999999,"5min":1.1850000000000001,"15min":1.232},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77900000000000003,"15min":0.79600000000000004},"min":{"1min":0.52300000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0489999999999999,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.86599999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.753,"15min":0.77900000000000003},"min":{"1min":0.505,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.81100000000000005,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.79600000000000004},"min":{"1min":0.54600000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.94699999999999995,"5min":1.0920000000000001,"15min":1.194},"last":0.82799999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.78100000000000003,"15min":0.81000000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.84799999999999998,"5min":1.129,"15min":1.196},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77700000000000002,"15min":0.80800000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.206,"15min":1.206},"last":0.85299999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79100000000000004,"15min":0.82799999999999996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.97099999999999997,"5min":1.167,"15min":1.167},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.79000000000000004,"15min":0.82599999999999996},"min":{"1min":0.56200000000000006,"5min":0.503,"15min":0.503},"max":{"1min":0.90500000000000003,"5min":1.238,"15min":1.238},"last":0.79700000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542339,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310796,"kb_used_data":1220,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061044,"statfs":{"total":96636764160,"available":96318509056,"internally_reserved":0,"allocated":1249280,"data_stored":594455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.441,"15min":0.44900000000000001},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.26700000000000002},"max":{"1min":0.63700000000000001,"5min":0.81000000000000005,"15min":1.258},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":0.80100000000000005,"15min":1.2290000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":1.1950000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":1.3200000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.501,"15min":0.51500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.94599999999999995,"15min":1.5820000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.69399999999999995,"5min":0.81399999999999995,"15min":1.052},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73399999999999999,"15min":0.75700000000000001},"min":{"1min":0.48099999999999998,"5min":0.46200000000000002,"15min":0.35699999999999998},"max":{"1min":0.81000000000000005,"5min":1.0660000000000001,"15min":1.4630000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93500000000000005,"5min":1.98,"15min":1.98},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.52500000000000002,"5min":0.52400000000000002,"15min":0.46100000000000002},"max":{"1min":0.88800000000000001,"5min":1.208,"15min":1.5169999999999999},"last":0.79000000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":0.877,"5min":1.931,"15min":1.931},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.59099999999999997,"5min":0.52800000000000002,"15min":0.47299999999999998},"max":{"1min":0.93700000000000006,"5min":1.306,"15min":1.8120000000000001},"last":0.77300000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.83899999999999997},"min":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.54200000000000004},"max":{"1min":0.89800000000000002,"5min":1.3520000000000001,"15min":1.3839999999999999},"last":0.80700000000000005}]}]},{"osd":3,"up_from":14,"seq":60129542340,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306720,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065120,"statfs":{"total":96636764160,"available":96322682880,"internally_reserved":0,"allocated":1269760,"data_stored":602101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45200000000000001,"15min":0.45700000000000002},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48299999999999998,"15min":0.50900000000000001},"min":{"1min":0.35399999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.755,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.42999999999999999,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.46600000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52300000000000002,"15min":0.54900000000000004},"min":{"1min":0.39900000000000002,"5min":0.317,"15min":0.317},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.437},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.39800000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54500000000000004,"5min":0.48899999999999999,"15min":0.44900000000000001},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.754,"5min":0.77100000000000002,"15min":0.78400000000000003},"min":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.3009999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.52500000000000002},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78400000000000003,"15min":0.79400000000000004},"min":{"1min":0.56599999999999995,"5min":0.53200000000000003,"15min":0.50900000000000001},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.93600000000000005}]},{"osd":6,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.56100000000000005,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80000000000000004,"15min":0.82099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.628,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":2.194,"5min":2.194,"15min":2.194},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.878,"5min":0.81999999999999995,"15min":0.83399999999999996},"min":{"1min":0.58099999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":0.96299999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306708,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065132,"statfs":{"total":96636764160,"available":96322695168,"internally_reserved":0,"allocated":1257472,"data_stored":600565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90000000000000002,"5min":1.069,"15min":1.069},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.69599999999999995,"15min":0.70699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84499999999999997,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.70999999999999996},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.874,"5min":0.96599999999999997,"15min":1.054},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.91900000000000004,"5min":1.478,"15min":1.478},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.74199999999999999,"15min":0.74299999999999999},"min":{"1min":0.55900000000000005,"5min":0.51100000000000001,"15min":0.432},"max":{"1min":1.0680000000000001,"5min":1.518,"15min":1.518},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68500000000000005,"15min":0.71399999999999997},"min":{"1min":0.53100000000000003,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.626}]},{"osd":3,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68999999999999995,"15min":0.71599999999999997},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86299999999999999,"5min":0.96899999999999997,"15min":1.032},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.752},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0249999999999999,"5min":1.036,"15min":1.036},"last":0.64000000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85999999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.627,"15min":0.626},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70799999999999996,"5min":1.639,"15min":1.639},"last":0.40799999999999997}]},{"osd":6,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.83599999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69599999999999995,"15min":0.69099999999999995},"min":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94699999999999995,"5min":1.585,"15min":1.585},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69999999999999996},"min":{"1min":0.45900000000000002,"5min":0.45100000000000001,"15min":0.443},"max":{"1min":0.96299999999999997,"5min":1.0049999999999999,"15min":1.0589999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.755,"15min":0.747},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.45200000000000001},"max":{"1min":1.012,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.72799999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302628,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069212,"statfs":{"total":96636764160,"available":96326873088,"internally_reserved":0,"allocated":1273856,"data_stored":602123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76300000000000001,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.49199999999999999,"15min":0.39900000000000002},"max":{"1min":0.94999999999999996,"5min":1.218,"15min":1.218},"last":1.115},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.436},"max":{"1min":0.95499999999999996,"5min":1.294,"15min":1.294},"last":0.91200000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.505,"5min":0.46400000000000002,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":1.1339999999999999,"15min":1.3049999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.73299999999999998,"15min":0.75600000000000001},"min":{"1min":0.58199999999999996,"5min":0.434,"15min":0.41699999999999998},"max":{"1min":1.028,"5min":1.194,"15min":6.9720000000000004},"last":0.88300000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71399999999999997,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":1.0049999999999999,"5min":1.119,"15min":5.9299999999999997},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.59899999999999998,"5min":0.48499999999999999,"15min":0.42299999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.704},"last":1.0369999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.60799999999999998,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":0.93799999999999994,"5min":1.2589999999999999,"15min":1.899},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51900000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94999999999999996,"5min":1.036,"15min":6.0620000000000003},"last":0.96299999999999997}]},{"osd":4,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.83499999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.64000000000000001,"15min":0.63600000000000001},"min":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.97499999999999998,"15min":0.998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50900000000000001,"5min":0.47799999999999998,"15min":0.42299999999999999},"max":{"1min":0.81599999999999995,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.81100000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:44:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70699999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":0.83399999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76500000000000001},"min":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.45600000000000002},"max":{"1min":0.89100000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:45:43.774 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:45:43.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:45:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.012+0000 7f8afabff700 1 Processor -- start 2023-10-07T09:45:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.012+0000 7f8afabff700 1 -- start start 2023-10-07T09:45:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.013+0000 7f8afabff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8af4158880 0x7f8af4162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.013+0000 7f8afabff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40ac5e0 con 0x7f8af40b1550 2023-10-07T09:45:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.013+0000 7f8afabff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40ac8c0 con 0x7f8af40ac100 2023-10-07T09:45:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.013+0000 7f8afabff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af40abae0 con 0x7f8af4158880 2023-10-07T09:45:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.013+0000 7f8af899b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8af40b1550 0x7f8af415bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53304/0 (socket says 172.21.15.29:53304) 2023-10-07T09:45:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.013+0000 7f8af899b700 1 -- 172.21.15.29:0/2219011289 learned_addr learned my addr 172.21.15.29:0/2219011289 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af919c700 1 --2- 172.21.15.29:0/2219011289 >> v2:172.21.15.144:3300/0 conn(0x7f8af4158880 0x7f8af4162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af37fe700 1 -- 172.21.15.29:0/2219011289 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654270557 0 0) 0x7f8af40ac5e0 con 0x7f8af40b1550 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af37fe700 1 -- 172.21.15.29:0/2219011289 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae4003570 con 0x7f8af40b1550 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af919c700 1 -- 172.21.15.29:0/2219011289 >> v1:172.21.15.144:6789/0 conn(0x7f8af40ac100 legacy=0x7f8af4157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af919c700 1 -- 172.21.15.29:0/2219011289 >> v1:172.21.15.29:6789/0 conn(0x7f8af40b1550 legacy=0x7f8af415bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af37fe700 1 -- 172.21.15.29:0/2219011289 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3724194945 0 0) 0x7f8ae4003570 con 0x7f8af40b1550 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af919c700 1 -- 172.21.15.29:0/2219011289 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af40acba0 con 0x7f8af4158880 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.014+0000 7f8af919c700 1 --2- 172.21.15.29:0/2219011289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af4158880 0x7f8af4162440 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009e60 tx=0x7f8aec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b651bc4706fb89ba server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8af37fe700 1 -- 172.21.15.29:0/2219011289 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec00e070 con 0x7f8af4158880 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8af37fe700 1 -- 172.21.15.29:0/2219011289 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec00bb60 con 0x7f8af4158880 2023-10-07T09:45:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8af37fe700 1 -- 172.21.15.29:0/2219011289 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec012430 con 0x7f8af4158880 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 -- 172.21.15.29:0/2219011289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af4158880 msgr2=0x7f8af4162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 --2- 172.21.15.29:0/2219011289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af4158880 0x7f8af4162440 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009e60 tx=0x7f8aec009780 comp rx=0 tx=0).stop 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 -- 172.21.15.29:0/2219011289 shutdown_connections 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 --2- 172.21.15.29:0/2219011289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af4158880 0x7f8af4162440 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 -- 172.21.15.29:0/2219011289 >> 172.21.15.29:0/2219011289 conn(0x7f8af4067610 msgr2=0x7f8af4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 -- 172.21.15.29:0/2219011289 shutdown_connections 2023-10-07T09:45:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.015+0000 7f8afabff700 1 -- 172.21.15.29:0/2219011289 wait complete. 2023-10-07T09:45:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.017+0000 7f8afabff700 1 Processor -- start 2023-10-07T09:45:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.018+0000 7f8afabff700 1 -- start start 2023-10-07T09:45:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8afabff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af40ac100 0x7f8af406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8afabff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8af40b1550 0x7f8af414fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8afabff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 0x7f8af406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8afabff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40ab300 con 0x7f8af4158880 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8afabff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af406dfd0 con 0x7f8af40ac100 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8afabff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af406e2b0 con 0x7f8af40b1550 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8af919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 0x7f8af406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8af919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 0x7f8af406d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47246/0 (socket says 172.21.15.29:47246) 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8af919c700 1 -- 172.21.15.29:0/1385515997 learned_addr learned my addr 172.21.15.29:0/1385515997 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8af899b700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af40ac100 0x7f8af406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.019+0000 7f8af3fff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8af40b1550 0x7f8af414fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af919c700 1 -- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8af40b1550 msgr2=0x7f8af414fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af919c700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8af40b1550 0x7f8af414fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af919c700 1 -- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af40ac100 msgr2=0x7f8af406f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af919c700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af40ac100 0x7f8af406f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af919c700 1 -- 172.21.15.29:0/1385515997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec009a20 con 0x7f8af4158880 2023-10-07T09:45:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af3fff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8af40b1550 0x7f8af414fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af919c700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 0x7f8af406d480 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8aec013000 tx=0x7f8aec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec026480 con 0x7f8af4158880 2023-10-07T09:45:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec026640 con 0x7f8af4158880 2023-10-07T09:45:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af406e590 con 0x7f8af4158880 2023-10-07T09:45:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.020+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec026a00 con 0x7f8af4158880 2023-10-07T09:45:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.021+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af406eb50 con 0x7f8af4158880 2023-10-07T09:45:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.022+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af406ed40 con 0x7f8af4158880 2023-10-07T09:45:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.022+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8aec01c400 con 0x7f8af4158880 2023-10-07T09:45:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.023+0000 7f8af0ff9700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8ad8066f30 0x7f8ad80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.023+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8aec086610 con 0x7f8af4158880 2023-10-07T09:45:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.024+0000 7f8af899b700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8ad8066f30 0x7f8ad80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.024+0000 7f8af899b700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8ad8066f30 0x7f8ad80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ae80011e0 tx=0x7f8ae8007af0 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.027+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8aec053700 con 0x7f8af4158880 2023-10-07T09:45:44.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.308+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8af40718b0 con 0x7f8af4158880 2023-10-07T09:45:44.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.313+0000 7f8af0ff9700 1 -- 172.21.15.29:0/1385515997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8aec056d20 con 0x7f8af4158880 2023-10-07T09:45:44.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8ad8066f30 msgr2=0x7f8ad80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8ad8066f30 0x7f8ad80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ae80011e0 tx=0x7f8ae8007af0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 msgr2=0x7f8af406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 0x7f8af406d480 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8aec013000 tx=0x7f8aec019000 comp rx=0 tx=0).stop 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 shutdown_connections 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8ad8066f30 0x7f8ad80693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8af4158880 0x7f8af406d480 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af40ac100 0x7f8af406f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 --2- 172.21.15.29:0/1385515997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8af40b1550 0x7f8af414fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 >> 172.21.15.29:0/1385515997 conn(0x7f8af4067610 msgr2=0x7f8af4156f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 shutdown_connections 2023-10-07T09:45:44.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.315+0000 7f8afabff700 1 -- 172.21.15.29:0/1385515997 wait complete. 2023-10-07T09:45:44.317 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:45:44.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:45:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.528+0000 7f81ba7ff700 1 Processor -- start 2023-10-07T09:45:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.528+0000 7f81ba7ff700 1 -- start start 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.529+0000 7f81ba7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.529+0000 7f81ba7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b414d790 con 0x7f81b40ad370 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.529+0000 7f81ba7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b40ae5d0 con 0x7f81b4159ec0 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.529+0000 7f81ba7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40ad740 con 0x7f81b415a340 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f81b40ad370 0x7f81b40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53308/0 (socket says 172.21.15.29:53308) 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b3fff700 1 -- 172.21.15.29:0/397279488 learned_addr learned my addr 172.21.15.29:0/397279488 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b8d9c700 1 --2- 172.21.15.29:0/397279488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660097079 0 0) 0x7f81b414d790 con 0x7f81b40ad370 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819c003570 con 0x7f81b40ad370 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574667153 0 0) 0x7f81b40ae5d0 con 0x7f81b4159ec0 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b414d790 con 0x7f81b4159ec0 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220761353 0 0) 0x7f819c003570 con 0x7f81b40ad370 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.530+0000 7f81b8d9c700 1 -- 172.21.15.29:0/397279488 >> v1:172.21.15.144:6789/0 conn(0x7f81b4159ec0 legacy=0x7f81b40af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.531+0000 7f81b8d9c700 1 -- 172.21.15.29:0/397279488 >> v1:172.21.15.29:6789/0 conn(0x7f81b40ad370 legacy=0x7f81b40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.531+0000 7f81b8d9c700 1 -- 172.21.15.29:0/397279488 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40ad9c0 con 0x7f81b415a340 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.531+0000 7f81b8d9c700 1 --2- 172.21.15.29:0/397279488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40afb10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009e90 tx=0x7f81ac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6648f8d4e86edf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.531+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00e070 con 0x7f81b415a340 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.531+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac00b870 con 0x7f81b415a340 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.531+0000 7f81b2ffd700 1 -- 172.21.15.29:0/397279488 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4001470 con 0x7f81b415a340 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 -- 172.21.15.29:0/397279488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 msgr2=0x7f81b40afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/397279488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40afb10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009e90 tx=0x7f81ac009780 comp rx=0 tx=0).stop 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 -- 172.21.15.29:0/397279488 shutdown_connections 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/397279488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40afb10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 -- 172.21.15.29:0/397279488 >> 172.21.15.29:0/397279488 conn(0x7f81b40680a0 msgr2=0x7f81b40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 -- 172.21.15.29:0/397279488 shutdown_connections 2023-10-07T09:45:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.532+0000 7f81ba7ff700 1 -- 172.21.15.29:0/397279488 wait complete. 2023-10-07T09:45:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.533+0000 7f81ba7ff700 1 Processor -- start 2023-10-07T09:45:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.533+0000 7f81ba7ff700 1 -- start start 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.534+0000 7f81ba7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 0x7f81b40a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.534+0000 7f81ba7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b4159ec0 0x7f81b40a88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.534+0000 7f81ba7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.534+0000 7f81ba7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40a8e30 con 0x7f81b40ad370 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.534+0000 7f81ba7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40a6520 con 0x7f81b4159ec0 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.534+0000 7f81ba7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b414bb50 con 0x7f81b415a340 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 0x7f81b40a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 0x7f81b40a83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47252/0 (socket says 172.21.15.29:47252) 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 -- 172.21.15.29:0/1570390238 learned_addr learned my addr 172.21.15.29:0/1570390238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b8d9c700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b37fe700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b4159ec0 0x7f81b40a88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 -- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 msgr2=0x7f81b40aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 -- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b4159ec0 msgr2=0x7f81b40a88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b4159ec0 0x7f81b40a88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 -- 172.21.15.29:0/1570390238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac009a20 con 0x7f81b40ad370 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b37fe700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b4159ec0 0x7f81b40a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.535+0000 7f81b3fff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 0x7f81b40a83b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f81a400c7f0 tx=0x7f81a4001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.536+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4007360 con 0x7f81b40ad370 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.536+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a40074e0 con 0x7f81b40ad370 2023-10-07T09:45:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.536+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4004830 con 0x7f81b40ad370 2023-10-07T09:45:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.536+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b40a9070 con 0x7f81b40ad370 2023-10-07T09:45:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.536+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b40a9630 con 0x7f81b40ad370 2023-10-07T09:45:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.537+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b404c3c0 con 0x7f81b40ad370 2023-10-07T09:45:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.538+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81a4004c00 con 0x7f81b40ad370 2023-10-07T09:45:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.539+0000 7f819bfff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a0066f30 0x7f81a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.539+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f81a4083e90 con 0x7f81b40ad370 2023-10-07T09:45:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.540+0000 7f81b37fe700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a0066f30 0x7f81a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.540+0000 7f81b37fe700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a0066f30 0x7f81a00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81b4150610 tx=0x7f81a8009040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.543+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81a4050f60 con 0x7f81b40ad370 2023-10-07T09:45:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.787+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b40ab660 con 0x7f81b40ad370 2023-10-07T09:45:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.787+0000 7f819bfff700 1 -- 172.21.15.29:0/1570390238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f81a4018020 con 0x7f81b40ad370 2023-10-07T09:45:44.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:44.788 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:38.377260+0000","modified":"2023-10-07T09:45:41.892365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3302133648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3302133648},{"type":"v1","addr":"172.21.15.29:6835","nonce":3302133648}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-10-07T09:45:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a0066f30 msgr2=0x7f81a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a0066f30 0x7f81a00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81b4150610 tx=0x7f81a8009040 comp rx=0 tx=0).stop 2023-10-07T09:45:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 msgr2=0x7f81b40a83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 0x7f81b40a83b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f81a400c7f0 tx=0x7f81a4001a10 comp rx=0 tx=0).stop 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 shutdown_connections 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a0066f30 0x7f81a00693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b415a340 0x7f81b40aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81b40ad370 0x7f81b40a83b0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 --2- 172.21.15.29:0/1570390238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b4159ec0 0x7f81b40a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 >> 172.21.15.29:0/1570390238 conn(0x7f81b40680a0 msgr2=0x7f81b414cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 shutdown_connections 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:44.790+0000 7f81ba7ff700 1 -- 172.21.15.29:0/1570390238 wait complete. 2023-10-07T09:45:44.791 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-07T09:45:44.842 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T09:45:44.842+0000 7fc95e5f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:45:45.210 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:45:38.377260+0000', 'modified': '2023-10-07T09:45:41.892365+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7206}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7206': {'gid': 7206, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/3302133648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 3302133648}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 3302133648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:45:45.210 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:45:45.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-07T09:45:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.411+0000 7f6a16f9d700 1 Processor -- start 2023-10-07T09:45:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.411+0000 7f6a16f9d700 1 -- start start 2023-10-07T09:45:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.412+0000 7f6a16f9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a10159530 0x7f6a100b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.412+0000 7f6a16f9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a100c6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.412+0000 7f6a16f9d700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a100c5da0 con 0x7f6a100c6770 2023-10-07T09:45:45.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.412+0000 7f6a16f9d700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a100cca90 con 0x7f6a10159530 2023-10-07T09:45:45.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.412+0000 7f6a16f9d700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100cc4d0 con 0x7f6a100d0e60 2023-10-07T09:45:45.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.412+0000 7f6a15f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a10159530 0x7f6a100b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a10159530 0x7f6a100b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47264/0 (socket says 172.21.15.29:47264) 2023-10-07T09:45:45.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1879223829 learned_addr learned my addr 172.21.15.29:0/1879223829 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a1579a700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a100c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1879223829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 msgr2=0x7f6a100c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a100c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1879223829 >> v1:172.21.15.144:6789/0 conn(0x7f6a100c6770 legacy=0x7f6a100cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1879223829 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100cc7b0 con 0x7f6a10159530 2023-10-07T09:45:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a1579a700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a100c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a10159530 0x7f6a100b31f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0099a0 tx=0x7f6a0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3db3e95ae94f5a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.413+0000 7f6a14f99700 1 -- 172.21.15.29:0/1879223829 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c00b860 con 0x7f6a10159530 2023-10-07T09:45:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a14f99700 1 -- 172.21.15.29:0/1879223829 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c00b9c0 con 0x7f6a10159530 2023-10-07T09:45:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a14f99700 1 -- 172.21.15.29:0/1879223829 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c00bd80 con 0x7f6a10159530 2023-10-07T09:45:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1879223829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a10159530 msgr2=0x7f6a100b31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a10159530 0x7f6a100b31f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0099a0 tx=0x7f6a0c009250 comp rx=0 tx=0).stop 2023-10-07T09:45:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1879223829 shutdown_connections 2023-10-07T09:45:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a10159530 0x7f6a100b31f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1879223829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a100c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1879223829 >> 172.21.15.29:0/1879223829 conn(0x7f6a10063a10 msgr2=0x7f6a100d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1879223829 shutdown_connections 2023-10-07T09:45:45.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.414+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1879223829 wait complete. 2023-10-07T09:45:45.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.416+0000 7f6a16f9d700 1 Processor -- start 2023-10-07T09:45:45.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.416+0000 7f6a16f9d700 1 -- start start 2023-10-07T09:45:45.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a16f9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 0x7f6a1015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a16f9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a10143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a16f9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a10159530 0x7f6a1015a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a16f9d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1015c2e0 con 0x7f6a100c6770 2023-10-07T09:45:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a16f9d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1015c5c0 con 0x7f6a10159530 2023-10-07T09:45:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a16f9d700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1015c8a0 con 0x7f6a100d0e60 2023-10-07T09:45:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a15f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 0x7f6a1015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a15f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 0x7f6a1015ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47270/0 (socket says 172.21.15.29:47270) 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1023212583 learned_addr learned my addr 172.21.15.29:0/1023212583 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a1679c700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a10159530 0x7f6a1015a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.417+0000 7f6a1579a700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a10143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 msgr2=0x7f6a10143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a10143d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a10159530 msgr2=0x7f6a1015a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a10159530 0x7f6a1015a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a15f9b700 1 -- 172.21.15.29:0/1023212583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c009490 con 0x7f6a100c6770 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a1579a700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a10143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 0x7f6a1015ba90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009290 tx=0x7f6a0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a05ffb700 1 -- 172.21.15.29:0/1023212583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c00bd40 con 0x7f6a100c6770 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a05ffb700 1 -- 172.21.15.29:0/1023212583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c026450 con 0x7f6a100c6770 2023-10-07T09:45:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1015cb80 con 0x7f6a100c6770 2023-10-07T09:45:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a05ffb700 1 -- 172.21.15.29:0/1023212583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c026810 con 0x7f6a100c6770 2023-10-07T09:45:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.418+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1015bfd0 con 0x7f6a100c6770 2023-10-07T09:45:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.421+0000 7f6a05ffb700 1 -- 172.21.15.29:0/1023212583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6a0c01a420 con 0x7f6a100c6770 2023-10-07T09:45:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.421+0000 7f6a05ffb700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69f4066f30 0x7f69f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.421+0000 7f6a05ffb700 1 -- 172.21.15.29:0/1023212583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6a0c086200 con 0x7f6a100c6770 2023-10-07T09:45:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.422+0000 7f6a1579a700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69f4066f30 0x7f69f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.423+0000 7f6a1579a700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69f4066f30 0x7f69f40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6a00006fd0 tx=0x7f6a00008040 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.423+0000 7f6a26936b80 1 Processor -- start 2023-10-07T09:45:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.423+0000 7f6a26936b80 1 -- start start 2023-10-07T09:45:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.424+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c74b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.424+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c776de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.424+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c775dd0 0x55652c776240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.424+0000 7f6a26936b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c663310 con 0x55652c74af00 2023-10-07T09:45:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.424+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c6a0130 con 0x55652c775dd0 2023-10-07T09:45:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.424+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55652c69daf0 con 0x55652c74c280 2023-10-07T09:45:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c74b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c74b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47272/0 (socket says 172.21.15.29:47272) 2023-10-07T09:45:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 -- 172.21.15.29:0/2491268877 learned_addr learned my addr 172.21.15.29:0/2491268877 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a1579a700 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c776de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a1679c700 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c775dd0 0x55652c776240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 -- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 msgr2=0x55652c776de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c776de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 -- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c775dd0 msgr2=0x55652c776240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c775dd0 0x55652c776240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 -- 172.21.15.29:0/2491268877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55652c58c0a0 con 0x55652c74af00 2023-10-07T09:45:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a1579a700 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c776de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c74b2d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0844d0 tx=0x7f6a0c05c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f687fc45d96a8d40 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.425+0000 7f69faffd700 1 -- 172.21.15.29:0/2491268877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c056f90 con 0x55652c74af00 2023-10-07T09:45:45.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f69faffd700 1 -- 172.21.15.29:0/2491268877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c05a1e0 con 0x55652c74af00 2023-10-07T09:45:45.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f69faffd700 1 -- 172.21.15.29:0/2491268877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c05df30 con 0x55652c74af00 2023-10-07T09:45:45.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 -- 172.21.15.29:0/2491268877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 msgr2=0x55652c74b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c74b2d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0844d0 tx=0x7f6a0c05c040 comp rx=0 tx=0).stop 2023-10-07T09:45:45.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 -- 172.21.15.29:0/2491268877 shutdown_connections 2023-10-07T09:45:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c775dd0 0x55652c776240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c74b2d0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 --2- 172.21.15.29:0/2491268877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c776de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 -- 172.21.15.29:0/2491268877 >> 172.21.15.29:0/2491268877 conn(0x55652c748510 msgr2=0x55652c794020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 -- 172.21.15.29:0/2491268877 shutdown_connections 2023-10-07T09:45:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.426+0000 7f6a26936b80 1 -- 172.21.15.29:0/2491268877 wait complete. 2023-10-07T09:45:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.427+0000 7f6a26936b80 1 -- start start 2023-10-07T09:45:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.427+0000 7f6a26936b80 1 Processor -- start 2023-10-07T09:45:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c76f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c79fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c775dd0 0x55652c745960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c6998e0 con 0x55652c74af00 2023-10-07T09:45:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c645980 con 0x55652c74c280 2023-10-07T09:45:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55652c645620 con 0x55652c775dd0 2023-10-07T09:45:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a26936b80 10 client.0 _ll_register_callbacks cb 0x55652c4a8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:45:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a15f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c76f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a15f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c76f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47276/0 (socket says 172.21.15.29:47276) 2023-10-07T09:45:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.429+0000 7f6a15f9b700 1 -- 172.21.15.29:0/457902872 learned_addr learned my addr 172.21.15.29:0/457902872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a1579a700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c79fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a15f9b700 1 -- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c775dd0 msgr2=0x55652c745960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c775dd0 0x55652c745960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a15f9b700 1 -- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 msgr2=0x55652c79fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c79fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a15f9b700 1 -- 172.21.15.29:0/457902872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1015bfd0 con 0x55652c74af00 2023-10-07T09:45:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a1579a700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c79fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c76f470 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009460 tx=0x7f6a0c028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d558ed3a2d4638 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f69effff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:45:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f69effff700 1 -- 172.21.15.29:0/457902872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c05dfb0 con 0x55652c74af00 2023-10-07T09:45:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f69effff700 1 -- 172.21.15.29:0/457902872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6a0c06d380 con 0x55652c74af00 2023-10-07T09:45:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55652c73e560 con 0x55652c74af00 2023-10-07T09:45:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.430+0000 7f69effff700 1 -- 172.21.15.29:0/457902872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a0c0842d0 con 0x55652c74af00 2023-10-07T09:45:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 10 client.7335 fetch_fsmap learned FSMap version 51 2023-10-07T09:45:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55652c6683c0 con 0x55652c74af00 2023-10-07T09:45:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f69effff700 1 -- 172.21.15.29:0/457902872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55652c73e560 con 0x55652c74af00 2023-10-07T09:45:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f69effff700 1 -- 172.21.15.29:0/457902872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 51) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6a0c0693f0 con 0x55652c74af00 2023-10-07T09:45:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 10 client.7335 fetch_fsmap finished waiting for FSMap version 51 2023-10-07T09:45:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 10 client.7335 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:45:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 20 client.7335 populate_metadata read hostname 'smithi029' 2023-10-07T09:45:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c7722e0 0x55652c7726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 4 client.7335 mds_command: new command op to 7206 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:45:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 --> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55652c71b0e0 con 0x55652c7722e0 2023-10-07T09:45:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.431+0000 7f6a1579a700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c7722e0 0x55652c7726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.432+0000 7f69effff700 10 client.7335 ms_handle_connect on v2:172.21.15.29:6834/3302133648 2023-10-07T09:45:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.432+0000 7f6a1579a700 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c7722e0 0x55652c7726b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6704491b78a7e65d server_cookie=1f7e16d2a020f3f3 in_seq=0 out_seq=0 2023-10-07T09:45:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.432+0000 7f69effff700 1 -- 172.21.15.29:0/457902872 <== mds.0 v2:172.21.15.29:6834/3302133648 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55652c71b0e0 con 0x55652c7722e0 2023-10-07T09:45:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.432+0000 7f69effff700 10 client.7335 handle_command_reply: tid=0 2023-10-07T09:45:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.432+0000 7f6a26936b80 1 client.7335 shutdown 2023-10-07T09:45:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 msgr2=0x55652c76f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c76f470 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009460 tx=0x7f6a0c028000 comp rx=0 tx=0).stop 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 shutdown_connections 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c7722e0 0x55652c7726b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c775dd0 0x55652c745960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c74af00 0x55652c76f470 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 --2- 172.21.15.29:0/457902872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c79fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f69effff700 0 client.7335 ms_handle_reset on v2:172.21.15.29:6834/3302133648 2023-10-07T09:45:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 >> 172.21.15.29:0/457902872 conn(0x55652c748510 msgr2=0x55652c774ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 shutdown_connections 2023-10-07T09:45:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 1 -- 172.21.15.29:0/457902872 wait complete. 2023-10-07T09:45:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.433+0000 7f6a26936b80 20 client.7335 trim_cache size 0 max 16384 2023-10-07T09:45:45.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.450+0000 7f6a26936b80 1 Processor -- start 2023-10-07T09:45:45.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 -- start start 2023-10-07T09:45:45.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74af00 0x55652c740060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c771480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c7726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c5311e0 con 0x55652c775dd0 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c74b370 con 0x55652c74c280 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.451+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55652c7405a0 con 0x55652c74af00 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c7726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c7726b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47278/0 (socket says 172.21.15.29:47278) 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 -- 172.21.15.29:0/3383441746 learned_addr learned my addr 172.21.15.29:0/3383441746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74af00 0x55652c740060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1579a700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c771480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 -- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74af00 msgr2=0x55652c740060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74af00 0x55652c740060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 -- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 msgr2=0x55652c771480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c771480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 -- 172.21.15.29:0/3383441746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55652c7422b0 con 0x55652c775dd0 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1579a700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c771480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74af00 0x55652c740060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f6a1679c700 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c7726b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f6a10169770 tx=0x7f6a08009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4651768d0337ba7c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.452+0000 7f69fb7fe700 1 -- 172.21.15.29:0/3383441746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a080108b0 con 0x55652c775dd0 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f69fb7fe700 1 -- 172.21.15.29:0/3383441746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a08010a10 con 0x55652c775dd0 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f69fb7fe700 1 -- 172.21.15.29:0/3383441746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08010dd0 con 0x55652c775dd0 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 -- 172.21.15.29:0/3383441746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 msgr2=0x55652c7726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c7726b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f6a10169770 tx=0x7f6a08009040 comp rx=0 tx=0).stop 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 -- 172.21.15.29:0/3383441746 shutdown_connections 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c7726b0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74af00 0x55652c740060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 --2- 172.21.15.29:0/3383441746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74c280 0x55652c771480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 -- 172.21.15.29:0/3383441746 >> 172.21.15.29:0/3383441746 conn(0x55652c748510 msgr2=0x55652c746620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 -- 172.21.15.29:0/3383441746 shutdown_connections 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.453+0000 7f6a26936b80 1 -- 172.21.15.29:0/3383441746 wait complete. 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.454+0000 7f6a26936b80 1 -- start start 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.454+0000 7f6a26936b80 1 Processor -- start 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74af00 0x55652c7726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c772bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c790020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c7f5be0 con 0x55652c775dd0 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55652c76e9e0 con 0x55652c74af00 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55652c7f56b0 con 0x55652c74c280 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a26936b80 10 client.0 _ll_register_callbacks cb 0x55652c4a8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a1679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c790020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a1679c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c790020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47280/0 (socket says 172.21.15.29:47280) 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a1679c700 1 -- 172.21.15.29:0/173567880 learned_addr learned my addr 172.21.15.29:0/173567880 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.455+0000 7f6a1579a700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c772bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74af00 0x55652c7726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a1679c700 1 -- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 msgr2=0x55652c772bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a1679c700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c772bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a1679c700 1 -- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74af00 msgr2=0x55652c7726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a1679c700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74af00 0x55652c7726b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a1679c700 1 -- 172.21.15.29:0/173567880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a100c5da0 con 0x55652c775dd0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a1679c700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c790020 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6a080013d0 tx=0x7f6a08017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab0bbf42746dc65 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f69effff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f69effff700 1 -- 172.21.15.29:0/173567880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08024070 con 0x55652c775dd0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f69effff700 1 -- 172.21.15.29:0/173567880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6a08014880 con 0x55652c775dd0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.456+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a0c05dfb0 con 0x55652c775dd0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f69effff700 1 -- 172.21.15.29:0/173567880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a08014c50 con 0x55652c775dd0 2023-10-07T09:45:45.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 10 client.7341 fetch_fsmap learned FSMap version 51 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55652c5001d0 con 0x55652c775dd0 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f69effff700 1 -- 172.21.15.29:0/173567880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c05dfb0 con 0x55652c775dd0 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f69effff700 1 -- 172.21.15.29:0/173567880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 51) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6a08012040 con 0x55652c775dd0 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 10 client.7341 fetch_fsmap finished waiting for FSMap version 51 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 10 client.7341 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 20 client.7341 populate_metadata read hostname 'smithi029' 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c790560 0x55652c790930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 4 client.7341 mds_command: new command op to 7206 tid=0[{"prefix": "status"}] 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 --> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6a0c05a1e0 con 0x55652c790560 2023-10-07T09:45:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.457+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c790560 0x55652c790930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a15f9b700 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c790560 0x55652c790930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e7dd141b29db26 server_cookie=2233b530d8aad7e6 in_seq=0 out_seq=0 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f69effff700 10 client.7341 ms_handle_connect on v2:172.21.15.29:6834/3302133648 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f69effff700 1 -- 172.21.15.29:0/173567880 <== mds.0 v2:172.21.15.29:6834/3302133648 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7f6a0c05a1e0 con 0x55652c790560 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f69effff700 10 client.7341 handle_command_reply: tid=0 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 client.7341 shutdown 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 msgr2=0x55652c790020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c790020 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6a080013d0 tx=0x7f6a08017000 comp rx=0 tx=0).stop 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 shutdown_connections 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:6834/3302133648,v1:172.21.15.29:6835/3302133648] conn(0x55652c790560 0x55652c790930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55652c775dd0 0x55652c790020 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55652c74af00 0x55652c7726b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 --2- 172.21.15.29:0/173567880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55652c74c280 0x55652c772bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f69effff700 0 client.7341 ms_handle_reset on v2:172.21.15.29:6834/3302133648 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 >> 172.21.15.29:0/173567880 conn(0x55652c748510 msgr2=0x55652c79f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 shutdown_connections 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.458+0000 7f6a26936b80 1 -- 172.21.15.29:0/173567880 wait complete. 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a26936b80 20 client.7341 trim_cache size 0 max 16384 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:45:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7206, 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5958832589999998, 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 51, 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 103, 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 103, 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.245172569999999 2023-10-07T09:45:45.469 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:45:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69f4066f30 msgr2=0x7f69f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69f4066f30 0x7f69f40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6a00006fd0 tx=0x7f6a00008040 comp rx=0 tx=0).stop 2023-10-07T09:45:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 msgr2=0x7f6a1015ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 0x7f6a1015ba90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009290 tx=0x7f6a0c017000 comp rx=0 tx=0).stop 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 shutdown_connections 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69f4066f30 0x7f69f40693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a10159530 0x7f6a1015a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a100c6770 0x7f6a1015ba90 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 --2- 172.21.15.29:0/1023212583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100d0e60 0x7f6a10143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 >> 172.21.15.29:0/1023212583 conn(0x7f6a10063a10 msgr2=0x7f6a10158590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 shutdown_connections 2023-10-07T09:45:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.459+0000 7f6a16f9d700 1 -- 172.21.15.29:0/1023212583 wait complete. 2023-10-07T09:45:45.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:45:45.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.671+0000 7fa1805ff700 1 Processor -- start 2023-10-07T09:45:45.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.671+0000 7fa1805ff700 1 -- start start 2023-10-07T09:45:45.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.672+0000 7fa1805ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa1780ad370 0x7fa1780cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.672+0000 7fa1805ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780b0410 0x7fa17815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.672+0000 7fa1805ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17814d790 con 0x7fa178159ec0 2023-10-07T09:45:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.672+0000 7fa1805ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17815a880 con 0x7fa1780ad370 2023-10-07T09:45:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.672+0000 7fa1805ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ae5d0 con 0x7fa1780b0410 2023-10-07T09:45:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17db9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa178159ec0 0x7fa1781568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53320/0 (socket says 172.21.15.29:53320) 2023-10-07T09:45:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17db9a700 1 -- 172.21.15.29:0/4092778592 learned_addr learned my addr 172.21.15.29:0/4092778592 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17eb9c700 1 --2- 172.21.15.29:0/4092778592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780b0410 0x7fa17815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17e39b700 1 --2- 172.21.15.29:0/4092778592 >> v2:172.21.15.144:3300/0 conn(0x7fa1780ad370 0x7fa1780cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17d399700 1 -- 172.21.15.29:0/4092778592 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411075695 0 0) 0x7fa17814d790 con 0x7fa178159ec0 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17d399700 1 -- 172.21.15.29:0/4092778592 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa164003570 con 0x7fa178159ec0 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17eb9c700 1 -- 172.21.15.29:0/4092778592 >> v2:172.21.15.144:3300/0 conn(0x7fa1780ad370 msgr2=0x7fa1780cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17eb9c700 1 --2- 172.21.15.29:0/4092778592 >> v2:172.21.15.144:3300/0 conn(0x7fa1780ad370 0x7fa1780cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17eb9c700 1 -- 172.21.15.29:0/4092778592 >> v1:172.21.15.29:6789/0 conn(0x7fa178159ec0 legacy=0x7fa1781568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.673+0000 7fa17eb9c700 1 -- 172.21.15.29:0/4092778592 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1780ad740 con 0x7fa1780b0410 2023-10-07T09:45:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.674+0000 7fa17eb9c700 1 --2- 172.21.15.29:0/4092778592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780b0410 0x7fa17815a340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa170009e90 tx=0x7fa170009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea4175853647df2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.674+0000 7fa17d399700 1 -- 172.21.15.29:0/4092778592 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17000e070 con 0x7fa1780b0410 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.674+0000 7fa17d399700 1 -- 172.21.15.29:0/4092778592 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17000bb40 con 0x7fa1780b0410 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.674+0000 7fa17d399700 1 -- 172.21.15.29:0/4092778592 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170012430 con 0x7fa1780b0410 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 -- 172.21.15.29:0/4092778592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780b0410 msgr2=0x7fa17815a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 --2- 172.21.15.29:0/4092778592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780b0410 0x7fa17815a340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa170009e90 tx=0x7fa170009780 comp rx=0 tx=0).stop 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 -- 172.21.15.29:0/4092778592 shutdown_connections 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 --2- 172.21.15.29:0/4092778592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780b0410 0x7fa17815a340 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 --2- 172.21.15.29:0/4092778592 >> v2:172.21.15.144:3300/0 conn(0x7fa1780ad370 0x7fa1780cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 -- 172.21.15.29:0/4092778592 >> 172.21.15.29:0/4092778592 conn(0x7fa1780680a0 msgr2=0x7fa1780cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 -- 172.21.15.29:0/4092778592 shutdown_connections 2023-10-07T09:45:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.675+0000 7fa1805ff700 1 -- 172.21.15.29:0/4092778592 wait complete. 2023-10-07T09:45:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.676+0000 7fa1805ff700 1 Processor -- start 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.676+0000 7fa1805ff700 1 -- start start 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.677+0000 7fa1805ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780ad370 0x7fa17813ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.677+0000 7fa1805ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 0x7fa1780cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.677+0000 7fa1805ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa178159ec0 0x7fa1780cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.677+0000 7fa1805ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ab2f0 con 0x7fa1780b0410 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.677+0000 7fa1805ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780aad30 con 0x7fa178159ec0 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.677+0000 7fa1805ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ab010 con 0x7fa1780ad370 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 0x7fa1780cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 0x7fa1780cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47290/0 (socket says 172.21.15.29:47290) 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 -- 172.21.15.29:0/3513824127 learned_addr learned my addr 172.21.15.29:0/3513824127 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17e39b700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780ad370 0x7fa17813ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17eb9c700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa178159ec0 0x7fa1780cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 -- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780ad370 msgr2=0x7fa17813ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780ad370 0x7fa17813ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 -- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa178159ec0 msgr2=0x7fa1780cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa178159ec0 0x7fa1780cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 -- 172.21.15.29:0/3513824127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170009a20 con 0x7fa1780b0410 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa17db9a700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 0x7fa1780cc050 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c590 tx=0x7fa168007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.678+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168005b30 con 0x7fa1780b0410 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.679+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168005cb0 con 0x7fa1780b0410 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.679+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168007320 con 0x7fa1780b0410 2023-10-07T09:45:45.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.679+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1780a5560 con 0x7fa1780b0410 2023-10-07T09:45:45.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.679+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17813ef50 con 0x7fa1780b0410 2023-10-07T09:45:45.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.680+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17804c3c0 con 0x7fa1780b0410 2023-10-07T09:45:45.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.681+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa16801b3b0 con 0x7fa1780b0410 2023-10-07T09:45:45.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.681+0000 7fa16e7fc700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa15c066f30 0x7fa15c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:45.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.682+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fa168083b30 con 0x7fa1780b0410 2023-10-07T09:45:45.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.682+0000 7fa17e39b700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa15c066f30 0x7fa15c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:45.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.682+0000 7fa17e39b700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa15c066f30 0x7fa15c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa174009db0 tx=0x7fa1740091a0 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:45.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.686+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa168050c20 con 0x7fa1780b0410 2023-10-07T09:45:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.935+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa1780ccb90 con 0x7fa1780b0410 2023-10-07T09:45:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.936+0000 7fa16e7fc700 1 -- 172.21.15.29:0/3513824127 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa168017020 con 0x7fa1780b0410 2023-10-07T09:45:45.938 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:45:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.938+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa15c066f30 msgr2=0x7fa15c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.938+0000 7fa1805ff700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa15c066f30 0x7fa15c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa174009db0 tx=0x7fa1740091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.938+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 msgr2=0x7fa1780cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.938+0000 7fa1805ff700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 0x7fa1780cc050 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c590 tx=0x7fa168007e00 comp rx=0 tx=0).stop 2023-10-07T09:45:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 shutdown_connections 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa15c066f30 0x7fa15c0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa178159ec0 0x7fa1780cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1780ad370 0x7fa17813ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 --2- 172.21.15.29:0/3513824127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1780b0410 0x7fa1780cc050 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 >> 172.21.15.29:0/3513824127 conn(0x7fa1780680a0 msgr2=0x7fa1780afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 shutdown_connections 2023-10-07T09:45:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:45.939+0000 7fa1805ff700 1 -- 172.21.15.29:0/3513824127 wait complete. 2023-10-07T09:45:45.957 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:45:45.958 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:45:45.959 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:45:45.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:45.959 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:45:46.006 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:45:46.007 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41283sec preferred_lft 41283sec 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.008 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:45:46.009 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:45:46.010 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.010 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:45:46.010 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:45:46.010 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:45:46.010 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:45:46.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:46.011 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:45:46.079 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:45:46.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:46.083 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:45:46.132 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:45:46.132 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:45:46.132 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:45:46.132 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:45:46.132 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:45:46.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:46.133 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:45:46.133 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:45:46.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:45:46.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:45:46.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:45:46.133 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:45:46.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:46.307 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:45:46.353 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:45:46.354 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:45:46.354 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:45:46.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:46.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:45:46.426 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:45:46.426 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:46.472 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:45:46.473 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:45:46.473 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:46.558 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:45:46.558 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:45:46.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:46.699 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:46.699 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:45:46.699 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:45:46.714 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:46.714 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:45:46.714 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:45:46.714 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:45:46.714 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:45:46.737 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:45:46.738 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:45:46.741 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:45:46.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:45:46.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:45:46.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:45:47.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.012+0000 7f9b0a7ff700 1 Processor -- start 2023-10-07T09:45:47.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.012+0000 7f9b0a7ff700 1 -- start start 2023-10-07T09:45:47.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b0a7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b040ad370 0x7f9b040cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b0a7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b0a7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0414d790 con 0x7f9b04159ec0 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b0a7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0415a880 con 0x7f9b040ad370 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b0a7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040ae5d0 con 0x7f9b040b0410 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b03fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b040ad370 0x7f9b040cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b03fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b040ad370 0x7f9b040cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47300/0 (socket says 172.21.15.29:47300) 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b037fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b04159ec0 0x7f9b041568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53328/0 (socket says 172.21.15.29:53328) 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b03fff700 1 -- 172.21.15.29:0/311707501 learned_addr learned my addr 172.21.15.29:0/311707501 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b03fff700 1 -- 172.21.15.29:0/311707501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 msgr2=0x7f9b0415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b08d9c700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b03fff700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b03fff700 1 -- 172.21.15.29:0/311707501 >> v1:172.21.15.29:6789/0 conn(0x7f9b04159ec0 legacy=0x7f9b041568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.014+0000 7f9b02ffd700 1 -- 172.21.15.29:0/311707501 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081152810 0 0) 0x7f9b0414d790 con 0x7f9b04159ec0 2023-10-07T09:45:47.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.015+0000 7f9b03fff700 1 -- 172.21.15.29:0/311707501 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040ad740 con 0x7f9b040ad370 2023-10-07T09:45:47.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.015+0000 7f9b08d9c700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:45:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.015+0000 7f9b03fff700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b040ad370 0x7f9b040cc870 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f9af40099e0 tx=0x7f9af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5677aa443b9a07 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.015+0000 7f9b02ffd700 1 -- 172.21.15.29:0/311707501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400b820 con 0x7f9b040ad370 2023-10-07T09:45:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.015+0000 7f9b02ffd700 1 -- 172.21.15.29:0/311707501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af400b980 con 0x7f9b040ad370 2023-10-07T09:45:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.015+0000 7f9b02ffd700 1 -- 172.21.15.29:0/311707501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400bd20 con 0x7f9b040ad370 2023-10-07T09:45:47.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/311707501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b040ad370 msgr2=0x7f9b040cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b040ad370 0x7f9b040cc870 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f9af40099e0 tx=0x7f9af4009250 comp rx=0 tx=0).stop 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/311707501 shutdown_connections 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/311707501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b040ad370 0x7f9b040cc870 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/311707501 >> 172.21.15.29:0/311707501 conn(0x7f9b040680a0 msgr2=0x7f9b040cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/311707501 shutdown_connections 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.016+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/311707501 wait complete. 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.017+0000 7f9b0a7ff700 1 Processor -- start 2023-10-07T09:45:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.017+0000 7f9b0a7ff700 1 -- start start 2023-10-07T09:45:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.018+0000 7f9b0a7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b040ad370 0x7f9b0416c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.018+0000 7f9b0a7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0416a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.018+0000 7f9b0a7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 0x7f9b0414f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.018+0000 7f9b0a7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0416cbe0 con 0x7f9b04159ec0 2023-10-07T09:45:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.018+0000 7f9b0a7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040c19b0 con 0x7f9b040ad370 2023-10-07T09:45:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.018+0000 7f9b0a7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b04156b10 con 0x7f9b040b0410 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 0x7f9b0414f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 0x7f9b0414f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47314/0 (socket says 172.21.15.29:47314) 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 -- 172.21.15.29:0/4130181741 learned_addr learned my addr 172.21.15.29:0/4130181741 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b03fff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b040ad370 0x7f9b0416c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b037fe700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0416a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 -- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 msgr2=0x7f9b0416a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0416a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 -- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b040ad370 msgr2=0x7f9b0416c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b040ad370 0x7f9b0416c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 -- 172.21.15.29:0/4130181741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40094f0 con 0x7f9b04159ec0 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.019+0000 7f9b08d9c700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 0x7f9b0414f7b0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00a3b0 tx=0x7f9afc007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.020+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc015440 con 0x7f9b04159ec0 2023-10-07T09:45:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.020+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc0155a0 con 0x7f9b04159ec0 2023-10-07T09:45:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.020+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b040c1080 con 0x7f9b04159ec0 2023-10-07T09:45:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.020+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc015960 con 0x7f9b04159ec0 2023-10-07T09:45:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.020+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b040c1640 con 0x7f9b04159ec0 2023-10-07T09:45:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.021+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b040c1b40 con 0x7f9b04159ec0 2023-10-07T09:45:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.022+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9afc015d50 con 0x7f9b04159ec0 2023-10-07T09:45:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.023+0000 7f9aebfff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9af0066f30 0x7f9af00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.023+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f9afc084fb0 con 0x7f9b04159ec0 2023-10-07T09:45:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.023+0000 7f9b03fff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9af0066f30 0x7f9af00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.024+0000 7f9b03fff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9af0066f30 0x7f9af00693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9af40099e0 tx=0x7f9af4019000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:47.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.027+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9afc052000 con 0x7f9b04159ec0 2023-10-07T09:45:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.256+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b04070d30 con 0x7f9af0066f30 2023-10-07T09:45:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.258+0000 7f9aebfff700 1 -- 172.21.15.29:0/4130181741 <== mgr.6721 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9af400bba0 con 0x7f9af0066f30 2023-10-07T09:45:47.259 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:45:47.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:45:47.259 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:45:47.259 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:45:47.260 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9af0066f30 msgr2=0x7f9af00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9af0066f30 0x7f9af00693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9af40099e0 tx=0x7f9af4019000 comp rx=0 tx=0).stop 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 msgr2=0x7f9b0414f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 0x7f9b0414f7b0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00a3b0 tx=0x7f9afc007900 comp rx=0 tx=0).stop 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 shutdown_connections 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9af0066f30 0x7f9af00693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04159ec0 0x7f9b0414f7b0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b040ad370 0x7f9b0416c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.260+0000 7f9b0a7ff700 1 --2- 172.21.15.29:0/4130181741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9b040b0410 0x7f9b0416a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.261+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 >> 172.21.15.29:0/4130181741 conn(0x7f9b040680a0 msgr2=0x7f9b0406ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.261+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 shutdown_connections 2023-10-07T09:45:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.261+0000 7f9b0a7ff700 1 -- 172.21.15.29:0/4130181741 wait complete. 2023-10-07T09:45:47.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:45:47.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.476+0000 7f99685ff700 1 Processor -- start 2023-10-07T09:45:47.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.476+0000 7f99685ff700 1 -- start start 2023-10-07T09:45:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.477+0000 7f99685ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99600b0c30 0x7f99600afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.477+0000 7f99685ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600ad4d0 con 0x7f99600ae570 2023-10-07T09:45:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.477+0000 7f99685ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600ad7b0 con 0x7f9960156180 2023-10-07T09:45:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.477+0000 7f99685ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99600ada90 con 0x7f99600b0c30 2023-10-07T09:45:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f99600ae570 0x7f99600b0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53334/0 (socket says 172.21.15.29:53334) 2023-10-07T09:45:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965b9a700 1 -- 172.21.15.29:0/898729436 learned_addr learned my addr 172.21.15.29:0/898729436 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9966b9c700 1 --2- 172.21.15.29:0/898729436 >> v2:172.21.15.144:3300/0 conn(0x7f99600b0c30 0x7f99600afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097111667 0 0) 0x7f99600ad4d0 con 0x7f99600ae570 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c003570 con 0x7f99600ae570 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2397182193 0 0) 0x7f994c003570 con 0x7f99600ae570 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 >> v2:172.21.15.144:3300/0 conn(0x7f99600b0c30 msgr2=0x7f99600afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.478+0000 7f9965399700 1 --2- 172.21.15.29:0/898729436 >> v2:172.21.15.144:3300/0 conn(0x7f99600b0c30 0x7f99600afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 >> v1:172.21.15.144:6789/0 conn(0x7f9960156180 legacy=0x7f99600accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600add70 con 0x7f99600ae570 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f9950001470 con 0x7f99600ae570 2023-10-07T09:45:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 >> v1:172.21.15.29:6789/0 conn(0x7f99600ae570 legacy=0x7f99600b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f994c003a60 0x7f994c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f994c004810 0x7f994c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c0051c0 0x7f994c00dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600ada90 con 0x7f994c003a60 2023-10-07T09:45:47.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c00e2e0 con 0x7f994c004810 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c00e4a0 con 0x7f994c0051c0 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f996639b700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f994c003a60 0x7f994c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9965b9a700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f994c004810 0x7f994c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9966b9c700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c0051c0 0x7f994c00dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f996639b700 1 -- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c0051c0 msgr2=0x7f994c00dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f996639b700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c0051c0 0x7f994c00dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f996639b700 1 -- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f994c004810 msgr2=0x7f994c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f996639b700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f994c004810 0x7f994c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f996639b700 1 -- 172.21.15.29:0/898729436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c003570 con 0x7f994c003a60 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9966b9c700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c0051c0 0x7f994c00dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:45:47.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.479+0000 7f9966b9c700 1 -- 172.21.15.29:0/898729436 reap_dead start 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f996639b700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f994c003a60 0x7f994c003e70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f995c0018e0 tx=0x7f995c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbcdd1731be1339 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c012420 con 0x7f994c003a60 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c0125a0 con 0x7f994c003a60 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f9965399700 1 -- 172.21.15.29:0/898729436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c012960 con 0x7f994c003a60 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f99685ff700 1 -- 172.21.15.29:0/898729436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f994c003a60 msgr2=0x7f994c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f99685ff700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f994c003a60 0x7f994c003e70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f995c0018e0 tx=0x7f995c007d30 comp rx=0 tx=0).stop 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f99685ff700 1 -- 172.21.15.29:0/898729436 shutdown_connections 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f99685ff700 1 --2- 172.21.15.29:0/898729436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f994c003a60 0x7f994c003e70 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f99685ff700 1 -- 172.21.15.29:0/898729436 >> 172.21.15.29:0/898729436 conn(0x7f9960067610 msgr2=0x7f9960067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.480+0000 7f99685ff700 1 -- 172.21.15.29:0/898729436 shutdown_connections 2023-10-07T09:45:47.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.481+0000 7f99685ff700 1 -- 172.21.15.29:0/898729436 wait complete. 2023-10-07T09:45:47.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.482+0000 7f99685ff700 1 Processor -- start 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.482+0000 7f99685ff700 1 -- start start 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f99685ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c003a60 0x7f99600ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f99685ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960147de0 0x7f99600aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f99685ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 0x7f99600c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f99685ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600ac130 con 0x7f99600aaec0 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f99685ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600ab520 con 0x7f9960147de0 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f99685ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99600ab800 con 0x7f994c003a60 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f9966b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 0x7f99600c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f996639b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c003a60 0x7f99600ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f9966b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 0x7f99600c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47342/0 (socket says 172.21.15.29:47342) 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.483+0000 7f9966b9c700 1 -- 172.21.15.29:0/2694002344 learned_addr learned my addr 172.21.15.29:0/2694002344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:47.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9965b9a700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960147de0 0x7f99600aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9966b9c700 1 -- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c003a60 msgr2=0x7f99600ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9966b9c700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c003a60 0x7f99600ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9966b9c700 1 -- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960147de0 msgr2=0x7f99600aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9966b9c700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960147de0 0x7f99600aa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9966b9c700 1 -- 172.21.15.29:0/2694002344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600abae0 con 0x7f99600aaec0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f9966b9c700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 0x7f99600c57e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f996006cd20 tx=0x7f995800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995801b070 con 0x7f99600aaec0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995800f040 con 0x7f99600aaec0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800e050 con 0x7f99600aaec0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.484+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99600abdc0 con 0x7f99600aaec0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.485+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9960148250 con 0x7f99600aaec0 2023-10-07T09:45:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.486+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996006b570 con 0x7f99600aaec0 2023-10-07T09:45:47.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.486+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9958003020 con 0x7f99600aaec0 2023-10-07T09:45:47.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.487+0000 7f99567fc700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9944066f30 0x7f99440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:47.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.487+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f9958084a50 con 0x7f99600aaec0 2023-10-07T09:45:47.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.488+0000 7f996639b700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9944066f30 0x7f99440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:47.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.488+0000 7f996639b700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9944066f30 0x7f99440693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f995c001050 tx=0x7f995c015000 comp rx=0 tx=0).ready entity=mgr.6721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:47.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.492+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9958051aa0 con 0x7f99600aaec0 2023-10-07T09:45:47.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:47.754+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f996004c3c0 con 0x7f99600aaec0 2023-10-07T09:45:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.169+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 41) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f9958051e70 con 0x7f99600aaec0 2023-10-07T09:45:48.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.170+0000 7f99567fc700 1 -- 172.21.15.29:0/2694002344 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v41) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9958051aa0 con 0x7f99600aaec0 2023-10-07T09:45:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9944066f30 msgr2=0x7f99440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9944066f30 0x7f99440693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f995c001050 tx=0x7f995c015000 comp rx=0 tx=0).stop 2023-10-07T09:45:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 msgr2=0x7f99600c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 0x7f99600c57e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f996006cd20 tx=0x7f995800b040 comp rx=0 tx=0).stop 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 shutdown_connections 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9944066f30 0x7f99440693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600aaec0 0x7f99600c57e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f994c003a60 0x7f99600ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 --2- 172.21.15.29:0/2694002344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960147de0 0x7f99600aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 >> 172.21.15.29:0/2694002344 conn(0x7f9960067610 msgr2=0x7f99600c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 shutdown_connections 2023-10-07T09:45:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.173+0000 7f99685ff700 1 -- 172.21.15.29:0/2694002344 wait complete. 2023-10-07T09:45:48.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:45:48.383 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:48.382+0000 7f90a788c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:45:48.386 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:48.385+0000 7f1bc94eb1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:45:48.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.407+0000 7fc2a11ff700 1 Processor -- start 2023-10-07T09:45:48.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.407+0000 7fc2a11ff700 1 -- start start 2023-10-07T09:45:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.408+0000 7fc2a11ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc29c159300 0x7fc29c069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.408+0000 7fc2a11ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c15da10 0x7fc29c0b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.408+0000 7fc2a11ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0b0a10 con 0x7fc29c0b0060 2023-10-07T09:45:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.408+0000 7fc2a11ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c15de70 con 0x7fc29c159300 2023-10-07T09:45:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.408+0000 7fc2a11ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c069400 con 0x7fc29c15da10 2023-10-07T09:45:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc29c159300 0x7fc29c069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29b59e700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c15da10 0x7fc29c0b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc29c159300 0x7fc29c069890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:48366/0 (socket says 172.21.15.29:48366) 2023-10-07T09:45:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 -- 172.21.15.29:0/4184011027 learned_addr learned my addr 172.21.15.29:0/4184011027 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc299d9b700 1 -- 172.21.15.29:0/4184011027 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103565048 0 0) 0x7fc29c0b0a10 con 0x7fc29c0b0060 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc299d9b700 1 -- 172.21.15.29:0/4184011027 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc284003570 con 0x7fc29c0b0060 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 -- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c15da10 msgr2=0x7fc29c0b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 --2- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c15da10 0x7fc29c0b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 -- 172.21.15.29:0/4184011027 >> v1:172.21.15.144:6789/0 conn(0x7fc29c0b0060 legacy=0x7fc29c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29ad9d700 1 -- 172.21.15.29:0/4184011027 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c069dd0 con 0x7fc29c159300 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.409+0000 7fc29b59e700 1 --2- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c15da10 0x7fc29c0b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc29ad9d700 1 --2- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c159300 0x7fc29c069890 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009990 tx=0x7fc28c009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0a0554d9a2dab8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc299d9b700 1 -- 172.21.15.29:0/4184011027 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c005c10 con 0x7fc29c159300 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc299d9b700 1 -- 172.21.15.29:0/4184011027 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c005d70 con 0x7fc29c159300 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc299d9b700 1 -- 172.21.15.29:0/4184011027 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c006260 con 0x7fc29c159300 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc2a11ff700 1 -- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c159300 msgr2=0x7fc29c069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c159300 0x7fc29c069890 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009990 tx=0x7fc28c009220 comp rx=0 tx=0).stop 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc2a11ff700 1 -- 172.21.15.29:0/4184011027 shutdown_connections 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.410+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c15da10 0x7fc29c0b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.411+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/4184011027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c159300 0x7fc29c069890 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.411+0000 7fc2a11ff700 1 -- 172.21.15.29:0/4184011027 >> 172.21.15.29:0/4184011027 conn(0x7fc29c067610 msgr2=0x7fc29c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.411+0000 7fc2a11ff700 1 -- 172.21.15.29:0/4184011027 shutdown_connections 2023-10-07T09:45:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.411+0000 7fc2a11ff700 1 -- 172.21.15.29:0/4184011027 wait complete. 2023-10-07T09:45:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.412+0000 7fc2a11ff700 1 Processor -- start 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.412+0000 7fc2a11ff700 1 -- start start 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc2a11ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 0x7fc29c0cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc2a11ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc29c159300 0x7fc29c0cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc2a11ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c15da10 0x7fc29c14fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc2a11ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0c9170 con 0x7fc29c159300 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc2a11ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0c8e70 con 0x7fc29c15da10 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc2a11ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0ce590 con 0x7fc29c0b0060 2023-10-07T09:45:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.413+0000 7fc29ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 0x7fc29c0cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 0x7fc29c0cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:41152/0 (socket says 172.21.15.29:41152) 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 -- 172.21.15.29:0/2365422282 learned_addr learned my addr 172.21.15.29:0/2365422282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29a59c700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc29c159300 0x7fc29c0cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29b59e700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c15da10 0x7fc29c14fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 -- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c15da10 msgr2=0x7fc29c14fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c15da10 0x7fc29c14fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 -- 172.21.15.29:0/2365422282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc29c159300 msgr2=0x7fc29c0cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc29c159300 0x7fc29c0cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 -- 172.21.15.29:0/2365422282 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294009a20 con 0x7fc29c0b0060 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc29ad9d700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 0x7fc29c0cdef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0add90 tx=0x7fc28c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.414+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0119b0 con 0x7fc29c0b0060 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.415+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c011b30 con 0x7fc29c0b0060 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.415+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c025470 con 0x7fc29c0b0060 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.415+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c009480 con 0x7fc29c0b0060 2023-10-07T09:45:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.415+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c0ceb50 con 0x7fc29c0b0060 2023-10-07T09:45:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.416+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c04c3c0 con 0x7fc29c0b0060 2023-10-07T09:45:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.417+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 41) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fc28c022070 con 0x7fc29c0b0060 2023-10-07T09:45:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.418+0000 7fc282ffd700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 0x7fc288069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.418+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fc28c0863c0 con 0x7fc29c0b0060 2023-10-07T09:45:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.418+0000 7fc29a59c700 1 -- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 msgr2=0x7fc288069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.418+0000 7fc29a59c700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 0x7fc288069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.422+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc28c053600 con 0x7fc29c0b0060 2023-10-07T09:45:48.561 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:48.560+0000 7f90a788c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:45:48.620 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:48.619+0000 7f1bc94eb1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:45:48.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.619+0000 7fc29a59c700 1 -- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 msgr2=0x7fc288069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:48.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.619+0000 7fc29a59c700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 0x7fc288069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.701+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc29c067df0 con 0x7fc29c0b0060 2023-10-07T09:45:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.707+0000 7fc282ffd700 1 -- 172.21.15.29:0/2365422282 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc28c056c20 con 0x7fc29c0b0060 2023-10-07T09:45:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 msgr2=0x7fc288069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 0x7fc288069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 msgr2=0x7fc29c0cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 0x7fc29c0cdef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0add90 tx=0x7fc28c016000 comp rx=0 tx=0).stop 2023-10-07T09:45:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 shutdown_connections 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc288066f80 0x7fc288069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc29c15da10 0x7fc29c14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc29c0b0060 0x7fc29c0cdef0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 --2- 172.21.15.29:0/2365422282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc29c159300 0x7fc29c0cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 >> 172.21.15.29:0/2365422282 conn(0x7fc29c067610 msgr2=0x7fc29c1540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 shutdown_connections 2023-10-07T09:45:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.710+0000 7fc2a11ff700 1 -- 172.21.15.29:0/2365422282 wait complete. 2023-10-07T09:45:48.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:45:48.873 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:48.872+0000 7f90a788c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:45:48.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.931+0000 7f1cf35ff700 1 Processor -- start 2023-10-07T09:45:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.932+0000 7f1cf35ff700 1 -- start start 2023-10-07T09:45:48.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf35ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159750 0x7f1cec0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf35ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 0x7f1cec0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf35ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0cbf20 con 0x7f1cec0ce6d0 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf35ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec0cc200 con 0x7f1cec159b20 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf35ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0cc4e0 con 0x7f1cec159750 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf0b9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 0x7f1cec0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.933+0000 7f1cf139b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159750 0x7f1cec0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf1b9c700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1cec0ce6d0 0x7f1cec15d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:54628/0 (socket says 172.21.15.29:54628) 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf1b9c700 1 -- 172.21.15.29:0/1188096508 learned_addr learned my addr 172.21.15.29:0/1188096508 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf0b9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 0x7f1cec0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:48376/0 (socket says 172.21.15.29:48376) 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1ce3fff700 1 -- 172.21.15.29:0/1188096508 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265977595 0 0) 0x7f1cec0cbf20 con 0x7f1cec0ce6d0 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1ce3fff700 1 -- 172.21.15.29:0/1188096508 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8003570 con 0x7f1cec0ce6d0 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf139b700 1 -- 172.21.15.29:0/1188096508 >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 msgr2=0x7f1cec0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf139b700 1 --2- 172.21.15.29:0/1188096508 >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 0x7f1cec0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf139b700 1 -- 172.21.15.29:0/1188096508 >> v1:172.21.15.144:6789/0 conn(0x7f1cec0ce6d0 legacy=0x7f1cec15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf139b700 1 -- 172.21.15.29:0/1188096508 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0cc7c0 con 0x7f1cec159750 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf0b9a700 1 --2- 172.21.15.29:0/1188096508 >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 0x7f1cec0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.934+0000 7f1cf139b700 1 --2- 172.21.15.29:0/1188096508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159750 0x7f1cec0d10a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098d0 tx=0x7f1ce8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fc3e2ff6bcd5c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1ce3fff700 1 -- 172.21.15.29:0/1188096508 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8005c10 con 0x7f1cec159750 2023-10-07T09:45:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1ce3fff700 1 -- 172.21.15.29:0/1188096508 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce8005d70 con 0x7f1cec159750 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1ce3fff700 1 -- 172.21.15.29:0/1188096508 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8006260 con 0x7f1cec159750 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 -- 172.21.15.29:0/1188096508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159750 msgr2=0x7f1cec0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/1188096508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159750 0x7f1cec0d10a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098d0 tx=0x7f1ce8009120 comp rx=0 tx=0).stop 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 -- 172.21.15.29:0/1188096508 shutdown_connections 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/1188096508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159750 0x7f1cec0d10a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/1188096508 >> v2:172.21.15.144:3300/0 conn(0x7f1cec159b20 0x7f1cec0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 -- 172.21.15.29:0/1188096508 >> 172.21.15.29:0/1188096508 conn(0x7f1cec063a10 msgr2=0x7f1cec0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 -- 172.21.15.29:0/1188096508 shutdown_connections 2023-10-07T09:45:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.935+0000 7f1cf35ff700 1 -- 172.21.15.29:0/1188096508 wait complete. 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.937+0000 7f1cf35ff700 1 Processor -- start 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.937+0000 7f1cf35ff700 1 -- start start 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf35ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cec0ce6d0 0x7f1cec151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf35ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 0x7f1cec16cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf35ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159b20 0x7f1cec0b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf35ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec16db70 con 0x7f1cec159750 2023-10-07T09:45:48.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf35ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec15ec90 con 0x7f1cec0ce6d0 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf35ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec16d5b0 con 0x7f1cec159b20 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf0b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 0x7f1cec16cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf0b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 0x7f1cec16cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47370/0 (socket says 172.21.15.29:47370) 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf0b9a700 1 -- 172.21.15.29:0/365245425 learned_addr learned my addr 172.21.15.29:0/365245425 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf1b9c700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159b20 0x7f1cec0b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.938+0000 7f1cf139b700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cec0ce6d0 0x7f1cec151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf0b9a700 1 -- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159b20 msgr2=0x7f1cec0b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf0b9a700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159b20 0x7f1cec0b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf0b9a700 1 -- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cec0ce6d0 msgr2=0x7f1cec151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf0b9a700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cec0ce6d0 0x7f1cec151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf0b9a700 1 -- 172.21.15.29:0/365245425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc009420 con 0x7f1cec159750 2023-10-07T09:45:48.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf0b9a700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 0x7f1cec16cba0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00ec80 tx=0x7f1cdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00c850 con 0x7f1cec159750 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc00c9b0 con 0x7f1cec159750 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00cd30 con 0x7f1cec159750 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.939+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce80093c0 con 0x7f1cec159750 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.940+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec15e920 con 0x7f1cec159750 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.941+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec15e780 con 0x7f1cec159750 2023-10-07T09:45:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.941+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f1cdc013680 con 0x7f1cec159750 2023-10-07T09:45:48.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.943+0000 7f1ce17fa700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 0x7f1cd0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:48.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.943+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1cdc084990 con 0x7f1cec159750 2023-10-07T09:45:48.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.943+0000 7f1cf139b700 1 -- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 msgr2=0x7f1cd0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:48.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.943+0000 7f1cf139b700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 0x7f1cd0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:48.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:48.947+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1cec15e780 con 0x7f1cec159750 2023-10-07T09:45:48.961 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:48.960+0000 7f1bc94eb1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:45:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.144+0000 7f1cf139b700 1 -- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 msgr2=0x7f1cd0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.144+0000 7f1cf139b700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 0x7f1cd0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:49.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.154+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f1cec15e780 con 0x7f1cec159750 2023-10-07T09:45:49.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.158+0000 7f1ce17fa700 1 -- 172.21.15.29:0/365245425 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1cec15e780 con 0x7f1cec159750 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 msgr2=0x7f1cd0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 0x7f1cd0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 msgr2=0x7f1cec16cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 0x7f1cec16cba0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00ec80 tx=0x7f1cdc0091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 shutdown_connections 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cd0066f80 0x7f1cd0069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cec159b20 0x7f1cec0b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cec0ce6d0 0x7f1cec151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 --2- 172.21.15.29:0/365245425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cec159750 0x7f1cec16cba0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.160+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 >> 172.21.15.29:0/365245425 conn(0x7f1cec063a10 msgr2=0x7f1cec0ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.161+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 shutdown_connections 2023-10-07T09:45:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.161+0000 7f1cf35ff700 1 -- 172.21.15.29:0/365245425 wait complete. 2023-10-07T09:45:49.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /bad-path 1M 2023-10-07T09:45:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.390+0000 7f322bbff700 1 Processor -- start 2023-10-07T09:45:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.390+0000 7f322bbff700 1 -- start start 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322bbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f32240b24e0 0x7f32240cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322bbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32240cccc0 con 0x7f32240aed00 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322bbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32240b30d0 con 0x7f3224159580 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322bbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322415d790 con 0x7f32240b24e0 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322a19c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32240aed00 0x7f3224159950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53350/0 (socket says 172.21.15.29:53350) 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322a19c700 1 -- 172.21.15.29:0/1831516422 learned_addr learned my addr 172.21.15.29:0/1831516422 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f322919a700 1 --2- 172.21.15.29:0/1831516422 >> v2:172.21.15.144:3300/0 conn(0x7f32240b24e0 0x7f32240cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f3228999700 1 -- 172.21.15.29:0/1831516422 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245285455 0 0) 0x7f32240cccc0 con 0x7f32240aed00 2023-10-07T09:45:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.392+0000 7f3228999700 1 -- 172.21.15.29:0/1831516422 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3210003570 con 0x7f32240aed00 2023-10-07T09:45:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f322919a700 1 -- 172.21.15.29:0/1831516422 >> v1:172.21.15.144:6789/0 conn(0x7f3224159580 legacy=0x7f322415dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f322919a700 1 -- 172.21.15.29:0/1831516422 >> v1:172.21.15.29:6789/0 conn(0x7f32240aed00 legacy=0x7f3224159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f322919a700 1 -- 172.21.15.29:0/1831516422 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32240b2b10 con 0x7f32240b24e0 2023-10-07T09:45:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f322919a700 1 --2- 172.21.15.29:0/1831516422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32240b24e0 0x7f32240cda60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32140098e0 tx=0x7f32140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f32dba69319f68ed server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f3228999700 1 -- 172.21.15.29:0/1831516422 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400b640 con 0x7f32240b24e0 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f3228999700 1 -- 172.21.15.29:0/1831516422 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321400b7a0 con 0x7f32240b24e0 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.393+0000 7f3228999700 1 -- 172.21.15.29:0/1831516422 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400bb40 con 0x7f32240b24e0 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 -- 172.21.15.29:0/1831516422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32240b24e0 msgr2=0x7f32240cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 --2- 172.21.15.29:0/1831516422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32240b24e0 0x7f32240cda60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32140098e0 tx=0x7f32140091a0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 -- 172.21.15.29:0/1831516422 shutdown_connections 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 --2- 172.21.15.29:0/1831516422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32240b24e0 0x7f32240cda60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 -- 172.21.15.29:0/1831516422 >> 172.21.15.29:0/1831516422 conn(0x7f3224068720 msgr2=0x7f3224068b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 -- 172.21.15.29:0/1831516422 shutdown_connections 2023-10-07T09:45:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.394+0000 7f322bbff700 1 -- 172.21.15.29:0/1831516422 wait complete. 2023-10-07T09:45:49.397 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:49.394+0000 7f90a788c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:45:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.396+0000 7f322bbff700 1 Processor -- start 2023-10-07T09:45:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.396+0000 7f322bbff700 1 -- start start 2023-10-07T09:45:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322bbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32240aed00 0x7f322415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322bbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 0x7f322416cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322bbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224159580 0x7f3224144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322bbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322416d490 con 0x7f32240b24e0 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322bbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3224169c60 con 0x7f3224159580 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322bbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3224169920 con 0x7f32240aed00 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322919a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 0x7f322416cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322919a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 0x7f322416cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47372/0 (socket says 172.21.15.29:47372) 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322919a700 1 -- 172.21.15.29:0/3949535745 learned_addr learned my addr 172.21.15.29:0/3949535745 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32240aed00 0x7f322415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322a19c700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224159580 0x7f3224144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.397+0000 7f322919a700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32240aed00 msgr2=0x7f322415b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322919a700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32240aed00 0x7f322415b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322919a700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224159580 msgr2=0x7f3224144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322919a700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224159580 0x7f3224144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322919a700 1 -- 172.21.15.29:0/3949535745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009420 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322919a700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 0x7f322416cf50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f32140093f0 tx=0x7f3214017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214000bc0 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321400b910 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214000d20 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322416d6c0 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.398+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322416dc80 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.400+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224169dd0 con 0x7f32240b24e0 2023-10-07T09:45:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.400+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f3214023070 con 0x7f32240b24e0 2023-10-07T09:45:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.401+0000 7f3219ffb700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.401+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f32140859b0 con 0x7f32240b24e0 2023-10-07T09:45:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.401+0000 7f322999b700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f3208069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.401+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:45:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.406+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3214052a00 con 0x7f32240b24e0 2023-10-07T09:45:49.522 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:49.521+0000 7f1bc94eb1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:45:49.565 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:49.564+0000 7f90a788c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:45:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.602+0000 7f322999b700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f3208069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.602+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:45:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:49.635+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f32240718b0 con 0x7f3208066f80 2023-10-07T09:45:49.698 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:49.697+0000 7f1bc94eb1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:45:49.975 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:49.974+0000 7f90a788c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:45:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:50.003+0000 7f322999b700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f3208069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:50.003+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:45:50.126 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:50.125+0000 7f1bc94eb1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:45:50.743 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:50.742+0000 7f90a788c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:45:50.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:50.804+0000 7f322999b700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f3208069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:50.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:50.804+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:45:50.830 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:50.829+0000 7f90a788c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:45:50.931 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:50.930+0000 7f1bc94eb1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:45:50.985 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:50.984+0000 7f90a788c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:45:51.022 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.021+0000 7f1bc94eb1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:45:51.070 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.069+0000 7f90a788c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:45:51.182 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.181+0000 7f1bc94eb1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:45:51.273 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.272+0000 7f1bc94eb1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:45:51.285 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.284+0000 7f90a788c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:45:51.379 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.378+0000 7f90a788c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:45:51.473 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.472+0000 7f90a788c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:45:51.493 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.492+0000 7f1bc94eb1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:45:51.567 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.566+0000 7f90a788c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:45:51.592 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.591+0000 7f1bc94eb1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:45:51.691 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.690+0000 7f1bc94eb1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:45:51.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:45:51.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:45:51.755 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.754+0000 7f90a788c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:45:51.791 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.790+0000 7f1bc94eb1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:45:51.855 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:51.855+0000 7f90a788c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:45:51.988 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:51.987+0000 7f1bc94eb1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:45:52.088 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:52.087+0000 7f1bc94eb1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:45:52.196 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:52.195+0000 7f90a788c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:45:52.283 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:52.282+0000 7f90a788c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:52.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:52.406+0000 7f322999b700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f3208069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:45:52.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:52.406+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:45:52.437 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:52.436+0000 7f1bc94eb1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:45:52.528 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:52.527+0000 7f1bc94eb1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:52.822 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:52.820+0000 7f90a788c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:45:53.006 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:53.005+0000 7f90a788c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:45:53.093 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:53.091+0000 7f1bc94eb1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:45:53.285 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:53.284+0000 7f1bc94eb1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:45:53.380 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:53.379+0000 7f90a788c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:53.668 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:53.667+0000 7f1bc94eb1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:53.794 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:53.793+0000 7f90a788c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:45:53.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:53.973+0000 7f90a788c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:54.109 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:54.108+0000 7f1bc94eb1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:45:54.159 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.158+0000 7f90a788c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:45:54.266 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.265+0000 7f90a788c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:54.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:54.276+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 42) v1 ==== 84468+0+0 (secure 0 0 0) 0x7f3214052dd0 con 0x7f32240b24e0 2023-10-07T09:45:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:54.277+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f3208069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:45:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:54.277+0000 7f3219ffb700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f3208069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:54.301 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:54.300+0000 7f1bc94eb1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:45:54.402 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f9018e31700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.402 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f9018e31700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.402 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f9018e31700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.402 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f9018e31700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.402 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f9018e31700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.403 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f901b636700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.403 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.401+0000 7f901b636700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.403 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.402+0000 7f901b636700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.403 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.402+0000 7f901b636700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.404 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:45:54.402+0000 7f901b636700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:45:54.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:54.504+0000 7f1bc94eb1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:45:54.618 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:45:54.617+0000 7f1bc94eb1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:45:55.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.290+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 43) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f321402e3f0 con 0x7f32240b24e0 2023-10-07T09:45:55.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.291+0000 7f3219ffb700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f32080693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.291+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f32240718b0 con 0x7f3208066f80 2023-10-07T09:45:55.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.296+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f32080693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.299+0000 7f322999b700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f32080693b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3224157f60 tx=0x7f3220007ec0 comp rx=0 tx=0).ready entity=mgr.7060 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:55.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.365+0000 7f3219ffb700 1 -- 172.21.15.29:0/3949535745 <== mgr.7060 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f32240718b0 con 0x7f3208066f80 2023-10-07T09:45:55.366 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /bad-path 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 msgr2=0x7f32080693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f32080693b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3224157f60 tx=0x7f3220007ec0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 msgr2=0x7f322416cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 0x7f322416cf50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f32140093f0 tx=0x7f3214017000 comp rx=0 tx=0).stop 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 shutdown_connections 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3208066f80 0x7f32080693b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224159580 0x7f3224144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32240aed00 0x7f322415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.367+0000 7f322bbff700 1 --2- 172.21.15.29:0/3949535745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32240b24e0 0x7f322416cf50 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.368+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 >> 172.21.15.29:0/3949535745 conn(0x7f3224068720 msgr2=0x7f3224154190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:55.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.368+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 shutdown_connections 2023-10-07T09:45:55.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.368+0000 7f322bbff700 1 -- 172.21.15.29:0/3949535745 wait complete. 2023-10-07T09:45:55.388 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-07T09:45:55.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-10-07T09:45:55.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.587+0000 7f42379ff700 1 Processor -- start 2023-10-07T09:45:55.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.588+0000 7f42379ff700 1 -- start start 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f42379ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230157a20 0x7f42300b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f42379ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42300cb4d0 0x7f42300b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f42379ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300aeae0 con 0x7f4230157650 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f42379ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42300ca430 con 0x7f42300cb4d0 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f42379ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42300ca710 con 0x7f4230157a20 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f423579b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4230157650 0x7f42300ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49014/0 (socket says 172.21.15.29:49014) 2023-10-07T09:45:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f423579b700 1 -- 172.21.15.29:0/2781219617 learned_addr learned my addr 172.21.15.29:0/2781219617 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f4235f9c700 1 --2- 172.21.15.29:0/2781219617 >> v2:172.21.15.29:3300/0 conn(0x7f42300cb4d0 0x7f42300b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.589+0000 7f4234f9a700 1 --2- 172.21.15.29:0/2781219617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230157a20 0x7f42300b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4235f9c700 1 -- 172.21.15.29:0/2781219617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230157a20 msgr2=0x7f42300b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4235f9c700 1 --2- 172.21.15.29:0/2781219617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230157a20 0x7f42300b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4235f9c700 1 -- 172.21.15.29:0/2781219617 >> v1:172.21.15.29:6789/0 conn(0x7f4230157650 legacy=0x7f42300ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4235f9c700 1 -- 172.21.15.29:0/2781219617 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300ca9f0 con 0x7f42300cb4d0 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4227fff700 1 -- 172.21.15.29:0/2781219617 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659546308 0 0) 0x7f42300aeae0 con 0x7f4230157650 2023-10-07T09:45:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4235f9c700 1 --2- 172.21.15.29:0/2781219617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300cb4d0 0x7f42300b17d0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f4228009e60 tx=0x7f4228009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae987f8a815c42d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4227fff700 1 -- 172.21.15.29:0/2781219617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800e070 con 0x7f42300cb4d0 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4227fff700 1 -- 172.21.15.29:0/2781219617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800bc30 con 0x7f42300cb4d0 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.590+0000 7f4227fff700 1 -- 172.21.15.29:0/2781219617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228012430 con 0x7f42300cb4d0 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 -- 172.21.15.29:0/2781219617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300cb4d0 msgr2=0x7f42300b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 --2- 172.21.15.29:0/2781219617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300cb4d0 0x7f42300b17d0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f4228009e60 tx=0x7f4228009780 comp rx=0 tx=0).stop 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 -- 172.21.15.29:0/2781219617 shutdown_connections 2023-10-07T09:45:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 --2- 172.21.15.29:0/2781219617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42300cb4d0 0x7f42300b17d0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 --2- 172.21.15.29:0/2781219617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4230157a20 0x7f42300b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 -- 172.21.15.29:0/2781219617 >> 172.21.15.29:0/2781219617 conn(0x7f4230067610 msgr2=0x7f4230067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 -- 172.21.15.29:0/2781219617 shutdown_connections 2023-10-07T09:45:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.591+0000 7f42379ff700 1 -- 172.21.15.29:0/2781219617 wait complete. 2023-10-07T09:45:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.593+0000 7f42379ff700 1 Processor -- start 2023-10-07T09:45:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.593+0000 7f42379ff700 1 -- start start 2023-10-07T09:45:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f42379ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 0x7f42300b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f42379ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230157650 0x7f4230153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f42379ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4230157a20 0x7f4230151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f42379ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4230154660 con 0x7f4230157a20 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f42379ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4230153a50 con 0x7f4230157650 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f42379ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4230153d30 con 0x7f42300cb4d0 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f423579b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 0x7f42300b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f4235f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4230157a20 0x7f4230151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f423579b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 0x7f42300b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:47284/0 (socket says 172.21.15.29:47284) 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f423579b700 1 -- 172.21.15.29:0/472354934 learned_addr learned my addr 172.21.15.29:0/472354934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f4234f9a700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230157650 0x7f4230153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f4235f9c700 1 -- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 msgr2=0x7f42300b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f4235f9c700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 0x7f42300b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.594+0000 7f4235f9c700 1 -- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230157650 msgr2=0x7f4230153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f4235f9c700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230157650 0x7f4230153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f4235f9c700 1 -- 172.21.15.29:0/472354934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009a20 con 0x7f4230157a20 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f423579b700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 0x7f42300b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f4235f9c700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4230157a20 0x7f4230151860 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f4228016000 tx=0x7f4228019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228027470 con 0x7f4230157a20 2023-10-07T09:45:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42280275f0 con 0x7f4230157a20 2023-10-07T09:45:55.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4230154010 con 0x7f4230157a20 2023-10-07T09:45:55.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280279b0 con 0x7f4230157a20 2023-10-07T09:45:55.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.595+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42300b1240 con 0x7f4230157a20 2023-10-07T09:45:55.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.596+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4230154810 con 0x7f4230157a20 2023-10-07T09:45:55.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.597+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f4228024070 con 0x7f4230157a20 2023-10-07T09:45:55.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.598+0000 7f42257fa700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4214066dd0 0x7f4214069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:45:55.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.598+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f4228086c40 con 0x7f4230157a20 2023-10-07T09:45:55.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.598+0000 7f423579b700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4214066dd0 0x7f4214069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:45:55.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.599+0000 7f423579b700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4214066dd0 0x7f4214069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f422c007fd0 tx=0x7f422c007df0 comp rx=0 tx=0).ready entity=mgr.7060 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:45:55.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.603+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4230154810 con 0x7f4230157a20 2023-10-07T09:45:55.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.828+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4230154810 con 0x7f4214066dd0 2023-10-07T09:45:55.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.837+0000 7f42257fa700 1 -- 172.21.15.29:0/472354934 <== mgr.7060 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7f4230154810 con 0x7f4214066dd0 2023-10-07T09:45:55.839 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:45:55.839 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:45:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4214066dd0 msgr2=0x7f4214069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4214066dd0 0x7f4214069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f422c007fd0 tx=0x7f422c007df0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4230157a20 msgr2=0x7f4230151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4230157a20 0x7f4230151860 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f4228016000 tx=0x7f4228019000 comp rx=0 tx=0).stop 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 shutdown_connections 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4214066dd0 0x7f4214069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4230157a20 0x7f4230151860 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f42300cb4d0 0x7f42300b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 --2- 172.21.15.29:0/472354934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4230157650 0x7f4230153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:45:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 >> 172.21.15.29:0/472354934 conn(0x7f4230067610 msgr2=0x7f42300abe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:45:55.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 shutdown_connections 2023-10-07T09:45:55.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:45:55.839+0000 7f42379ff700 1 -- 172.21.15.29:0/472354934 wait complete. 2023-10-07T09:45:55.858 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:45:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-07T09:46:00.379 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:00.378+0000 7f900b616700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:46:00.380 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:00.378+0000 7f900b616700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:46:00.380 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:00.378+0000 7f900b616700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:46:00.380 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:00.378+0000 7f900b616700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:46:00.380 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:00.378+0000 7f900b616700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:46:21.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:46:21.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:46:51.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:46:51.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:46:55.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-10-07T09:46:56.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.071+0000 7fdd4f9ff700 1 Processor -- start 2023-10-07T09:46:56.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.071+0000 7fdd4f9ff700 1 -- start start 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.072+0000 7fdd4f9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd480b2a20 0x7fdd480cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.072+0000 7fdd4f9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd480cf2c0 0x7fdd480c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.072+0000 7fdd4f9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd480cc940 con 0x7fdd480cb6f0 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.072+0000 7fdd4f9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd480b2df0 con 0x7fdd480cf2c0 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.072+0000 7fdd4f9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd480cbd30 con 0x7fdd480b2a20 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4cf9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdd480cb6f0 0x7fdd480cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54970/0 (socket says 172.21.15.29:54970) 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4cf9a700 1 -- 172.21.15.29:0/758480432 learned_addr learned my addr 172.21.15.29:0/758480432 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4df9c700 1 --2- 172.21.15.29:0/758480432 >> v2:172.21.15.29:3300/0 conn(0x7fdd480cf2c0 0x7fdd480c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4d79b700 1 --2- 172.21.15.29:0/758480432 >> v2:172.21.15.144:3300/0 conn(0x7fdd480b2a20 0x7fdd480cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd3ffff700 1 -- 172.21.15.29:0/758480432 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502808752 0 0) 0x7fdd480cc940 con 0x7fdd480cb6f0 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd3ffff700 1 -- 172.21.15.29:0/758480432 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34003570 con 0x7fdd480cb6f0 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4df9c700 1 -- 172.21.15.29:0/758480432 >> v2:172.21.15.144:3300/0 conn(0x7fdd480b2a20 msgr2=0x7fdd480cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4df9c700 1 --2- 172.21.15.29:0/758480432 >> v2:172.21.15.144:3300/0 conn(0x7fdd480b2a20 0x7fdd480cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4df9c700 1 -- 172.21.15.29:0/758480432 >> v1:172.21.15.29:6789/0 conn(0x7fdd480cb6f0 legacy=0x7fdd480cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4df9c700 1 -- 172.21.15.29:0/758480432 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480cc010 con 0x7fdd480cf2c0 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.073+0000 7fdd4df9c700 1 --2- 172.21.15.29:0/758480432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cf2c0 0x7fdd480c5400 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdd4808ac80 tx=0x7fdd40009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b00b787cc24f27 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:56.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd3ffff700 1 -- 172.21.15.29:0/758480432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000e070 con 0x7fdd480cf2c0 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd3ffff700 1 -- 172.21.15.29:0/758480432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4000bab0 con 0x7fdd480cf2c0 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd3ffff700 1 -- 172.21.15.29:0/758480432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000be70 con 0x7fdd480cf2c0 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/758480432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cf2c0 msgr2=0x7fdd480c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/758480432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cf2c0 0x7fdd480c5400 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdd4808ac80 tx=0x7fdd40009780 comp rx=0 tx=0).stop 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/758480432 shutdown_connections 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/758480432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cf2c0 0x7fdd480c5400 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/758480432 >> v2:172.21.15.144:3300/0 conn(0x7fdd480b2a20 0x7fdd480cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/758480432 >> 172.21.15.29:0/758480432 conn(0x7fdd48099820 msgr2=0x7fdd48063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/758480432 shutdown_connections 2023-10-07T09:46:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.074+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/758480432 wait complete. 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.076+0000 7fdd4f9ff700 1 Processor -- start 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.076+0000 7fdd4f9ff700 1 -- start start 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4f9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd480b2a20 0x7fdd48155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4f9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 0x7fdd4815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4f9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd480cf2c0 0x7fdd4816a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4f9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd48169c50 con 0x7fdd480cb6f0 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4f9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4815b680 con 0x7fdd480cf2c0 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4f9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4815b960 con 0x7fdd480b2a20 2023-10-07T09:46:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4cf9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 0x7fdd4815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4cf9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 0x7fdd4815ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37012/0 (socket says 172.21.15.29:37012) 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4cf9a700 1 -- 172.21.15.29:0/4055156621 learned_addr learned my addr 172.21.15.29:0/4055156621 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4d79b700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd480b2a20 0x7fdd48155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.077+0000 7fdd4df9c700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd480cf2c0 0x7fdd4816a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4cf9a700 1 -- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd480b2a20 msgr2=0x7fdd48155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4cf9a700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd480b2a20 0x7fdd48155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4cf9a700 1 -- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd480cf2c0 msgr2=0x7fdd4816a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4cf9a700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd480cf2c0 0x7fdd4816a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4cf9a700 1 -- 172.21.15.29:0/4055156621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40009a20 con 0x7fdd480cb6f0 2023-10-07T09:46:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4cf9a700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 0x7fdd4815ae50 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800c7c0 tx=0x7fdd38007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:56.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38005cd0 con 0x7fdd480cb6f0 2023-10-07T09:46:56.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd38005e30 con 0x7fdd480cb6f0 2023-10-07T09:46:56.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38007320 con 0x7fdd480cb6f0 2023-10-07T09:46:56.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.078+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4815bc40 con 0x7fdd480cb6f0 2023-10-07T09:46:56.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.079+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd48169860 con 0x7fdd480cb6f0 2023-10-07T09:46:56.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.080+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd48169e00 con 0x7fdd480cb6f0 2023-10-07T09:46:56.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.080+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 84973+0+0 (secure 0 0 0) 0x7fdd3801b3b0 con 0x7fdd480cb6f0 2023-10-07T09:46:56.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.081+0000 7fdd3d7fa700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdd2c0671c0 0x7fdd2c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.081+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdd380842b0 con 0x7fdd480cb6f0 2023-10-07T09:46:56.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.082+0000 7fdd4d79b700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdd2c0671c0 0x7fdd2c069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.082+0000 7fdd4d79b700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdd2c0671c0 0x7fdd2c069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009da0 tx=0x7fdd44009120 comp rx=0 tx=0).ready entity=mgr.7060 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:56.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.085+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd48169e00 con 0x7fdd480cb6f0 2023-10-07T09:46:56.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.316+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd480718b0 con 0x7fdd2c0671c0 2023-10-07T09:46:56.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.324+0000 7fdd3d7fa700 1 -- 172.21.15.29:0/4055156621 <== mgr.7060 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7fdd480718b0 con 0x7fdd2c0671c0 2023-10-07T09:46:56.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:46:56.325 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:45:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-07T09:46:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdd2c0671c0 msgr2=0x7fdd2c069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdd2c0671c0 0x7fdd2c069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009da0 tx=0x7fdd44009120 comp rx=0 tx=0).stop 2023-10-07T09:46:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 msgr2=0x7fdd4815ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 0x7fdd4815ae50 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800c7c0 tx=0x7fdd38007e00 comp rx=0 tx=0).stop 2023-10-07T09:46:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 shutdown_connections 2023-10-07T09:46:56.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdd2c0671c0 0x7fdd2c069670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd480cf2c0 0x7fdd4816a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd480b2a20 0x7fdd48155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 --2- 172.21.15.29:0/4055156621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480cb6f0 0x7fdd4815ae50 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 >> 172.21.15.29:0/4055156621 conn(0x7fdd48099820 msgr2=0x7fdd48158440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.326+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 shutdown_connections 2023-10-07T09:46:56.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.327+0000 7fdd4f9ff700 1 -- 172.21.15.29:0/4055156621 wait complete. 2023-10-07T09:46:56.345 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:45:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-07T09:46:56.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /bad-path 2023-10-07T09:46:56.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.547+0000 7f7754fff700 1 Processor -- start 2023-10-07T09:46:56.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.547+0000 7f7754fff700 1 -- start start 2023-10-07T09:46:56.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f7754fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f77500b0610 0x7f77500ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f7754fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77500b09e0 0x7f77500c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f7754fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f7754fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77500690f0 con 0x7f77500b09e0 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f7754fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77500c4740 con 0x7f77500b0610 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f7754fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500cf2e0 con 0x7f77500c8a20 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f774dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77500b09e0 0x7f77500c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f774dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77500b09e0 0x7f77500c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37014/0 (socket says 172.21.15.29:37014) 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.548+0000 7f774dd9b700 1 -- 172.21.15.29:0/3511724306 learned_addr learned my addr 172.21.15.29:0/3511724306 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774e59c700 1 --2- 172.21.15.29:0/3511724306 >> v2:172.21.15.144:3300/0 conn(0x7f77500b0610 0x7f77500ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774ed9d700 1 --2- 172.21.15.29:0/3511724306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774dd9b700 1 -- 172.21.15.29:0/3511724306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 msgr2=0x7f77500cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774dd9b700 1 --2- 172.21.15.29:0/3511724306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774dd9b700 1 -- 172.21.15.29:0/3511724306 >> v2:172.21.15.144:3300/0 conn(0x7f77500b0610 msgr2=0x7f77500ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774dd9b700 1 --2- 172.21.15.29:0/3511724306 >> v2:172.21.15.144:3300/0 conn(0x7f77500b0610 0x7f77500ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774dd9b700 1 -- 172.21.15.29:0/3511724306 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77500687c0 con 0x7f77500b09e0 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774dd9b700 1 --2- 172.21.15.29:0/3511724306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500c4200 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f77500659c0 tx=0x7f7744009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87133690e07c0d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.549+0000 7f774d59a700 1 -- 172.21.15.29:0/3511724306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774400b640 con 0x7f77500b09e0 2023-10-07T09:46:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f774d59a700 1 -- 172.21.15.29:0/3511724306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774400b7a0 con 0x7f77500b09e0 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f774d59a700 1 -- 172.21.15.29:0/3511724306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774400bb60 con 0x7f77500b09e0 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 -- 172.21.15.29:0/3511724306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 msgr2=0x7f77500c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 --2- 172.21.15.29:0/3511724306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500c4200 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f77500659c0 tx=0x7f7744009250 comp rx=0 tx=0).stop 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 -- 172.21.15.29:0/3511724306 shutdown_connections 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 --2- 172.21.15.29:0/3511724306 >> v2:172.21.15.144:3300/0 conn(0x7f77500b0610 0x7f77500ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 --2- 172.21.15.29:0/3511724306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 --2- 172.21.15.29:0/3511724306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500c4200 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 -- 172.21.15.29:0/3511724306 >> 172.21.15.29:0/3511724306 conn(0x7f7750063a10 msgr2=0x7f77500ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 -- 172.21.15.29:0/3511724306 shutdown_connections 2023-10-07T09:46:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.550+0000 7f7754fff700 1 -- 172.21.15.29:0/3511724306 wait complete. 2023-10-07T09:46:56.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.552+0000 7f7754fff700 1 Processor -- start 2023-10-07T09:46:56.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.552+0000 7f7754fff700 1 -- start start 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f7754fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77500b0610 0x7f77501556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f7754fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f7754fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f7754fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77500c5f40 con 0x7f77500b09e0 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f7754fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77500c5610 con 0x7f77500b0610 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f7754fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500c58f0 con 0x7f77500c8a20 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37026/0 (socket says 172.21.15.29:37026) 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 -- 172.21.15.29:0/2494576120 learned_addr learned my addr 172.21.15.29:0/2494576120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774e59c700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77500b0610 0x7f77501556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774ed9d700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 -- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 msgr2=0x7f77500c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 -- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77500b0610 msgr2=0x7f77501556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77500b0610 0x7f77501556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774dd9b700 1 -- 172.21.15.29:0/2494576120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77440094f0 con 0x7f77500b09e0 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774e59c700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77500b0610 0x7f77501556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.553+0000 7f774ed9d700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.554+0000 7f774dd9b700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500b1870 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f775006ea20 tx=0x7f7744017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.554+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7744000d10 con 0x7f77500b09e0 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.554+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774400bb20 con 0x7f77500b09e0 2023-10-07T09:46:56.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.554+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7744011630 con 0x7f77500b09e0 2023-10-07T09:46:56.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.554+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77500c5bd0 con 0x7f77500b09e0 2023-10-07T09:46:56.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.554+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77500b1510 con 0x7f77500b09e0 2023-10-07T09:46:56.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.555+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77500c5750 con 0x7f77500b09e0 2023-10-07T09:46:56.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.556+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 84973+0+0 (secure 0 0 0) 0x7f77440119a0 con 0x7f77500b09e0 2023-10-07T09:46:56.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.557+0000 7f773e7fc700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77300671c0 0x7f7730069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:56.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.557+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7744085f70 con 0x7f77500b09e0 2023-10-07T09:46:56.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.558+0000 7f774e59c700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77300671c0 0x7f7730069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:56.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.558+0000 7f774e59c700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77300671c0 0x7f7730069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7740009ce0 tx=0x7f7740008040 comp rx=0 tx=0).ready entity=mgr.7060 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:56.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.562+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77500c5750 con 0x7f77500b09e0 2023-10-07T09:46:56.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.792+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f77500c5750 con 0x7f77300671c0 2023-10-07T09:46:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.859+0000 7f773e7fc700 1 -- 172.21.15.29:0/2494576120 <== mgr.7060 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f77500c5750 con 0x7f77300671c0 2023-10-07T09:46:56.861 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /bad-path 2023-10-07T09:46:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77300671c0 msgr2=0x7f7730069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77300671c0 0x7f7730069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7740009ce0 tx=0x7f7740008040 comp rx=0 tx=0).stop 2023-10-07T09:46:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 msgr2=0x7f77500b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500b1870 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f775006ea20 tx=0x7f7744017000 comp rx=0 tx=0).stop 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 shutdown_connections 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77300671c0 0x7f7730069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77500c8a20 0x7f77500c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77500b0610 0x7f77501556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 --2- 172.21.15.29:0/2494576120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77500b09e0 0x7f77500b1870 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 >> 172.21.15.29:0/2494576120 conn(0x7f7750063a10 msgr2=0x7f775006c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 shutdown_connections 2023-10-07T09:46:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:56.862+0000 7f7754fff700 1 -- 172.21.15.29:0/2494576120 wait complete. 2023-10-07T09:46:56.881 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-07T09:46:56.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T09:46:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.079+0000 7f542a5ff700 1 Processor -- start 2023-10-07T09:46:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.079+0000 7f542a5ff700 1 -- start start 2023-10-07T09:46:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.080+0000 7f542a5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5424156180 0x7f54240afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.080+0000 7f542a5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54240ae570 0x7f54240accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.080+0000 7f542a5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54240b0520 0x7f54240b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.080+0000 7f542a5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240b0ea0 con 0x7f54240b0520 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.080+0000 7f542a5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240ad230 con 0x7f5424156180 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.080+0000 7f542a5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54240ad4d0 con 0x7f54240ae570 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54240b0520 0x7f54240b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54240b0520 0x7f54240b0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37030/0 (socket says 172.21.15.29:37030) 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 -- 172.21.15.29:0/1472816265 learned_addr learned my addr 172.21.15.29:0/1472816265 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f54237fe700 1 --2- 172.21.15.29:0/1472816265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54240ae570 0x7f54240accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5423fff700 1 --2- 172.21.15.29:0/1472816265 >> v2:172.21.15.144:3300/0 conn(0x7f5424156180 0x7f54240afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 -- 172.21.15.29:0/1472816265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54240ae570 msgr2=0x7f54240accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 --2- 172.21.15.29:0/1472816265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54240ae570 0x7f54240accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 -- 172.21.15.29:0/1472816265 >> v2:172.21.15.144:3300/0 conn(0x7f5424156180 msgr2=0x7f54240afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 --2- 172.21.15.29:0/1472816265 >> v2:172.21.15.144:3300/0 conn(0x7f5424156180 0x7f54240afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 -- 172.21.15.29:0/1472816265 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240ad7b0 con 0x7f54240b0520 2023-10-07T09:46:57.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5423fff700 1 --2- 172.21.15.29:0/1472816265 >> v2:172.21.15.144:3300/0 conn(0x7f5424156180 0x7f54240afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:46:57.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.081+0000 7f5428b9c700 1 --2- 172.21.15.29:0/1472816265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f54240b0960 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f54240b2250 tx=0x7f541c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363ae96c527ab9ca server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:57.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f5422ffd700 1 -- 172.21.15.29:0/1472816265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c00e070 con 0x7f54240b0520 2023-10-07T09:46:57.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f5422ffd700 1 -- 172.21.15.29:0/1472816265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00ba20 con 0x7f54240b0520 2023-10-07T09:46:57.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f5422ffd700 1 -- 172.21.15.29:0/1472816265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c0092e0 con 0x7f54240b0520 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 -- 172.21.15.29:0/1472816265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 msgr2=0x7f54240b0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 --2- 172.21.15.29:0/1472816265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f54240b0960 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f54240b2250 tx=0x7f541c009840 comp rx=0 tx=0).stop 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 -- 172.21.15.29:0/1472816265 shutdown_connections 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 --2- 172.21.15.29:0/1472816265 >> v2:172.21.15.144:3300/0 conn(0x7f5424156180 0x7f54240afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 --2- 172.21.15.29:0/1472816265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f54240b0960 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 --2- 172.21.15.29:0/1472816265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54240ae570 0x7f54240accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 -- 172.21.15.29:0/1472816265 >> 172.21.15.29:0/1472816265 conn(0x7f5424067610 msgr2=0x7f5424067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 -- 172.21.15.29:0/1472816265 shutdown_connections 2023-10-07T09:46:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.082+0000 7f542a5ff700 1 -- 172.21.15.29:0/1472816265 wait complete. 2023-10-07T09:46:57.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.084+0000 7f542a5ff700 1 Processor -- start 2023-10-07T09:46:57.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.084+0000 7f542a5ff700 1 -- start start 2023-10-07T09:46:57.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f542a5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54240ae570 0x7f5424152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f542a5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f542414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f542a5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5424156180 0x7f5424151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f542a5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424151d80 con 0x7f54240b0520 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f542a5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424152060 con 0x7f54240ae570 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f542a5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5424152340 con 0x7f5424156180 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.085+0000 7f54237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f542414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f542414ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37036/0 (socket says 172.21.15.29:37036) 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 -- 172.21.15.29:0/392118654 learned_addr learned my addr 172.21.15.29:0/392118654 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f5428b9c700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5424156180 0x7f5424151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 -- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5424156180 msgr2=0x7f5424151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5424156180 0x7f5424151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 -- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54240ae570 msgr2=0x7f5424152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:57.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54240ae570 0x7f5424152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 -- 172.21.15.29:0/392118654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c009ae0 con 0x7f54240b0520 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f54237fe700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f542414ef60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f541800ea70 tx=0x7f54180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541800c920 con 0x7f54240b0520 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541800caa0 con 0x7f54240b0520 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5424152620 con 0x7f54240b0520 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.086+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541800ce60 con 0x7f54240b0520 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.087+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542414f4a0 con 0x7f54240b0520 2023-10-07T09:46:57.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.087+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5424152b40 con 0x7f54240b0520 2023-10-07T09:46:57.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.088+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 84973+0+0 (secure 0 0 0) 0x7f5418013680 con 0x7f54240b0520 2023-10-07T09:46:57.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.089+0000 7f540bfff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f54100671c0 0x7f5410069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:57.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.089+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5418084fe0 con 0x7f54240b0520 2023-10-07T09:46:57.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.090+0000 7f5423fff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f54100671c0 0x7f5410069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:57.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.091+0000 7f5423fff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f54100671c0 0x7f5410069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5414006fd0 tx=0x7f5414008040 comp rx=0 tx=0).ready entity=mgr.7060 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:57.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.094+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5424152b40 con 0x7f54240b0520 2023-10-07T09:46:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:57.359+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f5424152b40 con 0x7f54240b0520 2023-10-07T09:46:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.301+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f5418052060 con 0x7f54240b0520 2023-10-07T09:46:58.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.303+0000 7f540bfff700 1 -- 172.21.15.29:0/392118654 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v46) v1 ==== 93+0+0 (secure 0 0 0) 0x7f5418055720 con 0x7f54240b0520 2023-10-07T09:46:58.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f54100671c0 msgr2=0x7f5410069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:58.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f54100671c0 0x7f5410069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5414006fd0 tx=0x7f5414008040 comp rx=0 tx=0).stop 2023-10-07T09:46:58.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 msgr2=0x7f542414ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:58.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f542414ef60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f541800ea70 tx=0x7f54180091a0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 shutdown_connections 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f54100671c0 0x7f5410069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5424156180 0x7f5424151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54240ae570 0x7f5424152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 --2- 172.21.15.29:0/392118654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54240b0520 0x7f542414ef60 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 >> 172.21.15.29:0/392118654 conn(0x7f5424067610 msgr2=0x7f5424156550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 shutdown_connections 2023-10-07T09:46:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.306+0000 7f542a5ff700 1 -- 172.21.15.29:0/392118654 wait complete. 2023-10-07T09:46:58.327 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:46:58.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:46:58.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:46:58.520 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:46:58.519+0000 7fb44f0681c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:46:58.524 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:58.523+0000 7fb67e09d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:46:58.541 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:46:58.541 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:46:58.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:46:58.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T09:46:58.632 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:46:58.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:46:58.633 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:46:58.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:46:58.703 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:58.702+0000 7fb67e09d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:46:58.711 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:46:58.710+0000 7fb44f0681c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:46:58.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.893+0000 7fe57bbff700 1 Processor -- start 2023-10-07T09:46:58.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.893+0000 7fe57bbff700 1 -- start start 2023-10-07T09:46:58.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57bbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe574157290 0x7fe5740cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:58.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57bbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5740b1a10 0x7fe574157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:58.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57bbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe574158ad0 con 0x7fe5740ae8b0 2023-10-07T09:46:58.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57bbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57415b140 con 0x7fe574157290 2023-10-07T09:46:58.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57bbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57415b420 con 0x7fe5740b1a10 2023-10-07T09:46:58.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57999b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe574157290 0x7fe5740cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.895+0000 7fe57a19c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5740b1a10 0x7fe574157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe574157290 0x7fe5740cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37064/0 (socket says 172.21.15.29:37064) 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 -- 172.21.15.29:0/3659229434 learned_addr learned my addr 172.21.15.29:0/3659229434 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 -- 172.21.15.29:0/3659229434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5740b1a10 msgr2=0x7fe574157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 --2- 172.21.15.29:0/3659229434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5740b1a10 0x7fe574157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 -- 172.21.15.29:0/3659229434 >> v1:172.21.15.29:6789/0 conn(0x7fe5740ae8b0 legacy=0x7fe5740b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 -- 172.21.15.29:0/3659229434 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57415b700 con 0x7fe574157290 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.896+0000 7fe57999b700 1 --2- 172.21.15.29:0/3659229434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574157290 0x7fe5740cc870 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe5700099b0 tx=0x7fe570009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b5f95d65364c0b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe578999700 1 -- 172.21.15.29:0/3659229434 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570009ea0 con 0x7fe574157290 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe578999700 1 -- 172.21.15.29:0/3659229434 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57000b860 con 0x7fe574157290 2023-10-07T09:46:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe578999700 1 -- 172.21.15.29:0/3659229434 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57000bc00 con 0x7fe574157290 2023-10-07T09:46:58.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 -- 172.21.15.29:0/3659229434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574157290 msgr2=0x7fe5740cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:58.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 --2- 172.21.15.29:0/3659229434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574157290 0x7fe5740cc870 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe5700099b0 tx=0x7fe570009250 comp rx=0 tx=0).stop 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 -- 172.21.15.29:0/3659229434 shutdown_connections 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 --2- 172.21.15.29:0/3659229434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5740b1a10 0x7fe574157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 --2- 172.21.15.29:0/3659229434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe574157290 0x7fe5740cc870 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 -- 172.21.15.29:0/3659229434 >> 172.21.15.29:0/3659229434 conn(0x7fe5740680a0 msgr2=0x7fe5740cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.897+0000 7fe57bbff700 1 -- 172.21.15.29:0/3659229434 shutdown_connections 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.898+0000 7fe57bbff700 1 -- 172.21.15.29:0/3659229434 wait complete. 2023-10-07T09:46:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.899+0000 7fe57bbff700 1 Processor -- start 2023-10-07T09:46:58.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.899+0000 7fe57bbff700 1 -- start start 2023-10-07T09:46:58.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57bbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 0x7fe57406ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:58.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57bbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574157290 0x7fe57406efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:58.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57bbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574144fd0 0x7fe57415e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:58.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57bbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57415efe0 con 0x7fe5740b1a10 2023-10-07T09:46:58.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57bbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574169aa0 con 0x7fe574144fd0 2023-10-07T09:46:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57bbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe574169ce0 con 0x7fe574157290 2023-10-07T09:46:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57999b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 0x7fe57406ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57999b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 0x7fe57406ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37074/0 (socket says 172.21.15.29:37074) 2023-10-07T09:46:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57999b700 1 -- 172.21.15.29:0/801943243 learned_addr learned my addr 172.21.15.29:0/801943243 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57a19c700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574144fd0 0x7fe57415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57919a700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574157290 0x7fe57406efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57999b700 1 -- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574157290 msgr2=0x7fe57406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.901+0000 7fe57999b700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574157290 0x7fe57406efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe57999b700 1 -- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574144fd0 msgr2=0x7fe57415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe57999b700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574144fd0 0x7fe57415e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe57999b700 1 -- 172.21.15.29:0/801943243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700094f0 con 0x7fe5740b1a10 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe57999b700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 0x7fe57406ea60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe574068dd0 tx=0x7fe570017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570000d10 con 0x7fe5740b1a10 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57000bbe0 con 0x7fe5740b1a10 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574169fc0 con 0x7fe5740b1a10 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570011650 con 0x7fe5740b1a10 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.902+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57416a580 con 0x7fe5740b1a10 2023-10-07T09:46:58.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.904+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57416a770 con 0x7fe5740b1a10 2023-10-07T09:46:58.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.905+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fe570011a80 con 0x7fe5740b1a10 2023-10-07T09:46:58.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.906+0000 7fe569ffb700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 0x7fe558069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:58.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.906+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe570085ff0 con 0x7fe5740b1a10 2023-10-07T09:46:58.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.906+0000 7fe57919a700 1 -- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 msgr2=0x7fe558069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:46:58.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.906+0000 7fe57919a700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 0x7fe558069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:46:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:58.911+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe570052d00 con 0x7fe5740b1a10 2023-10-07T09:46:59.020 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:59.019+0000 7fb67e09d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:46:59.052 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:46:59.051+0000 7fb44f0681c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:46:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.107+0000 7fe57919a700 1 -- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 msgr2=0x7fe558069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:46:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.107+0000 7fe57919a700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 0x7fe558069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:46:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.161+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe57404c3c0 con 0x7fe5740b1a10 2023-10-07T09:46:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.161+0000 7fe569ffb700 1 -- 172.21.15.29:0/801943243 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fe570056320 con 0x7fe5740b1a10 2023-10-07T09:46:59.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:46:59.163 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:38.377260+0000","modified":"2023-10-07T09:45:41.892365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3302133648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3302133648},{"type":"v1","addr":"172.21.15.29:6835","nonce":3302133648}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-10-07T09:46:59.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 msgr2=0x7fe558069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:46:59.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 0x7fe558069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 msgr2=0x7fe57406ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 0x7fe57406ea60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe574068dd0 tx=0x7fe570017000 comp rx=0 tx=0).stop 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 shutdown_connections 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe558067170 0x7fe558069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe574144fd0 0x7fe57415e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5740b1a10 0x7fe57406ea60 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 --2- 172.21.15.29:0/801943243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe574157290 0x7fe57406efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 >> 172.21.15.29:0/801943243 conn(0x7fe5740680a0 msgr2=0x7fe574157660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 shutdown_connections 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.164+0000 7fe57bbff700 1 -- 172.21.15.29:0/801943243 wait complete. 2023-10-07T09:46:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-07T09:46:59.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:46:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.422+0000 7f35217ff700 1 Processor -- start 2023-10-07T09:46:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.422+0000 7f35217ff700 1 -- start start 2023-10-07T09:46:59.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f35217ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f351c0ad370 0x7f351c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f35217ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c14d790 con 0x7f351c0b0410 2023-10-07T09:46:59.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f35217ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c0ae5d0 con 0x7f351c159ec0 2023-10-07T09:46:59.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f35217ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351c0ad740 con 0x7f351c0ad370 2023-10-07T09:46:59.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f351b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f351c0b0410 0x7f351c156fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54994/0 (socket says 172.21.15.29:54994) 2023-10-07T09:46:59.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f351b7fe700 1 -- 172.21.15.29:0/2648091786 learned_addr learned my addr 172.21.15.29:0/2648091786 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:46:59.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.424+0000 7f351affd700 1 --2- 172.21.15.29:0/2648091786 >> v2:172.21.15.144:3300/0 conn(0x7f351c0ad370 0x7f351c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f3519ffb700 1 -- 172.21.15.29:0/2648091786 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100006978 0 0) 0x7f351c0ae5d0 con 0x7f351c159ec0 2023-10-07T09:46:59.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f3519ffb700 1 -- 172.21.15.29:0/2648091786 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3504003570 con 0x7f351c159ec0 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f351affd700 1 -- 172.21.15.29:0/2648091786 >> v1:172.21.15.144:6789/0 conn(0x7f351c159ec0 legacy=0x7f351c1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f351affd700 1 -- 172.21.15.29:0/2648091786 >> v1:172.21.15.29:6789/0 conn(0x7f351c0b0410 legacy=0x7f351c156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f3519ffb700 1 -- 172.21.15.29:0/2648091786 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264030093 0 0) 0x7f351c14d790 con 0x7f351c0b0410 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f351affd700 1 -- 172.21.15.29:0/2648091786 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0ccdb0 con 0x7f351c0ad370 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f351affd700 1 --2- 172.21.15.29:0/2648091786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0ad370 0x7f351c0cc870 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f350c009910 tx=0x7f350c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67d138851268182e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f3519ffb700 1 -- 172.21.15.29:0/2648091786 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00b640 con 0x7f351c0ad370 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f3519ffb700 1 -- 172.21.15.29:0/2648091786 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c00b7a0 con 0x7f351c0ad370 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.425+0000 7f3519ffb700 1 -- 172.21.15.29:0/2648091786 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00bb60 con 0x7f351c0ad370 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 -- 172.21.15.29:0/2648091786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0ad370 msgr2=0x7f351c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 --2- 172.21.15.29:0/2648091786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0ad370 0x7f351c0cc870 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f350c009910 tx=0x7f350c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 -- 172.21.15.29:0/2648091786 shutdown_connections 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 --2- 172.21.15.29:0/2648091786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0ad370 0x7f351c0cc870 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 -- 172.21.15.29:0/2648091786 >> 172.21.15.29:0/2648091786 conn(0x7f351c0680a0 msgr2=0x7f351c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 -- 172.21.15.29:0/2648091786 shutdown_connections 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.426+0000 7f35217ff700 1 -- 172.21.15.29:0/2648091786 wait complete. 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.428+0000 7f35217ff700 1 Processor -- start 2023-10-07T09:46:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.429+0000 7f35217ff700 1 -- start start 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f35217ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 0x7f351c14ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f35217ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b0410 0x7f351c151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f35217ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351c159ec0 0x7f351c150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f35217ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c14f480 con 0x7f351c159ec0 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f35217ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1502a0 con 0x7f351c0b0410 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f35217ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c151770 con 0x7f351c0ad370 2023-10-07T09:46:59.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351c159ec0 0x7f351c150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351a7fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b0410 0x7f351c151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351affd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 0x7f351c14ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351affd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 0x7f351c14ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43746/0 (socket says 172.21.15.29:43746) 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351affd700 1 -- 172.21.15.29:0/376975158 learned_addr learned my addr 172.21.15.29:0/376975158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351a7fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b0410 0x7f351c151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:47138/0 (socket says 172.21.15.29:47138) 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.430+0000 7f351b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351c159ec0 0x7f351c150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37078/0 (socket says 172.21.15.29:37078) 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f351affd700 1 -- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b0410 msgr2=0x7f351c151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f351affd700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b0410 0x7f351c151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f351affd700 1 -- 172.21.15.29:0/376975158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351c159ec0 msgr2=0x7f351c150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f351affd700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351c159ec0 0x7f351c150730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f351affd700 1 -- 172.21.15.29:0/376975158 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c009420 con 0x7f351c0ad370 2023-10-07T09:46:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f351affd700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 0x7f351c14ef40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f350c0093f0 tx=0x7f350c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c01f070 con 0x7f351c0ad370 2023-10-07T09:46:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c000bc0 con 0x7f351c0ad370 2023-10-07T09:46:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00bb40 con 0x7f351c0ad370 2023-10-07T09:46:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.431+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c150c70 con 0x7f351c0ad370 2023-10-07T09:46:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.432+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c14f970 con 0x7f351c0ad370 2023-10-07T09:46:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.433+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c150580 con 0x7f351c0ad370 2023-10-07T09:46:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.433+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f350c023070 con 0x7f351c0ad370 2023-10-07T09:46:59.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.434+0000 7f3502ffd700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 0x7f3508069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.435+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f350c085ff0 con 0x7f351c0ad370 2023-10-07T09:46:59.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.435+0000 7f351a7fc700 1 -- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 msgr2=0x7f3508069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:46:59.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.435+0000 7f351a7fc700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 0x7f3508069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:46:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.439+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f350c052d00 con 0x7f351c0ad370 2023-10-07T09:46:59.535 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:59.534+0000 7fb67e09d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:46:59.629 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:46:59.628+0000 7fb44f0681c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:46:59.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.636+0000 7f351a7fc700 1 -- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 msgr2=0x7f3508069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:46:59.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.636+0000 7f351a7fc700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 0x7f3508069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:46:59.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.667+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f351c04c3c0 con 0x7f351c0ad370 2023-10-07T09:46:59.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.668+0000 7f3502ffd700 1 -- 172.21.15.29:0/376975158 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16504 (secure 0 0 0) 0x7f350c0562a0 con 0x7f351c0ad370 2023-10-07T09:46:59.669 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:46:59.669 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":106,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:46:56.275599+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:45:36.139381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:37.345605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]},{"pgid":"13.32","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1688934630":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/1525826437":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/2483656043":"2023-10-08T09:45:54.269145+0000","172.21.15.144:6836/114714":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/3266577652":"2023-10-08T09:45:54.269145+0000","172.21.15.144:6834/114714":"2023-10-08T09:45:54.269145+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 msgr2=0x7f3508069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 0x7f3508069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 msgr2=0x7f351c14ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 0x7f351c14ef40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f350c0093f0 tx=0x7f350c017000 comp rx=0 tx=0).stop 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 shutdown_connections 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3508067170 0x7f3508069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351c159ec0 0x7f351c150730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0ad370 0x7f351c14ef40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 --2- 172.21.15.29:0/376975158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b0410 0x7f351c151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 >> 172.21.15.29:0/376975158 conn(0x7f351c0680a0 msgr2=0x7f351c06d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.670+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 shutdown_connections 2023-10-07T09:46:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.671+0000 7f35217ff700 1 -- 172.21.15.29:0/376975158 wait complete. 2023-10-07T09:46:59.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:45:36.139381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:45:37.345605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:46:59.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:46:59.703 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:46:59.702+0000 7fb67e09d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:46:59.814 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:46:59.813+0000 7fb44f0681c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:46:59.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.896+0000 7f5b891ff700 1 Processor -- start 2023-10-07T09:46:59.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.896+0000 7f5b891ff700 1 -- start start 2023-10-07T09:46:59.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.897+0000 7f5b891ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.897+0000 7f5b891ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b8415ad30 0x7f5b840b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.897+0000 7f5b891ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b840acb40 con 0x7f5b8415a960 2023-10-07T09:46:59.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.897+0000 7f5b891ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b84158c60 con 0x7f5b8415ad30 2023-10-07T09:46:59.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.897+0000 7f5b891ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840abf30 con 0x7f5b840ae8a0 2023-10-07T09:46:59.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b8359e700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b8415ad30 0x7f5b840b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b8259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b82d9d700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f5b8415a960 0x7f5b840ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:57726/0 (socket says 172.21.15.29:57726) 2023-10-07T09:46:59.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b82d9d700 1 -- 172.21.15.29:0/1920357214 learned_addr learned my addr 172.21.15.29:0/1920357214 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:46:59.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b81d9b700 1 -- 172.21.15.29:0/1920357214 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430997042 0 0) 0x7f5b840acb40 con 0x7f5b8415a960 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b81d9b700 1 -- 172.21.15.29:0/1920357214 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b6c003570 con 0x7f5b8415a960 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b8259c700 1 -- 172.21.15.29:0/1920357214 >> v2:172.21.15.144:3300/0 conn(0x7f5b8415ad30 msgr2=0x7f5b840b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b8259c700 1 --2- 172.21.15.29:0/1920357214 >> v2:172.21.15.144:3300/0 conn(0x7f5b8415ad30 0x7f5b840b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b8259c700 1 -- 172.21.15.29:0/1920357214 >> v1:172.21.15.144:6789/0 conn(0x7f5b8415a960 legacy=0x7f5b840ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.898+0000 7f5b8259c700 1 -- 172.21.15.29:0/1920357214 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840ac210 con 0x7f5b840ae8a0 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.899+0000 7f5b8259c700 1 --2- 172.21.15.29:0/1920357214 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84158e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b780098e0 tx=0x7f5b780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86db4ea634a603db server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.899+0000 7f5b81d9b700 1 -- 172.21.15.29:0/1920357214 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78011630 con 0x7f5b840ae8a0 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.899+0000 7f5b81d9b700 1 -- 172.21.15.29:0/1920357214 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78011790 con 0x7f5b840ae8a0 2023-10-07T09:46:59.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.899+0000 7f5b81d9b700 1 -- 172.21.15.29:0/1920357214 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78011b30 con 0x7f5b840ae8a0 2023-10-07T09:46:59.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.899+0000 7f5b891ff700 1 -- 172.21.15.29:0/1920357214 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 msgr2=0x7f5b84158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.899+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1920357214 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84158e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b780098e0 tx=0x7f5b780091a0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.900+0000 7f5b891ff700 1 -- 172.21.15.29:0/1920357214 shutdown_connections 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.900+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1920357214 >> v2:172.21.15.144:3300/0 conn(0x7f5b8415ad30 0x7f5b840b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.900+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1920357214 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84158e20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.900+0000 7f5b891ff700 1 -- 172.21.15.29:0/1920357214 >> 172.21.15.29:0/1920357214 conn(0x7f5b84067610 msgr2=0x7f5b84067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.900+0000 7f5b891ff700 1 -- 172.21.15.29:0/1920357214 shutdown_connections 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.900+0000 7f5b891ff700 1 -- 172.21.15.29:0/1920357214 wait complete. 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.901+0000 7f5b891ff700 1 Processor -- start 2023-10-07T09:46:59.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.901+0000 7f5b891ff700 1 -- start start 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b891ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b891ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b8415a960 0x7f5b84156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b891ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 0x7f5b8415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b891ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8416a570 con 0x7f5b8415ad30 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b891ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840af870 con 0x7f5b8415a960 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b891ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8416a290 con 0x7f5b840ae8a0 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b8359e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 0x7f5b8415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.902+0000 7f5b82d9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 0x7f5b8415d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37082/0 (socket says 172.21.15.29:37082) 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 -- 172.21.15.29:0/1890632668 learned_addr learned my addr 172.21.15.29:0/1890632668 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8259c700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b8415a960 0x7f5b84156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 -- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 msgr2=0x7f5b84156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 -- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b8415a960 msgr2=0x7f5b84156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b8415a960 0x7f5b84156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 -- 172.21.15.29:0/1890632668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78009420 con 0x7f5b8415ad30 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b82d9d700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b8359e700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 0x7f5b8415d2e0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009d70 tx=0x7f5b7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:46:59.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.903+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c00caa0 con 0x7f5b8415ad30 2023-10-07T09:46:59.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.904+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c00f040 con 0x7f5b8415ad30 2023-10-07T09:46:59.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.904+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c00ce50 con 0x7f5b8415ad30 2023-10-07T09:46:59.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.904+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b840af220 con 0x7f5b8415ad30 2023-10-07T09:46:59.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.904+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8415d820 con 0x7f5b8415ad30 2023-10-07T09:46:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.905+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b840af080 con 0x7f5b8415ad30 2023-10-07T09:46:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.906+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f5b7c014690 con 0x7f5b8415ad30 2023-10-07T09:46:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.907+0000 7f5b6affd700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 0x7f5b70069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:46:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.907+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5b7c085860 con 0x7f5b8415ad30 2023-10-07T09:46:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.907+0000 7f5b82d9d700 1 -- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 msgr2=0x7f5b70069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:46:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.907+0000 7f5b82d9d700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 0x7f5b70069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:46:59.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:46:59.911+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b840af080 con 0x7f5b8415ad30 2023-10-07T09:47:00.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.108+0000 7f5b82d9d700 1 -- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 msgr2=0x7f5b70069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:00.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.108+0000 7f5b82d9d700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 0x7f5b70069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:00.114 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:00.113+0000 7fb67e09d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:47:00.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.115+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b840af080 con 0x7f5b8415ad30 2023-10-07T09:47:00.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.116+0000 7f5b6affd700 1 -- 172.21.15.29:0/1890632668 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b840af080 con 0x7f5b8415ad30 2023-10-07T09:47:00.117 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:47:00.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 msgr2=0x7f5b70069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:00.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 0x7f5b70069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 msgr2=0x7f5b8415d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 0x7f5b8415d2e0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009d70 tx=0x7f5b7c00b040 comp rx=0 tx=0).stop 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 shutdown_connections 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5b70067170 0x7f5b70069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b8415ad30 0x7f5b8415d2e0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b840ae8a0 0x7f5b84156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 --2- 172.21.15.29:0/1890632668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b8415a960 0x7f5b84156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 >> 172.21.15.29:0/1890632668 conn(0x7f5b84067610 msgr2=0x7f5b8415bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 shutdown_connections 2023-10-07T09:47:00.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.119+0000 7f5b891ff700 1 -- 172.21.15.29:0/1890632668 wait complete. 2023-10-07T09:47:00.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:47:00.260 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:00.259+0000 7fb44f0681c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:47:00.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.351+0000 7f80c6bff700 1 Processor -- start 2023-10-07T09:47:00.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.351+0000 7f80c6bff700 1 -- start start 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c6bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80c015a960 0x7f80c0158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c015ad30 0x7f80c00ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c6bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c00acb40 con 0x7f80c00ae6d0 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c6bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c00aeb50 con 0x7f80c015a960 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0158c60 con 0x7f80c015ad30 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c519c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f80c00ae6d0 0x7f80c00b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54998/0 (socket says 172.21.15.29:54998) 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c519c700 1 -- 172.21.15.29:0/3176128870 learned_addr learned my addr 172.21.15.29:0/3176128870 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80bffff700 1 --2- 172.21.15.29:0/3176128870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c015ad30 0x7f80c00ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.353+0000 7f80c499b700 1 --2- 172.21.15.29:0/3176128870 >> v2:172.21.15.144:3300/0 conn(0x7f80c015a960 0x7f80c0158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bf7fe700 1 -- 172.21.15.29:0/3176128870 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569026999 0 0) 0x7f80c00acb40 con 0x7f80c00ae6d0 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bf7fe700 1 -- 172.21.15.29:0/3176128870 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b0003570 con 0x7f80c00ae6d0 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bffff700 1 -- 172.21.15.29:0/3176128870 >> v2:172.21.15.144:3300/0 conn(0x7f80c015a960 msgr2=0x7f80c0158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bffff700 1 --2- 172.21.15.29:0/3176128870 >> v2:172.21.15.144:3300/0 conn(0x7f80c015a960 0x7f80c0158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bffff700 1 -- 172.21.15.29:0/3176128870 >> v1:172.21.15.29:6789/0 conn(0x7f80c00ae6d0 legacy=0x7f80c00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bffff700 1 -- 172.21.15.29:0/3176128870 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c00abf30 con 0x7f80c015ad30 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80c499b700 1 --2- 172.21.15.29:0/3176128870 >> v2:172.21.15.144:3300/0 conn(0x7f80c015a960 0x7f80c0158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:00.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bffff700 1 --2- 172.21.15.29:0/3176128870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c015ad30 0x7f80c00ae190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0098e0 tx=0x7f80ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08df8966f42456b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:00.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bf7fe700 1 -- 172.21.15.29:0/3176128870 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac011630 con 0x7f80c015ad30 2023-10-07T09:47:00.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bf7fe700 1 -- 172.21.15.29:0/3176128870 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac011790 con 0x7f80c015ad30 2023-10-07T09:47:00.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.354+0000 7f80bf7fe700 1 -- 172.21.15.29:0/3176128870 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac011b30 con 0x7f80c015ad30 2023-10-07T09:47:00.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 -- 172.21.15.29:0/3176128870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c015ad30 msgr2=0x7f80c00ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3176128870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c015ad30 0x7f80c00ae190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0098e0 tx=0x7f80ac0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 -- 172.21.15.29:0/3176128870 shutdown_connections 2023-10-07T09:47:00.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3176128870 >> v2:172.21.15.144:3300/0 conn(0x7f80c015a960 0x7f80c0158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3176128870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c015ad30 0x7f80c00ae190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 -- 172.21.15.29:0/3176128870 >> 172.21.15.29:0/3176128870 conn(0x7f80c0067610 msgr2=0x7f80c0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:00.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 -- 172.21.15.29:0/3176128870 shutdown_connections 2023-10-07T09:47:00.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.355+0000 7f80c6bff700 1 -- 172.21.15.29:0/3176128870 wait complete. 2023-10-07T09:47:00.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.357+0000 7f80c6bff700 1 Processor -- start 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.358+0000 7f80c6bff700 1 -- start start 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c00ae6d0 0x7f80c014f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c6bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80c015a960 0x7f80c00cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c6bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c015ad30 0x7f80c0169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c6bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c016d090 con 0x7f80c015ad30 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c6bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00cbb50 con 0x7f80c015a960 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c015c6f0 con 0x7f80c00ae6d0 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c519c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c015ad30 0x7f80c0169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c499b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c00ae6d0 0x7f80c014f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80bffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80c015a960 0x7f80c00cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c499b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c00ae6d0 0x7f80c014f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43764/0 (socket says 172.21.15.29:43764) 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.359+0000 7f80c499b700 1 -- 172.21.15.29:0/3946298123 learned_addr learned my addr 172.21.15.29:0/3946298123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bffff700 1 -- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c00ae6d0 msgr2=0x7f80c014f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bffff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c00ae6d0 0x7f80c014f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bffff700 1 -- 172.21.15.29:0/3946298123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c015ad30 msgr2=0x7f80c0169ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bffff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c015ad30 0x7f80c0169ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bffff700 1 -- 172.21.15.29:0/3946298123 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4009420 con 0x7f80c015a960 2023-10-07T09:47:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80c519c700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c015ad30 0x7f80c0169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bffff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80c015a960 0x7f80c00cac60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f80ac016000 tx=0x7f80ac017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac000d10 con 0x7f80c015a960 2023-10-07T09:47:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac011b10 con 0x7f80c015a960 2023-10-07T09:47:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac025660 con 0x7f80c015a960 2023-10-07T09:47:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ac009420 con 0x7f80c015a960 2023-10-07T09:47:00.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.360+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c015ccb0 con 0x7f80c015a960 2023-10-07T09:47:00.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.362+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c004c3c0 con 0x7f80c015a960 2023-10-07T09:47:00.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.362+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f80ac0193b0 con 0x7f80c015a960 2023-10-07T09:47:00.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.363+0000 7f80bcff9700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.363+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f80ac086390 con 0x7f80c015a960 2023-10-07T09:47:00.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.363+0000 7f80c499b700 1 -- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 msgr2=0x7f80a4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:00.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.363+0000 7f80c499b700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:00.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.372+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80ac053020 con 0x7f80c015a960 2023-10-07T09:47:00.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.564+0000 7f80c499b700 1 -- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 msgr2=0x7f80a4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:00.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.564+0000 7f80c499b700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.578+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80c00ac210 con 0x7f80c015a960 2023-10-07T09:47:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.578+0000 7f80bcff9700 1 -- 172.21.15.29:0/3946298123 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80ac056640 con 0x7f80c015a960 2023-10-07T09:47:00.580 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:47:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 msgr2=0x7f80a4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80c015a960 msgr2=0x7f80c00cac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80c015a960 0x7f80c00cac60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f80ac016000 tx=0x7f80ac017000 comp rx=0 tx=0).stop 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 shutdown_connections 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c015ad30 0x7f80c0169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80c00ae6d0 0x7f80c014f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 --2- 172.21.15.29:0/3946298123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80c015a960 0x7f80c00cac60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 >> 172.21.15.29:0/3946298123 conn(0x7f80c0067610 msgr2=0x7f80c01514b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 shutdown_connections 2023-10-07T09:47:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.581+0000 7f80c6bff700 1 -- 172.21.15.29:0/3946298123 wait complete. 2023-10-07T09:47:00.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:47:00.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.806+0000 7f46707ff700 1 Processor -- start 2023-10-07T09:47:00.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.806+0000 7f46707ff700 1 -- start start 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.807+0000 7f46707ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4668159300 0x7f4668069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.807+0000 7f46707ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46680b0060 0x7f46680b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.807+0000 7f46707ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680b0a10 con 0x7f466815d300 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.807+0000 7f46707ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466815de90 con 0x7f46680b0060 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.807+0000 7f46707ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4668069400 con 0x7f4668159300 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46680b0060 0x7f46680b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46680b0060 0x7f46680b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37106/0 (socket says 172.21.15.29:37106) 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 -- 172.21.15.29:0/633018344 learned_addr learned my addr 172.21.15.29:0/633018344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:00.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466e59b700 1 --2- 172.21.15.29:0/633018344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4668159300 0x7f4668069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 -- 172.21.15.29:0/633018344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4668159300 msgr2=0x7f4668069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 --2- 172.21.15.29:0/633018344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4668159300 0x7f4668069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 -- 172.21.15.29:0/633018344 >> v1:172.21.15.144:6789/0 conn(0x7f466815d300 legacy=0x7f466815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 -- 172.21.15.29:0/633018344 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668069dd0 con 0x7f46680b0060 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.808+0000 7f466dd9a700 1 --2- 172.21.15.29:0/633018344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46680b0060 0x7f46680b04d0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f46580099b0 tx=0x7f4658009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594046ac30c01583 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f466d599700 1 -- 172.21.15.29:0/633018344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658009ea0 con 0x7f46680b0060 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f466d599700 1 -- 172.21.15.29:0/633018344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465800b860 con 0x7f46680b0060 2023-10-07T09:47:00.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f466d599700 1 -- 172.21.15.29:0/633018344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465800bc00 con 0x7f46680b0060 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 -- 172.21.15.29:0/633018344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46680b0060 msgr2=0x7f46680b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 --2- 172.21.15.29:0/633018344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46680b0060 0x7f46680b04d0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f46580099b0 tx=0x7f4658009250 comp rx=0 tx=0).stop 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 -- 172.21.15.29:0/633018344 shutdown_connections 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 --2- 172.21.15.29:0/633018344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4668159300 0x7f4668069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 --2- 172.21.15.29:0/633018344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46680b0060 0x7f46680b04d0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 -- 172.21.15.29:0/633018344 >> 172.21.15.29:0/633018344 conn(0x7f4668067610 msgr2=0x7f4668067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.809+0000 7f46707ff700 1 -- 172.21.15.29:0/633018344 shutdown_connections 2023-10-07T09:47:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.810+0000 7f46707ff700 1 -- 172.21.15.29:0/633018344 wait complete. 2023-10-07T09:47:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.812+0000 7f46707ff700 1 Processor -- start 2023-10-07T09:47:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.812+0000 7f46707ff700 1 -- start start 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f46707ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 0x7f4668169580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f46707ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4668159300 0x7f4668148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f46707ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f466815d300 0x7f466815b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f46707ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680ccd60 con 0x7f466815d300 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f46707ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4668169bd0 con 0x7f4668159300 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f46707ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4668169eb0 con 0x7f46680b0060 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 0x7f4668169580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466ed9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f466815d300 0x7f466815b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 0x7f4668169580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43780/0 (socket says 172.21.15.29:43780) 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 -- 172.21.15.29:0/3235550077 learned_addr learned my addr 172.21.15.29:0/3235550077 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466dd9a700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4668159300 0x7f4668148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 -- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4668159300 msgr2=0x7f4668148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4668159300 0x7f4668148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 -- 172.21.15.29:0/3235550077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f466815d300 msgr2=0x7f466815b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f466815d300 0x7f466815b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.814+0000 7f466e59b700 1 -- 172.21.15.29:0/3235550077 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46580094f0 con 0x7f46680b0060 2023-10-07T09:47:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.815+0000 7f466e59b700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 0x7f4668169580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f466400ec80 tx=0x7f4664009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.815+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466400c850 con 0x7f46680b0060 2023-10-07T09:47:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.815+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466400c9b0 con 0x7f46680b0060 2023-10-07T09:47:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.815+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466400cd50 con 0x7f46680b0060 2023-10-07T09:47:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.815+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466816a190 con 0x7f46680b0060 2023-10-07T09:47:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.815+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4668154640 con 0x7f46680b0060 2023-10-07T09:47:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.817+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466804c3c0 con 0x7f46680b0060 2023-10-07T09:47:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.817+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f4664013680 con 0x7f46680b0060 2023-10-07T09:47:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.818+0000 7f465e7fc700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 0x7f464c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.818+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f46640850f0 con 0x7f46680b0060 2023-10-07T09:47:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.818+0000 7f466dd9a700 1 -- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 msgr2=0x7f464c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.818+0000 7f466dd9a700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 0x7f464c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:00.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:00.822+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4664051e20 con 0x7f46680b0060 2023-10-07T09:47:00.895 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:00.894+0000 7fb67e09d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:47:00.981 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:00.980+0000 7fb67e09d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:47:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.019+0000 7f466dd9a700 1 -- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 msgr2=0x7f464c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.019+0000 7f466dd9a700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 0x7f464c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.026+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f466806a930 con 0x7f46680b0060 2023-10-07T09:47:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.027+0000 7f465e7fc700 1 -- 172.21.15.29:0/3235550077 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4664055440 con 0x7f46680b0060 2023-10-07T09:47:01.028 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:47:01.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 msgr2=0x7f464c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:01.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 0x7f464c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 msgr2=0x7f4668169580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 0x7f4668169580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f466400ec80 tx=0x7f4664009120 comp rx=0 tx=0).stop 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 shutdown_connections 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f464c067170 0x7f464c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f466815d300 0x7f466815b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46680b0060 0x7f4668169580 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 --2- 172.21.15.29:0/3235550077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4668159300 0x7f4668148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 >> 172.21.15.29:0/3235550077 conn(0x7f4668067610 msgr2=0x7f4668158330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 shutdown_connections 2023-10-07T09:47:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.031+0000 7f46707ff700 1 -- 172.21.15.29:0/3235550077 wait complete. 2023-10-07T09:47:01.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:47:01.103 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.102+0000 7fb44f0681c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:47:01.135 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.134+0000 7fb67e09d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:47:01.196 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.195+0000 7fb44f0681c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:47:01.222 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.221+0000 7fb67e09d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:47:01.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.259+0000 7f56ff3ff700 1 Processor -- start 2023-10-07T09:47:01.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.259+0000 7f56ff3ff700 1 -- start start 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.260+0000 7f56ff3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56f8157290 0x7f56f80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.260+0000 7f56ff3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f8158ad0 con 0x7f56f80b1a10 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.260+0000 7f56ff3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f815b140 con 0x7f56f80ae8b0 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.260+0000 7f56ff3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f815b420 con 0x7f56f8157290 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56f8157290 0x7f56f80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56f8157290 0x7f56f80cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37118/0 (socket says 172.21.15.29:37118) 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 -- 172.21.15.29:0/1309069587 learned_addr learned my addr 172.21.15.29:0/1309069587 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:01.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd99c700 1 --1- 172.21.15.29:0/1309069587 >> v1:172.21.15.29:6789/0 conn(0x7f56f80b1a10 0x7f56f8157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55008/0 (socket says 172.21.15.29:55008) 2023-10-07T09:47:01.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 -- 172.21.15.29:0/1309069587 >> v1:172.21.15.144:6789/0 conn(0x7f56f80ae8b0 legacy=0x7f56f80b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 -- 172.21.15.29:0/1309069587 >> v1:172.21.15.29:6789/0 conn(0x7f56f80b1a10 legacy=0x7f56f8157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 -- 172.21.15.29:0/1309069587 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f815b700 con 0x7f56f8157290 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.261+0000 7f56fd19b700 1 --2- 172.21.15.29:0/1309069587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f80cc870 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f56f40099b0 tx=0x7f56f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0de00edfaccd91 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56effff700 1 -- 172.21.15.29:0/1309069587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f400b860 con 0x7f56f8157290 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56effff700 1 -- 172.21.15.29:0/1309069587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f400b9c0 con 0x7f56f8157290 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56effff700 1 -- 172.21.15.29:0/1309069587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f400bd60 con 0x7f56f8157290 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1309069587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 msgr2=0x7f56f80cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1309069587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f80cc870 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f56f40099b0 tx=0x7f56f4009250 comp rx=0 tx=0).stop 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1309069587 shutdown_connections 2023-10-07T09:47:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1309069587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f80cc870 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.262+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1309069587 >> 172.21.15.29:0/1309069587 conn(0x7f56f80680a0 msgr2=0x7f56f80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.263+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1309069587 shutdown_connections 2023-10-07T09:47:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.263+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1309069587 wait complete. 2023-10-07T09:47:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.264+0000 7f56ff3ff700 1 Processor -- start 2023-10-07T09:47:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.264+0000 7f56ff3ff700 1 -- start start 2023-10-07T09:47:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56ff3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56f80b1a10 0x7f56f816a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56ff3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f8155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56ff3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56f816d4b0 0x7f56f80cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56ff3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f816ae40 con 0x7f56f8157290 2023-10-07T09:47:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56ff3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f816d920 con 0x7f56f816d4b0 2023-10-07T09:47:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56ff3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f816dc00 con 0x7f56f80b1a10 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56fc99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f8155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56fc99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f8155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37122/0 (socket says 172.21.15.29:37122) 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56fd19b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56f80b1a10 0x7f56f816a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.265+0000 7f56fc99a700 1 -- 172.21.15.29:0/1035902238 learned_addr learned my addr 172.21.15.29:0/1035902238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fd99c700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56f816d4b0 0x7f56f80cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fc99a700 1 -- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56f80b1a10 msgr2=0x7f56f816a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fc99a700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56f80b1a10 0x7f56f816a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fc99a700 1 -- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56f816d4b0 msgr2=0x7f56f80cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fc99a700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56f816d4b0 0x7f56f80cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fc99a700 1 -- 172.21.15.29:0/1035902238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f40094f0 con 0x7f56f8157290 2023-10-07T09:47:01.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56fc99a700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f8155d30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c590 tx=0x7f56e800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:01.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011590 con 0x7f56f8157290 2023-10-07T09:47:01.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e8011710 con 0x7f56f8157290 2023-10-07T09:47:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.266+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011ad0 con 0x7f56f8157290 2023-10-07T09:47:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.267+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f816dee0 con 0x7f56f8157290 2023-10-07T09:47:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.267+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f80c6850 con 0x7f56f8157290 2023-10-07T09:47:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.268+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f80ca210 con 0x7f56f8157290 2023-10-07T09:47:01.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.270+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f56e801b3b0 con 0x7f56f8157290 2023-10-07T09:47:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.270+0000 7f56ed7fa700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 0x7f56dc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.271+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f56e8083ff0 con 0x7f56f8157290 2023-10-07T09:47:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.271+0000 7f56fd19b700 1 -- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 msgr2=0x7f56dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.271+0000 7f56fd19b700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 0x7f56dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:01.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.275+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56e8050c80 con 0x7f56f8157290 2023-10-07T09:47:01.365 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.363+0000 7fb44f0681c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:47:01.441 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.440+0000 7fb67e09d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:47:01.456 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.455+0000 7fb44f0681c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:47:01.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.472+0000 7f56fd19b700 1 -- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 msgr2=0x7f56dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:01.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.472+0000 7f56fd19b700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 0x7f56dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:01.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.525+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56f80c6a70 con 0x7f56f8157290 2023-10-07T09:47:01.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.525+0000 7f56ed7fa700 1 -- 172.21.15.29:0/1035902238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f56e8017070 con 0x7f56f8157290 2023-10-07T09:47:01.527 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:01.527 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:38.377260+0000","modified":"2023-10-07T09:45:41.892365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3302133648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3302133648},{"type":"v1","addr":"172.21.15.29:6835","nonce":3302133648}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-10-07T09:47:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.528+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 msgr2=0x7f56dc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.528+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 0x7f56dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.528+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 msgr2=0x7f56f8155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.528+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f8155d30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c590 tx=0x7f56e800c8a0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 shutdown_connections 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f56dc067170 0x7f56dc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56f816d4b0 0x7f56f80cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56f80b1a10 0x7f56f816a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 --2- 172.21.15.29:0/1035902238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f8157290 0x7f56f8155d30 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 >> 172.21.15.29:0/1035902238 conn(0x7f56f80680a0 msgr2=0x7f56f8157f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 shutdown_connections 2023-10-07T09:47:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.529+0000 7f56ff3ff700 1 -- 172.21.15.29:0/1035902238 wait complete. 2023-10-07T09:47:01.532 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-07T09:47:01.536 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.535+0000 7fb67e09d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:47:01.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:47:01.629 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.628+0000 7fb67e09d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:47:01.685 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.684+0000 7fb44f0681c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:47:01.723 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.722+0000 7fb67e09d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:47:01.786 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.785+0000 7fb44f0681c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:47:01.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.793+0000 7f435ddff700 1 Processor -- start 2023-10-07T09:47:01.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.793+0000 7f435ddff700 1 -- start start 2023-10-07T09:47:01.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.794+0000 7f435ddff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4358157300 0x7f43580b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.794+0000 7f435ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43581576d0 0x7f43580b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.794+0000 7f435ddff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43580cb330 con 0x7f435806f410 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.794+0000 7f435ddff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435806f980 con 0x7f4358157300 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.794+0000 7f435ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43580cbe90 con 0x7f43581576d0 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4357fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f435806f410 0x7f4358067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55014/0 (socket says 172.21.15.29:55014) 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4357fff700 1 -- 172.21.15.29:0/2688006472 learned_addr learned my addr 172.21.15.29:0/2688006472 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f43577fe700 1 --2- 172.21.15.29:0/2688006472 >> v2:172.21.15.144:3300/0 conn(0x7f4358157300 0x7f43580b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4356ffd700 1 --2- 172.21.15.29:0/2688006472 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43581576d0 0x7f43580b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f43567fc700 1 -- 172.21.15.29:0/2688006472 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478232417 0 0) 0x7f43580cb330 con 0x7f435806f410 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f43567fc700 1 -- 172.21.15.29:0/2688006472 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4340003570 con 0x7f435806f410 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4356ffd700 1 -- 172.21.15.29:0/2688006472 >> v2:172.21.15.144:3300/0 conn(0x7f4358157300 msgr2=0x7f43580b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4356ffd700 1 --2- 172.21.15.29:0/2688006472 >> v2:172.21.15.144:3300/0 conn(0x7f4358157300 0x7f43580b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4356ffd700 1 -- 172.21.15.29:0/2688006472 >> v1:172.21.15.29:6789/0 conn(0x7f435806f410 legacy=0x7f4358067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f4356ffd700 1 -- 172.21.15.29:0/2688006472 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43580cb560 con 0x7f43581576d0 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.795+0000 7f43577fe700 1 --2- 172.21.15.29:0/2688006472 >> v2:172.21.15.144:3300/0 conn(0x7f4358157300 0x7f43580b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f4356ffd700 1 --2- 172.21.15.29:0/2688006472 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43581576d0 0x7f43580b28c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f434c0098e0 tx=0x7f434c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57a7aae6a2ecfc49 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f43567fc700 1 -- 172.21.15.29:0/2688006472 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c011630 con 0x7f43581576d0 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f43567fc700 1 -- 172.21.15.29:0/2688006472 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c011790 con 0x7f43581576d0 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f43567fc700 1 -- 172.21.15.29:0/2688006472 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c011b30 con 0x7f43581576d0 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 -- 172.21.15.29:0/2688006472 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43581576d0 msgr2=0x7f43580b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 --2- 172.21.15.29:0/2688006472 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43581576d0 0x7f43580b28c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f434c0098e0 tx=0x7f434c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 -- 172.21.15.29:0/2688006472 shutdown_connections 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 --2- 172.21.15.29:0/2688006472 >> v2:172.21.15.144:3300/0 conn(0x7f4358157300 0x7f43580b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 --2- 172.21.15.29:0/2688006472 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43581576d0 0x7f43580b28c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 -- 172.21.15.29:0/2688006472 >> 172.21.15.29:0/2688006472 conn(0x7f4358069190 msgr2=0x7f435806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.796+0000 7f435ddff700 1 -- 172.21.15.29:0/2688006472 shutdown_connections 2023-10-07T09:47:01.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.797+0000 7f435ddff700 1 -- 172.21.15.29:0/2688006472 wait complete. 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.798+0000 7f435ddff700 1 Processor -- start 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.798+0000 7f435ddff700 1 -- start start 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f435ddff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 0x7f43580aa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f435ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4358157300 0x7f43580aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f435ddff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43581693c0 0x7f4358169840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f435ddff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43580aad10 con 0x7f435806f410 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f435ddff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43580ac310 con 0x7f43581693c0 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f435ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435816ad30 con 0x7f4358157300 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f43577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 0x7f43580aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f43577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 0x7f43580aa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37134/0 (socket says 172.21.15.29:37134) 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f43577fe700 1 -- 172.21.15.29:0/2300123777 learned_addr learned my addr 172.21.15.29:0/2300123777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f4357fff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43581693c0 0x7f4358169840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.799+0000 7f4356ffd700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4358157300 0x7f43580aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f43577fe700 1 -- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4358157300 msgr2=0x7f43580aa7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f43577fe700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4358157300 0x7f43580aa7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f43577fe700 1 -- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43581693c0 msgr2=0x7f4358169840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f43577fe700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43581693c0 0x7f4358169840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f43577fe700 1 -- 172.21.15.29:0/2300123777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4348009420 con 0x7f435806f410 2023-10-07T09:47:01.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f43577fe700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 0x7f43580aa290 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f434800ec80 tx=0x7f43480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:01.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434800c8c0 con 0x7f435806f410 2023-10-07T09:47:01.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434800ca20 con 0x7f435806f410 2023-10-07T09:47:01.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c009420 con 0x7f435806f410 2023-10-07T09:47:01.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434800cda0 con 0x7f435806f410 2023-10-07T09:47:01.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.800+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43580ab1d0 con 0x7f435806f410 2023-10-07T09:47:01.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.801+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435804c3c0 con 0x7f435806f410 2023-10-07T09:47:01.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.803+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f4348013680 con 0x7f435806f410 2023-10-07T09:47:01.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.804+0000 7f433f7fe700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 0x7f4344069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:01.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.804+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4348085720 con 0x7f435806f410 2023-10-07T09:47:01.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.804+0000 7f4356ffd700 1 -- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 msgr2=0x7f4344069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:01.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.804+0000 7f4356ffd700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 0x7f4344069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:01.806+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f43480523b0 con 0x7f435806f410 2023-10-07T09:47:01.888 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.887+0000 7fb44f0681c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:47:01.909 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:01.908+0000 7fb67e09d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:47:01.987 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:01.986+0000 7fb44f0681c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:47:02.005 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:02.004+0000 7fb67e09d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:47:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.005+0000 7f4356ffd700 1 -- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 msgr2=0x7f4344069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.005+0000 7f4356ffd700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 0x7f4344069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:02.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.010+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43580cb840 con 0x7f435806f410 2023-10-07T09:47:02.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.011+0000 7f433f7fe700 1 -- 172.21.15.29:0/2300123777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16504 (secure 0 0 0) 0x7f43480559d0 con 0x7f435806f410 2023-10-07T09:47:02.013 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:02.013 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":106,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:46:56.275599+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:45:36.139381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:37.345605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]},{"pgid":"13.32","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1688934630":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/1525826437":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/2483656043":"2023-10-08T09:45:54.269145+0000","172.21.15.144:6836/114714":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/3266577652":"2023-10-08T09:45:54.269145+0000","172.21.15.144:6834/114714":"2023-10-08T09:45:54.269145+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 msgr2=0x7f4344069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 0x7f4344069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 msgr2=0x7f43580aa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 0x7f43580aa290 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f434800ec80 tx=0x7f43480091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 shutdown_connections 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4344067170 0x7f4344069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43581693c0 0x7f4358169840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f435806f410 0x7f43580aa290 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 --2- 172.21.15.29:0/2300123777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4358157300 0x7f43580aa7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 >> 172.21.15.29:0/2300123777 conn(0x7f4358069190 msgr2=0x7f43581538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:02.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 shutdown_connections 2023-10-07T09:47:02.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.014+0000 7f435ddff700 1 -- 172.21.15.29:0/2300123777 wait complete. 2023-10-07T09:47:02.033 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:47:02.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:47:02.189 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:02.188+0000 7fb44f0681c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:47:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.231+0000 7ff0943ff700 1 Processor -- start 2023-10-07T09:47:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.231+0000 7ff0943ff700 1 -- start start 2023-10-07T09:47:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff0943ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff08c153cc0 0x7ff08c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff0943ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c15ade0 0x7ff08c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff0943ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff08c0b0000 0x7ff08c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff0943ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0b0980 con 0x7ff08c153cc0 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff0943ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c154090 con 0x7ff08c0b0000 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff0943ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c159f60 con 0x7ff08c15ade0 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff08c153cc0 0x7ff08c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff08c153cc0 0x7ff08c0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37136/0 (socket says 172.21.15.29:37136) 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 -- 172.21.15.29:0/1734233091 learned_addr learned my addr 172.21.15.29:0/1734233091 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09199a700 1 --2- 172.21.15.29:0/1734233091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c15ade0 0x7ff08c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09299c700 1 --2- 172.21.15.29:0/1734233091 >> v2:172.21.15.144:3300/0 conn(0x7ff08c0b0000 0x7ff08c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 -- 172.21.15.29:0/1734233091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c15ade0 msgr2=0x7ff08c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 --2- 172.21.15.29:0/1734233091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c15ade0 0x7ff08c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 -- 172.21.15.29:0/1734233091 >> v2:172.21.15.144:3300/0 conn(0x7ff08c0b0000 msgr2=0x7ff08c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 --2- 172.21.15.29:0/1734233091 >> v2:172.21.15.144:3300/0 conn(0x7ff08c0b0000 0x7ff08c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.233+0000 7ff09219b700 1 -- 172.21.15.29:0/1734233091 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c15a240 con 0x7ff08c153cc0 2023-10-07T09:47:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.234+0000 7ff09219b700 1 --2- 172.21.15.29:0/1734233091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c0b17c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff08c068230 tx=0x7ff088009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695d5cbcbb72f84f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.234+0000 7ff091199700 1 -- 172.21.15.29:0/1734233091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088005c10 con 0x7ff08c153cc0 2023-10-07T09:47:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.234+0000 7ff091199700 1 -- 172.21.15.29:0/1734233091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088005d70 con 0x7ff08c153cc0 2023-10-07T09:47:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.234+0000 7ff091199700 1 -- 172.21.15.29:0/1734233091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088006260 con 0x7ff08c153cc0 2023-10-07T09:47:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 -- 172.21.15.29:0/1734233091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 msgr2=0x7ff08c0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 --2- 172.21.15.29:0/1734233091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c0b17c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff08c068230 tx=0x7ff088009220 comp rx=0 tx=0).stop 2023-10-07T09:47:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 -- 172.21.15.29:0/1734233091 shutdown_connections 2023-10-07T09:47:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 --2- 172.21.15.29:0/1734233091 >> v2:172.21.15.144:3300/0 conn(0x7ff08c0b0000 0x7ff08c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 --2- 172.21.15.29:0/1734233091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c0b17c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 --2- 172.21.15.29:0/1734233091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c15ade0 0x7ff08c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 -- 172.21.15.29:0/1734233091 >> 172.21.15.29:0/1734233091 conn(0x7ff08c067610 msgr2=0x7ff08c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:02.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 -- 172.21.15.29:0/1734233091 shutdown_connections 2023-10-07T09:47:02.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.235+0000 7ff0943ff700 1 -- 172.21.15.29:0/1734233091 wait complete. 2023-10-07T09:47:02.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.236+0000 7ff0943ff700 1 Processor -- start 2023-10-07T09:47:02.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.237+0000 7ff0943ff700 1 -- start start 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.237+0000 7ff0943ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c0b0000 0x7ff08c1564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.237+0000 7ff0943ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c16a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff0943ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08c15ade0 0x7ff08c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff0943ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c85b0 con 0x7ff08c153cc0 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff0943ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c169ee0 con 0x7ff08c15ade0 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff0943ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c156a30 con 0x7ff08c0b0000 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09199a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c16a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09199a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c16a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37140/0 (socket says 172.21.15.29:37140) 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09199a700 1 -- 172.21.15.29:0/2570765825 learned_addr learned my addr 172.21.15.29:0/2570765825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09219b700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c0b0000 0x7ff08c1564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09299c700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08c15ade0 0x7ff08c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09219b700 1 -- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08c15ade0 msgr2=0x7ff08c15e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09219b700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08c15ade0 0x7ff08c15e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09219b700 1 -- 172.21.15.29:0/2570765825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 msgr2=0x7ff08c16a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09219b700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c16a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.238+0000 7ff09219b700 1 -- 172.21.15.29:0/2570765825 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088009480 con 0x7ff08c0b0000 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.239+0000 7ff09219b700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c0b0000 0x7ff08c1564f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff088015000 tx=0x7ff088016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.239+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0880117d0 con 0x7ff08c0b0000 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.239+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088011930 con 0x7ff08c0b0000 2023-10-07T09:47:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.239+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088011cf0 con 0x7ff08c0b0000 2023-10-07T09:47:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.239+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c0c7ff0 con 0x7ff08c0b0000 2023-10-07T09:47:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.239+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c169b70 con 0x7ff08c0b0000 2023-10-07T09:47:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.240+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c16a1c0 con 0x7ff08c0b0000 2023-10-07T09:47:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.241+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7ff08801b640 con 0x7ff08c0b0000 2023-10-07T09:47:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.242+0000 7ff0827fc700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 0x7ff070069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.242+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff088085ea0 con 0x7ff08c0b0000 2023-10-07T09:47:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.242+0000 7ff09199a700 1 -- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 msgr2=0x7ff070069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.242+0000 7ff09199a700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 0x7ff070069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:02.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.247+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff088052b30 con 0x7ff08c0b0000 2023-10-07T09:47:02.297 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:02.295+0000 7fb44f0681c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:47:02.349 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:02.348+0000 7fb67e09d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:47:02.439 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:02.438+0000 7fb67e09d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:02.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.443+0000 7ff09199a700 1 -- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 msgr2=0x7ff070069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:02.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.443+0000 7ff09199a700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 0x7ff070069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.494+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08c1699a0 con 0x7ff08c0b0000 2023-10-07T09:47:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.494+0000 7ff0827fc700 1 -- 172.21.15.29:0/2570765825 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4599 (secure 0 0 0) 0x7ff088056150 con 0x7ff08c0b0000 2023-10-07T09:47:02.496 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:02.496 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:38.377260+0000","modified":"2023-10-07T09:45:41.892365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3302133648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3302133648},{"type":"v1","addr":"172.21.15.29:6835","nonce":3302133648}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-10-07T09:47:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 msgr2=0x7ff070069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 0x7ff070069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c0b0000 msgr2=0x7ff08c1564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c0b0000 0x7ff08c1564f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff088015000 tx=0x7ff088016000 comp rx=0 tx=0).stop 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 shutdown_connections 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff070067170 0x7ff070069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08c15ade0 0x7ff08c15e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08c0b0000 0x7ff08c1564f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 --2- 172.21.15.29:0/2570765825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08c153cc0 0x7ff08c16a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 >> 172.21.15.29:0/2570765825 conn(0x7ff08c067610 msgr2=0x7ff08c0b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 shutdown_connections 2023-10-07T09:47:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.497+0000 7ff0943ff700 1 -- 172.21.15.29:0/2570765825 wait complete. 2023-10-07T09:47:02.500 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-07T09:47:02.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:47:02.652 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:02.651+0000 7fb44f0681c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:47:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.715+0000 7efe4d5ff700 1 Processor -- start 2023-10-07T09:47:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.715+0000 7efe4d5ff700 1 -- start start 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.716+0000 7efe4d5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe48157290 0x7efe480cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.716+0000 7efe4d5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe480ae8b0 0x7efe480b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.716+0000 7efe4d5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe48158ad0 con 0x7efe480b1840 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.716+0000 7efe4d5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe480b1cc0 con 0x7efe480ae8b0 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.716+0000 7efe4d5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4815b140 con 0x7efe48157290 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe467fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe480ae8b0 0x7efe480b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe477fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efe480b1840 0x7efe480ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55022/0 (socket says 172.21.15.29:55022) 2023-10-07T09:47:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe477fe700 1 -- 172.21.15.29:0/4018529828 learned_addr learned my addr 172.21.15.29:0/4018529828 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe46ffd700 1 --2- 172.21.15.29:0/4018529828 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe48157290 0x7efe480cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe45ffb700 1 -- 172.21.15.29:0/4018529828 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503503975 0 0) 0x7efe48158ad0 con 0x7efe480b1840 2023-10-07T09:47:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe45ffb700 1 -- 172.21.15.29:0/4018529828 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30003570 con 0x7efe480b1840 2023-10-07T09:47:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe467fc700 1 -- 172.21.15.29:0/4018529828 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe48157290 msgr2=0x7efe480cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe467fc700 1 --2- 172.21.15.29:0/4018529828 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe48157290 0x7efe480cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe467fc700 1 -- 172.21.15.29:0/4018529828 >> v1:172.21.15.29:6789/0 conn(0x7efe480b1840 legacy=0x7efe480ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.717+0000 7efe467fc700 1 -- 172.21.15.29:0/4018529828 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4815b420 con 0x7efe480ae8b0 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.718+0000 7efe467fc700 1 --2- 172.21.15.29:0/4018529828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe480ae8b0 0x7efe480b1300 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0098e0 tx=0x7efe3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f0befae7530762 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.718+0000 7efe45ffb700 1 -- 172.21.15.29:0/4018529828 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c011630 con 0x7efe480ae8b0 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.718+0000 7efe45ffb700 1 -- 172.21.15.29:0/4018529828 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c011790 con 0x7efe480ae8b0 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.718+0000 7efe45ffb700 1 -- 172.21.15.29:0/4018529828 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c011b30 con 0x7efe480ae8b0 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 -- 172.21.15.29:0/4018529828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe480ae8b0 msgr2=0x7efe480b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/4018529828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe480ae8b0 0x7efe480b1300 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0098e0 tx=0x7efe3c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 -- 172.21.15.29:0/4018529828 shutdown_connections 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/4018529828 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe48157290 0x7efe480cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/4018529828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe480ae8b0 0x7efe480b1300 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 -- 172.21.15.29:0/4018529828 >> 172.21.15.29:0/4018529828 conn(0x7efe480680a0 msgr2=0x7efe480cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 -- 172.21.15.29:0/4018529828 shutdown_connections 2023-10-07T09:47:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.719+0000 7efe4d5ff700 1 -- 172.21.15.29:0/4018529828 wait complete. 2023-10-07T09:47:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.720+0000 7efe4d5ff700 1 Processor -- start 2023-10-07T09:47:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.720+0000 7efe4d5ff700 1 -- start start 2023-10-07T09:47:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.721+0000 7efe4d5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe480b1840 0x7efe4814abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.721+0000 7efe4d5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 0x7efe4814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.721+0000 7efe4d5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe4814b120 0x7efe4815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.721+0000 7efe4d5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4814fbf0 con 0x7efe48157290 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.721+0000 7efe4d5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4814fed0 con 0x7efe480b1840 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.721+0000 7efe4d5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe481501b0 con 0x7efe4814b120 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 0x7efe4814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 0x7efe4814f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37166/0 (socket says 172.21.15.29:37166) 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 -- 172.21.15.29:0/3001502993 learned_addr learned my addr 172.21.15.29:0/3001502993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe46ffd700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe480b1840 0x7efe4814abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe477fe700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe4814b120 0x7efe4815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 -- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe4814b120 msgr2=0x7efe4815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe4814b120 0x7efe4815bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 -- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe480b1840 msgr2=0x7efe4814abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe480b1840 0x7efe4814abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 -- 172.21.15.29:0/3001502993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c009420 con 0x7efe48157290 2023-10-07T09:47:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.722+0000 7efe467fc700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 0x7efe4814f120 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efe48068dd0 tx=0x7efe3c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.723+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c011900 con 0x7efe48157290 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.723+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c011a60 con 0x7efe48157290 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.723+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe48150490 con 0x7efe48157290 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.723+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c025640 con 0x7efe48157290 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.723+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4815a060 con 0x7efe48157290 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.724+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4804c3c0 con 0x7efe48157290 2023-10-07T09:47:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.725+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7efe3c025a30 con 0x7efe48157290 2023-10-07T09:47:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.726+0000 7efe2effd700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 0x7efe34069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.726+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efe3c086810 con 0x7efe48157290 2023-10-07T09:47:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.726+0000 7efe46ffd700 1 -- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 msgr2=0x7efe34069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.726+0000 7efe46ffd700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 0x7efe34069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:02.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.730+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe3c053520 con 0x7efe48157290 2023-10-07T09:47:02.747 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:02.747+0000 7fb44f0681c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:02.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.927+0000 7efe46ffd700 1 -- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 msgr2=0x7efe34069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:02.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.927+0000 7efe46ffd700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 0x7efe34069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:02.972 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:02.971+0000 7fb67e09d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:47:02.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.978+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4815b700 con 0x7efe48157290 2023-10-07T09:47:02.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.978+0000 7efe2effd700 1 -- 172.21.15.29:0/3001502993 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4599 (secure 0 0 0) 0x7efe3c05c070 con 0x7efe48157290 2023-10-07T09:47:02.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:02.980 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:45:38.377260+0000","modified":"2023-10-07T09:45:41.892365+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3302133648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3302133648},{"type":"v1","addr":"172.21.15.29:6835","nonce":3302133648}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-10-07T09:47:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 msgr2=0x7efe34069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 0x7efe34069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 msgr2=0x7efe4814f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 0x7efe4814f120 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efe48068dd0 tx=0x7efe3c017000 comp rx=0 tx=0).stop 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 shutdown_connections 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7efe34067170 0x7efe34069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe4814b120 0x7efe4815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe480b1840 0x7efe4814abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 --2- 172.21.15.29:0/3001502993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe48157290 0x7efe4814f120 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 >> 172.21.15.29:0/3001502993 conn(0x7efe480680a0 msgr2=0x7efe4814cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 shutdown_connections 2023-10-07T09:47:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:02.981+0000 7efe4d5ff700 1 -- 172.21.15.29:0/3001502993 wait complete. 2023-10-07T09:47:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-07T09:47:03.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:47:03.156 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:03.155+0000 7fb67e09d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:47:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.197+0000 7f7965fff700 1 Processor -- start 2023-10-07T09:47:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.197+0000 7f7965fff700 1 -- start start 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.199+0000 7f7965fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f7960159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.199+0000 7f7965fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79600ad5f0 0x7f79600ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f7965fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7960068620 0x7f7960068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f7965fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7960068fc0 con 0x7f7960068620 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f7965fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796015af00 con 0x7f79600ad5f0 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f7965fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600ac9d0 con 0x7f79600ad220 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7960068620 0x7f7960068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7960068620 0x7f7960068a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53246/0 (socket says 172.21.15.29:53246) 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 -- 172.21.15.29:0/69563366 learned_addr learned my addr 172.21.15.29:0/69563366 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795effd700 1 --2- 172.21.15.29:0/69563366 >> v2:172.21.15.144:3300/0 conn(0x7f79600ad5f0 0x7f79600ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795f7fe700 1 --2- 172.21.15.29:0/69563366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f7960159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 -- 172.21.15.29:0/69563366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 msgr2=0x7f7960159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 --2- 172.21.15.29:0/69563366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f7960159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 -- 172.21.15.29:0/69563366 >> v2:172.21.15.144:3300/0 conn(0x7f79600ad5f0 msgr2=0x7f79600ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.200+0000 7f795ffff700 1 --2- 172.21.15.29:0/69563366 >> v2:172.21.15.144:3300/0 conn(0x7f79600ad5f0 0x7f79600ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.201+0000 7f795ffff700 1 -- 172.21.15.29:0/69563366 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79600ada60 con 0x7f7960068620 2023-10-07T09:47:03.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.201+0000 7f795ffff700 1 --2- 172.21.15.29:0/69563366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7960068620 0x7f7960068a80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f796006d710 tx=0x7f7958009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7c9edc37511f16 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:03.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.201+0000 7f795e7fc700 1 -- 172.21.15.29:0/69563366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795800e070 con 0x7f7960068620 2023-10-07T09:47:03.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.201+0000 7f795e7fc700 1 -- 172.21.15.29:0/69563366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795800b970 con 0x7f7960068620 2023-10-07T09:47:03.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.201+0000 7f795e7fc700 1 -- 172.21.15.29:0/69563366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79580092e0 con 0x7f7960068620 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 -- 172.21.15.29:0/69563366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7960068620 msgr2=0x7f7960068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 --2- 172.21.15.29:0/69563366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7960068620 0x7f7960068a80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f796006d710 tx=0x7f7958009840 comp rx=0 tx=0).stop 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 -- 172.21.15.29:0/69563366 shutdown_connections 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 --2- 172.21.15.29:0/69563366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7960068620 0x7f7960068a80 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 --2- 172.21.15.29:0/69563366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f7960159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 --2- 172.21.15.29:0/69563366 >> v2:172.21.15.144:3300/0 conn(0x7f79600ad5f0 0x7f79600ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 -- 172.21.15.29:0/69563366 >> 172.21.15.29:0/69563366 conn(0x7f796006dd00 msgr2=0x7f796006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 -- 172.21.15.29:0/69563366 shutdown_connections 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.202+0000 7f7965fff700 1 -- 172.21.15.29:0/69563366 wait complete. 2023-10-07T09:47:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.204+0000 7f7965fff700 1 Processor -- start 2023-10-07T09:47:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.204+0000 7f7965fff700 1 -- start start 2023-10-07T09:47:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f7965fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 0x7f79600b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f7965fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f79600cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f7965fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ad5f0 0x7f79601530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f7965fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79600b1a00 con 0x7f79600ad5f0 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f7965fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7960169910 con 0x7f7960068620 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f7965fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600b0f50 con 0x7f79600ad220 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 0x7f79600b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ad5f0 0x7f79601530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 0x7f79600b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:39668/0 (socket says 172.21.15.29:39668) 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 -- 172.21.15.29:0/565598122 learned_addr learned my addr 172.21.15.29:0/565598122 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795effd700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f79600cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 -- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 msgr2=0x7f79600cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f79600cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 -- 172.21.15.29:0/565598122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ad5f0 msgr2=0x7f79601530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ad5f0 0x7f79601530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795f7fe700 1 -- 172.21.15.29:0/565598122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958009ae0 con 0x7f7960068620 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.205+0000 7f795effd700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f79600cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.206+0000 7f795f7fe700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 0x7f79600b0a10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f795000e950 tx=0x7f79500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.206+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950013680 con 0x7f7960068620 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.206+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79500137e0 con 0x7f7960068620 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.206+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950013ba0 con 0x7f7960068620 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.206+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79600b11c0 con 0x7f7960068620 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.206+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7960169b20 con 0x7f7960068620 2023-10-07T09:47:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.208+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7960169d40 con 0x7f7960068620 2023-10-07T09:47:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.209+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f79500123d0 con 0x7f7960068620 2023-10-07T09:47:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.210+0000 7f79477fe700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 0x7f794c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.210+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7950084cb0 con 0x7f7960068620 2023-10-07T09:47:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.210+0000 7f795effd700 1 -- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 msgr2=0x7f794c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.210+0000 7f795effd700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 0x7f794c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:03.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.219+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79500519e0 con 0x7f7960068620 2023-10-07T09:47:03.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:03.325+0000 7fb44f0681c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:47:03.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.411+0000 7f795effd700 1 -- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 msgr2=0x7f794c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:03.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.411+0000 7f795effd700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 0x7f794c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.427+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f796004c3c0 con 0x7f7960068620 2023-10-07T09:47:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.428+0000 7f79477fe700 1 -- 172.21.15.29:0/565598122 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16504 (secure 0 0 0) 0x7f7950017090 con 0x7f7960068620 2023-10-07T09:47:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:03.430 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":106,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:46:56.275599+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:45:36.139381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-07T09:45:37.345605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]},{"pgid":"13.32","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1688934630":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/1525826437":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/2483656043":"2023-10-08T09:45:54.269145+0000","172.21.15.144:6836/114714":"2023-10-08T09:45:54.269145+0000","172.21.15.144:0/3266577652":"2023-10-08T09:45:54.269145+0000","172.21.15.144:6834/114714":"2023-10-08T09:45:54.269145+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 msgr2=0x7f794c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 0x7f794c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 msgr2=0x7f79600b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 0x7f79600b0a10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f795000e950 tx=0x7f79500091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 shutdown_connections 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f794c067170 0x7f794c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79600ad5f0 0x7f79601530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7960068620 0x7f79600b0a10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.430+0000 7f7965fff700 1 --2- 172.21.15.29:0/565598122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79600ad220 0x7f79600cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.431+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 >> 172.21.15.29:0/565598122 conn(0x7f796006dd00 msgr2=0x7f79600aaec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.431+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 shutdown_connections 2023-10-07T09:47:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.431+0000 7f7965fff700 1 -- 172.21.15.29:0/565598122 wait complete. 2023-10-07T09:47:03.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:47:03.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:03.524+0000 7fb44f0681c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:47:03.538 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:03.537+0000 7fb67e09d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.653+0000 7f5af37ff700 1 Processor -- start 2023-10-07T09:47:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.653+0000 7f5af37ff700 1 -- start start 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec159320 0x7f5aec1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af37ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5aec0afd20 0x7f5aec0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af37ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5aec0cc9c0 0x7f5aec0acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af37ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec0b0dc0 con 0x7f5aec0afd20 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af37ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec0b01b0 con 0x7f5aec0cc9c0 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0b0490 con 0x7f5aec159320 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af0d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5aec0afd20 0x7f5aec0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af0d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5aec0afd20 0x7f5aec0b1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53270/0 (socket says 172.21.15.29:53270) 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af0d9a700 1 -- 172.21.15.29:0/2377364777 learned_addr learned my addr 172.21.15.29:0/2377364777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af159b700 1 --2- 172.21.15.29:0/2377364777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec159320 0x7f5aec1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.655+0000 7f5af1d9c700 1 --2- 172.21.15.29:0/2377364777 >> v2:172.21.15.144:3300/0 conn(0x7f5aec0cc9c0 0x7f5aec0acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5af0d9a700 1 -- 172.21.15.29:0/2377364777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec159320 msgr2=0x7f5aec1596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5af0d9a700 1 --2- 172.21.15.29:0/2377364777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec159320 0x7f5aec1596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5af0d9a700 1 -- 172.21.15.29:0/2377364777 >> v2:172.21.15.144:3300/0 conn(0x7f5aec0cc9c0 msgr2=0x7f5aec0acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5af0d9a700 1 --2- 172.21.15.29:0/2377364777 >> v2:172.21.15.144:3300/0 conn(0x7f5aec0cc9c0 0x7f5aec0acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5af0d9a700 1 -- 172.21.15.29:0/2377364777 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0b0770 con 0x7f5aec0afd20 2023-10-07T09:47:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5af0d9a700 1 --2- 172.21.15.29:0/2377364777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec0b1250 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06cf20 tx=0x7f5adc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cded6c7607e23b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5ae3fff700 1 -- 172.21.15.29:0/2377364777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00b6d0 con 0x7f5aec0afd20 2023-10-07T09:47:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5ae3fff700 1 -- 172.21.15.29:0/2377364777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc00b830 con 0x7f5aec0afd20 2023-10-07T09:47:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.656+0000 7f5ae3fff700 1 -- 172.21.15.29:0/2377364777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00bbf0 con 0x7f5aec0afd20 2023-10-07T09:47:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 -- 172.21.15.29:0/2377364777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 msgr2=0x7f5aec0b1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 --2- 172.21.15.29:0/2377364777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec0b1250 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06cf20 tx=0x7f5adc009250 comp rx=0 tx=0).stop 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 -- 172.21.15.29:0/2377364777 shutdown_connections 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 --2- 172.21.15.29:0/2377364777 >> v2:172.21.15.144:3300/0 conn(0x7f5aec0cc9c0 0x7f5aec0acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 --2- 172.21.15.29:0/2377364777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec159320 0x7f5aec1596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 --2- 172.21.15.29:0/2377364777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec0b1250 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 -- 172.21.15.29:0/2377364777 >> 172.21.15.29:0/2377364777 conn(0x7f5aec0680a0 msgr2=0x7f5aec0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 -- 172.21.15.29:0/2377364777 shutdown_connections 2023-10-07T09:47:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.657+0000 7f5af37ff700 1 -- 172.21.15.29:0/2377364777 wait complete. 2023-10-07T09:47:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.659+0000 7f5af37ff700 1 Processor -- start 2023-10-07T09:47:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.659+0000 7f5af37ff700 1 -- start start 2023-10-07T09:47:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af37ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0cc9c0 0x7f5aec15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af37ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec159320 0x7f5aec15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af37ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec169a80 con 0x7f5aec0afd20 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af37ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec15b160 con 0x7f5aec159320 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec15d060 con 0x7f5aec0cc9c0 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af159b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af159b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec146c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53284/0 (socket says 172.21.15.29:53284) 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af159b700 1 -- 172.21.15.29:0/3745544563 learned_addr learned my addr 172.21.15.29:0/3745544563 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af0d9a700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0cc9c0 0x7f5aec15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.660+0000 7f5af1d9c700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec159320 0x7f5aec15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:03.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af159b700 1 -- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0cc9c0 msgr2=0x7f5aec15eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af159b700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0cc9c0 0x7f5aec15eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af159b700 1 -- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec159320 msgr2=0x7f5aec15f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af159b700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec159320 0x7f5aec15f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af159b700 1 -- 172.21.15.29:0/3745544563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0094f0 con 0x7f5aec0afd20 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af159b700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec146c30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800e950 tx=0x7f5ae80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8013680 con 0x7f5aec0afd20 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae80137e0 con 0x7f5aec0afd20 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8013ba0 con 0x7f5aec0afd20 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec15c730 con 0x7f5aec0afd20 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.661+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec15ccf0 con 0x7f5aec0afd20 2023-10-07T09:47:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.663+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec04c3c0 con 0x7f5aec0afd20 2023-10-07T09:47:03.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.664+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f5ae80123d0 con 0x7f5aec0afd20 2023-10-07T09:47:03.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.665+0000 7f5ae17fa700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 0x7f5ad0069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:03.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.665+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5ae8084c90 con 0x7f5aec0afd20 2023-10-07T09:47:03.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.665+0000 7f5af0d9a700 1 -- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 msgr2=0x7f5ad0069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:03.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.665+0000 7f5af0d9a700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 0x7f5ad0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:03.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.669+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ae80519c0 con 0x7f5aec0afd20 2023-10-07T09:47:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.866+0000 7f5af0d9a700 1 -- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 msgr2=0x7f5ad0069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.866+0000 7f5af0d9a700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 0x7f5ad0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:03.913 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:03.912+0000 7fb44f0681c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:03.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.918+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5aec0b0a50 con 0x7f5aec0afd20 2023-10-07T09:47:03.920 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:47:03.919+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:47:03.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.923+0000 7f5ae17fa700 1 -- 172.21.15.29:0/3745544563 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5ae8017090 con 0x7f5aec0afd20 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 msgr2=0x7f5ad0069650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 0x7f5ad0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 msgr2=0x7f5aec146c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec146c30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800e950 tx=0x7f5ae80091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 shutdown_connections 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad00671a0 0x7f5ad0069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec159320 0x7f5aec15f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0afd20 0x7f5aec146c30 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 --2- 172.21.15.29:0/3745544563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0cc9c0 0x7f5aec15eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 >> 172.21.15.29:0/3745544563 conn(0x7f5aec0680a0 msgr2=0x7f5aec0acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 shutdown_connections 2023-10-07T09:47:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:03.927+0000 7f5af37ff700 1 -- 172.21.15.29:0/3745544563 wait complete. 2023-10-07T09:47:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:47:03.935 INFO:tasks.ceph.mds.c.smithi029.stderr: -6013> 2023-10-07T09:45:44.842+0000 7fc95e5f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:47:03.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:47:03.958 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:03.956+0000 7fb67e09d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:47:03.983 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:47:04.140 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.139+0000 7fb67e09d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:04.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.168+0000 7f32ef9ff700 1 Processor -- start 2023-10-07T09:47:04.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.168+0000 7f32ef9ff700 1 -- start start 2023-10-07T09:47:04.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.169+0000 7f32ef9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80cc800 0x7f32e80acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:04.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.169+0000 7f32ef9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e8159ab0 con 0x7f32e80ad440 2023-10-07T09:47:04.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.169+0000 7f32ef9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e80a98d0 con 0x7f32e81596e0 2023-10-07T09:47:04.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.169+0000 7f32ef9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80aa8e0 con 0x7f32e80cc800 2023-10-07T09:47:04.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32edf9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32e80ad440 0x7f32e815afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48798/0 (socket says 172.21.15.29:48798) 2023-10-07T09:47:04.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32edf9c700 1 -- 172.21.15.29:0/3307655885 learned_addr learned my addr 172.21.15.29:0/3307655885 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:04.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32ecf9a700 1 --2- 172.21.15.29:0/3307655885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80cc800 0x7f32e80acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283348980 0 0) 0x7f32e00013c0 con 0x7f32e80ad440 2023-10-07T09:47:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d4003570 con 0x7f32e80ad440 2023-10-07T09:47:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253177618 0 0) 0x7f32e80a98d0 con 0x7f32e81596e0 2023-10-07T09:47:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e00013c0 con 0x7f32e81596e0 2023-10-07T09:47:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32ecf9a700 1 -- 172.21.15.29:0/3307655885 >> v1:172.21.15.144:6789/0 conn(0x7f32e81596e0 legacy=0x7f32e80cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32ecf9a700 1 -- 172.21.15.29:0/3307655885 >> v1:172.21.15.29:6789/0 conn(0x7f32e80ad440 legacy=0x7f32e815afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.170+0000 7f32ecf9a700 1 -- 172.21.15.29:0/3307655885 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e80aa450 con 0x7f32e80cc800 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.171+0000 7f32ecf9a700 1 --2- 172.21.15.29:0/3307655885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80cc800 0x7f32e80acf00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009910 tx=0x7f32d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9919c9ee73cbd307 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.171+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8011630 con 0x7f32e80cc800 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.171+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8011790 con 0x7f32e80cc800 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.171+0000 7f32dffff700 1 -- 172.21.15.29:0/3307655885 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8011b30 con 0x7f32e80cc800 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.171+0000 7f32ef9ff700 1 -- 172.21.15.29:0/3307655885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80cc800 msgr2=0x7f32e80acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:04.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.171+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/3307655885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80cc800 0x7f32e80acf00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009910 tx=0x7f32d80091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.172+0000 7f32ef9ff700 1 -- 172.21.15.29:0/3307655885 shutdown_connections 2023-10-07T09:47:04.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.172+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/3307655885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80cc800 0x7f32e80acf00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.172+0000 7f32ef9ff700 1 -- 172.21.15.29:0/3307655885 >> 172.21.15.29:0/3307655885 conn(0x7f32e8067610 msgr2=0x7f32e8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:04.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.172+0000 7f32ef9ff700 1 -- 172.21.15.29:0/3307655885 shutdown_connections 2023-10-07T09:47:04.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.172+0000 7f32ef9ff700 1 -- 172.21.15.29:0/3307655885 wait complete. 2023-10-07T09:47:04.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.173+0000 7f32ef9ff700 1 Processor -- start 2023-10-07T09:47:04.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.173+0000 7f32ef9ff700 1 -- start start 2023-10-07T09:47:04.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ef9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80ad440 0x7f32e80c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:04.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ef9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32e80cc800 0x7f32e81529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ef9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 0x7f32e8150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ef9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81531f0 con 0x7f32e81596e0 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ef9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80c8c20 con 0x7f32e80cc800 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ef9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80c8330 con 0x7f32e80ad440 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32edf9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 0x7f32e8150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32edf9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 0x7f32e8150290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53314/0 (socket says 172.21.15.29:53314) 2023-10-07T09:47:04.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32edf9c700 1 -- 172.21.15.29:0/251788729 learned_addr learned my addr 172.21.15.29:0/251788729 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:04.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ecf9a700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32e80cc800 0x7f32e81529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:04.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.174+0000 7f32ed79b700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80ad440 0x7f32e80c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:04.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32ecf9a700 1 -- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80ad440 msgr2=0x7f32e80c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:04.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32ecf9a700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80ad440 0x7f32e80c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32ecf9a700 1 -- 172.21.15.29:0/251788729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 msgr2=0x7f32e8150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:04.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32ecf9a700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 0x7f32e8150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32ecf9a700 1 -- 172.21.15.29:0/251788729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d8009420 con 0x7f32e80cc800 2023-10-07T09:47:04.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32edf9c700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 0x7f32e8150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:04.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32ecf9a700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32e80cc800 0x7f32e81529a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f32e806d230 tx=0x7f32d8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:04.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.175+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d801f070 con 0x7f32e80cc800 2023-10-07T09:47:04.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.176+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8002f70 con 0x7f32e80cc800 2023-10-07T09:47:04.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.176+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e80c85d0 con 0x7f32e80cc800 2023-10-07T09:47:04.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.176+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8011900 con 0x7f32e80cc800 2023-10-07T09:47:04.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.176+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e8152ee0 con 0x7f32e80cc800 2023-10-07T09:47:04.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.177+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e804c3c0 con 0x7f32e80cc800 2023-10-07T09:47:04.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.178+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 46) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f32d8023070 con 0x7f32e80cc800 2023-10-07T09:47:04.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.179+0000 7f32dd7fa700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 0x7f32cc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:04.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.179+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f32d8085ff0 con 0x7f32e80cc800 2023-10-07T09:47:04.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.179+0000 7f32ed79b700 1 -- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 msgr2=0x7f32cc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:04.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.179+0000 7f32ed79b700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 0x7f32cc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:04.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.186+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32d8052ce0 con 0x7f32e80cc800 2023-10-07T09:47:04.330 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.329+0000 7fb67e09d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:47:04.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.380+0000 7f32ed79b700 1 -- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 msgr2=0x7f32cc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:04.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.380+0000 7f32ed79b700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 0x7f32cc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:04.393 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:04.392+0000 7fb44f0681c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:47:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.437+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f32e80ccfb0 con 0x7f32e80cc800 2023-10-07T09:47:04.441 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.440+0000 7fb67e09d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.456+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 47) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f32d80530b0 con 0x7f32e80cc800 2023-10-07T09:47:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.456+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 msgr2=0x7f32cc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:04.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.456+0000 7f32dd7fa700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 0x7f32cc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.529 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.528+0000 7fb5f8ec4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.529 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.528+0000 7fb5f8ec4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.529 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.528+0000 7fb5f8ec4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.528+0000 7fb5f8ec4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.528+0000 7fb5f8ec4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.529+0000 7fb5f6ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.529+0000 7fb5f6ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.529+0000 7fb5f6ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.529+0000 7fb5f6ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:04.529+0000 7fb5f6ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:04.589 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:04.588+0000 7fb44f0681c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:04.784 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:04.783+0000 7fb44f0681c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:47:04.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:04.895+0000 7fb44f0681c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:04.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.924+0000 7f32dd7fa700 1 -- 172.21.15.29:0/251788729 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7f32d8069740 con 0x7f32e80cc800 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32e80cc800 msgr2=0x7f32e81529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32e80cc800 0x7f32e81529a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f32e806d230 tx=0x7f32d8017000 comp rx=0 tx=0).stop 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 shutdown_connections 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f32cc067170 0x7f32cc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e81596e0 0x7f32e8150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32e80ad440 0x7f32e80c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 --2- 172.21.15.29:0/251788729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32e80cc800 0x7f32e81529a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 >> 172.21.15.29:0/251788729 conn(0x7f32e8067610 msgr2=0x7f32e8153d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 shutdown_connections 2023-10-07T09:47:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:04.927+0000 7f32ef9ff700 1 -- 172.21.15.29:0/251788729 wait complete. 2023-10-07T09:47:04.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:47:05.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.184+0000 7f5f8dfff700 1 Processor -- start 2023-10-07T09:47:05.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.185+0000 7f5f8dfff700 1 -- start start 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.186+0000 7f5f8dfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f881559f0 0x7f5f880b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f8dfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f881596d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f8dfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f880c8c70 con 0x7f5f88159c10 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f8dfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f880c98f0 con 0x7f5f881559f0 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f8dfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8815a090 con 0x7f5f880b0660 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f87fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f88159c10 0x7f5f880adb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48814/0 (socket says 172.21.15.29:48814) 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f87fff700 1 -- 172.21.15.29:0/2933745630 learned_addr learned my addr 172.21.15.29:0/2933745630 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f877fe700 1 --2- 172.21.15.29:0/2933745630 >> v2:172.21.15.29:3300/0 conn(0x7f5f881559f0 0x7f5f880b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:05.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f86ffd700 1 --2- 172.21.15.29:0/2933745630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f881596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:05.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f867fc700 1 -- 172.21.15.29:0/2933745630 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817022989 0 0) 0x7f5f880c8c70 con 0x7f5f88159c10 2023-10-07T09:47:05.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f867fc700 1 -- 172.21.15.29:0/2933745630 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f70003570 con 0x7f5f88159c10 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f877fe700 1 -- 172.21.15.29:0/2933745630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 msgr2=0x7f5f881596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f877fe700 1 --2- 172.21.15.29:0/2933745630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f881596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f877fe700 1 -- 172.21.15.29:0/2933745630 >> v1:172.21.15.29:6789/0 conn(0x7f5f88159c10 legacy=0x7f5f880adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.187+0000 7f5f877fe700 1 -- 172.21.15.29:0/2933745630 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f880c8f50 con 0x7f5f881559f0 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.188+0000 7f5f877fe700 1 --2- 172.21.15.29:0/2933745630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880b0120 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009910 tx=0x7f5f780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc385baddf6cbfe server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.188+0000 7f5f867fc700 1 -- 172.21.15.29:0/2933745630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78011630 con 0x7f5f881559f0 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.188+0000 7f5f867fc700 1 -- 172.21.15.29:0/2933745630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f78011790 con 0x7f5f881559f0 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.188+0000 7f5f867fc700 1 -- 172.21.15.29:0/2933745630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78011b30 con 0x7f5f881559f0 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 -- 172.21.15.29:0/2933745630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 msgr2=0x7f5f880b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/2933745630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880b0120 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009910 tx=0x7f5f780091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 -- 172.21.15.29:0/2933745630 shutdown_connections 2023-10-07T09:47:05.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/2933745630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880b0120 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:05.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/2933745630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f881596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:05.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 -- 172.21.15.29:0/2933745630 >> 172.21.15.29:0/2933745630 conn(0x7f5f88067610 msgr2=0x7f5f88067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:05.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 -- 172.21.15.29:0/2933745630 shutdown_connections 2023-10-07T09:47:05.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.189+0000 7f5f8dfff700 1 -- 172.21.15.29:0/2933745630 wait complete. 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.190+0000 7f5f8dfff700 1 Processor -- start 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.190+0000 7f5f8dfff700 1 -- start start 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.191+0000 7f5f8dfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f88150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.191+0000 7f5f8dfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.191+0000 7f5f8dfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f88159c10 0x7f5f88169670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.191+0000 7f5f8dfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88151230 con 0x7f5f881559f0 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.191+0000 7f5f8dfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88151460 con 0x7f5f88159c10 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.191+0000 7f5f8dfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88151740 con 0x7f5f880b0660 2023-10-07T09:47:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880af7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53324/0 (socket says 172.21.15.29:53324) 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 -- 172.21.15.29:0/893237728 learned_addr learned my addr 172.21.15.29:0/893237728 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f87fff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f88159c10 0x7f5f88169670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f877fe700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f88150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 -- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 msgr2=0x7f5f88150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f88150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 -- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f88159c10 msgr2=0x7f5f88169670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f88159c10 0x7f5f88169670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 -- 172.21.15.29:0/893237728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f78009420 con 0x7f5f881559f0 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f86ffd700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880af7d0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00ec80 tx=0x7f5f7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.192+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00c9d0 con 0x7f5f881559f0 2023-10-07T09:47:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.193+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c00cb30 con 0x7f5f881559f0 2023-10-07T09:47:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.193+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c013470 con 0x7f5f881559f0 2023-10-07T09:47:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.193+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f88151a20 con 0x7f5f881559f0 2023-10-07T09:47:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.193+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f880c74c0 con 0x7f5f881559f0 2023-10-07T09:47:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.194+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f880c7aa0 con 0x7f5f881559f0 2023-10-07T09:47:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.195+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 47) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f5f7c013820 con 0x7f5f881559f0 2023-10-07T09:47:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.196+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6827+0+0 (secure 0 0 0) 0x7f5f7c084910 con 0x7f5f881559f0 2023-10-07T09:47:05.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.200+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f880c7aa0 con 0x7f5f881559f0 2023-10-07T09:47:05.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.406+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5f880c7aa0 con 0x7f5f881559f0 2023-10-07T09:47:05.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.456+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 48) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f5f7c0514a0 con 0x7f5f881559f0 2023-10-07T09:47:05.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.457+0000 7f5f6f7fe700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f74034340 0x7f5f7406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:05.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.458+0000 7f5f877fe700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f74034340 0x7f5f7406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:05.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:05.459+0000 7f5f877fe700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f74034340 0x7f5f7406eb30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009860 tx=0x7f5f78017000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.316+0000 7f5f6f7fe700 1 -- 172.21.15.29:0/893237728 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5f7c06ff40 con 0x7f5f881559f0 2023-10-07T09:47:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.321+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f74034340 msgr2=0x7f5f7406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:06.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.321+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f74034340 0x7f5f7406eb30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009860 tx=0x7f5f78017000 comp rx=0 tx=0).stop 2023-10-07T09:47:06.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.321+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 msgr2=0x7f5f880af7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:06.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.321+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880af7d0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00ec80 tx=0x7f5f7c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 shutdown_connections 2023-10-07T09:47:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5f74034340 0x7f5f7406eb30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f88159c10 0x7f5f88169670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f880b0660 0x7f5f88150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 --2- 172.21.15.29:0/893237728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f881559f0 0x7f5f880af7d0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 >> 172.21.15.29:0/893237728 conn(0x7f5f88067610 msgr2=0x7f5f88159350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.322+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 shutdown_connections 2023-10-07T09:47:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.323+0000 7f5f8dfff700 1 -- 172.21.15.29:0/893237728 wait complete. 2023-10-07T09:47:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:47:06.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:47:06.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.551+0000 7ff3579ff700 1 Processor -- start 2023-10-07T09:47:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.551+0000 7ff3579ff700 1 -- start start 2023-10-07T09:47:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff3579ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff3500afeb0 0x7ff3500cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff3579ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500accf0 0x7ff3500aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff3579ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3500ad160 con 0x7ff3500af4c0 2023-10-07T09:47:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff3579ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35015a140 con 0x7ff3500afeb0 2023-10-07T09:47:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff3579ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500adca0 con 0x7ff3500accf0 2023-10-07T09:47:06.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff35579b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff3500afeb0 0x7ff3500cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff355f9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3500af4c0 0x7ff35015a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48820/0 (socket says 172.21.15.29:48820) 2023-10-07T09:47:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff355f9c700 1 -- 172.21.15.29:0/2299381102 learned_addr learned my addr 172.21.15.29:0/2299381102 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.553+0000 7ff354f9a700 1 --2- 172.21.15.29:0/2299381102 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500accf0 0x7ff3500aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:06.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff35579b700 1 -- 172.21.15.29:0/2299381102 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500accf0 msgr2=0x7ff3500aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:06.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff35579b700 1 --2- 172.21.15.29:0/2299381102 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500accf0 0x7ff3500aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff35579b700 1 -- 172.21.15.29:0/2299381102 >> v1:172.21.15.29:6789/0 conn(0x7ff3500af4c0 legacy=0x7ff35015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff35579b700 1 -- 172.21.15.29:0/2299381102 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500ad400 con 0x7ff3500afeb0 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff35579b700 1 --2- 172.21.15.29:0/2299381102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500afeb0 0x7ff3500cc810 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0098e0 tx=0x7ff34c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b35928b677b750a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff347fff700 1 -- 172.21.15.29:0/2299381102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00b640 con 0x7ff3500afeb0 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff347fff700 1 -- 172.21.15.29:0/2299381102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c00b7a0 con 0x7ff3500afeb0 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.554+0000 7ff347fff700 1 -- 172.21.15.29:0/2299381102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00bb40 con 0x7ff3500afeb0 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 -- 172.21.15.29:0/2299381102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500afeb0 msgr2=0x7ff3500cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 --2- 172.21.15.29:0/2299381102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500afeb0 0x7ff3500cc810 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0098e0 tx=0x7ff34c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 -- 172.21.15.29:0/2299381102 shutdown_connections 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 --2- 172.21.15.29:0/2299381102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500afeb0 0x7ff3500cc810 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 --2- 172.21.15.29:0/2299381102 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500accf0 0x7ff3500aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 -- 172.21.15.29:0/2299381102 >> 172.21.15.29:0/2299381102 conn(0x7ff350067610 msgr2=0x7ff350067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 -- 172.21.15.29:0/2299381102 shutdown_connections 2023-10-07T09:47:06.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.555+0000 7ff3579ff700 1 -- 172.21.15.29:0/2299381102 wait complete. 2023-10-07T09:47:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.556+0000 7ff3579ff700 1 Processor -- start 2023-10-07T09:47:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.556+0000 7ff3579ff700 1 -- start start 2023-10-07T09:47:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.557+0000 7ff3579ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 0x7ff3500cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.557+0000 7ff3579ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3500af4c0 0x7ff350148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.557+0000 7ff3579ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500afeb0 0x7ff35015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.557+0000 7ff3579ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35015b050 con 0x7ff3500accf0 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.557+0000 7ff3579ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35015b610 con 0x7ff3500af4c0 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.557+0000 7ff3579ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500cc5d0 con 0x7ff3500afeb0 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 0x7ff3500cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 0x7ff3500cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53350/0 (socket says 172.21.15.29:53350) 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 -- 172.21.15.29:0/4063572996 learned_addr learned my addr 172.21.15.29:0/4063572996 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff354f9a700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3500af4c0 0x7ff350148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff355f9c700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500afeb0 0x7ff35015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 -- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500afeb0 msgr2=0x7ff35015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500afeb0 0x7ff35015baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 -- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3500af4c0 msgr2=0x7ff350148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3500af4c0 0x7ff350148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:06.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.558+0000 7ff35579b700 1 -- 172.21.15.29:0/4063572996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c009420 con 0x7ff3500accf0 2023-10-07T09:47:06.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.559+0000 7ff35579b700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 0x7ff3500cbd80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009f80 tx=0x7ff34c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:06.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.559+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c01bd90 con 0x7ff3500accf0 2023-10-07T09:47:06.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.559+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c011420 con 0x7ff3500accf0 2023-10-07T09:47:06.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.559+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c0117e0 con 0x7ff3500accf0 2023-10-07T09:47:06.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.559+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3500cc8b0 con 0x7ff3500accf0 2023-10-07T09:47:06.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.559+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3500cc2c0 con 0x7ff3500accf0 2023-10-07T09:47:06.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.560+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3500718b0 con 0x7ff3500accf0 2023-10-07T09:47:06.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.561+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 48) v1 ==== 84578+0+0 (secure 0 0 0) 0x7ff34c011b50 con 0x7ff3500accf0 2023-10-07T09:47:06.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.562+0000 7ff3457fa700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff334066d80 0x7ff334069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:06.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.562+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff34c085c80 con 0x7ff3500accf0 2023-10-07T09:47:06.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.562+0000 7ff354f9a700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff334066d80 0x7ff334069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:06.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.563+0000 7ff354f9a700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff334066d80 0x7ff334069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff3500b0f50 tx=0x7ff340008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:06.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.566+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff34c052e20 con 0x7ff3500accf0 2023-10-07T09:47:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:06.768+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3500b1730 con 0x7ff3500accf0 2023-10-07T09:47:07.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.327+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff34c0531f0 con 0x7ff3500accf0 2023-10-07T09:47:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.462+0000 7ff3457fa700 1 -- 172.21.15.29:0/4063572996 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff34c0568b0 con 0x7ff3500accf0 2023-10-07T09:47:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff334066d80 msgr2=0x7ff334069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff334066d80 0x7ff334069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff3500b0f50 tx=0x7ff340008040 comp rx=0 tx=0).stop 2023-10-07T09:47:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 msgr2=0x7ff3500cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 0x7ff3500cbd80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009f80 tx=0x7ff34c017000 comp rx=0 tx=0).stop 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 shutdown_connections 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff334066d80 0x7ff334069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3500afeb0 0x7ff35015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3500accf0 0x7ff3500cbd80 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 --2- 172.21.15.29:0/4063572996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3500af4c0 0x7ff350148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.466+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 >> 172.21.15.29:0/4063572996 conn(0x7ff350067610 msgr2=0x7ff35006d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.467+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 shutdown_connections 2023-10-07T09:47:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.467+0000 7ff3579ff700 1 -- 172.21.15.29:0/4063572996 wait complete. 2023-10-07T09:47:07.471 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:47:07.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T09:47:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.692+0000 7ffb1abff700 1 Processor -- start 2023-10-07T09:47:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.693+0000 7ffb1abff700 1 -- start start 2023-10-07T09:47:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb1abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb14159750 0x7ffb140d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb1abff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffb140ce8a0 0x7ffb140c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb1abff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140cbf20 con 0x7ffb14159b20 2023-10-07T09:47:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb1abff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb140cc200 con 0x7ffb140ce8a0 2023-10-07T09:47:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb1abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140cc4e0 con 0x7ffb14159750 2023-10-07T09:47:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb13fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffb14159b20 0x7ffb140ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48824/0 (socket says 172.21.15.29:48824) 2023-10-07T09:47:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb13fff700 1 -- 172.21.15.29:0/3497474433 learned_addr learned my addr 172.21.15.29:0/3497474433 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.694+0000 7ffb1919c700 1 --2- 172.21.15.29:0/3497474433 >> v2:172.21.15.29:3300/0 conn(0x7ffb140ce8a0 0x7ffb140c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb1899b700 1 --2- 172.21.15.29:0/3497474433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb14159750 0x7ffb140d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb137fe700 1 -- 172.21.15.29:0/3497474433 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757646620 0 0) 0x7ffb140cbf20 con 0x7ffb14159b20 2023-10-07T09:47:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb137fe700 1 -- 172.21.15.29:0/3497474433 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003570 con 0x7ffb14159b20 2023-10-07T09:47:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb1919c700 1 -- 172.21.15.29:0/3497474433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb14159750 msgr2=0x7ffb140d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb1919c700 1 --2- 172.21.15.29:0/3497474433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb14159750 0x7ffb140d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb1919c700 1 -- 172.21.15.29:0/3497474433 >> v1:172.21.15.29:6789/0 conn(0x7ffb14159b20 legacy=0x7ffb140ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb1919c700 1 -- 172.21.15.29:0/3497474433 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140cc7c0 con 0x7ffb140ce8a0 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb1919c700 1 --2- 172.21.15.29:0/3497474433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb140ce8a0 0x7ffb140c5460 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009e60 tx=0x7ffb0c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd94215a19e6c9e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb137fe700 1 -- 172.21.15.29:0/3497474433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c00e070 con 0x7ffb140ce8a0 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb137fe700 1 -- 172.21.15.29:0/3497474433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c00bab0 con 0x7ffb140ce8a0 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.695+0000 7ffb137fe700 1 -- 172.21.15.29:0/3497474433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c00be30 con 0x7ffb140ce8a0 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 -- 172.21.15.29:0/3497474433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb140ce8a0 msgr2=0x7ffb140c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 --2- 172.21.15.29:0/3497474433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb140ce8a0 0x7ffb140c5460 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009e60 tx=0x7ffb0c009780 comp rx=0 tx=0).stop 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 -- 172.21.15.29:0/3497474433 shutdown_connections 2023-10-07T09:47:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 --2- 172.21.15.29:0/3497474433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb140ce8a0 0x7ffb140c5460 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 --2- 172.21.15.29:0/3497474433 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb14159750 0x7ffb140d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 -- 172.21.15.29:0/3497474433 >> 172.21.15.29:0/3497474433 conn(0x7ffb14063a10 msgr2=0x7ffb140d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 -- 172.21.15.29:0/3497474433 shutdown_connections 2023-10-07T09:47:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.696+0000 7ffb1abff700 1 -- 172.21.15.29:0/3497474433 wait complete. 2023-10-07T09:47:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.698+0000 7ffb1abff700 1 Processor -- start 2023-10-07T09:47:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.699+0000 7ffb1abff700 1 -- start start 2023-10-07T09:47:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb140ce8a0 0x7ffb1414e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb14159750 0x7ffb1414ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 0x7ffb140b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1414ecf0 con 0x7ffb14159b20 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14157240 con 0x7ffb14159750 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1415ae20 con 0x7ffb140ce8a0 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 0x7ffb140b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 0x7ffb140b5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53380/0 (socket says 172.21.15.29:53380) 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1919c700 1 -- 172.21.15.29:0/1559440349 learned_addr learned my addr 172.21.15.29:0/1559440349 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb1899b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb140ce8a0 0x7ffb1414e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.700+0000 7ffb13fff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb14159750 0x7ffb1414ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1919c700 1 -- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb140ce8a0 msgr2=0x7ffb1414e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1919c700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb140ce8a0 0x7ffb1414e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1919c700 1 -- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb14159750 msgr2=0x7ffb1414ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1919c700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb14159750 0x7ffb1414ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1919c700 1 -- 172.21.15.29:0/1559440349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c009a20 con 0x7ffb14159b20 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1919c700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 0x7ffb140b5950 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ffb14088a40 tx=0x7ffb0c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c012e10 con 0x7ffb14159b20 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c027470 con 0x7ffb14159b20 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c027830 con 0x7ffb14159b20 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1415b100 con 0x7ffb14159b20 2023-10-07T09:47:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.701+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1415b6c0 con 0x7ffb14159b20 2023-10-07T09:47:07.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.702+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb141573f0 con 0x7ffb14159b20 2023-10-07T09:47:07.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.703+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffb0c024070 con 0x7ffb14159b20 2023-10-07T09:47:07.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.704+0000 7ffb10ff9700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffaf8066f30 0x7ffaf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:07.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.704+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7ffb0c086bf0 con 0x7ffb14159b20 2023-10-07T09:47:07.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.705+0000 7ffb1899b700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffaf8066f30 0x7ffaf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:07.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.705+0000 7ffb1899b700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffaf8066f30 0x7ffaf80693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009da0 tx=0x7ffb08009120 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:07.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.709+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb141573f0 con 0x7ffb14159b20 2023-10-07T09:47:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.912+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7ffb140c7490 con 0x7ffb14159b20 2023-10-07T09:47:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.916+0000 7ffb10ff9700 1 -- 172.21.15.29:0/1559440349 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7ffb0c02c0b0 con 0x7ffb14159b20 2023-10-07T09:47:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffaf8066f30 msgr2=0x7ffaf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffaf8066f30 0x7ffaf80693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009da0 tx=0x7ffb08009120 comp rx=0 tx=0).stop 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 msgr2=0x7ffb140b5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 0x7ffb140b5950 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ffb14088a40 tx=0x7ffb0c019000 comp rx=0 tx=0).stop 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 shutdown_connections 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ffaf8066f30 0x7ffaf80693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb14159b20 0x7ffb140b5950 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb140ce8a0 0x7ffb1414e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 --2- 172.21.15.29:0/1559440349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb14159750 0x7ffb1414ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 >> 172.21.15.29:0/1559440349 conn(0x7ffb14063a10 msgr2=0x7ffb141502b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 shutdown_connections 2023-10-07T09:47:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:07.919+0000 7ffb1abff700 1 -- 172.21.15.29:0/1559440349 wait complete. 2023-10-07T09:47:07.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T09:47:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.139+0000 7fa7135ff700 1 Processor -- start 2023-10-07T09:47:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.139+0000 7fa7135ff700 1 -- start start 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.140+0000 7fa7135ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa70c153cc0 0x7fa70c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.140+0000 7fa7135ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa70c15ade0 0x7fa70c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.140+0000 7fa7135ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c154090 con 0x7fa70c0b0000 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.140+0000 7fa7135ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70c159f60 con 0x7fa70c153cc0 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.140+0000 7fa7135ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70c15a240 con 0x7fa70c15ade0 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa71139b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa70c153cc0 0x7fa70c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa711b9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa70c0b0000 0x7fa70c15c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48836/0 (socket says 172.21.15.29:48836) 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa711b9c700 1 -- 172.21.15.29:0/2945029420 learned_addr learned my addr 172.21.15.29:0/2945029420 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa710b9a700 1 --2- 172.21.15.29:0/2945029420 >> v2:172.21.15.144:3300/0 conn(0x7fa70c15ade0 0x7fa70c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa71139b700 1 -- 172.21.15.29:0/2945029420 >> v2:172.21.15.144:3300/0 conn(0x7fa70c15ade0 msgr2=0x7fa70c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa71139b700 1 --2- 172.21.15.29:0/2945029420 >> v2:172.21.15.144:3300/0 conn(0x7fa70c15ade0 0x7fa70c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa703fff700 1 -- 172.21.15.29:0/2945029420 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023494323 0 0) 0x7fa70c154090 con 0x7fa70c0b0000 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa71139b700 1 -- 172.21.15.29:0/2945029420 >> v1:172.21.15.29:6789/0 conn(0x7fa70c0b0000 legacy=0x7fa70c15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa71139b700 1 -- 172.21.15.29:0/2945029420 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c15a520 con 0x7fa70c153cc0 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.141+0000 7fa71139b700 1 --2- 172.21.15.29:0/2945029420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c0b17c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0add60 tx=0x7fa708009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb6235820bcf099 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa703fff700 1 -- 172.21.15.29:0/2945029420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708005c10 con 0x7fa70c153cc0 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa703fff700 1 -- 172.21.15.29:0/2945029420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa708005d70 con 0x7fa70c153cc0 2023-10-07T09:47:08.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa703fff700 1 -- 172.21.15.29:0/2945029420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708006260 con 0x7fa70c153cc0 2023-10-07T09:47:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 -- 172.21.15.29:0/2945029420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 msgr2=0x7fa70c0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 --2- 172.21.15.29:0/2945029420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c0b17c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0add60 tx=0x7fa708009120 comp rx=0 tx=0).stop 2023-10-07T09:47:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 -- 172.21.15.29:0/2945029420 shutdown_connections 2023-10-07T09:47:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 --2- 172.21.15.29:0/2945029420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c0b17c0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 --2- 172.21.15.29:0/2945029420 >> v2:172.21.15.144:3300/0 conn(0x7fa70c15ade0 0x7fa70c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 -- 172.21.15.29:0/2945029420 >> 172.21.15.29:0/2945029420 conn(0x7fa70c067610 msgr2=0x7fa70c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 -- 172.21.15.29:0/2945029420 shutdown_connections 2023-10-07T09:47:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.142+0000 7fa7135ff700 1 -- 172.21.15.29:0/2945029420 wait complete. 2023-10-07T09:47:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.144+0000 7fa7135ff700 1 Processor -- start 2023-10-07T09:47:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.144+0000 7fa7135ff700 1 -- start start 2023-10-07T09:47:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa7135ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa70c0b0000 0x7fa70c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa7135ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa7135ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70c15ade0 0x7fa70c15e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa7135ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c16a540 con 0x7fa70c153cc0 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa7135ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c1560e0 con 0x7fa70c0b0000 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa7135ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c1563c0 con 0x7fa70c15ade0 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa710b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa710b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c15e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53390/0 (socket says 172.21.15.29:53390) 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa710b9a700 1 -- 172.21.15.29:0/3089954091 learned_addr learned my addr 172.21.15.29:0/3089954091 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa711b9c700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70c15ade0 0x7fa70c15e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.145+0000 7fa71139b700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa70c0b0000 0x7fa70c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa710b9a700 1 -- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70c15ade0 msgr2=0x7fa70c15e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa710b9a700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70c15ade0 0x7fa70c15e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa710b9a700 1 -- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa70c0b0000 msgr2=0x7fa70c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa710b9a700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa70c0b0000 0x7fa70c155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa710b9a700 1 -- 172.21.15.29:0/3089954091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7080093c0 con 0x7fa70c153cc0 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa71139b700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa70c0b0000 0x7fa70c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa711b9c700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70c15ade0 0x7fa70c15e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa710b9a700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c15e3c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00d890 tx=0x7fa6fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc012630 con 0x7fa70c153cc0 2023-10-07T09:47:08.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc012790 con 0x7fa70c153cc0 2023-10-07T09:47:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc012b30 con 0x7fa70c153cc0 2023-10-07T09:47:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c1566a0 con 0x7fa70c153cc0 2023-10-07T09:47:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.146+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c169550 con 0x7fa70c153cc0 2023-10-07T09:47:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.147+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c16a6f0 con 0x7fa70c153cc0 2023-10-07T09:47:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.148+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa6fc01e460 con 0x7fa70c153cc0 2023-10-07T09:47:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.149+0000 7fa7017fa700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa6f0066f30 0x7fa6f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.149+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7fa6fc084cf0 con 0x7fa70c153cc0 2023-10-07T09:47:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.150+0000 7fa71139b700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa6f0066f30 0x7fa6f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.150+0000 7fa71139b700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa6f0066f30 0x7fa6f00693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa708009340 tx=0x7fa708016000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.154+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa70c16a6f0 con 0x7fa70c153cc0 2023-10-07T09:47:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.358+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa70c04c3c0 con 0x7fa70c153cc0 2023-10-07T09:47:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.362+0000 7fa7017fa700 1 -- 172.21.15.29:0/3089954091 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6fc051df0 con 0x7fa70c153cc0 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa6f0066f30 msgr2=0x7fa6f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa6f0066f30 0x7fa6f00693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa708009340 tx=0x7fa708016000 comp rx=0 tx=0).stop 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 msgr2=0x7fa70c15e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c15e3c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00d890 tx=0x7fa6fc008040 comp rx=0 tx=0).stop 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 shutdown_connections 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa6f0066f30 0x7fa6f00693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa70c15ade0 0x7fa70c15e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa70c0b0000 0x7fa70c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 --2- 172.21.15.29:0/3089954091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa70c153cc0 0x7fa70c15e3c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 >> 172.21.15.29:0/3089954091 conn(0x7fa70c067610 msgr2=0x7fa70c0b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 shutdown_connections 2023-10-07T09:47:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.364+0000 7fa7135ff700 1 -- 172.21.15.29:0/3089954091 wait complete. 2023-10-07T09:47:08.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T09:47:08.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.581+0000 7fea5c3ff700 1 Processor -- start 2023-10-07T09:47:08.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.581+0000 7fea5c3ff700 1 -- start start 2023-10-07T09:47:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5c3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fea54159b20 0x7fea540d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5c3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 0x7fea540c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5c3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea540cbf20 con 0x7fea54159750 2023-10-07T09:47:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5c3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea540cc200 con 0x7fea54159b20 2023-10-07T09:47:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5c3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea540cc4e0 con 0x7fea540ce8a0 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5a19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fea54159750 0x7fea540ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48838/0 (socket says 172.21.15.29:48838) 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.582+0000 7fea5a19b700 1 -- 172.21.15.29:0/2096419364 learned_addr learned my addr 172.21.15.29:0/2096419364 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea5999a700 1 --2- 172.21.15.29:0/2096419364 >> v2:172.21.15.144:3300/0 conn(0x7fea54159b20 0x7fea540d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea5a99c700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 0x7fea540c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322510243 0 0) 0x7fea540cbf20 con 0x7fea54159750 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea40003570 con 0x7fea54159750 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2775232047 0 0) 0x7fea40003570 con 0x7fea54159750 2023-10-07T09:47:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 msgr2=0x7fea540c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 0x7fea540c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 >> v2:172.21.15.144:3300/0 conn(0x7fea54159b20 msgr2=0x7fea540d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 --2- 172.21.15.29:0/2096419364 >> v2:172.21.15.144:3300/0 conn(0x7fea54159b20 0x7fea540d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea540cc7c0 con 0x7fea54159750 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fea500015f0 con 0x7fea54159750 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 >> v1:172.21.15.29:6789/0 conn(0x7fea54159750 legacy=0x7fea540ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 0x7fea40003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea40003fe0 0x7fea40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea540cc200 con 0x7fea40004d50 2023-10-07T09:47:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea540cc4e0 con 0x7fea40003fe0 2023-10-07T09:47:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4000ba90 con 0x7fea540ce8a0 2023-10-07T09:47:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.583+0000 7fea5999a700 1 --2- 172.21.15.29:0/2096419364 >> v2:172.21.15.144:3300/0 conn(0x7fea54159b20 0x7fea540d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a19b700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 0x7fea40003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5999a700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea40003fe0 0x7fea40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 -- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 msgr2=0x7fea40003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea540ce8a0 0x7fea40003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 -- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea40003fe0 msgr2=0x7fea40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea40003fe0 0x7fea40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 -- 172.21.15.29:0/2096419364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea40003570 con 0x7fea40004d50 2023-10-07T09:47:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.584+0000 7fea5a99c700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea400051c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00e210 tx=0x7fea4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2063f086812f9e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00ea60 con 0x7fea40004d50 2023-10-07T09:47:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c00f040 con 0x7fea40004d50 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea59199700 1 -- 172.21.15.29:0/2096419364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00ee10 con 0x7fea40004d50 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5c3ff700 1 -- 172.21.15.29:0/2096419364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 msgr2=0x7fea400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/2096419364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea400051c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00e210 tx=0x7fea4c00b040 comp rx=0 tx=0).stop 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5a99c700 1 -- 172.21.15.29:0/2096419364 reap_dead start 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5c3ff700 1 -- 172.21.15.29:0/2096419364 shutdown_connections 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5c3ff700 1 -- 172.21.15.29:0/2096419364 >> 172.21.15.29:0/2096419364 conn(0x7fea54063a10 msgr2=0x7fea540d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5c3ff700 1 -- 172.21.15.29:0/2096419364 shutdown_connections 2023-10-07T09:47:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.585+0000 7fea5c3ff700 1 -- 172.21.15.29:0/2096419364 wait complete. 2023-10-07T09:47:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.587+0000 7fea5c3ff700 1 Processor -- start 2023-10-07T09:47:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.587+0000 7fea5c3ff700 1 -- start start 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5c3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea5415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5c3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea5415e560 0x7fea54152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5c3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54169640 0x7fea54169ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5c3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5415f490 con 0x7fea40004d50 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5c3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5415e9d0 con 0x7fea54169640 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5c3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea540b36b0 con 0x7fea5415e560 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea5415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea5415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53406/0 (socket says 172.21.15.29:53406) 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 -- 172.21.15.29:0/3702928982 learned_addr learned my addr 172.21.15.29:0/3702928982 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5999a700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea5415e560 0x7fea54152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 -- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea5415e560 msgr2=0x7fea54152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a99c700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54169640 0x7fea54169ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea5415e560 0x7fea54152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 -- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54169640 msgr2=0x7fea54169ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54169640 0x7fea54169ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a19b700 1 -- 172.21.15.29:0/3702928982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5416ce40 con 0x7fea40004d50 2023-10-07T09:47:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5999a700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea5415e560 0x7fea54152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.588+0000 7fea5a99c700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54169640 0x7fea54169ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.589+0000 7fea5a19b700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea5415fe60 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fea5000e8d0 tx=0x7fea50009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.589+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5000ee70 con 0x7fea40004d50 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.589+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea50010040 con 0x7fea40004d50 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.589+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea54152fc0 con 0x7fea40004d50 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.589+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50007530 con 0x7fea40004d50 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.589+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5415eed0 con 0x7fea40004d50 2023-10-07T09:47:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.590+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5404c3c0 con 0x7fea40004d50 2023-10-07T09:47:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.591+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fea50005a00 con 0x7fea40004d50 2023-10-07T09:47:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.592+0000 7fea4a7fc700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea38066f30 0x7fea380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.592+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7fea50084230 con 0x7fea40004d50 2023-10-07T09:47:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.592+0000 7fea5999a700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea38066f30 0x7fea380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.593+0000 7fea5999a700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea38066f30 0x7fea380693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea440093f0 tx=0x7fea44008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.596+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea50051330 con 0x7fea40004d50 2023-10-07T09:47:08.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.803+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fea54064040 con 0x7fea40004d50 2023-10-07T09:47:08.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.807+0000 7fea4a7fc700 1 -- 172.21.15.29:0/3702928982 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7fea50054950 con 0x7fea40004d50 2023-10-07T09:47:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.809+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea38066f30 msgr2=0x7fea380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.809+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea38066f30 0x7fea380693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea440093f0 tx=0x7fea44008040 comp rx=0 tx=0).stop 2023-10-07T09:47:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.809+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 msgr2=0x7fea5415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.809+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea5415fe60 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fea5000e8d0 tx=0x7fea50009040 comp rx=0 tx=0).stop 2023-10-07T09:47:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 shutdown_connections 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea38066f30 0x7fea380693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54169640 0x7fea54169ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40004d50 0x7fea5415fe60 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 --2- 172.21.15.29:0/3702928982 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea5415e560 0x7fea54152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 >> 172.21.15.29:0/3702928982 conn(0x7fea54063a10 msgr2=0x7fea540c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 shutdown_connections 2023-10-07T09:47:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:08.810+0000 7fea5c3ff700 1 -- 172.21.15.29:0/3702928982 wait complete. 2023-10-07T09:47:08.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-07T09:47:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.026+0000 7fb7853ff700 1 Processor -- start 2023-10-07T09:47:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.026+0000 7fb7853ff700 1 -- start start 2023-10-07T09:47:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb7853ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 0x7fb7800d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb7853ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7800ce8a0 0x7fb7800c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb7853ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7800cbf20 con 0x7fb780159b20 2023-10-07T09:47:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb7853ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7800cc200 con 0x7fb7800ce8a0 2023-10-07T09:47:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb7853ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800cc4e0 con 0x7fb780159750 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb77e7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb780159b20 0x7fb7800ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48854/0 (socket says 172.21.15.29:48854) 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.027+0000 7fb77e7fc700 1 -- 172.21.15.29:0/396358046 learned_addr learned my addr 172.21.15.29:0/396358046 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77effd700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 0x7fb7800d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77f7fe700 1 --2- 172.21.15.29:0/396358046 >> v2:172.21.15.144:3300/0 conn(0x7fb7800ce8a0 0x7fb7800c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709434790 0 0) 0x7fb7800cbf20 con 0x7fb780159b20 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb768003570 con 0x7fb780159b20 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3086642693 0 0) 0x7fb768003570 con 0x7fb780159b20 2023-10-07T09:47:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 msgr2=0x7fb7800d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 0x7fb7800d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 >> v2:172.21.15.144:3300/0 conn(0x7fb7800ce8a0 msgr2=0x7fb7800c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 --2- 172.21.15.29:0/396358046 >> v2:172.21.15.144:3300/0 conn(0x7fb7800ce8a0 0x7fb7800c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800cc7c0 con 0x7fb780159b20 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fb774001470 con 0x7fb780159b20 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 >> v1:172.21.15.29:6789/0 conn(0x7fb780159b20 legacy=0x7fb7800ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 0x7fb768003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.028+0000 7fb77dffb700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb768003fe0 0x7fb768004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77dffb700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb768004d50 0x7fb7680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800cc200 con 0x7fb768003fe0 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800cc4e0 con 0x7fb768004d50 2023-10-07T09:47:09.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76800ba90 con 0x7fb780159750 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb768003fe0 0x7fb768004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77effd700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 0x7fb768003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77f7fe700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb768004d50 0x7fb7680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 -- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 msgr2=0x7fb768003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb780159750 0x7fb768003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 -- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb768004d50 msgr2=0x7fb7680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb768004d50 0x7fb7680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 -- 172.21.15.29:0/396358046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb768003570 con 0x7fb768003fe0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.029+0000 7fb77e7fc700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb768003fe0 0x7fb768004810 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb77400c3c0 tx=0x7fb774001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de811fa2e48702d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77400cc30 con 0x7fb768003fe0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77400cdd0 con 0x7fb768003fe0 2023-10-07T09:47:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb77dffb700 1 -- 172.21.15.29:0/396358046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774005b70 con 0x7fb768003fe0 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb7853ff700 1 -- 172.21.15.29:0/396358046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb768003fe0 msgr2=0x7fb768004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb7853ff700 1 --2- 172.21.15.29:0/396358046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb768003fe0 0x7fb768004810 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb77400c3c0 tx=0x7fb774001e90 comp rx=0 tx=0).stop 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb77f7fe700 1 -- 172.21.15.29:0/396358046 reap_dead start 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb7853ff700 1 -- 172.21.15.29:0/396358046 shutdown_connections 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb7853ff700 1 -- 172.21.15.29:0/396358046 >> 172.21.15.29:0/396358046 conn(0x7fb780063a10 msgr2=0x7fb7800d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb7853ff700 1 -- 172.21.15.29:0/396358046 shutdown_connections 2023-10-07T09:47:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.030+0000 7fb7853ff700 1 -- 172.21.15.29:0/396358046 wait complete. 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.032+0000 7fb7853ff700 1 Processor -- start 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.032+0000 7fb7853ff700 1 -- start start 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb7853ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb768003fe0 0x7fb7800c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb7853ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb780155b00 0x7fb780155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb7853ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 0x7fb7800b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb7853ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7801564b0 con 0x7fb7800b4ef0 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb7853ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800b5c30 con 0x7fb780155b00 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb7853ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800c7670 con 0x7fb768003fe0 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb77f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 0x7fb7800b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb77f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 0x7fb7800b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53420/0 (socket says 172.21.15.29:53420) 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb77f7fe700 1 -- 172.21.15.29:0/3213431364 learned_addr learned my addr 172.21.15.29:0/3213431364 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb77e7fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb780155b00 0x7fb780155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.033+0000 7fb77effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb768003fe0 0x7fb7800c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb77f7fe700 1 -- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb768003fe0 msgr2=0x7fb7800c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb77f7fe700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb768003fe0 0x7fb7800c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb77f7fe700 1 -- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb780155b00 msgr2=0x7fb780155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb77f7fe700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb780155b00 0x7fb780155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb77f7fe700 1 -- 172.21.15.29:0/3213431364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800c7950 con 0x7fb7800b4ef0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb77f7fe700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 0x7fb7800b5370 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb78015e510 tx=0x7fb77800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77800e040 con 0x7fb7800b4ef0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77800f040 con 0x7fb7800b4ef0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77800fd70 con 0x7fb7800b4ef0 2023-10-07T09:47:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7800c7c30 con 0x7fb7800b4ef0 2023-10-07T09:47:09.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.034+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7800b58b0 con 0x7fb7800b4ef0 2023-10-07T09:47:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.035+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78004c3c0 con 0x7fb7800b4ef0 2023-10-07T09:47:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.036+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb778015640 con 0x7fb7800b4ef0 2023-10-07T09:47:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.037+0000 7fb766ffd700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb76c066f30 0x7fb76c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.037+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7fb778084e10 con 0x7fb7800b4ef0 2023-10-07T09:47:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.038+0000 7fb77effd700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb76c066f30 0x7fb76c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.039+0000 7fb77effd700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb76c066f30 0x7fb76c0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb770006ea0 tx=0x7fb770009120 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:09.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.042+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb778089070 con 0x7fb7800b4ef0 2023-10-07T09:47:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:09.326+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fb7800caa80 con 0x7fb7800b4ef0 2023-10-07T09:47:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.329+0000 7fb766ffd700 1 -- 172.21.15.29:0/3213431364 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v672) v1 ==== 183+0+0 (secure 0 0 0) 0x7fb778051f10 con 0x7fb7800b4ef0 2023-10-07T09:47:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb76c066f30 msgr2=0x7fb76c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb76c066f30 0x7fb76c0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb770006ea0 tx=0x7fb770009120 comp rx=0 tx=0).stop 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 msgr2=0x7fb7800b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 0x7fb7800b5370 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb78015e510 tx=0x7fb77800b040 comp rx=0 tx=0).stop 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 shutdown_connections 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb76c066f30 0x7fb76c0693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7800b4ef0 0x7fb7800b5370 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb768003fe0 0x7fb7800c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 --2- 172.21.15.29:0/3213431364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb780155b00 0x7fb780155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 >> 172.21.15.29:0/3213431364 conn(0x7fb780063a10 msgr2=0x7fb780069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 shutdown_connections 2023-10-07T09:47:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.332+0000 7fb7853ff700 1 -- 172.21.15.29:0/3213431364 wait complete. 2023-10-07T09:47:10.355 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-07T09:47:10.355 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-07T09:47:10.356 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:47:10.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-07T09:47:10.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.557+0000 7f7b849ff700 1 Processor -- start 2023-10-07T09:47:10.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.557+0000 7f7b849ff700 1 -- start start 2023-10-07T09:47:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.558+0000 7f7b849ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b800ad370 0x7f7b800cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.558+0000 7f7b849ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b80159ec0 0x7f7b8015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.558+0000 7f7b849ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8014d790 con 0x7f7b801568c0 2023-10-07T09:47:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.558+0000 7f7b849ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8015a880 con 0x7f7b80159ec0 2023-10-07T09:47:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.558+0000 7f7b849ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b800ae5d0 con 0x7f7b800ad370 2023-10-07T09:47:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b80159ec0 0x7f7b8015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:10.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b80159ec0 0x7f7b8015a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53424/0 (socket says 172.21.15.29:53424) 2023-10-07T09:47:10.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 -- 172.21.15.29:0/3652679873 learned_addr learned my addr 172.21.15.29:0/3652679873 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:10.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7e59c700 1 --2- 172.21.15.29:0/3652679873 >> v2:172.21.15.144:3300/0 conn(0x7f7b800ad370 0x7f7b800cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:10.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 -- 172.21.15.29:0/3652679873 >> v2:172.21.15.144:3300/0 conn(0x7f7b800ad370 msgr2=0x7f7b800cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:10.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 --2- 172.21.15.29:0/3652679873 >> v2:172.21.15.144:3300/0 conn(0x7f7b800ad370 0x7f7b800cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 -- 172.21.15.29:0/3652679873 >> v1:172.21.15.144:6789/0 conn(0x7f7b801568c0 legacy=0x7f7b80156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.559+0000 7f7b7dd9b700 1 -- 172.21.15.29:0/3652679873 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b800ad740 con 0x7f7b80159ec0 2023-10-07T09:47:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b7dd9b700 1 --2- 172.21.15.29:0/3652679873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80159ec0 0x7f7b8015a340 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7b740099b0 tx=0x7f7b74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f288f9a0b35914cf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b7d59a700 1 -- 172.21.15.29:0/3652679873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74009ea0 con 0x7f7b80159ec0 2023-10-07T09:47:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b7d59a700 1 -- 172.21.15.29:0/3652679873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7400b820 con 0x7f7b80159ec0 2023-10-07T09:47:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b7d59a700 1 -- 172.21.15.29:0/3652679873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7400bbc0 con 0x7f7b80159ec0 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b849ff700 1 -- 172.21.15.29:0/3652679873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80159ec0 msgr2=0x7f7b8015a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b849ff700 1 --2- 172.21.15.29:0/3652679873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80159ec0 0x7f7b8015a340 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7b740099b0 tx=0x7f7b74009250 comp rx=0 tx=0).stop 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b849ff700 1 -- 172.21.15.29:0/3652679873 shutdown_connections 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b849ff700 1 --2- 172.21.15.29:0/3652679873 >> v2:172.21.15.144:3300/0 conn(0x7f7b800ad370 0x7f7b800cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.560+0000 7f7b849ff700 1 --2- 172.21.15.29:0/3652679873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80159ec0 0x7f7b8015a340 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.561+0000 7f7b849ff700 1 -- 172.21.15.29:0/3652679873 >> 172.21.15.29:0/3652679873 conn(0x7f7b800680a0 msgr2=0x7f7b800cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.561+0000 7f7b849ff700 1 -- 172.21.15.29:0/3652679873 shutdown_connections 2023-10-07T09:47:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.561+0000 7f7b849ff700 1 -- 172.21.15.29:0/3652679873 wait complete. 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.562+0000 7f7b849ff700 1 Processor -- start 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.562+0000 7f7b849ff700 1 -- start start 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b849ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 0x7f7b80143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b849ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b801568c0 0x7f7b80146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b849ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b80159ec0 0x7f7b801482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b849ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b80147b90 con 0x7f7b800ad370 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b849ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b800c9a60 con 0x7f7b801568c0 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b849ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b80144500 con 0x7f7b80159ec0 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.563+0000 7f7b7e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 0x7f7b80143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 0x7f7b80143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53438/0 (socket says 172.21.15.29:53438) 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 -- 172.21.15.29:0/480591590 learned_addr learned my addr 172.21.15.29:0/480591590 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7dd9b700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b801568c0 0x7f7b80146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7ed9d700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b80159ec0 0x7f7b801482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 -- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b80159ec0 msgr2=0x7f7b801482a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b80159ec0 0x7f7b801482a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 -- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b801568c0 msgr2=0x7f7b80146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b801568c0 0x7f7b80146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 -- 172.21.15.29:0/480591590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740094f0 con 0x7f7b800ad370 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b7e59c700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 0x7f7b80143fc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000ec80 tx=0x7f7b700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000c850 con 0x7f7b800ad370 2023-10-07T09:47:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7000c9b0 con 0x7f7b800ad370 2023-10-07T09:47:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000cd50 con 0x7f7b800ad370 2023-10-07T09:47:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.564+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b801478b0 con 0x7f7b800ad370 2023-10-07T09:47:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.565+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b800c96f0 con 0x7f7b800ad370 2023-10-07T09:47:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.566+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b800c9d40 con 0x7f7b800ad370 2023-10-07T09:47:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.566+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b70013680 con 0x7f7b800ad370 2023-10-07T09:47:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.567+0000 7f7b6e7fc700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b60066f30 0x7f7b600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.567+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f7b70084960 con 0x7f7b800ad370 2023-10-07T09:47:10.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.568+0000 7f7b7dd9b700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b60066f30 0x7f7b600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:10.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.569+0000 7f7b7dd9b700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b60066f30 0x7f7b600693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7b740099b0 tx=0x7f7b74017000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:10.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.571+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b70051a60 con 0x7f7b800ad370 2023-10-07T09:47:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:10.854+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f7b8004c3c0 con 0x7f7b800ad370 2023-10-07T09:47:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.334+0000 7f7b6e7fc700 1 -- 172.21.15.29:0/480591590 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v673) v1 ==== 151+0+0 (secure 0 0 0) 0x7f7b70055080 con 0x7f7b800ad370 2023-10-07T09:47:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b60066f30 msgr2=0x7f7b600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b60066f30 0x7f7b600693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7b740099b0 tx=0x7f7b74017000 comp rx=0 tx=0).stop 2023-10-07T09:47:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 msgr2=0x7f7b80143fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 0x7f7b80143fc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000ec80 tx=0x7f7b700091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 shutdown_connections 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7b60066f30 0x7f7b600693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b80159ec0 0x7f7b801482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b800ad370 0x7f7b80143fc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 --2- 172.21.15.29:0/480591590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b801568c0 0x7f7b80146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 >> 172.21.15.29:0/480591590 conn(0x7f7b800680a0 msgr2=0x7f7b800b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 shutdown_connections 2023-10-07T09:47:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.337+0000 7f7b849ff700 1 -- 172.21.15.29:0/480591590 wait complete. 2023-10-07T09:47:11.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:47:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.554+0000 7f92c41ff700 1 Processor -- start 2023-10-07T09:47:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.554+0000 7f92c41ff700 1 -- start start 2023-10-07T09:47:11.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.556+0000 7f92c41ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92bc14cd70 0x7f92bc159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:11.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.556+0000 7f92c41ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc157790 con 0x7f92bc153cc0 2023-10-07T09:47:11.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.556+0000 7f92c41ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc157a70 con 0x7f92bc14c8d0 2023-10-07T09:47:11.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.556+0000 7f92c41ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc0b0950 con 0x7f92bc14cd70 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.556+0000 7f92c1f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f92bc153cc0 0x7f92bc0cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48862/0 (socket says 172.21.15.29:48862) 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.556+0000 7f92c1f9b700 1 -- 172.21.15.29:0/3871351260 learned_addr learned my addr 172.21.15.29:0/3871351260 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c279c700 1 --2- 172.21.15.29:0/3871351260 >> v2:172.21.15.29:3300/0 conn(0x7f92bc14cd70 0x7f92bc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c0f99700 1 -- 172.21.15.29:0/3871351260 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415846938 0 0) 0x7f92bc157790 con 0x7f92bc153cc0 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c0f99700 1 -- 172.21.15.29:0/3871351260 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a8003570 con 0x7f92bc153cc0 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c279c700 1 -- 172.21.15.29:0/3871351260 >> v1:172.21.15.144:6789/0 conn(0x7f92bc14c8d0 legacy=0x7f92bc0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c279c700 1 -- 172.21.15.29:0/3871351260 >> v1:172.21.15.29:6789/0 conn(0x7f92bc153cc0 legacy=0x7f92bc0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c279c700 1 -- 172.21.15.29:0/3871351260 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc157d50 con 0x7f92bc14cd70 2023-10-07T09:47:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.557+0000 7f92c279c700 1 --2- 172.21.15.29:0/3871351260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc14cd70 0x7f92bc159680 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009e60 tx=0x7f92b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86210c6daf22d61 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c0f99700 1 -- 172.21.15.29:0/3871351260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400e070 con 0x7f92bc14cd70 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c0f99700 1 -- 172.21.15.29:0/3871351260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400bb60 con 0x7f92bc14cd70 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c0f99700 1 -- 172.21.15.29:0/3871351260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012430 con 0x7f92bc14cd70 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 -- 172.21.15.29:0/3871351260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc14cd70 msgr2=0x7f92bc159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 --2- 172.21.15.29:0/3871351260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc14cd70 0x7f92bc159680 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009e60 tx=0x7f92b4009780 comp rx=0 tx=0).stop 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 -- 172.21.15.29:0/3871351260 shutdown_connections 2023-10-07T09:47:11.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 --2- 172.21.15.29:0/3871351260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc14cd70 0x7f92bc159680 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 -- 172.21.15.29:0/3871351260 >> 172.21.15.29:0/3871351260 conn(0x7f92bc067610 msgr2=0x7f92bc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:11.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 -- 172.21.15.29:0/3871351260 shutdown_connections 2023-10-07T09:47:11.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.558+0000 7f92c41ff700 1 -- 172.21.15.29:0/3871351260 wait complete. 2023-10-07T09:47:11.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.560+0000 7f92c41ff700 1 Processor -- start 2023-10-07T09:47:11.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.560+0000 7f92c41ff700 1 -- start start 2023-10-07T09:47:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c41ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92bc14c8d0 0x7f92bc14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c41ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92bc14cd70 0x7f92bc151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c41ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 0x7f92bc14daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c41ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc14b360 con 0x7f92bc153cc0 2023-10-07T09:47:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c41ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc150860 con 0x7f92bc14c8d0 2023-10-07T09:47:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c41ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc150b40 con 0x7f92bc14cd70 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 0x7f92bc14daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 0x7f92bc14daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53448/0 (socket says 172.21.15.29:53448) 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c279c700 1 -- 172.21.15.29:0/113307037 learned_addr learned my addr 172.21.15.29:0/113307037 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c1f9b700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92bc14c8d0 0x7f92bc14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.561+0000 7f92c179a700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92bc14cd70 0x7f92bc151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c279c700 1 -- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92bc14cd70 msgr2=0x7f92bc151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c279c700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92bc14cd70 0x7f92bc151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c279c700 1 -- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92bc14c8d0 msgr2=0x7f92bc14ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c279c700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92bc14c8d0 0x7f92bc14ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c279c700 1 -- 172.21.15.29:0/113307037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009a20 con 0x7f92bc153cc0 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c179a700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92bc14cd70 0x7f92bc151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c279c700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 0x7f92bc14daf0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0641e0 tx=0x7f92b4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4027470 con 0x7f92bc153cc0 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b40275f0 con 0x7f92bc153cc0 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc150e20 con 0x7f92bc153cc0 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b40279b0 con 0x7f92bc153cc0 2023-10-07T09:47:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.562+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc1513e0 con 0x7f92bc153cc0 2023-10-07T09:47:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.563+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc04c3c0 con 0x7f92bc153cc0 2023-10-07T09:47:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.564+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f92b4024070 con 0x7f92bc153cc0 2023-10-07T09:47:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.565+0000 7f92b1ffb700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a0066f30 0x7f92a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.565+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f92b4086fe0 con 0x7f92bc153cc0 2023-10-07T09:47:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.566+0000 7f92c1f9b700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a0066f30 0x7f92a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:11.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.566+0000 7f92c1f9b700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a0066f30 0x7f92a00693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007fd0 tx=0x7f92b8009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:11.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.569+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92b4054140 con 0x7f92bc153cc0 2023-10-07T09:47:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.772+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f92bc0ccd00 con 0x7f92bc153cc0 2023-10-07T09:47:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.776+0000 7f92b1ffb700 1 -- 172.21.15.29:0/113307037 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f92b4057760 con 0x7f92bc153cc0 2023-10-07T09:47:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a0066f30 msgr2=0x7f92a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a0066f30 0x7f92a00693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007fd0 tx=0x7f92b8009040 comp rx=0 tx=0).stop 2023-10-07T09:47:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 msgr2=0x7f92bc14daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 0x7f92bc14daf0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0641e0 tx=0x7f92b4019000 comp rx=0 tx=0).stop 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 shutdown_connections 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a0066f30 0x7f92a00693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92bc153cc0 0x7f92bc14daf0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92bc14c8d0 0x7f92bc14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 --2- 172.21.15.29:0/113307037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92bc14cd70 0x7f92bc151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 >> 172.21.15.29:0/113307037 conn(0x7f92bc067610 msgr2=0x7f92bc0af210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 shutdown_connections 2023-10-07T09:47:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:11.779+0000 7f92c41ff700 1 -- 172.21.15.29:0/113307037 wait complete. 2023-10-07T09:47:11.797 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:47:11.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:47:11.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:47:12.009 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:47:12.010 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:47:12.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:47:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.205+0000 7f26cf7ff700 1 Processor -- start 2023-10-07T09:47:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.205+0000 7f26cf7ff700 1 -- start start 2023-10-07T09:47:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.206+0000 7f26cf7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26c80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.206+0000 7f26cf7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f26c80b0710 0x7f26c80b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.206+0000 7f26cf7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8154090 con 0x7f26c815ade0 2023-10-07T09:47:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.206+0000 7f26cf7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c8159f60 con 0x7f26c80b0710 2023-10-07T09:47:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26cf7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c815a240 con 0x7f26c8153cc0 2023-10-07T09:47:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26ccd9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f26c815ade0 0x7f26c80b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48866/0 (socket says 172.21.15.29:48866) 2023-10-07T09:47:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26ccd9a700 1 -- 172.21.15.29:0/1208941054 learned_addr learned my addr 172.21.15.29:0/1208941054 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26cdd9c700 1 --2- 172.21.15.29:0/1208941054 >> v2:172.21.15.144:3300/0 conn(0x7f26c80b0710 0x7f26c80b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26cd59b700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26c80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590489042 0 0) 0x7f26c8154090 con 0x7f26c815ade0 2023-10-07T09:47:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b4003570 con 0x7f26c815ade0 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220724057 0 0) 0x7f26b4003570 con 0x7f26c815ade0 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 msgr2=0x7f26c80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26c80b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 >> v2:172.21.15.144:3300/0 conn(0x7f26c80b0710 msgr2=0x7f26c80b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 --2- 172.21.15.29:0/1208941054 >> v2:172.21.15.144:3300/0 conn(0x7f26c80b0710 0x7f26c80b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.207+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c815a520 con 0x7f26c815ade0 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f26b8001470 con 0x7f26c815ade0 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26cd59b700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26c80b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 >> v1:172.21.15.29:6789/0 conn(0x7f26c815ade0 legacy=0x7f26c80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4003a60 0x7f26b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b4004810 0x7f26b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8159f60 con 0x7f26b4004810 2023-10-07T09:47:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c815a240 con 0x7f26b4003a60 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b400ba60 con 0x7f26c8153cc0 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26cdd9c700 1 --2- 172.21.15.29:0/1208941054 >> v2:172.21.15.144:3300/0 conn(0x7f26c80b0710 0x7f26c80b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26ccd9a700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b4004810 0x7f26b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26cd59b700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4003a60 0x7f26b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26cdd9c700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26ccd9a700 1 -- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 msgr2=0x7f26b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26ccd9a700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26c8153cc0 0x7f26b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26ccd9a700 1 -- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4003a60 msgr2=0x7f26b4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26ccd9a700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4003a60 0x7f26b4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.208+0000 7f26ccd9a700 1 -- 172.21.15.29:0/1208941054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c40093c0 con 0x7f26b4004810 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26ccd9a700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b4004810 0x7f26b4004c80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c3c0 tx=0x7f26b8001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de03b37611eaf7fa server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b800cc90 con 0x7f26b4004810 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b800ce30 con 0x7f26b4004810 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26bffff700 1 -- 172.21.15.29:0/1208941054 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8005b70 con 0x7f26b4004810 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1208941054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b4004810 msgr2=0x7f26b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1208941054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b4004810 0x7f26b4004c80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c3c0 tx=0x7f26b8001e90 comp rx=0 tx=0).stop 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.209+0000 7f26cdd9c700 1 -- 172.21.15.29:0/1208941054 reap_dead start 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.210+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1208941054 shutdown_connections 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.210+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1208941054 >> 172.21.15.29:0/1208941054 conn(0x7f26c8067610 msgr2=0x7f26c8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.210+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1208941054 shutdown_connections 2023-10-07T09:47:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.210+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1208941054 wait complete. 2023-10-07T09:47:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.211+0000 7f26cf7ff700 1 Processor -- start 2023-10-07T09:47:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.211+0000 7f26cf7ff700 1 -- start start 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.212+0000 7f26cf7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4004810 0x7f26c8152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.212+0000 7f26cf7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c815b1d0 0x7f26c80c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.212+0000 7f26cf7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 0x7f26c81581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.212+0000 7f26cf7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80c9260 con 0x7f26c80c83d0 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.212+0000 7f26cf7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c815c2c0 con 0x7f26c815b1d0 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.212+0000 7f26cf7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8152640 con 0x7f26b4004810 2023-10-07T09:47:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 0x7f26c81581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cd59b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4004810 0x7f26c8152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 0x7f26c81581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53470/0 (socket says 172.21.15.29:53470) 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 -- 172.21.15.29:0/1445893191 learned_addr learned my addr 172.21.15.29:0/1445893191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26ccd9a700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c815b1d0 0x7f26c80c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 -- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4004810 msgr2=0x7f26c8152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4004810 0x7f26c8152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 -- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c815b1d0 msgr2=0x7f26c80c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c815b1d0 0x7f26c80c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 -- 172.21.15.29:0/1445893191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c815b6b0 con 0x7f26c80c83d0 2023-10-07T09:47:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26ccd9a700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c815b1d0 0x7f26c80c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26cdd9c700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 0x7f26c81581c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f26c80b44b0 tx=0x7f26c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.213+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c000e070 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.214+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c0011d00 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.214+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c815b990 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.214+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c00166c0 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.214+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c815bf50 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.215+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c815d5b0 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.216+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f26c0016ab0 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.216+0000 7f26bd7fa700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f26ac066f30 0x7f26ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.216+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f26c0085320 con 0x7f26c80c83d0 2023-10-07T09:47:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.217+0000 7f26cd59b700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f26ac066f30 0x7f26ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.217+0000 7f26cd59b700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f26ac066f30 0x7f26ac0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f26c4006ea0 tx=0x7f26c4009120 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.221+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26c815d5b0 con 0x7f26c80c83d0 2023-10-07T09:47:12.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.467+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26c815d5b0 con 0x7f26c80c83d0 2023-10-07T09:47:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.468+0000 7f26bd7fa700 1 -- 172.21.15.29:0/1445893191 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f26c815d5b0 con 0x7f26c80c83d0 2023-10-07T09:47:12.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:12.469 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/905161756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":905161756},{"type":"v1","addr":"172.21.15.144:6835","nonce":905161756}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6377,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4081197732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4081197732},{"type":"v1","addr":"172.21.15.144:6837","nonce":4081197732}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6650,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2079178165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2079178165},{"type":"v1","addr":"172.21.15.29:6835","nonce":2079178165}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/579365689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":579365689},{"type":"v1","addr":"172.21.15.29:6833","nonce":579365689}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[]} 2023-10-07T09:47:12.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f26ac066f30 msgr2=0x7f26ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f26ac066f30 0x7f26ac0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f26c4006ea0 tx=0x7f26c4009120 comp rx=0 tx=0).stop 2023-10-07T09:47:12.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 msgr2=0x7f26c81581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 0x7f26c81581c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f26c80b44b0 tx=0x7f26c000b040 comp rx=0 tx=0).stop 2023-10-07T09:47:12.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 shutdown_connections 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f26ac066f30 0x7f26ac0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26c80c83d0 0x7f26c81581c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4004810 0x7f26c8152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 --2- 172.21.15.29:0/1445893191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26c815b1d0 0x7f26c80c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 >> 172.21.15.29:0/1445893191 conn(0x7f26c8067610 msgr2=0x7f26c80b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 shutdown_connections 2023-10-07T09:47:12.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.470+0000 7f26cf7ff700 1 -- 172.21.15.29:0/1445893191 wait complete. 2023-10-07T09:47:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 53 2023-10-07T09:47:12.489 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:47:12.489 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:47:12.489 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:47:12.489 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:47:12.490 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:47:12.490 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:47:12.490 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:47:12.490 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:47:12.490 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:47:12.490 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:47:12.491 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:47:12.491 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:47:12.491 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:47:12.493 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:47:12.493 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:47:12.493 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:47:12.503 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:47:12.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:47:12.504 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:47:12.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:47:12.508 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:47:12.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:47:12.509 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:47:12.509 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:47:12.547 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:47:12.548 DEBUG:teuthology.parallel:result is None 2023-10-07T09:47:12.550 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:47:12.551 DEBUG:teuthology.parallel:result is None 2023-10-07T09:47:12.553 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:47:12.553 DEBUG:teuthology.parallel:result is None 2023-10-07T09:47:12.556 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:47:12.556 DEBUG:teuthology.parallel:result is None 2023-10-07T09:47:12.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:47:12.699 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:47:12.699 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:47:12.701 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:47:12.702 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:47:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.765+0000 7f76ed9ff700 1 Processor -- start 2023-10-07T09:47:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.765+0000 7f76ed9ff700 1 -- start start 2023-10-07T09:47:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.766+0000 7f76ed9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76e80ad220 0x7f76e8159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.766+0000 7f76ed9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76e80ad5f0 0x7f76e80ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.766+0000 7f76ed9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e815af00 con 0x7f76e8068620 2023-10-07T09:47:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.766+0000 7f76ed9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e80ac9d0 con 0x7f76e80ad5f0 2023-10-07T09:47:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.766+0000 7f76ed9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e80ada40 con 0x7f76e80ad220 2023-10-07T09:47:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76e80ad5f0 0x7f76e80ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76e80ad5f0 0x7f76e80ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53508/0 (socket says 172.21.15.29:53508) 2023-10-07T09:47:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 -- 172.21.15.29:0/2588391043 learned_addr learned my addr 172.21.15.29:0/2588391043 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e6ffd700 1 --2- 172.21.15.29:0/2588391043 >> v2:172.21.15.144:3300/0 conn(0x7f76e80ad220 0x7f76e8159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 -- 172.21.15.29:0/2588391043 >> v2:172.21.15.144:3300/0 conn(0x7f76e80ad220 msgr2=0x7f76e8159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 --2- 172.21.15.29:0/2588391043 >> v2:172.21.15.144:3300/0 conn(0x7f76e80ad220 0x7f76e8159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 -- 172.21.15.29:0/2588391043 >> v1:172.21.15.144:6789/0 conn(0x7f76e8068620 legacy=0x7f76e8068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 -- 172.21.15.29:0/2588391043 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e815a250 con 0x7f76e80ad5f0 2023-10-07T09:47:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e5ffb700 1 -- 172.21.15.29:0/2588391043 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765383902 0 0) 0x7f76e815af00 con 0x7f76e8068620 2023-10-07T09:47:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e6ffd700 1 --2- 172.21.15.29:0/2588391043 >> v2:172.21.15.144:3300/0 conn(0x7f76e80ad220 0x7f76e8159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.767+0000 7f76e67fc700 1 --2- 172.21.15.29:0/2588391043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e80ad5f0 0x7f76e80ac490 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f76e80691f0 tx=0x7f76dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f279e4c0792e7a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76e5ffb700 1 -- 172.21.15.29:0/2588391043 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc00b860 con 0x7f76e80ad5f0 2023-10-07T09:47:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76e5ffb700 1 -- 172.21.15.29:0/2588391043 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76dc00b9c0 con 0x7f76e80ad5f0 2023-10-07T09:47:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76e5ffb700 1 -- 172.21.15.29:0/2588391043 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc00bd60 con 0x7f76e80ad5f0 2023-10-07T09:47:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 -- 172.21.15.29:0/2588391043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e80ad5f0 msgr2=0x7f76e80ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/2588391043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e80ad5f0 0x7f76e80ac490 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f76e80691f0 tx=0x7f76dc009250 comp rx=0 tx=0).stop 2023-10-07T09:47:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 -- 172.21.15.29:0/2588391043 shutdown_connections 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/2588391043 >> v2:172.21.15.144:3300/0 conn(0x7f76e80ad220 0x7f76e8159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/2588391043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e80ad5f0 0x7f76e80ac490 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 -- 172.21.15.29:0/2588391043 >> 172.21.15.29:0/2588391043 conn(0x7f76e806dd00 msgr2=0x7f76e806e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 -- 172.21.15.29:0/2588391043 shutdown_connections 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.768+0000 7f76ed9ff700 1 -- 172.21.15.29:0/2588391043 wait complete. 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.770+0000 7f76ed9ff700 1 Processor -- start 2023-10-07T09:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.770+0000 7f76ed9ff700 1 -- start start 2023-10-07T09:47:12.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76ed9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 0x7f76e8150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76ed9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76e80ad220 0x7f76e8169be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76ed9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76e80ad5f0 0x7f76e816cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76ed9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8150ed0 con 0x7f76e8068620 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76ed9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e80b1d00 con 0x7f76e80ad220 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76ed9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e80b2550 con 0x7f76e80ad5f0 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76e6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 0x7f76e8150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76e6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 0x7f76e8150990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53524/0 (socket says 172.21.15.29:53524) 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76e6ffd700 1 -- 172.21.15.29:0/3128239187 learned_addr learned my addr 172.21.15.29:0/3128239187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76e67fc700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76e80ad220 0x7f76e8169be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.771+0000 7f76e77fe700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76e80ad5f0 0x7f76e816cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76e6ffd700 1 -- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76e80ad5f0 msgr2=0x7f76e816cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76e6ffd700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76e80ad5f0 0x7f76e816cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76e6ffd700 1 -- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76e80ad220 msgr2=0x7f76e8169be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76e6ffd700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76e80ad220 0x7f76e8169be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:12.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76e6ffd700 1 -- 172.21.15.29:0/3128239187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc0094f0 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76e6ffd700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 0x7f76e8150990 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f76d800e900 tx=0x7f76d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8013410 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d8013570 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e80b1780 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8013910 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.772+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e81696c0 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.774+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e81694f0 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.774+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f76d8013d00 con 0x7f76e8068620 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.775+0000 7f76ceffd700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76d4066f60 0x7f76d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:12.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.776+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f76d80841c0 con 0x7f76e8068620 2023-10-07T09:47:12.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.776+0000 7f76e67fc700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76d4066f60 0x7f76d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:12.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.776+0000 7f76e67fc700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76d4066f60 0x7f76d4069410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f76e80691f0 tx=0x7f76dc017000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:12.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.779+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76d8051360 con 0x7f76e8068620 2023-10-07T09:47:12.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:12.997+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f76e804c3c0 con 0x7f76e8068620 2023-10-07T09:47:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.346+0000 7f76ceffd700 1 -- 172.21.15.29:0/3128239187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v112) v1 ==== 97+0+0 (secure 0 0 0) 0x7f76d8054980 con 0x7f76e8068620 2023-10-07T09:47:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76d4066f60 msgr2=0x7f76d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76d4066f60 0x7f76d4069410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f76e80691f0 tx=0x7f76dc017000 comp rx=0 tx=0).stop 2023-10-07T09:47:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 msgr2=0x7f76e8150990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 0x7f76e8150990 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f76d800e900 tx=0x7f76d8008040 comp rx=0 tx=0).stop 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 shutdown_connections 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76d4066f60 0x7f76d4069410 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76e80ad5f0 0x7f76e816cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76e8068620 0x7f76e8150990 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 --2- 172.21.15.29:0/3128239187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76e80ad220 0x7f76e8169be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 >> 172.21.15.29:0/3128239187 conn(0x7f76e806dd00 msgr2=0x7f76e80a98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 shutdown_connections 2023-10-07T09:47:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.350+0000 7f76ed9ff700 1 -- 172.21.15.29:0/3128239187 wait complete. 2023-10-07T09:47:13.353 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:47:13.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:47:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.568+0000 7f0b243ff700 1 Processor -- start 2023-10-07T09:47:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.568+0000 7f0b243ff700 1 -- start start 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.569+0000 7f0b243ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b1c1537c0 0x7f0b1c156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.569+0000 7f0b243ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c159670 con 0x7f0b1c1533f0 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b243ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c15a1f0 con 0x7f0b1c0b04b0 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b243ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1c15a680 con 0x7f0b1c1537c0 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b2219b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b1c1533f0 0x7f0b1c0afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51880/0 (socket says 172.21.15.29:51880) 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b2219b700 1 -- 172.21.15.29:0/3977990730 learned_addr learned my addr 172.21.15.29:0/3977990730 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b2299c700 1 --2- 172.21.15.29:0/3977990730 >> v2:172.21.15.29:3300/0 conn(0x7f0b1c1537c0 0x7f0b1c156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b21199700 1 -- 172.21.15.29:0/3977990730 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561909714 0 0) 0x7f0b1c159670 con 0x7f0b1c1533f0 2023-10-07T09:47:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b21199700 1 -- 172.21.15.29:0/3977990730 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08003570 con 0x7f0b1c1533f0 2023-10-07T09:47:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b2299c700 1 -- 172.21.15.29:0/3977990730 >> v1:172.21.15.144:6789/0 conn(0x7f0b1c0b04b0 legacy=0x7f0b1c157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b21199700 1 -- 172.21.15.29:0/3977990730 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478755603 0 0) 0x7f0b1c15a1f0 con 0x7f0b1c0b04b0 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b2299c700 1 -- 172.21.15.29:0/3977990730 >> v1:172.21.15.29:6789/0 conn(0x7f0b1c1533f0 legacy=0x7f0b1c0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.570+0000 7f0b2299c700 1 -- 172.21.15.29:0/3977990730 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c1598c0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.571+0000 7f0b2299c700 1 --2- 172.21.15.29:0/3977990730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c156a90 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009e60 tx=0x7f0b14009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128c93aecc359244 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.571+0000 7f0b21199700 1 -- 172.21.15.29:0/3977990730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400bb60 con 0x7f0b1c1537c0 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.571+0000 7f0b21199700 1 -- 172.21.15.29:0/3977990730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1400bcc0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.571+0000 7f0b21199700 1 -- 172.21.15.29:0/3977990730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14011400 con 0x7f0b1c1537c0 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.571+0000 7f0b243ff700 1 -- 172.21.15.29:0/3977990730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 msgr2=0x7f0b1c156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.571+0000 7f0b243ff700 1 --2- 172.21.15.29:0/3977990730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c156a90 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009e60 tx=0x7f0b14009780 comp rx=0 tx=0).stop 2023-10-07T09:47:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.572+0000 7f0b243ff700 1 -- 172.21.15.29:0/3977990730 shutdown_connections 2023-10-07T09:47:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.572+0000 7f0b243ff700 1 --2- 172.21.15.29:0/3977990730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c156a90 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.572+0000 7f0b243ff700 1 -- 172.21.15.29:0/3977990730 >> 172.21.15.29:0/3977990730 conn(0x7f0b1c0680a0 msgr2=0x7f0b1c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.572+0000 7f0b243ff700 1 -- 172.21.15.29:0/3977990730 shutdown_connections 2023-10-07T09:47:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.572+0000 7f0b243ff700 1 -- 172.21.15.29:0/3977990730 wait complete. 2023-10-07T09:47:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.573+0000 7f0b243ff700 1 Processor -- start 2023-10-07T09:47:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.573+0000 7f0b243ff700 1 -- start start 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b243ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b1c0b04b0 0x7f0b1c14ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b243ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b1c1533f0 0x7f0b1c13fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b243ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c1508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b243ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c14f5d0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b243ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c150450 con 0x7f0b1c0b04b0 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b243ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c151fb0 con 0x7f0b1c1533f0 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.574+0000 7f0b2299c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c1508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c1508e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43952/0 (socket says 172.21.15.29:43952) 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 -- 172.21.15.29:0/2546191597 learned_addr learned my addr 172.21.15.29:0/2546191597 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2219b700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b1c0b04b0 0x7f0b1c14ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2199a700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b1c1533f0 0x7f0b1c13fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 -- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b1c1533f0 msgr2=0x7f0b1c13fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b1c1533f0 0x7f0b1c13fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 -- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b1c0b04b0 msgr2=0x7f0b1c14ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b1c0b04b0 0x7f0b1c14ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 -- 172.21.15.29:0/2546191597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14009a20 con 0x7f0b1c1537c0 2023-10-07T09:47:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b2299c700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c1508e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f0b14011fc0 tx=0x7f0b1401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14020070 con 0x7f0b1c1537c0 2023-10-07T09:47:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1401a730 con 0x7f0b1c1537c0 2023-10-07T09:47:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.575+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c1519f0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.576+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1401aad0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.576+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c140220 con 0x7f0b1c1537c0 2023-10-07T09:47:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.577+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1c04c3c0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.578+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0b1402d3b0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.578+0000 7f0b127fc700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b00066f30 0x7f0b000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.579+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f0b140860b0 con 0x7f0b1c1537c0 2023-10-07T09:47:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.579+0000 7f0b2219b700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b00066f30 0x7f0b000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.579+0000 7f0b2219b700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b00066f30 0x7f0b000693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0adb50 tx=0x7f0b18009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:13.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.582+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b14053460 con 0x7f0b1c1537c0 2023-10-07T09:47:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.866+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0b1c06f660 con 0x7f0b1c1537c0 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.867+0000 7f0b127fc700 1 -- 172.21.15.29:0/2546191597 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0b14056a80 con 0x7f0b1c1537c0 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.869 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:47:13.870 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:47:13.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:47:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.874 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.875 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:47:13.876 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:47:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T09:47:13.878 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:47:13.880 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.882 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:47:13.883 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:47:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b00066f30 msgr2=0x7f0b000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b00066f30 0x7f0b000693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0adb50 tx=0x7f0b18009040 comp rx=0 tx=0).stop 2023-10-07T09:47:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 msgr2=0x7f0b1c1508e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c1508e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f0b14011fc0 tx=0x7f0b1401a000 comp rx=0 tx=0).stop 2023-10-07T09:47:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 shutdown_connections 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0b00066f30 0x7f0b000693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c1537c0 0x7f0b1c1508e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b1c0b04b0 0x7f0b1c14ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 --2- 172.21.15.29:0/2546191597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b1c1533f0 0x7f0b1c13fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 >> 172.21.15.29:0/2546191597 conn(0x7f0b1c0680a0 msgr2=0x7f0b1c1525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 shutdown_connections 2023-10-07T09:47:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:13.869+0000 7f0b243ff700 1 -- 172.21.15.29:0/2546191597 wait complete. 2023-10-07T09:47:13.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:47:14.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.081+0000 7f9397bff700 1 Processor -- start 2023-10-07T09:47:14.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.081+0000 7f9397bff700 1 -- start start 2023-10-07T09:47:14.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.082+0000 7f9397bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9390159550 0x7f93900b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.082+0000 7f9397bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b0f50 0x7f93900b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.082+0000 7f9397bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93900b2bb0 0x7f939015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.082+0000 7f9397bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939006f4e0 con 0x7f9390159550 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.082+0000 7f9397bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939006e8d0 con 0x7f93900b2bb0 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.082+0000 7f9397bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939006ebb0 con 0x7f93900b0f50 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9390159550 0x7f93900b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9390159550 0x7f93900b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43956/0 (socket says 172.21.15.29:43956) 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 -- 172.21.15.29:0/4013467578 learned_addr learned my addr 172.21.15.29:0/4013467578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939519a700 1 --2- 172.21.15.29:0/4013467578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b0f50 0x7f93900b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939619c700 1 --2- 172.21.15.29:0/4013467578 >> v2:172.21.15.144:3300/0 conn(0x7f93900b2bb0 0x7f939015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 -- 172.21.15.29:0/4013467578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b0f50 msgr2=0x7f93900b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 --2- 172.21.15.29:0/4013467578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b0f50 0x7f93900b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 -- 172.21.15.29:0/4013467578 >> v2:172.21.15.144:3300/0 conn(0x7f93900b2bb0 msgr2=0x7f939015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 --2- 172.21.15.29:0/4013467578 >> v2:172.21.15.144:3300/0 conn(0x7f93900b2bb0 0x7f939015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.083+0000 7f939599b700 1 -- 172.21.15.29:0/4013467578 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939006ee90 con 0x7f9390159550 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f939599b700 1 --2- 172.21.15.29:0/4013467578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9390159550 0x7f93900b0a10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9390068200 tx=0x7f938c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f040654a689e0c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9394999700 1 -- 172.21.15.29:0/4013467578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00b860 con 0x7f9390159550 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9394999700 1 -- 172.21.15.29:0/4013467578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c00b9c0 con 0x7f9390159550 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9394999700 1 -- 172.21.15.29:0/4013467578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00bd60 con 0x7f9390159550 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 -- 172.21.15.29:0/4013467578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9390159550 msgr2=0x7f93900b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 --2- 172.21.15.29:0/4013467578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9390159550 0x7f93900b0a10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9390068200 tx=0x7f938c009250 comp rx=0 tx=0).stop 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 -- 172.21.15.29:0/4013467578 shutdown_connections 2023-10-07T09:47:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 --2- 172.21.15.29:0/4013467578 >> v2:172.21.15.144:3300/0 conn(0x7f93900b2bb0 0x7f939015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 --2- 172.21.15.29:0/4013467578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9390159550 0x7f93900b0a10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 --2- 172.21.15.29:0/4013467578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b0f50 0x7f93900b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.084+0000 7f9397bff700 1 -- 172.21.15.29:0/4013467578 >> 172.21.15.29:0/4013467578 conn(0x7f9390069190 msgr2=0x7f939006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.085+0000 7f9397bff700 1 -- 172.21.15.29:0/4013467578 shutdown_connections 2023-10-07T09:47:14.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.085+0000 7f9397bff700 1 -- 172.21.15.29:0/4013467578 wait complete. 2023-10-07T09:47:14.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.087+0000 7f9397bff700 1 Processor -- start 2023-10-07T09:47:14.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.087+0000 7f9397bff700 1 -- start start 2023-10-07T09:47:14.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f9397bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 0x7f939015d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f9397bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b2bb0 0x7f9390143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f9397bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9390159550 0x7f939015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f9397bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939015cbe0 con 0x7f93900b0f50 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f9397bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939015c010 con 0x7f9390159550 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f9397bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939015c2b0 con 0x7f93900b2bb0 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f939599b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 0x7f939015d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f939599b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 0x7f939015d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43958/0 (socket says 172.21.15.29:43958) 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.088+0000 7f939599b700 1 -- 172.21.15.29:0/176847663 learned_addr learned my addr 172.21.15.29:0/176847663 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f939599b700 1 -- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b2bb0 msgr2=0x7f9390143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f939599b700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b2bb0 0x7f9390143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f939599b700 1 -- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9390159550 msgr2=0x7f939015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f939599b700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9390159550 0x7f939015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f939599b700 1 -- 172.21.15.29:0/176847663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c0094f0 con 0x7f93900b0f50 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f939599b700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 0x7f939015d070 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f939006c520 tx=0x7f938c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c024440 con 0x7f93900b0f50 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c0245a0 con 0x7f93900b0f50 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c024960 con 0x7f93900b0f50 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.089+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939015c590 con 0x7f93900b0f50 2023-10-07T09:47:14.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.090+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939015d5b0 con 0x7f93900b0f50 2023-10-07T09:47:14.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.091+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939004c3c0 con 0x7f93900b0f50 2023-10-07T09:47:14.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.092+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f938c01a420 con 0x7f93900b0f50 2023-10-07T09:47:14.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.092+0000 7f9385ffb700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9374066f30 0x7f93740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.092+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f938c086260 con 0x7f93900b0f50 2023-10-07T09:47:14.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.093+0000 7f939519a700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9374066f30 0x7f93740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.093+0000 7f939519a700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9374066f30 0x7f93740693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93900ce1a0 tx=0x7f9380008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.097+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f938c0536b0 con 0x7f93900b0f50 2023-10-07T09:47:14.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.298+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93900683d0 con 0x7f93900b0f50 2023-10-07T09:47:14.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.299+0000 7f9385ffb700 1 -- 172.21.15.29:0/176847663 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+12811 (secure 0 0 0) 0x7f938c056cd0 con 0x7f93900b0f50 2023-10-07T09:47:14.301 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:14.301 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:47:13.353116+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:47:14.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9374066f30 msgr2=0x7f93740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9374066f30 0x7f93740693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93900ce1a0 tx=0x7f9380008040 comp rx=0 tx=0).stop 2023-10-07T09:47:14.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 msgr2=0x7f939015d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 0x7f939015d070 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f939006c520 tx=0x7f938c017000 comp rx=0 tx=0).stop 2023-10-07T09:47:14.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 shutdown_connections 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9374066f30 0x7f93740693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9390159550 0x7f939015a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93900b0f50 0x7f939015d070 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 --2- 172.21.15.29:0/176847663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93900b2bb0 0x7f9390143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 >> 172.21.15.29:0/176847663 conn(0x7f9390069190 msgr2=0x7f93900aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 shutdown_connections 2023-10-07T09:47:14.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.302+0000 7f9397bff700 1 -- 172.21.15.29:0/176847663 wait complete. 2023-10-07T09:47:14.321 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:47:14.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:47:14.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.522+0000 7f68637ff700 1 Processor -- start 2023-10-07T09:47:14.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.522+0000 7f68637ff700 1 -- start start 2023-10-07T09:47:14.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f68637ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f685c159750 0x7f685c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f68637ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f685c159b20 0x7f685c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f68637ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f68637ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685c0cbf20 con 0x7f685c159750 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f68637ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685c0cc200 con 0x7f685c159b20 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f68637ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685c0cc4e0 con 0x7f685c0ce6d0 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.523+0000 7f686159b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f685c159750 0x7f685c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f685c159750 0x7f685c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43972/0 (socket says 172.21.15.29:43972) 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 -- 172.21.15.29:0/2081589482 learned_addr learned my addr 172.21.15.29:0/2081589482 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f6860d9a700 1 --2- 172.21.15.29:0/2081589482 >> v2:172.21.15.144:3300/0 conn(0x7f685c159b20 0x7f685c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f6861d9c700 1 --2- 172.21.15.29:0/2081589482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 -- 172.21.15.29:0/2081589482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 msgr2=0x7f685c0ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 --2- 172.21.15.29:0/2081589482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 -- 172.21.15.29:0/2081589482 >> v2:172.21.15.144:3300/0 conn(0x7f685c159b20 msgr2=0x7f685c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 --2- 172.21.15.29:0/2081589482 >> v2:172.21.15.144:3300/0 conn(0x7f685c159b20 0x7f685c0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 -- 172.21.15.29:0/2081589482 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c0cc7c0 con 0x7f685c159750 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.524+0000 7f686159b700 1 --2- 172.21.15.29:0/2081589482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0d10a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f685c088a70 tx=0x7f6858009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed21302a876d8cb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f6853fff700 1 -- 172.21.15.29:0/2081589482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858005c10 con 0x7f685c159750 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f6853fff700 1 -- 172.21.15.29:0/2081589482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6858005d70 con 0x7f685c159750 2023-10-07T09:47:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f6853fff700 1 -- 172.21.15.29:0/2081589482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858006260 con 0x7f685c159750 2023-10-07T09:47:14.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 -- 172.21.15.29:0/2081589482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 msgr2=0x7f685c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 --2- 172.21.15.29:0/2081589482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0d10a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f685c088a70 tx=0x7f6858009220 comp rx=0 tx=0).stop 2023-10-07T09:47:14.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 -- 172.21.15.29:0/2081589482 shutdown_connections 2023-10-07T09:47:14.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 --2- 172.21.15.29:0/2081589482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 --2- 172.21.15.29:0/2081589482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0d10a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 --2- 172.21.15.29:0/2081589482 >> v2:172.21.15.144:3300/0 conn(0x7f685c159b20 0x7f685c0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 -- 172.21.15.29:0/2081589482 >> 172.21.15.29:0/2081589482 conn(0x7f685c063a10 msgr2=0x7f685c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 -- 172.21.15.29:0/2081589482 shutdown_connections 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.525+0000 7f68637ff700 1 -- 172.21.15.29:0/2081589482 wait complete. 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.527+0000 7f68637ff700 1 Processor -- start 2023-10-07T09:47:14.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.527+0000 7f68637ff700 1 -- start start 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f68637ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f68637ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f68637ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c159b20 0x7f685c1519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f68637ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c0c9010 con 0x7f685c159750 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f68637ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c0b47e0 con 0x7f685c159b20 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f68637ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685c16a8d0 con 0x7f685c0ce6d0 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f6860d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f6860d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43978/0 (socket says 172.21.15.29:43978) 2023-10-07T09:47:14.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f6860d9a700 1 -- 172.21.15.29:0/1435268769 learned_addr learned my addr 172.21.15.29:0/1435268769 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f686159b700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.528+0000 7f6861d9c700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c159b20 0x7f685c1519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f6860d9a700 1 -- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 msgr2=0x7f685c0c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f6860d9a700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f6860d9a700 1 -- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c159b20 msgr2=0x7f685c1519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f6860d9a700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c159b20 0x7f685c1519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f6860d9a700 1 -- 172.21.15.29:0/1435268769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6858009480 con 0x7f685c159750 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f6860d9a700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0b42a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f684c00e930 tx=0x7f684c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c013680 con 0x7f685c159750 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684c0137e0 con 0x7f685c159750 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c013b80 con 0x7f685c159750 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c169a60 con 0x7f685c159750 2023-10-07T09:47:14.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.529+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c16a280 con 0x7f685c159750 2023-10-07T09:47:14.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.530+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c04c3c0 con 0x7f685c159750 2023-10-07T09:47:14.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.531+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f684c00c3e0 con 0x7f685c159750 2023-10-07T09:47:14.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.532+0000 7f68517fa700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6840066f30 0x7f68400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.532+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f684c0844a0 con 0x7f685c159750 2023-10-07T09:47:14.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.533+0000 7f686159b700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6840066f30 0x7f68400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.533+0000 7f686159b700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6840066f30 0x7f68400693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f685c0ce150 tx=0x7f6858016000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.536+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f684c051850 con 0x7f685c159750 2023-10-07T09:47:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.741+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f685c15d1a0 con 0x7f685c159750 2023-10-07T09:47:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.742+0000 7f68517fa700 1 -- 172.21.15.29:0/1435268769 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f684c054e70 con 0x7f685c159750 2023-10-07T09:47:14.743 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:47:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6840066f30 msgr2=0x7f68400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6840066f30 0x7f68400693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f685c0ce150 tx=0x7f6858016000 comp rx=0 tx=0).stop 2023-10-07T09:47:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 msgr2=0x7f685c0b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0b42a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f684c00e930 tx=0x7f684c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 shutdown_connections 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6840066f30 0x7f68400693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685c159b20 0x7f685c1519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f685c0ce6d0 0x7f685c0c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 --2- 172.21.15.29:0/1435268769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f685c159750 0x7f685c0b42a0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.744+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 >> 172.21.15.29:0/1435268769 conn(0x7f685c063a10 msgr2=0x7f685c153b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.745+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 shutdown_connections 2023-10-07T09:47:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.745+0000 7f68637ff700 1 -- 172.21.15.29:0/1435268769 wait complete. 2023-10-07T09:47:14.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:47:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.958+0000 7f538f3ff700 1 Processor -- start 2023-10-07T09:47:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.959+0000 7f538f3ff700 1 -- start start 2023-10-07T09:47:14.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538f3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5388159750 0x7f53880d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538f3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f53880ce8a0 0x7f53880c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538f3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880cbf20 con 0x7f5388159b20 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538f3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53880cc200 con 0x7f5388159750 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538f3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53880cc4e0 con 0x7f53880ce8a0 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538d19b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5388159750 0x7f53880d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538c99a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5388159b20 0x7f53880ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51884/0 (socket says 172.21.15.29:51884) 2023-10-07T09:47:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.960+0000 7f538c99a700 1 -- 172.21.15.29:0/2694212232 learned_addr learned my addr 172.21.15.29:0/2694212232 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d99c700 1 --2- 172.21.15.29:0/2694212232 >> v2:172.21.15.144:3300/0 conn(0x7f53880ce8a0 0x7f53880c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f537ffff700 1 -- 172.21.15.29:0/2694212232 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253895992 0 0) 0x7f53880cbf20 con 0x7f5388159b20 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f537ffff700 1 -- 172.21.15.29:0/2694212232 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5374003570 con 0x7f5388159b20 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d19b700 1 -- 172.21.15.29:0/2694212232 >> v2:172.21.15.144:3300/0 conn(0x7f53880ce8a0 msgr2=0x7f53880c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d19b700 1 --2- 172.21.15.29:0/2694212232 >> v2:172.21.15.144:3300/0 conn(0x7f53880ce8a0 0x7f53880c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d19b700 1 -- 172.21.15.29:0/2694212232 >> v1:172.21.15.29:6789/0 conn(0x7f5388159b20 legacy=0x7f53880ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d19b700 1 -- 172.21.15.29:0/2694212232 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880cc7c0 con 0x7f5388159750 2023-10-07T09:47:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d99c700 1 --2- 172.21.15.29:0/2694212232 >> v2:172.21.15.144:3300/0 conn(0x7f53880ce8a0 0x7f53880c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f538d19b700 1 --2- 172.21.15.29:0/2694212232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388159750 0x7f53880d10a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f538815e510 tx=0x7f5384009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143c98f55bca2de0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f537ffff700 1 -- 172.21.15.29:0/2694212232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384005c10 con 0x7f5388159750 2023-10-07T09:47:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f537ffff700 1 -- 172.21.15.29:0/2694212232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5384005d70 con 0x7f5388159750 2023-10-07T09:47:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.961+0000 7f537ffff700 1 -- 172.21.15.29:0/2694212232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384006240 con 0x7f5388159750 2023-10-07T09:47:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 -- 172.21.15.29:0/2694212232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388159750 msgr2=0x7f53880d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2694212232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388159750 0x7f53880d10a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f538815e510 tx=0x7f5384009120 comp rx=0 tx=0).stop 2023-10-07T09:47:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 -- 172.21.15.29:0/2694212232 shutdown_connections 2023-10-07T09:47:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2694212232 >> v2:172.21.15.144:3300/0 conn(0x7f53880ce8a0 0x7f53880c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2694212232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5388159750 0x7f53880d10a0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 -- 172.21.15.29:0/2694212232 >> 172.21.15.29:0/2694212232 conn(0x7f5388063a10 msgr2=0x7f53880d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 -- 172.21.15.29:0/2694212232 shutdown_connections 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.962+0000 7f538f3ff700 1 -- 172.21.15.29:0/2694212232 wait complete. 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.964+0000 7f538f3ff700 1 Processor -- start 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.964+0000 7f538f3ff700 1 -- start start 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538f3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 0x7f53881470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538f3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5388159750 0x7f53880ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5388159b20 0x7f5388159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538f3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b53c0 con 0x7f53880ce8a0 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538f3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b47b0 con 0x7f5388159750 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b4a90 con 0x7f5388159b20 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 0x7f53881470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 0x7f53881470b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43982/0 (socket says 172.21.15.29:43982) 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 -- 172.21.15.29:0/2395182324 learned_addr learned my addr 172.21.15.29:0/2395182324 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d99c700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5388159b20 0x7f5388159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538c99a700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5388159750 0x7f53880ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 -- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5388159b20 msgr2=0x7f5388159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5388159b20 0x7f5388159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 -- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5388159750 msgr2=0x7f53880ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5388159750 0x7f53880ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.965+0000 7f538d19b700 1 -- 172.21.15.29:0/2395182324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840093c0 con 0x7f53880ce8a0 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.966+0000 7f538d19b700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 0x7f53881470b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5388088a40 tx=0x7f5384016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.966+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538401a630 con 0x7f53880ce8a0 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.966+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538401a7b0 con 0x7f53880ce8a0 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.966+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53880b4d70 con 0x7f53880ce8a0 2023-10-07T09:47:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.966+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538401ab70 con 0x7f53880ce8a0 2023-10-07T09:47:14.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.966+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880ca740 con 0x7f53880ce8a0 2023-10-07T09:47:14.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.967+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538804c3c0 con 0x7f53880ce8a0 2023-10-07T09:47:14.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.968+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5384026430 con 0x7f53880ce8a0 2023-10-07T09:47:14.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.969+0000 7f537d7fa700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f536c066f30 0x7f536c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:14.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.969+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f5384085b20 con 0x7f53880ce8a0 2023-10-07T09:47:14.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.970+0000 7f538c99a700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f536c066f30 0x7f536c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:14.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.970+0000 7f538c99a700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f536c066f30 0x7f536c0693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f538816d1d0 tx=0x7f5378009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:14.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:14.973+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5384052ed0 con 0x7f53880ce8a0 2023-10-07T09:47:15.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.177+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5388064d80 con 0x7f53880ce8a0 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.178+0000 7f537d7fa700 1 -- 172.21.15.29:0/2395182324 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v114) v1 ==== 84+0+65 (secure 0 0 0) 0x7f53840564f0 con 0x7f53880ce8a0 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:15.179 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:47:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f536c066f30 msgr2=0x7f536c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f536c066f30 0x7f536c0693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f538816d1d0 tx=0x7f5378009040 comp rx=0 tx=0).stop 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 msgr2=0x7f53881470b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 0x7f53881470b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5388088a40 tx=0x7f5384016000 comp rx=0 tx=0).stop 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 shutdown_connections 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f536c066f30 0x7f536c0693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5388159b20 0x7f5388159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53880ce8a0 0x7f53881470b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 --2- 172.21.15.29:0/2395182324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5388159750 0x7f53880ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 >> 172.21.15.29:0/2395182324 conn(0x7f5388063a10 msgr2=0x7f53880c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 shutdown_connections 2023-10-07T09:47:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.180+0000 7f538f3ff700 1 -- 172.21.15.29:0/2395182324 wait complete. 2023-10-07T09:47:15.199 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:47:15.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:47:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.398+0000 7fe83adff700 1 Processor -- start 2023-10-07T09:47:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.398+0000 7fe83adff700 1 -- start start 2023-10-07T09:47:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83adff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe8340b2a20 0x7fe8340cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83adff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8340cf2c0 0x7fe8340c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83adff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340cc940 con 0x7fe8340cb6f0 2023-10-07T09:47:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83adff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8340b2df0 con 0x7fe8340cf2c0 2023-10-07T09:47:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83adff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8340cbd30 con 0x7fe8340b2a20 2023-10-07T09:47:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83939c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8340cf2c0 0x7fe8340c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe833fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe8340cb6f0 0x7fe8340cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51896/0 (socket says 172.21.15.29:51896) 2023-10-07T09:47:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe833fff700 1 -- 172.21.15.29:0/1445799684 learned_addr learned my addr 172.21.15.29:0/1445799684 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe838b9b700 1 --2- 172.21.15.29:0/1445799684 >> v2:172.21.15.144:3300/0 conn(0x7fe8340b2a20 0x7fe8340cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:15.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83939c700 1 -- 172.21.15.29:0/1445799684 >> v2:172.21.15.144:3300/0 conn(0x7fe8340b2a20 msgr2=0x7fe8340cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83939c700 1 --2- 172.21.15.29:0/1445799684 >> v2:172.21.15.144:3300/0 conn(0x7fe8340b2a20 0x7fe8340cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83939c700 1 -- 172.21.15.29:0/1445799684 >> v1:172.21.15.29:6789/0 conn(0x7fe8340cb6f0 legacy=0x7fe8340cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe83939c700 1 -- 172.21.15.29:0/1445799684 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8340cc010 con 0x7fe8340cf2c0 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.400+0000 7fe8337fe700 1 -- 172.21.15.29:0/1445799684 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564865771 0 0) 0x7fe8340cc940 con 0x7fe8340cb6f0 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.401+0000 7fe83939c700 1 --2- 172.21.15.29:0/1445799684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe8340c5400 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe83408ac80 tx=0x7fe82c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1ca2ae5d89b2c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.401+0000 7fe8337fe700 1 -- 172.21.15.29:0/1445799684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c00e070 con 0x7fe8340cf2c0 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.401+0000 7fe8337fe700 1 -- 172.21.15.29:0/1445799684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c00bb60 con 0x7fe8340cf2c0 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.401+0000 7fe8337fe700 1 -- 172.21.15.29:0/1445799684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c012430 con 0x7fe8340cf2c0 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 -- 172.21.15.29:0/1445799684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 msgr2=0x7fe8340c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 --2- 172.21.15.29:0/1445799684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe8340c5400 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe83408ac80 tx=0x7fe82c009780 comp rx=0 tx=0).stop 2023-10-07T09:47:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 -- 172.21.15.29:0/1445799684 shutdown_connections 2023-10-07T09:47:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 --2- 172.21.15.29:0/1445799684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe8340c5400 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 --2- 172.21.15.29:0/1445799684 >> v2:172.21.15.144:3300/0 conn(0x7fe8340b2a20 0x7fe8340cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 -- 172.21.15.29:0/1445799684 >> 172.21.15.29:0/1445799684 conn(0x7fe834099820 msgr2=0x7fe834063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 -- 172.21.15.29:0/1445799684 shutdown_connections 2023-10-07T09:47:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.402+0000 7fe83adff700 1 -- 172.21.15.29:0/1445799684 wait complete. 2023-10-07T09:47:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.404+0000 7fe83adff700 1 Processor -- start 2023-10-07T09:47:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.404+0000 7fe83adff700 1 -- start start 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.405+0000 7fe83adff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8340b2a20 0x7fe8341550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.405+0000 7fe83adff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8340cb6f0 0x7fe8340b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.405+0000 7fe83adff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe834147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.405+0000 7fe83adff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340c8280 con 0x7fe8340cf2c0 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.405+0000 7fe83adff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340b3ed0 con 0x7fe8340cb6f0 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.405+0000 7fe83adff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340c84d0 con 0x7fe8340b2a20 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe834147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe834147be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44002/0 (socket says 172.21.15.29:44002) 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 -- 172.21.15.29:0/3226044436 learned_addr learned my addr 172.21.15.29:0/3226044436 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe833fff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8340cb6f0 0x7fe8340b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe838b9b700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8340b2a20 0x7fe8341550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 -- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8340b2a20 msgr2=0x7fe8341550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8340b2a20 0x7fe8341550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 -- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8340cb6f0 msgr2=0x7fe8340b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8340cb6f0 0x7fe8340b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 -- 172.21.15.29:0/3226044436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c009a20 con 0x7fe8340cf2c0 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe833fff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8340cb6f0 0x7fe8340b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.406+0000 7fe83939c700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe834147be0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe834064210 tx=0x7fe82c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.407+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c012e20 con 0x7fe8340cf2c0 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.407+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c01d630 con 0x7fe8340cf2c0 2023-10-07T09:47:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.407+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c01d9f0 con 0x7fe8340cf2c0 2023-10-07T09:47:15.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.407+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8340c87b0 con 0x7fe8340cf2c0 2023-10-07T09:47:15.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.407+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83415d280 con 0x7fe8340cf2c0 2023-10-07T09:47:15.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.408+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8340b3c80 con 0x7fe8340cf2c0 2023-10-07T09:47:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.409+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe82c024070 con 0x7fe8340cf2c0 2023-10-07T09:47:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.410+0000 7fe830ff9700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe818066f30 0x7fe8180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.410+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fe82c086bb0 con 0x7fe8340cf2c0 2023-10-07T09:47:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.410+0000 7fe838b9b700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe818066f30 0x7fe8180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.411+0000 7fe838b9b700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe818066f30 0x7fe8180693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe828009d30 tx=0x7fe828008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.414+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe82c053f60 con 0x7fe8340cf2c0 2023-10-07T09:47:15.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:15.622+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe8340b4060 con 0x7fe8340cf2c0 2023-10-07T09:47:16.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.362+0000 7fe830ff9700 1 -- 172.21.15.29:0/3226044436 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v115) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe82c02c0b0 con 0x7fe8340cf2c0 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.369+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe818066f30 msgr2=0x7fe8180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.369+0000 7fe83adff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe818066f30 0x7fe8180693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe828009d30 tx=0x7fe828008040 comp rx=0 tx=0).stop 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.369+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 msgr2=0x7fe834147be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.369+0000 7fe83adff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe834147be0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe834064210 tx=0x7fe82c019000 comp rx=0 tx=0).stop 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.370+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 shutdown_connections 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.370+0000 7fe83adff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe818066f30 0x7fe8180693e0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe828009d30 tx=0x7fe828008040 comp rx=0 tx=0).stop 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.370+0000 7fe83adff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8340cf2c0 0x7fe834147be0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.370+0000 7fe83adff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8340b2a20 0x7fe8341550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.370+0000 7fe83adff700 1 --2- 172.21.15.29:0/3226044436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8340cb6f0 0x7fe8340b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.370+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 >> 172.21.15.29:0/3226044436 conn(0x7fe834099820 msgr2=0x7fe834069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.371+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 shutdown_connections 2023-10-07T09:47:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.371+0000 7fe83adff700 1 -- 172.21.15.29:0/3226044436 wait complete. 2023-10-07T09:47:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:47:16.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:47:16.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.600+0000 7faa015ff700 1 Processor -- start 2023-10-07T09:47:16.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.600+0000 7faa015ff700 1 -- start start 2023-10-07T09:47:16.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.601+0000 7faa015ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa9fc0b1550 0x7fa9fc15bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.601+0000 7faa015ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9fc0ac100 0x7fa9fc0adc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.601+0000 7faa015ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0ac5e0 con 0x7fa9fc0b1920 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.601+0000 7faa015ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9fc0abae0 con 0x7fa9fc0ac100 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.601+0000 7faa015ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9fc0ac8c0 con 0x7fa9fc0b1550 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fa7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9fc0ac100 0x7fa9fc0adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fb7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa9fc0b1920 0x7fa9fc0ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51904/0 (socket says 172.21.15.29:51904) 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fb7fe700 1 -- 172.21.15.29:0/621852010 learned_addr learned my addr 172.21.15.29:0/621852010 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9faffd700 1 --2- 172.21.15.29:0/621852010 >> v2:172.21.15.144:3300/0 conn(0x7fa9fc0b1550 0x7fa9fc15bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fa7fc700 1 -- 172.21.15.29:0/621852010 >> v2:172.21.15.144:3300/0 conn(0x7fa9fc0b1550 msgr2=0x7fa9fc15bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fa7fc700 1 --2- 172.21.15.29:0/621852010 >> v2:172.21.15.144:3300/0 conn(0x7fa9fc0b1550 0x7fa9fc15bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fa7fc700 1 -- 172.21.15.29:0/621852010 >> v1:172.21.15.29:6789/0 conn(0x7fa9fc0b1920 legacy=0x7fa9fc0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fa7fc700 1 -- 172.21.15.29:0/621852010 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0acba0 con 0x7fa9fc0ac100 2023-10-07T09:47:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9faffd700 1 --2- 172.21.15.29:0/621852010 >> v2:172.21.15.144:3300/0 conn(0x7fa9fc0b1550 0x7fa9fc15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.602+0000 7fa9fa7fc700 1 --2- 172.21.15.29:0/621852010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0ac100 0x7fa9fc0adc50 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0cd220 tx=0x7fa9f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c34d4c0304f74b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7fa9f9ffb700 1 -- 172.21.15.29:0/621852010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000b6b0 con 0x7fa9fc0ac100 2023-10-07T09:47:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7fa9f9ffb700 1 -- 172.21.15.29:0/621852010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f000b810 con 0x7fa9fc0ac100 2023-10-07T09:47:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7fa9f9ffb700 1 -- 172.21.15.29:0/621852010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000bbb0 con 0x7fa9fc0ac100 2023-10-07T09:47:16.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 -- 172.21.15.29:0/621852010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0ac100 msgr2=0x7fa9fc0adc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:16.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 --2- 172.21.15.29:0/621852010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0ac100 0x7fa9fc0adc50 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0cd220 tx=0x7fa9f00091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 -- 172.21.15.29:0/621852010 shutdown_connections 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 --2- 172.21.15.29:0/621852010 >> v2:172.21.15.144:3300/0 conn(0x7fa9fc0b1550 0x7fa9fc15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 --2- 172.21.15.29:0/621852010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0ac100 0x7fa9fc0adc50 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 -- 172.21.15.29:0/621852010 >> 172.21.15.29:0/621852010 conn(0x7fa9fc067610 msgr2=0x7fa9fc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 -- 172.21.15.29:0/621852010 shutdown_connections 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.603+0000 7faa015ff700 1 -- 172.21.15.29:0/621852010 wait complete. 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.605+0000 7faa015ff700 1 Processor -- start 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.605+0000 7faa015ff700 1 -- start start 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7faa015ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9fc0ac100 0x7fa9fc14f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7faa015ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9fc0b1550 0x7fa9fc1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:16.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7faa015ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 0x7fa9fc1545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7faa015ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc154c90 con 0x7fa9fc0b1920 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7faa015ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc169a30 con 0x7fa9fc0b1550 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7faa015ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc16a5b0 con 0x7fa9fc0ac100 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7fa9fb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 0x7fa9fc1545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7fa9fb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 0x7fa9fc1545f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44022/0 (socket says 172.21.15.29:44022) 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7fa9fb7fe700 1 -- 172.21.15.29:0/1231004469 learned_addr learned my addr 172.21.15.29:0/1231004469 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7fa9faffd700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9fc0ac100 0x7fa9fc14f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.606+0000 7fa9fa7fc700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9fc0b1550 0x7fa9fc1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9fb7fe700 1 -- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9fc0ac100 msgr2=0x7fa9fc14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:16.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9fb7fe700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9fc0ac100 0x7fa9fc14f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9fb7fe700 1 -- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9fc0b1550 msgr2=0x7fa9fc1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9fb7fe700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9fc0b1550 0x7fa9fc1540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9fb7fe700 1 -- 172.21.15.29:0/1231004469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0009420 con 0x7fa9fc0b1920 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9fb7fe700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 0x7fa9fc1545f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400cbc0 tx=0x7fa9f40019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400a940 con 0x7fa9fc0b1920 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f400aaa0 con 0x7fa9fc0b1920 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc14fca0 con 0x7fa9fc0b1920 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400ae20 con 0x7fa9fc0b1920 2023-10-07T09:47:16.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.607+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc169f60 con 0x7fa9fc0b1920 2023-10-07T09:47:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.608+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc16a720 con 0x7fa9fc0b1920 2023-10-07T09:47:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.610+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa9f4007300 con 0x7fa9fc0b1920 2023-10-07T09:47:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.610+0000 7fa9e2ffd700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e8066f30 0x7fa9e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.610+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5599+0+0 (secure 0 0 0) 0x7fa9f4085480 con 0x7fa9fc0b1920 2023-10-07T09:47:16.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.611+0000 7fa9faffd700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e8066f30 0x7fa9e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:16.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.612+0000 7fa9faffd700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e8066f30 0x7fa9e80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec006f00 tx=0x7fa9ec008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:16.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.614+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9f4052730 con 0x7fa9fc0b1920 2023-10-07T09:47:16.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:16.816+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa9fc0718b0 con 0x7fa9fc0b1920 2023-10-07T09:47:17.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.369+0000 7fa9e2ffd700 1 -- 172.21.15.29:0/1231004469 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v116) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa9f4055d50 con 0x7fa9fc0b1920 2023-10-07T09:47:17.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.380+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e8066f30 msgr2=0x7fa9e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:17.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.380+0000 7faa015ff700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e8066f30 0x7fa9e80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec006f00 tx=0x7fa9ec008040 comp rx=0 tx=0).stop 2023-10-07T09:47:17.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.380+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 msgr2=0x7fa9fc1545f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:17.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.380+0000 7faa015ff700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 0x7fa9fc1545f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400cbc0 tx=0x7fa9f40019c0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.382+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 shutdown_connections 2023-10-07T09:47:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.382+0000 7faa015ff700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e8066f30 0x7fa9e80693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.382+0000 7faa015ff700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc0b1920 0x7fa9fc1545f0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.382+0000 7faa015ff700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9fc0ac100 0x7fa9fc14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.382+0000 7faa015ff700 1 --2- 172.21.15.29:0/1231004469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9fc0b1550 0x7fa9fc1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.382+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 >> 172.21.15.29:0/1231004469 conn(0x7fa9fc067610 msgr2=0x7fa9fc157980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:17.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.386+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 shutdown_connections 2023-10-07T09:47:17.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.389+0000 7faa015ff700 1 -- 172.21.15.29:0/1231004469 wait complete. 2023-10-07T09:47:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:47:17.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:47:17.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.612+0000 7ff33b5ff700 1 Processor -- start 2023-10-07T09:47:17.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.612+0000 7ff33b5ff700 1 -- start start 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.613+0000 7ff33b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159750 0x7ff3340d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.613+0000 7ff33b5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff334159b20 0x7ff3340ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.613+0000 7ff33b5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3340cbf20 con 0x7ff3340ce6d0 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.613+0000 7ff33b5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3340cc200 con 0x7ff334159b20 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.613+0000 7ff33b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340cc4e0 con 0x7ff334159750 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff339b9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3340ce6d0 0x7ff33415d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51916/0 (socket says 172.21.15.29:51916) 2023-10-07T09:47:17.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff339b9c700 1 -- 172.21.15.29:0/4233656685 learned_addr learned my addr 172.21.15.29:0/4233656685 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:17.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff33939b700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159750 0x7ff3340d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:17.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff338b9a700 1 --2- 172.21.15.29:0/4233656685 >> v2:172.21.15.144:3300/0 conn(0x7ff334159b20 0x7ff3340ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff32bfff700 1 -- 172.21.15.29:0/4233656685 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330790078 0 0) 0x7ff3340cbf20 con 0x7ff3340ce6d0 2023-10-07T09:47:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff32bfff700 1 -- 172.21.15.29:0/4233656685 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff320003570 con 0x7ff3340ce6d0 2023-10-07T09:47:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff338b9a700 1 -- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159750 msgr2=0x7ff3340d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff338b9a700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159750 0x7ff3340d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff338b9a700 1 -- 172.21.15.29:0/4233656685 >> v1:172.21.15.29:6789/0 conn(0x7ff3340ce6d0 legacy=0x7ff33415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff338b9a700 1 -- 172.21.15.29:0/4233656685 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340cc7c0 con 0x7ff334159b20 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.614+0000 7ff33939b700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159750 0x7ff3340d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.615+0000 7ff338b9a700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334159b20 0x7ff3340ce190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff3240098e0 tx=0x7ff3240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dca031cfbfd1d97 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.615+0000 7ff32bfff700 1 -- 172.21.15.29:0/4233656685 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324011630 con 0x7ff334159b20 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.615+0000 7ff32bfff700 1 -- 172.21.15.29:0/4233656685 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324011790 con 0x7ff334159b20 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.615+0000 7ff32bfff700 1 -- 172.21.15.29:0/4233656685 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324011b10 con 0x7ff334159b20 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.615+0000 7ff33b5ff700 1 -- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334159b20 msgr2=0x7ff3340ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.615+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334159b20 0x7ff3340ce190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff3240098e0 tx=0x7ff3240091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.616+0000 7ff33b5ff700 1 -- 172.21.15.29:0/4233656685 shutdown_connections 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.616+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159750 0x7ff3340d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.616+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/4233656685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334159b20 0x7ff3340ce190 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.616+0000 7ff33b5ff700 1 -- 172.21.15.29:0/4233656685 >> 172.21.15.29:0/4233656685 conn(0x7ff334063a10 msgr2=0x7ff3340d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.616+0000 7ff33b5ff700 1 -- 172.21.15.29:0/4233656685 shutdown_connections 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.616+0000 7ff33b5ff700 1 -- 172.21.15.29:0/4233656685 wait complete. 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.617+0000 7ff33b5ff700 1 Processor -- start 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.617+0000 7ff33b5ff700 1 -- start start 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff33b5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3340ce6d0 0x7ff3340c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff33b5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 0x7ff334152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff33b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159b20 0x7ff3341582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff33b5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340c9c90 con 0x7ff334159750 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff33b5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3341525d0 con 0x7ff3340ce6d0 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff33b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340c8bf0 con 0x7ff334159b20 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.618+0000 7ff338b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 0x7ff334152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 0x7ff334152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44028/0 (socket says 172.21.15.29:44028) 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 -- 172.21.15.29:0/999261585 learned_addr learned my addr 172.21.15.29:0/999261585 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff33939b700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3340ce6d0 0x7ff3340c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff339b9c700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159b20 0x7ff3341582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 -- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159b20 msgr2=0x7ff3341582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159b20 0x7ff3341582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 -- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3340ce6d0 msgr2=0x7ff3340c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3340ce6d0 0x7ff3340c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 -- 172.21.15.29:0/999261585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3300093c0 con 0x7ff334159750 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.619+0000 7ff338b9a700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 0x7ff334152090 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff334088a40 tx=0x7ff324017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.620+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324011900 con 0x7ff334159750 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.620+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324011a60 con 0x7ff334159750 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.620+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32401a680 con 0x7ff334159750 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.620+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff324009420 con 0x7ff334159750 2023-10-07T09:47:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.620+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3340c91b0 con 0x7ff334159750 2023-10-07T09:47:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.621+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33404c3c0 con 0x7ff334159750 2023-10-07T09:47:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.623+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff32401a9f0 con 0x7ff334159750 2023-10-07T09:47:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.624+0000 7ff3297fa700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff318066f30 0x7ff3180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.624+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5981+0+0 (secure 0 0 0) 0x7ff324086610 con 0x7ff334159750 2023-10-07T09:47:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.624+0000 7ff33939b700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff318066f30 0x7ff3180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.625+0000 7ff33939b700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff318066f30 0x7ff3180693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff330009f30 tx=0x7ff330009120 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.628+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3240537f0 con 0x7ff334159750 2023-10-07T09:47:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:17.879+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff3340895a0 con 0x7ff334159750 2023-10-07T09:47:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.372+0000 7ff3297fa700 1 -- 172.21.15.29:0/999261585 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v56) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff324056e10 con 0x7ff334159750 2023-10-07T09:47:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.382+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff318066f30 msgr2=0x7ff3180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.382+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff318066f30 0x7ff3180693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff330009f30 tx=0x7ff330009120 comp rx=0 tx=0).stop 2023-10-07T09:47:18.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.382+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 msgr2=0x7ff334152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.382+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 0x7ff334152090 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff334088a40 tx=0x7ff324017000 comp rx=0 tx=0).stop 2023-10-07T09:47:18.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.384+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 shutdown_connections 2023-10-07T09:47:18.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.384+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff318066f30 0x7ff3180693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.384+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff334159b20 0x7ff3341582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.384+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3340ce6d0 0x7ff3340c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.384+0000 7ff33b5ff700 1 --2- 172.21.15.29:0/999261585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334159750 0x7ff334152090 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.384+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 >> 172.21.15.29:0/999261585 conn(0x7ff334063a10 msgr2=0x7ff334152960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:18.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.389+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 shutdown_connections 2023-10-07T09:47:18.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.389+0000 7ff33b5ff700 1 -- 172.21.15.29:0/999261585 wait complete. 2023-10-07T09:47:18.406 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 14 and data pool 15 2023-10-07T09:47:18.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:47:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.618+0000 7fa940bff700 1 Processor -- start 2023-10-07T09:47:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.618+0000 7fa940bff700 1 -- start start 2023-10-07T09:47:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa940bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa93c1533f0 0x7fa93c156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa940bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c159670 con 0x7fa93c0b04b0 2023-10-07T09:47:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa940bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c15a1f0 con 0x7fa93c1537c0 2023-10-07T09:47:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa940bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93c15a680 con 0x7fa93c1533f0 2023-10-07T09:47:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa93ad9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa93c0b04b0 0x7fa93c157240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51928/0 (socket says 172.21.15.29:51928) 2023-10-07T09:47:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa93ad9d700 1 -- 172.21.15.29:0/901897899 learned_addr learned my addr 172.21.15.29:0/901897899 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:18.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.620+0000 7fa93a59c700 1 --2- 172.21.15.29:0/901897899 >> v2:172.21.15.144:3300/0 conn(0x7fa93c1533f0 0x7fa93c156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471532916 0 0) 0x7fa93c159670 con 0x7fa93c0b04b0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa924003570 con 0x7fa93c0b04b0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281375045 0 0) 0x7fa93c15a1f0 con 0x7fa93c1537c0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93c159670 con 0x7fa93c1537c0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877769339 0 0) 0x7fa924003570 con 0x7fa93c0b04b0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 >> v2:172.21.15.144:3300/0 conn(0x7fa93c1533f0 msgr2=0x7fa93c156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 --2- 172.21.15.29:0/901897899 >> v2:172.21.15.144:3300/0 conn(0x7fa93c1533f0 0x7fa93c156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 >> v1:172.21.15.144:6789/0 conn(0x7fa93c1537c0 legacy=0x7fa93c0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c1598c0 con 0x7fa93c0b04b0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fa9340017f0 con 0x7fa93c0b04b0 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 >> v1:172.21.15.29:6789/0 conn(0x7fa93c0b04b0 legacy=0x7fa93c157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003c40 0x7fa924004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa924004810 0x7fa924004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa924005210 0x7fa92400dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92400e390 con 0x7fa924004810 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92400e500 con 0x7fa924003c40 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92400e6c0 con 0x7fa924005210 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.621+0000 7fa939d9b700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa924004810 0x7fa924004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa93a59c700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003c40 0x7fa924004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa93ad9d700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa924005210 0x7fa92400dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa939d9b700 1 -- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa924005210 msgr2=0x7fa92400dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa939d9b700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa924005210 0x7fa92400dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa939d9b700 1 -- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003c40 msgr2=0x7fa924004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa939d9b700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003c40 0x7fa924004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa939d9b700 1 -- 172.21.15.29:0/901897899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924003570 con 0x7fa924004810 2023-10-07T09:47:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa93ad9d700 1 -- 172.21.15.29:0/901897899 reap_dead start 2023-10-07T09:47:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa939d9b700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa924004810 0x7fa924004c80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa9300019e0 tx=0x7fa930007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c80e84374579f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93000c5d0 con 0x7fa924004810 2023-10-07T09:47:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93000c730 con 0x7fa924004810 2023-10-07T09:47:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.622+0000 7fa93959a700 1 -- 172.21.15.29:0/901897899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93000cad0 con 0x7fa924004810 2023-10-07T09:47:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 -- 172.21.15.29:0/901897899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa924004810 msgr2=0x7fa924004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa924004810 0x7fa924004c80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa9300019e0 tx=0x7fa930007d30 comp rx=0 tx=0).stop 2023-10-07T09:47:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 -- 172.21.15.29:0/901897899 shutdown_connections 2023-10-07T09:47:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 --2- 172.21.15.29:0/901897899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa924004810 0x7fa924004c80 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 -- 172.21.15.29:0/901897899 >> 172.21.15.29:0/901897899 conn(0x7fa93c0680a0 msgr2=0x7fa93c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 -- 172.21.15.29:0/901897899 shutdown_connections 2023-10-07T09:47:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.623+0000 7fa940bff700 1 -- 172.21.15.29:0/901897899 wait complete. 2023-10-07T09:47:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.624+0000 7fa940bff700 1 Processor -- start 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.625+0000 7fa940bff700 1 -- start start 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.625+0000 7fa940bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004810 0x7fa93c16a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa940bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 0x7fa93c151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa940bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93c1520f0 0x7fa93c16be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa940bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c13a4c0 con 0x7fa93c169790 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa940bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0c6a40 con 0x7fa924004810 2023-10-07T09:47:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa940bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c152850 con 0x7fa93c1520f0 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 0x7fa93c151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 0x7fa93c151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44042/0 (socket says 172.21.15.29:44042) 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 -- 172.21.15.29:0/2157931844 learned_addr learned my addr 172.21.15.29:0/2157931844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa93a59c700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004810 0x7fa93c16a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa93ad9d700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93c1520f0 0x7fa93c16be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 -- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93c1520f0 msgr2=0x7fa93c16be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93c1520f0 0x7fa93c16be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 -- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004810 msgr2=0x7fa93c16a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004810 0x7fa93c16a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa939d9b700 1 -- 172.21.15.29:0/2157931844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c152570 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.626+0000 7fa93ad9d700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93c1520f0 0x7fa93c16be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.627+0000 7fa939d9b700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 0x7fa93c151bb0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa93000c3c0 tx=0x7fa930015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.627+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93000ce40 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.627+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93000c8a0 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.627+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930006820 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.627+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c0c63f0 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.627+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c14f9c0 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.628+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93c0b1e70 con 0x7fa93c169790 2023-10-07T09:47:18.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.629+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa930019030 con 0x7fa93c169790 2023-10-07T09:47:18.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.630+0000 7fa92a7fc700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa91c066f30 0x7fa91c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:18.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.630+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fa930085960 con 0x7fa93c169790 2023-10-07T09:47:18.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.630+0000 7fa93a59c700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa91c066f30 0x7fa91c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:18.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.631+0000 7fa93a59c700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa91c066f30 0x7fa91c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa93c153da0 tx=0x7fa92c0091a0 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:18.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.634+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa930052a50 con 0x7fa93c169790 2023-10-07T09:47:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.835+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa93c14fbb0 con 0x7fa93c169790 2023-10-07T09:47:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.836+0000 7fa92a7fc700 1 -- 172.21.15.29:0/2157931844 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16044 (secure 0 0 0) 0x7fa930022090 con 0x7fa93c169790 2023-10-07T09:47:18.838 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:18.838 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":117,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:47:17.881487+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:47:15.663387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-07T09:47:16.857437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa91c066f30 msgr2=0x7fa91c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa91c066f30 0x7fa91c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa93c153da0 tx=0x7fa92c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 msgr2=0x7fa93c151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 0x7fa93c151bb0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa93000c3c0 tx=0x7fa930015000 comp rx=0 tx=0).stop 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 shutdown_connections 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa91c066f30 0x7fa91c0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93c1520f0 0x7fa93c16be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004810 0x7fa93c16a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 --2- 172.21.15.29:0/2157931844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa93c169790 0x7fa93c151bb0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 >> 172.21.15.29:0/2157931844 conn(0x7fa93c0680a0 msgr2=0x7fa93c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.838+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 shutdown_connections 2023-10-07T09:47:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:18.839+0000 7fa940bff700 1 -- 172.21.15.29:0/2157931844 wait complete. 2023-10-07T09:47:18.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:47:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.053+0000 7f82f91ff700 1 Processor -- start 2023-10-07T09:47:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.053+0000 7f82f91ff700 1 -- start start 2023-10-07T09:47:19.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f91ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f91ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82f40b1a10 0x7f82f4157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f91ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f4158ad0 con 0x7f82f40ae8b0 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f91ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f415b140 con 0x7f82f40b1a10 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f91ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f415b420 con 0x7f82f4157290 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f359e700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82f40b1a10 0x7f82f4157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f259c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f82f40ae8b0 0x7f82f40b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51940/0 (socket says 172.21.15.29:51940) 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.054+0000 7f82f259c700 1 -- 172.21.15.29:0/1493659137 learned_addr learned my addr 172.21.15.29:0/1493659137 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:19.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f2d9d700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f1d9b700 1 -- 172.21.15.29:0/1493659137 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190012746 0 0) 0x7f82f4158ad0 con 0x7f82f40ae8b0 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f1d9b700 1 -- 172.21.15.29:0/1493659137 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82dc003570 con 0x7f82f40ae8b0 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f359e700 1 -- 172.21.15.29:0/1493659137 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 msgr2=0x7f82f40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f359e700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f359e700 1 -- 172.21.15.29:0/1493659137 >> v1:172.21.15.29:6789/0 conn(0x7f82f40ae8b0 legacy=0x7f82f40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f359e700 1 -- 172.21.15.29:0/1493659137 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f415b700 con 0x7f82f40b1a10 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f2d9d700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:19.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f359e700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f4157660 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009e60 tx=0x7f82ec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cd5bbcdb4e9ff5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f1d9b700 1 -- 172.21.15.29:0/1493659137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00e070 con 0x7f82f40b1a10 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.055+0000 7f82f1d9b700 1 -- 172.21.15.29:0/1493659137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec00bb60 con 0x7f82f40b1a10 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f1d9b700 1 -- 172.21.15.29:0/1493659137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec012430 con 0x7f82f40b1a10 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 -- 172.21.15.29:0/1493659137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 msgr2=0x7f82f4157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f4157660 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009e60 tx=0x7f82ec009780 comp rx=0 tx=0).stop 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 -- 172.21.15.29:0/1493659137 shutdown_connections 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f4157660 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1493659137 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 -- 172.21.15.29:0/1493659137 >> 172.21.15.29:0/1493659137 conn(0x7f82f40680a0 msgr2=0x7f82f40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 -- 172.21.15.29:0/1493659137 shutdown_connections 2023-10-07T09:47:19.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.056+0000 7f82f91ff700 1 -- 172.21.15.29:0/1493659137 wait complete. 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.058+0000 7f82f91ff700 1 Processor -- start 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.058+0000 7f82f91ff700 1 -- start start 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f91ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f415a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f91ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f414d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f91ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82f414de60 0x7f82f4158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f91ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4157900 con 0x7f82f40b1a10 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f91ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41581a0 con 0x7f82f414de60 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f91ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f414e6e0 con 0x7f82f4157290 2023-10-07T09:47:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f415a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f415a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44068/0 (socket says 172.21.15.29:44068) 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f2d9d700 1 -- 172.21.15.29:0/1373468461 learned_addr learned my addr 172.21.15.29:0/1373468461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f259c700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f414d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f2d9d700 1 -- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 msgr2=0x7f82f414d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.059+0000 7f82f359e700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82f414de60 0x7f82f4158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f2d9d700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f414d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f2d9d700 1 -- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82f414de60 msgr2=0x7f82f4158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f2d9d700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82f414de60 0x7f82f4158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f2d9d700 1 -- 172.21.15.29:0/1373468461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009a20 con 0x7f82f40b1a10 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f359e700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82f414de60 0x7f82f4158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f259c700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f414d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f2d9d700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f415a120 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f82e400d860 tx=0x7f82e4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e40125f0 con 0x7f82f40b1a10 2023-10-07T09:47:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e4012750 con 0x7f82f40b1a10 2023-10-07T09:47:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4012ad0 con 0x7f82f40b1a10 2023-10-07T09:47:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f4157b50 con 0x7f82f40b1a10 2023-10-07T09:47:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.060+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f415a660 con 0x7f82f40b1a10 2023-10-07T09:47:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.061+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f4158480 con 0x7f82f40b1a10 2023-10-07T09:47:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.062+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f82e4014430 con 0x7f82f40b1a10 2023-10-07T09:47:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.063+0000 7f82daffd700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82e0066f30 0x7f82e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.063+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f82e4084ba0 con 0x7f82f40b1a10 2023-10-07T09:47:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.064+0000 7f82f259c700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82e0066f30 0x7f82e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.065+0000 7f82f259c700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82e0066f30 0x7f82e00693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82e8007fd0 tx=0x7f82e8009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:19.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.067+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f82e4051bf0 con 0x7f82f40b1a10 2023-10-07T09:47:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.266+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f4064de0 con 0x7f82f40b1a10 2023-10-07T09:47:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.267+0000 7f82daffd700 1 -- 172.21.15.29:0/1373468461 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16044 (secure 0 0 0) 0x7f82e4055210 con 0x7f82f40b1a10 2023-10-07T09:47:19.269 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:19.269 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":117,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:47:17.881487+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:47:15.663387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-07T09:47:16.857437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:47:19.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82e0066f30 msgr2=0x7f82e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82e0066f30 0x7f82e00693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82e8007fd0 tx=0x7f82e8009040 comp rx=0 tx=0).stop 2023-10-07T09:47:19.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 msgr2=0x7f82f415a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f415a120 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f82e400d860 tx=0x7f82e4008040 comp rx=0 tx=0).stop 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 shutdown_connections 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f82e0066f30 0x7f82e00693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82f414de60 0x7f82f4158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f40b1a10 0x7f82f415a120 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 --2- 172.21.15.29:0/1373468461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82f4157290 0x7f82f414d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 >> 172.21.15.29:0/1373468461 conn(0x7f82f40680a0 msgr2=0x7f82f41527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 shutdown_connections 2023-10-07T09:47:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.270+0000 7f82f91ff700 1 -- 172.21.15.29:0/1373468461 wait complete. 2023-10-07T09:47:19.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:47:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.493+0000 7f1d155ff700 1 Processor -- start 2023-10-07T09:47:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.493+0000 7f1d155ff700 1 -- start start 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d155ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d155ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d100cbf20 con 0x7f1d10159750 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d155ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d100cc200 con 0x7f1d100ce8a0 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d155ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100cc4e0 con 0x7f1d10159b20 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d0effd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d10159750 0x7f1d100ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51956/0 (socket says 172.21.15.29:51956) 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d0effd700 1 -- 172.21.15.29:0/145809511 learned_addr learned my addr 172.21.15.29:0/145809511 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d0e7fc700 1 --2- 172.21.15.29:0/145809511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889489240 0 0) 0x7f1d100cbf20 con 0x7f1d10159750 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf8003570 con 0x7f1d10159750 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.495+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634331240 0 0) 0x7f1d100cc200 con 0x7f1d100ce8a0 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d100cbf20 con 0x7f1d100ce8a0 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/145809511 >> v1:172.21.15.144:6789/0 conn(0x7f1d100ce8a0 legacy=0x7f1d100c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/145809511 >> v1:172.21.15.29:6789/0 conn(0x7f1d10159750 legacy=0x7f1d100ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/145809511 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d100cc7c0 con 0x7f1d10159b20 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0e7fc700 1 --2- 172.21.15.29:0/145809511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100d10a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009910 tx=0x7f1d040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f265acdd98dd1b4d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04011630 con 0x7f1d10159b20 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d04011790 con 0x7f1d10159b20 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.496+0000 7f1d0dffb700 1 -- 172.21.15.29:0/145809511 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04011b30 con 0x7f1d10159b20 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 -- 172.21.15.29:0/145809511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 msgr2=0x7f1d100d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 --2- 172.21.15.29:0/145809511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100d10a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009910 tx=0x7f1d040091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 -- 172.21.15.29:0/145809511 shutdown_connections 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 --2- 172.21.15.29:0/145809511 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100d10a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 -- 172.21.15.29:0/145809511 >> 172.21.15.29:0/145809511 conn(0x7f1d10063a10 msgr2=0x7f1d100d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 -- 172.21.15.29:0/145809511 shutdown_connections 2023-10-07T09:47:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.497+0000 7f1d155ff700 1 -- 172.21.15.29:0/145809511 wait complete. 2023-10-07T09:47:19.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.498+0000 7f1d155ff700 1 Processor -- start 2023-10-07T09:47:19.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.499+0000 7f1d155ff700 1 -- start start 2023-10-07T09:47:19.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.499+0000 7f1d155ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d100ce8a0 0x7f1d1014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.499+0000 7f1d155ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 0x7f1d100c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d155ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d155ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100b3690 con 0x7f1d10159750 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d155ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100b3970 con 0x7f1d100ce8a0 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d155ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100b3c50 con 0x7f1d10159b20 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 0x7f1d100c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0effd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d100ce8a0 0x7f1d1014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 0x7f1d100c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44072/0 (socket says 172.21.15.29:44072) 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/1584429211 learned_addr learned my addr 172.21.15.29:0/1584429211 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0f7fe700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 msgr2=0x7f1d100ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d100ce8a0 msgr2=0x7f1d1014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:19.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d100ce8a0 0x7f1d1014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:19.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0e7fc700 1 -- 172.21.15.29:0/1584429211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04009420 con 0x7f1d10159750 2023-10-07T09:47:19.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.500+0000 7f1d0f7fe700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:19.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.501+0000 7f1d0e7fc700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 0x7f1d100c9d60 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009f00 tx=0x7f1d04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.501+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0401f070 con 0x7f1d10159750 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.501+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d04000bc0 con 0x7f1d10159750 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.501+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d100b3f30 con 0x7f1d10159750 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.501+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04025420 con 0x7f1d10159750 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.501+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1014b630 con 0x7f1d10159750 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.502+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1004c3c0 con 0x7f1d10159750 2023-10-07T09:47:19.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.503+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1d04023070 con 0x7f1d10159750 2023-10-07T09:47:19.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.504+0000 7f1cf6ffd700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cfc066f30 0x7f1cfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:19.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.504+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1d04085910 con 0x7f1d10159750 2023-10-07T09:47:19.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.504+0000 7f1d0effd700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cfc066f30 0x7f1cfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:19.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.505+0000 7f1d0effd700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cfc066f30 0x7f1cfc0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1d10069f20 tx=0x7f1d00009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:19.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.508+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d04052a00 con 0x7f1d10159750 2023-10-07T09:47:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:19.762+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1d10064040 con 0x7f1d10159750 2023-10-07T09:47:20.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.399+0000 7f1cf6ffd700 1 -- 172.21.15.29:0/1584429211 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v59) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1d0402a0b0 con 0x7f1d10159750 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cfc066f30 msgr2=0x7f1cfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cfc066f30 0x7f1cfc0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1d10069f20 tx=0x7f1d00009040 comp rx=0 tx=0).stop 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 msgr2=0x7f1d100c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 0x7f1d100c9d60 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009f00 tx=0x7f1d04017000 comp rx=0 tx=0).stop 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 shutdown_connections 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1cfc066f30 0x7f1cfc0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d10159b20 0x7f1d100ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d100ce8a0 0x7f1d1014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.402+0000 7f1d155ff700 1 --2- 172.21.15.29:0/1584429211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10159750 0x7f1d100c9d60 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.403+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 >> 172.21.15.29:0/1584429211 conn(0x7f1d10063a10 msgr2=0x7f1d100c5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.403+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 shutdown_connections 2023-10-07T09:47:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.403+0000 7f1d155ff700 1 -- 172.21.15.29:0/1584429211 wait complete. 2023-10-07T09:47:20.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:47:20.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.626+0000 7f31209ff700 1 Processor -- start 2023-10-07T09:47:20.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.627+0000 7f31209ff700 1 -- start start 2023-10-07T09:47:20.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f31209ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f311c156180 0x7f311c0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:20.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f31209ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0ad4d0 con 0x7f311c0ae570 2023-10-07T09:47:20.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f31209ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0ad7b0 con 0x7f311c0b0520 2023-10-07T09:47:20.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f31209ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0ada90 con 0x7f311c156180 2023-10-07T09:47:20.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f311a59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f311c156180 0x7f311c0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:20.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f3119d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f311c0ae570 0x7f311c0accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51972/0 (socket says 172.21.15.29:51972) 2023-10-07T09:47:20.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.628+0000 7f3119d9b700 1 -- 172.21.15.29:0/2449098981 learned_addr learned my addr 172.21.15.29:0/2449098981 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:20.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311959a700 1 -- 172.21.15.29:0/2449098981 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208739412 0 0) 0x7f311c0ad4d0 con 0x7f311c0ae570 2023-10-07T09:47:20.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311a59c700 1 -- 172.21.15.29:0/2449098981 >> v1:172.21.15.144:6789/0 conn(0x7f311c0b0520 legacy=0x7f311c0b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311a59c700 1 -- 172.21.15.29:0/2449098981 >> v1:172.21.15.29:6789/0 conn(0x7f311c0ae570 legacy=0x7f311c0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311a59c700 1 -- 172.21.15.29:0/2449098981 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0add70 con 0x7f311c156180 2023-10-07T09:47:20.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311a59c700 1 --2- 172.21.15.29:0/2449098981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c156180 0x7f311c0afc00 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f310c0098e0 tx=0x7f310c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff2b00aec3ffa62 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:20.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311959a700 1 -- 172.21.15.29:0/2449098981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c009e80 con 0x7f311c156180 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311959a700 1 -- 172.21.15.29:0/2449098981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c00b860 con 0x7f311c156180 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.629+0000 7f311959a700 1 -- 172.21.15.29:0/2449098981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c00bbe0 con 0x7f311c156180 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 -- 172.21.15.29:0/2449098981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c156180 msgr2=0x7f311c0afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 --2- 172.21.15.29:0/2449098981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c156180 0x7f311c0afc00 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f310c0098e0 tx=0x7f310c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 -- 172.21.15.29:0/2449098981 shutdown_connections 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 --2- 172.21.15.29:0/2449098981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c156180 0x7f311c0afc00 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 -- 172.21.15.29:0/2449098981 >> 172.21.15.29:0/2449098981 conn(0x7f311c067610 msgr2=0x7f311c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 -- 172.21.15.29:0/2449098981 shutdown_connections 2023-10-07T09:47:20.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.630+0000 7f31209ff700 1 -- 172.21.15.29:0/2449098981 wait complete. 2023-10-07T09:47:20.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.631+0000 7f31209ff700 1 Processor -- start 2023-10-07T09:47:20.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 -- start start 2023-10-07T09:47:20.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f311c0ae570 0x7f311c0caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:20.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 0x7f311c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:20.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f311c156180 0x7f311c169e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c16bd80 con 0x7f311c0b0520 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c147cd0 con 0x7f311c0ae570 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.632+0000 7f31209ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c16b170 con 0x7f311c156180 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 0x7f311c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 0x7f311c0cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44086/0 (socket says 172.21.15.29:44086) 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 -- 172.21.15.29:0/2774447368 learned_addr learned my addr 172.21.15.29:0/2774447368 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f311a59c700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f311c0ae570 0x7f311c0caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f311ad9d700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f311c156180 0x7f311c169e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 -- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f311c156180 msgr2=0x7f311c169e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f311c156180 0x7f311c169e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 -- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f311c0ae570 msgr2=0x7f311c0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f311c0ae570 0x7f311c0caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.633+0000 7f3119d9b700 1 -- 172.21.15.29:0/2774447368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c009420 con 0x7f311c0b0520 2023-10-07T09:47:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.634+0000 7f3119d9b700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 0x7f311c0cb440 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f311000c810 tx=0x7f3110007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.634+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110005c00 con 0x7f311c0b0520 2023-10-07T09:47:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.634+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110005d60 con 0x7f311c0b0520 2023-10-07T09:47:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.634+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c16b450 con 0x7f311c0b0520 2023-10-07T09:47:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.634+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110007320 con 0x7f311c0b0520 2023-10-07T09:47:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.634+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c16ba10 con 0x7f311c0b0520 2023-10-07T09:47:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.635+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c0c8350 con 0x7f311c0b0520 2023-10-07T09:47:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.636+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f311001b3b0 con 0x7f311c0b0520 2023-10-07T09:47:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.637+0000 7f310a7fc700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f30fc066f30 0x7f30fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.637+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3110083ba0 con 0x7f311c0b0520 2023-10-07T09:47:20.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.637+0000 7f311a59c700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f30fc066f30 0x7f30fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:20.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.638+0000 7f311a59c700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f30fc066f30 0x7f30fc0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f310c0093f0 tx=0x7f310c017000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:20.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.641+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3110050bf0 con 0x7f311c0b0520 2023-10-07T09:47:20.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:20.893+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f311c04c3c0 con 0x7f311c0b0520 2023-10-07T09:47:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.401+0000 7f310a7fc700 1 -- 172.21.15.29:0/2774447368 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v60) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3110017020 con 0x7f311c0b0520 2023-10-07T09:47:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f30fc066f30 msgr2=0x7f30fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f30fc066f30 0x7f30fc0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f310c0093f0 tx=0x7f310c017000 comp rx=0 tx=0).stop 2023-10-07T09:47:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 msgr2=0x7f311c0cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 0x7f311c0cb440 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f311000c810 tx=0x7f3110007d90 comp rx=0 tx=0).stop 2023-10-07T09:47:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 shutdown_connections 2023-10-07T09:47:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f30fc066f30 0x7f30fc0693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f311c156180 0x7f311c169e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f311c0ae570 0x7f311c0caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 --2- 172.21.15.29:0/2774447368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311c0b0520 0x7f311c0cb440 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 >> 172.21.15.29:0/2774447368 conn(0x7f311c067610 msgr2=0x7f311c0aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.404+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 shutdown_connections 2023-10-07T09:47:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.405+0000 7f31209ff700 1 -- 172.21.15.29:0/2774447368 wait complete. 2023-10-07T09:47:21.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:47:21.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.622+0000 7fa0227ff700 1 Processor -- start 2023-10-07T09:47:21.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.622+0000 7fa0227ff700 1 -- start start 2023-10-07T09:47:21.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa0227ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa01c157290 0x7fa01c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:21.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa0227ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c0b1a10 0x7fa01c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:21.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa0227ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c158ad0 con 0x7fa01c0ae8b0 2023-10-07T09:47:21.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa0227ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c15b140 con 0x7fa01c157290 2023-10-07T09:47:21.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa0227ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c15b420 con 0x7fa01c0b1a10 2023-10-07T09:47:21.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa01b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa01c0ae8b0 0x7fa01c0b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51974/0 (socket says 172.21.15.29:51974) 2023-10-07T09:47:21.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa01b7fe700 1 -- 172.21.15.29:0/357366897 learned_addr learned my addr 172.21.15.29:0/357366897 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:21.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa01bfff700 1 --2- 172.21.15.29:0/357366897 >> v2:172.21.15.29:3300/0 conn(0x7fa01c157290 0x7fa01c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:21.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa020d9c700 1 --2- 172.21.15.29:0/357366897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c0b1a10 0x7fa01c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:21.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.624+0000 7fa01affd700 1 -- 172.21.15.29:0/357366897 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616092626 0 0) 0x7fa01c158ad0 con 0x7fa01c0ae8b0 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01affd700 1 -- 172.21.15.29:0/357366897 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa004003570 con 0x7fa01c0ae8b0 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01bfff700 1 -- 172.21.15.29:0/357366897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c0b1a10 msgr2=0x7fa01c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01bfff700 1 --2- 172.21.15.29:0/357366897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c0b1a10 0x7fa01c157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01bfff700 1 -- 172.21.15.29:0/357366897 >> v1:172.21.15.29:6789/0 conn(0x7fa01c0ae8b0 legacy=0x7fa01c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01bfff700 1 -- 172.21.15.29:0/357366897 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c15b700 con 0x7fa01c157290 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01bfff700 1 --2- 172.21.15.29:0/357366897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c157290 0x7fa01c0cc870 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0098e0 tx=0x7fa00c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f8b9a314e66922 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01affd700 1 -- 172.21.15.29:0/357366897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c011630 con 0x7fa01c157290 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01affd700 1 -- 172.21.15.29:0/357366897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c011790 con 0x7fa01c157290 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.625+0000 7fa01affd700 1 -- 172.21.15.29:0/357366897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c011b10 con 0x7fa01c157290 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 -- 172.21.15.29:0/357366897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c157290 msgr2=0x7fa01c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 --2- 172.21.15.29:0/357366897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c157290 0x7fa01c0cc870 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0098e0 tx=0x7fa00c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 -- 172.21.15.29:0/357366897 shutdown_connections 2023-10-07T09:47:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 --2- 172.21.15.29:0/357366897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c0b1a10 0x7fa01c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 --2- 172.21.15.29:0/357366897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c157290 0x7fa01c0cc870 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 -- 172.21.15.29:0/357366897 >> 172.21.15.29:0/357366897 conn(0x7fa01c0680a0 msgr2=0x7fa01c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 -- 172.21.15.29:0/357366897 shutdown_connections 2023-10-07T09:47:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.626+0000 7fa0227ff700 1 -- 172.21.15.29:0/357366897 wait complete. 2023-10-07T09:47:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.628+0000 7fa0227ff700 1 Processor -- start 2023-10-07T09:47:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.628+0000 7fa0227ff700 1 -- start start 2023-10-07T09:47:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa0227ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa01c0b1a10 0x7fa01c0c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa0227ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c157290 0x7fa01c0b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa0227ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 0x7fa01c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa0227ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c14b540 con 0x7fa01c16aef0 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa0227ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0c6470 con 0x7fa01c0b1a10 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa0227ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c154e10 con 0x7fa01c157290 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 0x7fa01c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 0x7fa01c0c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44104/0 (socket says 172.21.15.29:44104) 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 -- 172.21.15.29:0/3947748415 learned_addr learned my addr 172.21.15.29:0/3947748415 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa01b7fe700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c157290 0x7fa01c0b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa01bfff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa01c0b1a10 0x7fa01c0c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 -- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c157290 msgr2=0x7fa01c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c157290 0x7fa01c0b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 -- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa01c0b1a10 msgr2=0x7fa01c0c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa01c0b1a10 0x7fa01c0c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.629+0000 7fa020d9c700 1 -- 172.21.15.29:0/3947748415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c009420 con 0x7fa01c16aef0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.630+0000 7fa020d9c700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 0x7fa01c0c6980 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fa014009ef0 tx=0x7fa01400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.630+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01400cb70 con 0x7fa01c16aef0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.630+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01400f040 con 0x7fa01c16aef0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.630+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014014480 con 0x7fa01c16aef0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.630+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c1550f0 con 0x7fa01c16aef0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.630+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c1556b0 con 0x7fa01c16aef0 2023-10-07T09:47:21.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.631+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c1558d0 con 0x7fa01c16aef0 2023-10-07T09:47:21.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.632+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa014014830 con 0x7fa01c16aef0 2023-10-07T09:47:21.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.633+0000 7fa003fff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa008066f30 0x7fa0080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:21.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.633+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fa0140852d0 con 0x7fa01c16aef0 2023-10-07T09:47:21.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.634+0000 7fa01bfff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa008066f30 0x7fa0080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:21.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.634+0000 7fa01bfff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa008066f30 0x7fa0080693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0091e0 tx=0x7fa00c017000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:21.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.637+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0140523c0 con 0x7fa01c16aef0 2023-10-07T09:47:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.881+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa01c04c3c0 con 0x7fa01c16aef0 2023-10-07T09:47:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.882+0000 7fa003fff700 1 -- 172.21.15.29:0/3947748415 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fa0140559e0 con 0x7fa01c16aef0 2023-10-07T09:47:21.883 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:21.883 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:47:17.881832+0000","modified":"2023-10-07T09:47:21.398903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/581264510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":581264510},{"type":"v1","addr":"172.21.15.29:6835","nonce":581264510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-10-07T09:47:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.884+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa008066f30 msgr2=0x7fa0080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.884+0000 7fa0227ff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa008066f30 0x7fa0080693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0091e0 tx=0x7fa00c017000 comp rx=0 tx=0).stop 2023-10-07T09:47:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 msgr2=0x7fa01c0c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 0x7fa01c0c6980 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fa014009ef0 tx=0x7fa01400b040 comp rx=0 tx=0).stop 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 shutdown_connections 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa008066f30 0x7fa0080693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa01c16aef0 0x7fa01c0c6980 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa01c0b1a10 0x7fa01c0c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 --2- 172.21.15.29:0/3947748415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa01c157290 0x7fa01c0b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 >> 172.21.15.29:0/3947748415 conn(0x7fa01c0680a0 msgr2=0x7fa01c0c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 shutdown_connections 2023-10-07T09:47:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:21.885+0000 7fa0227ff700 1 -- 172.21.15.29:0/3947748415 wait complete. 2023-10-07T09:47:21.888 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-07T09:47:21.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:47:22.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:47:22.105 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:47:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.098+0000 7fdc797ff700 1 Processor -- start 2023-10-07T09:47:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.098+0000 7fdc797ff700 1 -- start start 2023-10-07T09:47:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.100+0000 7fdc797ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc74159750 0x7fdc740d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.100+0000 7fdc797ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdc74159b20 0x7fdc740ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.100+0000 7fdc797ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc740ce6d0 0x7fdc740ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.100+0000 7fdc797ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc740cbf20 con 0x7fdc74159750 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc797ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc740cc200 con 0x7fdc74159b20 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc797ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc740cc4e0 con 0x7fdc740ce6d0 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc74159750 0x7fdc740d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc74159750 0x7fdc740d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44120/0 (socket says 172.21.15.29:44120) 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 -- 172.21.15.29:0/2550247036 learned_addr learned my addr 172.21.15.29:0/2550247036 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc727fc700 1 --2- 172.21.15.29:0/2550247036 >> v2:172.21.15.144:3300/0 conn(0x7fdc74159b20 0x7fdc740ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc737fe700 1 --2- 172.21.15.29:0/2550247036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc740ce6d0 0x7fdc740ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 -- 172.21.15.29:0/2550247036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc740ce6d0 msgr2=0x7fdc740ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:22.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 --2- 172.21.15.29:0/2550247036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc740ce6d0 0x7fdc740ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 -- 172.21.15.29:0/2550247036 >> v2:172.21.15.144:3300/0 conn(0x7fdc74159b20 msgr2=0x7fdc740ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 --2- 172.21.15.29:0/2550247036 >> v2:172.21.15.144:3300/0 conn(0x7fdc74159b20 0x7fdc740ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 -- 172.21.15.29:0/2550247036 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc740cc7c0 con 0x7fdc74159750 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc727fc700 1 --2- 172.21.15.29:0/2550247036 >> v2:172.21.15.144:3300/0 conn(0x7fdc74159b20 0x7fdc740ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.101+0000 7fdc72ffd700 1 --2- 172.21.15.29:0/2550247036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740d10a0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fdc74088a70 tx=0x7fdc64009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59406ebbc54b987e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc71ffb700 1 -- 172.21.15.29:0/2550247036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64005c10 con 0x7fdc74159750 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc71ffb700 1 -- 172.21.15.29:0/2550247036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc64005d70 con 0x7fdc74159750 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc71ffb700 1 -- 172.21.15.29:0/2550247036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400b880 con 0x7fdc74159750 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc797ff700 1 -- 172.21.15.29:0/2550247036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 msgr2=0x7fdc740d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc797ff700 1 --2- 172.21.15.29:0/2550247036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740d10a0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fdc74088a70 tx=0x7fdc64009220 comp rx=0 tx=0).stop 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc797ff700 1 -- 172.21.15.29:0/2550247036 shutdown_connections 2023-10-07T09:47:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc797ff700 1 --2- 172.21.15.29:0/2550247036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc740ce6d0 0x7fdc740ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc797ff700 1 --2- 172.21.15.29:0/2550247036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740d10a0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.102+0000 7fdc797ff700 1 --2- 172.21.15.29:0/2550247036 >> v2:172.21.15.144:3300/0 conn(0x7fdc74159b20 0x7fdc740ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.103+0000 7fdc797ff700 1 -- 172.21.15.29:0/2550247036 >> 172.21.15.29:0/2550247036 conn(0x7fdc74063a10 msgr2=0x7fdc740d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.103+0000 7fdc797ff700 1 -- 172.21.15.29:0/2550247036 shutdown_connections 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.103+0000 7fdc797ff700 1 -- 172.21.15.29:0/2550247036 wait complete. 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.104+0000 7fdc797ff700 1 Processor -- start 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.104+0000 7fdc797ff700 1 -- start start 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.105+0000 7fdc797ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc740ce6d0 0x7fdc74153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.105+0000 7fdc797ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.105+0000 7fdc797ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc74159b20 0x7fdc740c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.105+0000 7fdc797ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7415a720 con 0x7fdc74159750 2023-10-07T09:47:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.105+0000 7fdc797ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc740c85f0 con 0x7fdc740ce6d0 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.105+0000 7fdc797ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc740c79e0 con 0x7fdc74159b20 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44130/0 (socket says 172.21.15.29:44130) 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 -- 172.21.15.29:0/1723351027 learned_addr learned my addr 172.21.15.29:0/1723351027 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc72ffd700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc740ce6d0 0x7fdc74153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc737fe700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc74159b20 0x7fdc740c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 -- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc74159b20 msgr2=0x7fdc740c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc74159b20 0x7fdc740c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 -- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc740ce6d0 msgr2=0x7fdc74153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc740ce6d0 0x7fdc74153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 -- 172.21.15.29:0/1723351027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc64009480 con 0x7fdc74159750 2023-10-07T09:47:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.106+0000 7fdc727fc700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740c6820 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800d860 tx=0x7fdc68008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.107+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680125f0 con 0x7fdc74159750 2023-10-07T09:47:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.107+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc68012750 con 0x7fdc74159750 2023-10-07T09:47:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.107+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc740c7cc0 con 0x7fdc74159750 2023-10-07T09:47:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.107+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68012af0 con 0x7fdc74159750 2023-10-07T09:47:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.107+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc740c8300 con 0x7fdc74159750 2023-10-07T09:47:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.108+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc740c87a0 con 0x7fdc74159750 2023-10-07T09:47:22.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.109+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdc68013480 con 0x7fdc74159750 2023-10-07T09:47:22.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.110+0000 7fdc5affd700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdc60066f30 0x7fdc600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.110+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fdc680847b0 con 0x7fdc74159750 2023-10-07T09:47:22.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.110+0000 7fdc72ffd700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdc60066f30 0x7fdc600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.111+0000 7fdc72ffd700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdc60066f30 0x7fdc600693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdc740b4570 tx=0x7fdc64016000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:22.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.114+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc740c87a0 con 0x7fdc74159750 2023-10-07T09:47:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.318+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc741542c0 con 0x7fdc74159750 2023-10-07T09:47:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.319+0000 7fdc5affd700 1 -- 172.21.15.29:0/1723351027 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16031 (secure 0 0 0) 0x7fdc680518a0 con 0x7fdc74159750 2023-10-07T09:47:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:22.320 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":118,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:47:19.385184+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:47:15.663387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-07T09:47:16.857437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:47:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdc60066f30 msgr2=0x7fdc600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdc60066f30 0x7fdc600693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdc740b4570 tx=0x7fdc64016000 comp rx=0 tx=0).stop 2023-10-07T09:47:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 msgr2=0x7fdc740c6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740c6820 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800d860 tx=0x7fdc68008040 comp rx=0 tx=0).stop 2023-10-07T09:47:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 shutdown_connections 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdc60066f30 0x7fdc600693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc74159b20 0x7fdc740c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc740ce6d0 0x7fdc74153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 --2- 172.21.15.29:0/1723351027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc74159750 0x7fdc740c6820 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 >> 172.21.15.29:0/1723351027 conn(0x7fdc74063a10 msgr2=0x7fdc740ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 shutdown_connections 2023-10-07T09:47:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.322+0000 7fdc797ff700 1 -- 172.21.15.29:0/1723351027 wait complete. 2023-10-07T09:47:22.341 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:47:22.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:47:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.537+0000 7f7826dff700 1 Processor -- start 2023-10-07T09:47:22.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.537+0000 7f7826dff700 1 -- start start 2023-10-07T09:47:22.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.538+0000 7f7826dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78200c6470 0x7f78200d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.538+0000 7f7826dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78200cefb0 con 0x7f7820159b20 2023-10-07T09:47:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.538+0000 7f7826dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78200cbf20 con 0x7f7820159750 2023-10-07T09:47:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.538+0000 7f7826dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78200cc200 con 0x7f78200c6470 2023-10-07T09:47:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f781ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7820159b20 0x7f78200ce8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51978/0 (socket says 172.21.15.29:51978) 2023-10-07T09:47:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f781ffff700 1 -- 172.21.15.29:0/3846978668 learned_addr learned my addr 172.21.15.29:0/3846978668 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f782539c700 1 --2- 172.21.15.29:0/3846978668 >> v2:172.21.15.29:3300/0 conn(0x7f78200c6470 0x7f78200d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f781f7fe700 1 -- 172.21.15.29:0/3846978668 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102036293 0 0) 0x7f78200cefb0 con 0x7f7820159b20 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f781f7fe700 1 -- 172.21.15.29:0/3846978668 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810003570 con 0x7f7820159b20 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f782539c700 1 -- 172.21.15.29:0/3846978668 >> v1:172.21.15.144:6789/0 conn(0x7f7820159750 legacy=0x7f78200ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f782539c700 1 -- 172.21.15.29:0/3846978668 >> v1:172.21.15.29:6789/0 conn(0x7f7820159b20 legacy=0x7f78200ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.539+0000 7f782539c700 1 -- 172.21.15.29:0/3846978668 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78200cc4e0 con 0x7f78200c6470 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f782539c700 1 --2- 172.21.15.29:0/3846978668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f78200d10a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7818009e60 tx=0x7f7818009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ace0c667004bc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f781f7fe700 1 -- 172.21.15.29:0/3846978668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781800e070 con 0x7f78200c6470 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f781f7fe700 1 -- 172.21.15.29:0/3846978668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781800bb60 con 0x7f78200c6470 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f781f7fe700 1 -- 172.21.15.29:0/3846978668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818012430 con 0x7f78200c6470 2023-10-07T09:47:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f7826dff700 1 -- 172.21.15.29:0/3846978668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 msgr2=0x7f78200d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f7826dff700 1 --2- 172.21.15.29:0/3846978668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f78200d10a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7818009e60 tx=0x7f7818009780 comp rx=0 tx=0).stop 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f7826dff700 1 -- 172.21.15.29:0/3846978668 shutdown_connections 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f7826dff700 1 --2- 172.21.15.29:0/3846978668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f78200d10a0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.540+0000 7f7826dff700 1 -- 172.21.15.29:0/3846978668 >> 172.21.15.29:0/3846978668 conn(0x7f7820063a10 msgr2=0x7f78200d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.541+0000 7f7826dff700 1 -- 172.21.15.29:0/3846978668 shutdown_connections 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.541+0000 7f7826dff700 1 -- 172.21.15.29:0/3846978668 wait complete. 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.542+0000 7f7826dff700 1 Processor -- start 2023-10-07T09:47:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.542+0000 7f7826dff700 1 -- start start 2023-10-07T09:47:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7826dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f7820151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7826dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7820159750 0x7f78200c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7826dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7820159b20 0x7f78200c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7826dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78200b61f0 con 0x7f78200c6470 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7826dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7820156d70 con 0x7f7820159b20 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7826dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78200b5670 con 0x7f7820159750 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7824b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f7820151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7824b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f7820151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44160/0 (socket says 172.21.15.29:44160) 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f7824b9b700 1 -- 172.21.15.29:0/1384576764 learned_addr learned my addr 172.21.15.29:0/1384576764 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.543+0000 7f782539c700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7820159b20 0x7f78200c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f781ffff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7820159750 0x7f78200c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7824b9b700 1 -- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7820159750 msgr2=0x7f78200c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7824b9b700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7820159750 0x7f78200c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7824b9b700 1 -- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7820159b20 msgr2=0x7f78200c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7824b9b700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7820159b20 0x7f78200c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7824b9b700 1 -- 172.21.15.29:0/1384576764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818009a20 con 0x7f78200c6470 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f782539c700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7820159b20 0x7f78200c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7824b9b700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f7820151d40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f781400c7c0 tx=0x7f7814001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814006870 con 0x7f78200c6470 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78140069d0 con 0x7f78200c6470 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78200b5950 con 0x7f78200c6470 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814006d90 con 0x7f78200c6470 2023-10-07T09:47:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.544+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78200b5f10 con 0x7f78200c6470 2023-10-07T09:47:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.545+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782004c3c0 con 0x7f78200c6470 2023-10-07T09:47:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.546+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7814007240 con 0x7f78200c6470 2023-10-07T09:47:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.547+0000 7f781cff9700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066f30 0x7f78040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.547+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f7814084250 con 0x7f78200c6470 2023-10-07T09:47:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.548+0000 7f781ffff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066f30 0x7f78040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.548+0000 7f781ffff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066f30 0x7f78040693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7820156700 tx=0x7f780c009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.551+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7814051340 con 0x7f78200c6470 2023-10-07T09:47:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.748+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7820064040 con 0x7f7804066f30 2023-10-07T09:47:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.760+0000 7f781cff9700 1 -- 172.21.15.29:0/1384576764 <== mgr.7162 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289181 (secure 0 0 0) 0x7f7820064040 con 0x7f7804066f30 2023-10-07T09:47:22.764 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.763+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066f30 msgr2=0x7f78040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.763+0000 7f7826dff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066f30 0x7f78040693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7820156700 tx=0x7f780c009040 comp rx=0 tx=0).stop 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.763+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 msgr2=0x7f7820151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.763+0000 7f7826dff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f7820151d40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f781400c7c0 tx=0x7f7814001170 comp rx=0 tx=0).stop 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 shutdown_connections 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7804066f30 0x7f78040693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7820159b20 0x7f78200c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78200c6470 0x7f7820151d40 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 --2- 172.21.15.29:0/1384576764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7820159750 0x7f78200c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 >> 172.21.15.29:0/1384576764 conn(0x7f7820063a10 msgr2=0x7f7820153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 shutdown_connections 2023-10-07T09:47:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:22.764+0000 7f7826dff700 1 -- 172.21.15.29:0/1384576764 wait complete. 2023-10-07T09:47:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:47:22.782 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:47:22.472701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439984,"kb_used_data":12528,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752534736,"statfs":{"total":773094113280,"available":770595569664,"internally_reserved":0,"allocated":12828672,"data_stored":6755724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001699"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.395327+0000","last_change":"2023-10-07T09:47:18.393444+0000","last_active":"2023-10-07T09:47:18.395327+0000","last_peered":"2023-10-07T09:47:18.395327+0000","last_clean":"2023-10-07T09:47:18.395327+0000","last_became_active":"2023-10-07T09:47:18.393186+0000","last_became_peered":"2023-10-07T09:47:18.393186+0000","last_unstale":"2023-10-07T09:47:18.395327+0000","last_undegraded":"2023-10-07T09:47:18.395327+0000","last_fullsized":"2023-10-07T09:47:18.395327+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:02:07.742177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404898+0000","last_change":"2023-10-07T09:47:18.391919+0000","last_active":"2023-10-07T09:47:18.404898+0000","last_peered":"2023-10-07T09:47:18.404898+0000","last_clean":"2023-10-07T09:47:18.404898+0000","last_became_active":"2023-10-07T09:47:18.391600+0000","last_became_peered":"2023-10-07T09:47:18.391600+0000","last_unstale":"2023-10-07T09:47:18.404898+0000","last_undegraded":"2023-10-07T09:47:18.404898+0000","last_fullsized":"2023-10-07T09:47:18.404898+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:45:17.030160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.390564+0000","last_change":"2023-10-07T09:47:18.389275+0000","last_active":"2023-10-07T09:47:18.390564+0000","last_peered":"2023-10-07T09:47:18.390564+0000","last_clean":"2023-10-07T09:47:18.390564+0000","last_became_active":"2023-10-07T09:47:18.388996+0000","last_became_peered":"2023-10-07T09:47:18.388996+0000","last_unstale":"2023-10-07T09:47:18.390564+0000","last_undegraded":"2023-10-07T09:47:18.390564+0000","last_fullsized":"2023-10-07T09:47:18.390564+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:58.133179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403803+0000","last_change":"2023-10-07T09:47:18.398592+0000","last_active":"2023-10-07T09:47:18.403803+0000","last_peered":"2023-10-07T09:47:18.403803+0000","last_clean":"2023-10-07T09:47:18.403803+0000","last_became_active":"2023-10-07T09:47:18.398038+0000","last_became_peered":"2023-10-07T09:47:18.398038+0000","last_unstale":"2023-10-07T09:47:18.403803+0000","last_undegraded":"2023-10-07T09:47:18.403803+0000","last_fullsized":"2023-10-07T09:47:18.403803+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:58:13.384898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.402243+0000","last_change":"2023-10-07T09:47:18.393985+0000","last_active":"2023-10-07T09:47:18.402243+0000","last_peered":"2023-10-07T09:47:18.402243+0000","last_clean":"2023-10-07T09:47:18.402243+0000","last_became_active":"2023-10-07T09:47:18.393672+0000","last_became_peered":"2023-10-07T09:47:18.393672+0000","last_unstale":"2023-10-07T09:47:18.402243+0000","last_undegraded":"2023-10-07T09:47:18.402243+0000","last_fullsized":"2023-10-07T09:47:18.402243+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:54.054174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398502+0000","last_change":"2023-10-07T09:47:18.394136+0000","last_active":"2023-10-07T09:47:18.398502+0000","last_peered":"2023-10-07T09:47:18.398502+0000","last_clean":"2023-10-07T09:47:18.398502+0000","last_became_active":"2023-10-07T09:47:18.393879+0000","last_became_peered":"2023-10-07T09:47:18.393879+0000","last_unstale":"2023-10-07T09:47:18.398502+0000","last_undegraded":"2023-10-07T09:47:18.398502+0000","last_fullsized":"2023-10-07T09:47:18.398502+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:47:14.028161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398305+0000","last_change":"2023-10-07T09:47:18.393588+0000","last_active":"2023-10-07T09:47:18.398305+0000","last_peered":"2023-10-07T09:47:18.398305+0000","last_clean":"2023-10-07T09:47:18.398305+0000","last_became_active":"2023-10-07T09:47:18.393279+0000","last_became_peered":"2023-10-07T09:47:18.393279+0000","last_unstale":"2023-10-07T09:47:18.398305+0000","last_undegraded":"2023-10-07T09:47:18.398305+0000","last_fullsized":"2023-10-07T09:47:18.398305+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:51.792984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404821+0000","last_change":"2023-10-07T09:47:18.395396+0000","last_active":"2023-10-07T09:47:18.404821+0000","last_peered":"2023-10-07T09:47:18.404821+0000","last_clean":"2023-10-07T09:47:18.404821+0000","last_became_active":"2023-10-07T09:47:18.395079+0000","last_became_peered":"2023-10-07T09:47:18.395079+0000","last_unstale":"2023-10-07T09:47:18.404821+0000","last_undegraded":"2023-10-07T09:47:18.404821+0000","last_fullsized":"2023-10-07T09:47:18.404821+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:11.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.395991+0000","last_change":"2023-10-07T09:47:18.389344+0000","last_active":"2023-10-07T09:47:18.395991+0000","last_peered":"2023-10-07T09:47:18.395991+0000","last_clean":"2023-10-07T09:47:18.395991+0000","last_became_active":"2023-10-07T09:47:18.389036+0000","last_became_peered":"2023-10-07T09:47:18.389036+0000","last_unstale":"2023-10-07T09:47:18.395991+0000","last_undegraded":"2023-10-07T09:47:18.395991+0000","last_fullsized":"2023-10-07T09:47:18.395991+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:47:37.693705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401579+0000","last_change":"2023-10-07T09:47:18.394071+0000","last_active":"2023-10-07T09:47:18.401579+0000","last_peered":"2023-10-07T09:47:18.401579+0000","last_clean":"2023-10-07T09:47:18.401579+0000","last_became_active":"2023-10-07T09:47:18.393740+0000","last_became_peered":"2023-10-07T09:47:18.393740+0000","last_unstale":"2023-10-07T09:47:18.401579+0000","last_undegraded":"2023-10-07T09:47:18.401579+0000","last_fullsized":"2023-10-07T09:47:18.401579+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:17:57.209214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.400357+0000","last_change":"2023-10-07T09:47:18.398829+0000","last_active":"2023-10-07T09:47:18.400357+0000","last_peered":"2023-10-07T09:47:18.400357+0000","last_clean":"2023-10-07T09:47:18.400357+0000","last_became_active":"2023-10-07T09:47:18.398560+0000","last_became_peered":"2023-10-07T09:47:18.398560+0000","last_unstale":"2023-10-07T09:47:18.400357+0000","last_undegraded":"2023-10-07T09:47:18.400357+0000","last_fullsized":"2023-10-07T09:47:18.400357+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:31:25.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.396615+0000","last_change":"2023-10-07T09:47:18.389856+0000","last_active":"2023-10-07T09:47:18.396615+0000","last_peered":"2023-10-07T09:47:18.396615+0000","last_clean":"2023-10-07T09:47:18.396615+0000","last_became_active":"2023-10-07T09:47:18.389407+0000","last_became_peered":"2023-10-07T09:47:18.389407+0000","last_unstale":"2023-10-07T09:47:18.396615+0000","last_undegraded":"2023-10-07T09:47:18.396615+0000","last_fullsized":"2023-10-07T09:47:18.396615+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:11.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.388410+0000","last_change":"2023-10-07T09:47:18.379946+0000","last_active":"2023-10-07T09:47:18.388410+0000","last_peered":"2023-10-07T09:47:18.388410+0000","last_clean":"2023-10-07T09:47:18.388410+0000","last_became_active":"2023-10-07T09:47:18.379691+0000","last_became_peered":"2023-10-07T09:47:18.379691+0000","last_unstale":"2023-10-07T09:47:18.388410+0000","last_undegraded":"2023-10-07T09:47:18.388410+0000","last_fullsized":"2023-10-07T09:47:18.388410+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:47:37.693705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405185+0000","last_change":"2023-10-07T09:47:18.401256+0000","last_active":"2023-10-07T09:47:18.405185+0000","last_peered":"2023-10-07T09:47:18.405185+0000","last_clean":"2023-10-07T09:47:18.405185+0000","last_became_active":"2023-10-07T09:47:18.401004+0000","last_became_peered":"2023-10-07T09:47:18.401004+0000","last_unstale":"2023-10-07T09:47:18.405185+0000","last_undegraded":"2023-10-07T09:47:18.405185+0000","last_fullsized":"2023-10-07T09:47:18.405185+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:17.966226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.389822+0000","last_change":"2023-10-07T09:47:18.388682+0000","last_active":"2023-10-07T09:47:18.389822+0000","last_peered":"2023-10-07T09:47:18.389822+0000","last_clean":"2023-10-07T09:47:18.389822+0000","last_became_active":"2023-10-07T09:47:18.388322+0000","last_became_peered":"2023-10-07T09:47:18.388322+0000","last_unstale":"2023-10-07T09:47:18.389822+0000","last_undegraded":"2023-10-07T09:47:18.389822+0000","last_fullsized":"2023-10-07T09:47:18.389822+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:10:09.529294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407268+0000","last_change":"2023-10-07T09:47:18.406518+0000","last_active":"2023-10-07T09:47:18.407268+0000","last_peered":"2023-10-07T09:47:18.407268+0000","last_clean":"2023-10-07T09:47:18.407268+0000","last_became_active":"2023-10-07T09:47:18.406212+0000","last_became_peered":"2023-10-07T09:47:18.406212+0000","last_unstale":"2023-10-07T09:47:18.407268+0000","last_undegraded":"2023-10-07T09:47:18.407268+0000","last_fullsized":"2023-10-07T09:47:18.407268+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:15.752555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404857+0000","last_change":"2023-10-07T09:47:18.392294+0000","last_active":"2023-10-07T09:47:18.404857+0000","last_peered":"2023-10-07T09:47:18.404857+0000","last_clean":"2023-10-07T09:47:18.404857+0000","last_became_active":"2023-10-07T09:47:18.391969+0000","last_became_peered":"2023-10-07T09:47:18.391969+0000","last_unstale":"2023-10-07T09:47:18.404857+0000","last_undegraded":"2023-10-07T09:47:18.404857+0000","last_fullsized":"2023-10-07T09:47:18.404857+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:39:47.987946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403749+0000","last_change":"2023-10-07T09:47:18.398435+0000","last_active":"2023-10-07T09:47:18.403749+0000","last_peered":"2023-10-07T09:47:18.403749+0000","last_clean":"2023-10-07T09:47:18.403749+0000","last_became_active":"2023-10-07T09:47:18.398071+0000","last_became_peered":"2023-10-07T09:47:18.398071+0000","last_unstale":"2023-10-07T09:47:18.403749+0000","last_undegraded":"2023-10-07T09:47:18.403749+0000","last_fullsized":"2023-10-07T09:47:18.403749+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:29:02.888441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401276+0000","last_change":"2023-10-07T09:47:18.399925+0000","last_active":"2023-10-07T09:47:18.401276+0000","last_peered":"2023-10-07T09:47:18.401276+0000","last_clean":"2023-10-07T09:47:18.401276+0000","last_became_active":"2023-10-07T09:47:18.398090+0000","last_became_peered":"2023-10-07T09:47:18.398090+0000","last_unstale":"2023-10-07T09:47:18.401276+0000","last_undegraded":"2023-10-07T09:47:18.401276+0000","last_fullsized":"2023-10-07T09:47:18.401276+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:00:53.059580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.396594+0000","last_change":"2023-10-07T09:47:18.389676+0000","last_active":"2023-10-07T09:47:18.396594+0000","last_peered":"2023-10-07T09:47:18.396594+0000","last_clean":"2023-10-07T09:47:18.396594+0000","last_became_active":"2023-10-07T09:47:18.389014+0000","last_became_peered":"2023-10-07T09:47:18.389014+0000","last_unstale":"2023-10-07T09:47:18.396594+0000","last_undegraded":"2023-10-07T09:47:18.396594+0000","last_fullsized":"2023-10-07T09:47:18.396594+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:00.591766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401300+0000","last_change":"2023-10-07T09:47:18.399299+0000","last_active":"2023-10-07T09:47:18.401300+0000","last_peered":"2023-10-07T09:47:18.401300+0000","last_clean":"2023-10-07T09:47:18.401300+0000","last_became_active":"2023-10-07T09:47:18.398738+0000","last_became_peered":"2023-10-07T09:47:18.398738+0000","last_unstale":"2023-10-07T09:47:18.401300+0000","last_undegraded":"2023-10-07T09:47:18.401300+0000","last_fullsized":"2023-10-07T09:47:18.401300+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:35:21.217791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404798+0000","last_change":"2023-10-07T09:47:18.393673+0000","last_active":"2023-10-07T09:47:18.404798+0000","last_peered":"2023-10-07T09:47:18.404798+0000","last_clean":"2023-10-07T09:47:18.404798+0000","last_became_active":"2023-10-07T09:47:18.393410+0000","last_became_peered":"2023-10-07T09:47:18.393410+0000","last_unstale":"2023-10-07T09:47:18.404798+0000","last_undegraded":"2023-10-07T09:47:18.404798+0000","last_fullsized":"2023-10-07T09:47:18.404798+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:57:45.265868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404719+0000","last_change":"2023-10-07T09:47:18.393324+0000","last_active":"2023-10-07T09:47:18.404719+0000","last_peered":"2023-10-07T09:47:18.404719+0000","last_clean":"2023-10-07T09:47:18.404719+0000","last_became_active":"2023-10-07T09:47:18.393054+0000","last_became_peered":"2023-10-07T09:47:18.393054+0000","last_unstale":"2023-10-07T09:47:18.404719+0000","last_undegraded":"2023-10-07T09:47:18.404719+0000","last_fullsized":"2023-10-07T09:47:18.404719+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:55:36.237343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.399673+0000","last_change":"2023-10-07T09:47:18.397767+0000","last_active":"2023-10-07T09:47:18.399673+0000","last_peered":"2023-10-07T09:47:18.399673+0000","last_clean":"2023-10-07T09:47:18.399673+0000","last_became_active":"2023-10-07T09:47:18.396862+0000","last_became_peered":"2023-10-07T09:47:18.396862+0000","last_unstale":"2023-10-07T09:47:18.399673+0000","last_undegraded":"2023-10-07T09:47:18.399673+0000","last_fullsized":"2023-10-07T09:47:18.399673+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:36:22.235575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401691+0000","last_change":"2023-10-07T09:47:18.395897+0000","last_active":"2023-10-07T09:47:18.401691+0000","last_peered":"2023-10-07T09:47:18.401691+0000","last_clean":"2023-10-07T09:47:18.401691+0000","last_became_active":"2023-10-07T09:47:18.395251+0000","last_became_peered":"2023-10-07T09:47:18.395251+0000","last_unstale":"2023-10-07T09:47:18.401691+0000","last_undegraded":"2023-10-07T09:47:18.401691+0000","last_fullsized":"2023-10-07T09:47:18.401691+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:02:02.383028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404859+0000","last_change":"2023-10-07T09:47:18.391758+0000","last_active":"2023-10-07T09:47:18.404859+0000","last_peered":"2023-10-07T09:47:18.404859+0000","last_clean":"2023-10-07T09:47:18.404859+0000","last_became_active":"2023-10-07T09:47:18.391338+0000","last_became_peered":"2023-10-07T09:47:18.391338+0000","last_unstale":"2023-10-07T09:47:18.404859+0000","last_undegraded":"2023-10-07T09:47:18.404859+0000","last_fullsized":"2023-10-07T09:47:18.404859+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:11.717213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403737+0000","last_change":"2023-10-07T09:47:18.400589+0000","last_active":"2023-10-07T09:47:18.403737+0000","last_peered":"2023-10-07T09:47:18.403737+0000","last_clean":"2023-10-07T09:47:18.403737+0000","last_became_active":"2023-10-07T09:47:18.400177+0000","last_became_peered":"2023-10-07T09:47:18.400177+0000","last_unstale":"2023-10-07T09:47:18.403737+0000","last_undegraded":"2023-10-07T09:47:18.403737+0000","last_fullsized":"2023-10-07T09:47:18.403737+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:19.684823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401750+0000","last_change":"2023-10-07T09:47:18.394976+0000","last_active":"2023-10-07T09:47:18.401750+0000","last_peered":"2023-10-07T09:47:18.401750+0000","last_clean":"2023-10-07T09:47:18.401750+0000","last_became_active":"2023-10-07T09:47:18.394641+0000","last_became_peered":"2023-10-07T09:47:18.394641+0000","last_unstale":"2023-10-07T09:47:18.401750+0000","last_undegraded":"2023-10-07T09:47:18.401750+0000","last_fullsized":"2023-10-07T09:47:18.401750+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:11.717213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404564+0000","last_change":"2023-10-07T09:47:18.391236+0000","last_active":"2023-10-07T09:47:18.404564+0000","last_peered":"2023-10-07T09:47:18.404564+0000","last_clean":"2023-10-07T09:47:18.404564+0000","last_became_active":"2023-10-07T09:47:18.390982+0000","last_became_peered":"2023-10-07T09:47:18.390982+0000","last_unstale":"2023-10-07T09:47:18.404564+0000","last_undegraded":"2023-10-07T09:47:18.404564+0000","last_fullsized":"2023-10-07T09:47:18.404564+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:29:02.888441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405560+0000","last_change":"2023-10-07T09:47:18.404052+0000","last_active":"2023-10-07T09:47:18.405560+0000","last_peered":"2023-10-07T09:47:18.405560+0000","last_clean":"2023-10-07T09:47:18.405560+0000","last_became_active":"2023-10-07T09:47:18.402919+0000","last_became_peered":"2023-10-07T09:47:18.402919+0000","last_unstale":"2023-10-07T09:47:18.405560+0000","last_undegraded":"2023-10-07T09:47:18.405560+0000","last_fullsized":"2023-10-07T09:47:18.405560+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:29:20.084226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403781+0000","last_change":"2023-10-07T09:47:18.398648+0000","last_active":"2023-10-07T09:47:18.403781+0000","last_peered":"2023-10-07T09:47:18.403781+0000","last_clean":"2023-10-07T09:47:18.403781+0000","last_became_active":"2023-10-07T09:47:18.398329+0000","last_became_peered":"2023-10-07T09:47:18.398329+0000","last_unstale":"2023-10-07T09:47:18.403781+0000","last_undegraded":"2023-10-07T09:47:18.403781+0000","last_fullsized":"2023-10-07T09:47:18.403781+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:46:35.633305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.408498+0000","last_change":"2023-10-07T09:47:18.406760+0000","last_active":"2023-10-07T09:47:18.408498+0000","last_peered":"2023-10-07T09:47:18.408498+0000","last_clean":"2023-10-07T09:47:18.408498+0000","last_became_active":"2023-10-07T09:47:18.406499+0000","last_became_peered":"2023-10-07T09:47:18.406499+0000","last_unstale":"2023-10-07T09:47:18.408498+0000","last_undegraded":"2023-10-07T09:47:18.408498+0000","last_fullsized":"2023-10-07T09:47:18.408498+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:59.699128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405147+0000","last_change":"2023-10-07T09:47:18.400965+0000","last_active":"2023-10-07T09:47:18.405147+0000","last_peered":"2023-10-07T09:47:18.405147+0000","last_clean":"2023-10-07T09:47:18.405147+0000","last_became_active":"2023-10-07T09:47:18.400441+0000","last_became_peered":"2023-10-07T09:47:18.400441+0000","last_unstale":"2023-10-07T09:47:18.405147+0000","last_undegraded":"2023-10-07T09:47:18.405147+0000","last_fullsized":"2023-10-07T09:47:18.405147+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:42:11.163248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"117'4","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404377+0000","last_change":"2023-10-07T09:47:17.390190+0000","last_active":"2023-10-07T09:47:18.404377+0000","last_peered":"2023-10-07T09:47:18.404377+0000","last_clean":"2023-10-07T09:47:18.404377+0000","last_became_active":"2023-10-07T09:47:17.389900+0000","last_became_peered":"2023-10-07T09:47:17.389900+0000","last_unstale":"2023-10-07T09:47:18.404377+0000","last_undegraded":"2023-10-07T09:47:18.404377+0000","last_fullsized":"2023-10-07T09:47:18.404377+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:08.630717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407259+0000","last_change":"2023-10-07T09:47:18.399227+0000","last_active":"2023-10-07T09:47:18.407259+0000","last_peered":"2023-10-07T09:47:18.407259+0000","last_clean":"2023-10-07T09:47:18.407259+0000","last_became_active":"2023-10-07T09:47:18.398958+0000","last_became_peered":"2023-10-07T09:47:18.398958+0000","last_unstale":"2023-10-07T09:47:18.407259+0000","last_undegraded":"2023-10-07T09:47:18.407259+0000","last_fullsized":"2023-10-07T09:47:18.407259+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:11.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407911+0000","last_change":"2023-10-07T09:47:17.389152+0000","last_active":"2023-10-07T09:47:18.407911+0000","last_peered":"2023-10-07T09:47:18.407911+0000","last_clean":"2023-10-07T09:47:18.407911+0000","last_became_active":"2023-10-07T09:47:17.388822+0000","last_became_peered":"2023-10-07T09:47:17.388822+0000","last_unstale":"2023-10-07T09:47:18.407911+0000","last_undegraded":"2023-10-07T09:47:18.407911+0000","last_fullsized":"2023-10-07T09:47:18.407911+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:04.806906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.389691+0000","last_change":"2023-10-07T09:47:18.388286+0000","last_active":"2023-10-07T09:47:18.389691+0000","last_peered":"2023-10-07T09:47:18.389691+0000","last_clean":"2023-10-07T09:47:18.389691+0000","last_became_active":"2023-10-07T09:47:18.387954+0000","last_became_peered":"2023-10-07T09:47:18.387954+0000","last_unstale":"2023-10-07T09:47:18.389691+0000","last_undegraded":"2023-10-07T09:47:18.389691+0000","last_fullsized":"2023-10-07T09:47:18.389691+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:08.152980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.406084+0000","last_change":"2023-10-07T09:47:17.393730+0000","last_active":"2023-10-07T09:47:18.406084+0000","last_peered":"2023-10-07T09:47:18.406084+0000","last_clean":"2023-10-07T09:47:18.406084+0000","last_became_active":"2023-10-07T09:47:17.393270+0000","last_became_peered":"2023-10-07T09:47:17.393270+0000","last_unstale":"2023-10-07T09:47:18.406084+0000","last_undegraded":"2023-10-07T09:47:18.406084+0000","last_fullsized":"2023-10-07T09:47:18.406084+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:14:52.406463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384365+0000","last_change":"2023-10-07T09:47:18.379617+0000","last_active":"2023-10-07T09:47:18.384365+0000","last_peered":"2023-10-07T09:47:18.384365+0000","last_clean":"2023-10-07T09:47:18.384365+0000","last_became_active":"2023-10-07T09:47:18.379306+0000","last_became_peered":"2023-10-07T09:47:18.379306+0000","last_unstale":"2023-10-07T09:47:18.384365+0000","last_undegraded":"2023-10-07T09:47:18.384365+0000","last_fullsized":"2023-10-07T09:47:18.384365+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:46:35.633305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404456+0000","last_change":"2023-10-07T09:47:17.392660+0000","last_active":"2023-10-07T09:47:18.404456+0000","last_peered":"2023-10-07T09:47:18.404456+0000","last_clean":"2023-10-07T09:47:18.404456+0000","last_became_active":"2023-10-07T09:47:17.392398+0000","last_became_peered":"2023-10-07T09:47:17.392398+0000","last_unstale":"2023-10-07T09:47:18.404456+0000","last_undegraded":"2023-10-07T09:47:18.404456+0000","last_fullsized":"2023-10-07T09:47:18.404456+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:34.755567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.392700+0000","last_change":"2023-10-07T09:47:18.387310+0000","last_active":"2023-10-07T09:47:18.392700+0000","last_peered":"2023-10-07T09:47:18.392700+0000","last_clean":"2023-10-07T09:47:18.392700+0000","last_became_active":"2023-10-07T09:47:18.387010+0000","last_became_peered":"2023-10-07T09:47:18.387010+0000","last_unstale":"2023-10-07T09:47:18.392700+0000","last_undegraded":"2023-10-07T09:47:18.392700+0000","last_fullsized":"2023-10-07T09:47:18.392700+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:58.133179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.402752+0000","last_change":"2023-10-07T09:47:17.388067+0000","last_active":"2023-10-07T09:47:18.402752+0000","last_peered":"2023-10-07T09:47:18.402752+0000","last_clean":"2023-10-07T09:47:18.402752+0000","last_became_active":"2023-10-07T09:47:17.387799+0000","last_became_peered":"2023-10-07T09:47:17.387799+0000","last_unstale":"2023-10-07T09:47:18.402752+0000","last_undegraded":"2023-10-07T09:47:18.402752+0000","last_fullsized":"2023-10-07T09:47:18.402752+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:42:10.155445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398352+0000","last_change":"2023-10-07T09:47:18.396700+0000","last_active":"2023-10-07T09:47:18.398352+0000","last_peered":"2023-10-07T09:47:18.398352+0000","last_clean":"2023-10-07T09:47:18.398352+0000","last_became_active":"2023-10-07T09:47:18.396410+0000","last_became_peered":"2023-10-07T09:47:18.396410+0000","last_unstale":"2023-10-07T09:47:18.398352+0000","last_undegraded":"2023-10-07T09:47:18.398352+0000","last_fullsized":"2023-10-07T09:47:18.398352+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:36.644856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403780+0000","last_change":"2023-10-07T09:47:17.389738+0000","last_active":"2023-10-07T09:47:18.403780+0000","last_peered":"2023-10-07T09:47:18.403780+0000","last_clean":"2023-10-07T09:47:18.403780+0000","last_became_active":"2023-10-07T09:47:17.389432+0000","last_became_peered":"2023-10-07T09:47:17.389432+0000","last_unstale":"2023-10-07T09:47:18.403780+0000","last_undegraded":"2023-10-07T09:47:18.403780+0000","last_fullsized":"2023-10-07T09:47:18.403780+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:02:06.734374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.391686+0000","last_change":"2023-10-07T09:47:18.389121+0000","last_active":"2023-10-07T09:47:18.391686+0000","last_peered":"2023-10-07T09:47:18.391686+0000","last_clean":"2023-10-07T09:47:18.391686+0000","last_became_active":"2023-10-07T09:47:18.388851+0000","last_became_peered":"2023-10-07T09:47:18.388851+0000","last_unstale":"2023-10-07T09:47:18.391686+0000","last_undegraded":"2023-10-07T09:47:18.391686+0000","last_fullsized":"2023-10-07T09:47:18.391686+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:41.832204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.380545+0000","last_change":"2023-10-07T09:47:17.392020+0000","last_active":"2023-10-07T09:47:18.380545+0000","last_peered":"2023-10-07T09:47:18.380545+0000","last_clean":"2023-10-07T09:47:18.380545+0000","last_became_active":"2023-10-07T09:47:17.391769+0000","last_became_peered":"2023-10-07T09:47:17.391769+0000","last_unstale":"2023-10-07T09:47:18.380545+0000","last_undegraded":"2023-10-07T09:47:18.380545+0000","last_fullsized":"2023-10-07T09:47:18.380545+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:00.728816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384089+0000","last_change":"2023-10-07T09:47:18.381375+0000","last_active":"2023-10-07T09:47:18.384089+0000","last_peered":"2023-10-07T09:47:18.384089+0000","last_clean":"2023-10-07T09:47:18.384089+0000","last_became_active":"2023-10-07T09:47:18.381068+0000","last_became_peered":"2023-10-07T09:47:18.381068+0000","last_unstale":"2023-10-07T09:47:18.384089+0000","last_undegraded":"2023-10-07T09:47:18.384089+0000","last_fullsized":"2023-10-07T09:47:18.384089+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:41.832204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.382641+0000","last_change":"2023-10-07T09:47:17.384581+0000","last_active":"2023-10-07T09:47:18.382641+0000","last_peered":"2023-10-07T09:47:18.382641+0000","last_clean":"2023-10-07T09:47:18.382641+0000","last_became_active":"2023-10-07T09:47:17.384302+0000","last_became_peered":"2023-10-07T09:47:17.384302+0000","last_unstale":"2023-10-07T09:47:18.382641+0000","last_undegraded":"2023-10-07T09:47:18.382641+0000","last_fullsized":"2023-10-07T09:47:18.382641+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:35:20.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404881+0000","last_change":"2023-10-07T09:47:18.391855+0000","last_active":"2023-10-07T09:47:18.404881+0000","last_peered":"2023-10-07T09:47:18.404881+0000","last_clean":"2023-10-07T09:47:18.404881+0000","last_became_active":"2023-10-07T09:47:18.391510+0000","last_became_peered":"2023-10-07T09:47:18.391510+0000","last_unstale":"2023-10-07T09:47:18.404881+0000","last_undegraded":"2023-10-07T09:47:18.404881+0000","last_fullsized":"2023-10-07T09:47:18.404881+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:52:43.789995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.382341+0000","last_change":"2023-10-07T09:47:17.389013+0000","last_active":"2023-10-07T09:47:18.382341+0000","last_peered":"2023-10-07T09:47:18.382341+0000","last_clean":"2023-10-07T09:47:18.382341+0000","last_became_active":"2023-10-07T09:47:17.388745+0000","last_became_peered":"2023-10-07T09:47:17.388745+0000","last_unstale":"2023-10-07T09:47:18.382341+0000","last_undegraded":"2023-10-07T09:47:18.382341+0000","last_fullsized":"2023-10-07T09:47:18.382341+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:05.857907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384051+0000","last_change":"2023-10-07T09:47:18.380863+0000","last_active":"2023-10-07T09:47:18.384051+0000","last_peered":"2023-10-07T09:47:18.384051+0000","last_clean":"2023-10-07T09:47:18.384051+0000","last_became_active":"2023-10-07T09:47:18.380168+0000","last_became_peered":"2023-10-07T09:47:18.380168+0000","last_unstale":"2023-10-07T09:47:18.384051+0000","last_undegraded":"2023-10-07T09:47:18.384051+0000","last_fullsized":"2023-10-07T09:47:18.384051+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:58:13.384898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373252+0000","last_change":"2023-10-07T09:47:17.392199+0000","last_active":"2023-10-07T09:47:18.373252+0000","last_peered":"2023-10-07T09:47:18.373252+0000","last_clean":"2023-10-07T09:47:18.373252+0000","last_became_active":"2023-10-07T09:47:17.391849+0000","last_became_peered":"2023-10-07T09:47:17.391849+0000","last_unstale":"2023-10-07T09:47:18.373252+0000","last_undegraded":"2023-10-07T09:47:18.373252+0000","last_fullsized":"2023-10-07T09:47:18.373252+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:57.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405516+0000","last_change":"2023-10-07T09:47:18.404871+0000","last_active":"2023-10-07T09:47:18.405516+0000","last_peered":"2023-10-07T09:47:18.405516+0000","last_clean":"2023-10-07T09:47:18.405516+0000","last_became_active":"2023-10-07T09:47:18.404621+0000","last_became_peered":"2023-10-07T09:47:18.404621+0000","last_unstale":"2023-10-07T09:47:18.405516+0000","last_undegraded":"2023-10-07T09:47:18.405516+0000","last_fullsized":"2023-10-07T09:47:18.405516+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:29:34.580808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"117'3","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401148+0000","last_change":"2023-10-07T09:47:17.388738+0000","last_active":"2023-10-07T09:47:18.401148+0000","last_peered":"2023-10-07T09:47:18.401148+0000","last_clean":"2023-10-07T09:47:18.401148+0000","last_became_active":"2023-10-07T09:47:17.388426+0000","last_became_peered":"2023-10-07T09:47:17.388426+0000","last_unstale":"2023-10-07T09:47:18.401148+0000","last_undegraded":"2023-10-07T09:47:18.401148+0000","last_fullsized":"2023-10-07T09:47:18.401148+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:54:47.661303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407274+0000","last_change":"2023-10-07T09:47:18.406580+0000","last_active":"2023-10-07T09:47:18.407274+0000","last_peered":"2023-10-07T09:47:18.407274+0000","last_clean":"2023-10-07T09:47:18.407274+0000","last_became_active":"2023-10-07T09:47:18.406299+0000","last_became_peered":"2023-10-07T09:47:18.406299+0000","last_unstale":"2023-10-07T09:47:18.407274+0000","last_undegraded":"2023-10-07T09:47:18.407274+0000","last_fullsized":"2023-10-07T09:47:18.407274+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:09:07.870732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"117'6","reported_seq":16,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.408598+0000","last_change":"2023-10-07T09:47:17.383118+0000","last_active":"2023-10-07T09:47:18.408598+0000","last_peered":"2023-10-07T09:47:18.408598+0000","last_clean":"2023-10-07T09:47:18.408598+0000","last_became_active":"2023-10-07T09:47:17.382831+0000","last_became_peered":"2023-10-07T09:47:17.382831+0000","last_unstale":"2023-10-07T09:47:18.408598+0000","last_undegraded":"2023-10-07T09:47:18.408598+0000","last_fullsized":"2023-10-07T09:47:18.408598+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:31:21.528178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404340+0000","last_change":"2023-10-07T09:47:18.397611+0000","last_active":"2023-10-07T09:47:18.404340+0000","last_peered":"2023-10-07T09:47:18.404340+0000","last_clean":"2023-10-07T09:47:18.404340+0000","last_became_active":"2023-10-07T09:47:18.397284+0000","last_became_peered":"2023-10-07T09:47:18.397284+0000","last_unstale":"2023-10-07T09:47:18.404340+0000","last_undegraded":"2023-10-07T09:47:18.404340+0000","last_fullsized":"2023-10-07T09:47:18.404340+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:59.475818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404703+0000","last_change":"2023-10-07T09:47:17.386081+0000","last_active":"2023-10-07T09:47:18.404703+0000","last_peered":"2023-10-07T09:47:18.404703+0000","last_clean":"2023-10-07T09:47:18.404703+0000","last_became_active":"2023-10-07T09:47:17.385819+0000","last_became_peered":"2023-10-07T09:47:17.385819+0000","last_unstale":"2023-10-07T09:47:18.404703+0000","last_undegraded":"2023-10-07T09:47:18.404703+0000","last_fullsized":"2023-10-07T09:47:18.404703+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:40.385342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407957+0000","last_change":"2023-10-07T09:47:18.402616+0000","last_active":"2023-10-07T09:47:18.407957+0000","last_peered":"2023-10-07T09:47:18.407957+0000","last_clean":"2023-10-07T09:47:18.407957+0000","last_became_active":"2023-10-07T09:47:18.402099+0000","last_became_peered":"2023-10-07T09:47:18.402099+0000","last_unstale":"2023-10-07T09:47:18.407957+0000","last_undegraded":"2023-10-07T09:47:18.407957+0000","last_fullsized":"2023-10-07T09:47:18.407957+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:11.227058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.388179+0000","last_change":"2023-10-07T09:47:17.389362+0000","last_active":"2023-10-07T09:47:18.388179+0000","last_peered":"2023-10-07T09:47:18.388179+0000","last_clean":"2023-10-07T09:47:18.388179+0000","last_became_active":"2023-10-07T09:47:17.389052+0000","last_became_peered":"2023-10-07T09:47:17.389052+0000","last_unstale":"2023-10-07T09:47:18.388179+0000","last_undegraded":"2023-10-07T09:47:18.388179+0000","last_fullsized":"2023-10-07T09:47:18.388179+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:02:27.882294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.395445+0000","last_change":"2023-10-07T09:47:18.387451+0000","last_active":"2023-10-07T09:47:18.395445+0000","last_peered":"2023-10-07T09:47:18.395445+0000","last_clean":"2023-10-07T09:47:18.395445+0000","last_became_active":"2023-10-07T09:47:18.387107+0000","last_became_peered":"2023-10-07T09:47:18.387107+0000","last_unstale":"2023-10-07T09:47:18.395445+0000","last_undegraded":"2023-10-07T09:47:18.395445+0000","last_fullsized":"2023-10-07T09:47:18.395445+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:10:09.529294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373639+0000","last_change":"2023-10-07T09:47:17.386573+0000","last_active":"2023-10-07T09:47:18.373639+0000","last_peered":"2023-10-07T09:47:18.373639+0000","last_clean":"2023-10-07T09:47:18.373639+0000","last_became_active":"2023-10-07T09:47:17.386293+0000","last_became_peered":"2023-10-07T09:47:17.386293+0000","last_unstale":"2023-10-07T09:47:18.373639+0000","last_undegraded":"2023-10-07T09:47:18.373639+0000","last_fullsized":"2023-10-07T09:47:18.373639+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:57.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398507+0000","last_change":"2023-10-07T09:47:18.394454+0000","last_active":"2023-10-07T09:47:18.398507+0000","last_peered":"2023-10-07T09:47:18.398507+0000","last_clean":"2023-10-07T09:47:18.398507+0000","last_became_active":"2023-10-07T09:47:18.394089+0000","last_became_peered":"2023-10-07T09:47:18.394089+0000","last_unstale":"2023-10-07T09:47:18.398507+0000","last_undegraded":"2023-10-07T09:47:18.398507+0000","last_fullsized":"2023-10-07T09:47:18.398507+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:37:29.053053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"117'8","reported_seq":18,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.406316+0000","last_change":"2023-10-07T09:47:17.387809+0000","last_active":"2023-10-07T09:47:18.406316+0000","last_peered":"2023-10-07T09:47:18.406316+0000","last_clean":"2023-10-07T09:47:18.406316+0000","last_became_active":"2023-10-07T09:47:17.387500+0000","last_became_peered":"2023-10-07T09:47:17.387500+0000","last_unstale":"2023-10-07T09:47:18.406316+0000","last_undegraded":"2023-10-07T09:47:18.406316+0000","last_fullsized":"2023-10-07T09:47:18.406316+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:37:28.045250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401709+0000","last_change":"2023-10-07T09:47:18.396103+0000","last_active":"2023-10-07T09:47:18.401709+0000","last_peered":"2023-10-07T09:47:18.401709+0000","last_clean":"2023-10-07T09:47:18.401709+0000","last_became_active":"2023-10-07T09:47:18.395576+0000","last_became_peered":"2023-10-07T09:47:18.395576+0000","last_unstale":"2023-10-07T09:47:18.401709+0000","last_undegraded":"2023-10-07T09:47:18.401709+0000","last_fullsized":"2023-10-07T09:47:18.401709+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:57:45.265868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398007+0000","last_change":"2023-10-07T09:47:17.381899+0000","last_active":"2023-10-07T09:47:18.398007+0000","last_peered":"2023-10-07T09:47:18.398007+0000","last_clean":"2023-10-07T09:47:18.398007+0000","last_became_active":"2023-10-07T09:47:17.381601+0000","last_became_peered":"2023-10-07T09:47:17.381601+0000","last_unstale":"2023-10-07T09:47:18.398007+0000","last_undegraded":"2023-10-07T09:47:18.398007+0000","last_fullsized":"2023-10-07T09:47:18.398007+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:10.219255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.400933+0000","last_change":"2023-10-07T09:47:18.398350+0000","last_active":"2023-10-07T09:47:18.400933+0000","last_peered":"2023-10-07T09:47:18.400933+0000","last_clean":"2023-10-07T09:47:18.400933+0000","last_became_active":"2023-10-07T09:47:18.398051+0000","last_became_peered":"2023-10-07T09:47:18.398051+0000","last_unstale":"2023-10-07T09:47:18.400933+0000","last_undegraded":"2023-10-07T09:47:18.400933+0000","last_fullsized":"2023-10-07T09:47:18.400933+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:09.638520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403347+0000","last_change":"2023-10-07T09:47:17.383454+0000","last_active":"2023-10-07T09:47:18.403347+0000","last_peered":"2023-10-07T09:47:18.403347+0000","last_clean":"2023-10-07T09:47:18.403347+0000","last_became_active":"2023-10-07T09:47:17.383175+0000","last_became_peered":"2023-10-07T09:47:17.383175+0000","last_unstale":"2023-10-07T09:47:18.403347+0000","last_undegraded":"2023-10-07T09:47:18.403347+0000","last_fullsized":"2023-10-07T09:47:18.403347+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:47:36.685902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.400375+0000","last_change":"2023-10-07T09:47:18.399263+0000","last_active":"2023-10-07T09:47:18.400375+0000","last_peered":"2023-10-07T09:47:18.400375+0000","last_clean":"2023-10-07T09:47:18.400375+0000","last_became_active":"2023-10-07T09:47:18.398778+0000","last_became_peered":"2023-10-07T09:47:18.398778+0000","last_unstale":"2023-10-07T09:47:18.400375+0000","last_undegraded":"2023-10-07T09:47:18.400375+0000","last_fullsized":"2023-10-07T09:47:18.400375+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:28:32.954931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.377994+0000","last_change":"2023-10-07T09:47:17.392377+0000","last_active":"2023-10-07T09:47:18.377994+0000","last_peered":"2023-10-07T09:47:18.377994+0000","last_clean":"2023-10-07T09:47:18.377994+0000","last_became_active":"2023-10-07T09:47:17.392100+0000","last_became_peered":"2023-10-07T09:47:17.392100+0000","last_unstale":"2023-10-07T09:47:18.377994+0000","last_undegraded":"2023-10-07T09:47:18.377994+0000","last_fullsized":"2023-10-07T09:47:18.377994+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:53.163564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403327+0000","last_change":"2023-10-07T09:47:18.400757+0000","last_active":"2023-10-07T09:47:18.403327+0000","last_peered":"2023-10-07T09:47:18.403327+0000","last_clean":"2023-10-07T09:47:18.403327+0000","last_became_active":"2023-10-07T09:47:18.398310+0000","last_became_peered":"2023-10-07T09:47:18.398310+0000","last_unstale":"2023-10-07T09:47:18.403327+0000","last_undegraded":"2023-10-07T09:47:18.403327+0000","last_fullsized":"2023-10-07T09:47:18.403327+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:51.461351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.406640+0000","last_change":"2023-10-07T09:47:17.393547+0000","last_active":"2023-10-07T09:47:18.406640+0000","last_peered":"2023-10-07T09:47:18.406640+0000","last_clean":"2023-10-07T09:47:18.406640+0000","last_became_active":"2023-10-07T09:47:17.393238+0000","last_became_peered":"2023-10-07T09:47:17.393238+0000","last_unstale":"2023-10-07T09:47:18.406640+0000","last_undegraded":"2023-10-07T09:47:18.406640+0000","last_fullsized":"2023-10-07T09:47:18.406640+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:10.219255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401319+0000","last_change":"2023-10-07T09:47:18.399503+0000","last_active":"2023-10-07T09:47:18.401319+0000","last_peered":"2023-10-07T09:47:18.401319+0000","last_clean":"2023-10-07T09:47:18.401319+0000","last_became_active":"2023-10-07T09:47:18.399003+0000","last_became_peered":"2023-10-07T09:47:18.399003+0000","last_unstale":"2023-10-07T09:47:18.401319+0000","last_undegraded":"2023-10-07T09:47:18.401319+0000","last_fullsized":"2023-10-07T09:47:18.401319+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:36:58.155004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373952+0000","last_change":"2023-10-07T09:47:17.390287+0000","last_active":"2023-10-07T09:47:18.373952+0000","last_peered":"2023-10-07T09:47:18.373952+0000","last_clean":"2023-10-07T09:47:18.373952+0000","last_became_active":"2023-10-07T09:47:17.390027+0000","last_became_peered":"2023-10-07T09:47:17.390027+0000","last_unstale":"2023-10-07T09:47:18.373952+0000","last_undegraded":"2023-10-07T09:47:18.373952+0000","last_fullsized":"2023-10-07T09:47:18.373952+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:58.691325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398906+0000","last_change":"2023-10-07T09:47:18.395049+0000","last_active":"2023-10-07T09:47:18.398906+0000","last_peered":"2023-10-07T09:47:18.398906+0000","last_clean":"2023-10-07T09:47:18.398906+0000","last_became_active":"2023-10-07T09:47:18.394555+0000","last_became_peered":"2023-10-07T09:47:18.394555+0000","last_unstale":"2023-10-07T09:47:18.398906+0000","last_undegraded":"2023-10-07T09:47:18.398906+0000","last_fullsized":"2023-10-07T09:47:18.398906+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:36:09.513186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373222+0000","last_change":"2023-10-07T09:47:17.384019+0000","last_active":"2023-10-07T09:47:18.373222+0000","last_peered":"2023-10-07T09:47:18.373222+0000","last_clean":"2023-10-07T09:47:18.373222+0000","last_became_active":"2023-10-07T09:47:17.383721+0000","last_became_peered":"2023-10-07T09:47:17.383721+0000","last_unstale":"2023-10-07T09:47:18.373222+0000","last_undegraded":"2023-10-07T09:47:18.373222+0000","last_fullsized":"2023-10-07T09:47:18.373222+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:25:25.575632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401737+0000","last_change":"2023-10-07T09:47:18.395734+0000","last_active":"2023-10-07T09:47:18.401737+0000","last_peered":"2023-10-07T09:47:18.401737+0000","last_clean":"2023-10-07T09:47:18.401737+0000","last_became_active":"2023-10-07T09:47:18.395463+0000","last_became_peered":"2023-10-07T09:47:18.395463+0000","last_unstale":"2023-10-07T09:47:18.401737+0000","last_undegraded":"2023-10-07T09:47:18.401737+0000","last_fullsized":"2023-10-07T09:47:18.401737+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:45:17.030160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.379553+0000","last_change":"2023-10-07T09:47:17.389019+0000","last_active":"2023-10-07T09:47:18.379553+0000","last_peered":"2023-10-07T09:47:18.379553+0000","last_clean":"2023-10-07T09:47:18.379553+0000","last_became_active":"2023-10-07T09:47:17.388739+0000","last_became_peered":"2023-10-07T09:47:17.388739+0000","last_unstale":"2023-10-07T09:47:18.379553+0000","last_undegraded":"2023-10-07T09:47:18.379553+0000","last_fullsized":"2023-10-07T09:47:18.379553+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:31:21.528178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404844+0000","last_change":"2023-10-07T09:47:18.391128+0000","last_active":"2023-10-07T09:47:18.404844+0000","last_peered":"2023-10-07T09:47:18.404844+0000","last_clean":"2023-10-07T09:47:18.404844+0000","last_became_active":"2023-10-07T09:47:18.390854+0000","last_became_peered":"2023-10-07T09:47:18.390854+0000","last_unstale":"2023-10-07T09:47:18.404844+0000","last_undegraded":"2023-10-07T09:47:18.404844+0000","last_fullsized":"2023-10-07T09:47:18.404844+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:36.019820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398278+0000","last_change":"2023-10-07T09:47:17.384129+0000","last_active":"2023-10-07T09:47:18.398278+0000","last_peered":"2023-10-07T09:47:18.398278+0000","last_clean":"2023-10-07T09:47:18.398278+0000","last_became_active":"2023-10-07T09:47:17.383736+0000","last_became_peered":"2023-10-07T09:47:17.383736+0000","last_unstale":"2023-10-07T09:47:18.398278+0000","last_undegraded":"2023-10-07T09:47:18.398278+0000","last_fullsized":"2023-10-07T09:47:18.398278+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:54:43.114199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.402156+0000","last_change":"2023-10-07T09:47:18.397623+0000","last_active":"2023-10-07T09:47:18.402156+0000","last_peered":"2023-10-07T09:47:18.402156+0000","last_clean":"2023-10-07T09:47:18.402156+0000","last_became_active":"2023-10-07T09:47:18.397307+0000","last_became_peered":"2023-10-07T09:47:18.397307+0000","last_unstale":"2023-10-07T09:47:18.402156+0000","last_undegraded":"2023-10-07T09:47:18.402156+0000","last_fullsized":"2023-10-07T09:47:18.402156+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:09.638520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.377760+0000","last_change":"2023-10-07T09:47:17.391056+0000","last_active":"2023-10-07T09:47:18.377760+0000","last_peered":"2023-10-07T09:47:18.377760+0000","last_clean":"2023-10-07T09:47:18.377760+0000","last_became_active":"2023-10-07T09:47:17.390741+0000","last_became_peered":"2023-10-07T09:47:17.390741+0000","last_unstale":"2023-10-07T09:47:18.377760+0000","last_undegraded":"2023-10-07T09:47:18.377760+0000","last_fullsized":"2023-10-07T09:47:18.377760+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:58.691325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407928+0000","last_change":"2023-10-07T09:47:18.402491+0000","last_active":"2023-10-07T09:47:18.407928+0000","last_peered":"2023-10-07T09:47:18.407928+0000","last_clean":"2023-10-07T09:47:18.407928+0000","last_became_active":"2023-10-07T09:47:18.402241+0000","last_became_peered":"2023-10-07T09:47:18.402241+0000","last_unstale":"2023-10-07T09:47:18.407928+0000","last_undegraded":"2023-10-07T09:47:18.407928+0000","last_fullsized":"2023-10-07T09:47:18.407928+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:31:22.535981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.378965+0000","last_change":"2023-10-07T09:47:17.393524+0000","last_active":"2023-10-07T09:47:18.378965+0000","last_peered":"2023-10-07T09:47:18.378965+0000","last_clean":"2023-10-07T09:47:18.378965+0000","last_became_active":"2023-10-07T09:47:17.392943+0000","last_became_peered":"2023-10-07T09:47:17.392943+0000","last_unstale":"2023-10-07T09:47:18.378965+0000","last_undegraded":"2023-10-07T09:47:18.378965+0000","last_fullsized":"2023-10-07T09:47:18.378965+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:57.782277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401619+0000","last_change":"2023-10-07T09:47:18.394083+0000","last_active":"2023-10-07T09:47:18.401619+0000","last_peered":"2023-10-07T09:47:18.401619+0000","last_clean":"2023-10-07T09:47:18.401619+0000","last_became_active":"2023-10-07T09:47:18.393780+0000","last_became_peered":"2023-10-07T09:47:18.393780+0000","last_unstale":"2023-10-07T09:47:18.401619+0000","last_undegraded":"2023-10-07T09:47:18.401619+0000","last_fullsized":"2023-10-07T09:47:18.401619+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:39:47.987946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.380305+0000","last_change":"2023-10-07T09:47:17.387596+0000","last_active":"2023-10-07T09:47:18.380305+0000","last_peered":"2023-10-07T09:47:18.380305+0000","last_clean":"2023-10-07T09:47:18.380305+0000","last_became_active":"2023-10-07T09:47:17.387325+0000","last_became_peered":"2023-10-07T09:47:17.387325+0000","last_unstale":"2023-10-07T09:47:18.380305+0000","last_undegraded":"2023-10-07T09:47:18.380305+0000","last_fullsized":"2023-10-07T09:47:18.380305+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:21:16.370493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403309+0000","last_change":"2023-10-07T09:47:18.400564+0000","last_active":"2023-10-07T09:47:18.403309+0000","last_peered":"2023-10-07T09:47:18.403309+0000","last_clean":"2023-10-07T09:47:18.403309+0000","last_became_active":"2023-10-07T09:47:18.400260+0000","last_became_peered":"2023-10-07T09:47:18.400260+0000","last_unstale":"2023-10-07T09:47:18.403309+0000","last_undegraded":"2023-10-07T09:47:18.403309+0000","last_fullsized":"2023-10-07T09:47:18.403309+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:06.865710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.378666+0000","last_change":"2023-10-07T09:47:17.394513+0000","last_active":"2023-10-07T09:47:18.378666+0000","last_peered":"2023-10-07T09:47:18.378666+0000","last_clean":"2023-10-07T09:47:18.378666+0000","last_became_active":"2023-10-07T09:47:17.394224+0000","last_became_peered":"2023-10-07T09:47:17.394224+0000","last_unstale":"2023-10-07T09:47:18.378666+0000","last_undegraded":"2023-10-07T09:47:18.378666+0000","last_fullsized":"2023-10-07T09:47:18.378666+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:54:43.114199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403297+0000","last_change":"2023-10-07T09:47:18.397689+0000","last_active":"2023-10-07T09:47:18.403297+0000","last_peered":"2023-10-07T09:47:18.403297+0000","last_clean":"2023-10-07T09:47:18.403297+0000","last_became_active":"2023-10-07T09:47:18.396845+0000","last_became_peered":"2023-10-07T09:47:18.396845+0000","last_unstale":"2023-10-07T09:47:18.403297+0000","last_undegraded":"2023-10-07T09:47:18.403297+0000","last_fullsized":"2023-10-07T09:47:18.403297+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:59.475818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.379978+0000","last_change":"2023-10-07T09:47:17.389298+0000","last_active":"2023-10-07T09:47:18.379978+0000","last_peered":"2023-10-07T09:47:18.379978+0000","last_clean":"2023-10-07T09:47:18.379978+0000","last_became_active":"2023-10-07T09:47:17.388979+0000","last_became_peered":"2023-10-07T09:47:17.388979+0000","last_unstale":"2023-10-07T09:47:18.379978+0000","last_undegraded":"2023-10-07T09:47:18.379978+0000","last_fullsized":"2023-10-07T09:47:18.379978+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:21:27.986947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398888+0000","last_change":"2023-10-07T09:47:18.394879+0000","last_active":"2023-10-07T09:47:18.398888+0000","last_peered":"2023-10-07T09:47:18.398888+0000","last_clean":"2023-10-07T09:47:18.398888+0000","last_became_active":"2023-10-07T09:47:18.394192+0000","last_became_peered":"2023-10-07T09:47:18.394192+0000","last_unstale":"2023-10-07T09:47:18.398888+0000","last_undegraded":"2023-10-07T09:47:18.398888+0000","last_fullsized":"2023-10-07T09:47:18.398888+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:51:21.500686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.372194+0000","last_change":"2023-10-07T09:47:17.380592+0000","last_active":"2023-10-07T09:47:18.372194+0000","last_peered":"2023-10-07T09:47:18.372194+0000","last_clean":"2023-10-07T09:47:18.372194+0000","last_became_active":"2023-10-07T09:47:17.380304+0000","last_became_peered":"2023-10-07T09:47:17.380304+0000","last_unstale":"2023-10-07T09:47:18.372194+0000","last_undegraded":"2023-10-07T09:47:18.372194+0000","last_fullsized":"2023-10-07T09:47:18.372194+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:10.219255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.399839+0000","last_change":"2023-10-07T09:47:18.391011+0000","last_active":"2023-10-07T09:47:18.399839+0000","last_peered":"2023-10-07T09:47:18.399839+0000","last_clean":"2023-10-07T09:47:18.399839+0000","last_became_active":"2023-10-07T09:47:18.390628+0000","last_became_peered":"2023-10-07T09:47:18.390628+0000","last_unstale":"2023-10-07T09:47:18.399839+0000","last_undegraded":"2023-10-07T09:47:18.399839+0000","last_fullsized":"2023-10-07T09:47:18.399839+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:52:43.789995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.379674+0000","last_change":"2023-10-07T09:47:17.390637+0000","last_active":"2023-10-07T09:47:18.379674+0000","last_peered":"2023-10-07T09:47:18.379674+0000","last_clean":"2023-10-07T09:47:18.379674+0000","last_became_active":"2023-10-07T09:47:17.390375+0000","last_became_peered":"2023-10-07T09:47:17.390375+0000","last_unstale":"2023-10-07T09:47:18.379674+0000","last_undegraded":"2023-10-07T09:47:18.379674+0000","last_fullsized":"2023-10-07T09:47:18.379674+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:53:12.306910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.399151+0000","last_change":"2023-10-07T09:47:18.398220+0000","last_active":"2023-10-07T09:47:18.399151+0000","last_peered":"2023-10-07T09:47:18.399151+0000","last_clean":"2023-10-07T09:47:18.399151+0000","last_became_active":"2023-10-07T09:47:18.395665+0000","last_became_peered":"2023-10-07T09:47:18.395665+0000","last_unstale":"2023-10-07T09:47:18.399151+0000","last_undegraded":"2023-10-07T09:47:18.399151+0000","last_fullsized":"2023-10-07T09:47:18.399151+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:50.493566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404938+0000","last_change":"2023-10-07T09:47:17.384141+0000","last_active":"2023-10-07T09:47:18.404938+0000","last_peered":"2023-10-07T09:47:18.404938+0000","last_clean":"2023-10-07T09:47:18.404938+0000","last_became_active":"2023-10-07T09:47:17.383875+0000","last_became_peered":"2023-10-07T09:47:17.383875+0000","last_unstale":"2023-10-07T09:47:18.404938+0000","last_undegraded":"2023-10-07T09:47:18.404938+0000","last_fullsized":"2023-10-07T09:47:18.404938+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:41:13.214293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"108'227","reported_seq":634,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384116+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:47:18.384116+0000","last_peered":"2023-10-07T09:47:18.384116+0000","last_clean":"2023-10-07T09:47:18.384116+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:47:18.384116+0000","last_undegraded":"2023-10-07T09:47:18.384116+0000","last_fullsized":"2023-10-07T09:47:18.384116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476950,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303388,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068452,"statfs":{"total":96636764160,"available":96326094848,"internally_reserved":0,"allocated":2052096,"data_stored":1288443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.69699999999999995,"15min":0.71899999999999997},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":0.89600000000000002,"5min":0.96599999999999997,"15min":1.026},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69899999999999995,"15min":0.72399999999999998},"min":{"1min":0.51900000000000002,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":0.92500000000000004,"5min":0.94899999999999995,"15min":1.0349999999999999},"last":0.59599999999999997}]},{"osd":1,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71599999999999997,"15min":0.73499999999999999},"min":{"1min":0.46600000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.014},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.74099999999999999,"15min":0.746},"min":{"1min":0.58399999999999996,"5min":0.45900000000000002,"15min":0.45600000000000002},"max":{"1min":0.94499999999999995,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.378}]},{"osd":2,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.72399999999999998,"15min":0.752},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96899999999999997,"5min":1.206,"15min":1.206},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.74199999999999999},"min":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.92600000000000005,"5min":1.038,"15min":1.071},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.58499999999999996,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.01},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.96299999999999997,"5min":1.137,"15min":1.137},"last":0.81399999999999995}]},{"osd":4,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.439,"5min":0.438,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.93999999999999995,"15min":1.002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66600000000000004,"15min":0.64200000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.629}]},{"osd":5,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.36899999999999999},"max":{"1min":0.93700000000000006,"5min":1.27,"15min":1.27},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45000000000000001},"max":{"1min":0.877,"5min":1.0069999999999999,"15min":1.0189999999999999},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.75800000000000001,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.93400000000000005,"5min":1.077,"15min":1.077},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73499999999999999,"15min":0.72599999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.018,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.58599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509654,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302808,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069032,"statfs":{"total":96636764160,"available":96326688768,"internally_reserved":0,"allocated":1458176,"data_stored":698587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":1.012,"5min":1.274,"15min":1.274},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68300000000000005,"15min":0.69899999999999995},"min":{"1min":0.54200000000000004,"5min":0.40699999999999997,"15min":0.34999999999999998},"max":{"1min":1.012,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.69799999999999995,"15min":0.72199999999999998},"min":{"1min":0.55500000000000005,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":0.90000000000000002,"5min":1.169,"15min":1.169},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.51500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.93899999999999995,"5min":1.7,"15min":1.7},"last":0.57799999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.73599999999999999},"min":{"1min":0.46800000000000003,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":0.98199999999999998,"5min":1.482,"15min":1.482},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.038,"5min":1.643,"15min":1.643},"last":0.67700000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.45200000000000001},"max":{"1min":0.95699999999999996,"5min":1.458,"15min":1.458},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.73999999999999999,"15min":0.75800000000000001},"min":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97699999999999998,"5min":1.006,"15min":1.006},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.61599999999999999,"15min":0.61799999999999999},"min":{"1min":0.40500000000000003,"5min":0.36599999999999999,"15min":0.29899999999999999},"max":{"1min":0.879,"5min":0.92400000000000004,"15min":1.01},"last":0.624},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46000000000000002,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.94399999999999995,"5min":1.018,"15min":1.018},"last":0.76400000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67200000000000004,"15min":0.67300000000000004},"min":{"1min":0.50900000000000001,"5min":0.42299999999999999,"15min":0.41199999999999998},"max":{"1min":0.90800000000000003,"5min":1.036,"15min":1.0509999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.498,"5min":0.439,"15min":0.439},"max":{"1min":0.91700000000000004,"5min":1.0089999999999999,"15min":1.0449999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.69999999999999996,"15min":0.68899999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.39700000000000002},"max":{"1min":0.93200000000000005,"5min":1.099,"15min":1.099},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.57799999999999996,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95999999999999996,"5min":0.995,"15min":1.01},"last":0.78600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542356,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302800,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069040,"statfs":{"total":96636764160,"available":96326696960,"internally_reserved":0,"allocated":1449984,"data_stored":697963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.75600000000000001,"5min":0.78000000000000003,"15min":0.871},"last":0.439},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53000000000000003,"15min":0.55200000000000005},"min":{"1min":0.44900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71099999999999997,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.58799999999999997,"5min":0.73499999999999999,"15min":0.79600000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.39400000000000002,"5min":0.35699999999999998,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":0.78400000000000003,"15min":1.028},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.97599999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.54600000000000004},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69399999999999995,"5min":0.79200000000000004,"15min":0.95099999999999996},"last":0.496}]},{"osd":4,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.754,"15min":0.78300000000000003},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48599999999999999},"max":{"1min":0.90400000000000003,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72599999999999998,"15min":0.76700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97699999999999998,"5min":1.032,"15min":1.196},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.77900000000000003,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0049999999999999,"5min":1.099,"15min":1.2769999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.73899999999999999,"15min":0.78000000000000003},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92700000000000005,"5min":1.0680000000000001,"15min":1.3300000000000001},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.51500000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.131},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.76600000000000001,"15min":0.81899999999999995},"min":{"1min":0.59699999999999998,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":0.997,"5min":1.1040000000000001,"15min":1.165},"last":0.68500000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":1.038,"5min":1.0429999999999999,"15min":1.2450000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.78400000000000003,"15min":0.83899999999999997},"min":{"1min":0.59799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.03,"5min":1.137,"15min":1.385},"last":0.629}]}]},{"osd":0,"up_from":14,"seq":60129542356,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303372,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068468,"statfs":{"total":96636764160,"available":96326111232,"internally_reserved":0,"allocated":2035712,"data_stored":1282221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47299999999999998,"15min":0.48999999999999999},"min":{"1min":0.308,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.248},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.57199999999999995},"min":{"1min":0.42799999999999999,"5min":0.38600000000000001,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.434}]},{"osd":2,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.876,"15min":1.101},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47899999999999998,"15min":0.502},"min":{"1min":0.33800000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68200000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.28599999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51200000000000001,"15min":0.53400000000000003},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.749,"5min":0.83799999999999997,"15min":1.0009999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59099999999999997,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92100000000000004},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.752},"min":{"1min":0.54800000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88100000000000001,"5min":1.119,"15min":1.232},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.63600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.05,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.746,"15min":0.77900000000000003},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97799999999999998,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.76000000000000001,"15min":0.79300000000000004},"min":{"1min":0.60199999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.194},"last":0.63400000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78000000000000003,"15min":0.81100000000000005},"min":{"1min":0.64400000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.98799999999999999,"5min":1.081,"15min":1.129},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.80700000000000005},"min":{"1min":0.61699999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.206,"15min":1.206},"last":0.64700000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78600000000000003,"15min":0.82699999999999996},"min":{"1min":0.59099999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.98399999999999999,"5min":1.149,"15min":1.167},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78500000000000003,"15min":0.82399999999999995},"min":{"1min":0.64700000000000002,"5min":0.56200000000000006,"15min":0.503},"max":{"1min":0.995,"5min":1.238,"15min":1.238},"last":0.76300000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542357,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310996,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060844,"statfs":{"total":96636764160,"available":96318304256,"internally_reserved":0,"allocated":1454080,"data_stored":697997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":0.66200000000000003,"5min":0.67800000000000005,"15min":1.258},"last":0.438},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28999999999999998},"max":{"1min":0.79000000000000004,"5min":0.80100000000000005,"15min":1.2290000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.441,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.749,"5min":0.79600000000000004,"15min":1.1950000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":1.3200000000000001},"last":0.68500000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.495,"15min":0.51800000000000002},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.30599999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":1.5820000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39300000000000002,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":1.052},"last":0.69999999999999996}]},{"osd":4,"last update":"Sat Oct 7 09:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.75600000000000001},"min":{"1min":0.56699999999999995,"5min":0.46800000000000003,"15min":0.35699999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.4630000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.77500000000000002,"15min":0.80000000000000004},"min":{"1min":0.621,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":1.1499999999999999,"5min":1.327,"15min":1.8919999999999999},"last":0.97899999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.59099999999999997,"5min":0.52000000000000002,"15min":0.48999999999999999},"max":{"1min":1.095,"5min":1.173,"15min":1.98},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.78800000000000003},"min":{"1min":0.54100000000000004,"5min":0.52400000000000002,"15min":0.46100000000000002},"max":{"1min":1.042,"5min":1.208,"15min":1.5169999999999999},"last":1.075}]},{"osd":6,"last update":"Sat Oct 7 09:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78900000000000003,"15min":0.82599999999999996},"min":{"1min":0.59099999999999997,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.0720000000000001,"5min":1.1379999999999999,"15min":1.931},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.63800000000000001,"5min":0.55300000000000005,"15min":0.47299999999999998},"max":{"1min":1.194,"5min":1.2849999999999999,"15min":1.8120000000000001},"last":1.115}]},{"osd":7,"last update":"Sat Oct 7 09:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.81100000000000005,"15min":0.84499999999999997},"min":{"1min":0.66900000000000004,"5min":0.58599999999999997,"15min":0.56999999999999995},"max":{"1min":1.0800000000000001,"5min":1.2450000000000001,"15min":1.8500000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.3839999999999999},"last":1.0089999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542358,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306912,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064928,"statfs":{"total":96636764160,"available":96322486272,"internally_reserved":0,"allocated":1466368,"data_stored":699521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45600000000000002,"15min":0.45900000000000002},"min":{"1min":0.38100000000000001,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":0.76300000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":7.9939999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69499999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":3.246}]},{"osd":1,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.61299999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":5.5359999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.46600000000000003,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":1.024,"15min":1.024},"last":8.0990000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.41799999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71199999999999997,"5min":1.206,"15min":1.206},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55500000000000005,"15min":0.57399999999999995},"min":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":5.8719999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.54400000000000004,"5min":0.48899999999999999,"15min":0.44900000000000001},"max":{"1min":0.98399999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":7.3280000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.63,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":0.96599999999999997,"5min":1.1639999999999999,"15min":1.3009999999999999},"last":1.157}]},{"osd":5,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.79100000000000004,"15min":0.79900000000000004},"min":{"1min":0.57899999999999996,"5min":0.53200000000000003,"15min":0.52500000000000002},"max":{"1min":1.1000000000000001,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":3.0419999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.67700000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0529999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":5.8170000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.60199999999999998,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.0549999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":5.5789999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81299999999999994,"15min":0.82499999999999996},"min":{"1min":0.64800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.032,"5min":1.673,"15min":1.673},"last":7.9459999999999997}]},{"osd":7,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.65300000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0189999999999999,"5min":2.194,"15min":2.194},"last":6.0940000000000003},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.65500000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0700000000000001,"5min":2.403,"15min":2.403},"last":8.0470000000000006}]}]},{"osd":4,"up_from":15,"seq":64424509654,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306916,"kb_used_data":1436,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064924,"statfs":{"total":96636764160,"available":96322482176,"internally_reserved":0,"allocated":1470464,"data_stored":699623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.48599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":1.069},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.68799999999999994,"15min":0.71199999999999997},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0009999999999999},"last":0.75}]},{"osd":1,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67500000000000004,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":1.054},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.69999999999999996,"15min":0.73199999999999998},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.91700000000000004,"5min":1.478,"15min":1.478},"last":0.80100000000000005}]},{"osd":2,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.73799999999999999,"15min":0.75},"min":{"1min":0.504,"5min":0.504,"15min":0.432},"max":{"1min":0.91600000000000004,"5min":1.518,"15min":1.518},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.54500000000000004,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.93300000000000005,"5min":0.93700000000000006,"15min":1.0209999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.68500000000000005,"15min":0.71999999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.873,"5min":0.873,"15min":1.032},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.755},"min":{"1min":0.56499999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92000000000000004,"5min":1.0249999999999999,"15min":1.036},"last":0.81399999999999995}]},{"osd":5,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.427,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":1.639,"15min":1.639},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.42299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.46100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.272,"5min":1.585,"15min":1.585},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.443},"max":{"1min":0.80900000000000005,"5min":0.96299999999999997,"15min":1.0589999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.752},"min":{"1min":0.59199999999999997,"5min":0.52700000000000002,"15min":0.45200000000000001},"max":{"1min":1.5760000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.59499999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542356,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":1441792,"data_stored":691369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.75,"15min":0.75700000000000001},"min":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.39900000000000002},"max":{"1min":0.999,"5min":1.218,"15min":1.218},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77000000000000002,"15min":0.79800000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.436},"max":{"1min":0.98699999999999999,"5min":1.294,"15min":1.294},"last":0.81699999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70299999999999996,"15min":0.72899999999999998},"min":{"1min":0.52000000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.746,"15min":0.745},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94499999999999995,"5min":1.194,"15min":1.194},"last":0.78000000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.74299999999999999},"min":{"1min":0.503,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93500000000000005,"5min":1.054,"15min":1.054},"last":0.73599999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52500000000000002,"15min":0.48899999999999999},"max":{"1min":0.97399999999999998,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72199999999999998,"15min":0.74199999999999999},"min":{"1min":0.56599999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.88900000000000001,"5min":1.036,"15min":1.036},"last":0.69899999999999995}]},{"osd":4,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.40100000000000002,"15min":0.39800000000000002},"max":{"1min":0.83699999999999997,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.626,"15min":0.628},"min":{"1min":0.45700000000000002,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.86699999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63200000000000001},"min":{"1min":0.44400000000000001,"5min":0.39900000000000002,"15min":0.33900000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89700000000000002,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.86499999999999999}]},{"osd":7,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70799999999999996,"15min":0.69899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"max":{"1min":0.91800000000000004,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.91800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:47:22.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:47:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.018+0000 7f5e8abff700 1 Processor -- start 2023-10-07T09:47:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.018+0000 7f5e8abff700 1 -- start start 2023-10-07T09:47:23.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.019+0000 7f5e8abff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5e840b0710 0x7f5e8415c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.019+0000 7f5e8abff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e84154090 con 0x7f5e84153cc0 2023-10-07T09:47:23.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.019+0000 7f5e8abff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e84159f60 con 0x7f5e8415ade0 2023-10-07T09:47:23.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.019+0000 7f5e8abff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8415a240 con 0x7f5e840b0710 2023-10-07T09:47:23.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.019+0000 7f5e8899b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5e84153cc0 0x7f5e840b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39732/0 (socket says 172.21.15.29:39732) 2023-10-07T09:47:23.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e8899b700 1 -- 172.21.15.29:0/3013194007 learned_addr learned my addr 172.21.15.29:0/3013194007 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:23.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e8919c700 1 --2- 172.21.15.29:0/3013194007 >> v2:172.21.15.144:3300/0 conn(0x7f5e840b0710 0x7f5e8415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81553382 0 0) 0x7f5e84154090 con 0x7f5e84153cc0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74003570 con 0x7f5e84153cc0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780182914 0 0) 0x7f5e84159f60 con 0x7f5e8415ade0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84154090 con 0x7f5e8415ade0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4066866778 0 0) 0x7f5e74003570 con 0x7f5e84153cc0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e8919c700 1 -- 172.21.15.29:0/3013194007 >> v1:172.21.15.144:6789/0 conn(0x7f5e8415ade0 legacy=0x7f5e840b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e8919c700 1 -- 172.21.15.29:0/3013194007 >> v1:172.21.15.29:6789/0 conn(0x7f5e84153cc0 legacy=0x7f5e840b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.020+0000 7f5e8919c700 1 -- 172.21.15.29:0/3013194007 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8415a520 con 0x7f5e840b0710 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.021+0000 7f5e8919c700 1 --2- 172.21.15.29:0/3013194007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e840b0710 0x7f5e8415c240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009e60 tx=0x7f5e7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6dd7d12d336588 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:23.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.021+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00e070 con 0x7f5e840b0710 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.021+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c00b7a0 con 0x7f5e840b0710 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.021+0000 7f5e837fe700 1 -- 172.21.15.29:0/3013194007 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780015f0 con 0x7f5e840b0710 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.021+0000 7f5e8abff700 1 -- 172.21.15.29:0/3013194007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e840b0710 msgr2=0x7f5e8415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.021+0000 7f5e8abff700 1 --2- 172.21.15.29:0/3013194007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e840b0710 0x7f5e8415c240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009e60 tx=0x7f5e7c009780 comp rx=0 tx=0).stop 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.022+0000 7f5e8abff700 1 -- 172.21.15.29:0/3013194007 shutdown_connections 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.022+0000 7f5e8abff700 1 --2- 172.21.15.29:0/3013194007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e840b0710 0x7f5e8415c240 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.022+0000 7f5e8abff700 1 -- 172.21.15.29:0/3013194007 >> 172.21.15.29:0/3013194007 conn(0x7f5e84067610 msgr2=0x7f5e84067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.022+0000 7f5e8abff700 1 -- 172.21.15.29:0/3013194007 shutdown_connections 2023-10-07T09:47:23.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.022+0000 7f5e8abff700 1 -- 172.21.15.29:0/3013194007 wait complete. 2023-10-07T09:47:23.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.023+0000 7f5e8abff700 1 Processor -- start 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.023+0000 7f5e8abff700 1 -- start start 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.024+0000 7f5e8abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e840b0710 0x7f5e840c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.024+0000 7f5e8abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e84153cc0 0x7f5e84146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.024+0000 7f5e8abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 0x7f5e84147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.024+0000 7f5e8abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840c75b0 con 0x7f5e8415ade0 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.024+0000 7f5e8abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e84157d70 con 0x7f5e84153cc0 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.024+0000 7f5e8abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840c7800 con 0x7f5e840b0710 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 0x7f5e84147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 0x7f5e84147110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59976/0 (socket says 172.21.15.29:59976) 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 -- 172.21.15.29:0/2787418268 learned_addr learned my addr 172.21.15.29:0/2787418268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8899b700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e840b0710 0x7f5e840c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e83fff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e84153cc0 0x7f5e84146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 -- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e840b0710 msgr2=0x7f5e840c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e840b0710 0x7f5e840c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 -- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e84153cc0 msgr2=0x7f5e84146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e84153cc0 0x7f5e84146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 -- 172.21.15.29:0/2787418268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c009a20 con 0x7f5e8415ade0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e83fff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e84153cc0 0x7f5e84146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e8919c700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 0x7f5e84147110 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0097c0 tx=0x7f5e7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.025+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c012d60 con 0x7f5e8415ade0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.026+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c01e450 con 0x7f5e8415ade0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.026+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e840c7ae0 con 0x7f5e8415ade0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.026+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c01e7d0 con 0x7f5e8415ade0 2023-10-07T09:47:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.026+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e840ca4b0 con 0x7f5e8415ade0 2023-10-07T09:47:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.027+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e840ca6a0 con 0x7f5e8415ade0 2023-10-07T09:47:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.028+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5e7c01eb40 con 0x7f5e8415ade0 2023-10-07T09:47:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.028+0000 7f5e80ff9700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5e68066f30 0x7f5e680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.028+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5e7c086c40 con 0x7f5e8415ade0 2023-10-07T09:47:23.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.029+0000 7f5e8899b700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5e68066f30 0x7f5e680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.029+0000 7f5e8899b700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5e68066f30 0x7f5e680693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000c00 tx=0x7f5e78009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:23.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.032+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e7c053d30 con 0x7f5e8415ade0 2023-10-07T09:47:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.232+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e840c8000 con 0x7f5e68066f30 2023-10-07T09:47:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.241+0000 7f5e80ff9700 1 -- 172.21.15.29:0/2787418268 <== mgr.7162 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289181 (secure 0 0 0) 0x7f5e840c8000 con 0x7f5e68066f30 2023-10-07T09:47:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5e68066f30 msgr2=0x7f5e680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5e68066f30 0x7f5e680693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000c00 tx=0x7f5e78009040 comp rx=0 tx=0).stop 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 msgr2=0x7f5e84147110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 0x7f5e84147110 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0097c0 tx=0x7f5e7c017000 comp rx=0 tx=0).stop 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 shutdown_connections 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5e68066f30 0x7f5e680693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e8415ade0 0x7f5e84147110 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e840b0710 0x7f5e840c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 --2- 172.21.15.29:0/2787418268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e84153cc0 0x7f5e84146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 >> 172.21.15.29:0/2787418268 conn(0x7f5e84067610 msgr2=0x7f5e8415b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.245+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 shutdown_connections 2023-10-07T09:47:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.246+0000 7f5e8abff700 1 -- 172.21.15.29:0/2787418268 wait complete. 2023-10-07T09:47:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:47:23.264 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T09:47:22.472701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439984,"kb_used_data":12528,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752534736,"statfs":{"total":773094113280,"available":770595569664,"internally_reserved":0,"allocated":12828672,"data_stored":6755724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001699"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.395327+0000","last_change":"2023-10-07T09:47:18.393444+0000","last_active":"2023-10-07T09:47:18.395327+0000","last_peered":"2023-10-07T09:47:18.395327+0000","last_clean":"2023-10-07T09:47:18.395327+0000","last_became_active":"2023-10-07T09:47:18.393186+0000","last_became_peered":"2023-10-07T09:47:18.393186+0000","last_unstale":"2023-10-07T09:47:18.395327+0000","last_undegraded":"2023-10-07T09:47:18.395327+0000","last_fullsized":"2023-10-07T09:47:18.395327+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:02:07.742177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404898+0000","last_change":"2023-10-07T09:47:18.391919+0000","last_active":"2023-10-07T09:47:18.404898+0000","last_peered":"2023-10-07T09:47:18.404898+0000","last_clean":"2023-10-07T09:47:18.404898+0000","last_became_active":"2023-10-07T09:47:18.391600+0000","last_became_peered":"2023-10-07T09:47:18.391600+0000","last_unstale":"2023-10-07T09:47:18.404898+0000","last_undegraded":"2023-10-07T09:47:18.404898+0000","last_fullsized":"2023-10-07T09:47:18.404898+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:45:17.030160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.390564+0000","last_change":"2023-10-07T09:47:18.389275+0000","last_active":"2023-10-07T09:47:18.390564+0000","last_peered":"2023-10-07T09:47:18.390564+0000","last_clean":"2023-10-07T09:47:18.390564+0000","last_became_active":"2023-10-07T09:47:18.388996+0000","last_became_peered":"2023-10-07T09:47:18.388996+0000","last_unstale":"2023-10-07T09:47:18.390564+0000","last_undegraded":"2023-10-07T09:47:18.390564+0000","last_fullsized":"2023-10-07T09:47:18.390564+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:58.133179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403803+0000","last_change":"2023-10-07T09:47:18.398592+0000","last_active":"2023-10-07T09:47:18.403803+0000","last_peered":"2023-10-07T09:47:18.403803+0000","last_clean":"2023-10-07T09:47:18.403803+0000","last_became_active":"2023-10-07T09:47:18.398038+0000","last_became_peered":"2023-10-07T09:47:18.398038+0000","last_unstale":"2023-10-07T09:47:18.403803+0000","last_undegraded":"2023-10-07T09:47:18.403803+0000","last_fullsized":"2023-10-07T09:47:18.403803+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:58:13.384898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.402243+0000","last_change":"2023-10-07T09:47:18.393985+0000","last_active":"2023-10-07T09:47:18.402243+0000","last_peered":"2023-10-07T09:47:18.402243+0000","last_clean":"2023-10-07T09:47:18.402243+0000","last_became_active":"2023-10-07T09:47:18.393672+0000","last_became_peered":"2023-10-07T09:47:18.393672+0000","last_unstale":"2023-10-07T09:47:18.402243+0000","last_undegraded":"2023-10-07T09:47:18.402243+0000","last_fullsized":"2023-10-07T09:47:18.402243+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:54.054174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398502+0000","last_change":"2023-10-07T09:47:18.394136+0000","last_active":"2023-10-07T09:47:18.398502+0000","last_peered":"2023-10-07T09:47:18.398502+0000","last_clean":"2023-10-07T09:47:18.398502+0000","last_became_active":"2023-10-07T09:47:18.393879+0000","last_became_peered":"2023-10-07T09:47:18.393879+0000","last_unstale":"2023-10-07T09:47:18.398502+0000","last_undegraded":"2023-10-07T09:47:18.398502+0000","last_fullsized":"2023-10-07T09:47:18.398502+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:47:14.028161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398305+0000","last_change":"2023-10-07T09:47:18.393588+0000","last_active":"2023-10-07T09:47:18.398305+0000","last_peered":"2023-10-07T09:47:18.398305+0000","last_clean":"2023-10-07T09:47:18.398305+0000","last_became_active":"2023-10-07T09:47:18.393279+0000","last_became_peered":"2023-10-07T09:47:18.393279+0000","last_unstale":"2023-10-07T09:47:18.398305+0000","last_undegraded":"2023-10-07T09:47:18.398305+0000","last_fullsized":"2023-10-07T09:47:18.398305+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:55:51.792984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404821+0000","last_change":"2023-10-07T09:47:18.395396+0000","last_active":"2023-10-07T09:47:18.404821+0000","last_peered":"2023-10-07T09:47:18.404821+0000","last_clean":"2023-10-07T09:47:18.404821+0000","last_became_active":"2023-10-07T09:47:18.395079+0000","last_became_peered":"2023-10-07T09:47:18.395079+0000","last_unstale":"2023-10-07T09:47:18.404821+0000","last_undegraded":"2023-10-07T09:47:18.404821+0000","last_fullsized":"2023-10-07T09:47:18.404821+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:11.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.395991+0000","last_change":"2023-10-07T09:47:18.389344+0000","last_active":"2023-10-07T09:47:18.395991+0000","last_peered":"2023-10-07T09:47:18.395991+0000","last_clean":"2023-10-07T09:47:18.395991+0000","last_became_active":"2023-10-07T09:47:18.389036+0000","last_became_peered":"2023-10-07T09:47:18.389036+0000","last_unstale":"2023-10-07T09:47:18.395991+0000","last_undegraded":"2023-10-07T09:47:18.395991+0000","last_fullsized":"2023-10-07T09:47:18.395991+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:47:37.693705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401579+0000","last_change":"2023-10-07T09:47:18.394071+0000","last_active":"2023-10-07T09:47:18.401579+0000","last_peered":"2023-10-07T09:47:18.401579+0000","last_clean":"2023-10-07T09:47:18.401579+0000","last_became_active":"2023-10-07T09:47:18.393740+0000","last_became_peered":"2023-10-07T09:47:18.393740+0000","last_unstale":"2023-10-07T09:47:18.401579+0000","last_undegraded":"2023-10-07T09:47:18.401579+0000","last_fullsized":"2023-10-07T09:47:18.401579+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:17:57.209214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.400357+0000","last_change":"2023-10-07T09:47:18.398829+0000","last_active":"2023-10-07T09:47:18.400357+0000","last_peered":"2023-10-07T09:47:18.400357+0000","last_clean":"2023-10-07T09:47:18.400357+0000","last_became_active":"2023-10-07T09:47:18.398560+0000","last_became_peered":"2023-10-07T09:47:18.398560+0000","last_unstale":"2023-10-07T09:47:18.400357+0000","last_undegraded":"2023-10-07T09:47:18.400357+0000","last_fullsized":"2023-10-07T09:47:18.400357+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:31:25.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.396615+0000","last_change":"2023-10-07T09:47:18.389856+0000","last_active":"2023-10-07T09:47:18.396615+0000","last_peered":"2023-10-07T09:47:18.396615+0000","last_clean":"2023-10-07T09:47:18.396615+0000","last_became_active":"2023-10-07T09:47:18.389407+0000","last_became_peered":"2023-10-07T09:47:18.389407+0000","last_unstale":"2023-10-07T09:47:18.396615+0000","last_undegraded":"2023-10-07T09:47:18.396615+0000","last_fullsized":"2023-10-07T09:47:18.396615+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:11.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.388410+0000","last_change":"2023-10-07T09:47:18.379946+0000","last_active":"2023-10-07T09:47:18.388410+0000","last_peered":"2023-10-07T09:47:18.388410+0000","last_clean":"2023-10-07T09:47:18.388410+0000","last_became_active":"2023-10-07T09:47:18.379691+0000","last_became_peered":"2023-10-07T09:47:18.379691+0000","last_unstale":"2023-10-07T09:47:18.388410+0000","last_undegraded":"2023-10-07T09:47:18.388410+0000","last_fullsized":"2023-10-07T09:47:18.388410+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:47:37.693705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405185+0000","last_change":"2023-10-07T09:47:18.401256+0000","last_active":"2023-10-07T09:47:18.405185+0000","last_peered":"2023-10-07T09:47:18.405185+0000","last_clean":"2023-10-07T09:47:18.405185+0000","last_became_active":"2023-10-07T09:47:18.401004+0000","last_became_peered":"2023-10-07T09:47:18.401004+0000","last_unstale":"2023-10-07T09:47:18.405185+0000","last_undegraded":"2023-10-07T09:47:18.405185+0000","last_fullsized":"2023-10-07T09:47:18.405185+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:56:17.966226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.389822+0000","last_change":"2023-10-07T09:47:18.388682+0000","last_active":"2023-10-07T09:47:18.389822+0000","last_peered":"2023-10-07T09:47:18.389822+0000","last_clean":"2023-10-07T09:47:18.389822+0000","last_became_active":"2023-10-07T09:47:18.388322+0000","last_became_peered":"2023-10-07T09:47:18.388322+0000","last_unstale":"2023-10-07T09:47:18.389822+0000","last_undegraded":"2023-10-07T09:47:18.389822+0000","last_fullsized":"2023-10-07T09:47:18.389822+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:10:09.529294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407268+0000","last_change":"2023-10-07T09:47:18.406518+0000","last_active":"2023-10-07T09:47:18.407268+0000","last_peered":"2023-10-07T09:47:18.407268+0000","last_clean":"2023-10-07T09:47:18.407268+0000","last_became_active":"2023-10-07T09:47:18.406212+0000","last_became_peered":"2023-10-07T09:47:18.406212+0000","last_unstale":"2023-10-07T09:47:18.407268+0000","last_undegraded":"2023-10-07T09:47:18.407268+0000","last_fullsized":"2023-10-07T09:47:18.407268+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:15:15.752555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404857+0000","last_change":"2023-10-07T09:47:18.392294+0000","last_active":"2023-10-07T09:47:18.404857+0000","last_peered":"2023-10-07T09:47:18.404857+0000","last_clean":"2023-10-07T09:47:18.404857+0000","last_became_active":"2023-10-07T09:47:18.391969+0000","last_became_peered":"2023-10-07T09:47:18.391969+0000","last_unstale":"2023-10-07T09:47:18.404857+0000","last_undegraded":"2023-10-07T09:47:18.404857+0000","last_fullsized":"2023-10-07T09:47:18.404857+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:39:47.987946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403749+0000","last_change":"2023-10-07T09:47:18.398435+0000","last_active":"2023-10-07T09:47:18.403749+0000","last_peered":"2023-10-07T09:47:18.403749+0000","last_clean":"2023-10-07T09:47:18.403749+0000","last_became_active":"2023-10-07T09:47:18.398071+0000","last_became_peered":"2023-10-07T09:47:18.398071+0000","last_unstale":"2023-10-07T09:47:18.403749+0000","last_undegraded":"2023-10-07T09:47:18.403749+0000","last_fullsized":"2023-10-07T09:47:18.403749+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:29:02.888441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401276+0000","last_change":"2023-10-07T09:47:18.399925+0000","last_active":"2023-10-07T09:47:18.401276+0000","last_peered":"2023-10-07T09:47:18.401276+0000","last_clean":"2023-10-07T09:47:18.401276+0000","last_became_active":"2023-10-07T09:47:18.398090+0000","last_became_peered":"2023-10-07T09:47:18.398090+0000","last_unstale":"2023-10-07T09:47:18.401276+0000","last_undegraded":"2023-10-07T09:47:18.401276+0000","last_fullsized":"2023-10-07T09:47:18.401276+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:00:53.059580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.396594+0000","last_change":"2023-10-07T09:47:18.389676+0000","last_active":"2023-10-07T09:47:18.396594+0000","last_peered":"2023-10-07T09:47:18.396594+0000","last_clean":"2023-10-07T09:47:18.396594+0000","last_became_active":"2023-10-07T09:47:18.389014+0000","last_became_peered":"2023-10-07T09:47:18.389014+0000","last_unstale":"2023-10-07T09:47:18.396594+0000","last_undegraded":"2023-10-07T09:47:18.396594+0000","last_fullsized":"2023-10-07T09:47:18.396594+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:00.591766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401300+0000","last_change":"2023-10-07T09:47:18.399299+0000","last_active":"2023-10-07T09:47:18.401300+0000","last_peered":"2023-10-07T09:47:18.401300+0000","last_clean":"2023-10-07T09:47:18.401300+0000","last_became_active":"2023-10-07T09:47:18.398738+0000","last_became_peered":"2023-10-07T09:47:18.398738+0000","last_unstale":"2023-10-07T09:47:18.401300+0000","last_undegraded":"2023-10-07T09:47:18.401300+0000","last_fullsized":"2023-10-07T09:47:18.401300+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:35:21.217791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404798+0000","last_change":"2023-10-07T09:47:18.393673+0000","last_active":"2023-10-07T09:47:18.404798+0000","last_peered":"2023-10-07T09:47:18.404798+0000","last_clean":"2023-10-07T09:47:18.404798+0000","last_became_active":"2023-10-07T09:47:18.393410+0000","last_became_peered":"2023-10-07T09:47:18.393410+0000","last_unstale":"2023-10-07T09:47:18.404798+0000","last_undegraded":"2023-10-07T09:47:18.404798+0000","last_fullsized":"2023-10-07T09:47:18.404798+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:57:45.265868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404719+0000","last_change":"2023-10-07T09:47:18.393324+0000","last_active":"2023-10-07T09:47:18.404719+0000","last_peered":"2023-10-07T09:47:18.404719+0000","last_clean":"2023-10-07T09:47:18.404719+0000","last_became_active":"2023-10-07T09:47:18.393054+0000","last_became_peered":"2023-10-07T09:47:18.393054+0000","last_unstale":"2023-10-07T09:47:18.404719+0000","last_undegraded":"2023-10-07T09:47:18.404719+0000","last_fullsized":"2023-10-07T09:47:18.404719+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:55:36.237343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.399673+0000","last_change":"2023-10-07T09:47:18.397767+0000","last_active":"2023-10-07T09:47:18.399673+0000","last_peered":"2023-10-07T09:47:18.399673+0000","last_clean":"2023-10-07T09:47:18.399673+0000","last_became_active":"2023-10-07T09:47:18.396862+0000","last_became_peered":"2023-10-07T09:47:18.396862+0000","last_unstale":"2023-10-07T09:47:18.399673+0000","last_undegraded":"2023-10-07T09:47:18.399673+0000","last_fullsized":"2023-10-07T09:47:18.399673+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:36:22.235575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401691+0000","last_change":"2023-10-07T09:47:18.395897+0000","last_active":"2023-10-07T09:47:18.401691+0000","last_peered":"2023-10-07T09:47:18.401691+0000","last_clean":"2023-10-07T09:47:18.401691+0000","last_became_active":"2023-10-07T09:47:18.395251+0000","last_became_peered":"2023-10-07T09:47:18.395251+0000","last_unstale":"2023-10-07T09:47:18.401691+0000","last_undegraded":"2023-10-07T09:47:18.401691+0000","last_fullsized":"2023-10-07T09:47:18.401691+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:02:02.383028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404859+0000","last_change":"2023-10-07T09:47:18.391758+0000","last_active":"2023-10-07T09:47:18.404859+0000","last_peered":"2023-10-07T09:47:18.404859+0000","last_clean":"2023-10-07T09:47:18.404859+0000","last_became_active":"2023-10-07T09:47:18.391338+0000","last_became_peered":"2023-10-07T09:47:18.391338+0000","last_unstale":"2023-10-07T09:47:18.404859+0000","last_undegraded":"2023-10-07T09:47:18.404859+0000","last_fullsized":"2023-10-07T09:47:18.404859+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:11.717213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403737+0000","last_change":"2023-10-07T09:47:18.400589+0000","last_active":"2023-10-07T09:47:18.403737+0000","last_peered":"2023-10-07T09:47:18.403737+0000","last_clean":"2023-10-07T09:47:18.403737+0000","last_became_active":"2023-10-07T09:47:18.400177+0000","last_became_peered":"2023-10-07T09:47:18.400177+0000","last_unstale":"2023-10-07T09:47:18.403737+0000","last_undegraded":"2023-10-07T09:47:18.403737+0000","last_fullsized":"2023-10-07T09:47:18.403737+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:19.684823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401750+0000","last_change":"2023-10-07T09:47:18.394976+0000","last_active":"2023-10-07T09:47:18.401750+0000","last_peered":"2023-10-07T09:47:18.401750+0000","last_clean":"2023-10-07T09:47:18.401750+0000","last_became_active":"2023-10-07T09:47:18.394641+0000","last_became_peered":"2023-10-07T09:47:18.394641+0000","last_unstale":"2023-10-07T09:47:18.401750+0000","last_undegraded":"2023-10-07T09:47:18.401750+0000","last_fullsized":"2023-10-07T09:47:18.401750+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:11.717213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404564+0000","last_change":"2023-10-07T09:47:18.391236+0000","last_active":"2023-10-07T09:47:18.404564+0000","last_peered":"2023-10-07T09:47:18.404564+0000","last_clean":"2023-10-07T09:47:18.404564+0000","last_became_active":"2023-10-07T09:47:18.390982+0000","last_became_peered":"2023-10-07T09:47:18.390982+0000","last_unstale":"2023-10-07T09:47:18.404564+0000","last_undegraded":"2023-10-07T09:47:18.404564+0000","last_fullsized":"2023-10-07T09:47:18.404564+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:29:02.888441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405560+0000","last_change":"2023-10-07T09:47:18.404052+0000","last_active":"2023-10-07T09:47:18.405560+0000","last_peered":"2023-10-07T09:47:18.405560+0000","last_clean":"2023-10-07T09:47:18.405560+0000","last_became_active":"2023-10-07T09:47:18.402919+0000","last_became_peered":"2023-10-07T09:47:18.402919+0000","last_unstale":"2023-10-07T09:47:18.405560+0000","last_undegraded":"2023-10-07T09:47:18.405560+0000","last_fullsized":"2023-10-07T09:47:18.405560+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:29:20.084226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403781+0000","last_change":"2023-10-07T09:47:18.398648+0000","last_active":"2023-10-07T09:47:18.403781+0000","last_peered":"2023-10-07T09:47:18.403781+0000","last_clean":"2023-10-07T09:47:18.403781+0000","last_became_active":"2023-10-07T09:47:18.398329+0000","last_became_peered":"2023-10-07T09:47:18.398329+0000","last_unstale":"2023-10-07T09:47:18.403781+0000","last_undegraded":"2023-10-07T09:47:18.403781+0000","last_fullsized":"2023-10-07T09:47:18.403781+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:46:35.633305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.408498+0000","last_change":"2023-10-07T09:47:18.406760+0000","last_active":"2023-10-07T09:47:18.408498+0000","last_peered":"2023-10-07T09:47:18.408498+0000","last_clean":"2023-10-07T09:47:18.408498+0000","last_became_active":"2023-10-07T09:47:18.406499+0000","last_became_peered":"2023-10-07T09:47:18.406499+0000","last_unstale":"2023-10-07T09:47:18.408498+0000","last_undegraded":"2023-10-07T09:47:18.408498+0000","last_fullsized":"2023-10-07T09:47:18.408498+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:59.699128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405147+0000","last_change":"2023-10-07T09:47:18.400965+0000","last_active":"2023-10-07T09:47:18.405147+0000","last_peered":"2023-10-07T09:47:18.405147+0000","last_clean":"2023-10-07T09:47:18.405147+0000","last_became_active":"2023-10-07T09:47:18.400441+0000","last_became_peered":"2023-10-07T09:47:18.400441+0000","last_unstale":"2023-10-07T09:47:18.405147+0000","last_undegraded":"2023-10-07T09:47:18.405147+0000","last_fullsized":"2023-10-07T09:47:18.405147+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:42:11.163248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"117'4","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404377+0000","last_change":"2023-10-07T09:47:17.390190+0000","last_active":"2023-10-07T09:47:18.404377+0000","last_peered":"2023-10-07T09:47:18.404377+0000","last_clean":"2023-10-07T09:47:18.404377+0000","last_became_active":"2023-10-07T09:47:17.389900+0000","last_became_peered":"2023-10-07T09:47:17.389900+0000","last_unstale":"2023-10-07T09:47:18.404377+0000","last_undegraded":"2023-10-07T09:47:18.404377+0000","last_fullsized":"2023-10-07T09:47:18.404377+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:08.630717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407259+0000","last_change":"2023-10-07T09:47:18.399227+0000","last_active":"2023-10-07T09:47:18.407259+0000","last_peered":"2023-10-07T09:47:18.407259+0000","last_clean":"2023-10-07T09:47:18.407259+0000","last_became_active":"2023-10-07T09:47:18.398958+0000","last_became_peered":"2023-10-07T09:47:18.398958+0000","last_unstale":"2023-10-07T09:47:18.407259+0000","last_undegraded":"2023-10-07T09:47:18.407259+0000","last_fullsized":"2023-10-07T09:47:18.407259+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:52:11.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407911+0000","last_change":"2023-10-07T09:47:17.389152+0000","last_active":"2023-10-07T09:47:18.407911+0000","last_peered":"2023-10-07T09:47:18.407911+0000","last_clean":"2023-10-07T09:47:18.407911+0000","last_became_active":"2023-10-07T09:47:17.388822+0000","last_became_peered":"2023-10-07T09:47:17.388822+0000","last_unstale":"2023-10-07T09:47:18.407911+0000","last_undegraded":"2023-10-07T09:47:18.407911+0000","last_fullsized":"2023-10-07T09:47:18.407911+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:04.806906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.389691+0000","last_change":"2023-10-07T09:47:18.388286+0000","last_active":"2023-10-07T09:47:18.389691+0000","last_peered":"2023-10-07T09:47:18.389691+0000","last_clean":"2023-10-07T09:47:18.389691+0000","last_became_active":"2023-10-07T09:47:18.387954+0000","last_became_peered":"2023-10-07T09:47:18.387954+0000","last_unstale":"2023-10-07T09:47:18.389691+0000","last_undegraded":"2023-10-07T09:47:18.389691+0000","last_fullsized":"2023-10-07T09:47:18.389691+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:08.152980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.406084+0000","last_change":"2023-10-07T09:47:17.393730+0000","last_active":"2023-10-07T09:47:18.406084+0000","last_peered":"2023-10-07T09:47:18.406084+0000","last_clean":"2023-10-07T09:47:18.406084+0000","last_became_active":"2023-10-07T09:47:17.393270+0000","last_became_peered":"2023-10-07T09:47:17.393270+0000","last_unstale":"2023-10-07T09:47:18.406084+0000","last_undegraded":"2023-10-07T09:47:18.406084+0000","last_fullsized":"2023-10-07T09:47:18.406084+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:14:52.406463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384365+0000","last_change":"2023-10-07T09:47:18.379617+0000","last_active":"2023-10-07T09:47:18.384365+0000","last_peered":"2023-10-07T09:47:18.384365+0000","last_clean":"2023-10-07T09:47:18.384365+0000","last_became_active":"2023-10-07T09:47:18.379306+0000","last_became_peered":"2023-10-07T09:47:18.379306+0000","last_unstale":"2023-10-07T09:47:18.384365+0000","last_undegraded":"2023-10-07T09:47:18.384365+0000","last_fullsized":"2023-10-07T09:47:18.384365+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:46:35.633305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404456+0000","last_change":"2023-10-07T09:47:17.392660+0000","last_active":"2023-10-07T09:47:18.404456+0000","last_peered":"2023-10-07T09:47:18.404456+0000","last_clean":"2023-10-07T09:47:18.404456+0000","last_became_active":"2023-10-07T09:47:17.392398+0000","last_became_peered":"2023-10-07T09:47:17.392398+0000","last_unstale":"2023-10-07T09:47:18.404456+0000","last_undegraded":"2023-10-07T09:47:18.404456+0000","last_fullsized":"2023-10-07T09:47:18.404456+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:34.755567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.392700+0000","last_change":"2023-10-07T09:47:18.387310+0000","last_active":"2023-10-07T09:47:18.392700+0000","last_peered":"2023-10-07T09:47:18.392700+0000","last_clean":"2023-10-07T09:47:18.392700+0000","last_became_active":"2023-10-07T09:47:18.387010+0000","last_became_peered":"2023-10-07T09:47:18.387010+0000","last_unstale":"2023-10-07T09:47:18.392700+0000","last_undegraded":"2023-10-07T09:47:18.392700+0000","last_fullsized":"2023-10-07T09:47:18.392700+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:58.133179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.402752+0000","last_change":"2023-10-07T09:47:17.388067+0000","last_active":"2023-10-07T09:47:18.402752+0000","last_peered":"2023-10-07T09:47:18.402752+0000","last_clean":"2023-10-07T09:47:18.402752+0000","last_became_active":"2023-10-07T09:47:17.387799+0000","last_became_peered":"2023-10-07T09:47:17.387799+0000","last_unstale":"2023-10-07T09:47:18.402752+0000","last_undegraded":"2023-10-07T09:47:18.402752+0000","last_fullsized":"2023-10-07T09:47:18.402752+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:42:10.155445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398352+0000","last_change":"2023-10-07T09:47:18.396700+0000","last_active":"2023-10-07T09:47:18.398352+0000","last_peered":"2023-10-07T09:47:18.398352+0000","last_clean":"2023-10-07T09:47:18.398352+0000","last_became_active":"2023-10-07T09:47:18.396410+0000","last_became_peered":"2023-10-07T09:47:18.396410+0000","last_unstale":"2023-10-07T09:47:18.398352+0000","last_undegraded":"2023-10-07T09:47:18.398352+0000","last_fullsized":"2023-10-07T09:47:18.398352+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:54:36.644856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403780+0000","last_change":"2023-10-07T09:47:17.389738+0000","last_active":"2023-10-07T09:47:18.403780+0000","last_peered":"2023-10-07T09:47:18.403780+0000","last_clean":"2023-10-07T09:47:18.403780+0000","last_became_active":"2023-10-07T09:47:17.389432+0000","last_became_peered":"2023-10-07T09:47:17.389432+0000","last_unstale":"2023-10-07T09:47:18.403780+0000","last_undegraded":"2023-10-07T09:47:18.403780+0000","last_fullsized":"2023-10-07T09:47:18.403780+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:02:06.734374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.391686+0000","last_change":"2023-10-07T09:47:18.389121+0000","last_active":"2023-10-07T09:47:18.391686+0000","last_peered":"2023-10-07T09:47:18.391686+0000","last_clean":"2023-10-07T09:47:18.391686+0000","last_became_active":"2023-10-07T09:47:18.388851+0000","last_became_peered":"2023-10-07T09:47:18.388851+0000","last_unstale":"2023-10-07T09:47:18.391686+0000","last_undegraded":"2023-10-07T09:47:18.391686+0000","last_fullsized":"2023-10-07T09:47:18.391686+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:41.832204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.380545+0000","last_change":"2023-10-07T09:47:17.392020+0000","last_active":"2023-10-07T09:47:18.380545+0000","last_peered":"2023-10-07T09:47:18.380545+0000","last_clean":"2023-10-07T09:47:18.380545+0000","last_became_active":"2023-10-07T09:47:17.391769+0000","last_became_peered":"2023-10-07T09:47:17.391769+0000","last_unstale":"2023-10-07T09:47:18.380545+0000","last_undegraded":"2023-10-07T09:47:18.380545+0000","last_fullsized":"2023-10-07T09:47:18.380545+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:24:00.728816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384089+0000","last_change":"2023-10-07T09:47:18.381375+0000","last_active":"2023-10-07T09:47:18.384089+0000","last_peered":"2023-10-07T09:47:18.384089+0000","last_clean":"2023-10-07T09:47:18.384089+0000","last_became_active":"2023-10-07T09:47:18.381068+0000","last_became_peered":"2023-10-07T09:47:18.381068+0000","last_unstale":"2023-10-07T09:47:18.384089+0000","last_undegraded":"2023-10-07T09:47:18.384089+0000","last_fullsized":"2023-10-07T09:47:18.384089+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:41.832204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.382641+0000","last_change":"2023-10-07T09:47:17.384581+0000","last_active":"2023-10-07T09:47:18.382641+0000","last_peered":"2023-10-07T09:47:18.382641+0000","last_clean":"2023-10-07T09:47:18.382641+0000","last_became_active":"2023-10-07T09:47:17.384302+0000","last_became_peered":"2023-10-07T09:47:17.384302+0000","last_unstale":"2023-10-07T09:47:18.382641+0000","last_undegraded":"2023-10-07T09:47:18.382641+0000","last_fullsized":"2023-10-07T09:47:18.382641+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:35:20.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404881+0000","last_change":"2023-10-07T09:47:18.391855+0000","last_active":"2023-10-07T09:47:18.404881+0000","last_peered":"2023-10-07T09:47:18.404881+0000","last_clean":"2023-10-07T09:47:18.404881+0000","last_became_active":"2023-10-07T09:47:18.391510+0000","last_became_peered":"2023-10-07T09:47:18.391510+0000","last_unstale":"2023-10-07T09:47:18.404881+0000","last_undegraded":"2023-10-07T09:47:18.404881+0000","last_fullsized":"2023-10-07T09:47:18.404881+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:52:43.789995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.382341+0000","last_change":"2023-10-07T09:47:17.389013+0000","last_active":"2023-10-07T09:47:18.382341+0000","last_peered":"2023-10-07T09:47:18.382341+0000","last_clean":"2023-10-07T09:47:18.382341+0000","last_became_active":"2023-10-07T09:47:17.388745+0000","last_became_peered":"2023-10-07T09:47:17.388745+0000","last_unstale":"2023-10-07T09:47:18.382341+0000","last_undegraded":"2023-10-07T09:47:18.382341+0000","last_fullsized":"2023-10-07T09:47:18.382341+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:05.857907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384051+0000","last_change":"2023-10-07T09:47:18.380863+0000","last_active":"2023-10-07T09:47:18.384051+0000","last_peered":"2023-10-07T09:47:18.384051+0000","last_clean":"2023-10-07T09:47:18.384051+0000","last_became_active":"2023-10-07T09:47:18.380168+0000","last_became_peered":"2023-10-07T09:47:18.380168+0000","last_unstale":"2023-10-07T09:47:18.384051+0000","last_undegraded":"2023-10-07T09:47:18.384051+0000","last_fullsized":"2023-10-07T09:47:18.384051+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:58:13.384898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373252+0000","last_change":"2023-10-07T09:47:17.392199+0000","last_active":"2023-10-07T09:47:18.373252+0000","last_peered":"2023-10-07T09:47:18.373252+0000","last_clean":"2023-10-07T09:47:18.373252+0000","last_became_active":"2023-10-07T09:47:17.391849+0000","last_became_peered":"2023-10-07T09:47:17.391849+0000","last_unstale":"2023-10-07T09:47:18.373252+0000","last_undegraded":"2023-10-07T09:47:18.373252+0000","last_fullsized":"2023-10-07T09:47:18.373252+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:57.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.405516+0000","last_change":"2023-10-07T09:47:18.404871+0000","last_active":"2023-10-07T09:47:18.405516+0000","last_peered":"2023-10-07T09:47:18.405516+0000","last_clean":"2023-10-07T09:47:18.405516+0000","last_became_active":"2023-10-07T09:47:18.404621+0000","last_became_peered":"2023-10-07T09:47:18.404621+0000","last_unstale":"2023-10-07T09:47:18.405516+0000","last_undegraded":"2023-10-07T09:47:18.405516+0000","last_fullsized":"2023-10-07T09:47:18.405516+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:29:34.580808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"117'3","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401148+0000","last_change":"2023-10-07T09:47:17.388738+0000","last_active":"2023-10-07T09:47:18.401148+0000","last_peered":"2023-10-07T09:47:18.401148+0000","last_clean":"2023-10-07T09:47:18.401148+0000","last_became_active":"2023-10-07T09:47:17.388426+0000","last_became_peered":"2023-10-07T09:47:17.388426+0000","last_unstale":"2023-10-07T09:47:18.401148+0000","last_undegraded":"2023-10-07T09:47:18.401148+0000","last_fullsized":"2023-10-07T09:47:18.401148+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:54:47.661303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407274+0000","last_change":"2023-10-07T09:47:18.406580+0000","last_active":"2023-10-07T09:47:18.407274+0000","last_peered":"2023-10-07T09:47:18.407274+0000","last_clean":"2023-10-07T09:47:18.407274+0000","last_became_active":"2023-10-07T09:47:18.406299+0000","last_became_peered":"2023-10-07T09:47:18.406299+0000","last_unstale":"2023-10-07T09:47:18.407274+0000","last_undegraded":"2023-10-07T09:47:18.407274+0000","last_fullsized":"2023-10-07T09:47:18.407274+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:09:07.870732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"117'6","reported_seq":16,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.408598+0000","last_change":"2023-10-07T09:47:17.383118+0000","last_active":"2023-10-07T09:47:18.408598+0000","last_peered":"2023-10-07T09:47:18.408598+0000","last_clean":"2023-10-07T09:47:18.408598+0000","last_became_active":"2023-10-07T09:47:17.382831+0000","last_became_peered":"2023-10-07T09:47:17.382831+0000","last_unstale":"2023-10-07T09:47:18.408598+0000","last_undegraded":"2023-10-07T09:47:18.408598+0000","last_fullsized":"2023-10-07T09:47:18.408598+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:31:21.528178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404340+0000","last_change":"2023-10-07T09:47:18.397611+0000","last_active":"2023-10-07T09:47:18.404340+0000","last_peered":"2023-10-07T09:47:18.404340+0000","last_clean":"2023-10-07T09:47:18.404340+0000","last_became_active":"2023-10-07T09:47:18.397284+0000","last_became_peered":"2023-10-07T09:47:18.397284+0000","last_unstale":"2023-10-07T09:47:18.404340+0000","last_undegraded":"2023-10-07T09:47:18.404340+0000","last_fullsized":"2023-10-07T09:47:18.404340+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:59.475818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404703+0000","last_change":"2023-10-07T09:47:17.386081+0000","last_active":"2023-10-07T09:47:18.404703+0000","last_peered":"2023-10-07T09:47:18.404703+0000","last_clean":"2023-10-07T09:47:18.404703+0000","last_became_active":"2023-10-07T09:47:17.385819+0000","last_became_peered":"2023-10-07T09:47:17.385819+0000","last_unstale":"2023-10-07T09:47:18.404703+0000","last_undegraded":"2023-10-07T09:47:18.404703+0000","last_fullsized":"2023-10-07T09:47:18.404703+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:16:40.385342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407957+0000","last_change":"2023-10-07T09:47:18.402616+0000","last_active":"2023-10-07T09:47:18.407957+0000","last_peered":"2023-10-07T09:47:18.407957+0000","last_clean":"2023-10-07T09:47:18.407957+0000","last_became_active":"2023-10-07T09:47:18.402099+0000","last_became_peered":"2023-10-07T09:47:18.402099+0000","last_unstale":"2023-10-07T09:47:18.407957+0000","last_undegraded":"2023-10-07T09:47:18.407957+0000","last_fullsized":"2023-10-07T09:47:18.407957+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:11.227058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.388179+0000","last_change":"2023-10-07T09:47:17.389362+0000","last_active":"2023-10-07T09:47:18.388179+0000","last_peered":"2023-10-07T09:47:18.388179+0000","last_clean":"2023-10-07T09:47:18.388179+0000","last_became_active":"2023-10-07T09:47:17.389052+0000","last_became_peered":"2023-10-07T09:47:17.389052+0000","last_unstale":"2023-10-07T09:47:18.388179+0000","last_undegraded":"2023-10-07T09:47:18.388179+0000","last_fullsized":"2023-10-07T09:47:18.388179+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:02:27.882294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.395445+0000","last_change":"2023-10-07T09:47:18.387451+0000","last_active":"2023-10-07T09:47:18.395445+0000","last_peered":"2023-10-07T09:47:18.395445+0000","last_clean":"2023-10-07T09:47:18.395445+0000","last_became_active":"2023-10-07T09:47:18.387107+0000","last_became_peered":"2023-10-07T09:47:18.387107+0000","last_unstale":"2023-10-07T09:47:18.395445+0000","last_undegraded":"2023-10-07T09:47:18.395445+0000","last_fullsized":"2023-10-07T09:47:18.395445+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:10:09.529294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373639+0000","last_change":"2023-10-07T09:47:17.386573+0000","last_active":"2023-10-07T09:47:18.373639+0000","last_peered":"2023-10-07T09:47:18.373639+0000","last_clean":"2023-10-07T09:47:18.373639+0000","last_became_active":"2023-10-07T09:47:17.386293+0000","last_became_peered":"2023-10-07T09:47:17.386293+0000","last_unstale":"2023-10-07T09:47:18.373639+0000","last_undegraded":"2023-10-07T09:47:18.373639+0000","last_fullsized":"2023-10-07T09:47:18.373639+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:57.125376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398507+0000","last_change":"2023-10-07T09:47:18.394454+0000","last_active":"2023-10-07T09:47:18.398507+0000","last_peered":"2023-10-07T09:47:18.398507+0000","last_clean":"2023-10-07T09:47:18.398507+0000","last_became_active":"2023-10-07T09:47:18.394089+0000","last_became_peered":"2023-10-07T09:47:18.394089+0000","last_unstale":"2023-10-07T09:47:18.398507+0000","last_undegraded":"2023-10-07T09:47:18.398507+0000","last_fullsized":"2023-10-07T09:47:18.398507+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:37:29.053053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"117'8","reported_seq":18,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.406316+0000","last_change":"2023-10-07T09:47:17.387809+0000","last_active":"2023-10-07T09:47:18.406316+0000","last_peered":"2023-10-07T09:47:18.406316+0000","last_clean":"2023-10-07T09:47:18.406316+0000","last_became_active":"2023-10-07T09:47:17.387500+0000","last_became_peered":"2023-10-07T09:47:17.387500+0000","last_unstale":"2023-10-07T09:47:18.406316+0000","last_undegraded":"2023-10-07T09:47:18.406316+0000","last_fullsized":"2023-10-07T09:47:18.406316+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:37:28.045250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401709+0000","last_change":"2023-10-07T09:47:18.396103+0000","last_active":"2023-10-07T09:47:18.401709+0000","last_peered":"2023-10-07T09:47:18.401709+0000","last_clean":"2023-10-07T09:47:18.401709+0000","last_became_active":"2023-10-07T09:47:18.395576+0000","last_became_peered":"2023-10-07T09:47:18.395576+0000","last_unstale":"2023-10-07T09:47:18.401709+0000","last_undegraded":"2023-10-07T09:47:18.401709+0000","last_fullsized":"2023-10-07T09:47:18.401709+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:57:45.265868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398007+0000","last_change":"2023-10-07T09:47:17.381899+0000","last_active":"2023-10-07T09:47:18.398007+0000","last_peered":"2023-10-07T09:47:18.398007+0000","last_clean":"2023-10-07T09:47:18.398007+0000","last_became_active":"2023-10-07T09:47:17.381601+0000","last_became_peered":"2023-10-07T09:47:17.381601+0000","last_unstale":"2023-10-07T09:47:18.398007+0000","last_undegraded":"2023-10-07T09:47:18.398007+0000","last_fullsized":"2023-10-07T09:47:18.398007+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:10.219255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.400933+0000","last_change":"2023-10-07T09:47:18.398350+0000","last_active":"2023-10-07T09:47:18.400933+0000","last_peered":"2023-10-07T09:47:18.400933+0000","last_clean":"2023-10-07T09:47:18.400933+0000","last_became_active":"2023-10-07T09:47:18.398051+0000","last_became_peered":"2023-10-07T09:47:18.398051+0000","last_unstale":"2023-10-07T09:47:18.400933+0000","last_undegraded":"2023-10-07T09:47:18.400933+0000","last_fullsized":"2023-10-07T09:47:18.400933+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:09.638520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403347+0000","last_change":"2023-10-07T09:47:17.383454+0000","last_active":"2023-10-07T09:47:18.403347+0000","last_peered":"2023-10-07T09:47:18.403347+0000","last_clean":"2023-10-07T09:47:18.403347+0000","last_became_active":"2023-10-07T09:47:17.383175+0000","last_became_peered":"2023-10-07T09:47:17.383175+0000","last_unstale":"2023-10-07T09:47:18.403347+0000","last_undegraded":"2023-10-07T09:47:18.403347+0000","last_fullsized":"2023-10-07T09:47:18.403347+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:47:36.685902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.400375+0000","last_change":"2023-10-07T09:47:18.399263+0000","last_active":"2023-10-07T09:47:18.400375+0000","last_peered":"2023-10-07T09:47:18.400375+0000","last_clean":"2023-10-07T09:47:18.400375+0000","last_became_active":"2023-10-07T09:47:18.398778+0000","last_became_peered":"2023-10-07T09:47:18.398778+0000","last_unstale":"2023-10-07T09:47:18.400375+0000","last_undegraded":"2023-10-07T09:47:18.400375+0000","last_fullsized":"2023-10-07T09:47:18.400375+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:28:32.954931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.377994+0000","last_change":"2023-10-07T09:47:17.392377+0000","last_active":"2023-10-07T09:47:18.377994+0000","last_peered":"2023-10-07T09:47:18.377994+0000","last_clean":"2023-10-07T09:47:18.377994+0000","last_became_active":"2023-10-07T09:47:17.392100+0000","last_became_peered":"2023-10-07T09:47:17.392100+0000","last_unstale":"2023-10-07T09:47:18.377994+0000","last_undegraded":"2023-10-07T09:47:18.377994+0000","last_fullsized":"2023-10-07T09:47:18.377994+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:53.163564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403327+0000","last_change":"2023-10-07T09:47:18.400757+0000","last_active":"2023-10-07T09:47:18.403327+0000","last_peered":"2023-10-07T09:47:18.403327+0000","last_clean":"2023-10-07T09:47:18.403327+0000","last_became_active":"2023-10-07T09:47:18.398310+0000","last_became_peered":"2023-10-07T09:47:18.398310+0000","last_unstale":"2023-10-07T09:47:18.403327+0000","last_undegraded":"2023-10-07T09:47:18.403327+0000","last_fullsized":"2023-10-07T09:47:18.403327+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:51.461351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.406640+0000","last_change":"2023-10-07T09:47:17.393547+0000","last_active":"2023-10-07T09:47:18.406640+0000","last_peered":"2023-10-07T09:47:18.406640+0000","last_clean":"2023-10-07T09:47:18.406640+0000","last_became_active":"2023-10-07T09:47:17.393238+0000","last_became_peered":"2023-10-07T09:47:17.393238+0000","last_unstale":"2023-10-07T09:47:18.406640+0000","last_undegraded":"2023-10-07T09:47:18.406640+0000","last_fullsized":"2023-10-07T09:47:18.406640+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:10.219255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401319+0000","last_change":"2023-10-07T09:47:18.399503+0000","last_active":"2023-10-07T09:47:18.401319+0000","last_peered":"2023-10-07T09:47:18.401319+0000","last_clean":"2023-10-07T09:47:18.401319+0000","last_became_active":"2023-10-07T09:47:18.399003+0000","last_became_peered":"2023-10-07T09:47:18.399003+0000","last_unstale":"2023-10-07T09:47:18.401319+0000","last_undegraded":"2023-10-07T09:47:18.401319+0000","last_fullsized":"2023-10-07T09:47:18.401319+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:36:58.155004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373952+0000","last_change":"2023-10-07T09:47:17.390287+0000","last_active":"2023-10-07T09:47:18.373952+0000","last_peered":"2023-10-07T09:47:18.373952+0000","last_clean":"2023-10-07T09:47:18.373952+0000","last_became_active":"2023-10-07T09:47:17.390027+0000","last_became_peered":"2023-10-07T09:47:17.390027+0000","last_unstale":"2023-10-07T09:47:18.373952+0000","last_undegraded":"2023-10-07T09:47:18.373952+0000","last_fullsized":"2023-10-07T09:47:18.373952+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:58.691325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398906+0000","last_change":"2023-10-07T09:47:18.395049+0000","last_active":"2023-10-07T09:47:18.398906+0000","last_peered":"2023-10-07T09:47:18.398906+0000","last_clean":"2023-10-07T09:47:18.398906+0000","last_became_active":"2023-10-07T09:47:18.394555+0000","last_became_peered":"2023-10-07T09:47:18.394555+0000","last_unstale":"2023-10-07T09:47:18.398906+0000","last_undegraded":"2023-10-07T09:47:18.398906+0000","last_fullsized":"2023-10-07T09:47:18.398906+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:36:09.513186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.373222+0000","last_change":"2023-10-07T09:47:17.384019+0000","last_active":"2023-10-07T09:47:18.373222+0000","last_peered":"2023-10-07T09:47:18.373222+0000","last_clean":"2023-10-07T09:47:18.373222+0000","last_became_active":"2023-10-07T09:47:17.383721+0000","last_became_peered":"2023-10-07T09:47:17.383721+0000","last_unstale":"2023-10-07T09:47:18.373222+0000","last_undegraded":"2023-10-07T09:47:18.373222+0000","last_fullsized":"2023-10-07T09:47:18.373222+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:25:25.575632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401737+0000","last_change":"2023-10-07T09:47:18.395734+0000","last_active":"2023-10-07T09:47:18.401737+0000","last_peered":"2023-10-07T09:47:18.401737+0000","last_clean":"2023-10-07T09:47:18.401737+0000","last_became_active":"2023-10-07T09:47:18.395463+0000","last_became_peered":"2023-10-07T09:47:18.395463+0000","last_unstale":"2023-10-07T09:47:18.401737+0000","last_undegraded":"2023-10-07T09:47:18.401737+0000","last_fullsized":"2023-10-07T09:47:18.401737+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:45:17.030160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.379553+0000","last_change":"2023-10-07T09:47:17.389019+0000","last_active":"2023-10-07T09:47:18.379553+0000","last_peered":"2023-10-07T09:47:18.379553+0000","last_clean":"2023-10-07T09:47:18.379553+0000","last_became_active":"2023-10-07T09:47:17.388739+0000","last_became_peered":"2023-10-07T09:47:17.388739+0000","last_unstale":"2023-10-07T09:47:18.379553+0000","last_undegraded":"2023-10-07T09:47:18.379553+0000","last_fullsized":"2023-10-07T09:47:18.379553+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:31:21.528178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404844+0000","last_change":"2023-10-07T09:47:18.391128+0000","last_active":"2023-10-07T09:47:18.404844+0000","last_peered":"2023-10-07T09:47:18.404844+0000","last_clean":"2023-10-07T09:47:18.404844+0000","last_became_active":"2023-10-07T09:47:18.390854+0000","last_became_peered":"2023-10-07T09:47:18.390854+0000","last_unstale":"2023-10-07T09:47:18.404844+0000","last_undegraded":"2023-10-07T09:47:18.404844+0000","last_fullsized":"2023-10-07T09:47:18.404844+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:36.019820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398278+0000","last_change":"2023-10-07T09:47:17.384129+0000","last_active":"2023-10-07T09:47:18.398278+0000","last_peered":"2023-10-07T09:47:18.398278+0000","last_clean":"2023-10-07T09:47:18.398278+0000","last_became_active":"2023-10-07T09:47:17.383736+0000","last_became_peered":"2023-10-07T09:47:17.383736+0000","last_unstale":"2023-10-07T09:47:18.398278+0000","last_undegraded":"2023-10-07T09:47:18.398278+0000","last_fullsized":"2023-10-07T09:47:18.398278+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:54:43.114199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.402156+0000","last_change":"2023-10-07T09:47:18.397623+0000","last_active":"2023-10-07T09:47:18.402156+0000","last_peered":"2023-10-07T09:47:18.402156+0000","last_clean":"2023-10-07T09:47:18.402156+0000","last_became_active":"2023-10-07T09:47:18.397307+0000","last_became_peered":"2023-10-07T09:47:18.397307+0000","last_unstale":"2023-10-07T09:47:18.402156+0000","last_undegraded":"2023-10-07T09:47:18.402156+0000","last_fullsized":"2023-10-07T09:47:18.402156+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:09.638520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.377760+0000","last_change":"2023-10-07T09:47:17.391056+0000","last_active":"2023-10-07T09:47:18.377760+0000","last_peered":"2023-10-07T09:47:18.377760+0000","last_clean":"2023-10-07T09:47:18.377760+0000","last_became_active":"2023-10-07T09:47:17.390741+0000","last_became_peered":"2023-10-07T09:47:17.390741+0000","last_unstale":"2023-10-07T09:47:18.377760+0000","last_undegraded":"2023-10-07T09:47:18.377760+0000","last_fullsized":"2023-10-07T09:47:18.377760+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:31:58.691325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.407928+0000","last_change":"2023-10-07T09:47:18.402491+0000","last_active":"2023-10-07T09:47:18.407928+0000","last_peered":"2023-10-07T09:47:18.407928+0000","last_clean":"2023-10-07T09:47:18.407928+0000","last_became_active":"2023-10-07T09:47:18.402241+0000","last_became_peered":"2023-10-07T09:47:18.402241+0000","last_unstale":"2023-10-07T09:47:18.407928+0000","last_undegraded":"2023-10-07T09:47:18.407928+0000","last_fullsized":"2023-10-07T09:47:18.407928+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:31:22.535981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.378965+0000","last_change":"2023-10-07T09:47:17.393524+0000","last_active":"2023-10-07T09:47:18.378965+0000","last_peered":"2023-10-07T09:47:18.378965+0000","last_clean":"2023-10-07T09:47:18.378965+0000","last_became_active":"2023-10-07T09:47:17.392943+0000","last_became_peered":"2023-10-07T09:47:17.392943+0000","last_unstale":"2023-10-07T09:47:18.378965+0000","last_undegraded":"2023-10-07T09:47:18.378965+0000","last_fullsized":"2023-10-07T09:47:18.378965+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:57.782277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.401619+0000","last_change":"2023-10-07T09:47:18.394083+0000","last_active":"2023-10-07T09:47:18.401619+0000","last_peered":"2023-10-07T09:47:18.401619+0000","last_clean":"2023-10-07T09:47:18.401619+0000","last_became_active":"2023-10-07T09:47:18.393780+0000","last_became_peered":"2023-10-07T09:47:18.393780+0000","last_unstale":"2023-10-07T09:47:18.401619+0000","last_undegraded":"2023-10-07T09:47:18.401619+0000","last_fullsized":"2023-10-07T09:47:18.401619+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:39:47.987946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.380305+0000","last_change":"2023-10-07T09:47:17.387596+0000","last_active":"2023-10-07T09:47:18.380305+0000","last_peered":"2023-10-07T09:47:18.380305+0000","last_clean":"2023-10-07T09:47:18.380305+0000","last_became_active":"2023-10-07T09:47:17.387325+0000","last_became_peered":"2023-10-07T09:47:17.387325+0000","last_unstale":"2023-10-07T09:47:18.380305+0000","last_undegraded":"2023-10-07T09:47:18.380305+0000","last_fullsized":"2023-10-07T09:47:18.380305+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:21:16.370493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403309+0000","last_change":"2023-10-07T09:47:18.400564+0000","last_active":"2023-10-07T09:47:18.403309+0000","last_peered":"2023-10-07T09:47:18.403309+0000","last_clean":"2023-10-07T09:47:18.403309+0000","last_became_active":"2023-10-07T09:47:18.400260+0000","last_became_peered":"2023-10-07T09:47:18.400260+0000","last_unstale":"2023-10-07T09:47:18.403309+0000","last_undegraded":"2023-10-07T09:47:18.403309+0000","last_fullsized":"2023-10-07T09:47:18.403309+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:06.865710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.378666+0000","last_change":"2023-10-07T09:47:17.394513+0000","last_active":"2023-10-07T09:47:18.378666+0000","last_peered":"2023-10-07T09:47:18.378666+0000","last_clean":"2023-10-07T09:47:18.378666+0000","last_became_active":"2023-10-07T09:47:17.394224+0000","last_became_peered":"2023-10-07T09:47:17.394224+0000","last_unstale":"2023-10-07T09:47:18.378666+0000","last_undegraded":"2023-10-07T09:47:18.378666+0000","last_fullsized":"2023-10-07T09:47:18.378666+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:54:43.114199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.403297+0000","last_change":"2023-10-07T09:47:18.397689+0000","last_active":"2023-10-07T09:47:18.403297+0000","last_peered":"2023-10-07T09:47:18.403297+0000","last_clean":"2023-10-07T09:47:18.403297+0000","last_became_active":"2023-10-07T09:47:18.396845+0000","last_became_peered":"2023-10-07T09:47:18.396845+0000","last_unstale":"2023-10-07T09:47:18.403297+0000","last_undegraded":"2023-10-07T09:47:18.403297+0000","last_fullsized":"2023-10-07T09:47:18.403297+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:01:59.475818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.379978+0000","last_change":"2023-10-07T09:47:17.389298+0000","last_active":"2023-10-07T09:47:18.379978+0000","last_peered":"2023-10-07T09:47:18.379978+0000","last_clean":"2023-10-07T09:47:18.379978+0000","last_became_active":"2023-10-07T09:47:17.388979+0000","last_became_peered":"2023-10-07T09:47:17.388979+0000","last_unstale":"2023-10-07T09:47:18.379978+0000","last_undegraded":"2023-10-07T09:47:18.379978+0000","last_fullsized":"2023-10-07T09:47:18.379978+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:21:27.986947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.398888+0000","last_change":"2023-10-07T09:47:18.394879+0000","last_active":"2023-10-07T09:47:18.398888+0000","last_peered":"2023-10-07T09:47:18.398888+0000","last_clean":"2023-10-07T09:47:18.398888+0000","last_became_active":"2023-10-07T09:47:18.394192+0000","last_became_peered":"2023-10-07T09:47:18.394192+0000","last_unstale":"2023-10-07T09:47:18.398888+0000","last_undegraded":"2023-10-07T09:47:18.398888+0000","last_fullsized":"2023-10-07T09:47:18.398888+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:51:21.500686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.372194+0000","last_change":"2023-10-07T09:47:17.380592+0000","last_active":"2023-10-07T09:47:18.372194+0000","last_peered":"2023-10-07T09:47:18.372194+0000","last_clean":"2023-10-07T09:47:18.372194+0000","last_became_active":"2023-10-07T09:47:17.380304+0000","last_became_peered":"2023-10-07T09:47:17.380304+0000","last_unstale":"2023-10-07T09:47:18.372194+0000","last_undegraded":"2023-10-07T09:47:18.372194+0000","last_fullsized":"2023-10-07T09:47:18.372194+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:47:10.219255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.399839+0000","last_change":"2023-10-07T09:47:18.391011+0000","last_active":"2023-10-07T09:47:18.399839+0000","last_peered":"2023-10-07T09:47:18.399839+0000","last_clean":"2023-10-07T09:47:18.399839+0000","last_became_active":"2023-10-07T09:47:18.390628+0000","last_became_peered":"2023-10-07T09:47:18.390628+0000","last_unstale":"2023-10-07T09:47:18.399839+0000","last_undegraded":"2023-10-07T09:47:18.399839+0000","last_fullsized":"2023-10-07T09:47:18.399839+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:52:43.789995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.379674+0000","last_change":"2023-10-07T09:47:17.390637+0000","last_active":"2023-10-07T09:47:18.379674+0000","last_peered":"2023-10-07T09:47:18.379674+0000","last_clean":"2023-10-07T09:47:18.379674+0000","last_became_active":"2023-10-07T09:47:17.390375+0000","last_became_peered":"2023-10-07T09:47:17.390375+0000","last_unstale":"2023-10-07T09:47:18.379674+0000","last_undegraded":"2023-10-07T09:47:18.379674+0000","last_fullsized":"2023-10-07T09:47:18.379674+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:53:12.306910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.399151+0000","last_change":"2023-10-07T09:47:18.398220+0000","last_active":"2023-10-07T09:47:18.399151+0000","last_peered":"2023-10-07T09:47:18.399151+0000","last_clean":"2023-10-07T09:47:18.399151+0000","last_became_active":"2023-10-07T09:47:18.395665+0000","last_became_peered":"2023-10-07T09:47:18.395665+0000","last_unstale":"2023-10-07T09:47:18.399151+0000","last_undegraded":"2023-10-07T09:47:18.399151+0000","last_fullsized":"2023-10-07T09:47:18.399151+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:17.362167+0000","last_clean_scrub_stamp":"2023-10-07T09:47:17.362167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:46:50.493566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.404938+0000","last_change":"2023-10-07T09:47:17.384141+0000","last_active":"2023-10-07T09:47:18.404938+0000","last_peered":"2023-10-07T09:47:18.404938+0000","last_clean":"2023-10-07T09:47:18.404938+0000","last_became_active":"2023-10-07T09:47:17.383875+0000","last_became_peered":"2023-10-07T09:47:17.383875+0000","last_unstale":"2023-10-07T09:47:18.404938+0000","last_undegraded":"2023-10-07T09:47:18.404938+0000","last_fullsized":"2023-10-07T09:47:18.404938+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:47:16.354364+0000","last_clean_scrub_stamp":"2023-10-07T09:47:16.354364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:41:13.214293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"108'227","reported_seq":634,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-07T09:47:18.384116+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:47:18.384116+0000","last_peered":"2023-10-07T09:47:18.384116+0000","last_clean":"2023-10-07T09:47:18.384116+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:47:18.384116+0000","last_undegraded":"2023-10-07T09:47:18.384116+0000","last_fullsized":"2023-10-07T09:47:18.384116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476950,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303388,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068452,"statfs":{"total":96636764160,"available":96326094848,"internally_reserved":0,"allocated":2052096,"data_stored":1288443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.69699999999999995,"15min":0.71899999999999997},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":0.89600000000000002,"5min":0.96599999999999997,"15min":1.026},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69899999999999995,"15min":0.72399999999999998},"min":{"1min":0.51900000000000002,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":0.92500000000000004,"5min":0.94899999999999995,"15min":1.0349999999999999},"last":0.59599999999999997}]},{"osd":1,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71599999999999997,"15min":0.73499999999999999},"min":{"1min":0.46600000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.014},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.74099999999999999,"15min":0.746},"min":{"1min":0.58399999999999996,"5min":0.45900000000000002,"15min":0.45600000000000002},"max":{"1min":0.94499999999999995,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.378}]},{"osd":2,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.72399999999999998,"15min":0.752},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96899999999999997,"5min":1.206,"15min":1.206},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.74199999999999999},"min":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.92600000000000005,"5min":1.038,"15min":1.071},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.58499999999999996,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.01},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.96299999999999997,"5min":1.137,"15min":1.137},"last":0.81399999999999995}]},{"osd":4,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.439,"5min":0.438,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.93999999999999995,"15min":1.002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66600000000000004,"15min":0.64200000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.629}]},{"osd":5,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.36899999999999999},"max":{"1min":0.93700000000000006,"5min":1.27,"15min":1.27},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45000000000000001},"max":{"1min":0.877,"5min":1.0069999999999999,"15min":1.0189999999999999},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.75800000000000001,"15min":0.745},"min":{"1min":0.57299999999999995,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.93400000000000005,"5min":1.077,"15min":1.077},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73499999999999999,"15min":0.72599999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.018,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.58599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509654,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302808,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069032,"statfs":{"total":96636764160,"available":96326688768,"internally_reserved":0,"allocated":1458176,"data_stored":698587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.39300000000000002},"max":{"1min":1.012,"5min":1.274,"15min":1.274},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68300000000000005,"15min":0.69899999999999995},"min":{"1min":0.54200000000000004,"5min":0.40699999999999997,"15min":0.34999999999999998},"max":{"1min":1.012,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.69799999999999995,"15min":0.72199999999999998},"min":{"1min":0.55500000000000005,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":0.90000000000000002,"5min":1.169,"15min":1.169},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.51500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.93899999999999995,"5min":1.7,"15min":1.7},"last":0.57799999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.73599999999999999},"min":{"1min":0.46800000000000003,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":0.98199999999999998,"5min":1.482,"15min":1.482},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.038,"5min":1.643,"15min":1.643},"last":0.67700000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.45200000000000001},"max":{"1min":0.95699999999999996,"5min":1.458,"15min":1.458},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.73999999999999999,"15min":0.75800000000000001},"min":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97699999999999998,"5min":1.006,"15min":1.006},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.61599999999999999,"15min":0.61799999999999999},"min":{"1min":0.40500000000000003,"5min":0.36599999999999999,"15min":0.29899999999999999},"max":{"1min":0.879,"5min":0.92400000000000004,"15min":1.01},"last":0.624},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46000000000000002,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.94399999999999995,"5min":1.018,"15min":1.018},"last":0.76400000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67200000000000004,"15min":0.67300000000000004},"min":{"1min":0.50900000000000001,"5min":0.42299999999999999,"15min":0.41199999999999998},"max":{"1min":0.90800000000000003,"5min":1.036,"15min":1.0509999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.498,"5min":0.439,"15min":0.439},"max":{"1min":0.91700000000000004,"5min":1.0089999999999999,"15min":1.0449999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.69999999999999996,"15min":0.68899999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.39700000000000002},"max":{"1min":0.93200000000000005,"5min":1.099,"15min":1.099},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.57799999999999996,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95999999999999996,"5min":0.995,"15min":1.01},"last":0.78600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542356,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302800,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069040,"statfs":{"total":96636764160,"available":96326696960,"internally_reserved":0,"allocated":1449984,"data_stored":697963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.75600000000000001,"5min":0.78000000000000003,"15min":0.871},"last":0.439},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53000000000000003,"15min":0.55200000000000005},"min":{"1min":0.44900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71099999999999997,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.58799999999999997,"5min":0.73499999999999999,"15min":0.79600000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.39400000000000002,"5min":0.35699999999999998,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":0.78400000000000003,"15min":1.028},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.97599999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.54600000000000004},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69399999999999995,"5min":0.79200000000000004,"15min":0.95099999999999996},"last":0.496}]},{"osd":4,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.754,"15min":0.78300000000000003},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48599999999999999},"max":{"1min":0.90400000000000003,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72599999999999998,"15min":0.76700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97699999999999998,"5min":1.032,"15min":1.196},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.77900000000000003,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0049999999999999,"5min":1.099,"15min":1.2769999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.73899999999999999,"15min":0.78000000000000003},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92700000000000005,"5min":1.0680000000000001,"15min":1.3300000000000001},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.51500000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.131},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.76600000000000001,"15min":0.81899999999999995},"min":{"1min":0.59699999999999998,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":0.997,"5min":1.1040000000000001,"15min":1.165},"last":0.68500000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":1.038,"5min":1.0429999999999999,"15min":1.2450000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.78400000000000003,"15min":0.83899999999999997},"min":{"1min":0.59799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.03,"5min":1.137,"15min":1.385},"last":0.629}]}]},{"osd":0,"up_from":14,"seq":60129542356,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303372,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068468,"statfs":{"total":96636764160,"available":96326111232,"internally_reserved":0,"allocated":2035712,"data_stored":1282221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47299999999999998,"15min":0.48999999999999999},"min":{"1min":0.308,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.248},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.57199999999999995},"min":{"1min":0.42799999999999999,"5min":0.38600000000000001,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.434}]},{"osd":2,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.876,"15min":1.101},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47899999999999998,"15min":0.502},"min":{"1min":0.33800000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68200000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.28599999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51200000000000001,"15min":0.53400000000000003},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.749,"5min":0.83799999999999997,"15min":1.0009999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59099999999999997,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92100000000000004},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.752},"min":{"1min":0.54800000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88100000000000001,"5min":1.119,"15min":1.232},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.63600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.05,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.746,"15min":0.77900000000000003},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97799999999999998,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.76000000000000001,"15min":0.79300000000000004},"min":{"1min":0.60199999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.194},"last":0.63400000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78000000000000003,"15min":0.81100000000000005},"min":{"1min":0.64400000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.98799999999999999,"5min":1.081,"15min":1.129},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.80700000000000005},"min":{"1min":0.61699999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.206,"15min":1.206},"last":0.64700000000000002}]},{"osd":7,"last update":"Sat Oct 7 09:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78600000000000003,"15min":0.82699999999999996},"min":{"1min":0.59099999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.98399999999999999,"5min":1.149,"15min":1.167},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78500000000000003,"15min":0.82399999999999995},"min":{"1min":0.64700000000000002,"5min":0.56200000000000006,"15min":0.503},"max":{"1min":0.995,"5min":1.238,"15min":1.238},"last":0.76300000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542357,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310996,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060844,"statfs":{"total":96636764160,"available":96318304256,"internally_reserved":0,"allocated":1454080,"data_stored":697997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":0.66200000000000003,"5min":0.67800000000000005,"15min":1.258},"last":0.438},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28999999999999998},"max":{"1min":0.79000000000000004,"5min":0.80100000000000005,"15min":1.2290000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.441,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.749,"5min":0.79600000000000004,"15min":1.1950000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":1.3200000000000001},"last":0.68500000000000005}]},{"osd":3,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.495,"15min":0.51800000000000002},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.30599999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":1.5820000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39300000000000002,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":1.052},"last":0.69999999999999996}]},{"osd":4,"last update":"Sat Oct 7 09:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.75600000000000001},"min":{"1min":0.56699999999999995,"5min":0.46800000000000003,"15min":0.35699999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.4630000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.77500000000000002,"15min":0.80000000000000004},"min":{"1min":0.621,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":1.1499999999999999,"5min":1.327,"15min":1.8919999999999999},"last":0.97899999999999998}]},{"osd":5,"last update":"Sat Oct 7 09:46:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.59099999999999997,"5min":0.52000000000000002,"15min":0.48999999999999999},"max":{"1min":1.095,"5min":1.173,"15min":1.98},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.78800000000000003},"min":{"1min":0.54100000000000004,"5min":0.52400000000000002,"15min":0.46100000000000002},"max":{"1min":1.042,"5min":1.208,"15min":1.5169999999999999},"last":1.075}]},{"osd":6,"last update":"Sat Oct 7 09:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78900000000000003,"15min":0.82599999999999996},"min":{"1min":0.59099999999999997,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.0720000000000001,"5min":1.1379999999999999,"15min":1.931},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.63800000000000001,"5min":0.55300000000000005,"15min":0.47299999999999998},"max":{"1min":1.194,"5min":1.2849999999999999,"15min":1.8120000000000001},"last":1.115}]},{"osd":7,"last update":"Sat Oct 7 09:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.81100000000000005,"15min":0.84499999999999997},"min":{"1min":0.66900000000000004,"5min":0.58599999999999997,"15min":0.56999999999999995},"max":{"1min":1.0800000000000001,"5min":1.2450000000000001,"15min":1.8500000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.3839999999999999},"last":1.0089999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542358,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306912,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064928,"statfs":{"total":96636764160,"available":96322486272,"internally_reserved":0,"allocated":1466368,"data_stored":699521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45600000000000002,"15min":0.45900000000000002},"min":{"1min":0.38100000000000001,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":0.76300000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":7.9939999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69499999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":3.246}]},{"osd":1,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.61299999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":5.5359999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.46600000000000003,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":1.024,"15min":1.024},"last":8.0990000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.41799999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71199999999999997,"5min":1.206,"15min":1.206},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55500000000000005,"15min":0.57399999999999995},"min":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":5.8719999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.54400000000000004,"5min":0.48899999999999999,"15min":0.44900000000000001},"max":{"1min":0.98399999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":7.3280000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.63,"5min":0.48399999999999999,"15min":0.48099999999999998},"max":{"1min":0.96599999999999997,"5min":1.1639999999999999,"15min":1.3009999999999999},"last":1.157}]},{"osd":5,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.79100000000000004,"15min":0.79900000000000004},"min":{"1min":0.57899999999999996,"5min":0.53200000000000003,"15min":0.52500000000000002},"max":{"1min":1.1000000000000001,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":3.0419999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.67700000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0529999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":5.8170000000000002}]},{"osd":6,"last update":"Sat Oct 7 09:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.60199999999999998,"5min":0.52400000000000002,"15min":0.505},"max":{"1min":1.0549999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":5.5789999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81299999999999994,"15min":0.82499999999999996},"min":{"1min":0.64800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.032,"5min":1.673,"15min":1.673},"last":7.9459999999999997}]},{"osd":7,"last update":"Sat Oct 7 09:46:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.65300000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0189999999999999,"5min":2.194,"15min":2.194},"last":6.0940000000000003},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.65500000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0700000000000001,"5min":2.403,"15min":2.403},"last":8.0470000000000006}]}]},{"osd":4,"up_from":15,"seq":64424509654,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306916,"kb_used_data":1436,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064924,"statfs":{"total":96636764160,"available":96322482176,"internally_reserved":0,"allocated":1470464,"data_stored":699623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.48599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":1.069},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.68799999999999994,"15min":0.71199999999999997},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.0009999999999999},"last":0.75}]},{"osd":1,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67500000000000004,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":1.054},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.69999999999999996,"15min":0.73199999999999998},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.91700000000000004,"5min":1.478,"15min":1.478},"last":0.80100000000000005}]},{"osd":2,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.73799999999999999,"15min":0.75},"min":{"1min":0.504,"5min":0.504,"15min":0.432},"max":{"1min":0.91600000000000004,"5min":1.518,"15min":1.518},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.54500000000000004,"5min":0.436,"15min":0.40000000000000002},"max":{"1min":0.93300000000000005,"5min":0.93700000000000006,"15min":1.0209999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.68500000000000005,"15min":0.71999999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.873,"5min":0.873,"15min":1.032},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.755},"min":{"1min":0.56499999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92000000000000004,"5min":1.0249999999999999,"15min":1.036},"last":0.81399999999999995}]},{"osd":5,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.427,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":1.639,"15min":1.639},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.42299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.46100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.272,"5min":1.585,"15min":1.585},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:46:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.443},"max":{"1min":0.80900000000000005,"5min":0.96299999999999997,"15min":1.0589999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.752},"min":{"1min":0.59199999999999997,"5min":0.52700000000000002,"15min":0.45200000000000001},"max":{"1min":1.5760000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.59499999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542356,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":1441792,"data_stored":691369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.75,"15min":0.75700000000000001},"min":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.39900000000000002},"max":{"1min":0.999,"5min":1.218,"15min":1.218},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77000000000000002,"15min":0.79800000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.436},"max":{"1min":0.98699999999999999,"5min":1.294,"15min":1.294},"last":0.81699999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70299999999999996,"15min":0.72899999999999998},"min":{"1min":0.52000000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85099999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.746,"15min":0.745},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94499999999999995,"5min":1.194,"15min":1.194},"last":0.78000000000000003}]},{"osd":2,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.74299999999999999},"min":{"1min":0.503,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93500000000000005,"5min":1.054,"15min":1.054},"last":0.73599999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52500000000000002,"15min":0.48899999999999999},"max":{"1min":0.97399999999999998,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72199999999999998,"15min":0.74199999999999999},"min":{"1min":0.56599999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.88900000000000001,"5min":1.036,"15min":1.036},"last":0.69899999999999995}]},{"osd":4,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.40100000000000002,"15min":0.39800000000000002},"max":{"1min":0.83699999999999997,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.626,"15min":0.628},"min":{"1min":0.45700000000000002,"5min":0.41799999999999998,"15min":0.39400000000000002},"max":{"1min":0.86699999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63200000000000001},"min":{"1min":0.44400000000000001,"5min":0.39900000000000002,"15min":0.33900000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89700000000000002,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.86499999999999999}]},{"osd":7,"last update":"Sat Oct 7 09:46:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70799999999999996,"15min":0.69899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"max":{"1min":0.91800000000000004,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.91800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:47:23.271 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:47:23.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:47:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.509+0000 7ff59d9ff700 1 Processor -- start 2023-10-07T09:47:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.509+0000 7ff59d9ff700 1 -- start start 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.510+0000 7ff59d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cb800 0x7ff5980cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.510+0000 7ff59d9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5980d0000 0x7ff5980b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff59d9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5980c62d0 con 0x7ff5980cfbb0 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff59d9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5980d0420 con 0x7ff5980d0000 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff59d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff598159e10 con 0x7ff5980cb800 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5967fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff5980cfbb0 0x7ff5980c6760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39736/0 (socket says 172.21.15.29:39736) 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5977fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5980d0000 0x7ff5980b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5967fc700 1 -- 172.21.15.29:0/3487312047 learned_addr learned my addr 172.21.15.29:0/3487312047 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff596ffd700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cb800 0x7ff5980cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5977fe700 1 -- 172.21.15.29:0/3487312047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cb800 msgr2=0x7ff5980cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5977fe700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cb800 0x7ff5980cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5977fe700 1 -- 172.21.15.29:0/3487312047 >> v1:172.21.15.29:6789/0 conn(0x7ff5980cfbb0 legacy=0x7ff5980c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff5977fe700 1 -- 172.21.15.29:0/3487312047 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5980cc110 con 0x7ff5980d0000 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.511+0000 7ff596ffd700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cb800 0x7ff5980cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff5977fe700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff5980b3340 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff590009e60 tx=0x7ff590009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c59ba791c5e7f8b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff595ffb700 1 -- 172.21.15.29:0/3487312047 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59000e070 con 0x7ff5980d0000 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff595ffb700 1 -- 172.21.15.29:0/3487312047 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59000bcc0 con 0x7ff5980d0000 2023-10-07T09:47:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff595ffb700 1 -- 172.21.15.29:0/3487312047 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590012680 con 0x7ff5980d0000 2023-10-07T09:47:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 -- 172.21.15.29:0/3487312047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 msgr2=0x7ff5980b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff5980b3340 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff590009e60 tx=0x7ff590009780 comp rx=0 tx=0).stop 2023-10-07T09:47:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 -- 172.21.15.29:0/3487312047 shutdown_connections 2023-10-07T09:47:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff5980b3340 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/3487312047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cb800 0x7ff5980cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 -- 172.21.15.29:0/3487312047 >> 172.21.15.29:0/3487312047 conn(0x7ff598063a10 msgr2=0x7ff5980d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.512+0000 7ff59d9ff700 1 -- 172.21.15.29:0/3487312047 shutdown_connections 2023-10-07T09:47:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.513+0000 7ff59d9ff700 1 -- 172.21.15.29:0/3487312047 wait complete. 2023-10-07T09:47:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.514+0000 7ff59d9ff700 1 Processor -- start 2023-10-07T09:47:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.514+0000 7ff59d9ff700 1 -- start start 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff59d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5980cb800 0x7ff5980b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff59d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cfbb0 0x7ff59816d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff59d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff59816dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff59d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980b4c00 con 0x7ff5980d0000 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff59d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980b4ee0 con 0x7ff5980cb800 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff59d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980b51c0 con 0x7ff5980cfbb0 2023-10-07T09:47:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff5977fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff59816dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff5977fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff59816dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59980/0 (socket says 172.21.15.29:59980) 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff5977fe700 1 -- 172.21.15.29:0/1291152843 learned_addr learned my addr 172.21.15.29:0/1291152843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff596ffd700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5980cb800 0x7ff5980b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.515+0000 7ff5967fc700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cfbb0 0x7ff59816d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff5977fe700 1 -- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cfbb0 msgr2=0x7ff59816d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff5977fe700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cfbb0 0x7ff59816d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff5977fe700 1 -- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5980cb800 msgr2=0x7ff5980b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff5977fe700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5980cb800 0x7ff5980b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff5977fe700 1 -- 172.21.15.29:0/1291152843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590009a20 con 0x7ff5980d0000 2023-10-07T09:47:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff5977fe700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff59816dd40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff598088b40 tx=0x7ff590019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59001d630 con 0x7ff5980d0000 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59001d7b0 con 0x7ff5980d0000 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5980b54a0 con 0x7ff5980d0000 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59001db70 con 0x7ff5980d0000 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.516+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5980b3470 con 0x7ff5980d0000 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.517+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5980b59c0 con 0x7ff5980d0000 2023-10-07T09:47:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.518+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff590024070 con 0x7ff5980d0000 2023-10-07T09:47:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.519+0000 7ff57effd700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff584066f30 0x7ff5840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.519+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7ff590087120 con 0x7ff5980d0000 2023-10-07T09:47:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.520+0000 7ff596ffd700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff584066f30 0x7ff5840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.520+0000 7ff596ffd700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff584066f30 0x7ff5840693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff58800aaf0 tx=0x7ff588008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.523+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5980b59c0 con 0x7ff5980d0000 2023-10-07T09:47:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.804+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff5980b59c0 con 0x7ff5980d0000 2023-10-07T09:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.808+0000 7ff57effd700 1 -- 172.21.15.29:0/1291152843 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff5980b59c0 con 0x7ff5980d0000 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff584066f30 msgr2=0x7ff5840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff584066f30 0x7ff5840693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff58800aaf0 tx=0x7ff588008040 comp rx=0 tx=0).stop 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 msgr2=0x7ff59816dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff59816dd40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff598088b40 tx=0x7ff590019000 comp rx=0 tx=0).stop 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 shutdown_connections 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff584066f30 0x7ff5840693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5980d0000 0x7ff59816dd40 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5980cb800 0x7ff5980b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 --2- 172.21.15.29:0/1291152843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5980cfbb0 0x7ff59816d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.810+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 >> 172.21.15.29:0/1291152843 conn(0x7ff598063a10 msgr2=0x7ff598157510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.811+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 shutdown_connections 2023-10-07T09:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:23.811+0000 7ff59d9ff700 1 -- 172.21.15.29:0/1291152843 wait complete. 2023-10-07T09:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:47:23.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:47:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.024+0000 7fba709ff700 1 Processor -- start 2023-10-07T09:47:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.024+0000 7fba709ff700 1 -- start start 2023-10-07T09:47:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.025+0000 7fba709ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba6c15a340 0x7fba6c0afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.025+0000 7fba709ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c14d790 con 0x7fba6c0ad370 2023-10-07T09:47:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.025+0000 7fba709ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0ae5d0 con 0x7fba6c159ec0 2023-10-07T09:47:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.025+0000 7fba709ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c0ad740 con 0x7fba6c15a340 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.025+0000 7fba6a59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fba6c0ad370 0x7fba6c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39744/0 (socket says 172.21.15.29:39744) 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.025+0000 7fba6a59c700 1 -- 172.21.15.29:0/3824315845 learned_addr learned my addr 172.21.15.29:0/3824315845 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6ad9d700 1 --2- 172.21.15.29:0/3824315845 >> v2:172.21.15.29:3300/0 conn(0x7fba6c15a340 0x7fba6c0afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6959a700 1 -- 172.21.15.29:0/3824315845 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243605236 0 0) 0x7fba6c14d790 con 0x7fba6c0ad370 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6959a700 1 -- 172.21.15.29:0/3824315845 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba54003570 con 0x7fba6c0ad370 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6ad9d700 1 -- 172.21.15.29:0/3824315845 >> v1:172.21.15.144:6789/0 conn(0x7fba6c159ec0 legacy=0x7fba6c0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6ad9d700 1 -- 172.21.15.29:0/3824315845 >> v1:172.21.15.29:6789/0 conn(0x7fba6c0ad370 legacy=0x7fba6c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6ad9d700 1 -- 172.21.15.29:0/3824315845 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0ad9c0 con 0x7fba6c15a340 2023-10-07T09:47:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.026+0000 7fba6ad9d700 1 --2- 172.21.15.29:0/3824315845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c0afb10 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fba64009e60 tx=0x7fba64009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640064249b5d4edf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba6959a700 1 -- 172.21.15.29:0/3824315845 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6400bb60 con 0x7fba6c15a340 2023-10-07T09:47:24.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba6959a700 1 -- 172.21.15.29:0/3824315845 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6400bcc0 con 0x7fba6c15a340 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba6959a700 1 -- 172.21.15.29:0/3824315845 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64011670 con 0x7fba6c15a340 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 -- 172.21.15.29:0/3824315845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 msgr2=0x7fba6c0afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 --2- 172.21.15.29:0/3824315845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c0afb10 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fba64009e60 tx=0x7fba64009780 comp rx=0 tx=0).stop 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 -- 172.21.15.29:0/3824315845 shutdown_connections 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 --2- 172.21.15.29:0/3824315845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c0afb10 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 -- 172.21.15.29:0/3824315845 >> 172.21.15.29:0/3824315845 conn(0x7fba6c0680a0 msgr2=0x7fba6c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 -- 172.21.15.29:0/3824315845 shutdown_connections 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.027+0000 7fba709ff700 1 -- 172.21.15.29:0/3824315845 wait complete. 2023-10-07T09:47:24.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.029+0000 7fba709ff700 1 Processor -- start 2023-10-07T09:47:24.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.029+0000 7fba709ff700 1 -- start start 2023-10-07T09:47:24.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba709ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba6c0ad370 0x7fba6c1503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba709ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba6c159ec0 0x7fba6c0cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba709ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c16ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba709ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0cacb0 con 0x7fba6c15a340 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba709ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c14ef50 con 0x7fba6c0ad370 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba709ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c14f230 con 0x7fba6c159ec0 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba6ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c16ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba6ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c16ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59998/0 (socket says 172.21.15.29:59998) 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba6ad9d700 1 -- 172.21.15.29:0/2435104371 learned_addr learned my addr 172.21.15.29:0/2435104371 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba6a59c700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba6c0ad370 0x7fba6c1503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba6ad9d700 1 -- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba6c159ec0 msgr2=0x7fba6c0cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.030+0000 7fba69d9b700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba6c159ec0 0x7fba6c0cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba6ad9d700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba6c159ec0 0x7fba6c0cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba6ad9d700 1 -- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba6c0ad370 msgr2=0x7fba6c1503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba6ad9d700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba6c0ad370 0x7fba6c1503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba6ad9d700 1 -- 172.21.15.29:0/2435104371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64009a20 con 0x7fba6c15a340 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba6a59c700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba6c0ad370 0x7fba6c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba69d9b700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba6c159ec0 0x7fba6c0cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba6ad9d700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c16ce10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fba6c068dd0 tx=0x7fba6401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64019710 con 0x7fba6c15a340 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6401e040 con 0x7fba6c15a340 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c14f510 con 0x7fba6c15a340 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64019ae0 con 0x7fba6c15a340 2023-10-07T09:47:24.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.031+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c1566c0 con 0x7fba6c15a340 2023-10-07T09:47:24.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.032+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c0cae20 con 0x7fba6c15a340 2023-10-07T09:47:24.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.033+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba64003040 con 0x7fba6c15a340 2023-10-07T09:47:24.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.034+0000 7fba5a7fc700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fba4c066f30 0x7fba4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.035+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fba64086480 con 0x7fba6c15a340 2023-10-07T09:47:24.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.035+0000 7fba6a59c700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fba4c066f30 0x7fba4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.036+0000 7fba6a59c700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fba4c066f30 0x7fba4c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba5c007fd0 tx=0x7fba5c009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.039+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba64053570 con 0x7fba6c15a340 2023-10-07T09:47:24.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.286+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba6c04c3c0 con 0x7fba6c15a340 2023-10-07T09:47:24.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.286+0000 7fba5a7fc700 1 -- 172.21.15.29:0/2435104371 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fba64056b90 con 0x7fba6c15a340 2023-10-07T09:47:24.288 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:24.288 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:47:17.881832+0000","modified":"2023-10-07T09:47:21.398903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/581264510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":581264510},{"type":"v1","addr":"172.21.15.29:6835","nonce":581264510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-10-07T09:47:24.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fba4c066f30 msgr2=0x7fba4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fba4c066f30 0x7fba4c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba5c007fd0 tx=0x7fba5c009040 comp rx=0 tx=0).stop 2023-10-07T09:47:24.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 msgr2=0x7fba6c16ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c16ce10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fba6c068dd0 tx=0x7fba6401b000 comp rx=0 tx=0).stop 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 shutdown_connections 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fba4c066f30 0x7fba4c0693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba6c15a340 0x7fba6c16ce10 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba6c0ad370 0x7fba6c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 --2- 172.21.15.29:0/2435104371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba6c159ec0 0x7fba6c0cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 >> 172.21.15.29:0/2435104371 conn(0x7fba6c0680a0 msgr2=0x7fba6c0ac4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 shutdown_connections 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.289+0000 7fba709ff700 1 -- 172.21.15.29:0/2435104371 wait complete. 2023-10-07T09:47:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-07T09:47:24.357 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-07T09:47:24.355+0000 7f4b209f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:47:24.711 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:47:17.881832+0000', 'modified': '2023-10-07T09:47:21.398903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7677}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7677': {'gid': 7677, 'name': 'a', 'rank': 0, 'incarnation': 56, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/581264510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 581264510}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 581264510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:47:24.711 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:47:24.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-07T09:47:24.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.908+0000 7fd9dafff700 1 Processor -- start 2023-10-07T09:47:24.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.908+0000 7fd9dafff700 1 -- start start 2023-10-07T09:47:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9dafff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40ae8b0 0x7fd9d40b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9dafff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40b1ca0 con 0x7fd9d4157290 2023-10-07T09:47:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9dafff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4158ad0 con 0x7fd9d40b1840 2023-10-07T09:47:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9dafff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d415b140 con 0x7fd9d40ae8b0 2023-10-07T09:47:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9d8d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9d4157290 0x7fd9d40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39756/0 (socket says 172.21.15.29:39756) 2023-10-07T09:47:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/3014395064 learned_addr learned my addr 172.21.15.29:0/3014395064 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40ae8b0 0x7fd9d40b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801143925 0 0) 0x7fd9c8001110 con 0x7fd9d4157290 2023-10-07T09:47:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.910+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c4003570 con 0x7fd9d4157290 2023-10-07T09:47:24.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1950215609 0 0) 0x7fd9c4003570 con 0x7fd9d4157290 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40ae8b0 msgr2=0x7fd9d40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40ae8b0 0x7fd9d40b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 >> v1:172.21.15.144:6789/0 conn(0x7fd9d40b1840 legacy=0x7fd9d4157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d415b420 con 0x7fd9d4157290 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fd9c8001600 con 0x7fd9d4157290 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 >> v1:172.21.15.29:6789/0 conn(0x7fd9d4157290 legacy=0x7fd9d40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9c4003a90 0x7fd9c4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9c40043e0 0x7fd9c40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9c400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d415b140 con 0x7fd9c40050b0 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c400bfa0 con 0x7fd9c40043e0 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c400c160 con 0x7fd9c4003a90 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d959c700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9c400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9c4003a90 0x7fd9c4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.911+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9c40043e0 0x7fd9c40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d959c700 1 -- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9c4003a90 msgr2=0x7fd9c4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d959c700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9c4003a90 0x7fd9c4003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d959c700 1 -- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9c40043e0 msgr2=0x7fd9c40048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d959c700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9c40043e0 0x7fd9c40048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d959c700 1 -- 172.21.15.29:0/3014395064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c4003570 con 0x7fd9c40050b0 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9c4003a90 0x7fd9c4003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d959c700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9c400ba60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc001fc0 tx=0x7fd9cc001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d681bb8d647d0bec server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc001950 con 0x7fd9c40050b0 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc00c8b0 con 0x7fd9c40050b0 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.912+0000 7fd9d37fe700 1 -- 172.21.15.29:0/3014395064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc00cc50 con 0x7fd9c40050b0 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9dafff700 1 -- 172.21.15.29:0/3014395064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 msgr2=0x7fd9c400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9dafff700 1 --2- 172.21.15.29:0/3014395064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9c400ba60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc001fc0 tx=0x7fd9cc001640 comp rx=0 tx=0).stop 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9d959c700 1 -- 172.21.15.29:0/3014395064 reap_dead start 2023-10-07T09:47:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9dafff700 1 -- 172.21.15.29:0/3014395064 shutdown_connections 2023-10-07T09:47:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9dafff700 1 -- 172.21.15.29:0/3014395064 >> 172.21.15.29:0/3014395064 conn(0x7fd9d40680a0 msgr2=0x7fd9d40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9dafff700 1 -- 172.21.15.29:0/3014395064 shutdown_connections 2023-10-07T09:47:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.913+0000 7fd9dafff700 1 -- 172.21.15.29:0/3014395064 wait complete. 2023-10-07T09:47:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.914+0000 7fd9dafff700 1 Processor -- start 2023-10-07T09:47:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.915+0000 7fd9dafff700 1 -- start start 2023-10-07T09:47:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.915+0000 7fd9dafff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9d40b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.915+0000 7fd9dafff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40c5380 0x7fd9d40c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9dafff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9d4145b90 0x7fd9d406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9dafff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40c6770 con 0x7fd9c40050b0 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9dafff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40c5b60 con 0x7fd9d4145b90 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9dafff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40c5e40 con 0x7fd9d40c5380 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9d40b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9d40b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60004/0 (socket says 172.21.15.29:60004) 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/622817162 learned_addr learned my addr 172.21.15.29:0/622817162 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d959c700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9d4145b90 0x7fd9d406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40c5380 0x7fd9d40c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40c5380 msgr2=0x7fd9d40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40c5380 0x7fd9d40c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9d4145b90 msgr2=0x7fd9d406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9d4145b90 0x7fd9d406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/622817162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40c6120 con 0x7fd9c40050b0 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d959c700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9d4145b90 0x7fd9d406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.916+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9d40b12f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80073c0 tx=0x7fd9d40cc020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.917+0000 7fd9d0ff9700 1 -- 172.21.15.29:0/622817162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8018030 con 0x7fd9c40050b0 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.917+0000 7fd9d0ff9700 1 -- 172.21.15.29:0/622817162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c800ca60 con 0x7fd9c40050b0 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.917+0000 7fd9d0ff9700 1 -- 172.21.15.29:0/622817162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800cde0 con 0x7fd9c40050b0 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.917+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d40c6400 con 0x7fd9c40050b0 2023-10-07T09:47:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.917+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d40c57f0 con 0x7fd9c40050b0 2023-10-07T09:47:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.919+0000 7fd9d0ff9700 1 -- 172.21.15.29:0/622817162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd9c801b070 con 0x7fd9c40050b0 2023-10-07T09:47:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.920+0000 7fd9d0ff9700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066f30 0x7fd9b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.920+0000 7fd9d0ff9700 1 -- 172.21.15.29:0/622817162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd9c8019070 con 0x7fd9c40050b0 2023-10-07T09:47:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.920+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066f30 0x7fd9b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.921+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066f30 0x7fd9b80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9d41598d0 tx=0x7fd9c0008040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.922+0000 7fd9e9770b80 1 Processor -- start 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.922+0000 7fd9e9770b80 1 -- start start 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e8c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e8d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907d56310 con 0x563907e83a40 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907d93130 con 0x563907e69530 2023-10-07T09:47:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x563907d90af0 con 0x563907e8cbc0 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9d8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9d8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e83e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60010/0 (socket says 172.21.15.29:60010) 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/378044438 learned_addr learned my addr 172.21.15.29:0/378044438 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e8c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.923+0000 7fd9d959c700 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e8d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 msgr2=0x563907e8d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e8d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 msgr2=0x563907e8c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e8c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d8d9b700 1 -- 172.21.15.29:0/378044438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563907c7f0a0 con 0x563907e83a40 2023-10-07T09:47:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e8c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e83e10 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80519a0 tx=0x7fd9c8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab51a8f14f10e595 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9bdffb700 1 -- 172.21.15.29:0/378044438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c80554a0 con 0x563907e83a40 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9bdffb700 1 -- 172.21.15.29:0/378044438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c80586f0 con 0x563907e83a40 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.924+0000 7fd9bdffb700 1 -- 172.21.15.29:0/378044438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c805c440 con 0x563907e83a40 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 -- 172.21.15.29:0/378044438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 msgr2=0x563907e83e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e83e10 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80519a0 tx=0x7fd9c8009040 comp rx=0 tx=0).stop 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 -- 172.21.15.29:0/378044438 shutdown_connections 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e8d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e83e10 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/378044438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e8c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 -- 172.21.15.29:0/378044438 >> 172.21.15.29:0/378044438 conn(0x563907e86e70 msgr2=0x563907e87240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 -- 172.21.15.29:0/378044438 shutdown_connections 2023-10-07T09:47:24.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.925+0000 7fd9e9770b80 1 -- 172.21.15.29:0/378044438 wait complete. 2023-10-07T09:47:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.926+0000 7fd9e9770b80 1 -- start start 2023-10-07T09:47:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.926+0000 7fd9e9770b80 1 Processor -- start 2023-10-07T09:47:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e63dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907d8c8e0 con 0x563907e83a40 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907d38980 con 0x563907e69530 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x563907d38620 con 0x563907e8cbc0 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9e9770b80 10 client.0 _ll_register_callbacks cb 0x563907b9ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e63dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e63dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60018/0 (socket says 172.21.15.29:60018) 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.927+0000 7fd9d3fff700 1 -- 172.21.15.29:0/4011112093 learned_addr learned my addr 172.21.15.29:0/4011112093 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d959c700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d3fff700 1 -- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 msgr2=0x563907e674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e674f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d3fff700 1 -- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 msgr2=0x563907e653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d3fff700 1 -- 172.21.15.29:0/4011112093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d40c57f0 con 0x563907e83a40 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d959c700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e63dc0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0016b30 tx=0x7fd9c001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3116435094e25c6b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9b2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/4011112093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0015a60 con 0x563907e83a40 2023-10-07T09:47:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/4011112093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd9c0015bc0 con 0x563907e83a40 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.928+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563907e31560 con 0x563907e83a40 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/4011112093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9c0024430 con 0x563907e83a40 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 10 client.7827 fetch_fsmap learned FSMap version 60 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x563907d5b3c0 con 0x563907e83a40 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/4011112093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563907e31560 con 0x563907e83a40 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/4011112093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 60) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fd9c002a020 con 0x563907e83a40 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 10 client.7827 fetch_fsmap finished waiting for FSMap version 60 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 10 client.7827 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 20 client.7827 populate_metadata read hostname 'smithi029' 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e6bd40 0x563907e6c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 4 client.7827 mds_command: new command op to 7677 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 --> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563907e0e0e0 con 0x563907e6bd40 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.929+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e6bd40 0x563907e6c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.930+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e6bd40 0x563907e6c110 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe52e875f0ef6b1e server_cookie=7f241fc43cb50a5 in_seq=0 out_seq=0 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.930+0000 7fd9b2ffd700 10 client.7827 ms_handle_connect on v2:172.21.15.29:6833/581264510 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.930+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/4011112093 <== mds.0 v2:172.21.15.29:6833/581264510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x563907e0e0e0 con 0x563907e6bd40 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.930+0000 7fd9b2ffd700 10 client.7827 handle_command_reply: tid=0 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 client.7827 shutdown 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 msgr2=0x563907e63dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e63dc0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0016b30 tx=0x7fd9c001b000 comp rx=0 tx=0).stop 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 shutdown_connections 2023-10-07T09:47:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e6bd40 0x563907e6c110 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/4011112093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e63dc0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9b2ffd700 0 client.7827 ms_handle_reset on v2:172.21.15.29:6833/581264510 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 >> 172.21.15.29:0/4011112093 conn(0x563907e86e70 msgr2=0x563907e39250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 shutdown_connections 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 1 -- 172.21.15.29:0/4011112093 wait complete. 2023-10-07T09:47:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.931+0000 7fd9e9770b80 20 client.7827 trim_cache size 0 max 16384 2023-10-07T09:47:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.949+0000 7fd9e9770b80 1 Processor -- start 2023-10-07T09:47:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.949+0000 7fd9e9770b80 1 -- start start 2023-10-07T09:47:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e60d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e38850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907c241e0 con 0x563907e83a40 2023-10-07T09:47:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907e3d860 con 0x563907e69530 2023-10-07T09:47:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x563907e35300 con 0x563907e8cbc0 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e60d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e60d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60024/0 (socket says 172.21.15.29:60024) 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 -- 172.21.15.29:0/2166345698 learned_addr learned my addr 172.21.15.29:0/2166345698 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d959c700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e38850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 -- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 msgr2=0x563907e38850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e38850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 -- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 msgr2=0x563907e60810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e60810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d3fff700 1 -- 172.21.15.29:0/2166345698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563907bf31d0 con 0x563907e83a40 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e60810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.950+0000 7fd9d959c700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e38850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.951+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e60d50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0008010 tx=0x7fd9c0026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36aef33fd50ccbe4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.951+0000 7fd9be7fc700 1 -- 172.21.15.29:0/2166345698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0024430 con 0x563907e83a40 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.951+0000 7fd9be7fc700 1 -- 172.21.15.29:0/2166345698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c0024590 con 0x563907e83a40 2023-10-07T09:47:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.951+0000 7fd9be7fc700 1 -- 172.21.15.29:0/2166345698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0005a00 con 0x563907e83a40 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.951+0000 7fd9e9770b80 1 -- 172.21.15.29:0/2166345698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 msgr2=0x563907e60d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.951+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e60d50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0008010 tx=0x7fd9c0026000 comp rx=0 tx=0).stop 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 -- 172.21.15.29:0/2166345698 shutdown_connections 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e38850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e60810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/2166345698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907e60d50 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 -- 172.21.15.29:0/2166345698 >> 172.21.15.29:0/2166345698 conn(0x563907e86e70 msgr2=0x563907e843b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 -- 172.21.15.29:0/2166345698 shutdown_connections 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 -- 172.21.15.29:0/2166345698 wait complete. 2023-10-07T09:47:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.952+0000 7fd9e9770b80 1 -- start start 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.953+0000 7fd9e9770b80 1 Processor -- start 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.953+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e62060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.953+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907ee4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.953+0000 7fd9e9770b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907e63f80 con 0x563907e83a40 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563907ee5500 con 0x563907e69530 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9e9770b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x563907ee8a10 con 0x563907e8cbc0 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9e9770b80 10 client.0 _ll_register_callbacks cb 0x563907b9ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907ee4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907ee4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60032/0 (socket says 172.21.15.29:60032) 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 -- 172.21.15.29:0/3413387072 learned_addr learned my addr 172.21.15.29:0/3413387072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:24.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d959c700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e62060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 -- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 msgr2=0x563907e32ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e32ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 -- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 msgr2=0x563907e62060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e62060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 -- 172.21.15.29:0/3413387072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563907d5b3c0 con 0x563907e83a40 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9d3fff700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907ee4fc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0015010 tx=0x7fd9c0039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5297a8b17b6193 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.954+0000 7fd9b2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/3413387072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c004a070 con 0x563907e83a40 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9d40af1e0 con 0x563907e83a40 2023-10-07T09:47:24.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/3413387072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd9c00248c0 con 0x563907e83a40 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/3413387072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9c0024c90 con 0x563907e83a40 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 10 client.7833 fetch_fsmap learned FSMap version 60 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x563907bccd60 con 0x563907e83a40 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/3413387072 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d40af1e0 con 0x563907e83a40 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/3413387072 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 60) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fd9c004d020 con 0x563907e83a40 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 10 client.7833 fetch_fsmap finished waiting for FSMap version 60 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 10 client.7833 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 20 client.7833 populate_metadata read hostname 'smithi029' 2023-10-07T09:47:24.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e330e0 0x563907e38850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 4 client.7833 mds_command: new command op to 7677 tid=0[{"prefix": "status"}] 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.955+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 --> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fd9c80586f0 con 0x563907e330e0 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e330e0 0x563907e38850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9d8d9b700 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e330e0 0x563907e38850 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88edab003336c5ff server_cookie=62302d980943a2e6 in_seq=0 out_seq=0 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9b2ffd700 10 client.7833 ms_handle_connect on v2:172.21.15.29:6833/581264510 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/3413387072 <== mds.0 v2:172.21.15.29:6833/581264510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7fd9c80586f0 con 0x563907e330e0 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9b2ffd700 10 client.7833 handle_command_reply: tid=0 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9e9770b80 1 client.7833 shutdown 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 msgr2=0x563907ee4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.956+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907ee4fc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0015010 tx=0x7fd9c0039000 comp rx=0 tx=0).stop 2023-10-07T09:47:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 shutdown_connections 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:6833/581264510,v1:172.21.15.29:6835/581264510] conn(0x563907e330e0 0x563907e38850 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x563907e8cbc0 0x563907e32ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563907e69530 0x563907e62060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9b2ffd700 0 client.7833 ms_handle_reset on v2:172.21.15.29:6833/581264510 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 --2- 172.21.15.29:0/3413387072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563907e83a40 0x563907ee4fc0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 >> 172.21.15.29:0/3413387072 conn(0x563907e86e70 msgr2=0x563907e674f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 shutdown_connections 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 1 -- 172.21.15.29:0/3413387072 wait complete. 2023-10-07T09:47:24.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9e9770b80 20 client.7833 trim_cache size 0 max 16384 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7677, 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5879565500000004, 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 60, 2023-10-07T09:47:24.966 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 118, 2023-10-07T09:47:24.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 118, 2023-10-07T09:47:24.967 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.251991911999999 2023-10-07T09:47:24.967 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:47:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066f30 msgr2=0x7fd9b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9dafff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066f30 0x7fd9b80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9d41598d0 tx=0x7fd9c0008040 comp rx=0 tx=0).stop 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 msgr2=0x7fd9d40b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.957+0000 7fd9dafff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9d40b12f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80073c0 tx=0x7fd9d40cc020 comp rx=0 tx=0).stop 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 shutdown_connections 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066f30 0x7fd9b80693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9d4145b90 0x7fd9d406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9c40050b0 0x7fd9d40b12f0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 --2- 172.21.15.29:0/622817162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9d40c5380 0x7fd9d40c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 >> 172.21.15.29:0/622817162 conn(0x7fd9d40680a0 msgr2=0x7fd9d40cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 shutdown_connections 2023-10-07T09:47:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:24.958+0000 7fd9dafff700 1 -- 172.21.15.29:0/622817162 wait complete. 2023-10-07T09:47:24.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:47:25.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.172+0000 7f05ad3ff700 1 Processor -- start 2023-10-07T09:47:25.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.172+0000 7f05ad3ff700 1 -- start start 2023-10-07T09:47:25.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05ad3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05a8159c00 0x7f05a80ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:25.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05ad3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a815af00 con 0x7f05a80ad220 2023-10-07T09:47:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05ad3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a815a250 con 0x7f05a8068620 2023-10-07T09:47:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05ad3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a815a050 con 0x7f05a8159c00 2023-10-07T09:47:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05a6ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f05a80ad220 0x7f05a80ac490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39770/0 (socket says 172.21.15.29:39770) 2023-10-07T09:47:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05a6ffd700 1 -- 172.21.15.29:0/3648902567 learned_addr learned my addr 172.21.15.29:0/3648902567 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.173+0000 7f05a67fc700 1 --2- 172.21.15.29:0/3648902567 >> v2:172.21.15.29:3300/0 conn(0x7f05a8159c00 0x7f05a80ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a5ffb700 1 -- 172.21.15.29:0/3648902567 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504485887 0 0) 0x7f05a815af00 con 0x7f05a80ad220 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a5ffb700 1 -- 172.21.15.29:0/3648902567 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0590003570 con 0x7f05a80ad220 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a67fc700 1 -- 172.21.15.29:0/3648902567 >> v1:172.21.15.144:6789/0 conn(0x7f05a8068620 legacy=0x7f05a8068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a67fc700 1 -- 172.21.15.29:0/3648902567 >> v1:172.21.15.29:6789/0 conn(0x7f05a80ad220 legacy=0x7f05a80ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a67fc700 1 -- 172.21.15.29:0/3648902567 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a815a530 con 0x7f05a8159c00 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a67fc700 1 --2- 172.21.15.29:0/3648902567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a80ad5f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f059c0098e0 tx=0x7f059c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd963b794ddfc0fe server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a5ffb700 1 -- 172.21.15.29:0/3648902567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00b640 con 0x7f05a8159c00 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.174+0000 7f05a5ffb700 1 -- 172.21.15.29:0/3648902567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c00b7a0 con 0x7f05a8159c00 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05a5ffb700 1 -- 172.21.15.29:0/3648902567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00bb40 con 0x7f05a8159c00 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 -- 172.21.15.29:0/3648902567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 msgr2=0x7f05a80ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/3648902567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a80ad5f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f059c0098e0 tx=0x7f059c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 -- 172.21.15.29:0/3648902567 shutdown_connections 2023-10-07T09:47:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/3648902567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a80ad5f0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 -- 172.21.15.29:0/3648902567 >> 172.21.15.29:0/3648902567 conn(0x7f05a806dd00 msgr2=0x7f05a806e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:25.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 -- 172.21.15.29:0/3648902567 shutdown_connections 2023-10-07T09:47:25.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.175+0000 7f05ad3ff700 1 -- 172.21.15.29:0/3648902567 wait complete. 2023-10-07T09:47:25.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.177+0000 7f05ad3ff700 1 Processor -- start 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.178+0000 7f05ad3ff700 1 -- start start 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05ad3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05a8068620 0x7f05a815e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05ad3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05a80ad220 0x7f05a81515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05ad3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a81422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05ad3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8151b20 con 0x7f05a8159c00 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05ad3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a815e850 con 0x7f05a80ad220 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05ad3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a815eb30 con 0x7f05a8068620 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a81422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a6ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05a8068620 0x7f05a815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a81422b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60050/0 (socket says 172.21.15.29:60050) 2023-10-07T09:47:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a77fe700 1 -- 172.21.15.29:0/612265149 learned_addr learned my addr 172.21.15.29:0/612265149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a67fc700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05a80ad220 0x7f05a81515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a77fe700 1 -- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05a8068620 msgr2=0x7f05a815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a77fe700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05a8068620 0x7f05a815e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.179+0000 7f05a77fe700 1 -- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05a80ad220 msgr2=0x7f05a81515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f05a77fe700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05a80ad220 0x7f05a81515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f05a77fe700 1 -- 172.21.15.29:0/612265149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c009420 con 0x7f05a8159c00 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f05a77fe700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a81422b0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c840 tx=0x7f05a0001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0012630 con 0x7f05a8159c00 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a00127b0 con 0x7f05a8159c00 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a815ee10 con 0x7f05a8159c00 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0012b70 con 0x7f05a8159c00 2023-10-07T09:47:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.180+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80c9eb0 con 0x7f05a8159c00 2023-10-07T09:47:25.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.181+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a804c3c0 con 0x7f05a8159c00 2023-10-07T09:47:25.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.182+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f05a001a3a0 con 0x7f05a8159c00 2023-10-07T09:47:25.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.183+0000 7f058effd700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0594066f30 0x7f05940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:25.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.183+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f05a0084830 con 0x7f05a8159c00 2023-10-07T09:47:25.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.183+0000 7f05a6ffd700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0594066f30 0x7f05940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:25.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.184+0000 7f05a6ffd700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0594066f30 0x7f05940693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0598007fd0 tx=0x7f0598009040 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:25.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.187+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05a0051880 con 0x7f05a8159c00 2023-10-07T09:47:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.439+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f05a806ec80 con 0x7f05a8159c00 2023-10-07T09:47:25.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.440+0000 7f058effd700 1 -- 172.21.15.29:0/612265149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v60) v1 ==== 53+0+73 (secure 0 0 0) 0x7f05a0054ea0 con 0x7f05a8159c00 2023-10-07T09:47:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:47:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0594066f30 msgr2=0x7f05940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0594066f30 0x7f05940693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0598007fd0 tx=0x7f0598009040 comp rx=0 tx=0).stop 2023-10-07T09:47:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 msgr2=0x7f05a81422b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a81422b0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c840 tx=0x7f05a0001670 comp rx=0 tx=0).stop 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 shutdown_connections 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0594066f30 0x7f05940693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a8159c00 0x7f05a81422b0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05a8068620 0x7f05a815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 --2- 172.21.15.29:0/612265149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05a80ad220 0x7f05a81515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 >> 172.21.15.29:0/612265149 conn(0x7f05a806dd00 msgr2=0x7f05a80aacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 shutdown_connections 2023-10-07T09:47:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:25.442+0000 7f05ad3ff700 1 -- 172.21.15.29:0/612265149 wait complete. 2023-10-07T09:47:25.460 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:47:25.461 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:47:25.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:25.461 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:47:25.509 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:47:25.510 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41183sec preferred_lft 41183sec 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:47:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:47:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:47:25.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:25.513 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:47:25.618 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:47:25.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:25.621 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:47:25.670 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:47:25.670 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:47:25.670 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:47:25.670 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:47:25.670 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:47:25.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:25.671 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:47:25.672 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:47:25.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:47:25.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:47:25.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:47:25.672 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:47:25.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:25.845 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:47:25.892 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:47:25.892 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:47:25.893 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:47:25.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:25.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:47:25.965 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:47:25.965 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:47:26.013 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:47:26.015 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:47:26.015 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:47:26.106 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:47:26.107 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:47:26.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:47:26.252 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:47:26.252 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:47:26.253 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:47:26.267 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:47:26.267 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:47:26.267 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:47:26.267 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:47:26.267 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:47:26.291 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:47:26.291 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:47:26.291 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:47:26.291 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:47:26.291 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:47:26.291 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:47:26.292 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:47:26.292 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:47:26.295 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:47:26.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:47:26.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:47:26.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:47:26.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.566+0000 7fc5bebff700 1 Processor -- start 2023-10-07T09:47:26.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.566+0000 7fc5bebff700 1 -- start start 2023-10-07T09:47:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.567+0000 7fc5bebff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc5b8159750 0x7fc5b80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.567+0000 7fc5bebff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cbf20 con 0x7fc5b8159b20 2023-10-07T09:47:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.567+0000 7fc5bebff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cc200 con 0x7fc5b80ce8a0 2023-10-07T09:47:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.567+0000 7fc5bebff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b80cc4e0 con 0x7fc5b8159750 2023-10-07T09:47:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5b7fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5b8159b20 0x7fc5b80ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39776/0 (socket says 172.21.15.29:39776) 2023-10-07T09:47:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5bc99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc5b8159750 0x7fc5b80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5b7fff700 1 -- 172.21.15.29:0/447107445 learned_addr learned my addr 172.21.15.29:0/447107445 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:26.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5bc99b700 1 -- 172.21.15.29:0/447107445 >> v1:172.21.15.144:6789/0 conn(0x7fc5b80ce8a0 legacy=0x7fc5b80c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5b77fe700 1 -- 172.21.15.29:0/447107445 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250017440 0 0) 0x7fc5b80cbf20 con 0x7fc5b8159b20 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5bc99b700 1 -- 172.21.15.29:0/447107445 >> v1:172.21.15.29:6789/0 conn(0x7fc5b8159b20 legacy=0x7fc5b80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5bc99b700 1 -- 172.21.15.29:0/447107445 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80cc7c0 con 0x7fc5b8159750 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.568+0000 7fc5bc99b700 1 --2- 172.21.15.29:0/447107445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159750 0x7fc5b80d10a0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0098d0 tx=0x7fc5ac009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28a4083fafd8a32 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5b77fe700 1 -- 172.21.15.29:0/447107445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac009e70 con 0x7fc5b8159750 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5b77fe700 1 -- 172.21.15.29:0/447107445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac005c10 con 0x7fc5b8159750 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5b77fe700 1 -- 172.21.15.29:0/447107445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac00b650 con 0x7fc5b8159750 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 -- 172.21.15.29:0/447107445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159750 msgr2=0x7fc5b80d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 --2- 172.21.15.29:0/447107445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159750 0x7fc5b80d10a0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0098d0 tx=0x7fc5ac009120 comp rx=0 tx=0).stop 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 -- 172.21.15.29:0/447107445 shutdown_connections 2023-10-07T09:47:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 --2- 172.21.15.29:0/447107445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159750 0x7fc5b80d10a0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 -- 172.21.15.29:0/447107445 >> 172.21.15.29:0/447107445 conn(0x7fc5b8063a10 msgr2=0x7fc5b80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:26.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 -- 172.21.15.29:0/447107445 shutdown_connections 2023-10-07T09:47:26.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.569+0000 7fc5bebff700 1 -- 172.21.15.29:0/447107445 wait complete. 2023-10-07T09:47:26.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.571+0000 7fc5bebff700 1 Processor -- start 2023-10-07T09:47:26.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.571+0000 7fc5bebff700 1 -- start start 2023-10-07T09:47:26.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bebff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b80ce8a0 0x7fc5b814b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:26.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bebff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b8159750 0x7fc5b8153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bebff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 0x7fc5b8153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bebff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c7e70 con 0x7fc5b8159b20 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bebff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c8150 con 0x7fc5b8159750 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bebff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c8430 con 0x7fc5b80ce8a0 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bd19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 0x7fc5b8153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bd19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 0x7fc5b8153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60064/0 (socket says 172.21.15.29:60064) 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bd19c700 1 -- 172.21.15.29:0/2651230059 learned_addr learned my addr 172.21.15.29:0/2651230059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5b7fff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b8159750 0x7fc5b8153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.572+0000 7fc5bc99b700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b80ce8a0 0x7fc5b814b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bd19c700 1 -- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b80ce8a0 msgr2=0x7fc5b814b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bd19c700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b80ce8a0 0x7fc5b814b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bd19c700 1 -- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b8159750 msgr2=0x7fc5b8153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bd19c700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b8159750 0x7fc5b8153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bd19c700 1 -- 172.21.15.29:0/2651230059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac0093c0 con 0x7fc5b8159b20 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bc99b700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b80ce8a0 0x7fc5b814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bd19c700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 0x7fc5b8153a80 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c910 tx=0x7fc5b0002720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0006ba0 con 0x7fc5b8159b20 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b0006d00 con 0x7fc5b8159b20 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0012880 con 0x7fc5b8159b20 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80c8710 con 0x7fc5b8159b20 2023-10-07T09:47:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.573+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b80ca4c0 con 0x7fc5b8159b20 2023-10-07T09:47:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.574+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b804c3c0 con 0x7fc5b8159b20 2023-10-07T09:47:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.575+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc5b0012cb0 con 0x7fc5b8159b20 2023-10-07T09:47:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.576+0000 7fc5b4ff9700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc59c066f30 0x7fc59c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.576+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fc5b0084780 con 0x7fc5b8159b20 2023-10-07T09:47:26.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.577+0000 7fc5bc99b700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc59c066f30 0x7fc59c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:26.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.577+0000 7fc5bc99b700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc59c066f30 0x7fc59c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac003080 tx=0x7fc5ac015000 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.580+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5b00517d0 con 0x7fc5b8159b20 2023-10-07T09:47:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.811+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5b8064040 con 0x7fc59c066f30 2023-10-07T09:47:26.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.814+0000 7fc5b4ff9700 1 -- 172.21.15.29:0/2651230059 <== mgr.7162 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc5b8064040 con 0x7fc59c066f30 2023-10-07T09:47:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:47:26.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:47:26.815 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:47:26.815 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:47:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc59c066f30 msgr2=0x7fc59c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc59c066f30 0x7fc59c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac003080 tx=0x7fc5ac015000 comp rx=0 tx=0).stop 2023-10-07T09:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 msgr2=0x7fc5b8153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 0x7fc5b8153a80 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c910 tx=0x7fc5b0002720 comp rx=0 tx=0).stop 2023-10-07T09:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 shutdown_connections 2023-10-07T09:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc59c066f30 0x7fc59c0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5b8159b20 0x7fc5b8153a80 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b80ce8a0 0x7fc5b814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 --2- 172.21.15.29:0/2651230059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b8159750 0x7fc5b8153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 >> 172.21.15.29:0/2651230059 conn(0x7fc5b8063a10 msgr2=0x7fc5b8155050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 shutdown_connections 2023-10-07T09:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:26.816+0000 7fc5bebff700 1 -- 172.21.15.29:0/2651230059 wait complete. 2023-10-07T09:47:26.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:47:27.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.033+0000 7f9cb33ff700 1 Processor -- start 2023-10-07T09:47:27.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.033+0000 7f9cb33ff700 1 -- start start 2023-10-07T09:47:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.034+0000 7f9cb33ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9cac0ce390 0x7f9cac0ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.034+0000 7f9cb33ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac1578a0 con 0x7f9cac0b00b0 2023-10-07T09:47:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.034+0000 7f9cb33ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac0cc830 con 0x7f9cac0b3e60 2023-10-07T09:47:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb33ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac157ce0 con 0x7f9cac0ce390 2023-10-07T09:47:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb199c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9cac0b00b0 0x7f9cac0cccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39786/0 (socket says 172.21.15.29:39786) 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb199c700 1 -- 172.21.15.29:0/2359229573 learned_addr learned my addr 172.21.15.29:0/2359229573 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb099a700 1 --2- 172.21.15.29:0/2359229573 >> v2:172.21.15.144:3300/0 conn(0x7f9cac0ce390 0x7f9cac0ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9ca3fff700 1 -- 172.21.15.29:0/2359229573 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280721036 0 0) 0x7f9cac1578a0 con 0x7f9cac0b00b0 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9ca3fff700 1 -- 172.21.15.29:0/2359229573 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c98003570 con 0x7f9cac0b00b0 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb099a700 1 -- 172.21.15.29:0/2359229573 >> v1:172.21.15.144:6789/0 conn(0x7f9cac0b3e60 legacy=0x7f9cac158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb099a700 1 -- 172.21.15.29:0/2359229573 >> v1:172.21.15.29:6789/0 conn(0x7f9cac0b00b0 legacy=0x7f9cac0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.035+0000 7f9cb099a700 1 -- 172.21.15.29:0/2359229573 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac0cee50 con 0x7f9cac0ce390 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.036+0000 7f9cb099a700 1 --2- 172.21.15.29:0/2359229573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0ce390 0x7f9cac0ce830 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098e0 tx=0x7f9c9c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=346d2f6406d097fb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.036+0000 7f9ca3fff700 1 -- 172.21.15.29:0/2359229573 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c00b640 con 0x7f9cac0ce390 2023-10-07T09:47:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.036+0000 7f9ca3fff700 1 -- 172.21.15.29:0/2359229573 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c00b7a0 con 0x7f9cac0ce390 2023-10-07T09:47:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.036+0000 7f9ca3fff700 1 -- 172.21.15.29:0/2359229573 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c00bb40 con 0x7f9cac0ce390 2023-10-07T09:47:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.036+0000 7f9cb33ff700 1 -- 172.21.15.29:0/2359229573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0ce390 msgr2=0x7f9cac0ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.036+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/2359229573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0ce390 0x7f9cac0ce830 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098e0 tx=0x7f9c9c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.037+0000 7f9cb33ff700 1 -- 172.21.15.29:0/2359229573 shutdown_connections 2023-10-07T09:47:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.037+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/2359229573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0ce390 0x7f9cac0ce830 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.037+0000 7f9cb33ff700 1 -- 172.21.15.29:0/2359229573 >> 172.21.15.29:0/2359229573 conn(0x7f9cac067610 msgr2=0x7f9cac067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.037+0000 7f9cb33ff700 1 -- 172.21.15.29:0/2359229573 shutdown_connections 2023-10-07T09:47:27.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.037+0000 7f9cb33ff700 1 -- 172.21.15.29:0/2359229573 wait complete. 2023-10-07T09:47:27.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.039+0000 7f9cb33ff700 1 Processor -- start 2023-10-07T09:47:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.039+0000 7f9cb33ff700 1 -- start start 2023-10-07T09:47:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.040+0000 7f9cb33ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0b00b0 0x7f9cac150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.040+0000 7f9cb33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0b3e60 0x7f9cac1561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.040+0000 7f9cb33ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 0x7f9cac156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.040+0000 7f9cb33ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac151c10 con 0x7f9cac0ce390 2023-10-07T09:47:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.040+0000 7f9cb33ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac151ed0 con 0x7f9cac0b00b0 2023-10-07T09:47:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.040+0000 7f9cb33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac1521b0 con 0x7f9cac0b3e60 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 0x7f9cac156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb099a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0b3e60 0x7f9cac1561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb119b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0b00b0 0x7f9cac150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 0x7f9cac156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60068/0 (socket says 172.21.15.29:60068) 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 -- 172.21.15.29:0/273814952 learned_addr learned my addr 172.21.15.29:0/273814952 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 -- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0b3e60 msgr2=0x7f9cac1561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0b3e60 0x7f9cac1561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 -- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0b00b0 msgr2=0x7f9cac150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0b00b0 0x7f9cac150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 -- 172.21.15.29:0/273814952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c009420 con 0x7f9cac0ce390 2023-10-07T09:47:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.041+0000 7f9cb199c700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 0x7f9cac156730 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4001460 tx=0x7f9ca4001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.042+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca400a890 con 0x7f9cac0ce390 2023-10-07T09:47:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.042+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca400a9f0 con 0x7f9cac0ce390 2023-10-07T09:47:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.042+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca400ad70 con 0x7f9cac0ce390 2023-10-07T09:47:27.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.042+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac152490 con 0x7f9cac0ce390 2023-10-07T09:47:27.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.042+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac150590 con 0x7f9cac0ce390 2023-10-07T09:47:27.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.043+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac1674d0 con 0x7f9cac0ce390 2023-10-07T09:47:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.044+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9ca4007300 con 0x7f9cac0ce390 2023-10-07T09:47:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.045+0000 7f9ca17fa700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9c90066f30 0x7f9c900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.045+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f9ca4084b20 con 0x7f9cac0ce390 2023-10-07T09:47:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.045+0000 7f9cb119b700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9c90066f30 0x7f9c900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.046+0000 7f9cb119b700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9c90066f30 0x7f9c900693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8007fd0 tx=0x7f9ca8007eb0 comp rx=0 tx=0).ready entity=mgr.7162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.049+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ca4051b60 con 0x7f9cac0ce390 2023-10-07T09:47:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:27.310+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9cac04c3c0 con 0x7f9cac0ce390 2023-10-07T09:47:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.201+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 50) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f9ca4051f30 con 0x7f9cac0ce390 2023-10-07T09:47:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.202+0000 7f9ca17fa700 1 -- 172.21.15.29:0/273814952 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v50) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9ca40555f0 con 0x7f9cac0ce390 2023-10-07T09:47:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9c90066f30 msgr2=0x7f9c900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9c90066f30 0x7f9c900693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8007fd0 tx=0x7f9ca8007eb0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 msgr2=0x7f9cac156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 0x7f9cac156730 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4001460 tx=0x7f9ca4001ae0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 shutdown_connections 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9c90066f30 0x7f9c900693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0ce390 0x7f9cac156730 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cac0b00b0 0x7f9cac150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 --2- 172.21.15.29:0/273814952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9cac0b3e60 0x7f9cac1561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 >> 172.21.15.29:0/273814952 conn(0x7f9cac067610 msgr2=0x7f9cac0ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.206+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 shutdown_connections 2023-10-07T09:47:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.207+0000 7f9cb33ff700 1 -- 172.21.15.29:0/273814952 wait complete. 2023-10-07T09:47:28.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:47:28.414 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:28.414+0000 7f591e3311c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:47:28.430 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:28.429+0000 7f444c0661c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:47:28.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.444+0000 7f375abff700 1 Processor -- start 2023-10-07T09:47:28.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.444+0000 7f375abff700 1 -- start start 2023-10-07T09:47:28.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.445+0000 7f375abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 0x7f37541576e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.445+0000 7f375abff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37540ac100 0x7f37540ac570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.445+0000 7f375abff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37540acab0 con 0x7f3754067680 2023-10-07T09:47:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.445+0000 7f375abff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3754068210 con 0x7f37540ac100 2023-10-07T09:47:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.445+0000 7f375abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3754069000 con 0x7f3754157310 2023-10-07T09:47:28.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 0x7f37541576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f3753fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37540ac100 0x7f37540ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 0x7f37541576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:40228/0 (socket says 172.21.15.29:40228) 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 -- 172.21.15.29:0/1058890199 learned_addr learned my addr 172.21.15.29:0/1058890199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375919c700 1 --1- 172.21.15.29:0/1058890199 >> v1:172.21.15.144:6789/0 conn(0x7f3754067680 0x7f3754067b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:54346/0 (socket says 172.21.15.29:54346) 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 -- 172.21.15.29:0/1058890199 >> v2:172.21.15.144:3300/0 conn(0x7f37540ac100 msgr2=0x7f37540ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 --2- 172.21.15.29:0/1058890199 >> v2:172.21.15.144:3300/0 conn(0x7f37540ac100 0x7f37540ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 -- 172.21.15.29:0/1058890199 >> v1:172.21.15.144:6789/0 conn(0x7f3754067680 legacy=0x7f3754067b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 -- 172.21.15.29:0/1058890199 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540683f0 con 0x7f3754157310 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f3753fff700 1 --2- 172.21.15.29:0/1058890199 >> v2:172.21.15.144:3300/0 conn(0x7f37540ac100 0x7f37540ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.446+0000 7f375899b700 1 --2- 172.21.15.29:0/1058890199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 0x7f37541576e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f37480099e0 tx=0x7f3748009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9c0c6689744438e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f37537fe700 1 -- 172.21.15.29:0/1058890199 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374800b820 con 0x7f3754157310 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f37537fe700 1 -- 172.21.15.29:0/1058890199 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374800b980 con 0x7f3754157310 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f37537fe700 1 -- 172.21.15.29:0/1058890199 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374800bd20 con 0x7f3754157310 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f375abff700 1 -- 172.21.15.29:0/1058890199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 msgr2=0x7f37541576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f375abff700 1 --2- 172.21.15.29:0/1058890199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 0x7f37541576e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f37480099e0 tx=0x7f3748009250 comp rx=0 tx=0).stop 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f375abff700 1 -- 172.21.15.29:0/1058890199 shutdown_connections 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f375abff700 1 --2- 172.21.15.29:0/1058890199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754157310 0x7f37541576e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f375abff700 1 --2- 172.21.15.29:0/1058890199 >> v2:172.21.15.144:3300/0 conn(0x7f37540ac100 0x7f37540ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.447+0000 7f375abff700 1 -- 172.21.15.29:0/1058890199 >> 172.21.15.29:0/1058890199 conn(0x7f37540c9410 msgr2=0x7f37540c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.448+0000 7f375abff700 1 -- 172.21.15.29:0/1058890199 shutdown_connections 2023-10-07T09:47:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.448+0000 7f375abff700 1 -- 172.21.15.29:0/1058890199 wait complete. 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.449+0000 7f375abff700 1 Processor -- start 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.449+0000 7f375abff700 1 -- start start 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 0x7f37540c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37540ac100 0x7f375414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3754157310 0x7f37541502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540c6330 con 0x7f3754157310 2023-10-07T09:47:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540c67c0 con 0x7f37540ac100 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37540c5720 con 0x7f3754067680 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3754157310 0x7f37541502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375899b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 0x7f37540c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f3753fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37540ac100 0x7f375414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.450+0000 7f375899b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 0x7f37540c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:40232/0 (socket says 172.21.15.29:40232) 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 -- 172.21.15.29:0/2494707416 learned_addr learned my addr 172.21.15.29:0/2494707416 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 -- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37540ac100 msgr2=0x7f375414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37540ac100 0x7f375414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 -- 172.21.15.29:0/2494707416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3754157310 msgr2=0x7f37541502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3754157310 0x7f37541502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 -- 172.21.15.29:0/2494707416 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3740009420 con 0x7f3754067680 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f3753fff700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37540ac100 0x7f375414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f375899b700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 0x7f37540c5040 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37540ad1d0 tx=0x7f3748017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.451+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748011d70 con 0x7f3754067680 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.452+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748024440 con 0x7f3754067680 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.452+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748024800 con 0x7f3754067680 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.452+0000 7f375abff700 1 -- 172.21.15.29:0/2494707416 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37480094f0 con 0x7f3754067680 2023-10-07T09:47:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.452+0000 7f375abff700 1 -- 172.21.15.29:0/2494707416 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37540c5ce0 con 0x7f3754067680 2023-10-07T09:47:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.453+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37540c6610 con 0x7f3754067680 2023-10-07T09:47:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.454+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 50) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f374802d440 con 0x7f3754067680 2023-10-07T09:47:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.454+0000 7f3750ff9700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 0x7f3738069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.454+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3748085fa0 con 0x7f3754067680 2023-10-07T09:47:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.455+0000 7f3753fff700 1 -- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 msgr2=0x7f3738069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.455+0000 7f3753fff700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 0x7f3738069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.459+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3748053070 con 0x7f3754067680 2023-10-07T09:47:28.593 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:28.593+0000 7f591e3311c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:47:28.617 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:28.616+0000 7f444c0661c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:47:28.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.656+0000 7f3753fff700 1 -- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 msgr2=0x7f3738069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:28.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.656+0000 7f3753fff700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 0x7f3738069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:28.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.705+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f375404c3c0 con 0x7f3754067680 2023-10-07T09:47:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.710+0000 7f3750ff9700 1 -- 172.21.15.29:0/2494707416 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3748056690 con 0x7f3754067680 2023-10-07T09:47:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.712+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 msgr2=0x7f3738069460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.712+0000 7f373e7fc700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 0x7f3738069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.712+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 msgr2=0x7f37540c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.712+0000 7f373e7fc700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 0x7f37540c5040 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37540ad1d0 tx=0x7f3748017000 comp rx=0 tx=0).stop 2023-10-07T09:47:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 shutdown_connections 2023-10-07T09:47:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3738066fb0 0x7f3738069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3754157310 0x7f37541502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3754067680 0x7f37540c5040 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 --2- 172.21.15.29:0/2494707416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37540ac100 0x7f375414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 >> 172.21.15.29:0/2494707416 conn(0x7f37540c9410 msgr2=0x7f37540cc050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 shutdown_connections 2023-10-07T09:47:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.713+0000 7f373e7fc700 1 -- 172.21.15.29:0/2494707416 wait complete. 2023-10-07T09:47:28.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:47:28.905 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:28.905+0000 7f591e3311c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:47:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.934+0000 7f88681ff700 1 Processor -- start 2023-10-07T09:47:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.935+0000 7f88681ff700 1 -- start start 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f88681ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f886015a960 0x7f8860158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f88681ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600acb40 con 0x7f88600ae8a0 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f88681ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600abf30 con 0x7f886015ad30 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f88681ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8860158c60 con 0x7f886015a960 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f8865f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f886015a960 0x7f8860158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f8865f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f886015a960 0x7f8860158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60096/0 (socket says 172.21.15.29:60096) 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f8865f9b700 1 -- 172.21.15.29:0/947126541 learned_addr learned my addr 172.21.15.29:0/947126541 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.936+0000 7f886679c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f88600ae8a0 0x7f88600b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39794/0 (socket says 172.21.15.29:39794) 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.937+0000 7f8865f9b700 1 -- 172.21.15.29:0/947126541 >> v1:172.21.15.144:6789/0 conn(0x7f886015ad30 legacy=0x7f88600ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.937+0000 7f8865f9b700 1 -- 172.21.15.29:0/947126541 >> v1:172.21.15.29:6789/0 conn(0x7f88600ae8a0 legacy=0x7f88600b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.937+0000 7f8865f9b700 1 -- 172.21.15.29:0/947126541 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88600ac210 con 0x7f886015a960 2023-10-07T09:47:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.937+0000 7f8864f99700 1 -- 172.21.15.29:0/947126541 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809241156 0 0) 0x7f88600acb40 con 0x7f88600ae8a0 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.937+0000 7f8865f9b700 1 --2- 172.21.15.29:0/947126541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f8860158e20 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f885c0099b0 tx=0x7f885c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9bb17b96d3b93d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.937+0000 7f8864f99700 1 -- 172.21.15.29:0/947126541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c00b860 con 0x7f886015a960 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f8864f99700 1 -- 172.21.15.29:0/947126541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c00b9c0 con 0x7f886015a960 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f8864f99700 1 -- 172.21.15.29:0/947126541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c00bd60 con 0x7f886015a960 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 -- 172.21.15.29:0/947126541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 msgr2=0x7f8860158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 --2- 172.21.15.29:0/947126541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f8860158e20 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f885c0099b0 tx=0x7f885c009250 comp rx=0 tx=0).stop 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 -- 172.21.15.29:0/947126541 shutdown_connections 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 --2- 172.21.15.29:0/947126541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f8860158e20 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 -- 172.21.15.29:0/947126541 >> 172.21.15.29:0/947126541 conn(0x7f8860067610 msgr2=0x7f8860067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 -- 172.21.15.29:0/947126541 shutdown_connections 2023-10-07T09:47:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.938+0000 7f88681ff700 1 -- 172.21.15.29:0/947126541 wait complete. 2023-10-07T09:47:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.940+0000 7f88681ff700 1 Processor -- start 2023-10-07T09:47:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.940+0000 7f88681ff700 1 -- start start 2023-10-07T09:47:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f88681ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 0x7f8860150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f88681ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f886014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f88681ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886015ad30 0x7f8860156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f88681ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8860169fc0 con 0x7f886015a960 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f88681ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886014d770 con 0x7f88600ae8a0 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f88681ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886016d400 con 0x7f886015ad30 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f886579a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f886014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f8865f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 0x7f8860150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f8865f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 0x7f8860150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:55436/0 (socket says 172.21.15.29:55436) 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f8865f9b700 1 -- 172.21.15.29:0/3566029110 learned_addr learned my addr 172.21.15.29:0/3566029110 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.941+0000 7f886679c700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886015ad30 0x7f8860156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f886579a700 1 -- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886015ad30 msgr2=0x7f8860156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f886579a700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886015ad30 0x7f8860156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f886579a700 1 -- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 msgr2=0x7f8860150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f886579a700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 0x7f8860150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f886579a700 1 -- 172.21.15.29:0/3566029110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c0094f0 con 0x7f886015a960 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f8865f9b700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 0x7f8860150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f886579a700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f886014c210 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f885000c410 tx=0x7f8850007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8850012420 con 0x7f886015a960 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88500125a0 con 0x7f886015a960 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886014cb60 con 0x7f886015a960 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8850012960 con 0x7f886015a960 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.942+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886014d400 con 0x7f886015a960 2023-10-07T09:47:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.943+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886014d8b0 con 0x7f886015a960 2023-10-07T09:47:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.944+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 50) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f885001b3b0 con 0x7f886015a960 2023-10-07T09:47:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.945+0000 7f8855ffb700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 0x7f8844069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.945+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8850083cf0 con 0x7f886015a960 2023-10-07T09:47:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.945+0000 7f8865f9b700 1 -- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 msgr2=0x7f8844069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.945+0000 7f8865f9b700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 0x7f8844069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:28.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:28.950+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f886014d8b0 con 0x7f886015a960 2023-10-07T09:47:28.955 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:28.954+0000 7f444c0661c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:47:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.146+0000 7f8865f9b700 1 -- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 msgr2=0x7f8844069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.146+0000 7f8865f9b700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 0x7f8844069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:29.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.153+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f886014d8b0 con 0x7f886015a960 2023-10-07T09:47:29.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.157+0000 7f8855ffb700 1 -- 172.21.15.29:0/3566029110 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f886014d8b0 con 0x7f886015a960 2023-10-07T09:47:29.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 msgr2=0x7f8844069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:29.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 0x7f8844069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 msgr2=0x7f886014c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f886014c210 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f885000c410 tx=0x7f8850007ec0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 shutdown_connections 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8844066f80 0x7f8844069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886015ad30 0x7f8860156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88600ae8a0 0x7f8860150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 --2- 172.21.15.29:0/3566029110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f886015a960 0x7f886014c210 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 >> 172.21.15.29:0/3566029110 conn(0x7f8860067610 msgr2=0x7f8860069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 shutdown_connections 2023-10-07T09:47:29.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.160+0000 7f88681ff700 1 -- 172.21.15.29:0/3566029110 wait complete. 2023-10-07T09:47:29.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:29.179 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T09:47:29.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-10-07T09:47:29.426 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:29.426+0000 7f591e3311c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:47:29.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.435+0000 7ff75e3ff700 1 Processor -- start 2023-10-07T09:47:29.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.436+0000 7ff75e3ff700 1 -- start start 2023-10-07T09:47:29.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff75e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff7580ab560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:29.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff75e3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7580acf20 0x7ff7580ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:29.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff75e3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7580ac750 con 0x7ff7581597a0 2023-10-07T09:47:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff75e3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7580ad8e0 con 0x7ff7580acf20 2023-10-07T09:47:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff75e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580abb40 con 0x7ff7581593d0 2023-10-07T09:47:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff7577fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff7581597a0 0x7ff7580aa630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39796/0 (socket says 172.21.15.29:39796) 2023-10-07T09:47:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff7577fe700 1 -- 172.21.15.29:0/1313385521 learned_addr learned my addr 172.21.15.29:0/1313385521 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff757fff700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff7580ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.437+0000 7ff75c99c700 1 --2- 172.21.15.29:0/1313385521 >> v2:172.21.15.144:3300/0 conn(0x7ff7580acf20 0x7ff7580ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff756ffd700 1 -- 172.21.15.29:0/1313385521 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432959025 0 0) 0x7ff7580ac750 con 0x7ff7581597a0 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff756ffd700 1 -- 172.21.15.29:0/1313385521 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff740003570 con 0x7ff7581597a0 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff75c99c700 1 -- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 msgr2=0x7ff7580ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff75c99c700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff7580ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff75c99c700 1 -- 172.21.15.29:0/1313385521 >> v1:172.21.15.29:6789/0 conn(0x7ff7581597a0 legacy=0x7ff7580aa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff75c99c700 1 -- 172.21.15.29:0/1313385521 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7580abe20 con 0x7ff7580acf20 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff757fff700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff7580ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.438+0000 7ff75c99c700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7580acf20 0x7ff7580ad3a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff750009e60 tx=0x7ff750009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99c7398fe60dbce8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff756ffd700 1 -- 172.21.15.29:0/1313385521 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75000e070 con 0x7ff7580acf20 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff756ffd700 1 -- 172.21.15.29:0/1313385521 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75000bab0 con 0x7ff7580acf20 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff756ffd700 1 -- 172.21.15.29:0/1313385521 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75000be70 con 0x7ff7580acf20 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 -- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7580acf20 msgr2=0x7ff7580ad3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7580acf20 0x7ff7580ad3a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff750009e60 tx=0x7ff750009780 comp rx=0 tx=0).stop 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 -- 172.21.15.29:0/1313385521 shutdown_connections 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7580acf20 0x7ff7580ad3a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/1313385521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff7580ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 -- 172.21.15.29:0/1313385521 >> 172.21.15.29:0/1313385521 conn(0x7ff758067610 msgr2=0x7ff758067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 -- 172.21.15.29:0/1313385521 shutdown_connections 2023-10-07T09:47:29.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.439+0000 7ff75e3ff700 1 -- 172.21.15.29:0/1313385521 wait complete. 2023-10-07T09:47:29.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.441+0000 7ff75e3ff700 1 Processor -- start 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.441+0000 7ff75e3ff700 1 -- start start 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75e3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 0x7ff758159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75e3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff75813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75e3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7581597a0 0x7ff7580c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75e3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75815b000 con 0x7ff7580acf20 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75e3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75815a150 con 0x7ff7581597a0 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75e3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff758140430 con 0x7ff7581593d0 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff757fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 0x7ff758159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff757fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 0x7ff758159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60122/0 (socket says 172.21.15.29:60122) 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff75c99c700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7581597a0 0x7ff7580c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff757fff700 1 -- 172.21.15.29:0/2636819251 learned_addr learned my addr 172.21.15.29:0/2636819251 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.442+0000 7ff7577fe700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff75813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75c99c700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 msgr2=0x7ff75813fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75c99c700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff75813fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75c99c700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 msgr2=0x7ff758159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75c99c700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 0x7ff758159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75c99c700 1 -- 172.21.15.29:0/2636819251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748009420 con 0x7ff7581597a0 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 0x7ff758159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75c99c700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7581597a0 0x7ff7580c95b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff750016000 tx=0x7ff750019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750012e10 con 0x7ff7581597a0 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff750027470 con 0x7ff7581597a0 2023-10-07T09:47:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750027830 con 0x7ff7581597a0 2023-10-07T09:47:29.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.443+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff750009a20 con 0x7ff7581597a0 2023-10-07T09:47:29.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.444+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75815a6d0 con 0x7ff7581597a0 2023-10-07T09:47:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.445+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75815b170 con 0x7ff7581597a0 2023-10-07T09:47:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.445+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 50) v1 ==== 84757+0+0 (secure 0 0 0) 0x7ff750024070 con 0x7ff7581597a0 2023-10-07T09:47:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.446+0000 7ff73ffff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.446+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6081+0+0 (secure 0 0 0) 0x7ff750086cc0 con 0x7ff7581597a0 2023-10-07T09:47:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.446+0000 7ff757fff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.446+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:47:29.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.451+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff750053db0 con 0x7ff7581597a0 2023-10-07T09:47:29.509 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:29.508+0000 7f444c0661c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:47:29.596 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:29.596+0000 7f591e3311c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:47:29.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.647+0000 7ff757fff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:29.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.647+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:47:29.685 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:29.684+0000 7f444c0661c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:47:29.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:29.686+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7580718b0 con 0x7ff744066f80 2023-10-07T09:47:30.008 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:30.008+0000 7f591e3311c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:47:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:30.048+0000 7ff757fff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:30.048+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:47:30.110 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:30.109+0000 7f444c0661c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:47:30.778 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:30.777+0000 7f591e3311c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:47:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:30.849+0000 7ff757fff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:30.849+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:47:30.864 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:30.864+0000 7f591e3311c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:47:30.914 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:30.913+0000 7f444c0661c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:47:31.004 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.003+0000 7f444c0661c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:47:31.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.018+0000 7f591e3311c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:47:31.104 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.104+0000 7f591e3311c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:47:31.165 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.164+0000 7f444c0661c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:47:31.255 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.254+0000 7f444c0661c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:47:31.320 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.320+0000 7f591e3311c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:47:31.416 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.415+0000 7f591e3311c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:47:31.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.475+0000 7f444c0661c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:47:31.512 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.512+0000 7f591e3311c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:47:31.574 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.573+0000 7f444c0661c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:47:31.608 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.608+0000 7f591e3311c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:47:31.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.671+0000 7f444c0661c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:47:31.770 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.769+0000 7f444c0661c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:47:31.795 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.795+0000 7f591e3311c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:47:31.893 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:31.893+0000 7f591e3311c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:47:31.965 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:31.962+0000 7f444c0661c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:47:32.063 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:32.062+0000 7f444c0661c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:47:32.234 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:32.234+0000 7f591e3311c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:47:32.323 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:32.323+0000 7f591e3311c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:32.413 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:32.412+0000 7f444c0661c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:47:32.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:32.450+0000 7ff757fff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:47:32.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:32.450+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:47:32.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:32.504+0000 7f444c0661c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:32.860 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:32.859+0000 7f591e3311c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:47:33.044 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:33.044+0000 7f591e3311c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:47:33.067 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:33.067+0000 7f444c0661c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:47:33.259 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:33.258+0000 7f444c0661c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:47:33.421 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:33.420+0000 7f591e3311c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:33.642 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:33.641+0000 7f444c0661c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:33.841 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:33.841+0000 7f591e3311c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:47:34.022 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.021+0000 7f591e3311c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:34.082 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:34.081+0000 7f444c0661c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:47:34.208 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.208+0000 7f591e3311c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:47:34.269 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:34.268+0000 7f444c0661c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:47:34.316 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.316+0000 7f591e3311c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:34.329+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 51) v1 ==== 84468+0+0 (secure 0 0 0) 0x7ff750003040 con 0x7ff7581597a0 2023-10-07T09:47:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:34.329+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:47:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:34.329+0000 7ff73ffff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f588f931700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f588f931700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f588f931700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f588f931700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f588f931700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f5891935700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f5891935700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f5891935700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f5891935700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.455 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:47:34.453+0000 7f5891935700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:47:34.479 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:34.478+0000 7f444c0661c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:47:34.590 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:47:34.589+0000 7f444c0661c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:47:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.333+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mgrmap(e 52) v1 ==== 84595+0+0 (secure 0 0 0) 0x7ff750027c60 con 0x7ff7581597a0 2023-10-07T09:47:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.334+0000 7ff73ffff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.334+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7580718b0 con 0x7ff744066f80 2023-10-07T09:47:35.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.336+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:35.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.339+0000 7ff757fff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff748009d30 tx=0x7ff7480091a0 comp rx=0 tx=0).ready entity=mgr.6848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:35.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.407+0000 7ff73ffff700 1 -- 172.21.15.29:0/2636819251 <== mgr.6848 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7580718b0 con 0x7ff744066f80 2023-10-07T09:47:35.409 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-07T09:47:35.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 msgr2=0x7ff744069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff748009d30 tx=0x7ff7480091a0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7581597a0 msgr2=0x7ff7580c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7581597a0 0x7ff7580c95b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff750016000 tx=0x7ff750019000 comp rx=0 tx=0).stop 2023-10-07T09:47:35.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 shutdown_connections 2023-10-07T09:47:35.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff744066f80 0x7ff744069430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7581597a0 0x7ff7580c95b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7580acf20 0x7ff758159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 --2- 172.21.15.29:0/2636819251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7581593d0 0x7ff75813fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 >> 172.21.15.29:0/2636819251 conn(0x7ff758067610 msgr2=0x7ff75806dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:35.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 shutdown_connections 2023-10-07T09:47:35.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.410+0000 7ff75e3ff700 1 -- 172.21.15.29:0/2636819251 wait complete. 2023-10-07T09:47:35.431 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-07T09:47:35.431 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-07-09_48_00 in ~25s... 2023-10-07T09:47:35.431 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-10-07T09:47:35.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-10-07T09:47:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.631+0000 7fc205bff700 1 Processor -- start 2023-10-07T09:47:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.631+0000 7fc205bff700 1 -- start start 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.632+0000 7fc205bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000accf0 0x7fc2000b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.632+0000 7fc205bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000ad400 con 0x7fc2000afeb0 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.632+0000 7fc205bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000ad6e0 con 0x7fc2000cc810 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.632+0000 7fc205bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20015a140 con 0x7fc2000accf0 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1ff7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc2000afeb0 0x7fc2000aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46580/0 (socket says 172.21.15.29:46580) 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1ff7fe700 1 -- 172.21.15.29:0/3545560143 learned_addr learned my addr 172.21.15.29:0/3545560143 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:47:35.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fffff700 1 --2- 172.21.15.29:0/3545560143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000accf0 0x7fc2000b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fe7fc700 1 -- 172.21.15.29:0/3545560143 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891775640 0 0) 0x7fc2000ad400 con 0x7fc2000afeb0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fe7fc700 1 -- 172.21.15.29:0/3545560143 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e8003570 con 0x7fc2000afeb0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fffff700 1 -- 172.21.15.29:0/3545560143 >> v1:172.21.15.144:6789/0 conn(0x7fc2000cc810 legacy=0x7fc20015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fffff700 1 -- 172.21.15.29:0/3545560143 >> v1:172.21.15.29:6789/0 conn(0x7fc2000afeb0 legacy=0x7fc2000aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fe7fc700 1 -- 172.21.15.29:0/3545560143 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535443845 0 0) 0x7fc2000ad6e0 con 0x7fc2000cc810 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.633+0000 7fc1fffff700 1 -- 172.21.15.29:0/3545560143 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2000ad9c0 con 0x7fc2000accf0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.634+0000 7fc1fffff700 1 --2- 172.21.15.29:0/3545560143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000accf0 0x7fc2000b05d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8009e90 tx=0x7fc1f8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a982b0b8e05be0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.634+0000 7fc1fe7fc700 1 -- 172.21.15.29:0/3545560143 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f800bc30 con 0x7fc2000accf0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.634+0000 7fc1fe7fc700 1 -- 172.21.15.29:0/3545560143 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f800bd90 con 0x7fc2000accf0 2023-10-07T09:47:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.634+0000 7fc1fe7fc700 1 -- 172.21.15.29:0/3545560143 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80116b0 con 0x7fc2000accf0 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.634+0000 7fc205bff700 1 -- 172.21.15.29:0/3545560143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000accf0 msgr2=0x7fc2000b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.634+0000 7fc205bff700 1 --2- 172.21.15.29:0/3545560143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000accf0 0x7fc2000b05d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8009e90 tx=0x7fc1f8009780 comp rx=0 tx=0).stop 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.635+0000 7fc205bff700 1 -- 172.21.15.29:0/3545560143 shutdown_connections 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.635+0000 7fc205bff700 1 --2- 172.21.15.29:0/3545560143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000accf0 0x7fc2000b05d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.635+0000 7fc205bff700 1 -- 172.21.15.29:0/3545560143 >> 172.21.15.29:0/3545560143 conn(0x7fc200067610 msgr2=0x7fc200067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.635+0000 7fc205bff700 1 -- 172.21.15.29:0/3545560143 shutdown_connections 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.635+0000 7fc205bff700 1 -- 172.21.15.29:0/3545560143 wait complete. 2023-10-07T09:47:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.636+0000 7fc205bff700 1 Processor -- start 2023-10-07T09:47:35.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.636+0000 7fc205bff700 1 -- start start 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc205bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2000accf0 0x7fc200157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc205bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000afeb0 0x7fc200169da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc205bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 0x7fc2000b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc205bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2001577f0 con 0x7fc2000cc810 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc205bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20016a2e0 con 0x7fc2000accf0 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc205bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20006e380 con 0x7fc2000afeb0 2023-10-07T09:47:35.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc1fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 0x7fc2000b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc1fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 0x7fc2000b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54524/0 (socket says 172.21.15.29:54524) 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.637+0000 7fc1fffff700 1 -- 172.21.15.29:0/3445349843 learned_addr learned my addr 172.21.15.29:0/3445349843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1ff7fe700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2000accf0 0x7fc200157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1feffd700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000afeb0 0x7fc200169da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1fffff700 1 -- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000afeb0 msgr2=0x7fc200169da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1fffff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000afeb0 0x7fc200169da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1fffff700 1 -- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2000accf0 msgr2=0x7fc200157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1fffff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2000accf0 0x7fc200157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1fffff700 1 -- 172.21.15.29:0/3445349843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8009a20 con 0x7fc2000cc810 2023-10-07T09:47:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1fffff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 0x7fc2000b04d0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc20006f840 tx=0x7fc1f801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:35.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f800bc30 con 0x7fc2000cc810 2023-10-07T09:47:35.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f80032f0 con 0x7fc2000cc810 2023-10-07T09:47:35.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.638+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8011ea0 con 0x7fc2000cc810 2023-10-07T09:47:35.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.639+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc200156be0 con 0x7fc2000cc810 2023-10-07T09:47:35.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.639+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2001571a0 con 0x7fc2000cc810 2023-10-07T09:47:35.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.640+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc200157960 con 0x7fc2000cc810 2023-10-07T09:47:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.641+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fc1f8023610 con 0x7fc2000cc810 2023-10-07T09:47:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.641+0000 7fc1e77fe700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1ec066dd0 0x7fc1ec069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:47:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.642+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fc1f80861f0 con 0x7fc2000cc810 2023-10-07T09:47:35.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.642+0000 7fc1ff7fe700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1ec066dd0 0x7fc1ec069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:47:35.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.643+0000 7fc1ff7fe700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1ec066dd0 0x7fc1ec069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00058d0 tx=0x7fc1f0009040 comp rx=0 tx=0).ready entity=mgr.6848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:47:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.646+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1f8056fd0 con 0x7fc2000cc810 2023-10-07T09:47:35.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.873+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2000718b0 con 0x7fc1ec066dd0 2023-10-07T09:47:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.883+0000 7fc1e77fe700 1 -- 172.21.15.29:0/3445349843 <== mgr.6848 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fc2000718b0 con 0x7fc1ec066dd0 2023-10-07T09:47:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:47:35.884 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1ec066dd0 msgr2=0x7fc1ec069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1ec066dd0 0x7fc1ec069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00058d0 tx=0x7fc1f0009040 comp rx=0 tx=0).stop 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 msgr2=0x7fc2000b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 0x7fc2000b04d0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc20006f840 tx=0x7fc1f801b000 comp rx=0 tx=0).stop 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 shutdown_connections 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1ec066dd0 0x7fc1ec069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2000cc810 0x7fc2000b04d0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2000accf0 0x7fc200157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.885+0000 7fc205bff700 1 --2- 172.21.15.29:0/3445349843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2000afeb0 0x7fc200169da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:47:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.886+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 >> 172.21.15.29:0/3445349843 conn(0x7fc200067610 msgr2=0x7fc2000c8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:47:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.886+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 shutdown_connections 2023-10-07T09:47:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:47:35.886+0000 7fc205bff700 1 -- 172.21.15.29:0/3445349843 wait complete. 2023-10-07T09:47:35.904 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-10-07T09:47:35.904 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{}]} 2023-10-07T09:47:35.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:35.905 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:35.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:35.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:35.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:36.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:36.955 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:37.008 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:37.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:37.009 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:38.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:38.010 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:38.064 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:38.065 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:38.065 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:39.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:39.066 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:39.121 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:39.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:39.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:40.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:40.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:40.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:40.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:40.176 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:41.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:41.178 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:41.232 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:41.233 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:41.233 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:42.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:42.234 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:42.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:42.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:42.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:43.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:43.290 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:43.344 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:43.344 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:43.344 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:44.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:44.346 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:44.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:44.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:44.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:45.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:45.401 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:45.455 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:45.455 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:45.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:46.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:46.456 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:46.511 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:46.511 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:46.511 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:47.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:47.512 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:47.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:47.570 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:47.570 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:48.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:48.571 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:48.625 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:48.625 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:48.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:49.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:49.626 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:49.680 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:49.680 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:49.681 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:50.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:50.815 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:50.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:50.969 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:50.969 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:51.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:51.971 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:52.025 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:52.025 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:52.025 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:52.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:47:52.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:47:53.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:53.027 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:53.081 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:53.081 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:53.081 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:54.082 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:54.082 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:54.136 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:54.136 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:54.137 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:55.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:55.137 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:55.191 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:55.191 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:55.191 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:56.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:56.192 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:56.247 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:56.247 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:56.247 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:57.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:57.248 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:57.302 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:57.303 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:57.303 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:58.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:58.304 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:58.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:58.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:58.359 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:47:59.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:47:59.359 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:47:59.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T09:47:59.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T09:47:59.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:48:00.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:48:00.415 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:48:00.422 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:00.421+0000 7f5882116700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:00.422 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:00.421+0000 7f5882116700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:00.422 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:00.421+0000 7f5882116700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:00.423 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:00.421+0000 7f5882116700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:00.423 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:00.421+0000 7f5882116700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:00.471 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_48_00_UTC 2023-10-07T09:48:00.472 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-09_48_00_UTC'} 2023-10-07T09:48:00.473 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-07-09_48_00_UTC'} 2023-10-07T09:48:00.473 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T09:48:00.473 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-07-09_48_00_UTC'] 2023-10-07T09:48:00.473 DEBUG:tasks.cephfs.test_snap_schedules:exec=1696672055.4315116, now = 1696672080.4732943, timo = 25 2023-10-07T09:48:01.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-10-07T09:48:01.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.684+0000 7f1b293ff700 1 Processor -- start 2023-10-07T09:48:01.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.684+0000 7f1b293ff700 1 -- start start 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b293ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1b240afeb0 0x7f1b240cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b293ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b240accf0 0x7f1b240aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b293ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240ad160 con 0x7f1b240af4c0 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b293ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2415a140 con 0x7f1b240accf0 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b293ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b240adca0 con 0x7f1b240afeb0 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b227fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b240accf0 0x7f1b240aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:01.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b227fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b240accf0 0x7f1b240aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53468/0 (socket says 172.21.15.29:53468) 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.685+0000 7f1b227fc700 1 -- 172.21.15.29:0/4254544510 learned_addr learned my addr 172.21.15.29:0/4254544510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b22ffd700 1 --2- 172.21.15.29:0/4254544510 >> v2:172.21.15.144:3300/0 conn(0x7f1b240afeb0 0x7f1b240cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b227fc700 1 -- 172.21.15.29:0/4254544510 >> v2:172.21.15.144:3300/0 conn(0x7f1b240afeb0 msgr2=0x7f1b240cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b227fc700 1 --2- 172.21.15.29:0/4254544510 >> v2:172.21.15.144:3300/0 conn(0x7f1b240afeb0 0x7f1b240cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b227fc700 1 -- 172.21.15.29:0/4254544510 >> v1:172.21.15.144:6789/0 conn(0x7f1b240af4c0 legacy=0x7f1b2415a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b227fc700 1 -- 172.21.15.29:0/4254544510 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b240ad400 con 0x7f1b240accf0 2023-10-07T09:48:01.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b227fc700 1 --2- 172.21.15.29:0/4254544510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240aef80 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1b240685f0 tx=0x7f1b18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750fc1cee6bcb328 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:01.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b21ffb700 1 -- 172.21.15.29:0/4254544510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1800b820 con 0x7f1b240accf0 2023-10-07T09:48:01.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b21ffb700 1 -- 172.21.15.29:0/4254544510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1800b980 con 0x7f1b240accf0 2023-10-07T09:48:01.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.686+0000 7f1b21ffb700 1 -- 172.21.15.29:0/4254544510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1800bd20 con 0x7f1b240accf0 2023-10-07T09:48:01.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 -- 172.21.15.29:0/4254544510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 msgr2=0x7f1b240aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 --2- 172.21.15.29:0/4254544510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240aef80 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1b240685f0 tx=0x7f1b18009250 comp rx=0 tx=0).stop 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 -- 172.21.15.29:0/4254544510 shutdown_connections 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 --2- 172.21.15.29:0/4254544510 >> v2:172.21.15.144:3300/0 conn(0x7f1b240afeb0 0x7f1b240cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 --2- 172.21.15.29:0/4254544510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240aef80 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 -- 172.21.15.29:0/4254544510 >> 172.21.15.29:0/4254544510 conn(0x7f1b24067610 msgr2=0x7f1b24067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 -- 172.21.15.29:0/4254544510 shutdown_connections 2023-10-07T09:48:01.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.687+0000 7f1b293ff700 1 -- 172.21.15.29:0/4254544510 wait complete. 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.689+0000 7f1b293ff700 1 Processor -- start 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.689+0000 7f1b293ff700 1 -- start start 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b293ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b293ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b240af4c0 0x7f1b241548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b293ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b240afeb0 0x7f1b24154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b293ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240c9fb0 con 0x7f1b240accf0 2023-10-07T09:48:01.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b293ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2414c270 con 0x7f1b240afeb0 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b293ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240c99f0 con 0x7f1b240af4c0 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53480/0 (socket says 172.21.15.29:53480) 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 -- 172.21.15.29:0/2634569860 learned_addr learned my addr 172.21.15.29:0/2634569860 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 -- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b240af4c0 msgr2=0x7f1b241548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b240af4c0 0x7f1b241548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 -- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b240afeb0 msgr2=0x7f1b24154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b240afeb0 0x7f1b24154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.690+0000 7f1b22ffd700 1 -- 172.21.15.29:0/2634569860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b180094f0 con 0x7f1b240accf0 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.691+0000 7f1b22ffd700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240c9080 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400d860 tx=0x7f1b14008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:01.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.691+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b140125f0 con 0x7f1b240accf0 2023-10-07T09:48:01.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.691+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b14012750 con 0x7f1b240accf0 2023-10-07T09:48:01.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.691+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14012b10 con 0x7f1b240accf0 2023-10-07T09:48:01.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.691+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b240c9cd0 con 0x7f1b240accf0 2023-10-07T09:48:01.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.691+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b240c95c0 con 0x7f1b240accf0 2023-10-07T09:48:01.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.692+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b240c97b0 con 0x7f1b240accf0 2023-10-07T09:48:01.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.693+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f1b1401e460 con 0x7f1b240accf0 2023-10-07T09:48:01.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.694+0000 7f1b0affd700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1b10067130 0x7f1b100695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:01.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.694+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f1b14085290 con 0x7f1b240accf0 2023-10-07T09:48:01.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.695+0000 7f1b227fc700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1b10067130 0x7f1b100695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:01.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.695+0000 7f1b227fc700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1b10067130 0x7f1b100695e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b240ccd50 tx=0x7f1b18016000 comp rx=0 tx=0).ready entity=mgr.6848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:01.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.698+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b240c97b0 con 0x7f1b240accf0 2023-10-07T09:48:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:01.925+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b24155330 con 0x7f1b10067130 2023-10-07T09:48:02.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.000+0000 7f1b0affd700 1 -- 172.21.15.29:0/2634569860 <== mgr.6848 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1b24155330 con 0x7f1b10067130 2023-10-07T09:48:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-07T09:48:02.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1b10067130 msgr2=0x7f1b100695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1b10067130 0x7f1b100695e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b240ccd50 tx=0x7f1b18016000 comp rx=0 tx=0).stop 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 msgr2=0x7f1b240c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240c9080 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400d860 tx=0x7f1b14008040 comp rx=0 tx=0).stop 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 shutdown_connections 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1b10067130 0x7f1b100695e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b240afeb0 0x7f1b24154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b240accf0 0x7f1b240c9080 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 --2- 172.21.15.29:0/2634569860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b240af4c0 0x7f1b241548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 >> 172.21.15.29:0/2634569860 conn(0x7f1b24067610 msgr2=0x7f1b24156f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 shutdown_connections 2023-10-07T09:48:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.003+0000 7f1b293ff700 1 -- 172.21.15.29:0/2634569860 wait complete. 2023-10-07T09:48:02.022 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-07T09:48:02.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:48:02.022 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T09:48:02.069 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_48_00_UTC 2023-10-07T09:48:02.070 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_48_00_UTC 2023-10-07T09:48:02.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:48:02.071 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_48_00_UTC) 2023-10-07T09:48:02.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:48:02.123 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T09:48:02.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T09:48:02.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.371+0000 7fb042bff700 1 Processor -- start 2023-10-07T09:48:02.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.371+0000 7fb042bff700 1 -- start start 2023-10-07T09:48:02.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.372+0000 7fb042bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb03c15a960 0x7fb03c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:02.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.372+0000 7fb042bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0acb40 con 0x7fb03c0ae8a0 2023-10-07T09:48:02.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.372+0000 7fb042bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0abf30 con 0x7fb03c15ad30 2023-10-07T09:48:02.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.372+0000 7fb042bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c158c60 con 0x7fb03c15a960 2023-10-07T09:48:02.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.372+0000 7fb04099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb03c15a960 0x7fb03c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:02.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb04119c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb03c0ae8a0 0x7fb03c0b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56240/0 (socket says 172.21.15.29:56240) 2023-10-07T09:48:02.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb04119c700 1 -- 172.21.15.29:0/867404263 learned_addr learned my addr 172.21.15.29:0/867404263 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:02.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb03b7fe700 1 -- 172.21.15.29:0/867404263 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476722233 0 0) 0x7fb03c0acb40 con 0x7fb03c0ae8a0 2023-10-07T09:48:02.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb04099b700 1 -- 172.21.15.29:0/867404263 >> v1:172.21.15.144:6789/0 conn(0x7fb03c15ad30 legacy=0x7fb03c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb04099b700 1 -- 172.21.15.29:0/867404263 >> v1:172.21.15.29:6789/0 conn(0x7fb03c0ae8a0 legacy=0x7fb03c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb04099b700 1 -- 172.21.15.29:0/867404263 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0ac210 con 0x7fb03c15a960 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb04099b700 1 --2- 172.21.15.29:0/867404263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c158e20 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098e0 tx=0x7fb0300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c6d99396d427cf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.373+0000 7fb03b7fe700 1 -- 172.21.15.29:0/867404263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb030009e80 con 0x7fb03c15a960 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb03b7fe700 1 -- 172.21.15.29:0/867404263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03000b860 con 0x7fb03c15a960 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb03b7fe700 1 -- 172.21.15.29:0/867404263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03000bbe0 con 0x7fb03c15a960 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 -- 172.21.15.29:0/867404263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 msgr2=0x7fb03c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 --2- 172.21.15.29:0/867404263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c158e20 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098e0 tx=0x7fb0300091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 -- 172.21.15.29:0/867404263 shutdown_connections 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 --2- 172.21.15.29:0/867404263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c158e20 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 -- 172.21.15.29:0/867404263 >> 172.21.15.29:0/867404263 conn(0x7fb03c067610 msgr2=0x7fb03c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 -- 172.21.15.29:0/867404263 shutdown_connections 2023-10-07T09:48:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.374+0000 7fb042bff700 1 -- 172.21.15.29:0/867404263 wait complete. 2023-10-07T09:48:02.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.376+0000 7fb042bff700 1 Processor -- start 2023-10-07T09:48:02.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.376+0000 7fb042bff700 1 -- start start 2023-10-07T09:48:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb042bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb03c0ae8a0 0x7fb03c15fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb042bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb042bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb03c15ad30 0x7fb03c0aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb042bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c15d390 con 0x7fb03c15a960 2023-10-07T09:48:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb042bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0cb210 con 0x7fb03c15ad30 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb042bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0cb4f0 con 0x7fb03c0ae8a0 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb03bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb03bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53504/0 (socket says 172.21.15.29:53504) 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb03bfff700 1 -- 172.21.15.29:0/180020145 learned_addr learned my addr 172.21.15.29:0/180020145 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb04099b700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb03c0ae8a0 0x7fb03c15fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb04119c700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb03c15ad30 0x7fb03c0aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb03bfff700 1 -- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb03c0ae8a0 msgr2=0x7fb03c15fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.377+0000 7fb03bfff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb03c0ae8a0 0x7fb03c15fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb03bfff700 1 -- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb03c15ad30 msgr2=0x7fb03c0aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb03bfff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb03c15ad30 0x7fb03c0aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb03bfff700 1 -- 172.21.15.29:0/180020145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030009420 con 0x7fb03c15a960 2023-10-07T09:48:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb03bfff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c150c60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb02800c590 tx=0x7fb028007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb028011590 con 0x7fb03c15a960 2023-10-07T09:48:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb028011710 con 0x7fb03c15a960 2023-10-07T09:48:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c0cb7d0 con 0x7fb03c15a960 2023-10-07T09:48:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb028011ad0 con 0x7fb03c15a960 2023-10-07T09:48:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.378+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c160410 con 0x7fb03c15a960 2023-10-07T09:48:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.379+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c15d540 con 0x7fb03c15a960 2023-10-07T09:48:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.380+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 84916+0+0 (secure 0 0 0) 0x7fb02801a360 con 0x7fb03c15a960 2023-10-07T09:48:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.381+0000 7fb038ff9700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb020067130 0x7fb0200695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.382+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fb028083b20 con 0x7fb03c15a960 2023-10-07T09:48:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.382+0000 7fb04099b700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb020067130 0x7fb0200695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:02.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.383+0000 7fb04099b700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb020067130 0x7fb0200695e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0300093f0 tx=0x7fb030017000 comp rx=0 tx=0).ready entity=mgr.6848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.386+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb03c15d540 con 0x7fb03c15a960 2023-10-07T09:48:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:02.649+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb03c04c3c0 con 0x7fb03c15a960 2023-10-07T09:48:03.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.235+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fb028050bc0 con 0x7fb03c15a960 2023-10-07T09:48:03.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.237+0000 7fb038ff9700 1 -- 172.21.15.29:0/180020145 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v54) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb028049720 con 0x7fb03c15a960 2023-10-07T09:48:03.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb020067130 msgr2=0x7fb0200695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb020067130 0x7fb0200695e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0300093f0 tx=0x7fb030017000 comp rx=0 tx=0).stop 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 msgr2=0x7fb03c150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c150c60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb02800c590 tx=0x7fb028007f50 comp rx=0 tx=0).stop 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 shutdown_connections 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb020067130 0x7fb0200695e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb03c15ad30 0x7fb03c0aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb03c0ae8a0 0x7fb03c15fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 --2- 172.21.15.29:0/180020145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03c15a960 0x7fb03c150c60 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 >> 172.21.15.29:0/180020145 conn(0x7fb03c067610 msgr2=0x7fb03c15b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 shutdown_connections 2023-10-07T09:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.241+0000 7fb042bff700 1 -- 172.21.15.29:0/180020145 wait complete. 2023-10-07T09:48:03.264 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:48:03.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:48:03.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:48:03.457 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:03.456+0000 7fef0404a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:48:03.458 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:03.457+0000 7f2be375c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:48:03.479 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:48:03.479 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T09:48:03.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:48:03.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T09:48:03.568 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T09:48:03.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:03.568 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T09:48:03.637 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:03.635+0000 7f2be375c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:48:03.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:03.654 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:03.652+0000 7fef0404a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:48:03.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.856+0000 7f48419ff700 1 Processor -- start 2023-10-07T09:48:03.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.856+0000 7f48419ff700 1 -- start start 2023-10-07T09:48:03.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f48419ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f483c159750 0x7f483c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:03.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f48419ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f483c0ce8a0 0x7f483c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:03.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f48419ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0cbf20 con 0x7f483c159b20 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f48419ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c0cc200 con 0x7f483c159750 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f48419ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c0cc4e0 con 0x7f483c0ce8a0 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f483affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f483c159750 0x7f483c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f483affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f483c159750 0x7f483c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54570/0 (socket says 172.21.15.29:54570) 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f483affd700 1 -- 172.21.15.29:0/3455374861 learned_addr learned my addr 172.21.15.29:0/3455374861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f483a7fc700 1 --1- 172.21.15.29:0/3455374861 >> v1:172.21.15.29:6789/0 conn(0x7f483c159b20 0x7f483c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40308/0 (socket says 172.21.15.29:40308) 2023-10-07T09:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.858+0000 7f483b7fe700 1 --2- 172.21.15.29:0/3455374861 >> v2:172.21.15.144:3300/0 conn(0x7f483c0ce8a0 0x7f483c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f483affd700 1 -- 172.21.15.29:0/3455374861 >> v2:172.21.15.144:3300/0 conn(0x7f483c0ce8a0 msgr2=0x7f483c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f483affd700 1 --2- 172.21.15.29:0/3455374861 >> v2:172.21.15.144:3300/0 conn(0x7f483c0ce8a0 0x7f483c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f483affd700 1 -- 172.21.15.29:0/3455374861 >> v1:172.21.15.29:6789/0 conn(0x7f483c159b20 legacy=0x7f483c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f483affd700 1 -- 172.21.15.29:0/3455374861 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c0cc7c0 con 0x7f483c159750 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f4839ffb700 1 -- 172.21.15.29:0/3455374861 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879618959 0 0) 0x7f483c0cbf20 con 0x7f483c159b20 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f483affd700 1 --2- 172.21.15.29:0/3455374861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c0d10a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f483c15e510 tx=0x7f482c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d089258a75eecb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f4839ffb700 1 -- 172.21.15.29:0/3455374861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c005c10 con 0x7f483c159750 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f4839ffb700 1 -- 172.21.15.29:0/3455374861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c005d70 con 0x7f483c159750 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.859+0000 7f4839ffb700 1 -- 172.21.15.29:0/3455374861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c006260 con 0x7f483c159750 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 -- 172.21.15.29:0/3455374861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 msgr2=0x7f483c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 --2- 172.21.15.29:0/3455374861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c0d10a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f483c15e510 tx=0x7f482c009220 comp rx=0 tx=0).stop 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 -- 172.21.15.29:0/3455374861 shutdown_connections 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 --2- 172.21.15.29:0/3455374861 >> v2:172.21.15.144:3300/0 conn(0x7f483c0ce8a0 0x7f483c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 --2- 172.21.15.29:0/3455374861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c0d10a0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 -- 172.21.15.29:0/3455374861 >> 172.21.15.29:0/3455374861 conn(0x7f483c063a10 msgr2=0x7f483c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 -- 172.21.15.29:0/3455374861 shutdown_connections 2023-10-07T09:48:03.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.860+0000 7f48419ff700 1 -- 172.21.15.29:0/3455374861 wait complete. 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.862+0000 7f48419ff700 1 Processor -- start 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.862+0000 7f48419ff700 1 -- start start 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f48419ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f483c0ce8a0 0x7f483c1470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f48419ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c16a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f48419ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c159b20 0x7f483c15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f48419ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c147610 con 0x7f483c159750 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f48419ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0b4810 con 0x7f483c159b20 2023-10-07T09:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f48419ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0b4af0 con 0x7f483c0ce8a0 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c16a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c16a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54584/0 (socket says 172.21.15.29:54584) 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c159b20 0x7f483c15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 -- 172.21.15.29:0/3556856394 learned_addr learned my addr 172.21.15.29:0/3556856394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483affd700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f483c0ce8a0 0x7f483c1470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 -- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f483c0ce8a0 msgr2=0x7f483c1470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f483c0ce8a0 0x7f483c1470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 -- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c159b20 msgr2=0x7f483c15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c159b20 0x7f483c15ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483a7fc700 1 -- 172.21.15.29:0/3556856394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c009480 con 0x7f483c159750 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.863+0000 7f483b7fe700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c159b20 0x7f483c15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.864+0000 7f483a7fc700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c16a900 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f483000c5e0 tx=0x7f4830001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.864+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830005a70 con 0x7f483c159750 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.864+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4830005bd0 con 0x7f483c159750 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.864+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300070d0 con 0x7f483c159750 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.864+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c0c8790 con 0x7f483c159750 2023-10-07T09:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.864+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c0c8d50 con 0x7f483c159750 2023-10-07T09:48:03.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.865+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c0b4c80 con 0x7f483c159750 2023-10-07T09:48:03.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.866+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f4830007480 con 0x7f483c159750 2023-10-07T09:48:03.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.867+0000 7f4822ffd700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 0x7f4828069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.868+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f48300844e0 con 0x7f483c159750 2023-10-07T09:48:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.868+0000 7f483affd700 1 -- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 msgr2=0x7f4828069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.868+0000 7f483affd700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 0x7f4828069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:03.871+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4830051150 con 0x7f483c159750 2023-10-07T09:48:03.951 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:03.950+0000 7f2be375c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:48:03.996 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:03.995+0000 7fef0404a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:48:04.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.069+0000 7f483affd700 1 -- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 msgr2=0x7f4828069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:04.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.069+0000 7f483affd700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 0x7f4828069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:04.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.120+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f483c0c8f60 con 0x7f483c159750 2023-10-07T09:48:04.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.120+0000 7f4822ffd700 1 -- 172.21.15.29:0/3556856394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f4830054770 con 0x7f483c159750 2023-10-07T09:48:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:04.122 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:47:17.881832+0000","modified":"2023-10-07T09:47:21.398903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/581264510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":581264510},{"type":"v1","addr":"172.21.15.29:6835","nonce":581264510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-10-07T09:48:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 msgr2=0x7f4828069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 0x7f4828069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 msgr2=0x7f483c16a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c16a900 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f483000c5e0 tx=0x7f4830001040 comp rx=0 tx=0).stop 2023-10-07T09:48:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 shutdown_connections 2023-10-07T09:48:04.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4828067170 0x7f4828069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c159b20 0x7f483c15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f483c0ce8a0 0x7f483c1470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 --2- 172.21.15.29:0/3556856394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f483c159750 0x7f483c16a900 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 >> 172.21.15.29:0/3556856394 conn(0x7f483c063a10 msgr2=0x7f483c154950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 shutdown_connections 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.123+0000 7f48419ff700 1 -- 172.21.15.29:0/3556856394 wait complete. 2023-10-07T09:48:04.125 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-07T09:48:04.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:04.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.346+0000 7f4681bff700 1 Processor -- start 2023-10-07T09:48:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.346+0000 7f4681bff700 1 -- start start 2023-10-07T09:48:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.347+0000 7f4681bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f467c156a90 0x7f467c1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.347+0000 7f4681bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0afda0 0x7f467c0b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.347+0000 7f4681bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c159670 con 0x7f467c0af6d0 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f4681bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c15a680 con 0x7f467c156a90 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f4681bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0b2070 con 0x7f467c0afda0 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f467c156a90 0x7f467c1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f467c156a90 0x7f467c1533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54588/0 (socket says 172.21.15.29:54588) 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 -- 172.21.15.29:0/96828400 learned_addr learned my addr 172.21.15.29:0/96828400 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467b7fe700 1 --1- 172.21.15.29:0/96828400 >> v1:172.21.15.29:6789/0 conn(0x7f467c0af6d0 0x7f467c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40320/0 (socket says 172.21.15.29:40320) 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467bfff700 1 --2- 172.21.15.29:0/96828400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0afda0 0x7f467c0b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 -- 172.21.15.29:0/96828400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0afda0 msgr2=0x7f467c0b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 --2- 172.21.15.29:0/96828400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0afda0 0x7f467c0b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 -- 172.21.15.29:0/96828400 >> v1:172.21.15.29:6789/0 conn(0x7f467c0af6d0 legacy=0x7f467c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467affd700 1 -- 172.21.15.29:0/96828400 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c15a1f0 con 0x7f467c156a90 2023-10-07T09:48:04.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.348+0000 7f467a7fc700 1 -- 172.21.15.29:0/96828400 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022530403 0 0) 0x7f467c159670 con 0x7f467c0af6d0 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.349+0000 7f467affd700 1 --2- 172.21.15.29:0/96828400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c1533f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f467000bc80 tx=0x7f467000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4309e2ed726216 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.349+0000 7f467a7fc700 1 -- 172.21.15.29:0/96828400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467000dbb0 con 0x7f467c156a90 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.349+0000 7f467a7fc700 1 -- 172.21.15.29:0/96828400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467000dd10 con 0x7f467c156a90 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.349+0000 7f467a7fc700 1 -- 172.21.15.29:0/96828400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46700196b0 con 0x7f467c156a90 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 -- 172.21.15.29:0/96828400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 msgr2=0x7f467c1533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 --2- 172.21.15.29:0/96828400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c1533f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f467000bc80 tx=0x7f467000b4f0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 -- 172.21.15.29:0/96828400 shutdown_connections 2023-10-07T09:48:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 --2- 172.21.15.29:0/96828400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0afda0 0x7f467c0b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 --2- 172.21.15.29:0/96828400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c1533f0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 -- 172.21.15.29:0/96828400 >> 172.21.15.29:0/96828400 conn(0x7f467c0680a0 msgr2=0x7f467c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 -- 172.21.15.29:0/96828400 shutdown_connections 2023-10-07T09:48:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.350+0000 7f4681bff700 1 -- 172.21.15.29:0/96828400 wait complete. 2023-10-07T09:48:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.352+0000 7f4681bff700 1 Processor -- start 2023-10-07T09:48:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.352+0000 7f4681bff700 1 -- start start 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f4681bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0af6d0 0x7f467c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f4681bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f467c0afda0 0x7f467c0c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f4681bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f4681bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c8c20 con 0x7f467c156a90 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f4681bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c96f0 con 0x7f467c0afda0 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f4681bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c8e50 con 0x7f467c0af6d0 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f467bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f467b7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0af6d0 0x7f467c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f467bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c169990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54600/0 (socket says 172.21.15.29:54600) 2023-10-07T09:48:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f467bfff700 1 -- 172.21.15.29:0/1243783144 learned_addr learned my addr 172.21.15.29:0/1243783144 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.354+0000 7f467affd700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f467c0afda0 0x7f467c0c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f467b7fe700 1 -- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f467c0afda0 msgr2=0x7f467c0c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f467b7fe700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f467c0afda0 0x7f467c0c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f467b7fe700 1 -- 172.21.15.29:0/1243783144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 msgr2=0x7f467c169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f467b7fe700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c169990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f467b7fe700 1 -- 172.21.15.29:0/1243783144 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467000b790 con 0x7f467c0af6d0 2023-10-07T09:48:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f467b7fe700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0af6d0 0x7f467c0ca5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c5b0 tx=0x7f466c007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c005b40 con 0x7f467c0af6d0 2023-10-07T09:48:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c005ca0 con 0x7f467c0af6d0 2023-10-07T09:48:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c0c9130 con 0x7f467c0af6d0 2023-10-07T09:48:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c007320 con 0x7f467c0af6d0 2023-10-07T09:48:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.355+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c0c5420 con 0x7f467c0af6d0 2023-10-07T09:48:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.356+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c04c3c0 con 0x7f467c0af6d0 2023-10-07T09:48:04.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.358+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f466c01b3b0 con 0x7f467c0af6d0 2023-10-07T09:48:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.359+0000 7f46637fe700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 0x7f4668069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.359+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f466c083f10 con 0x7f467c0af6d0 2023-10-07T09:48:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.359+0000 7f467affd700 1 -- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 msgr2=0x7f4668069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:04.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.359+0000 7f467affd700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 0x7f4668069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:04.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.364+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f466c050c70 con 0x7f467c0af6d0 2023-10-07T09:48:04.472 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:04.471+0000 7f2be375c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:48:04.558 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:04.557+0000 7fef0404a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:48:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.560+0000 7f467affd700 1 -- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 msgr2=0x7f4668069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.560+0000 7f467affd700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 0x7f4668069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.568+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467c0c5610 con 0x7f467c0af6d0 2023-10-07T09:48:04.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.569+0000 7f46637fe700 1 -- 172.21.15.29:0/1243783144 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16405 (secure 0 0 0) 0x7f466c017020 con 0x7f467c0af6d0 2023-10-07T09:48:04.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:04.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":119,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:47:34.318934+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:47:15.663387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-07T09:47:16.857437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3818341042":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/1072858706":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/378227010":"2023-10-08T09:47:34.318899+0000","172.21.15.144:6836/114714":"2023-10-08T09:47:34.318899+0000","172.21.15.144:6834/114714":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/2477548044":"2023-10-08T09:47:34.318899+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:04.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 msgr2=0x7f4668069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:04.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 0x7f4668069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0af6d0 msgr2=0x7f467c0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0af6d0 0x7f467c0ca5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c5b0 tx=0x7f466c007d50 comp rx=0 tx=0).stop 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 shutdown_connections 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4668067170 0x7f4668069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467c156a90 0x7f467c169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f467c0af6d0 0x7f467c0ca5f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 --2- 172.21.15.29:0/1243783144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f467c0afda0 0x7f467c0c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 >> 172.21.15.29:0/1243783144 conn(0x7f467c0680a0 msgr2=0x7f467c157520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 shutdown_connections 2023-10-07T09:48:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.572+0000 7f4681bff700 1 -- 172.21.15.29:0/1243783144 wait complete. 2023-10-07T09:48:04.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:47:15.663387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:47:16.857437+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T09:48:04.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:48:04.641 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:04.640+0000 7f2be375c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:48:04.743 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:04.742+0000 7fef0404a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:48:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.793+0000 7fd06b9ff700 1 Processor -- start 2023-10-07T09:48:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.794+0000 7fd06b9ff700 1 -- start start 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06b9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd064153cc0 0x7fd064159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06b9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd06414cca0 0x7fd064159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06b9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd064157790 con 0x7fd06414c8d0 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06b9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0640b0950 con 0x7fd06414cca0 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06b9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd064157580 con 0x7fd064153cc0 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06979b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd06414c8d0 0x7fd0640afed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40332/0 (socket says 172.21.15.29:40332) 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd06979b700 1 -- 172.21.15.29:0/1844613034 learned_addr learned my addr 172.21.15.29:0/1844613034 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd068f9a700 1 --2- 172.21.15.29:0/1844613034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd064153cc0 0x7fd064159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd069f9c700 1 --2- 172.21.15.29:0/1844613034 >> v2:172.21.15.144:3300/0 conn(0x7fd06414cca0 0x7fd064159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd05bfff700 1 -- 172.21.15.29:0/1844613034 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478542587 0 0) 0x7fd064157790 con 0x7fd06414c8d0 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.796+0000 7fd05bfff700 1 -- 172.21.15.29:0/1844613034 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd050003570 con 0x7fd06414c8d0 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd068f9a700 1 -- 172.21.15.29:0/1844613034 >> v2:172.21.15.144:3300/0 conn(0x7fd06414cca0 msgr2=0x7fd064159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd068f9a700 1 --2- 172.21.15.29:0/1844613034 >> v2:172.21.15.144:3300/0 conn(0x7fd06414cca0 0x7fd064159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd068f9a700 1 -- 172.21.15.29:0/1844613034 >> v1:172.21.15.29:6789/0 conn(0x7fd06414c8d0 legacy=0x7fd0640afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd068f9a700 1 -- 172.21.15.29:0/1844613034 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd064157a70 con 0x7fd064153cc0 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd069f9c700 1 --2- 172.21.15.29:0/1844613034 >> v2:172.21.15.144:3300/0 conn(0x7fd06414cca0 0x7fd064159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd068f9a700 1 --2- 172.21.15.29:0/1844613034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd064153cc0 0x7fd064159680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd0641576c0 tx=0x7fd05400a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa46053a2cee4544 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd05bfff700 1 -- 172.21.15.29:0/1844613034 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400aea0 con 0x7fd064153cc0 2023-10-07T09:48:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd05bfff700 1 -- 172.21.15.29:0/1844613034 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05400c970 con 0x7fd064153cc0 2023-10-07T09:48:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.797+0000 7fd05bfff700 1 -- 172.21.15.29:0/1844613034 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400cd30 con 0x7fd064153cc0 2023-10-07T09:48:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 -- 172.21.15.29:0/1844613034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd064153cc0 msgr2=0x7fd064159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/1844613034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd064153cc0 0x7fd064159680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd0641576c0 tx=0x7fd05400a330 comp rx=0 tx=0).stop 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 -- 172.21.15.29:0/1844613034 shutdown_connections 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/1844613034 >> v2:172.21.15.144:3300/0 conn(0x7fd06414cca0 0x7fd064159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/1844613034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd064153cc0 0x7fd064159680 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 -- 172.21.15.29:0/1844613034 >> 172.21.15.29:0/1844613034 conn(0x7fd064067610 msgr2=0x7fd064067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 -- 172.21.15.29:0/1844613034 shutdown_connections 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.798+0000 7fd06b9ff700 1 -- 172.21.15.29:0/1844613034 wait complete. 2023-10-07T09:48:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.800+0000 7fd06b9ff700 1 Processor -- start 2023-10-07T09:48:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.800+0000 7fd06b9ff700 1 -- start start 2023-10-07T09:48:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06b9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd06414c8d0 0x7fd064148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06b9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 0x7fd06414ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06b9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd064153cc0 0x7fd06414f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06b9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd064151d10 con 0x7fd06414cca0 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06b9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0641544b0 con 0x7fd064153cc0 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06b9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd064154790 con 0x7fd06414c8d0 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd068f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 0x7fd06414ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd06979b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd06414c8d0 0x7fd064148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd068f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 0x7fd06414ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54614/0 (socket says 172.21.15.29:54614) 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd068f9a700 1 -- 172.21.15.29:0/769271948 learned_addr learned my addr 172.21.15.29:0/769271948 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.801+0000 7fd069f9c700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd064153cc0 0x7fd06414f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd068f9a700 1 -- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd06414c8d0 msgr2=0x7fd064148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd068f9a700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd06414c8d0 0x7fd064148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd068f9a700 1 -- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd064153cc0 msgr2=0x7fd06414f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd068f9a700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd064153cc0 0x7fd06414f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd068f9a700 1 -- 172.21.15.29:0/769271948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c009a20 con 0x7fd06414cca0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd06979b700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd06414c8d0 0x7fd064148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd068f9a700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 0x7fd06414ecf0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd05400a370 tx=0x7fd054019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd054020440 con 0x7fd06414cca0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0540205e0 con 0x7fd06414cca0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05400a5d0 con 0x7fd06414cca0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0540209a0 con 0x7fd06414cca0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.802+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd064154d50 con 0x7fd06414cca0 2023-10-07T09:48:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.804+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06404c3c0 con 0x7fd06414cca0 2023-10-07T09:48:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.804+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fd054020d10 con 0x7fd06414cca0 2023-10-07T09:48:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.805+0000 7fd0597fa700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 0x7fd048069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.805+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fd054085fb0 con 0x7fd06414cca0 2023-10-07T09:48:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.806+0000 7fd06979b700 1 -- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 msgr2=0x7fd048069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.806+0000 7fd06979b700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 0x7fd048069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:04.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:04.810+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd054052e60 con 0x7fd06414cca0 2023-10-07T09:48:05.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.007+0000 7fd06979b700 1 -- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 msgr2=0x7fd048069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:05.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.007+0000 7fd06979b700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 0x7fd048069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:05.060 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:05.058+0000 7f2be375c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:48:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.084+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0640ccd00 con 0x7fd06414cca0 2023-10-07T09:48:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.084+0000 7fd0597fa700 1 -- 172.21.15.29:0/769271948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd054056480 con 0x7fd06414cca0 2023-10-07T09:48:05.085 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 msgr2=0x7fd048069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 0x7fd048069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 msgr2=0x7fd06414ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 0x7fd06414ecf0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd05400a370 tx=0x7fd054019000 comp rx=0 tx=0).stop 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 shutdown_connections 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd048067170 0x7fd048069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd064153cc0 0x7fd06414f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd06414c8d0 0x7fd064148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.086+0000 7fd06b9ff700 1 --2- 172.21.15.29:0/769271948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06414cca0 0x7fd06414ecf0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.087+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 >> 172.21.15.29:0/769271948 conn(0x7fd064067610 msgr2=0x7fd0641595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:05.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.087+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 shutdown_connections 2023-10-07T09:48:05.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.087+0000 7fd06b9ff700 1 -- 172.21.15.29:0/769271948 wait complete. 2023-10-07T09:48:05.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T09:48:05.240 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:05.239+0000 7fef0404a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:48:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.316+0000 7f8fc4fff700 1 Processor -- start 2023-10-07T09:48:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.316+0000 7f8fc4fff700 1 -- start start 2023-10-07T09:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.317+0000 7f8fc4fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fc00b07c0 0x7f8fc00ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.317+0000 7f8fc4fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc01578a0 con 0x7f8fc00b3e60 2023-10-07T09:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.317+0000 7f8fc4fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc00cc830 con 0x7f8fc00ce390 2023-10-07T09:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.317+0000 7f8fc4fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc0157ce0 con 0x7f8fc00b07c0 2023-10-07T09:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbe59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8fc00b3e60 0x7f8fc0158b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40346/0 (socket says 172.21.15.29:40346) 2023-10-07T09:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbe59c700 1 -- 172.21.15.29:0/3166582332 learned_addr learned my addr 172.21.15.29:0/3166582332 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbed9d700 1 --2- 172.21.15.29:0/3166582332 >> v2:172.21.15.29:3300/0 conn(0x7f8fc00b07c0 0x7f8fc00ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbd59a700 1 -- 172.21.15.29:0/3166582332 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447360323 0 0) 0x7f8fc00cc830 con 0x7f8fc00ce390 2023-10-07T09:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbd59a700 1 -- 172.21.15.29:0/3166582332 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa8003570 con 0x7f8fc00ce390 2023-10-07T09:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbed9d700 1 -- 172.21.15.29:0/3166582332 >> v1:172.21.15.144:6789/0 conn(0x7f8fc00ce390 legacy=0x7f8fc00b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbed9d700 1 -- 172.21.15.29:0/3166582332 >> v1:172.21.15.29:6789/0 conn(0x7f8fc00b3e60 legacy=0x7f8fc0158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbed9d700 1 -- 172.21.15.29:0/3166582332 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc00cee50 con 0x7f8fc00b07c0 2023-10-07T09:48:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.318+0000 7f8fbed9d700 1 --2- 172.21.15.29:0/3166582332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc00ce830 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009e60 tx=0x7f8fb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a710b408d6f71c83 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.319+0000 7f8fbd59a700 1 -- 172.21.15.29:0/3166582332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb800e070 con 0x7f8fc00b07c0 2023-10-07T09:48:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.319+0000 7f8fbd59a700 1 -- 172.21.15.29:0/3166582332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb800bb60 con 0x7f8fc00b07c0 2023-10-07T09:48:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.319+0000 7f8fbd59a700 1 -- 172.21.15.29:0/3166582332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8012430 con 0x7f8fc00b07c0 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.319+0000 7f8fc4fff700 1 -- 172.21.15.29:0/3166582332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 msgr2=0x7f8fc00ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.319+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/3166582332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc00ce830 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009e60 tx=0x7f8fb8009780 comp rx=0 tx=0).stop 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.320+0000 7f8fc4fff700 1 -- 172.21.15.29:0/3166582332 shutdown_connections 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.320+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/3166582332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc00ce830 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.320+0000 7f8fc4fff700 1 -- 172.21.15.29:0/3166582332 >> 172.21.15.29:0/3166582332 conn(0x7f8fc0067610 msgr2=0x7f8fc0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.320+0000 7f8fc4fff700 1 -- 172.21.15.29:0/3166582332 shutdown_connections 2023-10-07T09:48:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.320+0000 7f8fc4fff700 1 -- 172.21.15.29:0/3166582332 wait complete. 2023-10-07T09:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.321+0000 7f8fc4fff700 1 Processor -- start 2023-10-07T09:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.321+0000 7f8fc4fff700 1 -- start start 2023-10-07T09:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.322+0000 7f8fc4fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc0157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.322+0000 7f8fc4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fc00b3e60 0x7f8fc00c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.322+0000 7f8fc4fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fc00ce390 0x7f8fc00c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.322+0000 7f8fc4fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc0069510 con 0x7f8fc00b07c0 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.322+0000 7f8fc4fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00697f0 con 0x7f8fc00ce390 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.322+0000 7f8fc4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc0069ad0 con 0x7f8fc00b3e60 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbe59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc0157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbe59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc0157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54634/0 (socket says 172.21.15.29:54634) 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbed9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fc00ce390 0x7f8fc00c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbe59c700 1 -- 172.21.15.29:0/1491057483 learned_addr learned my addr 172.21.15.29:0/1491057483 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbdd9b700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fc00b3e60 0x7f8fc00c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbed9d700 1 -- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fc00b3e60 msgr2=0x7f8fc00c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbed9d700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fc00b3e60 0x7f8fc00c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbed9d700 1 -- 172.21.15.29:0/1491057483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 msgr2=0x7f8fc0157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbed9d700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc0157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbed9d700 1 -- 172.21.15.29:0/1491057483 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb8009a20 con 0x7f8fc00ce390 2023-10-07T09:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.323+0000 7f8fbe59c700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc0157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.324+0000 7f8fbed9d700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fc00ce390 0x7f8fc00c9e10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8016000 tx=0x7f8fb8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.324+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8027470 con 0x7f8fc00ce390 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.324+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb80275f0 con 0x7f8fc00ce390 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.324+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb80279f0 con 0x7f8fc00ce390 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.324+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc0069db0 con 0x7f8fc00ce390 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.324+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc00c7af0 con 0x7f8fc00ce390 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.325+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc004c3c0 con 0x7f8fc00ce390 2023-10-07T09:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.327+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f8fb8024070 con 0x7f8fc00ce390 2023-10-07T09:48:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.328+0000 7f8fae7fc700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 0x7f8fa0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.328+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f8fb80874b0 con 0x7f8fc00ce390 2023-10-07T09:48:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.328+0000 7f8fbe59c700 1 -- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 msgr2=0x7f8fa0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.328+0000 7f8fbe59c700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 0x7f8fa0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.332+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fb8054140 con 0x7f8fc00ce390 2023-10-07T09:48:05.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.529+0000 7f8fbe59c700 1 -- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 msgr2=0x7f8fa0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:05.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.529+0000 7f8fbe59c700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 0x7f8fa0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.538+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8fc0069930 con 0x7f8fc00ce390 2023-10-07T09:48:05.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.539+0000 7f8fae7fc700 1 -- 172.21.15.29:0/1491057483 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v119) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8fc0069930 con 0x7f8fc00ce390 2023-10-07T09:48:05.540 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 msgr2=0x7f8fa0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 0x7f8fa0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fc00ce390 msgr2=0x7f8fc00c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fc00ce390 0x7f8fc00c9e10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8016000 tx=0x7f8fb8019000 comp rx=0 tx=0).stop 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 shutdown_connections 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8fa0067170 0x7f8fa0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fc00ce390 0x7f8fc00c9e10 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fc00b07c0 0x7f8fc0157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 --2- 172.21.15.29:0/1491057483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fc00b3e60 0x7f8fc00c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 >> 172.21.15.29:0/1491057483 conn(0x7f8fc0067610 msgr2=0x7f8fc00cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:05.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 shutdown_connections 2023-10-07T09:48:05.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.541+0000 7f8fc4fff700 1 -- 172.21.15.29:0/1491057483 wait complete. 2023-10-07T09:48:05.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T09:48:05.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.760+0000 7f4dda7ff700 1 Processor -- start 2023-10-07T09:48:05.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.760+0000 7f4dda7ff700 1 -- start start 2023-10-07T09:48:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.761+0000 7f4dda7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415a960 0x7f4dd4158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.761+0000 7f4dda7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 0x7f4dd40ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.761+0000 7f4dda7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40acb40 con 0x7f4dd40ae6d0 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.761+0000 7f4dda7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40aeb50 con 0x7f4dd415ad30 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.761+0000 7f4dda7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd4158c60 con 0x7f4dd415a960 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 0x7f4dd40ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 0x7f4dd40ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54650/0 (socket says 172.21.15.29:54650) 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd3fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415a960 0x7f4dd4158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd37fe700 1 -- 172.21.15.29:0/1463027748 learned_addr learned my addr 172.21.15.29:0/1463027748 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd8d9c700 1 --1- 172.21.15.29:0/1463027748 >> v1:172.21.15.144:6789/0 conn(0x7f4dd40ae6d0 0x7f4dd40b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:40424/0 (socket says 172.21.15.29:40424) 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd2ffd700 1 -- 172.21.15.29:0/1463027748 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769957298 0 0) 0x7f4dd40acb40 con 0x7f4dd40ae6d0 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd2ffd700 1 -- 172.21.15.29:0/1463027748 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dbc003570 con 0x7f4dd40ae6d0 2023-10-07T09:48:05.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd3fff700 1 -- 172.21.15.29:0/1463027748 >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 msgr2=0x7f4dd40ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd3fff700 1 --2- 172.21.15.29:0/1463027748 >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 0x7f4dd40ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd3fff700 1 -- 172.21.15.29:0/1463027748 >> v1:172.21.15.144:6789/0 conn(0x7f4dd40ae6d0 legacy=0x7f4dd40b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd3fff700 1 -- 172.21.15.29:0/1463027748 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40abf30 con 0x7f4dd415a960 2023-10-07T09:48:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.762+0000 7f4dd37fe700 1 --2- 172.21.15.29:0/1463027748 >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 0x7f4dd40ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.763+0000 7f4dd3fff700 1 --2- 172.21.15.29:0/1463027748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415a960 0x7f4dd4158e20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40098e0 tx=0x7f4dc40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1978647c78391d44 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.763+0000 7f4dd2ffd700 1 -- 172.21.15.29:0/1463027748 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4011630 con 0x7f4dd415a960 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.763+0000 7f4dd2ffd700 1 -- 172.21.15.29:0/1463027748 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4011790 con 0x7f4dd415a960 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.763+0000 7f4dd2ffd700 1 -- 172.21.15.29:0/1463027748 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4011b30 con 0x7f4dd415a960 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.763+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1463027748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415a960 msgr2=0x7f4dd4158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1463027748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415a960 0x7f4dd4158e20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40098e0 tx=0x7f4dc40091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1463027748 shutdown_connections 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1463027748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415a960 0x7f4dd4158e20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1463027748 >> v2:172.21.15.29:3300/0 conn(0x7f4dd415ad30 0x7f4dd40ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1463027748 >> 172.21.15.29:0/1463027748 conn(0x7f4dd4067610 msgr2=0x7f4dd4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1463027748 shutdown_connections 2023-10-07T09:48:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.764+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1463027748 wait complete. 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.765+0000 7f4dda7ff700 1 Processor -- start 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.765+0000 7f4dda7ff700 1 -- start start 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.766+0000 7f4dda7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd40ae6d0 0x7f4dd4149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.766+0000 7f4dda7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 0x7f4dd414d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.766+0000 7f4dda7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415ad30 0x7f4dd40cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.766+0000 7f4dda7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd4149fe0 con 0x7f4dd415a960 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.766+0000 7f4dda7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd415cb40 con 0x7f4dd40ae6d0 2023-10-07T09:48:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.766+0000 7f4dda7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd415ce20 con 0x7f4dd415ad30 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 0x7f4dd414d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd3fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd40ae6d0 0x7f4dd4149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 0x7f4dd414d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54654/0 (socket says 172.21.15.29:54654) 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 -- 172.21.15.29:0/1923009541 learned_addr learned my addr 172.21.15.29:0/1923009541 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd8d9c700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415ad30 0x7f4dd40cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 -- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415ad30 msgr2=0x7f4dd40cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415ad30 0x7f4dd40cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 -- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd40ae6d0 msgr2=0x7f4dd4149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd40ae6d0 0x7f4dd4149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd37fe700 1 -- 172.21.15.29:0/1923009541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc80094f0 con 0x7f4dd415a960 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.767+0000 7f4dd8d9c700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415ad30 0x7f4dd40cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.768+0000 7f4dd37fe700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 0x7f4dd414d0f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800ec50 tx=0x7f4dc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.768+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800c850 con 0x7f4dd415a960 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.768+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc800c9b0 con 0x7f4dd415a960 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.768+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800cd30 con 0x7f4dd415a960 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.768+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4009420 con 0x7f4dd415a960 2023-10-07T09:48:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.768+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd40cb4d0 con 0x7f4dd415a960 2023-10-07T09:48:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.769+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd404c3c0 con 0x7f4dd415a960 2023-10-07T09:48:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.770+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f4dc8013680 con 0x7f4dd415a960 2023-10-07T09:48:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.772+0000 7f4dbbfff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 0x7f4dc0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.772+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f4dc8084e00 con 0x7f4dd415a960 2023-10-07T09:48:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.772+0000 7f4dd3fff700 1 -- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 msgr2=0x7f4dc0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.772+0000 7f4dd3fff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 0x7f4dc0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.776+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4dc8051ac0 con 0x7f4dd415a960 2023-10-07T09:48:05.841 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:05.840+0000 7f2be375c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:48:05.930 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:05.929+0000 7f2be375c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:48:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.973+0000 7f4dd3fff700 1 -- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 msgr2=0x7f4dc0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.973+0000 7f4dd3fff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 0x7f4dc0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:05.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.982+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4dd40abbe0 con 0x7f4dd415a960 2023-10-07T09:48:05.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.982+0000 7f4dbbfff700 1 -- 172.21.15.29:0/1923009541 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v119) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4dc80550e0 con 0x7f4dd415a960 2023-10-07T09:48:05.984 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T09:48:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 msgr2=0x7f4dc0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 0x7f4dc0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 msgr2=0x7f4dd414d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 0x7f4dd414d0f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800ec50 tx=0x7f4dc8009250 comp rx=0 tx=0).stop 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 shutdown_connections 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4dc0067170 0x7f4dc0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd415ad30 0x7f4dd40cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd40ae6d0 0x7f4dd4149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 --2- 172.21.15.29:0/1923009541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415a960 0x7f4dd414d0f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 >> 172.21.15.29:0/1923009541 conn(0x7f4dd4067610 msgr2=0x7f4dd406ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 shutdown_connections 2023-10-07T09:48:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:05.985+0000 7f4dda7ff700 1 -- 172.21.15.29:0/1923009541 wait complete. 2023-10-07T09:48:06.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:06.065 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.064+0000 7fef0404a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:48:06.083 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.082+0000 7f2be375c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:48:06.160 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.159+0000 7fef0404a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:48:06.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.167+0000 7f2be375c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:48:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.204+0000 7fbc683ff700 1 Processor -- start 2023-10-07T09:48:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.205+0000 7fbc683ff700 1 -- start start 2023-10-07T09:48:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc683ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc600ad370 0x7fbc600cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc683ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc683ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6014d790 con 0x7fbc60159ec0 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc683ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6015a880 con 0x7fbc600ad370 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc683ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600ae5d0 con 0x7fbc600b0410 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc6619b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc600ad370 0x7fbc600cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc6619b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc600ad370 0x7fbc600cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54658/0 (socket says 172.21.15.29:54658) 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc6619b700 1 -- 172.21.15.29:0/2192292896 learned_addr learned my addr 172.21.15.29:0/2192292896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.206+0000 7fbc6699c700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc65199700 1 -- 172.21.15.29:0/2192292896 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956785884 0 0) 0x7fbc6014d790 con 0x7fbc60159ec0 2023-10-07T09:48:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc65199700 1 -- 172.21.15.29:0/2192292896 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c003570 con 0x7fbc60159ec0 2023-10-07T09:48:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc6619b700 1 -- 172.21.15.29:0/2192292896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 msgr2=0x7fbc6015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc6619b700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc6619b700 1 -- 172.21.15.29:0/2192292896 >> v1:172.21.15.144:6789/0 conn(0x7fbc60159ec0 legacy=0x7fbc601568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc6619b700 1 -- 172.21.15.29:0/2192292896 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600ad740 con 0x7fbc600ad370 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc6699c700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc6619b700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc600ad370 0x7fbc600cc870 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0099e0 tx=0x7fbc5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b41cfd30e7dad server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.207+0000 7fbc65199700 1 -- 172.21.15.29:0/2192292896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c00b640 con 0x7fbc600ad370 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc65199700 1 -- 172.21.15.29:0/2192292896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c00b7a0 con 0x7fbc600ad370 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc65199700 1 -- 172.21.15.29:0/2192292896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c00bb40 con 0x7fbc600ad370 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 -- 172.21.15.29:0/2192292896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc600ad370 msgr2=0x7fbc600cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc600ad370 0x7fbc600cc870 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0099e0 tx=0x7fbc5c009250 comp rx=0 tx=0).stop 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 -- 172.21.15.29:0/2192292896 shutdown_connections 2023-10-07T09:48:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 --2- 172.21.15.29:0/2192292896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc600ad370 0x7fbc600cc870 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 -- 172.21.15.29:0/2192292896 >> 172.21.15.29:0/2192292896 conn(0x7fbc600680a0 msgr2=0x7fbc600cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 -- 172.21.15.29:0/2192292896 shutdown_connections 2023-10-07T09:48:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.208+0000 7fbc683ff700 1 -- 172.21.15.29:0/2192292896 wait complete. 2023-10-07T09:48:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.211+0000 7fbc683ff700 1 Processor -- start 2023-10-07T09:48:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.211+0000 7fbc683ff700 1 -- start start 2023-10-07T09:48:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.212+0000 7fbc683ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc600ad370 0x7fbc60168f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc683ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6014f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc683ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 0x7fbc6014fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc683ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60155cb0 con 0x7fbc60159ec0 2023-10-07T09:48:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc683ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60169460 con 0x7fbc600ad370 2023-10-07T09:48:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc683ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60155380 con 0x7fbc600b0410 2023-10-07T09:48:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6619b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc600ad370 0x7fbc60168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 0x7fbc6014fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 0x7fbc6014fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54668/0 (socket says 172.21.15.29:54668) 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 -- 172.21.15.29:0/1061566751 learned_addr learned my addr 172.21.15.29:0/1061566751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6619b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc600ad370 0x7fbc60168f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:33074/0 (socket says 172.21.15.29:33074) 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6599a700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6014f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 -- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 msgr2=0x7fbc6014f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6014f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 -- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc600ad370 msgr2=0x7fbc60168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc600ad370 0x7fbc60168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.213+0000 7fbc6699c700 1 -- 172.21.15.29:0/1061566751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58009a20 con 0x7fbc60159ec0 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.214+0000 7fbc6699c700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 0x7fbc6014fd60 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c0f0 tx=0x7fbc5800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.214+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5800cc30 con 0x7fbc60159ec0 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.214+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5800f040 con 0x7fbc60159ec0 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.214+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c0094f0 con 0x7fbc60159ec0 2023-10-07T09:48:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.214+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58014480 con 0x7fbc60159ec0 2023-10-07T09:48:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.214+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60155940 con 0x7fbc60159ec0 2023-10-07T09:48:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.215+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60155e60 con 0x7fbc60159ec0 2023-10-07T09:48:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.216+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fbc58003020 con 0x7fbc60159ec0 2023-10-07T09:48:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.217+0000 7fbc567fc700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 0x7fbc44069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.218+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fbc58085670 con 0x7fbc60159ec0 2023-10-07T09:48:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.218+0000 7fbc6619b700 1 -- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 msgr2=0x7fbc44069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.218+0000 7fbc6619b700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 0x7fbc44069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.222+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc58052150 con 0x7fbc60159ec0 2023-10-07T09:48:06.329 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.328+0000 7fef0404a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:48:06.387 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.386+0000 7f2be375c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:48:06.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.419+0000 7fbc6619b700 1 -- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 msgr2=0x7fbc44069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:06.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.419+0000 7fbc6619b700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 0x7fbc44069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:06.437 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.435+0000 7fef0404a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:48:06.487 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.485+0000 7f2be375c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:48:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.493+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6004c3c0 con 0x7fbc60159ec0 2023-10-07T09:48:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.493+0000 7fbc567fc700 1 -- 172.21.15.29:0/1061566751 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fbc58055770 con 0x7fbc60159ec0 2023-10-07T09:48:06.495 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:06.495 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:47:17.881832+0000","modified":"2023-10-07T09:47:21.398903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/581264510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":581264510},{"type":"v1","addr":"172.21.15.29:6835","nonce":581264510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-10-07T09:48:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 msgr2=0x7fbc44069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 0x7fbc44069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 msgr2=0x7fbc6014fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 0x7fbc6014fd60 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c0f0 tx=0x7fbc5800b040 comp rx=0 tx=0).stop 2023-10-07T09:48:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 shutdown_connections 2023-10-07T09:48:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbc44067170 0x7fbc44069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc60159ec0 0x7fbc6014fd60 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc600ad370 0x7fbc60168f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 --2- 172.21.15.29:0/1061566751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc600b0410 0x7fbc6014f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 >> 172.21.15.29:0/1061566751 conn(0x7fbc600680a0 msgr2=0x7fbc6006e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 shutdown_connections 2023-10-07T09:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.496+0000 7fbc683ff700 1 -- 172.21.15.29:0/1061566751 wait complete. 2023-10-07T09:48:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-07T09:48:06.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:06.581 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.579+0000 7f2be375c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:48:06.661 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.660+0000 7fef0404a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:48:06.676 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.675+0000 7f2be375c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:48:06.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.716+0000 7f55d81ff700 1 Processor -- start 2023-10-07T09:48:06.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.716+0000 7f55d81ff700 1 -- start start 2023-10-07T09:48:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d81ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d81ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d00b0f60 con 0x7f55d00cc040 2023-10-07T09:48:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d81ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d00b0070 con 0x7f55d00ca280 2023-10-07T09:48:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d81ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d00b2cb0 con 0x7f55d00cccc0 2023-10-07T09:48:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d5f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f55d00cc040 0x7f55d006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40360/0 (socket says 172.21.15.29:40360) 2023-10-07T09:48:06.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d5f9b700 1 -- 172.21.15.29:0/1282893742 learned_addr learned my addr 172.21.15.29:0/1282893742 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d579a700 1 --2- 172.21.15.29:0/1282893742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.718+0000 7f55d4f99700 1 -- 172.21.15.29:0/1282893742 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149402691 0 0) 0x7f55d00b0f60 con 0x7f55d00cc040 2023-10-07T09:48:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d4f99700 1 -- 172.21.15.29:0/1282893742 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55bc003570 con 0x7f55d00cc040 2023-10-07T09:48:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d579a700 1 -- 172.21.15.29:0/1282893742 >> v1:172.21.15.144:6789/0 conn(0x7f55d00ca280 legacy=0x7f55d00b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d4f99700 1 -- 172.21.15.29:0/1282893742 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687637514 0 0) 0x7f55d00b0070 con 0x7f55d00ca280 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d579a700 1 -- 172.21.15.29:0/1282893742 >> v1:172.21.15.29:6789/0 conn(0x7f55d00cc040 legacy=0x7f55d006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d579a700 1 -- 172.21.15.29:0/1282893742 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d00b0350 con 0x7f55d00cccc0 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d579a700 1 --2- 172.21.15.29:0/1282893742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d006ec00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009910 tx=0x7f55c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0f016e9e123d4dc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d4f99700 1 -- 172.21.15.29:0/1282893742 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c000b680 con 0x7f55d00cccc0 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d4f99700 1 -- 172.21.15.29:0/1282893742 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c000b7e0 con 0x7f55d00cccc0 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.719+0000 7f55d4f99700 1 -- 172.21.15.29:0/1282893742 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c000bba0 con 0x7f55d00cccc0 2023-10-07T09:48:06.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 -- 172.21.15.29:0/1282893742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 msgr2=0x7f55d006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1282893742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d006ec00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009910 tx=0x7f55c00091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 -- 172.21.15.29:0/1282893742 shutdown_connections 2023-10-07T09:48:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1282893742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d006ec00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 -- 172.21.15.29:0/1282893742 >> 172.21.15.29:0/1282893742 conn(0x7f55d0069190 msgr2=0x7f55d006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 -- 172.21.15.29:0/1282893742 shutdown_connections 2023-10-07T09:48:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.720+0000 7f55d81ff700 1 -- 172.21.15.29:0/1282893742 wait complete. 2023-10-07T09:48:06.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.722+0000 7f55d81ff700 1 Processor -- start 2023-10-07T09:48:06.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.722+0000 7f55d81ff700 1 -- start start 2023-10-07T09:48:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d81ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 0x7f55d0153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d81ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 0x7f55d015a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d81ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d015aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d81ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d015c990 con 0x7f55d00cc040 2023-10-07T09:48:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d81ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d0150e60 con 0x7f55d00ca280 2023-10-07T09:48:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d81ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d015b110 con 0x7f55d00cccc0 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d579a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 0x7f55d015a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d5f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 0x7f55d0153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d579a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 0x7f55d015a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54684/0 (socket says 172.21.15.29:54684) 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d579a700 1 -- 172.21.15.29:0/1620686988 learned_addr learned my addr 172.21.15.29:0/1620686988 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d5f9b700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 0x7f55d0153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:33084/0 (socket says 172.21.15.29:33084) 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.723+0000 7f55d679c700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d015aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d579a700 1 -- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 msgr2=0x7f55d015aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d579a700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d015aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d579a700 1 -- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 msgr2=0x7f55d0153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d579a700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 0x7f55d0153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d579a700 1 -- 172.21.15.29:0/1620686988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c0009420 con 0x7f55d00cc040 2023-10-07T09:48:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d5f9b700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 0x7f55d0153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d579a700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 0x7f55d015a5a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f55d006c520 tx=0x7f55c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c000bb60 con 0x7f55d00cc040 2023-10-07T09:48:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c0000c90 con 0x7f55d00cc040 2023-10-07T09:48:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0027680 con 0x7f55d00cc040 2023-10-07T09:48:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.724+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d015b3f0 con 0x7f55d00cc040 2023-10-07T09:48:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.725+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d00c8370 con 0x7f55d00cc040 2023-10-07T09:48:06.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.726+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d00c85b0 con 0x7f55d00cc040 2023-10-07T09:48:06.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.727+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f55c001a430 con 0x7f55d00cc040 2023-10-07T09:48:06.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.727+0000 7f55c5ffb700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 0x7f55b4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:06.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.728+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f55c0086ab0 con 0x7f55d00cc040 2023-10-07T09:48:06.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.728+0000 7f55d5f9b700 1 -- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 msgr2=0x7f55b4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:06.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.728+0000 7f55d5f9b700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 0x7f55b4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:06.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.732+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55d00c85b0 con 0x7f55d00cc040 2023-10-07T09:48:06.765 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.764+0000 7fef0404a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:48:06.864 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.863+0000 7fef0404a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:48:06.865 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.863+0000 7f2be375c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:48:06.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.929+0000 7f55d5f9b700 1 -- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 msgr2=0x7f55b4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:06.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.929+0000 7f55d5f9b700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 0x7f55b4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:06.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.936+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55d00c85b0 con 0x7f55d00cc040 2023-10-07T09:48:06.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.937+0000 7f55c5ffb700 1 -- 172.21.15.29:0/1620686988 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16405 (secure 0 0 0) 0x7f55d00c85b0 con 0x7f55d00cc040 2023-10-07T09:48:06.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:06.939 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":119,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:47:34.318934+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:47:15.663387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-07T09:47:16.857437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3818341042":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/1072858706":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/378227010":"2023-10-08T09:47:34.318899+0000","172.21.15.144:6836/114714":"2023-10-08T09:47:34.318899+0000","172.21.15.144:6834/114714":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/2477548044":"2023-10-08T09:47:34.318899+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 msgr2=0x7f55b4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 0x7f55b4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 msgr2=0x7f55d015a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 0x7f55d015a5a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f55d006c520 tx=0x7f55c0017000 comp rx=0 tx=0).stop 2023-10-07T09:48:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 shutdown_connections 2023-10-07T09:48:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f55b4067170 0x7f55b4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55d00cccc0 0x7f55d015aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55d00ca280 0x7f55d0153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 --2- 172.21.15.29:0/1620686988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d00cc040 0x7f55d015a5a0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:06.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.939+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 >> 172.21.15.29:0/1620686988 conn(0x7f55d0069190 msgr2=0x7f55d014ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:06.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.940+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 shutdown_connections 2023-10-07T09:48:06.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:06.940+0000 7f55d81ff700 1 -- 172.21.15.29:0/1620686988 wait complete. 2023-10-07T09:48:06.959 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T09:48:06.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:06.962 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:06.959+0000 7f2be375c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:48:06.968 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:06.964+0000 7fef0404a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:48:07.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.160+0000 7f76f59ff700 1 Processor -- start 2023-10-07T09:48:07.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.160+0000 7f76f59ff700 1 -- start start 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.161+0000 7f76f59ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76f01576d0 0x7f76f00cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.161+0000 7f76f59ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00cc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76f59ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f00cb330 con 0x7f76f0157300 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76f59ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f006f980 con 0x7f76f01576d0 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76f59ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f00cbe90 con 0x7f76f00b2a90 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76eeffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76f0157300 0x7f76f00b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40370/0 (socket says 172.21.15.29:40370) 2023-10-07T09:48:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76eeffd700 1 -- 172.21.15.29:0/4109799581 learned_addr learned my addr 172.21.15.29:0/4109799581 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76ee7fc700 1 --2- 172.21.15.29:0/4109799581 >> v2:172.21.15.144:3300/0 conn(0x7f76f01576d0 0x7f76f00cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76ef7fe700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76edffb700 1 -- 172.21.15.29:0/4109799581 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848411420 0 0) 0x7f76f00cb330 con 0x7f76f0157300 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.162+0000 7f76edffb700 1 -- 172.21.15.29:0/4109799581 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d8003570 con 0x7f76f0157300 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76ee7fc700 1 -- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 msgr2=0x7f76f00cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76ee7fc700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76ee7fc700 1 -- 172.21.15.29:0/4109799581 >> v1:172.21.15.29:6789/0 conn(0x7f76f0157300 legacy=0x7f76f00b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76ee7fc700 1 -- 172.21.15.29:0/4109799581 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00cb560 con 0x7f76f01576d0 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76ef7fe700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76ee7fc700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f01576d0 0x7f76f00cc320 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f76e40098e0 tx=0x7f76e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fce026b7ccb0e08 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:07.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76edffb700 1 -- 172.21.15.29:0/4109799581 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4011630 con 0x7f76f01576d0 2023-10-07T09:48:07.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76edffb700 1 -- 172.21.15.29:0/4109799581 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e4011790 con 0x7f76f01576d0 2023-10-07T09:48:07.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.163+0000 7f76edffb700 1 -- 172.21.15.29:0/4109799581 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4011b30 con 0x7f76f01576d0 2023-10-07T09:48:07.166 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:07.163+0000 7fef0404a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:48:07.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 -- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f01576d0 msgr2=0x7f76f00cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f01576d0 0x7f76f00cc320 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f76e40098e0 tx=0x7f76e40091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 -- 172.21.15.29:0/4109799581 shutdown_connections 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 --2- 172.21.15.29:0/4109799581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f01576d0 0x7f76f00cc320 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 -- 172.21.15.29:0/4109799581 >> 172.21.15.29:0/4109799581 conn(0x7f76f0069190 msgr2=0x7f76f006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 -- 172.21.15.29:0/4109799581 shutdown_connections 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.164+0000 7f76f59ff700 1 -- 172.21.15.29:0/4109799581 wait complete. 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.166+0000 7f76f59ff700 1 Processor -- start 2023-10-07T09:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.166+0000 7f76f59ff700 1 -- start start 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76f59ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76f59ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 0x7f76f016a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76f59ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f00aa6f0 0x7f76f01523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76f59ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f00a9970 con 0x7f76f0157300 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76f59ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f00a9c10 con 0x7f76f00aa6f0 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76f59ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f00a9ef0 con 0x7f76f00b2a90 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 0x7f76f016a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 0x7f76f016a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54688/0 (socket says 172.21.15.29:54688) 2023-10-07T09:48:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 -- 172.21.15.29:0/768133276 learned_addr learned my addr 172.21.15.29:0/768133276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76eeffd700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ef7fe700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f00aa6f0 0x7f76f01523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 -- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 msgr2=0x7f76f00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 -- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f00aa6f0 msgr2=0x7f76f01523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.167+0000 7f76ee7fc700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f00aa6f0 0x7f76f01523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76ee7fc700 1 -- 172.21.15.29:0/768133276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8009a20 con 0x7f76f0157300 2023-10-07T09:48:07.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76ef7fe700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f00aa6f0 0x7f76f01523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:07.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76ee7fc700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 0x7f76f016a2d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f76f006c520 tx=0x7f76e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:07.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4011900 con 0x7f76f0157300 2023-10-07T09:48:07.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e4011a60 con 0x7f76f0157300 2023-10-07T09:48:07.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4025670 con 0x7f76f0157300 2023-10-07T09:48:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e4009420 con 0x7f76f0157300 2023-10-07T09:48:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.168+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f00afb00 con 0x7f76f0157300 2023-10-07T09:48:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.170+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f00aa410 con 0x7f76f0157300 2023-10-07T09:48:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.170+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f76e4023070 con 0x7f76f0157300 2023-10-07T09:48:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.171+0000 7f76d6ffd700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 0x7f76dc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.171+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f76e4085fe0 con 0x7f76f0157300 2023-10-07T09:48:07.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.172+0000 7f76eeffd700 1 -- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 msgr2=0x7f76dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:07.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.172+0000 7f76eeffd700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 0x7f76dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:07.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.175+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76f00aa410 con 0x7f76f0157300 2023-10-07T09:48:07.269 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:07.268+0000 7fef0404a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:48:07.307 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:07.305+0000 7f2be375c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:48:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.373+0000 7f76eeffd700 1 -- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 msgr2=0x7f76dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.373+0000 7f76eeffd700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 0x7f76dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:07.395 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:07.394+0000 7f2be375c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.426+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76f00aa410 con 0x7f76f0157300 2023-10-07T09:48:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.427+0000 7f76d6ffd700 1 -- 172.21.15.29:0/768133276 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f76f00aa410 con 0x7f76f0157300 2023-10-07T09:48:07.428 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:07.429 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:47:17.881832+0000","modified":"2023-10-07T09:47:21.398903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/581264510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":581264510},{"type":"v1","addr":"172.21.15.29:6835","nonce":581264510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-10-07T09:48:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 msgr2=0x7f76dc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 0x7f76dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 msgr2=0x7f76f016a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 0x7f76f016a2d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f76f006c520 tx=0x7f76e4017000 comp rx=0 tx=0).stop 2023-10-07T09:48:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 shutdown_connections 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f76dc067170 0x7f76dc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76f00aa6f0 0x7f76f01523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76f00b2a90 0x7f76f00af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 --2- 172.21.15.29:0/768133276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76f0157300 0x7f76f016a2d0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 >> 172.21.15.29:0/768133276 conn(0x7f76f0069190 msgr2=0x7f76f00b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 shutdown_connections 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.430+0000 7f76f59ff700 1 -- 172.21.15.29:0/768133276 wait complete. 2023-10-07T09:48:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-07T09:48:07.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:07.628 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:07.627+0000 7fef0404a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:48:07.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.649+0000 7faecadff700 1 Processor -- start 2023-10-07T09:48:07.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.650+0000 7faecadff700 1 -- start start 2023-10-07T09:48:07.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faecadff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faecadff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faec40ae8b0 0x7faec40b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faecadff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faec40b1840 0x7faec40ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faecadff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec40b1cc0 con 0x7faec40ae8b0 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faecadff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec4158ad0 con 0x7faec40b1840 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faecadff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec415b140 con 0x7faec4157290 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faec3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faec40ae8b0 0x7faec40b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faec3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faec40ae8b0 0x7faec40b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54698/0 (socket says 172.21.15.29:54698) 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faec3fff700 1 -- 172.21.15.29:0/2246947552 learned_addr learned my addr 172.21.15.29:0/2246947552 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faec939c700 1 --2- 172.21.15.29:0/2246947552 >> v2:172.21.15.144:3300/0 conn(0x7faec40b1840 0x7faec40ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.651+0000 7faec8b9b700 1 --2- 172.21.15.29:0/2246947552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec3fff700 1 -- 172.21.15.29:0/2246947552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 msgr2=0x7faec40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec3fff700 1 --2- 172.21.15.29:0/2246947552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec3fff700 1 -- 172.21.15.29:0/2246947552 >> v2:172.21.15.144:3300/0 conn(0x7faec40b1840 msgr2=0x7faec40ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec3fff700 1 --2- 172.21.15.29:0/2246947552 >> v2:172.21.15.144:3300/0 conn(0x7faec40b1840 0x7faec40ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec3fff700 1 -- 172.21.15.29:0/2246947552 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec415b420 con 0x7faec40ae8b0 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec939c700 1 --2- 172.21.15.29:0/2246947552 >> v2:172.21.15.144:3300/0 conn(0x7faec40b1840 0x7faec40ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec3fff700 1 --2- 172.21.15.29:0/2246947552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40ae8b0 0x7faec40b1300 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7faec40b2220 tx=0x7faeb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2e9a86b28f44d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec37fe700 1 -- 172.21.15.29:0/2246947552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000b640 con 0x7faec40ae8b0 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.652+0000 7faec37fe700 1 -- 172.21.15.29:0/2246947552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb000b7a0 con 0x7faec40ae8b0 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faec37fe700 1 -- 172.21.15.29:0/2246947552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000bb60 con 0x7faec40ae8b0 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 -- 172.21.15.29:0/2246947552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40ae8b0 msgr2=0x7faec40b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 --2- 172.21.15.29:0/2246947552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40ae8b0 0x7faec40b1300 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7faec40b2220 tx=0x7faeb0009250 comp rx=0 tx=0).stop 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 -- 172.21.15.29:0/2246947552 shutdown_connections 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 --2- 172.21.15.29:0/2246947552 >> v2:172.21.15.144:3300/0 conn(0x7faec40b1840 0x7faec40ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 --2- 172.21.15.29:0/2246947552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 --2- 172.21.15.29:0/2246947552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40ae8b0 0x7faec40b1300 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 -- 172.21.15.29:0/2246947552 >> 172.21.15.29:0/2246947552 conn(0x7faec40680a0 msgr2=0x7faec40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 -- 172.21.15.29:0/2246947552 shutdown_connections 2023-10-07T09:48:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.653+0000 7faecadff700 1 -- 172.21.15.29:0/2246947552 wait complete. 2023-10-07T09:48:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.656+0000 7faecadff700 1 Processor -- start 2023-10-07T09:48:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.656+0000 7faecadff700 1 -- start start 2023-10-07T09:48:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.657+0000 7faecadff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 0x7faec4138270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.657+0000 7faecadff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec4159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faecadff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faec415a2d0 0x7faec406e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faecadff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec415a770 con 0x7faec40b1840 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faecadff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec4154c00 con 0x7faec415a2d0 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faecadff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec4154310 con 0x7faec4157290 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 0x7faec4138270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 0x7faec4138270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54704/0 (socket says 172.21.15.29:54704) 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 -- 172.21.15.29:0/871506142 learned_addr learned my addr 172.21.15.29:0/871506142 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec3fff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec4159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec939c700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faec415a2d0 0x7faec406e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 -- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 msgr2=0x7faec4159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec4159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 -- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faec415a2d0 msgr2=0x7faec406e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faec415a2d0 0x7faec406e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 -- 172.21.15.29:0/871506142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc009a20 con 0x7faec40b1840 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.658+0000 7faec8b9b700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 0x7faec4138270 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7faeb800e950 tx=0x7faeb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.659+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8013680 con 0x7faec40b1840 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.659+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb80137e0 con 0x7faec40b1840 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.659+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb00094f0 con 0x7faec40b1840 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.659+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8013ba0 con 0x7faec40b1840 2023-10-07T09:48:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.659+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec41545b0 con 0x7faec40b1840 2023-10-07T09:48:07.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.660+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec404c3c0 con 0x7faec40b1840 2023-10-07T09:48:07.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.661+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7faeb800c490 con 0x7faec40b1840 2023-10-07T09:48:07.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.662+0000 7faec0ff9700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 0x7faea8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:07.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.662+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7faeb8084c10 con 0x7faec40b1840 2023-10-07T09:48:07.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.662+0000 7faec3fff700 1 -- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 msgr2=0x7faea8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:07.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.663+0000 7faec3fff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 0x7faea8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:07.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.666+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faeb8051990 con 0x7faec40b1840 2023-10-07T09:48:07.728 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:07.727+0000 7fef0404a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.864+0000 7faec3fff700 1 -- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 msgr2=0x7faea8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.864+0000 7faec3fff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 0x7faea8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.919+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faec415b700 con 0x7faec40b1840 2023-10-07T09:48:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.919+0000 7faec0ff9700 1 -- 172.21.15.29:0/871506142 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4599 (secure 0 0 0) 0x7faeb8054fb0 con 0x7faec40b1840 2023-10-07T09:48:07.920 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:07.920 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:47:17.881832+0000","modified":"2023-10-07T09:47:21.398903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7677},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7677":{"gid":7677,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/581264510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":581264510},{"type":"v1","addr":"172.21.15.29:6835","nonce":581264510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-10-07T09:48:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 msgr2=0x7faea8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 0x7faea8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 msgr2=0x7faec4138270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 0x7faec4138270 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7faeb800e950 tx=0x7faeb80091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 shutdown_connections 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faea8067170 0x7faea8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faec415a2d0 0x7faec406e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faec40b1840 0x7faec4138270 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 --2- 172.21.15.29:0/871506142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faec4157290 0x7faec4159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 >> 172.21.15.29:0/871506142 conn(0x7faec40680a0 msgr2=0x7faec4069c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 shutdown_connections 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:07.922+0000 7faecadff700 1 -- 172.21.15.29:0/871506142 wait complete. 2023-10-07T09:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-07T09:48:07.937 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:07.936+0000 7f2be375c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:48:07.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:08.126 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:08.125+0000 7f2be375c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:48:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.139+0000 7f35337ff700 1 Processor -- start 2023-10-07T09:48:08.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.139+0000 7f35337ff700 1 -- start start 2023-10-07T09:48:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.140+0000 7f35337ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f352c157290 0x7f352c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.140+0000 7f35337ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f352c0b1a10 0x7f352c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.140+0000 7f35337ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c158ad0 con 0x7f352c0ae8b0 2023-10-07T09:48:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.140+0000 7f35337ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f352c15b140 con 0x7f352c157290 2023-10-07T09:48:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.140+0000 7f35337ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f352c15b420 con 0x7f352c0b1a10 2023-10-07T09:48:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f352c157290 0x7f352c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f352c157290 0x7f352c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54708/0 (socket says 172.21.15.29:54708) 2023-10-07T09:48:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 -- 172.21.15.29:0/2997229939 learned_addr learned my addr 172.21.15.29:0/2997229939 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f3531d9c700 1 --2- 172.21.15.29:0/2997229939 >> v2:172.21.15.144:3300/0 conn(0x7f352c0b1a10 0x7f352c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 -- 172.21.15.29:0/2997229939 >> v2:172.21.15.144:3300/0 conn(0x7f352c0b1a10 msgr2=0x7f352c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 --2- 172.21.15.29:0/2997229939 >> v2:172.21.15.144:3300/0 conn(0x7f352c0b1a10 0x7f352c157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 -- 172.21.15.29:0/2997229939 >> v1:172.21.15.144:6789/0 conn(0x7f352c0ae8b0 legacy=0x7f352c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.141+0000 7f353159b700 1 -- 172.21.15.29:0/2997229939 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c15b700 con 0x7f352c157290 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f353159b700 1 --2- 172.21.15.29:0/2997229939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c157290 0x7f352c0cc870 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f352c0cbff0 tx=0x7f3528009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3da10c4139fea54 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f3523fff700 1 -- 172.21.15.29:0/2997229939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528009e70 con 0x7f352c157290 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f3523fff700 1 -- 172.21.15.29:0/2997229939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352800b860 con 0x7f352c157290 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f3523fff700 1 -- 172.21.15.29:0/2997229939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352800bc00 con 0x7f352c157290 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f35337ff700 1 -- 172.21.15.29:0/2997229939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c157290 msgr2=0x7f352c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f35337ff700 1 --2- 172.21.15.29:0/2997229939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c157290 0x7f352c0cc870 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f352c0cbff0 tx=0x7f3528009250 comp rx=0 tx=0).stop 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f35337ff700 1 -- 172.21.15.29:0/2997229939 shutdown_connections 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f35337ff700 1 --2- 172.21.15.29:0/2997229939 >> v2:172.21.15.144:3300/0 conn(0x7f352c0b1a10 0x7f352c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f35337ff700 1 --2- 172.21.15.29:0/2997229939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c157290 0x7f352c0cc870 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.142+0000 7f35337ff700 1 -- 172.21.15.29:0/2997229939 >> 172.21.15.29:0/2997229939 conn(0x7f352c0680a0 msgr2=0x7f352c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.143+0000 7f35337ff700 1 -- 172.21.15.29:0/2997229939 shutdown_connections 2023-10-07T09:48:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.143+0000 7f35337ff700 1 -- 172.21.15.29:0/2997229939 wait complete. 2023-10-07T09:48:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.144+0000 7f35337ff700 1 Processor -- start 2023-10-07T09:48:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.144+0000 7f35337ff700 1 -- start start 2023-10-07T09:48:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f35337ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f352c0b1a10 0x7f352c159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f35337ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f352c157290 0x7f352c14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f35337ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 0x7f352c1504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f35337ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c15e890 con 0x7f352c150b00 2023-10-07T09:48:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f35337ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c16d890 con 0x7f352c0b1a10 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f35337ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352c1535c0 con 0x7f352c157290 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f3531d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 0x7f352c1504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.145+0000 7f353159b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f352c0b1a10 0x7f352c159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f3531d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 0x7f352c1504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54718/0 (socket says 172.21.15.29:54718) 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f3531d9c700 1 -- 172.21.15.29:0/3042157452 learned_addr learned my addr 172.21.15.29:0/3042157452 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f3530d9a700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f352c157290 0x7f352c14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f353159b700 1 -- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f352c157290 msgr2=0x7f352c14ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f353159b700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f352c157290 0x7f352c14ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f353159b700 1 -- 172.21.15.29:0/3042157452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 msgr2=0x7f352c1504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f353159b700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 0x7f352c1504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f353159b700 1 -- 172.21.15.29:0/3042157452 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35280094f0 con 0x7f352c0b1a10 2023-10-07T09:48:08.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f3531d9c700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 0x7f352c1504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f353159b700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f352c0b1a10 0x7f352c159dd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f352c0c8d90 tx=0x7f3528017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.146+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352800bbe0 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.147+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528000c90 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.147+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528011650 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.147+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c16dd20 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.147+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c16d240 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.148+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352c04c3c0 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.149+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f3528011a40 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.149+0000 7f35217fa700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 0x7f3510069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.150+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f3528086240 con 0x7f352c0b1a10 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.150+0000 7f3530d9a700 1 -- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 msgr2=0x7f3510069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:08.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.150+0000 7f3530d9a700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 0x7f3510069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.154+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3528052f00 con 0x7f352c0b1a10 2023-10-07T09:48:08.306 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:08.305+0000 7fef0404a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:48:08.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.351+0000 7f3530d9a700 1 -- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 msgr2=0x7f3510069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:08.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.351+0000 7f3530d9a700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 0x7f3510069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.365+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f352c15b9e0 con 0x7f352c0b1a10 2023-10-07T09:48:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.366+0000 7f35217fa700 1 -- 172.21.15.29:0/3042157452 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16517 (secure 0 0 0) 0x7f3528056520 con 0x7f352c0b1a10 2023-10-07T09:48:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:08.369 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":120,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:48:08.237198+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:47:15.663387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-07T09:47:16.857437+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":120,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3818341042":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/1072858706":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/378227010":"2023-10-08T09:47:34.318899+0000","172.21.15.144:6836/114714":"2023-10-08T09:47:34.318899+0000","172.21.15.144:6834/114714":"2023-10-08T09:47:34.318899+0000","172.21.15.144:0/2477548044":"2023-10-08T09:47:34.318899+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 msgr2=0x7f3510069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 0x7f3510069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f352c0b1a10 msgr2=0x7f352c159dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f352c0b1a10 0x7f352c159dd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f352c0c8d90 tx=0x7f3528017000 comp rx=0 tx=0).stop 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 shutdown_connections 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3510067170 0x7f3510069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f352c150b00 0x7f352c1504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f352c0b1a10 0x7f352c159dd0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 --2- 172.21.15.29:0/3042157452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f352c157290 0x7f352c14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 >> 172.21.15.29:0/3042157452 conn(0x7f352c0680a0 msgr2=0x7f352c152ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 shutdown_connections 2023-10-07T09:48:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.369+0000 7f35337ff700 1 -- 172.21.15.29:0/3042157452 wait complete. 2023-10-07T09:48:08.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T09:48:08.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:08.503+0000 7fef0404a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:48:08.509 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:08.508+0000 7f2be375c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.592+0000 7f5dc79ff700 1 Processor -- start 2023-10-07T09:48:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.592+0000 7f5dc79ff700 1 -- start start 2023-10-07T09:48:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.594+0000 7f5dc79ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 0x7f5dc0156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.594+0000 7f5dc79ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0159670 con 0x7f5dc01533f0 2023-10-07T09:48:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.594+0000 7f5dc79ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc015a1f0 con 0x7f5dc01537c0 2023-10-07T09:48:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.594+0000 7f5dc79ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc015a680 con 0x7f5dc00b04b0 2023-10-07T09:48:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.594+0000 7f5dc579b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5dc01533f0 0x7f5dc00afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40372/0 (socket says 172.21.15.29:40372) 2023-10-07T09:48:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.594+0000 7f5dc579b700 1 -- 172.21.15.29:0/3997857364 learned_addr learned my addr 172.21.15.29:0/3997857364 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858982021 0 0) 0x7f5dc0159670 con 0x7f5dc01533f0 2023-10-07T09:48:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac003570 con 0x7f5dc01533f0 2023-10-07T09:48:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5dc4f9a700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 0x7f5dc0156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769283877 0 0) 0x7f5dac003570 con 0x7f5dc01533f0 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 msgr2=0x7f5dc0156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 0x7f5dc0156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 >> v1:172.21.15.144:6789/0 conn(0x7f5dc01537c0 legacy=0x7f5dc0157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc01598c0 con 0x7f5dc01533f0 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f5dbc001470 con 0x7f5dc01533f0 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 >> v1:172.21.15.29:6789/0 conn(0x7f5dc01533f0 legacy=0x7f5dc00afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dac003a90 0x7f5dac003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac004810 0x7f5dac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.595+0000 7f5db7fff700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 0x7f5dac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0054f0 con 0x7f5dac003a90 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac005660 con 0x7f5dac004810 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac00bfa0 con 0x7f5dc00b04b0 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc579b700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dac003a90 0x7f5dac003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc5f9c700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 0x7f5dac00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc4f9a700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac004810 0x7f5dac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc579b700 1 -- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 msgr2=0x7f5dac00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc579b700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc00b04b0 0x7f5dac00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc579b700 1 -- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac004810 msgr2=0x7f5dac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc579b700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac004810 0x7f5dac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.596+0000 7f5dc579b700 1 -- 172.21.15.29:0/3997857364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac003570 con 0x7f5dac003a90 2023-10-07T09:48:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.597+0000 7f5dc579b700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dac003a90 0x7f5dac003ea0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00c300 tx=0x7f5dbc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72da157acc4290f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.597+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc00cb90 con 0x7f5dac003a90 2023-10-07T09:48:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.597+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc00cd10 con 0x7f5dac003a90 2023-10-07T09:48:08.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.597+0000 7f5db7fff700 1 -- 172.21.15.29:0/3997857364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc005b70 con 0x7f5dac003a90 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc79ff700 1 -- 172.21.15.29:0/3997857364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dac003a90 msgr2=0x7f5dac003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/3997857364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dac003a90 0x7f5dac003ea0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00c300 tx=0x7f5dbc001d80 comp rx=0 tx=0).stop 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc5f9c700 1 -- 172.21.15.29:0/3997857364 reap_dead start 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc79ff700 1 -- 172.21.15.29:0/3997857364 shutdown_connections 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc79ff700 1 -- 172.21.15.29:0/3997857364 >> 172.21.15.29:0/3997857364 conn(0x7f5dc00680a0 msgr2=0x7f5dc00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc79ff700 1 -- 172.21.15.29:0/3997857364 shutdown_connections 2023-10-07T09:48:08.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.598+0000 7f5dc79ff700 1 -- 172.21.15.29:0/3997857364 wait complete. 2023-10-07T09:48:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.600+0000 7f5dc79ff700 1 Processor -- start 2023-10-07T09:48:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.601+0000 7f5dc79ff700 1 -- start start 2023-10-07T09:48:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc79ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac003a90 0x7f5dc0158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc79ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 0x7f5dc014b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc79ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc014bec0 0x7f5dc014ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc79ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0169a60 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc79ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0169cd0 con 0x7f5dac003a90 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc79ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0169fb0 con 0x7f5dc014bec0 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc4f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 0x7f5dc014b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc579b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac003a90 0x7f5dc0158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc4f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 0x7f5dc014b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54722/0 (socket says 172.21.15.29:54722) 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc4f9a700 1 -- 172.21.15.29:0/2750768496 learned_addr learned my addr 172.21.15.29:0/2750768496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.602+0000 7f5dc5f9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc014bec0 0x7f5dc014ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc4f9a700 1 -- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc014bec0 msgr2=0x7f5dc014ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc4f9a700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc014bec0 0x7f5dc014ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc4f9a700 1 -- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac003a90 msgr2=0x7f5dc0158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc4f9a700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac003a90 0x7f5dc0158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc4f9a700 1 -- 172.21.15.29:0/2750768496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc016a290 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc4f9a700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 0x7f5dc014b980 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f5db0010b90 tx=0x7f5db0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db000e760 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db000e8c0 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc016a570 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db000eca0 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.603+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc0158e60 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.604+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc01590b0 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.605+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f5db0019030 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.606+0000 7f5db57fa700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 0x7f5da4069610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:08.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.606+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6395+0+0 (secure 0 0 0) 0x7f5db0017070 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.607+0000 7f5dc579b700 1 -- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 msgr2=0x7f5da4069610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:08.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.607+0000 7f5dc579b700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 0x7f5da4069610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:08.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.610+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5dc01590b0 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.808+0000 7f5dc579b700 1 -- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 msgr2=0x7f5da4069610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:08.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.808+0000 7f5dc579b700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 0x7f5da4069610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:08.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.864+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5dc01590b0 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.866 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T09:48:08.865+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T09:48:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.870+0000 7f5db57fa700 1 -- 172.21.15.29:0/2750768496 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v61) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5dc01590b0 con 0x7f5dc00c9dc0 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 msgr2=0x7f5da4069610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 0x7f5da4069610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 msgr2=0x7f5dc014b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 0x7f5dc014b980 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f5db0010b90 tx=0x7f5db0008040 comp rx=0 tx=0).stop 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 shutdown_connections 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5da4067160 0x7f5da4069610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.872+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dc014bec0 0x7f5dc014ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.873+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac003a90 0x7f5dc0158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.873+0000 7f5dc79ff700 1 --2- 172.21.15.29:0/2750768496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc00c9dc0 0x7f5dc014b980 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.873+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 >> 172.21.15.29:0/2750768496 conn(0x7f5dc00680a0 msgr2=0x7f5dc00694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.873+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 shutdown_connections 2023-10-07T09:48:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:08.873+0000 7f5dc79ff700 1 -- 172.21.15.29:0/2750768496 wait complete. 2023-10-07T09:48:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T09:48:08.881 INFO:tasks.ceph.mds.a.smithi029.stderr: -5872> 2023-10-07T09:47:24.355+0000 7f4b209f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:48:08.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T09:48:08.909 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:08.908+0000 7fef0404a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:08.928 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:48:08.941 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:08.940+0000 7f2be375c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:48:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.101+0000 7fecd6fff700 1 Processor -- start 2023-10-07T09:48:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.102+0000 7fecd6fff700 1 -- start start 2023-10-07T09:48:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7fecd6fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fecd0159b20 0x7fecd00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7fecd6fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd00cbf20 con 0x7fecd0159750 2023-10-07T09:48:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7fecd6fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd00cc200 con 0x7fecd00ce8a0 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7fecd6fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd00cc4e0 con 0x7fecd0159b20 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7feccffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fecd0159b20 0x7fecd00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7feccffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fecd0159b20 0x7fecd00d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54738/0 (socket says 172.21.15.29:54738) 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.103+0000 7feccffff700 1 -- 172.21.15.29:0/872406320 learned_addr learned my addr 172.21.15.29:0/872406320 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccffff700 1 -- 172.21.15.29:0/872406320 >> v1:172.21.15.144:6789/0 conn(0x7fecd00ce8a0 legacy=0x7fecd00c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccffff700 1 -- 172.21.15.29:0/872406320 >> v1:172.21.15.29:6789/0 conn(0x7fecd0159750 legacy=0x7fecd00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccffff700 1 -- 172.21.15.29:0/872406320 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd00cc7c0 con 0x7fecd0159b20 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccffff700 1 --2- 172.21.15.29:0/872406320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd0159b20 0x7fecd00d10a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0099b0 tx=0x7fecbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54663dd5b733bf08 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccf7fe700 1 -- 172.21.15.29:0/872406320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc00b860 con 0x7fecd0159b20 2023-10-07T09:48:09.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccf7fe700 1 -- 172.21.15.29:0/872406320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc00b9c0 con 0x7fecd0159b20 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.104+0000 7feccf7fe700 1 -- 172.21.15.29:0/872406320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc00bd60 con 0x7fecd0159b20 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 -- 172.21.15.29:0/872406320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd0159b20 msgr2=0x7fecd00d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 --2- 172.21.15.29:0/872406320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd0159b20 0x7fecd00d10a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0099b0 tx=0x7fecbc009250 comp rx=0 tx=0).stop 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 -- 172.21.15.29:0/872406320 shutdown_connections 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 --2- 172.21.15.29:0/872406320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd0159b20 0x7fecd00d10a0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 -- 172.21.15.29:0/872406320 >> 172.21.15.29:0/872406320 conn(0x7fecd0063a10 msgr2=0x7fecd00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 -- 172.21.15.29:0/872406320 shutdown_connections 2023-10-07T09:48:09.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.105+0000 7fecd6fff700 1 -- 172.21.15.29:0/872406320 wait complete. 2023-10-07T09:48:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.107+0000 7fecd6fff700 1 Processor -- start 2023-10-07T09:48:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.108+0000 7fecd6fff700 1 -- start start 2023-10-07T09:48:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd6fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 0x7fecd013f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:09.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd6fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecd0159750 0x7fecd0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:09.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fecd0159b20 0x7fecd0069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:09.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd6fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015ab00 con 0x7fecd00ce8a0 2023-10-07T09:48:09.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd6fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015d270 con 0x7fecd0159750 2023-10-07T09:48:09.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015ccb0 con 0x7fecd0159b20 2023-10-07T09:48:09.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.109+0000 7fecd4d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 0x7fecd013f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:09.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7feccffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecd0159750 0x7fecd0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:09.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 0x7fecd013f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54744/0 (socket says 172.21.15.29:54744) 2023-10-07T09:48:09.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 -- 172.21.15.29:0/3716342119 learned_addr learned my addr 172.21.15.29:0/3716342119 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:09.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd559c700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fecd0159b20 0x7fecd0069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:09.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 -- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fecd0159b20 msgr2=0x7fecd0069990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fecd0159b20 0x7fecd0069990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 -- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecd0159750 msgr2=0x7fecd0069450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecd0159750 0x7fecd0069450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 -- 172.21.15.29:0/3716342119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc0094f0 con 0x7fecd00ce8a0 2023-10-07T09:48:09.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7feccffff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecd0159750 0x7fecd0069450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:09.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.110+0000 7fecd4d9b700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 0x7fecd013f940 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fecc400c7f0 tx=0x7fecc4007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:09.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.111+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc400f470 con 0x7fecd00ce8a0 2023-10-07T09:48:09.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.111+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc400f5f0 con 0x7fecd00ce8a0 2023-10-07T09:48:09.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.111+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd015cf90 con 0x7fecd00ce8a0 2023-10-07T09:48:09.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.111+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc400f9b0 con 0x7fecd00ce8a0 2023-10-07T09:48:09.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.111+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd00b4810 con 0x7fecd00ce8a0 2023-10-07T09:48:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.112+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd004c3c0 con 0x7fecd00ce8a0 2023-10-07T09:48:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.113+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fecc401b3b0 con 0x7fecd00ce8a0 2023-10-07T09:48:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.114+0000 7fecccff9700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 0x7fecb4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.114+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fecc4084130 con 0x7fecd00ce8a0 2023-10-07T09:48:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.114+0000 7feccffff700 1 -- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 msgr2=0x7fecb4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.115+0000 7feccffff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 0x7fecb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.118+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fecc4050d80 con 0x7fecd00ce8a0 2023-10-07T09:48:09.122 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.121+0000 7f2be375c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:09.310 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.309+0000 7f2be375c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:48:09.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.316+0000 7feccffff700 1 -- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 msgr2=0x7fecb4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:09.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.316+0000 7feccffff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 0x7fecb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:09.366 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:09.365+0000 7fef0404a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:48:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.369+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fecd00c7310 con 0x7fecd00ce8a0 2023-10-07T09:48:09.419 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.418+0000 7f2be375c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.429+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 55) v1 ==== 84451+0+0 (secure 0 0 0) 0x7fecc400fda0 con 0x7fecd00ce8a0 2023-10-07T09:48:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.429+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 msgr2=0x7fecb4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.429+0000 7fecccff9700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 0x7fecb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.501 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.500+0000 7f2b605c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.502 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.500+0000 7f2b605c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.502 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.500+0000 7f2b605c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.502 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.500+0000 7f2b605c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.502 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.500+0000 7f2b605c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.504 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.503+0000 7f2b5ddc3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.504 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.503+0000 7f2b5ddc3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.504 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.503+0000 7f2b5ddc3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.504 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.503+0000 7f2b5ddc3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.504 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:09.503+0000 7f2b5ddc3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:09.559 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:09.558+0000 7fef0404a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:09.753 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:09.751+0000 7fef0404a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:48:09.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:09.864+0000 7fef0404a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:09.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.870+0000 7fecccff9700 1 -- 172.21.15.29:0/3716342119 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7fecc406e260 con 0x7fecd00ce8a0 2023-10-07T09:48:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 msgr2=0x7fecd013f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 0x7fecd013f940 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fecc400c7f0 tx=0x7fecc4007f00 comp rx=0 tx=0).stop 2023-10-07T09:48:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 shutdown_connections 2023-10-07T09:48:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fecb4067170 0x7fecb4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fecd0159b20 0x7fecd0069990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd00ce8a0 0x7fecd013f940 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 --2- 172.21.15.29:0/3716342119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecd0159750 0x7fecd0069450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:09.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 >> 172.21.15.29:0/3716342119 conn(0x7fecd0063a10 msgr2=0x7fecd00c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:09.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 shutdown_connections 2023-10-07T09:48:09.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:09.873+0000 7fecd6fff700 1 -- 172.21.15.29:0/3716342119 wait complete. 2023-10-07T09:48:09.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T09:48:10.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.131+0000 7f69c11ff700 1 Processor -- start 2023-10-07T09:48:10.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.131+0000 7f69c11ff700 1 -- start start 2023-10-07T09:48:10.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69c11ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f69bc0b2a20 0x7f69bc0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69c11ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69bc0cf2c0 0x7f69bc0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69c11ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0cc940 con 0x7f69bc0cb6f0 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69c11ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69bc0b2df0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69c11ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69bc0cbd30 con 0x7f69bc0b2a20 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69ba59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f69bc0cb6f0 0x7f69bc0cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40382/0 (socket says 172.21.15.29:40382) 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69ba59c700 1 -- 172.21.15.29:0/119479628 learned_addr learned my addr 172.21.15.29:0/119479628 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.133+0000 7f69bb59e700 1 --2- 172.21.15.29:0/119479628 >> v2:172.21.15.29:3300/0 conn(0x7f69bc0cf2c0 0x7f69bc0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bad9d700 1 --2- 172.21.15.29:0/119479628 >> v2:172.21.15.144:3300/0 conn(0x7f69bc0b2a20 0x7f69bc0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bb59e700 1 -- 172.21.15.29:0/119479628 >> v2:172.21.15.144:3300/0 conn(0x7f69bc0b2a20 msgr2=0x7f69bc0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bb59e700 1 --2- 172.21.15.29:0/119479628 >> v2:172.21.15.144:3300/0 conn(0x7f69bc0b2a20 0x7f69bc0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bb59e700 1 -- 172.21.15.29:0/119479628 >> v1:172.21.15.29:6789/0 conn(0x7f69bc0cb6f0 legacy=0x7f69bc0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bb59e700 1 -- 172.21.15.29:0/119479628 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0cc010 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bad9d700 1 --2- 172.21.15.29:0/119479628 >> v2:172.21.15.144:3300/0 conn(0x7f69bc0b2a20 0x7f69bc0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69bb59e700 1 --2- 172.21.15.29:0/119479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0c5400 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f69bc08ac80 tx=0x7f69b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1d9292ee8f5cff server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.134+0000 7f69b9d9b700 1 -- 172.21.15.29:0/119479628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b400e070 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69b9d9b700 1 -- 172.21.15.29:0/119479628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b400bc30 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69b9d9b700 1 -- 172.21.15.29:0/119479628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4012430 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 -- 172.21.15.29:0/119479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 msgr2=0x7f69bc0c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 --2- 172.21.15.29:0/119479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0c5400 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f69bc08ac80 tx=0x7f69b4009780 comp rx=0 tx=0).stop 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 -- 172.21.15.29:0/119479628 shutdown_connections 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 --2- 172.21.15.29:0/119479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0c5400 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 --2- 172.21.15.29:0/119479628 >> v2:172.21.15.144:3300/0 conn(0x7f69bc0b2a20 0x7f69bc0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 -- 172.21.15.29:0/119479628 >> 172.21.15.29:0/119479628 conn(0x7f69bc099820 msgr2=0x7f69bc063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 -- 172.21.15.29:0/119479628 shutdown_connections 2023-10-07T09:48:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.135+0000 7f69c11ff700 1 -- 172.21.15.29:0/119479628 wait complete. 2023-10-07T09:48:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.137+0000 7f69c11ff700 1 Processor -- start 2023-10-07T09:48:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.137+0000 7f69c11ff700 1 -- start start 2023-10-07T09:48:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69c11ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69bc0b2a20 0x7f69bc15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69c11ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69bc0cb6f0 0x7f69bc0c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69c11ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69c11ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0b38e0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69c11ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0b3bc0 con 0x7f69bc0b2a20 2023-10-07T09:48:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69c11ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0b3ea0 con 0x7f69bc0cb6f0 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54768/0 (socket says 172.21.15.29:54768) 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 -- 172.21.15.29:0/643111377 learned_addr learned my addr 172.21.15.29:0/643111377 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bad9d700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69bc0b2a20 0x7f69bc15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69ba59c700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69bc0cb6f0 0x7f69bc0c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 -- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69bc0cb6f0 msgr2=0x7f69bc0c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69bc0cb6f0 0x7f69bc0c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 -- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69bc0b2a20 msgr2=0x7f69bc15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69bc0b2a20 0x7f69bc15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.138+0000 7f69bb59e700 1 -- 172.21.15.29:0/643111377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4009a20 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.139+0000 7f69bb59e700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0d0c50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f69b40097c0 tx=0x7f69b4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.139+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4027470 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.139+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b40275f0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.139+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc156170 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.139+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b40279b0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.139+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc0c7bc0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.140+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69bc1563b0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.141+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f69b4024070 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.142+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f69b4086d00 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.146+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69b4053e00 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.350+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f69bc0718b0 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.432+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 56) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f69b4003040 con 0x7f69bc0cf2c0 2023-10-07T09:48:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.433+0000 7f69a2ffd700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69a806ca90 0x7f69a806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.434+0000 7f69bad9d700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69a806ca90 0x7f69a806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:10.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:10.440+0000 7f69bad9d700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69a806ca90 0x7f69a806ee70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f69ac006ea0 tx=0x7f69ac008040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.258+0000 7f69a2ffd700 1 -- 172.21.15.29:0/643111377 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v123) v1 ==== 181+0+0 (secure 0 0 0) 0x7f69b4028eb0 con 0x7f69bc0cf2c0 2023-10-07T09:48:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69a806ca90 msgr2=0x7f69a806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69a806ca90 0x7f69a806ee70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f69ac006ea0 tx=0x7f69ac008040 comp rx=0 tx=0).stop 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 msgr2=0x7f69bc0d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0d0c50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f69b40097c0 tx=0x7f69b4019000 comp rx=0 tx=0).stop 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 shutdown_connections 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f69a806ca90 0x7f69a806ee70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0cf2c0 0x7f69bc0d0c50 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69bc0b2a20 0x7f69bc15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 --2- 172.21.15.29:0/643111377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69bc0cb6f0 0x7f69bc0c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 >> 172.21.15.29:0/643111377 conn(0x7f69bc099820 msgr2=0x7f69bc158ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 shutdown_connections 2023-10-07T09:48:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.263+0000 7f69c11ff700 1 -- 172.21.15.29:0/643111377 wait complete. 2023-10-07T09:48:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T09:48:11.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T09:48:11.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.492+0000 7fead7fff700 1 Processor -- start 2023-10-07T09:48:11.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.492+0000 7fead7fff700 1 -- start start 2023-10-07T09:48:11.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead7fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fead00cc810 0x7fead00accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead015a590 0x7fead015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead7fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead00ad400 con 0x7fead00afeb0 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead7fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead015a140 con 0x7fead00cc810 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead00adca0 con 0x7fead015a590 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fead00cc810 0x7fead00accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fead00cc810 0x7fead00accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54778/0 (socket says 172.21.15.29:54778) 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 -- 172.21.15.29:0/2157322261 learned_addr learned my addr 172.21.15.29:0/2157322261 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead659c700 1 --2- 172.21.15.29:0/2157322261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead015a590 0x7fead015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 -- 172.21.15.29:0/2157322261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead015a590 msgr2=0x7fead015a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:11.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 --2- 172.21.15.29:0/2157322261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead015a590 0x7fead015a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 -- 172.21.15.29:0/2157322261 >> v1:172.21.15.144:6789/0 conn(0x7fead00afeb0 legacy=0x7fead00aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:11.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.494+0000 7fead559a700 1 -- 172.21.15.29:0/2157322261 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead00ad6e0 con 0x7fead00cc810 2023-10-07T09:48:11.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.495+0000 7fead559a700 1 --2- 172.21.15.29:0/2157322261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00cc810 0x7fead00accf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7feac000bc50 tx=0x7feac000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3fb48919ccfa43 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:11.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.495+0000 7fead4d99700 1 -- 172.21.15.29:0/2157322261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac000dbf0 con 0x7fead00cc810 2023-10-07T09:48:11.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.495+0000 7fead4d99700 1 -- 172.21.15.29:0/2157322261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac000dd50 con 0x7fead00cc810 2023-10-07T09:48:11.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.495+0000 7fead4d99700 1 -- 172.21.15.29:0/2157322261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00196b0 con 0x7fead00cc810 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.495+0000 7fead7fff700 1 -- 172.21.15.29:0/2157322261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00cc810 msgr2=0x7fead00accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.495+0000 7fead7fff700 1 --2- 172.21.15.29:0/2157322261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00cc810 0x7fead00accf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7feac000bc50 tx=0x7feac000b4f0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.496+0000 7fead7fff700 1 -- 172.21.15.29:0/2157322261 shutdown_connections 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.496+0000 7fead7fff700 1 --2- 172.21.15.29:0/2157322261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead015a590 0x7fead015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.496+0000 7fead7fff700 1 --2- 172.21.15.29:0/2157322261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00cc810 0x7fead00accf0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.496+0000 7fead7fff700 1 -- 172.21.15.29:0/2157322261 >> 172.21.15.29:0/2157322261 conn(0x7fead0067610 msgr2=0x7fead0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.496+0000 7fead7fff700 1 -- 172.21.15.29:0/2157322261 shutdown_connections 2023-10-07T09:48:11.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.496+0000 7fead7fff700 1 -- 172.21.15.29:0/2157322261 wait complete. 2023-10-07T09:48:11.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.497+0000 7fead7fff700 1 Processor -- start 2023-10-07T09:48:11.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.497+0000 7fead7fff700 1 -- start start 2023-10-07T09:48:11.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 0x7fead014dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:11.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead00cc810 0x7fead015c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:11.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead7fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fead015a590 0x7fead00ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:11.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead7fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead015cb90 con 0x7fead00afeb0 2023-10-07T09:48:11.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead7fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead015ce70 con 0x7fead015a590 2023-10-07T09:48:11.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead015d150 con 0x7fead00cc810 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.498+0000 7fead5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 0x7fead014dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 0x7fead014dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54784/0 (socket says 172.21.15.29:54784) 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 -- 172.21.15.29:0/3795339622 learned_addr learned my addr 172.21.15.29:0/3795339622 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead659c700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fead015a590 0x7fead00ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead559a700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead00cc810 0x7fead015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 -- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead00cc810 msgr2=0x7fead015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead00cc810 0x7fead015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 -- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fead015a590 msgr2=0x7fead00ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fead015a590 0x7fead00ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 -- 172.21.15.29:0/3795339622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac000b790 con 0x7fead00afeb0 2023-10-07T09:48:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead559a700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead00cc810 0x7fead015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead5d9b700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 0x7fead014dc30 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7feacc00c590 tx=0x7feacc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc011590 con 0x7fead00afeb0 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc011710 con 0x7fead00afeb0 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.499+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead0156690 con 0x7fead00afeb0 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.500+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc011ad0 con 0x7fead00afeb0 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.500+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead014e170 con 0x7fead00afeb0 2023-10-07T09:48:11.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.501+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead0156bb0 con 0x7fead00afeb0 2023-10-07T09:48:11.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.502+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 84578+0+0 (secure 0 0 0) 0x7feacc01b3b0 con 0x7fead00afeb0 2023-10-07T09:48:11.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.502+0000 7feac5ffb700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feab4066d80 0x7feab4069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:11.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.503+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6359+0+0 (secure 0 0 0) 0x7feacc083a80 con 0x7fead00afeb0 2023-10-07T09:48:11.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.503+0000 7fead559a700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feab4066d80 0x7feab4069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:11.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.504+0000 7fead559a700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feab4066d80 0x7feab4069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feac000bc50 tx=0x7feac001b000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:11.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.506+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fead0156bb0 con 0x7fead00afeb0 2023-10-07T09:48:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:11.708+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fead0156bb0 con 0x7fead00afeb0 2023-10-07T09:48:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.196+0000 7feac5ffb700 1 -- 172.21.15.29:0/3795339622 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v124) v1 ==== 169+0+0 (secure 0 0 0) 0x7feacc017070 con 0x7fead00afeb0 2023-10-07T09:48:12.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.200+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feab4066d80 msgr2=0x7feab4069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.200+0000 7fead7fff700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feab4066d80 0x7feab4069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feac000bc50 tx=0x7feac001b000 comp rx=0 tx=0).stop 2023-10-07T09:48:12.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.200+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 msgr2=0x7fead014dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.200+0000 7fead7fff700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 0x7fead014dc30 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7feacc00c590 tx=0x7feacc00c8a0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 shutdown_connections 2023-10-07T09:48:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feab4066d80 0x7feab4069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fead015a590 0x7fead00ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fead00afeb0 0x7fead014dc30 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 --2- 172.21.15.29:0/3795339622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fead00cc810 0x7fead015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 >> 172.21.15.29:0/3795339622 conn(0x7fead0067610 msgr2=0x7fead006dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 shutdown_connections 2023-10-07T09:48:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.201+0000 7fead7fff700 1 -- 172.21.15.29:0/3795339622 wait complete. 2023-10-07T09:48:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T09:48:12.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T09:48:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.424+0000 7f2c817ff700 1 Processor -- start 2023-10-07T09:48:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.424+0000 7f2c817ff700 1 -- start start 2023-10-07T09:48:12.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.425+0000 7f2c817ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c159790 0x7f2c7c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.425+0000 7f2c817ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c06a430 con 0x7f2c7c06b3f0 2023-10-07T09:48:12.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.425+0000 7f2c817ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c06b830 con 0x7f2c7c06a8c0 2023-10-07T09:48:12.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.425+0000 7f2c817ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c159b60 con 0x7f2c7c159790 2023-10-07T09:48:12.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.425+0000 7f2c7b7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2c7c06b3f0 0x7f2c7c067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40392/0 (socket says 172.21.15.29:40392) 2023-10-07T09:48:12.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.425+0000 7f2c7b7fe700 1 -- 172.21.15.29:0/3439647866 learned_addr learned my addr 172.21.15.29:0/3439647866 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:12.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c7affd700 1 --2- 172.21.15.29:0/3439647866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c159790 0x7f2c7c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762125380 0 0) 0x7f2c7c06b830 con 0x7f2c7c06a8c0 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c64003570 con 0x7f2c7c06a8c0 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463801486 0 0) 0x7f2c7c06a430 con 0x7f2c7c06b3f0 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c06b830 con 0x7f2c7c06b3f0 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c7affd700 1 -- 172.21.15.29:0/3439647866 >> v1:172.21.15.144:6789/0 conn(0x7f2c7c06a8c0 legacy=0x7f2c7c0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c7affd700 1 -- 172.21.15.29:0/3439647866 >> v1:172.21.15.29:6789/0 conn(0x7f2c7c06b3f0 legacy=0x7f2c7c067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c7affd700 1 -- 172.21.15.29:0/3439647866 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c069820 con 0x7f2c7c159790 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.426+0000 7f2c7affd700 1 --2- 172.21.15.29:0/3439647866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c159790 0x7f2c7c06aeb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009910 tx=0x7f2c6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3810f874640c2bfc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c011630 con 0x7f2c7c159790 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c011790 con 0x7f2c7c159790 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c79ffb700 1 -- 172.21.15.29:0/3439647866 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c011b10 con 0x7f2c7c159790 2023-10-07T09:48:12.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 -- 172.21.15.29:0/3439647866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c159790 msgr2=0x7f2c7c06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 --2- 172.21.15.29:0/3439647866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c159790 0x7f2c7c06aeb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009910 tx=0x7f2c6c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 -- 172.21.15.29:0/3439647866 shutdown_connections 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 --2- 172.21.15.29:0/3439647866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c159790 0x7f2c7c06aeb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 -- 172.21.15.29:0/3439647866 >> 172.21.15.29:0/3439647866 conn(0x7f2c7c0c9410 msgr2=0x7f2c7c0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 -- 172.21.15.29:0/3439647866 shutdown_connections 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.427+0000 7f2c817ff700 1 -- 172.21.15.29:0/3439647866 wait complete. 2023-10-07T09:48:12.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.429+0000 7f2c817ff700 1 Processor -- start 2023-10-07T09:48:12.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.429+0000 7f2c817ff700 1 -- start start 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c817ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7c06b3f0 0x7f2c7c0b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c817ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 0x7f2c7c16d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c817ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c0b06a0 0x7f2c7c16d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c817ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0b0be0 con 0x7f2c7c159790 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c817ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0b0ec0 con 0x7f2c7c06b3f0 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c817ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0b11a0 con 0x7f2c7c0b06a0 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c7a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 0x7f2c7c16d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c7a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 0x7f2c7c16d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54792/0 (socket says 172.21.15.29:54792) 2023-10-07T09:48:12.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.430+0000 7f2c7a7fc700 1 -- 172.21.15.29:0/607741679 learned_addr learned my addr 172.21.15.29:0/607741679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7b7fe700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c0b06a0 0x7f2c7c16d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7affd700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7c06b3f0 0x7f2c7c0b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7a7fc700 1 -- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c0b06a0 msgr2=0x7f2c7c16d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7a7fc700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c0b06a0 0x7f2c7c16d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7a7fc700 1 -- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7c06b3f0 msgr2=0x7f2c7c0b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7a7fc700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7c06b3f0 0x7f2c7c0b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7a7fc700 1 -- 172.21.15.29:0/607741679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c009420 con 0x7f2c7c159790 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7affd700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7c06b3f0 0x7f2c7c0b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7b7fe700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c0b06a0 0x7f2c7c16d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:12.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c7a7fc700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 0x7f2c7c16d310 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000c5b0 tx=0x7f2c70007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70011680 con 0x7f2c7c159790 2023-10-07T09:48:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c0b1480 con 0x7f2c7c159790 2023-10-07T09:48:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.431+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c0aac40 con 0x7f2c7c159790 2023-10-07T09:48:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.433+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c0b1920 con 0x7f2c7c159790 2023-10-07T09:48:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.433+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c700119f0 con 0x7f2c7c159790 2023-10-07T09:48:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.433+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70011b60 con 0x7f2c7c159790 2023-10-07T09:48:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.435+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2c70011cc0 con 0x7f2c7c159790 2023-10-07T09:48:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.436+0000 7f2c62ffd700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c68066f30 0x7f2c680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.436+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7f2c70084fe0 con 0x7f2c7c159790 2023-10-07T09:48:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.437+0000 7f2c7affd700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c68066f30 0x7f2c680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.437+0000 7f2c7affd700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c68066f30 0x7f2c680693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c003080 tx=0x7f2c6c017000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.439+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c70052180 con 0x7f2c7c159790 2023-10-07T09:48:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.649+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2c7c04c3c0 con 0x7f2c7c159790 2023-10-07T09:48:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.654+0000 7f2c62ffd700 1 -- 172.21.15.29:0/607741679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2c700557a0 con 0x7f2c7c159790 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c68066f30 msgr2=0x7f2c680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c68066f30 0x7f2c680693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c003080 tx=0x7f2c6c017000 comp rx=0 tx=0).stop 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 msgr2=0x7f2c7c16d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 0x7f2c7c16d310 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000c5b0 tx=0x7f2c70007df0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 shutdown_connections 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2c68066f30 0x7f2c680693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c7c0b06a0 0x7f2c7c16d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7c06b3f0 0x7f2c7c0b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 --2- 172.21.15.29:0/607741679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159790 0x7f2c7c16d310 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 >> 172.21.15.29:0/607741679 conn(0x7f2c7c0c9410 msgr2=0x7f2c7c154000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:12.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 shutdown_connections 2023-10-07T09:48:12.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.656+0000 7f2c817ff700 1 -- 172.21.15.29:0/607741679 wait complete. 2023-10-07T09:48:12.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T09:48:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.872+0000 7f820d9ff700 1 Processor -- start 2023-10-07T09:48:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.872+0000 7f820d9ff700 1 -- start start 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.873+0000 7f820d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080b0c30 0x7f82080afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.873+0000 7f820d9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82080ad4d0 con 0x7f8208156180 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.873+0000 7f820d9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82080ad7b0 con 0x7f82080ae570 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.873+0000 7f820d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82080ada90 con 0x7f82080b0c30 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f8206ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8208156180 0x7f82080accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40398/0 (socket says 172.21.15.29:40398) 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f8206ffd700 1 -- 172.21.15.29:0/3838969649 learned_addr learned my addr 172.21.15.29:0/3838969649 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f82077fe700 1 --2- 172.21.15.29:0/3838969649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080b0c30 0x7f82080afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262776401 0 0) 0x7f82080ad4d0 con 0x7f8208156180 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0003570 con 0x7f8208156180 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214878180 0 0) 0x7f82080ad7b0 con 0x7f82080ae570 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82080ad4d0 con 0x7f82080ae570 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f82077fe700 1 -- 172.21.15.29:0/3838969649 >> v1:172.21.15.144:6789/0 conn(0x7f82080ae570 legacy=0x7f82080b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f82077fe700 1 -- 172.21.15.29:0/3838969649 >> v1:172.21.15.29:6789/0 conn(0x7f8208156180 legacy=0x7f82080accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.874+0000 7f82077fe700 1 -- 172.21.15.29:0/3838969649 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080add70 con 0x7f82080b0c30 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.875+0000 7f82077fe700 1 --2- 172.21.15.29:0/3838969649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080b0c30 0x7f82080afc00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8200009e90 tx=0x7f8200009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f651db3396d8862f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.875+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000e070 con 0x7f82080b0c30 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.875+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000bbd0 con 0x7f82080b0c30 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.875+0000 7f8205ffb700 1 -- 172.21.15.29:0/3838969649 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200012430 con 0x7f82080b0c30 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 -- 172.21.15.29:0/3838969649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080b0c30 msgr2=0x7f82080afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 --2- 172.21.15.29:0/3838969649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080b0c30 0x7f82080afc00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8200009e90 tx=0x7f8200009780 comp rx=0 tx=0).stop 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 -- 172.21.15.29:0/3838969649 shutdown_connections 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 --2- 172.21.15.29:0/3838969649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080b0c30 0x7f82080afc00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 -- 172.21.15.29:0/3838969649 >> 172.21.15.29:0/3838969649 conn(0x7f8208067610 msgr2=0x7f8208067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 -- 172.21.15.29:0/3838969649 shutdown_connections 2023-10-07T09:48:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.876+0000 7f820d9ff700 1 -- 172.21.15.29:0/3838969649 wait complete. 2023-10-07T09:48:12.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.878+0000 7f820d9ff700 1 Processor -- start 2023-10-07T09:48:12.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.878+0000 7f820d9ff700 1 -- start start 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.879+0000 7f820d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080ae570 0x7f8208149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.879+0000 7f820d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 0x7f82080c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.879+0000 7f820d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8208156180 0x7f82080a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.879+0000 7f820d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82080c6920 con 0x7f82080b0c30 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.879+0000 7f820d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82080c5c60 con 0x7f8208156180 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.879+0000 7f820d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82080c5f40 con 0x7f82080ae570 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 0x7f82080c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 0x7f82080c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54796/0 (socket says 172.21.15.29:54796) 2023-10-07T09:48:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 -- 172.21.15.29:0/2615506401 learned_addr learned my addr 172.21.15.29:0/2615506401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:12.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f8206ffd700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080ae570 0x7f8208149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82077fe700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8208156180 0x7f82080a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 -- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080ae570 msgr2=0x7f8208149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080ae570 0x7f8208149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 -- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8208156180 msgr2=0x7f82080a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8208156180 0x7f82080a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 -- 172.21.15.29:0/2615506401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200009a20 con 0x7f82080b0c30 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.880+0000 7f82067fc700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 0x7f82080c5310 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00c7c0 tx=0x7f81fc007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.881+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0015f0 con 0x7f82080b0c30 2023-10-07T09:48:12.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.881+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc001790 con 0x7f82080b0c30 2023-10-07T09:48:12.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.881+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82080c6220 con 0x7f82080b0c30 2023-10-07T09:48:12.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.881+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc001b50 con 0x7f82080b0c30 2023-10-07T09:48:12.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.881+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82080c5850 con 0x7f82080b0c30 2023-10-07T09:48:12.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.882+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820804c3c0 con 0x7f82080b0c30 2023-10-07T09:48:12.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.883+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81fc01b400 con 0x7f82080b0c30 2023-10-07T09:48:12.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.884+0000 7f81eeffd700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81f4066f30 0x7f81f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:12.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.884+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7f81fc084500 con 0x7f82080b0c30 2023-10-07T09:48:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.886+0000 7f8206ffd700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81f4066f30 0x7f81f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:12.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.887+0000 7f8206ffd700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81f4066f30 0x7f81f40693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81f8007fd0 tx=0x7f81f8009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:12.888+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81fc0515e0 con 0x7f82080b0c30 2023-10-07T09:48:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.099+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f82080c6a60 con 0x7f82080b0c30 2023-10-07T09:48:13.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.103+0000 7f81eeffd700 1 -- 172.21.15.29:0/2615506401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f81fc054c00 con 0x7f82080b0c30 2023-10-07T09:48:13.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81f4066f30 msgr2=0x7f81f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81f4066f30 0x7f81f40693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81f8007fd0 tx=0x7f81f8009040 comp rx=0 tx=0).stop 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 msgr2=0x7f82080c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 0x7f82080c5310 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00c7c0 tx=0x7f81fc007dd0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 shutdown_connections 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81f4066f30 0x7f81f40693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8208156180 0x7f82080a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82080ae570 0x7f8208149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 --2- 172.21.15.29:0/2615506401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82080b0c30 0x7f82080c5310 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 >> 172.21.15.29:0/2615506401 conn(0x7f8208067610 msgr2=0x7f82080accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 shutdown_connections 2023-10-07T09:48:13.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.106+0000 7f820d9ff700 1 -- 172.21.15.29:0/2615506401 wait complete. 2023-10-07T09:48:13.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T09:48:13.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.323+0000 7f92c87ff700 1 Processor -- start 2023-10-07T09:48:13.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.323+0000 7f92c87ff700 1 -- start start 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c87ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f92c00ce8a0 0x7f92c00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c87ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00cbf20 con 0x7f92c0159b20 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c87ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00cc200 con 0x7f92c00ce8a0 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cc4e0 con 0x7f92c0159750 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c5d9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f92c0159b20 0x7f92c00ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59152/0 (socket says 172.21.15.29:59152) 2023-10-07T09:48:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c5d9a700 1 -- 172.21.15.29:0/1548725148 learned_addr learned my addr 172.21.15.29:0/1548725148 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c659b700 1 --2- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c6d9c700 1 --2- 172.21.15.29:0/1548725148 >> v2:172.21.15.144:3300/0 conn(0x7f92c00ce8a0 0x7f92c00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c5599700 1 -- 172.21.15.29:0/1548725148 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173478376 0 0) 0x7f92c00cbf20 con 0x7f92c0159b20 2023-10-07T09:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.325+0000 7f92c5599700 1 -- 172.21.15.29:0/1548725148 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac003570 con 0x7f92c0159b20 2023-10-07T09:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c6d9c700 1 -- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 msgr2=0x7f92c00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c6d9c700 1 --2- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c6d9c700 1 -- 172.21.15.29:0/1548725148 >> v1:172.21.15.29:6789/0 conn(0x7f92c0159b20 legacy=0x7f92c00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c6d9c700 1 -- 172.21.15.29:0/1548725148 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00cc7c0 con 0x7f92c00ce8a0 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c6d9c700 1 --2- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c00ce8a0 0x7f92c00c5460 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009e60 tx=0x7f92b8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=350632fdffe3711c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c5599700 1 -- 172.21.15.29:0/1548725148 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b800e070 con 0x7f92c00ce8a0 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c5599700 1 -- 172.21.15.29:0/1548725148 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b800bab0 con 0x7f92c00ce8a0 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.326+0000 7f92c5599700 1 -- 172.21.15.29:0/1548725148 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b800be70 con 0x7f92c00ce8a0 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 -- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c00ce8a0 msgr2=0x7f92c00c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c00ce8a0 0x7f92c00c5460 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009e60 tx=0x7f92b8009780 comp rx=0 tx=0).stop 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 -- 172.21.15.29:0/1548725148 shutdown_connections 2023-10-07T09:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c00ce8a0 0x7f92c00c5460 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1548725148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 -- 172.21.15.29:0/1548725148 >> 172.21.15.29:0/1548725148 conn(0x7f92c0063a10 msgr2=0x7f92c00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 -- 172.21.15.29:0/1548725148 shutdown_connections 2023-10-07T09:48:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.327+0000 7f92c87ff700 1 -- 172.21.15.29:0/1548725148 wait complete. 2023-10-07T09:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.328+0000 7f92c87ff700 1 Processor -- start 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 -- start start 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 0x7f92c01697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c015e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c0159b20 0x7f92c016a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c015f850 con 0x7f92c00ce8a0 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00b6300 con 0x7f92c0159b20 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.329+0000 7f92c87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c015ec40 con 0x7f92c0159750 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 0x7f92c01697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 0x7f92c01697c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46884/0 (socket says 172.21.15.29:46884) 2023-10-07T09:48:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 -- 172.21.15.29:0/1485827721 learned_addr learned my addr 172.21.15.29:0/1485827721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c5d9a700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c6d9c700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c0159b20 0x7f92c016a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 -- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 msgr2=0x7f92c015e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c015e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 -- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c0159b20 msgr2=0x7f92c016a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c0159b20 0x7f92c016a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.330+0000 7f92c659b700 1 -- 172.21.15.29:0/1485827721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8009a20 con 0x7f92c00ce8a0 2023-10-07T09:48:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.331+0000 7f92c659b700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 0x7f92c01697c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00ec80 tx=0x7f92bc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.331+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00c9d0 con 0x7f92c00ce8a0 2023-10-07T09:48:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.331+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00cb30 con 0x7f92c00ce8a0 2023-10-07T09:48:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.331+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc013470 con 0x7f92c00ce8a0 2023-10-07T09:48:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.331+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c015ef20 con 0x7f92c00ce8a0 2023-10-07T09:48:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.331+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c015f4e0 con 0x7f92c00ce8a0 2023-10-07T09:48:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.332+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c015fb30 con 0x7f92c00ce8a0 2023-10-07T09:48:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.333+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f92bc013860 con 0x7f92c00ce8a0 2023-10-07T09:48:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.334+0000 7f92b67fc700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a4066f30 0x7f92a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.334+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7f92bc084e70 con 0x7f92c00ce8a0 2023-10-07T09:48:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.334+0000 7f92c5d9a700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a4066f30 0x7f92a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.335+0000 7f92c5d9a700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a4066f30 0x7f92a40693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f92c006a240 tx=0x7f92b0009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.338+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92bc051f50 con 0x7f92c00ce8a0 2023-10-07T09:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.539+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f92c004c3c0 con 0x7f92c00ce8a0 2023-10-07T09:48:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.543+0000 7f92b67fc700 1 -- 172.21.15.29:0/1485827721 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f92bc055570 con 0x7f92c00ce8a0 2023-10-07T09:48:13.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a4066f30 msgr2=0x7f92a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a4066f30 0x7f92a40693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f92c006a240 tx=0x7f92b0009040 comp rx=0 tx=0).stop 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 msgr2=0x7f92c01697c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 0x7f92c01697c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00ec80 tx=0x7f92bc009120 comp rx=0 tx=0).stop 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 shutdown_connections 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f92a4066f30 0x7f92a40693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92c0159b20 0x7f92c016a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92c00ce8a0 0x7f92c01697c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 --2- 172.21.15.29:0/1485827721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92c0159750 0x7f92c015e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 >> 172.21.15.29:0/1485827721 conn(0x7f92c0063a10 msgr2=0x7f92c00ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 shutdown_connections 2023-10-07T09:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.545+0000 7f92c87ff700 1 -- 172.21.15.29:0/1485827721 wait complete. 2023-10-07T09:48:13.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-07T09:48:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.759+0000 7f81859ff700 1 Processor -- start 2023-10-07T09:48:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.759+0000 7f81859ff700 1 -- start start 2023-10-07T09:48:13.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.760+0000 7f81859ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81800ad370 0x7f81800cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.760+0000 7f81859ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818014d790 con 0x7f81800b0410 2023-10-07T09:48:13.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.760+0000 7f81859ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800ae5d0 con 0x7f8180159ec0 2023-10-07T09:48:13.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.760+0000 7f81859ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81800ad740 con 0x7f81800ad370 2023-10-07T09:48:13.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.760+0000 7f817effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81800ad370 0x7f81800cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81800ad370 0x7f81800cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46898/0 (socket says 172.21.15.29:46898) 2023-10-07T09:48:13.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817effd700 1 -- 172.21.15.29:0/1123305747 learned_addr learned my addr 172.21.15.29:0/1123305747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817effd700 1 -- 172.21.15.29:0/1123305747 >> v1:172.21.15.144:6789/0 conn(0x7f8180159ec0 legacy=0x7f81801568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817effd700 1 -- 172.21.15.29:0/1123305747 >> v1:172.21.15.29:6789/0 conn(0x7f81800b0410 legacy=0x7f8180156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817effd700 1 -- 172.21.15.29:0/1123305747 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81800ccdb0 con 0x7f81800ad370 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817dffb700 1 -- 172.21.15.29:0/1123305747 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518646715 0 0) 0x7f818014d790 con 0x7f81800b0410 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.761+0000 7f817effd700 1 --2- 172.21.15.29:0/1123305747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81800ad370 0x7f81800cc870 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e47c18db8cb66d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f817dffb700 1 -- 172.21.15.29:0/1123305747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817000b820 con 0x7f81800ad370 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f817dffb700 1 -- 172.21.15.29:0/1123305747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817000b980 con 0x7f81800ad370 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f817dffb700 1 -- 172.21.15.29:0/1123305747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817000bd20 con 0x7f81800ad370 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 -- 172.21.15.29:0/1123305747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81800ad370 msgr2=0x7f81800cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 --2- 172.21.15.29:0/1123305747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81800ad370 0x7f81800cc870 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170009250 comp rx=0 tx=0).stop 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 -- 172.21.15.29:0/1123305747 shutdown_connections 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 --2- 172.21.15.29:0/1123305747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81800ad370 0x7f81800cc870 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 -- 172.21.15.29:0/1123305747 >> 172.21.15.29:0/1123305747 conn(0x7f81800680a0 msgr2=0x7f81800cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 -- 172.21.15.29:0/1123305747 shutdown_connections 2023-10-07T09:48:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.762+0000 7f81859ff700 1 -- 172.21.15.29:0/1123305747 wait complete. 2023-10-07T09:48:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.764+0000 7f81859ff700 1 Processor -- start 2023-10-07T09:48:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.764+0000 7f81859ff700 1 -- start start 2023-10-07T09:48:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f81859ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81800ad370 0x7f81800a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f81859ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81800b0410 0x7f8180168040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f81859ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 0x7f818014b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f81859ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81800a9f30 con 0x7f8180159ec0 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f81859ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81800a5830 con 0x7f81800b0410 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f81859ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8180168900 con 0x7f81800ad370 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f817f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 0x7f818014b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f817f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 0x7f818014b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46902/0 (socket says 172.21.15.29:46902) 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f817f7fe700 1 -- 172.21.15.29:0/2426360508 learned_addr learned my addr 172.21.15.29:0/2426360508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f817e7fc700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81800b0410 0x7f8180168040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.765+0000 7f817effd700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81800ad370 0x7f81800a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817f7fe700 1 -- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81800ad370 msgr2=0x7f81800a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817f7fe700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81800ad370 0x7f81800a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817f7fe700 1 -- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81800b0410 msgr2=0x7f8180168040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817f7fe700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81800b0410 0x7f8180168040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817f7fe700 1 -- 172.21.15.29:0/2426360508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700094f0 con 0x7f8180159ec0 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817effd700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81800ad370 0x7f81800a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f817f7fe700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 0x7f818014b3a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8178001560 tx=0x7f8178001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178003030 con 0x7f8180159ec0 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817800a7e0 con 0x7f8180159ec0 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817800aba0 con 0x7f8180159ec0 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8180168be0 con 0x7f8180159ec0 2023-10-07T09:48:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.766+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8180168580 con 0x7f8180159ec0 2023-10-07T09:48:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.768+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81800c8650 con 0x7f8180159ec0 2023-10-07T09:48:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.768+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8178012420 con 0x7f8180159ec0 2023-10-07T09:48:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.769+0000 7f8166ffd700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f816c066f30 0x7f816c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.770+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7f8178004070 con 0x7f8180159ec0 2023-10-07T09:48:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.770+0000 7f817effd700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f816c066f30 0x7f816c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.771+0000 7f817effd700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f816c066f30 0x7f816c0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170017000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:13.773+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8178051620 con 0x7f8180159ec0 2023-10-07T09:48:14.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.061+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f818004c3c0 con 0x7f8180159ec0 2023-10-07T09:48:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.272+0000 7f8166ffd700 1 -- 172.21.15.29:0/2426360508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v718) v1 ==== 148+0+0 (secure 0 0 0) 0x7f8178054c40 con 0x7f8180159ec0 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f816c066f30 msgr2=0x7f816c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f816c066f30 0x7f816c0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170017000 comp rx=0 tx=0).stop 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 msgr2=0x7f818014b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 0x7f818014b3a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8178001560 tx=0x7f8178001770 comp rx=0 tx=0).stop 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 shutdown_connections 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f816c066f30 0x7f816c0693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8180159ec0 0x7f818014b3a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81800ad370 0x7f81800a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 --2- 172.21.15.29:0/2426360508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81800b0410 0x7f8180168040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 >> 172.21.15.29:0/2426360508 conn(0x7f81800680a0 msgr2=0x7f818014c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.274+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 shutdown_connections 2023-10-07T09:48:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.275+0000 7f81859ff700 1 -- 172.21.15.29:0/2426360508 wait complete. 2023-10-07T09:48:14.294 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-07T09:48:14.294 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-07T09:48:14.294 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T09:48:14.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-07T09:48:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.489+0000 7feeb0bff700 1 Processor -- start 2023-10-07T09:48:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.489+0000 7feeb0bff700 1 -- start start 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeb0bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac0ad370 0x7feeac0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeb0bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feeac0b0410 0x7feeac15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeb0bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac14d790 con 0x7feeac159ec0 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeb0bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac0ae5d0 con 0x7feeac0b0410 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeb0bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac15a850 con 0x7feeac0ad370 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feeac0b0410 0x7feeac15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feeac0b0410 0x7feeac15a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46910/0 (socket says 172.21.15.29:46910) 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 -- 172.21.15.29:0/2195593119 learned_addr learned my addr 172.21.15.29:0/2195593119 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaa59c700 1 --2- 172.21.15.29:0/2195593119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac0ad370 0x7feeac0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 -- 172.21.15.29:0/2195593119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac0ad370 msgr2=0x7feeac0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 --2- 172.21.15.29:0/2195593119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac0ad370 0x7feeac0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 -- 172.21.15.29:0/2195593119 >> v1:172.21.15.144:6789/0 conn(0x7feeac159ec0 legacy=0x7feeac1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.491+0000 7feeaad9d700 1 -- 172.21.15.29:0/2195593119 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0ad740 con 0x7feeac0b0410 2023-10-07T09:48:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.492+0000 7feeaad9d700 1 --2- 172.21.15.29:0/2195593119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac15a310 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7feea4009f20 tx=0x7feea4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c7b19ec79fc608 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.492+0000 7feea959a700 1 -- 172.21.15.29:0/2195593119 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea400e070 con 0x7feeac0b0410 2023-10-07T09:48:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.492+0000 7feea959a700 1 -- 172.21.15.29:0/2195593119 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea400ba80 con 0x7feeac0b0410 2023-10-07T09:48:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.492+0000 7feea959a700 1 -- 172.21.15.29:0/2195593119 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea40092e0 con 0x7feeac0b0410 2023-10-07T09:48:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 -- 172.21.15.29:0/2195593119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 msgr2=0x7feeac15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 --2- 172.21.15.29:0/2195593119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac15a310 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7feea4009f20 tx=0x7feea4009840 comp rx=0 tx=0).stop 2023-10-07T09:48:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 -- 172.21.15.29:0/2195593119 shutdown_connections 2023-10-07T09:48:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 --2- 172.21.15.29:0/2195593119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac15a310 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 --2- 172.21.15.29:0/2195593119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac0ad370 0x7feeac0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 -- 172.21.15.29:0/2195593119 >> 172.21.15.29:0/2195593119 conn(0x7feeac0680a0 msgr2=0x7feeac0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 -- 172.21.15.29:0/2195593119 shutdown_connections 2023-10-07T09:48:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.493+0000 7feeb0bff700 1 -- 172.21.15.29:0/2195593119 wait complete. 2023-10-07T09:48:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.494+0000 7feeb0bff700 1 Processor -- start 2023-10-07T09:48:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.495+0000 7feeb0bff700 1 -- start start 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.495+0000 7feeb0bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feeac0ad370 0x7feeac069cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.495+0000 7feeb0bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac1495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.495+0000 7feeb0bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac159ec0 0x7feeac148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feeb0bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac14ec40 con 0x7feeac0b0410 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feeb0bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1493c0 con 0x7feeac0ad370 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feeb0bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac149dc0 con 0x7feeac159ec0 2023-10-07T09:48:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac1495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac1495a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46922/0 (socket says 172.21.15.29:46922) 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 -- 172.21.15.29:0/3344836776 learned_addr learned my addr 172.21.15.29:0/3344836776 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feeaad9d700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac159ec0 0x7feeac148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feeaa59c700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feeac0ad370 0x7feeac069cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 -- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac159ec0 msgr2=0x7feeac148e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac159ec0 0x7feeac148e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 -- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feeac0ad370 msgr2=0x7feeac069cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feeac0ad370 0x7feeac069cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feea9d9b700 1 -- 172.21.15.29:0/3344836776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4009ae0 con 0x7feeac0b0410 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.496+0000 7feeaa59c700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feeac0ad370 0x7feeac069cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.497+0000 7feea9d9b700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac1495a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7feea000c4e0 tx=0x7feea000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.497+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0011590 con 0x7feeac0b0410 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.497+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea0011710 con 0x7feeac0b0410 2023-10-07T09:48:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.497+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac06a2a0 con 0x7feeac0b0410 2023-10-07T09:48:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.497+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0011ad0 con 0x7feeac0b0410 2023-10-07T09:48:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.497+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac14e8d0 con 0x7feeac0b0410 2023-10-07T09:48:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.498+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac14edb0 con 0x7feeac0b0410 2023-10-07T09:48:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.499+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feea001b3b0 con 0x7feeac0b0410 2023-10-07T09:48:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.500+0000 7fee9a7fc700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee8c066f30 0x7fee8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.500+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7feea0083b60 con 0x7feeac0b0410 2023-10-07T09:48:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.500+0000 7feeaa59c700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee8c066f30 0x7fee8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.501+0000 7feeaa59c700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee8c066f30 0x7fee8c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006fd0 tx=0x7fee9c008040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:14.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.504+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feea0050ce0 con 0x7feeac0b0410 2023-10-07T09:48:14.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:14.784+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7feeac04c3c0 con 0x7feeac0b0410 2023-10-07T09:48:15.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.275+0000 7fee9a7fc700 1 -- 172.21.15.29:0/3344836776 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v719) v1 ==== 181+0+0 (secure 0 0 0) 0x7feea0017070 con 0x7feeac0b0410 2023-10-07T09:48:15.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee8c066f30 msgr2=0x7fee8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee8c066f30 0x7fee8c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006fd0 tx=0x7fee9c008040 comp rx=0 tx=0).stop 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 msgr2=0x7feeac1495a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac1495a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7feea000c4e0 tx=0x7feea000c8a0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 shutdown_connections 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fee8c066f30 0x7fee8c0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feeac159ec0 0x7feeac148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feeac0ad370 0x7feeac069cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 --2- 172.21.15.29:0/3344836776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feeac0b0410 0x7feeac1495a0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 >> 172.21.15.29:0/3344836776 conn(0x7feeac0680a0 msgr2=0x7feeac14cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 shutdown_connections 2023-10-07T09:48:15.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.277+0000 7feeb0bff700 1 -- 172.21.15.29:0/3344836776 wait complete. 2023-10-07T09:48:15.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T09:48:15.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.499+0000 7fc44d7ff700 1 Processor -- start 2023-10-07T09:48:15.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.499+0000 7fc44d7ff700 1 -- start start 2023-10-07T09:48:15.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.500+0000 7fc44d7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 0x7fc4480afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:15.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.500+0000 7fc44d7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4480ae570 0x7fc4480accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:15.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.500+0000 7fc44d7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4480ad230 con 0x7fc4480b0520 2023-10-07T09:48:15.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.500+0000 7fc44d7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4480ad4d0 con 0x7fc4480ae570 2023-10-07T09:48:15.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.500+0000 7fc44d7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480ad7b0 con 0x7fc448156180 2023-10-07T09:48:15.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 0x7fc4480afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:15.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 0x7fc4480afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44130/0 (socket says 172.21.15.29:44130) 2023-10-07T09:48:15.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc4467fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4480ae570 0x7fc4480accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:15.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc4477fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc4480b0520 0x7fc448159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:49040/0 (socket says 172.21.15.29:49040) 2023-10-07T09:48:15.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 -- 172.21.15.29:0/2983834084 learned_addr learned my addr 172.21.15.29:0/2983834084 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 -- 172.21.15.29:0/2983834084 >> v2:172.21.15.144:3300/0 conn(0x7fc4480ae570 msgr2=0x7fc4480accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 --2- 172.21.15.29:0/2983834084 >> v2:172.21.15.144:3300/0 conn(0x7fc4480ae570 0x7fc4480accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 -- 172.21.15.29:0/2983834084 >> v1:172.21.15.144:6789/0 conn(0x7fc4480b0520 legacy=0x7fc448159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc445ffb700 1 -- 172.21.15.29:0/2983834084 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040356633 0 0) 0x7fc4480ad230 con 0x7fc4480b0520 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc446ffd700 1 -- 172.21.15.29:0/2983834084 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4480ada90 con 0x7fc448156180 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.501+0000 7fc4467fc700 1 --2- 172.21.15.29:0/2983834084 >> v2:172.21.15.144:3300/0 conn(0x7fc4480ae570 0x7fc4480accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.502+0000 7fc446ffd700 1 --2- 172.21.15.29:0/2983834084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 0x7fc4480afc00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc4380099b0 tx=0x7fc438009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ca6bc265a7ad824 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.502+0000 7fc445ffb700 1 -- 172.21.15.29:0/2983834084 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438009ea0 con 0x7fc448156180 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.502+0000 7fc445ffb700 1 -- 172.21.15.29:0/2983834084 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43800b860 con 0x7fc448156180 2023-10-07T09:48:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.502+0000 7fc445ffb700 1 -- 172.21.15.29:0/2983834084 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43800bc00 con 0x7fc448156180 2023-10-07T09:48:15.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 -- 172.21.15.29:0/2983834084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 msgr2=0x7fc4480afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/2983834084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 0x7fc4480afc00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc4380099b0 tx=0x7fc438009250 comp rx=0 tx=0).stop 2023-10-07T09:48:15.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 -- 172.21.15.29:0/2983834084 shutdown_connections 2023-10-07T09:48:15.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/2983834084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc448156180 0x7fc4480afc00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/2983834084 >> v2:172.21.15.144:3300/0 conn(0x7fc4480ae570 0x7fc4480accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 -- 172.21.15.29:0/2983834084 >> 172.21.15.29:0/2983834084 conn(0x7fc448067610 msgr2=0x7fc448067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:15.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 -- 172.21.15.29:0/2983834084 shutdown_connections 2023-10-07T09:48:15.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.503+0000 7fc44d7ff700 1 -- 172.21.15.29:0/2983834084 wait complete. 2023-10-07T09:48:15.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.504+0000 7fc44d7ff700 1 Processor -- start 2023-10-07T09:48:15.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.504+0000 7fc44d7ff700 1 -- start start 2023-10-07T09:48:15.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.505+0000 7fc44d7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4480ae570 0x7fc44814bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:15.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.505+0000 7fc44d7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 0x7fc44816d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:15.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.505+0000 7fc44d7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc448156180 0x7fc4480aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:15.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.505+0000 7fc44d7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44816e270 con 0x7fc4480b0520 2023-10-07T09:48:15.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.505+0000 7fc44d7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44816dcb0 con 0x7fc448156180 2023-10-07T09:48:15.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.505+0000 7fc44d7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44816df90 con 0x7fc4480ae570 2023-10-07T09:48:15.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 0x7fc44816d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:15.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 0x7fc44816d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46928/0 (socket says 172.21.15.29:46928) 2023-10-07T09:48:15.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 -- 172.21.15.29:0/751786734 learned_addr learned my addr 172.21.15.29:0/751786734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:15.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc446ffd700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4480ae570 0x7fc44814bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:15.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4477fe700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc448156180 0x7fc4480aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:15.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 -- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4480ae570 msgr2=0x7fc44814bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4480ae570 0x7fc44814bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 -- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc448156180 msgr2=0x7fc4480aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc448156180 0x7fc4480aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 -- 172.21.15.29:0/751786734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c009420 con 0x7fc4480b0520 2023-10-07T09:48:15.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.506+0000 7fc4467fc700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 0x7fc44816d340 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00ec80 tx=0x7fc43c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:15.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.507+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c00c850 con 0x7fc4480b0520 2023-10-07T09:48:15.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.507+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43c00c9b0 con 0x7fc4480b0520 2023-10-07T09:48:15.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.507+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c00cd70 con 0x7fc4480b0520 2023-10-07T09:48:15.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.507+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4380094f0 con 0x7fc4480b0520 2023-10-07T09:48:15.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.507+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc448156a80 con 0x7fc4480b0520 2023-10-07T09:48:15.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.508+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44804c3c0 con 0x7fc4480b0520 2023-10-07T09:48:15.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.509+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc43c013680 con 0x7fc4480b0520 2023-10-07T09:48:15.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.510+0000 7fc42effd700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc434066f30 0x7fc4340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:15.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.510+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7fc43c084630 con 0x7fc4480b0520 2023-10-07T09:48:15.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.510+0000 7fc446ffd700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc434066f30 0x7fc4340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:15.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.511+0000 7fc446ffd700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc434066f30 0x7fc4340693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc4380099b0 tx=0x7fc438017000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:15.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.514+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc43c051710 con 0x7fc4480b0520 2023-10-07T09:48:15.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.714+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc4480add70 con 0x7fc4480b0520 2023-10-07T09:48:15.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.718+0000 7fc42effd700 1 -- 172.21.15.29:0/751786734 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc43c054d30 con 0x7fc4480b0520 2023-10-07T09:48:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc434066f30 msgr2=0x7fc4340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc434066f30 0x7fc4340693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc4380099b0 tx=0x7fc438017000 comp rx=0 tx=0).stop 2023-10-07T09:48:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 msgr2=0x7fc44816d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 0x7fc44816d340 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00ec80 tx=0x7fc43c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 shutdown_connections 2023-10-07T09:48:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc434066f30 0x7fc4340693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc448156180 0x7fc4480aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4480ae570 0x7fc44814bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 --2- 172.21.15.29:0/751786734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4480b0520 0x7fc44816d340 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 >> 172.21.15.29:0/751786734 conn(0x7fc448067610 msgr2=0x7fc4480aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 shutdown_connections 2023-10-07T09:48:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:15.721+0000 7fc44d7ff700 1 -- 172.21.15.29:0/751786734 wait complete. 2023-10-07T09:48:15.740 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T09:48:15.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:48:15.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T09:48:15.944 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T09:48:15.944 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T09:48:15.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.139+0000 7fae589ff700 1 Processor -- start 2023-10-07T09:48:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.139+0000 7fae589ff700 1 -- start start 2023-10-07T09:48:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae589ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fae54159750 0x7fae540d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae589ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae54159b20 0x7fae540ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae589ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae540cbf20 con 0x7fae540ce6d0 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae589ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540cc200 con 0x7fae54159b20 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae589ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540cc4e0 con 0x7fae54159750 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae51d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae54159b20 0x7fae540ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae51d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae54159b20 0x7fae540ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46932/0 (socket says 172.21.15.29:46932) 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae51d9b700 1 -- 172.21.15.29:0/3986262038 learned_addr learned my addr 172.21.15.29:0/3986262038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae5259c700 1 --2- 172.21.15.29:0/3986262038 >> v2:172.21.15.144:3300/0 conn(0x7fae54159750 0x7fae540d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae51d9b700 1 -- 172.21.15.29:0/3986262038 >> v2:172.21.15.144:3300/0 conn(0x7fae54159750 msgr2=0x7fae540d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae51d9b700 1 --2- 172.21.15.29:0/3986262038 >> v2:172.21.15.144:3300/0 conn(0x7fae54159750 0x7fae540d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.141+0000 7fae51d9b700 1 -- 172.21.15.29:0/3986262038 >> v1:172.21.15.144:6789/0 conn(0x7fae540ce6d0 legacy=0x7fae5415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.142+0000 7fae51d9b700 1 -- 172.21.15.29:0/3986262038 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae540cc7c0 con 0x7fae54159b20 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.142+0000 7fae51d9b700 1 --2- 172.21.15.29:0/3986262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae54159b20 0x7fae540ce190 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fae54089840 tx=0x7fae48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd83523c7b0d8a92 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.142+0000 7fae5159a700 1 -- 172.21.15.29:0/3986262038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800b860 con 0x7fae54159b20 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.142+0000 7fae5159a700 1 -- 172.21.15.29:0/3986262038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4800b9c0 con 0x7fae54159b20 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.142+0000 7fae5159a700 1 -- 172.21.15.29:0/3986262038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800bd60 con 0x7fae54159b20 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 -- 172.21.15.29:0/3986262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae54159b20 msgr2=0x7fae540ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 --2- 172.21.15.29:0/3986262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae54159b20 0x7fae540ce190 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fae54089840 tx=0x7fae48009250 comp rx=0 tx=0).stop 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 -- 172.21.15.29:0/3986262038 shutdown_connections 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 --2- 172.21.15.29:0/3986262038 >> v2:172.21.15.144:3300/0 conn(0x7fae54159750 0x7fae540d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 --2- 172.21.15.29:0/3986262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae54159b20 0x7fae540ce190 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 -- 172.21.15.29:0/3986262038 >> 172.21.15.29:0/3986262038 conn(0x7fae54063a10 msgr2=0x7fae540d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:16.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 -- 172.21.15.29:0/3986262038 shutdown_connections 2023-10-07T09:48:16.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.143+0000 7fae589ff700 1 -- 172.21.15.29:0/3986262038 wait complete. 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.144+0000 7fae589ff700 1 Processor -- start 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.144+0000 7fae589ff700 1 -- start start 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae589ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 0x7fae54144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae589ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae54159750 0x7fae540b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae589ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae54159b20 0x7fae540b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae589ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae54153750 con 0x7fae540ce6d0 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae589ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540b4ac0 con 0x7fae54159b20 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae589ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae540b3f40 con 0x7fae54159750 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.145+0000 7fae5259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 0x7fae54144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 0x7fae54144fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46948/0 (socket says 172.21.15.29:46948) 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 -- 172.21.15.29:0/3338045360 learned_addr learned my addr 172.21.15.29:0/3338045360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae51d9b700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae54159750 0x7fae540b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae52d9d700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae54159b20 0x7fae540b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 -- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae54159750 msgr2=0x7fae540b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae54159750 0x7fae540b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 -- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae54159b20 msgr2=0x7fae540b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae54159b20 0x7fae540b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 -- 172.21.15.29:0/3338045360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae480094f0 con 0x7fae540ce6d0 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae5259c700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 0x7fae54144fb0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fae4400d810 tx=0x7fae44008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.146+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440125f0 con 0x7fae540ce6d0 2023-10-07T09:48:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.147+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae44012750 con 0x7fae540ce6d0 2023-10-07T09:48:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.147+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae540b4220 con 0x7fae540ce6d0 2023-10-07T09:48:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.147+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44012b10 con 0x7fae540ce6d0 2023-10-07T09:48:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.147+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae540b47e0 con 0x7fae540ce6d0 2023-10-07T09:48:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.148+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5415a040 con 0x7fae540ce6d0 2023-10-07T09:48:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.148+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fae4401e460 con 0x7fae540ce6d0 2023-10-07T09:48:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.150+0000 7fae427fc700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fae34066f30 0x7fae340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.150+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7fae44084cb0 con 0x7fae540ce6d0 2023-10-07T09:48:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.150+0000 7fae51d9b700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fae34066f30 0x7fae340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.151+0000 7fae51d9b700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fae34066f30 0x7fae340693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fae54089840 tx=0x7fae48017000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.154+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae44051e30 con 0x7fae540ce6d0 2023-10-07T09:48:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.399+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae5404c3c0 con 0x7fae540ce6d0 2023-10-07T09:48:16.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.400+0000 7fae427fc700 1 -- 172.21.15.29:0/3338045360 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fae44055450 con 0x7fae540ce6d0 2023-10-07T09:48:16.401 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:16.401 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6998,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1922260327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1922260327},{"type":"v1","addr":"172.21.15.29:6835","nonce":1922260327}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":62},{"gid":7336,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3858070486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3858070486},{"type":"v1","addr":"172.21.15.144:6835","nonce":3858070486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55},{"gid":7671,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3029980950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3029980950},{"type":"v1","addr":"172.21.15.29:6834","nonce":3029980950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7674,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/620142282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":620142282},{"type":"v1","addr":"172.21.15.144:6837","nonce":620142282}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":55}],"filesystems":[]} 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fae34066f30 msgr2=0x7fae340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fae34066f30 0x7fae340693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fae54089840 tx=0x7fae48017000 comp rx=0 tx=0).stop 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 msgr2=0x7fae54144fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 0x7fae54144fb0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fae4400d810 tx=0x7fae44008040 comp rx=0 tx=0).stop 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 shutdown_connections 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fae34066f30 0x7fae340693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae54159b20 0x7fae540b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae540ce6d0 0x7fae54144fb0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 --2- 172.21.15.29:0/3338045360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae54159750 0x7fae540b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.402+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 >> 172.21.15.29:0/3338045360 conn(0x7fae54063a10 msgr2=0x7fae54154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.403+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 shutdown_connections 2023-10-07T09:48:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.403+0000 7fae589ff700 1 -- 172.21.15.29:0/3338045360 wait complete. 2023-10-07T09:48:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2023-10-07T09:48:16.421 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T09:48:16.422 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T09:48:16.422 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T09:48:16.422 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:48:16.422 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T09:48:16.422 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T09:48:16.422 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T09:48:16.423 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:48:16.423 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T09:48:16.423 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T09:48:16.423 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T09:48:16.423 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:48:16.423 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T09:48:16.423 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T09:48:16.424 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T09:48:16.424 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T09:48:16.435 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T09:48:16.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T09:48:16.436 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T09:48:16.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T09:48:16.439 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T09:48:16.439 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T09:48:16.440 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T09:48:16.440 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T09:48:16.481 INFO:tasks.ceph.mds.a:Started 2023-10-07T09:48:16.481 DEBUG:teuthology.parallel:result is None 2023-10-07T09:48:16.484 INFO:tasks.ceph.mds.b:Started 2023-10-07T09:48:16.484 DEBUG:teuthology.parallel:result is None 2023-10-07T09:48:16.485 INFO:tasks.ceph.mds.c:Started 2023-10-07T09:48:16.485 DEBUG:teuthology.parallel:result is None 2023-10-07T09:48:16.489 INFO:tasks.ceph.mds.d:Started 2023-10-07T09:48:16.489 DEBUG:teuthology.parallel:result is None 2023-10-07T09:48:16.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T09:48:16.629 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T09:48:16.630 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T09:48:16.632 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T09:48:16.633 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T09:48:16.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.700+0000 7f8c7f5ff700 1 Processor -- start 2023-10-07T09:48:16.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.700+0000 7f8c7f5ff700 1 -- start start 2023-10-07T09:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7f5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78157290 0x7f8c780cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7f5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c780ae8b0 0x7f8c780b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7f5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8c780b1840 0x7f8c780ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7f5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c780b1cc0 con 0x7f8c780ae8b0 2023-10-07T09:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7f5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c78158ad0 con 0x7f8c780b1840 2023-10-07T09:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7f5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7815b140 con 0x7f8c78157290 2023-10-07T09:48:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7cb9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c780ae8b0 0x7f8c780b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7cb9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c780ae8b0 0x7f8c780b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47002/0 (socket says 172.21.15.29:47002) 2023-10-07T09:48:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/4180369351 learned_addr learned my addr 172.21.15.29:0/4180369351 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7d39b700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78157290 0x7f8c780cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7db9c700 1 --2- 172.21.15.29:0/4180369351 >> v2:172.21.15.144:3300/0 conn(0x7f8c780b1840 0x7f8c780ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/4180369351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78157290 msgr2=0x7f8c780cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7cb9a700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78157290 0x7f8c780cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.702+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/4180369351 >> v2:172.21.15.144:3300/0 conn(0x7f8c780b1840 msgr2=0x7f8c780ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c7cb9a700 1 --2- 172.21.15.29:0/4180369351 >> v2:172.21.15.144:3300/0 conn(0x7f8c780b1840 0x7f8c780ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/4180369351 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7815b420 con 0x7f8c780ae8b0 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c7d39b700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78157290 0x7f8c780cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c7db9c700 1 --2- 172.21.15.29:0/4180369351 >> v2:172.21.15.144:3300/0 conn(0x7f8c780b1840 0x7f8c780ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c7cb9a700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c780ae8b0 0x7f8c780b1300 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8c780b2220 tx=0x7f8c68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6ca3d7ad415383 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c6ffff700 1 -- 172.21.15.29:0/4180369351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6800b860 con 0x7f8c780ae8b0 2023-10-07T09:48:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c6ffff700 1 -- 172.21.15.29:0/4180369351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6800b9c0 con 0x7f8c780ae8b0 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.703+0000 7f8c6ffff700 1 -- 172.21.15.29:0/4180369351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6800bd60 con 0x7f8c780ae8b0 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/4180369351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c780ae8b0 msgr2=0x7f8c780b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c780ae8b0 0x7f8c780b1300 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8c780b2220 tx=0x7f8c68009250 comp rx=0 tx=0).stop 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/4180369351 shutdown_connections 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/4180369351 >> v2:172.21.15.144:3300/0 conn(0x7f8c780b1840 0x7f8c780ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78157290 0x7f8c780cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/4180369351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c780ae8b0 0x7f8c780b1300 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/4180369351 >> 172.21.15.29:0/4180369351 conn(0x7f8c780680a0 msgr2=0x7f8c780cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/4180369351 shutdown_connections 2023-10-07T09:48:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.704+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/4180369351 wait complete. 2023-10-07T09:48:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.705+0000 7f8c7f5ff700 1 Processor -- start 2023-10-07T09:48:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.705+0000 7f8c7f5ff700 1 -- start start 2023-10-07T09:48:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.706+0000 7f8c7f5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c780b1840 0x7f8c7806e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.706+0000 7f8c7f5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 0x7f8c78153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.706+0000 7f8c7f5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78154090 0x7f8c7814aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.706+0000 7f8c7f5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78167a70 con 0x7f8c78157290 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.706+0000 7f8c7f5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7814d1e0 con 0x7f8c780b1840 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.706+0000 7f8c7f5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c780c8530 con 0x7f8c78154090 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 0x7f8c78153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7d39b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c780b1840 0x7f8c7806e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 0x7f8c78153b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47014/0 (socket says 172.21.15.29:47014) 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/2720672370 learned_addr learned my addr 172.21.15.29:0/2720672370 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7db9c700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78154090 0x7f8c7814aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78154090 msgr2=0x7f8c7814aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78154090 0x7f8c7814aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c780b1840 msgr2=0x7f8c7806e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c780b1840 0x7f8c7806e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 -- 172.21.15.29:0/2720672370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680094f0 con 0x7f8c78157290 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7d39b700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c780b1840 0x7f8c7806e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.707+0000 7f8c7cb9a700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 0x7f8c78153b50 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8c78068dd0 tx=0x7f8c68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.708+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68011dc0 con 0x7f8c78157290 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.708+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c68024440 con 0x7f8c78157290 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.708+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68024800 con 0x7f8c78157290 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.708+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c781546c0 con 0x7f8c78157290 2023-10-07T09:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.708+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7814ce70 con 0x7f8c78157290 2023-10-07T09:48:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.709+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c78154490 con 0x7f8c78157290 2023-10-07T09:48:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.709+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8c6801a420 con 0x7f8c78157290 2023-10-07T09:48:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.710+0000 7f8c6d7fa700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c5c066f30 0x7f8c5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.710+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5945+0+0 (secure 0 0 0) 0x7f8c680864b0 con 0x7f8c78157290 2023-10-07T09:48:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.711+0000 7f8c7d39b700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c5c066f30 0x7f8c5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.711+0000 7f8c7d39b700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c5c066f30 0x7f8c5c0693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c74006fd0 tx=0x7f8c74010040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.715+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c68053630 con 0x7f8c78157290 2023-10-07T09:48:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:16.931+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8c7814ccd0 con 0x7f8c78157290 2023-10-07T09:48:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.217+0000 7f8c6d7fa700 1 -- 172.21.15.29:0/2720672370 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v129) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8c7814ccd0 con 0x7f8c78157290 2023-10-07T09:48:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.220+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c5c066f30 msgr2=0x7f8c5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.220+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c5c066f30 0x7f8c5c0693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c74006fd0 tx=0x7f8c74010040 comp rx=0 tx=0).stop 2023-10-07T09:48:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.220+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 msgr2=0x7f8c78153b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.220+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 0x7f8c78153b50 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8c78068dd0 tx=0x7f8c68017000 comp rx=0 tx=0).stop 2023-10-07T09:48:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 shutdown_connections 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f8c5c066f30 0x7f8c5c0693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c78154090 0x7f8c7814aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c780b1840 0x7f8c7806e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 --2- 172.21.15.29:0/2720672370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c78157290 0x7f8c78153b50 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 >> 172.21.15.29:0/2720672370 conn(0x7f8c780680a0 msgr2=0x7f8c78159ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 shutdown_connections 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.221+0000 7f8c7f5ff700 1 -- 172.21.15.29:0/2720672370 wait complete. 2023-10-07T09:48:18.223 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T09:48:18.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T09:48:18.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.444+0000 7f9232fff700 1 Processor -- start 2023-10-07T09:48:18.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.445+0000 7f9232fff700 1 -- start start 2023-10-07T09:48:18.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.446+0000 7f9232fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f922c1533f0 0x7f922c156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.446+0000 7f9232fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1537c0 0x7f922c0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f9232fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f922c0af6d0 0x7f922c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f9232fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922c15a680 con 0x7f922c0af6d0 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f9232fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922c0b2070 con 0x7f922c1533f0 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f9232fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c159670 con 0x7f922c1537c0 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f922c0af6d0 0x7f922c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f922c0af6d0 0x7f922c0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47026/0 (socket says 172.21.15.29:47026) 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 -- 172.21.15.29:0/2240942075 learned_addr learned my addr 172.21.15.29:0/2240942075 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:18.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f922bfff700 1 --2- 172.21.15.29:0/2240942075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1537c0 0x7f922c0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f9230d9b700 1 --2- 172.21.15.29:0/2240942075 >> v2:172.21.15.144:3300/0 conn(0x7f922c1533f0 0x7f922c156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 -- 172.21.15.29:0/2240942075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1537c0 msgr2=0x7f922c0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 --2- 172.21.15.29:0/2240942075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1537c0 0x7f922c0afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 -- 172.21.15.29:0/2240942075 >> v2:172.21.15.144:3300/0 conn(0x7f922c1533f0 msgr2=0x7f922c156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 --2- 172.21.15.29:0/2240942075 >> v2:172.21.15.144:3300/0 conn(0x7f922c1533f0 0x7f922c156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f923159c700 1 -- 172.21.15.29:0/2240942075 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c15a1f0 con 0x7f922c0af6d0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.447+0000 7f9230d9b700 1 --2- 172.21.15.29:0/2240942075 >> v2:172.21.15.144:3300/0 conn(0x7f922c1533f0 0x7f922c156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.448+0000 7f923159c700 1 --2- 172.21.15.29:0/2240942075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c0af6d0 0x7f922c0b02e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f922c0b2240 tx=0x7f9224009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f3e3d3481777eb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.448+0000 7f922b7fe700 1 -- 172.21.15.29:0/2240942075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922400e070 con 0x7f922c0af6d0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.448+0000 7f922b7fe700 1 -- 172.21.15.29:0/2240942075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400ba50 con 0x7f922c0af6d0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.448+0000 7f922b7fe700 1 -- 172.21.15.29:0/2240942075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92240092e0 con 0x7f922c0af6d0 2023-10-07T09:48:18.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.448+0000 7f9232fff700 1 -- 172.21.15.29:0/2240942075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c0af6d0 msgr2=0x7f922c0b02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 --2- 172.21.15.29:0/2240942075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c0af6d0 0x7f922c0b02e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f922c0b2240 tx=0x7f9224009840 comp rx=0 tx=0).stop 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 -- 172.21.15.29:0/2240942075 shutdown_connections 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 --2- 172.21.15.29:0/2240942075 >> v2:172.21.15.144:3300/0 conn(0x7f922c1533f0 0x7f922c156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 --2- 172.21.15.29:0/2240942075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c0af6d0 0x7f922c0b02e0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 --2- 172.21.15.29:0/2240942075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1537c0 0x7f922c0afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 -- 172.21.15.29:0/2240942075 >> 172.21.15.29:0/2240942075 conn(0x7f922c0680a0 msgr2=0x7f922c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 -- 172.21.15.29:0/2240942075 shutdown_connections 2023-10-07T09:48:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.449+0000 7f9232fff700 1 -- 172.21.15.29:0/2240942075 wait complete. 2023-10-07T09:48:18.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.450+0000 7f9232fff700 1 Processor -- start 2023-10-07T09:48:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.450+0000 7f9232fff700 1 -- start start 2023-10-07T09:48:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.451+0000 7f9232fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f922c0af6d0 0x7f922c13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.451+0000 7f9232fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1533f0 0x7f922c1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.451+0000 7f9232fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 0x7f922c158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.451+0000 7f9232fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c16aa50 con 0x7f922c1537c0 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.451+0000 7f9232fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c16acc0 con 0x7f922c0af6d0 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.451+0000 7f9232fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c16afa0 con 0x7f922c1533f0 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 0x7f922c158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 0x7f922c158720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47028/0 (socket says 172.21.15.29:47028) 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 -- 172.21.15.29:0/2291685848 learned_addr learned my addr 172.21.15.29:0/2291685848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f9230d9b700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f922c0af6d0 0x7f922c13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f922bfff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1533f0 0x7f922c1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 -- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1533f0 msgr2=0x7f922c1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1533f0 0x7f922c1581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 -- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f922c0af6d0 msgr2=0x7f922c13fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f922c0af6d0 0x7f922c13fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 -- 172.21.15.29:0/2291685848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9224009ae0 con 0x7f922c1537c0 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f922bfff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1533f0 0x7f922c1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f9230d9b700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f922c0af6d0 0x7f922c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.452+0000 7f923159c700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 0x7f922c158720 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9224009fd0 tx=0x7f9224018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.453+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92240092e0 con 0x7f922c1537c0 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.453+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922401d420 con 0x7f922c1537c0 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.453+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922401d7e0 con 0x7f922c1537c0 2023-10-07T09:48:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.453+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c16b280 con 0x7f922c1537c0 2023-10-07T09:48:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.453+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c16b840 con 0x7f922c1537c0 2023-10-07T09:48:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.454+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c04c3c0 con 0x7f922c1537c0 2023-10-07T09:48:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.455+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9224003050 con 0x7f922c1537c0 2023-10-07T09:48:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.455+0000 7f9228ff9700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9210066f00 0x7f92100693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.456+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f9224086170 con 0x7f922c1537c0 2023-10-07T09:48:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.456+0000 7f9230d9b700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9210066f00 0x7f92100693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.457+0000 7f9230d9b700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9210066f00 0x7f92100693b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9220006f00 tx=0x7f9220008040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.460+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9224053520 con 0x7f922c1537c0 2023-10-07T09:48:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.743+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f922c06f660 con 0x7f922c1537c0 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.744+0000 7f9228ff9700 1 -- 172.21.15.29:0/2291685848 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f922405c090 con 0x7f922c1537c0 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T09:48:18.745 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.748 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.749 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T09:48:18.752 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.753 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.755 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:48:18.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.757 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:18.760 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T09:48:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9210066f00 msgr2=0x7f92100693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9210066f00 0x7f92100693b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9220006f00 tx=0x7f9220008040 comp rx=0 tx=0).stop 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 msgr2=0x7f922c158720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 0x7f922c158720 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9224009fd0 tx=0x7f9224018000 comp rx=0 tx=0).stop 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 shutdown_connections 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9210066f00 0x7f92100693b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f922c1537c0 0x7f922c158720 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f922c0af6d0 0x7f922c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 --2- 172.21.15.29:0/2291685848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f922c1533f0 0x7f922c1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 >> 172.21.15.29:0/2291685848 conn(0x7f922c0680a0 msgr2=0x7f922c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 shutdown_connections 2023-10-07T09:48:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.746+0000 7f9232fff700 1 -- 172.21.15.29:0/2291685848 wait complete. 2023-10-07T09:48:18.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.960+0000 7f6e91dff700 1 Processor -- start 2023-10-07T09:48:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.960+0000 7f6e91dff700 1 -- start start 2023-10-07T09:48:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.961+0000 7f6e91dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 0x7f6e8c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.961+0000 7f6e91dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 0x7f6e8c157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e91dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c158ad0 con 0x7f6e8c0ae8b0 2023-10-07T09:48:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e91dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8c15b140 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e91dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c15b420 con 0x7f6e8c157290 2023-10-07T09:48:18.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e8bfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 0x7f6e8c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e8b7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 0x7f6e8c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e8bfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 0x7f6e8c157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:51006/0 (socket says 172.21.15.29:51006) 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e8b7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 0x7f6e8c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44198/0 (socket says 172.21.15.29:44198) 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.962+0000 7f6e8bfff700 1 -- 172.21.15.29:0/3523143473 learned_addr learned my addr 172.21.15.29:0/3523143473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/3523143473 >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 msgr2=0x7f6e8c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8b7fe700 1 --2- 172.21.15.29:0/3523143473 >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 0x7f6e8c157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/3523143473 >> v1:172.21.15.144:6789/0 conn(0x7f6e8c0ae8b0 legacy=0x7f6e8c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/3523143473 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c15b700 con 0x7f6e8c157290 2023-10-07T09:48:18.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8bfff700 1 --2- 172.21.15.29:0/3523143473 >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 0x7f6e8c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:18.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8b7fe700 1 --2- 172.21.15.29:0/3523143473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 0x7f6e8c0cc870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0098e0 tx=0x7f6e7c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f3ed7a190835111 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:18.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.963+0000 7f6e8a7fc700 1 -- 172.21.15.29:0/3523143473 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00b640 con 0x7f6e8c157290 2023-10-07T09:48:18.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e8a7fc700 1 -- 172.21.15.29:0/3523143473 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c00b7a0 con 0x7f6e8c157290 2023-10-07T09:48:18.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e8a7fc700 1 -- 172.21.15.29:0/3523143473 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00bb40 con 0x7f6e8c157290 2023-10-07T09:48:18.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 -- 172.21.15.29:0/3523143473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 msgr2=0x7f6e8c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 --2- 172.21.15.29:0/3523143473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 0x7f6e8c0cc870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0098e0 tx=0x7f6e7c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 -- 172.21.15.29:0/3523143473 shutdown_connections 2023-10-07T09:48:18.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 --2- 172.21.15.29:0/3523143473 >> v2:172.21.15.144:3300/0 conn(0x7f6e8c0b1a10 0x7f6e8c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 --2- 172.21.15.29:0/3523143473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c157290 0x7f6e8c0cc870 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 -- 172.21.15.29:0/3523143473 >> 172.21.15.29:0/3523143473 conn(0x7f6e8c0680a0 msgr2=0x7f6e8c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:18.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 -- 172.21.15.29:0/3523143473 shutdown_connections 2023-10-07T09:48:18.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.964+0000 7f6e91dff700 1 -- 172.21.15.29:0/3523143473 wait complete. 2023-10-07T09:48:18.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.966+0000 7f6e91dff700 1 Processor -- start 2023-10-07T09:48:18.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.966+0000 7f6e91dff700 1 -- start start 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e91dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 0x7f6e8c0b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e91dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e8c157290 0x7f6e8c0afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e91dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c14ac70 0x7f6e8c159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e91dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c0b01c0 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e91dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c14c200 con 0x7f6e8c157290 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e91dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c14b910 con 0x7f6e8c14ac70 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 0x7f6e8c0b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 0x7f6e8c0b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47036/0 (socket says 172.21.15.29:47036) 2023-10-07T09:48:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/1113735896 learned_addr learned my addr 172.21.15.29:0/1113735896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8affd700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e8c157290 0x7f6e8c0afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8bfff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c14ac70 0x7f6e8c159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c14ac70 msgr2=0x7f6e8c159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c14ac70 0x7f6e8c159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e8c157290 msgr2=0x7f6e8c0afc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e8c157290 0x7f6e8c0afc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.967+0000 7f6e8b7fe700 1 -- 172.21.15.29:0/1113735896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84009ae0 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.968+0000 7f6e8b7fe700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 0x7f6e8c0b09c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00a010 tx=0x7f6e7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.968+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c01bcc0 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.968+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c01be20 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.968+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c009420 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.968+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c011670 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.968+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c14bbb0 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.969+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c14c4e0 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.970+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6e7c011a60 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.971+0000 7f6e737fe700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6e78066f30 0x7f6e780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.971+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f6e7c085760 con 0x7f6e8c0b1a10 2023-10-07T09:48:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.972+0000 7f6e8affd700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6e78066f30 0x7f6e780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.972+0000 7f6e8affd700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6e78066f30 0x7f6e780693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c16d2c0 tx=0x7f6e80007ec0 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:18.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:18.975+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e7c052b90 con 0x7f6e8c0b1a10 2023-10-07T09:48:19.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.182+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e8c04c3c0 con 0x7f6e8c0b1a10 2023-10-07T09:48:19.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.183+0000 7f6e737fe700 1 -- 172.21.15.29:0/1113735896 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+12811 (secure 0 0 0) 0x7f6e7c0561b0 con 0x7f6e8c0b1a10 2023-10-07T09:48:19.184 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:19.184 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:48:18.213871+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6e78066f30 msgr2=0x7f6e780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6e78066f30 0x7f6e780693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c16d2c0 tx=0x7f6e80007ec0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 msgr2=0x7f6e8c0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 0x7f6e8c0b09c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00a010 tx=0x7f6e7c017000 comp rx=0 tx=0).stop 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 shutdown_connections 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6e78066f30 0x7f6e780693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e8c14ac70 0x7f6e8c159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0b1a10 0x7f6e8c0b09c0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 --2- 172.21.15.29:0/1113735896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e8c157290 0x7f6e8c0afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 >> 172.21.15.29:0/1113735896 conn(0x7f6e8c0680a0 msgr2=0x7f6e8c158f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 shutdown_connections 2023-10-07T09:48:19.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.185+0000 7f6e91dff700 1 -- 172.21.15.29:0/1113735896 wait complete. 2023-10-07T09:48:19.204 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T09:48:19.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T09:48:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.401+0000 7f660d9ff700 1 Processor -- start 2023-10-07T09:48:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.401+0000 7f660d9ff700 1 -- start start 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f660d9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6608159b20 0x7f66080d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f660d9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f66080ce8a0 0x7f66080c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f660d9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66080cbf20 con 0x7f6608159750 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f660d9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66080cc200 con 0x7f6608159b20 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f660d9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66080cc4e0 con 0x7f66080ce8a0 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f66067fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6608159b20 0x7f66080d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f66067fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6608159b20 0x7f66080d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47048/0 (socket says 172.21.15.29:47048) 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f66067fc700 1 -- 172.21.15.29:0/829877687 learned_addr learned my addr 172.21.15.29:0/829877687 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f66077fe700 1 --2- 172.21.15.29:0/829877687 >> v2:172.21.15.144:3300/0 conn(0x7f66080ce8a0 0x7f66080c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f66067fc700 1 -- 172.21.15.29:0/829877687 >> v2:172.21.15.144:3300/0 conn(0x7f66080ce8a0 msgr2=0x7f66080c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.403+0000 7f66067fc700 1 --2- 172.21.15.29:0/829877687 >> v2:172.21.15.144:3300/0 conn(0x7f66080ce8a0 0x7f66080c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.404+0000 7f66067fc700 1 -- 172.21.15.29:0/829877687 >> v1:172.21.15.144:6789/0 conn(0x7f6608159750 legacy=0x7f66080ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.404+0000 7f66067fc700 1 -- 172.21.15.29:0/829877687 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66080cc7c0 con 0x7f6608159b20 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.404+0000 7f66067fc700 1 --2- 172.21.15.29:0/829877687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159b20 0x7f66080d10a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f660815e510 tx=0x7f65fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2071f1ab2f9b0102 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.404+0000 7f6605ffb700 1 -- 172.21.15.29:0/829877687 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc00b820 con 0x7f6608159b20 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.404+0000 7f6605ffb700 1 -- 172.21.15.29:0/829877687 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65fc00b980 con 0x7f6608159b20 2023-10-07T09:48:19.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.404+0000 7f6605ffb700 1 -- 172.21.15.29:0/829877687 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc00bd20 con 0x7f6608159b20 2023-10-07T09:48:19.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 -- 172.21.15.29:0/829877687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159b20 msgr2=0x7f66080d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 --2- 172.21.15.29:0/829877687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159b20 0x7f66080d10a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f660815e510 tx=0x7f65fc009250 comp rx=0 tx=0).stop 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 -- 172.21.15.29:0/829877687 shutdown_connections 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 --2- 172.21.15.29:0/829877687 >> v2:172.21.15.144:3300/0 conn(0x7f66080ce8a0 0x7f66080c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 --2- 172.21.15.29:0/829877687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159b20 0x7f66080d10a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 -- 172.21.15.29:0/829877687 >> 172.21.15.29:0/829877687 conn(0x7f6608063a10 msgr2=0x7f66080d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 -- 172.21.15.29:0/829877687 shutdown_connections 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.405+0000 7f660d9ff700 1 -- 172.21.15.29:0/829877687 wait complete. 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.406+0000 7f660d9ff700 1 Processor -- start 2023-10-07T09:48:19.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 -- start start 2023-10-07T09:48:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66080ce8a0 0x7f6608152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 0x7f66080c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6608159b20 0x7f6608155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66080ca150 con 0x7f6608159750 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66080ca430 con 0x7f6608159b20 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.407+0000 7f660d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66080c7710 con 0x7f66080ce8a0 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 0x7f66080c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 0x7f66080c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47062/0 (socket says 172.21.15.29:47062) 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 -- 172.21.15.29:0/1147769899 learned_addr learned my addr 172.21.15.29:0/1147769899 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66077fe700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6608159b20 0x7f6608155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f6606ffd700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66080ce8a0 0x7f6608152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 -- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66080ce8a0 msgr2=0x7f6608152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66080ce8a0 0x7f6608152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 -- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6608159b20 msgr2=0x7f6608155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6608159b20 0x7f6608155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 -- 172.21.15.29:0/1147769899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc0094f0 con 0x7f6608159750 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66077fe700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6608159b20 0x7f6608155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.408+0000 7f66067fc700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 0x7f66080c9770 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0030c0 tx=0x7f65fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.409+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc011e60 con 0x7f6608159750 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.409+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65fc020070 con 0x7f6608159750 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.409+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc01b6a0 con 0x7f6608159750 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.409+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66080c79f0 con 0x7f6608159750 2023-10-07T09:48:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.409+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6608152c90 con 0x7f6608159750 2023-10-07T09:48:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.410+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66080c7f10 con 0x7f6608159750 2023-10-07T09:48:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.411+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f65fc022030 con 0x7f6608159750 2023-10-07T09:48:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.413+0000 7f65eeffd700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f65f4066f30 0x7f65f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.413+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f65fc086390 con 0x7f6608159750 2023-10-07T09:48:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.413+0000 7f6606ffd700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f65f4066f30 0x7f65f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.414+0000 7f6606ffd700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f65f4066f30 0x7f65f40693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65f80038a0 tx=0x7f65f8009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:19.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.417+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66080c7f10 con 0x7f6608159750 2023-10-07T09:48:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.619+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66080c7f10 con 0x7f6608159750 2023-10-07T09:48:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.619+0000 7f65eeffd700 1 -- 172.21.15.29:0/1147769899 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66080c7f10 con 0x7f6608159750 2023-10-07T09:48:19.620 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T09:48:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f65f4066f30 msgr2=0x7f65f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f65f4066f30 0x7f65f40693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65f80038a0 tx=0x7f65f8009040 comp rx=0 tx=0).stop 2023-10-07T09:48:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 msgr2=0x7f66080c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 0x7f66080c9770 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0030c0 tx=0x7f65fc016000 comp rx=0 tx=0).stop 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 shutdown_connections 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f65f4066f30 0x7f65f40693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6608159b20 0x7f6608155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66080ce8a0 0x7f6608152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 --2- 172.21.15.29:0/1147769899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6608159750 0x7f66080c9770 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 >> 172.21.15.29:0/1147769899 conn(0x7f6608063a10 msgr2=0x7f66080cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 shutdown_connections 2023-10-07T09:48:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.622+0000 7f660d9ff700 1 -- 172.21.15.29:0/1147769899 wait complete. 2023-10-07T09:48:19.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T09:48:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.836+0000 7ff0059ff700 1 Processor -- start 2023-10-07T09:48:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.836+0000 7ff0059ff700 1 -- start start 2023-10-07T09:48:19.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.837+0000 7ff0059ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff000153cc0 0x7ff0000b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.837+0000 7ff0059ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff00015ade0 0x7ff0000b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.837+0000 7ff0059ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff000154090 con 0x7ff0000b0000 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.837+0000 7ff0059ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff000159f60 con 0x7ff000153cc0 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.837+0000 7ff0059ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00015a240 con 0x7ff00015ade0 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.837+0000 7feffeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff000153cc0 0x7ff0000b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff000153cc0 0x7ff0000b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47064/0 (socket says 172.21.15.29:47064) 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 -- 172.21.15.29:0/3587116203 learned_addr learned my addr 172.21.15.29:0/3587116203 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffe7fc700 1 --2- 172.21.15.29:0/3587116203 >> v2:172.21.15.144:3300/0 conn(0x7ff00015ade0 0x7ff0000b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 -- 172.21.15.29:0/3587116203 >> v2:172.21.15.144:3300/0 conn(0x7ff00015ade0 msgr2=0x7ff0000b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 --2- 172.21.15.29:0/3587116203 >> v2:172.21.15.144:3300/0 conn(0x7ff00015ade0 0x7ff0000b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 -- 172.21.15.29:0/3587116203 >> v1:172.21.15.144:6789/0 conn(0x7ff0000b0000 legacy=0x7ff00015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 -- 172.21.15.29:0/3587116203 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00015a520 con 0x7ff000153cc0 2023-10-07T09:48:19.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffeffd700 1 --2- 172.21.15.29:0/3587116203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000153cc0 0x7ff0000b17c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff0000add60 tx=0x7feff0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bdb2deef6038a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffdffb700 1 -- 172.21.15.29:0/3587116203 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0005c10 con 0x7ff000153cc0 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.838+0000 7feffdffb700 1 -- 172.21.15.29:0/3587116203 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0005d70 con 0x7ff000153cc0 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7feffdffb700 1 -- 172.21.15.29:0/3587116203 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff000b840 con 0x7ff000153cc0 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 -- 172.21.15.29:0/3587116203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000153cc0 msgr2=0x7ff0000b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 --2- 172.21.15.29:0/3587116203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000153cc0 0x7ff0000b17c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff0000add60 tx=0x7feff0009220 comp rx=0 tx=0).stop 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 -- 172.21.15.29:0/3587116203 shutdown_connections 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 --2- 172.21.15.29:0/3587116203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000153cc0 0x7ff0000b17c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 --2- 172.21.15.29:0/3587116203 >> v2:172.21.15.144:3300/0 conn(0x7ff00015ade0 0x7ff0000b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 -- 172.21.15.29:0/3587116203 >> 172.21.15.29:0/3587116203 conn(0x7ff000067610 msgr2=0x7ff000067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:19.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 -- 172.21.15.29:0/3587116203 shutdown_connections 2023-10-07T09:48:19.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.839+0000 7ff0059ff700 1 -- 172.21.15.29:0/3587116203 wait complete. 2023-10-07T09:48:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.841+0000 7ff0059ff700 1 Processor -- start 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.841+0000 7ff0059ff700 1 -- start start 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7ff0059ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0000b0000 0x7ff0000b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7ff0059ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff000153cc0 0x7ff000151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7ff0059ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 0x7ff0001523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7ff0059ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff000152930 con 0x7ff00015ade0 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7ff0059ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00016dbc0 con 0x7ff0000b0000 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7ff0059ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00016cff0 con 0x7ff000153cc0 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 0x7ff0001523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 0x7ff0001523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47078/0 (socket says 172.21.15.29:47078) 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 -- 172.21.15.29:0/540253953 learned_addr learned my addr 172.21.15.29:0/540253953 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7feffeffd700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0000b0000 0x7ff0000b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7feffe7fc700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff000153cc0 0x7ff000151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 -- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff000153cc0 msgr2=0x7ff000151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff000153cc0 0x7ff000151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 -- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0000b0000 msgr2=0x7ff0000b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0000b0000 0x7ff0000b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.842+0000 7fefff7fe700 1 -- 172.21.15.29:0/540253953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0009480 con 0x7ff00015ade0 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.843+0000 7fefff7fe700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 0x7ff0001523f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7feff800c930 tx=0x7feff80026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.843+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8012630 con 0x7ff00015ade0 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.843+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff80127d0 con 0x7ff00015ade0 2023-10-07T09:48:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.843+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8012b90 con 0x7ff00015ade0 2023-10-07T09:48:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.843+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00016d290 con 0x7ff00015ade0 2023-10-07T09:48:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.843+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00016d850 con 0x7ff00015ade0 2023-10-07T09:48:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.844+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00016dd70 con 0x7ff00015ade0 2023-10-07T09:48:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.845+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feff801b400 con 0x7ff00015ade0 2023-10-07T09:48:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.846+0000 7fefe6ffd700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fefec066f30 0x7fefec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.846+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5229+0+0 (secure 0 0 0) 0x7feff8084610 con 0x7ff00015ade0 2023-10-07T09:48:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.847+0000 7feffeffd700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fefec066f30 0x7fefec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.847+0000 7feffeffd700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fefec066f30 0x7fefec0693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feff0003080 tx=0x7feff0015000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:19.850+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff00016dd70 con 0x7ff00015ade0 2023-10-07T09:48:20.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.054+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff00016dd30 con 0x7ff00015ade0 2023-10-07T09:48:20.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.054+0000 7fefe6ffd700 1 -- 172.21.15.29:0/540253953 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v129) v1 ==== 84+0+65 (secure 0 0 0) 0x7feff8055450 con 0x7ff00015ade0 2023-10-07T09:48:20.055 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:20.056 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:48:20.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:20.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T09:48:20.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T09:48:20.056 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:20.056 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fefec066f30 msgr2=0x7fefec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fefec066f30 0x7fefec0693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feff0003080 tx=0x7feff0015000 comp rx=0 tx=0).stop 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 msgr2=0x7ff0001523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 0x7ff0001523f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7feff800c930 tx=0x7feff80026e0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 shutdown_connections 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fefec066f30 0x7fefec0693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff00015ade0 0x7ff0001523f0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0000b0000 0x7ff0000b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 --2- 172.21.15.29:0/540253953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff000153cc0 0x7ff000151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 >> 172.21.15.29:0/540253953 conn(0x7ff000067610 msgr2=0x7ff0000af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.056+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 shutdown_connections 2023-10-07T09:48:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.057+0000 7ff0059ff700 1 -- 172.21.15.29:0/540253953 wait complete. 2023-10-07T09:48:20.075 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T09:48:20.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T09:48:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.274+0000 7f81c7bff700 1 Processor -- start 2023-10-07T09:48:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.275+0000 7f81c7bff700 1 -- start start 2023-10-07T09:48:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c7bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c00b19a0 0x7f81c00cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c7bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c00aeae0 con 0x7f81c0157a20 2023-10-07T09:48:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c7bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c00ca430 con 0x7f81c0157650 2023-10-07T09:48:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c7bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00ca710 con 0x7f81c00b19a0 2023-10-07T09:48:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c519a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f81c0157a20 0x7f81c00b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59174/0 (socket says 172.21.15.29:59174) 2023-10-07T09:48:20.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c519a700 1 -- 172.21.15.29:0/3456842310 learned_addr learned my addr 172.21.15.29:0/3456842310 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:20.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c619c700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c00b19a0 0x7f81c00cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.276+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764296502 0 0) 0x7f81c00aeae0 con 0x7f81c0157a20 2023-10-07T09:48:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac003570 con 0x7f81c0157a20 2023-10-07T09:48:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852377753 0 0) 0x7f81c00ca430 con 0x7f81c0157650 2023-10-07T09:48:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c00aeae0 con 0x7f81c0157650 2023-10-07T09:48:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3059196237 0 0) 0x7f81ac003570 con 0x7f81c0157a20 2023-10-07T09:48:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c00b19a0 msgr2=0x7f81c00cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c00b19a0 0x7f81c00cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 >> v1:172.21.15.144:6789/0 conn(0x7f81c0157650 legacy=0x7f81c00ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c00ca9f0 con 0x7f81c0157a20 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f81b0001470 con 0x7f81c0157a20 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 >> v1:172.21.15.29:6789/0 conn(0x7f81c0157a20 legacy=0x7f81c00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81ac004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81ac004810 0x7f81ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81ac005210 0x7f81ac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00ca710 con 0x7f81ac003c20 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac00c160 con 0x7f81ac004810 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac00c320 con 0x7f81ac005210 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c619c700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c00b19a0 0x7f81c00cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c599b700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81ac004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.277+0000 7f81c519a700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81ac004810 0x7f81ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c619c700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81ac005210 0x7f81ac00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c599b700 1 -- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81ac005210 msgr2=0x7f81ac00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c599b700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81ac005210 0x7f81ac00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c599b700 1 -- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81ac004810 msgr2=0x7f81ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c599b700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81ac004810 0x7f81ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c599b700 1 -- 172.21.15.29:0/3456842310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac003570 con 0x7f81ac003c20 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c519a700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81ac004810 0x7f81ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c599b700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81ac004010 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0019e0 tx=0x7f81bc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdc1c8afa3ed437 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00c5d0 con 0x7f81ac003c20 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc00c730 con 0x7f81ac003c20 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.278+0000 7f81c4999700 1 -- 172.21.15.29:0/3456842310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00cad0 con 0x7f81ac003c20 2023-10-07T09:48:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c7bff700 1 -- 172.21.15.29:0/3456842310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 msgr2=0x7f81ac004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c7bff700 1 --2- 172.21.15.29:0/3456842310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81ac004010 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0019e0 tx=0x7f81bc007d30 comp rx=0 tx=0).stop 2023-10-07T09:48:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c619c700 1 -- 172.21.15.29:0/3456842310 reap_dead start 2023-10-07T09:48:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c7bff700 1 -- 172.21.15.29:0/3456842310 shutdown_connections 2023-10-07T09:48:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c7bff700 1 -- 172.21.15.29:0/3456842310 >> 172.21.15.29:0/3456842310 conn(0x7f81c0067610 msgr2=0x7f81c0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c7bff700 1 -- 172.21.15.29:0/3456842310 shutdown_connections 2023-10-07T09:48:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.279+0000 7f81c7bff700 1 -- 172.21.15.29:0/3456842310 wait complete. 2023-10-07T09:48:20.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.280+0000 7f81c7bff700 1 Processor -- start 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 -- start start 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81c00af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c0151150 0x7f81c0149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81c006dd20 0x7f81c006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c006e700 con 0x7f81ac003c20 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00b0270 con 0x7f81c006dd20 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.281+0000 7f81c7bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c0152180 con 0x7f81c0151150 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81c00af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81c00af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47084/0 (socket says 172.21.15.29:47084) 2023-10-07T09:48:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 -- 172.21.15.29:0/2963807381 learned_addr learned my addr 172.21.15.29:0/2963807381 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c619c700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81c006dd20 0x7f81c006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c519a700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c0151150 0x7f81c0149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 -- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c0151150 msgr2=0x7f81c0149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c0151150 0x7f81c0149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 -- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81c006dd20 msgr2=0x7f81c006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81c006dd20 0x7f81c006e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 -- 172.21.15.29:0/2963807381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c00af660 con 0x7f81ac003c20 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.282+0000 7f81c599b700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81c00af080 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c3c0 tx=0x7f81bc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.283+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc023070 con 0x7f81ac003c20 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.283+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc00c3f0 con 0x7f81ac003c20 2023-10-07T09:48:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.283+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00cad0 con 0x7f81ac003c20 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.283+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c00af940 con 0x7f81ac003c20 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.283+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c00aff00 con 0x7f81ac003c20 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.284+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c00718b0 con 0x7f81ac003c20 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.285+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81bc019070 con 0x7f81ac003c20 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.286+0000 7f81b5ffb700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a4066f30 0x7f81a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.286+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f81bc085110 con 0x7f81ac003c20 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.286+0000 7f81c519a700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a4066f30 0x7f81a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.287+0000 7f81c519a700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a4066f30 0x7f81a40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000e10 tx=0x7f81b0009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:20.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.290+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81bc052560 con 0x7f81ac003c20 2023-10-07T09:48:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:20.495+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f81c00cacd0 con 0x7f81ac003c20 2023-10-07T09:48:21.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.309+0000 7f81b5ffb700 1 -- 172.21.15.29:0/2963807381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v130) v1 ==== 138+0+0 (secure 0 0 0) 0x7f81bc017070 con 0x7f81ac003c20 2023-10-07T09:48:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.314+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a4066f30 msgr2=0x7f81a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a4066f30 0x7f81a40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000e10 tx=0x7f81b0009040 comp rx=0 tx=0).stop 2023-10-07T09:48:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 msgr2=0x7f81c00af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81c00af080 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c3c0 tx=0x7f81bc015000 comp rx=0 tx=0).stop 2023-10-07T09:48:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 shutdown_connections 2023-10-07T09:48:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f81a4066f30 0x7f81a40693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81c006dd20 0x7f81c006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81ac003c20 0x7f81c00af080 secure :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c3c0 tx=0x7f81bc015000 comp rx=0 tx=0).stop 2023-10-07T09:48:21.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.315+0000 7f81c7bff700 1 --2- 172.21.15.29:0/2963807381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81c0151150 0x7f81c0149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.316+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 >> 172.21.15.29:0/2963807381 conn(0x7f81c0067610 msgr2=0x7f81c00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.316+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 shutdown_connections 2023-10-07T09:48:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.316+0000 7f81c7bff700 1 -- 172.21.15.29:0/2963807381 wait complete. 2023-10-07T09:48:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T09:48:21.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T09:48:21.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.548+0000 7f58f6fff700 1 Processor -- start 2023-10-07T09:48:21.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.548+0000 7f58f6fff700 1 -- start start 2023-10-07T09:48:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f6fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58f00cb800 0x7f58f00cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f6fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f58f00cfbb0 0x7f58f00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00c6760 0x7f58f00c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f6fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f00c62d0 con 0x7f58f00cb800 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f6fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f0159e10 con 0x7f58f00cfbb0 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00cc110 con 0x7f58f00c6760 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.549+0000 7f58f4d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58f00cb800 0x7f58f00cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58f00cb800 0x7f58f00cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47092/0 (socket says 172.21.15.29:47092) 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 -- 172.21.15.29:0/2790238972 learned_addr learned my addr 172.21.15.29:0/2790238972 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f559c700 1 --2- 172.21.15.29:0/2790238972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00c6760 0x7f58f00c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58effff700 1 --2- 172.21.15.29:0/2790238972 >> v2:172.21.15.144:3300/0 conn(0x7f58f00cfbb0 0x7f58f00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 -- 172.21.15.29:0/2790238972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00c6760 msgr2=0x7f58f00c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 --2- 172.21.15.29:0/2790238972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00c6760 0x7f58f00c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 -- 172.21.15.29:0/2790238972 >> v2:172.21.15.144:3300/0 conn(0x7f58f00cfbb0 msgr2=0x7f58f00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 --2- 172.21.15.29:0/2790238972 >> v2:172.21.15.144:3300/0 conn(0x7f58f00cfbb0 0x7f58f00d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 -- 172.21.15.29:0/2790238972 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00c56c0 con 0x7f58f00cb800 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58f4d9b700 1 --2- 172.21.15.29:0/2790238972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cb800 0x7f58f00cbbd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f58f0067610 tx=0x7f58e4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a75783cc2a623 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.550+0000 7f58ef7fe700 1 -- 172.21.15.29:0/2790238972 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4005c10 con 0x7f58f00cb800 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58ef7fe700 1 -- 172.21.15.29:0/2790238972 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e4005d70 con 0x7f58f00cb800 2023-10-07T09:48:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58ef7fe700 1 -- 172.21.15.29:0/2790238972 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e400b910 con 0x7f58f00cb800 2023-10-07T09:48:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 -- 172.21.15.29:0/2790238972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cb800 msgr2=0x7f58f00cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2790238972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cb800 0x7f58f00cbbd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f58f0067610 tx=0x7f58e4009220 comp rx=0 tx=0).stop 2023-10-07T09:48:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 -- 172.21.15.29:0/2790238972 shutdown_connections 2023-10-07T09:48:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2790238972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00c6760 0x7f58f00c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2790238972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cb800 0x7f58f00cbbd0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2790238972 >> v2:172.21.15.144:3300/0 conn(0x7f58f00cfbb0 0x7f58f00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 -- 172.21.15.29:0/2790238972 >> 172.21.15.29:0/2790238972 conn(0x7f58f0063a10 msgr2=0x7f58f00d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 -- 172.21.15.29:0/2790238972 shutdown_connections 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.551+0000 7f58f6fff700 1 -- 172.21.15.29:0/2790238972 wait complete. 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.553+0000 7f58f6fff700 1 Processor -- start 2023-10-07T09:48:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.553+0000 7f58f6fff700 1 -- start start 2023-10-07T09:48:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f6fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f00c6760 0x7f58f0155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00cb800 0x7f58f0169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f6fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 0x7f58f00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f6fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f009e190 con 0x7f58f00cfbb0 2023-10-07T09:48:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f6fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00c8000 con 0x7f58f00c6760 2023-10-07T09:48:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f0155b80 con 0x7f58f00cb800 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f559c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 0x7f58f00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f559c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 0x7f58f00c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47094/0 (socket says 172.21.15.29:47094) 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f559c700 1 -- 172.21.15.29:0/2368331565 learned_addr learned my addr 172.21.15.29:0/2368331565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58f4d9b700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f00c6760 0x7f58f0155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.554+0000 7f58effff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00cb800 0x7f58f0169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f559c700 1 -- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00cb800 msgr2=0x7f58f0169cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f559c700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00cb800 0x7f58f0169cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f559c700 1 -- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f00c6760 msgr2=0x7f58f0155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f559c700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f00c6760 0x7f58f0155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f559c700 1 -- 172.21.15.29:0/2368331565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4009480 con 0x7f58f00cfbb0 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f559c700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 0x7f58f00c9be0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009fd0 tx=0x7f58e8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e800ed50 con 0x7f58f00cfbb0 2023-10-07T09:48:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e800eeb0 con 0x7f58f00cfbb0 2023-10-07T09:48:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f016a200 con 0x7f58f00cfbb0 2023-10-07T09:48:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8014650 con 0x7f58f00cfbb0 2023-10-07T09:48:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.555+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f00c76d0 con 0x7f58f00cfbb0 2023-10-07T09:48:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.556+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f00718b0 con 0x7f58f00cfbb0 2023-10-07T09:48:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.558+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f58e80149c0 con 0x7f58f00cfbb0 2023-10-07T09:48:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.558+0000 7f58ecff9700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f58d4066f30 0x7f58d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.559+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5599+0+0 (secure 0 0 0) 0x7f58e8085bb0 con 0x7f58f00cfbb0 2023-10-07T09:48:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.559+0000 7f58f4d9b700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f58d4066f30 0x7f58d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.560+0000 7f58f4d9b700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f58d4066f30 0x7f58d40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f58f00b30e0 tx=0x7f58e4016000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:21.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.562+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58e8052e70 con 0x7f58f00cfbb0 2023-10-07T09:48:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:21.766+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f58f00cbef0 con 0x7f58f00cfbb0 2023-10-07T09:48:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.315+0000 7f58ecff9700 1 -- 172.21.15.29:0/2368331565 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v131) v1 ==== 170+0+0 (secure 0 0 0) 0x7f58e8056490 con 0x7f58f00cfbb0 2023-10-07T09:48:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.321+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f58d4066f30 msgr2=0x7f58d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.321+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f58d4066f30 0x7f58d40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f58f00b30e0 tx=0x7f58e4016000 comp rx=0 tx=0).stop 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.321+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 msgr2=0x7f58f00c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.321+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 0x7f58f00c9be0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009fd0 tx=0x7f58e8009900 comp rx=0 tx=0).stop 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.322+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 shutdown_connections 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.322+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f58d4066f30 0x7f58d40693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.322+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58f00cfbb0 0x7f58f00c9be0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.322+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f00c6760 0x7f58f0155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.322+0000 7f58f6fff700 1 --2- 172.21.15.29:0/2368331565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00cb800 0x7f58f0169cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.322+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 >> 172.21.15.29:0/2368331565 conn(0x7f58f0063a10 msgr2=0x7f58f00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.323+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 shutdown_connections 2023-10-07T09:48:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.323+0000 7f58f6fff700 1 -- 172.21.15.29:0/2368331565 wait complete. 2023-10-07T09:48:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T09:48:22.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T09:48:22.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:48:22.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:48:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.555+0000 7fa0d77ff700 1 Processor -- start 2023-10-07T09:48:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.555+0000 7fa0d77ff700 1 -- start start 2023-10-07T09:48:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d77ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d0159750 0x7fa0d00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d77ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0d00ce8a0 0x7fa0d00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d77ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d00cbf20 con 0x7fa0d0159b20 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d77ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d00cc200 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d77ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00cc4e0 con 0x7fa0d0159750 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d5d9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0d00ce8a0 0x7fa0d00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d5d9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0d00ce8a0 0x7fa0d00c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47100/0 (socket says 172.21.15.29:47100) 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d5d9c700 1 -- 172.21.15.29:0/571406559 learned_addr learned my addr 172.21.15.29:0/571406559 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.557+0000 7fa0d559b700 1 --2- 172.21.15.29:0/571406559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d0159750 0x7fa0d00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.558+0000 7fa0d5d9c700 1 -- 172.21.15.29:0/571406559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d0159750 msgr2=0x7fa0d00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.558+0000 7fa0d5d9c700 1 --2- 172.21.15.29:0/571406559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d0159750 0x7fa0d00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.558+0000 7fa0d5d9c700 1 -- 172.21.15.29:0/571406559 >> v1:172.21.15.144:6789/0 conn(0x7fa0d0159b20 legacy=0x7fa0d00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.558+0000 7fa0d5d9c700 1 -- 172.21.15.29:0/571406559 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00cc7c0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.558+0000 7fa0d5d9c700 1 --2- 172.21.15.29:0/571406559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d00ce8a0 0x7fa0d00c5460 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009f20 tx=0x7fa0c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be9ad168d1ec140e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.558+0000 7fa0c7fff700 1 -- 172.21.15.29:0/571406559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c800e070 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0c7fff700 1 -- 172.21.15.29:0/571406559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c800b9d0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0c7fff700 1 -- 172.21.15.29:0/571406559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c80092e0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 -- 172.21.15.29:0/571406559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d00ce8a0 msgr2=0x7fa0d00c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/571406559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d00ce8a0 0x7fa0d00c5460 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009f20 tx=0x7fa0c8009840 comp rx=0 tx=0).stop 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 -- 172.21.15.29:0/571406559 shutdown_connections 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/571406559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d00ce8a0 0x7fa0d00c5460 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/571406559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d0159750 0x7fa0d00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 -- 172.21.15.29:0/571406559 >> 172.21.15.29:0/571406559 conn(0x7fa0d0063a10 msgr2=0x7fa0d00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 -- 172.21.15.29:0/571406559 shutdown_connections 2023-10-07T09:48:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.559+0000 7fa0d77ff700 1 -- 172.21.15.29:0/571406559 wait complete. 2023-10-07T09:48:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.561+0000 7fa0d77ff700 1 Processor -- start 2023-10-07T09:48:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.561+0000 7fa0d77ff700 1 -- start start 2023-10-07T09:48:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d77ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 0x7fa0d0169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d77ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d0159750 0x7fa0d015b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d77ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0d0159b20 0x7fa0d015b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d77ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d016ac60 con 0x7fa0d0159750 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d77ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d0157520 con 0x7fa0d0159b20 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d77ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d016a6a0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d559b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 0x7fa0d0169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d4d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d0159750 0x7fa0d015b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d559b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 0x7fa0d0169ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44276/0 (socket says 172.21.15.29:44276) 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d559b700 1 -- 172.21.15.29:0/1205805719 learned_addr learned my addr 172.21.15.29:0/1205805719 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.562+0000 7fa0d5d9c700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0d0159b20 0x7fa0d015b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0d559b700 1 -- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0d0159b20 msgr2=0x7fa0d015b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0d559b700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0d0159b20 0x7fa0d015b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0d559b700 1 -- 172.21.15.29:0/1205805719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d0159750 msgr2=0x7fa0d015b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0d559b700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d0159750 0x7fa0d015b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0d559b700 1 -- 172.21.15.29:0/1205805719 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c8009ae0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0d559b700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 0x7fa0d0169ce0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00ecb0 tx=0x7fa0cc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.563+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc00c850 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.564+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc00c9b0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.564+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d016a980 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.564+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc00cd30 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.564+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d016ae90 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.565+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d0157800 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.566+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa0cc013680 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.567+0000 7fa0c57fa700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa0b4066f30 0x7fa0b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.567+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5981+0+0 (secure 0 0 0) 0x7fa0cc084ca0 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.567+0000 7fa0d4d9a700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa0b4066f30 0x7fa0b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.568+0000 7fa0d4d9a700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa0b4066f30 0x7fa0b40693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0159100 tx=0x7fa0c0007ec0 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.572+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0cc051d60 con 0x7fa0d00ce8a0 2023-10-07T09:48:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:22.814+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa0d004c3c0 con 0x7fa0d00ce8a0 2023-10-07T09:48:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.332+0000 7fa0c57fa700 1 -- 172.21.15.29:0/1205805719 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v64) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa0cc0210c0 con 0x7fa0d00ce8a0 2023-10-07T09:48:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa0b4066f30 msgr2=0x7fa0b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa0b4066f30 0x7fa0b40693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0159100 tx=0x7fa0c0007ec0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 msgr2=0x7fa0d0169ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 0x7fa0d0169ce0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00ecb0 tx=0x7fa0cc009120 comp rx=0 tx=0).stop 2023-10-07T09:48:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 shutdown_connections 2023-10-07T09:48:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa0b4066f30 0x7fa0b40693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0d0159b20 0x7fa0d015b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0d00ce8a0 0x7fa0d0169ce0 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00ecb0 tx=0x7fa0cc009120 comp rx=0 tx=0).stop 2023-10-07T09:48:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 --2- 172.21.15.29:0/1205805719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0d0159750 0x7fa0d015b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.339+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 >> 172.21.15.29:0/1205805719 conn(0x7fa0d0063a10 msgr2=0x7fa0d014d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.346+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 shutdown_connections 2023-10-07T09:48:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.348+0000 7fa0d77ff700 1 -- 172.21.15.29:0/1205805719 wait complete. 2023-10-07T09:48:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 16 and data pool 17 2023-10-07T09:48:23.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.579+0000 7f11eb3ff700 1 Processor -- start 2023-10-07T09:48:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.579+0000 7f11eb3ff700 1 -- start start 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11eb3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159750 0x7f11e40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11eb3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11e40ce8a0 0x7f11e40c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11eb3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40cbf20 con 0x7f11e4159b20 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11eb3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e40cc200 con 0x7f11e40ce8a0 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11eb3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40cc4e0 con 0x7f11e4159750 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11e899a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f11e4159b20 0x7f11e40ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45324/0 (socket says 172.21.15.29:45324) 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11e899a700 1 -- 172.21.15.29:0/422449863 learned_addr learned my addr 172.21.15.29:0/422449863 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11e999c700 1 --2- 172.21.15.29:0/422449863 >> v2:172.21.15.144:3300/0 conn(0x7f11e40ce8a0 0x7f11e40c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:23.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11e919b700 1 --2- 172.21.15.29:0/422449863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159750 0x7f11e40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11dbfff700 1 -- 172.21.15.29:0/422449863 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399542684 0 0) 0x7f11e40cbf20 con 0x7f11e4159b20 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.581+0000 7f11dbfff700 1 -- 172.21.15.29:0/422449863 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d0003570 con 0x7f11e4159b20 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11e919b700 1 -- 172.21.15.29:0/422449863 >> v2:172.21.15.144:3300/0 conn(0x7f11e40ce8a0 msgr2=0x7f11e40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11e919b700 1 --2- 172.21.15.29:0/422449863 >> v2:172.21.15.144:3300/0 conn(0x7f11e40ce8a0 0x7f11e40c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11e919b700 1 -- 172.21.15.29:0/422449863 >> v1:172.21.15.29:6789/0 conn(0x7f11e4159b20 legacy=0x7f11e40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11e919b700 1 -- 172.21.15.29:0/422449863 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e40cc7c0 con 0x7f11e4159750 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11e999c700 1 --2- 172.21.15.29:0/422449863 >> v2:172.21.15.144:3300/0 conn(0x7f11e40ce8a0 0x7f11e40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11e919b700 1 --2- 172.21.15.29:0/422449863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159750 0x7f11e40d10a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f11e00098d0 tx=0x7f11e0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ca28cbcb5032db9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:23.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11dbfff700 1 -- 172.21.15.29:0/422449863 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0005c10 con 0x7f11e4159750 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11dbfff700 1 -- 172.21.15.29:0/422449863 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0005d70 con 0x7f11e4159750 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.582+0000 7f11dbfff700 1 -- 172.21.15.29:0/422449863 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0006260 con 0x7f11e4159750 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 -- 172.21.15.29:0/422449863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159750 msgr2=0x7f11e40d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/422449863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159750 0x7f11e40d10a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f11e00098d0 tx=0x7f11e0009120 comp rx=0 tx=0).stop 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 -- 172.21.15.29:0/422449863 shutdown_connections 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/422449863 >> v2:172.21.15.144:3300/0 conn(0x7f11e40ce8a0 0x7f11e40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/422449863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159750 0x7f11e40d10a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 -- 172.21.15.29:0/422449863 >> 172.21.15.29:0/422449863 conn(0x7f11e4063a10 msgr2=0x7f11e40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 -- 172.21.15.29:0/422449863 shutdown_connections 2023-10-07T09:48:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.583+0000 7f11eb3ff700 1 -- 172.21.15.29:0/422449863 wait complete. 2023-10-07T09:48:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.584+0000 7f11eb3ff700 1 Processor -- start 2023-10-07T09:48:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 -- start start 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11e40ce8a0 0x7f11e41537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 0x7f11e4153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159b20 0x7f11e4149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4069410 con 0x7f11e4159750 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40696f0 con 0x7f11e40ce8a0 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.585+0000 7f11eb3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40699d0 con 0x7f11e4159b20 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 0x7f11e4153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e919b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11e40ce8a0 0x7f11e41537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 0x7f11e4153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49990/0 (socket says 172.21.15.29:49990) 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e999c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159b20 0x7f11e4149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e919b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11e40ce8a0 0x7f11e41537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52948/0 (socket says 172.21.15.29:52948) 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 -- 172.21.15.29:0/2716326589 learned_addr learned my addr 172.21.15.29:0/2716326589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 -- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159b20 msgr2=0x7f11e4149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159b20 0x7f11e4149c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 -- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11e40ce8a0 msgr2=0x7f11e41537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11e40ce8a0 0x7f11e41537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 -- 172.21.15.29:0/2716326589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc009a20 con 0x7f11e4159750 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.586+0000 7f11e899a700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 0x7f11e4153ce0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c590 tx=0x7f11d4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.587+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4005cd0 con 0x7f11e4159750 2023-10-07T09:48:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.587+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d4005e50 con 0x7f11e4159750 2023-10-07T09:48:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.587+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4007320 con 0x7f11e4159750 2023-10-07T09:48:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.587+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e00093c0 con 0x7f11e4159750 2023-10-07T09:48:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.587+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e4069f90 con 0x7f11e4159750 2023-10-07T09:48:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.588+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e404c3c0 con 0x7f11e4159750 2023-10-07T09:48:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.589+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f11d401b3b0 con 0x7f11e4159750 2023-10-07T09:48:23.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.591+0000 7f11d97fa700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f11c8066f30 0x7f11c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:23.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.591+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f11d4084150 con 0x7f11e4159750 2023-10-07T09:48:23.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.591+0000 7f11e919b700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f11c8066f30 0x7f11c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.592+0000 7f11e919b700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f11c8066f30 0x7f11c80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009160 tx=0x7f11e0016000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:23.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.594+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11d40512e0 con 0x7f11e4159750 2023-10-07T09:48:23.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.798+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11e40ca970 con 0x7f11e4159750 2023-10-07T09:48:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.799+0000 7f11d97fa700 1 -- 172.21.15.29:0/2716326589 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16044 (secure 0 0 0) 0x7f11d4054900 con 0x7f11e4159750 2023-10-07T09:48:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:23.800 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":132,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:48:22.816690+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:48:20.536626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-07T09:48:21.807909+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.801+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f11c8066f30 msgr2=0x7f11c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.801+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f11c8066f30 0x7f11c80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009160 tx=0x7f11e0016000 comp rx=0 tx=0).stop 2023-10-07T09:48:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.801+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 msgr2=0x7f11e4153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.801+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 0x7f11e4153ce0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c590 tx=0x7f11d4007e00 comp rx=0 tx=0).stop 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 shutdown_connections 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f11c8066f30 0x7f11c80693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11e4159b20 0x7f11e4149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11e40ce8a0 0x7f11e41537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 --2- 172.21.15.29:0/2716326589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e4159750 0x7f11e4153ce0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 >> 172.21.15.29:0/2716326589 conn(0x7f11e4063a10 msgr2=0x7f11e4155130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 shutdown_connections 2023-10-07T09:48:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:23.802+0000 7f11eb3ff700 1 -- 172.21.15.29:0/2716326589 wait complete. 2023-10-07T09:48:23.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.016+0000 7f4808dff700 1 Processor -- start 2023-10-07T09:48:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.016+0000 7f4808dff700 1 -- start start 2023-10-07T09:48:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4808dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4808dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f480415ade0 0x7f48040b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4808dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4804154090 con 0x7f48040b0000 2023-10-07T09:48:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4808dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4804159f60 con 0x7f480415ade0 2023-10-07T09:48:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4808dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480415a240 con 0x7f4804153cc0 2023-10-07T09:48:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4801d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f480415ade0 0x7f48040b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4801d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f480415ade0 0x7f48040b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50004/0 (socket says 172.21.15.29:50004) 2023-10-07T09:48:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4801d9b700 1 -- 172.21.15.29:0/2339391858 learned_addr learned my addr 172.21.15.29:0/2339391858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f480259c700 1 --2- 172.21.15.29:0/2339391858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4801d9b700 1 -- 172.21.15.29:0/2339391858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 msgr2=0x7f48040b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:24.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.018+0000 7f4801d9b700 1 --2- 172.21.15.29:0/2339391858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.019+0000 7f4801d9b700 1 -- 172.21.15.29:0/2339391858 >> v1:172.21.15.144:6789/0 conn(0x7f48040b0000 legacy=0x7f480415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.019+0000 7f4801d9b700 1 -- 172.21.15.29:0/2339391858 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480415a520 con 0x7f480415ade0 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.019+0000 7f4801d9b700 1 --2- 172.21.15.29:0/2339391858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040b1d00 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f47f80099b0 tx=0x7f47f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439d6598967db389 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.019+0000 7f480159a700 1 -- 172.21.15.29:0/2339391858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f8009ea0 con 0x7f480415ade0 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.019+0000 7f480159a700 1 -- 172.21.15.29:0/2339391858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f800b860 con 0x7f480415ade0 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.019+0000 7f480159a700 1 -- 172.21.15.29:0/2339391858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800bc00 con 0x7f480415ade0 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 -- 172.21.15.29:0/2339391858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 msgr2=0x7f48040b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 --2- 172.21.15.29:0/2339391858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040b1d00 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f47f80099b0 tx=0x7f47f8009250 comp rx=0 tx=0).stop 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 -- 172.21.15.29:0/2339391858 shutdown_connections 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 --2- 172.21.15.29:0/2339391858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 --2- 172.21.15.29:0/2339391858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040b1d00 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 -- 172.21.15.29:0/2339391858 >> 172.21.15.29:0/2339391858 conn(0x7f4804067610 msgr2=0x7f4804067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:24.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 -- 172.21.15.29:0/2339391858 shutdown_connections 2023-10-07T09:48:24.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.020+0000 7f4808dff700 1 -- 172.21.15.29:0/2339391858 wait complete. 2023-10-07T09:48:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.021+0000 7f4808dff700 1 Processor -- start 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.021+0000 7f4808dff700 1 -- start start 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4808dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48040b0000 0x7f48040ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4808dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4808dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4808dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041697d0 con 0x7f480415ade0 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4808dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041561d0 con 0x7f48040b0000 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4808dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48041564b0 con 0x7f4804153cc0 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.022+0000 7f4802d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040c8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50012/0 (socket says 172.21.15.29:50012) 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 -- 172.21.15.29:0/2872713473 learned_addr learned my addr 172.21.15.29:0/2872713473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f480259c700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48040b0000 0x7f48040ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4801d9b700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 -- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 msgr2=0x7f48040c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 -- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48040b0000 msgr2=0x7f48040ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48040b0000 0x7f48040ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 -- 172.21.15.29:0/2872713473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80094f0 con 0x7f480415ade0 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4801d9b700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f480259c700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48040b0000 0x7f48040ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4802d9d700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040c8500 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00c8e0 tx=0x7f47fc001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00ce80 con 0x7f480415ade0 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc00a7e0 con 0x7f480415ade0 2023-10-07T09:48:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.023+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804156790 con 0x7f480415ade0 2023-10-07T09:48:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.024+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00ab80 con 0x7f480415ade0 2023-10-07T09:48:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.024+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48040cee00 con 0x7f480415ade0 2023-10-07T09:48:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.025+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4804156f90 con 0x7f480415ade0 2023-10-07T09:48:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.026+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f47fc012420 con 0x7f480415ade0 2023-10-07T09:48:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.027+0000 7f47f27fc700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47e4066f30 0x7f47e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.027+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f47fc084a40 con 0x7f480415ade0 2023-10-07T09:48:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.027+0000 7f480259c700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47e4066f30 0x7f47e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.028+0000 7f480259c700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47e4066f30 0x7f47e40693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006ea0 tx=0x7f47f4008040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:24.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.031+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47fc051a90 con 0x7f480415ade0 2023-10-07T09:48:24.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.235+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4804156da0 con 0x7f480415ade0 2023-10-07T09:48:24.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.236+0000 7f47f27fc700 1 -- 172.21.15.29:0/2872713473 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16044 (secure 0 0 0) 0x7f47fc0550b0 con 0x7f480415ade0 2023-10-07T09:48:24.237 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:24.237 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":132,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:48:22.816690+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:48:20.536626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-07T09:48:21.807909+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47e4066f30 msgr2=0x7f47e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47e4066f30 0x7f47e40693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006ea0 tx=0x7f47f4008040 comp rx=0 tx=0).stop 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 msgr2=0x7f48040c8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040c8500 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00c8e0 tx=0x7f47fc001710 comp rx=0 tx=0).stop 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 shutdown_connections 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f47e4066f30 0x7f47e40693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f480415ade0 0x7f48040c8500 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48040b0000 0x7f48040ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 --2- 172.21.15.29:0/2872713473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4804153cc0 0x7f48040c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 >> 172.21.15.29:0/2872713473 conn(0x7f4804067610 msgr2=0x7f48040aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 shutdown_connections 2023-10-07T09:48:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.238+0000 7f4808dff700 1 -- 172.21.15.29:0/2872713473 wait complete. 2023-10-07T09:48:24.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T09:48:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.459+0000 7f31465ff700 1 Processor -- start 2023-10-07T09:48:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.459+0000 7f31465ff700 1 -- start start 2023-10-07T09:48:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.460+0000 7f31465ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3140159750 0x7f31400d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.460+0000 7f31465ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31400cbf20 con 0x7f31400ce8a0 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.460+0000 7f31465ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31400cc200 con 0x7f3140159b20 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.460+0000 7f31465ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31400cc4e0 con 0x7f3140159750 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.460+0000 7f313ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3140159750 0x7f31400d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3140159750 0x7f31400d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50016/0 (socket says 172.21.15.29:50016) 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f3144b9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31400ce8a0 0x7f31400c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45338/0 (socket says 172.21.15.29:45338) 2023-10-07T09:48:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313ffff700 1 -- 172.21.15.29:0/1789127631 learned_addr learned my addr 172.21.15.29:0/1789127631 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313ffff700 1 -- 172.21.15.29:0/1789127631 >> v1:172.21.15.144:6789/0 conn(0x7f3140159b20 legacy=0x7f31400ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313ffff700 1 -- 172.21.15.29:0/1789127631 >> v1:172.21.15.29:6789/0 conn(0x7f31400ce8a0 legacy=0x7f31400c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313ffff700 1 -- 172.21.15.29:0/1789127631 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400cc7c0 con 0x7f3140159750 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313effd700 1 -- 172.21.15.29:0/1789127631 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584850706 0 0) 0x7f31400cbf20 con 0x7f31400ce8a0 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313ffff700 1 --2- 172.21.15.29:0/1789127631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3140159750 0x7f31400d10a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f3130009990 tx=0x7f3130009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2552f5bf3dbc6fcb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.461+0000 7f313effd700 1 -- 172.21.15.29:0/1789127631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130005c10 con 0x7f3140159750 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f313effd700 1 -- 172.21.15.29:0/1789127631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3130005d70 con 0x7f3140159750 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f313effd700 1 -- 172.21.15.29:0/1789127631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000b860 con 0x7f3140159750 2023-10-07T09:48:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 -- 172.21.15.29:0/1789127631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3140159750 msgr2=0x7f31400d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 --2- 172.21.15.29:0/1789127631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3140159750 0x7f31400d10a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f3130009990 tx=0x7f3130009220 comp rx=0 tx=0).stop 2023-10-07T09:48:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 -- 172.21.15.29:0/1789127631 shutdown_connections 2023-10-07T09:48:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 --2- 172.21.15.29:0/1789127631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3140159750 0x7f31400d10a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 -- 172.21.15.29:0/1789127631 >> 172.21.15.29:0/1789127631 conn(0x7f3140063a10 msgr2=0x7f31400d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 -- 172.21.15.29:0/1789127631 shutdown_connections 2023-10-07T09:48:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.462+0000 7f31465ff700 1 -- 172.21.15.29:0/1789127631 wait complete. 2023-10-07T09:48:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.464+0000 7f31465ff700 1 Processor -- start 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.464+0000 7f31465ff700 1 -- start start 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f31465ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 0x7f31400b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f31465ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3140159750 0x7f31401480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f31465ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3140159b20 0x7f31400695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f31465ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400b4010 con 0x7f31400ce8a0 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f31465ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400b3400 con 0x7f3140159b20 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f31465ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31400b36e0 con 0x7f3140159750 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 0x7f31400b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 0x7f31400b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50030/0 (socket says 172.21.15.29:50030) 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 -- 172.21.15.29:0/1579928822 learned_addr learned my addr 172.21.15.29:0/1579928822 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313f7fe700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3140159750 0x7f31401480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f3144b9c700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3140159b20 0x7f31400695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 -- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3140159750 msgr2=0x7f31401480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3140159750 0x7f31401480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 -- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3140159b20 msgr2=0x7f31400695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3140159b20 0x7f31400695c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f313ffff700 1 -- 172.21.15.29:0/1579928822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130009480 con 0x7f31400ce8a0 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.465+0000 7f3144b9c700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3140159b20 0x7f31400695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.466+0000 7f313ffff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 0x7f31400b44a0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f3140088a40 tx=0x7f3130016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.466+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313001d860 con 0x7f31400ce8a0 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.466+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313001d9e0 con 0x7f31400ce8a0 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.466+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313001dda0 con 0x7f31400ce8a0 2023-10-07T09:48:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.466+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31400b39c0 con 0x7f31400ce8a0 2023-10-07T09:48:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.466+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3140148600 con 0x7f31400ce8a0 2023-10-07T09:48:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.467+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314004c3c0 con 0x7f31400ce8a0 2023-10-07T09:48:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.468+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f313001b690 con 0x7f31400ce8a0 2023-10-07T09:48:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.470+0000 7f3127fff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f312c066f30 0x7f312c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.470+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3130086690 con 0x7f31400ce8a0 2023-10-07T09:48:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.470+0000 7f313f7fe700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f312c066f30 0x7f312c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.471+0000 7f313f7fe700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f312c066f30 0x7f312c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f314016d020 tx=0x7f3134009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.474+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31300536e0 con 0x7f31400ce8a0 2023-10-07T09:48:24.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:24.721+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3140064040 con 0x7f31400ce8a0 2023-10-07T09:48:25.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.355+0000 7f3127fff700 1 -- 172.21.15.29:0/1579928822 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v67) v1 ==== 118+0+0 (secure 0 0 0) 0x7f313002a0b0 con 0x7f31400ce8a0 2023-10-07T09:48:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f312c066f30 msgr2=0x7f312c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f312c066f30 0x7f312c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f314016d020 tx=0x7f3134009040 comp rx=0 tx=0).stop 2023-10-07T09:48:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 msgr2=0x7f31400b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 0x7f31400b44a0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f3140088a40 tx=0x7f3130016000 comp rx=0 tx=0).stop 2023-10-07T09:48:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 shutdown_connections 2023-10-07T09:48:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f312c066f30 0x7f312c0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3140159b20 0x7f31400695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400ce8a0 0x7f31400b44a0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 --2- 172.21.15.29:0/1579928822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3140159750 0x7f31401480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 >> 172.21.15.29:0/1579928822 conn(0x7f3140063a10 msgr2=0x7f3140155f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 shutdown_connections 2023-10-07T09:48:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.358+0000 7f31465ff700 1 -- 172.21.15.29:0/1579928822 wait complete. 2023-10-07T09:48:25.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T09:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.587+0000 7f2bbc9ff700 1 Processor -- start 2023-10-07T09:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.587+0000 7f2bbc9ff700 1 -- start start 2023-10-07T09:48:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.588+0000 7f2bbc9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bb8159750 0x7f2bb80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.588+0000 7f2bbc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb80ce8a0 0x7f2bb80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.588+0000 7f2bbc9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80cbf20 con 0x7f2bb8159b20 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.588+0000 7f2bbc9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80cc200 con 0x7f2bb8159750 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.588+0000 7f2bbc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80cc4e0 con 0x7f2bb80ce8a0 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bb8159750 0x7f2bb80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bb8159750 0x7f2bb80d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50036/0 (socket says 172.21.15.29:50036) 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 -- 172.21.15.29:0/3859875461 learned_addr learned my addr 172.21.15.29:0/3859875461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb6d9d700 1 --2- 172.21.15.29:0/3859875461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb80ce8a0 0x7f2bb80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 -- 172.21.15.29:0/3859875461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb80ce8a0 msgr2=0x7f2bb80c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 --2- 172.21.15.29:0/3859875461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb80ce8a0 0x7f2bb80c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 -- 172.21.15.29:0/3859875461 >> v1:172.21.15.144:6789/0 conn(0x7f2bb8159b20 legacy=0x7f2bb80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 -- 172.21.15.29:0/3859875461 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80cc7c0 con 0x7f2bb8159750 2023-10-07T09:48:25.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.589+0000 7f2bb659c700 1 --2- 172.21.15.29:0/3859875461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159750 0x7f2bb80d10a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009990 tx=0x7f2ba8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8998cfe6f9f44f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bb559a700 1 -- 172.21.15.29:0/3859875461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8005c10 con 0x7f2bb8159750 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bb559a700 1 -- 172.21.15.29:0/3859875461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8005d70 con 0x7f2bb8159750 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bb559a700 1 -- 172.21.15.29:0/3859875461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba800b880 con 0x7f2bb8159750 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/3859875461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159750 msgr2=0x7f2bb80d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/3859875461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159750 0x7f2bb80d10a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009990 tx=0x7f2ba8009220 comp rx=0 tx=0).stop 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/3859875461 shutdown_connections 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/3859875461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb80ce8a0 0x7f2bb80c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/3859875461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159750 0x7f2bb80d10a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/3859875461 >> 172.21.15.29:0/3859875461 conn(0x7f2bb8063a10 msgr2=0x7f2bb80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/3859875461 shutdown_connections 2023-10-07T09:48:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.590+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/3859875461 wait complete. 2023-10-07T09:48:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.592+0000 7f2bbc9ff700 1 Processor -- start 2023-10-07T09:48:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.592+0000 7f2bbc9ff700 1 -- start start 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bbc9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bb80ce8a0 0x7f2bb8145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bbc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb8159750 0x7f2bb815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bbc9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 0x7f2bb815c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bbc9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8159f10 con 0x7f2bb8159b20 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bbc9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb815a7b0 con 0x7f2bb80ce8a0 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bbc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80b4b30 con 0x7f2bb8159750 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bb6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 0x7f2bb815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bb6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 0x7f2bb815c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50046/0 (socket says 172.21.15.29:50046) 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bb6d9d700 1 -- 172.21.15.29:0/589148282 learned_addr learned my addr 172.21.15.29:0/589148282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bb659c700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bb80ce8a0 0x7f2bb8145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.593+0000 7f2bb5d9b700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb8159750 0x7f2bb815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb6d9d700 1 -- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb8159750 msgr2=0x7f2bb815c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb6d9d700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb8159750 0x7f2bb815c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb6d9d700 1 -- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bb80ce8a0 msgr2=0x7f2bb8145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb6d9d700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bb80ce8a0 0x7f2bb8145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb6d9d700 1 -- 172.21.15.29:0/589148282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8009480 con 0x7f2bb8159b20 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb5d9b700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb8159750 0x7f2bb815c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bb6d9d700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 0x7f2bb815c720 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009fd0 tx=0x7f2bb0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb000ed50 con 0x7f2bb8159b20 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb000eeb0 con 0x7f2bb8159b20 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0014650 con 0x7f2bb8159b20 2023-10-07T09:48:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb815a160 con 0x7f2bb8159b20 2023-10-07T09:48:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.594+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb80c84f0 con 0x7f2bb8159b20 2023-10-07T09:48:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.595+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb80c8770 con 0x7f2bb8159b20 2023-10-07T09:48:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.596+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2bb00149c0 con 0x7f2bb8159b20 2023-10-07T09:48:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.597+0000 7f2ba67fc700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2b98066f30 0x7f2b980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:25.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.597+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f2bb0084f00 con 0x7f2bb8159b20 2023-10-07T09:48:25.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.598+0000 7f2bb659c700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2b98066f30 0x7f2b980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:25.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.599+0000 7f2bb659c700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2b98066f30 0x7f2b980693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009260 tx=0x7f2ba8016000 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:25.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.601+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2bb0051f50 con 0x7f2bb8159b20 2023-10-07T09:48:25.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:25.848+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2bb804c3c0 con 0x7f2bb8159b20 2023-10-07T09:48:26.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.361+0000 7f2ba67fc700 1 -- 172.21.15.29:0/589148282 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v68) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2bb0055570 con 0x7f2bb8159b20 2023-10-07T09:48:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2b98066f30 msgr2=0x7f2b980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2b98066f30 0x7f2b980693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009260 tx=0x7f2ba8016000 comp rx=0 tx=0).stop 2023-10-07T09:48:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 msgr2=0x7f2bb815c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 0x7f2bb815c720 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009fd0 tx=0x7f2bb0009900 comp rx=0 tx=0).stop 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 shutdown_connections 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2b98066f30 0x7f2b980693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb8159b20 0x7f2bb815c720 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bb80ce8a0 0x7f2bb8145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 --2- 172.21.15.29:0/589148282 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bb8159750 0x7f2bb815c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 >> 172.21.15.29:0/589148282 conn(0x7f2bb8063a10 msgr2=0x7f2bb81564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 shutdown_connections 2023-10-07T09:48:26.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.364+0000 7f2bbc9ff700 1 -- 172.21.15.29:0/589148282 wait complete. 2023-10-07T09:48:26.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.583+0000 7f63ca7ff700 1 Processor -- start 2023-10-07T09:48:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.583+0000 7f63ca7ff700 1 -- start start 2023-10-07T09:48:26.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.584+0000 7f63ca7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c40b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.584+0000 7f63ca7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63c40b0540 0x7f63c415c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.584+0000 7f63ca7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40b0980 con 0x7f63c4153cc0 2023-10-07T09:48:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.584+0000 7f63ca7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c4154090 con 0x7f63c40b0540 2023-10-07T09:48:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.584+0000 7f63ca7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4159f60 con 0x7f63c415ade0 2023-10-07T09:48:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f63c4153cc0 0x7f63c40b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45342/0 (socket says 172.21.15.29:45342) 2023-10-07T09:48:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c3fff700 1 -- 172.21.15.29:0/3636744362 learned_addr learned my addr 172.21.15.29:0/3636744362 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c37fe700 1 --2- 172.21.15.29:0/3636744362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c40b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c8d9c700 1 --2- 172.21.15.29:0/3636744362 >> v2:172.21.15.144:3300/0 conn(0x7f63c40b0540 0x7f63c415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c2ffd700 1 -- 172.21.15.29:0/3636744362 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181941836 0 0) 0x7f63c40b0980 con 0x7f63c4153cc0 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c2ffd700 1 -- 172.21.15.29:0/3636744362 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003570 con 0x7f63c4153cc0 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c37fe700 1 -- 172.21.15.29:0/3636744362 >> v2:172.21.15.144:3300/0 conn(0x7f63c40b0540 msgr2=0x7f63c415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c37fe700 1 --2- 172.21.15.29:0/3636744362 >> v2:172.21.15.144:3300/0 conn(0x7f63c40b0540 0x7f63c415c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c37fe700 1 -- 172.21.15.29:0/3636744362 >> v1:172.21.15.29:6789/0 conn(0x7f63c4153cc0 legacy=0x7f63c40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c37fe700 1 -- 172.21.15.29:0/3636744362 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c415a240 con 0x7f63c415ade0 2023-10-07T09:48:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.585+0000 7f63c8d9c700 1 --2- 172.21.15.29:0/3636744362 >> v2:172.21.15.144:3300/0 conn(0x7f63c40b0540 0x7f63c415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.586+0000 7f63c37fe700 1 --2- 172.21.15.29:0/3636744362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c40b0000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098e0 tx=0x7f63b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fabd460fd59443f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.586+0000 7f63c2ffd700 1 -- 172.21.15.29:0/3636744362 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8011630 con 0x7f63c415ade0 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.586+0000 7f63c2ffd700 1 -- 172.21.15.29:0/3636744362 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8011790 con 0x7f63c415ade0 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.586+0000 7f63c2ffd700 1 -- 172.21.15.29:0/3636744362 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8011b30 con 0x7f63c415ade0 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.586+0000 7f63ca7ff700 1 -- 172.21.15.29:0/3636744362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 msgr2=0x7f63c40b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.586+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/3636744362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c40b0000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098e0 tx=0x7f63b80091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.587+0000 7f63ca7ff700 1 -- 172.21.15.29:0/3636744362 shutdown_connections 2023-10-07T09:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.587+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/3636744362 >> v2:172.21.15.144:3300/0 conn(0x7f63c40b0540 0x7f63c415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.587+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/3636744362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c40b0000 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.587+0000 7f63ca7ff700 1 -- 172.21.15.29:0/3636744362 >> 172.21.15.29:0/3636744362 conn(0x7f63c4067610 msgr2=0x7f63c4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.587+0000 7f63ca7ff700 1 -- 172.21.15.29:0/3636744362 shutdown_connections 2023-10-07T09:48:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.587+0000 7f63ca7ff700 1 -- 172.21.15.29:0/3636744362 wait complete. 2023-10-07T09:48:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.588+0000 7f63ca7ff700 1 Processor -- start 2023-10-07T09:48:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.588+0000 7f63ca7ff700 1 -- start start 2023-10-07T09:48:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.589+0000 7f63ca7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40b0540 0x7f63c4154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.589+0000 7f63ca7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 0x7f63c40c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.589+0000 7f63ca7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c41583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.589+0000 7f63ca7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c416abe0 con 0x7f63c4153cc0 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.589+0000 7f63ca7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c41578d0 con 0x7f63c40b0540 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.589+0000 7f63ca7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4157bb0 con 0x7f63c415ade0 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 0x7f63c40c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 0x7f63c40c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50050/0 (socket says 172.21.15.29:50050) 2023-10-07T09:48:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 -- 172.21.15.29:0/1742695962 learned_addr learned my addr 172.21.15.29:0/1742695962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c3fff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40b0540 0x7f63c4154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c8d9c700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c41583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 -- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 msgr2=0x7f63c41583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c41583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 -- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40b0540 msgr2=0x7f63c4154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40b0540 0x7f63c4154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 -- 172.21.15.29:0/1742695962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc009a20 con 0x7f63c4153cc0 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c3fff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40b0540 0x7f63c4154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.590+0000 7f63c37fe700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 0x7f63c40c8920 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f63c40b0b10 tx=0x7f63b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.591+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8011b10 con 0x7f63c4153cc0 2023-10-07T09:48:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.591+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b80253f0 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.591+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b8009420 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.591+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80257b0 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.591+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4154ec0 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.592+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c404c3c0 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.593+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f63b8025ba0 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.594+0000 7f63abfff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.594+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f63b8085fc0 con 0x7f63c4153cc0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.594+0000 7f63c3fff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.595+0000 7f63c3fff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63b4001360 tx=0x7f63b4009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:26.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.598+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63b80b75a0 con 0x7f63c4153cc0 2023-10-07T09:48:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.848+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c415a520 con 0x7f63c4153cc0 2023-10-07T09:48:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.848+0000 7f63abfff700 1 -- 172.21.15.29:0/1742695962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f63b80530b0 con 0x7f63c4153cc0 2023-10-07T09:48:26.850 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:26.850 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:48:22.817019+0000","modified":"2023-10-07T09:48:26.355188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/2749584891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2749584891},{"type":"v1","addr":"172.21.15.29:6834","nonce":2749584891}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-10-07T09:48:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 msgr2=0x7f63b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63b4001360 tx=0x7f63b4009040 comp rx=0 tx=0).stop 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 msgr2=0x7f63c40c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 0x7f63c40c8920 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f63c40b0b10 tx=0x7f63b8017000 comp rx=0 tx=0).stop 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 shutdown_connections 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63b0066f30 0x7f63b00693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c415ade0 0x7f63c41583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c40b0540 0x7f63c4154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 --2- 172.21.15.29:0/1742695962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c4153cc0 0x7f63c40c8920 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 >> 172.21.15.29:0/1742695962 conn(0x7f63c4067610 msgr2=0x7f63c40af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 shutdown_connections 2023-10-07T09:48:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:26.851+0000 7f63ca7ff700 1 -- 172.21.15.29:0/1742695962 wait complete. 2023-10-07T09:48:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-07T09:48:26.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T09:48:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.064+0000 7f0934dff700 1 Processor -- start 2023-10-07T09:48:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.064+0000 7f0934dff700 1 -- start start 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.065+0000 7f0934dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f093015ade0 0x7f09300b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.065+0000 7f0934dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0930154090 con 0x7f09300b0540 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.065+0000 7f0934dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0930159f60 con 0x7f0930153cc0 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.065+0000 7f0934dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09300b09e0 con 0x7f093015ade0 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092ed9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f09300b0540 0x7f093015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45352/0 (socket says 172.21.15.29:45352) 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092ed9d700 1 -- 172.21.15.29:0/383694856 learned_addr learned my addr 172.21.15.29:0/383694856 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092dd9b700 1 --2- 172.21.15.29:0/383694856 >> v2:172.21.15.144:3300/0 conn(0x7f093015ade0 0x7f09300b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092d59a700 1 -- 172.21.15.29:0/383694856 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345875262 0 0) 0x7f0930154090 con 0x7f09300b0540 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092d59a700 1 -- 172.21.15.29:0/383694856 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0918003570 con 0x7f09300b0540 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092dd9b700 1 -- 172.21.15.29:0/383694856 >> v1:172.21.15.144:6789/0 conn(0x7f0930153cc0 legacy=0x7f09300b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092dd9b700 1 -- 172.21.15.29:0/383694856 >> v1:172.21.15.29:6789/0 conn(0x7f09300b0540 legacy=0x7f093015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.066+0000 7f092dd9b700 1 -- 172.21.15.29:0/383694856 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093015a240 con 0x7f093015ade0 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.067+0000 7f092dd9b700 1 --2- 172.21.15.29:0/383694856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f093015ade0 0x7f09300b0000 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f09240098e0 tx=0x7f09240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=141a2eff359eaaa1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.067+0000 7f092d59a700 1 -- 172.21.15.29:0/383694856 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400b640 con 0x7f093015ade0 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.067+0000 7f092d59a700 1 -- 172.21.15.29:0/383694856 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092400b7a0 con 0x7f093015ade0 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.067+0000 7f092d59a700 1 -- 172.21.15.29:0/383694856 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400bb40 con 0x7f093015ade0 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.067+0000 7f0934dff700 1 -- 172.21.15.29:0/383694856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f093015ade0 msgr2=0x7f09300b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.067+0000 7f0934dff700 1 --2- 172.21.15.29:0/383694856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f093015ade0 0x7f09300b0000 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f09240098e0 tx=0x7f09240091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.068+0000 7f0934dff700 1 -- 172.21.15.29:0/383694856 shutdown_connections 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.068+0000 7f0934dff700 1 --2- 172.21.15.29:0/383694856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f093015ade0 0x7f09300b0000 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.068+0000 7f0934dff700 1 -- 172.21.15.29:0/383694856 >> 172.21.15.29:0/383694856 conn(0x7f0930067610 msgr2=0x7f0930067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.068+0000 7f0934dff700 1 -- 172.21.15.29:0/383694856 shutdown_connections 2023-10-07T09:48:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.068+0000 7f0934dff700 1 -- 172.21.15.29:0/383694856 wait complete. 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.069+0000 7f0934dff700 1 Processor -- start 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.069+0000 7f0934dff700 1 -- start start 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f0934dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09300b0540 0x7f09300c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f0934dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 0x7f0930151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f0934dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f093015ade0 0x7f09301570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f0934dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0930158b20 con 0x7f0930153cc0 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f0934dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0930158230 con 0x7f09300b0540 2023-10-07T09:48:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f0934dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0930152030 con 0x7f093015ade0 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f092dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 0x7f0930151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f092dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 0x7f0930151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50066/0 (socket says 172.21.15.29:50066) 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.070+0000 7f092dd9b700 1 -- 172.21.15.29:0/1257067978 learned_addr learned my addr 172.21.15.29:0/1257067978 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092e59c700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09300b0540 0x7f09300c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092ed9d700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f093015ade0 0x7f09301570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092dd9b700 1 -- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f093015ade0 msgr2=0x7f09301570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092dd9b700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f093015ade0 0x7f09301570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092dd9b700 1 -- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09300b0540 msgr2=0x7f09300c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092dd9b700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09300b0540 0x7f09300c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092dd9b700 1 -- 172.21.15.29:0/1257067978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924009420 con 0x7f0930153cc0 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092ed9d700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f093015ade0 0x7f09301570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f092dd9b700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 0x7f0930151af0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f0924013000 tx=0x7f0924017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400bb20 con 0x7f0930153cc0 2023-10-07T09:48:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924011420 con 0x7f0930153cc0 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09240117c0 con 0x7f0930153cc0 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.071+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0930157610 con 0x7f0930153cc0 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.072+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09301584d0 con 0x7f0930153cc0 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.072+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0930158c90 con 0x7f0930153cc0 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.073+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0924011b70 con 0x7f0930153cc0 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.074+0000 7f091e7fc700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0910066f30 0x7f09100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.074+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f0924085cc0 con 0x7f0930153cc0 2023-10-07T09:48:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.075+0000 7f092e59c700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0910066f30 0x7f09100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.075+0000 7f092e59c700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0910066f30 0x7f09100693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09301536e0 tx=0x7f0920009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.078+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0924052db0 con 0x7f0930153cc0 2023-10-07T09:48:27.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.281+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f093004c3c0 con 0x7f0930153cc0 2023-10-07T09:48:27.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.282+0000 7f091e7fc700 1 -- 172.21.15.29:0/1257067978 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+16031 (secure 0 0 0) 0x7f09240563d0 con 0x7f0930153cc0 2023-10-07T09:48:27.284 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:27.284 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":133,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T09:48:24.327143+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:48:20.536626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-07T09:48:21.807909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T09:48:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0910066f30 msgr2=0x7f09100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0910066f30 0x7f09100693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09301536e0 tx=0x7f0920009040 comp rx=0 tx=0).stop 2023-10-07T09:48:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 msgr2=0x7f0930151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 0x7f0930151af0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f0924013000 tx=0x7f0924017000 comp rx=0 tx=0).stop 2023-10-07T09:48:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 shutdown_connections 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0910066f30 0x7f09100693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f093015ade0 0x7f09301570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09300b0540 0x7f09300c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 --2- 172.21.15.29:0/1257067978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0930153cc0 0x7f0930151af0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 >> 172.21.15.29:0/1257067978 conn(0x7f0930067610 msgr2=0x7f0930152b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 shutdown_connections 2023-10-07T09:48:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.285+0000 7f0934dff700 1 -- 172.21.15.29:0/1257067978 wait complete. 2023-10-07T09:48:27.287 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T09:48:27.286+0000 7f6b629f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T09:48:27.304 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T09:48:27.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:48:27.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.501+0000 7fe5c9bff700 1 Processor -- start 2023-10-07T09:48:27.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.502+0000 7fe5c9bff700 1 -- start start 2023-10-07T09:48:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c9bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5c4159750 0x7fe5c40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c9bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5c40ce8a0 0x7fe5c40c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c9bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c40cbf20 con 0x7fe5c4159b20 2023-10-07T09:48:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c9bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c40cc200 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c9bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c40cc4e0 con 0x7fe5c4159750 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c2ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe5c4159b20 0x7fe5c40ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45364/0 (socket says 172.21.15.29:45364) 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c2ffd700 1 -- 172.21.15.29:0/601206354 learned_addr learned my addr 172.21.15.29:0/601206354 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c3fff700 1 --2- 172.21.15.29:0/601206354 >> v2:172.21.15.29:3300/0 conn(0x7fe5c40ce8a0 0x7fe5c40c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.503+0000 7fe5c37fe700 1 --2- 172.21.15.29:0/601206354 >> v2:172.21.15.144:3300/0 conn(0x7fe5c4159750 0x7fe5c40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c27fc700 1 -- 172.21.15.29:0/601206354 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451734654 0 0) 0x7fe5c40cbf20 con 0x7fe5c4159b20 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c27fc700 1 -- 172.21.15.29:0/601206354 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ac003570 con 0x7fe5c4159b20 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c3fff700 1 -- 172.21.15.29:0/601206354 >> v2:172.21.15.144:3300/0 conn(0x7fe5c4159750 msgr2=0x7fe5c40d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c3fff700 1 --2- 172.21.15.29:0/601206354 >> v2:172.21.15.144:3300/0 conn(0x7fe5c4159750 0x7fe5c40d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c3fff700 1 -- 172.21.15.29:0/601206354 >> v1:172.21.15.29:6789/0 conn(0x7fe5c4159b20 legacy=0x7fe5c40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c3fff700 1 -- 172.21.15.29:0/601206354 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c40cc7c0 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c3fff700 1 --2- 172.21.15.29:0/601206354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c40c5460 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4089840 tx=0x7fe5bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba345abb1d92f8e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c27fc700 1 -- 172.21.15.29:0/601206354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00e070 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c27fc700 1 -- 172.21.15.29:0/601206354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5bc00bab0 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.504+0000 7fe5c27fc700 1 -- 172.21.15.29:0/601206354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00be70 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 -- 172.21.15.29:0/601206354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 msgr2=0x7fe5c40c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/601206354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c40c5460 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4089840 tx=0x7fe5bc009780 comp rx=0 tx=0).stop 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 -- 172.21.15.29:0/601206354 shutdown_connections 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/601206354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c40c5460 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/601206354 >> v2:172.21.15.144:3300/0 conn(0x7fe5c4159750 0x7fe5c40d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 -- 172.21.15.29:0/601206354 >> 172.21.15.29:0/601206354 conn(0x7fe5c4063a10 msgr2=0x7fe5c40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 -- 172.21.15.29:0/601206354 shutdown_connections 2023-10-07T09:48:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.505+0000 7fe5c9bff700 1 -- 172.21.15.29:0/601206354 wait complete. 2023-10-07T09:48:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.506+0000 7fe5c9bff700 1 Processor -- start 2023-10-07T09:48:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.507+0000 7fe5c9bff700 1 -- start start 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.507+0000 7fe5c9bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c415f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.507+0000 7fe5c9bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5c4159750 0x7fe5c4159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.507+0000 7fe5c9bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c4159b20 0x7fe5c415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c9bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40b4330 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c9bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c415cca0 con 0x7fe5c4159b20 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c9bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c415cf80 con 0x7fe5c4159750 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c415f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c415f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50092/0 (socket says 172.21.15.29:50092) 2023-10-07T09:48:27.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 -- 172.21.15.29:0/566544802 learned_addr learned my addr 172.21.15.29:0/566544802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c3fff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c4159b20 0x7fe5c415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c2ffd700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5c4159750 0x7fe5c4159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 -- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5c4159750 msgr2=0x7fe5c4159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5c4159750 0x7fe5c4159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 -- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c4159b20 msgr2=0x7fe5c415a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c4159b20 0x7fe5c415a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c37fe700 1 -- 172.21.15.29:0/566544802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc009a20 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.508+0000 7fe5c2ffd700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5c4159750 0x7fe5c4159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.509+0000 7fe5c37fe700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c415f860 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400ec80 tx=0x7fe5b4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.509+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400c850 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.509+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b400c9b0 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.509+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400cd30 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.509+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c40b4580 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.509+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c415dd70 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.510+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c40b4bd0 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.511+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe5b4013680 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.512+0000 7fe5ab7fe700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5b0066f30 0x7fe5b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.512+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe5b4084e50 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.512+0000 7fe5c2ffd700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5b0066f30 0x7fe5b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.513+0000 7fe5c2ffd700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5b0066f30 0x7fe5b00693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe5c41605c0 tx=0x7fe5b8009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.516+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5b4051ea0 con 0x7fe5c40ce8a0 2023-10-07T09:48:27.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.712+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5c415d0f0 con 0x7fe5b0066f30 2023-10-07T09:48:27.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.724+0000 7fe5ab7fe700 1 -- 172.21.15.29:0/566544802 <== mgr.7582 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289269 (secure 0 0 0) 0x7fe5c415d0f0 con 0x7fe5b0066f30 2023-10-07T09:48:27.728 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5b0066f30 msgr2=0x7fe5b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5b0066f30 0x7fe5b00693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe5c41605c0 tx=0x7fe5b8009040 comp rx=0 tx=0).stop 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 msgr2=0x7fe5c415f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c415f860 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400ec80 tx=0x7fe5b4009120 comp rx=0 tx=0).stop 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 shutdown_connections 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5b0066f30 0x7fe5b00693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5c4159b20 0x7fe5c415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5c40ce8a0 0x7fe5c415f860 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 --2- 172.21.15.29:0/566544802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5c4159750 0x7fe5c4159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 >> 172.21.15.29:0/566544802 conn(0x7fe5c4063a10 msgr2=0x7fe5c4069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 shutdown_connections 2023-10-07T09:48:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.728+0000 7fe5c9bff700 1 -- 172.21.15.29:0/566544802 wait complete. 2023-10-07T09:48:27.733 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:48:27.746 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-07T09:48:27.446414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2445596,"kb_used_data":14044,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529124,"statfs":{"total":773094113280,"available":770589822976,"internally_reserved":0,"allocated":14381056,"data_stored":7550940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001428"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351091+0000","last_change":"2023-10-07T09:48:23.341874+0000","last_active":"2023-10-07T09:48:23.351091+0000","last_peered":"2023-10-07T09:48:23.351091+0000","last_clean":"2023-10-07T09:48:23.351091+0000","last_became_active":"2023-10-07T09:48:23.341608+0000","last_became_peered":"2023-10-07T09:48:23.341608+0000","last_unstale":"2023-10-07T09:48:23.351091+0000","last_undegraded":"2023-10-07T09:48:23.351091+0000","last_fullsized":"2023-10-07T09:48:23.351091+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:25.918638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348448+0000","last_change":"2023-10-07T09:48:23.346038+0000","last_active":"2023-10-07T09:48:23.348448+0000","last_peered":"2023-10-07T09:48:23.348448+0000","last_clean":"2023-10-07T09:48:23.348448+0000","last_became_active":"2023-10-07T09:48:23.345564+0000","last_became_peered":"2023-10-07T09:48:23.345564+0000","last_unstale":"2023-10-07T09:48:23.348448+0000","last_undegraded":"2023-10-07T09:48:23.348448+0000","last_fullsized":"2023-10-07T09:48:23.348448+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:56:57.842209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351273+0000","last_change":"2023-10-07T09:48:23.347956+0000","last_active":"2023-10-07T09:48:23.351273+0000","last_peered":"2023-10-07T09:48:23.351273+0000","last_clean":"2023-10-07T09:48:23.351273+0000","last_became_active":"2023-10-07T09:48:23.346753+0000","last_became_peered":"2023-10-07T09:48:23.346753+0000","last_unstale":"2023-10-07T09:48:23.351273+0000","last_undegraded":"2023-10-07T09:48:23.351273+0000","last_fullsized":"2023-10-07T09:48:23.351273+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:34:54.914609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.343025+0000","last_change":"2023-10-07T09:48:23.337931+0000","last_active":"2023-10-07T09:48:23.343025+0000","last_peered":"2023-10-07T09:48:23.343025+0000","last_clean":"2023-10-07T09:48:23.343025+0000","last_became_active":"2023-10-07T09:48:23.337598+0000","last_became_peered":"2023-10-07T09:48:23.337598+0000","last_unstale":"2023-10-07T09:48:23.343025+0000","last_undegraded":"2023-10-07T09:48:23.343025+0000","last_fullsized":"2023-10-07T09:48:23.343025+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:13:03.173904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.355277+0000","last_change":"2023-10-07T09:48:23.353245+0000","last_active":"2023-10-07T09:48:23.355277+0000","last_peered":"2023-10-07T09:48:23.355277+0000","last_clean":"2023-10-07T09:48:23.355277+0000","last_became_active":"2023-10-07T09:48:23.352963+0000","last_became_peered":"2023-10-07T09:48:23.352963+0000","last_unstale":"2023-10-07T09:48:23.355277+0000","last_undegraded":"2023-10-07T09:48:23.355277+0000","last_fullsized":"2023-10-07T09:48:23.355277+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:54.574890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350880+0000","last_change":"2023-10-07T09:48:23.344753+0000","last_active":"2023-10-07T09:48:23.350880+0000","last_peered":"2023-10-07T09:48:23.350880+0000","last_clean":"2023-10-07T09:48:23.350880+0000","last_became_active":"2023-10-07T09:48:23.344469+0000","last_became_peered":"2023-10-07T09:48:23.344469+0000","last_unstale":"2023-10-07T09:48:23.350880+0000","last_undegraded":"2023-10-07T09:48:23.350880+0000","last_fullsized":"2023-10-07T09:48:23.350880+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:34:17.956936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345210+0000","last_change":"2023-10-07T09:48:23.339263+0000","last_active":"2023-10-07T09:48:23.345210+0000","last_peered":"2023-10-07T09:48:23.345210+0000","last_clean":"2023-10-07T09:48:23.345210+0000","last_became_active":"2023-10-07T09:48:23.338797+0000","last_became_peered":"2023-10-07T09:48:23.338797+0000","last_unstale":"2023-10-07T09:48:23.345210+0000","last_undegraded":"2023-10-07T09:48:23.345210+0000","last_fullsized":"2023-10-07T09:48:23.345210+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:04.865056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345325+0000","last_change":"2023-10-07T09:48:23.340253+0000","last_active":"2023-10-07T09:48:23.345325+0000","last_peered":"2023-10-07T09:48:23.345325+0000","last_clean":"2023-10-07T09:48:23.345325+0000","last_became_active":"2023-10-07T09:48:23.339633+0000","last_became_peered":"2023-10-07T09:48:23.339633+0000","last_unstale":"2023-10-07T09:48:23.345325+0000","last_undegraded":"2023-10-07T09:48:23.345325+0000","last_fullsized":"2023-10-07T09:48:23.345325+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:01:43.759657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.341980+0000","last_change":"2023-10-07T09:48:23.338796+0000","last_active":"2023-10-07T09:48:23.341980+0000","last_peered":"2023-10-07T09:48:23.341980+0000","last_clean":"2023-10-07T09:48:23.341980+0000","last_became_active":"2023-10-07T09:48:23.338511+0000","last_became_peered":"2023-10-07T09:48:23.338511+0000","last_unstale":"2023-10-07T09:48:23.341980+0000","last_undegraded":"2023-10-07T09:48:23.341980+0000","last_fullsized":"2023-10-07T09:48:23.341980+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:59.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.352459+0000","last_change":"2023-10-07T09:48:23.344760+0000","last_active":"2023-10-07T09:48:23.352459+0000","last_peered":"2023-10-07T09:48:23.352459+0000","last_clean":"2023-10-07T09:48:23.352459+0000","last_became_active":"2023-10-07T09:48:23.344488+0000","last_became_peered":"2023-10-07T09:48:23.344488+0000","last_unstale":"2023-10-07T09:48:23.352459+0000","last_undegraded":"2023-10-07T09:48:23.352459+0000","last_fullsized":"2023-10-07T09:48:23.352459+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:20.024623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347835+0000","last_change":"2023-10-07T09:48:23.340993+0000","last_active":"2023-10-07T09:48:23.347835+0000","last_peered":"2023-10-07T09:48:23.347835+0000","last_clean":"2023-10-07T09:48:23.347835+0000","last_became_active":"2023-10-07T09:48:23.340640+0000","last_became_peered":"2023-10-07T09:48:23.340640+0000","last_unstale":"2023-10-07T09:48:23.347835+0000","last_undegraded":"2023-10-07T09:48:23.347835+0000","last_fullsized":"2023-10-07T09:48:23.347835+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:42.543975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350864+0000","last_change":"2023-10-07T09:48:23.344117+0000","last_active":"2023-10-07T09:48:23.350864+0000","last_peered":"2023-10-07T09:48:23.350864+0000","last_clean":"2023-10-07T09:48:23.350864+0000","last_became_active":"2023-10-07T09:48:23.343859+0000","last_became_peered":"2023-10-07T09:48:23.343859+0000","last_unstale":"2023-10-07T09:48:23.350864+0000","last_undegraded":"2023-10-07T09:48:23.350864+0000","last_fullsized":"2023-10-07T09:48:23.350864+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:57:42.592436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346226+0000","last_change":"2023-10-07T09:48:23.343372+0000","last_active":"2023-10-07T09:48:23.346226+0000","last_peered":"2023-10-07T09:48:23.346226+0000","last_clean":"2023-10-07T09:48:23.346226+0000","last_became_active":"2023-10-07T09:48:23.342925+0000","last_became_peered":"2023-10-07T09:48:23.342925+0000","last_unstale":"2023-10-07T09:48:23.346226+0000","last_undegraded":"2023-10-07T09:48:23.346226+0000","last_fullsized":"2023-10-07T09:48:23.346226+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:22:05.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345208+0000","last_change":"2023-10-07T09:48:23.339955+0000","last_active":"2023-10-07T09:48:23.345208+0000","last_peered":"2023-10-07T09:48:23.345208+0000","last_clean":"2023-10-07T09:48:23.345208+0000","last_became_active":"2023-10-07T09:48:23.339691+0000","last_became_peered":"2023-10-07T09:48:23.339691+0000","last_unstale":"2023-10-07T09:48:23.345208+0000","last_undegraded":"2023-10-07T09:48:23.345208+0000","last_fullsized":"2023-10-07T09:48:23.345208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:22:04.816400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350650+0000","last_change":"2023-10-07T09:48:23.346072+0000","last_active":"2023-10-07T09:48:23.350650+0000","last_peered":"2023-10-07T09:48:23.350650+0000","last_clean":"2023-10-07T09:48:23.350650+0000","last_became_active":"2023-10-07T09:48:23.345607+0000","last_became_peered":"2023-10-07T09:48:23.345607+0000","last_unstale":"2023-10-07T09:48:23.350650+0000","last_undegraded":"2023-10-07T09:48:23.350650+0000","last_fullsized":"2023-10-07T09:48:23.350650+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:31:44.235802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350380+0000","last_change":"2023-10-07T09:48:23.341085+0000","last_active":"2023-10-07T09:48:23.350380+0000","last_peered":"2023-10-07T09:48:23.350380+0000","last_clean":"2023-10-07T09:48:23.350380+0000","last_became_active":"2023-10-07T09:48:23.340450+0000","last_became_peered":"2023-10-07T09:48:23.340450+0000","last_unstale":"2023-10-07T09:48:23.350380+0000","last_undegraded":"2023-10-07T09:48:23.350380+0000","last_fullsized":"2023-10-07T09:48:23.350380+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:02:15.361286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347805+0000","last_change":"2023-10-07T09:48:23.340363+0000","last_active":"2023-10-07T09:48:23.347805+0000","last_peered":"2023-10-07T09:48:23.347805+0000","last_clean":"2023-10-07T09:48:23.347805+0000","last_became_active":"2023-10-07T09:48:23.339927+0000","last_became_peered":"2023-10-07T09:48:23.339927+0000","last_unstale":"2023-10-07T09:48:23.347805+0000","last_undegraded":"2023-10-07T09:48:23.347805+0000","last_fullsized":"2023-10-07T09:48:23.347805+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:04.326985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344368+0000","last_change":"2023-10-07T09:48:23.336029+0000","last_active":"2023-10-07T09:48:23.344368+0000","last_peered":"2023-10-07T09:48:23.344368+0000","last_clean":"2023-10-07T09:48:23.344368+0000","last_became_active":"2023-10-07T09:48:23.335725+0000","last_became_peered":"2023-10-07T09:48:23.335725+0000","last_unstale":"2023-10-07T09:48:23.344368+0000","last_undegraded":"2023-10-07T09:48:23.344368+0000","last_fullsized":"2023-10-07T09:48:23.344368+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:56:37.006142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346447+0000","last_change":"2023-10-07T09:48:23.343333+0000","last_active":"2023-10-07T09:48:23.346447+0000","last_peered":"2023-10-07T09:48:23.346447+0000","last_clean":"2023-10-07T09:48:23.346447+0000","last_became_active":"2023-10-07T09:48:23.342944+0000","last_became_peered":"2023-10-07T09:48:23.342944+0000","last_unstale":"2023-10-07T09:48:23.346447+0000","last_undegraded":"2023-10-07T09:48:23.346447+0000","last_fullsized":"2023-10-07T09:48:23.346447+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:23:10.235787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348443+0000","last_change":"2023-10-07T09:48:23.337224+0000","last_active":"2023-10-07T09:48:23.348443+0000","last_peered":"2023-10-07T09:48:23.348443+0000","last_clean":"2023-10-07T09:48:23.348443+0000","last_became_active":"2023-10-07T09:48:23.336889+0000","last_became_peered":"2023-10-07T09:48:23.336889+0000","last_unstale":"2023-10-07T09:48:23.348443+0000","last_undegraded":"2023-10-07T09:48:23.348443+0000","last_fullsized":"2023-10-07T09:48:23.348443+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:46:27.519448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332057+0000","last_change":"2023-10-07T09:48:23.330437+0000","last_active":"2023-10-07T09:48:23.332057+0000","last_peered":"2023-10-07T09:48:23.332057+0000","last_clean":"2023-10-07T09:48:23.332057+0000","last_became_active":"2023-10-07T09:48:23.330111+0000","last_became_peered":"2023-10-07T09:48:23.330111+0000","last_unstale":"2023-10-07T09:48:23.332057+0000","last_undegraded":"2023-10-07T09:48:23.332057+0000","last_fullsized":"2023-10-07T09:48:23.332057+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:32:56.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349181+0000","last_change":"2023-10-07T09:48:23.343109+0000","last_active":"2023-10-07T09:48:23.349181+0000","last_peered":"2023-10-07T09:48:23.349181+0000","last_clean":"2023-10-07T09:48:23.349181+0000","last_became_active":"2023-10-07T09:48:23.342838+0000","last_became_peered":"2023-10-07T09:48:23.342838+0000","last_unstale":"2023-10-07T09:48:23.349181+0000","last_undegraded":"2023-10-07T09:48:23.349181+0000","last_fullsized":"2023-10-07T09:48:23.349181+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:56:57.842209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347964+0000","last_change":"2023-10-07T09:48:23.341800+0000","last_active":"2023-10-07T09:48:23.347964+0000","last_peered":"2023-10-07T09:48:23.347964+0000","last_clean":"2023-10-07T09:48:23.347964+0000","last_became_active":"2023-10-07T09:48:23.341564+0000","last_became_peered":"2023-10-07T09:48:23.341564+0000","last_unstale":"2023-10-07T09:48:23.347964+0000","last_undegraded":"2023-10-07T09:48:23.347964+0000","last_fullsized":"2023-10-07T09:48:23.347964+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:36.593363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349109+0000","last_change":"2023-10-07T09:48:23.339774+0000","last_active":"2023-10-07T09:48:23.349109+0000","last_peered":"2023-10-07T09:48:23.349109+0000","last_clean":"2023-10-07T09:48:23.349109+0000","last_became_active":"2023-10-07T09:48:23.339493+0000","last_became_peered":"2023-10-07T09:48:23.339493+0000","last_unstale":"2023-10-07T09:48:23.349109+0000","last_undegraded":"2023-10-07T09:48:23.349109+0000","last_fullsized":"2023-10-07T09:48:23.349109+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:04.326985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347785+0000","last_change":"2023-10-07T09:48:23.340177+0000","last_active":"2023-10-07T09:48:23.347785+0000","last_peered":"2023-10-07T09:48:23.347785+0000","last_clean":"2023-10-07T09:48:23.347785+0000","last_became_active":"2023-10-07T09:48:23.339605+0000","last_became_peered":"2023-10-07T09:48:23.339605+0000","last_unstale":"2023-10-07T09:48:23.347785+0000","last_undegraded":"2023-10-07T09:48:23.347785+0000","last_fullsized":"2023-10-07T09:48:23.347785+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:21.035432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.340049+0000","last_change":"2023-10-07T09:48:23.334256+0000","last_active":"2023-10-07T09:48:23.340049+0000","last_peered":"2023-10-07T09:48:23.340049+0000","last_clean":"2023-10-07T09:48:23.340049+0000","last_became_active":"2023-10-07T09:48:23.333972+0000","last_became_peered":"2023-10-07T09:48:23.333972+0000","last_unstale":"2023-10-07T09:48:23.340049+0000","last_undegraded":"2023-10-07T09:48:23.340049+0000","last_fullsized":"2023-10-07T09:48:23.340049+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:27:42.938442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350962+0000","last_change":"2023-10-07T09:48:23.344515+0000","last_active":"2023-10-07T09:48:23.350962+0000","last_peered":"2023-10-07T09:48:23.350962+0000","last_clean":"2023-10-07T09:48:23.350962+0000","last_became_active":"2023-10-07T09:48:23.344155+0000","last_became_peered":"2023-10-07T09:48:23.344155+0000","last_unstale":"2023-10-07T09:48:23.350962+0000","last_undegraded":"2023-10-07T09:48:23.350962+0000","last_fullsized":"2023-10-07T09:48:23.350962+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:50.645114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349191+0000","last_change":"2023-10-07T09:48:23.341891+0000","last_active":"2023-10-07T09:48:23.349191+0000","last_peered":"2023-10-07T09:48:23.349191+0000","last_clean":"2023-10-07T09:48:23.349191+0000","last_became_active":"2023-10-07T09:48:23.341606+0000","last_became_peered":"2023-10-07T09:48:23.341606+0000","last_unstale":"2023-10-07T09:48:23.349191+0000","last_undegraded":"2023-10-07T09:48:23.349191+0000","last_fullsized":"2023-10-07T09:48:23.349191+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:46:12.906428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347762+0000","last_change":"2023-10-07T09:48:23.341305+0000","last_active":"2023-10-07T09:48:23.347762+0000","last_peered":"2023-10-07T09:48:23.347762+0000","last_clean":"2023-10-07T09:48:23.347762+0000","last_became_active":"2023-10-07T09:48:23.340977+0000","last_became_peered":"2023-10-07T09:48:23.340977+0000","last_unstale":"2023-10-07T09:48:23.347762+0000","last_undegraded":"2023-10-07T09:48:23.347762+0000","last_fullsized":"2023-10-07T09:48:23.347762+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:00:36.261617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344600+0000","last_change":"2023-10-07T09:48:23.337008+0000","last_active":"2023-10-07T09:48:23.344600+0000","last_peered":"2023-10-07T09:48:23.344600+0000","last_clean":"2023-10-07T09:48:23.344600+0000","last_became_active":"2023-10-07T09:48:23.336638+0000","last_became_peered":"2023-10-07T09:48:23.336638+0000","last_unstale":"2023-10-07T09:48:23.344600+0000","last_undegraded":"2023-10-07T09:48:23.344600+0000","last_fullsized":"2023-10-07T09:48:23.344600+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:05:07.769412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.342075+0000","last_change":"2023-10-07T09:48:23.336496+0000","last_active":"2023-10-07T09:48:23.342075+0000","last_peered":"2023-10-07T09:48:23.342075+0000","last_clean":"2023-10-07T09:48:23.342075+0000","last_became_active":"2023-10-07T09:48:23.336069+0000","last_became_peered":"2023-10-07T09:48:23.336069+0000","last_unstale":"2023-10-07T09:48:23.342075+0000","last_undegraded":"2023-10-07T09:48:23.342075+0000","last_fullsized":"2023-10-07T09:48:23.342075+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:42:12.564256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350605+0000","last_change":"2023-10-07T09:48:23.345549+0000","last_active":"2023-10-07T09:48:23.350605+0000","last_peered":"2023-10-07T09:48:23.350605+0000","last_clean":"2023-10-07T09:48:23.350605+0000","last_became_active":"2023-10-07T09:48:23.345037+0000","last_became_peered":"2023-10-07T09:48:23.345037+0000","last_unstale":"2023-10-07T09:48:23.350605+0000","last_undegraded":"2023-10-07T09:48:23.350605+0000","last_fullsized":"2023-10-07T09:48:23.350605+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:04.865056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350335+0000","last_change":"2023-10-07T09:48:23.344155+0000","last_active":"2023-10-07T09:48:23.350335+0000","last_peered":"2023-10-07T09:48:23.350335+0000","last_clean":"2023-10-07T09:48:23.350335+0000","last_became_active":"2023-10-07T09:48:23.343800+0000","last_became_peered":"2023-10-07T09:48:23.343800+0000","last_unstale":"2023-10-07T09:48:23.350335+0000","last_undegraded":"2023-10-07T09:48:23.350335+0000","last_fullsized":"2023-10-07T09:48:23.350335+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:40:52.846704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"132'4","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347832+0000","last_change":"2023-10-07T09:48:22.334213+0000","last_active":"2023-10-07T09:48:23.347832+0000","last_peered":"2023-10-07T09:48:23.347832+0000","last_clean":"2023-10-07T09:48:23.347832+0000","last_became_active":"2023-10-07T09:48:22.333881+0000","last_became_peered":"2023-10-07T09:48:22.333881+0000","last_unstale":"2023-10-07T09:48:23.347832+0000","last_undegraded":"2023-10-07T09:48:23.347832+0000","last_fullsized":"2023-10-07T09:48:23.347832+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:56.497314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345735+0000","last_change":"2023-10-07T09:48:23.339647+0000","last_active":"2023-10-07T09:48:23.345735+0000","last_peered":"2023-10-07T09:48:23.345735+0000","last_clean":"2023-10-07T09:48:23.345735+0000","last_became_active":"2023-10-07T09:48:23.339402+0000","last_became_peered":"2023-10-07T09:48:23.339402+0000","last_unstale":"2023-10-07T09:48:23.345735+0000","last_undegraded":"2023-10-07T09:48:23.345735+0000","last_fullsized":"2023-10-07T09:48:23.345735+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:36.593363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347601+0000","last_change":"2023-10-07T09:48:22.330987+0000","last_active":"2023-10-07T09:48:23.347601+0000","last_peered":"2023-10-07T09:48:23.347601+0000","last_clean":"2023-10-07T09:48:23.347601+0000","last_became_active":"2023-10-07T09:48:22.330714+0000","last_became_peered":"2023-10-07T09:48:22.330714+0000","last_unstale":"2023-10-07T09:48:23.347601+0000","last_undegraded":"2023-10-07T09:48:23.347601+0000","last_fullsized":"2023-10-07T09:48:23.347601+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:01.031834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351015+0000","last_change":"2023-10-07T09:48:23.345479+0000","last_active":"2023-10-07T09:48:23.351015+0000","last_peered":"2023-10-07T09:48:23.351015+0000","last_clean":"2023-10-07T09:48:23.351015+0000","last_became_active":"2023-10-07T09:48:23.345085+0000","last_became_peered":"2023-10-07T09:48:23.345085+0000","last_unstale":"2023-10-07T09:48:23.351015+0000","last_undegraded":"2023-10-07T09:48:23.351015+0000","last_fullsized":"2023-10-07T09:48:23.351015+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:05:07.769412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351513+0000","last_change":"2023-10-07T09:48:22.338315+0000","last_active":"2023-10-07T09:48:23.351513+0000","last_peered":"2023-10-07T09:48:23.351513+0000","last_clean":"2023-10-07T09:48:23.351513+0000","last_became_active":"2023-10-07T09:48:22.338094+0000","last_became_peered":"2023-10-07T09:48:22.338094+0000","last_unstale":"2023-10-07T09:48:23.351513+0000","last_undegraded":"2023-10-07T09:48:23.351513+0000","last_fullsized":"2023-10-07T09:48:23.351513+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:30:48.653184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347288+0000","last_change":"2023-10-07T09:48:23.345765+0000","last_active":"2023-10-07T09:48:23.347288+0000","last_peered":"2023-10-07T09:48:23.347288+0000","last_clean":"2023-10-07T09:48:23.347288+0000","last_became_active":"2023-10-07T09:48:23.345492+0000","last_became_peered":"2023-10-07T09:48:23.345492+0000","last_unstale":"2023-10-07T09:48:23.347288+0000","last_undegraded":"2023-10-07T09:48:23.347288+0000","last_fullsized":"2023-10-07T09:48:23.347288+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:06:27.733172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351759+0000","last_change":"2023-10-07T09:48:22.337285+0000","last_active":"2023-10-07T09:48:23.351759+0000","last_peered":"2023-10-07T09:48:23.351759+0000","last_clean":"2023-10-07T09:48:23.351759+0000","last_became_active":"2023-10-07T09:48:22.336987+0000","last_became_peered":"2023-10-07T09:48:22.336987+0000","last_unstale":"2023-10-07T09:48:23.351759+0000","last_undegraded":"2023-10-07T09:48:23.351759+0000","last_fullsized":"2023-10-07T09:48:23.351759+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:11.162155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.361283+0000","last_change":"2023-10-07T09:48:23.353331+0000","last_active":"2023-10-07T09:48:23.361283+0000","last_peered":"2023-10-07T09:48:23.361283+0000","last_clean":"2023-10-07T09:48:23.361283+0000","last_became_active":"2023-10-07T09:48:23.353031+0000","last_became_peered":"2023-10-07T09:48:23.353031+0000","last_unstale":"2023-10-07T09:48:23.361283+0000","last_undegraded":"2023-10-07T09:48:23.361283+0000","last_fullsized":"2023-10-07T09:48:23.361283+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:19:19.377309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351603+0000","last_change":"2023-10-07T09:48:22.325792+0000","last_active":"2023-10-07T09:48:23.351603+0000","last_peered":"2023-10-07T09:48:23.351603+0000","last_clean":"2023-10-07T09:48:23.351603+0000","last_became_active":"2023-10-07T09:48:22.325488+0000","last_became_peered":"2023-10-07T09:48:22.325488+0000","last_unstale":"2023-10-07T09:48:23.351603+0000","last_undegraded":"2023-10-07T09:48:23.351603+0000","last_fullsized":"2023-10-07T09:48:23.351603+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:29:39.822105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348550+0000","last_change":"2023-10-07T09:48:23.346681+0000","last_active":"2023-10-07T09:48:23.348550+0000","last_peered":"2023-10-07T09:48:23.348550+0000","last_clean":"2023-10-07T09:48:23.348550+0000","last_became_active":"2023-10-07T09:48:23.346303+0000","last_became_peered":"2023-10-07T09:48:23.346303+0000","last_unstale":"2023-10-07T09:48:23.348550+0000","last_undegraded":"2023-10-07T09:48:23.348550+0000","last_fullsized":"2023-10-07T09:48:23.348550+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:48.391128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349442+0000","last_change":"2023-10-07T09:48:22.324869+0000","last_active":"2023-10-07T09:48:23.349442+0000","last_peered":"2023-10-07T09:48:23.349442+0000","last_clean":"2023-10-07T09:48:23.349442+0000","last_became_active":"2023-10-07T09:48:22.324599+0000","last_became_peered":"2023-10-07T09:48:22.324599+0000","last_unstale":"2023-10-07T09:48:23.349442+0000","last_undegraded":"2023-10-07T09:48:23.349442+0000","last_fullsized":"2023-10-07T09:48:23.349442+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:40:51.839090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350410+0000","last_change":"2023-10-07T09:48:23.341280+0000","last_active":"2023-10-07T09:48:23.350410+0000","last_peered":"2023-10-07T09:48:23.350410+0000","last_clean":"2023-10-07T09:48:23.350410+0000","last_became_active":"2023-10-07T09:48:23.340838+0000","last_became_peered":"2023-10-07T09:48:23.340838+0000","last_unstale":"2023-10-07T09:48:23.350410+0000","last_undegraded":"2023-10-07T09:48:23.350410+0000","last_fullsized":"2023-10-07T09:48:23.350410+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:09:32.323274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324734+0000","last_change":"2023-10-07T09:48:22.332754+0000","last_active":"2023-10-07T09:48:23.324734+0000","last_peered":"2023-10-07T09:48:23.324734+0000","last_clean":"2023-10-07T09:48:23.324734+0000","last_became_active":"2023-10-07T09:48:22.332423+0000","last_became_peered":"2023-10-07T09:48:22.332423+0000","last_unstale":"2023-10-07T09:48:23.324734+0000","last_undegraded":"2023-10-07T09:48:23.324734+0000","last_fullsized":"2023-10-07T09:48:23.324734+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:34:16.949322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345167+0000","last_change":"2023-10-07T09:48:23.339083+0000","last_active":"2023-10-07T09:48:23.345167+0000","last_peered":"2023-10-07T09:48:23.345167+0000","last_clean":"2023-10-07T09:48:23.345167+0000","last_became_active":"2023-10-07T09:48:23.338465+0000","last_became_peered":"2023-10-07T09:48:23.338465+0000","last_unstale":"2023-10-07T09:48:23.345167+0000","last_undegraded":"2023-10-07T09:48:23.345167+0000","last_fullsized":"2023-10-07T09:48:23.345167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:20:07.941612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.325072+0000","last_change":"2023-10-07T09:48:22.336899+0000","last_active":"2023-10-07T09:48:23.325072+0000","last_peered":"2023-10-07T09:48:23.325072+0000","last_clean":"2023-10-07T09:48:23.325072+0000","last_became_active":"2023-10-07T09:48:22.331349+0000","last_became_peered":"2023-10-07T09:48:22.331349+0000","last_unstale":"2023-10-07T09:48:23.325072+0000","last_undegraded":"2023-10-07T09:48:23.325072+0000","last_fullsized":"2023-10-07T09:48:23.325072+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:56:56.834596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346251+0000","last_change":"2023-10-07T09:48:23.345316+0000","last_active":"2023-10-07T09:48:23.346251+0000","last_peered":"2023-10-07T09:48:23.346251+0000","last_clean":"2023-10-07T09:48:23.346251+0000","last_became_active":"2023-10-07T09:48:23.344917+0000","last_became_peered":"2023-10-07T09:48:23.344917+0000","last_unstale":"2023-10-07T09:48:23.346251+0000","last_undegraded":"2023-10-07T09:48:23.346251+0000","last_fullsized":"2023-10-07T09:48:23.346251+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:31:44.235802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332274+0000","last_change":"2023-10-07T09:48:22.337849+0000","last_active":"2023-10-07T09:48:23.332274+0000","last_peered":"2023-10-07T09:48:23.332274+0000","last_clean":"2023-10-07T09:48:23.332274+0000","last_became_active":"2023-10-07T09:48:22.337596+0000","last_became_peered":"2023-10-07T09:48:22.337596+0000","last_unstale":"2023-10-07T09:48:23.332274+0000","last_undegraded":"2023-10-07T09:48:23.332274+0000","last_fullsized":"2023-10-07T09:48:23.332274+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:12:22.941001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345752+0000","last_change":"2023-10-07T09:48:23.342247+0000","last_active":"2023-10-07T09:48:23.345752+0000","last_peered":"2023-10-07T09:48:23.345752+0000","last_clean":"2023-10-07T09:48:23.345752+0000","last_became_active":"2023-10-07T09:48:23.341934+0000","last_became_peered":"2023-10-07T09:48:23.341934+0000","last_unstale":"2023-10-07T09:48:23.345752+0000","last_undegraded":"2023-10-07T09:48:23.345752+0000","last_fullsized":"2023-10-07T09:48:23.345752+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:46:12.906428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.319537+0000","last_change":"2023-10-07T09:48:22.329924+0000","last_active":"2023-10-07T09:48:23.319537+0000","last_peered":"2023-10-07T09:48:23.319537+0000","last_clean":"2023-10-07T09:48:23.319537+0000","last_became_active":"2023-10-07T09:48:22.329619+0000","last_became_peered":"2023-10-07T09:48:22.329619+0000","last_unstale":"2023-10-07T09:48:23.319537+0000","last_undegraded":"2023-10-07T09:48:23.319537+0000","last_fullsized":"2023-10-07T09:48:23.319537+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:49.637500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338932+0000","last_change":"2023-10-07T09:48:23.334684+0000","last_active":"2023-10-07T09:48:23.338932+0000","last_peered":"2023-10-07T09:48:23.338932+0000","last_clean":"2023-10-07T09:48:23.338932+0000","last_became_active":"2023-10-07T09:48:23.334412+0000","last_became_peered":"2023-10-07T09:48:23.334412+0000","last_unstale":"2023-10-07T09:48:23.338932+0000","last_undegraded":"2023-10-07T09:48:23.338932+0000","last_fullsized":"2023-10-07T09:48:23.338932+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:46:27.519448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"132'3","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.363780+0000","last_change":"2023-10-07T09:48:22.331754+0000","last_active":"2023-10-07T09:48:23.363780+0000","last_peered":"2023-10-07T09:48:23.363780+0000","last_clean":"2023-10-07T09:48:23.363780+0000","last_became_active":"2023-10-07T09:48:22.331416+0000","last_became_peered":"2023-10-07T09:48:22.331416+0000","last_unstale":"2023-10-07T09:48:23.363780+0000","last_undegraded":"2023-10-07T09:48:23.363780+0000","last_fullsized":"2023-10-07T09:48:23.363780+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:51.675105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344769+0000","last_change":"2023-10-07T09:48:23.339889+0000","last_active":"2023-10-07T09:48:23.344769+0000","last_peered":"2023-10-07T09:48:23.344769+0000","last_clean":"2023-10-07T09:48:23.344769+0000","last_became_active":"2023-10-07T09:48:23.339624+0000","last_became_peered":"2023-10-07T09:48:23.339624+0000","last_unstale":"2023-10-07T09:48:23.344769+0000","last_undegraded":"2023-10-07T09:48:23.344769+0000","last_fullsized":"2023-10-07T09:48:23.344769+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:34:17.956936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"132'6","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.365618+0000","last_change":"2023-10-07T09:48:22.336553+0000","last_active":"2023-10-07T09:48:23.365618+0000","last_peered":"2023-10-07T09:48:23.365618+0000","last_clean":"2023-10-07T09:48:23.365618+0000","last_became_active":"2023-10-07T09:48:22.334979+0000","last_became_peered":"2023-10-07T09:48:22.334979+0000","last_unstale":"2023-10-07T09:48:23.365618+0000","last_undegraded":"2023-10-07T09:48:23.365618+0000","last_fullsized":"2023-10-07T09:48:23.365618+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:41.536361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338939+0000","last_change":"2023-10-07T09:48:23.331997+0000","last_active":"2023-10-07T09:48:23.338939+0000","last_peered":"2023-10-07T09:48:23.338939+0000","last_clean":"2023-10-07T09:48:23.338939+0000","last_became_active":"2023-10-07T09:48:23.331745+0000","last_became_peered":"2023-10-07T09:48:23.331745+0000","last_unstale":"2023-10-07T09:48:23.338939+0000","last_undegraded":"2023-10-07T09:48:23.338939+0000","last_fullsized":"2023-10-07T09:48:23.338939+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:42.543975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.355154+0000","last_change":"2023-10-07T09:48:22.331792+0000","last_active":"2023-10-07T09:48:23.355154+0000","last_peered":"2023-10-07T09:48:23.355154+0000","last_clean":"2023-10-07T09:48:23.355154+0000","last_became_active":"2023-10-07T09:48:22.331546+0000","last_became_peered":"2023-10-07T09:48:22.331546+0000","last_unstale":"2023-10-07T09:48:23.355154+0000","last_undegraded":"2023-10-07T09:48:23.355154+0000","last_fullsized":"2023-10-07T09:48:23.355154+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:27.785695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338733+0000","last_change":"2023-10-07T09:48:23.333225+0000","last_active":"2023-10-07T09:48:23.338733+0000","last_peered":"2023-10-07T09:48:23.338733+0000","last_clean":"2023-10-07T09:48:23.338733+0000","last_became_active":"2023-10-07T09:48:23.332964+0000","last_became_peered":"2023-10-07T09:48:23.332964+0000","last_unstale":"2023-10-07T09:48:23.338733+0000","last_undegraded":"2023-10-07T09:48:23.338733+0000","last_fullsized":"2023-10-07T09:48:23.338733+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:17.740390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.328148+0000","last_change":"2023-10-07T09:48:22.336761+0000","last_active":"2023-10-07T09:48:23.328148+0000","last_peered":"2023-10-07T09:48:23.328148+0000","last_clean":"2023-10-07T09:48:23.328148+0000","last_became_active":"2023-10-07T09:48:22.336441+0000","last_became_peered":"2023-10-07T09:48:22.336441+0000","last_unstale":"2023-10-07T09:48:23.328148+0000","last_undegraded":"2023-10-07T09:48:23.328148+0000","last_fullsized":"2023-10-07T09:48:23.328148+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:41.536361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338773+0000","last_change":"2023-10-07T09:48:23.328101+0000","last_active":"2023-10-07T09:48:23.338773+0000","last_peered":"2023-10-07T09:48:23.338773+0000","last_clean":"2023-10-07T09:48:23.338773+0000","last_became_active":"2023-10-07T09:48:23.327810+0000","last_became_peered":"2023-10-07T09:48:23.327810+0000","last_unstale":"2023-10-07T09:48:23.338773+0000","last_undegraded":"2023-10-07T09:48:23.338773+0000","last_fullsized":"2023-10-07T09:48:23.338773+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:00:36.261617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"132'8","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.363837+0000","last_change":"2023-10-07T09:48:22.341514+0000","last_active":"2023-10-07T09:48:23.363837+0000","last_peered":"2023-10-07T09:48:23.363837+0000","last_clean":"2023-10-07T09:48:23.363837+0000","last_became_active":"2023-10-07T09:48:22.341191+0000","last_became_peered":"2023-10-07T09:48:22.341191+0000","last_unstale":"2023-10-07T09:48:23.363837+0000","last_undegraded":"2023-10-07T09:48:23.363837+0000","last_fullsized":"2023-10-07T09:48:23.363837+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:47:53.945223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.354603+0000","last_change":"2023-10-07T09:48:23.344238+0000","last_active":"2023-10-07T09:48:23.354603+0000","last_peered":"2023-10-07T09:48:23.354603+0000","last_clean":"2023-10-07T09:48:23.354603+0000","last_became_active":"2023-10-07T09:48:23.343965+0000","last_became_peered":"2023-10-07T09:48:23.343965+0000","last_unstale":"2023-10-07T09:48:23.354603+0000","last_undegraded":"2023-10-07T09:48:23.354603+0000","last_fullsized":"2023-10-07T09:48:23.354603+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:58.953342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.363556+0000","last_change":"2023-10-07T09:48:22.330758+0000","last_active":"2023-10-07T09:48:23.363556+0000","last_peered":"2023-10-07T09:48:23.363556+0000","last_clean":"2023-10-07T09:48:23.363556+0000","last_became_active":"2023-10-07T09:48:22.326927+0000","last_became_peered":"2023-10-07T09:48:22.326927+0000","last_unstale":"2023-10-07T09:48:23.363556+0000","last_undegraded":"2023-10-07T09:48:23.363556+0000","last_fullsized":"2023-10-07T09:48:23.363556+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:32:55.194513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350662+0000","last_change":"2023-10-07T09:48:23.345936+0000","last_active":"2023-10-07T09:48:23.350662+0000","last_peered":"2023-10-07T09:48:23.350662+0000","last_clean":"2023-10-07T09:48:23.350662+0000","last_became_active":"2023-10-07T09:48:23.345644+0000","last_became_peered":"2023-10-07T09:48:23.345644+0000","last_unstale":"2023-10-07T09:48:23.350662+0000","last_undegraded":"2023-10-07T09:48:23.350662+0000","last_fullsized":"2023-10-07T09:48:23.350662+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:02.947201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.352553+0000","last_change":"2023-10-07T09:48:22.336692+0000","last_active":"2023-10-07T09:48:23.352553+0000","last_peered":"2023-10-07T09:48:23.352553+0000","last_clean":"2023-10-07T09:48:23.352553+0000","last_became_active":"2023-10-07T09:48:22.336443+0000","last_became_peered":"2023-10-07T09:48:22.336443+0000","last_unstale":"2023-10-07T09:48:23.352553+0000","last_undegraded":"2023-10-07T09:48:23.352553+0000","last_fullsized":"2023-10-07T09:48:23.352553+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:41:57.402686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347821+0000","last_change":"2023-10-07T09:48:23.344863+0000","last_active":"2023-10-07T09:48:23.347821+0000","last_peered":"2023-10-07T09:48:23.347821+0000","last_clean":"2023-10-07T09:48:23.347821+0000","last_became_active":"2023-10-07T09:48:23.344588+0000","last_became_peered":"2023-10-07T09:48:23.344588+0000","last_unstale":"2023-10-07T09:48:23.347821+0000","last_undegraded":"2023-10-07T09:48:23.347821+0000","last_fullsized":"2023-10-07T09:48:23.347821+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:26:43.640061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349614+0000","last_change":"2023-10-07T09:48:22.323276+0000","last_active":"2023-10-07T09:48:23.349614+0000","last_peered":"2023-10-07T09:48:23.349614+0000","last_clean":"2023-10-07T09:48:23.349614+0000","last_became_active":"2023-10-07T09:48:22.322888+0000","last_became_peered":"2023-10-07T09:48:22.322888+0000","last_unstale":"2023-10-07T09:48:23.349614+0000","last_undegraded":"2023-10-07T09:48:23.349614+0000","last_fullsized":"2023-10-07T09:48:23.349614+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:47.383514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348365+0000","last_change":"2023-10-07T09:48:23.346816+0000","last_active":"2023-10-07T09:48:23.348365+0000","last_peered":"2023-10-07T09:48:23.348365+0000","last_clean":"2023-10-07T09:48:23.348365+0000","last_became_active":"2023-10-07T09:48:23.346273+0000","last_became_peered":"2023-10-07T09:48:23.346273+0000","last_unstale":"2023-10-07T09:48:23.348365+0000","last_undegraded":"2023-10-07T09:48:23.348365+0000","last_fullsized":"2023-10-07T09:48:23.348365+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:30.228699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.327170+0000","last_change":"2023-10-07T09:48:22.336458+0000","last_active":"2023-10-07T09:48:23.327170+0000","last_peered":"2023-10-07T09:48:23.327170+0000","last_clean":"2023-10-07T09:48:23.327170+0000","last_became_active":"2023-10-07T09:48:22.336172+0000","last_became_peered":"2023-10-07T09:48:22.336172+0000","last_unstale":"2023-10-07T09:48:23.327170+0000","last_undegraded":"2023-10-07T09:48:23.327170+0000","last_fullsized":"2023-10-07T09:48:23.327170+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:27.785695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345179+0000","last_change":"2023-10-07T09:48:23.338398+0000","last_active":"2023-10-07T09:48:23.345179+0000","last_peered":"2023-10-07T09:48:23.345179+0000","last_clean":"2023-10-07T09:48:23.345179+0000","last_became_active":"2023-10-07T09:48:23.338120+0000","last_became_peered":"2023-10-07T09:48:23.338120+0000","last_unstale":"2023-10-07T09:48:23.345179+0000","last_undegraded":"2023-10-07T09:48:23.345179+0000","last_fullsized":"2023-10-07T09:48:23.345179+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:28:45.863077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.355674+0000","last_change":"2023-10-07T09:48:22.335032+0000","last_active":"2023-10-07T09:48:23.355674+0000","last_peered":"2023-10-07T09:48:23.355674+0000","last_clean":"2023-10-07T09:48:23.355674+0000","last_became_active":"2023-10-07T09:48:22.334751+0000","last_became_peered":"2023-10-07T09:48:22.334751+0000","last_unstale":"2023-10-07T09:48:23.355674+0000","last_undegraded":"2023-10-07T09:48:23.355674+0000","last_fullsized":"2023-10-07T09:48:23.355674+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:56.497314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346964+0000","last_change":"2023-10-07T09:48:23.340717+0000","last_active":"2023-10-07T09:48:23.346964+0000","last_peered":"2023-10-07T09:48:23.346964+0000","last_clean":"2023-10-07T09:48:23.346964+0000","last_became_active":"2023-10-07T09:48:23.340416+0000","last_became_peered":"2023-10-07T09:48:23.340416+0000","last_unstale":"2023-10-07T09:48:23.346964+0000","last_undegraded":"2023-10-07T09:48:23.346964+0000","last_fullsized":"2023-10-07T09:48:23.346964+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:53:03.934767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332401+0000","last_change":"2023-10-07T09:48:22.340681+0000","last_active":"2023-10-07T09:48:23.332401+0000","last_peered":"2023-10-07T09:48:23.332401+0000","last_clean":"2023-10-07T09:48:23.332401+0000","last_became_active":"2023-10-07T09:48:22.340338+0000","last_became_peered":"2023-10-07T09:48:22.340338+0000","last_unstale":"2023-10-07T09:48:23.332401+0000","last_undegraded":"2023-10-07T09:48:23.332401+0000","last_fullsized":"2023-10-07T09:48:23.332401+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:56.497314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351254+0000","last_change":"2023-10-07T09:48:23.347384+0000","last_active":"2023-10-07T09:48:23.351254+0000","last_peered":"2023-10-07T09:48:23.351254+0000","last_clean":"2023-10-07T09:48:23.351254+0000","last_became_active":"2023-10-07T09:48:23.347124+0000","last_became_peered":"2023-10-07T09:48:23.347124+0000","last_unstale":"2023-10-07T09:48:23.351254+0000","last_undegraded":"2023-10-07T09:48:23.351254+0000","last_fullsized":"2023-10-07T09:48:23.351254+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:26.743514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.329367+0000","last_change":"2023-10-07T09:48:22.335023+0000","last_active":"2023-10-07T09:48:23.329367+0000","last_peered":"2023-10-07T09:48:23.329367+0000","last_clean":"2023-10-07T09:48:23.329367+0000","last_became_active":"2023-10-07T09:48:22.334730+0000","last_became_peered":"2023-10-07T09:48:22.334730+0000","last_unstale":"2023-10-07T09:48:23.329367+0000","last_undegraded":"2023-10-07T09:48:23.329367+0000","last_fullsized":"2023-10-07T09:48:23.329367+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:47:53.945223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346697+0000","last_change":"2023-10-07T09:48:23.337705+0000","last_active":"2023-10-07T09:48:23.346697+0000","last_peered":"2023-10-07T09:48:23.346697+0000","last_clean":"2023-10-07T09:48:23.346697+0000","last_became_active":"2023-10-07T09:48:23.337382+0000","last_became_peered":"2023-10-07T09:48:23.337382+0000","last_unstale":"2023-10-07T09:48:23.346697+0000","last_undegraded":"2023-10-07T09:48:23.346697+0000","last_fullsized":"2023-10-07T09:48:23.346697+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:36.593363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324825+0000","last_change":"2023-10-07T09:48:22.327707+0000","last_active":"2023-10-07T09:48:23.324825+0000","last_peered":"2023-10-07T09:48:23.324825+0000","last_clean":"2023-10-07T09:48:23.324825+0000","last_became_active":"2023-10-07T09:48:22.326474+0000","last_became_peered":"2023-10-07T09:48:22.326474+0000","last_unstale":"2023-10-07T09:48:23.324825+0000","last_undegraded":"2023-10-07T09:48:23.324825+0000","last_fullsized":"2023-10-07T09:48:23.324825+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:47.383514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345689+0000","last_change":"2023-10-07T09:48:23.341784+0000","last_active":"2023-10-07T09:48:23.345689+0000","last_peered":"2023-10-07T09:48:23.345689+0000","last_clean":"2023-10-07T09:48:23.345689+0000","last_became_active":"2023-10-07T09:48:23.341552+0000","last_became_peered":"2023-10-07T09:48:23.341552+0000","last_unstale":"2023-10-07T09:48:23.345689+0000","last_undegraded":"2023-10-07T09:48:23.345689+0000","last_fullsized":"2023-10-07T09:48:23.345689+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:12:18.525773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.361871+0000","last_change":"2023-10-07T09:48:22.342396+0000","last_active":"2023-10-07T09:48:23.361871+0000","last_peered":"2023-10-07T09:48:23.361871+0000","last_clean":"2023-10-07T09:48:23.361871+0000","last_became_active":"2023-10-07T09:48:22.342056+0000","last_became_peered":"2023-10-07T09:48:22.342056+0000","last_unstale":"2023-10-07T09:48:23.361871+0000","last_undegraded":"2023-10-07T09:48:23.361871+0000","last_fullsized":"2023-10-07T09:48:23.361871+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:01.939588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.339631+0000","last_change":"2023-10-07T09:48:23.332504+0000","last_active":"2023-10-07T09:48:23.339631+0000","last_peered":"2023-10-07T09:48:23.339631+0000","last_clean":"2023-10-07T09:48:23.339631+0000","last_became_active":"2023-10-07T09:48:23.332211+0000","last_became_peered":"2023-10-07T09:48:23.332211+0000","last_unstale":"2023-10-07T09:48:23.339631+0000","last_undegraded":"2023-10-07T09:48:23.339631+0000","last_fullsized":"2023-10-07T09:48:23.339631+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:26:10.005451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.327249+0000","last_change":"2023-10-07T09:48:22.323757+0000","last_active":"2023-10-07T09:48:23.327249+0000","last_peered":"2023-10-07T09:48:23.327249+0000","last_clean":"2023-10-07T09:48:23.327249+0000","last_became_active":"2023-10-07T09:48:22.323518+0000","last_became_peered":"2023-10-07T09:48:22.323518+0000","last_unstale":"2023-10-07T09:48:23.327249+0000","last_undegraded":"2023-10-07T09:48:23.327249+0000","last_fullsized":"2023-10-07T09:48:23.327249+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:32:55.194513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350842+0000","last_change":"2023-10-07T09:48:23.342913+0000","last_active":"2023-10-07T09:48:23.350842+0000","last_peered":"2023-10-07T09:48:23.350842+0000","last_clean":"2023-10-07T09:48:23.350842+0000","last_became_active":"2023-10-07T09:48:23.342663+0000","last_became_peered":"2023-10-07T09:48:23.342663+0000","last_unstale":"2023-10-07T09:48:23.350842+0000","last_undegraded":"2023-10-07T09:48:23.350842+0000","last_fullsized":"2023-10-07T09:48:23.350842+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:19:22.128645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324025+0000","last_change":"2023-10-07T09:48:22.330583+0000","last_active":"2023-10-07T09:48:23.324025+0000","last_peered":"2023-10-07T09:48:23.324025+0000","last_clean":"2023-10-07T09:48:23.324025+0000","last_became_active":"2023-10-07T09:48:22.327011+0000","last_became_peered":"2023-10-07T09:48:22.327011+0000","last_unstale":"2023-10-07T09:48:23.324025+0000","last_undegraded":"2023-10-07T09:48:23.324025+0000","last_fullsized":"2023-10-07T09:48:23.324025+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:23:46.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345651+0000","last_change":"2023-10-07T09:48:23.341553+0000","last_active":"2023-10-07T09:48:23.345651+0000","last_peered":"2023-10-07T09:48:23.345651+0000","last_clean":"2023-10-07T09:48:23.345651+0000","last_became_active":"2023-10-07T09:48:23.341286+0000","last_became_peered":"2023-10-07T09:48:23.341286+0000","last_unstale":"2023-10-07T09:48:23.345651+0000","last_undegraded":"2023-10-07T09:48:23.345651+0000","last_fullsized":"2023-10-07T09:48:23.345651+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:52:03.541591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.320043+0000","last_change":"2023-10-07T09:48:22.325907+0000","last_active":"2023-10-07T09:48:23.320043+0000","last_peered":"2023-10-07T09:48:23.320043+0000","last_clean":"2023-10-07T09:48:23.320043+0000","last_became_active":"2023-10-07T09:48:22.325640+0000","last_became_peered":"2023-10-07T09:48:22.325640+0000","last_unstale":"2023-10-07T09:48:23.320043+0000","last_undegraded":"2023-10-07T09:48:23.320043+0000","last_fullsized":"2023-10-07T09:48:23.320043+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:31.570060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.336164+0000","last_change":"2023-10-07T09:48:23.332805+0000","last_active":"2023-10-07T09:48:23.336164+0000","last_peered":"2023-10-07T09:48:23.336164+0000","last_clean":"2023-10-07T09:48:23.336164+0000","last_became_active":"2023-10-07T09:48:23.332531+0000","last_became_peered":"2023-10-07T09:48:23.332531+0000","last_unstale":"2023-10-07T09:48:23.336164+0000","last_undegraded":"2023-10-07T09:48:23.336164+0000","last_fullsized":"2023-10-07T09:48:23.336164+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:09:59.093385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.326783+0000","last_change":"2023-10-07T09:48:22.331697+0000","last_active":"2023-10-07T09:48:23.326783+0000","last_peered":"2023-10-07T09:48:23.326783+0000","last_clean":"2023-10-07T09:48:23.326783+0000","last_became_active":"2023-10-07T09:48:22.331401+0000","last_became_peered":"2023-10-07T09:48:22.331401+0000","last_unstale":"2023-10-07T09:48:23.326783+0000","last_undegraded":"2023-10-07T09:48:23.326783+0000","last_fullsized":"2023-10-07T09:48:23.326783+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:12:22.941001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348567+0000","last_change":"2023-10-07T09:48:23.346948+0000","last_active":"2023-10-07T09:48:23.348567+0000","last_peered":"2023-10-07T09:48:23.348567+0000","last_clean":"2023-10-07T09:48:23.348567+0000","last_became_active":"2023-10-07T09:48:23.346657+0000","last_became_peered":"2023-10-07T09:48:23.346657+0000","last_unstale":"2023-10-07T09:48:23.348567+0000","last_undegraded":"2023-10-07T09:48:23.348567+0000","last_fullsized":"2023-10-07T09:48:23.348567+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:17.740390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324889+0000","last_change":"2023-10-07T09:48:22.335219+0000","last_active":"2023-10-07T09:48:23.324889+0000","last_peered":"2023-10-07T09:48:23.324889+0000","last_clean":"2023-10-07T09:48:23.324889+0000","last_became_active":"2023-10-07T09:48:22.334988+0000","last_became_peered":"2023-10-07T09:48:22.334988+0000","last_unstale":"2023-10-07T09:48:23.324889+0000","last_undegraded":"2023-10-07T09:48:23.324889+0000","last_fullsized":"2023-10-07T09:48:23.324889+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:05:06.761798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345308+0000","last_change":"2023-10-07T09:48:23.339125+0000","last_active":"2023-10-07T09:48:23.345308+0000","last_peered":"2023-10-07T09:48:23.345308+0000","last_clean":"2023-10-07T09:48:23.345308+0000","last_became_active":"2023-10-07T09:48:23.338859+0000","last_became_peered":"2023-10-07T09:48:23.338859+0000","last_unstale":"2023-10-07T09:48:23.345308+0000","last_undegraded":"2023-10-07T09:48:23.345308+0000","last_fullsized":"2023-10-07T09:48:23.345308+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:11:15.941025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332752+0000","last_change":"2023-10-07T09:48:22.337369+0000","last_active":"2023-10-07T09:48:23.332752+0000","last_peered":"2023-10-07T09:48:23.332752+0000","last_clean":"2023-10-07T09:48:23.332752+0000","last_became_active":"2023-10-07T09:48:22.337080+0000","last_became_peered":"2023-10-07T09:48:22.337080+0000","last_unstale":"2023-10-07T09:48:23.332752+0000","last_undegraded":"2023-10-07T09:48:23.332752+0000","last_fullsized":"2023-10-07T09:48:23.332752+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:45.881008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.352206+0000","last_change":"2023-10-07T09:48:23.341978+0000","last_active":"2023-10-07T09:48:23.352206+0000","last_peered":"2023-10-07T09:48:23.352206+0000","last_clean":"2023-10-07T09:48:23.352206+0000","last_became_active":"2023-10-07T09:48:23.341510+0000","last_became_peered":"2023-10-07T09:48:23.341510+0000","last_unstale":"2023-10-07T09:48:23.352206+0000","last_undegraded":"2023-10-07T09:48:23.352206+0000","last_fullsized":"2023-10-07T09:48:23.352206+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:54.623904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324971+0000","last_change":"2023-10-07T09:48:22.326218+0000","last_active":"2023-10-07T09:48:23.324971+0000","last_peered":"2023-10-07T09:48:23.324971+0000","last_clean":"2023-10-07T09:48:23.324971+0000","last_became_active":"2023-10-07T09:48:22.325941+0000","last_became_peered":"2023-10-07T09:48:22.325941+0000","last_unstale":"2023-10-07T09:48:23.324971+0000","last_undegraded":"2023-10-07T09:48:23.324971+0000","last_fullsized":"2023-10-07T09:48:23.324971+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:26:11.281727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344685+0000","last_change":"2023-10-07T09:48:23.337780+0000","last_active":"2023-10-07T09:48:23.344685+0000","last_peered":"2023-10-07T09:48:23.344685+0000","last_clean":"2023-10-07T09:48:23.344685+0000","last_became_active":"2023-10-07T09:48:23.337512+0000","last_became_peered":"2023-10-07T09:48:23.337512+0000","last_unstale":"2023-10-07T09:48:23.344685+0000","last_undegraded":"2023-10-07T09:48:23.344685+0000","last_fullsized":"2023-10-07T09:48:23.344685+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:21.035432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.326709+0000","last_change":"2023-10-07T09:48:22.328166+0000","last_active":"2023-10-07T09:48:23.326709+0000","last_peered":"2023-10-07T09:48:23.326709+0000","last_clean":"2023-10-07T09:48:23.326709+0000","last_became_active":"2023-10-07T09:48:22.327881+0000","last_became_peered":"2023-10-07T09:48:22.327881+0000","last_unstale":"2023-10-07T09:48:23.326709+0000","last_undegraded":"2023-10-07T09:48:23.326709+0000","last_fullsized":"2023-10-07T09:48:23.326709+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:47:53.945223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"122'249","reported_seq":725,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.333427+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:48:23.333427+0000","last_peered":"2023-10-07T09:48:23.333427+0000","last_clean":"2023-10-07T09:48:23.333427+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:48:23.333427+0000","last_undegraded":"2023-10-07T09:48:23.333427+0000","last_fullsized":"2023-10-07T09:48:23.333427+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303576,"kb_used_data":2192,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068264,"statfs":{"total":96636764160,"available":96325902336,"internally_reserved":0,"allocated":2244608,"data_stored":1387488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71099999999999997,"15min":0.71699999999999997},"min":{"1min":0.433,"5min":0.39300000000000002,"15min":0.374},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69799999999999995,"15min":0.71999999999999997},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0349999999999999},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71699999999999997,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94799999999999995,"5min":0.98599999999999999,"15min":1.014},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72699999999999998,"15min":0.73999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.087,"5min":1.087,"15min":1.2430000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.54000000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.206},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71799999999999997,"15min":0.73699999999999999},"min":{"1min":0.50700000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.875,"5min":1.0309999999999999,"15min":1.071},"last":0.64700000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":0.73999999999999999},"min":{"1min":0.52000000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.56599999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.48999999999999999,"5min":0.438,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.93999999999999995,"15min":1.002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.64500000000000002},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.88100000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.45200000000000001,"5min":0.40699999999999997,"15min":0.36899999999999999},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68899999999999995,"15min":0.69099999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.625}]},{"osd":6,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46400000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72699999999999998,"15min":0.72199999999999998},"min":{"1min":0.52500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.65200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509665,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302992,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068848,"statfs":{"total":96636764160,"available":96326500352,"internally_reserved":0,"allocated":1646592,"data_stored":796602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70099999999999996,"15min":0.71099999999999997},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.41999999999999998},"max":{"1min":0.98899999999999999,"5min":1.274,"15min":1.274},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.45600000000000002,"5min":0.45200000000000001,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70399999999999996,"15min":0.71699999999999997},"min":{"1min":0.46700000000000003,"5min":0.46300000000000002,"15min":0.33700000000000002},"max":{"1min":0.95699999999999996,"5min":1.169,"15min":1.169},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.35899999999999999},"max":{"1min":0.83299999999999996,"5min":1.7,"15min":1.7},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.71799999999999997,"15min":0.73099999999999998},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":0.95499999999999996,"5min":1.482,"15min":1.482},"last":0.62},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76100000000000001,"15min":0.75900000000000001},"min":{"1min":0.63100000000000001,"5min":0.48899999999999999,"15min":0.40699999999999997},"max":{"1min":0.88800000000000001,"5min":1.643,"15min":1.643},"last":0.45700000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.753},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45200000000000001},"max":{"1min":1.012,"5min":1.458,"15min":1.458},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.752,"15min":0.755},"min":{"1min":0.53800000000000003,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.89100000000000001,"5min":1.006,"15min":1.006},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61899999999999999,"15min":0.61799999999999999},"min":{"1min":0.42999999999999999,"5min":0.36599999999999999,"15min":0.29899999999999999},"max":{"1min":0.85199999999999998,"5min":0.91000000000000003,"15min":1.01},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.85899999999999999,"5min":1.018,"15min":1.018},"last":0.81899999999999995}]},{"osd":5,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.51700000000000002,"5min":0.42299999999999999,"15min":0.41199999999999998},"max":{"1min":0.88300000000000001,"5min":1.036,"15min":1.0509999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72199999999999998,"15min":0.71699999999999997},"min":{"1min":0.58499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78100000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.099,"15min":1.099},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98799999999999999,"5min":0.99099999999999999,"15min":1.01},"last":0.74299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542367,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302996,"kb_used_data":1612,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068844,"statfs":{"total":96636764160,"available":96326496256,"internally_reserved":0,"allocated":1650688,"data_stored":796656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.41599999999999998,"5min":0.36299999999999999,"15min":0.316},"max":{"1min":0.746,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46300000000000002,"15min":0.47699999999999998},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.72099999999999997,"5min":0.73499999999999999,"15min":0.79600000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.55300000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":1.028},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.438,"5min":0.379,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.97599999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.38400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.89200000000000002},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76300000000000001,"15min":0.78300000000000003},"min":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.48599999999999999},"max":{"1min":1.0029999999999999,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73599999999999999,"15min":0.76600000000000001},"min":{"1min":0.58199999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94499999999999995,"5min":0.98399999999999999,"15min":1.196},"last":0.72099999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.61099999999999999,"5min":0.55000000000000004,"15min":0.46300000000000002},"max":{"1min":1.038,"5min":1.0660000000000001,"15min":1.2769999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.751,"15min":0.77900000000000003},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.98099999999999998,"5min":1.0680000000000001,"15min":1.3300000000000001},"last":0.755}]},{"osd":6,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78000000000000003,"15min":0.81100000000000005},"min":{"1min":0.58399999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77500000000000002,"15min":0.81599999999999995},"min":{"1min":0.59499999999999997,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80100000000000005,"15min":0.83099999999999996},"min":{"1min":0.57999999999999996,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.2450000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78900000000000003,"15min":0.83699999999999997},"min":{"1min":0.57599999999999996,"5min":0.53900000000000003,"15min":0.51700000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.385},"last":0.73199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303588,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068252,"statfs":{"total":96636764160,"available":96325890048,"internally_reserved":0,"allocated":2256896,"data_stored":1388548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47899999999999998,"15min":0.48799999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.622,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55200000000000005,"15min":0.56799999999999995},"min":{"1min":0.39700000000000002,"5min":0.38600000000000001,"15min":0.311},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85999999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.40100000000000002,"5min":0.34000000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.876,"15min":1.101},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.60799999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.72499999999999998,"15min":0.751},"min":{"1min":0.58499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.232},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78600000000000003,"15min":0.79400000000000004},"min":{"1min":0.59699999999999998,"5min":0.52300000000000002,"15min":0.48599999999999999},"max":{"1min":1.069,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.90000000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.745,"15min":0.77600000000000002},"min":{"1min":0.58099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0229999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.45500000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.194},"last":0.86399999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.59599999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.048,"5min":1.048,"15min":1.129},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77500000000000002,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97799999999999998,"5min":1.206,"15min":1.206},"last":0.92800000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78800000000000003,"15min":0.82299999999999995},"min":{"1min":0.61599999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.0820000000000001,"5min":1.0920000000000001,"15min":1.167},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78700000000000003,"15min":0.81999999999999995},"min":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.503},"max":{"1min":0.99299999999999999,"5min":1.238,"15min":1.238},"last":0.88200000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542368,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060660,"statfs":{"total":96636764160,"available":96318115840,"internally_reserved":0,"allocated":1642496,"data_stored":796532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29399999999999998,"15min":0.27500000000000002},"max":{"1min":0.59999999999999998,"5min":0.66200000000000003,"15min":1.258},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.32100000000000001,"15min":0.28999999999999998},"max":{"1min":0.65900000000000003,"5min":0.79000000000000004,"15min":1.2290000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":1.1950000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":1.3200000000000001},"last":0.61799999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.30599999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":1.5820000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.72099999999999997,"5min":0.79900000000000004,"15min":1.052},"last":0.437}]},{"osd":4,"last update":"Sat Oct 7 09:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.59499999999999997,"5min":0.48099999999999998,"15min":0.35699999999999998},"max":{"1min":0.94999999999999996,"5min":1.1120000000000001,"15min":1.4630000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.76400000000000001,"15min":0.79600000000000004},"min":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.327,"15min":1.8919999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.61199999999999999,"5min":0.52000000000000002,"15min":0.48999999999999999},"max":{"1min":0.95199999999999996,"5min":1.173,"15min":1.98},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.77000000000000002,"15min":0.78900000000000003},"min":{"1min":0.54800000000000004,"5min":0.52500000000000002,"15min":0.46100000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.5169999999999999},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.77500000000000002,"15min":0.81899999999999995},"min":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":0.91100000000000003,"5min":1.1379999999999999,"15min":1.931},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80200000000000005,"15min":0.82599999999999996},"min":{"1min":0.66000000000000003,"5min":0.56399999999999995,"15min":0.47299999999999998},"max":{"1min":1.115,"5min":1.2849999999999999,"15min":1.8120000000000001},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Oct 7 09:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.80500000000000005,"15min":0.83799999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.56999999999999995},"max":{"1min":1.0529999999999999,"5min":1.2450000000000001,"15min":1.8500000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80100000000000005,"15min":0.82799999999999996},"min":{"1min":0.61099999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":1.1439999999999999,"15min":1.3839999999999999},"last":0.50900000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542369,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311196,"kb_used_data":1620,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060644,"statfs":{"total":96636764160,"available":96318099456,"internally_reserved":0,"allocated":1658880,"data_stored":798138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.28299999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.9939999999999998,"5min":7.9939999999999998,"15min":7.9939999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.32200000000000001},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":5.5359999999999996,"5min":5.5359999999999996,"15min":5.5359999999999996},"last":0.439},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.65500000000000003,"15min":0.61399999999999999},"min":{"1min":0.443,"5min":0.42699999999999999,"15min":0.42199999999999999},"max":{"1min":8.0990000000000002,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":1.0509999999999999,"5min":1.206,"15min":1.206},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.62,"15min":0.59199999999999997},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"max":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"last":0.48899999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.83699999999999997,"15min":0.79700000000000004},"min":{"1min":0.61299999999999999,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.78900000000000003},"min":{"1min":0.58499999999999996,"5min":0.56699999999999995,"15min":0.48399999999999999},"max":{"1min":1.157,"5min":1.1639999999999999,"15min":1.3009999999999999},"last":1.0109999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.83599999999999997,"15min":0.80800000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"max":{"1min":3.0419999999999998,"5min":3.0419999999999998,"15min":3.0419999999999998},"last":1.079},{"interface":"front","average":{"1min":1.0860000000000001,"5min":0.86699999999999999,"15min":0.81799999999999995},"min":{"1min":0.622,"5min":0.56599999999999995,"15min":0.53200000000000003},"max":{"1min":5.8170000000000002,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":0.82299999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":0.86499999999999999,"15min":0.82599999999999996},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"max":{"1min":5.5789999999999997,"5min":5.5789999999999997,"15min":5.5789999999999997},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.331,"5min":0.92900000000000005,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":7.9459999999999997,"5min":7.9459999999999997,"15min":7.9459999999999997},"last":1.101}]},{"osd":7,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.90500000000000003,"15min":0.85099999999999998},"min":{"1min":0.622,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.208,"5min":0.92200000000000004,"15min":0.86099999999999999},"min":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.55600000000000005},"max":{"1min":8.0470000000000006,"5min":8.0470000000000006,"15min":8.0470000000000006},"last":1.0409999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509665,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307072,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064768,"statfs":{"total":96636764160,"available":96322322432,"internally_reserved":0,"allocated":1630208,"data_stored":789868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67800000000000005,"15min":0.69099999999999995},"min":{"1min":0.44400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":1.069},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.0009999999999999},"last":0.75}]},{"osd":1,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69599999999999995,"15min":0.71399999999999997},"min":{"1min":0.47299999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.054},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.478},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.749},"min":{"1min":0.56899999999999995,"5min":0.504,"15min":0.432},"max":{"1min":1.014,"5min":1.0680000000000001,"15min":1.518},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.71199999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.40000000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.0209999999999999},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69599999999999995,"15min":0.71499999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46700000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.98699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.52300000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89500000000000002,"5min":1.0249999999999999,"15min":1.036},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.92300000000000004,"5min":1.427,"15min":1.5509999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.611,"5min":1.611,"15min":1.639},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63600000000000001,"15min":0.63200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.5389999999999999,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.46000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98899999999999999,"5min":1.272,"15min":1.585},"last":0.72299999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.753,"15min":0.752},"min":{"1min":0.502,"5min":0.502,"15min":0.45200000000000001},"max":{"1min":1.0660000000000001,"5min":1.5760000000000001,"15min":1.6180000000000001},"last":0.629}]}]},{"osd":5,"up_from":14,"seq":60129542367,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302996,"kb_used_data":1612,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068844,"statfs":{"total":96636764160,"available":96326496256,"internally_reserved":0,"allocated":1650688,"data_stored":797108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.74199999999999999,"15min":0.751},"min":{"1min":0.51400000000000001,"5min":0.46100000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":1.115,"15min":1.218},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.52900000000000003,"5min":0.47099999999999997,"15min":0.436},"max":{"1min":0.98799999999999999,"5min":1.0189999999999999,"15min":1.294},"last":0.69299999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.72399999999999998},"min":{"1min":0.46999999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91300000000000003,"5min":1.0780000000000001,"15min":1.1339999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.74199999999999999,"15min":0.73899999999999999},"min":{"1min":0.50700000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88200000000000001,"5min":1.0640000000000001,"15min":1.194},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.69299999999999995,"15min":0.70999999999999996},"min":{"1min":0.48299999999999998,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":0.79800000000000004,"5min":1.012,"15min":1.119},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.54100000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.76500000000000001,"5min":1.054,"15min":1.054},"last":0.82099999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.748,"15min":0.76000000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83699999999999997,"5min":1.0569999999999999,"15min":1.2589999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.46899999999999997},"max":{"1min":0.79500000000000004,"5min":1.02,"15min":1.036},"last":0.63}]},{"osd":4,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.628,"15min":0.623},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82599999999999996,"5min":0.89100000000000001,"15min":1.0489999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63100000000000001,"15min":0.627},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39400000000000002},"max":{"1min":0.83199999999999996,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.33900000000000002},"max":{"1min":0.84999999999999998,"5min":1.052,"15min":1.052},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.47799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.1040000000000001},"last":0.755}]},{"osd":7,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.69799999999999995},"min":{"1min":0.439,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88700000000000001,"5min":1.095,"15min":1.095},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76100000000000001,"15min":0.76500000000000001},"min":{"1min":0.54500000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.92900000000000005,"5min":0.95799999999999996,"15min":1.3360000000000001},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:48:27.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T09:48:27.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.982+0000 7f5ad15ff700 1 Processor -- start 2023-10-07T09:48:27.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.982+0000 7f5ad15ff700 1 -- start start 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.983+0000 7f5ad15ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159750 0x7f5acc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.983+0000 7f5ad15ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5acc0ce8a0 0x7f5acc0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.983+0000 7f5ad15ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc0cbf20 con 0x7f5acc159b20 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.983+0000 7f5ad15ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5acc0cc200 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.983+0000 7f5ad15ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0cc4e0 con 0x7f5acc159750 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.983+0000 7f5acb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5acc0ce8a0 0x7f5acc0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5acc0ce8a0 0x7f5acc0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50100/0 (socket says 172.21.15.29:50100) 2023-10-07T09:48:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 -- 172.21.15.29:0/541001177 learned_addr learned my addr 172.21.15.29:0/541001177 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 -- 172.21.15.29:0/541001177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159750 msgr2=0x7f5acc0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 --2- 172.21.15.29:0/541001177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159750 0x7f5acc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 -- 172.21.15.29:0/541001177 >> v1:172.21.15.144:6789/0 conn(0x7f5acc159b20 legacy=0x7f5acc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 -- 172.21.15.29:0/541001177 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc0cc7c0 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5acb7fe700 1 --2- 172.21.15.29:0/541001177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc0c5460 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009f20 tx=0x7f5ac4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f17013042428b10 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5ac9ffb700 1 -- 172.21.15.29:0/541001177 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400e070 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5ac9ffb700 1 -- 172.21.15.29:0/541001177 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac400ba40 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.984+0000 7f5ac9ffb700 1 -- 172.21.15.29:0/541001177 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40092e0 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 -- 172.21.15.29:0/541001177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 msgr2=0x7f5acc0c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/541001177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc0c5460 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009f20 tx=0x7f5ac4009840 comp rx=0 tx=0).stop 2023-10-07T09:48:27.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 -- 172.21.15.29:0/541001177 shutdown_connections 2023-10-07T09:48:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/541001177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc0c5460 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/541001177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159750 0x7f5acc0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 -- 172.21.15.29:0/541001177 >> 172.21.15.29:0/541001177 conn(0x7f5acc063a10 msgr2=0x7f5acc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 -- 172.21.15.29:0/541001177 shutdown_connections 2023-10-07T09:48:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.985+0000 7f5ad15ff700 1 -- 172.21.15.29:0/541001177 wait complete. 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.987+0000 7f5ad15ff700 1 Processor -- start 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.987+0000 7f5ad15ff700 1 -- start start 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5ad15ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5ad15ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5acc159750 0x7f5acc150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5ad15ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159b20 0x7f5acc15fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5ad15ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc15f5b0 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5ad15ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc15e9a0 con 0x7f5acc159750 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5ad15ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc15ec80 con 0x7f5acc159b20 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc14f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50112/0 (socket says 172.21.15.29:50112) 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 -- 172.21.15.29:0/833569906 learned_addr learned my addr 172.21.15.29:0/833569906 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5aca7fc700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5acc159750 0x7f5acc150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acb7fe700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159b20 0x7f5acc15fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 -- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159b20 msgr2=0x7f5acc15fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159b20 0x7f5acc15fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 -- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5acc159750 msgr2=0x7f5acc150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5acc159750 0x7f5acc150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5acaffd700 1 -- 172.21.15.29:0/833569906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4009ae0 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.988+0000 7f5aca7fc700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5acc159750 0x7f5acc150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.989+0000 7f5acaffd700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc14f4f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00b810 tx=0x7f5abc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.989+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc00f5f0 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.989+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc00f770 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.989+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc00fb30 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.989+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc15ef60 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.989+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc14fa30 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.990+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc15f890 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.991+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5abc010480 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.993+0000 7f5ab2ffd700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ab8066f30 0x7f5ab80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.993+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5abc084720 con 0x7f5acc0ce8a0 2023-10-07T09:48:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.993+0000 7f5aca7fc700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ab8066f30 0x7f5ab80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.994+0000 7f5aca7fc700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ab8066f30 0x7f5ab80693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0006fd0 tx=0x7f5ac0008040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:27.997+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5abc0517f0 con 0x7f5acc0ce8a0 2023-10-07T09:48:28.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.193+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5acc15f6f0 con 0x7f5ab8066f30 2023-10-07T09:48:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.202+0000 7f5ab2ffd700 1 -- 172.21.15.29:0/833569906 <== mgr.7582 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289269 (secure 0 0 0) 0x7f5ac000d630 con 0x7f5ab8066f30 2023-10-07T09:48:28.208 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ab8066f30 msgr2=0x7f5ab80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ab8066f30 0x7f5ab80693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0006fd0 tx=0x7f5ac0008040 comp rx=0 tx=0).stop 2023-10-07T09:48:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 msgr2=0x7f5acc14f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc14f4f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00b810 tx=0x7f5abc009250 comp rx=0 tx=0).stop 2023-10-07T09:48:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 shutdown_connections 2023-10-07T09:48:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ab8066f30 0x7f5ab80693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5acc159b20 0x7f5acc15fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5acc0ce8a0 0x7f5acc14f4f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 --2- 172.21.15.29:0/833569906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5acc159750 0x7f5acc150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 >> 172.21.15.29:0/833569906 conn(0x7f5acc063a10 msgr2=0x7f5acc0ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 shutdown_connections 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.206+0000 7f5ad15ff700 1 -- 172.21.15.29:0/833569906 wait complete. 2023-10-07T09:48:28.211 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T09:48:28.225 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-07T09:48:27.446414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2445596,"kb_used_data":14044,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529124,"statfs":{"total":773094113280,"available":770589822976,"internally_reserved":0,"allocated":14381056,"data_stored":7550940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001428"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351091+0000","last_change":"2023-10-07T09:48:23.341874+0000","last_active":"2023-10-07T09:48:23.351091+0000","last_peered":"2023-10-07T09:48:23.351091+0000","last_clean":"2023-10-07T09:48:23.351091+0000","last_became_active":"2023-10-07T09:48:23.341608+0000","last_became_peered":"2023-10-07T09:48:23.341608+0000","last_unstale":"2023-10-07T09:48:23.351091+0000","last_undegraded":"2023-10-07T09:48:23.351091+0000","last_fullsized":"2023-10-07T09:48:23.351091+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:25.918638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348448+0000","last_change":"2023-10-07T09:48:23.346038+0000","last_active":"2023-10-07T09:48:23.348448+0000","last_peered":"2023-10-07T09:48:23.348448+0000","last_clean":"2023-10-07T09:48:23.348448+0000","last_became_active":"2023-10-07T09:48:23.345564+0000","last_became_peered":"2023-10-07T09:48:23.345564+0000","last_unstale":"2023-10-07T09:48:23.348448+0000","last_undegraded":"2023-10-07T09:48:23.348448+0000","last_fullsized":"2023-10-07T09:48:23.348448+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:56:57.842209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351273+0000","last_change":"2023-10-07T09:48:23.347956+0000","last_active":"2023-10-07T09:48:23.351273+0000","last_peered":"2023-10-07T09:48:23.351273+0000","last_clean":"2023-10-07T09:48:23.351273+0000","last_became_active":"2023-10-07T09:48:23.346753+0000","last_became_peered":"2023-10-07T09:48:23.346753+0000","last_unstale":"2023-10-07T09:48:23.351273+0000","last_undegraded":"2023-10-07T09:48:23.351273+0000","last_fullsized":"2023-10-07T09:48:23.351273+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:34:54.914609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.343025+0000","last_change":"2023-10-07T09:48:23.337931+0000","last_active":"2023-10-07T09:48:23.343025+0000","last_peered":"2023-10-07T09:48:23.343025+0000","last_clean":"2023-10-07T09:48:23.343025+0000","last_became_active":"2023-10-07T09:48:23.337598+0000","last_became_peered":"2023-10-07T09:48:23.337598+0000","last_unstale":"2023-10-07T09:48:23.343025+0000","last_undegraded":"2023-10-07T09:48:23.343025+0000","last_fullsized":"2023-10-07T09:48:23.343025+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:13:03.173904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.355277+0000","last_change":"2023-10-07T09:48:23.353245+0000","last_active":"2023-10-07T09:48:23.355277+0000","last_peered":"2023-10-07T09:48:23.355277+0000","last_clean":"2023-10-07T09:48:23.355277+0000","last_became_active":"2023-10-07T09:48:23.352963+0000","last_became_peered":"2023-10-07T09:48:23.352963+0000","last_unstale":"2023-10-07T09:48:23.355277+0000","last_undegraded":"2023-10-07T09:48:23.355277+0000","last_fullsized":"2023-10-07T09:48:23.355277+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:54.574890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350880+0000","last_change":"2023-10-07T09:48:23.344753+0000","last_active":"2023-10-07T09:48:23.350880+0000","last_peered":"2023-10-07T09:48:23.350880+0000","last_clean":"2023-10-07T09:48:23.350880+0000","last_became_active":"2023-10-07T09:48:23.344469+0000","last_became_peered":"2023-10-07T09:48:23.344469+0000","last_unstale":"2023-10-07T09:48:23.350880+0000","last_undegraded":"2023-10-07T09:48:23.350880+0000","last_fullsized":"2023-10-07T09:48:23.350880+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:34:17.956936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345210+0000","last_change":"2023-10-07T09:48:23.339263+0000","last_active":"2023-10-07T09:48:23.345210+0000","last_peered":"2023-10-07T09:48:23.345210+0000","last_clean":"2023-10-07T09:48:23.345210+0000","last_became_active":"2023-10-07T09:48:23.338797+0000","last_became_peered":"2023-10-07T09:48:23.338797+0000","last_unstale":"2023-10-07T09:48:23.345210+0000","last_undegraded":"2023-10-07T09:48:23.345210+0000","last_fullsized":"2023-10-07T09:48:23.345210+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:04.865056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345325+0000","last_change":"2023-10-07T09:48:23.340253+0000","last_active":"2023-10-07T09:48:23.345325+0000","last_peered":"2023-10-07T09:48:23.345325+0000","last_clean":"2023-10-07T09:48:23.345325+0000","last_became_active":"2023-10-07T09:48:23.339633+0000","last_became_peered":"2023-10-07T09:48:23.339633+0000","last_unstale":"2023-10-07T09:48:23.345325+0000","last_undegraded":"2023-10-07T09:48:23.345325+0000","last_fullsized":"2023-10-07T09:48:23.345325+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:01:43.759657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.341980+0000","last_change":"2023-10-07T09:48:23.338796+0000","last_active":"2023-10-07T09:48:23.341980+0000","last_peered":"2023-10-07T09:48:23.341980+0000","last_clean":"2023-10-07T09:48:23.341980+0000","last_became_active":"2023-10-07T09:48:23.338511+0000","last_became_peered":"2023-10-07T09:48:23.338511+0000","last_unstale":"2023-10-07T09:48:23.341980+0000","last_undegraded":"2023-10-07T09:48:23.341980+0000","last_fullsized":"2023-10-07T09:48:23.341980+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:59.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.352459+0000","last_change":"2023-10-07T09:48:23.344760+0000","last_active":"2023-10-07T09:48:23.352459+0000","last_peered":"2023-10-07T09:48:23.352459+0000","last_clean":"2023-10-07T09:48:23.352459+0000","last_became_active":"2023-10-07T09:48:23.344488+0000","last_became_peered":"2023-10-07T09:48:23.344488+0000","last_unstale":"2023-10-07T09:48:23.352459+0000","last_undegraded":"2023-10-07T09:48:23.352459+0000","last_fullsized":"2023-10-07T09:48:23.352459+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:20.024623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347835+0000","last_change":"2023-10-07T09:48:23.340993+0000","last_active":"2023-10-07T09:48:23.347835+0000","last_peered":"2023-10-07T09:48:23.347835+0000","last_clean":"2023-10-07T09:48:23.347835+0000","last_became_active":"2023-10-07T09:48:23.340640+0000","last_became_peered":"2023-10-07T09:48:23.340640+0000","last_unstale":"2023-10-07T09:48:23.347835+0000","last_undegraded":"2023-10-07T09:48:23.347835+0000","last_fullsized":"2023-10-07T09:48:23.347835+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:42.543975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350864+0000","last_change":"2023-10-07T09:48:23.344117+0000","last_active":"2023-10-07T09:48:23.350864+0000","last_peered":"2023-10-07T09:48:23.350864+0000","last_clean":"2023-10-07T09:48:23.350864+0000","last_became_active":"2023-10-07T09:48:23.343859+0000","last_became_peered":"2023-10-07T09:48:23.343859+0000","last_unstale":"2023-10-07T09:48:23.350864+0000","last_undegraded":"2023-10-07T09:48:23.350864+0000","last_fullsized":"2023-10-07T09:48:23.350864+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:57:42.592436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346226+0000","last_change":"2023-10-07T09:48:23.343372+0000","last_active":"2023-10-07T09:48:23.346226+0000","last_peered":"2023-10-07T09:48:23.346226+0000","last_clean":"2023-10-07T09:48:23.346226+0000","last_became_active":"2023-10-07T09:48:23.342925+0000","last_became_peered":"2023-10-07T09:48:23.342925+0000","last_unstale":"2023-10-07T09:48:23.346226+0000","last_undegraded":"2023-10-07T09:48:23.346226+0000","last_fullsized":"2023-10-07T09:48:23.346226+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:22:05.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345208+0000","last_change":"2023-10-07T09:48:23.339955+0000","last_active":"2023-10-07T09:48:23.345208+0000","last_peered":"2023-10-07T09:48:23.345208+0000","last_clean":"2023-10-07T09:48:23.345208+0000","last_became_active":"2023-10-07T09:48:23.339691+0000","last_became_peered":"2023-10-07T09:48:23.339691+0000","last_unstale":"2023-10-07T09:48:23.345208+0000","last_undegraded":"2023-10-07T09:48:23.345208+0000","last_fullsized":"2023-10-07T09:48:23.345208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:22:04.816400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350650+0000","last_change":"2023-10-07T09:48:23.346072+0000","last_active":"2023-10-07T09:48:23.350650+0000","last_peered":"2023-10-07T09:48:23.350650+0000","last_clean":"2023-10-07T09:48:23.350650+0000","last_became_active":"2023-10-07T09:48:23.345607+0000","last_became_peered":"2023-10-07T09:48:23.345607+0000","last_unstale":"2023-10-07T09:48:23.350650+0000","last_undegraded":"2023-10-07T09:48:23.350650+0000","last_fullsized":"2023-10-07T09:48:23.350650+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:31:44.235802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350380+0000","last_change":"2023-10-07T09:48:23.341085+0000","last_active":"2023-10-07T09:48:23.350380+0000","last_peered":"2023-10-07T09:48:23.350380+0000","last_clean":"2023-10-07T09:48:23.350380+0000","last_became_active":"2023-10-07T09:48:23.340450+0000","last_became_peered":"2023-10-07T09:48:23.340450+0000","last_unstale":"2023-10-07T09:48:23.350380+0000","last_undegraded":"2023-10-07T09:48:23.350380+0000","last_fullsized":"2023-10-07T09:48:23.350380+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:02:15.361286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347805+0000","last_change":"2023-10-07T09:48:23.340363+0000","last_active":"2023-10-07T09:48:23.347805+0000","last_peered":"2023-10-07T09:48:23.347805+0000","last_clean":"2023-10-07T09:48:23.347805+0000","last_became_active":"2023-10-07T09:48:23.339927+0000","last_became_peered":"2023-10-07T09:48:23.339927+0000","last_unstale":"2023-10-07T09:48:23.347805+0000","last_undegraded":"2023-10-07T09:48:23.347805+0000","last_fullsized":"2023-10-07T09:48:23.347805+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:04.326985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344368+0000","last_change":"2023-10-07T09:48:23.336029+0000","last_active":"2023-10-07T09:48:23.344368+0000","last_peered":"2023-10-07T09:48:23.344368+0000","last_clean":"2023-10-07T09:48:23.344368+0000","last_became_active":"2023-10-07T09:48:23.335725+0000","last_became_peered":"2023-10-07T09:48:23.335725+0000","last_unstale":"2023-10-07T09:48:23.344368+0000","last_undegraded":"2023-10-07T09:48:23.344368+0000","last_fullsized":"2023-10-07T09:48:23.344368+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:56:37.006142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346447+0000","last_change":"2023-10-07T09:48:23.343333+0000","last_active":"2023-10-07T09:48:23.346447+0000","last_peered":"2023-10-07T09:48:23.346447+0000","last_clean":"2023-10-07T09:48:23.346447+0000","last_became_active":"2023-10-07T09:48:23.342944+0000","last_became_peered":"2023-10-07T09:48:23.342944+0000","last_unstale":"2023-10-07T09:48:23.346447+0000","last_undegraded":"2023-10-07T09:48:23.346447+0000","last_fullsized":"2023-10-07T09:48:23.346447+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:23:10.235787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348443+0000","last_change":"2023-10-07T09:48:23.337224+0000","last_active":"2023-10-07T09:48:23.348443+0000","last_peered":"2023-10-07T09:48:23.348443+0000","last_clean":"2023-10-07T09:48:23.348443+0000","last_became_active":"2023-10-07T09:48:23.336889+0000","last_became_peered":"2023-10-07T09:48:23.336889+0000","last_unstale":"2023-10-07T09:48:23.348443+0000","last_undegraded":"2023-10-07T09:48:23.348443+0000","last_fullsized":"2023-10-07T09:48:23.348443+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:46:27.519448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332057+0000","last_change":"2023-10-07T09:48:23.330437+0000","last_active":"2023-10-07T09:48:23.332057+0000","last_peered":"2023-10-07T09:48:23.332057+0000","last_clean":"2023-10-07T09:48:23.332057+0000","last_became_active":"2023-10-07T09:48:23.330111+0000","last_became_peered":"2023-10-07T09:48:23.330111+0000","last_unstale":"2023-10-07T09:48:23.332057+0000","last_undegraded":"2023-10-07T09:48:23.332057+0000","last_fullsized":"2023-10-07T09:48:23.332057+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:32:56.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349181+0000","last_change":"2023-10-07T09:48:23.343109+0000","last_active":"2023-10-07T09:48:23.349181+0000","last_peered":"2023-10-07T09:48:23.349181+0000","last_clean":"2023-10-07T09:48:23.349181+0000","last_became_active":"2023-10-07T09:48:23.342838+0000","last_became_peered":"2023-10-07T09:48:23.342838+0000","last_unstale":"2023-10-07T09:48:23.349181+0000","last_undegraded":"2023-10-07T09:48:23.349181+0000","last_fullsized":"2023-10-07T09:48:23.349181+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:56:57.842209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347964+0000","last_change":"2023-10-07T09:48:23.341800+0000","last_active":"2023-10-07T09:48:23.347964+0000","last_peered":"2023-10-07T09:48:23.347964+0000","last_clean":"2023-10-07T09:48:23.347964+0000","last_became_active":"2023-10-07T09:48:23.341564+0000","last_became_peered":"2023-10-07T09:48:23.341564+0000","last_unstale":"2023-10-07T09:48:23.347964+0000","last_undegraded":"2023-10-07T09:48:23.347964+0000","last_fullsized":"2023-10-07T09:48:23.347964+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:36.593363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349109+0000","last_change":"2023-10-07T09:48:23.339774+0000","last_active":"2023-10-07T09:48:23.349109+0000","last_peered":"2023-10-07T09:48:23.349109+0000","last_clean":"2023-10-07T09:48:23.349109+0000","last_became_active":"2023-10-07T09:48:23.339493+0000","last_became_peered":"2023-10-07T09:48:23.339493+0000","last_unstale":"2023-10-07T09:48:23.349109+0000","last_undegraded":"2023-10-07T09:48:23.349109+0000","last_fullsized":"2023-10-07T09:48:23.349109+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:42:04.326985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347785+0000","last_change":"2023-10-07T09:48:23.340177+0000","last_active":"2023-10-07T09:48:23.347785+0000","last_peered":"2023-10-07T09:48:23.347785+0000","last_clean":"2023-10-07T09:48:23.347785+0000","last_became_active":"2023-10-07T09:48:23.339605+0000","last_became_peered":"2023-10-07T09:48:23.339605+0000","last_unstale":"2023-10-07T09:48:23.347785+0000","last_undegraded":"2023-10-07T09:48:23.347785+0000","last_fullsized":"2023-10-07T09:48:23.347785+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:21.035432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.340049+0000","last_change":"2023-10-07T09:48:23.334256+0000","last_active":"2023-10-07T09:48:23.340049+0000","last_peered":"2023-10-07T09:48:23.340049+0000","last_clean":"2023-10-07T09:48:23.340049+0000","last_became_active":"2023-10-07T09:48:23.333972+0000","last_became_peered":"2023-10-07T09:48:23.333972+0000","last_unstale":"2023-10-07T09:48:23.340049+0000","last_undegraded":"2023-10-07T09:48:23.340049+0000","last_fullsized":"2023-10-07T09:48:23.340049+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:27:42.938442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350962+0000","last_change":"2023-10-07T09:48:23.344515+0000","last_active":"2023-10-07T09:48:23.350962+0000","last_peered":"2023-10-07T09:48:23.350962+0000","last_clean":"2023-10-07T09:48:23.350962+0000","last_became_active":"2023-10-07T09:48:23.344155+0000","last_became_peered":"2023-10-07T09:48:23.344155+0000","last_unstale":"2023-10-07T09:48:23.350962+0000","last_undegraded":"2023-10-07T09:48:23.350962+0000","last_fullsized":"2023-10-07T09:48:23.350962+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:50.645114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349191+0000","last_change":"2023-10-07T09:48:23.341891+0000","last_active":"2023-10-07T09:48:23.349191+0000","last_peered":"2023-10-07T09:48:23.349191+0000","last_clean":"2023-10-07T09:48:23.349191+0000","last_became_active":"2023-10-07T09:48:23.341606+0000","last_became_peered":"2023-10-07T09:48:23.341606+0000","last_unstale":"2023-10-07T09:48:23.349191+0000","last_undegraded":"2023-10-07T09:48:23.349191+0000","last_fullsized":"2023-10-07T09:48:23.349191+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:46:12.906428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347762+0000","last_change":"2023-10-07T09:48:23.341305+0000","last_active":"2023-10-07T09:48:23.347762+0000","last_peered":"2023-10-07T09:48:23.347762+0000","last_clean":"2023-10-07T09:48:23.347762+0000","last_became_active":"2023-10-07T09:48:23.340977+0000","last_became_peered":"2023-10-07T09:48:23.340977+0000","last_unstale":"2023-10-07T09:48:23.347762+0000","last_undegraded":"2023-10-07T09:48:23.347762+0000","last_fullsized":"2023-10-07T09:48:23.347762+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:00:36.261617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344600+0000","last_change":"2023-10-07T09:48:23.337008+0000","last_active":"2023-10-07T09:48:23.344600+0000","last_peered":"2023-10-07T09:48:23.344600+0000","last_clean":"2023-10-07T09:48:23.344600+0000","last_became_active":"2023-10-07T09:48:23.336638+0000","last_became_peered":"2023-10-07T09:48:23.336638+0000","last_unstale":"2023-10-07T09:48:23.344600+0000","last_undegraded":"2023-10-07T09:48:23.344600+0000","last_fullsized":"2023-10-07T09:48:23.344600+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:05:07.769412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.342075+0000","last_change":"2023-10-07T09:48:23.336496+0000","last_active":"2023-10-07T09:48:23.342075+0000","last_peered":"2023-10-07T09:48:23.342075+0000","last_clean":"2023-10-07T09:48:23.342075+0000","last_became_active":"2023-10-07T09:48:23.336069+0000","last_became_peered":"2023-10-07T09:48:23.336069+0000","last_unstale":"2023-10-07T09:48:23.342075+0000","last_undegraded":"2023-10-07T09:48:23.342075+0000","last_fullsized":"2023-10-07T09:48:23.342075+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:42:12.564256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350605+0000","last_change":"2023-10-07T09:48:23.345549+0000","last_active":"2023-10-07T09:48:23.350605+0000","last_peered":"2023-10-07T09:48:23.350605+0000","last_clean":"2023-10-07T09:48:23.350605+0000","last_became_active":"2023-10-07T09:48:23.345037+0000","last_became_peered":"2023-10-07T09:48:23.345037+0000","last_unstale":"2023-10-07T09:48:23.350605+0000","last_undegraded":"2023-10-07T09:48:23.350605+0000","last_fullsized":"2023-10-07T09:48:23.350605+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:04.865056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350335+0000","last_change":"2023-10-07T09:48:23.344155+0000","last_active":"2023-10-07T09:48:23.350335+0000","last_peered":"2023-10-07T09:48:23.350335+0000","last_clean":"2023-10-07T09:48:23.350335+0000","last_became_active":"2023-10-07T09:48:23.343800+0000","last_became_peered":"2023-10-07T09:48:23.343800+0000","last_unstale":"2023-10-07T09:48:23.350335+0000","last_undegraded":"2023-10-07T09:48:23.350335+0000","last_fullsized":"2023-10-07T09:48:23.350335+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:40:52.846704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"132'4","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347832+0000","last_change":"2023-10-07T09:48:22.334213+0000","last_active":"2023-10-07T09:48:23.347832+0000","last_peered":"2023-10-07T09:48:23.347832+0000","last_clean":"2023-10-07T09:48:23.347832+0000","last_became_active":"2023-10-07T09:48:22.333881+0000","last_became_peered":"2023-10-07T09:48:22.333881+0000","last_unstale":"2023-10-07T09:48:23.347832+0000","last_undegraded":"2023-10-07T09:48:23.347832+0000","last_fullsized":"2023-10-07T09:48:23.347832+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:56.497314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345735+0000","last_change":"2023-10-07T09:48:23.339647+0000","last_active":"2023-10-07T09:48:23.345735+0000","last_peered":"2023-10-07T09:48:23.345735+0000","last_clean":"2023-10-07T09:48:23.345735+0000","last_became_active":"2023-10-07T09:48:23.339402+0000","last_became_peered":"2023-10-07T09:48:23.339402+0000","last_unstale":"2023-10-07T09:48:23.345735+0000","last_undegraded":"2023-10-07T09:48:23.345735+0000","last_fullsized":"2023-10-07T09:48:23.345735+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:36.593363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347601+0000","last_change":"2023-10-07T09:48:22.330987+0000","last_active":"2023-10-07T09:48:23.347601+0000","last_peered":"2023-10-07T09:48:23.347601+0000","last_clean":"2023-10-07T09:48:23.347601+0000","last_became_active":"2023-10-07T09:48:22.330714+0000","last_became_peered":"2023-10-07T09:48:22.330714+0000","last_unstale":"2023-10-07T09:48:23.347601+0000","last_undegraded":"2023-10-07T09:48:23.347601+0000","last_fullsized":"2023-10-07T09:48:23.347601+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:28:01.031834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351015+0000","last_change":"2023-10-07T09:48:23.345479+0000","last_active":"2023-10-07T09:48:23.351015+0000","last_peered":"2023-10-07T09:48:23.351015+0000","last_clean":"2023-10-07T09:48:23.351015+0000","last_became_active":"2023-10-07T09:48:23.345085+0000","last_became_peered":"2023-10-07T09:48:23.345085+0000","last_unstale":"2023-10-07T09:48:23.351015+0000","last_undegraded":"2023-10-07T09:48:23.351015+0000","last_fullsized":"2023-10-07T09:48:23.351015+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:05:07.769412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351513+0000","last_change":"2023-10-07T09:48:22.338315+0000","last_active":"2023-10-07T09:48:23.351513+0000","last_peered":"2023-10-07T09:48:23.351513+0000","last_clean":"2023-10-07T09:48:23.351513+0000","last_became_active":"2023-10-07T09:48:22.338094+0000","last_became_peered":"2023-10-07T09:48:22.338094+0000","last_unstale":"2023-10-07T09:48:23.351513+0000","last_undegraded":"2023-10-07T09:48:23.351513+0000","last_fullsized":"2023-10-07T09:48:23.351513+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:30:48.653184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347288+0000","last_change":"2023-10-07T09:48:23.345765+0000","last_active":"2023-10-07T09:48:23.347288+0000","last_peered":"2023-10-07T09:48:23.347288+0000","last_clean":"2023-10-07T09:48:23.347288+0000","last_became_active":"2023-10-07T09:48:23.345492+0000","last_became_peered":"2023-10-07T09:48:23.345492+0000","last_unstale":"2023-10-07T09:48:23.347288+0000","last_undegraded":"2023-10-07T09:48:23.347288+0000","last_fullsized":"2023-10-07T09:48:23.347288+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:06:27.733172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351759+0000","last_change":"2023-10-07T09:48:22.337285+0000","last_active":"2023-10-07T09:48:23.351759+0000","last_peered":"2023-10-07T09:48:23.351759+0000","last_clean":"2023-10-07T09:48:23.351759+0000","last_became_active":"2023-10-07T09:48:22.336987+0000","last_became_peered":"2023-10-07T09:48:22.336987+0000","last_unstale":"2023-10-07T09:48:23.351759+0000","last_undegraded":"2023-10-07T09:48:23.351759+0000","last_fullsized":"2023-10-07T09:48:23.351759+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:11.162155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.361283+0000","last_change":"2023-10-07T09:48:23.353331+0000","last_active":"2023-10-07T09:48:23.361283+0000","last_peered":"2023-10-07T09:48:23.361283+0000","last_clean":"2023-10-07T09:48:23.361283+0000","last_became_active":"2023-10-07T09:48:23.353031+0000","last_became_peered":"2023-10-07T09:48:23.353031+0000","last_unstale":"2023-10-07T09:48:23.361283+0000","last_undegraded":"2023-10-07T09:48:23.361283+0000","last_fullsized":"2023-10-07T09:48:23.361283+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:19:19.377309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351603+0000","last_change":"2023-10-07T09:48:22.325792+0000","last_active":"2023-10-07T09:48:23.351603+0000","last_peered":"2023-10-07T09:48:23.351603+0000","last_clean":"2023-10-07T09:48:23.351603+0000","last_became_active":"2023-10-07T09:48:22.325488+0000","last_became_peered":"2023-10-07T09:48:22.325488+0000","last_unstale":"2023-10-07T09:48:23.351603+0000","last_undegraded":"2023-10-07T09:48:23.351603+0000","last_fullsized":"2023-10-07T09:48:23.351603+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:29:39.822105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348550+0000","last_change":"2023-10-07T09:48:23.346681+0000","last_active":"2023-10-07T09:48:23.348550+0000","last_peered":"2023-10-07T09:48:23.348550+0000","last_clean":"2023-10-07T09:48:23.348550+0000","last_became_active":"2023-10-07T09:48:23.346303+0000","last_became_peered":"2023-10-07T09:48:23.346303+0000","last_unstale":"2023-10-07T09:48:23.348550+0000","last_undegraded":"2023-10-07T09:48:23.348550+0000","last_fullsized":"2023-10-07T09:48:23.348550+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:48.391128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349442+0000","last_change":"2023-10-07T09:48:22.324869+0000","last_active":"2023-10-07T09:48:23.349442+0000","last_peered":"2023-10-07T09:48:23.349442+0000","last_clean":"2023-10-07T09:48:23.349442+0000","last_became_active":"2023-10-07T09:48:22.324599+0000","last_became_peered":"2023-10-07T09:48:22.324599+0000","last_unstale":"2023-10-07T09:48:23.349442+0000","last_undegraded":"2023-10-07T09:48:23.349442+0000","last_fullsized":"2023-10-07T09:48:23.349442+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:40:51.839090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350410+0000","last_change":"2023-10-07T09:48:23.341280+0000","last_active":"2023-10-07T09:48:23.350410+0000","last_peered":"2023-10-07T09:48:23.350410+0000","last_clean":"2023-10-07T09:48:23.350410+0000","last_became_active":"2023-10-07T09:48:23.340838+0000","last_became_peered":"2023-10-07T09:48:23.340838+0000","last_unstale":"2023-10-07T09:48:23.350410+0000","last_undegraded":"2023-10-07T09:48:23.350410+0000","last_fullsized":"2023-10-07T09:48:23.350410+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:09:32.323274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324734+0000","last_change":"2023-10-07T09:48:22.332754+0000","last_active":"2023-10-07T09:48:23.324734+0000","last_peered":"2023-10-07T09:48:23.324734+0000","last_clean":"2023-10-07T09:48:23.324734+0000","last_became_active":"2023-10-07T09:48:22.332423+0000","last_became_peered":"2023-10-07T09:48:22.332423+0000","last_unstale":"2023-10-07T09:48:23.324734+0000","last_undegraded":"2023-10-07T09:48:23.324734+0000","last_fullsized":"2023-10-07T09:48:23.324734+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:34:16.949322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345167+0000","last_change":"2023-10-07T09:48:23.339083+0000","last_active":"2023-10-07T09:48:23.345167+0000","last_peered":"2023-10-07T09:48:23.345167+0000","last_clean":"2023-10-07T09:48:23.345167+0000","last_became_active":"2023-10-07T09:48:23.338465+0000","last_became_peered":"2023-10-07T09:48:23.338465+0000","last_unstale":"2023-10-07T09:48:23.345167+0000","last_undegraded":"2023-10-07T09:48:23.345167+0000","last_fullsized":"2023-10-07T09:48:23.345167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:20:07.941612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.325072+0000","last_change":"2023-10-07T09:48:22.336899+0000","last_active":"2023-10-07T09:48:23.325072+0000","last_peered":"2023-10-07T09:48:23.325072+0000","last_clean":"2023-10-07T09:48:23.325072+0000","last_became_active":"2023-10-07T09:48:22.331349+0000","last_became_peered":"2023-10-07T09:48:22.331349+0000","last_unstale":"2023-10-07T09:48:23.325072+0000","last_undegraded":"2023-10-07T09:48:23.325072+0000","last_fullsized":"2023-10-07T09:48:23.325072+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:56:56.834596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346251+0000","last_change":"2023-10-07T09:48:23.345316+0000","last_active":"2023-10-07T09:48:23.346251+0000","last_peered":"2023-10-07T09:48:23.346251+0000","last_clean":"2023-10-07T09:48:23.346251+0000","last_became_active":"2023-10-07T09:48:23.344917+0000","last_became_peered":"2023-10-07T09:48:23.344917+0000","last_unstale":"2023-10-07T09:48:23.346251+0000","last_undegraded":"2023-10-07T09:48:23.346251+0000","last_fullsized":"2023-10-07T09:48:23.346251+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:31:44.235802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332274+0000","last_change":"2023-10-07T09:48:22.337849+0000","last_active":"2023-10-07T09:48:23.332274+0000","last_peered":"2023-10-07T09:48:23.332274+0000","last_clean":"2023-10-07T09:48:23.332274+0000","last_became_active":"2023-10-07T09:48:22.337596+0000","last_became_peered":"2023-10-07T09:48:22.337596+0000","last_unstale":"2023-10-07T09:48:23.332274+0000","last_undegraded":"2023-10-07T09:48:23.332274+0000","last_fullsized":"2023-10-07T09:48:23.332274+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:12:22.941001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345752+0000","last_change":"2023-10-07T09:48:23.342247+0000","last_active":"2023-10-07T09:48:23.345752+0000","last_peered":"2023-10-07T09:48:23.345752+0000","last_clean":"2023-10-07T09:48:23.345752+0000","last_became_active":"2023-10-07T09:48:23.341934+0000","last_became_peered":"2023-10-07T09:48:23.341934+0000","last_unstale":"2023-10-07T09:48:23.345752+0000","last_undegraded":"2023-10-07T09:48:23.345752+0000","last_fullsized":"2023-10-07T09:48:23.345752+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:46:12.906428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.319537+0000","last_change":"2023-10-07T09:48:22.329924+0000","last_active":"2023-10-07T09:48:23.319537+0000","last_peered":"2023-10-07T09:48:23.319537+0000","last_clean":"2023-10-07T09:48:23.319537+0000","last_became_active":"2023-10-07T09:48:22.329619+0000","last_became_peered":"2023-10-07T09:48:22.329619+0000","last_unstale":"2023-10-07T09:48:23.319537+0000","last_undegraded":"2023-10-07T09:48:23.319537+0000","last_fullsized":"2023-10-07T09:48:23.319537+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:49.637500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338932+0000","last_change":"2023-10-07T09:48:23.334684+0000","last_active":"2023-10-07T09:48:23.338932+0000","last_peered":"2023-10-07T09:48:23.338932+0000","last_clean":"2023-10-07T09:48:23.338932+0000","last_became_active":"2023-10-07T09:48:23.334412+0000","last_became_peered":"2023-10-07T09:48:23.334412+0000","last_unstale":"2023-10-07T09:48:23.338932+0000","last_undegraded":"2023-10-07T09:48:23.338932+0000","last_fullsized":"2023-10-07T09:48:23.338932+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:46:27.519448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"132'3","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.363780+0000","last_change":"2023-10-07T09:48:22.331754+0000","last_active":"2023-10-07T09:48:23.363780+0000","last_peered":"2023-10-07T09:48:23.363780+0000","last_clean":"2023-10-07T09:48:23.363780+0000","last_became_active":"2023-10-07T09:48:22.331416+0000","last_became_peered":"2023-10-07T09:48:22.331416+0000","last_unstale":"2023-10-07T09:48:23.363780+0000","last_undegraded":"2023-10-07T09:48:23.363780+0000","last_fullsized":"2023-10-07T09:48:23.363780+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:51.675105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344769+0000","last_change":"2023-10-07T09:48:23.339889+0000","last_active":"2023-10-07T09:48:23.344769+0000","last_peered":"2023-10-07T09:48:23.344769+0000","last_clean":"2023-10-07T09:48:23.344769+0000","last_became_active":"2023-10-07T09:48:23.339624+0000","last_became_peered":"2023-10-07T09:48:23.339624+0000","last_unstale":"2023-10-07T09:48:23.344769+0000","last_undegraded":"2023-10-07T09:48:23.344769+0000","last_fullsized":"2023-10-07T09:48:23.344769+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:34:17.956936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"132'6","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.365618+0000","last_change":"2023-10-07T09:48:22.336553+0000","last_active":"2023-10-07T09:48:23.365618+0000","last_peered":"2023-10-07T09:48:23.365618+0000","last_clean":"2023-10-07T09:48:23.365618+0000","last_became_active":"2023-10-07T09:48:22.334979+0000","last_became_peered":"2023-10-07T09:48:22.334979+0000","last_unstale":"2023-10-07T09:48:23.365618+0000","last_undegraded":"2023-10-07T09:48:23.365618+0000","last_fullsized":"2023-10-07T09:48:23.365618+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:41.536361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338939+0000","last_change":"2023-10-07T09:48:23.331997+0000","last_active":"2023-10-07T09:48:23.338939+0000","last_peered":"2023-10-07T09:48:23.338939+0000","last_clean":"2023-10-07T09:48:23.338939+0000","last_became_active":"2023-10-07T09:48:23.331745+0000","last_became_peered":"2023-10-07T09:48:23.331745+0000","last_unstale":"2023-10-07T09:48:23.338939+0000","last_undegraded":"2023-10-07T09:48:23.338939+0000","last_fullsized":"2023-10-07T09:48:23.338939+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:42.543975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.355154+0000","last_change":"2023-10-07T09:48:22.331792+0000","last_active":"2023-10-07T09:48:23.355154+0000","last_peered":"2023-10-07T09:48:23.355154+0000","last_clean":"2023-10-07T09:48:23.355154+0000","last_became_active":"2023-10-07T09:48:22.331546+0000","last_became_peered":"2023-10-07T09:48:22.331546+0000","last_unstale":"2023-10-07T09:48:23.355154+0000","last_undegraded":"2023-10-07T09:48:23.355154+0000","last_fullsized":"2023-10-07T09:48:23.355154+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:27.785695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338733+0000","last_change":"2023-10-07T09:48:23.333225+0000","last_active":"2023-10-07T09:48:23.338733+0000","last_peered":"2023-10-07T09:48:23.338733+0000","last_clean":"2023-10-07T09:48:23.338733+0000","last_became_active":"2023-10-07T09:48:23.332964+0000","last_became_peered":"2023-10-07T09:48:23.332964+0000","last_unstale":"2023-10-07T09:48:23.338733+0000","last_undegraded":"2023-10-07T09:48:23.338733+0000","last_fullsized":"2023-10-07T09:48:23.338733+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:17.740390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.328148+0000","last_change":"2023-10-07T09:48:22.336761+0000","last_active":"2023-10-07T09:48:23.328148+0000","last_peered":"2023-10-07T09:48:23.328148+0000","last_clean":"2023-10-07T09:48:23.328148+0000","last_became_active":"2023-10-07T09:48:22.336441+0000","last_became_peered":"2023-10-07T09:48:22.336441+0000","last_unstale":"2023-10-07T09:48:23.328148+0000","last_undegraded":"2023-10-07T09:48:23.328148+0000","last_fullsized":"2023-10-07T09:48:23.328148+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:41.536361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.338773+0000","last_change":"2023-10-07T09:48:23.328101+0000","last_active":"2023-10-07T09:48:23.338773+0000","last_peered":"2023-10-07T09:48:23.338773+0000","last_clean":"2023-10-07T09:48:23.338773+0000","last_became_active":"2023-10-07T09:48:23.327810+0000","last_became_peered":"2023-10-07T09:48:23.327810+0000","last_unstale":"2023-10-07T09:48:23.338773+0000","last_undegraded":"2023-10-07T09:48:23.338773+0000","last_fullsized":"2023-10-07T09:48:23.338773+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:00:36.261617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"132'8","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.363837+0000","last_change":"2023-10-07T09:48:22.341514+0000","last_active":"2023-10-07T09:48:23.363837+0000","last_peered":"2023-10-07T09:48:23.363837+0000","last_clean":"2023-10-07T09:48:23.363837+0000","last_became_active":"2023-10-07T09:48:22.341191+0000","last_became_peered":"2023-10-07T09:48:22.341191+0000","last_unstale":"2023-10-07T09:48:23.363837+0000","last_undegraded":"2023-10-07T09:48:23.363837+0000","last_fullsized":"2023-10-07T09:48:23.363837+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:47:53.945223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.354603+0000","last_change":"2023-10-07T09:48:23.344238+0000","last_active":"2023-10-07T09:48:23.354603+0000","last_peered":"2023-10-07T09:48:23.354603+0000","last_clean":"2023-10-07T09:48:23.354603+0000","last_became_active":"2023-10-07T09:48:23.343965+0000","last_became_peered":"2023-10-07T09:48:23.343965+0000","last_unstale":"2023-10-07T09:48:23.354603+0000","last_undegraded":"2023-10-07T09:48:23.354603+0000","last_fullsized":"2023-10-07T09:48:23.354603+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:33:58.953342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.363556+0000","last_change":"2023-10-07T09:48:22.330758+0000","last_active":"2023-10-07T09:48:23.363556+0000","last_peered":"2023-10-07T09:48:23.363556+0000","last_clean":"2023-10-07T09:48:23.363556+0000","last_became_active":"2023-10-07T09:48:22.326927+0000","last_became_peered":"2023-10-07T09:48:22.326927+0000","last_unstale":"2023-10-07T09:48:23.363556+0000","last_undegraded":"2023-10-07T09:48:23.363556+0000","last_fullsized":"2023-10-07T09:48:23.363556+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:32:55.194513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350662+0000","last_change":"2023-10-07T09:48:23.345936+0000","last_active":"2023-10-07T09:48:23.350662+0000","last_peered":"2023-10-07T09:48:23.350662+0000","last_clean":"2023-10-07T09:48:23.350662+0000","last_became_active":"2023-10-07T09:48:23.345644+0000","last_became_peered":"2023-10-07T09:48:23.345644+0000","last_unstale":"2023-10-07T09:48:23.350662+0000","last_undegraded":"2023-10-07T09:48:23.350662+0000","last_fullsized":"2023-10-07T09:48:23.350662+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:02.947201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.352553+0000","last_change":"2023-10-07T09:48:22.336692+0000","last_active":"2023-10-07T09:48:23.352553+0000","last_peered":"2023-10-07T09:48:23.352553+0000","last_clean":"2023-10-07T09:48:23.352553+0000","last_became_active":"2023-10-07T09:48:22.336443+0000","last_became_peered":"2023-10-07T09:48:22.336443+0000","last_unstale":"2023-10-07T09:48:23.352553+0000","last_undegraded":"2023-10-07T09:48:23.352553+0000","last_fullsized":"2023-10-07T09:48:23.352553+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:41:57.402686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.347821+0000","last_change":"2023-10-07T09:48:23.344863+0000","last_active":"2023-10-07T09:48:23.347821+0000","last_peered":"2023-10-07T09:48:23.347821+0000","last_clean":"2023-10-07T09:48:23.347821+0000","last_became_active":"2023-10-07T09:48:23.344588+0000","last_became_peered":"2023-10-07T09:48:23.344588+0000","last_unstale":"2023-10-07T09:48:23.347821+0000","last_undegraded":"2023-10-07T09:48:23.347821+0000","last_fullsized":"2023-10-07T09:48:23.347821+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:26:43.640061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.349614+0000","last_change":"2023-10-07T09:48:22.323276+0000","last_active":"2023-10-07T09:48:23.349614+0000","last_peered":"2023-10-07T09:48:23.349614+0000","last_clean":"2023-10-07T09:48:23.349614+0000","last_became_active":"2023-10-07T09:48:22.322888+0000","last_became_peered":"2023-10-07T09:48:22.322888+0000","last_unstale":"2023-10-07T09:48:23.349614+0000","last_undegraded":"2023-10-07T09:48:23.349614+0000","last_fullsized":"2023-10-07T09:48:23.349614+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:47.383514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348365+0000","last_change":"2023-10-07T09:48:23.346816+0000","last_active":"2023-10-07T09:48:23.348365+0000","last_peered":"2023-10-07T09:48:23.348365+0000","last_clean":"2023-10-07T09:48:23.348365+0000","last_became_active":"2023-10-07T09:48:23.346273+0000","last_became_peered":"2023-10-07T09:48:23.346273+0000","last_unstale":"2023-10-07T09:48:23.348365+0000","last_undegraded":"2023-10-07T09:48:23.348365+0000","last_fullsized":"2023-10-07T09:48:23.348365+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:18:30.228699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.327170+0000","last_change":"2023-10-07T09:48:22.336458+0000","last_active":"2023-10-07T09:48:23.327170+0000","last_peered":"2023-10-07T09:48:23.327170+0000","last_clean":"2023-10-07T09:48:23.327170+0000","last_became_active":"2023-10-07T09:48:22.336172+0000","last_became_peered":"2023-10-07T09:48:22.336172+0000","last_unstale":"2023-10-07T09:48:23.327170+0000","last_undegraded":"2023-10-07T09:48:23.327170+0000","last_fullsized":"2023-10-07T09:48:23.327170+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:33:27.785695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345179+0000","last_change":"2023-10-07T09:48:23.338398+0000","last_active":"2023-10-07T09:48:23.345179+0000","last_peered":"2023-10-07T09:48:23.345179+0000","last_clean":"2023-10-07T09:48:23.345179+0000","last_became_active":"2023-10-07T09:48:23.338120+0000","last_became_peered":"2023-10-07T09:48:23.338120+0000","last_unstale":"2023-10-07T09:48:23.345179+0000","last_undegraded":"2023-10-07T09:48:23.345179+0000","last_fullsized":"2023-10-07T09:48:23.345179+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:28:45.863077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.355674+0000","last_change":"2023-10-07T09:48:22.335032+0000","last_active":"2023-10-07T09:48:23.355674+0000","last_peered":"2023-10-07T09:48:23.355674+0000","last_clean":"2023-10-07T09:48:23.355674+0000","last_became_active":"2023-10-07T09:48:22.334751+0000","last_became_peered":"2023-10-07T09:48:22.334751+0000","last_unstale":"2023-10-07T09:48:23.355674+0000","last_undegraded":"2023-10-07T09:48:23.355674+0000","last_fullsized":"2023-10-07T09:48:23.355674+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:56.497314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346964+0000","last_change":"2023-10-07T09:48:23.340717+0000","last_active":"2023-10-07T09:48:23.346964+0000","last_peered":"2023-10-07T09:48:23.346964+0000","last_clean":"2023-10-07T09:48:23.346964+0000","last_became_active":"2023-10-07T09:48:23.340416+0000","last_became_peered":"2023-10-07T09:48:23.340416+0000","last_unstale":"2023-10-07T09:48:23.346964+0000","last_undegraded":"2023-10-07T09:48:23.346964+0000","last_fullsized":"2023-10-07T09:48:23.346964+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:53:03.934767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332401+0000","last_change":"2023-10-07T09:48:22.340681+0000","last_active":"2023-10-07T09:48:23.332401+0000","last_peered":"2023-10-07T09:48:23.332401+0000","last_clean":"2023-10-07T09:48:23.332401+0000","last_became_active":"2023-10-07T09:48:22.340338+0000","last_became_peered":"2023-10-07T09:48:22.340338+0000","last_unstale":"2023-10-07T09:48:23.332401+0000","last_undegraded":"2023-10-07T09:48:23.332401+0000","last_fullsized":"2023-10-07T09:48:23.332401+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:33:56.497314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.351254+0000","last_change":"2023-10-07T09:48:23.347384+0000","last_active":"2023-10-07T09:48:23.351254+0000","last_peered":"2023-10-07T09:48:23.351254+0000","last_clean":"2023-10-07T09:48:23.351254+0000","last_became_active":"2023-10-07T09:48:23.347124+0000","last_became_peered":"2023-10-07T09:48:23.347124+0000","last_unstale":"2023-10-07T09:48:23.351254+0000","last_undegraded":"2023-10-07T09:48:23.351254+0000","last_fullsized":"2023-10-07T09:48:23.351254+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:26.743514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.329367+0000","last_change":"2023-10-07T09:48:22.335023+0000","last_active":"2023-10-07T09:48:23.329367+0000","last_peered":"2023-10-07T09:48:23.329367+0000","last_clean":"2023-10-07T09:48:23.329367+0000","last_became_active":"2023-10-07T09:48:22.334730+0000","last_became_peered":"2023-10-07T09:48:22.334730+0000","last_unstale":"2023-10-07T09:48:23.329367+0000","last_undegraded":"2023-10-07T09:48:23.329367+0000","last_fullsized":"2023-10-07T09:48:23.329367+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:47:53.945223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.346697+0000","last_change":"2023-10-07T09:48:23.337705+0000","last_active":"2023-10-07T09:48:23.346697+0000","last_peered":"2023-10-07T09:48:23.346697+0000","last_clean":"2023-10-07T09:48:23.346697+0000","last_became_active":"2023-10-07T09:48:23.337382+0000","last_became_peered":"2023-10-07T09:48:23.337382+0000","last_unstale":"2023-10-07T09:48:23.346697+0000","last_undegraded":"2023-10-07T09:48:23.346697+0000","last_fullsized":"2023-10-07T09:48:23.346697+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:42:36.593363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324825+0000","last_change":"2023-10-07T09:48:22.327707+0000","last_active":"2023-10-07T09:48:23.324825+0000","last_peered":"2023-10-07T09:48:23.324825+0000","last_clean":"2023-10-07T09:48:23.324825+0000","last_became_active":"2023-10-07T09:48:22.326474+0000","last_became_peered":"2023-10-07T09:48:22.326474+0000","last_unstale":"2023-10-07T09:48:23.324825+0000","last_undegraded":"2023-10-07T09:48:23.324825+0000","last_fullsized":"2023-10-07T09:48:23.324825+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:47.383514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345689+0000","last_change":"2023-10-07T09:48:23.341784+0000","last_active":"2023-10-07T09:48:23.345689+0000","last_peered":"2023-10-07T09:48:23.345689+0000","last_clean":"2023-10-07T09:48:23.345689+0000","last_became_active":"2023-10-07T09:48:23.341552+0000","last_became_peered":"2023-10-07T09:48:23.341552+0000","last_unstale":"2023-10-07T09:48:23.345689+0000","last_undegraded":"2023-10-07T09:48:23.345689+0000","last_fullsized":"2023-10-07T09:48:23.345689+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:12:18.525773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.361871+0000","last_change":"2023-10-07T09:48:22.342396+0000","last_active":"2023-10-07T09:48:23.361871+0000","last_peered":"2023-10-07T09:48:23.361871+0000","last_clean":"2023-10-07T09:48:23.361871+0000","last_became_active":"2023-10-07T09:48:22.342056+0000","last_became_peered":"2023-10-07T09:48:22.342056+0000","last_unstale":"2023-10-07T09:48:23.361871+0000","last_undegraded":"2023-10-07T09:48:23.361871+0000","last_fullsized":"2023-10-07T09:48:23.361871+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:28:01.939588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.339631+0000","last_change":"2023-10-07T09:48:23.332504+0000","last_active":"2023-10-07T09:48:23.339631+0000","last_peered":"2023-10-07T09:48:23.339631+0000","last_clean":"2023-10-07T09:48:23.339631+0000","last_became_active":"2023-10-07T09:48:23.332211+0000","last_became_peered":"2023-10-07T09:48:23.332211+0000","last_unstale":"2023-10-07T09:48:23.339631+0000","last_undegraded":"2023-10-07T09:48:23.339631+0000","last_fullsized":"2023-10-07T09:48:23.339631+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:26:10.005451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.327249+0000","last_change":"2023-10-07T09:48:22.323757+0000","last_active":"2023-10-07T09:48:23.327249+0000","last_peered":"2023-10-07T09:48:23.327249+0000","last_clean":"2023-10-07T09:48:23.327249+0000","last_became_active":"2023-10-07T09:48:22.323518+0000","last_became_peered":"2023-10-07T09:48:22.323518+0000","last_unstale":"2023-10-07T09:48:23.327249+0000","last_undegraded":"2023-10-07T09:48:23.327249+0000","last_fullsized":"2023-10-07T09:48:23.327249+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:32:55.194513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.350842+0000","last_change":"2023-10-07T09:48:23.342913+0000","last_active":"2023-10-07T09:48:23.350842+0000","last_peered":"2023-10-07T09:48:23.350842+0000","last_clean":"2023-10-07T09:48:23.350842+0000","last_became_active":"2023-10-07T09:48:23.342663+0000","last_became_peered":"2023-10-07T09:48:23.342663+0000","last_unstale":"2023-10-07T09:48:23.350842+0000","last_undegraded":"2023-10-07T09:48:23.350842+0000","last_fullsized":"2023-10-07T09:48:23.350842+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:19:22.128645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324025+0000","last_change":"2023-10-07T09:48:22.330583+0000","last_active":"2023-10-07T09:48:23.324025+0000","last_peered":"2023-10-07T09:48:23.324025+0000","last_clean":"2023-10-07T09:48:23.324025+0000","last_became_active":"2023-10-07T09:48:22.327011+0000","last_became_peered":"2023-10-07T09:48:22.327011+0000","last_unstale":"2023-10-07T09:48:23.324025+0000","last_undegraded":"2023-10-07T09:48:23.324025+0000","last_fullsized":"2023-10-07T09:48:23.324025+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:23:46.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345651+0000","last_change":"2023-10-07T09:48:23.341553+0000","last_active":"2023-10-07T09:48:23.345651+0000","last_peered":"2023-10-07T09:48:23.345651+0000","last_clean":"2023-10-07T09:48:23.345651+0000","last_became_active":"2023-10-07T09:48:23.341286+0000","last_became_peered":"2023-10-07T09:48:23.341286+0000","last_unstale":"2023-10-07T09:48:23.345651+0000","last_undegraded":"2023-10-07T09:48:23.345651+0000","last_fullsized":"2023-10-07T09:48:23.345651+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T09:52:03.541591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.320043+0000","last_change":"2023-10-07T09:48:22.325907+0000","last_active":"2023-10-07T09:48:23.320043+0000","last_peered":"2023-10-07T09:48:23.320043+0000","last_clean":"2023-10-07T09:48:23.320043+0000","last_became_active":"2023-10-07T09:48:22.325640+0000","last_became_peered":"2023-10-07T09:48:22.325640+0000","last_unstale":"2023-10-07T09:48:23.320043+0000","last_undegraded":"2023-10-07T09:48:23.320043+0000","last_fullsized":"2023-10-07T09:48:23.320043+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:36:31.570060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.336164+0000","last_change":"2023-10-07T09:48:23.332805+0000","last_active":"2023-10-07T09:48:23.336164+0000","last_peered":"2023-10-07T09:48:23.336164+0000","last_clean":"2023-10-07T09:48:23.336164+0000","last_became_active":"2023-10-07T09:48:23.332531+0000","last_became_peered":"2023-10-07T09:48:23.332531+0000","last_unstale":"2023-10-07T09:48:23.336164+0000","last_undegraded":"2023-10-07T09:48:23.336164+0000","last_fullsized":"2023-10-07T09:48:23.336164+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:09:59.093385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.326783+0000","last_change":"2023-10-07T09:48:22.331697+0000","last_active":"2023-10-07T09:48:23.326783+0000","last_peered":"2023-10-07T09:48:23.326783+0000","last_clean":"2023-10-07T09:48:23.326783+0000","last_became_active":"2023-10-07T09:48:22.331401+0000","last_became_peered":"2023-10-07T09:48:22.331401+0000","last_unstale":"2023-10-07T09:48:23.326783+0000","last_undegraded":"2023-10-07T09:48:23.326783+0000","last_fullsized":"2023-10-07T09:48:23.326783+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:12:22.941001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.348567+0000","last_change":"2023-10-07T09:48:23.346948+0000","last_active":"2023-10-07T09:48:23.348567+0000","last_peered":"2023-10-07T09:48:23.348567+0000","last_clean":"2023-10-07T09:48:23.348567+0000","last_became_active":"2023-10-07T09:48:23.346657+0000","last_became_peered":"2023-10-07T09:48:23.346657+0000","last_unstale":"2023-10-07T09:48:23.348567+0000","last_undegraded":"2023-10-07T09:48:23.348567+0000","last_fullsized":"2023-10-07T09:48:23.348567+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:17.740390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324889+0000","last_change":"2023-10-07T09:48:22.335219+0000","last_active":"2023-10-07T09:48:23.324889+0000","last_peered":"2023-10-07T09:48:23.324889+0000","last_clean":"2023-10-07T09:48:23.324889+0000","last_became_active":"2023-10-07T09:48:22.334988+0000","last_became_peered":"2023-10-07T09:48:22.334988+0000","last_unstale":"2023-10-07T09:48:23.324889+0000","last_undegraded":"2023-10-07T09:48:23.324889+0000","last_fullsized":"2023-10-07T09:48:23.324889+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:05:06.761798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.345308+0000","last_change":"2023-10-07T09:48:23.339125+0000","last_active":"2023-10-07T09:48:23.345308+0000","last_peered":"2023-10-07T09:48:23.345308+0000","last_clean":"2023-10-07T09:48:23.345308+0000","last_became_active":"2023-10-07T09:48:23.338859+0000","last_became_peered":"2023-10-07T09:48:23.338859+0000","last_unstale":"2023-10-07T09:48:23.345308+0000","last_undegraded":"2023-10-07T09:48:23.345308+0000","last_fullsized":"2023-10-07T09:48:23.345308+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:11:15.941025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.332752+0000","last_change":"2023-10-07T09:48:22.337369+0000","last_active":"2023-10-07T09:48:23.332752+0000","last_peered":"2023-10-07T09:48:23.332752+0000","last_clean":"2023-10-07T09:48:23.332752+0000","last_became_active":"2023-10-07T09:48:22.337080+0000","last_became_peered":"2023-10-07T09:48:22.337080+0000","last_unstale":"2023-10-07T09:48:23.332752+0000","last_undegraded":"2023-10-07T09:48:23.332752+0000","last_fullsized":"2023-10-07T09:48:23.332752+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:50:45.881008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.352206+0000","last_change":"2023-10-07T09:48:23.341978+0000","last_active":"2023-10-07T09:48:23.352206+0000","last_peered":"2023-10-07T09:48:23.352206+0000","last_clean":"2023-10-07T09:48:23.352206+0000","last_became_active":"2023-10-07T09:48:23.341510+0000","last_became_peered":"2023-10-07T09:48:23.341510+0000","last_unstale":"2023-10-07T09:48:23.352206+0000","last_undegraded":"2023-10-07T09:48:23.352206+0000","last_fullsized":"2023-10-07T09:48:23.352206+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:54.623904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.324971+0000","last_change":"2023-10-07T09:48:22.326218+0000","last_active":"2023-10-07T09:48:23.324971+0000","last_peered":"2023-10-07T09:48:23.324971+0000","last_clean":"2023-10-07T09:48:23.324971+0000","last_became_active":"2023-10-07T09:48:22.325941+0000","last_became_peered":"2023-10-07T09:48:22.325941+0000","last_unstale":"2023-10-07T09:48:23.324971+0000","last_undegraded":"2023-10-07T09:48:23.324971+0000","last_fullsized":"2023-10-07T09:48:23.324971+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:26:11.281727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.344685+0000","last_change":"2023-10-07T09:48:23.337780+0000","last_active":"2023-10-07T09:48:23.344685+0000","last_peered":"2023-10-07T09:48:23.344685+0000","last_clean":"2023-10-07T09:48:23.344685+0000","last_became_active":"2023-10-07T09:48:23.337512+0000","last_became_peered":"2023-10-07T09:48:23.337512+0000","last_unstale":"2023-10-07T09:48:23.344685+0000","last_undegraded":"2023-10-07T09:48:23.344685+0000","last_fullsized":"2023-10-07T09:48:23.344685+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:22.308512+0000","last_clean_scrub_stamp":"2023-10-07T09:48:22.308512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:21.035432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.326709+0000","last_change":"2023-10-07T09:48:22.328166+0000","last_active":"2023-10-07T09:48:23.326709+0000","last_peered":"2023-10-07T09:48:23.326709+0000","last_clean":"2023-10-07T09:48:23.326709+0000","last_became_active":"2023-10-07T09:48:22.327881+0000","last_became_peered":"2023-10-07T09:48:22.327881+0000","last_unstale":"2023-10-07T09:48:23.326709+0000","last_undegraded":"2023-10-07T09:48:23.326709+0000","last_fullsized":"2023-10-07T09:48:23.326709+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:48:21.300898+0000","last_clean_scrub_stamp":"2023-10-07T09:48:21.300898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:47:53.945223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"122'249","reported_seq":725,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-07T09:48:23.333427+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T09:48:23.333427+0000","last_peered":"2023-10-07T09:48:23.333427+0000","last_clean":"2023-10-07T09:48:23.333427+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T09:48:23.333427+0000","last_undegraded":"2023-10-07T09:48:23.333427+0000","last_fullsized":"2023-10-07T09:48:23.333427+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303576,"kb_used_data":2192,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068264,"statfs":{"total":96636764160,"available":96325902336,"internally_reserved":0,"allocated":2244608,"data_stored":1387488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71099999999999997,"15min":0.71699999999999997},"min":{"1min":0.433,"5min":0.39300000000000002,"15min":0.374},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69799999999999995,"15min":0.71999999999999997},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0349999999999999},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71699999999999997,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94799999999999995,"5min":0.98599999999999999,"15min":1.014},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72699999999999998,"15min":0.73999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.087,"5min":1.087,"15min":1.2430000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.54000000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.206},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71799999999999997,"15min":0.73699999999999999},"min":{"1min":0.50700000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.875,"5min":1.0309999999999999,"15min":1.071},"last":0.64700000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":0.73999999999999999},"min":{"1min":0.52000000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.56599999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.48999999999999999,"5min":0.438,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.93999999999999995,"15min":1.002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.64500000000000002},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.88100000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.45200000000000001,"5min":0.40699999999999997,"15min":0.36899999999999999},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68899999999999995,"15min":0.69099999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.625}]},{"osd":6,"last update":"Sat Oct 7 09:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46400000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72699999999999998,"15min":0.72199999999999998},"min":{"1min":0.52500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.65200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509665,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302992,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068848,"statfs":{"total":96636764160,"available":96326500352,"internally_reserved":0,"allocated":1646592,"data_stored":796602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70099999999999996,"15min":0.71099999999999997},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.41999999999999998},"max":{"1min":0.98899999999999999,"5min":1.274,"15min":1.274},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.45600000000000002,"5min":0.45200000000000001,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70399999999999996,"15min":0.71699999999999997},"min":{"1min":0.46700000000000003,"5min":0.46300000000000002,"15min":0.33700000000000002},"max":{"1min":0.95699999999999996,"5min":1.169,"15min":1.169},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.35899999999999999},"max":{"1min":0.83299999999999996,"5min":1.7,"15min":1.7},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.71799999999999997,"15min":0.73099999999999998},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":0.95499999999999996,"5min":1.482,"15min":1.482},"last":0.62},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76100000000000001,"15min":0.75900000000000001},"min":{"1min":0.63100000000000001,"5min":0.48899999999999999,"15min":0.40699999999999997},"max":{"1min":0.88800000000000001,"5min":1.643,"15min":1.643},"last":0.45700000000000002}]},{"osd":3,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.753},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45200000000000001},"max":{"1min":1.012,"5min":1.458,"15min":1.458},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.752,"15min":0.755},"min":{"1min":0.53800000000000003,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.89100000000000001,"5min":1.006,"15min":1.006},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61899999999999999,"15min":0.61799999999999999},"min":{"1min":0.42999999999999999,"5min":0.36599999999999999,"15min":0.29899999999999999},"max":{"1min":0.85199999999999998,"5min":0.91000000000000003,"15min":1.01},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.85899999999999999,"5min":1.018,"15min":1.018},"last":0.81899999999999995}]},{"osd":5,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.51700000000000002,"5min":0.42299999999999999,"15min":0.41199999999999998},"max":{"1min":0.88300000000000001,"5min":1.036,"15min":1.0509999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72199999999999998,"15min":0.71699999999999997},"min":{"1min":0.58499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78100000000000003}]},{"osd":7,"last update":"Sat Oct 7 09:47:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":1.099,"15min":1.099},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98799999999999999,"5min":0.99099999999999999,"15min":1.01},"last":0.74299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542367,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302996,"kb_used_data":1612,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068844,"statfs":{"total":96636764160,"available":96326496256,"internally_reserved":0,"allocated":1650688,"data_stored":796656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.41599999999999998,"5min":0.36299999999999999,"15min":0.316},"max":{"1min":0.746,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46300000000000002,"15min":0.47699999999999998},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.72099999999999997,"5min":0.73499999999999999,"15min":0.79600000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.55300000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.33600000000000002},"max":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":1.028},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.438,"5min":0.379,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.97599999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.38400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.89200000000000002},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76300000000000001,"15min":0.78300000000000003},"min":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.48599999999999999},"max":{"1min":1.0029999999999999,"5min":1.0549999999999999,"15min":1.4259999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73599999999999999,"15min":0.76600000000000001},"min":{"1min":0.58199999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94499999999999995,"5min":0.98399999999999999,"15min":1.196},"last":0.72099999999999997}]},{"osd":5,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.61099999999999999,"5min":0.55000000000000004,"15min":0.46300000000000002},"max":{"1min":1.038,"5min":1.0660000000000001,"15min":1.2769999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.751,"15min":0.77900000000000003},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.98099999999999998,"5min":1.0680000000000001,"15min":1.3300000000000001},"last":0.755}]},{"osd":6,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78000000000000003,"15min":0.81100000000000005},"min":{"1min":0.58399999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77500000000000002,"15min":0.81599999999999995},"min":{"1min":0.59499999999999997,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80100000000000005,"15min":0.83099999999999996},"min":{"1min":0.57999999999999996,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.2450000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78900000000000003,"15min":0.83699999999999997},"min":{"1min":0.57599999999999996,"5min":0.53900000000000003,"15min":0.51700000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.385},"last":0.73199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303588,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068252,"statfs":{"total":96636764160,"available":96325890048,"internally_reserved":0,"allocated":2256896,"data_stored":1388548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47899999999999998,"15min":0.48799999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.622,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55200000000000005,"15min":0.56799999999999995},"min":{"1min":0.39700000000000002,"5min":0.38600000000000001,"15min":0.311},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85999999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.40100000000000002,"5min":0.34000000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.876,"15min":1.101},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.60799999999999998}]},{"osd":4,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.72499999999999998,"15min":0.751},"min":{"1min":0.58499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.232},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78600000000000003,"15min":0.79400000000000004},"min":{"1min":0.59699999999999998,"5min":0.52300000000000002,"15min":0.48599999999999999},"max":{"1min":1.069,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.90000000000000002}]},{"osd":5,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.745,"15min":0.77600000000000002},"min":{"1min":0.58099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0229999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.45500000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.194},"last":0.86399999999999999}]},{"osd":6,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.59599999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.048,"5min":1.048,"15min":1.129},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77500000000000002,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97799999999999998,"5min":1.206,"15min":1.206},"last":0.92800000000000005}]},{"osd":7,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78800000000000003,"15min":0.82299999999999995},"min":{"1min":0.61599999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.0820000000000001,"5min":1.0920000000000001,"15min":1.167},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78700000000000003,"15min":0.81999999999999995},"min":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.503},"max":{"1min":0.99299999999999999,"5min":1.238,"15min":1.238},"last":0.88200000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542368,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060660,"statfs":{"total":96636764160,"available":96318115840,"internally_reserved":0,"allocated":1642496,"data_stored":796532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.442,"15min":0.45100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29399999999999998,"15min":0.27500000000000002},"max":{"1min":0.59999999999999998,"5min":0.66200000000000003,"15min":1.258},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.32100000000000001,"15min":0.28999999999999998},"max":{"1min":0.65900000000000003,"5min":0.79000000000000004,"15min":1.2290000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":1.1950000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.40999999999999998,"15min":0.38400000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":1.3200000000000001},"last":0.61799999999999999}]},{"osd":3,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.30599999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":1.5820000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.72099999999999997,"5min":0.79900000000000004,"15min":1.052},"last":0.437}]},{"osd":4,"last update":"Sat Oct 7 09:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.59499999999999997,"5min":0.48099999999999998,"15min":0.35699999999999998},"max":{"1min":0.94999999999999996,"5min":1.1120000000000001,"15min":1.4630000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.76400000000000001,"15min":0.79600000000000004},"min":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.327,"15min":1.8919999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Sat Oct 7 09:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.61199999999999999,"5min":0.52000000000000002,"15min":0.48999999999999999},"max":{"1min":0.95199999999999996,"5min":1.173,"15min":1.98},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.77000000000000002,"15min":0.78900000000000003},"min":{"1min":0.54800000000000004,"5min":0.52500000000000002,"15min":0.46100000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.5169999999999999},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Oct 7 09:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.77500000000000002,"15min":0.81899999999999995},"min":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":0.91100000000000003,"5min":1.1379999999999999,"15min":1.931},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80200000000000005,"15min":0.82599999999999996},"min":{"1min":0.66000000000000003,"5min":0.56399999999999995,"15min":0.47299999999999998},"max":{"1min":1.115,"5min":1.2849999999999999,"15min":1.8120000000000001},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Oct 7 09:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.80500000000000005,"15min":0.83799999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.56999999999999995},"max":{"1min":1.0529999999999999,"5min":1.2450000000000001,"15min":1.8500000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80100000000000005,"15min":0.82799999999999996},"min":{"1min":0.61099999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":1.1439999999999999,"15min":1.3839999999999999},"last":0.50900000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542369,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311196,"kb_used_data":1620,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060644,"statfs":{"total":96636764160,"available":96318099456,"internally_reserved":0,"allocated":1658880,"data_stored":798138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.28299999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.9939999999999998,"5min":7.9939999999999998,"15min":7.9939999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.32200000000000001},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":5.5359999999999996,"5min":5.5359999999999996,"15min":5.5359999999999996},"last":0.439},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.65500000000000003,"15min":0.61399999999999999},"min":{"1min":0.443,"5min":0.42699999999999999,"15min":0.42199999999999999},"max":{"1min":8.0990000000000002,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":1.0509999999999999,"5min":1.206,"15min":1.206},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.62,"15min":0.59199999999999997},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"max":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"last":0.48899999999999999}]},{"osd":4,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.83699999999999997,"15min":0.79700000000000004},"min":{"1min":0.61299999999999999,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.78900000000000003},"min":{"1min":0.58499999999999996,"5min":0.56699999999999995,"15min":0.48399999999999999},"max":{"1min":1.157,"5min":1.1639999999999999,"15min":1.3009999999999999},"last":1.0109999999999999}]},{"osd":5,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.83599999999999997,"15min":0.80800000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"max":{"1min":3.0419999999999998,"5min":3.0419999999999998,"15min":3.0419999999999998},"last":1.079},{"interface":"front","average":{"1min":1.0860000000000001,"5min":0.86699999999999999,"15min":0.81799999999999995},"min":{"1min":0.622,"5min":0.56599999999999995,"15min":0.53200000000000003},"max":{"1min":5.8170000000000002,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":0.82299999999999995}]},{"osd":6,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":0.86499999999999999,"15min":0.82599999999999996},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"max":{"1min":5.5789999999999997,"5min":5.5789999999999997,"15min":5.5789999999999997},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.331,"5min":0.92900000000000005,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":7.9459999999999997,"5min":7.9459999999999997,"15min":7.9459999999999997},"last":1.101}]},{"osd":7,"last update":"Sat Oct 7 09:47:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.90500000000000003,"15min":0.85099999999999998},"min":{"1min":0.622,"5min":0.58499999999999996,"15min":0.54400000000000004},"max":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.208,"5min":0.92200000000000004,"15min":0.86099999999999999},"min":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.55600000000000005},"max":{"1min":8.0470000000000006,"5min":8.0470000000000006,"15min":8.0470000000000006},"last":1.0409999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509665,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307072,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064768,"statfs":{"total":96636764160,"available":96322322432,"internally_reserved":0,"allocated":1630208,"data_stored":789868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67800000000000005,"15min":0.69099999999999995},"min":{"1min":0.44400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":1.069},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.0009999999999999},"last":0.75}]},{"osd":1,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69599999999999995,"15min":0.71399999999999997},"min":{"1min":0.47299999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.054},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.478},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.749},"min":{"1min":0.56899999999999995,"5min":0.504,"15min":0.432},"max":{"1min":1.014,"5min":1.0680000000000001,"15min":1.518},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.71199999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.40000000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.0209999999999999},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69599999999999995,"15min":0.71499999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46700000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.98699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.52300000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89500000000000002,"5min":1.0249999999999999,"15min":1.036},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.92300000000000004,"5min":1.427,"15min":1.5509999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.611,"5min":1.611,"15min":1.639},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63600000000000001,"15min":0.63200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.5389999999999999,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.46000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98899999999999999,"5min":1.272,"15min":1.585},"last":0.72299999999999998}]},{"osd":7,"last update":"Sat Oct 7 09:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.753,"15min":0.752},"min":{"1min":0.502,"5min":0.502,"15min":0.45200000000000001},"max":{"1min":1.0660000000000001,"5min":1.5760000000000001,"15min":1.6180000000000001},"last":0.629}]}]},{"osd":5,"up_from":14,"seq":60129542367,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302996,"kb_used_data":1612,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068844,"statfs":{"total":96636764160,"available":96326496256,"internally_reserved":0,"allocated":1650688,"data_stored":797108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.74199999999999999,"15min":0.751},"min":{"1min":0.51400000000000001,"5min":0.46100000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":1.115,"15min":1.218},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.52900000000000003,"5min":0.47099999999999997,"15min":0.436},"max":{"1min":0.98799999999999999,"5min":1.0189999999999999,"15min":1.294},"last":0.69299999999999995}]},{"osd":1,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.72399999999999998},"min":{"1min":0.46999999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91300000000000003,"5min":1.0780000000000001,"15min":1.1339999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.74199999999999999,"15min":0.73899999999999999},"min":{"1min":0.50700000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88200000000000001,"5min":1.0640000000000001,"15min":1.194},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.69299999999999995,"15min":0.70999999999999996},"min":{"1min":0.48299999999999998,"5min":0.47299999999999998,"15min":0.41099999999999998},"max":{"1min":0.79800000000000004,"5min":1.012,"15min":1.119},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.54100000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.76500000000000001,"5min":1.054,"15min":1.054},"last":0.82099999999999995}]},{"osd":3,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.748,"15min":0.76000000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83699999999999997,"5min":1.0569999999999999,"15min":1.2589999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.46899999999999997},"max":{"1min":0.79500000000000004,"5min":1.02,"15min":1.036},"last":0.63}]},{"osd":4,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.628,"15min":0.623},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82599999999999996,"5min":0.89100000000000001,"15min":1.0489999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63100000000000001,"15min":0.627},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39400000000000002},"max":{"1min":0.83199999999999996,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.33900000000000002},"max":{"1min":0.84999999999999998,"5min":1.052,"15min":1.052},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.47799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.1040000000000001},"last":0.755}]},{"osd":7,"last update":"Sat Oct 7 09:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.69799999999999995},"min":{"1min":0.439,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88700000000000001,"5min":1.095,"15min":1.095},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76100000000000001,"15min":0.76500000000000001},"min":{"1min":0.54500000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.92900000000000005,"5min":0.95799999999999996,"15min":1.3360000000000001},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T09:48:28.235 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T09:48:28.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T09:48:28.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.470+0000 7f1be85ff700 1 Processor -- start 2023-10-07T09:48:28.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.471+0000 7f1be85ff700 1 -- start start 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.472+0000 7f1be85ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 0x7f1be015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.472+0000 7f1be85ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 0x7f1be00ae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.472+0000 7f1be85ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be00ac5e0 con 0x7f1be00ac100 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.472+0000 7f1be85ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be00abae0 con 0x7f1be00b1550 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.472+0000 7f1be85ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00ac8c0 con 0x7f1be00b1920 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be639b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 0x7f1be015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 0x7f1be00ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 0x7f1be00ae360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:37018/0 (socket says 172.21.15.29:37018) 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be639b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 0x7f1be015bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:53030/0 (socket says 172.21.15.29:53030) 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 -- 172.21.15.29:0/4076565731 learned_addr learned my addr 172.21.15.29:0/4076565731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 -- 172.21.15.29:0/4076565731 >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 msgr2=0x7f1be015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 --2- 172.21.15.29:0/4076565731 >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 0x7f1be015bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 -- 172.21.15.29:0/4076565731 >> v1:172.21.15.144:6789/0 conn(0x7f1be00ac100 legacy=0x7f1be00adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 -- 172.21.15.29:0/4076565731 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be00acba0 con 0x7f1be00b1920 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be639b700 1 --2- 172.21.15.29:0/4076565731 >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 0x7f1be015bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.473+0000 7f1be6b9c700 1 --2- 172.21.15.29:0/4076565731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 0x7f1be00ae360 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009f20 tx=0x7f1bd8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac7f728114953df server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be5399700 1 -- 172.21.15.29:0/4076565731 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd800e070 con 0x7f1be00b1920 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be5399700 1 -- 172.21.15.29:0/4076565731 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd800b9d0 con 0x7f1be00b1920 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be5399700 1 -- 172.21.15.29:0/4076565731 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80092e0 con 0x7f1be00b1920 2023-10-07T09:48:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 -- 172.21.15.29:0/4076565731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 msgr2=0x7f1be00ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 --2- 172.21.15.29:0/4076565731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 0x7f1be00ae360 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009f20 tx=0x7f1bd8009840 comp rx=0 tx=0).stop 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 -- 172.21.15.29:0/4076565731 shutdown_connections 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 --2- 172.21.15.29:0/4076565731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1920 0x7f1be00ae360 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 --2- 172.21.15.29:0/4076565731 >> v2:172.21.15.144:3300/0 conn(0x7f1be00b1550 0x7f1be015bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 -- 172.21.15.29:0/4076565731 >> 172.21.15.29:0/4076565731 conn(0x7f1be0067610 msgr2=0x7f1be0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 -- 172.21.15.29:0/4076565731 shutdown_connections 2023-10-07T09:48:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.474+0000 7f1be85ff700 1 -- 172.21.15.29:0/4076565731 wait complete. 2023-10-07T09:48:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.476+0000 7f1be85ff700 1 Processor -- start 2023-10-07T09:48:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.476+0000 7f1be85ff700 1 -- start start 2023-10-07T09:48:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be85ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1be00ac100 0x7f1be014f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be85ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1550 0x7f1be014bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be85ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 0x7f1be0154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be85ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be014c8a0 con 0x7f1be00b1920 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be85ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be014ce20 con 0x7f1be00ac100 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be85ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0154580 con 0x7f1be00b1550 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be6b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 0x7f1be0154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be6b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 0x7f1be0154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50122/0 (socket says 172.21.15.29:50122) 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be6b9c700 1 -- 172.21.15.29:0/1142033947 learned_addr learned my addr 172.21.15.29:0/1142033947 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be639b700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1be00ac100 0x7f1be014f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.477+0000 7f1be5b9a700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1550 0x7f1be014bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be6b9c700 1 -- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1550 msgr2=0x7f1be014bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be6b9c700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1550 0x7f1be014bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be6b9c700 1 -- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1be00ac100 msgr2=0x7f1be014f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be6b9c700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1be00ac100 0x7f1be014f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be6b9c700 1 -- 172.21.15.29:0/1142033947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc00d8d0 con 0x7f1be00b1920 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be6b9c700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 0x7f1be0154ed0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800b610 tx=0x7f1bd8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8007ac0 con 0x7f1be00b1920 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd8007c20 con 0x7f1be00b1920 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd8009ae0 con 0x7f1be00b1920 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.478+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd801d420 con 0x7f1be00b1920 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.479+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be014cab0 con 0x7f1be00b1920 2023-10-07T09:48:28.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.480+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be014cfb0 con 0x7f1be00b1920 2023-10-07T09:48:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.481+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1bd8003040 con 0x7f1be00b1920 2023-10-07T09:48:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.481+0000 7f1bd67fc700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1bc4066f30 0x7f1bc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.482+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1bd80861c0 con 0x7f1be00b1920 2023-10-07T09:48:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.482+0000 7f1be639b700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1bc4066f30 0x7f1bc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.483+0000 7f1be639b700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1bc4066f30 0x7f1bc40693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00dfd0 tx=0x7f1bdc00f040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:28.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.486+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1bd8053210 con 0x7f1be00b1920 2023-10-07T09:48:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.768+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1be00718b0 con 0x7f1be00b1920 2023-10-07T09:48:28.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.772+0000 7f1bd67fc700 1 -- 172.21.15.29:0/1142033947 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+0 (secure 0 0 0) 0x7f1bd805ae60 con 0x7f1be00b1920 2023-10-07T09:48:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.774+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1bc4066f30 msgr2=0x7f1bc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.774+0000 7f1be85ff700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1bc4066f30 0x7f1bc40693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00dfd0 tx=0x7f1bdc00f040 comp rx=0 tx=0).stop 2023-10-07T09:48:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.774+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 msgr2=0x7f1be0154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.774+0000 7f1be85ff700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 0x7f1be0154ed0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800b610 tx=0x7f1bd8018000 comp rx=0 tx=0).stop 2023-10-07T09:48:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 shutdown_connections 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1bc4066f30 0x7f1bc40693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be00b1920 0x7f1be0154ed0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1be00ac100 0x7f1be014f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 --2- 172.21.15.29:0/1142033947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1be00b1550 0x7f1be014bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 >> 172.21.15.29:0/1142033947 conn(0x7f1be0067610 msgr2=0x7f1be006d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 shutdown_connections 2023-10-07T09:48:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.775+0000 7f1be85ff700 1 -- 172.21.15.29:0/1142033947 wait complete. 2023-10-07T09:48:28.778 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T09:48:28.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T09:48:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.987+0000 7febed1ff700 1 Processor -- start 2023-10-07T09:48:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.987+0000 7febed1ff700 1 -- start start 2023-10-07T09:48:28.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.988+0000 7febed1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febe8159750 0x7febe80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febed1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febed1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe80cbf20 con 0x7febe8159b20 2023-10-07T09:48:28.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febed1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe80cc200 con 0x7febe8159750 2023-10-07T09:48:28.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febed1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe80cc4e0 con 0x7febe80ce8a0 2023-10-07T09:48:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febe759e700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febe6d9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febe8159750 0x7febe80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febe759e700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:37032/0 (socket says 172.21.15.29:37032) 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.989+0000 7febe759e700 1 -- 172.21.15.29:0/38141171 learned_addr learned my addr 172.21.15.29:0/38141171 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe759e700 1 -- 172.21.15.29:0/38141171 >> v2:172.21.15.144:3300/0 conn(0x7febe8159750 msgr2=0x7febe80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe759e700 1 --2- 172.21.15.29:0/38141171 >> v2:172.21.15.144:3300/0 conn(0x7febe8159750 0x7febe80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe759e700 1 -- 172.21.15.29:0/38141171 >> v1:172.21.15.144:6789/0 conn(0x7febe8159b20 legacy=0x7febe80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe759e700 1 -- 172.21.15.29:0/38141171 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe80cc7c0 con 0x7febe80ce8a0 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe6d9d700 1 --2- 172.21.15.29:0/38141171 >> v2:172.21.15.144:3300/0 conn(0x7febe8159750 0x7febe80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe5d9b700 1 -- 172.21.15.29:0/38141171 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430617674 0 0) 0x7febe80cbf20 con 0x7febe8159b20 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe759e700 1 --2- 172.21.15.29:0/38141171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80c5460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7febe0009f20 tx=0x7febe0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35bc9e1d0dacd17 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.990+0000 7febe5d9b700 1 -- 172.21.15.29:0/38141171 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe000e070 con 0x7febe80ce8a0 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febe5d9b700 1 -- 172.21.15.29:0/38141171 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe000b9d0 con 0x7febe80ce8a0 2023-10-07T09:48:28.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febe5d9b700 1 -- 172.21.15.29:0/38141171 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe00092e0 con 0x7febe80ce8a0 2023-10-07T09:48:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 -- 172.21.15.29:0/38141171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 msgr2=0x7febe80c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 --2- 172.21.15.29:0/38141171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80c5460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7febe0009f20 tx=0x7febe0009840 comp rx=0 tx=0).stop 2023-10-07T09:48:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 -- 172.21.15.29:0/38141171 shutdown_connections 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 --2- 172.21.15.29:0/38141171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80c5460 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 --2- 172.21.15.29:0/38141171 >> v2:172.21.15.144:3300/0 conn(0x7febe8159750 0x7febe80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 -- 172.21.15.29:0/38141171 >> 172.21.15.29:0/38141171 conn(0x7febe8063a10 msgr2=0x7febe80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 -- 172.21.15.29:0/38141171 shutdown_connections 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.991+0000 7febed1ff700 1 -- 172.21.15.29:0/38141171 wait complete. 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.993+0000 7febed1ff700 1 Processor -- start 2023-10-07T09:48:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.993+0000 7febed1ff700 1 -- start start 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febed1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febed1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febe8159750 0x7febe8143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febed1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 0x7febe8154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febed1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe8154ad0 con 0x7febe8159b20 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febed1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe8153ec0 con 0x7febe8159750 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febed1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe81541a0 con 0x7febe80ce8a0 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febe759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 0x7febe8154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febe759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 0x7febe8154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50132/0 (socket says 172.21.15.29:50132) 2023-10-07T09:48:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febe759e700 1 -- 172.21.15.29:0/2247868930 learned_addr learned my addr 172.21.15.29:0/2247868930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febe6d9d700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.994+0000 7febe659c700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febe8159750 0x7febe8143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febe759e700 1 -- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 msgr2=0x7febe80ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febe759e700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febe759e700 1 -- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febe8159750 msgr2=0x7febe8143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febe759e700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febe8159750 0x7febe8143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febe759e700 1 -- 172.21.15.29:0/2247868930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd80093c0 con 0x7febe8159b20 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febe759e700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 0x7febe8154f60 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7febe80cc620 tx=0x7febe0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0008b10 con 0x7febe8159b20 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe0008c70 con 0x7febe8159b20 2023-10-07T09:48:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0026460 con 0x7febe8159b20 2023-10-07T09:48:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.995+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe0009ae0 con 0x7febe8159b20 2023-10-07T09:48:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.996+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe8154760 con 0x7febe8159b20 2023-10-07T09:48:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.996+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe804c3c0 con 0x7febe8159b20 2023-10-07T09:48:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.997+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7febe0003040 con 0x7febe8159b20 2023-10-07T09:48:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.999+0000 7febceffd700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7febd4066f30 0x7febd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.999+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7febe0086570 con 0x7febe8159b20 2023-10-07T09:48:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:28.999+0000 7febe6d9d700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7febd4066f30 0x7febd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.000+0000 7febe6d9d700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7febd4066f30 0x7febd40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7febe8160290 tx=0x7febd8009120 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.002+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7febe0053670 con 0x7febe8159b20 2023-10-07T09:48:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.242+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febe80ce030 con 0x7febe8159b20 2023-10-07T09:48:29.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.243+0000 7febceffd700 1 -- 172.21.15.29:0/2247868930 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4599 (secure 0 0 0) 0x7febe0056c90 con 0x7febe8159b20 2023-10-07T09:48:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:29.244 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:48:22.817019+0000","modified":"2023-10-07T09:48:26.355188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/2749584891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2749584891},{"type":"v1","addr":"172.21.15.29:6834","nonce":2749584891}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-10-07T09:48:29.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7febd4066f30 msgr2=0x7febd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7febd4066f30 0x7febd40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7febe8160290 tx=0x7febd8009120 comp rx=0 tx=0).stop 2023-10-07T09:48:29.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 msgr2=0x7febe8154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 0x7febe8154f60 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7febe80cc620 tx=0x7febe0018000 comp rx=0 tx=0).stop 2023-10-07T09:48:29.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 shutdown_connections 2023-10-07T09:48:29.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7febd4066f30 0x7febd40693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe8159b20 0x7febe8154f60 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febe80ce8a0 0x7febe80ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 --2- 172.21.15.29:0/2247868930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febe8159750 0x7febe8143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 >> 172.21.15.29:0/2247868930 conn(0x7febe8063a10 msgr2=0x7febe80c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 shutdown_connections 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.245+0000 7febed1ff700 1 -- 172.21.15.29:0/2247868930 wait complete. 2023-10-07T09:48:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-07T09:48:29.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T09:48:22.817019+0000', 'modified': '2023-10-07T09:48:26.355188+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8160}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8160': {'gid': 8160, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6834/2749584891', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 2749584891}, {'type': 'v1', 'addr': '172.21.15.29:6834', 'nonce': 2749584891}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T09:48:29.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T09:48:29.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-07T09:48:29.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.865+0000 7f5af2bff700 1 Processor -- start 2023-10-07T09:48:29.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.865+0000 7f5af2bff700 1 -- start start 2023-10-07T09:48:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.866+0000 7f5af2bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5aec153cc0 0x7f5aec0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 0x7f5aec0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af2bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec154090 con 0x7f5aec15ade0 2023-10-07T09:48:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af2bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec159f60 con 0x7f5aec153cc0 2023-10-07T09:48:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec15a240 con 0x7f5aec0b0710 2023-10-07T09:48:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af099b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5aec153cc0 0x7f5aec0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af119c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 0x7f5aec0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af119c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 0x7f5aec0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:37052/0 (socket says 172.21.15.29:37052) 2023-10-07T09:48:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.867+0000 7f5af119c700 1 -- 172.21.15.29:0/2542323131 learned_addr learned my addr 172.21.15.29:0/2542323131 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5af119c700 1 -- 172.21.15.29:0/2542323131 >> v2:172.21.15.144:3300/0 conn(0x7f5aec153cc0 msgr2=0x7f5aec0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5af119c700 1 --2- 172.21.15.29:0/2542323131 >> v2:172.21.15.144:3300/0 conn(0x7f5aec153cc0 0x7f5aec0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5af119c700 1 -- 172.21.15.29:0/2542323131 >> v1:172.21.15.144:6789/0 conn(0x7f5aec15ade0 legacy=0x7f5aec0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5af119c700 1 -- 172.21.15.29:0/2542323131 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec15a520 con 0x7f5aec0b0710 2023-10-07T09:48:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5af119c700 1 --2- 172.21.15.29:0/2542323131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 0x7f5aec0b1d00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009f20 tx=0x7f5ae4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d2c01a873931a60 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5aeb7fe700 1 -- 172.21.15.29:0/2542323131 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400e070 con 0x7f5aec0b0710 2023-10-07T09:48:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5aeb7fe700 1 -- 172.21.15.29:0/2542323131 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae400b9d0 con 0x7f5aec0b0710 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.868+0000 7f5aeb7fe700 1 -- 172.21.15.29:0/2542323131 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40092e0 con 0x7f5aec0b0710 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 -- 172.21.15.29:0/2542323131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 msgr2=0x7f5aec0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2542323131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 0x7f5aec0b1d00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009f20 tx=0x7f5ae4009840 comp rx=0 tx=0).stop 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 -- 172.21.15.29:0/2542323131 shutdown_connections 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2542323131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec0b0710 0x7f5aec0b1d00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2542323131 >> v2:172.21.15.144:3300/0 conn(0x7f5aec153cc0 0x7f5aec0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 -- 172.21.15.29:0/2542323131 >> 172.21.15.29:0/2542323131 conn(0x7f5aec067610 msgr2=0x7f5aec067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 -- 172.21.15.29:0/2542323131 shutdown_connections 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.869+0000 7f5af2bff700 1 -- 172.21.15.29:0/2542323131 wait complete. 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.870+0000 7f5af2bff700 1 Processor -- start 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.870+0000 7f5af2bff700 1 -- start start 2023-10-07T09:48:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.871+0000 7f5af2bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 0x7f5aec151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.871+0000 7f5af2bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec153cc0 0x7f5aec0c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.871+0000 7f5af2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec15ade0 0x7f5aec156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.871+0000 7f5af2bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1561b0 con 0x7f5aec0b0710 2023-10-07T09:48:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.871+0000 7f5af2bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1555a0 con 0x7f5aec153cc0 2023-10-07T09:48:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec155880 con 0x7f5aec15ade0 2023-10-07T09:48:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 0x7f5aec151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 0x7f5aec151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50148/0 (socket says 172.21.15.29:50148) 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 -- 172.21.15.29:0/2553488703 learned_addr learned my addr 172.21.15.29:0/2553488703 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af119c700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec15ade0 0x7f5aec156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5aebfff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec153cc0 0x7f5aec0c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 -- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec15ade0 msgr2=0x7f5aec156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec15ade0 0x7f5aec156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 -- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec153cc0 msgr2=0x7f5aec0c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec153cc0 0x7f5aec0c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.872+0000 7f5af099b700 1 -- 172.21.15.29:0/2553488703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4009ae0 con 0x7f5aec0b0710 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.873+0000 7f5af099b700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 0x7f5aec151bb0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000ec80 tx=0x7f5ae0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.873+0000 7f5ae8ff9700 1 -- 172.21.15.29:0/2553488703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae000c850 con 0x7f5aec0b0710 2023-10-07T09:48:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.873+0000 7f5ae8ff9700 1 -- 172.21.15.29:0/2553488703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae000c9b0 con 0x7f5aec0b0710 2023-10-07T09:48:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.873+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec155b60 con 0x7f5aec0b0710 2023-10-07T09:48:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.873+0000 7f5ae8ff9700 1 -- 172.21.15.29:0/2553488703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae000cd30 con 0x7f5aec0b0710 2023-10-07T09:48:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.873+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec1520f0 con 0x7f5aec0b0710 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.875+0000 7f5ae8ff9700 1 -- 172.21.15.29:0/2553488703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5ae0013680 con 0x7f5aec0b0710 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.876+0000 7f5ae8ff9700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad0066f30 0x7f5ad00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.876+0000 7f5ae8ff9700 1 -- 172.21.15.29:0/2553488703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5ae0084cb0 con 0x7f5aec0b0710 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.876+0000 7f5aebfff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad0066f30 0x7f5ad00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.877+0000 7f5aebfff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad0066f30 0x7f5ad00693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0cf3e0 tx=0x7f5ad8007ec0 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.878+0000 7f5b013f2b80 1 Processor -- start 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.879+0000 7f5b013f2b80 1 -- start start 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 0x562110723e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56211072cbc0 0x56211072d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621105f6310 con 0x562110723a40 2023-10-07T09:48:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562110633130 con 0x56211072cbc0 2023-10-07T09:48:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562110630af0 con 0x562110709530 2023-10-07T09:48:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 0x562110723e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 0x562110723e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50150/0 (socket says 172.21.15.29:50150) 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 -- 172.21.15.29:0/2996551028 learned_addr learned my addr 172.21.15.29:0/2996551028 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5aebfff700 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af119c700 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56211072cbc0 0x56211072d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 -- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 msgr2=0x56211072c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 -- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56211072cbc0 msgr2=0x56211072d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56211072cbc0 0x56211072d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.880+0000 7f5af099b700 1 -- 172.21.15.29:0/2996551028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56211051f0a0 con 0x562110723a40 2023-10-07T09:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.881+0000 7f5af099b700 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 0x562110723e10 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0051ec0 tx=0x7f5ae0082f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9bb7291dc0503e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.881+0000 7f5ad5ffb700 1 -- 172.21.15.29:0/2996551028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0056140 con 0x562110723a40 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.881+0000 7f5ad5ffb700 1 -- 172.21.15.29:0/2996551028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae00559c0 con 0x562110723a40 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.881+0000 7f5ad5ffb700 1 -- 172.21.15.29:0/2996551028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0058c10 con 0x562110723a40 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 -- 172.21.15.29:0/2996551028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 msgr2=0x562110723e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 0x562110723e10 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0051ec0 tx=0x7f5ae0082f10 comp rx=0 tx=0).stop 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 -- 172.21.15.29:0/2996551028 shutdown_connections 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56211072cbc0 0x56211072d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110723a40 0x562110723e10 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/2996551028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 -- 172.21.15.29:0/2996551028 >> 172.21.15.29:0/2996551028 conn(0x562110726e70 msgr2=0x562110727240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 -- 172.21.15.29:0/2996551028 shutdown_connections 2023-10-07T09:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.882+0000 7f5b013f2b80 1 -- 172.21.15.29:0/2996551028 wait complete. 2023-10-07T09:48:29.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.883+0000 7f5b013f2b80 1 -- start start 2023-10-07T09:48:29.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.883+0000 7f5b013f2b80 1 Processor -- start 2023-10-07T09:48:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x56211070b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621107074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56211062c8e0 con 0x56211072cbc0 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621105d8980 con 0x562110723a40 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5621105d8620 con 0x562110709530 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5b013f2b80 10 client.0 _ll_register_callbacks cb 0x56211043ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621107074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621107074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50156/0 (socket says 172.21.15.29:50156) 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 -- 172.21.15.29:0/3166652319 learned_addr learned my addr 172.21.15.29:0/3166652319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af099b700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5aebfff700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x56211070b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 -- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 msgr2=0x56211072b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 -- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 msgr2=0x56211070b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x56211070b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.884+0000 7f5af119c700 1 -- 172.21.15.29:0/3166652319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec1520f0 con 0x56211072cbc0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5af119c700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621107074f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4012000 tx=0x7f5ae4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fba2ff4b28a22a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5acaffd700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5acaffd700 1 -- 172.21.15.29:0/3166652319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4023070 con 0x56211072cbc0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621106d1560 con 0x56211072cbc0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5acaffd700 1 -- 172.21.15.29:0/3166652319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5ae4027400 con 0x56211072cbc0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5acaffd700 1 -- 172.21.15.29:0/3166652319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ae40277d0 con 0x56211072cbc0 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5b013f2b80 10 client.8283 fetch_fsmap learned FSMap version 68 2023-10-07T09:48:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.885+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x5621105fb3c0 con 0x56211072cbc0 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5acaffd700 1 -- 172.21.15.29:0/3166652319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5621106d1560 con 0x56211072cbc0 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5acaffd700 1 -- 172.21.15.29:0/3166652319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 68) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5ae400b050 con 0x56211072cbc0 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5b013f2b80 10 client.8283 fetch_fsmap finished waiting for FSMap version 68 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5b013f2b80 10 client.8283 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5b013f2b80 20 client.8283 populate_metadata read hostname 'smithi029' 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x562110709e50 0x56211070a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5b013f2b80 4 client.8283 mds_command: new command op to 8160 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 --> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621106ae0e0 con 0x562110709e50 2023-10-07T09:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.886+0000 7f5af099b700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x562110709e50 0x56211070a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5af099b700 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x562110709e50 0x56211070a220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbbd5ab5aa2b2ef8 server_cookie=a6ecc54b696962bd in_seq=0 out_seq=0 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5acaffd700 10 client.8283 ms_handle_connect on v2:172.21.15.29:6832/2749584891 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5acaffd700 1 -- 172.21.15.29:0/3166652319 <== mds.0 v2:172.21.15.29:6832/2749584891 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5621106ae0e0 con 0x562110709e50 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5acaffd700 10 client.8283 handle_command_reply: tid=0 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5b013f2b80 1 client.8283 shutdown 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 msgr2=0x5621107074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.887+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621107074f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4012000 tx=0x7f5ae4018000 comp rx=0 tx=0).stop 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 shutdown_connections 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x562110709e50 0x56211070a220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621107074f0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110709530 0x56211072b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5acaffd700 0 client.8283 ms_handle_reset on v2:172.21.15.29:6832/2749584891 2023-10-07T09:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3166652319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x56211070b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 >> 172.21.15.29:0/3166652319 conn(0x562110726e70 msgr2=0x5621106d9250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 shutdown_connections 2023-10-07T09:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3166652319 wait complete. 2023-10-07T09:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.888+0000 7f5b013f2b80 20 client.8283 trim_cache size 0 max 16384 2023-10-07T09:48:29.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.906+0000 7f5b013f2b80 1 Processor -- start 2023-10-07T09:48:29.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.906+0000 7f5b013f2b80 1 -- start start 2023-10-07T09:48:29.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110709530 0x56211072a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110723a40 0x5621106dc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621106dc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621104c41e0 con 0x56211072cbc0 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562110724800 con 0x562110709530 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5621107254d0 con 0x562110723a40 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5af119c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621106dc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5af119c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621106dc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50164/0 (socket says 172.21.15.29:50164) 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5af119c700 1 -- 172.21.15.29:0/3100030753 learned_addr learned my addr 172.21.15.29:0/3100030753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5af099b700 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110709530 0x56211072a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.907+0000 7f5aebfff700 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110723a40 0x5621106dc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5af119c700 1 -- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110723a40 msgr2=0x5621106dc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5af119c700 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110723a40 0x5621106dc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5af119c700 1 -- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110709530 msgr2=0x56211072a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5af119c700 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110709530 0x56211072a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5af119c700 1 -- 172.21.15.29:0/3100030753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621104931d0 con 0x56211072cbc0 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5af119c700 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621106dc950 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4002230 tx=0x7f5ae4023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e604d276f51d46 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5ad67fc700 1 -- 172.21.15.29:0/3100030753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40277d0 con 0x56211072cbc0 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5ad67fc700 1 -- 172.21.15.29:0/3100030753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4027400 con 0x56211072cbc0 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.908+0000 7f5ad67fc700 1 -- 172.21.15.29:0/3100030753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400bd50 con 0x56211072cbc0 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3100030753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 msgr2=0x5621106dc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621106dc950 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4002230 tx=0x7f5ae4023000 comp rx=0 tx=0).stop 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3100030753 shutdown_connections 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56211072cbc0 0x5621106dc950 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110709530 0x56211072a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/3100030753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562110723a40 0x5621106dc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3100030753 >> 172.21.15.29:0/3100030753 conn(0x562110726e70 msgr2=0x56211072b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3100030753 shutdown_connections 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.909+0000 7f5b013f2b80 1 -- 172.21.15.29:0/3100030753 wait complete. 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.910+0000 7f5b013f2b80 1 -- start start 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.910+0000 7f5b013f2b80 1 Processor -- start 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 0x56211070ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x562110771700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56211072cbc0 0x5621106d27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562110789930 con 0x562110709530 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621106dc410 con 0x562110723a40 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5621106dd270 con 0x56211072cbc0 2023-10-07T09:48:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5b013f2b80 10 client.0 _ll_register_callbacks cb 0x56211043ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 0x56211070ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 0x56211070ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50168/0 (socket says 172.21.15.29:50168) 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 -- 172.21.15.29:0/4011097030 learned_addr learned my addr 172.21.15.29:0/4011097030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5aebfff700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x562110771700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af119c700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56211072cbc0 0x5621106d27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 -- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56211072cbc0 msgr2=0x5621106d27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56211072cbc0 0x5621106d27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 -- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 msgr2=0x562110771700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x562110771700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.911+0000 7f5af099b700 1 -- 172.21.15.29:0/4011097030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621105fb3c0 con 0x562110709530 2023-10-07T09:48:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5aebfff700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x562110771700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5af099b700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 0x56211070ac30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000f010 tx=0x7f5ae006c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b674793a59e9a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5acaffd700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5acaffd700 1 -- 172.21.15.29:0/4011097030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0058c10 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5acaffd700 1 -- 172.21.15.29:0/4011097030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5ae0053b00 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5aec1589a0 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5acaffd700 1 -- 172.21.15.29:0/4011097030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ae0055e30 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5b013f2b80 10 client.8289 fetch_fsmap learned FSMap version 68 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x56211046cd60 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5acaffd700 1 -- 172.21.15.29:0/4011097030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec1589a0 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5acaffd700 1 -- 172.21.15.29:0/4011097030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 68) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5ae006a070 con 0x562110709530 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5b013f2b80 10 client.8289 fetch_fsmap finished waiting for FSMap version 68 2023-10-07T09:48:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5b013f2b80 10 client.8289 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.912+0000 7f5b013f2b80 20 client.8289 populate_metadata read hostname 'smithi029' 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.913+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x5621107887b0 0x562110788b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.913+0000 7f5b013f2b80 4 client.8289 mds_command: new command op to 8160 tid=0[{"prefix": "status"}] 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.913+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 --> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f5ae00559c0 con 0x5621107887b0 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.913+0000 7f5aebfff700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x5621107887b0 0x562110788b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.913+0000 7f5acaffd700 10 client.8289 ms_handle_connect on v2:172.21.15.29:6832/2749584891 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.913+0000 7f5aebfff700 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x5621107887b0 0x562110788b80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67a7249ffe1028d1 server_cookie=2d3a4649e4d0e115 in_seq=0 out_seq=0 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5acaffd700 1 -- 172.21.15.29:0/4011097030 <== mds.0 v2:172.21.15.29:6832/2749584891 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f5ae00559c0 con 0x5621107887b0 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5acaffd700 10 client.8289 handle_command_reply: tid=0 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 client.8289 shutdown 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 msgr2=0x56211070ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 0x56211070ac30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000f010 tx=0x7f5ae006c000 comp rx=0 tx=0).stop 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 shutdown_connections 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:6832/2749584891,v1:172.21.15.29:6834/2749584891] conn(0x5621107887b0 0x562110788b80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56211072cbc0 0x5621106d27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562110709530 0x56211070ac30 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5acaffd700 0 client.8289 ms_handle_reset on v2:172.21.15.29:6832/2749584891 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 --2- 172.21.15.29:0/4011097030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562110723a40 0x562110771700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 >> 172.21.15.29:0/4011097030 conn(0x562110726e70 msgr2=0x56211072b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 shutdown_connections 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 1 -- 172.21.15.29:0/4011097030 wait complete. 2023-10-07T09:48:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.914+0000 7f5b013f2b80 20 client.8289 trim_cache size 0 max 16384 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8160, 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5985360010000003, 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 68, 2023-10-07T09:48:29.923 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 133, 2023-10-07T09:48:29.924 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 133, 2023-10-07T09:48:29.924 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.281566524 2023-10-07T09:48:29.924 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T09:48:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad0066f30 msgr2=0x7f5ad00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad0066f30 0x7f5ad00693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0cf3e0 tx=0x7f5ad8007ec0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 msgr2=0x7f5aec151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 0x7f5aec151bb0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000ec80 tx=0x7f5ae0009120 comp rx=0 tx=0).stop 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 shutdown_connections 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5ad0066f30 0x7f5ad00693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5aec15ade0 0x7f5aec156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5aec0b0710 0x7f5aec151bb0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 --2- 172.21.15.29:0/2553488703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5aec153cc0 0x7f5aec0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 >> 172.21.15.29:0/2553488703 conn(0x7f5aec067610 msgr2=0x7f5aec15b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 shutdown_connections 2023-10-07T09:48:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:29.915+0000 7f5af2bff700 1 -- 172.21.15.29:0/2553488703 wait complete. 2023-10-07T09:48:29.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T09:48:30.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.129+0000 7fd1c69ff700 1 Processor -- start 2023-10-07T09:48:30.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.129+0000 7fd1c69ff700 1 -- start start 2023-10-07T09:48:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.130+0000 7fd1c69ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd1c0157290 0x7fd1c00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.130+0000 7fd1c69ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c00ae8b0 0x7fd1c00b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.130+0000 7fd1c69ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1c00b1840 0x7fd1c00ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.130+0000 7fd1c69ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c00b1ca0 con 0x7fd1c00b1840 2023-10-07T09:48:30.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.130+0000 7fd1c69ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c0158ad0 con 0x7fd1c0157290 2023-10-07T09:48:30.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.130+0000 7fd1c69ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c015b140 con 0x7fd1c00ae8b0 2023-10-07T09:48:30.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1c00b1840 0x7fd1c00ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1c00b1840 0x7fd1c00ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50182/0 (socket says 172.21.15.29:50182) 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 -- 172.21.15.29:0/3686696655 learned_addr learned my addr 172.21.15.29:0/3686696655 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1bf7fe700 1 --2- 172.21.15.29:0/3686696655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c00ae8b0 0x7fd1c00b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1bffff700 1 --2- 172.21.15.29:0/3686696655 >> v2:172.21.15.144:3300/0 conn(0x7fd1c0157290 0x7fd1c00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 -- 172.21.15.29:0/3686696655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c00ae8b0 msgr2=0x7fd1c00b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 --2- 172.21.15.29:0/3686696655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c00ae8b0 0x7fd1c00b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 -- 172.21.15.29:0/3686696655 >> v2:172.21.15.144:3300/0 conn(0x7fd1c0157290 msgr2=0x7fd1c00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 --2- 172.21.15.29:0/3686696655 >> v2:172.21.15.144:3300/0 conn(0x7fd1c0157290 0x7fd1c00cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 -- 172.21.15.29:0/3686696655 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c015b420 con 0x7fd1c00b1840 2023-10-07T09:48:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.131+0000 7fd1c4f9c700 1 --2- 172.21.15.29:0/3686696655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00ca310 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00685f0 tx=0x7fd1b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88eb9f2c26b91af server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1beffd700 1 -- 172.21.15.29:0/3686696655 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800e070 con 0x7fd1c00b1840 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1beffd700 1 -- 172.21.15.29:0/3686696655 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b800ba20 con 0x7fd1c00b1840 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1beffd700 1 -- 172.21.15.29:0/3686696655 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b80092e0 con 0x7fd1c00b1840 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 -- 172.21.15.29:0/3686696655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 msgr2=0x7fd1c00ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/3686696655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00ca310 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00685f0 tx=0x7fd1b8009840 comp rx=0 tx=0).stop 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 -- 172.21.15.29:0/3686696655 shutdown_connections 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/3686696655 >> v2:172.21.15.144:3300/0 conn(0x7fd1c0157290 0x7fd1c00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/3686696655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00ca310 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/3686696655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c00ae8b0 0x7fd1c00b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 -- 172.21.15.29:0/3686696655 >> 172.21.15.29:0/3686696655 conn(0x7fd1c00680a0 msgr2=0x7fd1c00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 -- 172.21.15.29:0/3686696655 shutdown_connections 2023-10-07T09:48:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.132+0000 7fd1c69ff700 1 -- 172.21.15.29:0/3686696655 wait complete. 2023-10-07T09:48:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.134+0000 7fd1c69ff700 1 Processor -- start 2023-10-07T09:48:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.134+0000 7fd1c69ff700 1 -- start start 2023-10-07T09:48:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1c69ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1c69ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c0157290 0x7fd1c0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1c69ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1c00c5cc0 0x7fd1c00c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1c69ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0166f60 con 0x7fd1c00b1840 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1c69ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c00c58c0 con 0x7fd1c00c5cc0 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1c69ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c00c66a0 con 0x7fd1c0157290 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50190/0 (socket says 172.21.15.29:50190) 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.135+0000 7fd1bffff700 1 -- 172.21.15.29:0/1079210229 learned_addr learned my addr 172.21.15.29:0/1079210229 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bf7fe700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c0157290 0x7fd1c0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1c4f9c700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1c00c5cc0 0x7fd1c00c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bffff700 1 -- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c0157290 msgr2=0x7fd1c0150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bffff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c0157290 0x7fd1c0150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bffff700 1 -- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1c00c5cc0 msgr2=0x7fd1c00c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bffff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1c00c5cc0 0x7fd1c00c6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bffff700 1 -- 172.21.15.29:0/1079210229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8009ae0 con 0x7fd1c00b1840 2023-10-07T09:48:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1c4f9c700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1c00c5cc0 0x7fd1c00c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T09:48:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1bffff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00c5380 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000d890 tx=0x7fd1b0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0009610 con 0x7fd1c00b1840 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b0009770 con 0x7fd1c00b1840 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0009b30 con 0x7fd1c00b1840 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.136+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c0150db0 con 0x7fd1c00b1840 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.137+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c0169a20 con 0x7fd1c00b1840 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.138+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c004c3c0 con 0x7fd1c00b1840 2023-10-07T09:48:30.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.138+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd1b001e460 con 0x7fd1c00b1840 2023-10-07T09:48:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.139+0000 7fd1a7fff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1ac066f30 0x7fd1ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.139+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd1b0084dc0 con 0x7fd1c00b1840 2023-10-07T09:48:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.140+0000 7fd1bf7fe700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1ac066f30 0x7fd1ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.140+0000 7fd1bf7fe700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1ac066f30 0x7fd1ac0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009d40 tx=0x7fd1b4008040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:30.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.143+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1b0051e10 con 0x7fd1c00b1840 2023-10-07T09:48:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.397+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd1c015b700 con 0x7fd1c00b1840 2023-10-07T09:48:30.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.397+0000 7fd1a7fff700 1 -- 172.21.15.29:0/1079210229 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd1b0055430 con 0x7fd1c00b1840 2023-10-07T09:48:30.398 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1ac066f30 msgr2=0x7fd1ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1ac066f30 0x7fd1ac0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009d40 tx=0x7fd1b4008040 comp rx=0 tx=0).stop 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 msgr2=0x7fd1c00c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00c5380 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000d890 tx=0x7fd1b0008040 comp rx=0 tx=0).stop 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 shutdown_connections 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1ac066f30 0x7fd1ac0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1c00c5cc0 0x7fd1c00c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1c00b1840 0x7fd1c00c5380 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 --2- 172.21.15.29:0/1079210229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1c0157290 0x7fd1c0150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 >> 172.21.15.29:0/1079210229 conn(0x7fd1c00680a0 msgr2=0x7fd1c01599b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 shutdown_connections 2023-10-07T09:48:30.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:30.399+0000 7fd1c69ff700 1 -- 172.21.15.29:0/1079210229 wait complete. 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T09:48:30.418 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:48:30.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:30.419 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T09:48:30.465 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:48:30.466 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41118sec preferred_lft 41118sec 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T09:48:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T09:48:30.468 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T09:48:30.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:30.470 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T09:48:30.539 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T09:48:30.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:30.542 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T09:48:30.590 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T09:48:30.590 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T09:48:30.590 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T09:48:30.591 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T09:48:30.591 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T09:48:30.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:30.592 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T09:48:30.592 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T09:48:30.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T09:48:30.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T09:48:30.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T09:48:30.592 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T09:48:30.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:30.765 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T09:48:30.812 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T09:48:30.812 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T09:48:30.813 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:48:30.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:30.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T09:48:30.885 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:48:30.885 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T09:48:30.932 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T09:48:30.932 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T09:48:30.932 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:48:30.978 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T09:48:30.978 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T09:48:30.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:48:31.119 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:48:31.119 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:48:31.119 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:48:31.134 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:48:31.134 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:48:31.134 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:48:31.134 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:48:31.134 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:48:31.157 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T09:48:31.157 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T09:48:31.157 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T09:48:31.157 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T09:48:31.157 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T09:48:31.158 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T09:48:31.158 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T09:48:31.158 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T09:48:31.163 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T09:48:31.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T09:48:31.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T09:48:31.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T09:48:31.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.434+0000 7f431ddff700 1 Processor -- start 2023-10-07T09:48:31.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.434+0000 7f431ddff700 1 -- start start 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.435+0000 7f431ddff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43181533f0 0x7f4318156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.435+0000 7f431ddff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318159670 con 0x7f43180b04b0 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f431ddff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431815a1f0 con 0x7f43181537c0 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f431ddff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431815a680 con 0x7f43181533f0 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f43177fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43181533f0 0x7f4318156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f4317fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f43180b04b0 0x7f4318157240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45376/0 (socket says 172.21.15.29:45376) 2023-10-07T09:48:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f4317fff700 1 -- 172.21.15.29:0/2157461064 learned_addr learned my addr 172.21.15.29:0/2157461064 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f43177fe700 1 -- 172.21.15.29:0/2157461064 >> v1:172.21.15.144:6789/0 conn(0x7f43181537c0 legacy=0x7f43180afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f43177fe700 1 -- 172.21.15.29:0/2157461064 >> v1:172.21.15.29:6789/0 conn(0x7f43180b04b0 legacy=0x7f4318157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f43177fe700 1 -- 172.21.15.29:0/2157461064 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43181598c0 con 0x7f43181533f0 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.436+0000 7f43167fc700 1 -- 172.21.15.29:0/2157461064 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310510768 0 0) 0x7f4318159670 con 0x7f43180b04b0 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f43177fe700 1 --2- 172.21.15.29:0/2157461064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43181533f0 0x7f4318156a90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f43080098e0 tx=0x7f43080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4224dcc6fba2f0f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f43167fc700 1 -- 172.21.15.29:0/2157461064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308009e80 con 0x7f43181533f0 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f43167fc700 1 -- 172.21.15.29:0/2157461064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430800b860 con 0x7f43181533f0 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f43167fc700 1 -- 172.21.15.29:0/2157461064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800bbe0 con 0x7f43181533f0 2023-10-07T09:48:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f431ddff700 1 -- 172.21.15.29:0/2157461064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43181533f0 msgr2=0x7f4318156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f431ddff700 1 --2- 172.21.15.29:0/2157461064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43181533f0 0x7f4318156a90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f43080098e0 tx=0x7f43080091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f431ddff700 1 -- 172.21.15.29:0/2157461064 shutdown_connections 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f431ddff700 1 --2- 172.21.15.29:0/2157461064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43181533f0 0x7f4318156a90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.437+0000 7f431ddff700 1 -- 172.21.15.29:0/2157461064 >> 172.21.15.29:0/2157461064 conn(0x7f43180680a0 msgr2=0x7f43180cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.438+0000 7f431ddff700 1 -- 172.21.15.29:0/2157461064 shutdown_connections 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.438+0000 7f431ddff700 1 -- 172.21.15.29:0/2157461064 wait complete. 2023-10-07T09:48:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.439+0000 7f431ddff700 1 Processor -- start 2023-10-07T09:48:31.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.439+0000 7f431ddff700 1 -- start start 2023-10-07T09:48:31.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f431ddff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 0x7f431814ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f431ddff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43181533f0 0x7f43180ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f431ddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43181537c0 0x7f43180cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f431ddff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318168360 con 0x7f43180b04b0 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f431ddff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318167770 con 0x7f43181533f0 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f431ddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4318167a30 con 0x7f43181537c0 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f43177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 0x7f431814ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f43177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 0x7f431814ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50202/0 (socket says 172.21.15.29:50202) 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.440+0000 7f43177fe700 1 -- 172.21.15.29:0/161706937 learned_addr learned my addr 172.21.15.29:0/161706937 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f4317fff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43181537c0 0x7f43180cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f4316ffd700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43181533f0 0x7f43180ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f43177fe700 1 -- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43181537c0 msgr2=0x7f43180cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f43177fe700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43181537c0 0x7f43180cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f43177fe700 1 -- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43181533f0 msgr2=0x7f43180ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f43177fe700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43181533f0 0x7f43180ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f43177fe700 1 -- 172.21.15.29:0/161706937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308009420 con 0x7f43180b04b0 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f43177fe700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 0x7f431814ab40 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f4318068dd0 tx=0x7f4308017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430801f070 con 0x7f43180b04b0 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430800b9d0 con 0x7f43180b04b0 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4318167d10 con 0x7f43180b04b0 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308025670 con 0x7f43180b04b0 2023-10-07T09:48:31.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.441+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431814b080 con 0x7f43180b04b0 2023-10-07T09:48:31.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.442+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431804c3c0 con 0x7f43180b04b0 2023-10-07T09:48:31.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.443+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4308023070 con 0x7f43180b04b0 2023-10-07T09:48:31.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.444+0000 7f42ff7fe700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4304066f30 0x7f43040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.444+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f4308085db0 con 0x7f43180b04b0 2023-10-07T09:48:31.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.445+0000 7f4316ffd700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4304066f30 0x7f43040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.445+0000 7f4316ffd700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4304066f30 0x7f43040693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4318152a50 tx=0x7f430c009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:31.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.448+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f430808a070 con 0x7f43180b04b0 2023-10-07T09:48:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.678+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f43181684a0 con 0x7f4304066f30 2023-10-07T09:48:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.680+0000 7f42ff7fe700 1 -- 172.21.15.29:0/161706937 <== mgr.7582 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f430c013420 con 0x7f4304066f30 2023-10-07T09:48:31.682 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T09:48:31.682 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T09:48:31.682 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T09:48:31.682 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T09:48:31.682 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T09:48:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4304066f30 msgr2=0x7f43040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4304066f30 0x7f43040693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4318152a50 tx=0x7f430c009040 comp rx=0 tx=0).stop 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 msgr2=0x7f431814ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 0x7f431814ab40 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f4318068dd0 tx=0x7f4308017000 comp rx=0 tx=0).stop 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 shutdown_connections 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4304066f30 0x7f43040693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43181537c0 0x7f43180cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43180b04b0 0x7f431814ab40 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 --2- 172.21.15.29:0/161706937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43181533f0 0x7f43180ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 >> 172.21.15.29:0/161706937 conn(0x7f43180680a0 msgr2=0x7f4318154b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 shutdown_connections 2023-10-07T09:48:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.683+0000 7f431ddff700 1 -- 172.21.15.29:0/161706937 wait complete. 2023-10-07T09:48:31.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T09:48:31.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.900+0000 7ff8501ff700 1 Processor -- start 2023-10-07T09:48:31.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.900+0000 7ff8501ff700 1 -- start start 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.901+0000 7ff8501ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff8480afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.901+0000 7ff8501ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8480b0520 0x7ff8480b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.901+0000 7ff8501ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8480b0ec0 con 0x7ff848156180 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.901+0000 7ff8501ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8480ad4d0 con 0x7ff8480b0520 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.901+0000 7ff8501ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480ad7b0 con 0x7ff8480ae570 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84df9b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff848156180 0x7ff8480accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:46972/0 (socket says 172.21.15.29:46972) 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84df9b700 1 -- 172.21.15.29:0/3532632400 learned_addr learned my addr 172.21.15.29:0/3532632400 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84d79a700 1 --2- 172.21.15.29:0/3532632400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff8480afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84e79c700 1 --2- 172.21.15.29:0/3532632400 >> v2:172.21.15.144:3300/0 conn(0x7ff8480b0520 0x7ff8480b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84d79a700 1 -- 172.21.15.29:0/3532632400 >> v2:172.21.15.144:3300/0 conn(0x7ff8480b0520 msgr2=0x7ff8480b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:31.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84d79a700 1 --2- 172.21.15.29:0/3532632400 >> v2:172.21.15.144:3300/0 conn(0x7ff8480b0520 0x7ff8480b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84d79a700 1 -- 172.21.15.29:0/3532632400 >> v1:172.21.15.144:6789/0 conn(0x7ff848156180 legacy=0x7ff8480accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84d79a700 1 -- 172.21.15.29:0/3532632400 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480ada90 con 0x7ff8480ae570 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.902+0000 7ff84e79c700 1 --2- 172.21.15.29:0/3532632400 >> v2:172.21.15.144:3300/0 conn(0x7ff8480b0520 0x7ff8480b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff84d79a700 1 --2- 172.21.15.29:0/3532632400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff8480afc00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff8380098e0 tx=0x7ff8380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=499fbe378684b4a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff84cf99700 1 -- 172.21.15.29:0/3532632400 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83800b640 con 0x7ff8480ae570 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff84cf99700 1 -- 172.21.15.29:0/3532632400 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83800b7a0 con 0x7ff8480ae570 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff84cf99700 1 -- 172.21.15.29:0/3532632400 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83800bb40 con 0x7ff8480ae570 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff8501ff700 1 -- 172.21.15.29:0/3532632400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 msgr2=0x7ff8480afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff8501ff700 1 --2- 172.21.15.29:0/3532632400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff8480afc00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff8380098e0 tx=0x7ff8380091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff8501ff700 1 -- 172.21.15.29:0/3532632400 shutdown_connections 2023-10-07T09:48:31.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff8501ff700 1 --2- 172.21.15.29:0/3532632400 >> v2:172.21.15.144:3300/0 conn(0x7ff8480b0520 0x7ff8480b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff8501ff700 1 --2- 172.21.15.29:0/3532632400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff8480afc00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.903+0000 7ff8501ff700 1 -- 172.21.15.29:0/3532632400 >> 172.21.15.29:0/3532632400 conn(0x7ff848067610 msgr2=0x7ff848067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:31.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.904+0000 7ff8501ff700 1 -- 172.21.15.29:0/3532632400 shutdown_connections 2023-10-07T09:48:31.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.904+0000 7ff8501ff700 1 -- 172.21.15.29:0/3532632400 wait complete. 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.905+0000 7ff8501ff700 1 Processor -- start 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.905+0000 7ff8501ff700 1 -- start start 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.906+0000 7ff8501ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff84816abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.906+0000 7ff8501ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8480b0520 0x7ff8480abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.906+0000 7ff8501ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 0x7ff84816b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.906+0000 7ff8501ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84816b120 con 0x7ff848156180 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.906+0000 7ff8501ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480a6440 con 0x7ff8480b0520 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.906+0000 7ff8501ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480ac340 con 0x7ff8480ae570 2023-10-07T09:48:31.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 0x7ff84816b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84df9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff84816abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 0x7ff84816b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50210/0 (socket says 172.21.15.29:50210) 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 -- 172.21.15.29:0/1570561387 learned_addr learned my addr 172.21.15.29:0/1570561387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84d79a700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8480b0520 0x7ff8480abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 -- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 msgr2=0x7ff84816abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff84816abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 -- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8480b0520 msgr2=0x7ff8480abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8480b0520 0x7ff8480abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 -- 172.21.15.29:0/1570561387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840009a20 con 0x7ff848156180 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.907+0000 7ff84e79c700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 0x7ff84816b330 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7ff840009fc0 tx=0x7ff84000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.908+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84000cbd0 con 0x7ff848156180 2023-10-07T09:48:31.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.908+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84000f040 con 0x7ff848156180 2023-10-07T09:48:31.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.908+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff838009420 con 0x7ff848156180 2023-10-07T09:48:31.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.908+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840014480 con 0x7ff848156180 2023-10-07T09:48:31.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.908+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8480a5b10 con 0x7ff848156180 2023-10-07T09:48:31.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.909+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84804c3c0 con 0x7ff848156180 2023-10-07T09:48:31.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.910+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff840003020 con 0x7ff848156180 2023-10-07T09:48:31.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.911+0000 7ff83dffb700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff82c066f30 0x7ff82c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:31.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.911+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7ff840084fe0 con 0x7ff848156180 2023-10-07T09:48:31.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.912+0000 7ff84df9b700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff82c066f30 0x7ff82c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:31.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.912+0000 7ff84df9b700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff82c066f30 0x7ff82c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff8480c78f0 tx=0x7ff844009040 comp rx=0 tx=0).ready entity=mgr.7582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:31.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:31.915+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8400520d0 con 0x7ff848156180 2023-10-07T09:48:32.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.170+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff8480a6580 con 0x7ff848156180 2023-10-07T09:48:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.386+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 58) v1 ==== 84757+0+0 (secure 0 0 0) 0x7ff8400524a0 con 0x7ff848156180 2023-10-07T09:48:32.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.387+0000 7ff83dffb700 1 -- 172.21.15.29:0/1570561387 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff840055b60 con 0x7ff848156180 2023-10-07T09:48:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff82c066f30 msgr2=0x7ff82c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff82c066f30 0x7ff82c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff8480c78f0 tx=0x7ff844009040 comp rx=0 tx=0).stop 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 msgr2=0x7ff84816b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 0x7ff84816b330 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7ff840009fc0 tx=0x7ff84000b040 comp rx=0 tx=0).stop 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 shutdown_connections 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff82c066f30 0x7ff82c0693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff848156180 0x7ff84816b330 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8480ae570 0x7ff84816abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 --2- 172.21.15.29:0/1570561387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8480b0520 0x7ff8480abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 >> 172.21.15.29:0/1570561387 conn(0x7ff848067610 msgr2=0x7ff8480aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:32.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 shutdown_connections 2023-10-07T09:48:32.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.390+0000 7ff8501ff700 1 -- 172.21.15.29:0/1570561387 wait complete. 2023-10-07T09:48:32.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T09:48:32.604 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:32.604+0000 7f567e56d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:48:32.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:32.604+0000 7fc80910f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T09:48:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.616+0000 7f0662bff700 1 Processor -- start 2023-10-07T09:48:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.616+0000 7f0662bff700 1 -- start start 2023-10-07T09:48:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f0662bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f065c159750 0x7f065c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f0662bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f065c0ce8a0 0x7f065c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f0662bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c0cbf20 con 0x7f065c159b20 2023-10-07T09:48:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f0662bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c0cc200 con 0x7f065c0ce8a0 2023-10-07T09:48:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f0662bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c0cc4e0 con 0x7f065c159750 2023-10-07T09:48:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f066119c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f065c0ce8a0 0x7f065c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f066119c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f065c0ce8a0 0x7f065c0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50230/0 (socket says 172.21.15.29:50230) 2023-10-07T09:48:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f066119c700 1 -- 172.21.15.29:0/789503224 learned_addr learned my addr 172.21.15.29:0/789503224 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.618+0000 7f066099b700 1 --2- 172.21.15.29:0/789503224 >> v2:172.21.15.144:3300/0 conn(0x7f065c159750 0x7f065c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f066119c700 1 -- 172.21.15.29:0/789503224 >> v2:172.21.15.144:3300/0 conn(0x7f065c159750 msgr2=0x7f065c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f066119c700 1 --2- 172.21.15.29:0/789503224 >> v2:172.21.15.144:3300/0 conn(0x7f065c159750 0x7f065c0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f066119c700 1 -- 172.21.15.29:0/789503224 >> v1:172.21.15.29:6789/0 conn(0x7f065c159b20 legacy=0x7f065c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f066119c700 1 -- 172.21.15.29:0/789503224 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c0cc7c0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f066119c700 1 --2- 172.21.15.29:0/789503224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c0c5460 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f065c089840 tx=0x7f0654009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f473f73301085aa0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f065b7fe700 1 -- 172.21.15.29:0/789503224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065400e070 con 0x7f065c0ce8a0 2023-10-07T09:48:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f065b7fe700 1 -- 172.21.15.29:0/789503224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065400ba20 con 0x7f065c0ce8a0 2023-10-07T09:48:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.619+0000 7f065b7fe700 1 -- 172.21.15.29:0/789503224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540092e0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 -- 172.21.15.29:0/789503224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 msgr2=0x7f065c0c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 --2- 172.21.15.29:0/789503224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c0c5460 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f065c089840 tx=0x7f0654009840 comp rx=0 tx=0).stop 2023-10-07T09:48:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 -- 172.21.15.29:0/789503224 shutdown_connections 2023-10-07T09:48:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 --2- 172.21.15.29:0/789503224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c0c5460 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 --2- 172.21.15.29:0/789503224 >> v2:172.21.15.144:3300/0 conn(0x7f065c159750 0x7f065c0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 -- 172.21.15.29:0/789503224 >> 172.21.15.29:0/789503224 conn(0x7f065c063a10 msgr2=0x7f065c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 -- 172.21.15.29:0/789503224 shutdown_connections 2023-10-07T09:48:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.620+0000 7f0662bff700 1 -- 172.21.15.29:0/789503224 wait complete. 2023-10-07T09:48:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.621+0000 7f0662bff700 1 Processor -- start 2023-10-07T09:48:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.622+0000 7f0662bff700 1 -- start start 2023-10-07T09:48:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.622+0000 7f0662bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c15a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f0662bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065c159750 0x7f065c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f0662bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f065c159b20 0x7f065c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f0662bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c15e7a0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f0662bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0ca3f0 con 0x7f065c159750 2023-10-07T09:48:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f0662bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c15af00 con 0x7f065c159b20 2023-10-07T09:48:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c15a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c15a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50240/0 (socket says 172.21.15.29:50240) 2023-10-07T09:48:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 -- 172.21.15.29:0/3602879542 learned_addr learned my addr 172.21.15.29:0/3602879542 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066119c700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f065c159b20 0x7f065c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f065bfff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065c159750 0x7f065c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 -- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f065c159b20 msgr2=0x7f065c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f065c159b20 0x7f065c156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 -- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065c159750 msgr2=0x7f065c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065c159750 0x7f065c0c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.623+0000 7f066099b700 1 -- 172.21.15.29:0/3602879542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654009ae0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.624+0000 7f066099b700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c15a8d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f065000ec80 tx=0x7f0650009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.624+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065000c8e0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.624+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065000ca40 con 0x7f065c0ce8a0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.624+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065000cdc0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.624+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c15b1e0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.624+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c15b7a0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.625+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c15e570 con 0x7f065c0ce8a0 2023-10-07T09:48:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.626+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 58) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f0650013680 con 0x7f065c0ce8a0 2023-10-07T09:48:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.627+0000 7f0658ff9700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 0x7f0640069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.627+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f0650084dd0 con 0x7f065c0ce8a0 2023-10-07T09:48:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.627+0000 7f065bfff700 1 -- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 msgr2=0x7f0640069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.627+0000 7f065bfff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 0x7f0640069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.631+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f065c15e570 con 0x7f065c0ce8a0 2023-10-07T09:48:32.781 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:32.781+0000 7f567e56d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:48:32.817 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:32.815+0000 7fc80910f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T09:48:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.828+0000 7f065bfff700 1 -- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 msgr2=0x7f0640069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.828+0000 7f065bfff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 0x7f0640069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.873+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f065c15e570 con 0x7f065c0ce8a0 2023-10-07T09:48:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.877+0000 7f0658ff9700 1 -- 172.21.15.29:0/3602879542 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f065c15e570 con 0x7f065c0ce8a0 2023-10-07T09:48:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 msgr2=0x7f0640069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 0x7f0640069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 msgr2=0x7f065c15a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c15a8d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f065000ec80 tx=0x7f0650009120 comp rx=0 tx=0).stop 2023-10-07T09:48:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 shutdown_connections 2023-10-07T09:48:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0640066f80 0x7f0640069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f065c159b20 0x7f065c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c0ce8a0 0x7f065c15a8d0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 --2- 172.21.15.29:0/3602879542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065c159750 0x7f065c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 >> 172.21.15.29:0/3602879542 conn(0x7f065c063a10 msgr2=0x7f065c0cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 shutdown_connections 2023-10-07T09:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:32.880+0000 7f0662bff700 1 -- 172.21.15.29:0/3602879542 wait complete. 2023-10-07T09:48:32.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T09:48:33.094 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:33.094+0000 7f567e56d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:48:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.101+0000 7f9920bff700 1 Processor -- start 2023-10-07T09:48:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.101+0000 7f9920bff700 1 -- start start 2023-10-07T09:48:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.103+0000 7f9920bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f991c159790 0x7f991c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.103+0000 7f9920bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c06a8c0 0x7f991c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.103+0000 7f9920bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c06a430 con 0x7f991c0c6f40 2023-10-07T09:48:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.103+0000 7f9920bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991c159b60 con 0x7f991c159790 2023-10-07T09:48:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.103+0000 7f9920bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c7a70 con 0x7f991c06a8c0 2023-10-07T09:48:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.103+0000 7f991a59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f991c159790 0x7f991c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f991c159790 0x7f991c06aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39838/0 (socket says 172.21.15.29:39838) 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 -- 172.21.15.29:0/673486589 learned_addr learned my addr 172.21.15.29:0/673486589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f9919d9b700 1 --2- 172.21.15.29:0/673486589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c06a8c0 0x7f991c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 -- 172.21.15.29:0/673486589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c06a8c0 msgr2=0x7f991c06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 --2- 172.21.15.29:0/673486589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c06a8c0 0x7f991c06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 -- 172.21.15.29:0/673486589 >> v1:172.21.15.144:6789/0 conn(0x7f991c0c6f40 legacy=0x7f991c0c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 -- 172.21.15.29:0/673486589 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c069820 con 0x7f991c159790 2023-10-07T09:48:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991a59c700 1 --2- 172.21.15.29:0/673486589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c159790 0x7f991c06aeb0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f990c0099b0 tx=0x7f990c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491741748738dabf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.104+0000 7f991959a700 1 -- 172.21.15.29:0/673486589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c00b860 con 0x7f991c159790 2023-10-07T09:48:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.105+0000 7f991959a700 1 -- 172.21.15.29:0/673486589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c00b9c0 con 0x7f991c159790 2023-10-07T09:48:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.105+0000 7f991959a700 1 -- 172.21.15.29:0/673486589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c00bd60 con 0x7f991c159790 2023-10-07T09:48:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.105+0000 7f9920bff700 1 -- 172.21.15.29:0/673486589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c159790 msgr2=0x7f991c06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.105+0000 7f9920bff700 1 --2- 172.21.15.29:0/673486589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c159790 0x7f991c06aeb0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f990c0099b0 tx=0x7f990c009250 comp rx=0 tx=0).stop 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.106+0000 7f9920bff700 1 -- 172.21.15.29:0/673486589 shutdown_connections 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.106+0000 7f9920bff700 1 --2- 172.21.15.29:0/673486589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c159790 0x7f991c06aeb0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.106+0000 7f9920bff700 1 --2- 172.21.15.29:0/673486589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c06a8c0 0x7f991c06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.106+0000 7f9920bff700 1 -- 172.21.15.29:0/673486589 >> 172.21.15.29:0/673486589 conn(0x7f991c0c9410 msgr2=0x7f991c0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.106+0000 7f9920bff700 1 -- 172.21.15.29:0/673486589 shutdown_connections 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.106+0000 7f9920bff700 1 -- 172.21.15.29:0/673486589 wait complete. 2023-10-07T09:48:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.107+0000 7f9920bff700 1 Processor -- start 2023-10-07T09:48:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.107+0000 7f9920bff700 1 -- start start 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.108+0000 7f9920bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 0x7f991c0c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.108+0000 7f9920bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c159790 0x7f991c0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.108+0000 7f9920bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c0b1f10 0x7f991c0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.108+0000 7f9920bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c39b0 con 0x7f991c0b1f10 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.108+0000 7f9920bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c44c0 con 0x7f991c0c6f40 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.108+0000 7f9920bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c6660 con 0x7f991c159790 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 0x7f991c0c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c0b1f10 0x7f991c0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 0x7f991c0c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:56708/0 (socket says 172.21.15.29:56708) 2023-10-07T09:48:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 -- 172.21.15.29:0/3847944639 learned_addr learned my addr 172.21.15.29:0/3847944639 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:33.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f9919d9b700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c159790 0x7f991c0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 -- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c159790 msgr2=0x7f991c0b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c159790 0x7f991c0b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 -- 172.21.15.29:0/3847944639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c0b1f10 msgr2=0x7f991c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c0b1f10 0x7f991c0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 -- 172.21.15.29:0/3847944639 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c0094f0 con 0x7f991c0c6f40 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991ad9d700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c0b1f10 0x7f991c0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.109+0000 7f991a59c700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 0x7f991c0c4950 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f990c006fd0 tx=0x7f990c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.110+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c011e00 con 0x7f991c0c6f40 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.110+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c024440 con 0x7f991c0c6f40 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.110+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c024800 con 0x7f991c0c6f40 2023-10-07T09:48:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.110+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c0c3b90 con 0x7f991c0c6f40 2023-10-07T09:48:33.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.110+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c0c4150 con 0x7f991c0c6f40 2023-10-07T09:48:33.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.111+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c0b2310 con 0x7f991c0c6f40 2023-10-07T09:48:33.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.112+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 58) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f990c01a420 con 0x7f991c0c6f40 2023-10-07T09:48:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.113+0000 7f990a7fc700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 0x7f98fc069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.113+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f990c086540 con 0x7f991c0c6f40 2023-10-07T09:48:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.113+0000 7f9919d9b700 1 -- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 msgr2=0x7f98fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.113+0000 7f9919d9b700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 0x7f98fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:33.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.118+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f990c053590 con 0x7f991c0c6f40 2023-10-07T09:48:33.155 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:33.154+0000 7fc80910f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T09:48:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.314+0000 7f9919d9b700 1 -- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 msgr2=0x7f98fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.314+0000 7f9919d9b700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 0x7f98fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.321+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f991c04c3c0 con 0x7f991c0c6f40 2023-10-07T09:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.327+0000 7f990a7fc700 1 -- 172.21.15.29:0/3847944639 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f990c056bb0 con 0x7f991c0c6f40 2023-10-07T09:48:33.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 msgr2=0x7f98fc069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:33.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 0x7f98fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 msgr2=0x7f991c0c4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 0x7f991c0c4950 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f990c006fd0 tx=0x7f990c017000 comp rx=0 tx=0).stop 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 shutdown_connections 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98fc066f80 0x7f98fc069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f991c0b1f10 0x7f991c0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f991c0c6f40 0x7f991c0c4950 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 --2- 172.21.15.29:0/3847944639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f991c159790 0x7f991c0b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 >> 172.21.15.29:0/3847944639 conn(0x7f991c0c9410 msgr2=0x7f991c06aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 shutdown_connections 2023-10-07T09:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.330+0000 7f9920bff700 1 -- 172.21.15.29:0/3847944639 wait complete. 2023-10-07T09:48:33.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T09:48:33.350 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T09:48:33.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1 1M 2023-10-07T09:48:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.607+0000 7f9eb2bff700 1 Processor -- start 2023-10-07T09:48:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.607+0000 7f9eb2bff700 1 -- start start 2023-10-07T09:48:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.608+0000 7f9eb2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0b0610 0x7f9eac0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.608+0000 7f9eb2bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9eac0b09e0 0x7f9eac0c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.608+0000 7f9eb2bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eac0690f0 con 0x7f9eac0c8a20 2023-10-07T09:48:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.608+0000 7f9eb2bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac0c4740 con 0x7f9eac0b09e0 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.608+0000 7f9eb2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0cf2e0 con 0x7f9eac0b0610 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9eac0b09e0 0x7f9eac0c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9eac0b09e0 0x7f9eac0c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39846/0 (socket says 172.21.15.29:39846) 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 -- 172.21.15.29:0/336476234 learned_addr learned my addr 172.21.15.29:0/336476234 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eb099b700 1 --2- 172.21.15.29:0/336476234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0b0610 0x7f9eac0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 -- 172.21.15.29:0/336476234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0b0610 msgr2=0x7f9eac0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 --2- 172.21.15.29:0/336476234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0b0610 0x7f9eac0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 -- 172.21.15.29:0/336476234 >> v1:172.21.15.144:6789/0 conn(0x7f9eac0c8a20 legacy=0x7f9eac0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eabfff700 1 -- 172.21.15.29:0/336476234 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac0687c0 con 0x7f9eac0b09e0 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.609+0000 7f9eab7fe700 1 -- 172.21.15.29:0/336476234 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562029270 0 0) 0x7f9eac0690f0 con 0x7f9eac0c8a20 2023-10-07T09:48:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.610+0000 7f9eabfff700 1 --2- 172.21.15.29:0/336476234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c4200 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f9e980099b0 tx=0x7f9e98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e0880d168969b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.610+0000 7f9eab7fe700 1 -- 172.21.15.29:0/336476234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98009ea0 con 0x7f9eac0b09e0 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.610+0000 7f9eab7fe700 1 -- 172.21.15.29:0/336476234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9800b860 con 0x7f9eac0b09e0 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.610+0000 7f9eab7fe700 1 -- 172.21.15.29:0/336476234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800bc00 con 0x7f9eac0b09e0 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.610+0000 7f9eb2bff700 1 -- 172.21.15.29:0/336476234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b09e0 msgr2=0x7f9eac0c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.610+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/336476234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c4200 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f9e980099b0 tx=0x7f9e98009250 comp rx=0 tx=0).stop 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.611+0000 7f9eb2bff700 1 -- 172.21.15.29:0/336476234 shutdown_connections 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.611+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/336476234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0b0610 0x7f9eac0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.611+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/336476234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c4200 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.611+0000 7f9eb2bff700 1 -- 172.21.15.29:0/336476234 >> 172.21.15.29:0/336476234 conn(0x7f9eac063a10 msgr2=0x7f9eac0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.611+0000 7f9eb2bff700 1 -- 172.21.15.29:0/336476234 shutdown_connections 2023-10-07T09:48:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.611+0000 7f9eb2bff700 1 -- 172.21.15.29:0/336476234 wait complete. 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.612+0000 7f9eb2bff700 1 Processor -- start 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.612+0000 7f9eb2bff700 1 -- start start 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb2bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 0x7f9eac1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb2bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0c8a20 0x7f9eac0c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb2bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac152030 con 0x7f9eac0b0610 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb2bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0b1ab0 con 0x7f9eac0b09e0 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0c6410 con 0x7f9eac0c8a20 2023-10-07T09:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 0x7f9eac1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.613+0000 7f9eb099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 0x7f9eac1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39850/0 (socket says 172.21.15.29:39850) 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 -- 172.21.15.29:0/3632105129 learned_addr learned my addr 172.21.15.29:0/3632105129 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb119c700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0c8a20 0x7f9eac0c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0c8a20 msgr2=0x7f9eac0c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0c8a20 0x7f9eac0c3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9eac0b09e0 msgr2=0x7f9eac0c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 -- 172.21.15.29:0/3632105129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e980094f0 con 0x7f9eac0b0610 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.614+0000 7f9eb099b700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 0x7f9eac1517c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000e960 tx=0x7f9ea00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.615+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0013680 con 0x7f9eac0b0610 2023-10-07T09:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.615+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea00137e0 con 0x7f9eac0b0610 2023-10-07T09:48:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.615+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac0b1180 con 0x7f9eac0b0610 2023-10-07T09:48:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.615+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0013b80 con 0x7f9eac0b0610 2023-10-07T09:48:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.615+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac0b1740 con 0x7f9eac0b0610 2023-10-07T09:48:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.616+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac0718b0 con 0x7f9eac0b0610 2023-10-07T09:48:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.618+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 58) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f9ea00123d0 con 0x7f9eac0b0610 2023-10-07T09:48:33.619 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:33.619+0000 7f567e56d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:48:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.619+0000 7f9ea8ff9700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.619+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f9ea0084830 con 0x7f9eac0b0610 2023-10-07T09:48:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.619+0000 7f9eabfff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.619+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T09:48:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.623+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ea0051880 con 0x7f9eac0b0610 2023-10-07T09:48:33.718 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:33.717+0000 7fc80910f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T09:48:33.790 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:33.789+0000 7f567e56d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:48:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.820+0000 7f9eabfff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.820+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T09:48:33.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:33.855+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9eac068aa0 con 0x7f9e90066f80 2023-10-07T09:48:33.895 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:33.894+0000 7fc80910f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T09:48:34.206 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:34.205+0000 7f567e56d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:48:34.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:34.221+0000 7f9eabfff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:34.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:34.221+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T09:48:34.323 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:34.322+0000 7fc80910f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T09:48:34.979 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:34.979+0000 7f567e56d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:48:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:35.022+0000 7f9eabfff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:35.022+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T09:48:35.066 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.066+0000 7f567e56d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:48:35.132 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.129+0000 7fc80910f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T09:48:35.219 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.219+0000 7f567e56d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:48:35.220 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.219+0000 7fc80910f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T09:48:35.306 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.306+0000 7f567e56d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:48:35.382 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.381+0000 7fc80910f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T09:48:35.474 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.473+0000 7fc80910f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T09:48:35.523 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.523+0000 7f567e56d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:48:35.618 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.618+0000 7f567e56d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:48:35.694 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.693+0000 7fc80910f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T09:48:35.712 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.712+0000 7f567e56d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:48:35.793 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.792+0000 7fc80910f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T09:48:35.806 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.806+0000 7f567e56d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:48:35.892 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.891+0000 7fc80910f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T09:48:35.991 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:35.990+0000 7fc80910f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T09:48:35.992 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:35.992+0000 7f567e56d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:48:36.088 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:36.087+0000 7f567e56d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:48:36.187 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:36.186+0000 7fc80910f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T09:48:36.288 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:36.287+0000 7fc80910f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T09:48:36.429 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:36.428+0000 7f567e56d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:48:36.519 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:36.518+0000 7f567e56d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:36.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:36.624+0000 7f9eabfff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T09:48:36.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:36.624+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T09:48:36.636 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:36.635+0000 7fc80910f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T09:48:36.728 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:36.728+0000 7fc80910f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:37.055 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:37.054+0000 7f567e56d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:48:37.240 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:37.239+0000 7f567e56d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:48:37.291 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:37.291+0000 7fc80910f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T09:48:37.484 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:37.484+0000 7fc80910f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T09:48:37.613 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:37.612+0000 7f567e56d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:37.869 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:37.869+0000 7fc80910f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:38.032 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.031+0000 7f567e56d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:48:38.212 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.211+0000 7f567e56d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:38.311 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:38.310+0000 7fc80910f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T09:48:38.399 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.398+0000 7f567e56d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:48:38.501 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:38.501+0000 7fc80910f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T09:48:38.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.506+0000 7f567e56d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:38.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:38.517+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 59) v1 ==== 84468+0+0 (secure 0 0 0) 0x7f9ea0051c50 con 0x7f9eac0b0610 2023-10-07T09:48:38.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:38.518+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T09:48:38.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:38.518+0000 7f9ea8ff9700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:38.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55f2b77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55f2b77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55f2b77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55f2b77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55f2b77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55eeb6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55eeb6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55eeb6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55eeb6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:48:38.647+0000 7f55eeb6f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:48:38.709 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:38.709+0000 7fc80910f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T09:48:38.822 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T09:48:38.822+0000 7fc80910f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T09:48:39.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.523+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 60) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f9ea0052c30 con 0x7f9eac0b0610 2023-10-07T09:48:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.524+0000 7f9ea8ff9700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.524+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9eac068aa0 con 0x7f9e90066f80 2023-10-07T09:48:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.524+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.533+0000 7f9eabfff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e980099b0 tx=0x7f9e98017000 comp rx=0 tx=0).ready entity=mgr.7939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.593+0000 7f9ea8ff9700 1 -- 172.21.15.29:0/3632105129 <== mgr.7939 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9eac068aa0 con 0x7f9e90066f80 2023-10-07T09:48:39.594 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 msgr2=0x7f9e90069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e980099b0 tx=0x7f9e98017000 comp rx=0 tx=0).stop 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 msgr2=0x7f9eac1517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 0x7f9eac1517c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000e960 tx=0x7f9ea00091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 shutdown_connections 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9e90066f80 0x7f9e90069430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9eac0c8a20 0x7f9eac0c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0b0610 0x7f9eac1517c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 --2- 172.21.15.29:0/3632105129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9eac0b09e0 0x7f9eac0c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 >> 172.21.15.29:0/3632105129 conn(0x7f9eac063a10 msgr2=0x7f9eac0cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.596+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 shutdown_connections 2023-10-07T09:48:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.597+0000 7f9eb2bff700 1 -- 172.21.15.29:0/3632105129 wait complete. 2023-10-07T09:48:39.616 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-07T09:48:39.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs /snap_test_dir1 55n 2023-10-07T09:48:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.820+0000 7fc3ad3ff700 1 Processor -- start 2023-10-07T09:48:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.820+0000 7fc3ad3ff700 1 -- start start 2023-10-07T09:48:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3ad3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3a80cc570 0x7fc3a815a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3ad3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3a8156540 0x7fc3a815a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3ad3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a815adf0 con 0x7fc3a8156170 2023-10-07T09:48:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3ad3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80ad7c0 con 0x7fc3a8156540 2023-10-07T09:48:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3ad3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80b0000 con 0x7fc3a80cc570 2023-10-07T09:48:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3a6ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3a8156170 0x7fc3a80cbe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46916/0 (socket says 172.21.15.29:46916) 2023-10-07T09:48:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3a6ffd700 1 -- 172.21.15.29:0/3040195399 learned_addr learned my addr 172.21.15.29:0/3040195399 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3a77fe700 1 --2- 172.21.15.29:0/3040195399 >> v2:172.21.15.29:3300/0 conn(0x7fc3a8156540 0x7fc3a815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.822+0000 7fc3a67fc700 1 --2- 172.21.15.29:0/3040195399 >> v2:172.21.15.144:3300/0 conn(0x7fc3a80cc570 0x7fc3a815a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a77fe700 1 -- 172.21.15.29:0/3040195399 >> v2:172.21.15.144:3300/0 conn(0x7fc3a80cc570 msgr2=0x7fc3a815a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a77fe700 1 --2- 172.21.15.29:0/3040195399 >> v2:172.21.15.144:3300/0 conn(0x7fc3a80cc570 0x7fc3a815a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a77fe700 1 -- 172.21.15.29:0/3040195399 >> v1:172.21.15.29:6789/0 conn(0x7fc3a8156170 legacy=0x7fc3a80cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a77fe700 1 -- 172.21.15.29:0/3040195399 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a815ab40 con 0x7fc3a8156540 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a77fe700 1 --2- 172.21.15.29:0/3040195399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8156540 0x7fc3a815a600 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc3a806cb00 tx=0x7fc3a0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8813a1596141517 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a5ffb700 1 -- 172.21.15.29:0/3040195399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a000e070 con 0x7fc3a8156540 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a5ffb700 1 -- 172.21.15.29:0/3040195399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a000bb60 con 0x7fc3a8156540 2023-10-07T09:48:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.823+0000 7fc3a5ffb700 1 -- 172.21.15.29:0/3040195399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0012430 con 0x7fc3a8156540 2023-10-07T09:48:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3040195399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8156540 msgr2=0x7fc3a815a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3040195399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8156540 0x7fc3a815a600 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc3a806cb00 tx=0x7fc3a0009780 comp rx=0 tx=0).stop 2023-10-07T09:48:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3040195399 shutdown_connections 2023-10-07T09:48:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3040195399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8156540 0x7fc3a815a600 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3040195399 >> v2:172.21.15.144:3300/0 conn(0x7fc3a80cc570 0x7fc3a815a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3040195399 >> 172.21.15.29:0/3040195399 conn(0x7fc3a8067610 msgr2=0x7fc3a8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3040195399 shutdown_connections 2023-10-07T09:48:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.824+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3040195399 wait complete. 2023-10-07T09:48:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.826+0000 7fc3ad3ff700 1 Processor -- start 2023-10-07T09:48:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.826+0000 7fc3ad3ff700 1 -- start start 2023-10-07T09:48:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3ad3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 0x7fc3a814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3ad3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3a8156170 0x7fc3a8156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3ad3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a8156540 0x7fc3a8156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3ad3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81574e0 con 0x7fc3a80cc570 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3ad3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8159ec0 con 0x7fc3a8156540 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3ad3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80a9d70 con 0x7fc3a8156170 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 0x7fc3a814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 0x7fc3a814e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39866/0 (socket says 172.21.15.29:39866) 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 -- 172.21.15.29:0/3571636330 learned_addr learned my addr 172.21.15.29:0/3571636330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a67fc700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3a8156170 0x7fc3a8156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a77fe700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a8156540 0x7fc3a8156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 -- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3a8156170 msgr2=0x7fc3a8156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3a8156170 0x7fc3a8156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 -- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a8156540 msgr2=0x7fc3a8156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a8156540 0x7fc3a8156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a6ffd700 1 -- 172.21.15.29:0/3571636330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a0009a20 con 0x7fc3a80cc570 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.827+0000 7fc3a77fe700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a8156540 0x7fc3a8156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.828+0000 7fc3a6ffd700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 0x7fc3a814e6f0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc39800c7c0 tx=0x7fc398007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.828+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398005b30 con 0x7fc3a80cc570 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.828+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398005c90 con 0x7fc3a80cc570 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.828+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a8159900 con 0x7fc3a80cc570 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.828+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398004a80 con 0x7fc3a80cc570 2023-10-07T09:48:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.828+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a80a9a00 con 0x7fc3a80cc570 2023-10-07T09:48:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.829+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a80b0a70 con 0x7fc3a80cc570 2023-10-07T09:48:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.830+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 60) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fc3980070d0 con 0x7fc3a80cc570 2023-10-07T09:48:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.831+0000 7fc38effd700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc394066dd0 0x7fc394069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.831+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6339+0+0 (secure 0 0 0) 0x7fc398083a00 con 0x7fc3a80cc570 2023-10-07T09:48:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.831+0000 7fc3a67fc700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc394066dd0 0x7fc394069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:39.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.832+0000 7fc3a67fc700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc394066dd0 0x7fc394069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3a806fa10 tx=0x7fc39c0091a0 comp rx=0 tx=0).ready entity=mgr.7939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:39.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:39.836+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc398050bd0 con 0x7fc3a80cc570 2023-10-07T09:48:40.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.064+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3a80a9eb0 con 0x7fc394066dd0 2023-10-07T09:48:40.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.122+0000 7fc38effd700 1 -- 172.21.15.29:0/3571636330 <== mgr.7939 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7fc39c009e20 con 0x7fc394066dd0 2023-10-07T09:48:40.123 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path /snap_test_dir1 2023-10-07T09:48:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.124+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc394066dd0 msgr2=0x7fc394069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.124+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc394066dd0 0x7fc394069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3a806fa10 tx=0x7fc39c0091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.124+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 msgr2=0x7fc3a814e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.124+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 0x7fc3a814e6f0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc39800c7c0 tx=0x7fc398007f70 comp rx=0 tx=0).stop 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 shutdown_connections 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc394066dd0 0x7fc394069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a8156540 0x7fc3a8156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a80cc570 0x7fc3a814e6f0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 --2- 172.21.15.29:0/3571636330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3a8156170 0x7fc3a8156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 >> 172.21.15.29:0/3571636330 conn(0x7fc3a8067610 msgr2=0x7fc3a806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 shutdown_connections 2023-10-07T09:48:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.125+0000 7fc3ad3ff700 1 -- 172.21.15.29:0/3571636330 wait complete. 2023-10-07T09:48:40.142 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-07T09:48:40.142 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-10-07T09:48:40.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json /snap_test_dir1 2023-10-07T09:48:40.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.342+0000 7f4d27fff700 1 Processor -- start 2023-10-07T09:48:40.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.342+0000 7f4d27fff700 1 -- start start 2023-10-07T09:48:40.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.343+0000 7f4d27fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d200cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:40.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.343+0000 7f4d27fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d200cb6f0 0x7f4d200cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:40.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.343+0000 7f4d27fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d200cc940 con 0x7f4d200cf0f0 2023-10-07T09:48:40.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.343+0000 7f4d27fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200cf530 con 0x7f4d200cb6f0 2023-10-07T09:48:40.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.343+0000 7f4d27fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200b2df0 con 0x7f4d200b2a20 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.344+0000 7f4d25d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d200cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.344+0000 7f4d2559a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d200cb6f0 0x7f4d200cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.344+0000 7f4d25d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d200cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:38708/0 (socket says 172.21.15.29:38708) 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.344+0000 7f4d2559a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d200cb6f0 0x7f4d200cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:56736/0 (socket says 172.21.15.29:56736) 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.344+0000 7f4d25d9b700 1 -- 172.21.15.29:0/3296192824 learned_addr learned my addr 172.21.15.29:0/3296192824 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d25d9b700 1 -- 172.21.15.29:0/3296192824 >> v2:172.21.15.144:3300/0 conn(0x7f4d200cb6f0 msgr2=0x7f4d200cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d24d99700 1 -- 172.21.15.29:0/3296192824 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951378764 0 0) 0x7f4d200cc940 con 0x7f4d200cf0f0 2023-10-07T09:48:40.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d25d9b700 1 --2- 172.21.15.29:0/3296192824 >> v2:172.21.15.144:3300/0 conn(0x7f4d200cb6f0 0x7f4d200cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d25d9b700 1 -- 172.21.15.29:0/3296192824 >> v1:172.21.15.144:6789/0 conn(0x7f4d200cf0f0 legacy=0x7f4d20159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d25d9b700 1 -- 172.21.15.29:0/3296192824 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200cbd30 con 0x7f4d200b2a20 2023-10-07T09:48:40.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d25d9b700 1 --2- 172.21.15.29:0/3296192824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d200cb1b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009990 tx=0x7f4d1c009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcd954384cef1754 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d24d99700 1 -- 172.21.15.29:0/3296192824 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c009e80 con 0x7f4d200b2a20 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d24d99700 1 -- 172.21.15.29:0/3296192824 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c005c10 con 0x7f4d200b2a20 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.345+0000 7f4d24d99700 1 -- 172.21.15.29:0/3296192824 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00b650 con 0x7f4d200b2a20 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 -- 172.21.15.29:0/3296192824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 msgr2=0x7f4d200cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 --2- 172.21.15.29:0/3296192824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d200cb1b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009990 tx=0x7f4d1c009220 comp rx=0 tx=0).stop 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 -- 172.21.15.29:0/3296192824 shutdown_connections 2023-10-07T09:48:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 --2- 172.21.15.29:0/3296192824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d200cb1b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 --2- 172.21.15.29:0/3296192824 >> v2:172.21.15.144:3300/0 conn(0x7f4d200cb6f0 0x7f4d200cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 -- 172.21.15.29:0/3296192824 >> 172.21.15.29:0/3296192824 conn(0x7f4d20099820 msgr2=0x7f4d20063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:40.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 -- 172.21.15.29:0/3296192824 shutdown_connections 2023-10-07T09:48:40.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.346+0000 7f4d27fff700 1 -- 172.21.15.29:0/3296192824 wait complete. 2023-10-07T09:48:40.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.347+0000 7f4d27fff700 1 Processor -- start 2023-10-07T09:48:40.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 -- start start 2023-10-07T09:48:40.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d20152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 0x7f4d20152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d200cf0f0 0x7f4d200c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20150be0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d201531f0 con 0x7f4d200cf0f0 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.348+0000 7f4d27fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20152560 con 0x7f4d200b2a20 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 0x7f4d20152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 0x7f4d20152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39876/0 (socket says 172.21.15.29:39876) 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 -- 172.21.15.29:0/2709790480 learned_addr learned my addr 172.21.15.29:0/2709790480 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d25d9b700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d20152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 -- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 msgr2=0x7f4d20152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d20152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 -- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d200cf0f0 msgr2=0x7f4d200c6410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-07T09:48:40.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d200cf0f0 0x7f4d200c6410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 -- 172.21.15.29:0/2709790480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c009480 con 0x7f4d200cb6f0 2023-10-07T09:48:40.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.349+0000 7f4d2559a700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 0x7f4d20152020 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000e960 tx=0x7f4d100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.350+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10013680 con 0x7f4d200cb6f0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.350+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d100137e0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.350+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d20169ef0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.350+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10013b80 con 0x7f4d200cb6f0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.350+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2016a4b0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.351+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200718b0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.352+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 60) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f4d1000c490 con 0x7f4d200cb6f0 2023-10-07T09:48:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.353+0000 7f4d15ffb700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4d04066e00 0x7f4d040692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T09:48:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.353+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6339+0+0 (secure 0 0 0) 0x7f4d10084700 con 0x7f4d200cb6f0 2023-10-07T09:48:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.353+0000 7f4d25d9b700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4d04066e00 0x7f4d040692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T09:48:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.354+0000 7f4d25d9b700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4d04066e00 0x7f4d040692b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009260 tx=0x7f4d1c016000 comp rx=0 tx=0).ready entity=mgr.7939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T09:48:40.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.357+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d100518d0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.570+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 61) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f4d10051ca0 con 0x7f4d200cb6f0 2023-10-07T09:48:40.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.588+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d200cce70 con 0x7f4d04066e00 2023-10-07T09:48:40.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.596+0000 7f4d15ffb700 1 -- 172.21.15.29:0/2709790480 <== mgr.7939 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f4d200cce70 con 0x7f4d04066e00 2023-10-07T09:48:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T09:48:40.597 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "/snap_test_dir1", "schedule": ["1M"], "retention": [{"n": 55}]} 2023-10-07T09:48:40.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4d04066e00 msgr2=0x7f4d040692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4d04066e00 0x7f4d040692b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009260 tx=0x7f4d1c016000 comp rx=0 tx=0).stop 2023-10-07T09:48:40.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 msgr2=0x7f4d20152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 0x7f4d20152020 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000e960 tx=0x7f4d100091a0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 shutdown_connections 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4d04066e00 0x7f4d040692b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d200cf0f0 0x7f4d200c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d200b2a20 0x7f4d20152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 --2- 172.21.15.29:0/2709790480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d200cb6f0 0x7f4d20152020 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 >> 172.21.15.29:0/2709790480 conn(0x7f4d20099820 msgr2=0x7f4d200c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 shutdown_connections 2023-10-07T09:48:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T09:48:40.599+0000 7f4d27fff700 1 -- 172.21.15.29:0/2709790480 wait complete. 2023-10-07T09:48:40.617 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1M"], "retention": [{"n": 55}]} 2023-10-07T09:48:40.617 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1M'], 'retention': [{'n': 55}]} 2023-10-07T09:48:52.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:48:52.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:49:00.134 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:49:00.133+0000 7f55e1b55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:49:00.135 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:49:00.133+0000 7f55e1b55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:49:00.135 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:49:00.133+0000 7f55e1b55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:49:00.135 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:49:00.133+0000 7f55e1b55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:49:00.135 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T09:49:00.133+0000 7f55e1b55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T09:49:22.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:49:22.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:49:52.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:49:52.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:50:23.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:50:23.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:50:53.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:50:53.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:51:23.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:51:23.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:51:53.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:51:53.382 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:52:23.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:52:23.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:52:53.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:52:53.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:53:23.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:53:23.750 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:53:53.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:53:53.875 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:54:23.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:54:24.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:54:54.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:54:54.116 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:55:24.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:55:24.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:55:54.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:55:54.375 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:56:24.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:56:24.506 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:56:54.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:56:54.633 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:57:24.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:57:24.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:57:54.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:57:54.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:58:25.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:58:25.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:58:55.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:58:55.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:59:25.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:59:25.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:59:55.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T09:59:55.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:00:25.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:00:25.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:00:55.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:00:55.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:01:25.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:01:25.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:01:55.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:01:55.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:02:26.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:02:26.017 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:02:56.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:02:56.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:03:26.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:03:26.280 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:03:56.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:03:56.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:04:26.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:04:26.577 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:04:56.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:04:56.703 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:05:26.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:05:27.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:05:57.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:05:57.155 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:06:27.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:06:27.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:06:57.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:06:57.414 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:07:27.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:07:27.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:07:57.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:07:57.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:08:27.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:08:27.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:08:57.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:08:57.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:09:28.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:09:28.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:09:58.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:09:58.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:10:28.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:10:28.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:10:58.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:10:58.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:11:28.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:11:28.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:11:58.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:11:58.679 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:12:28.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:12:28.807 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:12:58.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:12:58.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:13:29.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:13:29.040 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:13:59.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:13:59.165 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:14:29.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:14:29.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:14:59.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:14:59.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:15:29.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:15:29.599 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:15:59.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:15:59.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:16:29.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:16:29.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:16:59.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:16:59.935 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:17:30.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:17:30.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:18:00.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:18:00.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:18:30.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:18:30.394 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:19:00.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:19:00.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:19:30.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:19:30.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:20:00.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:20:00.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:20:30.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:20:30.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:21:01.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:21:01.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:21:31.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:21:31.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:22:01.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:22:01.284 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:22:31.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:22:31.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:23:01.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:23:01.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:23:31.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:23:31.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:24:01.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:24:01.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:24:31.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:24:31.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:25:02.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:25:02.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:25:32.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:25:32.227 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:26:02.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:26:02.348 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:26:32.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:26:32.469 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:27:02.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:27:02.593 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:27:32.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:27:32.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:28:02.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:28:02.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:28:33.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:28:33.061 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:29:03.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:29:03.194 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:29:33.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:29:33.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:30:03.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:30:03.454 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:30:33.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:30:33.585 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:31:03.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:31:03.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:31:33.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:31:33.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:32:03.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:32:03.942 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:32:34.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:32:34.061 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:33:04.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:33:04.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:33:34.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:33:34.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:34:04.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:34:04.443 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:34:34.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:34:34.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:35:04.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:35:04.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:35:34.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:35:34.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:36:05.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:36:05.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:36:35.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:36:35.149 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:37:05.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:37:05.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:37:35.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:37:35.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:38:05.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:38:05.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:38:35.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:38:35.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:39:05.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:39:05.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:39:35.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:39:36.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:40:06.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:40:06.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:40:36.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:40:36.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:41:06.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:41:06.454 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:41:36.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:41:36.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:42:06.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:42:06.713 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:42:36.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:42:36.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:43:07.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:43:07.034 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:43:37.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:43:37.163 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:44:07.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:44:07.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:44:15.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:15.618 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_50_00_UTC 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_51_00_UTC 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_52_00_UTC 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_53_00_UTC 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_54_00_UTC 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_55_00_UTC 2023-10-07T10:44:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_56_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_57_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_58_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_59_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_00_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_01_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_02_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_03_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_04_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_05_00_UTC 2023-10-07T10:44:15.677 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_06_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_07_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_08_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_09_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_10_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_11_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_12_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_13_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_14_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_15_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_16_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_17_00_UTC 2023-10-07T10:44:15.678 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_18_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_19_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_20_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_21_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_22_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_23_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_24_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_25_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_26_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_27_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_28_00_UTC 2023-10-07T10:44:15.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_29_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_30_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_31_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_32_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_33_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_34_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_35_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_36_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_37_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_38_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_39_00_UTC 2023-10-07T10:44:15.680 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_40_00_UTC 2023-10-07T10:44:15.681 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_41_00_UTC 2023-10-07T10:44:15.681 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_42_00_UTC 2023-10-07T10:44:15.681 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_43_00_UTC 2023-10-07T10:44:15.681 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_44_00_UTC 2023-10-07T10:44:15.682 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-09_50_00_UTC', 'scheduled-2023-10-07-09_51_00_UTC', 'scheduled-2023-10-07-09_52_00_UTC', 'scheduled-2023-10-07-09_53_00_UTC', 'scheduled-2023-10-07-09_54_00_UTC', 'scheduled-2023-10-07-09_55_00_UTC', 'scheduled-2023-10-07-09_56_00_UTC', 'scheduled-2023-10-07-09_57_00_UTC', 'scheduled-2023-10-07-09_58_00_UTC', 'scheduled-2023-10-07-09_59_00_UTC', 'scheduled-2023-10-07-10_00_00_UTC', 'scheduled-2023-10-07-10_01_00_UTC', 'scheduled-2023-10-07-10_02_00_UTC', 'scheduled-2023-10-07-10_03_00_UTC', 'scheduled-2023-10-07-10_04_00_UTC', 'scheduled-2023-10-07-10_05_00_UTC', 'scheduled-2023-10-07-10_06_00_UTC', 'scheduled-2023-10-07-10_07_00_UTC', 'scheduled-2023-10-07-10_08_00_UTC', 'scheduled-2023-10-07-10_09_00_UTC', 'scheduled-2023-10-07-10_10_00_UTC', 'scheduled-2023-10-07-10_11_00_UTC', 'scheduled-2023-10-07-10_12_00_UTC', 'scheduled-2023-10-07-10_13_00_UTC', 'scheduled-2023-10-07-10_14_00_UTC', 'scheduled-2023-10-07-10_15_00_UTC', 'scheduled-2023-10-07-10_16_00_UTC', 'scheduled-2023-10-07-10_17_00_UTC', 'scheduled-2023-10-07-10_18_00_UTC', 'scheduled-2023-10-07-10_19_00_UTC', 'scheduled-2023-10-07-10_20_00_UTC', 'scheduled-2023-10-07-10_21_00_UTC', 'scheduled-2023-10-07-10_22_00_UTC', 'scheduled-2023-10-07-10_23_00_UTC', 'scheduled-2023-10-07-10_24_00_UTC', 'scheduled-2023-10-07-10_25_00_UTC', 'scheduled-2023-10-07-10_26_00_UTC', 'scheduled-2023-10-07-10_27_00_UTC', 'scheduled-2023-10-07-10_28_00_UTC', 'scheduled-2023-10-07-10_29_00_UTC', 'scheduled-2023-10-07-10_30_00_UTC', 'scheduled-2023-10-07-10_31_00_UTC', 'scheduled-2023-10-07-10_32_00_UTC', 'scheduled-2023-10-07-10_33_00_UTC', 'scheduled-2023-10-07-10_34_00_UTC', 'scheduled-2023-10-07-10_35_00_UTC', 'scheduled-2023-10-07-10_36_00_UTC', 'scheduled-2023-10-07-10_37_00_UTC', 'scheduled-2023-10-07-10_38_00_UTC', 'scheduled-2023-10-07-10_39_00_UTC', 'scheduled-2023-10-07-10_40_00_UTC', 'scheduled-2023-10-07-10_41_00_UTC', 'scheduled-2023-10-07-10_42_00_UTC', 'scheduled-2023-10-07-10_43_00_UTC', 'scheduled-2023-10-07-10_44_00_UTC'] 2023-10-07T10:44:15.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1 2023-10-07T10:44:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.885+0000 7fc1f9dff700 1 Processor -- start 2023-10-07T10:44:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.885+0000 7fc1f9dff700 1 -- start start 2023-10-07T10:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.886+0000 7fc1f9dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1f40cc810 0x7fc1f40accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.886+0000 7fc1f9dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f40ad400 con 0x7fc1f40afeb0 2023-10-07T10:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.886+0000 7fc1f9dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f40ad6e0 con 0x7fc1f415a590 2023-10-07T10:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.886+0000 7fc1f9dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f415a140 con 0x7fc1f40cc810 2023-10-07T10:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.886+0000 7fc1f2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1f40cc810 0x7fc1f40accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1f40cc810 0x7fc1f40accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33516/0 (socket says 172.21.15.29:33516) 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f2ffd700 1 -- 172.21.15.29:0/3778650398 learned_addr learned my addr 172.21.15.29:0/3778650398 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f37fe700 1 --1- 172.21.15.29:0/3778650398 >> v1:172.21.15.29:6789/0 conn(0x7fc1f40afeb0 0x7fc1f40aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60190/0 (socket says 172.21.15.29:60190) 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f2ffd700 1 -- 172.21.15.29:0/3778650398 >> v1:172.21.15.144:6789/0 conn(0x7fc1f415a590 legacy=0x7fc1f40b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f2ffd700 1 -- 172.21.15.29:0/3778650398 >> v1:172.21.15.29:6789/0 conn(0x7fc1f40afeb0 legacy=0x7fc1f40aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f2ffd700 1 -- 172.21.15.29:0/3778650398 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f40ad9c0 con 0x7fc1f40cc810 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f27fc700 1 -- 172.21.15.29:0/3778650398 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397141044 0 0) 0x7fc1f40ad400 con 0x7fc1f40afeb0 2023-10-07T10:44:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.887+0000 7fc1f2ffd700 1 --2- 172.21.15.29:0/3778650398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f40cc810 0x7fc1f40accf0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80099b0 tx=0x7fc1e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865baf29c12533db server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f27fc700 1 -- 172.21.15.29:0/3778650398 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800b820 con 0x7fc1f40cc810 2023-10-07T10:44:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f27fc700 1 -- 172.21.15.29:0/3778650398 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e800b980 con 0x7fc1f40cc810 2023-10-07T10:44:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f27fc700 1 -- 172.21.15.29:0/3778650398 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800bd20 con 0x7fc1f40cc810 2023-10-07T10:44:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 -- 172.21.15.29:0/3778650398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f40cc810 msgr2=0x7fc1f40accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/3778650398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f40cc810 0x7fc1f40accf0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80099b0 tx=0x7fc1e8009250 comp rx=0 tx=0).stop 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 -- 172.21.15.29:0/3778650398 shutdown_connections 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/3778650398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f40cc810 0x7fc1f40accf0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 -- 172.21.15.29:0/3778650398 >> 172.21.15.29:0/3778650398 conn(0x7fc1f4067610 msgr2=0x7fc1f4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 -- 172.21.15.29:0/3778650398 shutdown_connections 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.888+0000 7fc1f9dff700 1 -- 172.21.15.29:0/3778650398 wait complete. 2023-10-07T10:44:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.890+0000 7fc1f9dff700 1 Processor -- start 2023-10-07T10:44:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.890+0000 7fc1f9dff700 1 -- start start 2023-10-07T10:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f40afeb0 0x7fc1f41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f9dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f40cc810 0x7fc1f4156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f9dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 0x7fc1f4154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f9dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f4152f00 con 0x7fc1f415a590 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f9dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f4158440 con 0x7fc1f40cc810 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f4157830 con 0x7fc1f40afeb0 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 0x7fc1f4154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 0x7fc1f4154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33528/0 (socket says 172.21.15.29:33528) 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.891+0000 7fc1f3fff700 1 -- 172.21.15.29:0/855424257 learned_addr learned my addr 172.21.15.29:0/855424257 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f37fe700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f40afeb0 0x7fc1f41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f2ffd700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f40cc810 0x7fc1f4156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f3fff700 1 -- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f40afeb0 msgr2=0x7fc1f41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f3fff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f40afeb0 0x7fc1f41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f3fff700 1 -- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f40cc810 msgr2=0x7fc1f4156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f3fff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f40cc810 0x7fc1f4156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f3fff700 1 -- 172.21.15.29:0/855424257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e80094f0 con 0x7fc1f415a590 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f3fff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 0x7fc1f4154560 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00bc30 tx=0x7fc1ec009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec00c9b0 con 0x7fc1f415a590 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec00cb30 con 0x7fc1f415a590 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f4157b10 con 0x7fc1f415a590 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.892+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec014410 con 0x7fc1f415a590 2023-10-07T10:44:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.893+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f41580d0 con 0x7fc1f415a590 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.893+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f41585d0 con 0x7fc1f415a590 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.894+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 84871+0+0 (secure 0 0 0) 0x7fc1ec014840 con 0x7fc1f415a590 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.895+0000 7fc1db7fe700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1e00670a0 0x7fc1e0069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.895+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6441+0+0 (secure 0 0 0) 0x7fc1ec084ab0 con 0x7fc1f415a590 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.896+0000 7fc1f37fe700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1e00670a0 0x7fc1e0069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.896+0000 7fc1f37fe700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1e00670a0 0x7fc1e0069550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40011e0 tx=0x7fc1e4007bf0 comp rx=0 tx=0).ready entity=mgr.7939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:15.900+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1ec04c5a0 con 0x7fc1f415a590 2023-10-07T10:44:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.126+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1f41587d0 con 0x7fc1e00670a0 2023-10-07T10:44:16.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.136+0000 7fc1db7fe700 1 -- 172.21.15.29:0/855424257 <== mgr.7939 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fc1f41587d0 con 0x7fc1e00670a0 2023-10-07T10:44:16.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:16.137 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1M", "retention": {"n": 55}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:48:39", "first": "2023-10-07T09:49:00", "last": "2023-10-07T10:44:00", "last_pruned": "2023-10-07T10:44:01", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-07T10:44:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1e00670a0 msgr2=0x7fc1e0069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1e00670a0 0x7fc1e0069550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40011e0 tx=0x7fc1e4007bf0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 msgr2=0x7fc1f4154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 0x7fc1f4154560 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00bc30 tx=0x7fc1ec009820 comp rx=0 tx=0).stop 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 shutdown_connections 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc1e00670a0 0x7fc1e0069550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1f415a590 0x7fc1f4154560 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f40afeb0 0x7fc1f41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 --2- 172.21.15.29:0/855424257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f40cc810 0x7fc1f4156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 >> 172.21.15.29:0/855424257 conn(0x7fc1f4067610 msgr2=0x7fc1f4155c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 shutdown_connections 2023-10-07T10:44:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.138+0000 7fc1f9dff700 1 -- 172.21.15.29:0/855424257 wait complete. 2023-10-07T10:44:16.156 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1M", "retention": {"n": 55}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T09:48:39", "first": "2023-10-07T09:49:00", "last": "2023-10-07T10:44:00", "last_pruned": "2023-10-07T10:44:01", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-07T10:44:16.157 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1M', 'retention': {'n': 55}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T09:48:39', 'first': '2023-10-07T09:49:00', 'last': '2023-10-07T10:44:00', 'last_pruned': '2023-10-07T10:44:01', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-07T10:44:16.157 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-07T10:44:16.157 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-07T10:44:16.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1 2023-10-07T10:44:16.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.355+0000 7f87e7bff700 1 Processor -- start 2023-10-07T10:44:16.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.355+0000 7f87e7bff700 1 -- start start 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.356+0000 7f87e7bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f87e00b09e0 0x7f87e00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.356+0000 7f87e7bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e00690f0 con 0x7f87e00ced00 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e7bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e00cf2e0 con 0x7f87e00b0610 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e7bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e00687c0 con 0x7f87e00b09e0 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e519a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f87e00ced00 0x7f87e00c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60192/0 (socket says 172.21.15.29:60192) 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e519a700 1 -- 172.21.15.29:0/2617710436 learned_addr learned my addr 172.21.15.29:0/2617710436 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e619c700 1 --2- 172.21.15.29:0/2617710436 >> v2:172.21.15.144:3300/0 conn(0x7f87e00b09e0 0x7f87e00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161311468 0 0) 0x7f87e00690f0 con 0x7f87e00ced00 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc003570 con 0x7f87e00ced00 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169025152 0 0) 0x7f87e00cf2e0 con 0x7f87e00b0610 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.357+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e00690f0 con 0x7f87e00b0610 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e619c700 1 -- 172.21.15.29:0/2617710436 >> v1:172.21.15.144:6789/0 conn(0x7f87e00b0610 legacy=0x7f87e00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e619c700 1 -- 172.21.15.29:0/2617710436 >> v1:172.21.15.29:6789/0 conn(0x7f87e00ced00 legacy=0x7f87e00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e619c700 1 -- 172.21.15.29:0/2617710436 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0068aa0 con 0x7f87e00b09e0 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e619c700 1 --2- 172.21.15.29:0/2617710436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e00cc7e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009e60 tx=0x7f87d8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a1e835d0a0ce91a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d800e070 con 0x7f87e00b09e0 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d800bab0 con 0x7f87e00b09e0 2023-10-07T10:44:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.358+0000 7f87e4999700 1 -- 172.21.15.29:0/2617710436 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d800be70 con 0x7f87e00b09e0 2023-10-07T10:44:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 -- 172.21.15.29:0/2617710436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 msgr2=0x7f87e00cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 --2- 172.21.15.29:0/2617710436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e00cc7e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009e60 tx=0x7f87d8009780 comp rx=0 tx=0).stop 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 -- 172.21.15.29:0/2617710436 shutdown_connections 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 --2- 172.21.15.29:0/2617710436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e00cc7e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 -- 172.21.15.29:0/2617710436 >> 172.21.15.29:0/2617710436 conn(0x7f87e0063a10 msgr2=0x7f87e00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 -- 172.21.15.29:0/2617710436 shutdown_connections 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.359+0000 7f87e7bff700 1 -- 172.21.15.29:0/2617710436 wait complete. 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.361+0000 7f87e7bff700 1 Processor -- start 2023-10-07T10:44:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.361+0000 7f87e7bff700 1 -- start start 2023-10-07T10:44:16.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.362+0000 7f87e7bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87e00b0610 0x7f87e00af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.362+0000 7f87e7bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e0169860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.362+0000 7f87e7bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 0x7f87e00c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.362+0000 7f87e7bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e016a1b0 con 0x7f87e00ced00 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.362+0000 7f87e7bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e016a490 con 0x7f87e00b09e0 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.362+0000 7f87e7bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00c6500 con 0x7f87e00b0610 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 0x7f87e00c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 0x7f87e00c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33532/0 (socket says 172.21.15.29:33532) 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 -- 172.21.15.29:0/681602190 learned_addr learned my addr 172.21.15.29:0/681602190 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e519a700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e0169860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e599b700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87e00b0610 0x7f87e00af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 -- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87e00b0610 msgr2=0x7f87e00af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87e00b0610 0x7f87e00af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 -- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 msgr2=0x7f87e0169860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e0169860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 -- 172.21.15.29:0/681602190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d8009a20 con 0x7f87e00ced00 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e519a700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e0169860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e599b700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87e00b0610 0x7f87e00af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.363+0000 7f87e619c700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 0x7f87e00c6ce0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f87e006ea20 tx=0x7f87d8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.364+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8026480 con 0x7f87e00ced00 2023-10-07T10:44:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.364+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d80265e0 con 0x7f87e00ced00 2023-10-07T10:44:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.364+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e00c67e0 con 0x7f87e00ced00 2023-10-07T10:44:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.364+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d80269a0 con 0x7f87e00ced00 2023-10-07T10:44:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.364+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e0169da0 con 0x7f87e00ced00 2023-10-07T10:44:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.365+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e00c6b50 con 0x7f87e00ced00 2023-10-07T10:44:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.366+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f87d801c400 con 0x7f87e00ced00 2023-10-07T10:44:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.366+0000 7f87d5ffb700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f87c40670a0 0x7f87c4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.367+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f87d8086e10 con 0x7f87e00ced00 2023-10-07T10:44:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.367+0000 7f87e599b700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f87c40670a0 0x7f87c4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:16.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.368+0000 7f87e599b700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f87c40670a0 0x7f87c4069550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0038f0 tx=0x7f87dc009040 comp rx=0 tx=0).ready entity=mgr.7939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.371+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87d808a030 con 0x7f87e00ced00 2023-10-07T10:44:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.598+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f87e016a600 con 0x7f87c40670a0 2023-10-07T10:44:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.672+0000 7f87d5ffb700 1 -- 172.21.15.29:0/681602190 <== mgr.7939 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f87e016a600 con 0x7f87c40670a0 2023-10-07T10:44:16.673 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1 2023-10-07T10:44:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f87c40670a0 msgr2=0x7f87c4069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f87c40670a0 0x7f87c4069550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0038f0 tx=0x7f87dc009040 comp rx=0 tx=0).stop 2023-10-07T10:44:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 msgr2=0x7f87e00c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 0x7f87e00c6ce0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f87e006ea20 tx=0x7f87d8019000 comp rx=0 tx=0).stop 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 shutdown_connections 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f87c40670a0 0x7f87c4069550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e00ced00 0x7f87e00c6ce0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87e00b0610 0x7f87e00af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 --2- 172.21.15.29:0/681602190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e00b09e0 0x7f87e0169860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 >> 172.21.15.29:0/681602190 conn(0x7f87e0063a10 msgr2=0x7f87e00ca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 shutdown_connections 2023-10-07T10:44:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:16.674+0000 7f87e7bff700 1 -- 172.21.15.29:0/681602190 wait complete. 2023-10-07T10:44:16.692 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-07T10:44:16.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:16.693 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_50_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_51_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_52_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_53_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_54_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_55_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_56_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_57_00_UTC 2023-10-07T10:44:16.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_58_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-09_59_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_00_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_01_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_02_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_03_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_04_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_05_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_06_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_07_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_08_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_09_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_10_00_UTC 2023-10-07T10:44:16.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_11_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_12_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_13_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_14_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_15_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_16_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_17_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_18_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_19_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_20_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_21_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_22_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_23_00_UTC 2023-10-07T10:44:16.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_24_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_25_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_26_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_27_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_28_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_29_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_30_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_31_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_32_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_33_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_34_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_35_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_36_00_UTC 2023-10-07T10:44:16.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_37_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_38_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_39_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_40_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_41_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_42_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_43_00_UTC 2023-10-07T10:44:16.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_44_00_UTC 2023-10-07T10:44:16.747 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_50_00_UTC 2023-10-07T10:44:16.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:16.747 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_50_00_UTC) 2023-10-07T10:44:16.800 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_51_00_UTC 2023-10-07T10:44:16.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:16.800 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_51_00_UTC) 2023-10-07T10:44:16.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_52_00_UTC 2023-10-07T10:44:16.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:16.853 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_52_00_UTC) 2023-10-07T10:44:16.906 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_53_00_UTC 2023-10-07T10:44:16.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:16.906 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_53_00_UTC) 2023-10-07T10:44:16.959 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_54_00_UTC 2023-10-07T10:44:16.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:16.959 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_54_00_UTC) 2023-10-07T10:44:17.011 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_55_00_UTC 2023-10-07T10:44:17.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.012 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_55_00_UTC) 2023-10-07T10:44:17.065 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_56_00_UTC 2023-10-07T10:44:17.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.066 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_56_00_UTC) 2023-10-07T10:44:17.118 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_57_00_UTC 2023-10-07T10:44:17.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.118 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_57_00_UTC) 2023-10-07T10:44:17.171 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_58_00_UTC 2023-10-07T10:44:17.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.171 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_58_00_UTC) 2023-10-07T10:44:17.226 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-09_59_00_UTC 2023-10-07T10:44:17.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.227 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-09_59_00_UTC) 2023-10-07T10:44:17.280 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_00_00_UTC 2023-10-07T10:44:17.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.280 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_00_00_UTC) 2023-10-07T10:44:17.332 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_01_00_UTC 2023-10-07T10:44:17.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.332 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_01_00_UTC) 2023-10-07T10:44:17.385 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_02_00_UTC 2023-10-07T10:44:17.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.386 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_02_00_UTC) 2023-10-07T10:44:17.438 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_03_00_UTC 2023-10-07T10:44:17.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.438 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_03_00_UTC) 2023-10-07T10:44:17.491 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_04_00_UTC 2023-10-07T10:44:17.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.491 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_04_00_UTC) 2023-10-07T10:44:17.544 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_05_00_UTC 2023-10-07T10:44:17.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.545 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_05_00_UTC) 2023-10-07T10:44:17.598 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_06_00_UTC 2023-10-07T10:44:17.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.598 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_06_00_UTC) 2023-10-07T10:44:17.651 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_07_00_UTC 2023-10-07T10:44:17.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.651 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_07_00_UTC) 2023-10-07T10:44:17.703 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_08_00_UTC 2023-10-07T10:44:17.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.704 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_08_00_UTC) 2023-10-07T10:44:17.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_09_00_UTC 2023-10-07T10:44:17.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.757 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_09_00_UTC) 2023-10-07T10:44:17.810 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_10_00_UTC 2023-10-07T10:44:17.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.810 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_10_00_UTC) 2023-10-07T10:44:17.862 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_11_00_UTC 2023-10-07T10:44:17.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.863 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_11_00_UTC) 2023-10-07T10:44:17.915 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_12_00_UTC 2023-10-07T10:44:17.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.915 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_12_00_UTC) 2023-10-07T10:44:17.968 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_13_00_UTC 2023-10-07T10:44:17.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:17.968 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_13_00_UTC) 2023-10-07T10:44:18.021 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_14_00_UTC 2023-10-07T10:44:18.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.021 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_14_00_UTC) 2023-10-07T10:44:18.074 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_15_00_UTC 2023-10-07T10:44:18.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.074 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_15_00_UTC) 2023-10-07T10:44:18.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_16_00_UTC 2023-10-07T10:44:18.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.127 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_16_00_UTC) 2023-10-07T10:44:18.179 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_17_00_UTC 2023-10-07T10:44:18.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.180 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_17_00_UTC) 2023-10-07T10:44:18.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_18_00_UTC 2023-10-07T10:44:18.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.232 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_18_00_UTC) 2023-10-07T10:44:18.284 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_19_00_UTC 2023-10-07T10:44:18.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.285 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_19_00_UTC) 2023-10-07T10:44:18.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_20_00_UTC 2023-10-07T10:44:18.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.338 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_20_00_UTC) 2023-10-07T10:44:18.390 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_21_00_UTC 2023-10-07T10:44:18.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.391 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_21_00_UTC) 2023-10-07T10:44:18.444 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_22_00_UTC 2023-10-07T10:44:18.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.445 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_22_00_UTC) 2023-10-07T10:44:18.497 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_23_00_UTC 2023-10-07T10:44:18.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.497 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_23_00_UTC) 2023-10-07T10:44:18.550 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_24_00_UTC 2023-10-07T10:44:18.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.550 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_24_00_UTC) 2023-10-07T10:44:18.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_25_00_UTC 2023-10-07T10:44:18.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.603 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_25_00_UTC) 2023-10-07T10:44:18.655 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_26_00_UTC 2023-10-07T10:44:18.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.655 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_26_00_UTC) 2023-10-07T10:44:18.708 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_27_00_UTC 2023-10-07T10:44:18.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.709 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_27_00_UTC) 2023-10-07T10:44:18.761 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_28_00_UTC 2023-10-07T10:44:18.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.761 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_28_00_UTC) 2023-10-07T10:44:18.814 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_29_00_UTC 2023-10-07T10:44:18.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.814 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_29_00_UTC) 2023-10-07T10:44:18.867 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_30_00_UTC 2023-10-07T10:44:18.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.867 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_30_00_UTC) 2023-10-07T10:44:18.919 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_31_00_UTC 2023-10-07T10:44:18.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.919 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_31_00_UTC) 2023-10-07T10:44:18.973 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_32_00_UTC 2023-10-07T10:44:18.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:18.973 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_32_00_UTC) 2023-10-07T10:44:19.026 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_33_00_UTC 2023-10-07T10:44:19.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.026 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_33_00_UTC) 2023-10-07T10:44:19.079 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_34_00_UTC 2023-10-07T10:44:19.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.079 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_34_00_UTC) 2023-10-07T10:44:19.132 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_35_00_UTC 2023-10-07T10:44:19.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.133 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_35_00_UTC) 2023-10-07T10:44:19.185 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_36_00_UTC 2023-10-07T10:44:19.186 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.186 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_36_00_UTC) 2023-10-07T10:44:19.239 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_37_00_UTC 2023-10-07T10:44:19.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.240 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_37_00_UTC) 2023-10-07T10:44:19.293 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_38_00_UTC 2023-10-07T10:44:19.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.293 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_38_00_UTC) 2023-10-07T10:44:19.345 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_39_00_UTC 2023-10-07T10:44:19.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.346 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_39_00_UTC) 2023-10-07T10:44:19.398 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_40_00_UTC 2023-10-07T10:44:19.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.399 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_40_00_UTC) 2023-10-07T10:44:19.453 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_41_00_UTC 2023-10-07T10:44:19.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.453 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_41_00_UTC) 2023-10-07T10:44:19.506 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_42_00_UTC 2023-10-07T10:44:19.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.507 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_42_00_UTC) 2023-10-07T10:44:19.560 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_43_00_UTC 2023-10-07T10:44:19.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.560 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_43_00_UTC) 2023-10-07T10:44:19.614 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-10_44_00_UTC 2023-10-07T10:44:19.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.614 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-10_44_00_UTC) 2023-10-07T10:44:19.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:19.667 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T10:44:19.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T10:44:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.914+0000 7f170afff700 1 Processor -- start 2023-10-07T10:44:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.914+0000 7f170afff700 1 -- start start 2023-10-07T10:44:19.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 0x7f17040afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170afff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 0x7f17040b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170afff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040ad4d0 con 0x7f17040ae570 2023-10-07T10:44:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170afff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17040b0ee0 con 0x7f17040b0520 2023-10-07T10:44:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17040ad7b0 con 0x7f1704156180 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170959c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 0x7f17040b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f1708d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 0x7f17040afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170959c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 0x7f17040b09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:37250/0 (socket says 172.21.15.29:37250) 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f170959c700 1 -- 172.21.15.29:0/2888966256 learned_addr learned my addr 172.21.15.29:0/2888966256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f1703fff700 1 --1- 172.21.15.29:0/2888966256 >> v1:172.21.15.144:6789/0 conn(0x7f17040ae570 0x7f17040accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:44730/0 (socket says 172.21.15.29:44730) 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.916+0000 7f1708d9b700 1 --2- 172.21.15.29:0/2888966256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 0x7f17040afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:36388/0 (socket says 172.21.15.29:36388) 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f17037fe700 1 -- 172.21.15.29:0/2888966256 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619895573 0 0) 0x7f17040ad4d0 con 0x7f17040ae570 2023-10-07T10:44:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f17037fe700 1 -- 172.21.15.29:0/2888966256 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f4003570 con 0x7f17040ae570 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f1708d9b700 1 -- 172.21.15.29:0/2888966256 >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 msgr2=0x7f17040b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f1708d9b700 1 --2- 172.21.15.29:0/2888966256 >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 0x7f17040b09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f1708d9b700 1 -- 172.21.15.29:0/2888966256 >> v1:172.21.15.144:6789/0 conn(0x7f17040ae570 legacy=0x7f17040accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f1708d9b700 1 -- 172.21.15.29:0/2888966256 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040ada90 con 0x7f1704156180 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f170959c700 1 --2- 172.21.15.29:0/2888966256 >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 0x7f17040b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f1708d9b700 1 --2- 172.21.15.29:0/2888966256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 0x7f17040afc00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f16f80098e0 tx=0x7f16f80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acb59a5b5a7e2f4c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.917+0000 7f17037fe700 1 -- 172.21.15.29:0/2888966256 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8011630 con 0x7f1704156180 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f17037fe700 1 -- 172.21.15.29:0/2888966256 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f8011790 con 0x7f1704156180 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f17037fe700 1 -- 172.21.15.29:0/2888966256 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8011b30 con 0x7f1704156180 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 -- 172.21.15.29:0/2888966256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 msgr2=0x7f17040afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 --2- 172.21.15.29:0/2888966256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 0x7f17040afc00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f16f80098e0 tx=0x7f16f80091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 -- 172.21.15.29:0/2888966256 shutdown_connections 2023-10-07T10:44:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 --2- 172.21.15.29:0/2888966256 >> v2:172.21.15.144:3300/0 conn(0x7f17040b0520 0x7f17040b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 --2- 172.21.15.29:0/2888966256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1704156180 0x7f17040afc00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 -- 172.21.15.29:0/2888966256 >> 172.21.15.29:0/2888966256 conn(0x7f1704067610 msgr2=0x7f1704067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 -- 172.21.15.29:0/2888966256 shutdown_connections 2023-10-07T10:44:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.918+0000 7f170afff700 1 -- 172.21.15.29:0/2888966256 wait complete. 2023-10-07T10:44:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.920+0000 7f170afff700 1 Processor -- start 2023-10-07T10:44:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.920+0000 7f170afff700 1 -- start start 2023-10-07T10:44:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17040ae570 0x7f17040caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170afff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17040b0520 0x7f1704149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170afff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 0x7f17041522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170afff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040cabf0 con 0x7f1704156180 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170afff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040cb430 con 0x7f17040b0520 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170414ad40 con 0x7f17040ae570 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170959c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 0x7f17041522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170959c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 0x7f17041522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33542/0 (socket says 172.21.15.29:33542) 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.921+0000 7f170959c700 1 -- 172.21.15.29:0/829404560 learned_addr learned my addr 172.21.15.29:0/829404560 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f1708d9b700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17040ae570 0x7f17040caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f1703fff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17040b0520 0x7f1704149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170959c700 1 -- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17040ae570 msgr2=0x7f17040caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170959c700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17040ae570 0x7f17040caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170959c700 1 -- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17040b0520 msgr2=0x7f1704149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170959c700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17040b0520 0x7f1704149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170959c700 1 -- 172.21.15.29:0/829404560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc009ae0 con 0x7f1704156180 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170959c700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 0x7f17041522a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc018070 con 0x7f1704156180 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc00f040 con 0x7f1704156180 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f8009420 con 0x7f1704156180 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.922+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00cd30 con 0x7f1704156180 2023-10-07T10:44:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.923+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17040aa0b0 con 0x7f1704156180 2023-10-07T10:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.924+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170404c3c0 con 0x7f1704156180 2023-10-07T10:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.924+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f16fc003020 con 0x7f1704156180 2023-10-07T10:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.925+0000 7f1700ff9700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f16e80670a0 0x7f16e8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.925+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f16fc085120 con 0x7f1704156180 2023-10-07T10:44:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.926+0000 7f1708d9b700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f16e80670a0 0x7f16e8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.927+0000 7f1708d9b700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f16e80670a0 0x7f16e8069550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f16f80091e0 tx=0x7f16f8017000 comp rx=0 tx=0).ready entity=mgr.7939 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:19.930+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16fc051df0 con 0x7f1704156180 2023-10-07T10:44:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:20.187+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f17040ade00 con 0x7f1704156180 2023-10-07T10:44:21.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.029+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f16fc0521c0 con 0x7f1704156180 2023-10-07T10:44:21.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.031+0000 7f1700ff9700 1 -- 172.21.15.29:0/829404560 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) v1 ==== 93+0+0 (secure 0 0 0) 0x7f16fc051df0 con 0x7f1704156180 2023-10-07T10:44:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.034+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f16e80670a0 msgr2=0x7f16e8069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.034+0000 7f170afff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f16e80670a0 0x7f16e8069550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f16f80091e0 tx=0x7f16f8017000 comp rx=0 tx=0).stop 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.034+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 msgr2=0x7f17041522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.034+0000 7f170afff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 0x7f17041522a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc00b040 comp rx=0 tx=0).stop 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 shutdown_connections 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f16e80670a0 0x7f16e8069550 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1704156180 0x7f17041522a0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17040ae570 0x7f17040caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 --2- 172.21.15.29:0/829404560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17040b0520 0x7f1704149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 >> 172.21.15.29:0/829404560 conn(0x7f1704067610 msgr2=0x7f1704158f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 shutdown_connections 2023-10-07T10:44:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.035+0000 7f170afff700 1 -- 172.21.15.29:0/829404560 wait complete. 2023-10-07T10:44:21.056 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T10:44:21.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:44:21.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:44:21.251 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:21.250+0000 7f4dba76e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:44:21.252 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:21.250+0000 7f6ff0e911c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:44:21.278 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T10:44:21.278 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T10:44:21.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:21.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T10:44:21.382 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T10:44:21.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:21.382 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T10:44:21.429 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:21.428+0000 7f4dba76e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:44:21.446 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:21.445+0000 7f6ff0e911c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:44:21.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.656+0000 7fabfdfff700 1 Processor -- start 2023-10-07T10:44:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.657+0000 7fabfdfff700 1 -- start start 2023-10-07T10:44:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabfdfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fabf80ad370 0x7fabf80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabfdfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fabf80b0410 0x7fabf815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabfdfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf814d790 con 0x7fabf8159ec0 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabfdfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf815a880 con 0x7fabf80ad370 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabfdfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf80ae5d0 con 0x7fabf80b0410 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabf77fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fabf80ad370 0x7fabf80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabf77fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fabf80ad370 0x7fabf80cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33570/0 (socket says 172.21.15.29:33570) 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabf77fe700 1 -- 172.21.15.29:0/3800152187 learned_addr learned my addr 172.21.15.29:0/3800152187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.658+0000 7fabf7fff700 1 --2- 172.21.15.29:0/3800152187 >> v2:172.21.15.144:3300/0 conn(0x7fabf80b0410 0x7fabf815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:21.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf77fe700 1 -- 172.21.15.29:0/3800152187 >> v2:172.21.15.144:3300/0 conn(0x7fabf80b0410 msgr2=0x7fabf815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:21.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf77fe700 1 --2- 172.21.15.29:0/3800152187 >> v2:172.21.15.144:3300/0 conn(0x7fabf80b0410 0x7fabf815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf77fe700 1 -- 172.21.15.29:0/3800152187 >> v1:172.21.15.144:6789/0 conn(0x7fabf8159ec0 legacy=0x7fabf81568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf77fe700 1 -- 172.21.15.29:0/3800152187 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80ad740 con 0x7fabf80ad370 2023-10-07T10:44:21.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf77fe700 1 --2- 172.21.15.29:0/3800152187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ad370 0x7fabf80cc870 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fabe80099b0 tx=0x7fabe8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d98fa77d6690e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:21.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf67fc700 1 -- 172.21.15.29:0/3800152187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe8009ea0 con 0x7fabf80ad370 2023-10-07T10:44:21.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf67fc700 1 -- 172.21.15.29:0/3800152187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe800b820 con 0x7fabf80ad370 2023-10-07T10:44:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.659+0000 7fabf67fc700 1 -- 172.21.15.29:0/3800152187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe800bbc0 con 0x7fabf80ad370 2023-10-07T10:44:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 -- 172.21.15.29:0/3800152187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ad370 msgr2=0x7fabf80cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3800152187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ad370 0x7fabf80cc870 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fabe80099b0 tx=0x7fabe8009250 comp rx=0 tx=0).stop 2023-10-07T10:44:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 -- 172.21.15.29:0/3800152187 shutdown_connections 2023-10-07T10:44:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3800152187 >> v2:172.21.15.144:3300/0 conn(0x7fabf80b0410 0x7fabf815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3800152187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80ad370 0x7fabf80cc870 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 -- 172.21.15.29:0/3800152187 >> 172.21.15.29:0/3800152187 conn(0x7fabf80680a0 msgr2=0x7fabf80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 -- 172.21.15.29:0/3800152187 shutdown_connections 2023-10-07T10:44:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.660+0000 7fabfdfff700 1 -- 172.21.15.29:0/3800152187 wait complete. 2023-10-07T10:44:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.662+0000 7fabfdfff700 1 Processor -- start 2023-10-07T10:44:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.662+0000 7fabfdfff700 1 -- start start 2023-10-07T10:44:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabfdfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80ad370 0x7fabf81484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabfdfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 0x7fabf806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabfdfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8159ec0 0x7fabf81478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabfdfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81489e0 con 0x7fabf80b0410 2023-10-07T10:44:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabfdfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8148c10 con 0x7fabf8159ec0 2023-10-07T10:44:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabfdfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8148ef0 con 0x7fabf80ad370 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 0x7fabf806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf77fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80ad370 0x7fabf81484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 0x7fabf806d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33584/0 (socket says 172.21.15.29:33584) 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf6ffd700 1 -- 172.21.15.29:0/3898759518 learned_addr learned my addr 172.21.15.29:0/3898759518 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf7fff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8159ec0 0x7fabf81478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf77fe700 1 -- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8159ec0 msgr2=0x7fabf81478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf77fe700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8159ec0 0x7fabf81478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf77fe700 1 -- 172.21.15.29:0/3898759518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 msgr2=0x7fabf806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.663+0000 7fabf77fe700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 0x7fabf806d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabf77fe700 1 -- 172.21.15.29:0/3898759518 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80094f0 con 0x7fabf80ad370 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabf6ffd700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 0x7fabf806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabf77fe700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80ad370 0x7fabf81484a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fabf80b12a0 tx=0x7fabe8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:21.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe8000d10 con 0x7fabf80ad370 2023-10-07T10:44:21.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe800bba0 con 0x7fabf80ad370 2023-10-07T10:44:21.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe8011670 con 0x7fabf80ad370 2023-10-07T10:44:21.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf8146e10 con 0x7fabf80ad370 2023-10-07T10:44:21.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.664+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf81473d0 con 0x7fabf80ad370 2023-10-07T10:44:21.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.665+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf804c3c0 con 0x7fabf80ad370 2023-10-07T10:44:21.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.666+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fabe8011a60 con 0x7fabf80ad370 2023-10-07T10:44:21.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.667+0000 7fabdf7fe700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 0x7fabe4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:21.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.667+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6441+0+0 (secure 0 0 0) 0x7fabe8086230 con 0x7fabf80ad370 2023-10-07T10:44:21.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.667+0000 7fabf6ffd700 1 -- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 msgr2=0x7fabe4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:21.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.667+0000 7fabf6ffd700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 0x7fabe4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:21.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.672+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabe8053090 con 0x7fabf80ad370 2023-10-07T10:44:21.739 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:21.738+0000 7f4dba76e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:44:21.793 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:21.792+0000 7f6ff0e911c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:44:21.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.868+0000 7fabf6ffd700 1 -- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 msgr2=0x7fabe4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:21.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.868+0000 7fabf6ffd700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 0x7fabe4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:21.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.922+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf81475c0 con 0x7fabf80ad370 2023-10-07T10:44:21.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.923+0000 7fabdf7fe700 1 -- 172.21.15.29:0/3898759518 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fabe80566b0 con 0x7fabf80ad370 2023-10-07T10:44:21.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:21.924 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:48:22.817019+0000","modified":"2023-10-07T09:48:26.355188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/2749584891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2749584891},{"type":"v1","addr":"172.21.15.29:6834","nonce":2749584891}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-10-07T10:44:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 msgr2=0x7fabe4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 0x7fabe4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80ad370 msgr2=0x7fabf81484a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80ad370 0x7fabf81484a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fabf80b12a0 tx=0x7fabe8017000 comp rx=0 tx=0).stop 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 shutdown_connections 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fabe4067050 0x7fabe4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabf8159ec0 0x7fabf81478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabf80ad370 0x7fabf81484a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 --2- 172.21.15.29:0/3898759518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabf80b0410 0x7fabf806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 >> 172.21.15.29:0/3898759518 conn(0x7fabf80680a0 msgr2=0x7fabf80cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.925+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 shutdown_connections 2023-10-07T10:44:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:21.926+0000 7fabfdfff700 1 -- 172.21.15.29:0/3898759518 wait complete. 2023-10-07T10:44:21.928 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-07T10:44:21.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.252+0000 7fe9143ff700 1 Processor -- start 2023-10-07T10:44:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.253+0000 7fe9143ff700 1 -- start start 2023-10-07T10:44:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.255+0000 7fe9143ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe90c0b1c90 0x7fe90c0aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.255+0000 7fe9143ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c0af530 con 0x7fe90c0afac0 2023-10-07T10:44:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.255+0000 7fe9143ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c167f90 con 0x7fe90c087640 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.255+0000 7fe9143ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c06a4c0 con 0x7fe90c0b1c90 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.259+0000 7fe91299c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe90c0b1c90 0x7fe90c0aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.259+0000 7fe91299c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe90c0b1c90 0x7fe90c0aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33588/0 (socket says 172.21.15.29:33588) 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.259+0000 7fe91299c700 1 -- 172.21.15.29:0/2330697750 learned_addr learned my addr 172.21.15.29:0/2330697750 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.265+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274927463 0 0) 0x7fe90c0af530 con 0x7fe90c0afac0 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.314+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f8003570 con 0x7fe90c0afac0 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.314+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353444260 0 0) 0x7fe90c167f90 con 0x7fe90c087640 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.314+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe90c0af530 con 0x7fe90c087640 2023-10-07T10:44:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.315+0000 7fe91299c700 1 -- 172.21.15.29:0/2330697750 >> v1:172.21.15.144:6789/0 conn(0x7fe90c087640 legacy=0x7fe90c0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.315+0000 7fe91299c700 1 -- 172.21.15.29:0/2330697750 >> v1:172.21.15.29:6789/0 conn(0x7fe90c0afac0 legacy=0x7fe90c168420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.315+0000 7fe91299c700 1 -- 172.21.15.29:0/2330697750 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c1673c0 con 0x7fe90c0b1c90 2023-10-07T10:44:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.317+0000 7fe91299c700 1 --2- 172.21.15.29:0/2330697750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0b1c90 0x7fe90c0aeff0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00bb80 tx=0x7fe8fc0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc3012beffa568 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.319+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc00b150 con 0x7fe90c0b1c90 2023-10-07T10:44:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.319+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc00b2b0 con 0x7fe90c0b1c90 2023-10-07T10:44:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.319+0000 7fe911199700 1 -- 172.21.15.29:0/2330697750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc00b670 con 0x7fe90c0b1c90 2023-10-07T10:44:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.320+0000 7fe9143ff700 1 -- 172.21.15.29:0/2330697750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0b1c90 msgr2=0x7fe90c0aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.320+0000 7fe9143ff700 1 --2- 172.21.15.29:0/2330697750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0b1c90 0x7fe90c0aeff0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00bb80 tx=0x7fe8fc0096a0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.321+0000 7fe9143ff700 1 -- 172.21.15.29:0/2330697750 shutdown_connections 2023-10-07T10:44:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.321+0000 7fe9143ff700 1 --2- 172.21.15.29:0/2330697750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0b1c90 0x7fe90c0aeff0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.321+0000 7fe9143ff700 1 -- 172.21.15.29:0/2330697750 >> 172.21.15.29:0/2330697750 conn(0x7fe90c0ccc30 msgr2=0x7fe90c069570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.323+0000 7fe9143ff700 1 -- 172.21.15.29:0/2330697750 shutdown_connections 2023-10-07T10:44:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.324+0000 7fe9143ff700 1 -- 172.21.15.29:0/2330697750 wait complete. 2023-10-07T10:44:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.326+0000 7fe9143ff700 1 Processor -- start 2023-10-07T10:44:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.328+0000 7fe9143ff700 1 -- start start 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe9143ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c087640 0x7fe90c164b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe9143ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 0x7fe90c0c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe9143ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0b1c90 0x7fe90c0c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe9143ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c27d0 con 0x7fe90c0afac0 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe9143ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c1bc0 con 0x7fe90c087640 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe9143ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c1ea0 con 0x7fe90c0b1c90 2023-10-07T10:44:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.330+0000 7fe91199a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 0x7fe90c0c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.331+0000 7fe91199a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 0x7fe90c0c2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33596/0 (socket says 172.21.15.29:33596) 2023-10-07T10:44:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.331+0000 7fe91199a700 1 -- 172.21.15.29:0/3757053767 learned_addr learned my addr 172.21.15.29:0/3757053767 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.332+0000 7fe91199a700 1 -- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0b1c90 msgr2=0x7fe90c0c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.332+0000 7fe91199a700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0b1c90 0x7fe90c0c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.332+0000 7fe91199a700 1 -- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c087640 msgr2=0x7fe90c164b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.332+0000 7fe91199a700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c087640 0x7fe90c164b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.332+0000 7fe91199a700 1 -- 172.21.15.29:0/3757053767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc009260 con 0x7fe90c0afac0 2023-10-07T10:44:22.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.333+0000 7fe91199a700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 0x7fe90c0c2c60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c790 tx=0x7fe908007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.334+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908010510 con 0x7fe90c0afac0 2023-10-07T10:44:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.334+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe908010670 con 0x7fe90c0afac0 2023-10-07T10:44:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.334+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080109f0 con 0x7fe90c0afac0 2023-10-07T10:44:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.334+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0c2180 con 0x7fe90c0afac0 2023-10-07T10:44:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.335+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c165050 con 0x7fe90c0afac0 2023-10-07T10:44:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.337+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f40052b0 con 0x7fe90c0afac0 2023-10-07T10:44:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.347+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fe90801b400 con 0x7fe90c0afac0 2023-10-07T10:44:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.348+0000 7fe9027fc700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 0x7fe8f0069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.349+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fe908084670 con 0x7fe90c0afac0 2023-10-07T10:44:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.350+0000 7fe91219b700 1 -- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 msgr2=0x7fe8f0069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.350+0000 7fe91219b700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 0x7fe8f0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.351+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9080514a0 con 0x7fe90c0afac0 2023-10-07T10:44:22.429 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:22.425+0000 7f4dba76e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.551+0000 7fe91219b700 1 -- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 msgr2=0x7fe8f0069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.551+0000 7fe91219b700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 0x7fe8f0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:22.585 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:22.584+0000 7f6ff0e911c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:44:22.662 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:22.661+0000 7f4dba76e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:44:22.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.715+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8f4005830 con 0x7fe90c0afac0 2023-10-07T10:44:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.716+0000 7fe9027fc700 1 -- 172.21.15.29:0/3757053767 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16669 (secure 0 0 0) 0x7fe908054ac0 con 0x7fe90c0afac0 2023-10-07T10:44:22.717 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:22.717 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:22.037853+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:48:20.536626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-07T09:48:21.807909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":139,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.b","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1547178489":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/3550726117":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/1718325147":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/2288398636":"2023-10-08T09:48:38.510621+0000","172.21.15.144:6836/114714":"2023-10-08T09:48:38.510621+0000","172.21.15.144:6834/114714":"2023-10-08T09:48:38.510621+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 msgr2=0x7fe8f0069650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 0x7fe8f0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 msgr2=0x7fe90c0c2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 0x7fe90c0c2c60 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c790 tx=0x7fe908007e30 comp rx=0 tx=0).stop 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 shutdown_connections 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe8f00671a0 0x7fe8f0069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0b1c90 0x7fe90c0c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c087640 0x7fe90c164b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 --2- 172.21.15.29:0/3757053767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe90c0afac0 0x7fe90c0c2c60 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 >> 172.21.15.29:0/3757053767 conn(0x7fe90c0ccc30 msgr2=0x7fe90c0a54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:22.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 shutdown_connections 2023-10-07T10:44:22.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.718+0000 7fe9143ff700 1 -- 172.21.15.29:0/3757053767 wait complete. 2023-10-07T10:44:22.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T09:48:20.536626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T09:48:21.807909+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 139, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T10:44:22.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T10:44:22.800 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:22.799+0000 7f6ff0e911c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:44:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.939+0000 7fe5a5fff700 1 Processor -- start 2023-10-07T10:44:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.940+0000 7fe5a5fff700 1 -- start start 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe5a5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a0159790 0x7fe5a006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe5a5fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5a006b3f0 0x7fe5a00c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe5a5fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a006a430 con 0x7fe5a006a8c0 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe5a5fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a006b830 con 0x7fe5a006b3f0 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe5a5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a0159b60 con 0x7fe5a0159790 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe59effd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe5a006a8c0 0x7fe5a00c6f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56368/0 (socket says 172.21.15.29:56368) 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe59effd700 1 -- 172.21.15.29:0/2528741101 learned_addr learned my addr 172.21.15.29:0/2528741101 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe59f7fe700 1 --2- 172.21.15.29:0/2528741101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a0159790 0x7fe5a006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.941+0000 7fe59ffff700 1 --2- 172.21.15.29:0/2528741101 >> v2:172.21.15.144:3300/0 conn(0x7fe5a006b3f0 0x7fe5a00c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59e7fc700 1 -- 172.21.15.29:0/2528741101 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911014887 0 0) 0x7fe5a006a430 con 0x7fe5a006a8c0 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59e7fc700 1 -- 172.21.15.29:0/2528741101 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe588003570 con 0x7fe5a006a8c0 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59f7fe700 1 -- 172.21.15.29:0/2528741101 >> v2:172.21.15.144:3300/0 conn(0x7fe5a006b3f0 msgr2=0x7fe5a00c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59f7fe700 1 --2- 172.21.15.29:0/2528741101 >> v2:172.21.15.144:3300/0 conn(0x7fe5a006b3f0 0x7fe5a00c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59f7fe700 1 -- 172.21.15.29:0/2528741101 >> v1:172.21.15.29:6789/0 conn(0x7fe5a006a8c0 legacy=0x7fe5a00c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59f7fe700 1 -- 172.21.15.29:0/2528741101 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a0069820 con 0x7fe5a0159790 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.942+0000 7fe59f7fe700 1 --2- 172.21.15.29:0/2528741101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a0159790 0x7fe5a006aeb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5900098e0 tx=0x7fe5900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7519d01c0f33867 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe59e7fc700 1 -- 172.21.15.29:0/2528741101 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590011630 con 0x7fe5a0159790 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe59e7fc700 1 -- 172.21.15.29:0/2528741101 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe590011790 con 0x7fe5a0159790 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe59e7fc700 1 -- 172.21.15.29:0/2528741101 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590011b30 con 0x7fe5a0159790 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe5a5fff700 1 -- 172.21.15.29:0/2528741101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a0159790 msgr2=0x7fe5a006aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/2528741101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a0159790 0x7fe5a006aeb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5900098e0 tx=0x7fe5900091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe5a5fff700 1 -- 172.21.15.29:0/2528741101 shutdown_connections 2023-10-07T10:44:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/2528741101 >> v2:172.21.15.144:3300/0 conn(0x7fe5a006b3f0 0x7fe5a00c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/2528741101 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a0159790 0x7fe5a006aeb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.943+0000 7fe5a5fff700 1 -- 172.21.15.29:0/2528741101 >> 172.21.15.29:0/2528741101 conn(0x7fe5a00c9410 msgr2=0x7fe5a00c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.944+0000 7fe5a5fff700 1 -- 172.21.15.29:0/2528741101 shutdown_connections 2023-10-07T10:44:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.944+0000 7fe5a5fff700 1 -- 172.21.15.29:0/2528741101 wait complete. 2023-10-07T10:44:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.946+0000 7fe5a5fff700 1 Processor -- start 2023-10-07T10:44:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.946+0000 7fe5a5fff700 1 -- start start 2023-10-07T10:44:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe5a5fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a006b3f0 0x7fe5a00b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe5a5fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5a0159790 0x7fe5a00b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe5a5fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5a00b13c0 0x7fe5a016d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe5a5fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00ae3e0 con 0x7fe5a0159790 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe5a5fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00ac7b0 con 0x7fe5a00b13c0 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe5a5fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00ac1a0 con 0x7fe5a006b3f0 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.947+0000 7fe59effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5a0159790 0x7fe5a00b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5a0159790 0x7fe5a00b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58984/0 (socket says 172.21.15.29:58984) 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a006b3f0 0x7fe5a00b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59effd700 1 -- 172.21.15.29:0/1745635435 learned_addr learned my addr 172.21.15.29:0/1745635435 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59ffff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5a00b13c0 0x7fe5a016d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 -- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5a00b13c0 msgr2=0x7fe5a016d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5a00b13c0 0x7fe5a016d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 -- 172.21.15.29:0/1745635435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5a0159790 msgr2=0x7fe5a00b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5a0159790 0x7fe5a00b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 -- 172.21.15.29:0/1745635435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe590009420 con 0x7fe5a006b3f0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.948+0000 7fe59f7fe700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a006b3f0 0x7fe5a00b1f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe590016000 tx=0x7fe590017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.949+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590011900 con 0x7fe5a006b3f0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.949+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe590011a60 con 0x7fe5a006b3f0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.949+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590023620 con 0x7fe5a006b3f0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.949+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a016de50 con 0x7fe5a006b3f0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.949+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a016e130 con 0x7fe5a006b3f0 2023-10-07T10:44:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.950+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a00ac600 con 0x7fe5a006b3f0 2023-10-07T10:44:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.951+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fe59002c3b0 con 0x7fe5a006b3f0 2023-10-07T10:44:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.951+0000 7fe5877fe700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 0x7fe58c0695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.952+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fe590085d60 con 0x7fe5a006b3f0 2023-10-07T10:44:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.952+0000 7fe59effd700 1 -- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 msgr2=0x7fe58c0695d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.952+0000 7fe59effd700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 0x7fe58c0695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:22.956+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe590052c10 con 0x7fe5a006b3f0 2023-10-07T10:44:23.080 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:23.079+0000 7f4dba76e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:44:23.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.153+0000 7fe59effd700 1 -- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 msgr2=0x7fe58c0695d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:23.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.153+0000 7fe59effd700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 0x7fe58c0695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.161+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5a004c3c0 con 0x7fe5a006b3f0 2023-10-07T10:44:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.162+0000 7fe5877fe700 1 -- 172.21.15.29:0/1745635435 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe590056230 con 0x7fe5a006b3f0 2023-10-07T10:44:23.163 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 msgr2=0x7fe58c0695d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 0x7fe58c0695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a006b3f0 msgr2=0x7fe5a00b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a006b3f0 0x7fe5a00b1f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe590016000 tx=0x7fe590017000 comp rx=0 tx=0).stop 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 shutdown_connections 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe58c067120 0x7fe58c0695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5a00b13c0 0x7fe5a016d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a006b3f0 0x7fe5a00b1f30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 --2- 172.21.15.29:0/1745635435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5a0159790 0x7fe5a00b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 >> 172.21.15.29:0/1745635435 conn(0x7fe5a00c9410 msgr2=0x7fe5a0154130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 shutdown_connections 2023-10-07T10:44:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.165+0000 7fe5a5fff700 1 -- 172.21.15.29:0/1745635435 wait complete. 2023-10-07T10:44:23.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T10:44:23.242 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:23.241+0000 7f6ff0e911c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:44:23.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.389+0000 7f06acbff700 1 Processor -- start 2023-10-07T10:44:23.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.389+0000 7f06acbff700 1 -- start start 2023-10-07T10:44:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06acbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f06a813fba0 0x7f06a806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06acbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a813ff70 0x7f06a8067610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06acbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06a80cdc50 0x7f06a8067b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06acbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a80c78c0 con 0x7f06a80cdc50 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06acbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a806fa50 con 0x7f06a813fba0 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06acbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a806c0f0 con 0x7f06a813ff70 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06a6d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06a80cdc50 0x7f06a8067b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06a6d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06a80cdc50 0x7f06a8067b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58998/0 (socket says 172.21.15.29:58998) 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06a6d9d700 1 -- 172.21.15.29:0/274453266 learned_addr learned my addr 172.21.15.29:0/274453266 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06a5d9b700 1 --2- 172.21.15.29:0/274453266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a813ff70 0x7f06a8067610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.391+0000 7f06a659c700 1 --2- 172.21.15.29:0/274453266 >> v2:172.21.15.144:3300/0 conn(0x7f06a813fba0 0x7f06a806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a6d9d700 1 -- 172.21.15.29:0/274453266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a813ff70 msgr2=0x7f06a8067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a6d9d700 1 --2- 172.21.15.29:0/274453266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a813ff70 0x7f06a8067610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a6d9d700 1 -- 172.21.15.29:0/274453266 >> v2:172.21.15.144:3300/0 conn(0x7f06a813fba0 msgr2=0x7f06a806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a6d9d700 1 --2- 172.21.15.29:0/274453266 >> v2:172.21.15.144:3300/0 conn(0x7f06a813fba0 0x7f06a806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a6d9d700 1 -- 172.21.15.29:0/274453266 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a806c3d0 con 0x7f06a80cdc50 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a6d9d700 1 --2- 172.21.15.29:0/274453266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a80cdc50 0x7f06a8067b50 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f06a80cd1e0 tx=0x7f06a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c547f01ad6ac5202 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a559a700 1 -- 172.21.15.29:0/274453266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a000e070 con 0x7f06a80cdc50 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a559a700 1 -- 172.21.15.29:0/274453266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a000ba00 con 0x7f06a80cdc50 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.392+0000 7f06a559a700 1 -- 172.21.15.29:0/274453266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00092e0 con 0x7f06a80cdc50 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 -- 172.21.15.29:0/274453266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a80cdc50 msgr2=0x7f06a8067b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 --2- 172.21.15.29:0/274453266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a80cdc50 0x7f06a8067b50 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f06a80cd1e0 tx=0x7f06a0009840 comp rx=0 tx=0).stop 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 -- 172.21.15.29:0/274453266 shutdown_connections 2023-10-07T10:44:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 --2- 172.21.15.29:0/274453266 >> v2:172.21.15.144:3300/0 conn(0x7f06a813fba0 0x7f06a806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 --2- 172.21.15.29:0/274453266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a80cdc50 0x7f06a8067b50 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 --2- 172.21.15.29:0/274453266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a813ff70 0x7f06a8067610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 -- 172.21.15.29:0/274453266 >> 172.21.15.29:0/274453266 conn(0x7f06a806b1d0 msgr2=0x7f06a80cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:23.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 -- 172.21.15.29:0/274453266 shutdown_connections 2023-10-07T10:44:23.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.393+0000 7f06acbff700 1 -- 172.21.15.29:0/274453266 wait complete. 2023-10-07T10:44:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.395+0000 7f06acbff700 1 Processor -- start 2023-10-07T10:44:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.395+0000 7f06acbff700 1 -- start start 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06acbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 0x7f06a813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06acbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a813fba0 0x7f06a80b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06acbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a813ff70 0x7f06a80b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06acbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a815a980 con 0x7f06a813ff70 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06acbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a815ac60 con 0x7f06a813fba0 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06acbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a815af40 con 0x7f06a80cdc50 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 0x7f06a813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a813ff70 0x7f06a80b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 0x7f06a813e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53866/0 (socket says 172.21.15.29:53866) 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 -- 172.21.15.29:0/1562946841 learned_addr learned my addr 172.21.15.29:0/1562946841 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a5d9b700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a813fba0 0x7f06a80b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 -- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a813fba0 msgr2=0x7f06a80b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a813fba0 0x7f06a80b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 -- 172.21.15.29:0/1562946841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a813ff70 msgr2=0x7f06a80b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a813ff70 0x7f06a80b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.397+0000 7f06a659c700 1 -- 172.21.15.29:0/1562946841 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0009ae0 con 0x7f06a80cdc50 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.398+0000 7f06a659c700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 0x7f06a813e1a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f069800ec80 tx=0x7f06980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.398+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800c850 con 0x7f06a80cdc50 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.398+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800c9b0 con 0x7f06a80cdc50 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.398+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a815de20 con 0x7f06a80cdc50 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.398+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800cd50 con 0x7f06a80cdc50 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.398+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a813f170 con 0x7f06a80cdc50 2023-10-07T10:44:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.399+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a815dc80 con 0x7f06a80cdc50 2023-10-07T10:44:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.400+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f0698013680 con 0x7f06a80cdc50 2023-10-07T10:44:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.401+0000 7f06967fc700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 0x7f0688069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.401+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f0698084c50 con 0x7f06a80cdc50 2023-10-07T10:44:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.401+0000 7f06a5d9b700 1 -- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 msgr2=0x7f0688069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.401+0000 7f06a5d9b700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 0x7f0688069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.405+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06a815dc80 con 0x7f06a80cdc50 2023-10-07T10:44:23.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.602+0000 7f06a5d9b700 1 -- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 msgr2=0x7f0688069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:23.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.602+0000 7f06a5d9b700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 0x7f0688069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:23.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.614+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06a815dc80 con 0x7f06a80cdc50 2023-10-07T10:44:23.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.615+0000 7f06967fc700 1 -- 172.21.15.29:0/1562946841 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v139) v1 ==== 104+0+11 (secure 0 0 0) 0x7f06a815dc80 con 0x7f06a80cdc50 2023-10-07T10:44:23.616 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 msgr2=0x7f0688069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 0x7f0688069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 msgr2=0x7f06a813e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 0x7f06a813e1a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f069800ec80 tx=0x7f06980091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 shutdown_connections 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0688067050 0x7f0688069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06a813ff70 0x7f06a80b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06a80cdc50 0x7f06a813e1a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 --2- 172.21.15.29:0/1562946841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a813fba0 0x7f06a80b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 >> 172.21.15.29:0/1562946841 conn(0x7f06a806b1d0 msgr2=0x7f06a80c2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 shutdown_connections 2023-10-07T10:44:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.618+0000 7f06acbff700 1 -- 172.21.15.29:0/1562946841 wait complete. 2023-10-07T10:44:23.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T10:44:23.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.838+0000 7feca49ff700 1 Processor -- start 2023-10-07T10:44:23.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.838+0000 7feca49ff700 1 -- start start 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.839+0000 7feca49ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feca0067d20 0x7feca006ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.839+0000 7feca49ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feca013ff70 0x7feca00c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.839+0000 7feca49ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca00c78c0 con 0x7feca013fba0 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.839+0000 7feca49ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca006fa50 con 0x7feca0067d20 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.839+0000 7feca49ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca006c0f0 con 0x7feca013ff70 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feca0067d20 0x7feca006ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feca0067d20 0x7feca006ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59026/0 (socket says 172.21.15.29:59026) 2023-10-07T10:44:23.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 -- 172.21.15.29:0/2180315473 learned_addr learned my addr 172.21.15.29:0/2180315473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:23.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9ed9d700 1 --2- 172.21.15.29:0/2180315473 >> v2:172.21.15.144:3300/0 conn(0x7feca013ff70 0x7feca00c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 -- 172.21.15.29:0/2180315473 >> v2:172.21.15.144:3300/0 conn(0x7feca013ff70 msgr2=0x7feca00c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:23.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 --2- 172.21.15.29:0/2180315473 >> v2:172.21.15.144:3300/0 conn(0x7feca013ff70 0x7feca00c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 -- 172.21.15.29:0/2180315473 >> v1:172.21.15.29:6789/0 conn(0x7feca013fba0 legacy=0x7feca0067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 -- 172.21.15.29:0/2180315473 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca006c3d0 con 0x7feca0067d20 2023-10-07T10:44:23.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.840+0000 7fec9dd9b700 1 --2- 172.21.15.29:0/2180315473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca006ba10 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7feca00c7850 tx=0x7fec94009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b98f3d51f0ebb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:23.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7fec9d59a700 1 -- 172.21.15.29:0/2180315473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94005c10 con 0x7feca0067d20 2023-10-07T10:44:23.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7fec9d59a700 1 -- 172.21.15.29:0/2180315473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94005d70 con 0x7feca0067d20 2023-10-07T10:44:23.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7fec9d59a700 1 -- 172.21.15.29:0/2180315473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9400b910 con 0x7feca0067d20 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7feca49ff700 1 -- 172.21.15.29:0/2180315473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 msgr2=0x7feca006ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7feca49ff700 1 --2- 172.21.15.29:0/2180315473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca006ba10 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7feca00c7850 tx=0x7fec94009220 comp rx=0 tx=0).stop 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7feca49ff700 1 -- 172.21.15.29:0/2180315473 shutdown_connections 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.841+0000 7feca49ff700 1 --2- 172.21.15.29:0/2180315473 >> v2:172.21.15.144:3300/0 conn(0x7feca013ff70 0x7feca00c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.842+0000 7feca49ff700 1 --2- 172.21.15.29:0/2180315473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca006ba10 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.842+0000 7feca49ff700 1 -- 172.21.15.29:0/2180315473 >> 172.21.15.29:0/2180315473 conn(0x7feca006b1d0 msgr2=0x7feca00cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.842+0000 7feca49ff700 1 -- 172.21.15.29:0/2180315473 shutdown_connections 2023-10-07T10:44:23.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.842+0000 7feca49ff700 1 -- 172.21.15.29:0/2180315473 wait complete. 2023-10-07T10:44:23.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.844+0000 7feca49ff700 1 Processor -- start 2023-10-07T10:44:23.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.844+0000 7feca49ff700 1 -- start start 2023-10-07T10:44:23.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7feca49ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca00c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7feca49ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feca013fba0 0x7feca013b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7feca49ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feca013ff70 0x7feca01572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7feca49ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca00c4ce0 con 0x7feca0067d20 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7feca49ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca00c4fc0 con 0x7feca013fba0 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7feca49ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca00c52a0 con 0x7feca013ff70 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.845+0000 7fec9e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca00c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca00c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59028/0 (socket says 172.21.15.29:59028) 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 -- 172.21.15.29:0/527654928 learned_addr learned my addr 172.21.15.29:0/527654928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9dd9b700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feca013fba0 0x7feca013b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9ed9d700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feca013ff70 0x7feca01572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 -- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feca013ff70 msgr2=0x7feca01572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feca013ff70 0x7feca01572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 -- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feca013fba0 msgr2=0x7feca013b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feca013fba0 0x7feca013b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 -- 172.21.15.29:0/527654928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94009480 con 0x7feca0067d20 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec9e59c700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca00c6de0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fec9000c590 tx=0x7fec9000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.846+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec90011590 con 0x7feca0067d20 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.847+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec90011710 con 0x7feca0067d20 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.847+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec90011ad0 con 0x7feca0067d20 2023-10-07T10:44:23.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.847+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca00c5580 con 0x7feca0067d20 2023-10-07T10:44:23.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.847+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca00c5860 con 0x7feca0067d20 2023-10-07T10:44:23.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.848+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca004c3c0 con 0x7feca0067d20 2023-10-07T10:44:23.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.849+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fec9001b3b0 con 0x7feca0067d20 2023-10-07T10:44:23.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.850+0000 7fec8e7fc700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 0x7fec80069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:23.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.850+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fec90083e30 con 0x7feca0067d20 2023-10-07T10:44:23.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.850+0000 7fec9dd9b700 1 -- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 msgr2=0x7fec80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:23.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.850+0000 7fec9dd9b700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 0x7fec80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:23.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:23.854+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec90050c80 con 0x7feca0067d20 2023-10-07T10:44:23.859 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:23.858+0000 7f4dba76e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:44:23.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:23.945+0000 7f4dba76e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:44:24.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.051+0000 7fec9dd9b700 1 -- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 msgr2=0x7fec80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:24.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.051+0000 7fec9dd9b700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 0x7fec80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:24.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.059+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feca006c6b0 con 0x7feca0067d20 2023-10-07T10:44:24.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.060+0000 7fec8e7fc700 1 -- 172.21.15.29:0/527654928 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v139) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec90017070 con 0x7feca0067d20 2023-10-07T10:44:24.060 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T10:44:24.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 msgr2=0x7fec80069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:24.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 0x7fec80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 msgr2=0x7feca00c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca00c6de0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fec9000c590 tx=0x7fec9000c8a0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 shutdown_connections 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fec80067050 0x7fec80069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feca013ff70 0x7feca01572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca0067d20 0x7feca00c6de0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 --2- 172.21.15.29:0/527654928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feca013fba0 0x7feca013b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 >> 172.21.15.29:0/527654928 conn(0x7feca006b1d0 msgr2=0x7feca00cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 shutdown_connections 2023-10-07T10:44:24.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.062+0000 7feca49ff700 1 -- 172.21.15.29:0/527654928 wait complete. 2023-10-07T10:44:24.068 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.068+0000 7f6ff0e911c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:44:24.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:24.103 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.102+0000 7f4dba76e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:44:24.165 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.164+0000 7f6ff0e911c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:44:24.193 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.192+0000 7f4dba76e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:44:24.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.282+0000 7ff54e7ff700 1 Processor -- start 2023-10-07T10:44:24.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.282+0000 7ff54e7ff700 1 -- start start 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff54e7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff5480ad370 0x7ff5480cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff54e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff548159ec0 0x7ff54815a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff54e7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54814d790 con 0x7ff5481568c0 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff54e7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5480ae5d0 con 0x7ff5480ad370 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff54e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54815a850 con 0x7ff548159ec0 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff5477fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff548159ec0 0x7ff54815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff547fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff5480ad370 0x7ff5480cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff5477fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff548159ec0 0x7ff54815a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53882/0 (socket says 172.21.15.29:53882) 2023-10-07T10:44:24.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.285+0000 7ff5477fe700 1 -- 172.21.15.29:0/2803403917 learned_addr learned my addr 172.21.15.29:0/2803403917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:24.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff547fff700 1 -- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff548159ec0 msgr2=0x7ff54815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:24.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff547fff700 1 --2- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff548159ec0 0x7ff54815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff547fff700 1 -- 172.21.15.29:0/2803403917 >> v1:172.21.15.144:6789/0 conn(0x7ff5481568c0 legacy=0x7ff548156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff547fff700 1 -- 172.21.15.29:0/2803403917 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480ad740 con 0x7ff5480ad370 2023-10-07T10:44:24.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff547fff700 1 --2- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5480ad370 0x7ff5480cc870 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff5380098e0 tx=0x7ff5380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b8debe4b6d0cf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:24.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff546ffd700 1 -- 172.21.15.29:0/2803403917 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53800b640 con 0x7ff5480ad370 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff546ffd700 1 -- 172.21.15.29:0/2803403917 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53800b7a0 con 0x7ff5480ad370 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.286+0000 7ff546ffd700 1 -- 172.21.15.29:0/2803403917 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53800bb40 con 0x7ff5480ad370 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 -- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5480ad370 msgr2=0x7ff5480cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5480ad370 0x7ff5480cc870 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff5380098e0 tx=0x7ff5380091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 -- 172.21.15.29:0/2803403917 shutdown_connections 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5480ad370 0x7ff5480cc870 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/2803403917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff548159ec0 0x7ff54815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 -- 172.21.15.29:0/2803403917 >> 172.21.15.29:0/2803403917 conn(0x7ff5480680a0 msgr2=0x7ff5480cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 -- 172.21.15.29:0/2803403917 shutdown_connections 2023-10-07T10:44:24.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.287+0000 7ff54e7ff700 1 -- 172.21.15.29:0/2803403917 wait complete. 2023-10-07T10:44:24.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.289+0000 7ff54e7ff700 1 Processor -- start 2023-10-07T10:44:24.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.289+0000 7ff54e7ff700 1 -- start start 2023-10-07T10:44:24.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 0x7ff5481554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54e7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5481568c0 0x7ff54814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54e7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff548159ec0 0x7ff54814f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54e7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff548150520 con 0x7ff5481568c0 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54e7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54814f8a0 con 0x7ff548159ec0 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54814fb80 con 0x7ff5480ad370 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff5477fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5481568c0 0x7ff54814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff547fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 0x7ff5481554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff547fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 0x7ff5481554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53890/0 (socket says 172.21.15.29:53890) 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff547fff700 1 -- 172.21.15.29:0/1365680735 learned_addr learned my addr 172.21.15.29:0/1365680735 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.290+0000 7ff54cd9c700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff548159ec0 0x7ff54814f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54cd9c700 1 -- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 msgr2=0x7ff5481554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54cd9c700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 0x7ff5481554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54cd9c700 1 -- 172.21.15.29:0/1365680735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5481568c0 msgr2=0x7ff54814eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54cd9c700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5481568c0 0x7ff54814eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54cd9c700 1 -- 172.21.15.29:0/1365680735 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538009420 con 0x7ff548159ec0 2023-10-07T10:44:24.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff547fff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 0x7ff5481554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:24.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff5477fe700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5481568c0 0x7ff54814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:24.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54cd9c700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff548159ec0 0x7ff54814f070 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff54000c830 tx=0x7ff540001690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540012630 con 0x7ff548159ec0 2023-10-07T10:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff540012790 con 0x7ff548159ec0 2023-10-07T10:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540012b50 con 0x7ff548159ec0 2023-10-07T10:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54814fe60 con 0x7ff548159ec0 2023-10-07T10:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.291+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5481501f0 con 0x7ff548159ec0 2023-10-07T10:44:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.292+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548150660 con 0x7ff548159ec0 2023-10-07T10:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.293+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7ff540011380 con 0x7ff548159ec0 2023-10-07T10:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.294+0000 7ff52ffff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 0x7ff534069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.294+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7ff540084b50 con 0x7ff548159ec0 2023-10-07T10:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.294+0000 7ff547fff700 1 -- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 msgr2=0x7ff534069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.294+0000 7ff547fff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 0x7ff534069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.299+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff540051900 con 0x7ff548159ec0 2023-10-07T10:44:24.333 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.332+0000 7f6ff0e911c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:44:24.409 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.408+0000 7f4dba76e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:44:24.424 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.423+0000 7f6ff0e911c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:44:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.495+0000 7ff547fff700 1 -- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 msgr2=0x7ff534069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.495+0000 7ff547fff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 0x7ff534069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:24.503 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.502+0000 7f4dba76e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:44:24.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.548+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff54804c3c0 con 0x7ff548159ec0 2023-10-07T10:44:24.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.549+0000 7ff52ffff700 1 -- 172.21.15.29:0/1365680735 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4599 (secure 0 0 0) 0x7ff540054f20 con 0x7ff548159ec0 2023-10-07T10:44:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:24.550 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:48:22.817019+0000","modified":"2023-10-07T09:48:26.355188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/2749584891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2749584891},{"type":"v1","addr":"172.21.15.29:6834","nonce":2749584891}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-10-07T10:44:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 msgr2=0x7ff534069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 0x7ff534069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff548159ec0 msgr2=0x7ff54814f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff548159ec0 0x7ff54814f070 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff54000c830 tx=0x7ff540001690 comp rx=0 tx=0).stop 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 shutdown_connections 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff534067050 0x7ff534069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff548159ec0 0x7ff54814f070 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5480ad370 0x7ff5481554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 --2- 172.21.15.29:0/1365680735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5481568c0 0x7ff54814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.551+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 >> 172.21.15.29:0/1365680735 conn(0x7ff5480680a0 msgr2=0x7ff54806d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.552+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 shutdown_connections 2023-10-07T10:44:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.552+0000 7ff54e7ff700 1 -- 172.21.15.29:0/1365680735 wait complete. 2023-10-07T10:44:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-07T10:44:24.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:24.599 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.599+0000 7f4dba76e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:44:24.648 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.647+0000 7f6ff0e911c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:44:24.694 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.693+0000 7f4dba76e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:44:24.749 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.748+0000 7f6ff0e911c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:44:24.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.772+0000 7f91451ff700 1 Processor -- start 2023-10-07T10:44:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.773+0000 7f91451ff700 1 -- start start 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.774+0000 7f91451ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9140162590 0x7f9140164970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.774+0000 7f91451ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914006f4e0 con 0x7f91400b2670 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.774+0000 7f91451ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91400b1120 con 0x7f9140159550 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.774+0000 7f91451ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914006e8d0 con 0x7f9140162590 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.774+0000 7f913e59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91400b2670 0x7f91400b2ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56392/0 (socket says 172.21.15.29:56392) 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.774+0000 7f913e59c700 1 -- 172.21.15.29:0/3653954468 learned_addr learned my addr 172.21.15.29:0/3653954468 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913f59e700 1 --2- 172.21.15.29:0/3653954468 >> v2:172.21.15.144:3300/0 conn(0x7f9140162590 0x7f9140164970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913dd9b700 1 -- 172.21.15.29:0/3653954468 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358247628 0 0) 0x7f914006f4e0 con 0x7f91400b2670 2023-10-07T10:44:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913dd9b700 1 -- 172.21.15.29:0/3653954468 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9128003570 con 0x7f91400b2670 2023-10-07T10:44:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913f59e700 1 -- 172.21.15.29:0/3653954468 >> v1:172.21.15.144:6789/0 conn(0x7f9140159550 legacy=0x7f91400b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913f59e700 1 -- 172.21.15.29:0/3653954468 >> v1:172.21.15.29:6789/0 conn(0x7f91400b2670 legacy=0x7f91400b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913f59e700 1 -- 172.21.15.29:0/3653954468 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914006ebb0 con 0x7f9140162590 2023-10-07T10:44:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.775+0000 7f913f59e700 1 --2- 172.21.15.29:0/3653954468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9140162590 0x7f9140164970 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9138009e60 tx=0x7f9138009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4239652286fc50 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f913dd9b700 1 -- 172.21.15.29:0/3653954468 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913800e070 con 0x7f9140162590 2023-10-07T10:44:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f913dd9b700 1 -- 172.21.15.29:0/3653954468 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913800bb60 con 0x7f9140162590 2023-10-07T10:44:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f913dd9b700 1 -- 172.21.15.29:0/3653954468 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138012430 con 0x7f9140162590 2023-10-07T10:44:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 -- 172.21.15.29:0/3653954468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9140162590 msgr2=0x7f9140164970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 --2- 172.21.15.29:0/3653954468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9140162590 0x7f9140164970 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9138009e60 tx=0x7f9138009780 comp rx=0 tx=0).stop 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 -- 172.21.15.29:0/3653954468 shutdown_connections 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 --2- 172.21.15.29:0/3653954468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9140162590 0x7f9140164970 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 -- 172.21.15.29:0/3653954468 >> 172.21.15.29:0/3653954468 conn(0x7f9140069190 msgr2=0x7f914006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 -- 172.21.15.29:0/3653954468 shutdown_connections 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.776+0000 7f91451ff700 1 -- 172.21.15.29:0/3653954468 wait complete. 2023-10-07T10:44:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.778+0000 7f91451ff700 1 Processor -- start 2023-10-07T10:44:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.778+0000 7f91451ff700 1 -- start start 2023-10-07T10:44:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f91451ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91400b2670 0x7f9140151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f91451ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 0x7f91400ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f91451ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9140162590 0x7f9140149960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f91451ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914014fb20 con 0x7f9140159550 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f91451ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914014ef10 con 0x7f91400b2670 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f91451ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914014f1f0 con 0x7f9140162590 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f913e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 0x7f91400ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f913e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 0x7f91400ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59038/0 (socket says 172.21.15.29:59038) 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f913e59c700 1 -- 172.21.15.29:0/4272281437 learned_addr learned my addr 172.21.15.29:0/4272281437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f913ed9d700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91400b2670 0x7f9140151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.779+0000 7f913f59e700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9140162590 0x7f9140149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913ed9d700 1 -- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9140162590 msgr2=0x7f9140149960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913ed9d700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9140162590 0x7f9140149960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913ed9d700 1 -- 172.21.15.29:0/4272281437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 msgr2=0x7f91400ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913ed9d700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 0x7f91400ab7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913ed9d700 1 -- 172.21.15.29:0/4272281437 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138009a20 con 0x7f91400b2670 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913e59c700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 0x7f91400ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f913ed9d700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91400b2670 0x7f9140151020 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f913000c590 tx=0x7f9130007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130011590 con 0x7f91400b2670 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9130011710 con 0x7f91400b2670 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130011ad0 con 0x7f91400b2670 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914014f4d0 con 0x7f91400b2670 2023-10-07T10:44:24.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.780+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9140151560 con 0x7f91400b2670 2023-10-07T10:44:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.782+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91400b3030 con 0x7f91400b2670 2023-10-07T10:44:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.782+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f913001b3b0 con 0x7f91400b2670 2023-10-07T10:44:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.783+0000 7f9126ffd700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 0x7f912c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.784+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f9130083dc0 con 0x7f91400b2670 2023-10-07T10:44:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.784+0000 7f913e59c700 1 -- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 msgr2=0x7f912c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.784+0000 7f913e59c700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 0x7f912c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:24.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.788+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9130050b70 con 0x7f91400b2670 2023-10-07T10:44:24.852 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.851+0000 7f6ff0e911c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:44:24.882 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.882+0000 7f4dba76e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:44:24.952 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:24.951+0000 7f6ff0e911c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:44:24.979 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:24.978+0000 7f4dba76e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:44:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.985+0000 7f913e59c700 1 -- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 msgr2=0x7f912c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.985+0000 7f913e59c700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 0x7f912c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.993+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f914006b3c0 con 0x7f91400b2670 2023-10-07T10:44:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.995+0000 7f9126ffd700 1 -- 172.21.15.29:0/4272281437 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f9130017070 con 0x7f91400b2670 2023-10-07T10:44:24.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:24.996 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:22.037853+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:48:20.536626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-07T09:48:21.807909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":139,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.b","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1547178489":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/3550726117":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/1718325147":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/2288398636":"2023-10-08T09:48:38.510621+0000","172.21.15.144:6836/114714":"2023-10-08T09:48:38.510621+0000","172.21.15.144:6834/114714":"2023-10-08T09:48:38.510621+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.998+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 msgr2=0x7f912c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.998+0000 7f91451ff700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 0x7f912c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.998+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91400b2670 msgr2=0x7f9140151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.998+0000 7f91451ff700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91400b2670 0x7f9140151020 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f913000c590 tx=0x7f9130007ec0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 shutdown_connections 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f912c067050 0x7f912c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9140162590 0x7f9140149960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91400b2670 0x7f9140151020 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 --2- 172.21.15.29:0/4272281437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9140159550 0x7f91400ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 >> 172.21.15.29:0/4272281437 conn(0x7f9140069190 msgr2=0x7f91400ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 shutdown_connections 2023-10-07T10:44:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:24.999+0000 7f91451ff700 1 -- 172.21.15.29:0/4272281437 wait complete. 2023-10-07T10:44:25.025 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T10:44:25.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:25.162 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:25.161+0000 7f6ff0e911c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:44:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.224+0000 7f47123ff700 1 Processor -- start 2023-10-07T10:44:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.224+0000 7f47123ff700 1 -- start start 2023-10-07T10:44:25.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.225+0000 7f47123ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0cc040 0x7f470c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f47123ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0b0f60 con 0x7f470c0ca280 2023-10-07T10:44:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f47123ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0b0070 con 0x7f470c0cccc0 2023-10-07T10:44:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f47123ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0b2cb0 con 0x7f470c0cc040 2023-10-07T10:44:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f471099c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f470c0ca280 0x7f470c0b13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56406/0 (socket says 172.21.15.29:56406) 2023-10-07T10:44:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f471099c700 1 -- 172.21.15.29:0/4028233192 learned_addr learned my addr 172.21.15.29:0/4028233192 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f470bfff700 1 --2- 172.21.15.29:0/4028233192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0cc040 0x7f470c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836273672 0 0) 0x7f470c0b0f60 con 0x7f470c0ca280 2023-10-07T10:44:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f4003570 con 0x7f470c0ca280 2023-10-07T10:44:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658832767 0 0) 0x7f470c0b0070 con 0x7f470c0cccc0 2023-10-07T10:44:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.226+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c0b0f60 con 0x7f470c0cccc0 2023-10-07T10:44:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470bfff700 1 -- 172.21.15.29:0/4028233192 >> v1:172.21.15.144:6789/0 conn(0x7f470c0cccc0 legacy=0x7f470c06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470bfff700 1 -- 172.21.15.29:0/4028233192 >> v1:172.21.15.29:6789/0 conn(0x7f470c0ca280 legacy=0x7f470c0b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470bfff700 1 -- 172.21.15.29:0/4028233192 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0b0350 con 0x7f470c0cc040 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470bfff700 1 --2- 172.21.15.29:0/4028233192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0cc040 0x7f470c06e4f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009910 tx=0x7f46fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f500feacd49ff275 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc011630 con 0x7f470c0cc040 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc011790 con 0x7f470c0cc040 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.227+0000 7f470affd700 1 -- 172.21.15.29:0/4028233192 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc011b30 con 0x7f470c0cc040 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 -- 172.21.15.29:0/4028233192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0cc040 msgr2=0x7f470c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 --2- 172.21.15.29:0/4028233192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0cc040 0x7f470c06e4f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009910 tx=0x7f46fc0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 -- 172.21.15.29:0/4028233192 shutdown_connections 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 --2- 172.21.15.29:0/4028233192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0cc040 0x7f470c06e4f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 -- 172.21.15.29:0/4028233192 >> 172.21.15.29:0/4028233192 conn(0x7f470c069190 msgr2=0x7f470c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 -- 172.21.15.29:0/4028233192 shutdown_connections 2023-10-07T10:44:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.228+0000 7f47123ff700 1 -- 172.21.15.29:0/4028233192 wait complete. 2023-10-07T10:44:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.229+0000 7f47123ff700 1 Processor -- start 2023-10-07T10:44:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.230+0000 7f47123ff700 1 -- start start 2023-10-07T10:44:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.230+0000 7f47123ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0ca280 0x7f470c14c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.230+0000 7f47123ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 0x7f470c1572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f47123ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f470c0cccc0 0x7f470c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f47123ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0c8d90 con 0x7f470c0cc040 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f47123ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c14c560 con 0x7f470c0cccc0 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f47123ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c157ff0 con 0x7f470c0ca280 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f470b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 0x7f470c1572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f470bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0ca280 0x7f470c14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f470b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 0x7f470c1572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59040/0 (socket says 172.21.15.29:59040) 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f471099c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f470c0cccc0 0x7f470c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f470b7fe700 1 -- 172.21.15.29:0/1253716666 learned_addr learned my addr 172.21.15.29:0/1253716666 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f471099c700 1 -- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0ca280 msgr2=0x7f470c14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f471099c700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0ca280 0x7f470c14c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f471099c700 1 -- 172.21.15.29:0/1253716666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 msgr2=0x7f470c1572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f471099c700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 0x7f470c1572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f471099c700 1 -- 172.21.15.29:0/1253716666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc009420 con 0x7f470c0cccc0 2023-10-07T10:44:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.231+0000 7f470b7fe700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 0x7f470c1572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.232+0000 7f471099c700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f470c0cccc0 0x7f470c06ea80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4704001460 tx=0x7f47040019a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.232+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470400a7b0 con 0x7f470c0cccc0 2023-10-07T10:44:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.232+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470400a910 con 0x7f470c0cccc0 2023-10-07T10:44:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.232+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470400acb0 con 0x7f470c0cccc0 2023-10-07T10:44:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.232+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c0c8180 con 0x7f470c0cccc0 2023-10-07T10:44:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.232+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c0c8740 con 0x7f470c0cccc0 2023-10-07T10:44:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.233+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c0c9120 con 0x7f470c0cccc0 2023-10-07T10:44:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.234+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f4704007300 con 0x7f470c0cccc0 2023-10-07T10:44:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.235+0000 7f46f3fff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 0x7f46f8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.235+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f4704084c50 con 0x7f470c0cccc0 2023-10-07T10:44:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.235+0000 7f470bfff700 1 -- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 msgr2=0x7f46f8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:25.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.235+0000 7f470bfff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 0x7f46f8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:25.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.239+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4704051a80 con 0x7f470c0cccc0 2023-10-07T10:44:25.268 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:25.267+0000 7f6ff0e911c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:44:25.322 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:25.321+0000 7f4dba76e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:44:25.410 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:25.409+0000 7f4dba76e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.436+0000 7f470bfff700 1 -- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 msgr2=0x7f46f8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.436+0000 7f470bfff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 0x7f46f8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:25.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.490+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f470c0c8f00 con 0x7f470c0cccc0 2023-10-07T10:44:25.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.491+0000 7f46f3fff700 1 -- 172.21.15.29:0/1253716666 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f47040550a0 con 0x7f470c0cccc0 2023-10-07T10:44:25.492 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:25.492 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:48:22.817019+0000","modified":"2023-10-07T09:48:26.355188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/2749584891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2749584891},{"type":"v1","addr":"172.21.15.29:6834","nonce":2749584891}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-10-07T10:44:25.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 msgr2=0x7f46f8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 0x7f46f8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f470c0cccc0 msgr2=0x7f470c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f470c0cccc0 0x7f470c06ea80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4704001460 tx=0x7f47040019a0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 shutdown_connections 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f46f8067050 0x7f46f8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f470c0cccc0 0x7f470c06ea80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f470c0ca280 0x7f470c14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 --2- 172.21.15.29:0/1253716666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470c0cc040 0x7f470c1572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 >> 172.21.15.29:0/1253716666 conn(0x7f470c069190 msgr2=0x7f470c0cb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.493+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 shutdown_connections 2023-10-07T10:44:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.494+0000 7f47123ff700 1 -- 172.21.15.29:0/1253716666 wait complete. 2023-10-07T10:44:25.496 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-07T10:44:25.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:25.649 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:25.648+0000 7f6ff0e911c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:44:25.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.747+0000 7fbe1e7ff700 1 Processor -- start 2023-10-07T10:44:25.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.747+0000 7fbe1e7ff700 1 -- start start 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe1e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe1815a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe1e7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1814d790 con 0x7fbe180af400 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe1e7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180ae5d0 con 0x7fbe180ad370 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe1e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1815a850 con 0x7fbe18159ec0 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe1cd9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbe180af400 0x7fbe180af810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56414/0 (socket says 172.21.15.29:56414) 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe1cd9c700 1 -- 172.21.15.29:0/2819318795 learned_addr learned my addr 172.21.15.29:0/2819318795 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe177fe700 1 --2- 172.21.15.29:0/2819318795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe1815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020769213 0 0) 0x7fbe180ae5d0 con 0x7fbe180ad370 2023-10-07T10:44:25.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe00003570 con 0x7fbe180ad370 2023-10-07T10:44:25.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.750+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397398841 0 0) 0x7fbe1814d790 con 0x7fbe180af400 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe180ae5d0 con 0x7fbe180af400 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe177fe700 1 -- 172.21.15.29:0/2819318795 >> v1:172.21.15.144:6789/0 conn(0x7fbe180ad370 legacy=0x7fbe180cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe177fe700 1 -- 172.21.15.29:0/2819318795 >> v1:172.21.15.29:6789/0 conn(0x7fbe180af400 legacy=0x7fbe180af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe177fe700 1 -- 172.21.15.29:0/2819318795 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe180ad740 con 0x7fbe18159ec0 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe177fe700 1 --2- 172.21.15.29:0/2819318795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe1815a310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009910 tx=0x7fbe0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed928876a856bbd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c011630 con 0x7fbe18159ec0 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c011790 con 0x7fbe18159ec0 2023-10-07T10:44:25.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.751+0000 7fbe16ffd700 1 -- 172.21.15.29:0/2819318795 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c011b30 con 0x7fbe18159ec0 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/2819318795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 msgr2=0x7fbe1815a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/2819318795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe1815a310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009910 tx=0x7fbe0c0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/2819318795 shutdown_connections 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/2819318795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe1815a310 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/2819318795 >> 172.21.15.29:0/2819318795 conn(0x7fbe180680a0 msgr2=0x7fbe180cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/2819318795 shutdown_connections 2023-10-07T10:44:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.752+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/2819318795 wait complete. 2023-10-07T10:44:25.756 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:25.753+0000 7f6ff0e911c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:25.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.753+0000 7fbe1e7ff700 1 Processor -- start 2023-10-07T10:44:25.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.754+0000 7fbe1e7ff700 1 -- start start 2023-10-07T10:44:25.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.754+0000 7fbe1e7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 0x7fbe18144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.754+0000 7fbe1e7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe180af400 0x7fbe18169db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe1e7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe18150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe1e7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1814d1c0 con 0x7fbe180ad370 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe1e7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1806e950 con 0x7fbe180af400 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe1e7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1816aec0 con 0x7fbe18159ec0 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 0x7fbe18144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe177fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe180af400 0x7fbe18169db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 0x7fbe18144e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59044/0 (socket says 172.21.15.29:59044) 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe1cd9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe18150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 -- 172.21.15.29:0/3716910600 learned_addr learned my addr 172.21.15.29:0/3716910600 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 -- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 msgr2=0x7fbe18150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe18150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 -- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe180af400 msgr2=0x7fbe18169db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe180af400 0x7fbe18169db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe17fff700 1 -- 172.21.15.29:0/3716910600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c009420 con 0x7fbe180ad370 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.755+0000 7fbe177fe700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe180af400 0x7fbe18169db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.756+0000 7fbe17fff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 0x7fbe18144e40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800c7f0 tx=0x7fbe08007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.756+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080015b0 con 0x7fbe180ad370 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.756+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe08001710 con 0x7fbe180ad370 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.756+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe08001a90 con 0x7fbe180ad370 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.756+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1816a2f0 con 0x7fbe180ad370 2023-10-07T10:44:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.756+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1816a870 con 0x7fbe180ad370 2023-10-07T10:44:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.757+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1804c3c0 con 0x7fbe180ad370 2023-10-07T10:44:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.758+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fbe0801b400 con 0x7fbe180ad370 2023-10-07T10:44:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.759+0000 7fbdfffff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 0x7fbe04069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.759+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fbe080847c0 con 0x7fbe180ad370 2023-10-07T10:44:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.759+0000 7fbe177fe700 1 -- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 msgr2=0x7fbe04069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.759+0000 7fbe177fe700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 0x7fbe04069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.763+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe08051570 con 0x7fbe180ad370 2023-10-07T10:44:25.952 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:25.951+0000 7f4dba76e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:44:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.960+0000 7fbe177fe700 1 -- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 msgr2=0x7fbe04069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:25.960+0000 7fbe177fe700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 0x7fbe04069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.013+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe1806b740 con 0x7fbe180ad370 2023-10-07T10:44:26.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.013+0000 7fbdfffff700 1 -- 172.21.15.29:0/3716910600 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fbe08054b90 con 0x7fbe180ad370 2023-10-07T10:44:26.014 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:26.014 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T09:48:22.817019+0000","modified":"2023-10-07T09:48:26.355188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8160},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8160":{"gid":8160,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/2749584891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2749584891},{"type":"v1","addr":"172.21.15.29:6834","nonce":2749584891}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-10-07T10:44:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 msgr2=0x7fbe04069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 0x7fbe04069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 msgr2=0x7fbe18144e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 0x7fbe18144e40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800c7f0 tx=0x7fbe08007e00 comp rx=0 tx=0).stop 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 shutdown_connections 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbe04067050 0x7fbe04069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe18159ec0 0x7fbe18150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe180ad370 0x7fbe18144e40 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 --2- 172.21.15.29:0/3716910600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe180af400 0x7fbe18169db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 >> 172.21.15.29:0/3716910600 conn(0x7fbe180680a0 msgr2=0x7fbe180af810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 shutdown_connections 2023-10-07T10:44:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.016+0000 7fbe1e7ff700 1 -- 172.21.15.29:0/3716910600 wait complete. 2023-10-07T10:44:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-07T10:44:26.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:26.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:26.138+0000 7f4dba76e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:44:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.232+0000 7f0aa4dff700 1 Processor -- start 2023-10-07T10:44:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.232+0000 7f0aa4dff700 1 -- start start 2023-10-07T10:44:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.233+0000 7f0aa4dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0aa0153cc0 0x7f0aa00b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.233+0000 7f0aa4dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0aa00b0710 0x7f0aa00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.233+0000 7f0aa4dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa0154090 con 0x7f0aa015ade0 2023-10-07T10:44:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.233+0000 7f0aa4dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa0159f60 con 0x7f0aa00b0710 2023-10-07T10:44:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.233+0000 7f0aa4dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa015a240 con 0x7f0aa0153cc0 2023-10-07T10:44:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0aa00b0710 0x7f0aa00b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0aa00b0710 0x7f0aa00b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59046/0 (socket says 172.21.15.29:59046) 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 -- 172.21.15.29:0/2120605786 learned_addr learned my addr 172.21.15.29:0/2120605786 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9e59c700 1 --2- 172.21.15.29:0/2120605786 >> v2:172.21.15.144:3300/0 conn(0x7f0aa0153cc0 0x7f0aa00b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 -- 172.21.15.29:0/2120605786 >> v2:172.21.15.144:3300/0 conn(0x7f0aa0153cc0 msgr2=0x7f0aa00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 --2- 172.21.15.29:0/2120605786 >> v2:172.21.15.144:3300/0 conn(0x7f0aa0153cc0 0x7f0aa00b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 -- 172.21.15.29:0/2120605786 >> v1:172.21.15.29:6789/0 conn(0x7f0aa015ade0 legacy=0x7f0aa00b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 -- 172.21.15.29:0/2120605786 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa015a520 con 0x7f0aa00b0710 2023-10-07T10:44:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.234+0000 7f0a9ed9d700 1 --2- 172.21.15.29:0/2120605786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa00b0710 0x7f0aa00b1d00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00cf440 tx=0x7f0a98009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e573476250dd85d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:26.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0a9d59a700 1 -- 172.21.15.29:0/2120605786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9800e070 con 0x7f0aa00b0710 2023-10-07T10:44:26.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0a9d59a700 1 -- 172.21.15.29:0/2120605786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9800ba20 con 0x7f0aa00b0710 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0a9d59a700 1 -- 172.21.15.29:0/2120605786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a980092e0 con 0x7f0aa00b0710 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 -- 172.21.15.29:0/2120605786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa00b0710 msgr2=0x7f0aa00b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/2120605786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa00b0710 0x7f0aa00b1d00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00cf440 tx=0x7f0a98009840 comp rx=0 tx=0).stop 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 -- 172.21.15.29:0/2120605786 shutdown_connections 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/2120605786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa00b0710 0x7f0aa00b1d00 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/2120605786 >> v2:172.21.15.144:3300/0 conn(0x7f0aa0153cc0 0x7f0aa00b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 -- 172.21.15.29:0/2120605786 >> 172.21.15.29:0/2120605786 conn(0x7f0aa0067610 msgr2=0x7f0aa0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.235+0000 7f0aa4dff700 1 -- 172.21.15.29:0/2120605786 shutdown_connections 2023-10-07T10:44:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.236+0000 7f0aa4dff700 1 -- 172.21.15.29:0/2120605786 wait complete. 2023-10-07T10:44:26.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.237+0000 7f0aa4dff700 1 Processor -- start 2023-10-07T10:44:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.237+0000 7f0aa4dff700 1 -- start start 2023-10-07T10:44:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0aa4dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00b0710 0x7f0aa0152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0aa4dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 0x7f0aa0169410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0aa4dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aa015ade0 0x7f0aa0169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0aa4dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00ca1c0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0aa4dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0156d40 con 0x7f0aa00b0710 2023-10-07T10:44:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0aa4dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0152740 con 0x7f0aa015ade0 2023-10-07T10:44:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0a9dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 0x7f0aa0169410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.238+0000 7f0a9dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 0x7f0aa0169410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59048/0 (socket says 172.21.15.29:59048) 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 -- 172.21.15.29:0/1959290461 learned_addr learned my addr 172.21.15.29:0/1959290461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9ed9d700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aa015ade0 0x7f0aa0169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9e59c700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00b0710 0x7f0aa0152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 -- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aa015ade0 msgr2=0x7f0aa0169950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aa015ade0 0x7f0aa0169950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 -- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00b0710 msgr2=0x7f0aa0152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00b0710 0x7f0aa0152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 -- 172.21.15.29:0/1959290461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98009ae0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.239+0000 7f0a9dd9b700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 0x7f0aa0169410 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400c560 tx=0x7f0a94007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.240+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94011590 con 0x7f0aa0153cc0 2023-10-07T10:44:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.240+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a94011710 con 0x7f0aa0153cc0 2023-10-07T10:44:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.240+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94011ad0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.240+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa0156130 con 0x7f0aa0153cc0 2023-10-07T10:44:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.240+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa01566f0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.241+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa004c3c0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.243+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f0a9401a360 con 0x7f0aa0153cc0 2023-10-07T10:44:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.244+0000 7f0a8e7fc700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 0x7f0a80069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.244+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f0a940837f0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.244+0000 7f0a9e59c700 1 -- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 msgr2=0x7f0a80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.244+0000 7f0a9e59c700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 0x7f0a80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:26.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.248+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a94050640 con 0x7f0aa0153cc0 2023-10-07T10:44:26.344 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:26.343+0000 7f6ff0e911c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:44:26.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.445+0000 7f0a9e59c700 1 -- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 msgr2=0x7f0a80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.445+0000 7f0a9e59c700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 0x7f0a80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:26.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.451+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0aa00689d0 con 0x7f0aa0153cc0 2023-10-07T10:44:26.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.452+0000 7f0a8e7fc700 1 -- 172.21.15.29:0/1959290461 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f0a94053c60 con 0x7f0aa0153cc0 2023-10-07T10:44:26.453 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:26.453 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:22.037853+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-07T09:48:20.536626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-07T09:48:21.807909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":139,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.b","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1547178489":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/3550726117":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/1718325147":"2023-10-08T09:48:38.510621+0000","172.21.15.144:0/2288398636":"2023-10-08T09:48:38.510621+0000","172.21.15.144:6836/114714":"2023-10-08T09:48:38.510621+0000","172.21.15.144:6834/114714":"2023-10-08T09:48:38.510621+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 msgr2=0x7f0a80069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 0x7f0a80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 msgr2=0x7f0aa0169410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 0x7f0aa0169410 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400c560 tx=0x7f0a94007f20 comp rx=0 tx=0).stop 2023-10-07T10:44:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 shutdown_connections 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0a80067050 0x7f0a80069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aa015ade0 0x7f0aa0169950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00b0710 0x7f0aa0152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 --2- 172.21.15.29:0/1959290461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aa0153cc0 0x7f0aa0169410 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 >> 172.21.15.29:0/1959290461 conn(0x7f0aa0067610 msgr2=0x7f0aa00aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 shutdown_connections 2023-10-07T10:44:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.454+0000 7f0aa4dff700 1 -- 172.21.15.29:0/1959290461 wait complete. 2023-10-07T10:44:26.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T10:44:26.516 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:26.515+0000 7f4dba76e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:26.544 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:26.543+0000 7f6ff0e911c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:44:26.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.675+0000 7f5a19bff700 1 Processor -- start 2023-10-07T10:44:26.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.675+0000 7f5a19bff700 1 -- start start 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a19bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a140ad370 0x7f5a140cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a19bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a140b0410 0x7f5a1415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a19bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1414d790 con 0x7f5a14159ec0 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a19bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1415a880 con 0x7f5a140b0410 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a19bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140ae5d0 con 0x7f5a140ad370 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a13fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a140b0410 0x7f5a1415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a13fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a140b0410 0x7f5a1415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59050/0 (socket says 172.21.15.29:59050) 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a13fff700 1 -- 172.21.15.29:0/534800284 learned_addr learned my addr 172.21.15.29:0/534800284 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a12ffd700 1 --1- 172.21.15.29:0/534800284 >> v1:172.21.15.29:6789/0 conn(0x7f5a14159ec0 0x7f5a141568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56432/0 (socket says 172.21.15.29:56432) 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.677+0000 7f5a137fe700 1 --2- 172.21.15.29:0/534800284 >> v2:172.21.15.144:3300/0 conn(0x7f5a140ad370 0x7f5a140cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a13fff700 1 -- 172.21.15.29:0/534800284 >> v2:172.21.15.144:3300/0 conn(0x7f5a140ad370 msgr2=0x7f5a140cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a13fff700 1 --2- 172.21.15.29:0/534800284 >> v2:172.21.15.144:3300/0 conn(0x7f5a140ad370 0x7f5a140cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a13fff700 1 -- 172.21.15.29:0/534800284 >> v1:172.21.15.29:6789/0 conn(0x7f5a14159ec0 legacy=0x7f5a141568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a13fff700 1 -- 172.21.15.29:0/534800284 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140ad740 con 0x7f5a140b0410 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a127fc700 1 -- 172.21.15.29:0/534800284 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37751870 0 0) 0x7f5a1414d790 con 0x7f5a14159ec0 2023-10-07T10:44:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a13fff700 1 --2- 172.21.15.29:0/534800284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140b0410 0x7f5a1415a340 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009f20 tx=0x7f5a0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f62d81927533c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a127fc700 1 -- 172.21.15.29:0/534800284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00e070 con 0x7f5a140b0410 2023-10-07T10:44:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a127fc700 1 -- 172.21.15.29:0/534800284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00b990 con 0x7f5a140b0410 2023-10-07T10:44:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.678+0000 7f5a127fc700 1 -- 172.21.15.29:0/534800284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0092e0 con 0x7f5a140b0410 2023-10-07T10:44:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 -- 172.21.15.29:0/534800284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140b0410 msgr2=0x7f5a1415a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 --2- 172.21.15.29:0/534800284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140b0410 0x7f5a1415a340 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009f20 tx=0x7f5a0c009840 comp rx=0 tx=0).stop 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 -- 172.21.15.29:0/534800284 shutdown_connections 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 --2- 172.21.15.29:0/534800284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140b0410 0x7f5a1415a340 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 --2- 172.21.15.29:0/534800284 >> v2:172.21.15.144:3300/0 conn(0x7f5a140ad370 0x7f5a140cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 -- 172.21.15.29:0/534800284 >> 172.21.15.29:0/534800284 conn(0x7f5a140680a0 msgr2=0x7f5a140cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 -- 172.21.15.29:0/534800284 shutdown_connections 2023-10-07T10:44:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.679+0000 7f5a19bff700 1 -- 172.21.15.29:0/534800284 wait complete. 2023-10-07T10:44:26.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.681+0000 7f5a19bff700 1 Processor -- start 2023-10-07T10:44:26.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.682+0000 7f5a19bff700 1 -- start start 2023-10-07T10:44:26.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a19bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a140ad370 0x7f5a140aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a19bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a140b0410 0x7f5a140a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a19bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 0x7f5a140cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a19bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140a6be0 con 0x7f5a14159ec0 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a19bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1413f850 con 0x7f5a140ad370 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a19bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140a6e80 con 0x7f5a140b0410 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a13fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 0x7f5a140cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a13fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 0x7f5a140cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59056/0 (socket says 172.21.15.29:59056) 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a13fff700 1 -- 172.21.15.29:0/2457908558 learned_addr learned my addr 172.21.15.29:0/2457908558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a137fe700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a140ad370 0x7f5a140aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.683+0000 7f5a12ffd700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a140b0410 0x7f5a140a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a13fff700 1 -- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a140b0410 msgr2=0x7f5a140a66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a13fff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a140b0410 0x7f5a140a66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a13fff700 1 -- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a140ad370 msgr2=0x7f5a140aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a13fff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a140ad370 0x7f5a140aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a13fff700 1 -- 172.21.15.29:0/2457908558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c009ae0 con 0x7f5a14159ec0 2023-10-07T10:44:26.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a137fe700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a140ad370 0x7f5a140aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:26.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a13fff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 0x7f5a140cb120 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009ab0 tx=0x7f5a0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c007ac0 con 0x7f5a14159ec0 2023-10-07T10:44:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c007c40 con 0x7f5a14159ec0 2023-10-07T10:44:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a140a7160 con 0x7f5a14159ec0 2023-10-07T10:44:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.684+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c026460 con 0x7f5a14159ec0 2023-10-07T10:44:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.685+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a140a7720 con 0x7f5a14159ec0 2023-10-07T10:44:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.686+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1404c3c0 con 0x7f5a14159ec0 2023-10-07T10:44:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.687+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f5a0c01d420 con 0x7f5a14159ec0 2023-10-07T10:44:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.687+0000 7f59fb7fe700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 0x7f5a00069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.687+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6485+0+0 (secure 0 0 0) 0x7f5a0c086a40 con 0x7f5a14159ec0 2023-10-07T10:44:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.688+0000 7f5a137fe700 1 -- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 msgr2=0x7f5a00069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.688+0000 7f5a137fe700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 0x7f5a00069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:26.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.692+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a0c053790 con 0x7f5a14159ec0 2023-10-07T10:44:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.889+0000 7f5a137fe700 1 -- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 msgr2=0x7f5a00069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.889+0000 7f5a137fe700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 0x7f5a00069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:26.936 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:26.935+0000 7f6ff0e911c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:26.939 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:26.939+0000 7f4dba76e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:44:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.940+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5a1415a550 con 0x7f5a14159ec0 2023-10-07T10:44:26.942 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T10:44:26.941+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T10:44:26.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.946+0000 7f59fb7fe700 1 -- 172.21.15.29:0/2457908558 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5a0c056db0 con 0x7f5a14159ec0 2023-10-07T10:44:26.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 msgr2=0x7f5a00069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 0x7f5a00069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 msgr2=0x7f5a140cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 0x7f5a140cb120 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009ab0 tx=0x7f5a0c018000 comp rx=0 tx=0).stop 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 shutdown_connections 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5a00067050 0x7f5a00069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a14159ec0 0x7f5a140cb120 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a140ad370 0x7f5a140aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 --2- 172.21.15.29:0/2457908558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a140b0410 0x7f5a140a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.948+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 >> 172.21.15.29:0/2457908558 conn(0x7f5a140680a0 msgr2=0x7f5a140afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.949+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 shutdown_connections 2023-10-07T10:44:26.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:26.949+0000 7f5a19bff700 1 -- 172.21.15.29:0/2457908558 wait complete. 2023-10-07T10:44:26.951 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T10:44:26.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T10:44:27.035 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T10:44:27.125 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.124+0000 7f4dba76e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.179+0000 7f472bfff700 1 Processor -- start 2023-10-07T10:44:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.180+0000 7f472bfff700 1 -- start start 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f472bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f472413fba0 0x7f472406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f472bfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4724067d20 0x7f47240c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f472bfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240c78c0 con 0x7f472413ff70 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f472bfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472406fa50 con 0x7f472413fba0 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f472bfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472406c0f0 con 0x7f4724067d20 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f4729d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f472413fba0 0x7f472406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f4729d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f472413fba0 0x7f472406ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59072/0 (socket says 172.21.15.29:59072) 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f4729d9b700 1 -- 172.21.15.29:0/2479011285 learned_addr learned my addr 172.21.15.29:0/2479011285 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.182+0000 7f472959a700 1 --1- 172.21.15.29:0/2479011285 >> v1:172.21.15.29:6789/0 conn(0x7f472413ff70 0x7f4724067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56460/0 (socket says 172.21.15.29:56460) 2023-10-07T10:44:27.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f472a59c700 1 --2- 172.21.15.29:0/2479011285 >> v2:172.21.15.144:3300/0 conn(0x7f4724067d20 0x7f47240c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4729d9b700 1 -- 172.21.15.29:0/2479011285 >> v2:172.21.15.144:3300/0 conn(0x7f4724067d20 msgr2=0x7f47240c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4729d9b700 1 --2- 172.21.15.29:0/2479011285 >> v2:172.21.15.144:3300/0 conn(0x7f4724067d20 0x7f47240c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4729d9b700 1 -- 172.21.15.29:0/2479011285 >> v1:172.21.15.29:6789/0 conn(0x7f472413ff70 legacy=0x7f4724067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4729d9b700 1 -- 172.21.15.29:0/2479011285 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472406c3d0 con 0x7f472413fba0 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f472a59c700 1 --2- 172.21.15.29:0/2479011285 >> v2:172.21.15.144:3300/0 conn(0x7f4724067d20 0x7f47240c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4728d99700 1 -- 172.21.15.29:0/2479011285 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585563757 0 0) 0x7f47240c78c0 con 0x7f472413ff70 2023-10-07T10:44:27.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4729d9b700 1 --2- 172.21.15.29:0/2479011285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472406ba10 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f472406e090 tx=0x7f4720009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff055fc477931406 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:27.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4728d99700 1 -- 172.21.15.29:0/2479011285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000b8f0 con 0x7f472413fba0 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.183+0000 7f4728d99700 1 -- 172.21.15.29:0/2479011285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472000ba50 con 0x7f472413fba0 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f4728d99700 1 -- 172.21.15.29:0/2479011285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000bdf0 con 0x7f472413fba0 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 -- 172.21.15.29:0/2479011285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 msgr2=0x7f472406ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 --2- 172.21.15.29:0/2479011285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472406ba10 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f472406e090 tx=0x7f4720009250 comp rx=0 tx=0).stop 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 -- 172.21.15.29:0/2479011285 shutdown_connections 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 --2- 172.21.15.29:0/2479011285 >> v2:172.21.15.144:3300/0 conn(0x7f4724067d20 0x7f47240c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 --2- 172.21.15.29:0/2479011285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472406ba10 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 -- 172.21.15.29:0/2479011285 >> 172.21.15.29:0/2479011285 conn(0x7f472406b1d0 msgr2=0x7f47240cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 -- 172.21.15.29:0/2479011285 shutdown_connections 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.184+0000 7f472bfff700 1 -- 172.21.15.29:0/2479011285 wait complete. 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.186+0000 7f472bfff700 1 Processor -- start 2023-10-07T10:44:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.186+0000 7f472bfff700 1 -- start start 2023-10-07T10:44:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472bfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4724067d20 0x7f472413eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472415b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:27.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472413ff70 0x7f472412cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4724158950 con 0x7f472413fba0 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472bfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472415c630 con 0x7f4724067d20 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472bfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472413f080 con 0x7f472413ff70 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472959a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472415b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472959a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472415b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59086/0 (socket says 172.21.15.29:59086) 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f4729d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4724067d20 0x7f472413eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472959a700 1 -- 172.21.15.29:0/416432329 learned_addr learned my addr 172.21.15.29:0/416432329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.187+0000 7f472a59c700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472413ff70 0x7f472412cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472959a700 1 -- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472413ff70 msgr2=0x7f472412cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472959a700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472413ff70 0x7f472412cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472959a700 1 -- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4724067d20 msgr2=0x7f472413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:27.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472959a700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4724067d20 0x7f472413eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472959a700 1 -- 172.21.15.29:0/416432329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47200094f0 con 0x7f472413fba0 2023-10-07T10:44:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472959a700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472415b060 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f471400c5e0 tx=0x7f4714007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47140072e0 con 0x7f472413fba0 2023-10-07T10:44:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4714007440 con 0x7f472413fba0 2023-10-07T10:44:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472415ba20 con 0x7f472413fba0 2023-10-07T10:44:27.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47140057f0 con 0x7f472413fba0 2023-10-07T10:44:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.188+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472415c060 con 0x7f472413fba0 2023-10-07T10:44:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.189+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472404c3c0 con 0x7f472413fba0 2023-10-07T10:44:27.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.190+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f4714005be0 con 0x7f472413fba0 2023-10-07T10:44:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.192+0000 7f4719ffb700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 0x7f4708069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.192+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f4714083e50 con 0x7f472413fba0 2023-10-07T10:44:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.192+0000 7f4729d9b700 1 -- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 msgr2=0x7f4708069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:27.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.192+0000 7f4729d9b700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 0x7f4708069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:27.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.196+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4714050be0 con 0x7f472413fba0 2023-10-07T10:44:27.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.310+0000 7f4dba76e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:44:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.393+0000 7f4729d9b700 1 -- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 msgr2=0x7f4708069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.393+0000 7f4729d9b700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 0x7f4708069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:27.412 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:27.411+0000 7f6ff0e911c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:44:27.419 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.418+0000 7f4dba76e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.430+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 65) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f4714050fb0 con 0x7f472413fba0 2023-10-07T10:44:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.430+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 msgr2=0x7f4708069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.430+0000 7f4719ffb700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 0x7f4708069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.450+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f472415c770 con 0x7f472413fba0 2023-10-07T10:44:27.506 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.504+0000 7f4d33600700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.506 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.504+0000 7f4d33600700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.506 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.504+0000 7f4d33600700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.506 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.504+0000 7f4d33600700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.506 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.504+0000 7f4d33600700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.506+0000 7f4d37608700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.506+0000 7f4d37608700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.506+0000 7f4d37608700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.506+0000 7f4d37608700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:27.506+0000 7f4d37608700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:27.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:27.606+0000 7f6ff0e911c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:27.801 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:27.800+0000 7f6ff0e911c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:44:27.914 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:27.913+0000 7f6ff0e911c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:27.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.946+0000 7f4719ffb700 1 -- 172.21.15.29:0/416432329 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v70) v1 ==== 104+0+0 (secure 0 0 0) 0x7f471406ef60 con 0x7f472413fba0 2023-10-07T10:44:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 msgr2=0x7f472415b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472415b060 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f471400c5e0 tx=0x7f4714007f10 comp rx=0 tx=0).stop 2023-10-07T10:44:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 shutdown_connections 2023-10-07T10:44:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4708067050 0x7f4708069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f472413ff70 0x7f472412cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4724067d20 0x7f472413eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 --2- 172.21.15.29:0/416432329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f472413fba0 0x7f472415b060 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 >> 172.21.15.29:0/416432329 conn(0x7f472406b1d0 msgr2=0x7f47240ca220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 shutdown_connections 2023-10-07T10:44:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:27.949+0000 7f472bfff700 1 -- 172.21.15.29:0/416432329 wait complete. 2023-10-07T10:44:27.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T10:44:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.181+0000 7f17c7bff700 1 Processor -- start 2023-10-07T10:44:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.181+0000 7f17c7bff700 1 -- start start 2023-10-07T10:44:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.182+0000 7f17c7bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17c0157650 0x7f17c00ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.182+0000 7f17c7bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c00ca430 con 0x7f17c0157a20 2023-10-07T10:44:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.182+0000 7f17c7bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c00ca710 con 0x7f17c00ae910 2023-10-07T10:44:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.182+0000 7f17c7bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c00ca9f0 con 0x7f17c0157650 2023-10-07T10:44:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c619c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f17c0157a20 0x7f17c00cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56462/0 (socket says 172.21.15.29:56462) 2023-10-07T10:44:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c619c700 1 -- 172.21.15.29:0/3552004310 learned_addr learned my addr 172.21.15.29:0/3552004310 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c599b700 1 --2- 172.21.15.29:0/3552004310 >> v2:172.21.15.144:3300/0 conn(0x7f17c0157650 0x7f17c00ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017529734 0 0) 0x7f17c00ca430 con 0x7f17c0157a20 2023-10-07T10:44:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac003570 con 0x7f17c0157a20 2023-10-07T10:44:28.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244440509 0 0) 0x7f17c00ca710 con 0x7f17c00ae910 2023-10-07T10:44:28.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c00ca430 con 0x7f17c00ae910 2023-10-07T10:44:28.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145640009 0 0) 0x7f17ac003570 con 0x7f17c0157a20 2023-10-07T10:44:28.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 >> v2:172.21.15.144:3300/0 conn(0x7f17c0157650 msgr2=0x7f17c00ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:28.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 --2- 172.21.15.29:0/3552004310 >> v2:172.21.15.144:3300/0 conn(0x7f17c0157650 0x7f17c00ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 >> v1:172.21.15.144:6789/0 conn(0x7f17c00ae910 legacy=0x7f17c00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:28.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c00cacd0 con 0x7f17c0157a20 2023-10-07T10:44:28.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f17b80017f0 con 0x7f17c0157a20 2023-10-07T10:44:28.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.183+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 >> v1:172.21.15.29:6789/0 conn(0x7f17c0157a20 legacy=0x7f17c00cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:28.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c4999700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17ac003c40 0x7f17ac004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c4999700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac004810 0x7f17ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c4999700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17ac005280 0x7f17ac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c00ca9f0 con 0x7f17ac003c40 2023-10-07T10:44:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac00c250 con 0x7f17ac004810 2023-10-07T10:44:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac00c410 con 0x7f17ac005280 2023-10-07T10:44:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c599b700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17ac003c40 0x7f17ac004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c619c700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17ac005280 0x7f17ac00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c519a700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac004810 0x7f17ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c599b700 1 -- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17ac005280 msgr2=0x7f17ac00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:28.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c599b700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17ac005280 0x7f17ac00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c599b700 1 -- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac004810 msgr2=0x7f17ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:28.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c599b700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac004810 0x7f17ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c599b700 1 -- 172.21.15.29:0/3552004310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac003570 con 0x7f17ac003c40 2023-10-07T10:44:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.184+0000 7f17c619c700 1 -- 172.21.15.29:0/3552004310 reap_dead start 2023-10-07T10:44:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.185+0000 7f17c599b700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17ac003c40 0x7f17ac004030 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009c00 tx=0x7f17bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469f88168cc2290b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.185+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00e710 con 0x7f17ac003c40 2023-10-07T10:44:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.185+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc00e870 con 0x7f17ac003c40 2023-10-07T10:44:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.185+0000 7f17c4999700 1 -- 172.21.15.29:0/3552004310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00ec30 con 0x7f17ac003c40 2023-10-07T10:44:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.185+0000 7f17c7bff700 1 -- 172.21.15.29:0/3552004310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17ac003c40 msgr2=0x7f17ac004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.185+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17ac003c40 0x7f17ac004030 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009c00 tx=0x7f17bc0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.186+0000 7f17c7bff700 1 -- 172.21.15.29:0/3552004310 shutdown_connections 2023-10-07T10:44:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.186+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3552004310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17ac003c40 0x7f17ac004030 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.186+0000 7f17c7bff700 1 -- 172.21.15.29:0/3552004310 >> 172.21.15.29:0/3552004310 conn(0x7f17c0067610 msgr2=0x7f17c0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.186+0000 7f17c7bff700 1 -- 172.21.15.29:0/3552004310 shutdown_connections 2023-10-07T10:44:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.186+0000 7f17c7bff700 1 -- 172.21.15.29:0/3552004310 wait complete. 2023-10-07T10:44:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.188+0000 7f17c7bff700 1 Processor -- start 2023-10-07T10:44:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.188+0000 7f17c7bff700 1 -- start start 2023-10-07T10:44:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c7bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac003c40 0x7f17c0159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c7bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 0x7f17c00aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c7bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c0168a80 0x7f17c0168f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c7bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0159770 con 0x7f17c00aab30 2023-10-07T10:44:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c7bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0158190 con 0x7f17ac003c40 2023-10-07T10:44:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c7bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0158470 con 0x7f17c0168a80 2023-10-07T10:44:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 0x7f17c00aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 0x7f17c00aaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59106/0 (socket says 172.21.15.29:59106) 2023-10-07T10:44:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.189+0000 7f17c519a700 1 -- 172.21.15.29:0/3354508513 learned_addr learned my addr 172.21.15.29:0/3354508513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c599b700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac003c40 0x7f17c0159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c619c700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c0168a80 0x7f17c0168f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c519a700 1 -- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c0168a80 msgr2=0x7f17c0168f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c519a700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c0168a80 0x7f17c0168f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c519a700 1 -- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac003c40 msgr2=0x7f17c0159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c519a700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac003c40 0x7f17c0159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c519a700 1 -- 172.21.15.29:0/3354508513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c0158750 con 0x7f17c00aab30 2023-10-07T10:44:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c619c700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c0168a80 0x7f17c0168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c519a700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 0x7f17c00aaf80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f17b00010e0 tx=0x7f17b0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0012590 con 0x7f17c00aab30 2023-10-07T10:44:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b0012710 con 0x7f17c00aab30 2023-10-07T10:44:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0012ad0 con 0x7f17c00aab30 2023-10-07T10:44:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.190+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c0158a30 con 0x7f17c00aab30 2023-10-07T10:44:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.191+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c0157e20 con 0x7f17c00aab30 2023-10-07T10:44:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.192+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c00718b0 con 0x7f17c00aab30 2023-10-07T10:44:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.193+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f17b001d460 con 0x7f17c00aab30 2023-10-07T10:44:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.193+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6785+0+0 (secure 0 0 0) 0x7f17b0084760 con 0x7f17c00aab30 2023-10-07T10:44:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.198+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17b0051860 con 0x7f17c00aab30 2023-10-07T10:44:28.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.398+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17c00cafb0 con 0x7f17c00aab30 2023-10-07T10:44:28.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.434+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 66) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f17b0051310 con 0x7f17c00aab30 2023-10-07T10:44:28.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.434+0000 7f17b5ffb700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f17a406cae0 0x7f17a406eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:28.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.435+0000 7f17c599b700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f17a406cae0 0x7f17a406eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:28.436+0000 7f17c599b700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f17a406cae0 0x7f17a406eec0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0091e0 tx=0x7f17bc01a000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.061+0000 7f17b5ffb700 1 -- 172.21.15.29:0/3354508513 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) v1 ==== 181+0+0 (secure 0 0 0) 0x7f17b006f670 con 0x7f17c00aab30 2023-10-07T10:44:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f17a406cae0 msgr2=0x7f17a406eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f17a406cae0 0x7f17a406eec0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0091e0 tx=0x7f17bc01a000 comp rx=0 tx=0).stop 2023-10-07T10:44:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 msgr2=0x7f17c00aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 0x7f17c00aaf80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f17b00010e0 tx=0x7f17b0009040 comp rx=0 tx=0).stop 2023-10-07T10:44:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 shutdown_connections 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f17a406cae0 0x7f17a406eec0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c0168a80 0x7f17c0168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17ac003c40 0x7f17c0159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 --2- 172.21.15.29:0/3354508513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c00aab30 0x7f17c00aaf80 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 >> 172.21.15.29:0/3354508513 conn(0x7f17c0067610 msgr2=0x7f17c0152ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.065+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 shutdown_connections 2023-10-07T10:44:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.066+0000 7f17c7bff700 1 -- 172.21.15.29:0/3354508513 wait complete. 2023-10-07T10:44:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T10:44:29.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T10:44:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.298+0000 7fa4f7fff700 1 Processor -- start 2023-10-07T10:44:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.299+0000 7fa4f7fff700 1 -- start start 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f7fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa4f00ced00 0x7fa4f00b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00c8a20 0x7fa4f00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f7fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f00690f0 con 0x7fa4f00b0610 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f7fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f00cf2e0 con 0x7fa4f00ced00 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f00687c0 con 0x7fa4f00c8a20 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f5d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa4f00b0610 0x7fa4f00c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56464/0 (socket says 172.21.15.29:56464) 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f5d9b700 1 -- 172.21.15.29:0/2389004493 learned_addr learned my addr 172.21.15.29:0/2389004493 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.300+0000 7fa4f559a700 1 --2- 172.21.15.29:0/2389004493 >> v2:172.21.15.144:3300/0 conn(0x7fa4f00ced00 0x7fa4f00b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f659c700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00c8a20 0x7fa4f00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f4d99700 1 -- 172.21.15.29:0/2389004493 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131434628 0 0) 0x7fa4f00690f0 con 0x7fa4f00b0610 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f4d99700 1 -- 172.21.15.29:0/2389004493 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc003570 con 0x7fa4f00b0610 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f559a700 1 -- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00c8a20 msgr2=0x7fa4f00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f559a700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00c8a20 0x7fa4f00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f559a700 1 -- 172.21.15.29:0/2389004493 >> v1:172.21.15.29:6789/0 conn(0x7fa4f00b0610 legacy=0x7fa4f00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f559a700 1 -- 172.21.15.29:0/2389004493 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f0068aa0 con 0x7fa4f00ced00 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f659c700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00c8a20 0x7fa4f00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.301+0000 7fa4f559a700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00ced00 0x7fa4f00b09e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00098e0 tx=0x7fa4e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5b6c5a28dfb16a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:29.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f4d99700 1 -- 172.21.15.29:0/2389004493 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0011630 con 0x7fa4f00ced00 2023-10-07T10:44:29.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f4d99700 1 -- 172.21.15.29:0/2389004493 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e0011790 con 0x7fa4f00ced00 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f4d99700 1 -- 172.21.15.29:0/2389004493 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0011b30 con 0x7fa4f00ced00 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 -- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00ced00 msgr2=0x7fa4f00b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00ced00 0x7fa4f00b09e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00098e0 tx=0x7fa4e00091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 -- 172.21.15.29:0/2389004493 shutdown_connections 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00c8a20 0x7fa4f00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/2389004493 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00ced00 0x7fa4f00b09e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 -- 172.21.15.29:0/2389004493 >> 172.21.15.29:0/2389004493 conn(0x7fa4f0063a10 msgr2=0x7fa4f00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 -- 172.21.15.29:0/2389004493 shutdown_connections 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.302+0000 7fa4f7fff700 1 -- 172.21.15.29:0/2389004493 wait complete. 2023-10-07T10:44:29.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.304+0000 7fa4f7fff700 1 Processor -- start 2023-10-07T10:44:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.304+0000 7fa4f7fff700 1 -- start start 2023-10-07T10:44:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 0x7fa4f0143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f7fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00c8a20 0x7fa4f0158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f7fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00ced00 0x7fa4f00b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f7fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f015bad0 con 0x7fa4f00b0610 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f7fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0158570 con 0x7fa4f00c8a20 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f7fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f015aec0 con 0x7fa4f00ced00 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 0x7fa4f0143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 0x7fa4f0143a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59108/0 (socket says 172.21.15.29:59108) 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.305+0000 7fa4f5d9b700 1 -- 172.21.15.29:0/793301393 learned_addr learned my addr 172.21.15.29:0/793301393 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f559a700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00c8a20 0x7fa4f0158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f659c700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00ced00 0x7fa4f00b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f5d9b700 1 -- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00ced00 msgr2=0x7fa4f00b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f5d9b700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00ced00 0x7fa4f00b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f5d9b700 1 -- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00c8a20 msgr2=0x7fa4f0158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f5d9b700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00c8a20 0x7fa4f0158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f5d9b700 1 -- 172.21.15.29:0/793301393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8009a20 con 0x7fa4f00b0610 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f559a700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00c8a20 0x7fa4f0158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f5d9b700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 0x7fa4f0143a40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c590 tx=0x7fa4ec007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec005b30 con 0x7fa4f00b0610 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec005cb0 con 0x7fa4f00b0610 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec007320 con 0x7fa4f00b0610 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e0009420 con 0x7fa4f00b0610 2023-10-07T10:44:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.306+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f015b480 con 0x7fa4f00b0610 2023-10-07T10:44:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.307+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f00718b0 con 0x7fa4f00b0610 2023-10-07T10:44:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.308+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 66) v1 ==== 84578+0+0 (secure 0 0 0) 0x7fa4ec01b3b0 con 0x7fa4f00b0610 2023-10-07T10:44:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.309+0000 7fa4e5ffb700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d4066d80 0x7fa4d4069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.309+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6347+0+0 (secure 0 0 0) 0x7fa4ec083af0 con 0x7fa4f00b0610 2023-10-07T10:44:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.310+0000 7fa4f559a700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d4066d80 0x7fa4d4069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.311+0000 7fa4f559a700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d4066d80 0x7fa4d4069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00098e0 tx=0x7fa4e0017000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:29.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.314+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4ec050cc0 con 0x7fa4f00b0610 2023-10-07T10:44:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:29.518+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa4f0068d80 con 0x7fa4f00b0610 2023-10-07T10:44:30.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.071+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa4ec051090 con 0x7fa4f00b0610 2023-10-07T10:44:30.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.441+0000 7fa4e5ffb700 1 -- 172.21.15.29:0/793301393 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa4ec054750 con 0x7fa4f00b0610 2023-10-07T10:44:30.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d4066d80 msgr2=0x7fa4d4069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d4066d80 0x7fa4d4069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00098e0 tx=0x7fa4e0017000 comp rx=0 tx=0).stop 2023-10-07T10:44:30.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 msgr2=0x7fa4f0143a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 0x7fa4f0143a40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c590 tx=0x7fa4ec007e00 comp rx=0 tx=0).stop 2023-10-07T10:44:30.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 shutdown_connections 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d4066d80 0x7fa4d4069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4f00ced00 0x7fa4f00b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4f00b0610 0x7fa4f0143a40 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 --2- 172.21.15.29:0/793301393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f00c8a20 0x7fa4f0158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 >> 172.21.15.29:0/793301393 conn(0x7fa4f0063a10 msgr2=0x7fa4f00c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 shutdown_connections 2023-10-07T10:44:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.446+0000 7fa4f7fff700 1 -- 172.21.15.29:0/793301393 wait complete. 2023-10-07T10:44:30.450 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T10:44:30.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T10:44:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.671+0000 7fcc399ff700 1 Processor -- start 2023-10-07T10:44:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.671+0000 7fcc399ff700 1 -- start start 2023-10-07T10:44:30.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.672+0000 7fcc399ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc34159750 0x7fcc340d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:30.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.672+0000 7fcc399ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 0x7fcc340c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:30.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.672+0000 7fcc399ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc340cbf20 con 0x7fcc34159b20 2023-10-07T10:44:30.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.672+0000 7fcc399ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc340cc200 con 0x7fcc34159750 2023-10-07T10:44:30.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.672+0000 7fcc399ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340cc4e0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc32ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc34159750 0x7fcc340d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 0x7fcc340c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 0x7fcc340c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:54014/0 (socket says 172.21.15.29:54014) 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 -- 172.21.15.29:0/2188493062 learned_addr learned my addr 172.21.15.29:0/2188493062 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc327fc700 1 --1- 172.21.15.29:0/2188493062 >> v1:172.21.15.144:6789/0 conn(0x7fcc34159b20 0x7fcc340ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:49222/0 (socket says 172.21.15.29:49222) 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 -- 172.21.15.29:0/2188493062 >> v2:172.21.15.144:3300/0 conn(0x7fcc34159750 msgr2=0x7fcc340d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 --2- 172.21.15.29:0/2188493062 >> v2:172.21.15.144:3300/0 conn(0x7fcc34159750 0x7fcc340d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 -- 172.21.15.29:0/2188493062 >> v1:172.21.15.144:6789/0 conn(0x7fcc34159b20 legacy=0x7fcc340ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc337fe700 1 -- 172.21.15.29:0/2188493062 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc340cc7c0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.673+0000 7fcc32ffd700 1 --2- 172.21.15.29:0/2188493062 >> v2:172.21.15.144:3300/0 conn(0x7fcc34159750 0x7fcc340d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.674+0000 7fcc337fe700 1 --2- 172.21.15.29:0/2188493062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 0x7fcc340c5460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009f20 tx=0x7fcc2c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67c03a085b4490fa server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.674+0000 7fcc31ffb700 1 -- 172.21.15.29:0/2188493062 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c00e070 con 0x7fcc340ce8a0 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.674+0000 7fcc31ffb700 1 -- 172.21.15.29:0/2188493062 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2c00b9d0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.674+0000 7fcc31ffb700 1 -- 172.21.15.29:0/2188493062 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c0092e0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.674+0000 7fcc399ff700 1 -- 172.21.15.29:0/2188493062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 msgr2=0x7fcc340c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.674+0000 7fcc399ff700 1 --2- 172.21.15.29:0/2188493062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 0x7fcc340c5460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009f20 tx=0x7fcc2c009840 comp rx=0 tx=0).stop 2023-10-07T10:44:30.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.675+0000 7fcc399ff700 1 -- 172.21.15.29:0/2188493062 shutdown_connections 2023-10-07T10:44:30.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.675+0000 7fcc399ff700 1 --2- 172.21.15.29:0/2188493062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc340ce8a0 0x7fcc340c5460 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.675+0000 7fcc399ff700 1 --2- 172.21.15.29:0/2188493062 >> v2:172.21.15.144:3300/0 conn(0x7fcc34159750 0x7fcc340d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.675+0000 7fcc399ff700 1 -- 172.21.15.29:0/2188493062 >> 172.21.15.29:0/2188493062 conn(0x7fcc34063a10 msgr2=0x7fcc340d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:30.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.675+0000 7fcc399ff700 1 -- 172.21.15.29:0/2188493062 shutdown_connections 2023-10-07T10:44:30.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.675+0000 7fcc399ff700 1 -- 172.21.15.29:0/2188493062 wait complete. 2023-10-07T10:44:30.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.676+0000 7fcc399ff700 1 Processor -- start 2023-10-07T10:44:30.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.676+0000 7fcc399ff700 1 -- start start 2023-10-07T10:44:30.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.677+0000 7fcc399ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 0x7fcc34150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc399ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc34159750 0x7fcc34068f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc399ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34159b20 0x7fcc340694c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc399ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc341512d0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc399ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340c8000 con 0x7fcc34159750 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc399ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340c7710 con 0x7fcc34159b20 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 0x7fcc34150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 0x7fcc34150d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59112/0 (socket says 172.21.15.29:59112) 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 -- 172.21.15.29:0/298795432 learned_addr learned my addr 172.21.15.29:0/298795432 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc327fc700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc34159750 0x7fcc34068f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc337fe700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34159b20 0x7fcc340694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:30.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 -- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34159b20 msgr2=0x7fcc340694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34159b20 0x7fcc340694c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 -- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc34159750 msgr2=0x7fcc34068f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc34159750 0x7fcc34068f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.678+0000 7fcc32ffd700 1 -- 172.21.15.29:0/298795432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc240093c0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.679+0000 7fcc32ffd700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 0x7fcc34150d90 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400ec80 tx=0x7fcc24009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:30.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.679+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24005a00 con 0x7fcc340ce8a0 2023-10-07T10:44:30.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.679+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24005b60 con 0x7fcc340ce8a0 2023-10-07T10:44:30.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.679+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2400c9d0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.680+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c009ae0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.680+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc340c7c90 con 0x7fcc340ce8a0 2023-10-07T10:44:30.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.682+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3404c3c0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.683+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc2400cd80 con 0x7fcc340ce8a0 2023-10-07T10:44:30.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.685+0000 7fcc1affd700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc20067000 0x7fcc200694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:30.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.685+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7fcc24084b90 con 0x7fcc340ce8a0 2023-10-07T10:44:30.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.685+0000 7fcc327fc700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc20067000 0x7fcc200694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:30.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.686+0000 7fcc327fc700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc20067000 0x7fcc200694b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc28007fd0 tx=0x7fcc28007d70 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:30.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.689+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc24051dd0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.890+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fcc34064040 con 0x7fcc340ce8a0 2023-10-07T10:44:30.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.894+0000 7fcc1affd700 1 -- 172.21.15.29:0/298795432 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcc240553f0 con 0x7fcc340ce8a0 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.896+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc20067000 msgr2=0x7fcc200694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc20067000 0x7fcc200694b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc28007fd0 tx=0x7fcc28007d70 comp rx=0 tx=0).stop 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 msgr2=0x7fcc34150d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 0x7fcc34150d90 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400ec80 tx=0x7fcc24009120 comp rx=0 tx=0).stop 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 shutdown_connections 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc20067000 0x7fcc200694b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc34159b20 0x7fcc340694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc340ce8a0 0x7fcc34150d90 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 --2- 172.21.15.29:0/298795432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc34159750 0x7fcc34068f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:30.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 >> 172.21.15.29:0/298795432 conn(0x7fcc34063a10 msgr2=0x7fcc340ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:30.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 shutdown_connections 2023-10-07T10:44:30.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:30.897+0000 7fcc399ff700 1 -- 172.21.15.29:0/298795432 wait complete. 2023-10-07T10:44:30.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T10:44:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.112+0000 7f34b21ff700 1 Processor -- start 2023-10-07T10:44:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.112+0000 7f34b21ff700 1 -- start start 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.113+0000 7f34b21ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34ac15ade0 0x7f34ac0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.113+0000 7f34b21ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f34ac0b0540 0x7f34ac15c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.113+0000 7f34b21ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac154090 con 0x7f34ac153cc0 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.113+0000 7f34b21ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac0b09e0 con 0x7f34ac15ade0 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.113+0000 7f34b21ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac159f60 con 0x7f34ac0b0540 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34ac15ade0 0x7f34ac0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34ac15ade0 0x7f34ac0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59120/0 (socket says 172.21.15.29:59120) 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 -- 172.21.15.29:0/815762532 learned_addr learned my addr 172.21.15.29:0/815762532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34abfff700 1 --2- 172.21.15.29:0/815762532 >> v2:172.21.15.144:3300/0 conn(0x7f34ac0b0540 0x7f34ac15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 -- 172.21.15.29:0/815762532 >> v2:172.21.15.144:3300/0 conn(0x7f34ac0b0540 msgr2=0x7f34ac15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 --2- 172.21.15.29:0/815762532 >> v2:172.21.15.144:3300/0 conn(0x7f34ac0b0540 0x7f34ac15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 -- 172.21.15.29:0/815762532 >> v1:172.21.15.144:6789/0 conn(0x7f34ac153cc0 legacy=0x7f34ac0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.114+0000 7f34aaffd700 1 -- 172.21.15.29:0/815762532 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac15a240 con 0x7f34ac15ade0 2023-10-07T10:44:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.115+0000 7f34aaffd700 1 --2- 172.21.15.29:0/815762532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac15ade0 0x7f34ac0b0000 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f34ac06c400 tx=0x7f34a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8589477ad944ad6d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.115+0000 7f34aa7fc700 1 -- 172.21.15.29:0/815762532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0009e70 con 0x7f34ac15ade0 2023-10-07T10:44:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.115+0000 7f34aa7fc700 1 -- 172.21.15.29:0/815762532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a000b860 con 0x7f34ac15ade0 2023-10-07T10:44:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.115+0000 7f34aa7fc700 1 -- 172.21.15.29:0/815762532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000bc00 con 0x7f34ac15ade0 2023-10-07T10:44:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.115+0000 7f34b21ff700 1 -- 172.21.15.29:0/815762532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac15ade0 msgr2=0x7f34ac0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.115+0000 7f34b21ff700 1 --2- 172.21.15.29:0/815762532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac15ade0 0x7f34ac0b0000 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f34ac06c400 tx=0x7f34a0009250 comp rx=0 tx=0).stop 2023-10-07T10:44:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.116+0000 7f34b21ff700 1 -- 172.21.15.29:0/815762532 shutdown_connections 2023-10-07T10:44:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.116+0000 7f34b21ff700 1 --2- 172.21.15.29:0/815762532 >> v2:172.21.15.144:3300/0 conn(0x7f34ac0b0540 0x7f34ac15c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.116+0000 7f34b21ff700 1 --2- 172.21.15.29:0/815762532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac15ade0 0x7f34ac0b0000 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.116+0000 7f34b21ff700 1 -- 172.21.15.29:0/815762532 >> 172.21.15.29:0/815762532 conn(0x7f34ac067610 msgr2=0x7f34ac067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.116+0000 7f34b21ff700 1 -- 172.21.15.29:0/815762532 shutdown_connections 2023-10-07T10:44:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.116+0000 7f34b21ff700 1 -- 172.21.15.29:0/815762532 wait complete. 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.117+0000 7f34b21ff700 1 Processor -- start 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.117+0000 7f34b21ff700 1 -- start start 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34b21ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ac0b0540 0x7f34ac13dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34b21ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 0x7f34ac140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34b21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34ac15ade0 0x7f34ac0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34b21ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac140af0 con 0x7f34ac153cc0 2023-10-07T10:44:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34b21ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac13fee0 con 0x7f34ac0b0540 2023-10-07T10:44:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34b21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac1401c0 con 0x7f34ac15ade0 2023-10-07T10:44:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34aaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 0x7f34ac140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34aaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 0x7f34ac140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59124/0 (socket says 172.21.15.29:59124) 2023-10-07T10:44:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.118+0000 7f34aaffd700 1 -- 172.21.15.29:0/494616854 learned_addr learned my addr 172.21.15.29:0/494616854 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34abfff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34ac15ade0 0x7f34ac0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34ab7fe700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ac0b0540 0x7f34ac13dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34aaffd700 1 -- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34ac15ade0 msgr2=0x7f34ac0b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34aaffd700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34ac15ade0 0x7f34ac0b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34aaffd700 1 -- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ac0b0540 msgr2=0x7f34ac13dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34aaffd700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ac0b0540 0x7f34ac13dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34aaffd700 1 -- 172.21.15.29:0/494616854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00094f0 con 0x7f34ac153cc0 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34abfff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34ac15ade0 0x7f34ac0b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34ab7fe700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ac0b0540 0x7f34ac13dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34aaffd700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 0x7f34ac140f80 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0b0be0 tx=0x7f34a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.119+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0011420 con 0x7f34ac153cc0 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.120+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a00115a0 con 0x7f34ac153cc0 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.120+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0011960 con 0x7f34ac153cc0 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.120+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac1404a0 con 0x7f34ac153cc0 2023-10-07T10:44:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.120+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac13e2d0 con 0x7f34ac153cc0 2023-10-07T10:44:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.121+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac140c60 con 0x7f34ac153cc0 2023-10-07T10:44:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.121+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f34a00253f0 con 0x7f34ac153cc0 2023-10-07T10:44:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.122+0000 7f34937fe700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3498066f30 0x7f34980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.122+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f34a0085f30 con 0x7f34ac153cc0 2023-10-07T10:44:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.123+0000 7f34ab7fe700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3498066f30 0x7f34980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.124+0000 7f34ab7fe700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3498066f30 0x7f34980693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f349c0018f0 tx=0x7f349c009040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:31.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.126+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34a00530f0 con 0x7f34ac153cc0 2023-10-07T10:44:31.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.336+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f34ac04c3c0 con 0x7f34ac153cc0 2023-10-07T10:44:31.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.340+0000 7f34937fe700 1 -- 172.21.15.29:0/494616854 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34a0056710 con 0x7f34ac153cc0 2023-10-07T10:44:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3498066f30 msgr2=0x7f34980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3498066f30 0x7f34980693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f349c0018f0 tx=0x7f349c009040 comp rx=0 tx=0).stop 2023-10-07T10:44:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 msgr2=0x7f34ac140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 0x7f34ac140f80 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0b0be0 tx=0x7f34a0017000 comp rx=0 tx=0).stop 2023-10-07T10:44:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 shutdown_connections 2023-10-07T10:44:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3498066f30 0x7f34980693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34ac15ade0 0x7f34ac0b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34ac0b0540 0x7f34ac13dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 --2- 172.21.15.29:0/494616854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34ac153cc0 0x7f34ac140f80 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 >> 172.21.15.29:0/494616854 conn(0x7f34ac067610 msgr2=0x7f34ac069d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 shutdown_connections 2023-10-07T10:44:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.343+0000 7f34b21ff700 1 -- 172.21.15.29:0/494616854 wait complete. 2023-10-07T10:44:31.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T10:44:31.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.559+0000 7f2d911ff700 1 Processor -- start 2023-10-07T10:44:31.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.559+0000 7f2d911ff700 1 -- start start 2023-10-07T10:44:31.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.560+0000 7f2d911ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d8c0ae8a0 0x7f2d8c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.560+0000 7f2d911ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c15ad30 0x7f2d8c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.560+0000 7f2d911ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c0acb40 con 0x7f2d8c15a960 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.560+0000 7f2d911ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c158c60 con 0x7f2d8c0ae8a0 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.560+0000 7f2d911ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0abf30 con 0x7f2d8c15ad30 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d8c0ae8a0 0x7f2d8c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d8c0ae8a0 0x7f2d8c158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59128/0 (socket says 172.21.15.29:59128) 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 -- 172.21.15.29:0/773805423 learned_addr learned my addr 172.21.15.29:0/773805423 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8ad9d700 1 --1- 172.21.15.29:0/773805423 >> v1:172.21.15.29:6789/0 conn(0x7f2d8c15a960 0x7f2d8c0ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56476/0 (socket says 172.21.15.29:56476) 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8b59e700 1 --2- 172.21.15.29:0/773805423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c15ad30 0x7f2d8c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 -- 172.21.15.29:0/773805423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c15ad30 msgr2=0x7f2d8c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 --2- 172.21.15.29:0/773805423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c15ad30 0x7f2d8c0b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 -- 172.21.15.29:0/773805423 >> v1:172.21.15.29:6789/0 conn(0x7f2d8c15a960 legacy=0x7f2d8c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 -- 172.21.15.29:0/773805423 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0ac210 con 0x7f2d8c0ae8a0 2023-10-07T10:44:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d89d9b700 1 -- 172.21.15.29:0/773805423 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820976020 0 0) 0x7f2d8c0acb40 con 0x7f2d8c15a960 2023-10-07T10:44:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.561+0000 7f2d8a59c700 1 --2- 172.21.15.29:0/773805423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c0ae8a0 0x7f2d8c158e20 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2d800099b0 tx=0x7f2d80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c573b97d4ecffd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d89d9b700 1 -- 172.21.15.29:0/773805423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000b860 con 0x7f2d8c0ae8a0 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d89d9b700 1 -- 172.21.15.29:0/773805423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8000b9c0 con 0x7f2d8c0ae8a0 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d89d9b700 1 -- 172.21.15.29:0/773805423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000bd60 con 0x7f2d8c0ae8a0 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 -- 172.21.15.29:0/773805423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c0ae8a0 msgr2=0x7f2d8c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 --2- 172.21.15.29:0/773805423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c0ae8a0 0x7f2d8c158e20 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2d800099b0 tx=0x7f2d80009250 comp rx=0 tx=0).stop 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 -- 172.21.15.29:0/773805423 shutdown_connections 2023-10-07T10:44:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 --2- 172.21.15.29:0/773805423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c15ad30 0x7f2d8c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 --2- 172.21.15.29:0/773805423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c0ae8a0 0x7f2d8c158e20 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 -- 172.21.15.29:0/773805423 >> 172.21.15.29:0/773805423 conn(0x7f2d8c067610 msgr2=0x7f2d8c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 -- 172.21.15.29:0/773805423 shutdown_connections 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.562+0000 7f2d911ff700 1 -- 172.21.15.29:0/773805423 wait complete. 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.564+0000 7f2d911ff700 1 Processor -- start 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.564+0000 7f2d911ff700 1 -- start start 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d911ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c0ae8a0 0x7f2d8c0a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d911ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8c15a960 0x7f2d8c146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d911ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 0x7f2d8c0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d911ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0aff00 con 0x7f2d8c15ad30 2023-10-07T10:44:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d911ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c1500d0 con 0x7f2d8c15a960 2023-10-07T10:44:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d911ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0afc20 con 0x7f2d8c0ae8a0 2023-10-07T10:44:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d8b59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 0x7f2d8c0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d8b59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 0x7f2d8c0af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59136/0 (socket says 172.21.15.29:59136) 2023-10-07T10:44:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d8b59e700 1 -- 172.21.15.29:0/102056381 learned_addr learned my addr 172.21.15.29:0/102056381 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d8b59e700 1 -- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c0ae8a0 msgr2=0x7f2d8c0a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.565+0000 7f2d8b59e700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c0ae8a0 0x7f2d8c0a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d8b59e700 1 -- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8c15a960 msgr2=0x7f2d8c146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d8b59e700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8c15a960 0x7f2d8c146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d8b59e700 1 -- 172.21.15.29:0/102056381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d800094f0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d8b59e700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 0x7f2d8c0af080 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f2d84007ef0 tx=0x7f2d84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8400ec10 con 0x7f2d8c15ad30 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8400ed70 con 0x7f2d8c15ad30 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d84014670 con 0x7f2d8c15ad30 2023-10-07T10:44:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0a55f0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.566+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c0a5bb0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.567+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c04c3c0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.568+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2d84014a60 con 0x7f2d8c15ad30 2023-10-07T10:44:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.569+0000 7f2d72ffd700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d78066f30 0x7f2d780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.569+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f2d84084dd0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.570+0000 7f2d8ad9d700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d78066f30 0x7f2d780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.570+0000 7f2d8ad9d700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d78066f30 0x7f2d780693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0011e0 tx=0x7f2d7c0079b0 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.573+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d84051ef0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.776+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2d8c0a5da0 con 0x7f2d8c15ad30 2023-10-07T10:44:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.780+0000 7f2d72ffd700 1 -- 172.21.15.29:0/102056381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2d84055510 con 0x7f2d8c15ad30 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.782+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d78066f30 msgr2=0x7f2d780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.782+0000 7f2d911ff700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d78066f30 0x7f2d780693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0011e0 tx=0x7f2d7c0079b0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 msgr2=0x7f2d8c0af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 0x7f2d8c0af080 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f2d84007ef0 tx=0x7f2d84009840 comp rx=0 tx=0).stop 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 shutdown_connections 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2d78066f30 0x7f2d780693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d8c15ad30 0x7f2d8c0af080 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8c0ae8a0 0x7f2d8c0a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 --2- 172.21.15.29:0/102056381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8c15a960 0x7f2d8c146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:31.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 >> 172.21.15.29:0/102056381 conn(0x7f2d8c067610 msgr2=0x7f2d8c15b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:31.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 shutdown_connections 2023-10-07T10:44:31.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.783+0000 7f2d911ff700 1 -- 172.21.15.29:0/102056381 wait complete. 2023-10-07T10:44:31.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-07T10:44:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.997+0000 7fe4e4fff700 1 Processor -- start 2023-10-07T10:44:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.997+0000 7fe4e4fff700 1 -- start start 2023-10-07T10:44:32.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.998+0000 7fe4e4fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4e0159550 0x7fe4e00b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:32.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.998+0000 7fe4e4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b2d80 0x7fe4e00a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.998+0000 7fe4e4fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e006f4e0 con 0x7fe4e00b0f50 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.998+0000 7fe4e4fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e006e8d0 con 0x7fe4e0159550 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.998+0000 7fe4e4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e006ebb0 con 0x7fe4e00b2d80 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.998+0000 7fe4de59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4e0159550 0x7fe4e00b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4ddd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe4e00b0f50 0x7fe4e00b2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56480/0 (socket says 172.21.15.29:56480) 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4de59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4e0159550 0x7fe4e00b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59140/0 (socket says 172.21.15.29:59140) 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4ddd9b700 1 -- 172.21.15.29:0/4276554949 learned_addr learned my addr 172.21.15.29:0/4276554949 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4de59c700 1 -- 172.21.15.29:0/4276554949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b2d80 msgr2=0x7fe4e00a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4ded9d700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b2d80 0x7fe4e00a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4dd59a700 1 -- 172.21.15.29:0/4276554949 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686640704 0 0) 0x7fe4e006f4e0 con 0x7fe4e00b0f50 2023-10-07T10:44:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4de59c700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b2d80 0x7fe4e00a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4de59c700 1 -- 172.21.15.29:0/4276554949 >> v1:172.21.15.29:6789/0 conn(0x7fe4e00b0f50 legacy=0x7fe4e00b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4de59c700 1 -- 172.21.15.29:0/4276554949 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e006ee90 con 0x7fe4e0159550 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4ded9d700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b2d80 0x7fe4e00a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:31.999+0000 7fe4de59c700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e00b0a10 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00098e0 tx=0x7fe4d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a46e0215d2e4aca server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4dd59a700 1 -- 172.21.15.29:0/4276554949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0009e80 con 0x7fe4e0159550 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4dd59a700 1 -- 172.21.15.29:0/4276554949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d000b860 con 0x7fe4e0159550 2023-10-07T10:44:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4dd59a700 1 -- 172.21.15.29:0/4276554949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d000bbe0 con 0x7fe4e0159550 2023-10-07T10:44:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 -- 172.21.15.29:0/4276554949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 msgr2=0x7fe4e00b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e00b0a10 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00098e0 tx=0x7fe4d00091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 -- 172.21.15.29:0/4276554949 shutdown_connections 2023-10-07T10:44:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b2d80 0x7fe4e00a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/4276554949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e00b0a10 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 -- 172.21.15.29:0/4276554949 >> 172.21.15.29:0/4276554949 conn(0x7fe4e0069190 msgr2=0x7fe4e006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 -- 172.21.15.29:0/4276554949 shutdown_connections 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.000+0000 7fe4e4fff700 1 -- 172.21.15.29:0/4276554949 wait complete. 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.002+0000 7fe4e4fff700 1 Processor -- start 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.002+0000 7fe4e4fff700 1 -- start start 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4e4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b0f50 0x7fe4e0156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4e4fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4e00b2d80 0x7fe4e0150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4e4fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e0159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4e4fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0151080 con 0x7fe4e0159550 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4e4fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0155e80 con 0x7fe4e00b2d80 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4e4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0156160 con 0x7fe4e00b0f50 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4ded9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e0159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4ded9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e0159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59148/0 (socket says 172.21.15.29:59148) 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4ded9d700 1 -- 172.21.15.29:0/1744294347 learned_addr learned my addr 172.21.15.29:0/1744294347 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4de59c700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b0f50 0x7fe4e0156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.003+0000 7fe4ddd9b700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4e00b2d80 0x7fe4e0150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ded9d700 1 -- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b0f50 msgr2=0x7fe4e0156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ded9d700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b0f50 0x7fe4e0156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ded9d700 1 -- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4e00b2d80 msgr2=0x7fe4e0150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ded9d700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4e00b2d80 0x7fe4e0150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ded9d700 1 -- 172.21.15.29:0/1744294347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0009420 con 0x7fe4e0159550 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4de59c700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b0f50 0x7fe4e0156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ddd9b700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4e00b2d80 0x7fe4e0150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ded9d700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e0159a00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800cef0 tx=0x7fe4d8007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800ae00 con 0x7fe4e0159550 2023-10-07T10:44:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8016420 con 0x7fe4e0159550 2023-10-07T10:44:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80167e0 con 0x7fe4e0159550 2023-10-07T10:44:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e0156440 con 0x7fe4e0159550 2023-10-07T10:44:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.004+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e015c590 con 0x7fe4e0159550 2023-10-07T10:44:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.005+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e015c7b0 con 0x7fe4e0159550 2023-10-07T10:44:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.006+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe4d8016bd0 con 0x7fe4e0159550 2023-10-07T10:44:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.007+0000 7fe4ce7fc700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe4c0066f30 0x7fe4c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.008+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7fe4d8012070 con 0x7fe4e0159550 2023-10-07T10:44:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.008+0000 7fe4de59c700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe4c0066f30 0x7fe4c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.009+0000 7fe4de59c700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe4c0066f30 0x7fe4c00693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00091e0 tx=0x7fe4d0017000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:32.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.011+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4d804e5a0 con 0x7fe4e0159550 2023-10-07T10:44:32.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:32.301+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fe4e004c3c0 con 0x7fe4e0159550 2023-10-07T10:44:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.075+0000 7fe4ce7fc700 1 -- 172.21.15.29:0/1744294347 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2427) v1 ==== 178+0+0 (secure 0 0 0) 0x7fe4d8055bd0 con 0x7fe4e0159550 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe4c0066f30 msgr2=0x7fe4c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe4c0066f30 0x7fe4c00693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00091e0 tx=0x7fe4d0017000 comp rx=0 tx=0).stop 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 msgr2=0x7fe4e0159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e0159a00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800cef0 tx=0x7fe4d8007860 comp rx=0 tx=0).stop 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 shutdown_connections 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe4c0066f30 0x7fe4c00693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159550 0x7fe4e0159a00 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4e00b0f50 0x7fe4e0156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 --2- 172.21.15.29:0/1744294347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4e00b2d80 0x7fe4e0150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.078+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 >> 172.21.15.29:0/1744294347 conn(0x7fe4e0069190 msgr2=0x7fe4e0152910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.079+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 shutdown_connections 2023-10-07T10:44:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.079+0000 7fe4e4fff700 1 -- 172.21.15.29:0/1744294347 wait complete. 2023-10-07T10:44:33.100 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-07T10:44:33.100 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-07T10:44:33.100 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T10:44:33.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-07T10:44:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.300+0000 7f57635ff700 1 Processor -- start 2023-10-07T10:44:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.300+0000 7f57635ff700 1 -- start start 2023-10-07T10:44:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f57635ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f57635ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f575c159ec0 0x7f575c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f57635ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c14d790 con 0x7f575c1568c0 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f57635ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0ae5d0 con 0x7f575c159ec0 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f57635ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c15a850 con 0x7f575c0ad370 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f5760b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f575c159ec0 0x7f575c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f5760b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f575c159ec0 0x7f575c15a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53122/0 (socket says 172.21.15.29:53122) 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.301+0000 7f5760b9a700 1 -- 172.21.15.29:0/3001385236 learned_addr learned my addr 172.21.15.29:0/3001385236 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f576139b700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5760b9a700 1 -- 172.21.15.29:0/3001385236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 msgr2=0x7f575c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5760b9a700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5760b9a700 1 -- 172.21.15.29:0/3001385236 >> v1:172.21.15.144:6789/0 conn(0x7f575c1568c0 legacy=0x7f575c156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5760b9a700 1 -- 172.21.15.29:0/3001385236 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0ad740 con 0x7f575c159ec0 2023-10-07T10:44:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f576139b700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5760b9a700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c159ec0 0x7f575c15a310 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f574c0099b0 tx=0x7f574c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d4c7b6ddb7258d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5753fff700 1 -- 172.21.15.29:0/3001385236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c00b820 con 0x7f575c159ec0 2023-10-07T10:44:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5753fff700 1 -- 172.21.15.29:0/3001385236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c00b980 con 0x7f575c159ec0 2023-10-07T10:44:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.302+0000 7f5753fff700 1 -- 172.21.15.29:0/3001385236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c00bd20 con 0x7f575c159ec0 2023-10-07T10:44:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 -- 172.21.15.29:0/3001385236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c159ec0 msgr2=0x7f575c15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c159ec0 0x7f575c15a310 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f574c0099b0 tx=0x7f574c009250 comp rx=0 tx=0).stop 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 -- 172.21.15.29:0/3001385236 shutdown_connections 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 --2- 172.21.15.29:0/3001385236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c159ec0 0x7f575c15a310 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 -- 172.21.15.29:0/3001385236 >> 172.21.15.29:0/3001385236 conn(0x7f575c0680a0 msgr2=0x7f575c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 -- 172.21.15.29:0/3001385236 shutdown_connections 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.303+0000 7f57635ff700 1 -- 172.21.15.29:0/3001385236 wait complete. 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.305+0000 7f57635ff700 1 Processor -- start 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.305+0000 7f57635ff700 1 -- start start 2023-10-07T10:44:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f57635ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c169740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f57635ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 0x7f575c158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f57635ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c159ec0 0x7f575c0aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f57635ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c156d20 con 0x7f575c1568c0 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f57635ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c14f3b0 con 0x7f575c159ec0 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f57635ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c169d20 con 0x7f575c0ad370 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 0x7f575c158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 0x7f575c158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53134/0 (socket says 172.21.15.29:53134) 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 -- 172.21.15.29:0/1574002299 learned_addr learned my addr 172.21.15.29:0/1574002299 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f576139b700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c169740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5761b9c700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c159ec0 0x7f575c0aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 -- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 msgr2=0x7f575c169740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c169740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 -- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c159ec0 msgr2=0x7f575c0aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c159ec0 0x7f575c0aae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.306+0000 7f5760b9a700 1 -- 172.21.15.29:0/1574002299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0094f0 con 0x7f575c1568c0 2023-10-07T10:44:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.307+0000 7f5760b9a700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 0x7f575c158840 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f574c009290 tx=0x7f574c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.307+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c011d70 con 0x7f575c1568c0 2023-10-07T10:44:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.307+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c020070 con 0x7f575c1568c0 2023-10-07T10:44:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.307+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c01b6a0 con 0x7f575c1568c0 2023-10-07T10:44:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.307+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c16a000 con 0x7f575c1568c0 2023-10-07T10:44:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.307+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c16a5c0 con 0x7f575c1568c0 2023-10-07T10:44:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.308+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c159c20 con 0x7f575c1568c0 2023-10-07T10:44:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.309+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f574c022030 con 0x7f575c1568c0 2023-10-07T10:44:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.310+0000 7f57517fa700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5740066f30 0x7f57400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.310+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f574c086600 con 0x7f575c1568c0 2023-10-07T10:44:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.311+0000 7f576139b700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5740066f30 0x7f57400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.311+0000 7f576139b700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5740066f30 0x7f57400693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f575c150e80 tx=0x7f5758006f90 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.314+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f574c053720 con 0x7f575c1568c0 2023-10-07T10:44:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:33.594+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f575c159c20 con 0x7f575c1568c0 2023-10-07T10:44:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.080+0000 7f57517fa700 1 -- 172.21.15.29:0/1574002299 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2428) v1 ==== 177+0+0 (secure 0 0 0) 0x7f574c056d40 con 0x7f575c1568c0 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5740066f30 msgr2=0x7f57400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5740066f30 0x7f57400693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f575c150e80 tx=0x7f5758006f90 comp rx=0 tx=0).stop 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 msgr2=0x7f575c158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 0x7f575c158840 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f574c009290 tx=0x7f574c016000 comp rx=0 tx=0).stop 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 shutdown_connections 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5740066f30 0x7f57400693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c159ec0 0x7f575c0aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0ad370 0x7f575c169740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 --2- 172.21.15.29:0/1574002299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c1568c0 0x7f575c158840 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 >> 172.21.15.29:0/1574002299 conn(0x7f575c0680a0 msgr2=0x7f575c14cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 shutdown_connections 2023-10-07T10:44:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.083+0000 7f57635ff700 1 -- 172.21.15.29:0/1574002299 wait complete. 2023-10-07T10:44:34.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T10:44:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.303+0000 7f51469ff700 1 Processor -- start 2023-10-07T10:44:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.304+0000 7f51469ff700 1 -- start start 2023-10-07T10:44:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f51469ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f514006b5c0 0x7f51400c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f51469ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514006a430 con 0x7f514006a8c0 2023-10-07T10:44:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f51469ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5140159b60 con 0x7f5140159790 2023-10-07T10:44:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f51469ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5140069820 con 0x7f514006b5c0 2023-10-07T10:44:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f513f7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f514006a8c0 0x7f51400c6f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47316/0 (socket says 172.21.15.29:47316) 2023-10-07T10:44:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f513f7fe700 1 -- 172.21.15.29:0/2083999637 learned_addr learned my addr 172.21.15.29:0/2083999637 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f5144f9c700 1 --2- 172.21.15.29:0/2083999637 >> v2:172.21.15.144:3300/0 conn(0x7f514006b5c0 0x7f51400c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.305+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276867068 0 0) 0x7f514006a430 con 0x7f514006a8c0 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5128003570 con 0x7f514006a8c0 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3339295674 0 0) 0x7f5128003570 con 0x7f514006a8c0 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 >> v2:172.21.15.144:3300/0 conn(0x7f514006b5c0 msgr2=0x7f51400c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 --2- 172.21.15.29:0/2083999637 >> v2:172.21.15.144:3300/0 conn(0x7f514006b5c0 0x7f51400c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 >> v1:172.21.15.144:6789/0 conn(0x7f5140159790 legacy=0x7f514006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5140069b00 con 0x7f514006a8c0 2023-10-07T10:44:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f5134001470 con 0x7f514006a8c0 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 >> v1:172.21.15.29:6789/0 conn(0x7f514006a8c0 legacy=0x7f51400c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5128003a60 0x7f5128003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5128004810 0x7f5128004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f512800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5140069820 con 0x7f51280051c0 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512800c0d0 con 0x7f5128003a60 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512800c290 con 0x7f5128004810 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f5144f9c700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f512800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513f7fe700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5128004810 0x7f5128004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.306+0000 7f513ffff700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5128003a60 0x7f5128003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 -- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5128004810 msgr2=0x7f5128004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5128004810 0x7f5128004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 -- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5128003a60 msgr2=0x7f5128003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5128003a60 0x7f5128003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 -- 172.21.15.29:0/2083999637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5128003570 con 0x7f51280051c0 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 -- 172.21.15.29:0/2083999637 reap_dead start 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f5144f9c700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f512800ba60 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f514006b9c0 tx=0x7f513800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8f21ead5bed2b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513800ea90 con 0x7f51280051c0 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513800f040 con 0x7f51280051c0 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.307+0000 7f513effd700 1 -- 172.21.15.29:0/2083999637 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513800ee40 con 0x7f51280051c0 2023-10-07T10:44:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 -- 172.21.15.29:0/2083999637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 msgr2=0x7f512800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f512800ba60 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f514006b9c0 tx=0x7f513800b040 comp rx=0 tx=0).stop 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 -- 172.21.15.29:0/2083999637 shutdown_connections 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 --2- 172.21.15.29:0/2083999637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f512800ba60 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 -- 172.21.15.29:0/2083999637 >> 172.21.15.29:0/2083999637 conn(0x7f51400c9410 msgr2=0x7f51400c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 -- 172.21.15.29:0/2083999637 shutdown_connections 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.308+0000 7f51469ff700 1 -- 172.21.15.29:0/2083999637 wait complete. 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.310+0000 7f51469ff700 1 Processor -- start 2023-10-07T10:44:34.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.310+0000 7f51469ff700 1 -- start start 2023-10-07T10:44:34.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f51469ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f51400ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f51469ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f514014b210 0x7f51400cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f51469ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51401699e0 0x7f51400cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f51469ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51400cc700 con 0x7f51280051c0 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f51469ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51400ae1a0 con 0x7f51401699e0 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f51469ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514014b000 con 0x7f514014b210 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f51400ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f51400ac6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53160/0 (socket says 172.21.15.29:53160) 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 -- 172.21.15.29:0/3247025052 learned_addr learned my addr 172.21.15.29:0/3247025052 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513f7fe700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f514014b210 0x7f51400cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f5144f9c700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51401699e0 0x7f51400cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 -- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f514014b210 msgr2=0x7f51400cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f514014b210 0x7f51400cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 -- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51401699e0 msgr2=0x7f51400cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51401699e0 0x7f51400cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.311+0000 7f513ffff700 1 -- 172.21.15.29:0/3247025052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51400aece0 con 0x7f51280051c0 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.312+0000 7f513ffff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f51400ac6f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f5130002980 tx=0x7f5130007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.312+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513000f8d0 con 0x7f51280051c0 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.312+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513000fa30 con 0x7f51280051c0 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.312+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513000fdd0 con 0x7f51280051c0 2023-10-07T10:44:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.312+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51400ae3b0 con 0x7f51280051c0 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.312+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51400ae970 con 0x7f51280051c0 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.313+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514004c3c0 con 0x7f51280051c0 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.314+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5130013600 con 0x7f51280051c0 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.315+0000 7f5127fff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f512c066f30 0x7f512c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.315+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f5130084a80 con 0x7f51280051c0 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.315+0000 7f513f7fe700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f512c066f30 0x7f512c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.316+0000 7f513f7fe700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f512c066f30 0x7f512c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5134001050 tx=0x7f5134009040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.319+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5130051c40 con 0x7f51280051c0 2023-10-07T10:44:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.518+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5140063640 con 0x7f51280051c0 2023-10-07T10:44:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.522+0000 7f5127fff700 1 -- 172.21.15.29:0/3247025052 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5130055260 con 0x7f51280051c0 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f512c066f30 msgr2=0x7f512c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f512c066f30 0x7f512c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5134001050 tx=0x7f5134009040 comp rx=0 tx=0).stop 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 msgr2=0x7f51400ac6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f51400ac6f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f5130002980 tx=0x7f5130007d30 comp rx=0 tx=0).stop 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 shutdown_connections 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f512c066f30 0x7f512c0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51401699e0 0x7f51400cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51280051c0 0x7f51400ac6f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 --2- 172.21.15.29:0/3247025052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f514014b210 0x7f51400cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.524+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 >> 172.21.15.29:0/3247025052 conn(0x7f51400c9410 msgr2=0x7f514015ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.525+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 shutdown_connections 2023-10-07T10:44:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.525+0000 7f51469ff700 1 -- 172.21.15.29:0/3247025052 wait complete. 2023-10-07T10:44:34.543 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T10:44:34.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:44:34.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:44:34.750 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T10:44:34.750 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T10:44:34.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:34.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.945+0000 7f500a9ff700 1 Processor -- start 2023-10-07T10:44:34.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.946+0000 7f500a9ff700 1 -- start start 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f500a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50040ad370 0x7f50040cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f500a9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50040b0410 0x7f500415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f500a9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500414d790 con 0x7f5004159ec0 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f500a9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040ae5d0 con 0x7f50040b0410 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f500a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500415a850 con 0x7f50040ad370 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f5008f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50040b0410 0x7f500415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f5008f9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50040b0410 0x7f500415a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53166/0 (socket says 172.21.15.29:53166) 2023-10-07T10:44:34.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f50037fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5004159ec0 0x7f50041568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47322/0 (socket says 172.21.15.29:47322) 2023-10-07T10:44:34.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f5008f9c700 1 -- 172.21.15.29:0/107512187 learned_addr learned my addr 172.21.15.29:0/107512187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:34.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.947+0000 7f5003fff700 1 --2- 172.21.15.29:0/107512187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50040ad370 0x7f50040cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5008f9c700 1 -- 172.21.15.29:0/107512187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50040ad370 msgr2=0x7f50040cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:34.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5008f9c700 1 --2- 172.21.15.29:0/107512187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50040ad370 0x7f50040cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5008f9c700 1 -- 172.21.15.29:0/107512187 >> v1:172.21.15.29:6789/0 conn(0x7f5004159ec0 legacy=0x7f50041568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5008f9c700 1 -- 172.21.15.29:0/107512187 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040ad740 con 0x7f50040b0410 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5002ffd700 1 -- 172.21.15.29:0/107512187 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125745465 0 0) 0x7f500414d790 con 0x7f5004159ec0 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5008f9c700 1 --2- 172.21.15.29:0/107512187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040b0410 0x7f500415a310 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009f20 tx=0x7f4ffc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1959b2f73a82bdb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5002ffd700 1 -- 172.21.15.29:0/107512187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc00e070 con 0x7f50040b0410 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5002ffd700 1 -- 172.21.15.29:0/107512187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc00b9d0 con 0x7f50040b0410 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.948+0000 7f5002ffd700 1 -- 172.21.15.29:0/107512187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc0092e0 con 0x7f50040b0410 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 -- 172.21.15.29:0/107512187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040b0410 msgr2=0x7f500415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 --2- 172.21.15.29:0/107512187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040b0410 0x7f500415a310 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009f20 tx=0x7f4ffc009840 comp rx=0 tx=0).stop 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 -- 172.21.15.29:0/107512187 shutdown_connections 2023-10-07T10:44:34.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 --2- 172.21.15.29:0/107512187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040b0410 0x7f500415a310 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 --2- 172.21.15.29:0/107512187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50040ad370 0x7f50040cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 -- 172.21.15.29:0/107512187 >> 172.21.15.29:0/107512187 conn(0x7f50040680a0 msgr2=0x7f50040cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:34.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 -- 172.21.15.29:0/107512187 shutdown_connections 2023-10-07T10:44:34.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.949+0000 7f500a9ff700 1 -- 172.21.15.29:0/107512187 wait complete. 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.950+0000 7f500a9ff700 1 Processor -- start 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.951+0000 7f500a9ff700 1 -- start start 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.951+0000 7f500a9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 0x7f50040c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f500a9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50040b0410 0x7f500415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f500a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5004159ec0 0x7f500414fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f500a9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004150c10 con 0x7f50040ad370 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f500a9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500414a760 con 0x7f50040b0410 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f500a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040ab080 con 0x7f5004159ec0 2023-10-07T10:44:34.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 0x7f50040c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 0x7f50040c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53182/0 (socket says 172.21.15.29:53182) 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 -- 172.21.15.29:0/2452371448 learned_addr learned my addr 172.21.15.29:0/2452371448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5008f9c700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5004159ec0 0x7f500414fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f50037fe700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50040b0410 0x7f500415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 -- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5004159ec0 msgr2=0x7f500414fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5004159ec0 0x7f500414fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 -- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50040b0410 msgr2=0x7f500415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50040b0410 0x7f500415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f5003fff700 1 -- 172.21.15.29:0/2452371448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc009ae0 con 0x7f50040ad370 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.952+0000 7f50037fe700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50040b0410 0x7f500415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:34.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.953+0000 7f5003fff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 0x7f50040c8f80 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400ec80 tx=0x7f4ff40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.953+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400c850 con 0x7f50040ad370 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.953+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff400c9b0 con 0x7f50040ad370 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.953+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50040cabc0 con 0x7f50040ad370 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.953+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400cd50 con 0x7f50040ad370 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.953+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50040aad10 con 0x7f50040ad370 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.954+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50040c8460 con 0x7f50040ad370 2023-10-07T10:44:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.955+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4ff4013680 con 0x7f50040ad370 2023-10-07T10:44:34.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.956+0000 7f4febfff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4ff0066f30 0x7f4ff00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:34.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.956+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7f4ff4084ba0 con 0x7f50040ad370 2023-10-07T10:44:34.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.957+0000 7f50037fe700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4ff0066f30 0x7f4ff00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:34.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.957+0000 7f50037fe700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4ff0066f30 0x7f4ff00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8007fd0 tx=0x7f4ff8007e30 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:34.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:34.960+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ff4051d60 con 0x7f50040ad370 2023-10-07T10:44:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.205+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500404c3c0 con 0x7f50040ad370 2023-10-07T10:44:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.206+0000 7f4febfff700 1 -- 172.21.15.29:0/2452371448 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f4ff4021de0 con 0x7f50040ad370 2023-10-07T10:44:35.207 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:35.207 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":70,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7091,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/319345381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":319345381},{"type":"v1","addr":"172.21.15.144:6835","nonce":319345381}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7774,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2356079578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2356079578},{"type":"v1","addr":"172.21.15.29:6835","nonce":2356079578}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7780,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/263071015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":263071015},{"type":"v1","addr":"172.21.15.144:6837","nonce":263071015}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":8511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2887107930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2887107930},{"type":"v1","addr":"172.21.15.29:6834","nonce":2887107930}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[]} 2023-10-07T10:44:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4ff0066f30 msgr2=0x7f4ff00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4ff0066f30 0x7f4ff00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8007fd0 tx=0x7f4ff8007e30 comp rx=0 tx=0).stop 2023-10-07T10:44:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 msgr2=0x7f50040c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 0x7f50040c8f80 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400ec80 tx=0x7f4ff40091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 shutdown_connections 2023-10-07T10:44:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4ff0066f30 0x7f4ff00693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5004159ec0 0x7f500414fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50040ad370 0x7f50040c8f80 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 --2- 172.21.15.29:0/2452371448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50040b0410 0x7f500415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 >> 172.21.15.29:0/2452371448 conn(0x7f50040680a0 msgr2=0x7f50040694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 shutdown_connections 2023-10-07T10:44:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.208+0000 7f500a9ff700 1 -- 172.21.15.29:0/2452371448 wait complete. 2023-10-07T10:44:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 70 2023-10-07T10:44:35.227 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T10:44:35.227 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T10:44:35.227 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T10:44:35.228 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:44:35.228 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T10:44:35.228 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T10:44:35.228 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T10:44:35.228 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:44:35.228 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T10:44:35.229 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T10:44:35.229 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T10:44:35.230 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:44:35.230 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T10:44:35.231 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T10:44:35.231 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T10:44:35.231 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:44:35.246 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T10:44:35.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T10:44:35.247 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T10:44:35.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T10:44:35.248 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T10:44:35.248 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T10:44:35.249 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T10:44:35.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T10:44:35.290 INFO:tasks.ceph.mds.c:Started 2023-10-07T10:44:35.290 DEBUG:teuthology.parallel:result is None 2023-10-07T10:44:35.293 INFO:tasks.ceph.mds.a:Started 2023-10-07T10:44:35.293 DEBUG:teuthology.parallel:result is None 2023-10-07T10:44:35.293 INFO:tasks.ceph.mds.b:Started 2023-10-07T10:44:35.294 DEBUG:teuthology.parallel:result is None 2023-10-07T10:44:35.296 INFO:tasks.ceph.mds.d:Started 2023-10-07T10:44:35.296 DEBUG:teuthology.parallel:result is None 2023-10-07T10:44:35.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T10:44:35.439 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T10:44:35.439 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T10:44:35.440 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T10:44:35.440 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T10:44:35.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.505+0000 7fa9fa9ff700 1 Processor -- start 2023-10-07T10:44:35.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.505+0000 7fa9fa9ff700 1 -- start start 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9fa9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9f40b04b0 0x7fa9f4156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9fa9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f4159670 con 0x7fa9f41533f0 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9fa9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f415a1f0 con 0x7fa9f41537c0 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9fa9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f415a680 con 0x7fa9f40b04b0 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9f3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa9f41533f0 0x7fa9f40afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47332/0 (socket says 172.21.15.29:47332) 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9f3fff700 1 -- 172.21.15.29:0/3107830137 learned_addr learned my addr 172.21.15.29:0/3107830137 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:35.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.507+0000 7fa9f37fe700 1 --2- 172.21.15.29:0/3107830137 >> v2:172.21.15.29:3300/0 conn(0x7fa9f40b04b0 0x7fa9f4156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:35.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f2ffd700 1 -- 172.21.15.29:0/3107830137 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298502595 0 0) 0x7fa9f4159670 con 0x7fa9f41533f0 2023-10-07T10:44:35.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f2ffd700 1 -- 172.21.15.29:0/3107830137 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9dc003570 con 0x7fa9f41533f0 2023-10-07T10:44:35.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f37fe700 1 -- 172.21.15.29:0/3107830137 >> v1:172.21.15.144:6789/0 conn(0x7fa9f41537c0 legacy=0x7fa9f4157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f37fe700 1 -- 172.21.15.29:0/3107830137 >> v1:172.21.15.29:6789/0 conn(0x7fa9f41533f0 legacy=0x7fa9f40afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f37fe700 1 -- 172.21.15.29:0/3107830137 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f41598c0 con 0x7fa9f40b04b0 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f37fe700 1 --2- 172.21.15.29:0/3107830137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40b04b0 0x7fa9f4156a90 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098e0 tx=0x7fa9e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6e2ac595bf3552 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f2ffd700 1 -- 172.21.15.29:0/3107830137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800b640 con 0x7fa9f40b04b0 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f2ffd700 1 -- 172.21.15.29:0/3107830137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800b7a0 con 0x7fa9f40b04b0 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.508+0000 7fa9f2ffd700 1 -- 172.21.15.29:0/3107830137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800bb40 con 0x7fa9f40b04b0 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/3107830137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40b04b0 msgr2=0x7fa9f4156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/3107830137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40b04b0 0x7fa9f4156a90 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098e0 tx=0x7fa9e80091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/3107830137 shutdown_connections 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/3107830137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40b04b0 0x7fa9f4156a90 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/3107830137 >> 172.21.15.29:0/3107830137 conn(0x7fa9f40680a0 msgr2=0x7fa9f40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/3107830137 shutdown_connections 2023-10-07T10:44:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.509+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/3107830137 wait complete. 2023-10-07T10:44:35.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.511+0000 7fa9fa9ff700 1 Processor -- start 2023-10-07T10:44:35.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.511+0000 7fa9fa9ff700 1 -- start start 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9fa9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9f40b04b0 0x7fa9f416a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9fa9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 0x7fa9f414ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9fa9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f41537c0 0x7fa9f414fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9fa9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f414bcf0 con 0x7fa9f41537c0 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9fa9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f41508f0 con 0x7fa9f41533f0 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9fa9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f414b3c0 con 0x7fa9f40b04b0 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f8f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f41537c0 0x7fa9f414fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 0x7fa9f414ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 0x7fa9f414ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:58124/0 (socket says 172.21.15.29:58124) 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 -- 172.21.15.29:0/2041133124 learned_addr learned my addr 172.21.15.29:0/2041133124 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f3fff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9f40b04b0 0x7fa9f416a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 -- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9f40b04b0 msgr2=0x7fa9f416a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9f40b04b0 0x7fa9f416a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 -- 172.21.15.29:0/2041133124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f41537c0 msgr2=0x7fa9f414fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f41537c0 0x7fa9f414fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.512+0000 7fa9f37fe700 1 -- 172.21.15.29:0/2041133124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8009420 con 0x7fa9f41533f0 2023-10-07T10:44:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.513+0000 7fa9f37fe700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 0x7fa9f414ab30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4068dd0 tx=0x7fa9e8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.513+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8000d10 con 0x7fa9f41533f0 2023-10-07T10:44:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.513+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800bb20 con 0x7fa9f41533f0 2023-10-07T10:44:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.513+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8011670 con 0x7fa9f41533f0 2023-10-07T10:44:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.513+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f414b6a0 con 0x7fa9f41533f0 2023-10-07T10:44:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.513+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f416a730 con 0x7fa9f41533f0 2023-10-07T10:44:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.514+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f404c3c0 con 0x7fa9f41533f0 2023-10-07T10:44:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.515+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa9e80119e0 con 0x7fa9f41533f0 2023-10-07T10:44:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.516+0000 7fa9dbfff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e0066f30 0x7fa9e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.516+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5875+0+0 (secure 0 0 0) 0x7fa9e8085c10 con 0x7fa9f41533f0 2023-10-07T10:44:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.516+0000 7fa9f3fff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e0066f30 0x7fa9e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:35.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.517+0000 7fa9f3fff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e0066f30 0x7fa9e00693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4007fd0 tx=0x7fa9e4009040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:35.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.521+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9e8052d30 con 0x7fa9f41533f0 2023-10-07T10:44:35.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:35.739+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa9f414be30 con 0x7fa9f41533f0 2023-10-07T10:44:36.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.091+0000 7fa9dbfff700 1 -- 172.21.15.29:0/2041133124 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v145) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa9e8056350 con 0x7fa9f41533f0 2023-10-07T10:44:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e0066f30 msgr2=0x7fa9e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e0066f30 0x7fa9e00693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4007fd0 tx=0x7fa9e4009040 comp rx=0 tx=0).stop 2023-10-07T10:44:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 msgr2=0x7fa9f414ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 0x7fa9f414ab30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4068dd0 tx=0x7fa9e8017000 comp rx=0 tx=0).stop 2023-10-07T10:44:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 shutdown_connections 2023-10-07T10:44:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa9e0066f30 0x7fa9e00693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f41537c0 0x7fa9f414fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9f40b04b0 0x7fa9f416a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 --2- 172.21.15.29:0/2041133124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9f41533f0 0x7fa9f414ab30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.094+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 >> 172.21.15.29:0/2041133124 conn(0x7fa9f40680a0 msgr2=0x7fa9f4153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.095+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 shutdown_connections 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.095+0000 7fa9fa9ff700 1 -- 172.21.15.29:0/2041133124 wait complete. 2023-10-07T10:44:36.099 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T10:44:36.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T10:44:36.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.313+0000 7f1ecadff700 1 Processor -- start 2023-10-07T10:44:36.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.313+0000 7f1ecadff700 1 -- start start 2023-10-07T10:44:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ecadff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ec40ad370 0x7f1ec40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ecadff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ecadff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ec41568c0 0x7f1ec4156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ecadff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec414d790 con 0x7f1ec41568c0 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ecadff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec40ae5d0 con 0x7f1ec40ad370 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ecadff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec415a850 con 0x7f1ec4159ec0 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ec939c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ec41568c0 0x7f1ec4156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ec8b9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ec40ad370 0x7f1ec40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ec3fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.315+0000 7f1ec939c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ec41568c0 0x7f1ec4156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53234/0 (socket says 172.21.15.29:53234) 2023-10-07T10:44:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 -- 172.21.15.29:0/1395286299 learned_addr learned my addr 172.21.15.29:0/1395286299 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 -- 172.21.15.29:0/1395286299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 msgr2=0x7f1ec415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 --2- 172.21.15.29:0/1395286299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 -- 172.21.15.29:0/1395286299 >> v2:172.21.15.144:3300/0 conn(0x7f1ec40ad370 msgr2=0x7f1ec40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 --2- 172.21.15.29:0/1395286299 >> v2:172.21.15.144:3300/0 conn(0x7f1ec40ad370 0x7f1ec40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 -- 172.21.15.29:0/1395286299 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec40ad740 con 0x7f1ec41568c0 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec939c700 1 --2- 172.21.15.29:0/1395286299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4156d00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40685f0 tx=0x7f1ebc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85699ece1ccda5f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.316+0000 7f1ec37fe700 1 -- 172.21.15.29:0/1395286299 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc00e070 con 0x7f1ec41568c0 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ec37fe700 1 -- 172.21.15.29:0/1395286299 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc00b970 con 0x7f1ec41568c0 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ec37fe700 1 -- 172.21.15.29:0/1395286299 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc0092e0 con 0x7f1ec41568c0 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 -- 172.21.15.29:0/1395286299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 msgr2=0x7f1ec4156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1395286299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4156d00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40685f0 tx=0x7f1ebc009840 comp rx=0 tx=0).stop 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 -- 172.21.15.29:0/1395286299 shutdown_connections 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1395286299 >> v2:172.21.15.144:3300/0 conn(0x7f1ec40ad370 0x7f1ec40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1395286299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4156d00 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1395286299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 -- 172.21.15.29:0/1395286299 >> 172.21.15.29:0/1395286299 conn(0x7f1ec40680a0 msgr2=0x7f1ec40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 -- 172.21.15.29:0/1395286299 shutdown_connections 2023-10-07T10:44:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.317+0000 7f1ecadff700 1 -- 172.21.15.29:0/1395286299 wait complete. 2023-10-07T10:44:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.319+0000 7f1ecadff700 1 Processor -- start 2023-10-07T10:44:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.319+0000 7f1ecadff700 1 -- start start 2023-10-07T10:44:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ecadff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ec40ad370 0x7f1ec416c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ecadff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4169870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ecadff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ecadff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec416ccf0 con 0x7f1ec41568c0 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ecadff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec40c9ee0 con 0x7f1ec40ad370 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ecadff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec40c92d0 con 0x7f1ec4159ec0 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4169870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4169870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53244/0 (socket says 172.21.15.29:53244) 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 -- 172.21.15.29:0/1547641014 learned_addr learned my addr 172.21.15.29:0/1547641014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec8b9b700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ec40ad370 0x7f1ec416c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec939c700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 -- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 msgr2=0x7f1ec414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec414f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 -- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ec40ad370 msgr2=0x7f1ec416c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ec40ad370 0x7f1ec416c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec3fff700 1 -- 172.21.15.29:0/1547641014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc009ae0 con 0x7f1ec41568c0 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.320+0000 7f1ec939c700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.321+0000 7f1ec3fff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4169870 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e950 tx=0x7f1eb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.321+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0013680 con 0x7f1ec41568c0 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.321+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb00137e0 con 0x7f1ec41568c0 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.321+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec40c95b0 con 0x7f1ec41568c0 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.321+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0013ba0 con 0x7f1ec41568c0 2023-10-07T10:44:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.321+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec40c9b70 con 0x7f1ec41568c0 2023-10-07T10:44:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.322+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec40aec20 con 0x7f1ec41568c0 2023-10-07T10:44:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.323+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1eb000c490 con 0x7f1ec41568c0 2023-10-07T10:44:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.324+0000 7f1ec0ff9700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.324+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f1eb00845a0 con 0x7f1ec41568c0 2023-10-07T10:44:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.325+0000 7f1ec8b9b700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.325+0000 7f1ec8b9b700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009db0 tx=0x7f1eb80091a0 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.328+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eb0051920 con 0x7f1ec41568c0 2023-10-07T10:44:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.609+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1ec40ca050 con 0x7f1ec41568c0 2023-10-07T10:44:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.610+0000 7f1ec0ff9700 1 -- 172.21.15.29:0/1547641014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1eb0054f40 con 0x7f1ec41568c0 2023-10-07T10:44:36.611 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:36.611 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T10:44:36.611 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:44:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.613 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.614 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:44:36.614 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:44:36.614 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:44:36.615 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:44:36.616 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.616 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.616 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T10:44:36.616 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.618 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.619 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T10:44:36.619 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T10:44:36.619 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.620 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.620 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.620 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.620 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.621 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T10:44:36.621 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T10:44:36.621 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.621 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.621 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.621 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.622 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.623 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.624 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.625 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.626 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T10:44:36.627 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T10:44:36.628 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T10:44:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T10:44:36.630 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T10:44:36.631 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T10:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T10:44:36.633 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T10:44:36.634 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ea8066f30 msgr2=0x7f1ea80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009db0 tx=0x7f1eb80091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 msgr2=0x7f1ec4169870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4169870 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e950 tx=0x7f1eb00091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 shutdown_connections 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1ea8066f30 0x7f1ea80693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ec4159ec0 0x7f1ec414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ec40ad370 0x7f1ec416c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 --2- 172.21.15.29:0/1547641014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec41568c0 0x7f1ec4169870 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 >> 172.21.15.29:0/1547641014 conn(0x7f1ec40680a0 msgr2=0x7f1ec414cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:36.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 shutdown_connections 2023-10-07T10:44:36.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.612+0000 7f1ecadff700 1 -- 172.21.15.29:0/1547641014 wait complete. 2023-10-07T10:44:36.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.834+0000 7fab479ff700 1 Processor -- start 2023-10-07T10:44:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.834+0000 7fab479ff700 1 -- start start 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.835+0000 7fab479ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fab40153cc0 0x7fab400b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.835+0000 7fab479ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab4015ade0 0x7fab400b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.835+0000 7fab479ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40154090 con 0x7fab400b0000 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.835+0000 7fab479ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab40159f60 con 0x7fab4015ade0 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.835+0000 7fab479ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4015a240 con 0x7fab40153cc0 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab4015ade0 0x7fab400b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab4015ade0 0x7fab400b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53256/0 (socket says 172.21.15.29:53256) 2023-10-07T10:44:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 -- 172.21.15.29:0/3228601960 learned_addr learned my addr 172.21.15.29:0/3228601960 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab4579b700 1 --2- 172.21.15.29:0/3228601960 >> v2:172.21.15.144:3300/0 conn(0x7fab40153cc0 0x7fab400b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 -- 172.21.15.29:0/3228601960 >> v2:172.21.15.144:3300/0 conn(0x7fab40153cc0 msgr2=0x7fab400b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 --2- 172.21.15.29:0/3228601960 >> v2:172.21.15.144:3300/0 conn(0x7fab40153cc0 0x7fab400b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 -- 172.21.15.29:0/3228601960 >> v1:172.21.15.144:6789/0 conn(0x7fab400b0000 legacy=0x7fab4015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 -- 172.21.15.29:0/3228601960 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4015a520 con 0x7fab4015ade0 2023-10-07T10:44:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.836+0000 7fab44f9a700 1 --2- 172.21.15.29:0/3228601960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4015ade0 0x7fab400b1d00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fab400cf440 tx=0x7fab30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf06ca8db1a0082f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab37fff700 1 -- 172.21.15.29:0/3228601960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30009e70 con 0x7fab4015ade0 2023-10-07T10:44:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab37fff700 1 -- 172.21.15.29:0/3228601960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3000b860 con 0x7fab4015ade0 2023-10-07T10:44:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab37fff700 1 -- 172.21.15.29:0/3228601960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3000bc00 con 0x7fab4015ade0 2023-10-07T10:44:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 -- 172.21.15.29:0/3228601960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4015ade0 msgr2=0x7fab400b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 --2- 172.21.15.29:0/3228601960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4015ade0 0x7fab400b1d00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fab400cf440 tx=0x7fab30009250 comp rx=0 tx=0).stop 2023-10-07T10:44:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 -- 172.21.15.29:0/3228601960 shutdown_connections 2023-10-07T10:44:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 --2- 172.21.15.29:0/3228601960 >> v2:172.21.15.144:3300/0 conn(0x7fab40153cc0 0x7fab400b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 --2- 172.21.15.29:0/3228601960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4015ade0 0x7fab400b1d00 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 -- 172.21.15.29:0/3228601960 >> 172.21.15.29:0/3228601960 conn(0x7fab40067610 msgr2=0x7fab40067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 -- 172.21.15.29:0/3228601960 shutdown_connections 2023-10-07T10:44:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.837+0000 7fab479ff700 1 -- 172.21.15.29:0/3228601960 wait complete. 2023-10-07T10:44:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.839+0000 7fab479ff700 1 Processor -- start 2023-10-07T10:44:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.839+0000 7fab479ff700 1 -- start start 2023-10-07T10:44:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab479ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 0x7fab400ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab479ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab40153cc0 0x7fab400c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab479ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab4015ade0 0x7fab4013e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab479ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40156900 con 0x7fab400b0000 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab479ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40157c50 con 0x7fab4015ade0 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab479ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab40156b50 con 0x7fab40153cc0 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab4579b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 0x7fab400ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab4579b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 0x7fab400ceb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53260/0 (socket says 172.21.15.29:53260) 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab4579b700 1 -- 172.21.15.29:0/3536564698 learned_addr learned my addr 172.21.15.29:0/3536564698 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.840+0000 7fab45f9c700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab4015ade0 0x7fab4013e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab44f9a700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab40153cc0 0x7fab400c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab4579b700 1 -- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab40153cc0 msgr2=0x7fab400c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab4579b700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab40153cc0 0x7fab400c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab4579b700 1 -- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab4015ade0 msgr2=0x7fab4013e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab4579b700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab4015ade0 0x7fab4013e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab4579b700 1 -- 172.21.15.29:0/3536564698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab300094f0 con 0x7fab400b0000 2023-10-07T10:44:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab45f9c700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab4015ade0 0x7fab4013e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab4579b700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 0x7fab400ceb40 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00ec80 tx=0x7fab3c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00c9d0 con 0x7fab400b0000 2023-10-07T10:44:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c00cb30 con 0x7fab400b0000 2023-10-07T10:44:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c013470 con 0x7fab400b0000 2023-10-07T10:44:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.841+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab40156e30 con 0x7fab400b0000 2023-10-07T10:44:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.842+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab401581d0 con 0x7fab400b0000 2023-10-07T10:44:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.842+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab401583c0 con 0x7fab400b0000 2023-10-07T10:44:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.843+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fab3c013820 con 0x7fab400b0000 2023-10-07T10:44:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.844+0000 7fab357fa700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fab24066f30 0x7fab240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.844+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fab3c084b60 con 0x7fab400b0000 2023-10-07T10:44:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.845+0000 7fab44f9a700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fab24066f30 0x7fab240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.845+0000 7fab44f9a700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fab24066f30 0x7fab240693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab400cf440 tx=0x7fab30017000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:36.849+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab3c051ee0 con 0x7fab400b0000 2023-10-07T10:44:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.052+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab4004c3c0 con 0x7fab400b0000 2023-10-07T10:44:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.053+0000 7fab357fa700 1 -- 172.21.15.29:0/3536564698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+12882 (secure 0 0 0) 0x7fab3c017090 con 0x7fab400b0000 2023-10-07T10:44:37.054 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:37.054 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":147,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:36.099355+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fab24066f30 msgr2=0x7fab240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fab24066f30 0x7fab240693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab400cf440 tx=0x7fab30017000 comp rx=0 tx=0).stop 2023-10-07T10:44:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 msgr2=0x7fab400ceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 0x7fab400ceb40 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00ec80 tx=0x7fab3c009120 comp rx=0 tx=0).stop 2023-10-07T10:44:37.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 shutdown_connections 2023-10-07T10:44:37.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fab24066f30 0x7fab240693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab4015ade0 0x7fab4013e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab400b0000 0x7fab400ceb40 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 --2- 172.21.15.29:0/3536564698 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab40153cc0 0x7fab400c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.055+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 >> 172.21.15.29:0/3536564698 conn(0x7fab40067610 msgr2=0x7fab4015b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.056+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 shutdown_connections 2023-10-07T10:44:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.056+0000 7fab479ff700 1 -- 172.21.15.29:0/3536564698 wait complete. 2023-10-07T10:44:37.074 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T10:44:37.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T10:44:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.274+0000 7f06153ff700 1 Processor -- start 2023-10-07T10:44:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.274+0000 7f06153ff700 1 -- start start 2023-10-07T10:44:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.275+0000 7f06153ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f06100b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.275+0000 7f06153ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f061015ade0 0x7f06100b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.275+0000 7f06153ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0610154090 con 0x7f06100b0000 2023-10-07T10:44:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.275+0000 7f06153ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0610159f60 con 0x7f061015ade0 2023-10-07T10:44:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.275+0000 7f06153ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061015a240 con 0x7f0610153cc0 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f061015ade0 0x7f06100b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060effd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f06100b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f061015ade0 0x7f06100b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:58154/0 (socket says 172.21.15.29:58154) 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 -- 172.21.15.29:0/40796543 learned_addr learned my addr 172.21.15.29:0/40796543 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060f7fe700 1 --1- 172.21.15.29:0/40796543 >> v1:172.21.15.144:6789/0 conn(0x7f06100b0000 0x7f061015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:45566/0 (socket says 172.21.15.29:45566) 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 -- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 msgr2=0x7f06100b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 --2- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f06100b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 -- 172.21.15.29:0/40796543 >> v1:172.21.15.144:6789/0 conn(0x7f06100b0000 legacy=0x7f061015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.276+0000 7f060e7fc700 1 -- 172.21.15.29:0/40796543 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061015a520 con 0x7f061015ade0 2023-10-07T10:44:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f060e7fc700 1 --2- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061015ade0 0x7f06100b1d00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f06040099b0 tx=0x7f0604009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69746fd31120cdb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f060dffb700 1 -- 172.21.15.29:0/40796543 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604009ea0 con 0x7f061015ade0 2023-10-07T10:44:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f060dffb700 1 -- 172.21.15.29:0/40796543 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060400b860 con 0x7f061015ade0 2023-10-07T10:44:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f060dffb700 1 -- 172.21.15.29:0/40796543 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060400bc00 con 0x7f061015ade0 2023-10-07T10:44:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f06153ff700 1 -- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061015ade0 msgr2=0x7f06100b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f06153ff700 1 --2- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061015ade0 0x7f06100b1d00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f06040099b0 tx=0x7f0604009250 comp rx=0 tx=0).stop 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.277+0000 7f06153ff700 1 -- 172.21.15.29:0/40796543 shutdown_connections 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.278+0000 7f06153ff700 1 --2- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f06100b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.278+0000 7f06153ff700 1 --2- 172.21.15.29:0/40796543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061015ade0 0x7f06100b1d00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.278+0000 7f06153ff700 1 -- 172.21.15.29:0/40796543 >> 172.21.15.29:0/40796543 conn(0x7f0610067610 msgr2=0x7f0610067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.278+0000 7f06153ff700 1 -- 172.21.15.29:0/40796543 shutdown_connections 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.278+0000 7f06153ff700 1 -- 172.21.15.29:0/40796543 wait complete. 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.279+0000 7f06153ff700 1 Processor -- start 2023-10-07T10:44:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.279+0000 7f06153ff700 1 -- start start 2023-10-07T10:44:37.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f06153ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06100b0000 0x7f061015e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f06153ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f061014aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f06153ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 0x7f061015c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f06153ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061015e940 con 0x7f061015ade0 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f06153ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061015df70 con 0x7f06100b0000 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f06153ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061015d360 con 0x7f0610153cc0 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f060f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 0x7f061015c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f060f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 0x7f061015c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53272/0 (socket says 172.21.15.29:53272) 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.280+0000 7f060f7fe700 1 -- 172.21.15.29:0/1088232802 learned_addr learned my addr 172.21.15.29:0/1088232802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060e7fc700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f061014aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060effd700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06100b0000 0x7f061015e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060f7fe700 1 -- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 msgr2=0x7f061014aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060f7fe700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f061014aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060f7fe700 1 -- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06100b0000 msgr2=0x7f061015e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060f7fe700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06100b0000 0x7f061015e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060f7fe700 1 -- 172.21.15.29:0/1088232802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06040094f0 con 0x7f061015ade0 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f060f7fe700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 0x7f061015c8c0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f060800cd00 tx=0x7f0608001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.281+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060800ce60 con 0x7f061015ade0 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.282+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0608005df0 con 0x7f061015ade0 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.282+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060800ab00 con 0x7f061015ade0 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.282+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061015d640 con 0x7f061015ade0 2023-10-07T10:44:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.282+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061015dc00 con 0x7f061015ade0 2023-10-07T10:44:37.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.283+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061015e0e0 con 0x7f061015ade0 2023-10-07T10:44:37.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.284+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f06080070f0 con 0x7f061015ade0 2023-10-07T10:44:37.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.285+0000 7f05f6ffd700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f05fc066f30 0x7f05fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.285+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f0608084320 con 0x7f061015ade0 2023-10-07T10:44:37.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.285+0000 7f060effd700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f05fc066f30 0x7f05fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.286+0000 7f060effd700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f05fc066f30 0x7f05fc0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0600009da0 tx=0x7f0600009120 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:37.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.289+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06080517c0 con 0x7f061015ade0 2023-10-07T10:44:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.495+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f061004c3c0 con 0x7f061015ade0 2023-10-07T10:44:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.496+0000 7f05f6ffd700 1 -- 172.21.15.29:0/1088232802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0608054de0 con 0x7f061015ade0 2023-10-07T10:44:37.497 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T10:44:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f05fc066f30 msgr2=0x7f05fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f05fc066f30 0x7f05fc0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0600009da0 tx=0x7f0600009120 comp rx=0 tx=0).stop 2023-10-07T10:44:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 msgr2=0x7f061015c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 0x7f061015c8c0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f060800cd00 tx=0x7f0608001b10 comp rx=0 tx=0).stop 2023-10-07T10:44:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 shutdown_connections 2023-10-07T10:44:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f05fc066f30 0x7f05fc0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061015ade0 0x7f061015c8c0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06100b0000 0x7f061015e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 --2- 172.21.15.29:0/1088232802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0610153cc0 0x7f061014aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 >> 172.21.15.29:0/1088232802 conn(0x7f0610067610 msgr2=0x7f061015b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 shutdown_connections 2023-10-07T10:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.498+0000 7f06153ff700 1 -- 172.21.15.29:0/1088232802 wait complete. 2023-10-07T10:44:37.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T10:44:37.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:44:37.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:44:37.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.720+0000 7fd9733ff700 1 Processor -- start 2023-10-07T10:44:37.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.720+0000 7fd9733ff700 1 -- start start 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd9733ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd96c159ec0 0x7fd96c15a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd9733ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 0x7fd96c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd9733ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c14d790 con 0x7fd96c0ad370 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd9733ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c15a880 con 0x7fd96c159ec0 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd9733ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0ae5d0 con 0x7fd96c0af400 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd97119b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd96c0ad370 0x7fd96c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47340/0 (socket says 172.21.15.29:47340) 2023-10-07T10:44:37.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd97119b700 1 -- 172.21.15.29:0/3191653659 learned_addr learned my addr 172.21.15.29:0/3191653659 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd97099a700 1 --2- 172.21.15.29:0/3191653659 >> v2:172.21.15.144:3300/0 conn(0x7fd96c159ec0 0x7fd96c15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592641839 0 0) 0x7fd96c14d790 con 0x7fd96c0ad370 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.722+0000 7fd97199c700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 0x7fd96c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd958003570 con 0x7fd96c0ad370 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320162061 0 0) 0x7fd958003570 con 0x7fd96c0ad370 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 msgr2=0x7fd96c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 0x7fd96c0af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 >> v2:172.21.15.144:3300/0 conn(0x7fd96c159ec0 msgr2=0x7fd96c15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:37.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 --2- 172.21.15.29:0/3191653659 >> v2:172.21.15.144:3300/0 conn(0x7fd96c159ec0 0x7fd96c15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0ad740 con 0x7fd96c0ad370 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fd968001470 con 0x7fd96c0ad370 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 >> v1:172.21.15.29:6789/0 conn(0x7fd96c0ad370 legacy=0x7fd96c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd958003ae0 0x7fd958003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 0x7fd958004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958004d50 0x7fd9580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c15a880 con 0x7fd958003ae0 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0ae5d0 con 0x7fd958004d50 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95800dd20 con 0x7fd96c0af400 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd97119b700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd958003ae0 0x7fd958003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd97099a700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 0x7fd958004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.723+0000 7fd97199c700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958004d50 0x7fd9580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd97119b700 1 -- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 msgr2=0x7fd958004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd97119b700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0af400 0x7fd958004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd97119b700 1 -- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958004d50 msgr2=0x7fd9580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd97119b700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958004d50 0x7fd9580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd97119b700 1 -- 172.21.15.29:0/3191653659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958003570 con 0x7fd958003ae0 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd97119b700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd958003ae0 0x7fd958003ef0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fd96800c2d0 tx=0x7fd96c1578d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee696d3bdd630e08 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:37.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96800cbc0 con 0x7fd958003ae0 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96800cd20 con 0x7fd958003ae0 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.724+0000 7fd963fff700 1 -- 172.21.15.29:0/3191653659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968005b70 con 0x7fd958003ae0 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd9733ff700 1 -- 172.21.15.29:0/3191653659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd958003ae0 msgr2=0x7fd958003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd9733ff700 1 --2- 172.21.15.29:0/3191653659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd958003ae0 0x7fd958003ef0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fd96800c2d0 tx=0x7fd96c1578d0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd97199c700 1 -- 172.21.15.29:0/3191653659 reap_dead start 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd9733ff700 1 -- 172.21.15.29:0/3191653659 shutdown_connections 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd9733ff700 1 -- 172.21.15.29:0/3191653659 >> 172.21.15.29:0/3191653659 conn(0x7fd96c0680a0 msgr2=0x7fd96c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd9733ff700 1 -- 172.21.15.29:0/3191653659 shutdown_connections 2023-10-07T10:44:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.725+0000 7fd9733ff700 1 -- 172.21.15.29:0/3191653659 wait complete. 2023-10-07T10:44:37.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.726+0000 7fd9733ff700 1 Processor -- start 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.727+0000 7fd9733ff700 1 -- start start 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd9733ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958003ae0 0x7fd96c14c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd9733ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0c2720 0x7fd96c0c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd9733ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd96c0cb2d0 0x7fd96c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd9733ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0cae40 con 0x7fd96c0cb2d0 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd9733ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0c9f50 con 0x7fd958003ae0 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd9733ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0ca230 con 0x7fd96c0c2720 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97119b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958003ae0 0x7fd96c14c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97119b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958003ae0 0x7fd96c14c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:58202/0 (socket says 172.21.15.29:58202) 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97119b700 1 -- 172.21.15.29:0/165003520 learned_addr learned my addr 172.21.15.29:0/165003520 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97199c700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd96c0cb2d0 0x7fd96c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97099a700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0c2720 0x7fd96c0c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97099a700 1 -- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958003ae0 msgr2=0x7fd96c14c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97099a700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958003ae0 0x7fd96c14c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97099a700 1 -- 172.21.15.29:0/165003520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd96c0cb2d0 msgr2=0x7fd96c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97099a700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd96c0cb2d0 0x7fd96c0cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97099a700 1 -- 172.21.15.29:0/165003520 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0ca510 con 0x7fd96c0c2720 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.728+0000 7fd97199c700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd96c0cb2d0 0x7fd96c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.729+0000 7fd97099a700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0c2720 0x7fd96c0c2b90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd95c011b10 tx=0x7fd95c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.729+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00f940 con 0x7fd96c0c2720 2023-10-07T10:44:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.729+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c00faa0 con 0x7fd96c0c2720 2023-10-07T10:44:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.729+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00fe40 con 0x7fd96c0c2720 2023-10-07T10:44:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.729+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c0ca7f0 con 0x7fd96c0c2720 2023-10-07T10:44:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.729+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c14c7c0 con 0x7fd96c0c2720 2023-10-07T10:44:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.730+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c04c3c0 con 0x7fd96c0c2720 2023-10-07T10:44:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.731+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd95c014650 con 0x7fd96c0c2720 2023-10-07T10:44:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.732+0000 7fd9617fa700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd950066f30 0x7fd9500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.732+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fd95c018030 con 0x7fd96c0c2720 2023-10-07T10:44:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.732+0000 7fd97119b700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd950066f30 0x7fd9500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.733+0000 7fd97119b700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd950066f30 0x7fd9500693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd968007c60 tx=0x7fd968015000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:37.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.737+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd95c051f90 con 0x7fd96c0c2720 2023-10-07T10:44:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.940+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd96c0cca90 con 0x7fd96c0c2720 2023-10-07T10:44:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.941+0000 7fd9617fa700 1 -- 172.21.15.29:0/165003520 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd95c022090 con 0x7fd96c0c2720 2023-10-07T10:44:37.943 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:37.943 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T10:44:37.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:37.944 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T10:44:37.944 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T10:44:37.944 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:37.944 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T10:44:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd950066f30 msgr2=0x7fd9500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd950066f30 0x7fd9500693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd968007c60 tx=0x7fd968015000 comp rx=0 tx=0).stop 2023-10-07T10:44:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0c2720 msgr2=0x7fd96c0c2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0c2720 0x7fd96c0c2b90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd95c011b10 tx=0x7fd95c0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 shutdown_connections 2023-10-07T10:44:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd950066f30 0x7fd9500693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd96c0cb2d0 0x7fd96c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd958003ae0 0x7fd96c14c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 --2- 172.21.15.29:0/165003520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd96c0c2720 0x7fd96c0c2b90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 >> 172.21.15.29:0/165003520 conn(0x7fd96c0680a0 msgr2=0x7fd96c06d760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 shutdown_connections 2023-10-07T10:44:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:37.943+0000 7fd9733ff700 1 -- 172.21.15.29:0/165003520 wait complete. 2023-10-07T10:44:37.962 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T10:44:37.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T10:44:38.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.205+0000 7fa8a5dff700 1 Processor -- start 2023-10-07T10:44:38.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.205+0000 7fa8a5dff700 1 -- start start 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa8a5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa8a00b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa8a5dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa8a00c8a20 0x7fa8a00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa8a5dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a00690f0 con 0x7fa8a00b0610 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa8a5dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a00cf2e0 con 0x7fa8a00c8a20 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa8a5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00687c0 con 0x7fa8a00ced00 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa89f7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8a00b0610 0x7fa8a00c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47354/0 (socket says 172.21.15.29:47354) 2023-10-07T10:44:38.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa89f7fe700 1 -- 172.21.15.29:0/1142846773 learned_addr learned my addr 172.21.15.29:0/1142846773 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa89effd700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa8a00b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa89ffff700 1 --2- 172.21.15.29:0/1142846773 >> v2:172.21.15.144:3300/0 conn(0x7fa8a00c8a20 0x7fa8a00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815708239 0 0) 0x7fa8a00690f0 con 0x7fa8a00b0610 2023-10-07T10:44:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.207+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888003570 con 0x7fa8a00b0610 2023-10-07T10:44:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177416800 0 0) 0x7fa888003570 con 0x7fa8a00b0610 2023-10-07T10:44:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 msgr2=0x7fa8a00b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa8a00b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 >> v2:172.21.15.144:3300/0 conn(0x7fa8a00c8a20 msgr2=0x7fa8a00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 --2- 172.21.15.29:0/1142846773 >> v2:172.21.15.144:3300/0 conn(0x7fa8a00c8a20 0x7fa8a00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a0068aa0 con 0x7fa8a00b0610 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fa890001470 con 0x7fa8a00b0610 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89ffff700 1 --2- 172.21.15.29:0/1142846773 >> v2:172.21.15.144:3300/0 conn(0x7fa8a00c8a20 0x7fa8a00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 >> v1:172.21.15.29:6789/0 conn(0x7fa8a00b0610 legacy=0x7fa8a00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa888003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8880040b0 0x7fa888004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00cf2e0 con 0x7fa888004d50 2023-10-07T10:44:38.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00687c0 con 0x7fa8880040b0 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88800bb60 con 0x7fa8a00ced00 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89ffff700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89f7fe700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa888003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89effd700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8880040b0 0x7fa888004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89ffff700 1 -- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 msgr2=0x7fa888003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89ffff700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa888003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89ffff700 1 -- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8880040b0 msgr2=0x7fa888004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.208+0000 7fa89ffff700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8880040b0 0x7fa888004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89ffff700 1 -- 172.21.15.29:0/1142846773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888003570 con 0x7fa888004d50 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89f7fe700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a00ced00 0x7fa888003ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89ffff700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8880051c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fa8980094d0 tx=0x7fa89800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3de01ce991ec91 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:38.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89800ec40 con 0x7fa888004d50 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89800f040 con 0x7fa888004d50 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89e7fc700 1 -- 172.21.15.29:0/1142846773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898014480 con 0x7fa888004d50 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa8a5dff700 1 -- 172.21.15.29:0/1142846773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 msgr2=0x7fa8880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/1142846773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8880051c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fa8980094d0 tx=0x7fa89800b040 comp rx=0 tx=0).stop 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.209+0000 7fa89ffff700 1 -- 172.21.15.29:0/1142846773 reap_dead start 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.210+0000 7fa8a5dff700 1 -- 172.21.15.29:0/1142846773 shutdown_connections 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.210+0000 7fa8a5dff700 1 -- 172.21.15.29:0/1142846773 >> 172.21.15.29:0/1142846773 conn(0x7fa8a0063a10 msgr2=0x7fa8a00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:38.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.210+0000 7fa8a5dff700 1 -- 172.21.15.29:0/1142846773 shutdown_connections 2023-10-07T10:44:38.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.210+0000 7fa8a5dff700 1 -- 172.21.15.29:0/1142846773 wait complete. 2023-10-07T10:44:38.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.211+0000 7fa8a5dff700 1 Processor -- start 2023-10-07T10:44:38.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.211+0000 7fa8a5dff700 1 -- start start 2023-10-07T10:44:38.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa8a5dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8a00b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa8a5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a0151590 0x7fa8a016cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa8a5dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8a016d490 0x7fa8a00b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa8a5dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a016d930 con 0x7fa888004d50 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa8a5dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0152660 con 0x7fa8a016d490 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa8a5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0158880 con 0x7fa8a0151590 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa89f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8a00b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa89f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8a00b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53302/0 (socket says 172.21.15.29:53302) 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.212+0000 7fa89f7fe700 1 -- 172.21.15.29:0/2908106023 learned_addr learned my addr 172.21.15.29:0/2908106023 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89effd700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a0151590 0x7fa8a016cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89ffff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8a016d490 0x7fa8a00b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89f7fe700 1 -- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a0151590 msgr2=0x7fa8a016cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89f7fe700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a0151590 0x7fa8a016cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89f7fe700 1 -- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8a016d490 msgr2=0x7fa8a00b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:38.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89f7fe700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8a016d490 0x7fa8a00b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89f7fe700 1 -- 172.21.15.29:0/2908106023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a00b3c30 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89ffff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8a016d490 0x7fa8a00b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa89f7fe700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8a00b36f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fa89000eb00 tx=0x7fa890001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000a760 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa890010040 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a0158ad0 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.213+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000aad0 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.214+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a0152870 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.214+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a0068580 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.215+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa8900143f0 con 0x7fa888004d50 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.216+0000 7fa8877fe700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa88c066f30 0x7fa88c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.216+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fa890083770 con 0x7fa888004d50 2023-10-07T10:44:38.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.217+0000 7fa89effd700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa88c066f30 0x7fa88c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:38.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.217+0000 7fa89effd700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa88c066f30 0x7fa88c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00b2de0 tx=0x7fa8940091a0 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:38.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.221+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa89004d5a0 con 0x7fa888004d50 2023-10-07T10:44:38.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:38.424+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa8a00718b0 con 0x7fa888004d50 2023-10-07T10:44:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.109+0000 7fa8877fe700 1 -- 172.21.15.29:0/2908106023 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v148) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa890019020 con 0x7fa888004d50 2023-10-07T10:44:39.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.114+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa88c066f30 msgr2=0x7fa88c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.114+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa88c066f30 0x7fa88c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00b2de0 tx=0x7fa8940091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.114+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 msgr2=0x7fa8a00b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.114+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8a00b36f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fa89000eb00 tx=0x7fa890001e40 comp rx=0 tx=0).stop 2023-10-07T10:44:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.115+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 shutdown_connections 2023-10-07T10:44:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.115+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa88c066f30 0x7fa88c0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.115+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8a016d490 0x7fa8a00b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.115+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa888004d50 0x7fa8a00b36f0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.115+0000 7fa8a5dff700 1 --2- 172.21.15.29:0/2908106023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8a0151590 0x7fa8a016cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.115+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 >> 172.21.15.29:0/2908106023 conn(0x7fa8a0063a10 msgr2=0x7fa8a0151f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:39.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.116+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 shutdown_connections 2023-10-07T10:44:39.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.116+0000 7fa8a5dff700 1 -- 172.21.15.29:0/2908106023 wait complete. 2023-10-07T10:44:39.121 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T10:44:39.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T10:44:39.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.345+0000 7fc6d4fff700 1 Processor -- start 2023-10-07T10:44:39.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.345+0000 7fc6d4fff700 1 -- start start 2023-10-07T10:44:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.346+0000 7fc6d4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0159e10 0x7fc6d0160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.346+0000 7fc6d4fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d00c97e0 con 0x7fc6d0155cb0 2023-10-07T10:44:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.346+0000 7fc6d4fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d00c8bd0 con 0x7fc6d01558e0 2023-10-07T10:44:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.346+0000 7fc6d4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c8eb0 con 0x7fc6d0159e10 2023-10-07T10:44:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.346+0000 7fc6cdd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc6d0155cb0 0x7fc6d0159700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47368/0 (socket says 172.21.15.29:47368) 2023-10-07T10:44:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.346+0000 7fc6cdd9b700 1 -- 172.21.15.29:0/1094876443 learned_addr learned my addr 172.21.15.29:0/1094876443 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6ced9d700 1 --2- 172.21.15.29:0/1094876443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0159e10 0x7fc6d0160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443849762 0 0) 0x7fc6d00c97e0 con 0x7fc6d0155cb0 2023-10-07T10:44:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b8003570 con 0x7fc6d0155cb0 2023-10-07T10:44:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140231413 0 0) 0x7fc6d00c8bd0 con 0x7fc6d01558e0 2023-10-07T10:44:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d00c97e0 con 0x7fc6d01558e0 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6ced9d700 1 -- 172.21.15.29:0/1094876443 >> v1:172.21.15.144:6789/0 conn(0x7fc6d01558e0 legacy=0x7fc6d00b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6ced9d700 1 -- 172.21.15.29:0/1094876443 >> v1:172.21.15.29:6789/0 conn(0x7fc6d0155cb0 legacy=0x7fc6d0159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.347+0000 7fc6ced9d700 1 -- 172.21.15.29:0/1094876443 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00c9190 con 0x7fc6d0159e10 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6ced9d700 1 --2- 172.21.15.29:0/1094876443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0159e10 0x7fc6d0160aa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800afd0 tx=0x7fc6c800a890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290684bcb1cf815e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c800f070 con 0x7fc6d0159e10 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c800a2e0 con 0x7fc6d0159e10 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6cd59a700 1 -- 172.21.15.29:0/1094876443 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c800cbe0 con 0x7fc6d0159e10 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1094876443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0159e10 msgr2=0x7fc6d0160aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1094876443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0159e10 0x7fc6d0160aa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800afd0 tx=0x7fc6c800a890 comp rx=0 tx=0).stop 2023-10-07T10:44:39.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1094876443 shutdown_connections 2023-10-07T10:44:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1094876443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0159e10 0x7fc6d0160aa0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.348+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1094876443 >> 172.21.15.29:0/1094876443 conn(0x7fc6d0067610 msgr2=0x7fc6d0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.349+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1094876443 shutdown_connections 2023-10-07T10:44:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.349+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1094876443 wait complete. 2023-10-07T10:44:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.350+0000 7fc6d4fff700 1 Processor -- start 2023-10-07T10:44:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.350+0000 7fc6d4fff700 1 -- start start 2023-10-07T10:44:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6d4fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 0x7fc6d0150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6d4fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0155cb0 0x7fc6d00c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6d4fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d0159e10 0x7fc6d00afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6d4fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d01514f0 con 0x7fc6d01558e0 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6d4fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c6ef0 con 0x7fc6d0159e10 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6d4fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c62e0 con 0x7fc6d0155cb0 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.351+0000 7fc6ce59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 0x7fc6d0150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 0x7fc6d0150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53312/0 (socket says 172.21.15.29:53312) 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 -- 172.21.15.29:0/1548682770 learned_addr learned my addr 172.21.15.29:0/1548682770 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ced9d700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d0159e10 0x7fc6d00afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6cdd9b700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0155cb0 0x7fc6d00c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 -- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0155cb0 msgr2=0x7fc6d00c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0155cb0 0x7fc6d00c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 -- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d0159e10 msgr2=0x7fc6d00afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d0159e10 0x7fc6d00afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 -- 172.21.15.29:0/1548682770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c800ab30 con 0x7fc6d01558e0 2023-10-07T10:44:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6ce59c700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 0x7fc6d0150fb0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000c790 tx=0x7fc6c0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.352+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0010510 con 0x7fc6d01558e0 2023-10-07T10:44:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.353+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c0010670 con 0x7fc6d01558e0 2023-10-07T10:44:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.353+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0010a10 con 0x7fc6d01558e0 2023-10-07T10:44:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.353+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d00c65c0 con 0x7fc6d01558e0 2023-10-07T10:44:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.353+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d00c6b80 con 0x7fc6d01558e0 2023-10-07T10:44:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.354+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d00c71d0 con 0x7fc6d01558e0 2023-10-07T10:44:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.356+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc6c0025430 con 0x7fc6d01558e0 2023-10-07T10:44:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.357+0000 7fc6be7fc700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6b0066f30 0x7fc6b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.357+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5643+0+0 (secure 0 0 0) 0x7fc6c0084df0 con 0x7fc6d01558e0 2023-10-07T10:44:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.358+0000 7fc6cdd9b700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6b0066f30 0x7fc6b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:39.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.358+0000 7fc6cdd9b700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6b0066f30 0x7fc6b00693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00b0f20 tx=0x7fc6c4009040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.361+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6c0052130 con 0x7fc6d01558e0 2023-10-07T10:44:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:39.566+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc6d00718b0 con 0x7fc6d01558e0 2023-10-07T10:44:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.117+0000 7fc6be7fc700 1 -- 172.21.15.29:0/1548682770 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v149) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc6c0055750 con 0x7fc6d01558e0 2023-10-07T10:44:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.122+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6b0066f30 msgr2=0x7fc6b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.122+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6b0066f30 0x7fc6b00693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00b0f20 tx=0x7fc6c4009040 comp rx=0 tx=0).stop 2023-10-07T10:44:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.122+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 msgr2=0x7fc6d0150fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.122+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 0x7fc6d0150fb0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000c790 tx=0x7fc6c0007e10 comp rx=0 tx=0).stop 2023-10-07T10:44:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.126+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 shutdown_connections 2023-10-07T10:44:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.126+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6b0066f30 0x7fc6b00693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.126+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d0159e10 0x7fc6d00afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.126+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d01558e0 0x7fc6d0150fb0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.126+0000 7fc6d4fff700 1 --2- 172.21.15.29:0/1548682770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d0155cb0 0x7fc6d00c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.126+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 >> 172.21.15.29:0/1548682770 conn(0x7fc6d0067610 msgr2=0x7fc6d0156130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.127+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 shutdown_connections 2023-10-07T10:44:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.127+0000 7fc6d4fff700 1 -- 172.21.15.29:0/1548682770 wait complete. 2023-10-07T10:44:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T10:44:40.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T10:44:40.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.358+0000 7f50d1bff700 1 Processor -- start 2023-10-07T10:44:40.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.359+0000 7f50d1bff700 1 -- start start 2023-10-07T10:44:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50d1bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50cc06a8c0 0x7f50cc0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50d1bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc06a430 con 0x7f50cc06b5c0 2023-10-07T10:44:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50d1bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc159b60 con 0x7f50cc159790 2023-10-07T10:44:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50d1bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc069820 con 0x7f50cc06a8c0 2023-10-07T10:44:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50cbfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f50cc06b5c0 0x7f50cc0c7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47382/0 (socket says 172.21.15.29:47382) 2023-10-07T10:44:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50cbfff700 1 -- 172.21.15.29:0/2541380108 learned_addr learned my addr 172.21.15.29:0/2541380108 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.360+0000 7f50caffd700 1 --2- 172.21.15.29:0/2541380108 >> v2:172.21.15.144:3300/0 conn(0x7f50cc06a8c0 0x7f50cc0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730092442 0 0) 0x7f50cc06a430 con 0x7f50cc06b5c0 2023-10-07T10:44:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4003570 con 0x7f50cc06b5c0 2023-10-07T10:44:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516022242 0 0) 0x7f50b4003570 con 0x7f50cc06b5c0 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 >> v2:172.21.15.144:3300/0 conn(0x7f50cc06a8c0 msgr2=0x7f50cc0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 --2- 172.21.15.29:0/2541380108 >> v2:172.21.15.144:3300/0 conn(0x7f50cc06a8c0 0x7f50cc0c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 >> v1:172.21.15.144:6789/0 conn(0x7f50cc159790 legacy=0x7f50cc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc069b00 con 0x7f50cc06b5c0 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f50c40017f0 con 0x7f50cc06b5c0 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 >> v1:172.21.15.29:6789/0 conn(0x7f50cc06b5c0 legacy=0x7f50cc0c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4003a60 0x7f50b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b4004810 0x7f50b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b40051c0 0x7f50b400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc069820 con 0x7f50b4004810 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b400c0d0 con 0x7f50b4003a60 2023-10-07T10:44:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b400c290 con 0x7f50b40051c0 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50caffd700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b4004810 0x7f50b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.361+0000 7f50cb7fe700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4003a60 0x7f50b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50cbfff700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b40051c0 0x7f50b400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50caffd700 1 -- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b40051c0 msgr2=0x7f50b400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50caffd700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b40051c0 0x7f50b400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50caffd700 1 -- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4003a60 msgr2=0x7f50b4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50caffd700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4003a60 0x7f50b4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50caffd700 1 -- 172.21.15.29:0/2541380108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4003570 con 0x7f50b4004810 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50cbfff700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b40051c0 0x7f50b400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50cbfff700 1 -- 172.21.15.29:0/2541380108 reap_dead start 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50caffd700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b4004810 0x7f50b4004c80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009c00 tx=0x7f50c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3c4da38f2a1f85 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c000e710 con 0x7f50b4004810 2023-10-07T10:44:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c000e870 con 0x7f50b4004810 2023-10-07T10:44:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.362+0000 7f50ca7fc700 1 -- 172.21.15.29:0/2541380108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c000ec30 con 0x7f50b4004810 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 -- 172.21.15.29:0/2541380108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b4004810 msgr2=0x7f50b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b4004810 0x7f50b4004c80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009c00 tx=0x7f50c00091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 -- 172.21.15.29:0/2541380108 shutdown_connections 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 --2- 172.21.15.29:0/2541380108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b4004810 0x7f50b4004c80 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 -- 172.21.15.29:0/2541380108 >> 172.21.15.29:0/2541380108 conn(0x7f50cc0c9410 msgr2=0x7f50cc0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 -- 172.21.15.29:0/2541380108 shutdown_connections 2023-10-07T10:44:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.363+0000 7f50d1bff700 1 -- 172.21.15.29:0/2541380108 wait complete. 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.365+0000 7f50d1bff700 1 Processor -- start 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.365+0000 7f50d1bff700 1 -- start start 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50d1bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4004810 0x7f50cc151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50d1bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 0x7f50cc0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50d1bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50cc06dc70 0x7f50cc06e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50d1bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0b17e0 con 0x7f50cc0ac100 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50d1bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc06e630 con 0x7f50b4004810 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50d1bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0b2310 con 0x7f50cc06dc70 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 0x7f50cc0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 0x7f50cc0ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53326/0 (socket says 172.21.15.29:53326) 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 -- 172.21.15.29:0/3751468889 learned_addr learned my addr 172.21.15.29:0/3751468889 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50cb7fe700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4004810 0x7f50cc151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50cbfff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50cc06dc70 0x7f50cc06e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 -- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50cc06dc70 msgr2=0x7f50cc06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50cc06dc70 0x7f50cc06e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 -- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4004810 msgr2=0x7f50cc151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4004810 0x7f50cc151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.366+0000 7f50caffd700 1 -- 172.21.15.29:0/3751468889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc0b19e0 con 0x7f50cc0ac100 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.367+0000 7f50caffd700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 0x7f50cc0ae6a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f50c000e4b0 tx=0x7f50c001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.367+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0022070 con 0x7f50cc0ac100 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.367+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c000e9e0 con 0x7f50cc0ac100 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.367+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c001c450 con 0x7f50cc0ac100 2023-10-07T10:44:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.367+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc0b1cc0 con 0x7f50cc0ac100 2023-10-07T10:44:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.367+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc0aebe0 con 0x7f50cc0ac100 2023-10-07T10:44:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.368+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc0b24c0 con 0x7f50cc0ac100 2023-10-07T10:44:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.370+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f50c001c8a0 con 0x7f50cc0ac100 2023-10-07T10:44:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.370+0000 7f50b37fe700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50b8066f30 0x7f50b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:40.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.370+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f50c0086210 con 0x7f50cc0ac100 2023-10-07T10:44:40.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.371+0000 7f50cb7fe700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50b8066f30 0x7f50b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:40.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.371+0000 7f50cb7fe700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50b8066f30 0x7f50b80693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0c6b90 tx=0x7f50bc007d30 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.374+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50c00533e0 con 0x7f50cc0ac100 2023-10-07T10:44:40.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:40.621+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f50cc0b2160 con 0x7f50cc0ac100 2023-10-07T10:44:41.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.122+0000 7f50b37fe700 1 -- 172.21.15.29:0/3751468889 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v73) v1 ==== 174+0+0 (secure 0 0 0) 0x7f50c0056a00 con 0x7f50cc0ac100 2023-10-07T10:44:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.127+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50b8066f30 msgr2=0x7f50b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.127+0000 7f50d1bff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50b8066f30 0x7f50b80693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0c6b90 tx=0x7f50bc007d30 comp rx=0 tx=0).stop 2023-10-07T10:44:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.127+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 msgr2=0x7f50cc0ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.127+0000 7f50d1bff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 0x7f50cc0ae6a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f50c000e4b0 tx=0x7f50c001a000 comp rx=0 tx=0).stop 2023-10-07T10:44:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.131+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 shutdown_connections 2023-10-07T10:44:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.131+0000 7f50d1bff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f50b8066f30 0x7f50b80693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.131+0000 7f50d1bff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50cc06dc70 0x7f50cc06e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.131+0000 7f50d1bff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b4004810 0x7f50cc151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.131+0000 7f50d1bff700 1 --2- 172.21.15.29:0/3751468889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50cc0ac100 0x7f50cc0ae6a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.131+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 >> 172.21.15.29:0/3751468889 conn(0x7f50cc0c9410 msgr2=0x7f50cc0c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:41.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.133+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 shutdown_connections 2023-10-07T10:44:41.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.141+0000 7f50d1bff700 1 -- 172.21.15.29:0/3751468889 wait complete. 2023-10-07T10:44:41.153 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 18 and data pool 19 2023-10-07T10:44:41.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.367+0000 7fc582dff700 1 Processor -- start 2023-10-07T10:44:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.367+0000 7fc582dff700 1 -- start start 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc582dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c159550 0x7fc57c0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc582dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc57c0b0f50 0x7fc57c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc582dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c06f4e0 con 0x7fc57c0b2bb0 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc582dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c06e8d0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc582dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c06ebb0 con 0x7fc57c159550 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc57bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc57c0b0f50 0x7fc57c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc57bfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc57c0b0f50 0x7fc57c0b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53330/0 (socket says 172.21.15.29:53330) 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.368+0000 7fc57bfff700 1 -- 172.21.15.29:0/1923438306 learned_addr learned my addr 172.21.15.29:0/1923438306 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57bfff700 1 -- 172.21.15.29:0/1923438306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c159550 msgr2=0x7fc57c0b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57bfff700 1 --2- 172.21.15.29:0/1923438306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c159550 0x7fc57c0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57bfff700 1 -- 172.21.15.29:0/1923438306 >> v1:172.21.15.144:6789/0 conn(0x7fc57c0b2bb0 legacy=0x7fc57c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57bfff700 1 -- 172.21.15.29:0/1923438306 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c06ee90 con 0x7fc57c0b0f50 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57bfff700 1 --2- 172.21.15.29:0/1923438306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c0b2670 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc5680099b0 tx=0x7fc568009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51938e0c16588dbe server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57b7fe700 1 -- 172.21.15.29:0/1923438306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56800b820 con 0x7fc57c0b0f50 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57b7fe700 1 -- 172.21.15.29:0/1923438306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56800b980 con 0x7fc57c0b0f50 2023-10-07T10:44:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.369+0000 7fc57b7fe700 1 -- 172.21.15.29:0/1923438306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56800bd20 con 0x7fc57c0b0f50 2023-10-07T10:44:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 -- 172.21.15.29:0/1923438306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 msgr2=0x7fc57c0b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 --2- 172.21.15.29:0/1923438306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c0b2670 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc5680099b0 tx=0x7fc568009250 comp rx=0 tx=0).stop 2023-10-07T10:44:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 -- 172.21.15.29:0/1923438306 shutdown_connections 2023-10-07T10:44:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 --2- 172.21.15.29:0/1923438306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c159550 0x7fc57c0b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 --2- 172.21.15.29:0/1923438306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c0b2670 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 -- 172.21.15.29:0/1923438306 >> 172.21.15.29:0/1923438306 conn(0x7fc57c069190 msgr2=0x7fc57c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 -- 172.21.15.29:0/1923438306 shutdown_connections 2023-10-07T10:44:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.370+0000 7fc582dff700 1 -- 172.21.15.29:0/1923438306 wait complete. 2023-10-07T10:44:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.372+0000 7fc582dff700 1 Processor -- start 2023-10-07T10:44:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.372+0000 7fc582dff700 1 -- start start 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc582dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc582dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c0b2bb0 0x7fc57c0c2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc582dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc57c159550 0x7fc57c0c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc582dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c153450 con 0x7fc57c0b0f50 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc582dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c158920 con 0x7fc57c159550 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc582dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c157d50 con 0x7fc57c0b2bb0 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53346/0 (socket says 172.21.15.29:53346) 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 -- 172.21.15.29:0/2993091801 learned_addr learned my addr 172.21.15.29:0/2993091801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc57bfff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c0b2bb0 0x7fc57c0c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc58139c700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc57c159550 0x7fc57c0c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 -- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c0b2bb0 msgr2=0x7fc57c0c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c0b2bb0 0x7fc57c0c2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 -- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc57c159550 msgr2=0x7fc57c0c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc57c159550 0x7fc57c0c2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.373+0000 7fc580b9b700 1 -- 172.21.15.29:0/2993091801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680094f0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.374+0000 7fc580b9b700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c152f10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc57000b810 tx=0x7fc570009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.374+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000f5f0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.374+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57000f770 con 0x7fc57c0b0f50 2023-10-07T10:44:41.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.374+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c157ff0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.374+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000fb30 con 0x7fc57c0b0f50 2023-10-07T10:44:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.374+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c1585b0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.375+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c04c3c0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.376+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc570010480 con 0x7fc57c0b0f50 2023-10-07T10:44:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.377+0000 7fc578ff9700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc560066f30 0x7fc5600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.377+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fc5700846b0 con 0x7fc57c0b0f50 2023-10-07T10:44:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.378+0000 7fc57bfff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc560066f30 0x7fc5600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.378+0000 7fc57bfff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc560066f30 0x7fc5600693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5680099b0 tx=0x7fc568016000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.381+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc570051760 con 0x7fc57c0b0f50 2023-10-07T10:44:41.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.584+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc57c0af030 con 0x7fc57c0b0f50 2023-10-07T10:44:41.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.585+0000 7fc578ff9700 1 -- 172.21.15.29:0/2993091801 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16115 (secure 0 0 0) 0x7fc570054d80 con 0x7fc57c0b0f50 2023-10-07T10:44:41.586 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:41.586 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":150,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:40.623068+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:44:38.468584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-07T10:44:39.609895+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.587+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc560066f30 msgr2=0x7fc5600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.587+0000 7fc582dff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc560066f30 0x7fc5600693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5680099b0 tx=0x7fc568016000 comp rx=0 tx=0).stop 2023-10-07T10:44:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.587+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 msgr2=0x7fc57c152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.587+0000 7fc582dff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c152f10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc57000b810 tx=0x7fc570009250 comp rx=0 tx=0).stop 2023-10-07T10:44:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 shutdown_connections 2023-10-07T10:44:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc560066f30 0x7fc5600693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc57c159550 0x7fc57c0c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc57c0b0f50 0x7fc57c152f10 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 --2- 172.21.15.29:0/2993091801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc57c0b2bb0 0x7fc57c0c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 >> 172.21.15.29:0/2993091801 conn(0x7fc57c069190 msgr2=0x7fc57c0ab270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 shutdown_connections 2023-10-07T10:44:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.588+0000 7fc582dff700 1 -- 172.21.15.29:0/2993091801 wait complete. 2023-10-07T10:44:41.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:41.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.801+0000 7f2acfbff700 1 Processor -- start 2023-10-07T10:44:41.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.801+0000 7f2acfbff700 1 -- start start 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.802+0000 7f2acfbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159b20 0x7f2ac80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.802+0000 7f2acfbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ac80ce8a0 0x7f2ac80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.802+0000 7f2acfbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80cbf20 con 0x7f2ac8159750 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.802+0000 7f2acfbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac80cc200 con 0x7f2ac80ce8a0 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.802+0000 7f2acfbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80cc4e0 con 0x7f2ac8159b20 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ac80ce8a0 0x7f2ac80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ac80ce8a0 0x7f2ac80c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53356/0 (socket says 172.21.15.29:53356) 2023-10-07T10:44:41.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 -- 172.21.15.29:0/3527715517 learned_addr learned my addr 172.21.15.29:0/3527715517 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:41.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2acd19a700 1 --2- 172.21.15.29:0/3527715517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159b20 0x7f2ac80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 -- 172.21.15.29:0/3527715517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159b20 msgr2=0x7f2ac80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:41.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 --2- 172.21.15.29:0/3527715517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159b20 0x7f2ac80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 -- 172.21.15.29:0/3527715517 >> v1:172.21.15.144:6789/0 conn(0x7f2ac8159750 legacy=0x7f2ac80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 -- 172.21.15.29:0/3527715517 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80cc7c0 con 0x7f2ac80ce8a0 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.803+0000 7f2ace19c700 1 --2- 172.21.15.29:0/3527715517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac80c5460 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009f20 tx=0x7f2ac0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4a0a727cc0ffcf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acc999700 1 -- 172.21.15.29:0/3527715517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000e070 con 0x7f2ac80ce8a0 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acc999700 1 -- 172.21.15.29:0/3527715517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac000b9d0 con 0x7f2ac80ce8a0 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acc999700 1 -- 172.21.15.29:0/3527715517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac00092e0 con 0x7f2ac80ce8a0 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 -- 172.21.15.29:0/3527715517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac80ce8a0 msgr2=0x7f2ac80c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 --2- 172.21.15.29:0/3527715517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac80c5460 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009f20 tx=0x7f2ac0009840 comp rx=0 tx=0).stop 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 -- 172.21.15.29:0/3527715517 shutdown_connections 2023-10-07T10:44:41.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 --2- 172.21.15.29:0/3527715517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac80c5460 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 --2- 172.21.15.29:0/3527715517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159b20 0x7f2ac80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 -- 172.21.15.29:0/3527715517 >> 172.21.15.29:0/3527715517 conn(0x7f2ac8063a10 msgr2=0x7f2ac80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.804+0000 7f2acfbff700 1 -- 172.21.15.29:0/3527715517 shutdown_connections 2023-10-07T10:44:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.805+0000 7f2acfbff700 1 -- 172.21.15.29:0/3527715517 wait complete. 2023-10-07T10:44:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.806+0000 7f2acfbff700 1 Processor -- start 2023-10-07T10:44:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.806+0000 7f2acfbff700 1 -- start start 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2acfbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac8154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2acfbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159750 0x7f2ac8151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2acfbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 0x7f2ac80b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2acfbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8154c70 con 0x7f2ac8159b20 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2acfbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac815e8b0 con 0x7f2ac80ce8a0 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2acfbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81549d0 con 0x7f2ac8159750 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2ace19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 0x7f2ac80b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.807+0000 7f2ace19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 0x7f2ac80b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53360/0 (socket says 172.21.15.29:53360) 2023-10-07T10:44:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 -- 172.21.15.29:0/509089648 learned_addr learned my addr 172.21.15.29:0/509089648 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2acd19a700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159750 0x7f2ac8151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2acd99b700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac8154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 -- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159750 msgr2=0x7f2ac8151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159750 0x7f2ac8151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 -- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ac80ce8a0 msgr2=0x7f2ac8154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac8154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 -- 172.21.15.29:0/509089648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0009ae0 con 0x7f2ac8159b20 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2ace19c700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 0x7f2ac80b3620 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b610 tx=0x7f2ac0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.808+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0008c30 con 0x7f2ac8159b20 2023-10-07T10:44:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.809+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0008d90 con 0x7f2ac8159b20 2023-10-07T10:44:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.809+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0026690 con 0x7f2ac8159b20 2023-10-07T10:44:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.809+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac815eb50 con 0x7f2ac8159b20 2023-10-07T10:44:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.809+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac815f110 con 0x7f2ac8159b20 2023-10-07T10:44:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.810+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac804c3c0 con 0x7f2ac8159b20 2023-10-07T10:44:41.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.811+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ac0003040 con 0x7f2ac8159b20 2023-10-07T10:44:41.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.811+0000 7f2abdffb700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2aac066f30 0x7f2aac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:41.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.811+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f2ac0086720 con 0x7f2ac8159b20 2023-10-07T10:44:41.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.812+0000 7f2acd99b700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2aac066f30 0x7f2aac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:41.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.812+0000 7f2acd99b700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2aac066f30 0x7f2aac0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4003890 tx=0x7f2ac4009040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:41.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:41.816+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ac0053750 con 0x7f2ac8159b20 2023-10-07T10:44:42.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.020+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ac815f300 con 0x7f2ac8159b20 2023-10-07T10:44:42.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.021+0000 7f2abdffb700 1 -- 172.21.15.29:0/509089648 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16115 (secure 0 0 0) 0x7f2ac0056d70 con 0x7f2ac8159b20 2023-10-07T10:44:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:42.022 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":150,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:40.623068+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:44:38.468584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-07T10:44:39.609895+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.023+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2aac066f30 msgr2=0x7f2aac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.023+0000 7f2acfbff700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2aac066f30 0x7f2aac0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4003890 tx=0x7f2ac4009040 comp rx=0 tx=0).stop 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.023+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 msgr2=0x7f2ac80b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.023+0000 7f2acfbff700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 0x7f2ac80b3620 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b610 tx=0x7f2ac0018000 comp rx=0 tx=0).stop 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 shutdown_connections 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f2aac066f30 0x7f2aac0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8159b20 0x7f2ac80b3620 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ac80ce8a0 0x7f2ac8154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 --2- 172.21.15.29:0/509089648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ac8159750 0x7f2ac8151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 >> 172.21.15.29:0/509089648 conn(0x7f2ac8063a10 msgr2=0x7f2ac80c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:42.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 shutdown_connections 2023-10-07T10:44:42.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.024+0000 7f2acfbff700 1 -- 172.21.15.29:0/509089648 wait complete. 2023-10-07T10:44:42.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T10:44:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.243+0000 7f9d8cbff700 1 Processor -- start 2023-10-07T10:44:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.243+0000 7f9d8cbff700 1 -- start start 2023-10-07T10:44:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d8cbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d8814c8d0 0x7f9d880afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d8cbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88153cc0 0x7f9d88159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d8cbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d880b0410 0x7f9d88159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d8cbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d880b0950 con 0x7f9d880b0410 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d8cbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d88157580 con 0x7f9d8814c8d0 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d8cbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88157790 con 0x7f9d88153cc0 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.244+0000 7f9d86d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d880b0410 0x7f9d88159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d880b0410 0x7f9d88159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53370/0 (socket says 172.21.15.29:53370) 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 -- 172.21.15.29:0/3791526330 learned_addr learned my addr 172.21.15.29:0/3791526330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d8659c700 1 --2- 172.21.15.29:0/3791526330 >> v2:172.21.15.144:3300/0 conn(0x7f9d8814c8d0 0x7f9d880afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d85d9b700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88153cc0 0x7f9d88159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 -- 172.21.15.29:0/3791526330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88153cc0 msgr2=0x7f9d88159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88153cc0 0x7f9d88159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 -- 172.21.15.29:0/3791526330 >> v2:172.21.15.144:3300/0 conn(0x7f9d8814c8d0 msgr2=0x7f9d880afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 --2- 172.21.15.29:0/3791526330 >> v2:172.21.15.144:3300/0 conn(0x7f9d8814c8d0 0x7f9d880afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 -- 172.21.15.29:0/3791526330 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88157a70 con 0x7f9d880b0410 2023-10-07T10:44:42.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d85d9b700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88153cc0 0x7f9d88159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:42.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d86d9d700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d880b0410 0x7f9d88159bc0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f9d8806d0b0 tx=0x7f9d8000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88856d2fbd9cc0e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:42.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.245+0000 7f9d8559a700 1 -- 172.21.15.29:0/3791526330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80010070 con 0x7f9d880b0410 2023-10-07T10:44:42.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8559a700 1 -- 172.21.15.29:0/3791526330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8000dde0 con 0x7f9d880b0410 2023-10-07T10:44:42.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8559a700 1 -- 172.21.15.29:0/3791526330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000b320 con 0x7f9d880b0410 2023-10-07T10:44:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 -- 172.21.15.29:0/3791526330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d880b0410 msgr2=0x7f9d88159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d880b0410 0x7f9d88159bc0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f9d8806d0b0 tx=0x7f9d8000ba50 comp rx=0 tx=0).stop 2023-10-07T10:44:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 -- 172.21.15.29:0/3791526330 shutdown_connections 2023-10-07T10:44:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/3791526330 >> v2:172.21.15.144:3300/0 conn(0x7f9d8814c8d0 0x7f9d880afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d880b0410 0x7f9d88159bc0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/3791526330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88153cc0 0x7f9d88159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 -- 172.21.15.29:0/3791526330 >> 172.21.15.29:0/3791526330 conn(0x7f9d88067610 msgr2=0x7f9d88067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 -- 172.21.15.29:0/3791526330 shutdown_connections 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.246+0000 7f9d8cbff700 1 -- 172.21.15.29:0/3791526330 wait complete. 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.248+0000 7f9d8cbff700 1 Processor -- start 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.248+0000 7f9d8cbff700 1 -- start start 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d8cbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d880b0410 0x7f9d881510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d8cbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8814c8d0 0x7f9d88152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d8cbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 0x7f9d88169db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d8cbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8816a2f0 con 0x7f9d88153cc0 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d8cbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d881522a0 con 0x7f9d880b0410 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d8cbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88151690 con 0x7f9d8814c8d0 2023-10-07T10:44:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 0x7f9d88169db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 0x7f9d88169db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53380/0 (socket says 172.21.15.29:53380) 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 -- 172.21.15.29:0/1362682871 learned_addr learned my addr 172.21.15.29:0/1362682871 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 -- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8814c8d0 msgr2=0x7f9d88152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8814c8d0 0x7f9d88152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 -- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d880b0410 msgr2=0x7f9d881510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d880b0410 0x7f9d881510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.249+0000 7f9d86d9d700 1 -- 172.21.15.29:0/1362682871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8000bcf0 con 0x7f9d88153cc0 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.250+0000 7f9d86d9d700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 0x7f9d88169db0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f9d880641e0 tx=0x7f9d8001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.250+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000ab90 con 0x7f9d88153cc0 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.250+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8000ad30 con 0x7f9d88153cc0 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.250+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000aea0 con 0x7f9d88153cc0 2023-10-07T10:44:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.250+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d88151970 con 0x7f9d88153cc0 2023-10-07T10:44:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.250+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d88151f30 con 0x7f9d88153cc0 2023-10-07T10:44:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.251+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d88152580 con 0x7f9d88153cc0 2023-10-07T10:44:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.252+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9d80003040 con 0x7f9d88153cc0 2023-10-07T10:44:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.253+0000 7f9d767fc700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d68066f30 0x7f9d680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.253+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f9d800863b0 con 0x7f9d88153cc0 2023-10-07T10:44:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.254+0000 7f9d8659c700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d68066f30 0x7f9d680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.254+0000 7f9d8659c700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d68066f30 0x7f9d680693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9d8814af90 tx=0x7f9d780091a0 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:42.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.257+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d800533e0 con 0x7f9d88153cc0 2023-10-07T10:44:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:42.508+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9d881523e0 con 0x7f9d88153cc0 2023-10-07T10:44:43.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.142+0000 7f9d767fc700 1 -- 172.21.15.29:0/1362682871 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9d881523e0 con 0x7f9d88153cc0 2023-10-07T10:44:43.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d68066f30 msgr2=0x7f9d680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:43.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d68066f30 0x7f9d680693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9d8814af90 tx=0x7f9d780091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 msgr2=0x7f9d88169db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 0x7f9d88169db0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f9d880641e0 tx=0x7f9d8001a000 comp rx=0 tx=0).stop 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 shutdown_connections 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9d68066f30 0x7f9d680693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d88153cc0 0x7f9d88169db0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d880b0410 0x7f9d881510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 --2- 172.21.15.29:0/1362682871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d8814c8d0 0x7f9d88152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 >> 172.21.15.29:0/1362682871 conn(0x7f9d88067610 msgr2=0x7f9d88154120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.145+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 shutdown_connections 2023-10-07T10:44:43.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.146+0000 7f9d8cbff700 1 -- 172.21.15.29:0/1362682871 wait complete. 2023-10-07T10:44:43.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T10:44:43.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.365+0000 7f1f373ff700 1 Processor -- start 2023-10-07T10:44:43.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.365+0000 7f1f373ff700 1 -- start start 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f373ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f3015a960 0x7f1f30158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f373ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f3015ad30 0x7f1f300ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f373ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f300acb40 con 0x7f1f300ae6d0 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f373ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f300aeb50 con 0x7f1f3015ad30 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f373ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f30158c60 con 0x7f1f3015a960 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f3499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f3015ad30 0x7f1f300ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:43.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f3599c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1f300ae6d0 0x7f1f300b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45108/0 (socket says 172.21.15.29:45108) 2023-10-07T10:44:43.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f3599c700 1 -- 172.21.15.29:0/2031958080 learned_addr learned my addr 172.21.15.29:0/2031958080 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:43.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f3519b700 1 --2- 172.21.15.29:0/2031958080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f3015a960 0x7f1f30158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:43.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.367+0000 7f1f3499a700 1 -- 172.21.15.29:0/2031958080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f3015a960 msgr2=0x7f1f30158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:43.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f3499a700 1 --2- 172.21.15.29:0/2031958080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f3015a960 0x7f1f30158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f3499a700 1 -- 172.21.15.29:0/2031958080 >> v1:172.21.15.29:6789/0 conn(0x7f1f300ae6d0 legacy=0x7f1f300b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f3499a700 1 -- 172.21.15.29:0/2031958080 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300abf30 con 0x7f1f3015ad30 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f27fff700 1 -- 172.21.15.29:0/2031958080 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247442029 0 0) 0x7f1f300acb40 con 0x7f1f300ae6d0 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f3499a700 1 --2- 172.21.15.29:0/2031958080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300ae190 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f1f200098e0 tx=0x7f1f200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e8f895e28f120f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f27fff700 1 -- 172.21.15.29:0/2031958080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000b640 con 0x7f1f3015ad30 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f27fff700 1 -- 172.21.15.29:0/2031958080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000b7a0 con 0x7f1f3015ad30 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.368+0000 7f1f27fff700 1 -- 172.21.15.29:0/2031958080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000bb40 con 0x7f1f3015ad30 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 -- 172.21.15.29:0/2031958080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 msgr2=0x7f1f300ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 --2- 172.21.15.29:0/2031958080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300ae190 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f1f200098e0 tx=0x7f1f200091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 -- 172.21.15.29:0/2031958080 shutdown_connections 2023-10-07T10:44:43.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 --2- 172.21.15.29:0/2031958080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f3015a960 0x7f1f30158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 --2- 172.21.15.29:0/2031958080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300ae190 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 -- 172.21.15.29:0/2031958080 >> 172.21.15.29:0/2031958080 conn(0x7f1f30067610 msgr2=0x7f1f30067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:43.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 -- 172.21.15.29:0/2031958080 shutdown_connections 2023-10-07T10:44:43.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.369+0000 7f1f373ff700 1 -- 172.21.15.29:0/2031958080 wait complete. 2023-10-07T10:44:43.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.370+0000 7f1f373ff700 1 Processor -- start 2023-10-07T10:44:43.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.370+0000 7f1f373ff700 1 -- start start 2023-10-07T10:44:43.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.371+0000 7f1f373ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f300ae6d0 0x7f1f3013d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:43.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.371+0000 7f1f373ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f3015a960 0x7f1f300ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:43.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.371+0000 7f1f373ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.371+0000 7f1f373ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300b0f40 con 0x7f1f3015ad30 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.371+0000 7f1f373ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3015e840 con 0x7f1f3015a960 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.371+0000 7f1f373ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3015eb20 con 0x7f1f300ae6d0 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38766/0 (socket says 172.21.15.29:38766) 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 -- 172.21.15.29:0/363242308 learned_addr learned my addr 172.21.15.29:0/363242308 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3519b700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f300ae6d0 0x7f1f3013d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3499a700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f3015a960 0x7f1f300ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 -- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f300ae6d0 msgr2=0x7f1f3013d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:43.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f300ae6d0 0x7f1f3013d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 -- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f3015a960 msgr2=0x7f1f300ab2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f3015a960 0x7f1f300ab2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 -- 172.21.15.29:0/363242308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20009420 con 0x7f1f3015ad30 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f3599c700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300b02f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800cc90 tx=0x7f1f28001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2800a960 con 0x7f1f3015ad30 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2800aac0 con 0x7f1f3015ad30 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2800ae80 con 0x7f1f3015ad30 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.372+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3015ee00 con 0x7f1f3015ad30 2023-10-07T10:44:43.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.373+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3013dc60 con 0x7f1f3015ad30 2023-10-07T10:44:43.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.374+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3015f300 con 0x7f1f3015ad30 2023-10-07T10:44:43.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.375+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1f28007300 con 0x7f1f3015ad30 2023-10-07T10:44:43.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.376+0000 7f1f257fa700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1f14066f30 0x7f1f140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:43.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.376+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f1f28084ab0 con 0x7f1f3015ad30 2023-10-07T10:44:43.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.376+0000 7f1f3519b700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1f14066f30 0x7f1f140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:43.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.377+0000 7f1f3519b700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1f14066f30 0x7f1f140693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009db0 tx=0x7f1f2c0091a0 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:43.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.380+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f28051c00 con 0x7f1f3015ad30 2023-10-07T10:44:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:43.632+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1f3004c3c0 con 0x7f1f3015ad30 2023-10-07T10:44:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.149+0000 7f1f257fa700 1 -- 172.21.15.29:0/363242308 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1f28055220 con 0x7f1f3015ad30 2023-10-07T10:44:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1f14066f30 msgr2=0x7f1f140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1f14066f30 0x7f1f140693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009db0 tx=0x7f1f2c0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 msgr2=0x7f1f300b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300b02f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800cc90 tx=0x7f1f28001a80 comp rx=0 tx=0).stop 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 shutdown_connections 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1f14066f30 0x7f1f140693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f3015ad30 0x7f1f300b02f0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f300ae6d0 0x7f1f3013d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 --2- 172.21.15.29:0/363242308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f3015a960 0x7f1f300ab2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 >> 172.21.15.29:0/363242308 conn(0x7f1f30067610 msgr2=0x7f1f30156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 shutdown_connections 2023-10-07T10:44:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.152+0000 7f1f373ff700 1 -- 172.21.15.29:0/363242308 wait complete. 2023-10-07T10:44:44.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:44.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.369+0000 7f66a55ff700 1 Processor -- start 2023-10-07T10:44:44.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.369+0000 7f66a55ff700 1 -- start start 2023-10-07T10:44:44.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f66a55ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a0159750 0x7f66a00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f66a55ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66a0159b20 0x7f66a00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f66a55ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a00cbf20 con 0x7f66a00ce6d0 2023-10-07T10:44:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f66a55ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a00cc200 con 0x7f66a0159b20 2023-10-07T10:44:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f66a55ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00cc4e0 con 0x7f66a0159750 2023-10-07T10:44:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f669e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66a0159b20 0x7f66a00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f669e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66a0159b20 0x7f66a00ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38774/0 (socket says 172.21.15.29:38774) 2023-10-07T10:44:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f669e7fc700 1 -- 172.21.15.29:0/3576034558 learned_addr learned my addr 172.21.15.29:0/3576034558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.371+0000 7f669effd700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a0159750 0x7f66a00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669e7fc700 1 -- 172.21.15.29:0/3576034558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a0159750 msgr2=0x7f66a00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669e7fc700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a0159750 0x7f66a00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669e7fc700 1 -- 172.21.15.29:0/3576034558 >> v1:172.21.15.144:6789/0 conn(0x7f66a00ce6d0 legacy=0x7f66a015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669e7fc700 1 -- 172.21.15.29:0/3576034558 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00cc7c0 con 0x7f66a0159b20 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669effd700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a0159750 0x7f66a00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669e7fc700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a00ce190 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f66940099b0 tx=0x7f6694009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aaa1320fcf5309b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669dffb700 1 -- 172.21.15.29:0/3576034558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669400b860 con 0x7f66a0159b20 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669dffb700 1 -- 172.21.15.29:0/3576034558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669400b9c0 con 0x7f66a0159b20 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.372+0000 7f669dffb700 1 -- 172.21.15.29:0/3576034558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669400bd60 con 0x7f66a0159b20 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 -- 172.21.15.29:0/3576034558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 msgr2=0x7f66a00ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a00ce190 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f66940099b0 tx=0x7f6694009250 comp rx=0 tx=0).stop 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 -- 172.21.15.29:0/3576034558 shutdown_connections 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a0159750 0x7f66a00d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 --2- 172.21.15.29:0/3576034558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a00ce190 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 -- 172.21.15.29:0/3576034558 >> 172.21.15.29:0/3576034558 conn(0x7f66a0063a10 msgr2=0x7f66a00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 -- 172.21.15.29:0/3576034558 shutdown_connections 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.373+0000 7f66a55ff700 1 -- 172.21.15.29:0/3576034558 wait complete. 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.374+0000 7f66a55ff700 1 Processor -- start 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.375+0000 7f66a55ff700 1 -- start start 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f66a55ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a00ce6d0 0x7f66a0153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f66a55ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66a0159750 0x7f66a00695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f66a55ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a0069b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f66a55ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00b4940 con 0x7f66a0159b20 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f66a55ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00c81b0 con 0x7f66a0159750 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f66a55ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00c8490 con 0x7f66a00ce6d0 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a0069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a0069b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38778/0 (socket says 172.21.15.29:38778) 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 -- 172.21.15.29:0/142817742 learned_addr learned my addr 172.21.15.29:0/142817742 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669e7fc700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66a0159750 0x7f66a00695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669effd700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a00ce6d0 0x7f66a0153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 -- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a00ce6d0 msgr2=0x7f66a0153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a00ce6d0 0x7f66a0153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 -- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66a0159750 msgr2=0x7f66a00695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66a0159750 0x7f66a00695c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.376+0000 7f669f7fe700 1 -- 172.21.15.29:0/142817742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66940094f0 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.377+0000 7f669f7fe700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a0069b00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f669800c840 tx=0x7f6698001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.377+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698012630 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.377+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66980127b0 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.377+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698012b70 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.377+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a00c8770 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.377+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a0153770 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.378+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a004c3c0 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.379+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6698003030 con 0x7f66a0159b20 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.380+0000 7f6686ffd700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f668c066f60 0x7f668c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.380+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6698084710 con 0x7f66a0159b20 2023-10-07T10:44:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.381+0000 7f669effd700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f668c066f60 0x7f668c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.382+0000 7f669effd700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f668c066f60 0x7f668c069410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f669000ab80 tx=0x7f6690008040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.384+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66980551d0 con 0x7f66a0159b20 2023-10-07T10:44:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.632+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66a0064040 con 0x7f66a0159b20 2023-10-07T10:44:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.633+0000 7f6686ffd700 1 -- 172.21.15.29:0/142817742 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f6698054d60 con 0x7f66a0159b20 2023-10-07T10:44:44.634 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:44.634 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:44:40.623432+0000","modified":"2023-10-07T10:44:44.142903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8634":{"gid":8634,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3670759585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3670759585},{"type":"v1","addr":"172.21.15.29:6835","nonce":3670759585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-10-07T10:44:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f668c066f60 msgr2=0x7f668c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f668c066f60 0x7f668c069410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f669000ab80 tx=0x7f6690008040 comp rx=0 tx=0).stop 2023-10-07T10:44:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 msgr2=0x7f66a0069b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a0069b00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f669800c840 tx=0x7f6698001670 comp rx=0 tx=0).stop 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 shutdown_connections 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f668c066f60 0x7f668c069410 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a0159b20 0x7f66a0069b00 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66a00ce6d0 0x7f66a0153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 --2- 172.21.15.29:0/142817742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66a0159750 0x7f66a00695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 >> 172.21.15.29:0/142817742 conn(0x7f66a0063a10 msgr2=0x7f66a00c5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 shutdown_connections 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.635+0000 7f66a55ff700 1 -- 172.21.15.29:0/142817742 wait complete. 2023-10-07T10:44:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-07T10:44:44.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:44:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.849+0000 7f0973bff700 1 Processor -- start 2023-10-07T10:44:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.849+0000 7f0973bff700 1 -- start start 2023-10-07T10:44:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.850+0000 7f0973bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f096c153cc0 0x7f096c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.850+0000 7f0973bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f096c15ade0 0x7f096c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.850+0000 7f0973bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096c154090 con 0x7f096c0b0000 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.850+0000 7f0973bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f096c159f60 con 0x7f096c15ade0 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.850+0000 7f0973bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f096c15a240 con 0x7f096c153cc0 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f096c15ade0 0x7f096c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f096c15ade0 0x7f096c0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38780/0 (socket says 172.21.15.29:38780) 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 -- 172.21.15.29:0/3621527121 learned_addr learned my addr 172.21.15.29:0/3621527121 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097199b700 1 --2- 172.21.15.29:0/3621527121 >> v2:172.21.15.144:3300/0 conn(0x7f096c153cc0 0x7f096c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 -- 172.21.15.29:0/3621527121 >> v2:172.21.15.144:3300/0 conn(0x7f096c153cc0 msgr2=0x7f096c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 --2- 172.21.15.29:0/3621527121 >> v2:172.21.15.144:3300/0 conn(0x7f096c153cc0 0x7f096c0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 -- 172.21.15.29:0/3621527121 >> v1:172.21.15.144:6789/0 conn(0x7f096c0b0000 legacy=0x7f096c15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 -- 172.21.15.29:0/3621527121 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c15a520 con 0x7f096c15ade0 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.851+0000 7f097119a700 1 --2- 172.21.15.29:0/3621527121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c15ade0 0x7f096c0b1d00 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f096c0cf440 tx=0x7f095c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7696e3fbfe499cbb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0970999700 1 -- 172.21.15.29:0/3621527121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c009e70 con 0x7f096c15ade0 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0970999700 1 -- 172.21.15.29:0/3621527121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c00b860 con 0x7f096c15ade0 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0970999700 1 -- 172.21.15.29:0/3621527121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c00bc00 con 0x7f096c15ade0 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 -- 172.21.15.29:0/3621527121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c15ade0 msgr2=0x7f096c0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 --2- 172.21.15.29:0/3621527121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c15ade0 0x7f096c0b1d00 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f096c0cf440 tx=0x7f095c009250 comp rx=0 tx=0).stop 2023-10-07T10:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 -- 172.21.15.29:0/3621527121 shutdown_connections 2023-10-07T10:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 --2- 172.21.15.29:0/3621527121 >> v2:172.21.15.144:3300/0 conn(0x7f096c153cc0 0x7f096c0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 --2- 172.21.15.29:0/3621527121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c15ade0 0x7f096c0b1d00 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 -- 172.21.15.29:0/3621527121 >> 172.21.15.29:0/3621527121 conn(0x7f096c067610 msgr2=0x7f096c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 -- 172.21.15.29:0/3621527121 shutdown_connections 2023-10-07T10:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.852+0000 7f0973bff700 1 -- 172.21.15.29:0/3621527121 wait complete. 2023-10-07T10:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.854+0000 7f0973bff700 1 Processor -- start 2023-10-07T10:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.854+0000 7f0973bff700 1 -- start start 2023-10-07T10:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f0973bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f096c0b0000 0x7f096c145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f0973bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 0x7f096c157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f0973bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f096c15ade0 0x7f096c0adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f0973bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c146450 con 0x7f096c153cc0 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f0973bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c1557b0 con 0x7f096c15ade0 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f0973bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096c169e70 con 0x7f096c0b0000 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f097119a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 0x7f096c157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f097119a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 0x7f096c157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38784/0 (socket says 172.21.15.29:38784) 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.855+0000 7f097119a700 1 -- 172.21.15.29:0/72997736 learned_addr learned my addr 172.21.15.29:0/72997736 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097199b700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f096c0b0000 0x7f096c145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097119a700 1 -- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f096c0b0000 msgr2=0x7f096c145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097219c700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f096c15ade0 0x7f096c0adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097119a700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f096c0b0000 0x7f096c145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097119a700 1 -- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f096c15ade0 msgr2=0x7f096c0adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097119a700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f096c15ade0 0x7f096c0adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097119a700 1 -- 172.21.15.29:0/72997736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c0094f0 con 0x7f096c153cc0 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097199b700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f096c0b0000 0x7f096c145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097219c700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f096c15ade0 0x7f096c0adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f097119a700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 0x7f096c157200 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f096c0b0be0 tx=0x7f095c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c011420 con 0x7f096c153cc0 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c0115a0 con 0x7f096c153cc0 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c011960 con 0x7f096c153cc0 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c16a150 con 0x7f096c153cc0 2023-10-07T10:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.856+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c16a710 con 0x7f096c153cc0 2023-10-07T10:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.857+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c1559d0 con 0x7f096c153cc0 2023-10-07T10:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.858+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f095c0253f0 con 0x7f096c153cc0 2023-10-07T10:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.859+0000 7f0961ffb700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0950066f30 0x7f09500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.859+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f095c0860c0 con 0x7f096c153cc0 2023-10-07T10:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.860+0000 7f097199b700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0950066f30 0x7f09500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.860+0000 7f097199b700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0950066f30 0x7f09500693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0968006ea0 tx=0x7f0968008040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:44.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:44.863+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f096c1559d0 con 0x7f096c153cc0 2023-10-07T10:44:45.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.063+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f096c1559d0 con 0x7f096c153cc0 2023-10-07T10:44:45.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.064+0000 7f0961ffb700 1 -- 172.21.15.29:0/72997736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f096c1559d0 con 0x7f096c153cc0 2023-10-07T10:44:45.065 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:45.065 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":151,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:42.129879+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:44:38.468584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-07T10:44:39.609895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:44:45.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0950066f30 msgr2=0x7f09500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0950066f30 0x7f09500693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0968006ea0 tx=0x7f0968008040 comp rx=0 tx=0).stop 2023-10-07T10:44:45.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 msgr2=0x7f096c157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 0x7f096c157200 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f096c0b0be0 tx=0x7f095c017000 comp rx=0 tx=0).stop 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 shutdown_connections 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0950066f30 0x7f09500693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f096c15ade0 0x7f096c0adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f096c0b0000 0x7f096c145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 --2- 172.21.15.29:0/72997736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f096c153cc0 0x7f096c157200 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 >> 172.21.15.29:0/72997736 conn(0x7f096c067610 msgr2=0x7f096c0ad770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 shutdown_connections 2023-10-07T10:44:45.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.066+0000 7f0973bff700 1 -- 172.21.15.29:0/72997736 wait complete. 2023-10-07T10:44:45.085 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T10:44:45.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T10:44:45.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.281+0000 7f28d6fff700 1 Processor -- start 2023-10-07T10:44:45.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.282+0000 7f28d6fff700 1 -- start start 2023-10-07T10:44:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.283+0000 7f28d6fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f28d0157290 0x7f28d00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.283+0000 7f28d6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00ae8b0 0x7f28d00b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.283+0000 7f28d6fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d00b1ca0 con 0x7f28d00b1840 2023-10-07T10:44:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.283+0000 7f28d6fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d0158ad0 con 0x7f28d0157290 2023-10-07T10:44:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.283+0000 7f28d6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d015b140 con 0x7f28d00ae8b0 2023-10-07T10:44:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28cffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00ae8b0 0x7f28d00b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f28d0157290 0x7f28d00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f28d0157290 0x7f28d00cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:40158/0 (socket says 172.21.15.29:40158) 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 -- 172.21.15.29:0/1365088921 learned_addr learned my addr 172.21.15.29:0/1365088921 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 -- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00ae8b0 msgr2=0x7f28d00b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 --2- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00ae8b0 0x7f28d00b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 -- 172.21.15.29:0/1365088921 >> v1:172.21.15.144:6789/0 conn(0x7f28d00b1840 legacy=0x7f28d00ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 -- 172.21.15.29:0/1365088921 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d015b420 con 0x7f28d0157290 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28cffff700 1 --2- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00ae8b0 0x7f28d00b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.284+0000 7f28d4d9b700 1 --2- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0157290 0x7f28d00cc870 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f28c40099b0 tx=0x7f28c4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd4b5be4c7ecfef server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28cf7fe700 1 -- 172.21.15.29:0/1365088921 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4009ea0 con 0x7f28d0157290 2023-10-07T10:44:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28cf7fe700 1 -- 172.21.15.29:0/1365088921 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c400b860 con 0x7f28d0157290 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28cf7fe700 1 -- 172.21.15.29:0/1365088921 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c400bc00 con 0x7f28d0157290 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 -- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0157290 msgr2=0x7f28d00cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 --2- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0157290 0x7f28d00cc870 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f28c40099b0 tx=0x7f28c4009250 comp rx=0 tx=0).stop 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 -- 172.21.15.29:0/1365088921 shutdown_connections 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 --2- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0157290 0x7f28d00cc870 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 --2- 172.21.15.29:0/1365088921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00ae8b0 0x7f28d00b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 -- 172.21.15.29:0/1365088921 >> 172.21.15.29:0/1365088921 conn(0x7f28d00680a0 msgr2=0x7f28d00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 -- 172.21.15.29:0/1365088921 shutdown_connections 2023-10-07T10:44:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.285+0000 7f28d6fff700 1 -- 172.21.15.29:0/1365088921 wait complete. 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.287+0000 7f28d6fff700 1 Processor -- start 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.287+0000 7f28d6fff700 1 -- start start 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28d6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00b1840 0x7f28d00c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28d6fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 0x7f28d00c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28d6fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0145b90 0x7f28d0153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28d6fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00c7bd0 con 0x7f28d0157290 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28d6fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d01521a0 con 0x7f28d0145b90 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28d6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00c6fc0 con 0x7f28d00b1840 2023-10-07T10:44:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 0x7f28d00c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 0x7f28d00c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38792/0 (socket says 172.21.15.29:38792) 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.288+0000 7f28cffff700 1 -- 172.21.15.29:0/2574937369 learned_addr learned my addr 172.21.15.29:0/2574937369 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28d4d9b700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00b1840 0x7f28d00c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28d559c700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0145b90 0x7f28d0153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28cffff700 1 -- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00b1840 msgr2=0x7f28d00c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28cffff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00b1840 0x7f28d00c6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28cffff700 1 -- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0145b90 msgr2=0x7f28d0153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28cffff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0145b90 0x7f28d0153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28cffff700 1 -- 172.21.15.29:0/2574937369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc009420 con 0x7f28d0157290 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28d4d9b700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00b1840 0x7f28d00c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28d559c700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0145b90 0x7f28d0153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28cffff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 0x7f28d00c8060 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00ec80 tx=0x7f28bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc00c9d0 con 0x7f28d0157290 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc00cb30 con 0x7f28d0157290 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c40094f0 con 0x7f28d0157290 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc00ceb0 con 0x7f28d0157290 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.289+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d00c7580 con 0x7f28d0157290 2023-10-07T10:44:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.290+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d00c7d80 con 0x7f28d0157290 2023-10-07T10:44:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.291+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f28bc013680 con 0x7f28d0157290 2023-10-07T10:44:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.292+0000 7f28ccff9700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f28b4066f30 0x7f28b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.292+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f28bc084b90 con 0x7f28d0157290 2023-10-07T10:44:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.293+0000 7f28d4d9b700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f28b4066f30 0x7f28b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.293+0000 7f28d4d9b700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f28b4066f30 0x7f28b40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28c40099b0 tx=0x7f28c4017000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:45.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.296+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28d00c7d80 con 0x7f28d0157290 2023-10-07T10:44:45.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.497+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28d004c3c0 con 0x7f28b4066f30 2023-10-07T10:44:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.508+0000 7f28ccff9700 1 -- 172.21.15.29:0/2574937369 <== mgr.7310 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289194 (secure 0 0 0) 0x7f28d004c3c0 con 0x7f28b4066f30 2023-10-07T10:44:45.512 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f28b4066f30 msgr2=0x7f28b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f28b4066f30 0x7f28b40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28c40099b0 tx=0x7f28c4017000 comp rx=0 tx=0).stop 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 msgr2=0x7f28d00c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 0x7f28d00c8060 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00ec80 tx=0x7f28bc0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 shutdown_connections 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f28b4066f30 0x7f28b40693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d0145b90 0x7f28d0153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28d00b1840 0x7f28d00c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 --2- 172.21.15.29:0/2574937369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d0157290 0x7f28d00c8060 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 >> 172.21.15.29:0/2574937369 conn(0x7f28d00680a0 msgr2=0x7f28d0159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 shutdown_connections 2023-10-07T10:44:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.512+0000 7f28d6fff700 1 -- 172.21.15.29:0/2574937369 wait complete. 2023-10-07T10:44:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T10:44:45.531 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T10:44:45.446410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467868,"kb_used_data":15836,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752506852,"statfs":{"total":773094113280,"available":770567016448,"internally_reserved":0,"allocated":16216064,"data_stored":8497512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001414"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154896+0000","last_change":"2023-10-07T10:44:41.146486+0000","last_active":"2023-10-07T10:44:41.154896+0000","last_peered":"2023-10-07T10:44:41.154896+0000","last_clean":"2023-10-07T10:44:41.154896+0000","last_became_active":"2023-10-07T10:44:41.146225+0000","last_became_peered":"2023-10-07T10:44:41.146225+0000","last_unstale":"2023-10-07T10:44:41.154896+0000","last_undegraded":"2023-10-07T10:44:41.154896+0000","last_fullsized":"2023-10-07T10:44:41.154896+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:35:14.213371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145049+0000","last_change":"2023-10-07T10:44:41.141108+0000","last_active":"2023-10-07T10:44:41.145049+0000","last_peered":"2023-10-07T10:44:41.145049+0000","last_clean":"2023-10-07T10:44:41.145049+0000","last_became_active":"2023-10-07T10:44:41.140500+0000","last_became_peered":"2023-10-07T10:44:41.140500+0000","last_unstale":"2023-10-07T10:44:41.145049+0000","last_undegraded":"2023-10-07T10:44:41.145049+0000","last_fullsized":"2023-10-07T10:44:41.145049+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:29:32.978953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154229+0000","last_change":"2023-10-07T10:44:41.151114+0000","last_active":"2023-10-07T10:44:41.154229+0000","last_peered":"2023-10-07T10:44:41.154229+0000","last_clean":"2023-10-07T10:44:41.154229+0000","last_became_active":"2023-10-07T10:44:41.150860+0000","last_became_peered":"2023-10-07T10:44:41.150860+0000","last_unstale":"2023-10-07T10:44:41.154229+0000","last_undegraded":"2023-10-07T10:44:41.154229+0000","last_fullsized":"2023-10-07T10:44:41.154229+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:31:08.272160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154045+0000","last_change":"2023-10-07T10:44:41.143913+0000","last_active":"2023-10-07T10:44:41.154045+0000","last_peered":"2023-10-07T10:44:41.154045+0000","last_clean":"2023-10-07T10:44:41.154045+0000","last_became_active":"2023-10-07T10:44:41.143651+0000","last_became_peered":"2023-10-07T10:44:41.143651+0000","last_unstale":"2023-10-07T10:44:41.154045+0000","last_undegraded":"2023-10-07T10:44:41.154045+0000","last_fullsized":"2023-10-07T10:44:41.154045+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:50.192545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.151499+0000","last_change":"2023-10-07T10:44:41.150156+0000","last_active":"2023-10-07T10:44:41.151499+0000","last_peered":"2023-10-07T10:44:41.151499+0000","last_clean":"2023-10-07T10:44:41.151499+0000","last_became_active":"2023-10-07T10:44:41.149644+0000","last_became_peered":"2023-10-07T10:44:41.149644+0000","last_unstale":"2023-10-07T10:44:41.151499+0000","last_undegraded":"2023-10-07T10:44:41.151499+0000","last_fullsized":"2023-10-07T10:44:41.151499+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:33.745656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142679+0000","last_change":"2023-10-07T10:44:41.139741+0000","last_active":"2023-10-07T10:44:41.142679+0000","last_peered":"2023-10-07T10:44:41.142679+0000","last_clean":"2023-10-07T10:44:41.142679+0000","last_became_active":"2023-10-07T10:44:41.139467+0000","last_became_peered":"2023-10-07T10:44:41.139467+0000","last_unstale":"2023-10-07T10:44:41.142679+0000","last_undegraded":"2023-10-07T10:44:41.142679+0000","last_fullsized":"2023-10-07T10:44:41.142679+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:31:08.272160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143501+0000","last_change":"2023-10-07T10:44:41.141074+0000","last_active":"2023-10-07T10:44:41.143501+0000","last_peered":"2023-10-07T10:44:41.143501+0000","last_clean":"2023-10-07T10:44:41.143501+0000","last_became_active":"2023-10-07T10:44:41.140780+0000","last_became_peered":"2023-10-07T10:44:41.140780+0000","last_unstale":"2023-10-07T10:44:41.143501+0000","last_undegraded":"2023-10-07T10:44:41.143501+0000","last_fullsized":"2023-10-07T10:44:41.143501+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:26.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.144865+0000","last_change":"2023-10-07T10:44:41.140044+0000","last_active":"2023-10-07T10:44:41.144865+0000","last_peered":"2023-10-07T10:44:41.144865+0000","last_clean":"2023-10-07T10:44:41.144865+0000","last_became_active":"2023-10-07T10:44:41.139460+0000","last_became_peered":"2023-10-07T10:44:41.139460+0000","last_unstale":"2023-10-07T10:44:41.144865+0000","last_undegraded":"2023-10-07T10:44:41.144865+0000","last_fullsized":"2023-10-07T10:44:41.144865+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:12:23.908691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154122+0000","last_change":"2023-10-07T10:44:41.150391+0000","last_active":"2023-10-07T10:44:41.154122+0000","last_peered":"2023-10-07T10:44:41.154122+0000","last_clean":"2023-10-07T10:44:41.154122+0000","last_became_active":"2023-10-07T10:44:41.150133+0000","last_became_peered":"2023-10-07T10:44:41.150133+0000","last_unstale":"2023-10-07T10:44:41.154122+0000","last_undegraded":"2023-10-07T10:44:41.154122+0000","last_fullsized":"2023-10-07T10:44:41.154122+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:12:49.484952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145118+0000","last_change":"2023-10-07T10:44:41.141287+0000","last_active":"2023-10-07T10:44:41.145118+0000","last_peered":"2023-10-07T10:44:41.145118+0000","last_clean":"2023-10-07T10:44:41.145118+0000","last_became_active":"2023-10-07T10:44:41.140850+0000","last_became_peered":"2023-10-07T10:44:41.140850+0000","last_unstale":"2023-10-07T10:44:41.145118+0000","last_undegraded":"2023-10-07T10:44:41.145118+0000","last_fullsized":"2023-10-07T10:44:41.145118+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:44:35.721020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150006+0000","last_change":"2023-10-07T10:44:41.148120+0000","last_active":"2023-10-07T10:44:41.150006+0000","last_peered":"2023-10-07T10:44:41.150006+0000","last_clean":"2023-10-07T10:44:41.150006+0000","last_became_active":"2023-10-07T10:44:41.147825+0000","last_became_peered":"2023-10-07T10:44:41.147825+0000","last_unstale":"2023-10-07T10:44:41.150006+0000","last_undegraded":"2023-10-07T10:44:41.150006+0000","last_fullsized":"2023-10-07T10:44:41.150006+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:32:12.770853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153595+0000","last_change":"2023-10-07T10:44:41.146499+0000","last_active":"2023-10-07T10:44:41.153595+0000","last_peered":"2023-10-07T10:44:41.153595+0000","last_clean":"2023-10-07T10:44:41.153595+0000","last_became_active":"2023-10-07T10:44:41.146218+0000","last_became_peered":"2023-10-07T10:44:41.146218+0000","last_unstale":"2023-10-07T10:44:41.153595+0000","last_undegraded":"2023-10-07T10:44:41.153595+0000","last_fullsized":"2023-10-07T10:44:41.153595+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:20.347596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154444+0000","last_change":"2023-10-07T10:44:41.140573+0000","last_active":"2023-10-07T10:44:41.154444+0000","last_peered":"2023-10-07T10:44:41.154444+0000","last_clean":"2023-10-07T10:44:41.154444+0000","last_became_active":"2023-10-07T10:44:41.140218+0000","last_became_peered":"2023-10-07T10:44:41.140218+0000","last_unstale":"2023-10-07T10:44:41.154444+0000","last_undegraded":"2023-10-07T10:44:41.154444+0000","last_fullsized":"2023-10-07T10:44:41.154444+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:19.016810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149647+0000","last_change":"2023-10-07T10:44:41.145649+0000","last_active":"2023-10-07T10:44:41.149647+0000","last_peered":"2023-10-07T10:44:41.149647+0000","last_clean":"2023-10-07T10:44:41.149647+0000","last_became_active":"2023-10-07T10:44:41.145362+0000","last_became_peered":"2023-10-07T10:44:41.145362+0000","last_unstale":"2023-10-07T10:44:41.149647+0000","last_undegraded":"2023-10-07T10:44:41.149647+0000","last_fullsized":"2023-10-07T10:44:41.149647+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:35:55.938916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154674+0000","last_change":"2023-10-07T10:44:41.149774+0000","last_active":"2023-10-07T10:44:41.154674+0000","last_peered":"2023-10-07T10:44:41.154674+0000","last_clean":"2023-10-07T10:44:41.154674+0000","last_became_active":"2023-10-07T10:44:41.147509+0000","last_became_peered":"2023-10-07T10:44:41.147509+0000","last_unstale":"2023-10-07T10:44:41.154674+0000","last_undegraded":"2023-10-07T10:44:41.154674+0000","last_fullsized":"2023-10-07T10:44:41.154674+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:32:25.508518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142944+0000","last_change":"2023-10-07T10:44:41.138658+0000","last_active":"2023-10-07T10:44:41.142944+0000","last_peered":"2023-10-07T10:44:41.142944+0000","last_clean":"2023-10-07T10:44:41.142944+0000","last_became_active":"2023-10-07T10:44:41.138365+0000","last_became_peered":"2023-10-07T10:44:41.138365+0000","last_unstale":"2023-10-07T10:44:41.142944+0000","last_undegraded":"2023-10-07T10:44:41.142944+0000","last_fullsized":"2023-10-07T10:44:41.142944+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:23:49.101932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152762+0000","last_change":"2023-10-07T10:44:41.151589+0000","last_active":"2023-10-07T10:44:41.152762+0000","last_peered":"2023-10-07T10:44:41.152762+0000","last_clean":"2023-10-07T10:44:41.152762+0000","last_became_active":"2023-10-07T10:44:41.151135+0000","last_became_peered":"2023-10-07T10:44:41.151135+0000","last_unstale":"2023-10-07T10:44:41.152762+0000","last_undegraded":"2023-10-07T10:44:41.152762+0000","last_fullsized":"2023-10-07T10:44:41.152762+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:51:56.094466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143627+0000","last_change":"2023-10-07T10:44:41.139532+0000","last_active":"2023-10-07T10:44:41.143627+0000","last_peered":"2023-10-07T10:44:41.143627+0000","last_clean":"2023-10-07T10:44:41.143627+0000","last_became_active":"2023-10-07T10:44:41.139178+0000","last_became_peered":"2023-10-07T10:44:41.139178+0000","last_unstale":"2023-10-07T10:44:41.143627+0000","last_undegraded":"2023-10-07T10:44:41.143627+0000","last_fullsized":"2023-10-07T10:44:41.143627+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:48:31.051226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150023+0000","last_change":"2023-10-07T10:44:41.148162+0000","last_active":"2023-10-07T10:44:41.150023+0000","last_peered":"2023-10-07T10:44:41.150023+0000","last_clean":"2023-10-07T10:44:41.150023+0000","last_became_active":"2023-10-07T10:44:41.147848+0000","last_became_peered":"2023-10-07T10:44:41.147848+0000","last_unstale":"2023-10-07T10:44:41.150023+0000","last_undegraded":"2023-10-07T10:44:41.150023+0000","last_fullsized":"2023-10-07T10:44:41.150023+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:04.046119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142014+0000","last_change":"2023-10-07T10:44:41.135108+0000","last_active":"2023-10-07T10:44:41.142014+0000","last_peered":"2023-10-07T10:44:41.142014+0000","last_clean":"2023-10-07T10:44:41.142014+0000","last_became_active":"2023-10-07T10:44:41.134849+0000","last_became_peered":"2023-10-07T10:44:41.134849+0000","last_unstale":"2023-10-07T10:44:41.142014+0000","last_undegraded":"2023-10-07T10:44:41.142014+0000","last_fullsized":"2023-10-07T10:44:41.142014+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:17.616542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.151518+0000","last_change":"2023-10-07T10:44:41.150428+0000","last_active":"2023-10-07T10:44:41.151518+0000","last_peered":"2023-10-07T10:44:41.151518+0000","last_clean":"2023-10-07T10:44:41.151518+0000","last_became_active":"2023-10-07T10:44:41.149982+0000","last_became_peered":"2023-10-07T10:44:41.149982+0000","last_unstale":"2023-10-07T10:44:41.151518+0000","last_undegraded":"2023-10-07T10:44:41.151518+0000","last_fullsized":"2023-10-07T10:44:41.151518+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:58.711846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.151958+0000","last_change":"2023-10-07T10:44:41.148227+0000","last_active":"2023-10-07T10:44:41.151958+0000","last_peered":"2023-10-07T10:44:41.151958+0000","last_clean":"2023-10-07T10:44:41.151958+0000","last_became_active":"2023-10-07T10:44:41.147847+0000","last_became_peered":"2023-10-07T10:44:41.147847+0000","last_unstale":"2023-10-07T10:44:41.151958+0000","last_undegraded":"2023-10-07T10:44:41.151958+0000","last_fullsized":"2023-10-07T10:44:41.151958+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:27:50.412472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154133+0000","last_change":"2023-10-07T10:44:41.149817+0000","last_active":"2023-10-07T10:44:41.154133+0000","last_peered":"2023-10-07T10:44:41.154133+0000","last_clean":"2023-10-07T10:44:41.154133+0000","last_became_active":"2023-10-07T10:44:41.149541+0000","last_became_peered":"2023-10-07T10:44:41.149541+0000","last_unstale":"2023-10-07T10:44:41.154133+0000","last_undegraded":"2023-10-07T10:44:41.154133+0000","last_fullsized":"2023-10-07T10:44:41.154133+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:21.610466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.155461+0000","last_change":"2023-10-07T10:44:41.150258+0000","last_active":"2023-10-07T10:44:41.155461+0000","last_peered":"2023-10-07T10:44:41.155461+0000","last_clean":"2023-10-07T10:44:41.155461+0000","last_became_active":"2023-10-07T10:44:41.150015+0000","last_became_peered":"2023-10-07T10:44:41.150015+0000","last_unstale":"2023-10-07T10:44:41.155461+0000","last_undegraded":"2023-10-07T10:44:41.155461+0000","last_fullsized":"2023-10-07T10:44:41.155461+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:19.904126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149617+0000","last_change":"2023-10-07T10:44:41.146555+0000","last_active":"2023-10-07T10:44:41.149617+0000","last_peered":"2023-10-07T10:44:41.149617+0000","last_clean":"2023-10-07T10:44:41.149617+0000","last_became_active":"2023-10-07T10:44:41.146287+0000","last_became_peered":"2023-10-07T10:44:41.146287+0000","last_unstale":"2023-10-07T10:44:41.149617+0000","last_undegraded":"2023-10-07T10:44:41.149617+0000","last_fullsized":"2023-10-07T10:44:41.149617+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:18:10.617203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143609+0000","last_change":"2023-10-07T10:44:41.138871+0000","last_active":"2023-10-07T10:44:41.143609+0000","last_peered":"2023-10-07T10:44:41.143609+0000","last_clean":"2023-10-07T10:44:41.143609+0000","last_became_active":"2023-10-07T10:44:41.138536+0000","last_became_peered":"2023-10-07T10:44:41.138536+0000","last_unstale":"2023-10-07T10:44:41.143609+0000","last_undegraded":"2023-10-07T10:44:41.143609+0000","last_fullsized":"2023-10-07T10:44:41.143609+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:27:54.800429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153009+0000","last_change":"2023-10-07T10:44:41.140208+0000","last_active":"2023-10-07T10:44:41.153009+0000","last_peered":"2023-10-07T10:44:41.153009+0000","last_clean":"2023-10-07T10:44:41.153009+0000","last_became_active":"2023-10-07T10:44:41.139942+0000","last_became_peered":"2023-10-07T10:44:41.139942+0000","last_unstale":"2023-10-07T10:44:41.153009+0000","last_undegraded":"2023-10-07T10:44:41.153009+0000","last_fullsized":"2023-10-07T10:44:41.153009+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:36:55.696997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147285+0000","last_change":"2023-10-07T10:44:41.139644+0000","last_active":"2023-10-07T10:44:41.147285+0000","last_peered":"2023-10-07T10:44:41.147285+0000","last_clean":"2023-10-07T10:44:41.147285+0000","last_became_active":"2023-10-07T10:44:41.139371+0000","last_became_peered":"2023-10-07T10:44:41.139371+0000","last_unstale":"2023-10-07T10:44:41.147285+0000","last_undegraded":"2023-10-07T10:44:41.147285+0000","last_fullsized":"2023-10-07T10:44:41.147285+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:20:42.569790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154471+0000","last_change":"2023-10-07T10:44:41.147049+0000","last_active":"2023-10-07T10:44:41.154471+0000","last_peered":"2023-10-07T10:44:41.154471+0000","last_clean":"2023-10-07T10:44:41.154471+0000","last_became_active":"2023-10-07T10:44:41.146797+0000","last_became_peered":"2023-10-07T10:44:41.146797+0000","last_unstale":"2023-10-07T10:44:41.154471+0000","last_undegraded":"2023-10-07T10:44:41.154471+0000","last_fullsized":"2023-10-07T10:44:41.154471+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:01.539906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.155161+0000","last_change":"2023-10-07T10:44:41.148618+0000","last_active":"2023-10-07T10:44:41.155161+0000","last_peered":"2023-10-07T10:44:41.155161+0000","last_clean":"2023-10-07T10:44:41.155161+0000","last_became_active":"2023-10-07T10:44:41.147234+0000","last_became_peered":"2023-10-07T10:44:41.147234+0000","last_unstale":"2023-10-07T10:44:41.155161+0000","last_undegraded":"2023-10-07T10:44:41.155161+0000","last_fullsized":"2023-10-07T10:44:41.155161+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:05:07.923201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149074+0000","last_change":"2023-10-07T10:44:41.145520+0000","last_active":"2023-10-07T10:44:41.149074+0000","last_peered":"2023-10-07T10:44:41.149074+0000","last_clean":"2023-10-07T10:44:41.149074+0000","last_became_active":"2023-10-07T10:44:41.145049+0000","last_became_peered":"2023-10-07T10:44:41.145049+0000","last_unstale":"2023-10-07T10:44:41.149074+0000","last_undegraded":"2023-10-07T10:44:41.149074+0000","last_fullsized":"2023-10-07T10:44:41.149074+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:31:03.014537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.140978+0000","last_change":"2023-10-07T10:44:41.133265+0000","last_active":"2023-10-07T10:44:41.140978+0000","last_peered":"2023-10-07T10:44:41.140978+0000","last_clean":"2023-10-07T10:44:41.140978+0000","last_became_active":"2023-10-07T10:44:41.132962+0000","last_became_peered":"2023-10-07T10:44:41.132962+0000","last_unstale":"2023-10-07T10:44:41.140978+0000","last_undegraded":"2023-10-07T10:44:41.140978+0000","last_fullsized":"2023-10-07T10:44:41.140978+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:12:49.484952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154258+0000","last_change":"2023-10-07T10:44:41.148394+0000","last_active":"2023-10-07T10:44:41.154258+0000","last_peered":"2023-10-07T10:44:41.154258+0000","last_clean":"2023-10-07T10:44:41.154258+0000","last_became_active":"2023-10-07T10:44:41.148118+0000","last_became_peered":"2023-10-07T10:44:41.148118+0000","last_unstale":"2023-10-07T10:44:41.154258+0000","last_undegraded":"2023-10-07T10:44:41.154258+0000","last_fullsized":"2023-10-07T10:44:41.154258+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:50:11.176162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"150'4","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.157069+0000","last_change":"2023-10-07T10:44:40.141891+0000","last_active":"2023-10-07T10:44:41.157069+0000","last_peered":"2023-10-07T10:44:41.157069+0000","last_clean":"2023-10-07T10:44:41.157069+0000","last_became_active":"2023-10-07T10:44:40.141601+0000","last_became_peered":"2023-10-07T10:44:40.141601+0000","last_unstale":"2023-10-07T10:44:41.157069+0000","last_undegraded":"2023-10-07T10:44:41.157069+0000","last_fullsized":"2023-10-07T10:44:41.157069+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:21.972708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153876+0000","last_change":"2023-10-07T10:44:41.147988+0000","last_active":"2023-10-07T10:44:41.153876+0000","last_peered":"2023-10-07T10:44:41.153876+0000","last_clean":"2023-10-07T10:44:41.153876+0000","last_became_active":"2023-10-07T10:44:41.147740+0000","last_became_peered":"2023-10-07T10:44:41.147740+0000","last_unstale":"2023-10-07T10:44:41.153876+0000","last_undegraded":"2023-10-07T10:44:41.153876+0000","last_fullsized":"2023-10-07T10:44:41.153876+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:57:26.369780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150171+0000","last_change":"2023-10-07T10:44:40.138233+0000","last_active":"2023-10-07T10:44:41.150171+0000","last_peered":"2023-10-07T10:44:41.150171+0000","last_clean":"2023-10-07T10:44:41.150171+0000","last_became_active":"2023-10-07T10:44:40.137964+0000","last_became_peered":"2023-10-07T10:44:40.137964+0000","last_unstale":"2023-10-07T10:44:41.150171+0000","last_undegraded":"2023-10-07T10:44:41.150171+0000","last_fullsized":"2023-10-07T10:44:41.150171+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:00.532197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148978+0000","last_change":"2023-10-07T10:44:41.140918+0000","last_active":"2023-10-07T10:44:41.148978+0000","last_peered":"2023-10-07T10:44:41.148978+0000","last_clean":"2023-10-07T10:44:41.148978+0000","last_became_active":"2023-10-07T10:44:41.139574+0000","last_became_peered":"2023-10-07T10:44:41.139574+0000","last_unstale":"2023-10-07T10:44:41.148978+0000","last_undegraded":"2023-10-07T10:44:41.148978+0000","last_fullsized":"2023-10-07T10:44:41.148978+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:19.840943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148696+0000","last_change":"2023-10-07T10:44:40.132372+0000","last_active":"2023-10-07T10:44:41.148696+0000","last_peered":"2023-10-07T10:44:41.148696+0000","last_clean":"2023-10-07T10:44:41.148696+0000","last_became_active":"2023-10-07T10:44:40.132097+0000","last_became_peered":"2023-10-07T10:44:40.132097+0000","last_unstale":"2023-10-07T10:44:41.148696+0000","last_undegraded":"2023-10-07T10:44:41.148696+0000","last_fullsized":"2023-10-07T10:44:41.148696+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:03.386461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143898+0000","last_change":"2023-10-07T10:44:41.137096+0000","last_active":"2023-10-07T10:44:41.143898+0000","last_peered":"2023-10-07T10:44:41.143898+0000","last_clean":"2023-10-07T10:44:41.143898+0000","last_became_active":"2023-10-07T10:44:41.136825+0000","last_became_peered":"2023-10-07T10:44:41.136825+0000","last_unstale":"2023-10-07T10:44:41.143898+0000","last_undegraded":"2023-10-07T10:44:41.143898+0000","last_fullsized":"2023-10-07T10:44:41.143898+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:36.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152456+0000","last_change":"2023-10-07T10:44:40.129668+0000","last_active":"2023-10-07T10:44:41.152456+0000","last_peered":"2023-10-07T10:44:41.152456+0000","last_clean":"2023-10-07T10:44:41.152456+0000","last_became_active":"2023-10-07T10:44:40.129391+0000","last_became_peered":"2023-10-07T10:44:40.129391+0000","last_unstale":"2023-10-07T10:44:41.152456+0000","last_undegraded":"2023-10-07T10:44:41.152456+0000","last_fullsized":"2023-10-07T10:44:41.152456+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:32:24.500809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142866+0000","last_change":"2023-10-07T10:44:41.140180+0000","last_active":"2023-10-07T10:44:41.142866+0000","last_peered":"2023-10-07T10:44:41.142866+0000","last_clean":"2023-10-07T10:44:41.142866+0000","last_became_active":"2023-10-07T10:44:41.139900+0000","last_became_peered":"2023-10-07T10:44:41.139900+0000","last_unstale":"2023-10-07T10:44:41.142866+0000","last_undegraded":"2023-10-07T10:44:41.142866+0000","last_fullsized":"2023-10-07T10:44:41.142866+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:04.394170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.146711+0000","last_change":"2023-10-07T10:44:40.129304+0000","last_active":"2023-10-07T10:44:41.146711+0000","last_peered":"2023-10-07T10:44:41.146711+0000","last_clean":"2023-10-07T10:44:41.146711+0000","last_became_active":"2023-10-07T10:44:40.129005+0000","last_became_peered":"2023-10-07T10:44:40.129005+0000","last_unstale":"2023-10-07T10:44:41.146711+0000","last_undegraded":"2023-10-07T10:44:41.146711+0000","last_fullsized":"2023-10-07T10:44:41.146711+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:35:13.205662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149500+0000","last_change":"2023-10-07T10:44:41.148562+0000","last_active":"2023-10-07T10:44:41.149500+0000","last_peered":"2023-10-07T10:44:41.149500+0000","last_clean":"2023-10-07T10:44:41.149500+0000","last_became_active":"2023-10-07T10:44:41.148258+0000","last_became_peered":"2023-10-07T10:44:41.148258+0000","last_unstale":"2023-10-07T10:44:41.149500+0000","last_undegraded":"2023-10-07T10:44:41.149500+0000","last_fullsized":"2023-10-07T10:44:41.149500+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:12:23.908691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143943+0000","last_change":"2023-10-07T10:44:40.129052+0000","last_active":"2023-10-07T10:44:41.143943+0000","last_peered":"2023-10-07T10:44:41.143943+0000","last_clean":"2023-10-07T10:44:41.143943+0000","last_became_active":"2023-10-07T10:44:40.128796+0000","last_became_peered":"2023-10-07T10:44:40.128796+0000","last_unstale":"2023-10-07T10:44:41.143943+0000","last_undegraded":"2023-10-07T10:44:41.143943+0000","last_fullsized":"2023-10-07T10:44:41.143943+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:46:23.804562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.144885+0000","last_change":"2023-10-07T10:44:41.140223+0000","last_active":"2023-10-07T10:44:41.144885+0000","last_peered":"2023-10-07T10:44:41.144885+0000","last_clean":"2023-10-07T10:44:41.144885+0000","last_became_active":"2023-10-07T10:44:41.139752+0000","last_became_peered":"2023-10-07T10:44:41.139752+0000","last_unstale":"2023-10-07T10:44:41.144885+0000","last_undegraded":"2023-10-07T10:44:41.144885+0000","last_fullsized":"2023-10-07T10:44:41.144885+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:17.616542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.121445+0000","last_change":"2023-10-07T10:44:40.140008+0000","last_active":"2023-10-07T10:44:41.121445+0000","last_peered":"2023-10-07T10:44:41.121445+0000","last_clean":"2023-10-07T10:44:41.121445+0000","last_became_active":"2023-10-07T10:44:40.139723+0000","last_became_peered":"2023-10-07T10:44:40.139723+0000","last_unstale":"2023-10-07T10:44:41.121445+0000","last_undegraded":"2023-10-07T10:44:41.121445+0000","last_fullsized":"2023-10-07T10:44:41.121445+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:55:31.279084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148258+0000","last_change":"2023-10-07T10:44:41.144814+0000","last_active":"2023-10-07T10:44:41.148258+0000","last_peered":"2023-10-07T10:44:41.148258+0000","last_clean":"2023-10-07T10:44:41.148258+0000","last_became_active":"2023-10-07T10:44:41.144513+0000","last_became_peered":"2023-10-07T10:44:41.144513+0000","last_unstale":"2023-10-07T10:44:41.148258+0000","last_undegraded":"2023-10-07T10:44:41.148258+0000","last_fullsized":"2023-10-07T10:44:41.148258+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:53:59.503087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.131318+0000","last_change":"2023-10-07T10:44:40.139058+0000","last_active":"2023-10-07T10:44:41.131318+0000","last_peered":"2023-10-07T10:44:41.131318+0000","last_clean":"2023-10-07T10:44:41.131318+0000","last_became_active":"2023-10-07T10:44:40.138797+0000","last_became_peered":"2023-10-07T10:44:40.138797+0000","last_unstale":"2023-10-07T10:44:41.131318+0000","last_undegraded":"2023-10-07T10:44:41.131318+0000","last_fullsized":"2023-10-07T10:44:41.131318+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:00.532197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154785+0000","last_change":"2023-10-07T10:44:41.150501+0000","last_active":"2023-10-07T10:44:41.154785+0000","last_peered":"2023-10-07T10:44:41.154785+0000","last_clean":"2023-10-07T10:44:41.154785+0000","last_became_active":"2023-10-07T10:44:41.150251+0000","last_became_peered":"2023-10-07T10:44:41.150251+0000","last_unstale":"2023-10-07T10:44:41.154785+0000","last_undegraded":"2023-10-07T10:44:41.154785+0000","last_fullsized":"2023-10-07T10:44:41.154785+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:17:14.592996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122370+0000","last_change":"2023-10-07T10:44:40.140996+0000","last_active":"2023-10-07T10:44:41.122370+0000","last_peered":"2023-10-07T10:44:41.122370+0000","last_clean":"2023-10-07T10:44:41.122370+0000","last_became_active":"2023-10-07T10:44:40.139010+0000","last_became_peered":"2023-10-07T10:44:40.139010+0000","last_unstale":"2023-10-07T10:44:41.122370+0000","last_undegraded":"2023-10-07T10:44:41.122370+0000","last_fullsized":"2023-10-07T10:44:41.122370+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:18:44.321434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147388+0000","last_change":"2023-10-07T10:44:41.140563+0000","last_active":"2023-10-07T10:44:41.147388+0000","last_peered":"2023-10-07T10:44:41.147388+0000","last_clean":"2023-10-07T10:44:41.147388+0000","last_became_active":"2023-10-07T10:44:41.140301+0000","last_became_peered":"2023-10-07T10:44:41.140301+0000","last_unstale":"2023-10-07T10:44:41.147388+0000","last_undegraded":"2023-10-07T10:44:41.147388+0000","last_fullsized":"2023-10-07T10:44:41.147388+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:36.231006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122967+0000","last_change":"2023-10-07T10:44:40.141545+0000","last_active":"2023-10-07T10:44:41.122967+0000","last_peered":"2023-10-07T10:44:41.122967+0000","last_clean":"2023-10-07T10:44:41.122967+0000","last_became_active":"2023-10-07T10:44:40.141243+0000","last_became_peered":"2023-10-07T10:44:40.141243+0000","last_unstale":"2023-10-07T10:44:41.122967+0000","last_undegraded":"2023-10-07T10:44:41.122967+0000","last_fullsized":"2023-10-07T10:44:41.122967+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:36:57.908957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149062+0000","last_change":"2023-10-07T10:44:41.145151+0000","last_active":"2023-10-07T10:44:41.149062+0000","last_peered":"2023-10-07T10:44:41.149062+0000","last_clean":"2023-10-07T10:44:41.149062+0000","last_became_active":"2023-10-07T10:44:41.144863+0000","last_became_peered":"2023-10-07T10:44:41.144863+0000","last_unstale":"2023-10-07T10:44:41.149062+0000","last_undegraded":"2023-10-07T10:44:41.149062+0000","last_fullsized":"2023-10-07T10:44:41.149062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:54:09.510578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"150'3","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147846+0000","last_change":"2023-10-07T10:44:40.135545+0000","last_active":"2023-10-07T10:44:41.147846+0000","last_peered":"2023-10-07T10:44:41.147846+0000","last_clean":"2023-10-07T10:44:41.147846+0000","last_became_active":"2023-10-07T10:44:40.135252+0000","last_became_peered":"2023-10-07T10:44:40.135252+0000","last_unstale":"2023-10-07T10:44:41.147846+0000","last_undegraded":"2023-10-07T10:44:41.147846+0000","last_fullsized":"2023-10-07T10:44:41.147846+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:18.009101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149102+0000","last_change":"2023-10-07T10:44:41.144367+0000","last_active":"2023-10-07T10:44:41.149102+0000","last_peered":"2023-10-07T10:44:41.149102+0000","last_clean":"2023-10-07T10:44:41.149102+0000","last_became_active":"2023-10-07T10:44:41.144072+0000","last_became_peered":"2023-10-07T10:44:41.144072+0000","last_unstale":"2023-10-07T10:44:41.149102+0000","last_undegraded":"2023-10-07T10:44:41.149102+0000","last_fullsized":"2023-10-07T10:44:41.149102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:25.419244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"150'6","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154686+0000","last_change":"2023-10-07T10:44:40.142649+0000","last_active":"2023-10-07T10:44:41.154686+0000","last_peered":"2023-10-07T10:44:41.154686+0000","last_clean":"2023-10-07T10:44:41.154686+0000","last_became_active":"2023-10-07T10:44:40.142336+0000","last_became_peered":"2023-10-07T10:44:40.142336+0000","last_unstale":"2023-10-07T10:44:41.154686+0000","last_undegraded":"2023-10-07T10:44:41.154686+0000","last_fullsized":"2023-10-07T10:44:41.154686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:43:43.376117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154849+0000","last_change":"2023-10-07T10:44:41.149453+0000","last_active":"2023-10-07T10:44:41.154849+0000","last_peered":"2023-10-07T10:44:41.154849+0000","last_clean":"2023-10-07T10:44:41.154849+0000","last_became_active":"2023-10-07T10:44:41.149189+0000","last_became_peered":"2023-10-07T10:44:41.149189+0000","last_unstale":"2023-10-07T10:44:41.154849+0000","last_undegraded":"2023-10-07T10:44:41.154849+0000","last_fullsized":"2023-10-07T10:44:41.154849+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:29:16.993784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.126801+0000","last_change":"2023-10-07T10:44:40.137862+0000","last_active":"2023-10-07T10:44:41.126801+0000","last_peered":"2023-10-07T10:44:41.126801+0000","last_clean":"2023-10-07T10:44:41.126801+0000","last_became_active":"2023-10-07T10:44:40.137443+0000","last_became_peered":"2023-10-07T10:44:40.137443+0000","last_unstale":"2023-10-07T10:44:41.126801+0000","last_undegraded":"2023-10-07T10:44:41.126801+0000","last_fullsized":"2023-10-07T10:44:41.126801+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:47:27.687000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148230+0000","last_change":"2023-10-07T10:44:41.144640+0000","last_active":"2023-10-07T10:44:41.148230+0000","last_peered":"2023-10-07T10:44:41.148230+0000","last_clean":"2023-10-07T10:44:41.148230+0000","last_became_active":"2023-10-07T10:44:41.144366+0000","last_became_peered":"2023-10-07T10:44:41.144366+0000","last_unstale":"2023-10-07T10:44:41.148230+0000","last_undegraded":"2023-10-07T10:44:41.148230+0000","last_fullsized":"2023-10-07T10:44:41.148230+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:55:32.286792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.130941+0000","last_change":"2023-10-07T10:44:40.140151+0000","last_active":"2023-10-07T10:44:41.130941+0000","last_peered":"2023-10-07T10:44:41.130941+0000","last_clean":"2023-10-07T10:44:41.130941+0000","last_became_active":"2023-10-07T10:44:40.139824+0000","last_became_peered":"2023-10-07T10:44:40.139824+0000","last_unstale":"2023-10-07T10:44:41.130941+0000","last_undegraded":"2023-10-07T10:44:41.130941+0000","last_fullsized":"2023-10-07T10:44:41.130941+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:36:54.689289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153797+0000","last_change":"2023-10-07T10:44:41.147074+0000","last_active":"2023-10-07T10:44:41.153797+0000","last_peered":"2023-10-07T10:44:41.153797+0000","last_clean":"2023-10-07T10:44:41.153797+0000","last_became_active":"2023-10-07T10:44:41.146796+0000","last_became_peered":"2023-10-07T10:44:41.146796+0000","last_unstale":"2023-10-07T10:44:41.153797+0000","last_undegraded":"2023-10-07T10:44:41.153797+0000","last_fullsized":"2023-10-07T10:44:41.153797+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:36:58.916666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"150'8","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.144815+0000","last_change":"2023-10-07T10:44:40.137704+0000","last_active":"2023-10-07T10:44:41.144815+0000","last_peered":"2023-10-07T10:44:41.144815+0000","last_clean":"2023-10-07T10:44:41.144815+0000","last_became_active":"2023-10-07T10:44:40.137390+0000","last_became_peered":"2023-10-07T10:44:40.137390+0000","last_unstale":"2023-10-07T10:44:41.144815+0000","last_undegraded":"2023-10-07T10:44:41.144815+0000","last_fullsized":"2023-10-07T10:44:41.144815+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:08.592940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148532+0000","last_change":"2023-10-07T10:44:41.143114+0000","last_active":"2023-10-07T10:44:41.148532+0000","last_peered":"2023-10-07T10:44:41.148532+0000","last_clean":"2023-10-07T10:44:41.148532+0000","last_became_active":"2023-10-07T10:44:41.142629+0000","last_became_peered":"2023-10-07T10:44:41.142629+0000","last_unstale":"2023-10-07T10:44:41.148532+0000","last_undegraded":"2023-10-07T10:44:41.148532+0000","last_fullsized":"2023-10-07T10:44:41.148532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:26.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147273+0000","last_change":"2023-10-07T10:44:40.142936+0000","last_active":"2023-10-07T10:44:41.147273+0000","last_peered":"2023-10-07T10:44:41.147273+0000","last_clean":"2023-10-07T10:44:41.147273+0000","last_became_active":"2023-10-07T10:44:40.142623+0000","last_became_peered":"2023-10-07T10:44:40.142623+0000","last_unstale":"2023-10-07T10:44:41.147273+0000","last_undegraded":"2023-10-07T10:44:41.147273+0000","last_fullsized":"2023-10-07T10:44:41.147273+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:35:13.205662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153104+0000","last_change":"2023-10-07T10:44:41.150568+0000","last_active":"2023-10-07T10:44:41.153104+0000","last_peered":"2023-10-07T10:44:41.153104+0000","last_clean":"2023-10-07T10:44:41.153104+0000","last_became_active":"2023-10-07T10:44:41.150087+0000","last_became_peered":"2023-10-07T10:44:41.150087+0000","last_unstale":"2023-10-07T10:44:41.153104+0000","last_undegraded":"2023-10-07T10:44:41.153104+0000","last_fullsized":"2023-10-07T10:44:41.153104+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:45:31.355758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149721+0000","last_change":"2023-10-07T10:44:40.138593+0000","last_active":"2023-10-07T10:44:41.149721+0000","last_peered":"2023-10-07T10:44:41.149721+0000","last_clean":"2023-10-07T10:44:41.149721+0000","last_became_active":"2023-10-07T10:44:40.138341+0000","last_became_peered":"2023-10-07T10:44:40.138341+0000","last_unstale":"2023-10-07T10:44:41.149721+0000","last_undegraded":"2023-10-07T10:44:41.149721+0000","last_fullsized":"2023-10-07T10:44:41.149721+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:31:07.264451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142776+0000","last_change":"2023-10-07T10:44:41.140201+0000","last_active":"2023-10-07T10:44:41.142776+0000","last_peered":"2023-10-07T10:44:41.142776+0000","last_clean":"2023-10-07T10:44:41.142776+0000","last_became_active":"2023-10-07T10:44:41.139849+0000","last_became_peered":"2023-10-07T10:44:41.139849+0000","last_unstale":"2023-10-07T10:44:41.142776+0000","last_undegraded":"2023-10-07T10:44:41.142776+0000","last_fullsized":"2023-10-07T10:44:41.142776+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:58:45.712787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142946+0000","last_change":"2023-10-07T10:44:40.132723+0000","last_active":"2023-10-07T10:44:41.142946+0000","last_peered":"2023-10-07T10:44:41.142946+0000","last_clean":"2023-10-07T10:44:41.142946+0000","last_became_active":"2023-10-07T10:44:40.132449+0000","last_became_peered":"2023-10-07T10:44:40.132449+0000","last_unstale":"2023-10-07T10:44:41.142946+0000","last_undegraded":"2023-10-07T10:44:41.142946+0000","last_fullsized":"2023-10-07T10:44:41.142946+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:25:13.934597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150293+0000","last_change":"2023-10-07T10:44:41.145085+0000","last_active":"2023-10-07T10:44:41.150293+0000","last_peered":"2023-10-07T10:44:41.150293+0000","last_clean":"2023-10-07T10:44:41.150293+0000","last_became_active":"2023-10-07T10:44:41.144661+0000","last_became_peered":"2023-10-07T10:44:41.144661+0000","last_unstale":"2023-10-07T10:44:41.150293+0000","last_undegraded":"2023-10-07T10:44:41.150293+0000","last_fullsized":"2023-10-07T10:44:41.150293+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:36:05.572351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122151+0000","last_change":"2023-10-07T10:44:40.137697+0000","last_active":"2023-10-07T10:44:41.122151+0000","last_peered":"2023-10-07T10:44:41.122151+0000","last_clean":"2023-10-07T10:44:41.122151+0000","last_became_active":"2023-10-07T10:44:40.137443+0000","last_became_peered":"2023-10-07T10:44:40.137443+0000","last_unstale":"2023-10-07T10:44:41.122151+0000","last_undegraded":"2023-10-07T10:44:41.122151+0000","last_fullsized":"2023-10-07T10:44:41.122151+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:17:13.585287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154821+0000","last_change":"2023-10-07T10:44:41.141102+0000","last_active":"2023-10-07T10:44:41.154821+0000","last_peered":"2023-10-07T10:44:41.154821+0000","last_clean":"2023-10-07T10:44:41.154821+0000","last_became_active":"2023-10-07T10:44:41.140838+0000","last_became_peered":"2023-10-07T10:44:41.140838+0000","last_unstale":"2023-10-07T10:44:41.154821+0000","last_undegraded":"2023-10-07T10:44:41.154821+0000","last_fullsized":"2023-10-07T10:44:41.154821+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:18:45.329143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152492+0000","last_change":"2023-10-07T10:44:40.140398+0000","last_active":"2023-10-07T10:44:41.152492+0000","last_peered":"2023-10-07T10:44:41.152492+0000","last_clean":"2023-10-07T10:44:41.152492+0000","last_became_active":"2023-10-07T10:44:40.140127+0000","last_became_peered":"2023-10-07T10:44:40.140127+0000","last_unstale":"2023-10-07T10:44:41.152492+0000","last_undegraded":"2023-10-07T10:44:41.152492+0000","last_fullsized":"2023-10-07T10:44:41.152492+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:36:54.689289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148657+0000","last_change":"2023-10-07T10:44:41.144404+0000","last_active":"2023-10-07T10:44:41.148657+0000","last_peered":"2023-10-07T10:44:41.148657+0000","last_clean":"2023-10-07T10:44:41.148657+0000","last_became_active":"2023-10-07T10:44:41.144121+0000","last_became_peered":"2023-10-07T10:44:41.144121+0000","last_unstale":"2023-10-07T10:44:41.148657+0000","last_undegraded":"2023-10-07T10:44:41.148657+0000","last_fullsized":"2023-10-07T10:44:41.148657+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:57:26.369780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122503+0000","last_change":"2023-10-07T10:44:40.143918+0000","last_active":"2023-10-07T10:44:41.122503+0000","last_peered":"2023-10-07T10:44:41.122503+0000","last_clean":"2023-10-07T10:44:41.122503+0000","last_became_active":"2023-10-07T10:44:40.143653+0000","last_became_peered":"2023-10-07T10:44:40.143653+0000","last_unstale":"2023-10-07T10:44:41.122503+0000","last_undegraded":"2023-10-07T10:44:41.122503+0000","last_fullsized":"2023-10-07T10:44:41.122503+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:50:57.857575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.155395+0000","last_change":"2023-10-07T10:44:41.151070+0000","last_active":"2023-10-07T10:44:41.155395+0000","last_peered":"2023-10-07T10:44:41.155395+0000","last_clean":"2023-10-07T10:44:41.155395+0000","last_became_active":"2023-10-07T10:44:41.150797+0000","last_became_peered":"2023-10-07T10:44:41.150797+0000","last_unstale":"2023-10-07T10:44:41.155395+0000","last_undegraded":"2023-10-07T10:44:41.155395+0000","last_fullsized":"2023-10-07T10:44:41.155395+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:43:44.383826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.121693+0000","last_change":"2023-10-07T10:44:40.133500+0000","last_active":"2023-10-07T10:44:41.121693+0000","last_peered":"2023-10-07T10:44:41.121693+0000","last_clean":"2023-10-07T10:44:41.121693+0000","last_became_active":"2023-10-07T10:44:40.132841+0000","last_became_peered":"2023-10-07T10:44:40.132841+0000","last_unstale":"2023-10-07T10:44:41.121693+0000","last_undegraded":"2023-10-07T10:44:41.121693+0000","last_fullsized":"2023-10-07T10:44:41.121693+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:05.014756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149046+0000","last_change":"2023-10-07T10:44:41.145335+0000","last_active":"2023-10-07T10:44:41.149046+0000","last_peered":"2023-10-07T10:44:41.149046+0000","last_clean":"2023-10-07T10:44:41.149046+0000","last_became_active":"2023-10-07T10:44:41.144671+0000","last_became_peered":"2023-10-07T10:44:41.144671+0000","last_unstale":"2023-10-07T10:44:41.149046+0000","last_undegraded":"2023-10-07T10:44:41.149046+0000","last_fullsized":"2023-10-07T10:44:41.149046+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:05:50.642947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.135866+0000","last_change":"2023-10-07T10:44:40.141168+0000","last_active":"2023-10-07T10:44:41.135866+0000","last_peered":"2023-10-07T10:44:41.135866+0000","last_clean":"2023-10-07T10:44:41.135866+0000","last_became_active":"2023-10-07T10:44:40.140927+0000","last_became_peered":"2023-10-07T10:44:40.140927+0000","last_unstale":"2023-10-07T10:44:41.135866+0000","last_undegraded":"2023-10-07T10:44:41.135866+0000","last_fullsized":"2023-10-07T10:44:41.135866+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:16.608833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.133173+0000","last_change":"2023-10-07T10:44:41.125440+0000","last_active":"2023-10-07T10:44:41.133173+0000","last_peered":"2023-10-07T10:44:41.133173+0000","last_clean":"2023-10-07T10:44:41.133173+0000","last_became_active":"2023-10-07T10:44:41.125118+0000","last_became_peered":"2023-10-07T10:44:41.125118+0000","last_unstale":"2023-10-07T10:44:41.133173+0000","last_undegraded":"2023-10-07T10:44:41.133173+0000","last_fullsized":"2023-10-07T10:44:41.133173+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:32:32.458450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152265+0000","last_change":"2023-10-07T10:44:40.136125+0000","last_active":"2023-10-07T10:44:41.152265+0000","last_peered":"2023-10-07T10:44:41.152265+0000","last_clean":"2023-10-07T10:44:41.152265+0000","last_became_active":"2023-10-07T10:44:40.135826+0000","last_became_peered":"2023-10-07T10:44:40.135826+0000","last_unstale":"2023-10-07T10:44:41.152265+0000","last_undegraded":"2023-10-07T10:44:41.152265+0000","last_fullsized":"2023-10-07T10:44:41.152265+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:23:58.168750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154914+0000","last_change":"2023-10-07T10:44:41.150823+0000","last_active":"2023-10-07T10:44:41.154914+0000","last_peered":"2023-10-07T10:44:41.154914+0000","last_clean":"2023-10-07T10:44:41.154914+0000","last_became_active":"2023-10-07T10:44:41.147430+0000","last_became_peered":"2023-10-07T10:44:41.147430+0000","last_unstale":"2023-10-07T10:44:41.154914+0000","last_undegraded":"2023-10-07T10:44:41.154914+0000","last_fullsized":"2023-10-07T10:44:41.154914+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:52.219810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.127113+0000","last_change":"2023-10-07T10:44:40.137441+0000","last_active":"2023-10-07T10:44:41.127113+0000","last_peered":"2023-10-07T10:44:41.127113+0000","last_clean":"2023-10-07T10:44:41.127113+0000","last_became_active":"2023-10-07T10:44:40.137111+0000","last_became_peered":"2023-10-07T10:44:40.137111+0000","last_unstale":"2023-10-07T10:44:41.127113+0000","last_undegraded":"2023-10-07T10:44:41.127113+0000","last_fullsized":"2023-10-07T10:44:41.127113+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:45:02.843088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145094+0000","last_change":"2023-10-07T10:44:41.137903+0000","last_active":"2023-10-07T10:44:41.145094+0000","last_peered":"2023-10-07T10:44:41.145094+0000","last_clean":"2023-10-07T10:44:41.145094+0000","last_became_active":"2023-10-07T10:44:41.137616+0000","last_became_peered":"2023-10-07T10:44:41.137616+0000","last_unstale":"2023-10-07T10:44:41.145094+0000","last_undegraded":"2023-10-07T10:44:41.145094+0000","last_fullsized":"2023-10-07T10:44:41.145094+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:25:14.942305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.131176+0000","last_change":"2023-10-07T10:44:40.138312+0000","last_active":"2023-10-07T10:44:41.131176+0000","last_peered":"2023-10-07T10:44:41.131176+0000","last_clean":"2023-10-07T10:44:41.131176+0000","last_became_active":"2023-10-07T10:44:40.137993+0000","last_became_peered":"2023-10-07T10:44:40.137993+0000","last_unstale":"2023-10-07T10:44:41.131176+0000","last_undegraded":"2023-10-07T10:44:41.131176+0000","last_fullsized":"2023-10-07T10:44:41.131176+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:17:13.585287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142745+0000","last_change":"2023-10-07T10:44:41.139651+0000","last_active":"2023-10-07T10:44:41.142745+0000","last_peered":"2023-10-07T10:44:41.142745+0000","last_clean":"2023-10-07T10:44:41.142745+0000","last_became_active":"2023-10-07T10:44:41.139308+0000","last_became_peered":"2023-10-07T10:44:41.139308+0000","last_unstale":"2023-10-07T10:44:41.142745+0000","last_undegraded":"2023-10-07T10:44:41.142745+0000","last_fullsized":"2023-10-07T10:44:41.142745+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:17.616542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.134850+0000","last_change":"2023-10-07T10:44:40.130192+0000","last_active":"2023-10-07T10:44:41.134850+0000","last_peered":"2023-10-07T10:44:41.134850+0000","last_clean":"2023-10-07T10:44:41.134850+0000","last_became_active":"2023-10-07T10:44:40.129910+0000","last_became_peered":"2023-10-07T10:44:40.129910+0000","last_unstale":"2023-10-07T10:44:41.134850+0000","last_undegraded":"2023-10-07T10:44:41.134850+0000","last_fullsized":"2023-10-07T10:44:41.134850+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:25.361797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148713+0000","last_change":"2023-10-07T10:44:41.139533+0000","last_active":"2023-10-07T10:44:41.148713+0000","last_peered":"2023-10-07T10:44:41.148713+0000","last_clean":"2023-10-07T10:44:41.148713+0000","last_became_active":"2023-10-07T10:44:41.139218+0000","last_became_peered":"2023-10-07T10:44:41.139218+0000","last_unstale":"2023-10-07T10:44:41.148713+0000","last_undegraded":"2023-10-07T10:44:41.148713+0000","last_fullsized":"2023-10-07T10:44:41.148713+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:40:18.432284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.131145+0000","last_change":"2023-10-07T10:44:40.142553+0000","last_active":"2023-10-07T10:44:41.131145+0000","last_peered":"2023-10-07T10:44:41.131145+0000","last_clean":"2023-10-07T10:44:41.131145+0000","last_became_active":"2023-10-07T10:44:40.142320+0000","last_became_peered":"2023-10-07T10:44:40.142320+0000","last_unstale":"2023-10-07T10:44:41.131145+0000","last_undegraded":"2023-10-07T10:44:41.131145+0000","last_fullsized":"2023-10-07T10:44:41.131145+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:18:44.321434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147164+0000","last_change":"2023-10-07T10:44:41.142175+0000","last_active":"2023-10-07T10:44:41.147164+0000","last_peered":"2023-10-07T10:44:41.147164+0000","last_clean":"2023-10-07T10:44:41.147164+0000","last_became_active":"2023-10-07T10:44:41.141923+0000","last_became_peered":"2023-10-07T10:44:41.141923+0000","last_unstale":"2023-10-07T10:44:41.147164+0000","last_undegraded":"2023-10-07T10:44:41.147164+0000","last_fullsized":"2023-10-07T10:44:41.147164+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:44:44.790717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.126261+0000","last_change":"2023-10-07T10:44:40.142046+0000","last_active":"2023-10-07T10:44:41.126261+0000","last_peered":"2023-10-07T10:44:41.126261+0000","last_clean":"2023-10-07T10:44:41.126261+0000","last_became_active":"2023-10-07T10:44:40.141697+0000","last_became_peered":"2023-10-07T10:44:40.141697+0000","last_unstale":"2023-10-07T10:44:41.126261+0000","last_undegraded":"2023-10-07T10:44:41.126261+0000","last_fullsized":"2023-10-07T10:44:41.126261+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:51.467881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145291+0000","last_change":"2023-10-07T10:44:41.144274+0000","last_active":"2023-10-07T10:44:41.145291+0000","last_peered":"2023-10-07T10:44:41.145291+0000","last_clean":"2023-10-07T10:44:41.145291+0000","last_became_active":"2023-10-07T10:44:41.143961+0000","last_became_peered":"2023-10-07T10:44:41.143961+0000","last_unstale":"2023-10-07T10:44:41.145291+0000","last_undegraded":"2023-10-07T10:44:41.145291+0000","last_fullsized":"2023-10-07T10:44:41.145291+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:33:08.496023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.121806+0000","last_change":"2023-10-07T10:44:40.139190+0000","last_active":"2023-10-07T10:44:41.121806+0000","last_peered":"2023-10-07T10:44:41.121806+0000","last_clean":"2023-10-07T10:44:41.121806+0000","last_became_active":"2023-10-07T10:44:40.138909+0000","last_became_peered":"2023-10-07T10:44:40.138909+0000","last_unstale":"2023-10-07T10:44:41.121806+0000","last_undegraded":"2023-10-07T10:44:41.121806+0000","last_fullsized":"2023-10-07T10:44:41.121806+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:20.602757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145141+0000","last_change":"2023-10-07T10:44:41.137601+0000","last_active":"2023-10-07T10:44:41.145141+0000","last_peered":"2023-10-07T10:44:41.145141+0000","last_clean":"2023-10-07T10:44:41.145141+0000","last_became_active":"2023-10-07T10:44:41.137310+0000","last_became_peered":"2023-10-07T10:44:41.137310+0000","last_unstale":"2023-10-07T10:44:41.145141+0000","last_undegraded":"2023-10-07T10:44:41.145141+0000","last_fullsized":"2023-10-07T10:44:41.145141+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:09.600649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.135918+0000","last_change":"2023-10-07T10:44:40.134547+0000","last_active":"2023-10-07T10:44:41.135918+0000","last_peered":"2023-10-07T10:44:41.135918+0000","last_clean":"2023-10-07T10:44:41.135918+0000","last_became_active":"2023-10-07T10:44:40.134284+0000","last_became_peered":"2023-10-07T10:44:40.134284+0000","last_unstale":"2023-10-07T10:44:41.135918+0000","last_undegraded":"2023-10-07T10:44:41.135918+0000","last_fullsized":"2023-10-07T10:44:41.135918+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:57:25.362071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149070+0000","last_change":"2023-10-07T10:44:41.144985+0000","last_active":"2023-10-07T10:44:41.149070+0000","last_peered":"2023-10-07T10:44:41.149070+0000","last_clean":"2023-10-07T10:44:41.149070+0000","last_became_active":"2023-10-07T10:44:41.144649+0000","last_became_peered":"2023-10-07T10:44:41.144649+0000","last_unstale":"2023-10-07T10:44:41.149070+0000","last_undegraded":"2023-10-07T10:44:41.149070+0000","last_fullsized":"2023-10-07T10:44:41.149070+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:07.425202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.156996+0000","last_change":"2023-10-07T10:44:40.143873+0000","last_active":"2023-10-07T10:44:41.156996+0000","last_peered":"2023-10-07T10:44:41.156996+0000","last_clean":"2023-10-07T10:44:41.156996+0000","last_became_active":"2023-10-07T10:44:40.143455+0000","last_became_peered":"2023-10-07T10:44:40.143455+0000","last_unstale":"2023-10-07T10:44:41.156996+0000","last_undegraded":"2023-10-07T10:44:41.156996+0000","last_fullsized":"2023-10-07T10:44:41.156996+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:37:33.934868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"141'271","reported_seq":819,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.125378+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T10:44:41.125378+0000","last_peered":"2023-10-07T10:44:41.125378+0000","last_clean":"2023-10-07T10:44:41.125378+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T10:44:41.125378+0000","last_undegraded":"2023-10-07T10:44:41.125378+0000","last_fullsized":"2023-10-07T10:44:41.125378+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477635,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307912,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063928,"statfs":{"total":96636764160,"available":96321462272,"internally_reserved":0,"allocated":2490368,"data_stored":1507373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.95699999999999996,"5min":1.018,"15min":1.018},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.028,"15min":1.028},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.748,"15min":0.752},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.432},"max":{"1min":0.90900000000000003,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.75900000000000001},"min":{"1min":0.56999999999999995,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.995,"5min":1.119,"15min":1.119},"last":0.504}]},{"osd":2,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88200000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78000000000000003,"15min":0.77100000000000002},"min":{"1min":0.65900000000000003,"5min":0.55900000000000005,"15min":0.45400000000000001},"max":{"1min":1.002,"5min":1.147,"15min":1.147},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.746,"15min":0.73999999999999999},"min":{"1min":0.51600000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.94199999999999995,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.59399999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93899999999999995,"5min":1.161,"15min":1.161},"last":0.67900000000000005}]},{"osd":4,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.47699999999999998,"5min":0.44700000000000001,"15min":0.36399999999999999},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.92700000000000005,"5min":1.069,"15min":1.069},"last":0.66000000000000003}]},{"osd":5,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.83899999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70799999999999996,"15min":0.69599999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.746,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71199999999999997},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86699999999999999,"5min":1.131,"15min":1.131},"last":0.76200000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311416,"kb_used_data":1840,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060424,"statfs":{"total":96636764160,"available":96317874176,"internally_reserved":0,"allocated":1884160,"data_stored":916395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":0.75900000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90700000000000003,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.54400000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92600000000000005,"5min":1.052,"15min":1.073},"last":0.73599999999999999}]},{"osd":1,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.75800000000000001,"15min":0.754},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48499999999999999},"max":{"1min":0.874,"5min":1.008,"15min":1.0600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.50700000000000001},"max":{"1min":0.89000000000000001,"5min":1.034,"15min":1.0389999999999999},"last":0.76800000000000002}]},{"osd":2,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46100000000000002},"max":{"1min":1.018,"5min":1.0289999999999999,"15min":1.3380000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59499999999999997,"5min":0.54900000000000004,"15min":0.46500000000000002},"max":{"1min":0.95999999999999996,"5min":1.03,"15min":1.367},"last":0.88100000000000001}]},{"osd":3,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.51100000000000001},"max":{"1min":0.94599999999999995,"5min":1.044,"15min":1.0900000000000001},"last":0.75},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":1.0289999999999999,"15min":1.1040000000000001},"last":0.91900000000000004}]},{"osd":4,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.621},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0609999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.80900000000000005}]},{"osd":5,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.499,"5min":0.42899999999999999,"15min":0.40999999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.627},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0900000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.52000000000000002,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.1060000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76700000000000002,"15min":0.77000000000000002},"min":{"1min":0.57999999999999996,"5min":0.57899999999999996,"15min":0.53600000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.127},"last":0.89300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129543041,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307296,"kb_used_data":1816,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064544,"statfs":{"total":96636764160,"available":96322093056,"internally_reserved":0,"allocated":1859584,"data_stored":908151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.495,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.28899999999999998},"max":{"1min":0.70099999999999996,"5min":0.76000000000000001,"15min":0.83099999999999996},"last":0.439},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.45900000000000002,"5min":0.40799999999999997,"15min":0.40500000000000003},"max":{"1min":0.69699999999999995,"5min":0.76900000000000002,"15min":0.84199999999999997},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.35899999999999999},"max":{"1min":0.71399999999999997,"5min":0.74399999999999999,"15min":0.81699999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.374},"max":{"1min":0.77600000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63500000000000001,"15min":0.64000000000000001},"min":{"1min":0.505,"5min":0.48099999999999998,"15min":0.41999999999999998},"max":{"1min":0.85399999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.93100000000000005},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.83499999999999996,"15min":0.83399999999999996},"min":{"1min":0.64600000000000002,"5min":0.58899999999999997,"15min":0.56999999999999995},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1719999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.80600000000000005,"15min":0.80400000000000005},"min":{"1min":0.49399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.141},"last":0.84499999999999997}]},{"osd":5,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.84999999999999998,"15min":0.84699999999999998},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.54400000000000004},"max":{"1min":1.155,"5min":1.196,"15min":1.196},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82099999999999995,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"max":{"1min":0.97799999999999998,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.93200000000000005}]},{"osd":6,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.68600000000000005,"5min":0.56399999999999995,"15min":0.56100000000000005},"max":{"1min":1.153,"5min":1.153,"15min":1.2050000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.84099999999999997,"15min":0.84899999999999998},"min":{"1min":0.67100000000000004,"5min":0.60999999999999999,"15min":0.57199999999999995},"max":{"1min":1.0389999999999999,"5min":1.133,"15min":1.196},"last":0.80200000000000005}]},{"osd":7,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.86499999999999999,"15min":0.877},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.60899999999999999},"max":{"1min":1.0600000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88600000000000001,"15min":0.89000000000000001},"min":{"1min":0.72699999999999998,"5min":0.67100000000000004,"15min":0.60799999999999998},"max":{"1min":1.085,"5min":1.179,"15min":1.179},"last":0.98899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543041,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307888,"kb_used_data":2408,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063952,"statfs":{"total":96636764160,"available":96321486848,"internally_reserved":0,"allocated":2465792,"data_stored":1505227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.61099999999999999,"5min":0.751,"15min":0.76400000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":0.82899999999999996},"last":0.443}]},{"osd":2,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75,"5min":0.88900000000000001,"15min":0.92800000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.68400000000000005,"5min":0.87,"15min":0.87},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70899999999999996,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.80600000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.52500000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81599999999999995,"15min":0.79700000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.499},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.64700000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0960000000000001,"5min":1.194,"15min":1.194},"last":0.86699999999999999}]},{"osd":5,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.077,"5min":1.278,"15min":1.278},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":1.0229999999999999,"5min":1.206,"15min":1.206},"last":0.88700000000000001}]},{"osd":6,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86199999999999999,"15min":0.84699999999999998},"min":{"1min":0.67100000000000004,"5min":0.60599999999999998,"15min":0.59499999999999997},"max":{"1min":1.0369999999999999,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85799999999999998,"15min":0.85099999999999998},"min":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"max":{"1min":1.1579999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.97499999999999998}]},{"osd":7,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.89300000000000002,"15min":0.872},"min":{"1min":0.627,"5min":0.627,"15min":0.59899999999999998},"max":{"1min":1.131,"5min":1.173,"15min":1.198},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.875,"15min":0.86599999999999999},"min":{"1min":0.63100000000000001,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":1.198,"5min":1.198,"15min":1.234},"last":0.90900000000000003}]}]},{"osd":2,"up_from":14,"seq":60129543042,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311412,"kb_used_data":1836,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060428,"statfs":{"total":96636764160,"available":96317878272,"internally_reserved":0,"allocated":1880064,"data_stored":915447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.438},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.38},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.64000000000000001},"min":{"1min":0.51200000000000001,"5min":0.502,"15min":0.44500000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64300000000000002,"15min":0.629},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":0.91400000000000003,"15min":1.0529999999999999},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.79900000000000004,"15min":0.79700000000000004},"min":{"1min":0.56499999999999995,"5min":0.504,"15min":0.48499999999999999},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.82999999999999996,"15min":0.82199999999999995},"min":{"1min":0.71799999999999997,"5min":0.59599999999999997,"15min":0.47299999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.66500000000000004}]},{"osd":5,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84599999999999997,"15min":0.84799999999999998},"min":{"1min":0.77800000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":1.212,"15min":1.349},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.81699999999999995,"15min":0.81999999999999995},"min":{"1min":0.62,"5min":0.56599999999999995,"15min":0.54000000000000004},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.86499999999999999,"15min":0.86199999999999999},"min":{"1min":0.77900000000000003,"5min":0.59399999999999997,"15min":0.55200000000000005},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.85899999999999999,"15min":0.85499999999999998},"min":{"1min":0.72399999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.2430000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.89000000000000001,"15min":0.879},"min":{"1min":0.80400000000000005,"5min":0.66300000000000003,"15min":0.59999999999999998},"max":{"1min":1.0820000000000001,"5min":1.1679999999999999,"15min":1.3160000000000001},"last":0.754},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.70599999999999996,"5min":0.61299999999999999,"15min":0.55500000000000005},"max":{"1min":1.139,"5min":1.139,"15min":1.2969999999999999},"last":0.76900000000000002}]}]},{"osd":3,"up_from":14,"seq":60129543043,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299124,"kb_used_data":1836,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072716,"statfs":{"total":96636764160,"available":96330461184,"internally_reserved":0,"allocated":1880064,"data_stored":915019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.49099999999999999,"15min":0.47399999999999998},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73099999999999998,"5min":1.073,"15min":1.073},"last":0.313},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81699999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52700000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76900000000000002,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.52600000000000002,"5min":0.46300000000000002,"15min":0.45000000000000001},"max":{"1min":0.76000000000000001,"5min":1.02,"15min":1.02},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.42599999999999999,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":1.456,"15min":1.456},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.48599999999999999,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.75,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.63900000000000001,"5min":0.59099999999999997,"15min":0.56299999999999994},"max":{"1min":0.97899999999999998,"5min":1.319,"15min":1.3919999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81299999999999994,"15min":0.81000000000000005},"min":{"1min":0.65000000000000002,"5min":0.61099999999999999,"15min":0.60899999999999999},"max":{"1min":1.141,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83599999999999997,"15min":0.82099999999999995},"min":{"1min":0.65500000000000003,"5min":0.61799999999999999,"15min":0.56799999999999995},"max":{"1min":1.097,"5min":1.1879999999999999,"15min":1.2829999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.85099999999999998,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.63300000000000001,"15min":0.55600000000000005},"max":{"1min":1.1619999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.81899999999999995}]},{"osd":6,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.63,"5min":0.57099999999999995,"15min":0.56599999999999995},"max":{"1min":1.214,"5min":1.357,"15min":1.357},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.86599999999999999,"15min":0.84999999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.58799999999999997},"max":{"1min":1.1279999999999999,"5min":1.177,"15min":1.321},"last":0.747}]},{"osd":7,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88300000000000001,"15min":0.85699999999999998},"min":{"1min":0.72099999999999997,"5min":0.63200000000000001,"15min":0.58499999999999996},"max":{"1min":1.137,"5min":1.492,"15min":1.492},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.872,"5min":0.88900000000000001,"15min":0.86899999999999999},"min":{"1min":0.68700000000000006,"5min":0.64000000000000001,"15min":0.59599999999999997},"max":{"1min":1.161,"5min":1.411,"15min":1.411},"last":0.63200000000000001}]}]},{"osd":4,"up_from":15,"seq":64424510339,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311408,"kb_used_data":1832,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060432,"statfs":{"total":96636764160,"available":96317882368,"internally_reserved":0,"allocated":1875968,"data_stored":914929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93600000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.745,"15min":0.76000000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86499999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.76400000000000001,"15min":0.76500000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.016,"15min":1.032},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46500000000000002},"max":{"1min":0.84599999999999997,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":0.90300000000000002,"5min":1.002,"15min":1.01},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88500000000000001,"5min":1.145,"15min":1.145},"last":0.88300000000000001}]},{"osd":3,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.75600000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.497},"max":{"1min":0.82899999999999996,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96599999999999997,"5min":1.129,"15min":1.129},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45000000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.441,"5min":0.441,"15min":0.39400000000000002},"max":{"1min":0.85599999999999998,"5min":0.93999999999999995,"15min":1.0349999999999999},"last":0.996}]},{"osd":6,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89200000000000002,"5min":0.98999999999999999,"15min":1.018},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.69899999999999995},"min":{"1min":0.54500000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85499999999999998,"5min":0.96799999999999997,"15min":1.042},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.54600000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90400000000000003,"5min":0.94099999999999995,"15min":0.97899999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0960000000000001},"last":0.85499999999999998}]}]},{"osd":5,"up_from":14,"seq":60129543041,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311412,"kb_used_data":1836,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060428,"statfs":{"total":96636764160,"available":96317878272,"internally_reserved":0,"allocated":1880064,"data_stored":914971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78700000000000003,"15min":0.79400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39500000000000002},"max":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.091},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.438},"max":{"1min":0.93500000000000005,"5min":1.0820000000000001,"15min":1.1279999999999999},"last":1.391}]},{"osd":1,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.746},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93400000000000005,"5min":1.032,"15min":1.032},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78100000000000003,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.46899999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":5.0780000000000003}]},{"osd":2,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89900000000000002,"5min":1.0760000000000001,"15min":1.093},"last":3.415},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.76700000000000002,"15min":0.76800000000000002},"min":{"1min":0.52300000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89100000000000001,"5min":1.077,"15min":1.343},"last":1.2989999999999999}]},{"osd":3,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.77600000000000002,"15min":0.78200000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.50800000000000001},"max":{"1min":0.872,"5min":1.121,"15min":1.121},"last":1.341},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75700000000000001,"15min":0.754},"min":{"1min":0.61799999999999999,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":3.3570000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.91400000000000003,"5min":0.94999999999999996,"15min":0.995},"last":1.9630000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.64700000000000002},"min":{"1min":0.47199999999999998,"5min":0.46500000000000002,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":1.0089999999999999,"15min":1.0529999999999999},"last":3.2999999999999998}]},{"osd":6,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.52400000000000002,"5min":0.44500000000000001,"15min":0.39800000000000002},"max":{"1min":0.91900000000000004,"5min":1.016,"15min":1.083},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.71499999999999997,"15min":0.71299999999999997},"min":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0289999999999999,"5min":1.032,"15min":1.101},"last":12.090999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.73499999999999999,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.075},"last":1.835},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":1.4379999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T10:44:45.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T10:44:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.765+0000 7faa67bff700 1 Processor -- start 2023-10-07T10:44:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.765+0000 7faa67bff700 1 -- start start 2023-10-07T10:44:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa67bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faa600ad370 0x7faa600cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa67bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6015a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa67bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa601568c0 0x7faa60156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa67bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6014d790 con 0x7faa601568c0 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa67bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa600ae5d0 con 0x7faa600ad370 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa67bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6015a850 con 0x7faa60159ec0 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa6619c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa601568c0 0x7faa60156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa6619c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa601568c0 0x7faa60156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38804/0 (socket says 172.21.15.29:38804) 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.766+0000 7faa6619c700 1 -- 172.21.15.29:0/399269993 learned_addr learned my addr 172.21.15.29:0/399269993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6599b700 1 --2- 172.21.15.29:0/399269993 >> v2:172.21.15.144:3300/0 conn(0x7faa600ad370 0x7faa600cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6519a700 1 --2- 172.21.15.29:0/399269993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6619c700 1 -- 172.21.15.29:0/399269993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 msgr2=0x7faa6015a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6619c700 1 --2- 172.21.15.29:0/399269993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6015a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6619c700 1 -- 172.21.15.29:0/399269993 >> v2:172.21.15.144:3300/0 conn(0x7faa600ad370 msgr2=0x7faa600cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6619c700 1 --2- 172.21.15.29:0/399269993 >> v2:172.21.15.144:3300/0 conn(0x7faa600ad370 0x7faa600cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6619c700 1 -- 172.21.15.29:0/399269993 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600ad740 con 0x7faa601568c0 2023-10-07T10:44:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6599b700 1 --2- 172.21.15.29:0/399269993 >> v2:172.21.15.144:3300/0 conn(0x7faa600ad370 0x7faa600cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa6619c700 1 --2- 172.21.15.29:0/399269993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa60156d00 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7faa600685f0 tx=0x7faa58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ffd69ce8c8c2b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.767+0000 7faa64999700 1 -- 172.21.15.29:0/399269993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5800e070 con 0x7faa601568c0 2023-10-07T10:44:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa64999700 1 -- 172.21.15.29:0/399269993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5800ba50 con 0x7faa601568c0 2023-10-07T10:44:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa64999700 1 -- 172.21.15.29:0/399269993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa580092e0 con 0x7faa601568c0 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 -- 172.21.15.29:0/399269993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 msgr2=0x7faa60156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 --2- 172.21.15.29:0/399269993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa60156d00 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7faa600685f0 tx=0x7faa58009840 comp rx=0 tx=0).stop 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 -- 172.21.15.29:0/399269993 shutdown_connections 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 --2- 172.21.15.29:0/399269993 >> v2:172.21.15.144:3300/0 conn(0x7faa600ad370 0x7faa600cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 --2- 172.21.15.29:0/399269993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa60156d00 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 --2- 172.21.15.29:0/399269993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 -- 172.21.15.29:0/399269993 >> 172.21.15.29:0/399269993 conn(0x7faa600680a0 msgr2=0x7faa600cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 -- 172.21.15.29:0/399269993 shutdown_connections 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.768+0000 7faa67bff700 1 -- 172.21.15.29:0/399269993 wait complete. 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.770+0000 7faa67bff700 1 Processor -- start 2023-10-07T10:44:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.770+0000 7faa67bff700 1 -- start start 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa67bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa600ad370 0x7faa6014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa67bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa600c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa67bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6006e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa67bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6014d2b0 con 0x7faa601568c0 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa67bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa600ca720 con 0x7faa600ad370 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa67bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa600c23a0 con 0x7faa60159ec0 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa600c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa600c1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38820/0 (socket says 172.21.15.29:38820) 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 -- 172.21.15.29:0/4183230165 learned_addr learned my addr 172.21.15.29:0/4183230165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6619c700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6006e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6599b700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa600ad370 0x7faa6014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 -- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 msgr2=0x7faa6006e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6006e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 -- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa600ad370 msgr2=0x7faa6014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa600ad370 0x7faa6014bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6519a700 1 -- 172.21.15.29:0/4183230165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa58009ae0 con 0x7faa601568c0 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6599b700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa600ad370 0x7faa6014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.771+0000 7faa6619c700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6006e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.772+0000 7faa6519a700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa600c1e60 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7faa5000d860 tx=0x7faa50008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.772+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50009610 con 0x7faa601568c0 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.772+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa50009770 con 0x7faa601568c0 2023-10-07T10:44:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.772+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50009b30 con 0x7faa601568c0 2023-10-07T10:44:45.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.772+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa600c28e0 con 0x7faa601568c0 2023-10-07T10:44:45.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.772+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa600ca0d0 con 0x7faa601568c0 2023-10-07T10:44:45.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.773+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa600ca8d0 con 0x7faa601568c0 2023-10-07T10:44:45.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.774+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faa5001e460 con 0x7faa601568c0 2023-10-07T10:44:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.775+0000 7faa55ffb700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faa44066f30 0x7faa440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.775+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7faa50084d60 con 0x7faa601568c0 2023-10-07T10:44:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.775+0000 7faa6599b700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faa44066f30 0x7faa440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.776+0000 7faa6599b700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faa44066f30 0x7faa440693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa5c006f00 tx=0x7faa5c008040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:45.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.779+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa600ca8d0 con 0x7faa601568c0 2023-10-07T10:44:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.979+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa600ca890 con 0x7faa44066f30 2023-10-07T10:44:45.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.988+0000 7faa55ffb700 1 -- 172.21.15.29:0/4183230165 <== mgr.7310 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289194 (secure 0 0 0) 0x7faa600ca890 con 0x7faa44066f30 2023-10-07T10:44:45.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faa44066f30 msgr2=0x7faa440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faa44066f30 0x7faa440693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa5c006f00 tx=0x7faa5c008040 comp rx=0 tx=0).stop 2023-10-07T10:44:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 msgr2=0x7faa600c1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa600c1e60 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7faa5000d860 tx=0x7faa50008040 comp rx=0 tx=0).stop 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 shutdown_connections 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faa44066f30 0x7faa440693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa60159ec0 0x7faa6006e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa600ad370 0x7faa6014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 --2- 172.21.15.29:0/4183230165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa601568c0 0x7faa600c1e60 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 >> 172.21.15.29:0/4183230165 conn(0x7faa600680a0 msgr2=0x7faa600ac810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 shutdown_connections 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:45.992+0000 7faa67bff700 1 -- 172.21.15.29:0/4183230165 wait complete. 2023-10-07T10:44:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T10:44:46.010 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T10:44:45.446410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467868,"kb_used_data":15836,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752506852,"statfs":{"total":773094113280,"available":770567016448,"internally_reserved":0,"allocated":16216064,"data_stored":8497512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001414"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154896+0000","last_change":"2023-10-07T10:44:41.146486+0000","last_active":"2023-10-07T10:44:41.154896+0000","last_peered":"2023-10-07T10:44:41.154896+0000","last_clean":"2023-10-07T10:44:41.154896+0000","last_became_active":"2023-10-07T10:44:41.146225+0000","last_became_peered":"2023-10-07T10:44:41.146225+0000","last_unstale":"2023-10-07T10:44:41.154896+0000","last_undegraded":"2023-10-07T10:44:41.154896+0000","last_fullsized":"2023-10-07T10:44:41.154896+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:35:14.213371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145049+0000","last_change":"2023-10-07T10:44:41.141108+0000","last_active":"2023-10-07T10:44:41.145049+0000","last_peered":"2023-10-07T10:44:41.145049+0000","last_clean":"2023-10-07T10:44:41.145049+0000","last_became_active":"2023-10-07T10:44:41.140500+0000","last_became_peered":"2023-10-07T10:44:41.140500+0000","last_unstale":"2023-10-07T10:44:41.145049+0000","last_undegraded":"2023-10-07T10:44:41.145049+0000","last_fullsized":"2023-10-07T10:44:41.145049+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:29:32.978953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154229+0000","last_change":"2023-10-07T10:44:41.151114+0000","last_active":"2023-10-07T10:44:41.154229+0000","last_peered":"2023-10-07T10:44:41.154229+0000","last_clean":"2023-10-07T10:44:41.154229+0000","last_became_active":"2023-10-07T10:44:41.150860+0000","last_became_peered":"2023-10-07T10:44:41.150860+0000","last_unstale":"2023-10-07T10:44:41.154229+0000","last_undegraded":"2023-10-07T10:44:41.154229+0000","last_fullsized":"2023-10-07T10:44:41.154229+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:31:08.272160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154045+0000","last_change":"2023-10-07T10:44:41.143913+0000","last_active":"2023-10-07T10:44:41.154045+0000","last_peered":"2023-10-07T10:44:41.154045+0000","last_clean":"2023-10-07T10:44:41.154045+0000","last_became_active":"2023-10-07T10:44:41.143651+0000","last_became_peered":"2023-10-07T10:44:41.143651+0000","last_unstale":"2023-10-07T10:44:41.154045+0000","last_undegraded":"2023-10-07T10:44:41.154045+0000","last_fullsized":"2023-10-07T10:44:41.154045+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:50.192545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.151499+0000","last_change":"2023-10-07T10:44:41.150156+0000","last_active":"2023-10-07T10:44:41.151499+0000","last_peered":"2023-10-07T10:44:41.151499+0000","last_clean":"2023-10-07T10:44:41.151499+0000","last_became_active":"2023-10-07T10:44:41.149644+0000","last_became_peered":"2023-10-07T10:44:41.149644+0000","last_unstale":"2023-10-07T10:44:41.151499+0000","last_undegraded":"2023-10-07T10:44:41.151499+0000","last_fullsized":"2023-10-07T10:44:41.151499+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:03:33.745656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142679+0000","last_change":"2023-10-07T10:44:41.139741+0000","last_active":"2023-10-07T10:44:41.142679+0000","last_peered":"2023-10-07T10:44:41.142679+0000","last_clean":"2023-10-07T10:44:41.142679+0000","last_became_active":"2023-10-07T10:44:41.139467+0000","last_became_peered":"2023-10-07T10:44:41.139467+0000","last_unstale":"2023-10-07T10:44:41.142679+0000","last_undegraded":"2023-10-07T10:44:41.142679+0000","last_fullsized":"2023-10-07T10:44:41.142679+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:31:08.272160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143501+0000","last_change":"2023-10-07T10:44:41.141074+0000","last_active":"2023-10-07T10:44:41.143501+0000","last_peered":"2023-10-07T10:44:41.143501+0000","last_clean":"2023-10-07T10:44:41.143501+0000","last_became_active":"2023-10-07T10:44:41.140780+0000","last_became_peered":"2023-10-07T10:44:41.140780+0000","last_unstale":"2023-10-07T10:44:41.143501+0000","last_undegraded":"2023-10-07T10:44:41.143501+0000","last_fullsized":"2023-10-07T10:44:41.143501+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:26.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.144865+0000","last_change":"2023-10-07T10:44:41.140044+0000","last_active":"2023-10-07T10:44:41.144865+0000","last_peered":"2023-10-07T10:44:41.144865+0000","last_clean":"2023-10-07T10:44:41.144865+0000","last_became_active":"2023-10-07T10:44:41.139460+0000","last_became_peered":"2023-10-07T10:44:41.139460+0000","last_unstale":"2023-10-07T10:44:41.144865+0000","last_undegraded":"2023-10-07T10:44:41.144865+0000","last_fullsized":"2023-10-07T10:44:41.144865+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:12:23.908691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154122+0000","last_change":"2023-10-07T10:44:41.150391+0000","last_active":"2023-10-07T10:44:41.154122+0000","last_peered":"2023-10-07T10:44:41.154122+0000","last_clean":"2023-10-07T10:44:41.154122+0000","last_became_active":"2023-10-07T10:44:41.150133+0000","last_became_peered":"2023-10-07T10:44:41.150133+0000","last_unstale":"2023-10-07T10:44:41.154122+0000","last_undegraded":"2023-10-07T10:44:41.154122+0000","last_fullsized":"2023-10-07T10:44:41.154122+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:12:49.484952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145118+0000","last_change":"2023-10-07T10:44:41.141287+0000","last_active":"2023-10-07T10:44:41.145118+0000","last_peered":"2023-10-07T10:44:41.145118+0000","last_clean":"2023-10-07T10:44:41.145118+0000","last_became_active":"2023-10-07T10:44:41.140850+0000","last_became_peered":"2023-10-07T10:44:41.140850+0000","last_unstale":"2023-10-07T10:44:41.145118+0000","last_undegraded":"2023-10-07T10:44:41.145118+0000","last_fullsized":"2023-10-07T10:44:41.145118+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:44:35.721020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150006+0000","last_change":"2023-10-07T10:44:41.148120+0000","last_active":"2023-10-07T10:44:41.150006+0000","last_peered":"2023-10-07T10:44:41.150006+0000","last_clean":"2023-10-07T10:44:41.150006+0000","last_became_active":"2023-10-07T10:44:41.147825+0000","last_became_peered":"2023-10-07T10:44:41.147825+0000","last_unstale":"2023-10-07T10:44:41.150006+0000","last_undegraded":"2023-10-07T10:44:41.150006+0000","last_fullsized":"2023-10-07T10:44:41.150006+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:32:12.770853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153595+0000","last_change":"2023-10-07T10:44:41.146499+0000","last_active":"2023-10-07T10:44:41.153595+0000","last_peered":"2023-10-07T10:44:41.153595+0000","last_clean":"2023-10-07T10:44:41.153595+0000","last_became_active":"2023-10-07T10:44:41.146218+0000","last_became_peered":"2023-10-07T10:44:41.146218+0000","last_unstale":"2023-10-07T10:44:41.153595+0000","last_undegraded":"2023-10-07T10:44:41.153595+0000","last_fullsized":"2023-10-07T10:44:41.153595+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:40:20.347596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154444+0000","last_change":"2023-10-07T10:44:41.140573+0000","last_active":"2023-10-07T10:44:41.154444+0000","last_peered":"2023-10-07T10:44:41.154444+0000","last_clean":"2023-10-07T10:44:41.154444+0000","last_became_active":"2023-10-07T10:44:41.140218+0000","last_became_peered":"2023-10-07T10:44:41.140218+0000","last_unstale":"2023-10-07T10:44:41.154444+0000","last_undegraded":"2023-10-07T10:44:41.154444+0000","last_fullsized":"2023-10-07T10:44:41.154444+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:19.016810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149647+0000","last_change":"2023-10-07T10:44:41.145649+0000","last_active":"2023-10-07T10:44:41.149647+0000","last_peered":"2023-10-07T10:44:41.149647+0000","last_clean":"2023-10-07T10:44:41.149647+0000","last_became_active":"2023-10-07T10:44:41.145362+0000","last_became_peered":"2023-10-07T10:44:41.145362+0000","last_unstale":"2023-10-07T10:44:41.149647+0000","last_undegraded":"2023-10-07T10:44:41.149647+0000","last_fullsized":"2023-10-07T10:44:41.149647+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:35:55.938916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154674+0000","last_change":"2023-10-07T10:44:41.149774+0000","last_active":"2023-10-07T10:44:41.154674+0000","last_peered":"2023-10-07T10:44:41.154674+0000","last_clean":"2023-10-07T10:44:41.154674+0000","last_became_active":"2023-10-07T10:44:41.147509+0000","last_became_peered":"2023-10-07T10:44:41.147509+0000","last_unstale":"2023-10-07T10:44:41.154674+0000","last_undegraded":"2023-10-07T10:44:41.154674+0000","last_fullsized":"2023-10-07T10:44:41.154674+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:32:25.508518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142944+0000","last_change":"2023-10-07T10:44:41.138658+0000","last_active":"2023-10-07T10:44:41.142944+0000","last_peered":"2023-10-07T10:44:41.142944+0000","last_clean":"2023-10-07T10:44:41.142944+0000","last_became_active":"2023-10-07T10:44:41.138365+0000","last_became_peered":"2023-10-07T10:44:41.138365+0000","last_unstale":"2023-10-07T10:44:41.142944+0000","last_undegraded":"2023-10-07T10:44:41.142944+0000","last_fullsized":"2023-10-07T10:44:41.142944+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:23:49.101932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152762+0000","last_change":"2023-10-07T10:44:41.151589+0000","last_active":"2023-10-07T10:44:41.152762+0000","last_peered":"2023-10-07T10:44:41.152762+0000","last_clean":"2023-10-07T10:44:41.152762+0000","last_became_active":"2023-10-07T10:44:41.151135+0000","last_became_peered":"2023-10-07T10:44:41.151135+0000","last_unstale":"2023-10-07T10:44:41.152762+0000","last_undegraded":"2023-10-07T10:44:41.152762+0000","last_fullsized":"2023-10-07T10:44:41.152762+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:51:56.094466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143627+0000","last_change":"2023-10-07T10:44:41.139532+0000","last_active":"2023-10-07T10:44:41.143627+0000","last_peered":"2023-10-07T10:44:41.143627+0000","last_clean":"2023-10-07T10:44:41.143627+0000","last_became_active":"2023-10-07T10:44:41.139178+0000","last_became_peered":"2023-10-07T10:44:41.139178+0000","last_unstale":"2023-10-07T10:44:41.143627+0000","last_undegraded":"2023-10-07T10:44:41.143627+0000","last_fullsized":"2023-10-07T10:44:41.143627+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:48:31.051226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150023+0000","last_change":"2023-10-07T10:44:41.148162+0000","last_active":"2023-10-07T10:44:41.150023+0000","last_peered":"2023-10-07T10:44:41.150023+0000","last_clean":"2023-10-07T10:44:41.150023+0000","last_became_active":"2023-10-07T10:44:41.147848+0000","last_became_peered":"2023-10-07T10:44:41.147848+0000","last_unstale":"2023-10-07T10:44:41.150023+0000","last_undegraded":"2023-10-07T10:44:41.150023+0000","last_fullsized":"2023-10-07T10:44:41.150023+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:04.046119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142014+0000","last_change":"2023-10-07T10:44:41.135108+0000","last_active":"2023-10-07T10:44:41.142014+0000","last_peered":"2023-10-07T10:44:41.142014+0000","last_clean":"2023-10-07T10:44:41.142014+0000","last_became_active":"2023-10-07T10:44:41.134849+0000","last_became_peered":"2023-10-07T10:44:41.134849+0000","last_unstale":"2023-10-07T10:44:41.142014+0000","last_undegraded":"2023-10-07T10:44:41.142014+0000","last_fullsized":"2023-10-07T10:44:41.142014+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:17.616542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.151518+0000","last_change":"2023-10-07T10:44:41.150428+0000","last_active":"2023-10-07T10:44:41.151518+0000","last_peered":"2023-10-07T10:44:41.151518+0000","last_clean":"2023-10-07T10:44:41.151518+0000","last_became_active":"2023-10-07T10:44:41.149982+0000","last_became_peered":"2023-10-07T10:44:41.149982+0000","last_unstale":"2023-10-07T10:44:41.151518+0000","last_undegraded":"2023-10-07T10:44:41.151518+0000","last_fullsized":"2023-10-07T10:44:41.151518+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:58.711846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.151958+0000","last_change":"2023-10-07T10:44:41.148227+0000","last_active":"2023-10-07T10:44:41.151958+0000","last_peered":"2023-10-07T10:44:41.151958+0000","last_clean":"2023-10-07T10:44:41.151958+0000","last_became_active":"2023-10-07T10:44:41.147847+0000","last_became_peered":"2023-10-07T10:44:41.147847+0000","last_unstale":"2023-10-07T10:44:41.151958+0000","last_undegraded":"2023-10-07T10:44:41.151958+0000","last_fullsized":"2023-10-07T10:44:41.151958+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:27:50.412472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154133+0000","last_change":"2023-10-07T10:44:41.149817+0000","last_active":"2023-10-07T10:44:41.154133+0000","last_peered":"2023-10-07T10:44:41.154133+0000","last_clean":"2023-10-07T10:44:41.154133+0000","last_became_active":"2023-10-07T10:44:41.149541+0000","last_became_peered":"2023-10-07T10:44:41.149541+0000","last_unstale":"2023-10-07T10:44:41.154133+0000","last_undegraded":"2023-10-07T10:44:41.154133+0000","last_fullsized":"2023-10-07T10:44:41.154133+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:21.610466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.155461+0000","last_change":"2023-10-07T10:44:41.150258+0000","last_active":"2023-10-07T10:44:41.155461+0000","last_peered":"2023-10-07T10:44:41.155461+0000","last_clean":"2023-10-07T10:44:41.155461+0000","last_became_active":"2023-10-07T10:44:41.150015+0000","last_became_peered":"2023-10-07T10:44:41.150015+0000","last_unstale":"2023-10-07T10:44:41.155461+0000","last_undegraded":"2023-10-07T10:44:41.155461+0000","last_fullsized":"2023-10-07T10:44:41.155461+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:25:19.904126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149617+0000","last_change":"2023-10-07T10:44:41.146555+0000","last_active":"2023-10-07T10:44:41.149617+0000","last_peered":"2023-10-07T10:44:41.149617+0000","last_clean":"2023-10-07T10:44:41.149617+0000","last_became_active":"2023-10-07T10:44:41.146287+0000","last_became_peered":"2023-10-07T10:44:41.146287+0000","last_unstale":"2023-10-07T10:44:41.149617+0000","last_undegraded":"2023-10-07T10:44:41.149617+0000","last_fullsized":"2023-10-07T10:44:41.149617+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:18:10.617203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143609+0000","last_change":"2023-10-07T10:44:41.138871+0000","last_active":"2023-10-07T10:44:41.143609+0000","last_peered":"2023-10-07T10:44:41.143609+0000","last_clean":"2023-10-07T10:44:41.143609+0000","last_became_active":"2023-10-07T10:44:41.138536+0000","last_became_peered":"2023-10-07T10:44:41.138536+0000","last_unstale":"2023-10-07T10:44:41.143609+0000","last_undegraded":"2023-10-07T10:44:41.143609+0000","last_fullsized":"2023-10-07T10:44:41.143609+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:27:54.800429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153009+0000","last_change":"2023-10-07T10:44:41.140208+0000","last_active":"2023-10-07T10:44:41.153009+0000","last_peered":"2023-10-07T10:44:41.153009+0000","last_clean":"2023-10-07T10:44:41.153009+0000","last_became_active":"2023-10-07T10:44:41.139942+0000","last_became_peered":"2023-10-07T10:44:41.139942+0000","last_unstale":"2023-10-07T10:44:41.153009+0000","last_undegraded":"2023-10-07T10:44:41.153009+0000","last_fullsized":"2023-10-07T10:44:41.153009+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:36:55.696997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147285+0000","last_change":"2023-10-07T10:44:41.139644+0000","last_active":"2023-10-07T10:44:41.147285+0000","last_peered":"2023-10-07T10:44:41.147285+0000","last_clean":"2023-10-07T10:44:41.147285+0000","last_became_active":"2023-10-07T10:44:41.139371+0000","last_became_peered":"2023-10-07T10:44:41.139371+0000","last_unstale":"2023-10-07T10:44:41.147285+0000","last_undegraded":"2023-10-07T10:44:41.147285+0000","last_fullsized":"2023-10-07T10:44:41.147285+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:20:42.569790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154471+0000","last_change":"2023-10-07T10:44:41.147049+0000","last_active":"2023-10-07T10:44:41.154471+0000","last_peered":"2023-10-07T10:44:41.154471+0000","last_clean":"2023-10-07T10:44:41.154471+0000","last_became_active":"2023-10-07T10:44:41.146797+0000","last_became_peered":"2023-10-07T10:44:41.146797+0000","last_unstale":"2023-10-07T10:44:41.154471+0000","last_undegraded":"2023-10-07T10:44:41.154471+0000","last_fullsized":"2023-10-07T10:44:41.154471+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:01.539906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.155161+0000","last_change":"2023-10-07T10:44:41.148618+0000","last_active":"2023-10-07T10:44:41.155161+0000","last_peered":"2023-10-07T10:44:41.155161+0000","last_clean":"2023-10-07T10:44:41.155161+0000","last_became_active":"2023-10-07T10:44:41.147234+0000","last_became_peered":"2023-10-07T10:44:41.147234+0000","last_unstale":"2023-10-07T10:44:41.155161+0000","last_undegraded":"2023-10-07T10:44:41.155161+0000","last_fullsized":"2023-10-07T10:44:41.155161+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:05:07.923201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149074+0000","last_change":"2023-10-07T10:44:41.145520+0000","last_active":"2023-10-07T10:44:41.149074+0000","last_peered":"2023-10-07T10:44:41.149074+0000","last_clean":"2023-10-07T10:44:41.149074+0000","last_became_active":"2023-10-07T10:44:41.145049+0000","last_became_peered":"2023-10-07T10:44:41.145049+0000","last_unstale":"2023-10-07T10:44:41.149074+0000","last_undegraded":"2023-10-07T10:44:41.149074+0000","last_fullsized":"2023-10-07T10:44:41.149074+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:31:03.014537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.140978+0000","last_change":"2023-10-07T10:44:41.133265+0000","last_active":"2023-10-07T10:44:41.140978+0000","last_peered":"2023-10-07T10:44:41.140978+0000","last_clean":"2023-10-07T10:44:41.140978+0000","last_became_active":"2023-10-07T10:44:41.132962+0000","last_became_peered":"2023-10-07T10:44:41.132962+0000","last_unstale":"2023-10-07T10:44:41.140978+0000","last_undegraded":"2023-10-07T10:44:41.140978+0000","last_fullsized":"2023-10-07T10:44:41.140978+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:12:49.484952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154258+0000","last_change":"2023-10-07T10:44:41.148394+0000","last_active":"2023-10-07T10:44:41.154258+0000","last_peered":"2023-10-07T10:44:41.154258+0000","last_clean":"2023-10-07T10:44:41.154258+0000","last_became_active":"2023-10-07T10:44:41.148118+0000","last_became_peered":"2023-10-07T10:44:41.148118+0000","last_unstale":"2023-10-07T10:44:41.154258+0000","last_undegraded":"2023-10-07T10:44:41.154258+0000","last_fullsized":"2023-10-07T10:44:41.154258+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:50:11.176162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"150'4","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.157069+0000","last_change":"2023-10-07T10:44:40.141891+0000","last_active":"2023-10-07T10:44:41.157069+0000","last_peered":"2023-10-07T10:44:41.157069+0000","last_clean":"2023-10-07T10:44:41.157069+0000","last_became_active":"2023-10-07T10:44:40.141601+0000","last_became_peered":"2023-10-07T10:44:40.141601+0000","last_unstale":"2023-10-07T10:44:41.157069+0000","last_undegraded":"2023-10-07T10:44:41.157069+0000","last_fullsized":"2023-10-07T10:44:41.157069+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:34:21.972708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153876+0000","last_change":"2023-10-07T10:44:41.147988+0000","last_active":"2023-10-07T10:44:41.153876+0000","last_peered":"2023-10-07T10:44:41.153876+0000","last_clean":"2023-10-07T10:44:41.153876+0000","last_became_active":"2023-10-07T10:44:41.147740+0000","last_became_peered":"2023-10-07T10:44:41.147740+0000","last_unstale":"2023-10-07T10:44:41.153876+0000","last_undegraded":"2023-10-07T10:44:41.153876+0000","last_fullsized":"2023-10-07T10:44:41.153876+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:57:26.369780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150171+0000","last_change":"2023-10-07T10:44:40.138233+0000","last_active":"2023-10-07T10:44:41.150171+0000","last_peered":"2023-10-07T10:44:41.150171+0000","last_clean":"2023-10-07T10:44:41.150171+0000","last_became_active":"2023-10-07T10:44:40.137964+0000","last_became_peered":"2023-10-07T10:44:40.137964+0000","last_unstale":"2023-10-07T10:44:41.150171+0000","last_undegraded":"2023-10-07T10:44:41.150171+0000","last_fullsized":"2023-10-07T10:44:41.150171+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:00.532197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148978+0000","last_change":"2023-10-07T10:44:41.140918+0000","last_active":"2023-10-07T10:44:41.148978+0000","last_peered":"2023-10-07T10:44:41.148978+0000","last_clean":"2023-10-07T10:44:41.148978+0000","last_became_active":"2023-10-07T10:44:41.139574+0000","last_became_peered":"2023-10-07T10:44:41.139574+0000","last_unstale":"2023-10-07T10:44:41.148978+0000","last_undegraded":"2023-10-07T10:44:41.148978+0000","last_fullsized":"2023-10-07T10:44:41.148978+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:19.840943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148696+0000","last_change":"2023-10-07T10:44:40.132372+0000","last_active":"2023-10-07T10:44:41.148696+0000","last_peered":"2023-10-07T10:44:41.148696+0000","last_clean":"2023-10-07T10:44:41.148696+0000","last_became_active":"2023-10-07T10:44:40.132097+0000","last_became_peered":"2023-10-07T10:44:40.132097+0000","last_unstale":"2023-10-07T10:44:41.148696+0000","last_undegraded":"2023-10-07T10:44:41.148696+0000","last_fullsized":"2023-10-07T10:44:41.148696+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:03.386461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143898+0000","last_change":"2023-10-07T10:44:41.137096+0000","last_active":"2023-10-07T10:44:41.143898+0000","last_peered":"2023-10-07T10:44:41.143898+0000","last_clean":"2023-10-07T10:44:41.143898+0000","last_became_active":"2023-10-07T10:44:41.136825+0000","last_became_peered":"2023-10-07T10:44:41.136825+0000","last_unstale":"2023-10-07T10:44:41.143898+0000","last_undegraded":"2023-10-07T10:44:41.143898+0000","last_fullsized":"2023-10-07T10:44:41.143898+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:36.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152456+0000","last_change":"2023-10-07T10:44:40.129668+0000","last_active":"2023-10-07T10:44:41.152456+0000","last_peered":"2023-10-07T10:44:41.152456+0000","last_clean":"2023-10-07T10:44:41.152456+0000","last_became_active":"2023-10-07T10:44:40.129391+0000","last_became_peered":"2023-10-07T10:44:40.129391+0000","last_unstale":"2023-10-07T10:44:41.152456+0000","last_undegraded":"2023-10-07T10:44:41.152456+0000","last_fullsized":"2023-10-07T10:44:41.152456+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:32:24.500809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142866+0000","last_change":"2023-10-07T10:44:41.140180+0000","last_active":"2023-10-07T10:44:41.142866+0000","last_peered":"2023-10-07T10:44:41.142866+0000","last_clean":"2023-10-07T10:44:41.142866+0000","last_became_active":"2023-10-07T10:44:41.139900+0000","last_became_peered":"2023-10-07T10:44:41.139900+0000","last_unstale":"2023-10-07T10:44:41.142866+0000","last_undegraded":"2023-10-07T10:44:41.142866+0000","last_fullsized":"2023-10-07T10:44:41.142866+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:04.394170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.146711+0000","last_change":"2023-10-07T10:44:40.129304+0000","last_active":"2023-10-07T10:44:41.146711+0000","last_peered":"2023-10-07T10:44:41.146711+0000","last_clean":"2023-10-07T10:44:41.146711+0000","last_became_active":"2023-10-07T10:44:40.129005+0000","last_became_peered":"2023-10-07T10:44:40.129005+0000","last_unstale":"2023-10-07T10:44:41.146711+0000","last_undegraded":"2023-10-07T10:44:41.146711+0000","last_fullsized":"2023-10-07T10:44:41.146711+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:35:13.205662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149500+0000","last_change":"2023-10-07T10:44:41.148562+0000","last_active":"2023-10-07T10:44:41.149500+0000","last_peered":"2023-10-07T10:44:41.149500+0000","last_clean":"2023-10-07T10:44:41.149500+0000","last_became_active":"2023-10-07T10:44:41.148258+0000","last_became_peered":"2023-10-07T10:44:41.148258+0000","last_unstale":"2023-10-07T10:44:41.149500+0000","last_undegraded":"2023-10-07T10:44:41.149500+0000","last_fullsized":"2023-10-07T10:44:41.149500+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:12:23.908691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.143943+0000","last_change":"2023-10-07T10:44:40.129052+0000","last_active":"2023-10-07T10:44:41.143943+0000","last_peered":"2023-10-07T10:44:41.143943+0000","last_clean":"2023-10-07T10:44:41.143943+0000","last_became_active":"2023-10-07T10:44:40.128796+0000","last_became_peered":"2023-10-07T10:44:40.128796+0000","last_unstale":"2023-10-07T10:44:41.143943+0000","last_undegraded":"2023-10-07T10:44:41.143943+0000","last_fullsized":"2023-10-07T10:44:41.143943+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:46:23.804562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.144885+0000","last_change":"2023-10-07T10:44:41.140223+0000","last_active":"2023-10-07T10:44:41.144885+0000","last_peered":"2023-10-07T10:44:41.144885+0000","last_clean":"2023-10-07T10:44:41.144885+0000","last_became_active":"2023-10-07T10:44:41.139752+0000","last_became_peered":"2023-10-07T10:44:41.139752+0000","last_unstale":"2023-10-07T10:44:41.144885+0000","last_undegraded":"2023-10-07T10:44:41.144885+0000","last_fullsized":"2023-10-07T10:44:41.144885+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:17.616542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.121445+0000","last_change":"2023-10-07T10:44:40.140008+0000","last_active":"2023-10-07T10:44:41.121445+0000","last_peered":"2023-10-07T10:44:41.121445+0000","last_clean":"2023-10-07T10:44:41.121445+0000","last_became_active":"2023-10-07T10:44:40.139723+0000","last_became_peered":"2023-10-07T10:44:40.139723+0000","last_unstale":"2023-10-07T10:44:41.121445+0000","last_undegraded":"2023-10-07T10:44:41.121445+0000","last_fullsized":"2023-10-07T10:44:41.121445+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:55:31.279084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148258+0000","last_change":"2023-10-07T10:44:41.144814+0000","last_active":"2023-10-07T10:44:41.148258+0000","last_peered":"2023-10-07T10:44:41.148258+0000","last_clean":"2023-10-07T10:44:41.148258+0000","last_became_active":"2023-10-07T10:44:41.144513+0000","last_became_peered":"2023-10-07T10:44:41.144513+0000","last_unstale":"2023-10-07T10:44:41.148258+0000","last_undegraded":"2023-10-07T10:44:41.148258+0000","last_fullsized":"2023-10-07T10:44:41.148258+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:53:59.503087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.131318+0000","last_change":"2023-10-07T10:44:40.139058+0000","last_active":"2023-10-07T10:44:41.131318+0000","last_peered":"2023-10-07T10:44:41.131318+0000","last_clean":"2023-10-07T10:44:41.131318+0000","last_became_active":"2023-10-07T10:44:40.138797+0000","last_became_peered":"2023-10-07T10:44:40.138797+0000","last_unstale":"2023-10-07T10:44:41.131318+0000","last_undegraded":"2023-10-07T10:44:41.131318+0000","last_fullsized":"2023-10-07T10:44:41.131318+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:48:00.532197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154785+0000","last_change":"2023-10-07T10:44:41.150501+0000","last_active":"2023-10-07T10:44:41.154785+0000","last_peered":"2023-10-07T10:44:41.154785+0000","last_clean":"2023-10-07T10:44:41.154785+0000","last_became_active":"2023-10-07T10:44:41.150251+0000","last_became_peered":"2023-10-07T10:44:41.150251+0000","last_unstale":"2023-10-07T10:44:41.154785+0000","last_undegraded":"2023-10-07T10:44:41.154785+0000","last_fullsized":"2023-10-07T10:44:41.154785+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:17:14.592996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122370+0000","last_change":"2023-10-07T10:44:40.140996+0000","last_active":"2023-10-07T10:44:41.122370+0000","last_peered":"2023-10-07T10:44:41.122370+0000","last_clean":"2023-10-07T10:44:41.122370+0000","last_became_active":"2023-10-07T10:44:40.139010+0000","last_became_peered":"2023-10-07T10:44:40.139010+0000","last_unstale":"2023-10-07T10:44:41.122370+0000","last_undegraded":"2023-10-07T10:44:41.122370+0000","last_fullsized":"2023-10-07T10:44:41.122370+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:18:44.321434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147388+0000","last_change":"2023-10-07T10:44:41.140563+0000","last_active":"2023-10-07T10:44:41.147388+0000","last_peered":"2023-10-07T10:44:41.147388+0000","last_clean":"2023-10-07T10:44:41.147388+0000","last_became_active":"2023-10-07T10:44:41.140301+0000","last_became_peered":"2023-10-07T10:44:41.140301+0000","last_unstale":"2023-10-07T10:44:41.147388+0000","last_undegraded":"2023-10-07T10:44:41.147388+0000","last_fullsized":"2023-10-07T10:44:41.147388+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:36.231006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122967+0000","last_change":"2023-10-07T10:44:40.141545+0000","last_active":"2023-10-07T10:44:41.122967+0000","last_peered":"2023-10-07T10:44:41.122967+0000","last_clean":"2023-10-07T10:44:41.122967+0000","last_became_active":"2023-10-07T10:44:40.141243+0000","last_became_peered":"2023-10-07T10:44:40.141243+0000","last_unstale":"2023-10-07T10:44:41.122967+0000","last_undegraded":"2023-10-07T10:44:41.122967+0000","last_fullsized":"2023-10-07T10:44:41.122967+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:36:57.908957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149062+0000","last_change":"2023-10-07T10:44:41.145151+0000","last_active":"2023-10-07T10:44:41.149062+0000","last_peered":"2023-10-07T10:44:41.149062+0000","last_clean":"2023-10-07T10:44:41.149062+0000","last_became_active":"2023-10-07T10:44:41.144863+0000","last_became_peered":"2023-10-07T10:44:41.144863+0000","last_unstale":"2023-10-07T10:44:41.149062+0000","last_undegraded":"2023-10-07T10:44:41.149062+0000","last_fullsized":"2023-10-07T10:44:41.149062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:54:09.510578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"150'3","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147846+0000","last_change":"2023-10-07T10:44:40.135545+0000","last_active":"2023-10-07T10:44:41.147846+0000","last_peered":"2023-10-07T10:44:41.147846+0000","last_clean":"2023-10-07T10:44:41.147846+0000","last_became_active":"2023-10-07T10:44:40.135252+0000","last_became_peered":"2023-10-07T10:44:40.135252+0000","last_unstale":"2023-10-07T10:44:41.147846+0000","last_undegraded":"2023-10-07T10:44:41.147846+0000","last_fullsized":"2023-10-07T10:44:41.147846+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:18.009101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149102+0000","last_change":"2023-10-07T10:44:41.144367+0000","last_active":"2023-10-07T10:44:41.149102+0000","last_peered":"2023-10-07T10:44:41.149102+0000","last_clean":"2023-10-07T10:44:41.149102+0000","last_became_active":"2023-10-07T10:44:41.144072+0000","last_became_peered":"2023-10-07T10:44:41.144072+0000","last_unstale":"2023-10-07T10:44:41.149102+0000","last_undegraded":"2023-10-07T10:44:41.149102+0000","last_fullsized":"2023-10-07T10:44:41.149102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:47:25.419244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"150'6","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154686+0000","last_change":"2023-10-07T10:44:40.142649+0000","last_active":"2023-10-07T10:44:41.154686+0000","last_peered":"2023-10-07T10:44:41.154686+0000","last_clean":"2023-10-07T10:44:41.154686+0000","last_became_active":"2023-10-07T10:44:40.142336+0000","last_became_peered":"2023-10-07T10:44:40.142336+0000","last_unstale":"2023-10-07T10:44:41.154686+0000","last_undegraded":"2023-10-07T10:44:41.154686+0000","last_fullsized":"2023-10-07T10:44:41.154686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:43:43.376117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154849+0000","last_change":"2023-10-07T10:44:41.149453+0000","last_active":"2023-10-07T10:44:41.154849+0000","last_peered":"2023-10-07T10:44:41.154849+0000","last_clean":"2023-10-07T10:44:41.154849+0000","last_became_active":"2023-10-07T10:44:41.149189+0000","last_became_peered":"2023-10-07T10:44:41.149189+0000","last_unstale":"2023-10-07T10:44:41.154849+0000","last_undegraded":"2023-10-07T10:44:41.154849+0000","last_fullsized":"2023-10-07T10:44:41.154849+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:29:16.993784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.126801+0000","last_change":"2023-10-07T10:44:40.137862+0000","last_active":"2023-10-07T10:44:41.126801+0000","last_peered":"2023-10-07T10:44:41.126801+0000","last_clean":"2023-10-07T10:44:41.126801+0000","last_became_active":"2023-10-07T10:44:40.137443+0000","last_became_peered":"2023-10-07T10:44:40.137443+0000","last_unstale":"2023-10-07T10:44:41.126801+0000","last_undegraded":"2023-10-07T10:44:41.126801+0000","last_fullsized":"2023-10-07T10:44:41.126801+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:47:27.687000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148230+0000","last_change":"2023-10-07T10:44:41.144640+0000","last_active":"2023-10-07T10:44:41.148230+0000","last_peered":"2023-10-07T10:44:41.148230+0000","last_clean":"2023-10-07T10:44:41.148230+0000","last_became_active":"2023-10-07T10:44:41.144366+0000","last_became_peered":"2023-10-07T10:44:41.144366+0000","last_unstale":"2023-10-07T10:44:41.148230+0000","last_undegraded":"2023-10-07T10:44:41.148230+0000","last_fullsized":"2023-10-07T10:44:41.148230+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:55:32.286792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.130941+0000","last_change":"2023-10-07T10:44:40.140151+0000","last_active":"2023-10-07T10:44:41.130941+0000","last_peered":"2023-10-07T10:44:41.130941+0000","last_clean":"2023-10-07T10:44:41.130941+0000","last_became_active":"2023-10-07T10:44:40.139824+0000","last_became_peered":"2023-10-07T10:44:40.139824+0000","last_unstale":"2023-10-07T10:44:41.130941+0000","last_undegraded":"2023-10-07T10:44:41.130941+0000","last_fullsized":"2023-10-07T10:44:41.130941+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:36:54.689289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153797+0000","last_change":"2023-10-07T10:44:41.147074+0000","last_active":"2023-10-07T10:44:41.153797+0000","last_peered":"2023-10-07T10:44:41.153797+0000","last_clean":"2023-10-07T10:44:41.153797+0000","last_became_active":"2023-10-07T10:44:41.146796+0000","last_became_peered":"2023-10-07T10:44:41.146796+0000","last_unstale":"2023-10-07T10:44:41.153797+0000","last_undegraded":"2023-10-07T10:44:41.153797+0000","last_fullsized":"2023-10-07T10:44:41.153797+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:36:58.916666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"150'8","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.144815+0000","last_change":"2023-10-07T10:44:40.137704+0000","last_active":"2023-10-07T10:44:41.144815+0000","last_peered":"2023-10-07T10:44:41.144815+0000","last_clean":"2023-10-07T10:44:41.144815+0000","last_became_active":"2023-10-07T10:44:40.137390+0000","last_became_peered":"2023-10-07T10:44:40.137390+0000","last_unstale":"2023-10-07T10:44:41.144815+0000","last_undegraded":"2023-10-07T10:44:41.144815+0000","last_fullsized":"2023-10-07T10:44:41.144815+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:08.592940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148532+0000","last_change":"2023-10-07T10:44:41.143114+0000","last_active":"2023-10-07T10:44:41.148532+0000","last_peered":"2023-10-07T10:44:41.148532+0000","last_clean":"2023-10-07T10:44:41.148532+0000","last_became_active":"2023-10-07T10:44:41.142629+0000","last_became_peered":"2023-10-07T10:44:41.142629+0000","last_unstale":"2023-10-07T10:44:41.148532+0000","last_undegraded":"2023-10-07T10:44:41.148532+0000","last_fullsized":"2023-10-07T10:44:41.148532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:26.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147273+0000","last_change":"2023-10-07T10:44:40.142936+0000","last_active":"2023-10-07T10:44:41.147273+0000","last_peered":"2023-10-07T10:44:41.147273+0000","last_clean":"2023-10-07T10:44:41.147273+0000","last_became_active":"2023-10-07T10:44:40.142623+0000","last_became_peered":"2023-10-07T10:44:40.142623+0000","last_unstale":"2023-10-07T10:44:41.147273+0000","last_undegraded":"2023-10-07T10:44:41.147273+0000","last_fullsized":"2023-10-07T10:44:41.147273+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:35:13.205662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.153104+0000","last_change":"2023-10-07T10:44:41.150568+0000","last_active":"2023-10-07T10:44:41.153104+0000","last_peered":"2023-10-07T10:44:41.153104+0000","last_clean":"2023-10-07T10:44:41.153104+0000","last_became_active":"2023-10-07T10:44:41.150087+0000","last_became_peered":"2023-10-07T10:44:41.150087+0000","last_unstale":"2023-10-07T10:44:41.153104+0000","last_undegraded":"2023-10-07T10:44:41.153104+0000","last_fullsized":"2023-10-07T10:44:41.153104+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:45:31.355758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149721+0000","last_change":"2023-10-07T10:44:40.138593+0000","last_active":"2023-10-07T10:44:41.149721+0000","last_peered":"2023-10-07T10:44:41.149721+0000","last_clean":"2023-10-07T10:44:41.149721+0000","last_became_active":"2023-10-07T10:44:40.138341+0000","last_became_peered":"2023-10-07T10:44:40.138341+0000","last_unstale":"2023-10-07T10:44:41.149721+0000","last_undegraded":"2023-10-07T10:44:41.149721+0000","last_fullsized":"2023-10-07T10:44:41.149721+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:31:07.264451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142776+0000","last_change":"2023-10-07T10:44:41.140201+0000","last_active":"2023-10-07T10:44:41.142776+0000","last_peered":"2023-10-07T10:44:41.142776+0000","last_clean":"2023-10-07T10:44:41.142776+0000","last_became_active":"2023-10-07T10:44:41.139849+0000","last_became_peered":"2023-10-07T10:44:41.139849+0000","last_unstale":"2023-10-07T10:44:41.142776+0000","last_undegraded":"2023-10-07T10:44:41.142776+0000","last_fullsized":"2023-10-07T10:44:41.142776+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:58:45.712787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"150'1","reported_seq":11,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142946+0000","last_change":"2023-10-07T10:44:40.132723+0000","last_active":"2023-10-07T10:44:41.142946+0000","last_peered":"2023-10-07T10:44:41.142946+0000","last_clean":"2023-10-07T10:44:41.142946+0000","last_became_active":"2023-10-07T10:44:40.132449+0000","last_became_peered":"2023-10-07T10:44:40.132449+0000","last_unstale":"2023-10-07T10:44:41.142946+0000","last_undegraded":"2023-10-07T10:44:41.142946+0000","last_fullsized":"2023-10-07T10:44:41.142946+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:25:13.934597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.150293+0000","last_change":"2023-10-07T10:44:41.145085+0000","last_active":"2023-10-07T10:44:41.150293+0000","last_peered":"2023-10-07T10:44:41.150293+0000","last_clean":"2023-10-07T10:44:41.150293+0000","last_became_active":"2023-10-07T10:44:41.144661+0000","last_became_peered":"2023-10-07T10:44:41.144661+0000","last_unstale":"2023-10-07T10:44:41.150293+0000","last_undegraded":"2023-10-07T10:44:41.150293+0000","last_fullsized":"2023-10-07T10:44:41.150293+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:36:05.572351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122151+0000","last_change":"2023-10-07T10:44:40.137697+0000","last_active":"2023-10-07T10:44:41.122151+0000","last_peered":"2023-10-07T10:44:41.122151+0000","last_clean":"2023-10-07T10:44:41.122151+0000","last_became_active":"2023-10-07T10:44:40.137443+0000","last_became_peered":"2023-10-07T10:44:40.137443+0000","last_unstale":"2023-10-07T10:44:41.122151+0000","last_undegraded":"2023-10-07T10:44:41.122151+0000","last_fullsized":"2023-10-07T10:44:41.122151+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:17:13.585287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154821+0000","last_change":"2023-10-07T10:44:41.141102+0000","last_active":"2023-10-07T10:44:41.154821+0000","last_peered":"2023-10-07T10:44:41.154821+0000","last_clean":"2023-10-07T10:44:41.154821+0000","last_became_active":"2023-10-07T10:44:41.140838+0000","last_became_peered":"2023-10-07T10:44:41.140838+0000","last_unstale":"2023-10-07T10:44:41.154821+0000","last_undegraded":"2023-10-07T10:44:41.154821+0000","last_fullsized":"2023-10-07T10:44:41.154821+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:18:45.329143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152492+0000","last_change":"2023-10-07T10:44:40.140398+0000","last_active":"2023-10-07T10:44:41.152492+0000","last_peered":"2023-10-07T10:44:41.152492+0000","last_clean":"2023-10-07T10:44:41.152492+0000","last_became_active":"2023-10-07T10:44:40.140127+0000","last_became_peered":"2023-10-07T10:44:40.140127+0000","last_unstale":"2023-10-07T10:44:41.152492+0000","last_undegraded":"2023-10-07T10:44:41.152492+0000","last_fullsized":"2023-10-07T10:44:41.152492+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:36:54.689289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148657+0000","last_change":"2023-10-07T10:44:41.144404+0000","last_active":"2023-10-07T10:44:41.148657+0000","last_peered":"2023-10-07T10:44:41.148657+0000","last_clean":"2023-10-07T10:44:41.148657+0000","last_became_active":"2023-10-07T10:44:41.144121+0000","last_became_peered":"2023-10-07T10:44:41.144121+0000","last_unstale":"2023-10-07T10:44:41.148657+0000","last_undegraded":"2023-10-07T10:44:41.148657+0000","last_fullsized":"2023-10-07T10:44:41.148657+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:57:26.369780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.122503+0000","last_change":"2023-10-07T10:44:40.143918+0000","last_active":"2023-10-07T10:44:41.122503+0000","last_peered":"2023-10-07T10:44:41.122503+0000","last_clean":"2023-10-07T10:44:41.122503+0000","last_became_active":"2023-10-07T10:44:40.143653+0000","last_became_peered":"2023-10-07T10:44:40.143653+0000","last_unstale":"2023-10-07T10:44:41.122503+0000","last_undegraded":"2023-10-07T10:44:41.122503+0000","last_fullsized":"2023-10-07T10:44:41.122503+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:50:57.857575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.155395+0000","last_change":"2023-10-07T10:44:41.151070+0000","last_active":"2023-10-07T10:44:41.155395+0000","last_peered":"2023-10-07T10:44:41.155395+0000","last_clean":"2023-10-07T10:44:41.155395+0000","last_became_active":"2023-10-07T10:44:41.150797+0000","last_became_peered":"2023-10-07T10:44:41.150797+0000","last_unstale":"2023-10-07T10:44:41.155395+0000","last_undegraded":"2023-10-07T10:44:41.155395+0000","last_fullsized":"2023-10-07T10:44:41.155395+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:43:44.383826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.121693+0000","last_change":"2023-10-07T10:44:40.133500+0000","last_active":"2023-10-07T10:44:41.121693+0000","last_peered":"2023-10-07T10:44:41.121693+0000","last_clean":"2023-10-07T10:44:41.121693+0000","last_became_active":"2023-10-07T10:44:40.132841+0000","last_became_peered":"2023-10-07T10:44:40.132841+0000","last_unstale":"2023-10-07T10:44:41.121693+0000","last_undegraded":"2023-10-07T10:44:41.121693+0000","last_fullsized":"2023-10-07T10:44:41.121693+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:27:05.014756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149046+0000","last_change":"2023-10-07T10:44:41.145335+0000","last_active":"2023-10-07T10:44:41.149046+0000","last_peered":"2023-10-07T10:44:41.149046+0000","last_clean":"2023-10-07T10:44:41.149046+0000","last_became_active":"2023-10-07T10:44:41.144671+0000","last_became_peered":"2023-10-07T10:44:41.144671+0000","last_unstale":"2023-10-07T10:44:41.149046+0000","last_undegraded":"2023-10-07T10:44:41.149046+0000","last_fullsized":"2023-10-07T10:44:41.149046+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:05:50.642947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.135866+0000","last_change":"2023-10-07T10:44:40.141168+0000","last_active":"2023-10-07T10:44:41.135866+0000","last_peered":"2023-10-07T10:44:41.135866+0000","last_clean":"2023-10-07T10:44:41.135866+0000","last_became_active":"2023-10-07T10:44:40.140927+0000","last_became_peered":"2023-10-07T10:44:40.140927+0000","last_unstale":"2023-10-07T10:44:41.135866+0000","last_undegraded":"2023-10-07T10:44:41.135866+0000","last_fullsized":"2023-10-07T10:44:41.135866+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:16.608833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.133173+0000","last_change":"2023-10-07T10:44:41.125440+0000","last_active":"2023-10-07T10:44:41.133173+0000","last_peered":"2023-10-07T10:44:41.133173+0000","last_clean":"2023-10-07T10:44:41.133173+0000","last_became_active":"2023-10-07T10:44:41.125118+0000","last_became_peered":"2023-10-07T10:44:41.125118+0000","last_unstale":"2023-10-07T10:44:41.133173+0000","last_undegraded":"2023-10-07T10:44:41.133173+0000","last_fullsized":"2023-10-07T10:44:41.133173+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:32:32.458450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.152265+0000","last_change":"2023-10-07T10:44:40.136125+0000","last_active":"2023-10-07T10:44:41.152265+0000","last_peered":"2023-10-07T10:44:41.152265+0000","last_clean":"2023-10-07T10:44:41.152265+0000","last_became_active":"2023-10-07T10:44:40.135826+0000","last_became_peered":"2023-10-07T10:44:40.135826+0000","last_unstale":"2023-10-07T10:44:41.152265+0000","last_undegraded":"2023-10-07T10:44:41.152265+0000","last_fullsized":"2023-10-07T10:44:41.152265+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:23:58.168750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.154914+0000","last_change":"2023-10-07T10:44:41.150823+0000","last_active":"2023-10-07T10:44:41.154914+0000","last_peered":"2023-10-07T10:44:41.154914+0000","last_clean":"2023-10-07T10:44:41.154914+0000","last_became_active":"2023-10-07T10:44:41.147430+0000","last_became_peered":"2023-10-07T10:44:41.147430+0000","last_unstale":"2023-10-07T10:44:41.154914+0000","last_undegraded":"2023-10-07T10:44:41.154914+0000","last_fullsized":"2023-10-07T10:44:41.154914+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:52.219810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.127113+0000","last_change":"2023-10-07T10:44:40.137441+0000","last_active":"2023-10-07T10:44:41.127113+0000","last_peered":"2023-10-07T10:44:41.127113+0000","last_clean":"2023-10-07T10:44:41.127113+0000","last_became_active":"2023-10-07T10:44:40.137111+0000","last_became_peered":"2023-10-07T10:44:40.137111+0000","last_unstale":"2023-10-07T10:44:41.127113+0000","last_undegraded":"2023-10-07T10:44:41.127113+0000","last_fullsized":"2023-10-07T10:44:41.127113+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:45:02.843088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145094+0000","last_change":"2023-10-07T10:44:41.137903+0000","last_active":"2023-10-07T10:44:41.145094+0000","last_peered":"2023-10-07T10:44:41.145094+0000","last_clean":"2023-10-07T10:44:41.145094+0000","last_became_active":"2023-10-07T10:44:41.137616+0000","last_became_peered":"2023-10-07T10:44:41.137616+0000","last_unstale":"2023-10-07T10:44:41.145094+0000","last_undegraded":"2023-10-07T10:44:41.145094+0000","last_fullsized":"2023-10-07T10:44:41.145094+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:25:14.942305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.131176+0000","last_change":"2023-10-07T10:44:40.138312+0000","last_active":"2023-10-07T10:44:41.131176+0000","last_peered":"2023-10-07T10:44:41.131176+0000","last_clean":"2023-10-07T10:44:41.131176+0000","last_became_active":"2023-10-07T10:44:40.137993+0000","last_became_peered":"2023-10-07T10:44:40.137993+0000","last_unstale":"2023-10-07T10:44:41.131176+0000","last_undegraded":"2023-10-07T10:44:41.131176+0000","last_fullsized":"2023-10-07T10:44:41.131176+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:17:13.585287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.142745+0000","last_change":"2023-10-07T10:44:41.139651+0000","last_active":"2023-10-07T10:44:41.142745+0000","last_peered":"2023-10-07T10:44:41.142745+0000","last_clean":"2023-10-07T10:44:41.142745+0000","last_became_active":"2023-10-07T10:44:41.139308+0000","last_became_peered":"2023-10-07T10:44:41.139308+0000","last_unstale":"2023-10-07T10:44:41.142745+0000","last_undegraded":"2023-10-07T10:44:41.142745+0000","last_fullsized":"2023-10-07T10:44:41.142745+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:58:17.616542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.134850+0000","last_change":"2023-10-07T10:44:40.130192+0000","last_active":"2023-10-07T10:44:41.134850+0000","last_peered":"2023-10-07T10:44:41.134850+0000","last_clean":"2023-10-07T10:44:41.134850+0000","last_became_active":"2023-10-07T10:44:40.129910+0000","last_became_peered":"2023-10-07T10:44:40.129910+0000","last_unstale":"2023-10-07T10:44:41.134850+0000","last_undegraded":"2023-10-07T10:44:41.134850+0000","last_fullsized":"2023-10-07T10:44:41.134850+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:48:25.361797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.148713+0000","last_change":"2023-10-07T10:44:41.139533+0000","last_active":"2023-10-07T10:44:41.148713+0000","last_peered":"2023-10-07T10:44:41.148713+0000","last_clean":"2023-10-07T10:44:41.148713+0000","last_became_active":"2023-10-07T10:44:41.139218+0000","last_became_peered":"2023-10-07T10:44:41.139218+0000","last_unstale":"2023-10-07T10:44:41.148713+0000","last_undegraded":"2023-10-07T10:44:41.148713+0000","last_fullsized":"2023-10-07T10:44:41.148713+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:40:18.432284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.131145+0000","last_change":"2023-10-07T10:44:40.142553+0000","last_active":"2023-10-07T10:44:41.131145+0000","last_peered":"2023-10-07T10:44:41.131145+0000","last_clean":"2023-10-07T10:44:41.131145+0000","last_became_active":"2023-10-07T10:44:40.142320+0000","last_became_peered":"2023-10-07T10:44:40.142320+0000","last_unstale":"2023-10-07T10:44:41.131145+0000","last_undegraded":"2023-10-07T10:44:41.131145+0000","last_fullsized":"2023-10-07T10:44:41.131145+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:18:44.321434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.147164+0000","last_change":"2023-10-07T10:44:41.142175+0000","last_active":"2023-10-07T10:44:41.147164+0000","last_peered":"2023-10-07T10:44:41.147164+0000","last_clean":"2023-10-07T10:44:41.147164+0000","last_became_active":"2023-10-07T10:44:41.141923+0000","last_became_peered":"2023-10-07T10:44:41.141923+0000","last_unstale":"2023-10-07T10:44:41.147164+0000","last_undegraded":"2023-10-07T10:44:41.147164+0000","last_fullsized":"2023-10-07T10:44:41.147164+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:44:44.790717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.126261+0000","last_change":"2023-10-07T10:44:40.142046+0000","last_active":"2023-10-07T10:44:41.126261+0000","last_peered":"2023-10-07T10:44:41.126261+0000","last_clean":"2023-10-07T10:44:41.126261+0000","last_became_active":"2023-10-07T10:44:40.141697+0000","last_became_peered":"2023-10-07T10:44:40.141697+0000","last_unstale":"2023-10-07T10:44:41.126261+0000","last_undegraded":"2023-10-07T10:44:41.126261+0000","last_fullsized":"2023-10-07T10:44:41.126261+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:55:51.467881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145291+0000","last_change":"2023-10-07T10:44:41.144274+0000","last_active":"2023-10-07T10:44:41.145291+0000","last_peered":"2023-10-07T10:44:41.145291+0000","last_clean":"2023-10-07T10:44:41.145291+0000","last_became_active":"2023-10-07T10:44:41.143961+0000","last_became_peered":"2023-10-07T10:44:41.143961+0000","last_unstale":"2023-10-07T10:44:41.145291+0000","last_undegraded":"2023-10-07T10:44:41.145291+0000","last_fullsized":"2023-10-07T10:44:41.145291+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:33:08.496023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.121806+0000","last_change":"2023-10-07T10:44:40.139190+0000","last_active":"2023-10-07T10:44:41.121806+0000","last_peered":"2023-10-07T10:44:41.121806+0000","last_clean":"2023-10-07T10:44:41.121806+0000","last_became_active":"2023-10-07T10:44:40.138909+0000","last_became_peered":"2023-10-07T10:44:40.138909+0000","last_unstale":"2023-10-07T10:44:41.121806+0000","last_undegraded":"2023-10-07T10:44:41.121806+0000","last_fullsized":"2023-10-07T10:44:41.121806+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:19:20.602757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.145141+0000","last_change":"2023-10-07T10:44:41.137601+0000","last_active":"2023-10-07T10:44:41.145141+0000","last_peered":"2023-10-07T10:44:41.145141+0000","last_clean":"2023-10-07T10:44:41.145141+0000","last_became_active":"2023-10-07T10:44:41.137310+0000","last_became_peered":"2023-10-07T10:44:41.137310+0000","last_unstale":"2023-10-07T10:44:41.145141+0000","last_undegraded":"2023-10-07T10:44:41.145141+0000","last_fullsized":"2023-10-07T10:44:41.145141+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:42:09.600649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.135918+0000","last_change":"2023-10-07T10:44:40.134547+0000","last_active":"2023-10-07T10:44:41.135918+0000","last_peered":"2023-10-07T10:44:41.135918+0000","last_clean":"2023-10-07T10:44:41.135918+0000","last_became_active":"2023-10-07T10:44:40.134284+0000","last_became_peered":"2023-10-07T10:44:40.134284+0000","last_unstale":"2023-10-07T10:44:41.135918+0000","last_undegraded":"2023-10-07T10:44:41.135918+0000","last_fullsized":"2023-10-07T10:44:41.135918+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:57:25.362071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.149070+0000","last_change":"2023-10-07T10:44:41.144985+0000","last_active":"2023-10-07T10:44:41.149070+0000","last_peered":"2023-10-07T10:44:41.149070+0000","last_clean":"2023-10-07T10:44:41.149070+0000","last_became_active":"2023-10-07T10:44:41.144649+0000","last_became_peered":"2023-10-07T10:44:41.144649+0000","last_unstale":"2023-10-07T10:44:41.149070+0000","last_undegraded":"2023-10-07T10:44:41.149070+0000","last_fullsized":"2023-10-07T10:44:41.149070+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:40.108976+0000","last_clean_scrub_stamp":"2023-10-07T10:44:40.108976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:41:07.425202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"150'2","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.156996+0000","last_change":"2023-10-07T10:44:40.143873+0000","last_active":"2023-10-07T10:44:41.156996+0000","last_peered":"2023-10-07T10:44:41.156996+0000","last_clean":"2023-10-07T10:44:41.156996+0000","last_became_active":"2023-10-07T10:44:40.143455+0000","last_became_peered":"2023-10-07T10:44:40.143455+0000","last_unstale":"2023-10-07T10:44:41.156996+0000","last_undegraded":"2023-10-07T10:44:41.156996+0000","last_fullsized":"2023-10-07T10:44:41.156996+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:44:39.101267+0000","last_clean_scrub_stamp":"2023-10-07T10:44:39.101267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:37:33.934868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"141'271","reported_seq":819,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-07T10:44:41.125378+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T10:44:41.125378+0000","last_peered":"2023-10-07T10:44:41.125378+0000","last_clean":"2023-10-07T10:44:41.125378+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T10:44:41.125378+0000","last_undegraded":"2023-10-07T10:44:41.125378+0000","last_fullsized":"2023-10-07T10:44:41.125378+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477635,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307912,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063928,"statfs":{"total":96636764160,"available":96321462272,"internally_reserved":0,"allocated":2490368,"data_stored":1507373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.95699999999999996,"5min":1.018,"15min":1.018},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.028,"15min":1.028},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.748,"15min":0.752},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.432},"max":{"1min":0.90900000000000003,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.75900000000000001},"min":{"1min":0.56999999999999995,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.995,"5min":1.119,"15min":1.119},"last":0.504}]},{"osd":2,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88200000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78000000000000003,"15min":0.77100000000000002},"min":{"1min":0.65900000000000003,"5min":0.55900000000000005,"15min":0.45400000000000001},"max":{"1min":1.002,"5min":1.147,"15min":1.147},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.746,"15min":0.73999999999999999},"min":{"1min":0.51600000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.94199999999999995,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.59399999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93899999999999995,"5min":1.161,"15min":1.161},"last":0.67900000000000005}]},{"osd":4,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.47699999999999998,"5min":0.44700000000000001,"15min":0.36399999999999999},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.92700000000000005,"5min":1.069,"15min":1.069},"last":0.66000000000000003}]},{"osd":5,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.83899999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70799999999999996,"15min":0.69599999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.746,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71199999999999997},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86699999999999999,"5min":1.131,"15min":1.131},"last":0.76200000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311416,"kb_used_data":1840,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060424,"statfs":{"total":96636764160,"available":96317874176,"internally_reserved":0,"allocated":1884160,"data_stored":916395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":0.75900000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90700000000000003,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.54400000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92600000000000005,"5min":1.052,"15min":1.073},"last":0.73599999999999999}]},{"osd":1,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.75800000000000001,"15min":0.754},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48499999999999999},"max":{"1min":0.874,"5min":1.008,"15min":1.0600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.50700000000000001},"max":{"1min":0.89000000000000001,"5min":1.034,"15min":1.0389999999999999},"last":0.76800000000000002}]},{"osd":2,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46100000000000002},"max":{"1min":1.018,"5min":1.0289999999999999,"15min":1.3380000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59499999999999997,"5min":0.54900000000000004,"15min":0.46500000000000002},"max":{"1min":0.95999999999999996,"5min":1.03,"15min":1.367},"last":0.88100000000000001}]},{"osd":3,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.51100000000000001},"max":{"1min":0.94599999999999995,"5min":1.044,"15min":1.0900000000000001},"last":0.75},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":1.0289999999999999,"15min":1.1040000000000001},"last":0.91900000000000004}]},{"osd":4,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.621},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0609999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.80900000000000005}]},{"osd":5,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.499,"5min":0.42899999999999999,"15min":0.40999999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.627},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0900000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.52000000000000002,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.1060000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76700000000000002,"15min":0.77000000000000002},"min":{"1min":0.57999999999999996,"5min":0.57899999999999996,"15min":0.53600000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.127},"last":0.89300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129543041,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307296,"kb_used_data":1816,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064544,"statfs":{"total":96636764160,"available":96322093056,"internally_reserved":0,"allocated":1859584,"data_stored":908151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.495,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.28899999999999998},"max":{"1min":0.70099999999999996,"5min":0.76000000000000001,"15min":0.83099999999999996},"last":0.439},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.45900000000000002,"5min":0.40799999999999997,"15min":0.40500000000000003},"max":{"1min":0.69699999999999995,"5min":0.76900000000000002,"15min":0.84199999999999997},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.35899999999999999},"max":{"1min":0.71399999999999997,"5min":0.74399999999999999,"15min":0.81699999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.374},"max":{"1min":0.77600000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63500000000000001,"15min":0.64000000000000001},"min":{"1min":0.505,"5min":0.48099999999999998,"15min":0.41999999999999998},"max":{"1min":0.85399999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.93100000000000005},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.83499999999999996,"15min":0.83399999999999996},"min":{"1min":0.64600000000000002,"5min":0.58899999999999997,"15min":0.56999999999999995},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1719999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.80600000000000005,"15min":0.80400000000000005},"min":{"1min":0.49399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.141},"last":0.84499999999999997}]},{"osd":5,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.84999999999999998,"15min":0.84699999999999998},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.54400000000000004},"max":{"1min":1.155,"5min":1.196,"15min":1.196},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82099999999999995,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"max":{"1min":0.97799999999999998,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.93200000000000005}]},{"osd":6,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.68600000000000005,"5min":0.56399999999999995,"15min":0.56100000000000005},"max":{"1min":1.153,"5min":1.153,"15min":1.2050000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.84099999999999997,"15min":0.84899999999999998},"min":{"1min":0.67100000000000004,"5min":0.60999999999999999,"15min":0.57199999999999995},"max":{"1min":1.0389999999999999,"5min":1.133,"15min":1.196},"last":0.80200000000000005}]},{"osd":7,"last update":"Sat Oct 7 10:43:50 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.86499999999999999,"15min":0.877},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.60899999999999999},"max":{"1min":1.0600000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88600000000000001,"15min":0.89000000000000001},"min":{"1min":0.72699999999999998,"5min":0.67100000000000004,"15min":0.60799999999999998},"max":{"1min":1.085,"5min":1.179,"15min":1.179},"last":0.98899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543041,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307888,"kb_used_data":2408,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063952,"statfs":{"total":96636764160,"available":96321486848,"internally_reserved":0,"allocated":2465792,"data_stored":1505227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.61099999999999999,"5min":0.751,"15min":0.76400000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":0.82899999999999996},"last":0.443}]},{"osd":2,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75,"5min":0.88900000000000001,"15min":0.92800000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.68400000000000005,"5min":0.87,"15min":0.87},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70899999999999996,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.80600000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.52500000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81599999999999995,"15min":0.79700000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.499},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.64700000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0960000000000001,"5min":1.194,"15min":1.194},"last":0.86699999999999999}]},{"osd":5,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.077,"5min":1.278,"15min":1.278},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":1.0229999999999999,"5min":1.206,"15min":1.206},"last":0.88700000000000001}]},{"osd":6,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86199999999999999,"15min":0.84699999999999998},"min":{"1min":0.67100000000000004,"5min":0.60599999999999998,"15min":0.59499999999999997},"max":{"1min":1.0369999999999999,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85799999999999998,"15min":0.85099999999999998},"min":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"max":{"1min":1.1579999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.97499999999999998}]},{"osd":7,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.89300000000000002,"15min":0.872},"min":{"1min":0.627,"5min":0.627,"15min":0.59899999999999998},"max":{"1min":1.131,"5min":1.173,"15min":1.198},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.875,"15min":0.86599999999999999},"min":{"1min":0.63100000000000001,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":1.198,"5min":1.198,"15min":1.234},"last":0.90900000000000003}]}]},{"osd":2,"up_from":14,"seq":60129543042,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311412,"kb_used_data":1836,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060428,"statfs":{"total":96636764160,"available":96317878272,"internally_reserved":0,"allocated":1880064,"data_stored":915447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.27500000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.438},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.38},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.64000000000000001},"min":{"1min":0.51200000000000001,"5min":0.502,"15min":0.44500000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64300000000000002,"15min":0.629},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":0.91400000000000003,"15min":1.0529999999999999},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.79900000000000004,"15min":0.79700000000000004},"min":{"1min":0.56499999999999995,"5min":0.504,"15min":0.48499999999999999},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.82999999999999996,"15min":0.82199999999999995},"min":{"1min":0.71799999999999997,"5min":0.59599999999999997,"15min":0.47299999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.66500000000000004}]},{"osd":5,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84599999999999997,"15min":0.84799999999999998},"min":{"1min":0.77800000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":1.212,"15min":1.349},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.81699999999999995,"15min":0.81999999999999995},"min":{"1min":0.62,"5min":0.56599999999999995,"15min":0.54000000000000004},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.86499999999999999,"15min":0.86199999999999999},"min":{"1min":0.77900000000000003,"5min":0.59399999999999997,"15min":0.55200000000000005},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.85899999999999999,"15min":0.85499999999999998},"min":{"1min":0.72399999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.2430000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.89000000000000001,"15min":0.879},"min":{"1min":0.80400000000000005,"5min":0.66300000000000003,"15min":0.59999999999999998},"max":{"1min":1.0820000000000001,"5min":1.1679999999999999,"15min":1.3160000000000001},"last":0.754},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.70599999999999996,"5min":0.61299999999999999,"15min":0.55500000000000005},"max":{"1min":1.139,"5min":1.139,"15min":1.2969999999999999},"last":0.76900000000000002}]}]},{"osd":3,"up_from":14,"seq":60129543043,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299124,"kb_used_data":1836,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072716,"statfs":{"total":96636764160,"available":96330461184,"internally_reserved":0,"allocated":1880064,"data_stored":915019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.49099999999999999,"15min":0.47399999999999998},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73099999999999998,"5min":1.073,"15min":1.073},"last":0.313},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81699999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.46300000000000002}]},{"osd":1,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52700000000000002,"15min":0.52000000000000002},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76900000000000002,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.52600000000000002,"5min":0.46300000000000002,"15min":0.45000000000000001},"max":{"1min":0.76000000000000001,"5min":1.02,"15min":1.02},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.42599999999999999,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":1.456,"15min":1.456},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.48599999999999999,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.75,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.63900000000000001,"5min":0.59099999999999997,"15min":0.56299999999999994},"max":{"1min":0.97899999999999998,"5min":1.319,"15min":1.3919999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81299999999999994,"15min":0.81000000000000005},"min":{"1min":0.65000000000000002,"5min":0.61099999999999999,"15min":0.60899999999999999},"max":{"1min":1.141,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83599999999999997,"15min":0.82099999999999995},"min":{"1min":0.65500000000000003,"5min":0.61799999999999999,"15min":0.56799999999999995},"max":{"1min":1.097,"5min":1.1879999999999999,"15min":1.2829999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.85099999999999998,"15min":0.84099999999999997},"min":{"1min":0.64400000000000002,"5min":0.63300000000000001,"15min":0.55600000000000005},"max":{"1min":1.1619999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.81899999999999995}]},{"osd":6,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.63,"5min":0.57099999999999995,"15min":0.56599999999999995},"max":{"1min":1.214,"5min":1.357,"15min":1.357},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.86599999999999999,"15min":0.84999999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.58799999999999997},"max":{"1min":1.1279999999999999,"5min":1.177,"15min":1.321},"last":0.747}]},{"osd":7,"last update":"Sat Oct 7 10:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88300000000000001,"15min":0.85699999999999998},"min":{"1min":0.72099999999999997,"5min":0.63200000000000001,"15min":0.58499999999999996},"max":{"1min":1.137,"5min":1.492,"15min":1.492},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.872,"5min":0.88900000000000001,"15min":0.86899999999999999},"min":{"1min":0.68700000000000006,"5min":0.64000000000000001,"15min":0.59599999999999997},"max":{"1min":1.161,"5min":1.411,"15min":1.411},"last":0.63200000000000001}]}]},{"osd":4,"up_from":15,"seq":64424510339,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311408,"kb_used_data":1832,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060432,"statfs":{"total":96636764160,"available":96317882368,"internally_reserved":0,"allocated":1875968,"data_stored":914929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93600000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.745,"15min":0.76000000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86499999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.76400000000000001,"15min":0.76500000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.016,"15min":1.032},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46500000000000002},"max":{"1min":0.84599999999999997,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":0.90300000000000002,"5min":1.002,"15min":1.01},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88500000000000001,"5min":1.145,"15min":1.145},"last":0.88300000000000001}]},{"osd":3,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.75600000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.497},"max":{"1min":0.82899999999999996,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96599999999999997,"5min":1.129,"15min":1.129},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45000000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.441,"5min":0.441,"15min":0.39400000000000002},"max":{"1min":0.85599999999999998,"5min":0.93999999999999995,"15min":1.0349999999999999},"last":0.996}]},{"osd":6,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89200000000000002,"5min":0.98999999999999999,"15min":1.018},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.69899999999999995},"min":{"1min":0.54500000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85499999999999998,"5min":0.96799999999999997,"15min":1.042},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.54600000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90400000000000003,"5min":0.94099999999999995,"15min":0.97899999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0960000000000001},"last":0.85499999999999998}]}]},{"osd":5,"up_from":14,"seq":60129543041,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311412,"kb_used_data":1836,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060428,"statfs":{"total":96636764160,"available":96317878272,"internally_reserved":0,"allocated":1880064,"data_stored":914971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78700000000000003,"15min":0.79400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39500000000000002},"max":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.091},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.438},"max":{"1min":0.93500000000000005,"5min":1.0820000000000001,"15min":1.1279999999999999},"last":1.391}]},{"osd":1,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.746},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93400000000000005,"5min":1.032,"15min":1.032},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78100000000000003,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.46899999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":5.0780000000000003}]},{"osd":2,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89900000000000002,"5min":1.0760000000000001,"15min":1.093},"last":3.415},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.76700000000000002,"15min":0.76800000000000002},"min":{"1min":0.52300000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89100000000000001,"5min":1.077,"15min":1.343},"last":1.2989999999999999}]},{"osd":3,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.77600000000000002,"15min":0.78200000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.50800000000000001},"max":{"1min":0.872,"5min":1.121,"15min":1.121},"last":1.341},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75700000000000001,"15min":0.754},"min":{"1min":0.61799999999999999,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":3.3570000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.91400000000000003,"5min":0.94999999999999996,"15min":0.995},"last":1.9630000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.64700000000000002},"min":{"1min":0.47199999999999998,"5min":0.46500000000000002,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":1.0089999999999999,"15min":1.0529999999999999},"last":3.2999999999999998}]},{"osd":6,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.52400000000000002,"5min":0.44500000000000001,"15min":0.39800000000000002},"max":{"1min":0.91900000000000004,"5min":1.016,"15min":1.083},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.71499999999999997,"15min":0.71299999999999997},"min":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0289999999999999,"5min":1.032,"15min":1.101},"last":12.090999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.73499999999999999,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.075},"last":1.835},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":1.4379999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T10:44:46.019 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T10:44:46.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T10:44:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.253+0000 7fce253ff700 1 Processor -- start 2023-10-07T10:44:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.253+0000 7fce253ff700 1 -- start start 2023-10-07T10:44:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce253ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce200cc570 0x7fce2015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce253ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce20156540 0x7fce2015a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce253ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2015adf0 con 0x7fce20156170 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce253ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200ad7c0 con 0x7fce20156540 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce253ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200b0000 con 0x7fce200cc570 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce20156540 0x7fce2015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1effd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fce20156170 0x7fce200cbe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45118/0 (socket says 172.21.15.29:45118) 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1effd700 1 -- 172.21.15.29:0/3096990115 learned_addr learned my addr 172.21.15.29:0/3096990115 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3096990115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce200cc570 msgr2=0x7fce2015a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1e7fc700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce200cc570 0x7fce2015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1dffb700 1 -- 172.21.15.29:0/3096990115 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879622988 0 0) 0x7fce2015adf0 con 0x7fce20156170 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1f7fe700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce200cc570 0x7fce2015a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3096990115 >> v1:172.21.15.29:6789/0 conn(0x7fce20156170 legacy=0x7fce200cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3096990115 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2015ab40 con 0x7fce20156540 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.255+0000 7fce1e7fc700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce200cc570 0x7fce2015a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.256+0000 7fce1f7fe700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce2015a600 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fce18009e60 tx=0x7fce18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb8020e76406ffd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.256+0000 7fce1dffb700 1 -- 172.21.15.29:0/3096990115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1800e070 con 0x7fce20156540 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.256+0000 7fce1dffb700 1 -- 172.21.15.29:0/3096990115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1800bc30 con 0x7fce20156540 2023-10-07T10:44:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.256+0000 7fce1dffb700 1 -- 172.21.15.29:0/3096990115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18012430 con 0x7fce20156540 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.256+0000 7fce253ff700 1 -- 172.21.15.29:0/3096990115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 msgr2=0x7fce2015a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.256+0000 7fce253ff700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce2015a600 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fce18009e60 tx=0x7fce18009780 comp rx=0 tx=0).stop 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.257+0000 7fce253ff700 1 -- 172.21.15.29:0/3096990115 shutdown_connections 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.257+0000 7fce253ff700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce2015a600 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.257+0000 7fce253ff700 1 --2- 172.21.15.29:0/3096990115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce200cc570 0x7fce2015a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.257+0000 7fce253ff700 1 -- 172.21.15.29:0/3096990115 >> 172.21.15.29:0/3096990115 conn(0x7fce20067610 msgr2=0x7fce20067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.257+0000 7fce253ff700 1 -- 172.21.15.29:0/3096990115 shutdown_connections 2023-10-07T10:44:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.257+0000 7fce253ff700 1 -- 172.21.15.29:0/3096990115 wait complete. 2023-10-07T10:44:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.258+0000 7fce253ff700 1 Processor -- start 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.258+0000 7fce253ff700 1 -- start start 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce253ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce200cc570 0x7fce20155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce253ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce20156170 0x7fce20165af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce253ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce200cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce253ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2015a710 con 0x7fce20156540 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce253ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce20153eb0 con 0x7fce200cc570 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce253ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce20166030 con 0x7fce20156170 2023-10-07T10:44:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce1f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce200cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce1f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce200cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38834/0 (socket says 172.21.15.29:38834) 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.259+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3715742229 learned_addr learned my addr 172.21.15.29:0/3715742229 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1effd700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce200cc570 0x7fce20155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1e7fc700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce20156170 0x7fce20165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce20156170 msgr2=0x7fce20165af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1f7fe700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce20156170 0x7fce20165af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce200cc570 msgr2=0x7fce20155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1f7fe700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce200cc570 0x7fce20155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1f7fe700 1 -- 172.21.15.29:0/3715742229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009a20 con 0x7fce20156540 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1e7fc700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce20156170 0x7fce20165af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1effd700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce200cc570 0x7fce20155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce1f7fe700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce200cb650 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fce18013000 tx=0x7fce18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1801d630 con 0x7fce20156540 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1801d790 con 0x7fce20156540 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce20159b00 con 0x7fce20156540 2023-10-07T10:44:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.260+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1801db30 con 0x7fce20156540 2023-10-07T10:44:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.261+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2015a0c0 con 0x7fce20156540 2023-10-07T10:44:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.262+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce200718b0 con 0x7fce20156540 2023-10-07T10:44:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.263+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fce18024070 con 0x7fce20156540 2023-10-07T10:44:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.264+0000 7fce06ffd700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fce0c066f30 0x7fce0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.264+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fce18087120 con 0x7fce20156540 2023-10-07T10:44:46.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.264+0000 7fce1effd700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fce0c066f30 0x7fce0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.265+0000 7fce1effd700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fce0c066f30 0x7fce0c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fce100011e0 tx=0x7fce10009040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:46.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.268+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce18054150 con 0x7fce20156540 2023-10-07T10:44:46.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.551+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fce200ace90 con 0x7fce20156540 2023-10-07T10:44:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.555+0000 7fce06ffd700 1 -- 172.21.15.29:0/3715742229 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+0 (secure 0 0 0) 0x7fce18057770 con 0x7fce20156540 2023-10-07T10:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.557+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fce0c066f30 msgr2=0x7fce0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.557+0000 7fce253ff700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fce0c066f30 0x7fce0c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fce100011e0 tx=0x7fce10009040 comp rx=0 tx=0).stop 2023-10-07T10:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.557+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 msgr2=0x7fce200cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.557+0000 7fce253ff700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce200cb650 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fce18013000 tx=0x7fce18019000 comp rx=0 tx=0).stop 2023-10-07T10:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 shutdown_connections 2023-10-07T10:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fce0c066f30 0x7fce0c0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce20156540 0x7fce200cb650 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce200cc570 0x7fce20155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 --2- 172.21.15.29:0/3715742229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce20156170 0x7fce20165af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 >> 172.21.15.29:0/3715742229 conn(0x7fce20067610 msgr2=0x7fce2006ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 shutdown_connections 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.558+0000 7fce253ff700 1 -- 172.21.15.29:0/3715742229 wait complete. 2023-10-07T10:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T10:44:46.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.771+0000 7f3c079ff700 1 Processor -- start 2023-10-07T10:44:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.771+0000 7f3c079ff700 1 -- start start 2023-10-07T10:44:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.772+0000 7f3c079ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c00159550 0x7f3c000b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.772+0000 7f3c079ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b0f50 0x7f3c000b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c079ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0006f4e0 con 0x7f3c000b2bb0 2023-10-07T10:44:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c079ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0006e8d0 con 0x7f3c00159550 2023-10-07T10:44:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c079ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0006ebb0 con 0x7f3c000b0f50 2023-10-07T10:44:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c0579b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c00159550 0x7f3c000b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c05f9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3c000b2bb0 0x7f3c0015d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45124/0 (socket says 172.21.15.29:45124) 2023-10-07T10:44:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c05f9c700 1 -- 172.21.15.29:0/3241369449 learned_addr learned my addr 172.21.15.29:0/3241369449 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c04f9a700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b0f50 0x7f3c000b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c0579b700 1 -- 172.21.15.29:0/3241369449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b0f50 msgr2=0x7f3c000b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c0579b700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b0f50 0x7f3c000b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c0579b700 1 -- 172.21.15.29:0/3241369449 >> v1:172.21.15.29:6789/0 conn(0x7f3c000b2bb0 legacy=0x7f3c0015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c0579b700 1 -- 172.21.15.29:0/3241369449 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0006ee90 con 0x7f3c00159550 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3c04f9a700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b0f50 0x7f3c000b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.773+0000 7f3bf7fff700 1 -- 172.21.15.29:0/3241369449 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949812860 0 0) 0x7f3c0006f4e0 con 0x7f3c000b2bb0 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c0579b700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c000b0a10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0098e0 tx=0x7f3bfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54685185c817e40 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3bf7fff700 1 -- 172.21.15.29:0/3241369449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc009e80 con 0x7f3c00159550 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3bf7fff700 1 -- 172.21.15.29:0/3241369449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc00b860 con 0x7f3c00159550 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3bf7fff700 1 -- 172.21.15.29:0/3241369449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00bbe0 con 0x7f3c00159550 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c079ff700 1 -- 172.21.15.29:0/3241369449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 msgr2=0x7f3c000b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c000b0a10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0098e0 tx=0x7f3bfc0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c079ff700 1 -- 172.21.15.29:0/3241369449 shutdown_connections 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c000b0a10 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3241369449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b0f50 0x7f3c000b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.774+0000 7f3c079ff700 1 -- 172.21.15.29:0/3241369449 >> 172.21.15.29:0/3241369449 conn(0x7f3c00069190 msgr2=0x7f3c0006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.775+0000 7f3c079ff700 1 -- 172.21.15.29:0/3241369449 shutdown_connections 2023-10-07T10:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.775+0000 7f3c079ff700 1 -- 172.21.15.29:0/3241369449 wait complete. 2023-10-07T10:44:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.776+0000 7f3c079ff700 1 Processor -- start 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.776+0000 7f3c079ff700 1 -- start start 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c079ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c000b0f50 0x7f3c000abc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c079ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b2bb0 0x7f3c0015c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c079ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c00153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c079ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0016e620 con 0x7f3c00159550 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c079ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0016da10 con 0x7f3c000b0f50 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c079ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0016dcf0 con 0x7f3c000b2bb0 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c05f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c00153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c05f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c00153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38858/0 (socket says 172.21.15.29:38858) 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.777+0000 7f3c05f9c700 1 -- 172.21.15.29:0/3335307926 learned_addr learned my addr 172.21.15.29:0/3335307926 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c04f9a700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b2bb0 0x7f3c0015c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c0579b700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c000b0f50 0x7f3c000abc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c05f9c700 1 -- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b2bb0 msgr2=0x7f3c0015c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c05f9c700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b2bb0 0x7f3c0015c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c05f9c700 1 -- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c000b0f50 msgr2=0x7f3c000abc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c05f9c700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c000b0f50 0x7f3c000abc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c05f9c700 1 -- 172.21.15.29:0/3335307926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc009420 con 0x7f3c00159550 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c04f9a700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b2bb0 0x7f3c0015c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c0579b700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c000b0f50 0x7f3c000abc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c05f9c700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c00153e50 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800cbf0 tx=0x7f3bf80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf800a9c0 con 0x7f3c00159550 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf800ab20 con 0x7f3c00159550 2023-10-07T10:44:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0016dfd0 con 0x7f3c00159550 2023-10-07T10:44:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80070f0 con 0x7f3c00159550 2023-10-07T10:44:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.778+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0016d6c0 con 0x7f3c00159550 2023-10-07T10:44:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.779+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0006a0a0 con 0x7f3c00159550 2023-10-07T10:44:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.780+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3bf8007520 con 0x7f3c00159550 2023-10-07T10:44:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.782+0000 7f3bf57fa700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3be4066f30 0x7f3be40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.782+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f3bf8084b70 con 0x7f3c00159550 2023-10-07T10:44:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.782+0000 7f3c0579b700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3be4066f30 0x7f3be40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.783+0000 7f3c0579b700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3be4066f30 0x7f3be40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc003080 tx=0x7f3bfc017000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:46.786+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bf8051c20 con 0x7f3c00159550 2023-10-07T10:44:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.033+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c0004c3c0 con 0x7f3c00159550 2023-10-07T10:44:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.034+0000 7f3bf57fa700 1 -- 172.21.15.29:0/3335307926 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f3bf8055240 con 0x7f3c00159550 2023-10-07T10:44:47.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:47.035 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:44:40.623432+0000","modified":"2023-10-07T10:44:44.142903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8634":{"gid":8634,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3670759585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3670759585},{"type":"v1","addr":"172.21.15.29:6835","nonce":3670759585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-10-07T10:44:47.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3be4066f30 msgr2=0x7f3be40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3be4066f30 0x7f3be40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc003080 tx=0x7f3bfc017000 comp rx=0 tx=0).stop 2023-10-07T10:44:47.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 msgr2=0x7f3c00153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c00153e50 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800cbf0 tx=0x7f3bf80019b0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 shutdown_connections 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3be4066f30 0x7f3be40693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c00159550 0x7f3c00153e50 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c000b0f50 0x7f3c000abc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 --2- 172.21.15.29:0/3335307926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c000b2bb0 0x7f3c0015c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 >> 172.21.15.29:0/3335307926 conn(0x7f3c00069190 msgr2=0x7f3c001550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 shutdown_connections 2023-10-07T10:44:47.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.036+0000 7f3c079ff700 1 -- 172.21.15.29:0/3335307926 wait complete. 2023-10-07T10:44:47.039 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-07T10:44:47.100 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T10:44:47.099+0000 7f25295f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T10:44:47.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T10:44:40.623432+0000', 'modified': '2023-10-07T10:44:44.142903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8634': {'gid': 8634, 'name': 'c', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/3670759585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 3670759585}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 3670759585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T10:44:47.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T10:44:47.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-07T10:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.654+0000 7f4383dff700 1 Processor -- start 2023-10-07T10:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.655+0000 7f4383dff700 1 -- start start 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.656+0000 7f4383dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c0afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.656+0000 7f4383dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c14d790 con 0x7f437c0ad370 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.656+0000 7f4383dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c0ae5d0 con 0x7f437c159ec0 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.656+0000 7f4383dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0ad740 con 0x7f437c15a340 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.656+0000 7f4381b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f437c0ad370 0x7f437c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45136/0 (socket says 172.21.15.29:45136) 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.656+0000 7f4381b9b700 1 -- 172.21.15.29:0/537489862 learned_addr learned my addr 172.21.15.29:0/537489862 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f438239c700 1 --2- 172.21.15.29:0/537489862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c0afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631468889 0 0) 0x7f437c14d790 con 0x7f437c0ad370 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4368003570 con 0x7f437c0ad370 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874560501 0 0) 0x7f437c0ae5d0 con 0x7f437c159ec0 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437c14d790 con 0x7f437c159ec0 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 708404493 0 0) 0x7f4368003570 con 0x7f437c0ad370 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f438239c700 1 -- 172.21.15.29:0/537489862 >> v1:172.21.15.144:6789/0 conn(0x7f437c159ec0 legacy=0x7f437c0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f438239c700 1 -- 172.21.15.29:0/537489862 >> v1:172.21.15.29:6789/0 conn(0x7f437c0ad370 legacy=0x7f437c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.657+0000 7f438239c700 1 -- 172.21.15.29:0/537489862 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0ad9c0 con 0x7f437c15a340 2023-10-07T10:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.658+0000 7f438239c700 1 --2- 172.21.15.29:0/537489862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c0afb10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4374009e90 tx=0x7f4374009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a12eb1997c639565 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.658+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437400e070 con 0x7f437c15a340 2023-10-07T10:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.658+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437400b870 con 0x7f437c15a340 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.658+0000 7f4380b99700 1 -- 172.21.15.29:0/537489862 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378001470 con 0x7f437c15a340 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 -- 172.21.15.29:0/537489862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 msgr2=0x7f437c0afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 --2- 172.21.15.29:0/537489862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c0afb10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4374009e90 tx=0x7f4374009780 comp rx=0 tx=0).stop 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 -- 172.21.15.29:0/537489862 shutdown_connections 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 --2- 172.21.15.29:0/537489862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c0afb10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 -- 172.21.15.29:0/537489862 >> 172.21.15.29:0/537489862 conn(0x7f437c0680a0 msgr2=0x7f437c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 -- 172.21.15.29:0/537489862 shutdown_connections 2023-10-07T10:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.659+0000 7f4383dff700 1 -- 172.21.15.29:0/537489862 wait complete. 2023-10-07T10:44:47.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.660+0000 7f4383dff700 1 Processor -- start 2023-10-07T10:44:47.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.661+0000 7f4383dff700 1 -- start start 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.661+0000 7f4383dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f437c0ad370 0x7f437c0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.661+0000 7f4383dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 0x7f437c16a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f4383dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c16a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f4383dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c14c550 con 0x7f437c159ec0 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f4383dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0c86a0 con 0x7f437c0ad370 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f4383dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c16a750 con 0x7f437c15a340 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 0x7f437c16a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 0x7f437c16a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38870/0 (socket says 172.21.15.29:38870) 2023-10-07T10:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438239c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c16a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 -- 172.21.15.29:0/3475364030 learned_addr learned my addr 172.21.15.29:0/3475364030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f4381b9b700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f437c0ad370 0x7f437c0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 -- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 msgr2=0x7f437c16a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c16a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 -- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f437c0ad370 msgr2=0x7f437c0c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f437c0ad370 0x7f437c0c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.662+0000 7f438139a700 1 -- 172.21.15.29:0/3475364030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4374009a20 con 0x7f437c159ec0 2023-10-07T10:44:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.663+0000 7f438139a700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 0x7f437c16a210 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f436c00c790 tx=0x7f436c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.663+0000 7f4371ffb700 1 -- 172.21.15.29:0/3475364030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c010510 con 0x7f437c159ec0 2023-10-07T10:44:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.663+0000 7f4371ffb700 1 -- 172.21.15.29:0/3475364030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c010670 con 0x7f437c159ec0 2023-10-07T10:44:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.663+0000 7f4371ffb700 1 -- 172.21.15.29:0/3475364030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c010a10 con 0x7f437c159ec0 2023-10-07T10:44:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.663+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c0c92d0 con 0x7f437c159ec0 2023-10-07T10:44:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.663+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c0c9890 con 0x7f437c159ec0 2023-10-07T10:44:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.665+0000 7f4371ffb700 1 -- 172.21.15.29:0/3475364030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f436c01b3b0 con 0x7f437c159ec0 2023-10-07T10:44:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.666+0000 7f4371ffb700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4360066f30 0x7f43600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.666+0000 7f4371ffb700 1 -- 172.21.15.29:0/3475364030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f436c083fd0 con 0x7f437c159ec0 2023-10-07T10:44:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.666+0000 7f4381b9b700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4360066f30 0x7f43600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.667+0000 7f4381b9b700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4360066f30 0x7f43600693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4378001ed0 tx=0x7f4378001950 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.668+0000 7f4392475b80 1 Processor -- start 2023-10-07T10:44:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.668+0000 7f4392475b80 1 -- start start 2023-10-07T10:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.669+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c5346a40 0x55e0c5346e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.669+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c534f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.669+0000 7f4392475b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 0x55e0c5350010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.669+0000 7f4392475b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c5219310 con 0x55e0c534fbc0 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.669+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c5256130 con 0x55e0c5346a40 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.669+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e0c5253af0 con 0x55e0c532c530 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 0x55e0c5350010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 0x55e0c5350010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38882/0 (socket says 172.21.15.29:38882) 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 -- 172.21.15.29:0/2420470476 learned_addr learned my addr 172.21.15.29:0/2420470476 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f4381b9b700 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c5346a40 0x55e0c5346e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438139a700 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c534f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 -- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 msgr2=0x55e0c534f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c534f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 -- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c5346a40 msgr2=0x55e0c5346e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c5346a40 0x55e0c5346e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 -- 172.21.15.29:0/2420470476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0c51420a0 con 0x55e0c534fbc0 2023-10-07T10:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.670+0000 7f438239c700 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 0x55e0c5350010 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f4374003000 tx=0x7f4374017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14084daf8a27996e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f435effd700 1 -- 172.21.15.29:0/2420470476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374007000 con 0x55e0c534fbc0 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f435effd700 1 -- 172.21.15.29:0/2420470476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437400b9e0 con 0x55e0c534fbc0 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f435effd700 1 -- 172.21.15.29:0/2420470476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437401e690 con 0x55e0c534fbc0 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 -- 172.21.15.29:0/2420470476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 msgr2=0x55e0c5350010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 0x55e0c5350010 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f4374003000 tx=0x7f4374017000 comp rx=0 tx=0).stop 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 -- 172.21.15.29:0/2420470476 shutdown_connections 2023-10-07T10:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c534fbc0 0x55e0c5350010 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c5346a40 0x55e0c5346e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 --2- 172.21.15.29:0/2420470476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c534f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 -- 172.21.15.29:0/2420470476 >> 172.21.15.29:0/2420470476 conn(0x55e0c5349e70 msgr2=0x55e0c534a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 -- 172.21.15.29:0/2420470476 shutdown_connections 2023-10-07T10:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.671+0000 7f4392475b80 1 -- 172.21.15.29:0/2420470476 wait complete. 2023-10-07T10:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.672+0000 7f4392475b80 1 -- start start 2023-10-07T10:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.672+0000 7f4392475b80 1 Processor -- start 2023-10-07T10:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c532a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 0x55e0c532aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5314aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c524f8e0 con 0x55e0c5346a40 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c51fb980 con 0x55e0c534fbc0 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e0c51fb620 con 0x55e0c532c530 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.673+0000 7f4392475b80 10 client.0 _ll_register_callbacks cb 0x55e0c505ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 0x55e0c532aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 0x55e0c532aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38898/0 (socket says 172.21.15.29:38898) 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 -- 172.21.15.29:0/795287920 learned_addr learned my addr 172.21.15.29:0/795287920 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 -- 172.21.15.29:0/795287920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 msgr2=0x55e0c532a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c532a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 -- 172.21.15.29:0/795287920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 msgr2=0x55e0c5314aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5314aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 -- 172.21.15.29:0/795287920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c0ad9c0 con 0x55e0c5346a40 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f438139a700 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 0x55e0c532aa30 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f436c00cd80 tx=0x7f436c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bb3d44ba7a59d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f4357fff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f4357fff700 1 -- 172.21.15.29:0/795287920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c06b100 con 0x55e0c5346a40 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f4357fff700 1 -- 172.21.15.29:0/795287920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f436c057f10 con 0x55e0c5346a40 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.674+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0c52f4560 con 0x55e0c5346a40 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4357fff700 1 -- 172.21.15.29:0/795287920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f436c05bc60 con 0x55e0c5346a40 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 10 client.8760 fetch_fsmap learned FSMap version 77 2023-10-07T10:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55e0c521e3c0 con 0x55e0c5346a40 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4357fff700 1 -- 172.21.15.29:0/795287920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e0c52f4560 con 0x55e0c5346a40 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4357fff700 1 -- 172.21.15.29:0/795287920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 77) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f436c05b7d0 con 0x55e0c5346a40 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 10 client.8760 fetch_fsmap finished waiting for FSMap version 77 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 10 client.8760 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 20 client.8760 populate_metadata read hostname 'smithi029' 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c534e530 0x55e0c53633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 4 client.8760 mds_command: new command op to 8634 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 --> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0c52d10e0 con 0x55e0c534e530 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.675+0000 7f4381b9b700 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c534e530 0x55e0c53633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.676+0000 7f4381b9b700 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c534e530 0x55e0c53633d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3fc9542e799047 server_cookie=1fe70a0472eb1f2 in_seq=0 out_seq=0 2023-10-07T10:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.676+0000 7f4357fff700 10 client.8760 ms_handle_connect on v2:172.21.15.29:6833/3670759585 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.676+0000 7f4357fff700 1 -- 172.21.15.29:0/795287920 <== mds.0 v2:172.21.15.29:6833/3670759585 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55e0c52d10e0 con 0x55e0c534e530 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.676+0000 7f4357fff700 10 client.8760 handle_command_reply: tid=0 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.676+0000 7f4392475b80 1 client.8760 shutdown 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 msgr2=0x55e0c532aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 0x55e0c532aa30 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f436c00cd80 tx=0x7f436c00f000 comp rx=0 tx=0).stop 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 shutdown_connections 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c534e530 0x55e0c53633d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5314aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c532c530 0x55e0c532a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4357fff700 0 client.8760 ms_handle_reset on v2:172.21.15.29:6833/3670759585 2023-10-07T10:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 --2- 172.21.15.29:0/795287920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c5346a40 0x55e0c532aa30 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 >> 172.21.15.29:0/795287920 conn(0x55e0c5349e70 msgr2=0x55e0c52fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 shutdown_connections 2023-10-07T10:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 1 -- 172.21.15.29:0/795287920 wait complete. 2023-10-07T10:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.677+0000 7f4392475b80 20 client.8760 trim_cache size 0 max 16384 2023-10-07T10:44:47.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.695+0000 7f4392475b80 1 Processor -- start 2023-10-07T10:44:47.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.695+0000 7f4392475b80 1 -- start start 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4392475b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5345650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c53276a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5327be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4392475b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c50e71e0 con 0x55e0c532c530 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c5302530 con 0x55e0c534fbc0 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e0c532bfa0 con 0x55e0c5346a40 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4381b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5345650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4381b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5345650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38908/0 (socket says 172.21.15.29:38908) 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.696+0000 7f4381b9b700 1 -- 172.21.15.29:0/1112841426 learned_addr learned my addr 172.21.15.29:0/1112841426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f438139a700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c53276a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f438239c700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5327be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f4381b9b700 1 -- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 msgr2=0x55e0c53276a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f4381b9b700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c53276a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f4381b9b700 1 -- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 msgr2=0x55e0c5327be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f4381b9b700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5327be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f4381b9b700 1 -- 172.21.15.29:0/1112841426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0c532c300 con 0x55e0c532c530 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f438239c700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5327be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f438139a700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c53276a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f4381b9b700 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5345650 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f43780011e0 tx=0x7f4378026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a33ca5ab38ef94 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f435f7fe700 1 -- 172.21.15.29:0/1112841426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378024a10 con 0x55e0c532c530 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f435f7fe700 1 -- 172.21.15.29:0/1112841426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4378024b70 con 0x55e0c532c530 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.697+0000 7f435f7fe700 1 -- 172.21.15.29:0/1112841426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437802e420 con 0x55e0c532c530 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 -- 172.21.15.29:0/1112841426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 msgr2=0x55e0c5345650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5345650 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f43780011e0 tx=0x7f4378026040 comp rx=0 tx=0).stop 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 -- 172.21.15.29:0/1112841426 shutdown_connections 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c5327be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5345650 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 --2- 172.21.15.29:0/1112841426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c53276a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 -- 172.21.15.29:0/1112841426 >> 172.21.15.29:0/1112841426 conn(0x55e0c5349e70 msgr2=0x55e0c532a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 -- 172.21.15.29:0/1112841426 shutdown_connections 2023-10-07T10:44:47.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.698+0000 7f4392475b80 1 -- 172.21.15.29:0/1112841426 wait complete. 2023-10-07T10:44:47.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.699+0000 7f4392475b80 1 -- start start 2023-10-07T10:44:47.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.699+0000 7f4392475b80 1 Processor -- start 2023-10-07T10:44:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5391970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c534adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c534b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c5391eb0 con 0x55e0c532c530 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0c5301bb0 con 0x55e0c534fbc0 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e0c53272e0 con 0x55e0c5346a40 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4392475b80 10 client.0 _ll_register_callbacks cb 0x55e0c505ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5391970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5391970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38916/0 (socket says 172.21.15.29:38916) 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 -- 172.21.15.29:0/2538693871 learned_addr learned my addr 172.21.15.29:0/2538693871 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f438139a700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c534adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 -- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 msgr2=0x55e0c534adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f438239c700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c534b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c534adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 -- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 msgr2=0x55e0c534b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c534b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.700+0000 7f4381b9b700 1 -- 172.21.15.29:0/2538693871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4378009040 con 0x55e0c532c530 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f438139a700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c534adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f438239c700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c534b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4381b9b700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5391970 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f4378029000 tx=0x7f4378021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2a9688c3596843 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4357fff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4357fff700 1 -- 172.21.15.29:0/2538693871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378024990 con 0x55e0c532c530 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4357fff700 1 -- 172.21.15.29:0/2538693871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f437801b040 con 0x55e0c532c530 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f437c0cb5b0 con 0x55e0c532c530 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4357fff700 1 -- 172.21.15.29:0/2538693871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f437800bb20 con 0x55e0c532c530 2023-10-07T10:44:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4392475b80 10 client.8766 fetch_fsmap learned FSMap version 77 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55e0c50b61d0 con 0x55e0c532c530 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.701+0000 7f4357fff700 1 -- 172.21.15.29:0/2538693871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437803e070 con 0x55e0c532c530 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4357fff700 1 -- 172.21.15.29:0/2538693871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 77) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f437800bc80 con 0x55e0c532c530 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4392475b80 10 client.8766 fetch_fsmap finished waiting for FSMap version 77 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4392475b80 10 client.8766 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4392475b80 20 client.8766 populate_metadata read hostname 'smithi029' 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4392475b80 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c5300b80 0x55e0c5300f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4392475b80 4 client.8766 mds_command: new command op to 8634 tid=0[{"prefix": "status"}] 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 --> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f437400b9e0 con 0x55e0c5300b80 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f438139a700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c5300b80 0x55e0c5300f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f438139a700 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c5300b80 0x55e0c5300f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b770d603c6880d server_cookie=17805f187bc56057 in_seq=0 out_seq=0 2023-10-07T10:44:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.702+0000 7f4357fff700 10 client.8766 ms_handle_connect on v2:172.21.15.29:6833/3670759585 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4357fff700 1 -- 172.21.15.29:0/2538693871 <== mds.0 v2:172.21.15.29:6833/3670759585 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f437400b9e0 con 0x55e0c5300b80 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4357fff700 10 client.8766 handle_command_reply: tid=0 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 client.8766 shutdown 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 msgr2=0x55e0c5391970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5391970 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f4378029000 tx=0x7f4378021000 comp rx=0 tx=0).stop 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 shutdown_connections 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:6833/3670759585,v1:172.21.15.29:6835/3670759585] conn(0x55e0c5300b80 0x55e0c5300f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e0c534fbc0 0x55e0c534b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0c532c530 0x55e0c5391970 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 --2- 172.21.15.29:0/2538693871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e0c5346a40 0x55e0c534adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 >> 172.21.15.29:0/2538693871 conn(0x55e0c5349e70 msgr2=0x55e0c532a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4357fff700 0 client.8766 ms_handle_reset on v2:172.21.15.29:6833/3670759585 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 shutdown_connections 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 1 -- 172.21.15.29:0/2538693871 wait complete. 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.703+0000 7f4392475b80 20 client.8766 trim_cache size 0 max 16384 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8634, 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T10:44:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5856412039999999, 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 77, 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 151, 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 151, 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.257003181 2023-10-07T10:44:47.712 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4360066f30 msgr2=0x7f43600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4360066f30 0x7f43600693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4378001ed0 tx=0x7f4378001950 comp rx=0 tx=0).stop 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 msgr2=0x7f437c16a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 0x7f437c16a210 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f436c00c790 tx=0x7f436c007e10 comp rx=0 tx=0).stop 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 shutdown_connections 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4360066f30 0x7f43600693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f437c15a340 0x7f437c16a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f437c0ad370 0x7f437c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 --2- 172.21.15.29:0/3475364030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c159ec0 0x7f437c16a210 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 >> 172.21.15.29:0/3475364030 conn(0x7f437c0680a0 msgr2=0x7f437c0aa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 shutdown_connections 2023-10-07T10:44:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.704+0000 7f4383dff700 1 -- 172.21.15.29:0/3475364030 wait complete. 2023-10-07T10:44:47.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T10:44:47.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.916+0000 7fc3663ff700 1 Processor -- start 2023-10-07T10:44:47.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.916+0000 7fc3663ff700 1 -- start start 2023-10-07T10:44:47.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.917+0000 7fc3663ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3600ad370 0x7fc3600cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.917+0000 7fc3663ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc36015a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.917+0000 7fc3663ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3601568c0 0x7fc360156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.917+0000 7fc3663ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36014d790 con 0x7fc3600ad370 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.917+0000 7fc3663ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3600ae5d0 con 0x7fc3601568c0 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc3663ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36015a850 con 0x7fc360159ec0 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3600ad370 0x7fc3600cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3600ad370 0x7fc3600cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38924/0 (socket says 172.21.15.29:38924) 2023-10-07T10:44:47.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 -- 172.21.15.29:0/560726915 learned_addr learned my addr 172.21.15.29:0/560726915 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35f7fe700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc36015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc36499c700 1 --2- 172.21.15.29:0/560726915 >> v2:172.21.15.144:3300/0 conn(0x7fc3601568c0 0x7fc360156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 -- 172.21.15.29:0/560726915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 msgr2=0x7fc36015a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc36015a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 -- 172.21.15.29:0/560726915 >> v2:172.21.15.144:3300/0 conn(0x7fc3601568c0 msgr2=0x7fc360156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 --2- 172.21.15.29:0/560726915 >> v2:172.21.15.144:3300/0 conn(0x7fc3601568c0 0x7fc360156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 -- 172.21.15.29:0/560726915 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600ad740 con 0x7fc3600ad370 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35f7fe700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc36015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.918+0000 7fc35ffff700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc3600cc870 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fc36006a6a0 tx=0x7fc350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a32bbec88f794a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc35effd700 1 -- 172.21.15.29:0/560726915 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35000b860 con 0x7fc3600ad370 2023-10-07T10:44:47.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc35effd700 1 -- 172.21.15.29:0/560726915 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35000b9c0 con 0x7fc3600ad370 2023-10-07T10:44:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc35effd700 1 -- 172.21.15.29:0/560726915 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35000bd60 con 0x7fc3600ad370 2023-10-07T10:44:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 -- 172.21.15.29:0/560726915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 msgr2=0x7fc3600cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc3600cc870 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fc36006a6a0 tx=0x7fc350009250 comp rx=0 tx=0).stop 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 -- 172.21.15.29:0/560726915 shutdown_connections 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 --2- 172.21.15.29:0/560726915 >> v2:172.21.15.144:3300/0 conn(0x7fc3601568c0 0x7fc360156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc3600cc870 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 --2- 172.21.15.29:0/560726915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc36015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.919+0000 7fc3663ff700 1 -- 172.21.15.29:0/560726915 >> 172.21.15.29:0/560726915 conn(0x7fc3600680a0 msgr2=0x7fc3600cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.920+0000 7fc3663ff700 1 -- 172.21.15.29:0/560726915 shutdown_connections 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.920+0000 7fc3663ff700 1 -- 172.21.15.29:0/560726915 wait complete. 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.921+0000 7fc3663ff700 1 Processor -- start 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.921+0000 7fc3663ff700 1 -- start start 2023-10-07T10:44:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.922+0000 7fc3663ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc36013ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.922+0000 7fc3663ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3601568c0 0x7fc3600c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.922+0000 7fc3663ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc3600c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.922+0000 7fc3663ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600cabc0 con 0x7fc3600ad370 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.922+0000 7fc3663ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600ca730 con 0x7fc3601568c0 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.922+0000 7fc3663ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36014b4b0 con 0x7fc360159ec0 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc36013ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc36013ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38930/0 (socket says 172.21.15.29:38930) 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 -- 172.21.15.29:0/1489338973 learned_addr learned my addr 172.21.15.29:0/1489338973 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc36499c700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc3600c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35f7fe700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3601568c0 0x7fc3600c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 -- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 msgr2=0x7fc3600c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc3600c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 -- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3601568c0 msgr2=0x7fc3600c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3601568c0 0x7fc3600c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 -- 172.21.15.29:0/1489338973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500094f0 con 0x7fc3600ad370 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35f7fe700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3601568c0 0x7fc3600c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.923+0000 7fc35ffff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc36013ea80 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc3500090b0 tx=0x7fc350017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.924+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350026480 con 0x7fc3600ad370 2023-10-07T10:44:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.924+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35001f070 con 0x7fc3600ad370 2023-10-07T10:44:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.924+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350026870 con 0x7fc3600ad370 2023-10-07T10:44:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.924+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3600c9b20 con 0x7fc3600ad370 2023-10-07T10:44:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.924+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3600ca0e0 con 0x7fc3600ad370 2023-10-07T10:44:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.925+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3600caa10 con 0x7fc3600ad370 2023-10-07T10:44:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.926+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc35001a430 con 0x7fc3600ad370 2023-10-07T10:44:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.927+0000 7fc347fff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc34c066f30 0x7fc34c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.927+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fc350021070 con 0x7fc3600ad370 2023-10-07T10:44:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.928+0000 7fc35f7fe700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc34c066f30 0x7fc34c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.928+0000 7fc35f7fe700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc34c066f30 0x7fc34c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc354006f00 tx=0x7fc354008040 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:47.931+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3500537e0 con 0x7fc3600ad370 2023-10-07T10:44:48.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.184+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc36004c3c0 con 0x7fc3600ad370 2023-10-07T10:44:48.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.184+0000 7fc347fff700 1 -- 172.21.15.29:0/1489338973 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc350015070 con 0x7fc3600ad370 2023-10-07T10:44:48.185 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc34c066f30 msgr2=0x7fc34c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc34c066f30 0x7fc34c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc354006f00 tx=0x7fc354008040 comp rx=0 tx=0).stop 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 msgr2=0x7fc36013ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc36013ea80 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc3500090b0 tx=0x7fc350017000 comp rx=0 tx=0).stop 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 shutdown_connections 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc34c066f30 0x7fc34c0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc360159ec0 0x7fc3600c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3600ad370 0x7fc36013ea80 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 --2- 172.21.15.29:0/1489338973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3601568c0 0x7fc3600c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.186+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 >> 172.21.15.29:0/1489338973 conn(0x7fc3600680a0 msgr2=0x7fc360158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.187+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 shutdown_connections 2023-10-07T10:44:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:48.187+0000 7fc3663ff700 1 -- 172.21.15.29:0/1489338973 wait complete. 2023-10-07T10:44:48.206 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T10:44:48.206 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T10:44:48.206 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T10:44:48.206 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T10:44:48.206 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T10:44:48.207 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T10:44:48.207 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T10:44:48.207 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T10:44:48.207 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T10:44:48.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.207 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T10:44:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 37740sec preferred_lft 37740sec 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T10:44:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.257 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T10:44:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T10:44:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T10:44:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:44:48.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.258 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T10:44:48.326 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T10:44:48.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.330 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T10:44:48.380 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T10:44:48.380 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T10:44:48.380 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T10:44:48.380 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T10:44:48.380 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T10:44:48.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.380 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T10:44:48.381 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T10:44:48.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T10:44:48.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T10:44:48.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T10:44:48.381 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T10:44:48.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.554 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T10:44:48.602 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T10:44:48.602 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T10:44:48.602 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T10:44:48.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T10:44:48.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T10:44:48.675 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T10:44:48.722 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T10:44:48.722 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T10:44:48.722 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T10:44:48.809 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T10:44:48.809 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T10:44:48.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:44:48.951 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:44:48.952 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T10:44:48.952 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T10:44:48.967 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:44:48.967 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T10:44:48.967 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T10:44:48.968 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T10:44:48.968 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T10:44:48.988 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:44:48.988 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T10:44:48.989 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T10:44:48.989 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T10:44:48.989 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T10:44:48.989 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T10:44:48.989 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T10:44:48.989 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T10:44:48.994 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T10:44:48.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:48.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T10:44:49.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T10:44:49.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.267+0000 7fbdbf5ff700 1 Processor -- start 2023-10-07T10:44:49.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.267+0000 7fbdbf5ff700 1 -- start start 2023-10-07T10:44:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.268+0000 7fbdbf5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8164970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.268+0000 7fbdbf5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb806f4e0 con 0x7fbdb8159550 2023-10-07T10:44:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.268+0000 7fbdbf5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb80b1120 con 0x7fbdb80b2670 2023-10-07T10:44:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.268+0000 7fbdbf5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb806e8d0 con 0x7fbdb8162590 2023-10-07T10:44:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdbd39b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbdb8159550 0x7fbdb80b0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45144/0 (socket says 172.21.15.29:45144) 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdbd39b700 1 -- 172.21.15.29:0/3813324324 learned_addr learned my addr 172.21.15.29:0/3813324324 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207823457 0 0) 0x7fbdb806f4e0 con 0x7fbdb8159550 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda4003570 con 0x7fbdb8159550 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdbdb9c700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8164970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823282621 0 0) 0x7fbda4003570 con 0x7fbdb8159550 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 msgr2=0x7fbdb8164970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8164970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 >> v1:172.21.15.144:6789/0 conn(0x7fbdb80b2670 legacy=0x7fbdb80b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb806ebb0 con 0x7fbdb8159550 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.269+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fbdb4001470 con 0x7fbdb8159550 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 >> v1:172.21.15.29:6789/0 conn(0x7fbdb8159550 legacy=0x7fbdb80b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbda4003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbda4004060 0x7fbda4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda4004d50 0x7fbda40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb806e8d0 con 0x7fbda4004d50 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda400bc20 con 0x7fbda4004060 2023-10-07T10:44:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda400bde0 con 0x7fbdb8162590 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbda4003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbcb9a700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbda4004060 0x7fbda4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbdb9c700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda4004d50 0x7fbda40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbd39b700 1 -- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbda4004060 msgr2=0x7fbda4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbda4004060 0x7fbda4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbd39b700 1 -- 172.21.15.29:0/3813324324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda4004d50 msgr2=0x7fbda40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda4004d50 0x7fbda40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbd39b700 1 -- 172.21.15.29:0/3813324324 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda4003570 con 0x7fbdb8162590 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.270+0000 7fbdbdb9c700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda4004d50 0x7fbda40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbda4003a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400c2d0 tx=0x7fbdb4001d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab6316d53f55d25 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb400cba0 con 0x7fbdb8162590 2023-10-07T10:44:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb400cd20 con 0x7fbdb8162590 2023-10-07T10:44:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdaffff700 1 -- 172.21.15.29:0/3813324324 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4005b70 con 0x7fbdb8162590 2023-10-07T10:44:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 msgr2=0x7fbda4003a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/3813324324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbda4003a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400c2d0 tx=0x7fbdb4001d80 comp rx=0 tx=0).stop 2023-10-07T10:44:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.271+0000 7fbdbdb9c700 1 -- 172.21.15.29:0/3813324324 reap_dead start 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.272+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/3813324324 shutdown_connections 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.272+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/3813324324 >> 172.21.15.29:0/3813324324 conn(0x7fbdb8069190 msgr2=0x7fbdb806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.272+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/3813324324 shutdown_connections 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.272+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/3813324324 wait complete. 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.273+0000 7fbdbf5ff700 1 Processor -- start 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.273+0000 7fbdbf5ff700 1 -- start start 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.274+0000 7fbdbf5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.274+0000 7fbdbf5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb8149530 0x7fbdb81483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.274+0000 7fbdbf5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 0x7fbdb8148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.274+0000 7fbdbf5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb814aee0 con 0x7fbdb81488f0 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.274+0000 7fbdbf5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8144ac0 con 0x7fbdb8149530 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.274+0000 7fbdbf5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8144da0 con 0x7fbdb8162590 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 0x7fbdb8148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 0x7fbdb8148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38960/0 (socket says 172.21.15.29:38960) 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 -- 172.21.15.29:0/2487900045 learned_addr learned my addr 172.21.15.29:0/2487900045 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbcb9a700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb8149530 0x7fbdb81483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 -- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 msgr2=0x7fbdb8143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 -- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb8149530 msgr2=0x7fbdb81483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb8149530 0x7fbdb81483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 -- 172.21.15.29:0/2487900045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8145080 con 0x7fbdb81488f0 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbcb9a700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb8149530 0x7fbdb81483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdbdb9c700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 0x7fbdb8148d70 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00094d0 tx=0x7fbdb000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.275+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb001c070 con 0x7fbdb81488f0 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.276+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb000f040 con 0x7fbdb81488f0 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.276+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb000e040 con 0x7fbdb81488f0 2023-10-07T10:44:49.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.276+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb8149a10 con 0x7fbdb81488f0 2023-10-07T10:44:49.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.276+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb814b3a0 con 0x7fbdb81488f0 2023-10-07T10:44:49.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.277+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb804c3c0 con 0x7fbdb81488f0 2023-10-07T10:44:49.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.278+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbdb0003020 con 0x7fbdb81488f0 2023-10-07T10:44:49.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.278+0000 7fbdad7fa700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbd9c066f30 0x7fbd9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.279+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fbdb0084b00 con 0x7fbdb81488f0 2023-10-07T10:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.279+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbd9c066f30 0x7fbd9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.280+0000 7fbdbd39b700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbd9c066f30 0x7fbd9c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4007c00 tx=0x7fbdb4015000 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.283+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbdb0086e60 con 0x7fbdb81488f0 2023-10-07T10:44:49.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.517+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb80683d0 con 0x7fbd9c066f30 2023-10-07T10:44:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.519+0000 7fbdad7fa700 1 -- 172.21.15.29:0/2487900045 <== mgr.7310 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbdb80683d0 con 0x7fbd9c066f30 2023-10-07T10:44:49.519 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T10:44:49.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:44:49.520 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T10:44:49.520 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:44:49.520 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T10:44:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbd9c066f30 msgr2=0x7fbd9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbd9c066f30 0x7fbd9c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4007c00 tx=0x7fbdb4015000 comp rx=0 tx=0).stop 2023-10-07T10:44:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 msgr2=0x7fbdb8148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 0x7fbdb8148d70 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00094d0 tx=0x7fbdb000b040 comp rx=0 tx=0).stop 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 shutdown_connections 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fbd9c066f30 0x7fbd9c0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb81488f0 0x7fbdb8148d70 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb8162590 0x7fbdb8143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 --2- 172.21.15.29:0/2487900045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb8149530 0x7fbdb81483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 >> 172.21.15.29:0/2487900045 conn(0x7fbdb8069190 msgr2=0x7fbdb814ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 shutdown_connections 2023-10-07T10:44:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.521+0000 7fbdbf5ff700 1 -- 172.21.15.29:0/2487900045 wait complete. 2023-10-07T10:44:49.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T10:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.737+0000 7fe3c6dff700 1 Processor -- start 2023-10-07T10:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.737+0000 7fe3c6dff700 1 -- start start 2023-10-07T10:44:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3c6dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c00ae8a0 0x7fe3c0158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3c6dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00acb40 con 0x7fe3c015ad30 2023-10-07T10:44:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3c6dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00abf30 con 0x7fe3c015a960 2023-10-07T10:44:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3c6dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c0158c60 con 0x7fe3c00ae8a0 2023-10-07T10:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3bffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c00ae8a0 0x7fe3c0158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3c539c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe3c015ad30 0x7fe3c00b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45150/0 (socket says 172.21.15.29:45150) 2023-10-07T10:44:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.738+0000 7fe3c539c700 1 -- 172.21.15.29:0/1268978721 learned_addr learned my addr 172.21.15.29:0/1268978721 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.739+0000 7fe3bf7fe700 1 -- 172.21.15.29:0/1268978721 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803243361 0 0) 0x7fe3c00acb40 con 0x7fe3c015ad30 2023-10-07T10:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.739+0000 7fe3bffff700 1 -- 172.21.15.29:0/1268978721 >> v1:172.21.15.144:6789/0 conn(0x7fe3c015a960 legacy=0x7fe3c00ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.739+0000 7fe3bffff700 1 -- 172.21.15.29:0/1268978721 >> v1:172.21.15.29:6789/0 conn(0x7fe3c015ad30 legacy=0x7fe3c00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.739+0000 7fe3bffff700 1 -- 172.21.15.29:0/1268978721 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00ac210 con 0x7fe3c00ae8a0 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.739+0000 7fe3bffff700 1 --2- 172.21.15.29:0/1268978721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c00ae8a0 0x7fe3c0158e20 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0098e0 tx=0x7fe3ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a63366e258906a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.739+0000 7fe3bf7fe700 1 -- 172.21.15.29:0/1268978721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac009e80 con 0x7fe3c00ae8a0 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3bf7fe700 1 -- 172.21.15.29:0/1268978721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac00b860 con 0x7fe3c00ae8a0 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3bf7fe700 1 -- 172.21.15.29:0/1268978721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac00bbe0 con 0x7fe3c00ae8a0 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 -- 172.21.15.29:0/1268978721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c00ae8a0 msgr2=0x7fe3c0158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/1268978721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c00ae8a0 0x7fe3c0158e20 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0098e0 tx=0x7fe3ac0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 -- 172.21.15.29:0/1268978721 shutdown_connections 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/1268978721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c00ae8a0 0x7fe3c0158e20 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 -- 172.21.15.29:0/1268978721 >> 172.21.15.29:0/1268978721 conn(0x7fe3c0067610 msgr2=0x7fe3c0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 -- 172.21.15.29:0/1268978721 shutdown_connections 2023-10-07T10:44:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.740+0000 7fe3c6dff700 1 -- 172.21.15.29:0/1268978721 wait complete. 2023-10-07T10:44:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.742+0000 7fe3c6dff700 1 Processor -- start 2023-10-07T10:44:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.742+0000 7fe3c6dff700 1 -- start start 2023-10-07T10:44:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c6dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3c00ae8a0 0x7fe3c00cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c6dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3c015a960 0x7fe3c014afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c6dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 0x7fe3c0151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c6dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c0157370 con 0x7fe3c015ad30 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c6dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c01520f0 con 0x7fe3c015a960 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c6dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00ca3e0 con 0x7fe3c00ae8a0 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c539c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 0x7fe3c0151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c539c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 0x7fe3c0151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38976/0 (socket says 172.21.15.29:38976) 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c539c700 1 -- 172.21.15.29:0/3878815556 learned_addr learned my addr 172.21.15.29:0/3878815556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3bffff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3c015a960 0x7fe3c014afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.743+0000 7fe3c4b9b700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3c00ae8a0 0x7fe3c00cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c539c700 1 -- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3c00ae8a0 msgr2=0x7fe3c00cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c539c700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3c00ae8a0 0x7fe3c00cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c539c700 1 -- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3c015a960 msgr2=0x7fe3c014afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c539c700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3c015a960 0x7fe3c014afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c539c700 1 -- 172.21.15.29:0/3878815556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac009420 con 0x7fe3c015ad30 2023-10-07T10:44:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c539c700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 0x7fe3c0151510 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800cbf0 tx=0x7fe3b80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800a8b0 con 0x7fe3c015ad30 2023-10-07T10:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b800aa10 con 0x7fe3c015ad30 2023-10-07T10:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c00cb6a0 con 0x7fe3c015ad30 2023-10-07T10:44:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800add0 con 0x7fe3c015ad30 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.744+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c00ca960 con 0x7fe3c015ad30 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.745+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c004c3c0 con 0x7fe3c015ad30 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.746+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe3b8007300 con 0x7fe3c015ad30 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.747+0000 7fe3bcff9700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe3a4066f30 0x7fe3a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.747+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fe3b8084a70 con 0x7fe3c015ad30 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.747+0000 7fe3c4b9b700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe3a4066f30 0x7fe3a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.748+0000 7fe3c4b9b700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe3a4066f30 0x7fe3a40693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4003ac0 tx=0x7fe3b4007ea0 comp rx=0 tx=0).ready entity=mgr.7310 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:49.751+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3b8051aa0 con 0x7fe3c015ad30 2023-10-07T10:44:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.011+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe3c00b19e0 con 0x7fe3c015ad30 2023-10-07T10:44:50.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.040+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 68) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fe3b8084d50 con 0x7fe3c015ad30 2023-10-07T10:44:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.041+0000 7fe3bcff9700 1 -- 172.21.15.29:0/3878815556 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v68) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe3b8051aa0 con 0x7fe3c015ad30 2023-10-07T10:44:50.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe3a4066f30 msgr2=0x7fe3a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe3a4066f30 0x7fe3a40693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4003ac0 tx=0x7fe3b4007ea0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 msgr2=0x7fe3c0151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 0x7fe3c0151510 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800cbf0 tx=0x7fe3b80019b0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 shutdown_connections 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe3a4066f30 0x7fe3a40693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015ad30 0x7fe3c0151510 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3c00ae8a0 0x7fe3c00cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 --2- 172.21.15.29:0/3878815556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3c015a960 0x7fe3c014afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.044+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 >> 172.21.15.29:0/3878815556 conn(0x7fe3c0067610 msgr2=0x7fe3c014f360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.045+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 shutdown_connections 2023-10-07T10:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.045+0000 7fe3c6dff700 1 -- 172.21.15.29:0/3878815556 wait complete. 2023-10-07T10:44:50.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T10:44:50.256 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:50.255+0000 7fdc278561c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:44:50.257 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:50.257+0000 7f3c2c8ef1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:44:50.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.268+0000 7fea349ff700 1 Processor -- start 2023-10-07T10:44:50.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.268+0000 7fea349ff700 1 -- start start 2023-10-07T10:44:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea349ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fea30159d90 0x7fea30160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea349ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30157790 con 0x7fea30153cc0 2023-10-07T10:44:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea349ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30157a70 con 0x7fea3014c8d0 2023-10-07T10:44:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea349ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300b0950 con 0x7fea30159d90 2023-10-07T10:44:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea2dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fea30153cc0 0x7fea30159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45160/0 (socket says 172.21.15.29:45160) 2023-10-07T10:44:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea2dd9b700 1 -- 172.21.15.29:0/2008741424 learned_addr learned my addr 172.21.15.29:0/2008741424 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.270+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/2008741424 >> v2:172.21.15.144:3300/0 conn(0x7fea30159d90 0x7fea30160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92521182 0 0) 0x7fea30157790 con 0x7fea30153cc0 2023-10-07T10:44:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea18004680 con 0x7fea30153cc0 2023-10-07T10:44:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898023291 0 0) 0x7fea30157a70 con 0x7fea3014c8d0 2023-10-07T10:44:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea30157790 con 0x7fea3014c8d0 2023-10-07T10:44:50.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2ed9d700 1 -- 172.21.15.29:0/2008741424 >> v1:172.21.15.144:6789/0 conn(0x7fea3014c8d0 legacy=0x7fea300afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2ed9d700 1 -- 172.21.15.29:0/2008741424 >> v1:172.21.15.29:6789/0 conn(0x7fea30153cc0 legacy=0x7fea30159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2ed9d700 1 -- 172.21.15.29:0/2008741424 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30157d50 con 0x7fea30159d90 2023-10-07T10:44:50.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/2008741424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea30159d90 0x7fea30160a30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fea28009e60 tx=0x7fea28009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22ef3ac6043af65e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:50.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800e070 con 0x7fea30159d90 2023-10-07T10:44:50.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.271+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2800bab0 con 0x7fea30159d90 2023-10-07T10:44:50.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea2d59a700 1 -- 172.21.15.29:0/2008741424 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800be30 con 0x7fea30159d90 2023-10-07T10:44:50.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 -- 172.21.15.29:0/2008741424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea30159d90 msgr2=0x7fea30160a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 --2- 172.21.15.29:0/2008741424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea30159d90 0x7fea30160a30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fea28009e60 tx=0x7fea28009780 comp rx=0 tx=0).stop 2023-10-07T10:44:50.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 -- 172.21.15.29:0/2008741424 shutdown_connections 2023-10-07T10:44:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 --2- 172.21.15.29:0/2008741424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea30159d90 0x7fea30160a30 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 -- 172.21.15.29:0/2008741424 >> 172.21.15.29:0/2008741424 conn(0x7fea30067610 msgr2=0x7fea30067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 -- 172.21.15.29:0/2008741424 shutdown_connections 2023-10-07T10:44:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.272+0000 7fea349ff700 1 -- 172.21.15.29:0/2008741424 wait complete. 2023-10-07T10:44:50.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.274+0000 7fea349ff700 1 Processor -- start 2023-10-07T10:44:50.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.274+0000 7fea349ff700 1 -- start start 2023-10-07T10:44:50.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea349ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea3014c8d0 0x7fea3016ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea349ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 0x7fea30151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea349ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159d90 0x7fea30152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea349ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30151690 con 0x7fea30153cc0 2023-10-07T10:44:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea349ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30150a80 con 0x7fea3014c8d0 2023-10-07T10:44:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea349ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30150d60 con 0x7fea30159d90 2023-10-07T10:44:50.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea2dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 0x7fea30151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea2dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 0x7fea30151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39010/0 (socket says 172.21.15.29:39010) 2023-10-07T10:44:50.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea2dd9b700 1 -- 172.21.15.29:0/761063746 learned_addr learned my addr 172.21.15.29:0/761063746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:50.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea2e59c700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea3014c8d0 0x7fea3016ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.275+0000 7fea2ed9d700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159d90 0x7fea30152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea2dd9b700 1 -- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159d90 msgr2=0x7fea30152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea2dd9b700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159d90 0x7fea30152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea2dd9b700 1 -- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea3014c8d0 msgr2=0x7fea3016ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea2dd9b700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea3014c8d0 0x7fea3016ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea2dd9b700 1 -- 172.21.15.29:0/761063746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28009a20 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea2dd9b700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 0x7fea30151b20 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fea2400c790 tx=0x7fea24007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24005b30 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea24005cd0 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea30151040 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24007320 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.276+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea30150730 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.278+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30151800 con 0x7fea30153cc0 2023-10-07T10:44:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.278+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fea24012400 con 0x7fea30153cc0 2023-10-07T10:44:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.280+0000 7fea1e7fc700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 0x7fea10069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.280+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fea24083b90 con 0x7fea30153cc0 2023-10-07T10:44:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.280+0000 7fea2e59c700 1 -- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 msgr2=0x7fea10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.280+0000 7fea2e59c700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 0x7fea10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.284+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea2404b5a0 con 0x7fea30153cc0 2023-10-07T10:44:50.434 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:50.433+0000 7f3c2c8ef1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:44:50.470 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:50.469+0000 7fdc278561c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:44:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.481+0000 7fea2e59c700 1 -- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 msgr2=0x7fea10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.481+0000 7fea2e59c700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 0x7fea10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.535+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fea3004c3c0 con 0x7fea30153cc0 2023-10-07T10:44:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.539+0000 7fea1e7fc700 1 -- 172.21.15.29:0/761063746 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fea240170d0 con 0x7fea30153cc0 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 msgr2=0x7fea10069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 0x7fea10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 msgr2=0x7fea30151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 0x7fea30151b20 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fea2400c790 tx=0x7fea24007e80 comp rx=0 tx=0).stop 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 shutdown_connections 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fea10066f80 0x7fea10069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea30159d90 0x7fea30152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea3014c8d0 0x7fea3016ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 --2- 172.21.15.29:0/761063746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea30153cc0 0x7fea30151b20 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.541+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 >> 172.21.15.29:0/761063746 conn(0x7fea30067610 msgr2=0x7fea300679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.542+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 shutdown_connections 2023-10-07T10:44:50.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.542+0000 7fea349ff700 1 -- 172.21.15.29:0/761063746 wait complete. 2023-10-07T10:44:50.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T10:44:50.750 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:50.749+0000 7f3c2c8ef1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:44:50.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.762+0000 7fcb93bff700 1 Processor -- start 2023-10-07T10:44:50.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.762+0000 7fcb93bff700 1 -- start start 2023-10-07T10:44:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.763+0000 7fcb93bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb8c15ade0 0x7fcb8c0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.763+0000 7fcb93bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c154090 con 0x7fcb8c153cc0 2023-10-07T10:44:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.763+0000 7fcb93bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c159f60 con 0x7fcb8c0b0540 2023-10-07T10:44:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.763+0000 7fcb93bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b09e0 con 0x7fcb8c15ade0 2023-10-07T10:44:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb9199b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcb8c153cc0 0x7fcb8c0b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45174/0 (socket says 172.21.15.29:45174) 2023-10-07T10:44:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb9199b700 1 -- 172.21.15.29:0/358356437 learned_addr learned my addr 172.21.15.29:0/358356437 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb9119a700 1 --2- 172.21.15.29:0/358356437 >> v2:172.21.15.144:3300/0 conn(0x7fcb8c15ade0 0x7fcb8c0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb90999700 1 -- 172.21.15.29:0/358356437 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667463855 0 0) 0x7fcb8c154090 con 0x7fcb8c153cc0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb90999700 1 -- 172.21.15.29:0/358356437 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb78003570 con 0x7fcb8c153cc0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb9119a700 1 -- 172.21.15.29:0/358356437 >> v1:172.21.15.144:6789/0 conn(0x7fcb8c0b0540 legacy=0x7fcb8c15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb9119a700 1 -- 172.21.15.29:0/358356437 >> v1:172.21.15.29:6789/0 conn(0x7fcb8c153cc0 legacy=0x7fcb8c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.764+0000 7fcb9119a700 1 -- 172.21.15.29:0/358356437 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c15a240 con 0x7fcb8c15ade0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.765+0000 7fcb9119a700 1 --2- 172.21.15.29:0/358356437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c0b0000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0098e0 tx=0x7fcb7c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee7dd3fad3dc7e53 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.765+0000 7fcb90999700 1 -- 172.21.15.29:0/358356437 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00b640 con 0x7fcb8c15ade0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.765+0000 7fcb90999700 1 -- 172.21.15.29:0/358356437 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c00b7a0 con 0x7fcb8c15ade0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.765+0000 7fcb90999700 1 -- 172.21.15.29:0/358356437 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00bb20 con 0x7fcb8c15ade0 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.765+0000 7fcb93bff700 1 -- 172.21.15.29:0/358356437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c15ade0 msgr2=0x7fcb8c0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.765+0000 7fcb93bff700 1 --2- 172.21.15.29:0/358356437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c0b0000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0098e0 tx=0x7fcb7c0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.766+0000 7fcb93bff700 1 -- 172.21.15.29:0/358356437 shutdown_connections 2023-10-07T10:44:50.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.766+0000 7fcb93bff700 1 --2- 172.21.15.29:0/358356437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c0b0000 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.766+0000 7fcb93bff700 1 -- 172.21.15.29:0/358356437 >> 172.21.15.29:0/358356437 conn(0x7fcb8c067610 msgr2=0x7fcb8c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:50.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.766+0000 7fcb93bff700 1 -- 172.21.15.29:0/358356437 shutdown_connections 2023-10-07T10:44:50.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.766+0000 7fcb93bff700 1 -- 172.21.15.29:0/358356437 wait complete. 2023-10-07T10:44:50.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.767+0000 7fcb93bff700 1 Processor -- start 2023-10-07T10:44:50.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.767+0000 7fcb93bff700 1 -- start start 2023-10-07T10:44:50.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.768+0000 7fcb93bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b0540 0x7fcb8c151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.768+0000 7fcb93bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 0x7fcb8c0c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.768+0000 7fcb93bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c16d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.768+0000 7fcb93bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0c8760 con 0x7fcb8c15ade0 2023-10-07T10:44:50.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.768+0000 7fcb93bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0ce690 con 0x7fcb8c153cc0 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.768+0000 7fcb93bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c1556e0 con 0x7fcb8c0b0540 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9219c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c16d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9119a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 0x7fcb8c0c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9199b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b0540 0x7fcb8c151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9119a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 0x7fcb8c0c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:40354/0 (socket says 172.21.15.29:40354) 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9219c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c16d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39026/0 (socket says 172.21.15.29:39026) 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9119a700 1 -- 172.21.15.29:0/562689072 learned_addr learned my addr 172.21.15.29:0/562689072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9199b700 1 -- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 msgr2=0x7fcb8c0c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9199b700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 0x7fcb8c0c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9199b700 1 -- 172.21.15.29:0/562689072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 msgr2=0x7fcb8c16d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9199b700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c16d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9199b700 1 -- 172.21.15.29:0/562689072 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c009420 con 0x7fcb8c0b0540 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9119a700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 0x7fcb8c0c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.769+0000 7fcb9219c700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c16d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.770+0000 7fcb9199b700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b0540 0x7fcb8c151fc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c790 tx=0x7fcb88007bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.770+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880072e0 con 0x7fcb8c0b0540 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.770+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88007460 con 0x7fcb8c0b0540 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.770+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880057f0 con 0x7fcb8c0b0540 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.770+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c1559c0 con 0x7fcb8c0b0540 2023-10-07T10:44:50.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.770+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c0ceb80 con 0x7fcb8c0b0540 2023-10-07T10:44:50.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.771+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c0c8500 con 0x7fcb8c0b0540 2023-10-07T10:44:50.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.772+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 68) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fcb88005c40 con 0x7fcb8c0b0540 2023-10-07T10:44:50.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.773+0000 7fcb81ffb700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 0x7fcb70069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:50.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.773+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fcb88083a90 con 0x7fcb8c0b0540 2023-10-07T10:44:50.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.773+0000 7fcb9119a700 1 -- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 msgr2=0x7fcb70069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:50.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.773+0000 7fcb9119a700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 0x7fcb70069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:50.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.778+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb8804b5a0 con 0x7fcb8c0b0540 2023-10-07T10:44:50.805 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:50.804+0000 7fdc278561c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.974+0000 7fcb9119a700 1 -- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 msgr2=0x7fcb70069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.974+0000 7fcb9119a700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 0x7fcb70069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:50.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.982+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcb8c04c3c0 con 0x7fcb8c0b0540 2023-10-07T10:44:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.989+0000 7fcb81ffb700 1 -- 172.21.15.29:0/562689072 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcb88017020 con 0x7fcb8c0b0540 2023-10-07T10:44:50.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 msgr2=0x7fcb70069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 0x7fcb70069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b0540 msgr2=0x7fcb8c151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b0540 0x7fcb8c151fc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c790 tx=0x7fcb88007bc0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 shutdown_connections 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcb70066f80 0x7fcb70069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb8c15ade0 0x7fcb8c16d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb8c0b0540 0x7fcb8c151fc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 --2- 172.21.15.29:0/562689072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb8c153cc0 0x7fcb8c0c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 >> 172.21.15.29:0/562689072 conn(0x7fcb8c067610 msgr2=0x7fcb8c15df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.991+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 shutdown_connections 2023-10-07T10:44:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:50.992+0000 7fcb93bff700 1 -- 172.21.15.29:0/562689072 wait complete. 2023-10-07T10:44:51.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:51.010 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T10:44:51.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1h 2023-10-07T10:44:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.269+0000 7f22aebff700 1 Processor -- start 2023-10-07T10:44:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.269+0000 7f22aebff700 1 -- start start 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22aebff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157a20 0x7f22a80b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22aebff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22a80cb4d0 0x7f22a80b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22aebff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80aeae0 con 0x7f22a8157650 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22aebff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a80ca430 con 0x7f22a80cb4d0 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22aebff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80ca710 con 0x7f22a8157a20 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22ad19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22a80cb4d0 0x7f22a80b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.270+0000 7f22ad19c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22a80cb4d0 0x7f22a80b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39034/0 (socket says 172.21.15.29:39034) 2023-10-07T10:44:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22ad19c700 1 -- 172.21.15.29:0/1293185597 learned_addr learned my addr 172.21.15.29:0/1293185597 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22a7fff700 1 --2- 172.21.15.29:0/1293185597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157a20 0x7f22a80b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22ad19c700 1 -- 172.21.15.29:0/1293185597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157a20 msgr2=0x7f22a80b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22ad19c700 1 --2- 172.21.15.29:0/1293185597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157a20 0x7f22a80b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22ad19c700 1 -- 172.21.15.29:0/1293185597 >> v1:172.21.15.29:6789/0 conn(0x7f22a8157650 legacy=0x7f22a80ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22ad19c700 1 -- 172.21.15.29:0/1293185597 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80ca9f0 con 0x7f22a80cb4d0 2023-10-07T10:44:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22ad19c700 1 --2- 172.21.15.29:0/1293185597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a80cb4d0 0x7f22a80b17d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009f20 tx=0x7f22a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bf3b6146ac70c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:51.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.271+0000 7f22a77fe700 1 -- 172.21.15.29:0/1293185597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a000e070 con 0x7f22a80cb4d0 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22a77fe700 1 -- 172.21.15.29:0/1293185597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a000b9d0 con 0x7f22a80cb4d0 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22a77fe700 1 -- 172.21.15.29:0/1293185597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00092e0 con 0x7f22a80cb4d0 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 -- 172.21.15.29:0/1293185597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a80cb4d0 msgr2=0x7f22a80b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 --2- 172.21.15.29:0/1293185597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a80cb4d0 0x7f22a80b17d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009f20 tx=0x7f22a0009840 comp rx=0 tx=0).stop 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 -- 172.21.15.29:0/1293185597 shutdown_connections 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 --2- 172.21.15.29:0/1293185597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a80cb4d0 0x7f22a80b17d0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 --2- 172.21.15.29:0/1293185597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157a20 0x7f22a80b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 -- 172.21.15.29:0/1293185597 >> 172.21.15.29:0/1293185597 conn(0x7f22a8067610 msgr2=0x7f22a8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 -- 172.21.15.29:0/1293185597 shutdown_connections 2023-10-07T10:44:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.272+0000 7f22aebff700 1 -- 172.21.15.29:0/1293185597 wait complete. 2023-10-07T10:44:51.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.274+0000 7f22aebff700 1 Processor -- start 2023-10-07T10:44:51.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.274+0000 7f22aebff700 1 -- start start 2023-10-07T10:44:51.276 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:51.274+0000 7f3c2c8ef1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:44:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22aebff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22a80cb4d0 0x7f22a8141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22aebff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157650 0x7f22a815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22aebff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 0x7f22a8151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22aebff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a815c750 con 0x7f22a8157a20 2023-10-07T10:44:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22aebff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a815ca30 con 0x7f22a80cb4d0 2023-10-07T10:44:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22aebff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a815cd10 con 0x7f22a8157650 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22ad19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 0x7f22a8151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22ad19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 0x7f22a8151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39040/0 (socket says 172.21.15.29:39040) 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22ad19c700 1 -- 172.21.15.29:0/360671499 learned_addr learned my addr 172.21.15.29:0/360671499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22a80cb4d0 0x7f22a8141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.275+0000 7f22a7fff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157650 0x7f22a815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22ad19c700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157650 msgr2=0x7f22a815bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22ad19c700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157650 0x7f22a815bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22ad19c700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22a80cb4d0 msgr2=0x7f22a8141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22ad19c700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22a80cb4d0 0x7f22a8141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22ad19c700 1 -- 172.21.15.29:0/360671499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a0009ae0 con 0x7f22a8157a20 2023-10-07T10:44:51.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22ad19c700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 0x7f22a8151080 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f22a000b610 tx=0x7f22a0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0008ad0 con 0x7f22a8157a20 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a815cff0 con 0x7f22a8157a20 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a0008c30 con 0x7f22a8157a20 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.276+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0026460 con 0x7f22a8157a20 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.277+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a8141ed0 con 0x7f22a8157a20 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.278+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a80718b0 con 0x7f22a8157a20 2023-10-07T10:44:51.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.279+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f22a0003040 con 0x7f22a8157a20 2023-10-07T10:44:51.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.279+0000 7f22a4ff9700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:51.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.279+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f22a00865b0 con 0x7f22a8157a20 2023-10-07T10:44:51.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.280+0000 7f22ac99b700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:51.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.280+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:51.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.284+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22a00535e0 con 0x7f22a8157a20 2023-10-07T10:44:51.361 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:51.360+0000 7fdc278561c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:44:51.443 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:51.443+0000 7f3c2c8ef1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:44:51.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.481+0000 7f22ac99b700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:51.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.481+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.513+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f22a80cacd0 con 0x7f228c066f80 2023-10-07T10:44:51.536 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:51.536+0000 7fdc278561c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:44:51.856 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:51.857+0000 7f3c2c8ef1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:44:51.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.882+0000 7f22ac99b700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:51.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:51.882+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T10:44:51.962 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:51.961+0000 7fdc278561c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:44:52.627 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:52.627+0000 7f3c2c8ef1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:44:52.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:52.683+0000 7f22ac99b700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:52.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:52.683+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T10:44:52.713 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:52.713+0000 7f3c2c8ef1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:44:52.761 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:52.760+0000 7fdc278561c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:44:52.851 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:52.850+0000 7fdc278561c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:44:52.866 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:52.866+0000 7f3c2c8ef1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:44:52.952 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:52.952+0000 7f3c2c8ef1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:44:53.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.010+0000 7fdc278561c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:44:53.101 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.100+0000 7fdc278561c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:44:53.169 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:53.169+0000 7f3c2c8ef1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:44:53.265 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:53.264+0000 7f3c2c8ef1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:44:53.319 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.319+0000 7fdc278561c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:44:53.359 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:53.359+0000 7f3c2c8ef1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:44:53.418 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.417+0000 7fdc278561c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:44:53.453 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:53.453+0000 7f3c2c8ef1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:44:53.516 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.515+0000 7fdc278561c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:44:53.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.613+0000 7fdc278561c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:44:53.638 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:53.638+0000 7f3c2c8ef1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:44:53.735 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:53.734+0000 7f3c2c8ef1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:44:53.807 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.807+0000 7fdc278561c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:44:53.907 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:53.906+0000 7fdc278561c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:44:54.074 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:54.074+0000 7f3c2c8ef1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:44:54.162 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:54.162+0000 7f3c2c8ef1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:54.255 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:54.254+0000 7fdc278561c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:44:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:54.285+0000 7f22ac99b700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:54.285+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T10:44:54.347 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:54.346+0000 7fdc278561c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:54.698 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:54.698+0000 7f3c2c8ef1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:44:54.881 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:54.881+0000 7f3c2c8ef1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:44:54.905 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:54.904+0000 7fdc278561c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:44:55.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:55.094+0000 7fdc278561c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:44:55.254 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:55.255+0000 7f3c2c8ef1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:55.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:55.475+0000 7fdc278561c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:55.673 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:55.674+0000 7f3c2c8ef1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:44:55.855 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:55.855+0000 7f3c2c8ef1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:55.911 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:55.911+0000 7fdc278561c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:44:56.040 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.040+0000 7f3c2c8ef1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:44:56.098 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:56.097+0000 7fdc278561c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:44:56.150 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.150+0000 7f3c2c8ef1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:56.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:56.160+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 69) v1 ==== 84468+0+0 (secure 0 0 0) 0x7f22a00539b0 con 0x7f22a8157a20 2023-10-07T10:44:56.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:56.160+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:56.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:56.160+0000 7f22a4ff9700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:56.288 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3ba0ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.288 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3ba0ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3ba0ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3ba0ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3ba0ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3b9def1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3b9def1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3b9def1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3b9def1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:56.288+0000 7f3b9def1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:44:56.306 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:56.305+0000 7fdc278561c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:44:56.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:56.417+0000 7fdc278561c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:44:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.166+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 70) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f22a0056790 con 0x7f22a8157a20 2023-10-07T10:44:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.167+0000 7f22a4ff9700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.167+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f22a80cacd0 con 0x7f228c066f80 2023-10-07T10:44:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.169+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.172+0000 7f22ac99b700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f229c0012a0 tx=0x7f229c007ec0 comp rx=0 tx=0).ready entity=mgr.8374 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:57.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.235+0000 7f22a4ff9700 1 -- 172.21.15.29:0/360671499 <== mgr.8374 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f22a80cacd0 con 0x7f228c066f80 2023-10-07T10:44:57.236 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-07T10:44:57.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.237+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 msgr2=0x7f228c069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.237+0000 7f22aebff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f229c0012a0 tx=0x7f229c007ec0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.237+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 msgr2=0x7f22a8151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.237+0000 7f22aebff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 0x7f22a8151080 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f22a000b610 tx=0x7f22a0018000 comp rx=0 tx=0).stop 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 shutdown_connections 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f228c066f80 0x7f228c069430 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a8157a20 0x7f22a8151080 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22a80cb4d0 0x7f22a8141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 --2- 172.21.15.29:0/360671499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22a8157650 0x7f22a815bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 >> 172.21.15.29:0/360671499 conn(0x7f22a8067610 msgr2=0x7f22a80b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 shutdown_connections 2023-10-07T10:44:57.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.238+0000 7f22aebff700 1 -- 172.21.15.29:0/360671499 wait complete. 2023-10-07T10:44:57.259 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-07T10:44:57.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-10-07T10:44:57.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.461+0000 7fd9d13ff700 1 Processor -- start 2023-10-07T10:44:57.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.461+0000 7fd9d13ff700 1 -- start start 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9d13ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9cc0afeb0 0x7fd9cc0cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9d13ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0accf0 0x7fd9cc0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9d13ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9cc0af4c0 0x7fd9cc15a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9d13ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9cc15a140 con 0x7fd9cc0af4c0 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9d13ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9cc0adca0 con 0x7fd9cc0afeb0 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9d13ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0ad160 con 0x7fd9cc0accf0 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9cb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9cc0af4c0 0x7fd9cc15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9cb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9cc0af4c0 0x7fd9cc15a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59784/0 (socket says 172.21.15.29:59784) 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.462+0000 7fd9cb7fe700 1 -- 172.21.15.29:0/2139085171 learned_addr learned my addr 172.21.15.29:0/2139085171 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:57.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9ca7fc700 1 --2- 172.21.15.29:0/2139085171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0accf0 0x7fd9cc0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9caffd700 1 --2- 172.21.15.29:0/2139085171 >> v2:172.21.15.144:3300/0 conn(0x7fd9cc0afeb0 0x7fd9cc0cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9cb7fe700 1 -- 172.21.15.29:0/2139085171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0accf0 msgr2=0x7fd9cc0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9cb7fe700 1 --2- 172.21.15.29:0/2139085171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0accf0 0x7fd9cc0aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9cb7fe700 1 -- 172.21.15.29:0/2139085171 >> v2:172.21.15.144:3300/0 conn(0x7fd9cc0afeb0 msgr2=0x7fd9cc0cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9cb7fe700 1 --2- 172.21.15.29:0/2139085171 >> v2:172.21.15.144:3300/0 conn(0x7fd9cc0afeb0 0x7fd9cc0cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9cb7fe700 1 -- 172.21.15.29:0/2139085171 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc0ad400 con 0x7fd9cc0af4c0 2023-10-07T10:44:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9cb7fe700 1 --2- 172.21.15.29:0/2139085171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc15a590 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0685f0 tx=0x7fd9c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c1a57b57a9c835 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:57.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.463+0000 7fd9c9ffb700 1 -- 172.21.15.29:0/2139085171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40092e0 con 0x7fd9cc0af4c0 2023-10-07T10:44:57.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9c9ffb700 1 -- 172.21.15.29:0/2139085171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c400b970 con 0x7fd9cc0af4c0 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9c9ffb700 1 -- 172.21.15.29:0/2139085171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c400bd30 con 0x7fd9cc0af4c0 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 -- 172.21.15.29:0/2139085171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0af4c0 msgr2=0x7fd9cc15a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/2139085171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc15a590 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0685f0 tx=0x7fd9c4009840 comp rx=0 tx=0).stop 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 -- 172.21.15.29:0/2139085171 shutdown_connections 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/2139085171 >> v2:172.21.15.144:3300/0 conn(0x7fd9cc0afeb0 0x7fd9cc0cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/2139085171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc15a590 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/2139085171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0accf0 0x7fd9cc0aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 -- 172.21.15.29:0/2139085171 >> 172.21.15.29:0/2139085171 conn(0x7fd9cc067610 msgr2=0x7fd9cc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 -- 172.21.15.29:0/2139085171 shutdown_connections 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.464+0000 7fd9d13ff700 1 -- 172.21.15.29:0/2139085171 wait complete. 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.466+0000 7fd9d13ff700 1 Processor -- start 2023-10-07T10:44:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.466+0000 7fd9d13ff700 1 -- start start 2023-10-07T10:44:57.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9d13ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 0x7fd9cc154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9d13ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9d13ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0afeb0 0x7fd9cc0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9d13ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1565f0 con 0x7fd9cc0accf0 2023-10-07T10:44:57.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9d13ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0c9f30 con 0x7fd9cc0af4c0 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9d13ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc156840 con 0x7fd9cc0afeb0 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 0x7fd9cc154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 0x7fd9cc154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59798/0 (socket says 172.21.15.29:59798) 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 -- 172.21.15.29:0/1169746977 learned_addr learned my addr 172.21.15.29:0/1169746977 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9ca7fc700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9cb7fe700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0afeb0 0x7fd9cc0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 -- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0afeb0 msgr2=0x7fd9cc0ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0afeb0 0x7fd9cc0ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 -- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9cc0af4c0 msgr2=0x7fd9cc143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.467+0000 7fd9caffd700 1 -- 172.21.15.29:0/1169746977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c4009ae0 con 0x7fd9cc0accf0 2023-10-07T10:44:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.468+0000 7fd9caffd700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 0x7fd9cc154640 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00ec80 tx=0x7fd9bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.468+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc00c850 con 0x7fd9cc0accf0 2023-10-07T10:44:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.468+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc00c9b0 con 0x7fd9cc0accf0 2023-10-07T10:44:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.468+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc00cd50 con 0x7fd9cc0accf0 2023-10-07T10:44:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.468+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc156b20 con 0x7fd9cc0accf0 2023-10-07T10:44:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.468+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc154b80 con 0x7fd9cc0accf0 2023-10-07T10:44:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.469+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc0ca210 con 0x7fd9cc0accf0 2023-10-07T10:44:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.470+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fd9bc013680 con 0x7fd9cc0accf0 2023-10-07T10:44:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.471+0000 7fd9b2ffd700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066dd0 0x7fd9b8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.471+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fd9bc084710 con 0x7fd9cc0accf0 2023-10-07T10:44:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.472+0000 7fd9ca7fc700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066dd0 0x7fd9b8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.472+0000 7fd9ca7fc700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066dd0 0x7fd9b8069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0caf00 tx=0x7fd9c00091a0 comp rx=0 tx=0).ready entity=mgr.8374 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:57.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.475+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9bc051990 con 0x7fd9cc0accf0 2023-10-07T10:44:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.706+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9cc0718b0 con 0x7fd9b8066dd0 2023-10-07T10:44:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.768+0000 7fd9b2ffd700 1 -- 172.21.15.29:0/1169746977 <== mgr.8374 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd9cc0718b0 con 0x7fd9b8066dd0 2023-10-07T10:44:57.768 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-07T10:44:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066dd0 msgr2=0x7fd9b8069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066dd0 0x7fd9b8069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0caf00 tx=0x7fd9c00091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 msgr2=0x7fd9cc154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 0x7fd9cc154640 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00ec80 tx=0x7fd9bc0091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 shutdown_connections 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd9b8066dd0 0x7fd9b8069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9cc0afeb0 0x7fd9cc0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9cc0accf0 0x7fd9cc154640 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 --2- 172.21.15.29:0/1169746977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9cc0af4c0 0x7fd9cc143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 >> 172.21.15.29:0/1169746977 conn(0x7fd9cc067610 msgr2=0x7fd9cc0cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 shutdown_connections 2023-10-07T10:44:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.770+0000 7fd9d13ff700 1 -- 172.21.15.29:0/1169746977 wait complete. 2023-10-07T10:44:57.789 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-07T10:44:57.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-10-07T10:44:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.987+0000 7f651ffff700 1 Processor -- start 2023-10-07T10:44:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.987+0000 7f651ffff700 1 -- start start 2023-10-07T10:44:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.988+0000 7f651ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65180afeb0 0x7f65180cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.988+0000 7f651ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f651815a590 0x7f651815aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.988+0000 7f651ffff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65180ad160 con 0x7f65180accf0 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.988+0000 7f651ffff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651815a140 con 0x7f651815a590 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.988+0000 7f651ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65180adca0 con 0x7f65180afeb0 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f651815a590 0x7f651815aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f651815a590 0x7f651815aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59804/0 (socket says 172.21.15.29:59804) 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 -- 172.21.15.29:0/1240536679 learned_addr learned my addr 172.21.15.29:0/1240536679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651dd9b700 1 --2- 172.21.15.29:0/1240536679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65180afeb0 0x7f65180cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 -- 172.21.15.29:0/1240536679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65180afeb0 msgr2=0x7f65180cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 --2- 172.21.15.29:0/1240536679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65180afeb0 0x7f65180cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 -- 172.21.15.29:0/1240536679 >> v1:172.21.15.144:6789/0 conn(0x7f65180accf0 legacy=0x7f65180aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 -- 172.21.15.29:0/1240536679 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65180ad400 con 0x7f651815a590 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651cd99700 1 -- 172.21.15.29:0/1240536679 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879612184 0 0) 0x7f65180ad160 con 0x7f65180accf0 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.989+0000 7f651e59c700 1 --2- 172.21.15.29:0/1240536679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651815a590 0x7f651815aa10 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f651000d1a0 tx=0x7f651000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3975d163e9135f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651cd99700 1 -- 172.21.15.29:0/1240536679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510010070 con 0x7f651815a590 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651cd99700 1 -- 172.21.15.29:0/1240536679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651000de10 con 0x7f651815a590 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651cd99700 1 -- 172.21.15.29:0/1240536679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651000b320 con 0x7f651815a590 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 -- 172.21.15.29:0/1240536679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651815a590 msgr2=0x7f651815aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 --2- 172.21.15.29:0/1240536679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651815a590 0x7f651815aa10 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f651000d1a0 tx=0x7f651000ba50 comp rx=0 tx=0).stop 2023-10-07T10:44:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 -- 172.21.15.29:0/1240536679 shutdown_connections 2023-10-07T10:44:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 --2- 172.21.15.29:0/1240536679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651815a590 0x7f651815aa10 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 --2- 172.21.15.29:0/1240536679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65180afeb0 0x7f65180cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 -- 172.21.15.29:0/1240536679 >> 172.21.15.29:0/1240536679 conn(0x7f6518067610 msgr2=0x7f6518067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 -- 172.21.15.29:0/1240536679 shutdown_connections 2023-10-07T10:44:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.990+0000 7f651ffff700 1 -- 172.21.15.29:0/1240536679 wait complete. 2023-10-07T10:44:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.992+0000 7f651ffff700 1 Processor -- start 2023-10-07T10:44:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.992+0000 7f651ffff700 1 -- start start 2023-10-07T10:44:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65180accf0 0x7f651816a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65180afeb0 0x7f6518153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651815a590 0x7f651815e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651815e400 con 0x7f65180accf0 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651816aa00 con 0x7f65180afeb0 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6518153e90 con 0x7f651815a590 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65180accf0 0x7f651816a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65180accf0 0x7f651816a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59808/0 (socket says 172.21.15.29:59808) 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651dd9b700 1 -- 172.21.15.29:0/2641627393 learned_addr learned my addr 172.21.15.29:0/2641627393 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651dd9b700 1 -- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651815a590 msgr2=0x7f651815e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.993+0000 7f651dd9b700 1 --2- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651815a590 0x7f651815e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f651dd9b700 1 -- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65180afeb0 msgr2=0x7f6518153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f651dd9b700 1 --2- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65180afeb0 0x7f6518153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f651dd9b700 1 -- 172.21.15.29:0/2641627393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651000bcf0 con 0x7f65180accf0 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f651dd9b700 1 --2- 172.21.15.29:0/2641627393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65180accf0 0x7f651816a4c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f651400ec80 tx=0x7f65140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651400c9d0 con 0x7f65180accf0 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651400cb30 con 0x7f65180accf0 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f651ffff700 1 -- 172.21.15.29:0/2641627393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651815d7f0 con 0x7f65180accf0 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514013470 con 0x7f65180accf0 2023-10-07T10:44:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.994+0000 7f651ffff700 1 -- 172.21.15.29:0/2641627393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651815ddb0 con 0x7f65180accf0 2023-10-07T10:44:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.995+0000 7f651ffff700 1 -- 172.21.15.29:0/2641627393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65180718b0 con 0x7f65180accf0 2023-10-07T10:44:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.996+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f6514013820 con 0x7f65180accf0 2023-10-07T10:44:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.997+0000 7f650dffb700 1 --2- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64fc066dd0 0x7f64fc069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.997+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f6514084970 con 0x7f65180accf0 2023-10-07T10:44:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.997+0000 7f651d59a700 1 --2- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64fc066dd0 0x7f64fc069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:57.998+0000 7f651d59a700 1 --2- 172.21.15.29:0/2641627393 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64fc066dd0 0x7f64fc069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f65180cc110 tx=0x7f6508007b40 comp rx=0 tx=0).ready entity=mgr.8374 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:58.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.001+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6514051b20 con 0x7f65180accf0 2023-10-07T10:44:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.229+0000 7f651ffff700 1 -- 172.21.15.29:0/2641627393 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f651815e540 con 0x7f64fc066dd0 2023-10-07T10:44:58.237 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:58.237+0000 7f3ba7f05700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-07T10:44:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.237+0000 7f650dffb700 1 -- 172.21.15.29:0/2641627393 <== mgr.8374 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f65080012b0 con 0x7f64fc066dd0 2023-10-07T10:44:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-07T10:44:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-07T10:44:58.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:58.248 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T10:44:58.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T10:44:58.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.494+0000 7f42739ff700 1 Processor -- start 2023-10-07T10:44:58.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.495+0000 7f42739ff700 1 -- start start 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f42739ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 0x7f426c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f42739ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f426c0ce8a0 0x7f426c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f42739ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c0cbf20 con 0x7f426c159b20 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f42739ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c0cc200 con 0x7f426c0ce8a0 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f42739ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0cc4e0 con 0x7f426c159750 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f4270f9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f426c159b20 0x7f426c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60214/0 (socket says 172.21.15.29:60214) 2023-10-07T10:44:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.496+0000 7f4270f9a700 1 -- 172.21.15.29:0/4163829990 learned_addr learned my addr 172.21.15.29:0/4163829990 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:58.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4271f9c700 1 --2- 172.21.15.29:0/4163829990 >> v2:172.21.15.144:3300/0 conn(0x7f426c0ce8a0 0x7f426c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f427179b700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 0x7f426c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501328039 0 0) 0x7f426c0cbf20 con 0x7f426c159b20 2023-10-07T10:44:58.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4258003570 con 0x7f426c159b20 2023-10-07T10:44:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4281938969 0 0) 0x7f4258003570 con 0x7f426c159b20 2023-10-07T10:44:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 msgr2=0x7f426c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 0x7f426c0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 >> v2:172.21.15.144:3300/0 conn(0x7f426c0ce8a0 msgr2=0x7f426c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 --2- 172.21.15.29:0/4163829990 >> v2:172.21.15.144:3300/0 conn(0x7f426c0ce8a0 0x7f426c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c0cc7c0 con 0x7f426c159b20 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f425c001470 con 0x7f426c159b20 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 >> v1:172.21.15.29:6789/0 conn(0x7f426c159b20 legacy=0x7f426c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4258003a60 0x7f4258003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 0x7f4258004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f42580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0cc200 con 0x7f4258004d50 2023-10-07T10:44:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0cc4e0 con 0x7f4258003a60 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.497+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425800ba60 con 0x7f426c159750 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f42580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4270f9a700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 0x7f4258004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f427179b700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4258003a60 0x7f4258003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 -- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 msgr2=0x7f4258004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c159750 0x7f4258004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 -- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4258003a60 msgr2=0x7f4258003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4258003a60 0x7f4258003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 -- 172.21.15.29:0/4163829990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258003570 con 0x7f4258004d50 2023-10-07T10:44:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4271f9c700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f42580051c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f426400e210 tx=0x7f426400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbbe3c551df5063 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426400ea60 con 0x7f4258004d50 2023-10-07T10:44:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426400f040 con 0x7f4258004d50 2023-10-07T10:44:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.498+0000 7f4263fff700 1 -- 172.21.15.29:0/4163829990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426400ee10 con 0x7f4258004d50 2023-10-07T10:44:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f42739ff700 1 -- 172.21.15.29:0/4163829990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 msgr2=0x7f42580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f42739ff700 1 --2- 172.21.15.29:0/4163829990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f42580051c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f426400e210 tx=0x7f426400b040 comp rx=0 tx=0).stop 2023-10-07T10:44:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f4271f9c700 1 -- 172.21.15.29:0/4163829990 reap_dead start 2023-10-07T10:44:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f42739ff700 1 -- 172.21.15.29:0/4163829990 shutdown_connections 2023-10-07T10:44:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f42739ff700 1 -- 172.21.15.29:0/4163829990 >> 172.21.15.29:0/4163829990 conn(0x7f426c063a10 msgr2=0x7f426c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f42739ff700 1 -- 172.21.15.29:0/4163829990 shutdown_connections 2023-10-07T10:44:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.499+0000 7f42739ff700 1 -- 172.21.15.29:0/4163829990 wait complete. 2023-10-07T10:44:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.501+0000 7f42739ff700 1 Processor -- start 2023-10-07T10:44:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.501+0000 7f42739ff700 1 -- start start 2023-10-07T10:44:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f42739ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f426c143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f42739ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c0c8610 0x7f426c0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f42739ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f426c0c8a80 0x7f426c0b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f42739ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0c9b20 con 0x7f4258004d50 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f42739ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0c9e00 con 0x7f426c0c8a80 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f42739ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0ca0e0 con 0x7f426c0c8610 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f426c143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f426c143180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59824/0 (socket says 172.21.15.29:59824) 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 -- 172.21.15.29:0/648463607 learned_addr learned my addr 172.21.15.29:0/648463607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 -- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c0c8610 msgr2=0x7f426c0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c0c8610 0x7f426c0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 -- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f426c0c8a80 msgr2=0x7f426c0b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f426c0c8a80 0x7f426c0b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.502+0000 7f427179b700 1 -- 172.21.15.29:0/648463607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c0ca3c0 con 0x7f4258004d50 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.503+0000 7f427179b700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f426c143180 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f4268011b10 tx=0x7f4268009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.503+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268005c10 con 0x7f4258004d50 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.503+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4268005d70 con 0x7f4258004d50 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.503+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426800fa10 con 0x7f4258004d50 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.503+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c0c5570 con 0x7f4258004d50 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.503+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c0c5a90 con 0x7f4258004d50 2023-10-07T10:44:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.504+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c0c58e0 con 0x7f4258004d50 2023-10-07T10:44:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.505+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f4268006050 con 0x7f4258004d50 2023-10-07T10:44:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.506+0000 7f42617fa700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4250066dd0 0x7f4250069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.506+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f4268084fe0 con 0x7f4258004d50 2023-10-07T10:44:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.507+0000 7f4270f9a700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4250066dd0 0x7f4250069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.507+0000 7f4270f9a700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4250066dd0 0x7f4250069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f425c0015a0 tx=0x7f425c001e90 comp rx=0 tx=0).ready entity=mgr.8374 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:58.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.511+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4268052190 con 0x7f4258004d50 2023-10-07T10:44:58.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:58.779+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f426c0c5700 con 0x7f4258004d50 2023-10-07T10:44:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.034+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f426805b080 con 0x7f4258004d50 2023-10-07T10:44:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.036+0000 7f42617fa700 1 -- 172.21.15.29:0/648463607 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v71) v1 ==== 93+0+0 (secure 0 0 0) 0x7f426807db80 con 0x7f4258004d50 2023-10-07T10:44:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4250066dd0 msgr2=0x7f4250069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4250066dd0 0x7f4250069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f425c0015a0 tx=0x7f425c001e90 comp rx=0 tx=0).stop 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 msgr2=0x7f426c143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f426c143180 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f4268011b10 tx=0x7f4268009120 comp rx=0 tx=0).stop 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 shutdown_connections 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4250066dd0 0x7f4250069280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f426c0c8a80 0x7f426c0b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.039+0000 7f42739ff700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4258004d50 0x7f426c143180 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.040+0000 7f42739ff700 1 --2- 172.21.15.29:0/648463607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f426c0c8610 0x7f426c0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.040+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 >> 172.21.15.29:0/648463607 conn(0x7f426c063a10 msgr2=0x7f426c0704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.040+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 shutdown_connections 2023-10-07T10:44:59.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.040+0000 7f42739ff700 1 -- 172.21.15.29:0/648463607 wait complete. 2023-10-07T10:44:59.068 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T10:44:59.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:44:59.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:44:59.255 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:59.255+0000 7fbd574fd1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:44:59.265 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:59.265+0000 7fcc81ab11c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:44:59.290 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T10:44:59.290 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T10:44:59.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:44:59.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T10:44:59.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T10:44:59.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:44:59.424 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T10:44:59.435 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:59.435+0000 7fbd574fd1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:44:59.460 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:59.459+0000 7fcc81ab11c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:44:59.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:44:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.722+0000 7fe22b5ff700 1 Processor -- start 2023-10-07T10:44:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.723+0000 7fe22b5ff700 1 -- start start 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2240ae8b0 0x7fe2240b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22b5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe2240b1840 0x7fe224157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22b5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2240b1ca0 con 0x7fe224157290 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22b5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe224158ad0 con 0x7fe2240b1840 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22415b140 con 0x7fe2240ae8b0 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22939b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe224157290 0x7fe2240cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60236/0 (socket says 172.21.15.29:60236) 2023-10-07T10:44:59.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe22939b700 1 -- 172.21.15.29:0/3984019752 learned_addr learned my addr 172.21.15.29:0/3984019752 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:44:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe229b9c700 1 --2- 172.21.15.29:0/3984019752 >> v2:172.21.15.144:3300/0 conn(0x7fe2240b1840 0x7fe224157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:59.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe228b9a700 1 --2- 172.21.15.29:0/3984019752 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2240ae8b0 0x7fe2240b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe21bfff700 1 -- 172.21.15.29:0/3984019752 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929046699 0 0) 0x7fe220001110 con 0x7fe224157290 2023-10-07T10:44:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe21bfff700 1 -- 172.21.15.29:0/3984019752 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe210003570 con 0x7fe224157290 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe228b9a700 1 -- 172.21.15.29:0/3984019752 >> v2:172.21.15.144:3300/0 conn(0x7fe2240b1840 msgr2=0x7fe224157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe228b9a700 1 --2- 172.21.15.29:0/3984019752 >> v2:172.21.15.144:3300/0 conn(0x7fe2240b1840 0x7fe224157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe228b9a700 1 -- 172.21.15.29:0/3984019752 >> v1:172.21.15.29:6789/0 conn(0x7fe224157290 legacy=0x7fe2240cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe228b9a700 1 -- 172.21.15.29:0/3984019752 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22415b420 con 0x7fe2240ae8b0 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe228b9a700 1 --2- 172.21.15.29:0/3984019752 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2240ae8b0 0x7fe2240b1300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe2140098e0 tx=0x7fe2140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981b0d0ccf81d453 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe21bfff700 1 -- 172.21.15.29:0/3984019752 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214011630 con 0x7fe2240ae8b0 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe21bfff700 1 -- 172.21.15.29:0/3984019752 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214011790 con 0x7fe2240ae8b0 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.725+0000 7fe21bfff700 1 -- 172.21.15.29:0/3984019752 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214011b30 con 0x7fe2240ae8b0 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3984019752 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2240ae8b0 msgr2=0x7fe2240b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3984019752 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2240ae8b0 0x7fe2240b1300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe2140098e0 tx=0x7fe2140091a0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3984019752 shutdown_connections 2023-10-07T10:44:59.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3984019752 >> v2:172.21.15.144:3300/0 conn(0x7fe2240b1840 0x7fe224157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3984019752 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2240ae8b0 0x7fe2240b1300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3984019752 >> 172.21.15.29:0/3984019752 conn(0x7fe2240680a0 msgr2=0x7fe2240cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3984019752 shutdown_connections 2023-10-07T10:44:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.726+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3984019752 wait complete. 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.728+0000 7fe22b5ff700 1 Processor -- start 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.728+0000 7fe22b5ff700 1 -- start start 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 0x7fe22406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe224157290 0x7fe2240b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe22414ae10 0x7fe224155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22414edc0 con 0x7fe2240b1840 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22406e980 con 0x7fe224157290 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22406ec60 con 0x7fe22414ae10 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 0x7fe22406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 0x7fe22406e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59870/0 (socket says 172.21.15.29:59870) 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 -- 172.21.15.29:0/3597783933 learned_addr learned my addr 172.21.15.29:0/3597783933 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe228b9a700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe224157290 0x7fe2240b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe229b9c700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe22414ae10 0x7fe224155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 -- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe22414ae10 msgr2=0x7fe224155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe22414ae10 0x7fe224155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 -- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe224157290 msgr2=0x7fe2240b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe224157290 0x7fe2240b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 -- 172.21.15.29:0/3597783933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe214009420 con 0x7fe2240b1840 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22939b700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 0x7fe22406e340 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c590 tx=0x7fe220007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220005b30 con 0x7fe2240b1840 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe220005cb0 con 0x7fe2240b1840 2023-10-07T10:44:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220007320 con 0x7fe2240b1840 2023-10-07T10:44:59.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22406ef40 con 0x7fe2240b1840 2023-10-07T10:44:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.730+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2240b1230 con 0x7fe2240b1840 2023-10-07T10:44:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.731+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22404c3c0 con 0x7fe2240b1840 2023-10-07T10:44:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.732+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fe22001b3b0 con 0x7fe2240b1840 2023-10-07T10:44:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.733+0000 7fe2197fa700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 0x7fe208069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:44:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.733+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fe220083e30 con 0x7fe2240b1840 2023-10-07T10:44:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.733+0000 7fe228b9a700 1 -- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 msgr2=0x7fe208069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.733+0000 7fe228b9a700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 0x7fe208069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:44:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.737+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe220050ae0 con 0x7fe2240b1840 2023-10-07T10:44:59.748 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:44:59.748+0000 7fbd574fd1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:44:59.806 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:44:59.805+0000 7fcc81ab11c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:44:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.935+0000 7fe228b9a700 1 -- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 msgr2=0x7fe208069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:44:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.935+0000 7fe228b9a700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 0x7fe208069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:44:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.988+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2240b1ca0 con 0x7fe2240b1840 2023-10-07T10:44:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.988+0000 7fe2197fa700 1 -- 172.21.15.29:0/3597783933 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fe2240b1ca0 con 0x7fe2240b1840 2023-10-07T10:44:59.988 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:44:59.989 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:44:40.623432+0000","modified":"2023-10-07T10:44:44.142903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8634":{"gid":8634,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3670759585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3670759585},{"type":"v1","addr":"172.21.15.29:6835","nonce":3670759585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-10-07T10:44:59.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 msgr2=0x7fe208069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:44:59.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 0x7fe208069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 msgr2=0x7fe22406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 0x7fe22406e340 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c590 tx=0x7fe220007c70 comp rx=0 tx=0).stop 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 shutdown_connections 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe2080670e0 0x7fe208069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe22414ae10 0x7fe224155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2240b1840 0x7fe22406e340 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 --2- 172.21.15.29:0/3597783933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe224157290 0x7fe2240b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 >> 172.21.15.29:0/3597783933 conn(0x7fe2240680a0 msgr2=0x7fe224158c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 shutdown_connections 2023-10-07T10:44:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:44:59.991+0000 7fe22b5ff700 1 -- 172.21.15.29:0/3597783933 wait complete. 2023-10-07T10:44:59.993 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-07T10:45:00.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:00.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.212+0000 7fd143dff700 1 Processor -- start 2023-10-07T10:45:00.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.212+0000 7fd143dff700 1 -- start start 2023-10-07T10:45:00.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.213+0000 7fd143dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c0ad490 0x7fd13c1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.213+0000 7fd143dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0b0dc0 con 0x7fd13c159320 2023-10-07T10:45:00.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.213+0000 7fd143dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0cd0d0 con 0x7fd13c0afd20 2023-10-07T10:45:00.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.213+0000 7fd143dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0b01b0 con 0x7fd13c0ad490 2023-10-07T10:45:00.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd141b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd13c159320 0x7fd13c0cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60248/0 (socket says 172.21.15.29:60248) 2023-10-07T10:45:00.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd141b9b700 1 -- 172.21.15.29:0/2077615130 learned_addr learned my addr 172.21.15.29:0/2077615130 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:00.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd14239c700 1 --2- 172.21.15.29:0/2077615130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c0ad490 0x7fd13c1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688192925 0 0) 0x7fd13c0b0dc0 con 0x7fd13c159320 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128003570 con 0x7fd13c159320 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231382448 0 0) 0x7fd13c0cd0d0 con 0x7fd13c0afd20 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd13c0b0dc0 con 0x7fd13c0afd20 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd14239c700 1 -- 172.21.15.29:0/2077615130 >> v1:172.21.15.144:6789/0 conn(0x7fd13c0afd20 legacy=0x7fd13c0acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd14239c700 1 -- 172.21.15.29:0/2077615130 >> v1:172.21.15.29:6789/0 conn(0x7fd13c159320 legacy=0x7fd13c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.214+0000 7fd14239c700 1 -- 172.21.15.29:0/2077615130 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0b0490 con 0x7fd13c0ad490 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.215+0000 7fd14239c700 1 --2- 172.21.15.29:0/2077615130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c0ad490 0x7fd13c1596f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd134009e90 tx=0x7fd134009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9f2fc9517811497 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.215+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13400e070 con 0x7fd13c0ad490 2023-10-07T10:45:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.215+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13400bc10 con 0x7fd13c0ad490 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.215+0000 7fd140b99700 1 -- 172.21.15.29:0/2077615130 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134012430 con 0x7fd13c0ad490 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.215+0000 7fd143dff700 1 -- 172.21.15.29:0/2077615130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c0ad490 msgr2=0x7fd13c1596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.215+0000 7fd143dff700 1 --2- 172.21.15.29:0/2077615130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c0ad490 0x7fd13c1596f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd134009e90 tx=0x7fd134009780 comp rx=0 tx=0).stop 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.216+0000 7fd143dff700 1 -- 172.21.15.29:0/2077615130 shutdown_connections 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.216+0000 7fd143dff700 1 --2- 172.21.15.29:0/2077615130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c0ad490 0x7fd13c1596f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.216+0000 7fd143dff700 1 -- 172.21.15.29:0/2077615130 >> 172.21.15.29:0/2077615130 conn(0x7fd13c0680a0 msgr2=0x7fd13c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.216+0000 7fd143dff700 1 -- 172.21.15.29:0/2077615130 shutdown_connections 2023-10-07T10:45:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.216+0000 7fd143dff700 1 -- 172.21.15.29:0/2077615130 wait complete. 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.217+0000 7fd143dff700 1 Processor -- start 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.217+0000 7fd143dff700 1 -- start start 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd143dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd13c0ad490 0x7fd13c14e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd143dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd13c0afd20 0x7fd13c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd143dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 0x7fd13c152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd143dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c14fc60 con 0x7fd13c0afd20 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd143dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c14f080 con 0x7fd13c0ad490 2023-10-07T10:45:00.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd143dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c14f360 con 0x7fd13c159320 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.218+0000 7fd14139a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd13c0afd20 0x7fd13c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 0x7fd13c152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd141b9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd13c0ad490 0x7fd13c14e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 0x7fd13c152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:52014/0 (socket says 172.21.15.29:52014) 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 -- 172.21.15.29:0/2873699209 learned_addr learned my addr 172.21.15.29:0/2873699209 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd141b9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd13c0ad490 0x7fd13c14e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:47306/0 (socket says 172.21.15.29:47306) 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 -- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd13c0ad490 msgr2=0x7fd13c14e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd13c0ad490 0x7fd13c14e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 -- 172.21.15.29:0/2873699209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd13c0afd20 msgr2=0x7fd13c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd13c0afd20 0x7fd13c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 -- 172.21.15.29:0/2873699209 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd134009a20 con 0x7fd13c159320 2023-10-07T10:45:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.219+0000 7fd14239c700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 0x7fd13c152760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd134013000 tx=0x7fd134019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.220+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13401b420 con 0x7fd13c159320 2023-10-07T10:45:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.220+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13401b5a0 con 0x7fd13c159320 2023-10-07T10:45:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.220+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13401b960 con 0x7fd13c159320 2023-10-07T10:45:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.220+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c14f640 con 0x7fd13c159320 2023-10-07T10:45:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.220+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c14ed70 con 0x7fd13c159320 2023-10-07T10:45:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.221+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c04c3c0 con 0x7fd13c159320 2023-10-07T10:45:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.222+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fd13401bcd0 con 0x7fd13c159320 2023-10-07T10:45:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.223+0000 7fd131ffb700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 0x7fd120069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.223+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fd1340866d0 con 0x7fd13c159320 2023-10-07T10:45:00.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.223+0000 7fd141b9b700 1 -- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 msgr2=0x7fd120069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:00.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.223+0000 7fd141b9b700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 0x7fd120069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.227+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd134053320 con 0x7fd13c159320 2023-10-07T10:45:00.267 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:00.267+0000 7fbd574fd1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:45:00.374 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:00.374+0000 7fcc81ab11c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:45:00.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.424+0000 7fd141b9b700 1 -- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 msgr2=0x7fd120069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:00.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.424+0000 7fd141b9b700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 0x7fd120069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:00.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.430+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd13c0b0770 con 0x7fd13c159320 2023-10-07T10:45:00.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.431+0000 7fd131ffb700 1 -- 172.21.15.29:0/2873699209 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16477 (secure 0 0 0) 0x7fd134056940 con 0x7fd13c159320 2023-10-07T10:45:00.431 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:00.431 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":152,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:56.153475+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:44:38.468584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-07T10:44:39.609895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3633091014":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/1131701620":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/3484621640":"2023-10-08T10:44:56.153443+0000","172.21.15.144:6836/114714":"2023-10-08T10:44:56.153443+0000","172.21.15.144:6834/114714":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/3284926740":"2023-10-08T10:44:56.153443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 msgr2=0x7fd120069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 0x7fd120069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 msgr2=0x7fd13c152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 0x7fd13c152760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd134013000 tx=0x7fd134019000 comp rx=0 tx=0).stop 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 shutdown_connections 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd1200670e0 0x7fd120069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd13c159320 0x7fd13c152760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd13c0ad490 0x7fd13c14e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 --2- 172.21.15.29:0/2873699209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd13c0afd20 0x7fd13c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.433+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 >> 172.21.15.29:0/2873699209 conn(0x7fd13c0680a0 msgr2=0x7fd13c0cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:00.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.434+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 shutdown_connections 2023-10-07T10:45:00.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.434+0000 7fd143dff700 1 -- 172.21.15.29:0/2873699209 wait complete. 2023-10-07T10:45:00.437 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:00.437+0000 7fbd574fd1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:45:00.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T10:44:38.468584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T10:44:39.609895+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T10:45:00.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T10:45:00.557 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:00.558+0000 7fcc81ab11c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:45:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.656+0000 7f8360dff700 1 Processor -- start 2023-10-07T10:45:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.656+0000 7f8360dff700 1 -- start start 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8360dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f835c06a8c0 0x7f835c0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8360dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c06b5c0 0x7f835c0c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8360dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c06a430 con 0x7f835c159790 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8360dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835c159b60 con 0x7f835c06a8c0 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8360dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c069820 con 0x7f835c06b5c0 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f835c06a8c0 0x7f835c0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f835c06a8c0 0x7f835c0c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59886/0 (socket says 172.21.15.29:59886) 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 -- 172.21.15.29:0/2242328119 learned_addr learned my addr 172.21.15.29:0/2242328119 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f835ad9d700 1 --2- 172.21.15.29:0/2242328119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c06b5c0 0x7f835c0c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 -- 172.21.15.29:0/2242328119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c06b5c0 msgr2=0x7f835c0c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 --2- 172.21.15.29:0/2242328119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c06b5c0 0x7f835c0c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 -- 172.21.15.29:0/2242328119 >> v1:172.21.15.144:6789/0 conn(0x7f835c159790 legacy=0x7f835c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f8359d9b700 1 -- 172.21.15.29:0/2242328119 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c069b00 con 0x7f835c06a8c0 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.658+0000 7f835959a700 1 -- 172.21.15.29:0/2242328119 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802318808 0 0) 0x7f835c06a430 con 0x7f835c159790 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.659+0000 7f8359d9b700 1 --2- 172.21.15.29:0/2242328119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06a8c0 0x7f835c0c6f40 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f83500099b0 tx=0x7f8350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5823dca15445cc31 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.659+0000 7f835959a700 1 -- 172.21.15.29:0/2242328119 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350009ea0 con 0x7f835c06a8c0 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.659+0000 7f835959a700 1 -- 172.21.15.29:0/2242328119 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835000b860 con 0x7f835c06a8c0 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.659+0000 7f835959a700 1 -- 172.21.15.29:0/2242328119 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835000bc00 con 0x7f835c06a8c0 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 -- 172.21.15.29:0/2242328119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06a8c0 msgr2=0x7f835c0c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 --2- 172.21.15.29:0/2242328119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06a8c0 0x7f835c0c6f40 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f83500099b0 tx=0x7f8350009250 comp rx=0 tx=0).stop 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 -- 172.21.15.29:0/2242328119 shutdown_connections 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 --2- 172.21.15.29:0/2242328119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c06b5c0 0x7f835c0c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 --2- 172.21.15.29:0/2242328119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06a8c0 0x7f835c0c6f40 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 -- 172.21.15.29:0/2242328119 >> 172.21.15.29:0/2242328119 conn(0x7f835c0c9410 msgr2=0x7f835c0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 -- 172.21.15.29:0/2242328119 shutdown_connections 2023-10-07T10:45:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.660+0000 7f8360dff700 1 -- 172.21.15.29:0/2242328119 wait complete. 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.661+0000 7f8360dff700 1 Processor -- start 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.661+0000 7f8360dff700 1 -- start start 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.662+0000 7f8360dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 0x7f835c1515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.662+0000 7f8360dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835c159790 0x7f835c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.662+0000 7f8360dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c067610 0x7f835c159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.662+0000 7f8360dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c068720 con 0x7f835c06b5c0 2023-10-07T10:45:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.662+0000 7f8360dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c151b30 con 0x7f835c159790 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.662+0000 7f8360dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c1524c0 con 0x7f835c067610 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 0x7f835c1515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 0x7f835c1515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59892/0 (socket says 172.21.15.29:59892) 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 -- 172.21.15.29:0/2674697964 learned_addr learned my addr 172.21.15.29:0/2674697964 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f8359d9b700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835c159790 0x7f835c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835ad9d700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c067610 0x7f835c159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 -- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c067610 msgr2=0x7f835c159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c067610 0x7f835c159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 -- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835c159790 msgr2=0x7f835c0ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835c159790 0x7f835c0ae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 -- 172.21.15.29:0/2674697964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500094f0 con 0x7f835c06b5c0 2023-10-07T10:45:00.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.663+0000 7f835a59c700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 0x7f835c1515f0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c7e0 tx=0x7f834c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.664+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c011590 con 0x7f835c06b5c0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.664+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c011710 con 0x7f835c06b5c0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.664+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c011ad0 con 0x7f835c06b5c0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.664+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c067aa0 con 0x7f835c06b5c0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.664+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c068060 con 0x7f835c06b5c0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.665+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835c06ee20 con 0x7f835c06b5c0 2023-10-07T10:45:00.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.666+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f834c01c460 con 0x7f835c06b5c0 2023-10-07T10:45:00.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.667+0000 7f834a7fc700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 0x7f833c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:00.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.667+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f834c0847d0 con 0x7f835c06b5c0 2023-10-07T10:45:00.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.667+0000 7f8359d9b700 1 -- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 msgr2=0x7f833c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:00.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.667+0000 7f8359d9b700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 0x7f833c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:00.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.671+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f834c051520 con 0x7f835c06b5c0 2023-10-07T10:45:00.849 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:00.849+0000 7fbd574fd1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:45:00.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.868+0000 7f8359d9b700 1 -- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 msgr2=0x7f833c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:00.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.868+0000 7f8359d9b700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 0x7f833c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:00.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.876+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f835c04c3c0 con 0x7f835c06b5c0 2023-10-07T10:45:00.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.877+0000 7f834a7fc700 1 -- 172.21.15.29:0/2674697964 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) v1 ==== 93+0+10 (secure 0 0 0) 0x7f834c05a020 con 0x7f835c06b5c0 2023-10-07T10:45:00.876 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T10:45:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 msgr2=0x7f833c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 0x7f833c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 msgr2=0x7f835c1515f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 0x7f835c1515f0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c7e0 tx=0x7f834c007df0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 shutdown_connections 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f833c0670e0 0x7f833c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f835c067610 0x7f835c159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c06b5c0 0x7f835c1515f0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 --2- 172.21.15.29:0/2674697964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835c159790 0x7f835c0ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 >> 172.21.15.29:0/2674697964 conn(0x7f835c0c9410 msgr2=0x7f835c15bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.879+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 shutdown_connections 2023-10-07T10:45:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:00.880+0000 7f8360dff700 1 -- 172.21.15.29:0/2674697964 wait complete. 2023-10-07T10:45:00.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T10:45:01.000 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:01.000+0000 7fcc81ab11c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:45:01.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.102+0000 7fc7347ff700 1 Processor -- start 2023-10-07T10:45:01.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.102+0000 7fc7347ff700 1 -- start start 2023-10-07T10:45:01.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc7347ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc72c13ff70 0x7fc72c06ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc7347ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c0c78c0 con 0x7fc72c13fba0 2023-10-07T10:45:01.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc7347ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c06fa50 con 0x7fc72c067d20 2023-10-07T10:45:01.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc7347ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c06c0f0 con 0x7fc72c13ff70 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc73259b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc72c13fba0 0x7fc72c067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60258/0 (socket says 172.21.15.29:60258) 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc73259b700 1 -- 172.21.15.29:0/660644399 learned_addr learned my addr 172.21.15.29:0/660644399 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc732d9c700 1 --2- 172.21.15.29:0/660644399 >> v2:172.21.15.144:3300/0 conn(0x7fc72c13ff70 0x7fc72c06ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc731599700 1 -- 172.21.15.29:0/660644399 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804434668 0 0) 0x7fc72c0c78c0 con 0x7fc72c13fba0 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.104+0000 7fc731599700 1 -- 172.21.15.29:0/660644399 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc718003570 con 0x7fc72c13fba0 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc731599700 1 -- 172.21.15.29:0/660644399 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066253676 0 0) 0x7fc72c06fa50 con 0x7fc72c067d20 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc732d9c700 1 -- 172.21.15.29:0/660644399 >> v1:172.21.15.144:6789/0 conn(0x7fc72c067d20 legacy=0x7fc72c0c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc732d9c700 1 -- 172.21.15.29:0/660644399 >> v1:172.21.15.29:6789/0 conn(0x7fc72c13fba0 legacy=0x7fc72c067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc732d9c700 1 -- 172.21.15.29:0/660644399 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c06c3d0 con 0x7fc72c13ff70 2023-10-07T10:45:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc732d9c700 1 --2- 172.21.15.29:0/660644399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13ff70 0x7fc72c06ba10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc724009e60 tx=0x7fc724009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=102dcccc9d0a2adc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:01.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc731599700 1 -- 172.21.15.29:0/660644399 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72400e070 con 0x7fc72c13ff70 2023-10-07T10:45:01.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc731599700 1 -- 172.21.15.29:0/660644399 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72400bbf0 con 0x7fc72c13ff70 2023-10-07T10:45:01.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.105+0000 7fc731599700 1 -- 172.21.15.29:0/660644399 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc724012430 con 0x7fc72c13ff70 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 -- 172.21.15.29:0/660644399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13ff70 msgr2=0x7fc72c06ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 --2- 172.21.15.29:0/660644399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13ff70 0x7fc72c06ba10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc724009e60 tx=0x7fc724009780 comp rx=0 tx=0).stop 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 -- 172.21.15.29:0/660644399 shutdown_connections 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 --2- 172.21.15.29:0/660644399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13ff70 0x7fc72c06ba10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 -- 172.21.15.29:0/660644399 >> 172.21.15.29:0/660644399 conn(0x7fc72c06b1d0 msgr2=0x7fc72c0cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 -- 172.21.15.29:0/660644399 shutdown_connections 2023-10-07T10:45:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.106+0000 7fc7347ff700 1 -- 172.21.15.29:0/660644399 wait complete. 2023-10-07T10:45:01.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.108+0000 7fc7347ff700 1 Processor -- start 2023-10-07T10:45:01.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.108+0000 7fc7347ff700 1 -- start start 2023-10-07T10:45:01.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.109+0000 7fc7347ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc72c067d20 0x7fc72c158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc7347ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13fba0 0x7fc72c0b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc7347ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 0x7fc72c0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc7347ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c159470 con 0x7fc72c13ff70 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc7347ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c15cd80 con 0x7fc72c13fba0 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc7347ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c15c200 con 0x7fc72c067d20 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 0x7fc72c0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc73259b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc72c067d20 0x7fc72c158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc731d9a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13fba0 0x7fc72c0b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 0x7fc72c0b0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59902/0 (socket says 172.21.15.29:59902) 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 -- 172.21.15.29:0/396667028 learned_addr learned my addr 172.21.15.29:0/396667028 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 -- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc72c067d20 msgr2=0x7fc72c158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc72c067d20 0x7fc72c158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 -- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13fba0 msgr2=0x7fc72c0b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13fba0 0x7fc72c0b27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc732d9c700 1 -- 172.21.15.29:0/396667028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009a20 con 0x7fc72c13ff70 2023-10-07T10:45:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.110+0000 7fc731d9a700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13fba0 0x7fc72c0b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.111+0000 7fc732d9c700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 0x7fc72c0b0020 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc72c06dd00 tx=0x7fc724019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.111+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc724026480 con 0x7fc72c13ff70 2023-10-07T10:45:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.111+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc724026600 con 0x7fc72c13ff70 2023-10-07T10:45:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.111+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7240269c0 con 0x7fc72c13ff70 2023-10-07T10:45:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.111+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c15c4e0 con 0x7fc72c13ff70 2023-10-07T10:45:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.111+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c15caa0 con 0x7fc72c13ff70 2023-10-07T10:45:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.112+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c04c3c0 con 0x7fc72c13ff70 2023-10-07T10:45:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.114+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fc72401c400 con 0x7fc72c13ff70 2023-10-07T10:45:01.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.115+0000 7fc7227fc700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 0x7fc710069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.115+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fc7240870c0 con 0x7fc72c13ff70 2023-10-07T10:45:01.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.115+0000 7fc73259b700 1 -- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 msgr2=0x7fc710069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:01.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.115+0000 7fc73259b700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 0x7fc710069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:01.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.119+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc724053c60 con 0x7fc72c13ff70 2023-10-07T10:45:01.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.316+0000 7fc73259b700 1 -- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 msgr2=0x7fc710069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:01.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.316+0000 7fc73259b700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 0x7fc710069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:01.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.325+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc72c06c6b0 con 0x7fc72c13ff70 2023-10-07T10:45:01.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.325+0000 7fc7227fc700 1 -- 172.21.15.29:0/396667028 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v152) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc724057280 con 0x7fc72c13ff70 2023-10-07T10:45:01.325 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T10:45:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 msgr2=0x7fc710069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 0x7fc710069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 msgr2=0x7fc72c0b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 0x7fc72c0b0020 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc72c06dd00 tx=0x7fc724019000 comp rx=0 tx=0).stop 2023-10-07T10:45:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 shutdown_connections 2023-10-07T10:45:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc7100670e0 0x7fc710069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc72c13ff70 0x7fc72c0b0020 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc72c067d20 0x7fc72c158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 --2- 172.21.15.29:0/396667028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc72c13fba0 0x7fc72c0b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 >> 172.21.15.29:0/396667028 conn(0x7fc72c06b1d0 msgr2=0x7fc72c0cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 shutdown_connections 2023-10-07T10:45:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.328+0000 7fc7347ff700 1 -- 172.21.15.29:0/396667028 wait complete. 2023-10-07T10:45:01.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T10:45:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.547+0000 7f6dc71ff700 1 Processor -- start 2023-10-07T10:45:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.547+0000 7f6dc71ff700 1 -- start start 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc71ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc71ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc006a430 con 0x7f6dc0159790 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc71ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc0159b60 con 0x7f6dc006a8c0 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc71ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0069820 con 0x7f6dc006b5c0 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc4f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6dc0159790 0x7f6dc006aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60266/0 (socket says 172.21.15.29:60266) 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc4f9b700 1 -- 172.21.15.29:0/658467597 learned_addr learned my addr 172.21.15.29:0/658467597 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dc579c700 1 --2- 172.21.15.29:0/658467597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.549+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250923504 0 0) 0x7f6dc006a430 con 0x7f6dc0159790 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0003570 con 0x7f6dc0159790 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369926643 0 0) 0x7f6dc0159b60 con 0x7f6dc006a8c0 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc006a430 con 0x7f6dc006a8c0 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dc579c700 1 -- 172.21.15.29:0/658467597 >> v1:172.21.15.144:6789/0 conn(0x7f6dc006a8c0 legacy=0x7f6dc00c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dc579c700 1 -- 172.21.15.29:0/658467597 >> v1:172.21.15.29:6789/0 conn(0x7f6dc0159790 legacy=0x7f6dc006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dc579c700 1 -- 172.21.15.29:0/658467597 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0069b00 con 0x7f6dc006b5c0 2023-10-07T10:45:01.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dc579c700 1 --2- 172.21.15.29:0/658467597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00c7650 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009e90 tx=0x7f6db8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7686ee43ceef6d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800e070 con 0x7f6dc006b5c0 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800bb80 con 0x7f6dc006b5c0 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.550+0000 7f6dbf7fe700 1 -- 172.21.15.29:0/658467597 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8012430 con 0x7f6dc006b5c0 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 -- 172.21.15.29:0/658467597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 msgr2=0x7f6dc00c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/658467597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00c7650 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009e90 tx=0x7f6db8009780 comp rx=0 tx=0).stop 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 -- 172.21.15.29:0/658467597 shutdown_connections 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/658467597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00c7650 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 -- 172.21.15.29:0/658467597 >> 172.21.15.29:0/658467597 conn(0x7f6dc00c9410 msgr2=0x7f6dc00c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 -- 172.21.15.29:0/658467597 shutdown_connections 2023-10-07T10:45:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.551+0000 7f6dc71ff700 1 -- 172.21.15.29:0/658467597 wait complete. 2023-10-07T10:45:01.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.553+0000 7f6dc71ff700 1 Processor -- start 2023-10-07T10:45:01.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.554+0000 7f6dc71ff700 1 -- start start 2023-10-07T10:45:01.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc71ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc71ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dc0159790 0x7f6dc0151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc71ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01553d0 0x7f6dc01526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc71ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00ad4d0 con 0x7f6dc01553d0 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc71ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00ae7e0 con 0x7f6dc0159790 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc71ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00adbd0 con 0x7f6dc006b5c0 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.555+0000 7f6dc579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01553d0 0x7f6dc01526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc4f9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc4f9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00acf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:52056/0 (socket says 172.21.15.29:52056) 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc4f9b700 1 -- 172.21.15.29:0/2405993958 learned_addr learned my addr 172.21.15.29:0/2405993958 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:01.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dbffff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dc0159790 0x7f6dc0151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc579c700 1 -- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 msgr2=0x7f6dc00acf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc579c700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00acf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc579c700 1 -- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dc0159790 msgr2=0x7f6dc0151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc579c700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dc0159790 0x7f6dc0151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc579c700 1 -- 172.21.15.29:0/2405993958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8009a20 con 0x7f6dc01553d0 2023-10-07T10:45:01.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.556+0000 7f6dc579c700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01553d0 0x7f6dc01526f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f6db8016000 tx=0x7f6db8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:01.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.557+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db801b420 con 0x7f6dc01553d0 2023-10-07T10:45:01.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.557+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db801b5e0 con 0x7f6dc01553d0 2023-10-07T10:45:01.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.557+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc00adeb0 con 0x7f6dc01553d0 2023-10-07T10:45:01.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.557+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db801b9a0 con 0x7f6dc01553d0 2023-10-07T10:45:01.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.557+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc00ae240 con 0x7f6dc01553d0 2023-10-07T10:45:01.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.558+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc004c3c0 con 0x7f6dc01553d0 2023-10-07T10:45:01.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.560+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f6db801bd10 con 0x7f6dc01553d0 2023-10-07T10:45:01.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.561+0000 7f6dbcff9700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 0x7f6da4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.561+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f6db8086850 con 0x7f6dc01553d0 2023-10-07T10:45:01.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.561+0000 7f6dc4f9b700 1 -- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 msgr2=0x7f6da4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:01.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.561+0000 7f6dc4f9b700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 0x7f6da4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:01.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.565+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6db8053490 con 0x7f6dc01553d0 2023-10-07T10:45:01.621 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:01.621+0000 7fbd574fd1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:45:01.708 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:01.707+0000 7fbd574fd1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:45:01.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.762+0000 7f6dc4f9b700 1 -- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 msgr2=0x7f6da4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:01.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.762+0000 7f6dc4f9b700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 0x7f6da4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:01.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.766+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6dc0063640 con 0x7f6dc01553d0 2023-10-07T10:45:01.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.766+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2405993958 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v152) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6db8056ab0 con 0x7f6dc01553d0 2023-10-07T10:45:01.766 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T10:45:01.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 msgr2=0x7f6da4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:01.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 0x7f6da4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01553d0 msgr2=0x7f6dc01526f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01553d0 0x7f6dc01526f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f6db8016000 tx=0x7f6db8019000 comp rx=0 tx=0).stop 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 shutdown_connections 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6da40670e0 0x7f6da4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01553d0 0x7f6dc01526f0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dc006b5c0 0x7f6dc00acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 --2- 172.21.15.29:0/2405993958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dc0159790 0x7f6dc0151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 >> 172.21.15.29:0/2405993958 conn(0x7f6dc00c9410 msgr2=0x7f6dc015d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 shutdown_connections 2023-10-07T10:45:01.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.769+0000 7f6dc71ff700 1 -- 172.21.15.29:0/2405993958 wait complete. 2023-10-07T10:45:01.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:45:01.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:01.841+0000 7fcc81ab11c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:45:01.861 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:01.861+0000 7fbd574fd1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:45:01.937 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:01.937+0000 7fcc81ab11c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:45:01.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:01.946+0000 7fbd574fd1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:45:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.985+0000 7f64e9bff700 1 Processor -- start 2023-10-07T10:45:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.985+0000 7f64e9bff700 1 -- start start 2023-10-07T10:45:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.986+0000 7f64e9bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f64e40b3e60 0x7f64e4158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.986+0000 7f64e9bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.986+0000 7f64e9bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e41578a0 con 0x7f64e40b00b0 2023-10-07T10:45:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.986+0000 7f64e9bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e40cc830 con 0x7f64e40b3e60 2023-10-07T10:45:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.986+0000 7f64e9bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e4157ce0 con 0x7f64e40ce390 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f64e40b00b0 0x7f64e40cccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60278/0 (socket says 172.21.15.29:60278) 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e3fff700 1 -- 172.21.15.29:0/441699896 learned_addr learned my addr 172.21.15.29:0/441699896 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e2ffd700 1 --2- 172.21.15.29:0/441699896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e37fe700 1 --2- 172.21.15.29:0/441699896 >> v2:172.21.15.144:3300/0 conn(0x7f64e40b3e60 0x7f64e4158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e27fc700 1 -- 172.21.15.29:0/441699896 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983168289 0 0) 0x7f64e41578a0 con 0x7f64e40b00b0 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e27fc700 1 -- 172.21.15.29:0/441699896 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc003570 con 0x7f64e40b00b0 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e2ffd700 1 -- 172.21.15.29:0/441699896 >> v2:172.21.15.144:3300/0 conn(0x7f64e40b3e60 msgr2=0x7f64e4158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e2ffd700 1 --2- 172.21.15.29:0/441699896 >> v2:172.21.15.144:3300/0 conn(0x7f64e40b3e60 0x7f64e4158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e2ffd700 1 -- 172.21.15.29:0/441699896 >> v1:172.21.15.29:6789/0 conn(0x7f64e40b00b0 legacy=0x7f64e40cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e2ffd700 1 -- 172.21.15.29:0/441699896 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40cee50 con 0x7f64e40ce390 2023-10-07T10:45:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e37fe700 1 --2- 172.21.15.29:0/441699896 >> v2:172.21.15.144:3300/0 conn(0x7f64e40b3e60 0x7f64e4158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.987+0000 7f64e2ffd700 1 --2- 172.21.15.29:0/441699896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e40ce800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f64d80098e0 tx=0x7f64d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be242986f7b6e640 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e27fc700 1 -- 172.21.15.29:0/441699896 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8011630 con 0x7f64e40ce390 2023-10-07T10:45:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e27fc700 1 -- 172.21.15.29:0/441699896 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8011790 con 0x7f64e40ce390 2023-10-07T10:45:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e27fc700 1 -- 172.21.15.29:0/441699896 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8011b30 con 0x7f64e40ce390 2023-10-07T10:45:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 -- 172.21.15.29:0/441699896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 msgr2=0x7f64e40ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 --2- 172.21.15.29:0/441699896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e40ce800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f64d80098e0 tx=0x7f64d80091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 -- 172.21.15.29:0/441699896 shutdown_connections 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 --2- 172.21.15.29:0/441699896 >> v2:172.21.15.144:3300/0 conn(0x7f64e40b3e60 0x7f64e4158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 --2- 172.21.15.29:0/441699896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e40ce800 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 -- 172.21.15.29:0/441699896 >> 172.21.15.29:0/441699896 conn(0x7f64e4067610 msgr2=0x7f64e4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.988+0000 7f64e9bff700 1 -- 172.21.15.29:0/441699896 shutdown_connections 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.989+0000 7f64e9bff700 1 -- 172.21.15.29:0/441699896 wait complete. 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.990+0000 7f64e9bff700 1 Processor -- start 2023-10-07T10:45:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.990+0000 7f64e9bff700 1 -- start start 2023-10-07T10:45:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e9bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e40b00b0 0x7f64e41671b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e9bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 0x7f64e4147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e9bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e4159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e9bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e41676f0 con 0x7f64e40b3e60 2023-10-07T10:45:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e9bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e4148470 con 0x7f64e40b00b0 2023-10-07T10:45:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e9bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e41681a0 con 0x7f64e40ce390 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 0x7f64e4147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 0x7f64e4147f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59932/0 (socket says 172.21.15.29:59932) 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e2ffd700 1 -- 172.21.15.29:0/1544861142 learned_addr learned my addr 172.21.15.29:0/1544861142 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e37fe700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e40b00b0 0x7f64e41671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.991+0000 7f64e3fff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e4159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e2ffd700 1 -- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 msgr2=0x7f64e4159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e2ffd700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e4159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e2ffd700 1 -- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e40b00b0 msgr2=0x7f64e41671b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e2ffd700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e40b00b0 0x7f64e41671b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e2ffd700 1 -- 172.21.15.29:0/1544861142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d40093c0 con 0x7f64e40b3e60 2023-10-07T10:45:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e2ffd700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 0x7f64e4147f30 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f64e40b0cb0 tx=0x7f64d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d801f070 con 0x7f64e40b3e60 2023-10-07T10:45:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8000bc0 con 0x7f64e40b3e60 2023-10-07T10:45:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d8009420 con 0x7f64e40b3e60 2023-10-07T10:45:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.992+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8011b30 con 0x7f64e40b3e60 2023-10-07T10:45:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.993+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e4155840 con 0x7f64e40b3e60 2023-10-07T10:45:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.994+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e40ad810 con 0x7f64e40b3e60 2023-10-07T10:45:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.996+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f64d8023070 con 0x7f64e40b3e60 2023-10-07T10:45:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.996+0000 7f64cb7fe700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 0x7f64d0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.996+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f64d8085d70 con 0x7f64e40b3e60 2023-10-07T10:45:01.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.997+0000 7f64e37fe700 1 -- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 msgr2=0x7f64d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:01.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:01.997+0000 7f64e37fe700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 0x7f64d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:02.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.001+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64d8052ac0 con 0x7f64e40b3e60 2023-10-07T10:45:02.106 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.106+0000 7fcc81ab11c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:45:02.163 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:02.162+0000 7fbd574fd1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:45:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.198+0000 7f64e37fe700 1 -- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 msgr2=0x7f64d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.198+0000 7f64e37fe700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 0x7f64d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:02.198 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.199+0000 7fcc81ab11c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:45:02.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.249+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64e40c3f20 con 0x7f64e40b3e60 2023-10-07T10:45:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.249+0000 7f64cb7fe700 1 -- 172.21.15.29:0/1544861142 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f64d802a020 con 0x7f64e40b3e60 2023-10-07T10:45:02.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:02.249 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:44:40.623432+0000","modified":"2023-10-07T10:44:44.142903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8634":{"gid":8634,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3670759585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3670759585},{"type":"v1","addr":"172.21.15.29:6835","nonce":3670759585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 msgr2=0x7f64d0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 0x7f64d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 msgr2=0x7f64e4147f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 0x7f64e4147f30 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f64e40b0cb0 tx=0x7f64d8017000 comp rx=0 tx=0).stop 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 shutdown_connections 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f64d00670e0 0x7f64d0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64e40ce390 0x7f64e4159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e40b00b0 0x7f64e41671b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 --2- 172.21.15.29:0/1544861142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64e40b3e60 0x7f64e4147f30 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 >> 172.21.15.29:0/1544861142 conn(0x7f64e4067610 msgr2=0x7f64e40b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:02.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 shutdown_connections 2023-10-07T10:45:02.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.252+0000 7f64e9bff700 1 -- 172.21.15.29:0/1544861142 wait complete. 2023-10-07T10:45:02.253 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-07T10:45:02.258 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:02.257+0000 7fbd574fd1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:45:02.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:02.351 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:02.350+0000 7fbd574fd1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:45:02.423 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.424+0000 7fcc81ab11c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:45:02.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:02.444+0000 7fbd574fd1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:45:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.472+0000 7fa4f43ff700 1 Processor -- start 2023-10-07T10:45:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.472+0000 7fa4f43ff700 1 -- start start 2023-10-07T10:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f43ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4ec0b04b0 0x7fa4ec156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f43ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1537c0 0x7fa4ec157240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f43ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ec159670 con 0x7fa4ec1533f0 2023-10-07T10:45:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f43ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec15a680 con 0x7fa4ec0b04b0 2023-10-07T10:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f43ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0b2070 con 0x7fa4ec1537c0 2023-10-07T10:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f199a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4ec0b04b0 0x7fa4ec156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f199a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4ec0b04b0 0x7fa4ec156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59938/0 (socket says 172.21.15.29:59938) 2023-10-07T10:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f199a700 1 -- 172.21.15.29:0/3719442759 learned_addr learned my addr 172.21.15.29:0/3719442759 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.474+0000 7fa4f299c700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1537c0 0x7fa4ec157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f1199700 1 -- 172.21.15.29:0/3719442759 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974032280 0 0) 0x7fa4ec159670 con 0x7fa4ec1533f0 2023-10-07T10:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f1199700 1 -- 172.21.15.29:0/3719442759 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d8004680 con 0x7fa4ec1533f0 2023-10-07T10:45:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f199a700 1 -- 172.21.15.29:0/3719442759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1537c0 msgr2=0x7fa4ec157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f199a700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1537c0 0x7fa4ec157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f199a700 1 -- 172.21.15.29:0/3719442759 >> v1:172.21.15.144:6789/0 conn(0x7fa4ec1533f0 legacy=0x7fa4ec0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f199a700 1 -- 172.21.15.29:0/3719442759 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec15a1f0 con 0x7fa4ec0b04b0 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f299c700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1537c0 0x7fa4ec157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.475+0000 7fa4f199a700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec156a90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0099e0 tx=0x7fa4dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403f55917b193769 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f1199700 1 -- 172.21.15.29:0/3719442759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00b640 con 0x7fa4ec0b04b0 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f1199700 1 -- 172.21.15.29:0/3719442759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc00b7a0 con 0x7fa4ec0b04b0 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f1199700 1 -- 172.21.15.29:0/3719442759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00bb40 con 0x7fa4ec0b04b0 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 -- 172.21.15.29:0/3719442759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 msgr2=0x7fa4ec156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec156a90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0099e0 tx=0x7fa4dc009250 comp rx=0 tx=0).stop 2023-10-07T10:45:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 -- 172.21.15.29:0/3719442759 shutdown_connections 2023-10-07T10:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1537c0 0x7fa4ec157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/3719442759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec156a90 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 -- 172.21.15.29:0/3719442759 >> 172.21.15.29:0/3719442759 conn(0x7fa4ec0680a0 msgr2=0x7fa4ec0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 -- 172.21.15.29:0/3719442759 shutdown_connections 2023-10-07T10:45:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.476+0000 7fa4f43ff700 1 -- 172.21.15.29:0/3719442759 wait complete. 2023-10-07T10:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.478+0000 7fa4f43ff700 1 Processor -- start 2023-10-07T10:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.478+0000 7fa4f43ff700 1 -- start start 2023-10-07T10:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f43ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec1513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f43ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1533f0 0x7fa4ec151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f43ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ec1537c0 0x7fa4ec0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f43ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1528d0 con 0x7fa4ec0b04b0 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f43ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec150f50 con 0x7fa4ec1537c0 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f43ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec150340 con 0x7fa4ec1533f0 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f219b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec1513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f219b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec1513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59954/0 (socket says 172.21.15.29:59954) 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f219b700 1 -- 172.21.15.29:0/23495768 learned_addr learned my addr 172.21.15.29:0/23495768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f199a700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1533f0 0x7fa4ec151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.479+0000 7fa4f299c700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ec1537c0 0x7fa4ec0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f199a700 1 -- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ec1537c0 msgr2=0x7fa4ec0a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f199a700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ec1537c0 0x7fa4ec0a5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f199a700 1 -- 172.21.15.29:0/23495768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 msgr2=0x7fa4ec1513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f199a700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec1513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f199a700 1 -- 172.21.15.29:0/23495768 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e4009a20 con 0x7fa4ec1533f0 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f219b700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec1513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4f199a700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1533f0 0x7fa4ec151920 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0b2240 tx=0x7fa4dc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.480+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00bb20 con 0x7fa4ec1533f0 2023-10-07T10:45:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.481+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc000c90 con 0x7fa4ec1533f0 2023-10-07T10:45:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.481+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc011690 con 0x7fa4ec1533f0 2023-10-07T10:45:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.481+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc0094f0 con 0x7fa4ec1533f0 2023-10-07T10:45:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.481+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec150900 con 0x7fa4ec1533f0 2023-10-07T10:45:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.482+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec04c3c0 con 0x7fa4ec1533f0 2023-10-07T10:45:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.483+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fa4dc011a80 con 0x7fa4ec1533f0 2023-10-07T10:45:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.484+0000 7fa4e27fc700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 0x7fa4d0069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.484+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fa4dc085cc0 con 0x7fa4ec1533f0 2023-10-07T10:45:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.484+0000 7fa4f219b700 1 -- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 msgr2=0x7fa4d0069660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.484+0000 7fa4f219b700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 0x7fa4d0069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.488+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4dc08c080 con 0x7fa4ec1533f0 2023-10-07T10:45:02.529 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.529+0000 7fcc81ab11c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:45:02.630 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.630+0000 7fcc81ab11c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:45:02.632 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:02.632+0000 7fbd574fd1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:45:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.685+0000 7fa4f219b700 1 -- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 msgr2=0x7fa4d0069660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.685+0000 7fa4f219b700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 0x7fa4d0069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.692+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4ec06f660 con 0x7fa4ec1533f0 2023-10-07T10:45:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.693+0000 7fa4e27fc700 1 -- 172.21.15.29:0/23495768 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16477 (secure 0 0 0) 0x7fa4dc052b60 con 0x7fa4ec1533f0 2023-10-07T10:45:02.693 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:02.694 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":152,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:56.153475+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:44:38.468584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-07T10:44:39.609895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3633091014":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/1131701620":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/3484621640":"2023-10-08T10:44:56.153443+0000","172.21.15.144:6836/114714":"2023-10-08T10:44:56.153443+0000","172.21.15.144:6834/114714":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/3284926740":"2023-10-08T10:44:56.153443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 msgr2=0x7fa4d0069660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 0x7fa4d0069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1533f0 msgr2=0x7fa4ec151920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1533f0 0x7fa4ec151920 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0b2240 tx=0x7fa4dc017000 comp rx=0 tx=0).stop 2023-10-07T10:45:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 shutdown_connections 2023-10-07T10:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fa4d00671b0 0x7fa4d0069660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ec1537c0 0x7fa4ec0a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec0b04b0 0x7fa4ec1513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 --2- 172.21.15.29:0/23495768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ec1533f0 0x7fa4ec151920 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 >> 172.21.15.29:0/23495768 conn(0x7fa4ec0680a0 msgr2=0x7fa4ec0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 shutdown_connections 2023-10-07T10:45:02.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.696+0000 7fa4f43ff700 1 -- 172.21.15.29:0/23495768 wait complete. 2023-10-07T10:45:02.714 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T10:45:02.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:45:02.728 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:02.728+0000 7fbd574fd1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:45:02.734 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.734+0000 7fcc81ab11c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:45:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.912+0000 7f2110dff700 1 Processor -- start 2023-10-07T10:45:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.913+0000 7f2110dff700 1 -- start start 2023-10-07T10:45:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f2110dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c159ec0 0x7f210c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f2110dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f210c0af400 0x7f210c0af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f2110dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c14d790 con 0x7f210c0ad370 2023-10-07T10:45:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f2110dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c0ae5d0 con 0x7f210c0af400 2023-10-07T10:45:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f2110dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c15a850 con 0x7f210c159ec0 2023-10-07T10:45:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f210ad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f210c0af400 0x7f210c0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f210ad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f210c0af400 0x7f210c0af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51996/0 (socket says 172.21.15.29:51996) 2023-10-07T10:45:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f210ad9d700 1 -- 172.21.15.29:0/204433651 learned_addr learned my addr 172.21.15.29:0/204433651 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.914+0000 7f2109d9b700 1 --2- 172.21.15.29:0/204433651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c159ec0 0x7f210c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210ad9d700 1 -- 172.21.15.29:0/204433651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c159ec0 msgr2=0x7f210c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210ad9d700 1 --2- 172.21.15.29:0/204433651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c159ec0 0x7f210c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210ad9d700 1 -- 172.21.15.29:0/204433651 >> v1:172.21.15.144:6789/0 conn(0x7f210c0ad370 legacy=0x7f210c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210ad9d700 1 -- 172.21.15.29:0/204433651 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c0ad740 con 0x7f210c0af400 2023-10-07T10:45:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210ad9d700 1 --2- 172.21.15.29:0/204433651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0af400 0x7f210c0af810 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2104009f20 tx=0x7f2104009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4846d92425998124 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210959a700 1 -- 172.21.15.29:0/204433651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210400e070 con 0x7f210c0af400 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210959a700 1 -- 172.21.15.29:0/204433651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210400b9d0 con 0x7f210c0af400 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.915+0000 7f210959a700 1 -- 172.21.15.29:0/204433651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040092e0 con 0x7f210c0af400 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 -- 172.21.15.29:0/204433651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0af400 msgr2=0x7f210c0af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 --2- 172.21.15.29:0/204433651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0af400 0x7f210c0af810 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2104009f20 tx=0x7f2104009840 comp rx=0 tx=0).stop 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 -- 172.21.15.29:0/204433651 shutdown_connections 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 --2- 172.21.15.29:0/204433651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0af400 0x7f210c0af810 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 --2- 172.21.15.29:0/204433651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c159ec0 0x7f210c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 -- 172.21.15.29:0/204433651 >> 172.21.15.29:0/204433651 conn(0x7f210c0680a0 msgr2=0x7f210c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 -- 172.21.15.29:0/204433651 shutdown_connections 2023-10-07T10:45:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.916+0000 7f2110dff700 1 -- 172.21.15.29:0/204433651 wait complete. 2023-10-07T10:45:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.918+0000 7f2110dff700 1 Processor -- start 2023-10-07T10:45:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.918+0000 7f2110dff700 1 -- start start 2023-10-07T10:45:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2110dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 0x7f210c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2110dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c0af400 0x7f210c1504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2110dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f210c159ec0 0x7f210c0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2110dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14dac0 con 0x7f210c0ad370 2023-10-07T10:45:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2110dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14dda0 con 0x7f210c159ec0 2023-10-07T10:45:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2110dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14e080 con 0x7f210c0af400 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f210a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 0x7f210c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f210a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 0x7f210c14ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52006/0 (socket says 172.21.15.29:52006) 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f210a59c700 1 -- 172.21.15.29:0/4098457064 learned_addr learned my addr 172.21.15.29:0/4098457064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f210ad9d700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f210c159ec0 0x7f210c0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f2109d9b700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c0af400 0x7f210c1504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.919+0000 7f210a59c700 1 -- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c0af400 msgr2=0x7f210c1504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f210a59c700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c0af400 0x7f210c1504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f210a59c700 1 -- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f210c159ec0 msgr2=0x7f210c0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f210a59c700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f210c159ec0 0x7f210c0caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f210a59c700 1 -- 172.21.15.29:0/4098457064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104009ae0 con 0x7f210c0ad370 2023-10-07T10:45:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f210a59c700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 0x7f210c14ff70 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c590 tx=0x7f20fc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc011590 con 0x7f210c0ad370 2023-10-07T10:45:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc011710 con 0x7f210c0ad370 2023-10-07T10:45:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc011ad0 con 0x7f210c0ad370 2023-10-07T10:45:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c14e360 con 0x7f210c0ad370 2023-10-07T10:45:02.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.920+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c0cb4d0 con 0x7f210c0ad370 2023-10-07T10:45:02.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.921+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c14e880 con 0x7f210c0ad370 2023-10-07T10:45:02.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.922+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f20fc01b3b0 con 0x7f210c0ad370 2023-10-07T10:45:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.923+0000 7f20fa7fc700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 0x7f20ec069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.923+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f20fc083fa0 con 0x7f210c0ad370 2023-10-07T10:45:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.923+0000 7f2109d9b700 1 -- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 msgr2=0x7f20ec069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.923+0000 7f2109d9b700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 0x7f20ec069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:02.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:02.927+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f210c14e880 con 0x7f210c0ad370 2023-10-07T10:45:02.943 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:02.943+0000 7fcc81ab11c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:45:03.049 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:03.049+0000 7fcc81ab11c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:45:03.070 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:03.069+0000 7fbd574fd1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:45:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.124+0000 7f2109d9b700 1 -- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 msgr2=0x7f20ec069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:03.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.124+0000 7f2109d9b700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 0x7f20ec069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:03.158 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:03.157+0000 7fbd574fd1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:03.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.176+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f210c14e880 con 0x7f210c0ad370 2023-10-07T10:45:03.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.177+0000 7f20fa7fc700 1 -- 172.21.15.29:0/4098457064 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f210c14e880 con 0x7f210c0ad370 2023-10-07T10:45:03.177 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:03.177 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:44:40.623432+0000","modified":"2023-10-07T10:44:44.142903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8634":{"gid":8634,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3670759585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3670759585},{"type":"v1","addr":"172.21.15.29:6835","nonce":3670759585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.179+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 msgr2=0x7f20ec069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.179+0000 7f2110dff700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 0x7f20ec069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.179+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 msgr2=0x7f210c14ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.179+0000 7f2110dff700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 0x7f210c14ff70 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c590 tx=0x7f20fc00c8a0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 shutdown_connections 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f20ec0670e0 0x7f20ec069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f210c159ec0 0x7f210c0caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f210c0ad370 0x7f210c14ff70 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 --2- 172.21.15.29:0/4098457064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f210c0af400 0x7f210c1504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 >> 172.21.15.29:0/4098457064 conn(0x7f210c0680a0 msgr2=0x7f210c0aadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:03.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 shutdown_connections 2023-10-07T10:45:03.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.180+0000 7f2110dff700 1 -- 172.21.15.29:0/4098457064 wait complete. 2023-10-07T10:45:03.181 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-07T10:45:03.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:45:03.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:03.410+0000 7fcc81ab11c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:45:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.443+0000 7fe9fa3ff700 1 Processor -- start 2023-10-07T10:45:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.443+0000 7fe9fa3ff700 1 -- start start 2023-10-07T10:45:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.444+0000 7fe9fa3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f4157300 0x7fe9f40b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.444+0000 7fe9fa3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe9f41576d0 0x7fe9f40b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.444+0000 7fe9fa3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9f406f410 0x7fe9f4067610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.444+0000 7fe9fa3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f40cb330 con 0x7fe9f406f410 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.444+0000 7fe9fa3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f406f980 con 0x7fe9f41576d0 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.444+0000 7fe9fa3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f40cbe90 con 0x7fe9f4157300 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f899c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9f406f410 0x7fe9f4067610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f37fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe9f41576d0 0x7fe9f40b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f899c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9f406f410 0x7fe9f4067610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52016/0 (socket says 172.21.15.29:52016) 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f3fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f4157300 0x7fe9f40b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f899c700 1 -- 172.21.15.29:0/1654534739 learned_addr learned my addr 172.21.15.29:0/1654534739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:03.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f3fff700 1 -- 172.21.15.29:0/1654534739 >> v2:172.21.15.144:3300/0 conn(0x7fe9f41576d0 msgr2=0x7fe9f40b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:03.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f3fff700 1 --2- 172.21.15.29:0/1654534739 >> v2:172.21.15.144:3300/0 conn(0x7fe9f41576d0 0x7fe9f40b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f3fff700 1 -- 172.21.15.29:0/1654534739 >> v2:172.21.15.29:3300/0 conn(0x7fe9f406f410 msgr2=0x7fe9f4067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f3fff700 1 --2- 172.21.15.29:0/1654534739 >> v2:172.21.15.29:3300/0 conn(0x7fe9f406f410 0x7fe9f4067610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f3fff700 1 -- 172.21.15.29:0/1654534739 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f40cb560 con 0x7fe9f4157300 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.445+0000 7fe9f37fe700 1 --2- 172.21.15.29:0/1654534739 >> v2:172.21.15.144:3300/0 conn(0x7fe9f41576d0 0x7fe9f40b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.446+0000 7fe9f3fff700 1 --2- 172.21.15.29:0/1654534739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f4157300 0x7fe9f40b2380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4069700 tx=0x7fe9e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d58776360b20ba server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.446+0000 7fe9f2ffd700 1 -- 172.21.15.29:0/1654534739 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4011630 con 0x7fe9f4157300 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.446+0000 7fe9f2ffd700 1 -- 172.21.15.29:0/1654534739 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e40117b0 con 0x7fe9f4157300 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.446+0000 7fe9f2ffd700 1 -- 172.21.15.29:0/1654534739 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4011b70 con 0x7fe9f4157300 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.446+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/1654534739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f4157300 msgr2=0x7fe9f40b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.446+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/1654534739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f4157300 0x7fe9f40b2380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4069700 tx=0x7fe9e40091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/1654534739 shutdown_connections 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/1654534739 >> v2:172.21.15.29:3300/0 conn(0x7fe9f406f410 0x7fe9f4067610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/1654534739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f4157300 0x7fe9f40b2380 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/1654534739 >> v2:172.21.15.144:3300/0 conn(0x7fe9f41576d0 0x7fe9f40b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/1654534739 >> 172.21.15.29:0/1654534739 conn(0x7fe9f4069190 msgr2=0x7fe9f406a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/1654534739 shutdown_connections 2023-10-07T10:45:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.447+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/1654534739 wait complete. 2023-10-07T10:45:03.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.448+0000 7fe9fa3ff700 1 Processor -- start 2023-10-07T10:45:03.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.449+0000 7fe9fa3ff700 1 -- start start 2023-10-07T10:45:03.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9fa3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f406f410 0x7fe9f40acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9fa3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 0x7fe9f416a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9fa3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40af6b0 0x7fe9f40aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9fa3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f415b1b0 con 0x7fe9f4157300 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9fa3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f40afb30 con 0x7fe9f40af6b0 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9fa3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f40af220 con 0x7fe9f406f410 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 0x7fe9f416a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 0x7fe9f416a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52022/0 (socket says 172.21.15.29:52022) 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 -- 172.21.15.29:0/2219981768 learned_addr learned my addr 172.21.15.29:0/2219981768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f3fff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f406f410 0x7fe9f40acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f899c700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40af6b0 0x7fe9f40aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 -- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f406f410 msgr2=0x7fe9f40acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f406f410 0x7fe9f40acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 -- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40af6b0 msgr2=0x7fe9f40aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40af6b0 0x7fe9f40aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.450+0000 7fe9f37fe700 1 -- 172.21.15.29:0/2219981768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e8009420 con 0x7fe9f4157300 2023-10-07T10:45:03.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.451+0000 7fe9f37fe700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 0x7fe9f416a3d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800ecb0 tx=0x7fe9e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:03.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.451+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e800c850 con 0x7fe9f4157300 2023-10-07T10:45:03.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.451+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e800c9b0 con 0x7fe9f4157300 2023-10-07T10:45:03.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.451+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e4009420 con 0x7fe9f4157300 2023-10-07T10:45:03.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.451+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e800cd30 con 0x7fe9f4157300 2023-10-07T10:45:03.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.451+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f40ae8f0 con 0x7fe9f4157300 2023-10-07T10:45:03.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.452+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f404c3c0 con 0x7fe9f4157300 2023-10-07T10:45:03.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.453+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fe9e8013680 con 0x7fe9f4157300 2023-10-07T10:45:03.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.454+0000 7fe9dbfff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 0x7fe9e0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.454+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fe9e8085140 con 0x7fe9f4157300 2023-10-07T10:45:03.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.454+0000 7fe9f3fff700 1 -- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 msgr2=0x7fe9e0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:03.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.454+0000 7fe9f3fff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 0x7fe9e0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:03.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.459+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9e8055880 con 0x7fe9f4157300 2023-10-07T10:45:03.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:03.507+0000 7fcc81ab11c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.655+0000 7fe9f3fff700 1 -- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 msgr2=0x7fe9e0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.655+0000 7fe9f3fff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 0x7fe9e0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:03.694 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:03.693+0000 7fbd574fd1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:45:03.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.704+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9f40af360 con 0x7fe9f4157300 2023-10-07T10:45:03.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.704+0000 7fe9dbfff700 1 -- 172.21.15.29:0/2219981768 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fe9e8055410 con 0x7fe9f4157300 2023-10-07T10:45:03.704 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:03.704 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:44:40.623432+0000","modified":"2023-10-07T10:44:44.142903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8634":{"gid":8634,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3670759585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3670759585},{"type":"v1","addr":"172.21.15.29:6835","nonce":3670759585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.706+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 msgr2=0x7fe9e0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.706+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 0x7fe9e0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.706+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 msgr2=0x7fe9f416a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.706+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 0x7fe9f416a3d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800ecb0 tx=0x7fe9e80091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 shutdown_connections 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe9e00670e0 0x7fe9e0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40af6b0 0x7fe9f40aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f406f410 0x7fe9f40acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 --2- 172.21.15.29:0/2219981768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9f4157300 0x7fe9f416a3d0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 >> 172.21.15.29:0/2219981768 conn(0x7fe9f4069190 msgr2=0x7fe9f4153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:03.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 shutdown_connections 2023-10-07T10:45:03.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.707+0000 7fe9fa3ff700 1 -- 172.21.15.29:0/2219981768 wait complete. 2023-10-07T10:45:03.708 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-07T10:45:03.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:03.877 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:03.876+0000 7fbd574fd1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:45:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.927+0000 7fceed5ff700 1 Processor -- start 2023-10-07T10:45:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.927+0000 7fceed5ff700 1 -- start start 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fceed5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee80b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fceed5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcee80b0540 0x7fcee815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fceed5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee80b0980 con 0x7fcee8153cc0 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fceed5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee8154090 con 0x7fcee80b0540 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fceed5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8159f60 con 0x7fcee815ade0 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fcee77fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcee80b0540 0x7fcee815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fcee67fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee80b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fcee77fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcee80b0540 0x7fcee815c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:40050/0 (socket says 172.21.15.29:40050) 2023-10-07T10:45:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.929+0000 7fcee77fe700 1 -- 172.21.15.29:0/908655591 learned_addr learned my addr 172.21.15.29:0/908655591 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee77fe700 1 -- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 msgr2=0x7fcee80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee77fe700 1 --2- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee80b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee77fe700 1 -- 172.21.15.29:0/908655591 >> v1:172.21.15.144:6789/0 conn(0x7fcee8153cc0 legacy=0x7fcee80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee77fe700 1 -- 172.21.15.29:0/908655591 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee815a240 con 0x7fcee80b0540 2023-10-07T10:45:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee77fe700 1 --2- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee80b0540 0x7fcee815c240 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009f20 tx=0x7fcee0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19ca3a11f02c5b4f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee5ffb700 1 -- 172.21.15.29:0/908655591 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee000e070 con 0x7fcee80b0540 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee5ffb700 1 -- 172.21.15.29:0/908655591 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee000b9d0 con 0x7fcee80b0540 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.930+0000 7fcee5ffb700 1 -- 172.21.15.29:0/908655591 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee00092e0 con 0x7fcee80b0540 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 -- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee80b0540 msgr2=0x7fcee815c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 --2- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee80b0540 0x7fcee815c240 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009f20 tx=0x7fcee0009840 comp rx=0 tx=0).stop 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 -- 172.21.15.29:0/908655591 shutdown_connections 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 --2- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee80b0540 0x7fcee815c240 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 --2- 172.21.15.29:0/908655591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee80b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 -- 172.21.15.29:0/908655591 >> 172.21.15.29:0/908655591 conn(0x7fcee8067610 msgr2=0x7fcee8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 -- 172.21.15.29:0/908655591 shutdown_connections 2023-10-07T10:45:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.931+0000 7fceed5ff700 1 -- 172.21.15.29:0/908655591 wait complete. 2023-10-07T10:45:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.932+0000 7fceed5ff700 1 Processor -- start 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.932+0000 7fceed5ff700 1 -- start start 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.933+0000 7fceed5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 0x7fcee8158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.933+0000 7fceed5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee8153cc0 0x7fcee8154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.933+0000 7fceed5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee8143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.933+0000 7fceed5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8156940 con 0x7fcee80b0540 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.933+0000 7fceed5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8155250 con 0x7fcee8153cc0 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.933+0000 7fceed5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee81554e0 con 0x7fcee815ade0 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 0x7fcee8158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 0x7fcee8158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52038/0 (socket says 172.21.15.29:52038) 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 -- 172.21.15.29:0/3916681024 learned_addr learned my addr 172.21.15.29:0/3916681024 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee77fe700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee8143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee67fc700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee8153cc0 0x7fcee8154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 -- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 msgr2=0x7fcee8143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee8143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 -- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee8153cc0 msgr2=0x7fcee8154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee8153cc0 0x7fcee8154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee6ffd700 1 -- 172.21.15.29:0/3916681024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0009ae0 con 0x7fcee80b0540 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee77fe700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee8143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.934+0000 7fcee67fc700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee8153cc0 0x7fcee8154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.935+0000 7fcee6ffd700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 0x7fcee8158d60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fced800c7c0 tx=0x7fced8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.935+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8011590 con 0x7fcee80b0540 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.935+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee8156010 con 0x7fcee80b0540 2023-10-07T10:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.935+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced80116f0 con 0x7fcee80b0540 2023-10-07T10:45:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.935+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8011ab0 con 0x7fcee80b0540 2023-10-07T10:45:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.935+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee81565d0 con 0x7fcee80b0540 2023-10-07T10:45:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.936+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee8155620 con 0x7fcee80b0540 2023-10-07T10:45:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.937+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fced801b3b0 con 0x7fcee80b0540 2023-10-07T10:45:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.938+0000 7fceceffd700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 0x7fced4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.938+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fced8083fc0 con 0x7fcee80b0540 2023-10-07T10:45:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.938+0000 7fcee67fc700 1 -- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 msgr2=0x7fced4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.938+0000 7fcee67fc700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 0x7fced4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:03.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:03.942+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fced8050d20 con 0x7fcee80b0540 2023-10-07T10:45:04.100 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:04.100+0000 7fcc81ab11c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:45:04.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.139+0000 7fcee67fc700 1 -- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 msgr2=0x7fced4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:04.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.139+0000 7fcee67fc700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 0x7fced4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.149+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcee804c3c0 con 0x7fcee80b0540 2023-10-07T10:45:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.150+0000 7fceceffd700 1 -- 172.21.15.29:0/3916681024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16477 (secure 0 0 0) 0x7fced8017070 con 0x7fcee80b0540 2023-10-07T10:45:04.151 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:04.151 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":152,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:44:56.153475+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:44:38.468584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-07T10:44:39.609895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3633091014":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/1131701620":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/3484621640":"2023-10-08T10:44:56.153443+0000","172.21.15.144:6836/114714":"2023-10-08T10:44:56.153443+0000","172.21.15.144:6834/114714":"2023-10-08T10:44:56.153443+0000","172.21.15.144:0/3284926740":"2023-10-08T10:44:56.153443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 msgr2=0x7fced4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 0x7fced4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 msgr2=0x7fcee8158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 0x7fcee8158d60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fced800c7c0 tx=0x7fced8007f70 comp rx=0 tx=0).stop 2023-10-07T10:45:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 shutdown_connections 2023-10-07T10:45:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fced40670e0 0x7fced4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcee815ade0 0x7fcee8143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee80b0540 0x7fcee8158d60 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 --2- 172.21.15.29:0/3916681024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee8153cc0 0x7fcee8154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 >> 172.21.15.29:0/3916681024 conn(0x7fcee8067610 msgr2=0x7fcee80af340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 shutdown_connections 2023-10-07T10:45:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.153+0000 7fceed5ff700 1 -- 172.21.15.29:0/3916681024 wait complete. 2023-10-07T10:45:04.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T10:45:04.257 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:04.256+0000 7fbd574fd1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:04.301 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:04.301+0000 7fcc81ab11c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:45:04.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.371+0000 7fafa9fff700 1 Processor -- start 2023-10-07T10:45:04.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.371+0000 7fafa9fff700 1 -- start start 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa9fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafa40ae8b0 0x7fafa40b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa9fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa4157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa9fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa4158ad0 con 0x7fafa4157290 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa9fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa40b1cc0 con 0x7fafa40ae8b0 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa9fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa415b140 con 0x7fafa40b1840 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafa40ae8b0 0x7fafa40b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafa40ae8b0 0x7fafa40b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52052/0 (socket says 172.21.15.29:52052) 2023-10-07T10:45:04.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 -- 172.21.15.29:0/1970962694 learned_addr learned my addr 172.21.15.29:0/1970962694 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa37fe700 1 --1- 172.21.15.29:0/1970962694 >> v1:172.21.15.29:6789/0 conn(0x7fafa4157290 0x7fafa40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47540/0 (socket says 172.21.15.29:47540) 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa3fff700 1 --2- 172.21.15.29:0/1970962694 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa4157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 -- 172.21.15.29:0/1970962694 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 msgr2=0x7fafa4157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 --2- 172.21.15.29:0/1970962694 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa4157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 -- 172.21.15.29:0/1970962694 >> v1:172.21.15.29:6789/0 conn(0x7fafa4157290 legacy=0x7fafa40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.373+0000 7fafa2ffd700 1 -- 172.21.15.29:0/1970962694 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa415b420 con 0x7fafa40ae8b0 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.374+0000 7fafa2ffd700 1 --2- 172.21.15.29:0/1970962694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa40ae8b0 0x7fafa40b1300 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7faf980099b0 tx=0x7faf98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52285f20685a7bc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.374+0000 7fafa27fc700 1 -- 172.21.15.29:0/1970962694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98009ea0 con 0x7fafa40ae8b0 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.374+0000 7fafa27fc700 1 -- 172.21.15.29:0/1970962694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9800b860 con 0x7fafa40ae8b0 2023-10-07T10:45:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.374+0000 7fafa27fc700 1 -- 172.21.15.29:0/1970962694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9800bc00 con 0x7fafa40ae8b0 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 -- 172.21.15.29:0/1970962694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa40ae8b0 msgr2=0x7fafa40b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 --2- 172.21.15.29:0/1970962694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa40ae8b0 0x7fafa40b1300 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7faf980099b0 tx=0x7faf98009250 comp rx=0 tx=0).stop 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 -- 172.21.15.29:0/1970962694 shutdown_connections 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 --2- 172.21.15.29:0/1970962694 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa4157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 --2- 172.21.15.29:0/1970962694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa40ae8b0 0x7fafa40b1300 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 -- 172.21.15.29:0/1970962694 >> 172.21.15.29:0/1970962694 conn(0x7fafa40680a0 msgr2=0x7fafa40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 -- 172.21.15.29:0/1970962694 shutdown_connections 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.375+0000 7fafa9fff700 1 -- 172.21.15.29:0/1970962694 wait complete. 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.377+0000 7fafa9fff700 1 Processor -- start 2023-10-07T10:45:04.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.377+0000 7fafa9fff700 1 -- start start 2023-10-07T10:45:04.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa9fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa41506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa9fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 0x7fafa40c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa9fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa4150bf0 0x7fafa416d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa9fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40af270 con 0x7fafa4157290 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa9fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40af550 con 0x7fafa4150bf0 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa9fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40af830 con 0x7fafa40b1840 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 0x7fafa40c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 0x7fafa40c5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52068/0 (socket says 172.21.15.29:52068) 2023-10-07T10:45:04.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa2ffd700 1 -- 172.21.15.29:0/3292775504 learned_addr learned my addr 172.21.15.29:0/3292775504 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:04.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa37fe700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa41506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.378+0000 7fafa3fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa4150bf0 0x7fafa416d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa2ffd700 1 -- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 msgr2=0x7fafa41506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa2ffd700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa41506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa2ffd700 1 -- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa4150bf0 msgr2=0x7fafa416d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa2ffd700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa4150bf0 0x7fafa416d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa2ffd700 1 -- 172.21.15.29:0/3292775504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980094f0 con 0x7fafa4157290 2023-10-07T10:45:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa37fe700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa41506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa2ffd700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 0x7fafa40c5cd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fafa4068dd0 tx=0x7faf98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98000d10 con 0x7fafa4157290 2023-10-07T10:45:04.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9800bbe0 con 0x7fafa4157290 2023-10-07T10:45:04.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98011670 con 0x7fafa4157290 2023-10-07T10:45:04.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.379+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa40afb10 con 0x7fafa4157290 2023-10-07T10:45:04.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.380+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa40c7f70 con 0x7fafa4157290 2023-10-07T10:45:04.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.381+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa40c8190 con 0x7fafa4157290 2023-10-07T10:45:04.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.382+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7faf98011aa0 con 0x7fafa4157290 2023-10-07T10:45:04.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.383+0000 7faf8b7fe700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 0x7faf90069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.383+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6383+0+0 (secure 0 0 0) 0x7faf98086200 con 0x7fafa4157290 2023-10-07T10:45:04.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.383+0000 7fafa37fe700 1 -- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 msgr2=0x7faf90069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:04.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.383+0000 7fafa37fe700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 0x7faf90069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:04.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.387+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf98052f50 con 0x7fafa4157290 2023-10-07T10:45:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.584+0000 7fafa37fe700 1 -- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 msgr2=0x7faf90069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.584+0000 7fafa37fe700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 0x7faf90069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:04.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.635+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fafa40af3b0 con 0x7fafa4157290 2023-10-07T10:45:04.635 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T10:45:04.635+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T10:45:04.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.640+0000 7faf8b7fe700 1 -- 172.21.15.29:0/3292775504 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) v1 ==== 161+0+0 (secure 0 0 0) 0x7fafa40af3b0 con 0x7fafa4157290 2023-10-07T10:45:04.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 msgr2=0x7faf90069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 0x7faf90069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 msgr2=0x7fafa40c5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 0x7fafa40c5cd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fafa4068dd0 tx=0x7faf98017000 comp rx=0 tx=0).stop 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 shutdown_connections 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7faf900670e0 0x7faf90069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa4150bf0 0x7fafa416d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa40b1840 0x7fafa41506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 --2- 172.21.15.29:0/3292775504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafa4157290 0x7fafa40c5cd0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.643+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 >> 172.21.15.29:0/3292775504 conn(0x7fafa40680a0 msgr2=0x7fafa4158c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.644+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 shutdown_connections 2023-10-07T10:45:04.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.644+0000 7fafa9fff700 1 -- 172.21.15.29:0/3292775504 wait complete. 2023-10-07T10:45:04.645 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T10:45:04.648 INFO:tasks.ceph.mds.c.smithi029.stderr: -2153> 2023-10-07T10:44:47.099+0000 7f25295f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T10:45:04.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T10:45:04.679 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:04.679+0000 7fbd574fd1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:45:04.697 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T10:45:04.703 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:04.704+0000 7fcc81ab11c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:04.864 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:04.863+0000 7fbd574fd1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.880+0000 7f29549ff700 1 Processor -- start 2023-10-07T10:45:04.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.880+0000 7f29549ff700 1 -- start start 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f29549ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2950159790 0x7f295006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f29549ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f29500c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f29549ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295006b850 con 0x7f295006a8c0 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f29549ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295006a430 con 0x7f2950159790 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f29549ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2950159b60 con 0x7f295006b3f0 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f294dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f295006a8c0 0x7f29500c6f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47548/0 (socket says 172.21.15.29:47548) 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f294dd9b700 1 -- 172.21.15.29:0/156673231 learned_addr learned my addr 172.21.15.29:0/156673231 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f294e59c700 1 --2- 172.21.15.29:0/156673231 >> v2:172.21.15.144:3300/0 conn(0x7f2950159790 0x7f295006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.881+0000 7f294ed9d700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f29500c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294d59a700 1 -- 172.21.15.29:0/156673231 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291390861 0 0) 0x7f2944001110 con 0x7f295006a8c0 2023-10-07T10:45:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294d59a700 1 -- 172.21.15.29:0/156673231 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2938003570 con 0x7f295006a8c0 2023-10-07T10:45:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294e59c700 1 -- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 msgr2=0x7f29500c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294e59c700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f29500c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294e59c700 1 -- 172.21.15.29:0/156673231 >> v1:172.21.15.29:6789/0 conn(0x7f295006a8c0 legacy=0x7f29500c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294e59c700 1 -- 172.21.15.29:0/156673231 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2950069820 con 0x7f2950159790 2023-10-07T10:45:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294ed9d700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f29500c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.882+0000 7f294e59c700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2950159790 0x7f295006aeb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f29400098e0 tx=0x7f29400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69638c2d67838200 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.883+0000 7f294d59a700 1 -- 172.21.15.29:0/156673231 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940011630 con 0x7f2950159790 2023-10-07T10:45:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.883+0000 7f294d59a700 1 -- 172.21.15.29:0/156673231 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940011790 con 0x7f2950159790 2023-10-07T10:45:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.883+0000 7f294d59a700 1 -- 172.21.15.29:0/156673231 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940011b30 con 0x7f2950159790 2023-10-07T10:45:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.883+0000 7f29549ff700 1 -- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2950159790 msgr2=0x7f295006aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.883+0000 7f29549ff700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2950159790 0x7f295006aeb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f29400098e0 tx=0x7f29400091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.884+0000 7f29549ff700 1 -- 172.21.15.29:0/156673231 shutdown_connections 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.884+0000 7f29549ff700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f29500c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.884+0000 7f29549ff700 1 --2- 172.21.15.29:0/156673231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2950159790 0x7f295006aeb0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.884+0000 7f29549ff700 1 -- 172.21.15.29:0/156673231 >> 172.21.15.29:0/156673231 conn(0x7f29500c9410 msgr2=0x7f29500c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.884+0000 7f29549ff700 1 -- 172.21.15.29:0/156673231 shutdown_connections 2023-10-07T10:45:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.884+0000 7f29549ff700 1 -- 172.21.15.29:0/156673231 wait complete. 2023-10-07T10:45:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.886+0000 7f29549ff700 1 Processor -- start 2023-10-07T10:45:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.886+0000 7f29549ff700 1 -- start start 2023-10-07T10:45:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f29549ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f2950152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f29549ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 0x7f29500c39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f29549ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29500ac100 0x7f29500acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f29549ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29500c61e0 con 0x7f2950159790 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f29549ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29500aa2f0 con 0x7f29500ac100 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f29549ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29500ac570 con 0x7f295006b3f0 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 0x7f29500c39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 0x7f29500c39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52106/0 (socket says 172.21.15.29:52106) 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 -- 172.21.15.29:0/3555073149 learned_addr learned my addr 172.21.15.29:0/3555073149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294ed9d700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29500ac100 0x7f29500acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294e59c700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f2950152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 -- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 msgr2=0x7f2950152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f2950152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 -- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29500ac100 msgr2=0x7f29500acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29500ac100 0x7f29500acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.888+0000 7f294dd9b700 1 -- 172.21.15.29:0/3555073149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948009a20 con 0x7f2950159790 2023-10-07T10:45:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.889+0000 7f294dd9b700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 0x7f29500c39b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f294400c590 tx=0x7f2944007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.889+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944005c00 con 0x7f2950159790 2023-10-07T10:45:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.889+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2944005d80 con 0x7f2950159790 2023-10-07T10:45:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.889+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944007320 con 0x7f2950159790 2023-10-07T10:45:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.889+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2940009420 con 0x7f2950159790 2023-10-07T10:45:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.889+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29500c6710 con 0x7f2950159790 2023-10-07T10:45:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.890+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29500aa460 con 0x7f2950159790 2023-10-07T10:45:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.891+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f294401b3b0 con 0x7f2950159790 2023-10-07T10:45:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.892+0000 7f293e7fc700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 0x7f2930069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.892+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f2944083f40 con 0x7f2950159790 2023-10-07T10:45:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.892+0000 7f294e59c700 1 -- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 msgr2=0x7f2930069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.892+0000 7f294e59c700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 0x7f2930069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:04.896+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2944050c50 con 0x7f2950159790 2023-10-07T10:45:05.048 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.047+0000 7fbd574fd1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:45:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.093+0000 7f294e59c700 1 -- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 msgr2=0x7f2930069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.093+0000 7f294e59c700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 0x7f2930069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:05.159 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.159+0000 7fbd574fd1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.170+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 72) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f2944051020 con 0x7f2950159790 2023-10-07T10:45:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.170+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 msgr2=0x7f2930069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.170+0000 7f293e7fc700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 0x7f2930069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.176+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f295006b850 con 0x7f2950159790 2023-10-07T10:45:05.194 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:05.193+0000 7fcc81ab11c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:45:05.243 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.242+0000 7fbcd4b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.243 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.242+0000 7fbcd4b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.243 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.242+0000 7fbcd4b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.243 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.242+0000 7fbcd4b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.243 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.242+0000 7fbcd4b89700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.243+0000 7fbcd0380700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.243+0000 7fbcd0380700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.243+0000 7fbcd0380700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.243+0000 7fbcd0380700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:05.243+0000 7fbcd0380700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:05.387 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:05.387+0000 7fcc81ab11c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:05.582 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:05.582+0000 7fcc81ab11c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:45:05.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.641+0000 7f293e7fc700 1 -- 172.21.15.29:0/3555073149 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7f295006b850 con 0x7f2950159790 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.643+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 msgr2=0x7f29500c39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.643+0000 7f29549ff700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 0x7f29500c39b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f294400c590 tx=0x7f2944007c70 comp rx=0 tx=0).stop 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 shutdown_connections 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f29300670e0 0x7f2930069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29500ac100 0x7f29500acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f295006b3f0 0x7f2950152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 --2- 172.21.15.29:0/3555073149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2950159790 0x7f29500c39b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 >> 172.21.15.29:0/3555073149 conn(0x7f29500c9410 msgr2=0x7f295015c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 shutdown_connections 2023-10-07T10:45:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.644+0000 7f29549ff700 1 -- 172.21.15.29:0/3555073149 wait complete. 2023-10-07T10:45:05.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T10:45:05.698 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:05.698+0000 7fcc81ab11c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.898+0000 7f467d3ff700 1 Processor -- start 2023-10-07T10:45:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.898+0000 7f467d3ff700 1 -- start start 2023-10-07T10:45:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.899+0000 7f467d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46780ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.899+0000 7f467d3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4678067aa0 0x7f4678067f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.899+0000 7f467d3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4678067610 con 0x7f467815a730 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.899+0000 7f467d3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46780caf20 con 0x7f4678067aa0 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.899+0000 7f467d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46780c1e70 con 0x7f46780ca590 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4678067aa0 0x7f4678067f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4678067aa0 0x7f4678067f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52112/0 (socket says 172.21.15.29:52112) 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 -- 172.21.15.29:0/1378071394 learned_addr learned my addr 172.21.15.29:0/1378071394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46767fc700 1 --2- 172.21.15.29:0/1378071394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46780ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f4675ffb700 1 -- 172.21.15.29:0/1378071394 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781769971 0 0) 0x7f4678067610 con 0x7f467815a730 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f4675ffb700 1 -- 172.21.15.29:0/1378071394 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660003570 con 0x7f467815a730 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 -- 172.21.15.29:0/1378071394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 msgr2=0x7f46780ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 --2- 172.21.15.29:0/1378071394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46780ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 -- 172.21.15.29:0/1378071394 >> v1:172.21.15.144:6789/0 conn(0x7f467815a730 legacy=0x7f46780c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.900+0000 7f46777fe700 1 -- 172.21.15.29:0/1378071394 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780c2140 con 0x7f4678067aa0 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.901+0000 7f46777fe700 1 --2- 172.21.15.29:0/1378071394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4678067aa0 0x7f4678067f20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4670009f20 tx=0x7f4670009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fe26dd535d0b5d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.901+0000 7f4675ffb700 1 -- 172.21.15.29:0/1378071394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467000e070 con 0x7f4678067aa0 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.901+0000 7f4675ffb700 1 -- 172.21.15.29:0/1378071394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467000b970 con 0x7f4678067aa0 2023-10-07T10:45:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.901+0000 7f4675ffb700 1 -- 172.21.15.29:0/1378071394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46700092e0 con 0x7f4678067aa0 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.901+0000 7f467d3ff700 1 -- 172.21.15.29:0/1378071394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4678067aa0 msgr2=0x7f4678067f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.901+0000 7f467d3ff700 1 --2- 172.21.15.29:0/1378071394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4678067aa0 0x7f4678067f20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4670009f20 tx=0x7f4670009840 comp rx=0 tx=0).stop 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.902+0000 7f467d3ff700 1 -- 172.21.15.29:0/1378071394 shutdown_connections 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.902+0000 7f467d3ff700 1 --2- 172.21.15.29:0/1378071394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4678067aa0 0x7f4678067f20 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.902+0000 7f467d3ff700 1 --2- 172.21.15.29:0/1378071394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46780ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.902+0000 7f467d3ff700 1 -- 172.21.15.29:0/1378071394 >> 172.21.15.29:0/1378071394 conn(0x7f46780c9430 msgr2=0x7f46780c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.902+0000 7f467d3ff700 1 -- 172.21.15.29:0/1378071394 shutdown_connections 2023-10-07T10:45:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.902+0000 7f467d3ff700 1 -- 172.21.15.29:0/1378071394 wait complete. 2023-10-07T10:45:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.903+0000 7f467d3ff700 1 Processor -- start 2023-10-07T10:45:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.903+0000 7f467d3ff700 1 -- start start 2023-10-07T10:45:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.904+0000 7f467d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4678067aa0 0x7f46780c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.904+0000 7f467d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46781510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.904+0000 7f467d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 0x7f46780c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.904+0000 7f467d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46780c4a60 con 0x7f467815a730 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.904+0000 7f467d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46780c4d40 con 0x7f4678067aa0 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.904+0000 7f467d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46780c5020 con 0x7f46780ca590 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 0x7f46780c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 0x7f46780c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52124/0 (socket says 172.21.15.29:52124) 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 -- 172.21.15.29:0/2967589023 learned_addr learned my addr 172.21.15.29:0/2967589023 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46767fc700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46781510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f4676ffd700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4678067aa0 0x7f46780c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:05.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 -- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 msgr2=0x7f46781510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46781510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 -- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4678067aa0 msgr2=0x7f46780c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4678067aa0 0x7f46780c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 -- 172.21.15.29:0/2967589023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4670009ae0 con 0x7f467815a730 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.905+0000 7f46777fe700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 0x7f46780c6040 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4670009880 tx=0x7f4670018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.906+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670007170 con 0x7f467815a730 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.906+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4670008b10 con 0x7f467815a730 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.906+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46780c5300 con 0x7f467815a730 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.906+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670025460 con 0x7f467815a730 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.906+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4678151610 con 0x7f467815a730 2023-10-07T10:45:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.907+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46780c5540 con 0x7f467815a730 2023-10-07T10:45:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.908+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f4670003040 con 0x7f467815a730 2023-10-07T10:45:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.909+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f4670085b70 con 0x7f467815a730 2023-10-07T10:45:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:05.913+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4670052d70 con 0x7f467815a730 2023-10-07T10:45:06.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:06.120+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46780718b0 con 0x7f467815a730 2023-10-07T10:45:06.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:06.175+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 73) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f4670052820 con 0x7f467815a730 2023-10-07T10:45:06.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:06.175+0000 7f465effd700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f466406ca70 0x7f466406ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:06.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:06.176+0000 7f4676ffd700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f466406ca70 0x7f466406ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:06.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:06.178+0000 7f4676ffd700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f466406ca70 0x7f466406ee50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4668007fd0 tx=0x7f4668007d30 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.060+0000 7f465effd700 1 -- 172.21.15.29:0/2967589023 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v155) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4670027170 con 0x7f467815a730 2023-10-07T10:45:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f466406ca70 msgr2=0x7f466406ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f466406ca70 0x7f466406ee50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4668007fd0 tx=0x7f4668007d30 comp rx=0 tx=0).stop 2023-10-07T10:45:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 msgr2=0x7f46780c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 0x7f46780c6040 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4670009880 tx=0x7f4670018000 comp rx=0 tx=0).stop 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 shutdown_connections 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f466406ca70 0x7f466406ee50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f467815a730 0x7f46780c6040 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4678067aa0 0x7f46780c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 --2- 172.21.15.29:0/2967589023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46780ca590 0x7f46781510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 >> 172.21.15.29:0/2967589023 conn(0x7f46780c9430 msgr2=0x7f467806e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.064+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 shutdown_connections 2023-10-07T10:45:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.065+0000 7f467d3ff700 1 -- 172.21.15.29:0/2967589023 wait complete. 2023-10-07T10:45:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T10:45:07.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T10:45:07.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.296+0000 7f96959ff700 1 Processor -- start 2023-10-07T10:45:07.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.296+0000 7f96959ff700 1 -- start start 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.297+0000 7f96959ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9690157a20 0x7f96900b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.297+0000 7f96959ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96900cb4d0 0x7f96900b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.297+0000 7f96959ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96900aeae0 con 0x7f9690157650 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.297+0000 7f96959ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96900ca430 con 0x7f9690157a20 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.297+0000 7f96959ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96900ca710 con 0x7f96900cb4d0 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9690157a20 0x7f96900b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9690157a20 0x7f96900b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52130/0 (socket says 172.21.15.29:52130) 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 -- 172.21.15.29:0/4257348183 learned_addr learned my addr 172.21.15.29:0/4257348183 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968effd700 1 --1- 172.21.15.29:0/4257348183 >> v1:172.21.15.29:6789/0 conn(0x7f9690157650 0x7f96900ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47564/0 (socket says 172.21.15.29:47564) 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968f7fe700 1 --2- 172.21.15.29:0/4257348183 >> v2:172.21.15.144:3300/0 conn(0x7f96900cb4d0 0x7f96900b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 -- 172.21.15.29:0/4257348183 >> v2:172.21.15.144:3300/0 conn(0x7f96900cb4d0 msgr2=0x7f96900b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 --2- 172.21.15.29:0/4257348183 >> v2:172.21.15.144:3300/0 conn(0x7f96900cb4d0 0x7f96900b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 -- 172.21.15.29:0/4257348183 >> v1:172.21.15.29:6789/0 conn(0x7f9690157650 legacy=0x7f96900ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.298+0000 7f968e7fc700 1 -- 172.21.15.29:0/4257348183 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96900ca9f0 con 0x7f9690157a20 2023-10-07T10:45:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.299+0000 7f968e7fc700 1 --2- 172.21.15.29:0/4257348183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157a20 0x7f96900b1290 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f96900c9810 tx=0x7f9684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523ac045d560c12e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.299+0000 7f968dffb700 1 -- 172.21.15.29:0/4257348183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400b860 con 0x7f9690157a20 2023-10-07T10:45:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.299+0000 7f968dffb700 1 -- 172.21.15.29:0/4257348183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968400b9c0 con 0x7f9690157a20 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.299+0000 7f968dffb700 1 -- 172.21.15.29:0/4257348183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400bd80 con 0x7f9690157a20 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 -- 172.21.15.29:0/4257348183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157a20 msgr2=0x7f96900b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 --2- 172.21.15.29:0/4257348183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157a20 0x7f96900b1290 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f96900c9810 tx=0x7f9684009250 comp rx=0 tx=0).stop 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 -- 172.21.15.29:0/4257348183 shutdown_connections 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 --2- 172.21.15.29:0/4257348183 >> v2:172.21.15.144:3300/0 conn(0x7f96900cb4d0 0x7f96900b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 --2- 172.21.15.29:0/4257348183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157a20 0x7f96900b1290 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 -- 172.21.15.29:0/4257348183 >> 172.21.15.29:0/4257348183 conn(0x7f9690067610 msgr2=0x7f9690067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 -- 172.21.15.29:0/4257348183 shutdown_connections 2023-10-07T10:45:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.300+0000 7f96959ff700 1 -- 172.21.15.29:0/4257348183 wait complete. 2023-10-07T10:45:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.301+0000 7f96959ff700 1 Processor -- start 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 -- start start 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96900cb4d0 0x7f9690169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 0x7f9690151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9690157a20 0x7f969006d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969006ddb0 con 0x7f9690157650 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969006e090 con 0x7f96900cb4d0 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.302+0000 7f96959ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969006e370 con 0x7f9690157a20 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 0x7f9690151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 0x7f9690151de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52146/0 (socket says 172.21.15.29:52146) 2023-10-07T10:45:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 -- 172.21.15.29:0/526860020 learned_addr learned my addr 172.21.15.29:0/526860020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968effd700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96900cb4d0 0x7f9690169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968f7fe700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9690157a20 0x7f969006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 -- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9690157a20 msgr2=0x7f969006d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9690157a20 0x7f969006d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 -- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96900cb4d0 msgr2=0x7f9690169a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96900cb4d0 0x7f9690169a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 -- 172.21.15.29:0/526860020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96840094f0 con 0x7f9690157650 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968f7fe700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9690157a20 0x7f969006d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.303+0000 7f968e7fc700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 0x7f9690151de0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f969006ce40 tx=0x7f9684017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.304+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968401f070 con 0x7f9690157650 2023-10-07T10:45:07.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.304+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9684002f30 con 0x7f9690157650 2023-10-07T10:45:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.304+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684011e00 con 0x7f9690157650 2023-10-07T10:45:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.304+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9690159ab0 con 0x7f9690157650 2023-10-07T10:45:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.304+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9690152320 con 0x7f9690157650 2023-10-07T10:45:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.305+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690159910 con 0x7f9690157650 2023-10-07T10:45:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.306+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f9684023070 con 0x7f9690157650 2023-10-07T10:45:07.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.307+0000 7f9676ffd700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f967c066d80 0x7f967c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:07.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.307+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f96840858b0 con 0x7f9690157650 2023-10-07T10:45:07.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.308+0000 7f968effd700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f967c066d80 0x7f967c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:07.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.308+0000 7f968effd700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f967c066d80 0x7f967c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9680004fd0 tx=0x7f9680007eb0 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.311+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9690159910 con 0x7f9690157650 2023-10-07T10:45:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:07.516+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9690159910 con 0x7f9690157650 2023-10-07T10:45:07.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:45:07.711 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:45:08.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.064+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9684052e60 con 0x7f9690157650 2023-10-07T10:45:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.179+0000 7f9676ffd700 1 -- 172.21.15.29:0/526860020 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v156) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9684052a90 con 0x7f9690157650 2023-10-07T10:45:08.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f967c066d80 msgr2=0x7f967c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f967c066d80 0x7f967c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9680004fd0 tx=0x7f9680007eb0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 msgr2=0x7f9690151de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 0x7f9690151de0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f969006ce40 tx=0x7f9684017000 comp rx=0 tx=0).stop 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 shutdown_connections 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f967c066d80 0x7f967c069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9690157a20 0x7f969006d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96900cb4d0 0x7f9690169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 --2- 172.21.15.29:0/526860020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9690157650 0x7f9690151de0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.183+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 >> 172.21.15.29:0/526860020 conn(0x7f9690067610 msgr2=0x7f96900ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.184+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 shutdown_connections 2023-10-07T10:45:08.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.184+0000 7f96959ff700 1 -- 172.21.15.29:0/526860020 wait complete. 2023-10-07T10:45:08.186 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T10:45:08.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T10:45:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.401+0000 7f572abff700 1 Processor -- start 2023-10-07T10:45:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.401+0000 7f572abff700 1 -- start start 2023-10-07T10:45:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.402+0000 7f572abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5724156180 0x7f57240afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.402+0000 7f572abff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57240ae570 0x7f57240accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.402+0000 7f572abff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57240b0520 0x7f57240b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.402+0000 7f572abff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57240b0ea0 con 0x7f57240b0520 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.402+0000 7f572abff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57240ad230 con 0x7f57240ae570 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.402+0000 7f572abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57240ad4d0 con 0x7f5724156180 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57240b0520 0x7f57240b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57240b0520 0x7f57240b0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52154/0 (socket says 172.21.15.29:52154) 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 -- 172.21.15.29:0/2721390587 learned_addr learned my addr 172.21.15.29:0/2721390587 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f5723fff700 1 --2- 172.21.15.29:0/2721390587 >> v2:172.21.15.144:3300/0 conn(0x7f57240ae570 0x7f57240accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572899b700 1 --2- 172.21.15.29:0/2721390587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5724156180 0x7f57240afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 -- 172.21.15.29:0/2721390587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5724156180 msgr2=0x7f57240afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 --2- 172.21.15.29:0/2721390587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5724156180 0x7f57240afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 -- 172.21.15.29:0/2721390587 >> v2:172.21.15.144:3300/0 conn(0x7f57240ae570 msgr2=0x7f57240accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 --2- 172.21.15.29:0/2721390587 >> v2:172.21.15.144:3300/0 conn(0x7f57240ae570 0x7f57240accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 -- 172.21.15.29:0/2721390587 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240ad7b0 con 0x7f57240b0520 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f5723fff700 1 --2- 172.21.15.29:0/2721390587 >> v2:172.21.15.144:3300/0 conn(0x7f57240ae570 0x7f57240accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.403+0000 7f572919c700 1 --2- 172.21.15.29:0/2721390587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57240b0520 0x7f57240b0960 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f572406d130 tx=0x7f571c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed74a23352bb0318 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f57237fe700 1 -- 172.21.15.29:0/2721390587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c010070 con 0x7f57240b0520 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f57237fe700 1 -- 172.21.15.29:0/2721390587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00dde0 con 0x7f57240b0520 2023-10-07T10:45:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f57237fe700 1 -- 172.21.15.29:0/2721390587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c00b320 con 0x7f57240b0520 2023-10-07T10:45:08.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 -- 172.21.15.29:0/2721390587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57240b0520 msgr2=0x7f57240b0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 --2- 172.21.15.29:0/2721390587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57240b0520 0x7f57240b0960 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f572406d130 tx=0x7f571c00ba50 comp rx=0 tx=0).stop 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 -- 172.21.15.29:0/2721390587 shutdown_connections 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 --2- 172.21.15.29:0/2721390587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57240b0520 0x7f57240b0960 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 --2- 172.21.15.29:0/2721390587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5724156180 0x7f57240afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 --2- 172.21.15.29:0/2721390587 >> v2:172.21.15.144:3300/0 conn(0x7f57240ae570 0x7f57240accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 -- 172.21.15.29:0/2721390587 >> 172.21.15.29:0/2721390587 conn(0x7f5724067610 msgr2=0x7f5724067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 -- 172.21.15.29:0/2721390587 shutdown_connections 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.404+0000 7f572abff700 1 -- 172.21.15.29:0/2721390587 wait complete. 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.406+0000 7f572abff700 1 Processor -- start 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.406+0000 7f572abff700 1 -- start start 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57240ae570 0x7f5724166280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57240b0520 0x7f5724151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 0x7f57240a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724165df0 con 0x7f5724156180 2023-10-07T10:45:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724152300 con 0x7f57240b0520 2023-10-07T10:45:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5724165810 con 0x7f57240ae570 2023-10-07T10:45:08.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 0x7f57240a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 0x7f57240a6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52158/0 (socket says 172.21.15.29:52158) 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572919c700 1 -- 172.21.15.29:0/2141056364 learned_addr learned my addr 172.21.15.29:0/2141056364 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f572899b700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57240ae570 0x7f5724166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.407+0000 7f5723fff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57240b0520 0x7f5724151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572919c700 1 -- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57240ae570 msgr2=0x7f5724166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572919c700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57240ae570 0x7f5724166280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572919c700 1 -- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57240b0520 msgr2=0x7f5724151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572919c700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57240b0520 0x7f5724151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572919c700 1 -- 172.21.15.29:0/2141056364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c00bcf0 con 0x7f5724156180 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572919c700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 0x7f57240a6fe0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f571c00d810 tx=0x7f571c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:08.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c00ab90 con 0x7f5724156180 2023-10-07T10:45:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00ad10 con 0x7f5724156180 2023-10-07T10:45:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57240a7830 con 0x7f5724156180 2023-10-07T10:45:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c0286b0 con 0x7f5724156180 2023-10-07T10:45:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.408+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57240a7520 con 0x7f5724156180 2023-10-07T10:45:08.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.409+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5724165fa0 con 0x7f5724156180 2023-10-07T10:45:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.410+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f571c003040 con 0x7f5724156180 2023-10-07T10:45:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.412+0000 7f5720ff9700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5708066f30 0x7f57080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.412+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7f571c088690 con 0x7f5724156180 2023-10-07T10:45:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.412+0000 7f572899b700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5708066f30 0x7f57080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.413+0000 7f572899b700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5708066f30 0x7f57080693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f57241534a0 tx=0x7f57180091a0 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:08.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.416+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5724165fa0 con 0x7f5724156180 2023-10-07T10:45:08.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.620+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5724165fa0 con 0x7f5724156180 2023-10-07T10:45:08.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.625+0000 7f5720ff9700 1 -- 172.21.15.29:0/2141056364 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5724165fa0 con 0x7f5724156180 2023-10-07T10:45:08.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5708066f30 msgr2=0x7f57080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5708066f30 0x7f57080693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f57241534a0 tx=0x7f57180091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 msgr2=0x7f57240a6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 0x7f57240a6fe0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f571c00d810 tx=0x7f571c01a000 comp rx=0 tx=0).stop 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 shutdown_connections 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f5708066f30 0x7f57080693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5724156180 0x7f57240a6fe0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57240ae570 0x7f5724166280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 --2- 172.21.15.29:0/2141056364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57240b0520 0x7f5724151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.627+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 >> 172.21.15.29:0/2141056364 conn(0x7f5724067610 msgr2=0x7f5724158910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.628+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 shutdown_connections 2023-10-07T10:45:08.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.628+0000 7f572abff700 1 -- 172.21.15.29:0/2141056364 wait complete. 2023-10-07T10:45:08.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T10:45:08.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.843+0000 7fe425bff700 1 Processor -- start 2023-10-07T10:45:08.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.843+0000 7fe425bff700 1 -- start start 2023-10-07T10:45:08.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.844+0000 7fe425bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe42015ade0 0x7fe4200b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.844+0000 7fe425bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe420154090 con 0x7fe4200b0540 2023-10-07T10:45:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.844+0000 7fe425bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe420159f60 con 0x7fe420153cc0 2023-10-07T10:45:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.844+0000 7fe425bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4200b09e0 con 0x7fe42015ade0 2023-10-07T10:45:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe42015ade0 0x7fe4200b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe4200b0540 0x7fe42015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47578/0 (socket says 172.21.15.29:47578) 2023-10-07T10:45:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41effd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe42015ade0 0x7fe4200b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52162/0 (socket says 172.21.15.29:52162) 2023-10-07T10:45:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41ffff700 1 -- 172.21.15.29:0/3314764322 learned_addr learned my addr 172.21.15.29:0/3314764322 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:08.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41effd700 1 -- 172.21.15.29:0/3314764322 >> v1:172.21.15.144:6789/0 conn(0x7fe420153cc0 legacy=0x7fe4200b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41effd700 1 -- 172.21.15.29:0/3314764322 >> v1:172.21.15.29:6789/0 conn(0x7fe4200b0540 legacy=0x7fe42015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41effd700 1 -- 172.21.15.29:0/3314764322 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42015a240 con 0x7fe42015ade0 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.845+0000 7fe41effd700 1 --2- 172.21.15.29:0/3314764322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe42015ade0 0x7fe4200b0000 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe4140098e0 tx=0x7fe4140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf7687059f924c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe41e7fc700 1 -- 172.21.15.29:0/3314764322 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414009e80 con 0x7fe42015ade0 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe41e7fc700 1 -- 172.21.15.29:0/3314764322 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41400b860 con 0x7fe42015ade0 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe41e7fc700 1 -- 172.21.15.29:0/3314764322 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41400bbe0 con 0x7fe42015ade0 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 -- 172.21.15.29:0/3314764322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe42015ade0 msgr2=0x7fe4200b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 --2- 172.21.15.29:0/3314764322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe42015ade0 0x7fe4200b0000 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe4140098e0 tx=0x7fe4140091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 -- 172.21.15.29:0/3314764322 shutdown_connections 2023-10-07T10:45:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 --2- 172.21.15.29:0/3314764322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe42015ade0 0x7fe4200b0000 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 -- 172.21.15.29:0/3314764322 >> 172.21.15.29:0/3314764322 conn(0x7fe420067610 msgr2=0x7fe420067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 -- 172.21.15.29:0/3314764322 shutdown_connections 2023-10-07T10:45:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.846+0000 7fe425bff700 1 -- 172.21.15.29:0/3314764322 wait complete. 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.848+0000 7fe425bff700 1 Processor -- start 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.848+0000 7fe425bff700 1 -- start start 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe425bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4200b0540 0x7fe420156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe425bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 0x7fe4201582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe425bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe42015ade0 0x7fe420145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe425bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe420157e10 con 0x7fe420153cc0 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe425bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe420157200 con 0x7fe42015ade0 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe425bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4201574e0 con 0x7fe4200b0540 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe41effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 0x7fe4201582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe41effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 0x7fe4201582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52164/0 (socket says 172.21.15.29:52164) 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.849+0000 7fe41effd700 1 -- 172.21.15.29:0/507178983 learned_addr learned my addr 172.21.15.29:0/507178983 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41ffff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe42015ade0 0x7fe420145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41f7fe700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4200b0540 0x7fe420156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41effd700 1 -- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4200b0540 msgr2=0x7fe420156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41effd700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4200b0540 0x7fe420156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41effd700 1 -- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe42015ade0 msgr2=0x7fe420145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41effd700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe42015ade0 0x7fe420145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41effd700 1 -- 172.21.15.29:0/507178983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414009420 con 0x7fe420153cc0 2023-10-07T10:45:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe41effd700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 0x7fe4201582a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe414013000 tx=0x7fe414017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41400bb20 con 0x7fe420153cc0 2023-10-07T10:45:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe414025420 con 0x7fe420153cc0 2023-10-07T10:45:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4140257c0 con 0x7fe420153cc0 2023-10-07T10:45:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.850+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4201577c0 con 0x7fe420153cc0 2023-10-07T10:45:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.851+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4200c77f0 con 0x7fe420153cc0 2023-10-07T10:45:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.851+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe420157f80 con 0x7fe420153cc0 2023-10-07T10:45:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.852+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe414023070 con 0x7fe420153cc0 2023-10-07T10:45:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.853+0000 7fe4077fe700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe40c066f30 0x7fe40c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.853+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7fe414085cf0 con 0x7fe420153cc0 2023-10-07T10:45:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.854+0000 7fe41f7fe700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe40c066f30 0x7fe40c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.855+0000 7fe41f7fe700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe40c066f30 0x7fe40c0693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe420069d80 tx=0x7fe410009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:08.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:08.858+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe414052df0 con 0x7fe420153cc0 2023-10-07T10:45:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.062+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe42004c3c0 con 0x7fe420153cc0 2023-10-07T10:45:09.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.067+0000 7fe4077fe700 1 -- 172.21.15.29:0/507178983 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe414056410 con 0x7fe420153cc0 2023-10-07T10:45:09.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.070+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe40c066f30 msgr2=0x7fe40c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.070+0000 7fe425bff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe40c066f30 0x7fe40c0693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe420069d80 tx=0x7fe410009040 comp rx=0 tx=0).stop 2023-10-07T10:45:09.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.070+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 msgr2=0x7fe4201582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.070+0000 7fe425bff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 0x7fe4201582a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe414013000 tx=0x7fe414017000 comp rx=0 tx=0).stop 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 shutdown_connections 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe40c066f30 0x7fe40c0693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe42015ade0 0x7fe420145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4200b0540 0x7fe420156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 --2- 172.21.15.29:0/507178983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe420153cc0 0x7fe4201582a0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 >> 172.21.15.29:0/507178983 conn(0x7fe420067610 msgr2=0x7fe4200af380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 shutdown_connections 2023-10-07T10:45:09.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.074+0000 7fe425bff700 1 -- 172.21.15.29:0/507178983 wait complete. 2023-10-07T10:45:09.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T10:45:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.293+0000 7f77dd3ff700 1 Processor -- start 2023-10-07T10:45:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.293+0000 7f77dd3ff700 1 -- start start 2023-10-07T10:45:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.294+0000 7f77dd3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77d8153cc0 0x7f77d80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.294+0000 7f77dd3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d815ade0 0x7f77d80b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.294+0000 7f77dd3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f77d80b0000 0x7f77d80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.294+0000 7f77dd3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d80b0980 con 0x7f77d8153cc0 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.294+0000 7f77dd3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d8154090 con 0x7f77d80b0000 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.294+0000 7f77dd3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d8159f60 con 0x7f77d815ade0 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77d8153cc0 0x7f77d80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77d8153cc0 0x7f77d80b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52170/0 (socket says 172.21.15.29:52170) 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 -- 172.21.15.29:0/1657001697 learned_addr learned my addr 172.21.15.29:0/1657001697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d77fe700 1 --2- 172.21.15.29:0/1657001697 >> v2:172.21.15.144:3300/0 conn(0x7f77d80b0000 0x7f77d80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d67fc700 1 --2- 172.21.15.29:0/1657001697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d815ade0 0x7f77d80b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 -- 172.21.15.29:0/1657001697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d815ade0 msgr2=0x7f77d80b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 --2- 172.21.15.29:0/1657001697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d815ade0 0x7f77d80b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 -- 172.21.15.29:0/1657001697 >> v2:172.21.15.144:3300/0 conn(0x7f77d80b0000 msgr2=0x7f77d80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 --2- 172.21.15.29:0/1657001697 >> v2:172.21.15.144:3300/0 conn(0x7f77d80b0000 0x7f77d80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 -- 172.21.15.29:0/1657001697 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d815a240 con 0x7f77d8153cc0 2023-10-07T10:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d77fe700 1 --2- 172.21.15.29:0/1657001697 >> v2:172.21.15.144:3300/0 conn(0x7f77d80b0000 0x7f77d80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.295+0000 7f77d6ffd700 1 --2- 172.21.15.29:0/1657001697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8153cc0 0x7f77d80b17c0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f77d8068230 tx=0x7f77c8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cc6e9bebb4e3c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77d5ffb700 1 -- 172.21.15.29:0/1657001697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8005c10 con 0x7f77d8153cc0 2023-10-07T10:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77d5ffb700 1 -- 172.21.15.29:0/1657001697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c8005d70 con 0x7f77d8153cc0 2023-10-07T10:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77d5ffb700 1 -- 172.21.15.29:0/1657001697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c800b880 con 0x7f77d8153cc0 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 -- 172.21.15.29:0/1657001697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8153cc0 msgr2=0x7f77d80b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/1657001697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8153cc0 0x7f77d80b17c0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f77d8068230 tx=0x7f77c8009220 comp rx=0 tx=0).stop 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 -- 172.21.15.29:0/1657001697 shutdown_connections 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/1657001697 >> v2:172.21.15.144:3300/0 conn(0x7f77d80b0000 0x7f77d80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/1657001697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8153cc0 0x7f77d80b17c0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/1657001697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d815ade0 0x7f77d80b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 -- 172.21.15.29:0/1657001697 >> 172.21.15.29:0/1657001697 conn(0x7f77d8067610 msgr2=0x7f77d8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 -- 172.21.15.29:0/1657001697 shutdown_connections 2023-10-07T10:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.296+0000 7f77dd3ff700 1 -- 172.21.15.29:0/1657001697 wait complete. 2023-10-07T10:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.298+0000 7f77dd3ff700 1 Processor -- start 2023-10-07T10:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.298+0000 7f77dd3ff700 1 -- start start 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77dd3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 0x7f77d813ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77dd3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d8153cc0 0x7f77d80c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77dd3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77d815ade0 0x7f77d80c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77dd3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81590f0 con 0x7f77d80b0000 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77dd3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d8159580 con 0x7f77d815ade0 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77dd3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81584e0 con 0x7f77d8153cc0 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77d6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 0x7f77d813ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77d6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 0x7f77d813ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52180/0 (socket says 172.21.15.29:52180) 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77d6ffd700 1 -- 172.21.15.29:0/3925048805 learned_addr learned my addr 172.21.15.29:0/3925048805 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77d67fc700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d8153cc0 0x7f77d80c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.299+0000 7f77d77fe700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77d815ade0 0x7f77d80c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77d6ffd700 1 -- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d8153cc0 msgr2=0x7f77d80c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77d6ffd700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d8153cc0 0x7f77d80c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77d6ffd700 1 -- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77d815ade0 msgr2=0x7f77d80c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77d6ffd700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77d815ade0 0x7f77d80c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77d6ffd700 1 -- 172.21.15.29:0/3925048805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c8009480 con 0x7f77d80b0000 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77d6ffd700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 0x7f77d813ed30 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f77c800b3c0 tx=0x7f77c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8011740 con 0x7f77d80b0000 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c8020070 con 0x7f77d80b0000 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8011ad0 con 0x7f77d80b0000 2023-10-07T10:45:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d81587c0 con 0x7f77d80b0000 2023-10-07T10:45:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.300+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d8158d80 con 0x7f77d80b0000 2023-10-07T10:45:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.301+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d804c3c0 con 0x7f77d80b0000 2023-10-07T10:45:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.302+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f77c8022030 con 0x7f77d80b0000 2023-10-07T10:45:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.303+0000 7f77beffd700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77c4066f30 0x7f77c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.303+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7f77c8086070 con 0x7f77d80b0000 2023-10-07T10:45:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.304+0000 7f77d67fc700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77c4066f30 0x7f77c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.304+0000 7f77d67fc700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77c4066f30 0x7f77c40693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f77d80b2f30 tx=0x7f77cc008040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.307+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77c80b75a0 con 0x7f77d80b0000 2023-10-07T10:45:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.512+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f77d81593d0 con 0x7f77d80b0000 2023-10-07T10:45:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.516+0000 7f77beffd700 1 -- 172.21.15.29:0/3925048805 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7f77c8053170 con 0x7f77d80b0000 2023-10-07T10:45:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77c4066f30 msgr2=0x7f77c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77c4066f30 0x7f77c40693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f77d80b2f30 tx=0x7f77cc008040 comp rx=0 tx=0).stop 2023-10-07T10:45:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 msgr2=0x7f77d813ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 0x7f77d813ed30 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f77c800b3c0 tx=0x7f77c8016000 comp rx=0 tx=0).stop 2023-10-07T10:45:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 shutdown_connections 2023-10-07T10:45:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77c4066f30 0x7f77c40693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77d815ade0 0x7f77d80c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d80b0000 0x7f77d813ed30 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 --2- 172.21.15.29:0/3925048805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77d8153cc0 0x7f77d80c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 >> 172.21.15.29:0/3925048805 conn(0x7f77d8067610 msgr2=0x7f77d80af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 shutdown_connections 2023-10-07T10:45:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.519+0000 7f77dd3ff700 1 -- 172.21.15.29:0/3925048805 wait complete. 2023-10-07T10:45:09.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-07T10:45:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.733+0000 7ff08dfff700 1 Processor -- start 2023-10-07T10:45:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.733+0000 7ff08dfff700 1 -- start start 2023-10-07T10:45:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.734+0000 7ff08dfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff088159750 0x7ff0880d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.734+0000 7ff08dfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0880ce8a0 0x7ff0880c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.734+0000 7ff08dfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880cbf20 con 0x7ff088159b20 2023-10-07T10:45:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.734+0000 7ff08dfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880cc200 con 0x7ff088159750 2023-10-07T10:45:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.734+0000 7ff08dfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880cc4e0 con 0x7ff0880ce8a0 2023-10-07T10:45:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff088159750 0x7ff0880d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff088159750 0x7ff0880d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52188/0 (socket says 172.21.15.29:52188) 2023-10-07T10:45:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 -- 172.21.15.29:0/3069168363 learned_addr learned my addr 172.21.15.29:0/3069168363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff087fff700 1 --2- 172.21.15.29:0/3069168363 >> v2:172.21.15.144:3300/0 conn(0x7ff0880ce8a0 0x7ff0880c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 -- 172.21.15.29:0/3069168363 >> v2:172.21.15.144:3300/0 conn(0x7ff0880ce8a0 msgr2=0x7ff0880c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 --2- 172.21.15.29:0/3069168363 >> v2:172.21.15.144:3300/0 conn(0x7ff0880ce8a0 0x7ff0880c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 -- 172.21.15.29:0/3069168363 >> v1:172.21.15.29:6789/0 conn(0x7ff088159b20 legacy=0x7ff0880ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 -- 172.21.15.29:0/3069168363 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880cc7c0 con 0x7ff088159750 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.735+0000 7ff0877fe700 1 --2- 172.21.15.29:0/3069168363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff088159750 0x7ff0880d10a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ff08815e510 tx=0x7ff078009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426560ab11c6bd22 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff0867fc700 1 -- 172.21.15.29:0/3069168363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800b860 con 0x7ff088159750 2023-10-07T10:45:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff0867fc700 1 -- 172.21.15.29:0/3069168363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07800b9c0 con 0x7ff088159750 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff0867fc700 1 -- 172.21.15.29:0/3069168363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800bd60 con 0x7ff088159750 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 -- 172.21.15.29:0/3069168363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff088159750 msgr2=0x7ff0880d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 --2- 172.21.15.29:0/3069168363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff088159750 0x7ff0880d10a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ff08815e510 tx=0x7ff078009250 comp rx=0 tx=0).stop 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 -- 172.21.15.29:0/3069168363 shutdown_connections 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 --2- 172.21.15.29:0/3069168363 >> v2:172.21.15.144:3300/0 conn(0x7ff0880ce8a0 0x7ff0880c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 --2- 172.21.15.29:0/3069168363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff088159750 0x7ff0880d10a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 -- 172.21.15.29:0/3069168363 >> 172.21.15.29:0/3069168363 conn(0x7ff088063a10 msgr2=0x7ff0880d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 -- 172.21.15.29:0/3069168363 shutdown_connections 2023-10-07T10:45:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.736+0000 7ff08dfff700 1 -- 172.21.15.29:0/3069168363 wait complete. 2023-10-07T10:45:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.738+0000 7ff08dfff700 1 Processor -- start 2023-10-07T10:45:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.738+0000 7ff08dfff700 1 -- start start 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff08dfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 0x7ff088147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff08dfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff088159750 0x7ff0880c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff08dfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff088159b20 0x7ff088156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff08dfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880c80c0 con 0x7ff0880ce8a0 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff08dfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880c83a0 con 0x7ff088159750 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff08dfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880c8680 con 0x7ff088159b20 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff0877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 0x7ff088147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff0877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 0x7ff088147eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52190/0 (socket says 172.21.15.29:52190) 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff0877fe700 1 -- 172.21.15.29:0/1515549093 learned_addr learned my addr 172.21.15.29:0/1515549093 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff087fff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff088159b20 0x7ff088156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff086ffd700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff088159750 0x7ff0880c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.739+0000 7ff0877fe700 1 -- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff088159b20 msgr2=0x7ff088156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff0877fe700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff088159b20 0x7ff088156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff0877fe700 1 -- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff088159750 msgr2=0x7ff0880c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff0877fe700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff088159750 0x7ff0880c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff0877fe700 1 -- 172.21.15.29:0/1515549093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009490 con 0x7ff0880ce8a0 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff0877fe700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 0x7ff088147eb0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ff088088a40 tx=0x7ff078017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800b860 con 0x7ff0880ce8a0 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078024440 con 0x7ff0880ce8a0 2023-10-07T10:45:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078024820 con 0x7ff0880ce8a0 2023-10-07T10:45:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0880c8960 con 0x7ff0880ce8a0 2023-10-07T10:45:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.740+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0880b5a40 con 0x7ff0880ce8a0 2023-10-07T10:45:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.741+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08804c3c0 con 0x7ff0880ce8a0 2023-10-07T10:45:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.742+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff07801a420 con 0x7ff0880ce8a0 2023-10-07T10:45:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.743+0000 7ff06f7fe700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff074066f30 0x7ff0740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.743+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7ff0780866d0 con 0x7ff0880ce8a0 2023-10-07T10:45:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.743+0000 7ff086ffd700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff074066f30 0x7ff0740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.744+0000 7ff086ffd700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff074066f30 0x7ff0740693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff08816ced0 tx=0x7ff07c009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:09.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:09.747+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff078053850 con 0x7ff0880ce8a0 2023-10-07T10:45:10.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.027+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7ff08815acd0 con 0x7ff0880ce8a0 2023-10-07T10:45:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.078+0000 7ff06f7fe700 1 -- 172.21.15.29:0/1515549093 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2458) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff078056e70 con 0x7ff0880ce8a0 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff074066f30 msgr2=0x7ff0740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff074066f30 0x7ff0740693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff08816ced0 tx=0x7ff07c009040 comp rx=0 tx=0).stop 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 msgr2=0x7ff088147eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 0x7ff088147eb0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ff088088a40 tx=0x7ff078017000 comp rx=0 tx=0).stop 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 shutdown_connections 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7ff074066f30 0x7ff0740693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff088159b20 0x7ff088156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880ce8a0 0x7ff088147eb0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 --2- 172.21.15.29:0/1515549093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff088159750 0x7ff0880c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 >> 172.21.15.29:0/1515549093 conn(0x7ff088063a10 msgr2=0x7ff0880d05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:10.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 shutdown_connections 2023-10-07T10:45:10.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.081+0000 7ff08dfff700 1 -- 172.21.15.29:0/1515549093 wait complete. 2023-10-07T10:45:10.099 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-07T10:45:10.099 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-07T10:45:10.099 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T10:45:10.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-07T10:45:10.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.295+0000 7fd5df7ff700 1 Processor -- start 2023-10-07T10:45:10.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.295+0000 7fd5df7ff700 1 -- start start 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.296+0000 7fd5df7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5d80cf040 0x7fd5d80cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.296+0000 7fd5df7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d80cc300 con 0x7fd5d81596e0 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.296+0000 7fd5df7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d80cb6f0 con 0x7fd5d8159310 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.296+0000 7fd5df7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d8159b50 con 0x7fd5d80cf040 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.296+0000 7fd5dcd9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd5d81596e0 0x7fd5d80ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47600/0 (socket says 172.21.15.29:47600) 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5dcd9a700 1 -- 172.21.15.29:0/695865142 learned_addr learned my addr 172.21.15.29:0/695865142 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5ddd9c700 1 --2- 172.21.15.29:0/695865142 >> v2:172.21.15.29:3300/0 conn(0x7fd5d80cf040 0x7fd5d80cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:10.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5ddd9c700 1 -- 172.21.15.29:0/695865142 >> v1:172.21.15.144:6789/0 conn(0x7fd5d8159310 legacy=0x7fd5d815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5ddd9c700 1 -- 172.21.15.29:0/695865142 >> v1:172.21.15.29:6789/0 conn(0x7fd5d81596e0 legacy=0x7fd5d80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5ddd9c700 1 -- 172.21.15.29:0/695865142 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d80cb9d0 con 0x7fd5d80cf040 2023-10-07T10:45:10.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5ddd9c700 1 --2- 172.21.15.29:0/695865142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d80cf040 0x7fd5d80cc790 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009e60 tx=0x7fd5d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2739093f8a2f0e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5cffff700 1 -- 172.21.15.29:0/695865142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d000e070 con 0x7fd5d80cf040 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.297+0000 7fd5cffff700 1 -- 172.21.15.29:0/695865142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d000bc30 con 0x7fd5d80cf040 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5cffff700 1 -- 172.21.15.29:0/695865142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0012430 con 0x7fd5d80cf040 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 -- 172.21.15.29:0/695865142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d80cf040 msgr2=0x7fd5d80cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/695865142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d80cf040 0x7fd5d80cc790 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009e60 tx=0x7fd5d0009780 comp rx=0 tx=0).stop 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 -- 172.21.15.29:0/695865142 shutdown_connections 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/695865142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d80cf040 0x7fd5d80cc790 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 -- 172.21.15.29:0/695865142 >> 172.21.15.29:0/695865142 conn(0x7fd5d8063230 msgr2=0x7fd5d8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 -- 172.21.15.29:0/695865142 shutdown_connections 2023-10-07T10:45:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.298+0000 7fd5df7ff700 1 -- 172.21.15.29:0/695865142 wait complete. 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.300+0000 7fd5df7ff700 1 Processor -- start 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.300+0000 7fd5df7ff700 1 -- start start 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5df7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5d80cf040 0x7fd5d814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5df7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 0x7fd5d80b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5df7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5d81596e0 0x7fd5d816ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5df7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d816d340 con 0x7fd5d8159310 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5df7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80b32d0 con 0x7fd5d81596e0 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5df7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80b3590 con 0x7fd5d80cf040 2023-10-07T10:45:10.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 0x7fd5d80b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 0x7fd5d80b5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52208/0 (socket says 172.21.15.29:52208) 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 -- 172.21.15.29:0/529844429 learned_addr learned my addr 172.21.15.29:0/529844429 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5ddd9c700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5d81596e0 0x7fd5d816ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dd59b700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5d80cf040 0x7fd5d814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 -- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5d80cf040 msgr2=0x7fd5d814a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5d80cf040 0x7fd5d814a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 -- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5d81596e0 msgr2=0x7fd5d816ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5d81596e0 0x7fd5d816ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.301+0000 7fd5dcd9a700 1 -- 172.21.15.29:0/529844429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0009a20 con 0x7fd5d8159310 2023-10-07T10:45:10.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.302+0000 7fd5dcd9a700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 0x7fd5d80b5bb0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c7c0 tx=0x7fd5c8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:10.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.302+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8005b30 con 0x7fd5d8159310 2023-10-07T10:45:10.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.302+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c8005c90 con 0x7fd5d8159310 2023-10-07T10:45:10.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.302+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d80b3870 con 0x7fd5d8159310 2023-10-07T10:45:10.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.302+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8004a80 con 0x7fd5d8159310 2023-10-07T10:45:10.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.302+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d80b60f0 con 0x7fd5d8159310 2023-10-07T10:45:10.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.303+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d816d110 con 0x7fd5d8159310 2023-10-07T10:45:10.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.304+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd5c80070d0 con 0x7fd5d8159310 2023-10-07T10:45:10.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.305+0000 7fd5cd7fa700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5bc066f30 0x7fd5bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:10.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.305+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7fd5c8083990 con 0x7fd5d8159310 2023-10-07T10:45:10.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.305+0000 7fd5dd59b700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5bc066f30 0x7fd5bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:10.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.306+0000 7fd5dd59b700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5bc066f30 0x7fd5bc0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4003a80 tx=0x7fd5d4007ec0 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:10.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.309+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5c804b5a0 con 0x7fd5d8159310 2023-10-07T10:45:10.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:10.589+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fd5d804c3c0 con 0x7fd5d8159310 2023-10-07T10:45:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.081+0000 7fd5cd7fa700 1 -- 172.21.15.29:0/529844429 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2459) v1 ==== 168+0+0 (secure 0 0 0) 0x7fd5c8050a90 con 0x7fd5d8159310 2023-10-07T10:45:11.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5bc066f30 msgr2=0x7fd5bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5bc066f30 0x7fd5bc0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4003a80 tx=0x7fd5d4007ec0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 msgr2=0x7fd5d80b5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 0x7fd5d80b5bb0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c7c0 tx=0x7fd5c8007f70 comp rx=0 tx=0).stop 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 shutdown_connections 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd5bc066f30 0x7fd5bc0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5d81596e0 0x7fd5d816ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5d80cf040 0x7fd5d814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 --2- 172.21.15.29:0/529844429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d8159310 0x7fd5d80b5bb0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.083+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 >> 172.21.15.29:0/529844429 conn(0x7fd5d8063230 msgr2=0x7fd5d814cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.084+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 shutdown_connections 2023-10-07T10:45:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.084+0000 7fd5df7ff700 1 -- 172.21.15.29:0/529844429 wait complete. 2023-10-07T10:45:11.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T10:45:11.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.304+0000 7f68047ff700 1 Processor -- start 2023-10-07T10:45:11.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.305+0000 7f68047ff700 1 -- start start 2023-10-07T10:45:11.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f68047ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc153cc0 0x7f67fc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f68047ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67fc15ade0 0x7f67fc0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f68047ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67fc0b0000 0x7f67fc0b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f68047ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67fc0b09a0 con 0x7f67fc15ade0 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f68047ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67fc154090 con 0x7f67fc0b0000 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f68047ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc159f60 con 0x7f67fc153cc0 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f6801d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67fc15ade0 0x7f67fc0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f6801d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67fc15ade0 0x7f67fc0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52220/0 (socket says 172.21.15.29:52220) 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.306+0000 7f6801d9a700 1 -- 172.21.15.29:0/118478577 learned_addr learned my addr 172.21.15.29:0/118478577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6802d9c700 1 --2- 172.21.15.29:0/118478577 >> v2:172.21.15.144:3300/0 conn(0x7f67fc0b0000 0x7f67fc0b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f680259b700 1 --2- 172.21.15.29:0/118478577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc153cc0 0x7f67fc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801d9a700 1 -- 172.21.15.29:0/118478577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc153cc0 msgr2=0x7f67fc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:11.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801d9a700 1 --2- 172.21.15.29:0/118478577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc153cc0 0x7f67fc0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801d9a700 1 -- 172.21.15.29:0/118478577 >> v2:172.21.15.144:3300/0 conn(0x7f67fc0b0000 msgr2=0x7f67fc0b0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:11.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801d9a700 1 --2- 172.21.15.29:0/118478577 >> v2:172.21.15.144:3300/0 conn(0x7f67fc0b0000 0x7f67fc0b0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801d9a700 1 -- 172.21.15.29:0/118478577 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc15a240 con 0x7f67fc15ade0 2023-10-07T10:45:11.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6802d9c700 1 --2- 172.21.15.29:0/118478577 >> v2:172.21.15.144:3300/0 conn(0x7f67fc0b0000 0x7f67fc0b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801d9a700 1 --2- 172.21.15.29:0/118478577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc15ade0 0x7f67fc0b1d00 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f67fc068230 tx=0x7f67ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dac356c331f5d8c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801599700 1 -- 172.21.15.29:0/118478577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00b860 con 0x7f67fc15ade0 2023-10-07T10:45:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801599700 1 -- 172.21.15.29:0/118478577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec00b9c0 con 0x7f67fc15ade0 2023-10-07T10:45:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.307+0000 7f6801599700 1 -- 172.21.15.29:0/118478577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec00bd60 con 0x7f67fc15ade0 2023-10-07T10:45:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 -- 172.21.15.29:0/118478577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc15ade0 msgr2=0x7f67fc0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 --2- 172.21.15.29:0/118478577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc15ade0 0x7f67fc0b1d00 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f67fc068230 tx=0x7f67ec009250 comp rx=0 tx=0).stop 2023-10-07T10:45:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 -- 172.21.15.29:0/118478577 shutdown_connections 2023-10-07T10:45:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 --2- 172.21.15.29:0/118478577 >> v2:172.21.15.144:3300/0 conn(0x7f67fc0b0000 0x7f67fc0b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 --2- 172.21.15.29:0/118478577 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc153cc0 0x7f67fc0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 --2- 172.21.15.29:0/118478577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc15ade0 0x7f67fc0b1d00 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 -- 172.21.15.29:0/118478577 >> 172.21.15.29:0/118478577 conn(0x7f67fc067610 msgr2=0x7f67fc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 -- 172.21.15.29:0/118478577 shutdown_connections 2023-10-07T10:45:11.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.308+0000 7f68047ff700 1 -- 172.21.15.29:0/118478577 wait complete. 2023-10-07T10:45:11.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.310+0000 7f68047ff700 1 Processor -- start 2023-10-07T10:45:11.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.310+0000 7f68047ff700 1 -- start start 2023-10-07T10:45:11.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f68047ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67fc0b0000 0x7f67fc0c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f68047ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 0x7f67fc16da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f68047ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc15ade0 0x7f67fc152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f68047ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc0c7850 con 0x7f67fc153cc0 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f68047ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc0c8730 con 0x7f67fc0b0000 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f68047ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc16dfc0 con 0x7f67fc15ade0 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 0x7f67fc16da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 0x7f67fc16da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52226/0 (socket says 172.21.15.29:52226) 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 -- 172.21.15.29:0/1174343183 learned_addr learned my addr 172.21.15.29:0/1174343183 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f680259b700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67fc0b0000 0x7f67fc0c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6802d9c700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc15ade0 0x7f67fc152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 -- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc15ade0 msgr2=0x7f67fc152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc15ade0 0x7f67fc152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 -- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67fc0b0000 msgr2=0x7f67fc0c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67fc0b0000 0x7f67fc0c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.311+0000 7f6801d9a700 1 -- 172.21.15.29:0/1174343183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec0094f0 con 0x7f67fc153cc0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.312+0000 7f6801d9a700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 0x7f67fc16da80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009290 tx=0x7f67ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.312+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec011e40 con 0x7f67fc153cc0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.312+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec01f070 con 0x7f67fc153cc0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.312+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0266d0 con 0x7f67fc153cc0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.312+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67fc153500 con 0x7f67fc153cc0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.312+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67fc0c7e00 con 0x7f67fc153cc0 2023-10-07T10:45:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.313+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67fc04c3c0 con 0x7f67fc153cc0 2023-10-07T10:45:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.314+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f67ec026a40 con 0x7f67fc153cc0 2023-10-07T10:45:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.315+0000 7f67f27fc700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f67e0066f30 0x7f67e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.315+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7f67ec021070 con 0x7f67fc153cc0 2023-10-07T10:45:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.315+0000 7f680259b700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f67e0066f30 0x7f67e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.316+0000 7f680259b700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f67e0066f30 0x7f67e00693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0cf370 tx=0x7f67f8008040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.319+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67ec053510 con 0x7f67fc153cc0 2023-10-07T10:45:11.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.526+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f67fc0c8870 con 0x7f67fc153cc0 2023-10-07T10:45:11.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.530+0000 7f67f27fc700 1 -- 172.21.15.29:0/1174343183 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f67ec056b30 con 0x7f67fc153cc0 2023-10-07T10:45:11.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.532+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f67e0066f30 msgr2=0x7f67e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.532+0000 7f68047ff700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f67e0066f30 0x7f67e00693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0cf370 tx=0x7f67f8008040 comp rx=0 tx=0).stop 2023-10-07T10:45:11.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.532+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 msgr2=0x7f67fc16da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.532+0000 7f68047ff700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 0x7f67fc16da80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009290 tx=0x7f67ec017000 comp rx=0 tx=0).stop 2023-10-07T10:45:11.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 shutdown_connections 2023-10-07T10:45:11.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f67e0066f30 0x7f67e00693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67fc15ade0 0x7f67fc152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67fc0b0000 0x7f67fc0c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 --2- 172.21.15.29:0/1174343183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67fc153cc0 0x7f67fc16da80 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 >> 172.21.15.29:0/1174343183 conn(0x7f67fc067610 msgr2=0x7f67fc0b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:11.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 shutdown_connections 2023-10-07T10:45:11.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.533+0000 7f68047ff700 1 -- 172.21.15.29:0/1174343183 wait complete. 2023-10-07T10:45:11.551 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T10:45:11.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:45:11.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T10:45:11.750 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T10:45:11.750 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T10:45:11.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:45:11.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.945+0000 7fb88d9ff700 1 Processor -- start 2023-10-07T10:45:11.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.945+0000 7fb88d9ff700 1 -- start start 2023-10-07T10:45:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb88d9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb888159750 0x7fb8880d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb88d9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880cbf20 con 0x7fb888159b20 2023-10-07T10:45:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb88d9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880cc200 con 0x7fb8880ce8a0 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb88d9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8880cc4e0 con 0x7fb888159750 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb886ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb888159750 0x7fb8880d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb886ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb888159750 0x7fb8880d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52240/0 (socket says 172.21.15.29:52240) 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb886ffd700 1 -- 172.21.15.29:0/2032108352 learned_addr learned my addr 172.21.15.29:0/2032108352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.947+0000 7fb8867fc700 1 --1- 172.21.15.29:0/2032108352 >> v1:172.21.15.29:6789/0 conn(0x7fb888159b20 0x7fb8880ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47604/0 (socket says 172.21.15.29:47604) 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb886ffd700 1 -- 172.21.15.29:0/2032108352 >> v1:172.21.15.144:6789/0 conn(0x7fb8880ce8a0 legacy=0x7fb8880c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb886ffd700 1 -- 172.21.15.29:0/2032108352 >> v1:172.21.15.29:6789/0 conn(0x7fb888159b20 legacy=0x7fb8880ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb886ffd700 1 -- 172.21.15.29:0/2032108352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880cc7c0 con 0x7fb888159750 2023-10-07T10:45:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb886ffd700 1 --2- 172.21.15.29:0/2032108352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159750 0x7fb8880d10a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb878009990 tx=0x7fb878009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554ec088f54b20b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb885ffb700 1 -- 172.21.15.29:0/2032108352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878005c10 con 0x7fb888159750 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb885ffb700 1 -- 172.21.15.29:0/2032108352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878005d70 con 0x7fb888159750 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.948+0000 7fb885ffb700 1 -- 172.21.15.29:0/2032108352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800b860 con 0x7fb888159750 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 -- 172.21.15.29:0/2032108352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159750 msgr2=0x7fb8880d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/2032108352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159750 0x7fb8880d10a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb878009990 tx=0x7fb878009220 comp rx=0 tx=0).stop 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 -- 172.21.15.29:0/2032108352 shutdown_connections 2023-10-07T10:45:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/2032108352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159750 0x7fb8880d10a0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 -- 172.21.15.29:0/2032108352 >> 172.21.15.29:0/2032108352 conn(0x7fb888063a10 msgr2=0x7fb8880d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 -- 172.21.15.29:0/2032108352 shutdown_connections 2023-10-07T10:45:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.949+0000 7fb88d9ff700 1 -- 172.21.15.29:0/2032108352 wait complete. 2023-10-07T10:45:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.950+0000 7fb88d9ff700 1 Processor -- start 2023-10-07T10:45:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.950+0000 7fb88d9ff700 1 -- start start 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.951+0000 7fb88d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8880ce8a0 0x7fb8880b5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.951+0000 7fb88d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb888159750 0x7fb8880b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.951+0000 7fb88d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 0x7fb888157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.951+0000 7fb88d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880ca820 con 0x7fb888159b20 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.951+0000 7fb88d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880b3800 con 0x7fb8880ce8a0 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.951+0000 7fb88d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880ca260 con 0x7fb888159750 2023-10-07T10:45:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 0x7fb888157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 0x7fb888157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52250/0 (socket says 172.21.15.29:52250) 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 -- 172.21.15.29:0/3300612252 learned_addr learned my addr 172.21.15.29:0/3300612252 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8867fc700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb888159750 0x7fb8880b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb886ffd700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8880ce8a0 0x7fb8880b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 -- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb888159750 msgr2=0x7fb8880b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb888159750 0x7fb8880b5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 -- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8880ce8a0 msgr2=0x7fb8880b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8880ce8a0 0x7fb8880b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 -- 172.21.15.29:0/3300612252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb878009480 con 0x7fb888159b20 2023-10-07T10:45:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb8877fe700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 0x7fb888157080 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c930 tx=0x7fb880001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:11.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88000a7e0 con 0x7fb888159b20 2023-10-07T10:45:11.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88000a940 con 0x7fb888159b20 2023-10-07T10:45:11.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.952+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88000ad00 con 0x7fb888159b20 2023-10-07T10:45:11.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.953+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8880ca540 con 0x7fb888159b20 2023-10-07T10:45:11.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.953+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8880b5fd0 con 0x7fb888159b20 2023-10-07T10:45:11.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.954+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8880b3970 con 0x7fb888159b20 2023-10-07T10:45:11.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.955+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb880012630 con 0x7fb888159b20 2023-10-07T10:45:11.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.956+0000 7fb86effd700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb874066f30 0x7fb8740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:11.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.956+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7fb880084900 con 0x7fb888159b20 2023-10-07T10:45:11.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.957+0000 7fb886ffd700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb874066f30 0x7fb8740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:11.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.957+0000 7fb886ffd700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb874066f30 0x7fb8740693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb878007fd0 tx=0x7fb878016000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:11.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:11.960+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb880051aa0 con 0x7fb888159b20 2023-10-07T10:45:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.212+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb88804c3c0 con 0x7fb888159b20 2023-10-07T10:45:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.213+0000 7fb86effd700 1 -- 172.21.15.29:0/3300612252 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fb880084070 con 0x7fb888159b20 2023-10-07T10:45:12.214 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:12.214 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3289689719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3289689719},{"type":"v1","addr":"172.21.15.144:6837","nonce":3289689719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2833505963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2833505963},{"type":"v1","addr":"172.21.15.29:6834","nonce":2833505963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8628,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4018215854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4018215854},{"type":"v1","addr":"172.21.15.144:6835","nonce":4018215854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71},{"gid":8967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/842728045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":842728045},{"type":"v1","addr":"172.21.15.29:6835","nonce":842728045}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":79}],"filesystems":[]} 2023-10-07T10:45:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb874066f30 msgr2=0x7fb8740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb874066f30 0x7fb8740693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb878007fd0 tx=0x7fb878016000 comp rx=0 tx=0).stop 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 msgr2=0x7fb888157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 0x7fb888157080 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c930 tx=0x7fb880001750 comp rx=0 tx=0).stop 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 shutdown_connections 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fb874066f30 0x7fb8740693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb888159b20 0x7fb888157080 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8880ce8a0 0x7fb8880b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 --2- 172.21.15.29:0/3300612252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb888159750 0x7fb8880b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 >> 172.21.15.29:0/3300612252 conn(0x7fb888063a10 msgr2=0x7fb888069c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.215+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 shutdown_connections 2023-10-07T10:45:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.216+0000 7fb88d9ff700 1 -- 172.21.15.29:0/3300612252 wait complete. 2023-10-07T10:45:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 79 2023-10-07T10:45:12.234 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T10:45:12.234 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T10:45:12.234 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T10:45:12.234 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:45:12.235 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T10:45:12.235 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T10:45:12.235 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T10:45:12.235 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:45:12.235 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T10:45:12.235 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T10:45:12.236 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T10:45:12.236 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:45:12.236 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T10:45:12.236 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T10:45:12.236 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T10:45:12.236 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:45:12.249 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T10:45:12.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T10:45:12.250 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T10:45:12.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T10:45:12.254 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T10:45:12.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T10:45:12.255 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T10:45:12.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T10:45:12.294 INFO:tasks.ceph.mds.a:Started 2023-10-07T10:45:12.295 DEBUG:teuthology.parallel:result is None 2023-10-07T10:45:12.297 INFO:tasks.ceph.mds.c:Started 2023-10-07T10:45:12.297 DEBUG:teuthology.parallel:result is None 2023-10-07T10:45:12.300 INFO:tasks.ceph.mds.b:Started 2023-10-07T10:45:12.300 DEBUG:teuthology.parallel:result is None 2023-10-07T10:45:12.304 INFO:tasks.ceph.mds.d:Started 2023-10-07T10:45:12.304 DEBUG:teuthology.parallel:result is None 2023-10-07T10:45:12.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T10:45:12.441 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T10:45:12.447 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T10:45:12.449 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T10:45:12.449 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T10:45:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.512+0000 7f1398bff700 1 Processor -- start 2023-10-07T10:45:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.513+0000 7f1398bff700 1 -- start start 2023-10-07T10:45:12.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1398bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1394159ec0 0x7f139415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1398bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13940af400 0x7f13940af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1398bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139414d790 con 0x7f13940ad370 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1398bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139415a880 con 0x7f13940af400 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1398bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13940ae5d0 con 0x7f1394159ec0 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1392d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13940af400 0x7f13940af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1392d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13940af400 0x7f13940af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52282/0 (socket says 172.21.15.29:52282) 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1392d9d700 1 -- 172.21.15.29:0/3598001186 learned_addr learned my addr 172.21.15.29:0/3598001186 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.514+0000 7f1391d9b700 1 --2- 172.21.15.29:0/3598001186 >> v2:172.21.15.144:3300/0 conn(0x7f1394159ec0 0x7f139415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f1392d9d700 1 -- 172.21.15.29:0/3598001186 >> v2:172.21.15.144:3300/0 conn(0x7f1394159ec0 msgr2=0x7f139415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f1392d9d700 1 --2- 172.21.15.29:0/3598001186 >> v2:172.21.15.144:3300/0 conn(0x7f1394159ec0 0x7f139415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f1392d9d700 1 -- 172.21.15.29:0/3598001186 >> v1:172.21.15.144:6789/0 conn(0x7f13940ad370 legacy=0x7f13940cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f1392d9d700 1 -- 172.21.15.29:0/3598001186 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940ad740 con 0x7f13940af400 2023-10-07T10:45:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f139159a700 1 -- 172.21.15.29:0/3598001186 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339953074 0 0) 0x7f139414d790 con 0x7f13940ad370 2023-10-07T10:45:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f1392d9d700 1 --2- 172.21.15.29:0/3598001186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13940af400 0x7f13940af840 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f13940ccfb0 tx=0x7f138c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6eba70ab6d2b4a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:12.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f139159a700 1 -- 172.21.15.29:0/3598001186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c00e070 con 0x7f13940af400 2023-10-07T10:45:12.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f139159a700 1 -- 172.21.15.29:0/3598001186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c00ba20 con 0x7f13940af400 2023-10-07T10:45:12.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.515+0000 7f139159a700 1 -- 172.21.15.29:0/3598001186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c0092e0 con 0x7f13940af400 2023-10-07T10:45:12.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 -- 172.21.15.29:0/3598001186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13940af400 msgr2=0x7f13940af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:12.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 --2- 172.21.15.29:0/3598001186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13940af400 0x7f13940af840 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f13940ccfb0 tx=0x7f138c009840 comp rx=0 tx=0).stop 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 -- 172.21.15.29:0/3598001186 shutdown_connections 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 --2- 172.21.15.29:0/3598001186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13940af400 0x7f13940af840 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 --2- 172.21.15.29:0/3598001186 >> v2:172.21.15.144:3300/0 conn(0x7f1394159ec0 0x7f139415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 -- 172.21.15.29:0/3598001186 >> 172.21.15.29:0/3598001186 conn(0x7f13940680a0 msgr2=0x7f13940cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 -- 172.21.15.29:0/3598001186 shutdown_connections 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.516+0000 7f1398bff700 1 -- 172.21.15.29:0/3598001186 wait complete. 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.517+0000 7f1398bff700 1 Processor -- start 2023-10-07T10:45:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.518+0000 7f1398bff700 1 -- start start 2023-10-07T10:45:12.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.518+0000 7f1398bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13940ad370 0x7f1394169b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:12.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.518+0000 7f1398bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13940af400 0x7f139406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:12.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.518+0000 7f1398bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 0x7f1394143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:12.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.518+0000 7f1398bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139406d9e0 con 0x7f1394159ec0 2023-10-07T10:45:12.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.518+0000 7f1398bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394169090 con 0x7f13940af400 2023-10-07T10:45:12.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1398bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1394144120 con 0x7f13940ad370 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 0x7f1394143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 0x7f1394143be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52298/0 (socket says 172.21.15.29:52298) 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 -- 172.21.15.29:0/822959526 learned_addr learned my addr 172.21.15.29:0/822959526 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1391d9b700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13940af400 0x7f139406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f139259c700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13940ad370 0x7f1394169b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 -- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13940ad370 msgr2=0x7f1394169b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13940ad370 0x7f1394169b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 -- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13940af400 msgr2=0x7f139406d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13940af400 0x7f139406d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 -- 172.21.15.29:0/822959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c009ae0 con 0x7f1394159ec0 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.519+0000 7f1392d9d700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 0x7f1394143be0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f138c00b8c0 tx=0x7f138c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:12.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.520+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c008b00 con 0x7f1394159ec0 2023-10-07T10:45:12.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.520+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c008c60 con 0x7f1394159ec0 2023-10-07T10:45:12.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.520+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13940a9aa0 con 0x7f1394159ec0 2023-10-07T10:45:12.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.520+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c026460 con 0x7f1394159ec0 2023-10-07T10:45:12.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.520+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13940aa060 con 0x7f1394159ec0 2023-10-07T10:45:12.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.521+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139404c3c0 con 0x7f1394159ec0 2023-10-07T10:45:12.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.522+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f138c003040 con 0x7f1394159ec0 2023-10-07T10:45:12.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.523+0000 7f13827fc700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:12.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.523+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5918+0+0 (secure 0 0 0) 0x7f138c085eb0 con 0x7f1394159ec0 2023-10-07T10:45:12.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.524+0000 7f139259c700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:12.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.524+0000 7f139259c700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1374066f30 0x7f13740693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1384007fd0 tx=0x7f1384007d30 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:12.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.527+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f138c052fb0 con 0x7f1394159ec0 2023-10-07T10:45:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:12.743+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1394068620 con 0x7f1394159ec0 2023-10-07T10:45:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.046+0000 7f13827fc700 1 -- 172.21.15.29:0/822959526 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v161) v1 ==== 97+0+0 (secure 0 0 0) 0x7f138c0565d0 con 0x7f1394159ec0 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.049+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1374066f30 msgr2=0x7f13740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.049+0000 7f1398bff700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1374066f30 0x7f13740693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1384007fd0 tx=0x7f1384007d30 comp rx=0 tx=0).stop 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.049+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 msgr2=0x7f1394143be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.049+0000 7f1398bff700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 0x7f1394143be0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f138c00b8c0 tx=0x7f138c018000 comp rx=0 tx=0).stop 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 shutdown_connections 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1394159ec0 0x7f1394143be0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13940ad370 0x7f1394169b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 --2- 172.21.15.29:0/822959526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13940af400 0x7f139406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 >> 172.21.15.29:0/822959526 conn(0x7f13940680a0 msgr2=0x7f139414cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 shutdown_connections 2023-10-07T10:45:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.050+0000 7f1398bff700 1 -- 172.21.15.29:0/822959526 wait complete. 2023-10-07T10:45:14.052 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T10:45:14.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T10:45:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.271+0000 7f6db07ff700 1 Processor -- start 2023-10-07T10:45:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.271+0000 7f6db07ff700 1 -- start start 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6db07ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6da8157290 0x7f6da80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6db07ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80b1a10 0x7f6da8157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6db07ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da8158ad0 con 0x7f6da80ae8b0 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6db07ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da815b140 con 0x7f6da8157290 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6db07ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da815b420 con 0x7f6da80b1a10 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6dae59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6da8157290 0x7f6da80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6dae59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6da8157290 0x7f6da80cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49906/0 (socket says 172.21.15.29:49906) 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6dadd9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6da80ae8b0 0x7f6da80b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47846/0 (socket says 172.21.15.29:47846) 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.272+0000 7f6dae59b700 1 -- 172.21.15.29:0/793161150 learned_addr learned my addr 172.21.15.29:0/793161150 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6daed9c700 1 --2- 172.21.15.29:0/793161150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80b1a10 0x7f6da8157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dad599700 1 -- 172.21.15.29:0/793161150 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375574829 0 0) 0x7f6da8158ad0 con 0x7f6da80ae8b0 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dae59b700 1 -- 172.21.15.29:0/793161150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80b1a10 msgr2=0x7f6da8157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dae59b700 1 --2- 172.21.15.29:0/793161150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80b1a10 0x7f6da8157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dae59b700 1 -- 172.21.15.29:0/793161150 >> v1:172.21.15.29:6789/0 conn(0x7f6da80ae8b0 legacy=0x7f6da80b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dae59b700 1 -- 172.21.15.29:0/793161150 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da815b700 con 0x7f6da8157290 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dae59b700 1 --2- 172.21.15.29:0/793161150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80cc870 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f6da40099b0 tx=0x7f6da4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b4f9bcaa1f67d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dad599700 1 -- 172.21.15.29:0/793161150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da400b860 con 0x7f6da8157290 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dad599700 1 -- 172.21.15.29:0/793161150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da400b9c0 con 0x7f6da8157290 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.273+0000 7f6dad599700 1 -- 172.21.15.29:0/793161150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da400bd60 con 0x7f6da8157290 2023-10-07T10:45:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 -- 172.21.15.29:0/793161150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 msgr2=0x7f6da80cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 --2- 172.21.15.29:0/793161150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80cc870 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f6da40099b0 tx=0x7f6da4009250 comp rx=0 tx=0).stop 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 -- 172.21.15.29:0/793161150 shutdown_connections 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 --2- 172.21.15.29:0/793161150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80b1a10 0x7f6da8157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 --2- 172.21.15.29:0/793161150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80cc870 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 -- 172.21.15.29:0/793161150 >> 172.21.15.29:0/793161150 conn(0x7f6da80680a0 msgr2=0x7f6da80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 -- 172.21.15.29:0/793161150 shutdown_connections 2023-10-07T10:45:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.274+0000 7f6db07ff700 1 -- 172.21.15.29:0/793161150 wait complete. 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.275+0000 7f6db07ff700 1 Processor -- start 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 -- start start 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6da80b1a10 0x7f6da806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80c14b0 0x7f6da8167b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da806ef00 con 0x7f6da8157290 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da80c1900 con 0x7f6da80b1a10 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.276+0000 7f6db07ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da806e2f0 con 0x7f6da80c14b0 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80c0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49922/0 (socket says 172.21.15.29:49922) 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 -- 172.21.15.29:0/3741517513 learned_addr learned my addr 172.21.15.29:0/3741517513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6daed9c700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80c14b0 0x7f6da8167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dae59b700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6da80b1a10 0x7f6da806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 -- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80c14b0 msgr2=0x7f6da8167b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80c14b0 0x7f6da8167b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 -- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6da80b1a10 msgr2=0x7f6da806d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6da80b1a10 0x7f6da806d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 -- 172.21.15.29:0/3741517513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da40094f0 con 0x7f6da8157290 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6daed9c700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80c14b0 0x7f6da8167b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:14.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dae59b700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6da80b1a10 0x7f6da806d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.277+0000 7f6dadd9a700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80c0f70 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f6d980011e0 tx=0x7f6d98007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.278+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98007300 con 0x7f6da8157290 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.278+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d98007460 con 0x7f6da8157290 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.278+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da806e5d0 con 0x7f6da8157290 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.278+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98004830 con 0x7f6da8157290 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.278+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da806eb90 con 0x7f6da8157290 2023-10-07T10:45:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.279+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da806f0b0 con 0x7f6da8157290 2023-10-07T10:45:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.280+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6d98004be0 con 0x7f6da8157290 2023-10-07T10:45:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.281+0000 7f6d9e7fc700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6d8c066f30 0x7f6d8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.281+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f6d98083930 con 0x7f6da8157290 2023-10-07T10:45:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.281+0000 7f6dae59b700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6d8c066f30 0x7f6d8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.282+0000 7f6dae59b700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6d8c066f30 0x7f6d8c0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6da40099b0 tx=0x7f6da4017000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:14.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.285+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6da806f0b0 con 0x7f6da8157290 2023-10-07T10:45:14.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.565+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6da804c3c0 con 0x7f6da8157290 2023-10-07T10:45:14.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.565+0000 7f6d9e7fc700 1 -- 172.21.15.29:0/3741517513 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6d98018020 con 0x7f6da8157290 2023-10-07T10:45:14.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:14.571 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T10:45:14.571 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T10:45:14.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.572 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T10:45:14.572 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T10:45:14.572 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.572 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.572 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:45:14.572 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:45:14.573 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.573 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.575 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T10:45:14.575 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:45:14.577 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.578 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.579 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T10:45:14.580 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T10:45:14.581 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T10:45:14.582 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.582 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.582 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.582 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.585 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.586 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T10:45:14.586 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T10:45:14.587 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.587 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.587 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.587 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.588 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.588 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.589 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.590 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T10:45:14.591 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T10:45:14.592 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T10:45:14.594 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T10:45:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T10:45:14.596 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.567+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6d8c066f30 msgr2=0x7f6d8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.567+0000 7f6db07ff700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6d8c066f30 0x7f6d8c0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6da40099b0 tx=0x7f6da4017000 comp rx=0 tx=0).stop 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.567+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 msgr2=0x7f6da80c0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.567+0000 7f6db07ff700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80c0f70 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f6d980011e0 tx=0x7f6d98007e10 comp rx=0 tx=0).stop 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.567+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 shutdown_connections 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6d8c066f30 0x7f6d8c0693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6da80c14b0 0x7f6da8167b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6da80b1a10 0x7f6da806d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 --2- 172.21.15.29:0/3741517513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da8157290 0x7f6da80c0f70 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 >> 172.21.15.29:0/3741517513 conn(0x7f6da80680a0 msgr2=0x7f6da8152940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 shutdown_connections 2023-10-07T10:45:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.568+0000 7f6db07ff700 1 -- 172.21.15.29:0/3741517513 wait complete. 2023-10-07T10:45:14.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:14.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.807+0000 7f60a93ff700 1 Processor -- start 2023-10-07T10:45:14.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.807+0000 7f60a93ff700 1 -- start start 2023-10-07T10:45:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.808+0000 7f60a93ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a4157290 0x7f60a40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.808+0000 7f60a93ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60a40b1a10 0x7f60a4157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.808+0000 7f60a93ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a4158ad0 con 0x7f60a40ae8b0 2023-10-07T10:45:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.808+0000 7f60a93ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a415b140 con 0x7f60a40b1a10 2023-10-07T10:45:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.808+0000 7f60a93ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a415b420 con 0x7f60a4157290 2023-10-07T10:45:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60a40b1a10 0x7f60a4157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60a40b1a10 0x7f60a4157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49934/0 (socket says 172.21.15.29:49934) 2023-10-07T10:45:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 -- 172.21.15.29:0/2482977832 learned_addr learned my addr 172.21.15.29:0/2482977832 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a2ffd700 1 --2- 172.21.15.29:0/2482977832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a4157290 0x7f60a40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 -- 172.21.15.29:0/2482977832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a4157290 msgr2=0x7f60a40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 --2- 172.21.15.29:0/2482977832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a4157290 0x7f60a40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 -- 172.21.15.29:0/2482977832 >> v1:172.21.15.144:6789/0 conn(0x7f60a40ae8b0 legacy=0x7f60a40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 -- 172.21.15.29:0/2482977832 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a415b700 con 0x7f60a40b1a10 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.809+0000 7f60a37fe700 1 --2- 172.21.15.29:0/2482977832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a40b1a10 0x7f60a4157660 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f609c009f20 tx=0x7f609c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda97908abefb967 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:14.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a1ffb700 1 -- 172.21.15.29:0/2482977832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c00e070 con 0x7f60a40b1a10 2023-10-07T10:45:14.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a1ffb700 1 -- 172.21.15.29:0/2482977832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c00ba80 con 0x7f60a40b1a10 2023-10-07T10:45:14.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a1ffb700 1 -- 172.21.15.29:0/2482977832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0092e0 con 0x7f60a40b1a10 2023-10-07T10:45:14.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 -- 172.21.15.29:0/2482977832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a40b1a10 msgr2=0x7f60a4157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 --2- 172.21.15.29:0/2482977832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a40b1a10 0x7f60a4157660 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f609c009f20 tx=0x7f609c009840 comp rx=0 tx=0).stop 2023-10-07T10:45:14.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 -- 172.21.15.29:0/2482977832 shutdown_connections 2023-10-07T10:45:14.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 --2- 172.21.15.29:0/2482977832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a40b1a10 0x7f60a4157660 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 --2- 172.21.15.29:0/2482977832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a4157290 0x7f60a40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 -- 172.21.15.29:0/2482977832 >> 172.21.15.29:0/2482977832 conn(0x7f60a40680a0 msgr2=0x7f60a40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:14.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 -- 172.21.15.29:0/2482977832 shutdown_connections 2023-10-07T10:45:14.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.810+0000 7f60a93ff700 1 -- 172.21.15.29:0/2482977832 wait complete. 2023-10-07T10:45:14.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.812+0000 7f60a93ff700 1 Processor -- start 2023-10-07T10:45:14.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.813+0000 7f60a93ff700 1 -- start start 2023-10-07T10:45:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a93ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a40b1a10 0x7f60a4150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a93ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 0x7f60a416a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a93ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60a406e910 0x7f60a41536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a93ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40c8b50 con 0x7f60a4157290 2023-10-07T10:45:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a93ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4169dd0 con 0x7f60a406e910 2023-10-07T10:45:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a93ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a416a380 con 0x7f60a40b1a10 2023-10-07T10:45:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 0x7f60a416a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 0x7f60a416a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49936/0 (socket says 172.21.15.29:49936) 2023-10-07T10:45:14.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.814+0000 7f60a27fc700 1 -- 172.21.15.29:0/944520314 learned_addr learned my addr 172.21.15.29:0/944520314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:14.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a27fc700 1 -- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a40b1a10 msgr2=0x7f60a4150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a27fc700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a40b1a10 0x7f60a4150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a27fc700 1 -- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60a406e910 msgr2=0x7f60a41536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:14.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a27fc700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60a406e910 0x7f60a41536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:14.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a27fc700 1 -- 172.21.15.29:0/944520314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c009ae0 con 0x7f60a4157290 2023-10-07T10:45:14.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a27fc700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 0x7f60a416a810 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f609800c5c0 tx=0x7f609800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:14.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6098011590 con 0x7f60a4157290 2023-10-07T10:45:14.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6098011710 con 0x7f60a4157290 2023-10-07T10:45:14.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6098011ad0 con 0x7f60a4157290 2023-10-07T10:45:14.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a416ad50 con 0x7f60a4157290 2023-10-07T10:45:14.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.815+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a40c8870 con 0x7f60a4157290 2023-10-07T10:45:14.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.816+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a404c3c0 con 0x7f60a4157290 2023-10-07T10:45:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.817+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f609801b3b0 con 0x7f60a4157290 2023-10-07T10:45:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.818+0000 7f608affd700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6090066f30 0x7f60900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.818+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f60980838d0 con 0x7f60a4157290 2023-10-07T10:45:14.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.819+0000 7f60a2ffd700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6090066f30 0x7f60900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:14.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.820+0000 7f60a2ffd700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6090066f30 0x7f60900693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f60a40c8130 tx=0x7f6094008040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:14.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:14.822+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6098050cf0 con 0x7f60a4157290 2023-10-07T10:45:15.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.024+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60a416a4c0 con 0x7f60a4157290 2023-10-07T10:45:15.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.024+0000 7f608affd700 1 -- 172.21.15.29:0/944520314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+12882 (secure 0 0 0) 0x7f6098017070 con 0x7f60a4157290 2023-10-07T10:45:15.026 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:15.027 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":161,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:45:14.043439+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6090066f30 msgr2=0x7f60900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6090066f30 0x7f60900693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f60a40c8130 tx=0x7f6094008040 comp rx=0 tx=0).stop 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 msgr2=0x7f60a416a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 0x7f60a416a810 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f609800c5c0 tx=0x7f609800c980 comp rx=0 tx=0).stop 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 shutdown_connections 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6090066f30 0x7f60900693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60a406e910 0x7f60a41536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f60a40b1a10 0x7f60a4150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 --2- 172.21.15.29:0/944520314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4157290 0x7f60a416a810 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 >> 172.21.15.29:0/944520314 conn(0x7f60a40680a0 msgr2=0x7f60a4152ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 shutdown_connections 2023-10-07T10:45:15.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.027+0000 7f60a93ff700 1 -- 172.21.15.29:0/944520314 wait complete. 2023-10-07T10:45:15.046 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T10:45:15.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T10:45:15.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.245+0000 7f06237ff700 1 Processor -- start 2023-10-07T10:45:15.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.245+0000 7f06237ff700 1 -- start start 2023-10-07T10:45:15.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f06237ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b3e60 0x7f061c158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f06237ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f061c0ce390 0x7f061c0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f06237ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061c1578a0 con 0x7f061c0b00b0 2023-10-07T10:45:15.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f06237ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061c0cc830 con 0x7f061c0ce390 2023-10-07T10:45:15.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f06237ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c157ce0 con 0x7f061c0b3e60 2023-10-07T10:45:15.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f0620d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f061c0ce390 0x7f061c0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f0620d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f061c0ce390 0x7f061c0ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49940/0 (socket says 172.21.15.29:49940) 2023-10-07T10:45:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f0620d9a700 1 -- 172.21.15.29:0/3986871493 learned_addr learned my addr 172.21.15.29:0/3986871493 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f0621d9c700 1 --1- 172.21.15.29:0/3986871493 >> v1:172.21.15.29:6789/0 conn(0x7f061c0b00b0 0x7f061c0cccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47854/0 (socket says 172.21.15.29:47854) 2023-10-07T10:45:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.247+0000 7f062159b700 1 --2- 172.21.15.29:0/3986871493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b3e60 0x7f061c158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0620d9a700 1 -- 172.21.15.29:0/3986871493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b3e60 msgr2=0x7f061c158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0620d9a700 1 --2- 172.21.15.29:0/3986871493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b3e60 0x7f061c158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0620d9a700 1 -- 172.21.15.29:0/3986871493 >> v1:172.21.15.29:6789/0 conn(0x7f061c0b00b0 legacy=0x7f061c0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0620d9a700 1 -- 172.21.15.29:0/3986871493 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c0cee50 con 0x7f061c0ce390 2023-10-07T10:45:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0613fff700 1 -- 172.21.15.29:0/3986871493 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983802039 0 0) 0x7f061c1578a0 con 0x7f061c0b00b0 2023-10-07T10:45:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0620d9a700 1 --2- 172.21.15.29:0/3986871493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0ce800 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f060c0099b0 tx=0x7f060c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26c76fc3440b0c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0613fff700 1 -- 172.21.15.29:0/3986871493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c009ea0 con 0x7f061c0ce390 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0613fff700 1 -- 172.21.15.29:0/3986871493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060c00b860 con 0x7f061c0ce390 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.248+0000 7f0613fff700 1 -- 172.21.15.29:0/3986871493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c00bc00 con 0x7f061c0ce390 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 -- 172.21.15.29:0/3986871493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 msgr2=0x7f061c0ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 --2- 172.21.15.29:0/3986871493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0ce800 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f060c0099b0 tx=0x7f060c009250 comp rx=0 tx=0).stop 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 -- 172.21.15.29:0/3986871493 shutdown_connections 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 --2- 172.21.15.29:0/3986871493 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b3e60 0x7f061c158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 --2- 172.21.15.29:0/3986871493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0ce800 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 -- 172.21.15.29:0/3986871493 >> 172.21.15.29:0/3986871493 conn(0x7f061c067610 msgr2=0x7f061c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 -- 172.21.15.29:0/3986871493 shutdown_connections 2023-10-07T10:45:15.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.249+0000 7f06237ff700 1 -- 172.21.15.29:0/3986871493 wait complete. 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.250+0000 7f06237ff700 1 Processor -- start 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.251+0000 7f06237ff700 1 -- start start 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f06237ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b00b0 0x7f061c069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f06237ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061c0b3e60 0x7f061c0aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f06237ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f06237ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0c75b0 con 0x7f061c0ce390 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f06237ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0c7800 con 0x7f061c0b3e60 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f06237ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0c7ae0 con 0x7f061c0b00b0 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0af3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49952/0 (socket says 172.21.15.29:49952) 2023-10-07T10:45:15.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f062159b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b00b0 0x7f061c069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 -- 172.21.15.29:0/2090823547 learned_addr learned my addr 172.21.15.29:0/2090823547 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0620d9a700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061c0b3e60 0x7f061c0aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 -- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b00b0 msgr2=0x7f061c069b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b00b0 0x7f061c069b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 -- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061c0b3e60 msgr2=0x7f061c0aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061c0b3e60 0x7f061c0aee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.252+0000 7f0621d9c700 1 -- 172.21.15.29:0/2090823547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c0094f0 con 0x7f061c0ce390 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.253+0000 7f0621d9c700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0af3c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f061400cd00 tx=0x7f0614001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.253+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061400ce60 con 0x7f061c0ce390 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.253+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0614005df0 con 0x7f061c0ce390 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.253+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c0c7dc0 con 0x7f061c0ce390 2023-10-07T10:45:15.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.253+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061400aae0 con 0x7f061c0ce390 2023-10-07T10:45:15.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.253+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c06a070 con 0x7f061c0ce390 2023-10-07T10:45:15.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.254+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061c0c82c0 con 0x7f061c0ce390 2023-10-07T10:45:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.255+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f06140070f0 con 0x7f061c0ce390 2023-10-07T10:45:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.256+0000 7f06117fa700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0600066f30 0x7f06000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.256+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f0614084460 con 0x7f061c0ce390 2023-10-07T10:45:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.256+0000 7f062159b700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0600066f30 0x7f06000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.257+0000 7f062159b700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0600066f30 0x7f06000693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0618009da0 tx=0x7f0618009120 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:15.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.260+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0614051860 con 0x7f061c0ce390 2023-10-07T10:45:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.462+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f061c04c3c0 con 0x7f061c0ce390 2023-10-07T10:45:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.463+0000 7f06117fa700 1 -- 172.21.15.29:0/2090823547 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v161) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0614054e80 con 0x7f061c0ce390 2023-10-07T10:45:15.464 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T10:45:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.465+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0600066f30 msgr2=0x7f06000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.465+0000 7f06237ff700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0600066f30 0x7f06000693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0618009da0 tx=0x7f0618009120 comp rx=0 tx=0).stop 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.465+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 msgr2=0x7f061c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.465+0000 7f06237ff700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0af3c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f061400cd00 tx=0x7f0614001ac0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 shutdown_connections 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f0600066f30 0x7f06000693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f061c0ce390 0x7f061c0af3c0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f061c0b00b0 0x7f061c069b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 --2- 172.21.15.29:0/2090823547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f061c0b3e60 0x7f061c0aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 >> 172.21.15.29:0/2090823547 conn(0x7f061c067610 msgr2=0x7f061c06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 shutdown_connections 2023-10-07T10:45:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.466+0000 7f06237ff700 1 -- 172.21.15.29:0/2090823547 wait complete. 2023-10-07T10:45:15.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T10:45:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.677+0000 7feb4cdff700 1 Processor -- start 2023-10-07T10:45:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.677+0000 7feb4cdff700 1 -- start start 2023-10-07T10:45:15.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb4cdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feb481533f0 0x7feb48156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb4cdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb48159670 con 0x7feb480b04b0 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb4cdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4815a1f0 con 0x7feb481537c0 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb4cdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4815a680 con 0x7feb481533f0 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb46d9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feb480b04b0 0x7feb48157240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47858/0 (socket says 172.21.15.29:47858) 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb46d9d700 1 -- 172.21.15.29:0/673908120 learned_addr learned my addr 172.21.15.29:0/673908120 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb4659c700 1 --2- 172.21.15.29:0/673908120 >> v2:172.21.15.144:3300/0 conn(0x7feb481533f0 0x7feb48156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.679+0000 7feb4559a700 1 -- 172.21.15.29:0/673908120 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223912163 0 0) 0x7feb48159670 con 0x7feb480b04b0 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4559a700 1 -- 172.21.15.29:0/673908120 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb30003570 con 0x7feb480b04b0 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4659c700 1 -- 172.21.15.29:0/673908120 >> v1:172.21.15.144:6789/0 conn(0x7feb481537c0 legacy=0x7feb480afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4659c700 1 -- 172.21.15.29:0/673908120 >> v1:172.21.15.29:6789/0 conn(0x7feb480b04b0 legacy=0x7feb48157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4659c700 1 -- 172.21.15.29:0/673908120 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb481598c0 con 0x7feb481533f0 2023-10-07T10:45:15.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4659c700 1 --2- 172.21.15.29:0/673908120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481533f0 0x7feb48156a90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7feb380098e0 tx=0x7feb380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe61129e349a5462 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4559a700 1 -- 172.21.15.29:0/673908120 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800b640 con 0x7feb481533f0 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4559a700 1 -- 172.21.15.29:0/673908120 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3800b7a0 con 0x7feb481533f0 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.680+0000 7feb4559a700 1 -- 172.21.15.29:0/673908120 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800bb40 con 0x7feb481533f0 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 -- 172.21.15.29:0/673908120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481533f0 msgr2=0x7feb48156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 --2- 172.21.15.29:0/673908120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481533f0 0x7feb48156a90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7feb380098e0 tx=0x7feb380091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 -- 172.21.15.29:0/673908120 shutdown_connections 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 --2- 172.21.15.29:0/673908120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481533f0 0x7feb48156a90 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 -- 172.21.15.29:0/673908120 >> 172.21.15.29:0/673908120 conn(0x7feb480680a0 msgr2=0x7feb480cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 -- 172.21.15.29:0/673908120 shutdown_connections 2023-10-07T10:45:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.681+0000 7feb4cdff700 1 -- 172.21.15.29:0/673908120 wait complete. 2023-10-07T10:45:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.682+0000 7feb4cdff700 1 Processor -- start 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 -- start start 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 0x7feb480c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb481533f0 0x7feb4813a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481537c0 0x7feb48151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4813a7a0 con 0x7feb480b04b0 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb480c9e80 con 0x7feb481537c0 2023-10-07T10:45:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.683+0000 7feb4cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4814e3a0 con 0x7feb481533f0 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 0x7feb480c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 0x7feb480c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49964/0 (socket says 172.21.15.29:49964) 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 -- 172.21.15.29:0/1832258527 learned_addr learned my addr 172.21.15.29:0/1832258527 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 -- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb481533f0 msgr2=0x7feb4813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb481533f0 0x7feb4813a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 -- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481537c0 msgr2=0x7feb48151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481537c0 0x7feb48151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 -- 172.21.15.29:0/1832258527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38009420 con 0x7feb480b04b0 2023-10-07T10:45:15.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb4659c700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 0x7feb480c5ce0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7feb38009f80 tx=0x7feb38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:15.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.684+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800bb20 con 0x7feb480b04b0 2023-10-07T10:45:15.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.685+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb38011420 con 0x7feb480b04b0 2023-10-07T10:45:15.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.685+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380117a0 con 0x7feb480b04b0 2023-10-07T10:45:15.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.685+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb48152ae0 con 0x7feb480b04b0 2023-10-07T10:45:15.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.685+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb480c9b10 con 0x7feb480b04b0 2023-10-07T10:45:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.686+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4804c3c0 con 0x7feb480b04b0 2023-10-07T10:45:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.687+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feb38011b10 con 0x7feb480b04b0 2023-10-07T10:45:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.688+0000 7feb367fc700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feb28066f30 0x7feb280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.688+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5273+0+0 (secure 0 0 0) 0x7feb380858a0 con 0x7feb480b04b0 2023-10-07T10:45:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.688+0000 7feb45d9b700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feb28066f30 0x7feb280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.689+0000 7feb45d9b700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feb28066f30 0x7feb280693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb4816d020 tx=0x7feb3c007b40 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:15.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.692+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb38052ca0 con 0x7feb480b04b0 2023-10-07T10:45:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.896+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feb4806f660 con 0x7feb480b04b0 2023-10-07T10:45:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.897+0000 7feb367fc700 1 -- 172.21.15.29:0/1832258527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v161) v1 ==== 84+0+65 (secure 0 0 0) 0x7feb38015090 con 0x7feb480b04b0 2023-10-07T10:45:15.897 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:15.898 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T10:45:15.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:15.898 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T10:45:15.898 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T10:45:15.898 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:15.898 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feb28066f30 msgr2=0x7feb280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feb28066f30 0x7feb280693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb4816d020 tx=0x7feb3c007b40 comp rx=0 tx=0).stop 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 msgr2=0x7feb480c5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 0x7feb480c5ce0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7feb38009f80 tx=0x7feb38017000 comp rx=0 tx=0).stop 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 shutdown_connections 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7feb28066f30 0x7feb280693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb481537c0 0x7feb48151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb480b04b0 0x7feb480c5ce0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 --2- 172.21.15.29:0/1832258527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb481533f0 0x7feb4813a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 >> 172.21.15.29:0/1832258527 conn(0x7feb480680a0 msgr2=0x7feb4806eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 shutdown_connections 2023-10-07T10:45:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:15.899+0000 7feb4cdff700 1 -- 172.21.15.29:0/1832258527 wait complete. 2023-10-07T10:45:15.917 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T10:45:15.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T10:45:16.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.117+0000 7f9fa21ff700 1 Processor -- start 2023-10-07T10:45:16.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.117+0000 7f9fa21ff700 1 -- start start 2023-10-07T10:45:16.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.118+0000 7f9fa21ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0ced00 0x7f9f9c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:16.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9fa21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0c8a20 0x7f9f9c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:16.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9fa21ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c0690f0 con 0x7f9f9c0b0610 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9fa21ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0cf2e0 con 0x7f9f9c0ced00 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9fa21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0687c0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0ced00 0x7f9f9c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f9c0ced00 0x7f9f9c0b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49976/0 (socket says 172.21.15.29:49976) 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 -- 172.21.15.29:0/415327912 learned_addr learned my addr 172.21.15.29:0/415327912 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9b7fe700 1 --1- 172.21.15.29:0/415327912 >> v1:172.21.15.29:6789/0 conn(0x7f9f9c0b0610 0x7f9f9c0c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47874/0 (socket says 172.21.15.29:47874) 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9bfff700 1 --2- 172.21.15.29:0/415327912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0c8a20 0x7f9f9c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 -- 172.21.15.29:0/415327912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0c8a20 msgr2=0x7f9f9c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 --2- 172.21.15.29:0/415327912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0c8a20 0x7f9f9c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 -- 172.21.15.29:0/415327912 >> v1:172.21.15.29:6789/0 conn(0x7f9f9c0b0610 legacy=0x7f9f9c0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:16.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.119+0000 7f9f9affd700 1 -- 172.21.15.29:0/415327912 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c068aa0 con 0x7f9f9c0ced00 2023-10-07T10:45:16.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9f9affd700 1 --2- 172.21.15.29:0/415327912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c0b09e0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f9f900099b0 tx=0x7f9f90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9242d88e7f8076 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:16.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9f9a7fc700 1 -- 172.21.15.29:0/415327912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9000b860 con 0x7f9f9c0ced00 2023-10-07T10:45:16.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9f9a7fc700 1 -- 172.21.15.29:0/415327912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9000b9c0 con 0x7f9f9c0ced00 2023-10-07T10:45:16.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9f9a7fc700 1 -- 172.21.15.29:0/415327912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9000bd60 con 0x7f9f9c0ced00 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9fa21ff700 1 -- 172.21.15.29:0/415327912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ced00 msgr2=0x7f9f9c0b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/415327912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c0b09e0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f9f900099b0 tx=0x7f9f90009250 comp rx=0 tx=0).stop 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9fa21ff700 1 -- 172.21.15.29:0/415327912 shutdown_connections 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/415327912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0c8a20 0x7f9f9c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/415327912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c0b09e0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.120+0000 7f9fa21ff700 1 -- 172.21.15.29:0/415327912 >> 172.21.15.29:0/415327912 conn(0x7f9f9c063a10 msgr2=0x7f9f9c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.121+0000 7f9fa21ff700 1 -- 172.21.15.29:0/415327912 shutdown_connections 2023-10-07T10:45:16.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.121+0000 7f9fa21ff700 1 -- 172.21.15.29:0/415327912 wait complete. 2023-10-07T10:45:16.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.122+0000 7f9fa21ff700 1 Processor -- start 2023-10-07T10:45:16.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.122+0000 7f9fa21ff700 1 -- start start 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9fa21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b0610 0x7f9f9c06b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9fa21ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 0x7f9f9c06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9fa21ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c14b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9fa21ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c14b840 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9fa21ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c06bd20 con 0x7f9f9c0ced00 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9fa21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c06c000 con 0x7f9f9c0b0610 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9f9affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 0x7f9f9c06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9f9affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 0x7f9f9c06cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49986/0 (socket says 172.21.15.29:49986) 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.123+0000 7f9f9affd700 1 -- 172.21.15.29:0/921851547 learned_addr learned my addr 172.21.15.29:0/921851547 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9b7fe700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b0610 0x7f9f9c06b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9bfff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c14b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9affd700 1 -- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b0610 msgr2=0x7f9f9c06b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9affd700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b0610 0x7f9f9c06b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9affd700 1 -- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0ced00 msgr2=0x7f9f9c14b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9affd700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c14b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9affd700 1 -- 172.21.15.29:0/921851547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f900094f0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9b7fe700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b0610 0x7f9f9c06b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f9affd700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 0x7f9f9c06cdc0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009290 tx=0x7f9f90017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9001a450 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9001a5b0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9001a950 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.124+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c06c2e0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.125+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c06c8a0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.126+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c06cae0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.126+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f9001ad00 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.127+0000 7f9f837fe700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f88066f30 0x7f9f880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.127+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f9f90086080 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.128+0000 7f9f9b7fe700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f88066f30 0x7f9f880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.128+0000 7f9f9b7fe700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f88066f30 0x7f9f880693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007fd0 tx=0x7f9f8c007e10 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.132+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f900b75a0 con 0x7f9f9c0c8a20 2023-10-07T10:45:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:16.334+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9f9c0718b0 con 0x7f9f9c0c8a20 2023-10-07T10:45:17.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.110+0000 7f9f837fe700 1 -- 172.21.15.29:0/921851547 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v162) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9f9c06cae0 con 0x7f9f9c0c8a20 2023-10-07T10:45:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.117+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f88066f30 msgr2=0x7f9f880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.117+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f88066f30 0x7f9f880693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007fd0 tx=0x7f9f8c007e10 comp rx=0 tx=0).stop 2023-10-07T10:45:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.117+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 msgr2=0x7f9f9c06cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.117+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 0x7f9f9c06cdc0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009290 tx=0x7f9f90017000 comp rx=0 tx=0).stop 2023-10-07T10:45:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 shutdown_connections 2023-10-07T10:45:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f9f88066f30 0x7f9f880693e0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007fd0 tx=0x7f9f8c007e10 comp rx=0 tx=0).stop 2023-10-07T10:45:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f9c0ced00 0x7f9f9c14b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f9c0b0610 0x7f9f9c06b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 --2- 172.21.15.29:0/921851547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9c0c8a20 0x7f9f9c06cdc0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 >> 172.21.15.29:0/921851547 conn(0x7f9f9c063a10 msgr2=0x7f9f9c1591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.118+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 shutdown_connections 2023-10-07T10:45:17.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.119+0000 7f9fa21ff700 1 -- 172.21.15.29:0/921851547 wait complete. 2023-10-07T10:45:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T10:45:17.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T10:45:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.347+0000 7f98d85ff700 1 Processor -- start 2023-10-07T10:45:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.348+0000 7f98d85ff700 1 -- start start 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d85ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98d00cc570 0x7f98d015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d85ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d015a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d85ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d015adf0 con 0x7f98d0156170 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d85ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00ad7c0 con 0x7f98d00cc570 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d85ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00b0000 con 0x7f98d0156540 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d5b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98d00cc570 0x7f98d015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d639b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f98d0156170 0x7f98d00cbe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47890/0 (socket says 172.21.15.29:47890) 2023-10-07T10:45:17.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d5b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98d00cc570 0x7f98d015a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49990/0 (socket says 172.21.15.29:49990) 2023-10-07T10:45:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d639b700 1 -- 172.21.15.29:0/2239670778 learned_addr learned my addr 172.21.15.29:0/2239670778 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.349+0000 7f98d6b9c700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5b9a700 1 -- 172.21.15.29:0/2239670778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 msgr2=0x7f98d015a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5b9a700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d015a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5b9a700 1 -- 172.21.15.29:0/2239670778 >> v1:172.21.15.29:6789/0 conn(0x7f98d0156170 legacy=0x7f98d00cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5b9a700 1 -- 172.21.15.29:0/2239670778 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d015ab40 con 0x7f98d00cc570 2023-10-07T10:45:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5399700 1 -- 172.21.15.29:0/2239670778 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645906511 0 0) 0x7f98d015adf0 con 0x7f98d0156170 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d6b9c700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5b9a700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d00cc570 0x7f98d015a0c0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098e0 tx=0x7f98c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e44d4fdf8f51aa server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5399700 1 -- 172.21.15.29:0/2239670778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0009e80 con 0x7f98d00cc570 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5399700 1 -- 172.21.15.29:0/2239670778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000b860 con 0x7f98d00cc570 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.350+0000 7f98d5399700 1 -- 172.21.15.29:0/2239670778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c000bbe0 con 0x7f98d00cc570 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 -- 172.21.15.29:0/2239670778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d00cc570 msgr2=0x7f98d015a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d00cc570 0x7f98d015a0c0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098e0 tx=0x7f98c00091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 -- 172.21.15.29:0/2239670778 shutdown_connections 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 --2- 172.21.15.29:0/2239670778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d00cc570 0x7f98d015a0c0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 -- 172.21.15.29:0/2239670778 >> 172.21.15.29:0/2239670778 conn(0x7f98d0067610 msgr2=0x7f98d0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 -- 172.21.15.29:0/2239670778 shutdown_connections 2023-10-07T10:45:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.351+0000 7f98d85ff700 1 -- 172.21.15.29:0/2239670778 wait complete. 2023-10-07T10:45:17.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.352+0000 7f98d85ff700 1 Processor -- start 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.352+0000 7f98d85ff700 1 -- start start 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.353+0000 7f98d85ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d00cc570 0x7f98d013ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.353+0000 7f98d85ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 0x7f98d014a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.353+0000 7f98d85ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d00b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.353+0000 7f98d85ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00aa110 con 0x7f98d0156170 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.353+0000 7f98d85ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d014b100 con 0x7f98d00cc570 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.353+0000 7f98d85ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d014b3e0 con 0x7f98d0156540 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 0x7f98d014a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 0x7f98d014a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49994/0 (socket says 172.21.15.29:49994) 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 -- 172.21.15.29:0/570022734 learned_addr learned my addr 172.21.15.29:0/570022734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d639b700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d00cc570 0x7f98d013ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d6b9c700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d00b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 -- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 msgr2=0x7f98d00b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d00b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 -- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d00cc570 msgr2=0x7f98d013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d00cc570 0x7f98d013ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 -- 172.21.15.29:0/570022734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009420 con 0x7f98d0156170 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d6b9c700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d00b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.354+0000 7f98d5b9a700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 0x7f98d014a610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f98c0013000 tx=0x7f98c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.355+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0000c40 con 0x7f98d0156170 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.355+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000b9d0 con 0x7f98d0156170 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.355+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0025650 con 0x7f98d0156170 2023-10-07T10:45:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.355+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d00a9ac0 con 0x7f98d0156170 2023-10-07T10:45:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.355+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d0145da0 con 0x7f98d0156170 2023-10-07T10:45:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.356+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d00718b0 con 0x7f98d0156170 2023-10-07T10:45:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.357+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f98c0023070 con 0x7f98d0156170 2023-10-07T10:45:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.358+0000 7f98c67fc700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98b4066f30 0x7f98b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.359+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5643+0+0 (secure 0 0 0) 0x7f98c0085ee0 con 0x7f98d0156170 2023-10-07T10:45:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.359+0000 7f98d639b700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98b4066f30 0x7f98b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.360+0000 7f98d639b700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98b4066f30 0x7f98b40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007fd0 tx=0x7f98cc007d30 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.363+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98c0053170 con 0x7f98d0156170 2023-10-07T10:45:17.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:17.566+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f98d00cd040 con 0x7f98d0156170 2023-10-07T10:45:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.117+0000 7f98c67fc700 1 -- 172.21.15.29:0/570022734 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v163) v1 ==== 170+0+0 (secure 0 0 0) 0x7f98c0056790 con 0x7f98d0156170 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.123+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98b4066f30 msgr2=0x7f98b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.123+0000 7f98d85ff700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98b4066f30 0x7f98b40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007fd0 tx=0x7f98cc007d30 comp rx=0 tx=0).stop 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.123+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 msgr2=0x7f98d014a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.123+0000 7f98d85ff700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 0x7f98d014a610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f98c0013000 tx=0x7f98c0017000 comp rx=0 tx=0).stop 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.124+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 shutdown_connections 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.124+0000 7f98d85ff700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f98b4066f30 0x7f98b40693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.124+0000 7f98d85ff700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d0156540 0x7f98d00b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.124+0000 7f98d85ff700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d00cc570 0x7f98d013ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.124+0000 7f98d85ff700 1 --2- 172.21.15.29:0/570022734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d0156170 0x7f98d014a610 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.124+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 >> 172.21.15.29:0/570022734 conn(0x7f98d0067610 msgr2=0x7f98d014c080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:18.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.128+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 shutdown_connections 2023-10-07T10:45:18.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.128+0000 7f98d85ff700 1 -- 172.21.15.29:0/570022734 wait complete. 2023-10-07T10:45:18.139 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T10:45:18.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T10:45:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.355+0000 7f33247ff700 1 Processor -- start 2023-10-07T10:45:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.355+0000 7f33247ff700 1 -- start start 2023-10-07T10:45:18.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f33247ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f331c13fba0 0x7f331c06ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f33247ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331c0c78c0 con 0x7f331c13ff70 2023-10-07T10:45:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f33247ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331c06fa50 con 0x7f331c067d20 2023-10-07T10:45:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f33247ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331c06c0f0 con 0x7f331c13fba0 2023-10-07T10:45:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f3321d9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f331c13ff70 0x7f331c067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47896/0 (socket says 172.21.15.29:47896) 2023-10-07T10:45:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f3321d9a700 1 -- 172.21.15.29:0/1553427128 learned_addr learned my addr 172.21.15.29:0/1553427128 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f332259b700 1 --2- 172.21.15.29:0/1553427128 >> v2:172.21.15.144:3300/0 conn(0x7f331c13fba0 0x7f331c06ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548259622 0 0) 0x7f331c0c78c0 con 0x7f331c13ff70 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.357+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3308003570 con 0x7f331c13ff70 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984628489 0 0) 0x7f331c06fa50 con 0x7f331c067d20 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f331c0c78c0 con 0x7f331c067d20 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012408395 0 0) 0x7f3308003570 con 0x7f331c13ff70 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 >> v2:172.21.15.144:3300/0 conn(0x7f331c13fba0 msgr2=0x7f331c06ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 --2- 172.21.15.29:0/1553427128 >> v2:172.21.15.144:3300/0 conn(0x7f331c13fba0 0x7f331c06ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 >> v1:172.21.15.144:6789/0 conn(0x7f331c067d20 legacy=0x7f331c0c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c06c3d0 con 0x7f331c13ff70 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f330c0015f0 con 0x7f331c13ff70 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 >> v1:172.21.15.29:6789/0 conn(0x7f331c13ff70 legacy=0x7f331c067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3308003bf0 0x7f3308003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3308004810 0x7f3308004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33080051c0 0x7f330800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c06c0f0 con 0x7f33080051c0 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330800c0b0 con 0x7f3308004810 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330800c270 con 0x7f3308003bf0 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f332259b700 1 --2- 172.21.15.29:0/1553427128 >> v2:172.21.15.144:3300/0 conn(0x7f331c13fba0 0x7f331c06ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3322d9c700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33080051c0 0x7f330800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f3321d9a700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3308004810 0x7f3308004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.358+0000 7f332259b700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3308003bf0 0x7f3308003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 -- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3308003bf0 msgr2=0x7f3308003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3308003bf0 0x7f3308003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 -- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3308004810 msgr2=0x7f3308004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3308004810 0x7f3308004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 -- 172.21.15.29:0/1553427128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308003570 con 0x7f33080051c0 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 -- 172.21.15.29:0/1553427128 reap_dead start 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3322d9c700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33080051c0 0x7f330800ba60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f331c0c7870 tx=0x7f33140017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad49ef4eecd9e95 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314006ba0 con 0x7f33080051c0 2023-10-07T10:45:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3314006d00 con 0x7f33080051c0 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.359+0000 7f3321599700 1 -- 172.21.15.29:0/1553427128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331400cac0 con 0x7f33080051c0 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 -- 172.21.15.29:0/1553427128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33080051c0 msgr2=0x7f330800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33080051c0 0x7f330800ba60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f331c0c7870 tx=0x7f33140017a0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 -- 172.21.15.29:0/1553427128 shutdown_connections 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 --2- 172.21.15.29:0/1553427128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33080051c0 0x7f330800ba60 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 -- 172.21.15.29:0/1553427128 >> 172.21.15.29:0/1553427128 conn(0x7f331c06b1d0 msgr2=0x7f331c0cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 -- 172.21.15.29:0/1553427128 shutdown_connections 2023-10-07T10:45:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.360+0000 7f33247ff700 1 -- 172.21.15.29:0/1553427128 wait complete. 2023-10-07T10:45:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.362+0000 7f33247ff700 1 Processor -- start 2023-10-07T10:45:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.362+0000 7f33247ff700 1 -- start start 2023-10-07T10:45:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.363+0000 7f33247ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33080051c0 0x7f331c157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.363+0000 7f33247ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 0x7f331c0c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.363+0000 7f33247ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331c0c6ce0 0x7f331c0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.363+0000 7f33247ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0b22e0 con 0x7f331c0c4330 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.363+0000 7f33247ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0c7160 con 0x7f33080051c0 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.363+0000 7f33247ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0b29f0 con 0x7f331c0c6ce0 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 0x7f331c0c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 0x7f331c0c67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50012/0 (socket says 172.21.15.29:50012) 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 -- 172.21.15.29:0/1725159090 learned_addr learned my addr 172.21.15.29:0/1725159090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f332259b700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33080051c0 0x7f331c157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3322d9c700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331c0c6ce0 0x7f331c0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 -- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331c0c6ce0 msgr2=0x7f331c0b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331c0c6ce0 0x7f331c0b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 -- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33080051c0 msgr2=0x7f331c157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33080051c0 0x7f331c157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 -- 172.21.15.29:0/1725159090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c0b1a40 con 0x7f331c0c4330 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.364+0000 7f3321d9a700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 0x7f331c0c67a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f330c00f880 tx=0x7f330c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.365+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0072e0 con 0x7f331c0c4330 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.365+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c007460 con 0x7f331c0c4330 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.365+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0057f0 con 0x7f331c0c4330 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.365+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c0b1d20 con 0x7f331c0c4330 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.365+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c159f60 con 0x7f331c0c4330 2023-10-07T10:45:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.366+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331c04c3c0 con 0x7f331c0c4330 2023-10-07T10:45:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.367+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f330c005b60 con 0x7f331c0c4330 2023-10-07T10:45:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.368+0000 7f33127fc700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3300066f30 0x7f33000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.368+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f330c085140 con 0x7f331c0c4330 2023-10-07T10:45:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.369+0000 7f332259b700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3300066f30 0x7f33000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.369+0000 7f332259b700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3300066f30 0x7f33000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33180093f0 tx=0x7f3318008040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.372+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f330c052300 con 0x7f331c0c4330 2023-10-07T10:45:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:18.617+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f331c06c6b0 con 0x7f331c0c4330 2023-10-07T10:45:19.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.122+0000 7f33127fc700 1 -- 172.21.15.29:0/1725159090 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v81) v1 ==== 174+0+0 (secure 0 0 0) 0x7f330c055920 con 0x7f331c0c4330 2023-10-07T10:45:19.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.128+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3300066f30 msgr2=0x7f33000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.128+0000 7f33247ff700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3300066f30 0x7f33000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33180093f0 tx=0x7f3318008040 comp rx=0 tx=0).stop 2023-10-07T10:45:19.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.128+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 msgr2=0x7f331c0c67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.128+0000 7f33247ff700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 0x7f331c0c67a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f330c00f880 tx=0x7f330c009040 comp rx=0 tx=0).stop 2023-10-07T10:45:19.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.129+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 shutdown_connections 2023-10-07T10:45:19.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.129+0000 7f33247ff700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3300066f30 0x7f33000693e0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33180093f0 tx=0x7f3318008040 comp rx=0 tx=0).stop 2023-10-07T10:45:19.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.129+0000 7f33247ff700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331c0c6ce0 0x7f331c0b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.129+0000 7f33247ff700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33080051c0 0x7f331c157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.129+0000 7f33247ff700 1 --2- 172.21.15.29:0/1725159090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f331c0c4330 0x7f331c0c67a0 secure :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f330c00f880 tx=0x7f330c009040 comp rx=0 tx=0).stop 2023-10-07T10:45:19.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.129+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 >> 172.21.15.29:0/1725159090 conn(0x7f331c06b1d0 msgr2=0x7f331c13ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:19.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.134+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 shutdown_connections 2023-10-07T10:45:19.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.138+0000 7f33247ff700 1 -- 172.21.15.29:0/1725159090 wait complete. 2023-10-07T10:45:19.147 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 20 and data pool 21 2023-10-07T10:45:19.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:19.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.369+0000 7fe5ff3ff700 1 Processor -- start 2023-10-07T10:45:19.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.369+0000 7fe5ff3ff700 1 -- start start 2023-10-07T10:45:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.370+0000 7fe5ff3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 0x7fe5f815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.370+0000 7fe5ff3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 0x7fe5f80af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.370+0000 7fe5ff3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f814d790 con 0x7fe5f80ad370 2023-10-07T10:45:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.370+0000 7fe5ff3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f815a880 con 0x7fe5f8159ec0 2023-10-07T10:45:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.370+0000 7fe5ff3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80ae5d0 con 0x7fe5f80af400 2023-10-07T10:45:19.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fc99a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 0x7fe5f815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 0x7fe5f80af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 0x7fe5f80af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:33378/0 (socket says 172.21.15.29:33378) 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fc99a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 0x7fe5f815a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:37256/0 (socket says 172.21.15.29:37256) 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 -- 172.21.15.29:0/2801092325 learned_addr learned my addr 172.21.15.29:0/2801092325 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 -- 172.21.15.29:0/2801092325 >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 msgr2=0x7fe5f815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 --2- 172.21.15.29:0/2801092325 >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 0x7fe5f815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 -- 172.21.15.29:0/2801092325 >> v1:172.21.15.144:6789/0 conn(0x7fe5f80ad370 legacy=0x7fe5f80cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 -- 172.21.15.29:0/2801092325 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80ad740 con 0x7fe5f80af400 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fc99a700 1 --2- 172.21.15.29:0/2801092325 >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 0x7fe5f815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.371+0000 7fe5fd99c700 1 --2- 172.21.15.29:0/2801092325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 0x7fe5f80af840 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009f20 tx=0x7fe5f0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e48c81f97723fb31 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5effff700 1 -- 172.21.15.29:0/2801092325 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f000e070 con 0x7fe5f80af400 2023-10-07T10:45:19.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5effff700 1 -- 172.21.15.29:0/2801092325 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f000b9d0 con 0x7fe5f80af400 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5effff700 1 -- 172.21.15.29:0/2801092325 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00092e0 con 0x7fe5f80af400 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/2801092325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 msgr2=0x7fe5f80af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/2801092325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 0x7fe5f80af840 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009f20 tx=0x7fe5f0009840 comp rx=0 tx=0).stop 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/2801092325 shutdown_connections 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/2801092325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80af400 0x7fe5f80af840 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/2801092325 >> v2:172.21.15.144:3300/0 conn(0x7fe5f8159ec0 0x7fe5f815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/2801092325 >> 172.21.15.29:0/2801092325 conn(0x7fe5f80680a0 msgr2=0x7fe5f80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.372+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/2801092325 shutdown_connections 2023-10-07T10:45:19.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.373+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/2801092325 wait complete. 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.374+0000 7fe5ff3ff700 1 Processor -- start 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.374+0000 7fe5ff3ff700 1 -- start start 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5ff3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 0x7fe5f80a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5ff3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80af400 0x7fe5f8157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5ff3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f8159ec0 0x7fe5f8158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5ff3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80a8070 con 0x7fe5f80ad370 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5ff3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80a7460 con 0x7fe5f80af400 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5ff3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80a7740 con 0x7fe5f8159ec0 2023-10-07T10:45:19.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5fd19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 0x7fe5f80a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5fd19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 0x7fe5f80a95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50016/0 (socket says 172.21.15.29:50016) 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.375+0000 7fe5fd19b700 1 -- 172.21.15.29:0/4289189285 learned_addr learned my addr 172.21.15.29:0/4289189285 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fc99a700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80af400 0x7fe5f8157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd99c700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f8159ec0 0x7fe5f8158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd19b700 1 -- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f8159ec0 msgr2=0x7fe5f8158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd19b700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f8159ec0 0x7fe5f8158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd19b700 1 -- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80af400 msgr2=0x7fe5f8157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd19b700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80af400 0x7fe5f8157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd19b700 1 -- 172.21.15.29:0/4289189285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8009420 con 0x7fe5f80ad370 2023-10-07T10:45:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5fd19b700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 0x7fe5f80a95e0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400c590 tx=0x7fe5f400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:19.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4011590 con 0x7fe5f80ad370 2023-10-07T10:45:19.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f4011710 con 0x7fe5f80ad370 2023-10-07T10:45:19.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4011ad0 con 0x7fe5f80ad370 2023-10-07T10:45:19.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.376+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0009ae0 con 0x7fe5f80ad370 2023-10-07T10:45:19.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.377+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f80a7d00 con 0x7fe5f80ad370 2023-10-07T10:45:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.378+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f804c3c0 con 0x7fe5f80ad370 2023-10-07T10:45:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.379+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe5f401b3b0 con 0x7fe5f80ad370 2023-10-07T10:45:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.379+0000 7fe5ed7fa700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5dc066f30 0x7fe5dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.380+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fe5f4083be0 con 0x7fe5f80ad370 2023-10-07T10:45:19.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.380+0000 7fe5fc99a700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5dc066f30 0x7fe5dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.380+0000 7fe5fc99a700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5dc066f30 0x7fe5dc0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe5f816aa80 tx=0x7fe5e80091a0 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:19.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.383+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5f4050c10 con 0x7fe5f80ad370 2023-10-07T10:45:19.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.591+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5f80cd020 con 0x7fe5f80ad370 2023-10-07T10:45:19.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.592+0000 7fe5ed7fa700 1 -- 172.21.15.29:0/4289189285 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16115 (secure 0 0 0) 0x7fe5f4017070 con 0x7fe5f80ad370 2023-10-07T10:45:19.593 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:19.593 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":164,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:45:18.619508+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:45:16.377745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-07T10:45:17.609874+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5dc066f30 msgr2=0x7fe5dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5dc066f30 0x7fe5dc0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe5f816aa80 tx=0x7fe5e80091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 msgr2=0x7fe5f80a95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 0x7fe5f80a95e0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400c590 tx=0x7fe5f400c8a0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 shutdown_connections 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fe5dc066f30 0x7fe5dc0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f8159ec0 0x7fe5f8158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f80ad370 0x7fe5f80a95e0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 --2- 172.21.15.29:0/4289189285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80af400 0x7fe5f8157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 >> 172.21.15.29:0/4289189285 conn(0x7fe5f80680a0 msgr2=0x7fe5f80b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:19.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 shutdown_connections 2023-10-07T10:45:19.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.594+0000 7fe5ff3ff700 1 -- 172.21.15.29:0/4289189285 wait complete. 2023-10-07T10:45:19.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:19.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.809+0000 7f3749bff700 1 Processor -- start 2023-10-07T10:45:19.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.809+0000 7f3749bff700 1 -- start start 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f3749bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37440ae8b0 0x7f37440b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f3749bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37440b1840 0x7f3744157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f3749bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3744158ad0 con 0x7f3744157290 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f3749bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440b1cc0 con 0x7f37440ae8b0 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f3749bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374415b140 con 0x7f37440b1840 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f3742ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37440ae8b0 0x7f37440b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f37437fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3744157290 0x7f37440cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47898/0 (socket says 172.21.15.29:47898) 2023-10-07T10:45:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.810+0000 7f37437fe700 1 -- 172.21.15.29:0/2925153645 learned_addr learned my addr 172.21.15.29:0/2925153645 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3743fff700 1 --2- 172.21.15.29:0/2925153645 >> v2:172.21.15.144:3300/0 conn(0x7f37440b1840 0x7f3744157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3742ffd700 1 -- 172.21.15.29:0/2925153645 >> v2:172.21.15.144:3300/0 conn(0x7f37440b1840 msgr2=0x7f3744157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3742ffd700 1 --2- 172.21.15.29:0/2925153645 >> v2:172.21.15.144:3300/0 conn(0x7f37440b1840 0x7f3744157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3742ffd700 1 -- 172.21.15.29:0/2925153645 >> v1:172.21.15.29:6789/0 conn(0x7f3744157290 legacy=0x7f37440cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3742ffd700 1 -- 172.21.15.29:0/2925153645 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374415b420 con 0x7f37440ae8b0 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3743fff700 1 --2- 172.21.15.29:0/2925153645 >> v2:172.21.15.144:3300/0 conn(0x7f37440b1840 0x7f3744157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f37427fc700 1 -- 172.21.15.29:0/2925153645 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480313603 0 0) 0x7f3744158ad0 con 0x7f3744157290 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f3742ffd700 1 --2- 172.21.15.29:0/2925153645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440ae8b0 0x7f37440b1300 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f37440cbff0 tx=0x7f37380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4befdc1fb44a9d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f37427fc700 1 -- 172.21.15.29:0/2925153645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373800b6b0 con 0x7f37440ae8b0 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f37427fc700 1 -- 172.21.15.29:0/2925153645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373800b810 con 0x7f37440ae8b0 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.811+0000 7f37427fc700 1 -- 172.21.15.29:0/2925153645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373800bbb0 con 0x7f37440ae8b0 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 -- 172.21.15.29:0/2925153645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440ae8b0 msgr2=0x7f37440b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 --2- 172.21.15.29:0/2925153645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440ae8b0 0x7f37440b1300 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f37440cbff0 tx=0x7f37380091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 -- 172.21.15.29:0/2925153645 shutdown_connections 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 --2- 172.21.15.29:0/2925153645 >> v2:172.21.15.144:3300/0 conn(0x7f37440b1840 0x7f3744157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 --2- 172.21.15.29:0/2925153645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37440ae8b0 0x7f37440b1300 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 -- 172.21.15.29:0/2925153645 >> 172.21.15.29:0/2925153645 conn(0x7f37440680a0 msgr2=0x7f37440cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 -- 172.21.15.29:0/2925153645 shutdown_connections 2023-10-07T10:45:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.812+0000 7f3749bff700 1 -- 172.21.15.29:0/2925153645 wait complete. 2023-10-07T10:45:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.814+0000 7f3749bff700 1 Processor -- start 2023-10-07T10:45:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.814+0000 7f3749bff700 1 -- start start 2023-10-07T10:45:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3749bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37440b1840 0x7f374416ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3749bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3744157290 0x7f374416c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3749bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 0x7f37441590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3749bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374415cca0 con 0x7f374416a740 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3749bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374415c0d0 con 0x7f37440b1840 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3749bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374415c370 con 0x7f3744157290 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3743fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 0x7f37441590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3743fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 0x7f37441590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50038/0 (socket says 172.21.15.29:50038) 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3743fff700 1 -- 172.21.15.29:0/2289477455 learned_addr learned my addr 172.21.15.29:0/2289477455 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f37437fe700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37440b1840 0x7f374416ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.815+0000 7f3742ffd700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3744157290 0x7f374416c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3743fff700 1 -- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3744157290 msgr2=0x7f374416c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3743fff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3744157290 0x7f374416c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3743fff700 1 -- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37440b1840 msgr2=0x7f374416ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3743fff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37440b1840 0x7f374416ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:19.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3743fff700 1 -- 172.21.15.29:0/2289477455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738009420 con 0x7f374416a740 2023-10-07T10:45:19.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f37437fe700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37440b1840 0x7f374416ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:19.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3743fff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 0x7f37441590d0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f373c00a380 tx=0x7f373c007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:19.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c015440 con 0x7f374416a740 2023-10-07T10:45:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c0155a0 con 0x7f374416a740 2023-10-07T10:45:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374415c650 con 0x7f374416a740 2023-10-07T10:45:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c015960 con 0x7f374416a740 2023-10-07T10:45:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.816+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374416cda0 con 0x7f374416a740 2023-10-07T10:45:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.817+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37440b20b0 con 0x7f374416a740 2023-10-07T10:45:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.818+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f373c00d440 con 0x7f374416a740 2023-10-07T10:45:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.819+0000 7f372b7fe700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3730066f60 0x7f3730069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.819+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f373c085210 con 0x7f374416a740 2023-10-07T10:45:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.820+0000 7f37437fe700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3730066f60 0x7f3730069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.820+0000 7f37437fe700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3730066f60 0x7f3730069410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f374406f2b0 tx=0x7f3734007d30 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:19.823+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f373c052240 con 0x7f374416a740 2023-10-07T10:45:20.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.027+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f374404c3c0 con 0x7f374416a740 2023-10-07T10:45:20.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.028+0000 7f372b7fe700 1 -- 172.21.15.29:0/2289477455 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+16115 (secure 0 0 0) 0x7f373c055860 con 0x7f374416a740 2023-10-07T10:45:20.029 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":164,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:45:18.619508+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:45:16.377745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-07T10:45:17.609874+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:20.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.030+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3730066f60 msgr2=0x7f3730069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.030+0000 7f3749bff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3730066f60 0x7f3730069410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f374406f2b0 tx=0x7f3734007d30 comp rx=0 tx=0).stop 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.030+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 msgr2=0x7f37441590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.030+0000 7f3749bff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 0x7f37441590d0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f373c00a380 tx=0x7f373c007900 comp rx=0 tx=0).stop 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 shutdown_connections 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3730066f60 0x7f3730069410 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f374416a740 0x7f37441590d0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37440b1840 0x7f374416ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 --2- 172.21.15.29:0/2289477455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3744157290 0x7f374416c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 >> 172.21.15.29:0/2289477455 conn(0x7f37440680a0 msgr2=0x7f37440b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:20.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 shutdown_connections 2023-10-07T10:45:20.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.031+0000 7f3749bff700 1 -- 172.21.15.29:0/2289477455 wait complete. 2023-10-07T10:45:20.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T10:45:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.252+0000 7fdbeffff700 1 Processor -- start 2023-10-07T10:45:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.252+0000 7fdbeffff700 1 -- start start 2023-10-07T10:45:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbeffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbe8156180 0x7fdbe80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbeffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80ad4d0 con 0x7fdbe80ae570 2023-10-07T10:45:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbeffff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80ad7b0 con 0x7fdbe80b0520 2023-10-07T10:45:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbeffff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe80ada90 con 0x7fdbe8156180 2023-10-07T10:45:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbedd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbe8156180 0x7fdbe80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbedd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbe8156180 0x7fdbe80afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50040/0 (socket says 172.21.15.29:50040) 2023-10-07T10:45:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbedd9b700 1 -- 172.21.15.29:0/3249619515 learned_addr learned my addr 172.21.15.29:0/3249619515 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.253+0000 7fdbed59a700 1 --1- 172.21.15.29:0/3249619515 >> v1:172.21.15.29:6789/0 conn(0x7fdbe80ae570 0x7fdbe80accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47906/0 (socket says 172.21.15.29:47906) 2023-10-07T10:45:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.254+0000 7fdbedd9b700 1 -- 172.21.15.29:0/3249619515 >> v1:172.21.15.144:6789/0 conn(0x7fdbe80b0520 legacy=0x7fdbe80b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.254+0000 7fdbedd9b700 1 -- 172.21.15.29:0/3249619515 >> v1:172.21.15.29:6789/0 conn(0x7fdbe80ae570 legacy=0x7fdbe80accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.254+0000 7fdbecd99700 1 -- 172.21.15.29:0/3249619515 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024541939 0 0) 0x7fdbe80ad4d0 con 0x7fdbe80ae570 2023-10-07T10:45:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.254+0000 7fdbedd9b700 1 -- 172.21.15.29:0/3249619515 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80add70 con 0x7fdbe8156180 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.254+0000 7fdbedd9b700 1 --2- 172.21.15.29:0/3249619515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe8156180 0x7fdbe80afc00 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40099b0 tx=0x7fdbe4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3844ad876c58bb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.254+0000 7fdbecd99700 1 -- 172.21.15.29:0/3249619515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400b860 con 0x7fdbe8156180 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbecd99700 1 -- 172.21.15.29:0/3249619515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe400b9c0 con 0x7fdbe8156180 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbecd99700 1 -- 172.21.15.29:0/3249619515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400bd60 con 0x7fdbe8156180 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 -- 172.21.15.29:0/3249619515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe8156180 msgr2=0x7fdbe80afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 --2- 172.21.15.29:0/3249619515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe8156180 0x7fdbe80afc00 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40099b0 tx=0x7fdbe4009250 comp rx=0 tx=0).stop 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 -- 172.21.15.29:0/3249619515 shutdown_connections 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 --2- 172.21.15.29:0/3249619515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe8156180 0x7fdbe80afc00 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 -- 172.21.15.29:0/3249619515 >> 172.21.15.29:0/3249619515 conn(0x7fdbe8067610 msgr2=0x7fdbe8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 -- 172.21.15.29:0/3249619515 shutdown_connections 2023-10-07T10:45:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.255+0000 7fdbeffff700 1 -- 172.21.15.29:0/3249619515 wait complete. 2023-10-07T10:45:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.257+0000 7fdbeffff700 1 Processor -- start 2023-10-07T10:45:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.257+0000 7fdbeffff700 1 -- start start 2023-10-07T10:45:20.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbeffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe80ae570 0x7fdbe816d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:20.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbeffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 0x7fdbe8145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbeffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbe8156180 0x7fdbe80cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbeffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe816a7f0 con 0x7fdbe80b0520 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbeffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe816dd00 con 0x7fdbe80ae570 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbeffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe8169c70 con 0x7fdbe8156180 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbed59a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 0x7fdbe8145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbed59a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 0x7fdbe8145c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50056/0 (socket says 172.21.15.29:50056) 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbed59a700 1 -- 172.21.15.29:0/711212371 learned_addr learned my addr 172.21.15.29:0/711212371 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbedd9b700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe80ae570 0x7fdbe816d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.258+0000 7fdbee59c700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbe8156180 0x7fdbe80cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbed59a700 1 -- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbe8156180 msgr2=0x7fdbe80cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbed59a700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbe8156180 0x7fdbe80cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbed59a700 1 -- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe80ae570 msgr2=0x7fdbe816d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbed59a700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe80ae570 0x7fdbe816d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbed59a700 1 -- 172.21.15.29:0/711212371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40094f0 con 0x7fdbe80b0520 2023-10-07T10:45:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbed59a700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 0x7fdbe8145c60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800c810 tx=0x7fdbd8007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8005b40 con 0x7fdbe80b0520 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd8005cc0 con 0x7fdbe80b0520 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8007320 con 0x7fdbe80b0520 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe8169f50 con 0x7fdbe80b0520 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.259+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe816a510 con 0x7fdbe80b0520 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.260+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe816aa80 con 0x7fdbe80b0520 2023-10-07T10:45:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.261+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdbd801b3b0 con 0x7fdbe80b0520 2023-10-07T10:45:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.262+0000 7fdbddffb700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdbcc066f30 0x7fdbcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.262+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fdbd8083c80 con 0x7fdbe80b0520 2023-10-07T10:45:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.263+0000 7fdbedd9b700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdbcc066f30 0x7fdbcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.264+0000 7fdbedd9b700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdbcc066f30 0x7fdbcc0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40099b0 tx=0x7fdbe4017000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:20.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.267+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdbe816aa80 con 0x7fdbe80b0520 2023-10-07T10:45:20.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:20.515+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdbe816aa80 con 0x7fdbe80b0520 2023-10-07T10:45:21.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.149+0000 7fdbddffb700 1 -- 172.21.15.29:0/711212371 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v84) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdbe816aa80 con 0x7fdbe80b0520 2023-10-07T10:45:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdbcc066f30 msgr2=0x7fdbcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdbcc066f30 0x7fdbcc0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40099b0 tx=0x7fdbe4017000 comp rx=0 tx=0).stop 2023-10-07T10:45:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 msgr2=0x7fdbe8145c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 0x7fdbe8145c60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800c810 tx=0x7fdbd8007d90 comp rx=0 tx=0).stop 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 shutdown_connections 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fdbcc066f30 0x7fdbcc0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbe8156180 0x7fdbe80cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe80ae570 0x7fdbe816d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 --2- 172.21.15.29:0/711212371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe80b0520 0x7fdbe8145c60 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 >> 172.21.15.29:0/711212371 conn(0x7fdbe8067610 msgr2=0x7fdbe8158770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.152+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 shutdown_connections 2023-10-07T10:45:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.153+0000 7fdbeffff700 1 -- 172.21.15.29:0/711212371 wait complete. 2023-10-07T10:45:21.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T10:45:21.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.377+0000 7f41e6dff700 1 Processor -- start 2023-10-07T10:45:21.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.377+0000 7f41e6dff700 1 -- start start 2023-10-07T10:45:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.378+0000 7f41e6dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41e015a960 0x7f41e0158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.378+0000 7f41e6dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e00ae8a0 0x7f41e00b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.378+0000 7f41e6dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e00acb40 con 0x7f41e015ad30 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.378+0000 7f41e6dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e0158c60 con 0x7f41e015a960 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.378+0000 7f41e6dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00abf30 con 0x7f41e00ae8a0 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41e015a960 0x7f41e0158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41e015a960 0x7f41e0158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50064/0 (socket says 172.21.15.29:50064) 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 -- 172.21.15.29:0/308833909 learned_addr learned my addr 172.21.15.29:0/308833909 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:21.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e539c700 1 --2- 172.21.15.29:0/308833909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e00ae8a0 0x7f41e00b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:21.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 -- 172.21.15.29:0/308833909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e00ae8a0 msgr2=0x7f41e00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 --2- 172.21.15.29:0/308833909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e00ae8a0 0x7f41e00b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 -- 172.21.15.29:0/308833909 >> v1:172.21.15.144:6789/0 conn(0x7f41e015ad30 legacy=0x7f41e00ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 -- 172.21.15.29:0/308833909 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e00ac210 con 0x7f41e015a960 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.379+0000 7f41e4b9b700 1 --2- 172.21.15.29:0/308833909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e015a960 0x7f41e0158e20 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f41d40099b0 tx=0x7f41d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa4f99a3f6cb0be server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41df7fe700 1 -- 172.21.15.29:0/308833909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4009ea0 con 0x7f41e015a960 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41df7fe700 1 -- 172.21.15.29:0/308833909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d400b860 con 0x7f41e015a960 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41df7fe700 1 -- 172.21.15.29:0/308833909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d400bc00 con 0x7f41e015a960 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41e6dff700 1 -- 172.21.15.29:0/308833909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e015a960 msgr2=0x7f41e0158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41e6dff700 1 --2- 172.21.15.29:0/308833909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e015a960 0x7f41e0158e20 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f41d40099b0 tx=0x7f41d4009250 comp rx=0 tx=0).stop 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41e6dff700 1 -- 172.21.15.29:0/308833909 shutdown_connections 2023-10-07T10:45:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41e6dff700 1 --2- 172.21.15.29:0/308833909 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e00ae8a0 0x7f41e00b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41e6dff700 1 --2- 172.21.15.29:0/308833909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e015a960 0x7f41e0158e20 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.380+0000 7f41e6dff700 1 -- 172.21.15.29:0/308833909 >> 172.21.15.29:0/308833909 conn(0x7f41e0067610 msgr2=0x7f41e0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.381+0000 7f41e6dff700 1 -- 172.21.15.29:0/308833909 shutdown_connections 2023-10-07T10:45:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.381+0000 7f41e6dff700 1 -- 172.21.15.29:0/308833909 wait complete. 2023-10-07T10:45:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.383+0000 7f41e6dff700 1 Processor -- start 2023-10-07T10:45:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.383+0000 7f41e6dff700 1 -- start start 2023-10-07T10:45:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e6dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 0x7f41e0168310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e6dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41e015a960 0x7f41e0151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e6dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e015ad30 0x7f41e014eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e6dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e0167e80 con 0x7f41e00ae8a0 2023-10-07T10:45:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e6dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00ab310 con 0x7f41e015a960 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e6dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00ab5f0 con 0x7f41e015ad30 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e4b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 0x7f41e0168310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e4b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 0x7f41e0168310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50074/0 (socket says 172.21.15.29:50074) 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.384+0000 7f41e4b9b700 1 -- 172.21.15.29:0/3728508488 learned_addr learned my addr 172.21.15.29:0/3728508488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e4b9b700 1 -- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e015ad30 msgr2=0x7f41e014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e539c700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e015ad30 0x7f41e014eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41dffff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41e015a960 0x7f41e0151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e4b9b700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e015ad30 0x7f41e014eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e4b9b700 1 -- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41e015a960 msgr2=0x7f41e0151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e4b9b700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41e015a960 0x7f41e0151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e4b9b700 1 -- 172.21.15.29:0/3728508488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40094f0 con 0x7f41e00ae8a0 2023-10-07T10:45:21.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41dffff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41e015a960 0x7f41e0151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e539c700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e015ad30 0x7f41e014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e4b9b700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 0x7f41e0168310 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f41e0064150 tx=0x7f41d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d400bbe0 con 0x7f41e00ae8a0 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4011420 con 0x7f41e00ae8a0 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40117c0 con 0x7f41e00ae8a0 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.385+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e0167b10 con 0x7f41e00ae8a0 2023-10-07T10:45:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.386+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e0167820 con 0x7f41e00ae8a0 2023-10-07T10:45:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.387+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e004c3c0 con 0x7f41e00ae8a0 2023-10-07T10:45:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.387+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f41d4011bf0 con 0x7f41e00ae8a0 2023-10-07T10:45:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.388+0000 7f41dcff9700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f41c4066f30 0x7f41c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.388+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f41d4085d10 con 0x7f41e00ae8a0 2023-10-07T10:45:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.389+0000 7f41dffff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f41c4066f30 0x7f41c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:21.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.389+0000 7f41dffff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f41c4066f30 0x7f41c40693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41e01582e0 tx=0x7f41cc009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.392+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41d408b020 con 0x7f41e00ae8a0 2023-10-07T10:45:21.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:21.641+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f41e00ac4f0 con 0x7f41e00ae8a0 2023-10-07T10:45:22.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.151+0000 7f41dcff9700 1 -- 172.21.15.29:0/3728508488 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v85) v1 ==== 122+0+0 (secure 0 0 0) 0x7f41d4052dc0 con 0x7f41e00ae8a0 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f41c4066f30 msgr2=0x7f41c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f41c4066f30 0x7f41c40693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41e01582e0 tx=0x7f41cc009040 comp rx=0 tx=0).stop 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 msgr2=0x7f41e0168310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 0x7f41e0168310 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f41e0064150 tx=0x7f41d4017000 comp rx=0 tx=0).stop 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 shutdown_connections 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f41c4066f30 0x7f41c40693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41e015ad30 0x7f41e014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e00ae8a0 0x7f41e0168310 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 --2- 172.21.15.29:0/3728508488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41e015a960 0x7f41e0151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 >> 172.21.15.29:0/3728508488 conn(0x7f41e0067610 msgr2=0x7f41e0159820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 shutdown_connections 2023-10-07T10:45:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.155+0000 7f41e6dff700 1 -- 172.21.15.29:0/3728508488 wait complete. 2023-10-07T10:45:22.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:45:22.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.376+0000 7fd2e4dff700 1 Processor -- start 2023-10-07T10:45:22.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.376+0000 7fd2e4dff700 1 -- start start 2023-10-07T10:45:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.377+0000 7fd2e4dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2e0159750 0x7fd2e00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.377+0000 7fd2e4dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd2e00ce8a0 0x7fd2e00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.377+0000 7fd2e4dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00cbf20 con 0x7fd2e0159b20 2023-10-07T10:45:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.377+0000 7fd2e4dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e00cc200 con 0x7fd2e0159750 2023-10-07T10:45:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.377+0000 7fd2e4dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e00cc4e0 con 0x7fd2e00ce8a0 2023-10-07T10:45:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2e0159750 0x7fd2e00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2e0159750 0x7fd2e00d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50086/0 (socket says 172.21.15.29:50086) 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2ddd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd2e0159b20 0x7fd2e00ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47918/0 (socket says 172.21.15.29:47918) 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 -- 172.21.15.29:0/1511888249 learned_addr learned my addr 172.21.15.29:0/1511888249 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2ded9d700 1 --2- 172.21.15.29:0/1511888249 >> v2:172.21.15.144:3300/0 conn(0x7fd2e00ce8a0 0x7fd2e00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2dd59a700 1 -- 172.21.15.29:0/1511888249 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344090426 0 0) 0x7fd2e00cbf20 con 0x7fd2e0159b20 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 -- 172.21.15.29:0/1511888249 >> v2:172.21.15.144:3300/0 conn(0x7fd2e00ce8a0 msgr2=0x7fd2e00c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 --2- 172.21.15.29:0/1511888249 >> v2:172.21.15.144:3300/0 conn(0x7fd2e00ce8a0 0x7fd2e00c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 -- 172.21.15.29:0/1511888249 >> v1:172.21.15.29:6789/0 conn(0x7fd2e0159b20 legacy=0x7fd2e00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 -- 172.21.15.29:0/1511888249 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00cc7c0 con 0x7fd2e0159750 2023-10-07T10:45:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.378+0000 7fd2de59c700 1 --2- 172.21.15.29:0/1511888249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159750 0x7fd2e00d10a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fd2e015e510 tx=0x7fd2d0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d49e665e85a91a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2dd59a700 1 -- 172.21.15.29:0/1511888249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0005c10 con 0x7fd2e0159750 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2dd59a700 1 -- 172.21.15.29:0/1511888249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0005d70 con 0x7fd2e0159750 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2dd59a700 1 -- 172.21.15.29:0/1511888249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000b880 con 0x7fd2e0159750 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1511888249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159750 msgr2=0x7fd2e00d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1511888249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159750 0x7fd2e00d10a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fd2e015e510 tx=0x7fd2d0009220 comp rx=0 tx=0).stop 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1511888249 shutdown_connections 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1511888249 >> v2:172.21.15.144:3300/0 conn(0x7fd2e00ce8a0 0x7fd2e00c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1511888249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159750 0x7fd2e00d10a0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1511888249 >> 172.21.15.29:0/1511888249 conn(0x7fd2e0063a10 msgr2=0x7fd2e00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1511888249 shutdown_connections 2023-10-07T10:45:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.379+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1511888249 wait complete. 2023-10-07T10:45:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.381+0000 7fd2e4dff700 1 Processor -- start 2023-10-07T10:45:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.381+0000 7fd2e4dff700 1 -- start start 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2e4dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2e00ce8a0 0x7fd2e00b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2e4dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2e0159750 0x7fd2e016ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2e4dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 0x7fd2e00c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2e4dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00b4f80 con 0x7fd2e0159b20 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2e4dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00c8820 con 0x7fd2e0159750 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2e4dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e016b190 con 0x7fd2e00ce8a0 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2ded9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 0x7fd2e00c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2ded9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 0x7fd2e00c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50102/0 (socket says 172.21.15.29:50102) 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2ded9d700 1 -- 172.21.15.29:0/1568743281 learned_addr learned my addr 172.21.15.29:0/1568743281 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2ddd9b700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2e0159750 0x7fd2e016ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2de59c700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2e00ce8a0 0x7fd2e00b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.382+0000 7fd2ded9d700 1 -- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2e00ce8a0 msgr2=0x7fd2e00b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ded9d700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2e00ce8a0 0x7fd2e00b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ded9d700 1 -- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2e0159750 msgr2=0x7fd2e016ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ded9d700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2e0159750 0x7fd2e016ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ded9d700 1 -- 172.21.15.29:0/1568743281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0009480 con 0x7fd2e0159b20 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ded9d700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 0x7fd2e00c9490 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009fa0 tx=0x7fd2d8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d800ed20 con 0x7fd2e0159b20 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d800ee80 con 0x7fd2e0159b20 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e00c8cb0 con 0x7fd2e0159b20 2023-10-07T10:45:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8014670 con 0x7fd2e0159b20 2023-10-07T10:45:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.383+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e00c7ef0 con 0x7fd2e0159b20 2023-10-07T10:45:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.384+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e004c3c0 con 0x7fd2e0159b20 2023-10-07T10:45:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.385+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd2d80149e0 con 0x7fd2e0159b20 2023-10-07T10:45:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.387+0000 7fd2ce7fc700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2c0066f30 0x7fd2c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.387+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fd2d8084f10 con 0x7fd2e0159b20 2023-10-07T10:45:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.387+0000 7fd2de59c700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2c0066f30 0x7fd2c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.388+0000 7fd2de59c700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2c0066f30 0x7fd2c00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd2e006a220 tx=0x7fd2d0016000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:22.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.391+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2d8051fe0 con 0x7fd2e0159b20 2023-10-07T10:45:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.639+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e00c8960 con 0x7fd2e0159b20 2023-10-07T10:45:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.639+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/1568743281 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fd2d8055600 con 0x7fd2e0159b20 2023-10-07T10:45:22.640 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:22.640 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:45:18.619852+0000","modified":"2023-10-07T10:45:22.148839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9090":{"gid":9090,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/4080992371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4080992371},{"type":"v1","addr":"172.21.15.29:6833","nonce":4080992371}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2c0066f30 msgr2=0x7fd2c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2c0066f30 0x7fd2c00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd2e006a220 tx=0x7fd2d0016000 comp rx=0 tx=0).stop 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 msgr2=0x7fd2e00c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 0x7fd2e00c9490 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009fa0 tx=0x7fd2d8009900 comp rx=0 tx=0).stop 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 shutdown_connections 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd2c0066f30 0x7fd2c00693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2e0159b20 0x7fd2e00c9490 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2e00ce8a0 0x7fd2e00b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 --2- 172.21.15.29:0/1568743281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2e0159750 0x7fd2e016ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.641+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 >> 172.21.15.29:0/1568743281 conn(0x7fd2e0063a10 msgr2=0x7fd2e00c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.642+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 shutdown_connections 2023-10-07T10:45:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.642+0000 7fd2e4dff700 1 -- 172.21.15.29:0/1568743281 wait complete. 2023-10-07T10:45:22.646 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-07T10:45:22.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T10:45:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.854+0000 7fcc595ff700 1 Processor -- start 2023-10-07T10:45:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.854+0000 7fcc595ff700 1 -- start start 2023-10-07T10:45:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.855+0000 7fcc595ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc54159ec0 0x7fcc5415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc595ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5414d790 con 0x7fcc540af400 2023-10-07T10:45:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc595ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc540ae5d0 con 0x7fcc540ad370 2023-10-07T10:45:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc595ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5415a850 con 0x7fcc54159ec0 2023-10-07T10:45:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc537fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcc540af400 0x7fcc540af810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47930/0 (socket says 172.21.15.29:47930) 2023-10-07T10:45:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc537fe700 1 -- 172.21.15.29:0/1321545763 learned_addr learned my addr 172.21.15.29:0/1321545763 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc527fc700 1 --2- 172.21.15.29:0/1321545763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc54159ec0 0x7fcc5415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216439143 0 0) 0x7fcc5414d790 con 0x7fcc540af400 2023-10-07T10:45:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c003570 con 0x7fcc540af400 2023-10-07T10:45:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882352166 0 0) 0x7fcc540ae5d0 con 0x7fcc540ad370 2023-10-07T10:45:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.856+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc5414d790 con 0x7fcc540ad370 2023-10-07T10:45:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc527fc700 1 -- 172.21.15.29:0/1321545763 >> v1:172.21.15.144:6789/0 conn(0x7fcc540ad370 legacy=0x7fcc540cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc527fc700 1 -- 172.21.15.29:0/1321545763 >> v1:172.21.15.29:6789/0 conn(0x7fcc540af400 legacy=0x7fcc540af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc527fc700 1 -- 172.21.15.29:0/1321545763 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc540ad740 con 0x7fcc54159ec0 2023-10-07T10:45:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc527fc700 1 --2- 172.21.15.29:0/1321545763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc54159ec0 0x7fcc5415a310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009910 tx=0x7fcc480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b71334d9bfb856a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48011630 con 0x7fcc54159ec0 2023-10-07T10:45:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc48011790 con 0x7fcc54159ec0 2023-10-07T10:45:22.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.857+0000 7fcc51ffb700 1 -- 172.21.15.29:0/1321545763 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48011b30 con 0x7fcc54159ec0 2023-10-07T10:45:22.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 -- 172.21.15.29:0/1321545763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc54159ec0 msgr2=0x7fcc5415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 --2- 172.21.15.29:0/1321545763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc54159ec0 0x7fcc5415a310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009910 tx=0x7fcc480091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 -- 172.21.15.29:0/1321545763 shutdown_connections 2023-10-07T10:45:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 --2- 172.21.15.29:0/1321545763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc54159ec0 0x7fcc5415a310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 -- 172.21.15.29:0/1321545763 >> 172.21.15.29:0/1321545763 conn(0x7fcc540680a0 msgr2=0x7fcc540cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 -- 172.21.15.29:0/1321545763 shutdown_connections 2023-10-07T10:45:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.858+0000 7fcc595ff700 1 -- 172.21.15.29:0/1321545763 wait complete. 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.859+0000 7fcc595ff700 1 Processor -- start 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 -- start start 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc540ad370 0x7fcc541693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc540af400 0x7fcc5414c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 0x7fcc5414e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5414cb40 con 0x7fcc54159ec0 2023-10-07T10:45:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5416a3e0 con 0x7fcc540af400 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.860+0000 7fcc595ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5414fd50 con 0x7fcc540ad370 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 0x7fcc5414e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 0x7fcc5414e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50118/0 (socket says 172.21.15.29:50118) 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 -- 172.21.15.29:0/4183185503 learned_addr learned my addr 172.21.15.29:0/4183185503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc52ffd700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc540ad370 0x7fcc541693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc527fc700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc540af400 0x7fcc5414c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 -- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc540ad370 msgr2=0x7fcc541693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc540ad370 0x7fcc541693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 -- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc540af400 msgr2=0x7fcc5414c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc540af400 0x7fcc5414c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 -- 172.21.15.29:0/4183185503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48009420 con 0x7fcc54159ec0 2023-10-07T10:45:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc52ffd700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc540ad370 0x7fcc541693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc527fc700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc540af400 0x7fcc5414c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.861+0000 7fcc537fe700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 0x7fcc5414e8f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00cc60 tx=0x7fcc4c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.862+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00ce70 con 0x7fcc54159ec0 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.862+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc541501e0 con 0x7fcc54159ec0 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.862+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c00ab10 con 0x7fcc54159ec0 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.862+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0070f0 con 0x7fcc54159ec0 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.862+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5414f420 con 0x7fcc54159ec0 2023-10-07T10:45:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.863+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5404c3c0 con 0x7fcc54159ec0 2023-10-07T10:45:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.864+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc4c0074a0 con 0x7fcc54159ec0 2023-10-07T10:45:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.864+0000 7fcc3affd700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc40066f30 0x7fcc400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.865+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fcc4c084c90 con 0x7fcc54159ec0 2023-10-07T10:45:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.865+0000 7fcc52ffd700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc40066f30 0x7fcc400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.866+0000 7fcc52ffd700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc40066f30 0x7fcc400693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc540647e0 tx=0x7fcc44009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:22.869+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc4c051cc0 con 0x7fcc54159ec0 2023-10-07T10:45:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.071+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc540ab660 con 0x7fcc54159ec0 2023-10-07T10:45:23.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.071+0000 7fcc3affd700 1 -- 172.21.15.29:0/4183185503 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+16102 (secure 0 0 0) 0x7fcc4c0552e0 con 0x7fcc54159ec0 2023-10-07T10:45:23.072 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:23.073 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":165,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:45:20.117377+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:45:16.377745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-07T10:45:17.609874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T10:45:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc40066f30 msgr2=0x7fcc400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc40066f30 0x7fcc400693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc540647e0 tx=0x7fcc44009040 comp rx=0 tx=0).stop 2023-10-07T10:45:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 msgr2=0x7fcc5414e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 0x7fcc5414e8f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00cc60 tx=0x7fcc4c001ae0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 shutdown_connections 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fcc40066f30 0x7fcc400693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc54159ec0 0x7fcc5414e8f0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc540ad370 0x7fcc541693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 --2- 172.21.15.29:0/4183185503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc540af400 0x7fcc5414c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 >> 172.21.15.29:0/4183185503 conn(0x7fcc540680a0 msgr2=0x7fcc540aa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 shutdown_connections 2023-10-07T10:45:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.074+0000 7fcc595ff700 1 -- 172.21.15.29:0/4183185503 wait complete. 2023-10-07T10:45:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T10:45:23.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T10:45:23.094 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-07T10:45:23.093+0000 7f6af47f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T10:45:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.290+0000 7f3cbb9ff700 1 Processor -- start 2023-10-07T10:45:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.290+0000 7f3cbb9ff700 1 -- start start 2023-10-07T10:45:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.291+0000 7f3cbb9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3cb41533a0 0x7f3cb40c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.291+0000 7f3cbb9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb4069ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.291+0000 7f3cbb9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb406db30 con 0x7f3cb4156040 2023-10-07T10:45:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.291+0000 7f3cbb9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb40c8750 con 0x7f3cb41533a0 2023-10-07T10:45:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.291+0000 7f3cbb9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40c7160 con 0x7f3cb4153770 2023-10-07T10:45:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3cb41533a0 0x7f3cb40c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb8f9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb4069ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3cb41533a0 0x7f3cb40c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:55480/0 (socket says 172.21.15.29:55480) 2023-10-07T10:45:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 -- 172.21.15.29:0/1561890660 learned_addr learned my addr 172.21.15.29:0/1561890660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb8f9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb4069ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:45992/0 (socket says 172.21.15.29:45992) 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 -- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 msgr2=0x7f3cb4069ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 --2- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb4069ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 -- 172.21.15.29:0/1561890660 >> v1:172.21.15.144:6789/0 conn(0x7f3cb4156040 legacy=0x7f3cb4159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb979b700 1 -- 172.21.15.29:0/1561890660 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb40c7440 con 0x7f3cb41533a0 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.292+0000 7f3cb8f9a700 1 --2- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb4069ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.293+0000 7f3cb979b700 1 --2- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40c6af0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000ab50 tx=0x7f3cb000a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cdd5c38fe302f83 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.293+0000 7f3cabfff700 1 -- 172.21.15.29:0/1561890660 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb000ca40 con 0x7f3cb41533a0 2023-10-07T10:45:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.293+0000 7f3cabfff700 1 -- 172.21.15.29:0/1561890660 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb000cba0 con 0x7f3cb41533a0 2023-10-07T10:45:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.293+0000 7f3cabfff700 1 -- 172.21.15.29:0/1561890660 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0018400 con 0x7f3cb41533a0 2023-10-07T10:45:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.293+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 msgr2=0x7f3cb40c6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.293+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40c6af0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000ab50 tx=0x7f3cb000a3f0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.294+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1561890660 shutdown_connections 2023-10-07T10:45:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.294+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40c6af0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.294+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1561890660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb4069ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.294+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1561890660 >> 172.21.15.29:0/1561890660 conn(0x7f3cb40cc0e0 msgr2=0x7f3cb40cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.294+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1561890660 shutdown_connections 2023-10-07T10:45:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.294+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1561890660 wait complete. 2023-10-07T10:45:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.295+0000 7f3cbb9ff700 1 Processor -- start 2023-10-07T10:45:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.295+0000 7f3cbb9ff700 1 -- start start 2023-10-07T10:45:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.296+0000 7f3cbb9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.296+0000 7f3cbb9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb415c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.296+0000 7f3cbb9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 0x7f3cb4152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.296+0000 7f3cbb9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40b14a0 con 0x7f3cb4156040 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.296+0000 7f3cbb9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb415ce30 con 0x7f3cb41533a0 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.296+0000 7f3cbb9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb41525d0 con 0x7f3cb4153770 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 0x7f3cb4152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 0x7f3cb4152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56852/0 (socket says 172.21.15.29:56852) 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 -- 172.21.15.29:0/1202255645 learned_addr learned my addr 172.21.15.29:0/1202255645 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb979b700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb8f9a700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb415c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 -- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 msgr2=0x7f3cb415c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb415c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 -- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 msgr2=0x7f3cb40b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 -- 172.21.15.29:0/1202255645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca4009420 con 0x7f3cb4156040 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.297+0000 7f3cb9f9c700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 0x7f3cb4152090 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00c870 tx=0x7f3cac001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.298+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac012630 con 0x7f3cb4156040 2023-10-07T10:45:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.298+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac0127b0 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.298+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb000a690 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.298+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac012b70 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.298+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb40b0b70 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.299+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb404c3c0 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.300+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3cac003030 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.301+0000 7f3ca97fa700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c98066f30 0x7f3c980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.301+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f3cac0846e0 con 0x7f3cb4156040 2023-10-07T10:45:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.301+0000 7f3cb979b700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c98066f30 0x7f3c980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.302+0000 7f3cb979b700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c98066f30 0x7f3c980693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000ab50 tx=0x7f3cb001a000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:23.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.305+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3cac051710 con 0x7f3cb4156040 2023-10-07T10:45:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.502+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cb40b15e0 con 0x7f3c98066f30 2023-10-07T10:45:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.513+0000 7f3ca97fa700 1 -- 172.21.15.29:0/1202255645 <== mgr.8449 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289364 (secure 0 0 0) 0x7f3cb40b15e0 con 0x7f3c98066f30 2023-10-07T10:45:23.517 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c98066f30 msgr2=0x7f3c980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c98066f30 0x7f3c980693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000ab50 tx=0x7f3cb001a000 comp rx=0 tx=0).stop 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 msgr2=0x7f3cb4152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 0x7f3cb4152090 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00c870 tx=0x7f3cac001670 comp rx=0 tx=0).stop 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 shutdown_connections 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f3c98066f30 0x7f3c980693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cb4156040 0x7f3cb4152090 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cb41533a0 0x7f3cb40b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 --2- 172.21.15.29:0/1202255645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3cb4153770 0x7f3cb415c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 >> 172.21.15.29:0/1202255645 conn(0x7f3cb40cc0e0 msgr2=0x7f3cb4152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 shutdown_connections 2023-10-07T10:45:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.517+0000 7f3cbb9ff700 1 -- 172.21.15.29:0/1202255645 wait complete. 2023-10-07T10:45:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T10:45:23.535 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-07T10:45:23.185943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2469252,"kb_used_data":17220,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752505468,"statfs":{"total":773094113280,"available":770565599232,"internally_reserved":0,"allocated":17633280,"data_stored":9217376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001553"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.159607+0000","last_change":"2023-10-07T10:45:19.148845+0000","last_active":"2023-10-07T10:45:19.159607+0000","last_peered":"2023-10-07T10:45:19.159607+0000","last_clean":"2023-10-07T10:45:19.159607+0000","last_became_active":"2023-10-07T10:45:19.148464+0000","last_became_peered":"2023-10-07T10:45:19.148464+0000","last_unstale":"2023-10-07T10:45:19.159607+0000","last_undegraded":"2023-10-07T10:45:19.159607+0000","last_fullsized":"2023-10-07T10:45:19.159607+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:16:38.783846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145718+0000","last_change":"2023-10-07T10:45:19.144734+0000","last_active":"2023-10-07T10:45:19.145718+0000","last_peered":"2023-10-07T10:45:19.145718+0000","last_clean":"2023-10-07T10:45:19.145718+0000","last_became_active":"2023-10-07T10:45:19.144379+0000","last_became_peered":"2023-10-07T10:45:19.144379+0000","last_unstale":"2023-10-07T10:45:19.145718+0000","last_undegraded":"2023-10-07T10:45:19.145718+0000","last_fullsized":"2023-10-07T10:45:19.145718+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:47.397055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148213+0000","last_change":"2023-10-07T10:45:19.136552+0000","last_active":"2023-10-07T10:45:19.148213+0000","last_peered":"2023-10-07T10:45:19.148213+0000","last_clean":"2023-10-07T10:45:19.148213+0000","last_became_active":"2023-10-07T10:45:19.136264+0000","last_became_peered":"2023-10-07T10:45:19.136264+0000","last_unstale":"2023-10-07T10:45:19.148213+0000","last_undegraded":"2023-10-07T10:45:19.148213+0000","last_fullsized":"2023-10-07T10:45:19.148213+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:52.338610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146861+0000","last_change":"2023-10-07T10:45:19.140026+0000","last_active":"2023-10-07T10:45:19.146861+0000","last_peered":"2023-10-07T10:45:19.146861+0000","last_clean":"2023-10-07T10:45:19.146861+0000","last_became_active":"2023-10-07T10:45:19.139544+0000","last_became_peered":"2023-10-07T10:45:19.139544+0000","last_unstale":"2023-10-07T10:45:19.146861+0000","last_undegraded":"2023-10-07T10:45:19.146861+0000","last_fullsized":"2023-10-07T10:45:19.146861+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:53.575149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.137852+0000","last_change":"2023-10-07T10:45:19.133768+0000","last_active":"2023-10-07T10:45:19.137852+0000","last_peered":"2023-10-07T10:45:19.137852+0000","last_clean":"2023-10-07T10:45:19.137852+0000","last_became_active":"2023-10-07T10:45:19.133451+0000","last_became_peered":"2023-10-07T10:45:19.133451+0000","last_unstale":"2023-10-07T10:45:19.137852+0000","last_undegraded":"2023-10-07T10:45:19.137852+0000","last_fullsized":"2023-10-07T10:45:19.137852+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:39.283636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.152120+0000","last_change":"2023-10-07T10:45:19.142721+0000","last_active":"2023-10-07T10:45:19.152120+0000","last_peered":"2023-10-07T10:45:19.152120+0000","last_clean":"2023-10-07T10:45:19.152120+0000","last_became_active":"2023-10-07T10:45:19.142323+0000","last_became_peered":"2023-10-07T10:45:19.142323+0000","last_unstale":"2023-10-07T10:45:19.152120+0000","last_undegraded":"2023-10-07T10:45:19.152120+0000","last_fullsized":"2023-10-07T10:45:19.152120+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:06:49.678247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143282+0000","last_change":"2023-10-07T10:45:19.136492+0000","last_active":"2023-10-07T10:45:19.143282+0000","last_peered":"2023-10-07T10:45:19.143282+0000","last_clean":"2023-10-07T10:45:19.143282+0000","last_became_active":"2023-10-07T10:45:19.136169+0000","last_became_peered":"2023-10-07T10:45:19.136169+0000","last_unstale":"2023-10-07T10:45:19.143282+0000","last_undegraded":"2023-10-07T10:45:19.143282+0000","last_fullsized":"2023-10-07T10:45:19.143282+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:27.595946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.133399+0000","last_change":"2023-10-07T10:45:19.128733+0000","last_active":"2023-10-07T10:45:19.133399+0000","last_peered":"2023-10-07T10:45:19.133399+0000","last_clean":"2023-10-07T10:45:19.133399+0000","last_became_active":"2023-10-07T10:45:19.128449+0000","last_became_peered":"2023-10-07T10:45:19.128449+0000","last_unstale":"2023-10-07T10:45:19.133399+0000","last_undegraded":"2023-10-07T10:45:19.133399+0000","last_fullsized":"2023-10-07T10:45:19.133399+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:51:31.538804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141861+0000","last_change":"2023-10-07T10:45:19.139258+0000","last_active":"2023-10-07T10:45:19.141861+0000","last_peered":"2023-10-07T10:45:19.141861+0000","last_clean":"2023-10-07T10:45:19.141861+0000","last_became_active":"2023-10-07T10:45:19.138951+0000","last_became_peered":"2023-10-07T10:45:19.138951+0000","last_unstale":"2023-10-07T10:45:19.141861+0000","last_undegraded":"2023-10-07T10:45:19.141861+0000","last_fullsized":"2023-10-07T10:45:19.141861+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:24.999401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151527+0000","last_change":"2023-10-07T10:45:19.149127+0000","last_active":"2023-10-07T10:45:19.151527+0000","last_peered":"2023-10-07T10:45:19.151527+0000","last_clean":"2023-10-07T10:45:19.151527+0000","last_became_active":"2023-10-07T10:45:19.148854+0000","last_became_peered":"2023-10-07T10:45:19.148854+0000","last_unstale":"2023-10-07T10:45:19.151527+0000","last_undegraded":"2023-10-07T10:45:19.151527+0000","last_fullsized":"2023-10-07T10:45:19.151527+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:29.555531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.133370+0000","last_change":"2023-10-07T10:45:19.127275+0000","last_active":"2023-10-07T10:45:19.133370+0000","last_peered":"2023-10-07T10:45:19.133370+0000","last_clean":"2023-10-07T10:45:19.133370+0000","last_became_active":"2023-10-07T10:45:19.126996+0000","last_became_peered":"2023-10-07T10:45:19.126996+0000","last_unstale":"2023-10-07T10:45:19.133370+0000","last_undegraded":"2023-10-07T10:45:19.133370+0000","last_fullsized":"2023-10-07T10:45:19.133370+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:15:37.883323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145139+0000","last_change":"2023-10-07T10:45:19.139208+0000","last_active":"2023-10-07T10:45:19.145139+0000","last_peered":"2023-10-07T10:45:19.145139+0000","last_clean":"2023-10-07T10:45:19.145139+0000","last_became_active":"2023-10-07T10:45:19.138939+0000","last_became_peered":"2023-10-07T10:45:19.138939+0000","last_unstale":"2023-10-07T10:45:19.145139+0000","last_undegraded":"2023-10-07T10:45:19.145139+0000","last_fullsized":"2023-10-07T10:45:19.145139+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:19:23.963788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.137832+0000","last_change":"2023-10-07T10:45:19.132566+0000","last_active":"2023-10-07T10:45:19.137832+0000","last_peered":"2023-10-07T10:45:19.137832+0000","last_clean":"2023-10-07T10:45:19.137832+0000","last_became_active":"2023-10-07T10:45:19.132279+0000","last_became_peered":"2023-10-07T10:45:19.132279+0000","last_unstale":"2023-10-07T10:45:19.137832+0000","last_undegraded":"2023-10-07T10:45:19.137832+0000","last_fullsized":"2023-10-07T10:45:19.137832+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:37.783382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146914+0000","last_change":"2023-10-07T10:45:19.134365+0000","last_active":"2023-10-07T10:45:19.146914+0000","last_peered":"2023-10-07T10:45:19.146914+0000","last_clean":"2023-10-07T10:45:19.146914+0000","last_became_active":"2023-10-07T10:45:19.133994+0000","last_became_peered":"2023-10-07T10:45:19.133994+0000","last_unstale":"2023-10-07T10:45:19.146914+0000","last_undegraded":"2023-10-07T10:45:19.146914+0000","last_fullsized":"2023-10-07T10:45:19.146914+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:55.959931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146705+0000","last_change":"2023-10-07T10:45:19.143816+0000","last_active":"2023-10-07T10:45:19.146705+0000","last_peered":"2023-10-07T10:45:19.146705+0000","last_clean":"2023-10-07T10:45:19.146705+0000","last_became_active":"2023-10-07T10:45:19.143508+0000","last_became_peered":"2023-10-07T10:45:19.143508+0000","last_unstale":"2023-10-07T10:45:19.146705+0000","last_undegraded":"2023-10-07T10:45:19.146705+0000","last_fullsized":"2023-10-07T10:45:19.146705+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:42:47.896740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.156216+0000","last_change":"2023-10-07T10:45:19.150588+0000","last_active":"2023-10-07T10:45:19.156216+0000","last_peered":"2023-10-07T10:45:19.156216+0000","last_clean":"2023-10-07T10:45:19.156216+0000","last_became_active":"2023-10-07T10:45:19.150289+0000","last_became_peered":"2023-10-07T10:45:19.150289+0000","last_unstale":"2023-10-07T10:45:19.156216+0000","last_undegraded":"2023-10-07T10:45:19.156216+0000","last_fullsized":"2023-10-07T10:45:19.156216+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:16:15.346473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147243+0000","last_change":"2023-10-07T10:45:19.146040+0000","last_active":"2023-10-07T10:45:19.147243+0000","last_peered":"2023-10-07T10:45:19.147243+0000","last_clean":"2023-10-07T10:45:19.147243+0000","last_became_active":"2023-10-07T10:45:19.145533+0000","last_became_peered":"2023-10-07T10:45:19.145533+0000","last_unstale":"2023-10-07T10:45:19.147243+0000","last_undegraded":"2023-10-07T10:45:19.147243+0000","last_fullsized":"2023-10-07T10:45:19.147243+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:02.464065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148359+0000","last_change":"2023-10-07T10:45:19.136919+0000","last_active":"2023-10-07T10:45:19.148359+0000","last_peered":"2023-10-07T10:45:19.148359+0000","last_clean":"2023-10-07T10:45:19.148359+0000","last_became_active":"2023-10-07T10:45:19.136630+0000","last_became_peered":"2023-10-07T10:45:19.136630+0000","last_unstale":"2023-10-07T10:45:19.148359+0000","last_undegraded":"2023-10-07T10:45:19.148359+0000","last_fullsized":"2023-10-07T10:45:19.148359+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:55.959931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.159787+0000","last_change":"2023-10-07T10:45:19.158281+0000","last_active":"2023-10-07T10:45:19.159787+0000","last_peered":"2023-10-07T10:45:19.159787+0000","last_clean":"2023-10-07T10:45:19.159787+0000","last_became_active":"2023-10-07T10:45:19.157984+0000","last_became_peered":"2023-10-07T10:45:19.157984+0000","last_unstale":"2023-10-07T10:45:19.159787+0000","last_undegraded":"2023-10-07T10:45:19.159787+0000","last_fullsized":"2023-10-07T10:45:19.159787+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:29.034536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.154629+0000","last_change":"2023-10-07T10:45:19.153620+0000","last_active":"2023-10-07T10:45:19.154629+0000","last_peered":"2023-10-07T10:45:19.154629+0000","last_clean":"2023-10-07T10:45:19.154629+0000","last_became_active":"2023-10-07T10:45:19.153341+0000","last_became_peered":"2023-10-07T10:45:19.153341+0000","last_unstale":"2023-10-07T10:45:19.154629+0000","last_undegraded":"2023-10-07T10:45:19.154629+0000","last_fullsized":"2023-10-07T10:45:19.154629+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:39.283636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151333+0000","last_change":"2023-10-07T10:45:19.148594+0000","last_active":"2023-10-07T10:45:19.151333+0000","last_peered":"2023-10-07T10:45:19.151333+0000","last_clean":"2023-10-07T10:45:19.151333+0000","last_became_active":"2023-10-07T10:45:19.148249+0000","last_became_peered":"2023-10-07T10:45:19.148249+0000","last_unstale":"2023-10-07T10:45:19.151333+0000","last_undegraded":"2023-10-07T10:45:19.151333+0000","last_fullsized":"2023-10-07T10:45:19.151333+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:36.084756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146948+0000","last_change":"2023-10-07T10:45:19.135737+0000","last_active":"2023-10-07T10:45:19.146948+0000","last_peered":"2023-10-07T10:45:19.146948+0000","last_clean":"2023-10-07T10:45:19.146948+0000","last_became_active":"2023-10-07T10:45:19.135483+0000","last_became_peered":"2023-10-07T10:45:19.135483+0000","last_unstale":"2023-10-07T10:45:19.146948+0000","last_undegraded":"2023-10-07T10:45:19.146948+0000","last_fullsized":"2023-10-07T10:45:19.146948+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:36.818457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146758+0000","last_change":"2023-10-07T10:45:19.142707+0000","last_active":"2023-10-07T10:45:19.146758+0000","last_peered":"2023-10-07T10:45:19.146758+0000","last_clean":"2023-10-07T10:45:19.146758+0000","last_became_active":"2023-10-07T10:45:19.140090+0000","last_became_peered":"2023-10-07T10:45:19.140090+0000","last_unstale":"2023-10-07T10:45:19.146758+0000","last_undegraded":"2023-10-07T10:45:19.146758+0000","last_fullsized":"2023-10-07T10:45:19.146758+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:56.491619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151366+0000","last_change":"2023-10-07T10:45:19.149067+0000","last_active":"2023-10-07T10:45:19.151366+0000","last_peered":"2023-10-07T10:45:19.151366+0000","last_clean":"2023-10-07T10:45:19.151366+0000","last_became_active":"2023-10-07T10:45:19.148731+0000","last_became_peered":"2023-10-07T10:45:19.148731+0000","last_unstale":"2023-10-07T10:45:19.151366+0000","last_undegraded":"2023-10-07T10:45:19.151366+0000","last_fullsized":"2023-10-07T10:45:19.151366+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:21:28.091881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.158635+0000","last_change":"2023-10-07T10:45:19.156808+0000","last_active":"2023-10-07T10:45:19.158635+0000","last_peered":"2023-10-07T10:45:19.158635+0000","last_clean":"2023-10-07T10:45:19.158635+0000","last_became_active":"2023-10-07T10:45:19.156475+0000","last_became_peered":"2023-10-07T10:45:19.156475+0000","last_unstale":"2023-10-07T10:45:19.158635+0000","last_undegraded":"2023-10-07T10:45:19.158635+0000","last_fullsized":"2023-10-07T10:45:19.158635+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:43:27.479906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151989+0000","last_change":"2023-10-07T10:45:19.144447+0000","last_active":"2023-10-07T10:45:19.151989+0000","last_peered":"2023-10-07T10:45:19.151989+0000","last_clean":"2023-10-07T10:45:19.151989+0000","last_became_active":"2023-10-07T10:45:19.144183+0000","last_became_peered":"2023-10-07T10:45:19.144183+0000","last_unstale":"2023-10-07T10:45:19.151989+0000","last_undegraded":"2023-10-07T10:45:19.151989+0000","last_fullsized":"2023-10-07T10:45:19.151989+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:33:48.518165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.157915+0000","last_change":"2023-10-07T10:45:19.155544+0000","last_active":"2023-10-07T10:45:19.157915+0000","last_peered":"2023-10-07T10:45:19.157915+0000","last_clean":"2023-10-07T10:45:19.157915+0000","last_became_active":"2023-10-07T10:45:19.155229+0000","last_became_peered":"2023-10-07T10:45:19.155229+0000","last_unstale":"2023-10-07T10:45:19.157915+0000","last_undegraded":"2023-10-07T10:45:19.157915+0000","last_fullsized":"2023-10-07T10:45:19.157915+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:37.783382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143664+0000","last_change":"2023-10-07T10:45:19.137326+0000","last_active":"2023-10-07T10:45:19.143664+0000","last_peered":"2023-10-07T10:45:19.143664+0000","last_clean":"2023-10-07T10:45:19.143664+0000","last_became_active":"2023-10-07T10:45:19.137030+0000","last_became_peered":"2023-10-07T10:45:19.137030+0000","last_unstale":"2023-10-07T10:45:19.143664+0000","last_undegraded":"2023-10-07T10:45:19.143664+0000","last_fullsized":"2023-10-07T10:45:19.143664+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:46:18.232666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149361+0000","last_change":"2023-10-07T10:45:19.141562+0000","last_active":"2023-10-07T10:45:19.149361+0000","last_peered":"2023-10-07T10:45:19.149361+0000","last_clean":"2023-10-07T10:45:19.149361+0000","last_became_active":"2023-10-07T10:45:19.141147+0000","last_became_peered":"2023-10-07T10:45:19.141147+0000","last_unstale":"2023-10-07T10:45:19.149361+0000","last_undegraded":"2023-10-07T10:45:19.149361+0000","last_fullsized":"2023-10-07T10:45:19.149361+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:48:51.721239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147036+0000","last_change":"2023-10-07T10:45:19.144266+0000","last_active":"2023-10-07T10:45:19.147036+0000","last_peered":"2023-10-07T10:45:19.147036+0000","last_clean":"2023-10-07T10:45:19.147036+0000","last_became_active":"2023-10-07T10:45:19.143943+0000","last_became_peered":"2023-10-07T10:45:19.143943+0000","last_unstale":"2023-10-07T10:45:19.147036+0000","last_undegraded":"2023-10-07T10:45:19.147036+0000","last_fullsized":"2023-10-07T10:45:19.147036+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:10:46.792904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141885+0000","last_change":"2023-10-07T10:45:19.138807+0000","last_active":"2023-10-07T10:45:19.141885+0000","last_peered":"2023-10-07T10:45:19.141885+0000","last_clean":"2023-10-07T10:45:19.141885+0000","last_became_active":"2023-10-07T10:45:19.138480+0000","last_became_peered":"2023-10-07T10:45:19.138480+0000","last_unstale":"2023-10-07T10:45:19.141885+0000","last_undegraded":"2023-10-07T10:45:19.141885+0000","last_fullsized":"2023-10-07T10:45:19.141885+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:48:13.786036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.158160+0000","last_change":"2023-10-07T10:45:19.156072+0000","last_active":"2023-10-07T10:45:19.158160+0000","last_peered":"2023-10-07T10:45:19.158160+0000","last_clean":"2023-10-07T10:45:19.158160+0000","last_became_active":"2023-10-07T10:45:19.155793+0000","last_became_peered":"2023-10-07T10:45:19.155793+0000","last_unstale":"2023-10-07T10:45:19.158160+0000","last_undegraded":"2023-10-07T10:45:19.158160+0000","last_fullsized":"2023-10-07T10:45:19.158160+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:12:01.286114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146978+0000","last_change":"2023-10-07T10:45:19.140218+0000","last_active":"2023-10-07T10:45:19.146978+0000","last_peered":"2023-10-07T10:45:19.146978+0000","last_clean":"2023-10-07T10:45:19.146978+0000","last_became_active":"2023-10-07T10:45:19.139954+0000","last_became_peered":"2023-10-07T10:45:19.139954+0000","last_unstale":"2023-10-07T10:45:19.146978+0000","last_undegraded":"2023-10-07T10:45:19.146978+0000","last_fullsized":"2023-10-07T10:45:19.146978+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:46:12.190595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"164'4","reported_seq":14,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149563+0000","last_change":"2023-10-07T10:45:18.142989+0000","last_active":"2023-10-07T10:45:19.149563+0000","last_peered":"2023-10-07T10:45:19.149563+0000","last_clean":"2023-10-07T10:45:19.149563+0000","last_became_active":"2023-10-07T10:45:18.142720+0000","last_became_peered":"2023-10-07T10:45:18.142720+0000","last_unstale":"2023-10-07T10:45:19.149563+0000","last_undegraded":"2023-10-07T10:45:19.149563+0000","last_fullsized":"2023-10-07T10:45:19.149563+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:52.732129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141206+0000","last_change":"2023-10-07T10:45:19.137410+0000","last_active":"2023-10-07T10:45:19.141206+0000","last_peered":"2023-10-07T10:45:19.141206+0000","last_clean":"2023-10-07T10:45:19.141206+0000","last_became_active":"2023-10-07T10:45:19.137106+0000","last_became_peered":"2023-10-07T10:45:19.137106+0000","last_unstale":"2023-10-07T10:45:19.141206+0000","last_undegraded":"2023-10-07T10:45:19.141206+0000","last_fullsized":"2023-10-07T10:45:19.141206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:40:55.053332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149990+0000","last_change":"2023-10-07T10:45:18.140674+0000","last_active":"2023-10-07T10:45:19.149990+0000","last_peered":"2023-10-07T10:45:19.149990+0000","last_clean":"2023-10-07T10:45:19.149990+0000","last_became_active":"2023-10-07T10:45:18.140397+0000","last_became_peered":"2023-10-07T10:45:18.140397+0000","last_unstale":"2023-10-07T10:45:19.149990+0000","last_undegraded":"2023-10-07T10:45:19.149990+0000","last_fullsized":"2023-10-07T10:45:19.149990+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:28:26.440612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140513+0000","last_change":"2023-10-07T10:45:19.137635+0000","last_active":"2023-10-07T10:45:19.140513+0000","last_peered":"2023-10-07T10:45:19.140513+0000","last_clean":"2023-10-07T10:45:19.140513+0000","last_became_active":"2023-10-07T10:45:19.137350+0000","last_became_peered":"2023-10-07T10:45:19.137350+0000","last_unstale":"2023-10-07T10:45:19.140513+0000","last_undegraded":"2023-10-07T10:45:19.140513+0000","last_fullsized":"2023-10-07T10:45:19.140513+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:48.056310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.154496+0000","last_change":"2023-10-07T10:45:18.134101+0000","last_active":"2023-10-07T10:45:19.154496+0000","last_peered":"2023-10-07T10:45:19.154496+0000","last_clean":"2023-10-07T10:45:19.154496+0000","last_became_active":"2023-10-07T10:45:18.133848+0000","last_became_peered":"2023-10-07T10:45:18.133848+0000","last_unstale":"2023-10-07T10:45:19.154496+0000","last_undegraded":"2023-10-07T10:45:19.154496+0000","last_fullsized":"2023-10-07T10:45:19.154496+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:47.048150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148416+0000","last_change":"2023-10-07T10:45:19.147540+0000","last_active":"2023-10-07T10:45:19.148416+0000","last_peered":"2023-10-07T10:45:19.148416+0000","last_clean":"2023-10-07T10:45:19.148416+0000","last_became_active":"2023-10-07T10:45:19.145202+0000","last_became_peered":"2023-10-07T10:45:19.145202+0000","last_unstale":"2023-10-07T10:45:19.148416+0000","last_undegraded":"2023-10-07T10:45:19.148416+0000","last_fullsized":"2023-10-07T10:45:19.148416+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:20:40.363427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.155032+0000","last_change":"2023-10-07T10:45:18.135363+0000","last_active":"2023-10-07T10:45:19.155032+0000","last_peered":"2023-10-07T10:45:19.155032+0000","last_clean":"2023-10-07T10:45:19.155032+0000","last_became_active":"2023-10-07T10:45:18.135072+0000","last_became_peered":"2023-10-07T10:45:18.135072+0000","last_unstale":"2023-10-07T10:45:19.155032+0000","last_undegraded":"2023-10-07T10:45:19.155032+0000","last_fullsized":"2023-10-07T10:45:19.155032+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:10.550922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141840+0000","last_change":"2023-10-07T10:45:19.140106+0000","last_active":"2023-10-07T10:45:19.141840+0000","last_peered":"2023-10-07T10:45:19.141840+0000","last_clean":"2023-10-07T10:45:19.141840+0000","last_became_active":"2023-10-07T10:45:19.139850+0000","last_became_peered":"2023-10-07T10:45:19.139850+0000","last_unstale":"2023-10-07T10:45:19.141840+0000","last_undegraded":"2023-10-07T10:45:19.141840+0000","last_fullsized":"2023-10-07T10:45:19.141840+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:20.315521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.156817+0000","last_change":"2023-10-07T10:45:18.140719+0000","last_active":"2023-10-07T10:45:19.156817+0000","last_peered":"2023-10-07T10:45:19.156817+0000","last_clean":"2023-10-07T10:45:19.156817+0000","last_became_active":"2023-10-07T10:45:18.140454+0000","last_became_peered":"2023-10-07T10:45:18.140454+0000","last_unstale":"2023-10-07T10:45:19.156817+0000","last_undegraded":"2023-10-07T10:45:19.156817+0000","last_fullsized":"2023-10-07T10:45:19.156817+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:02:04.522990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147107+0000","last_change":"2023-10-07T10:45:19.140209+0000","last_active":"2023-10-07T10:45:19.147107+0000","last_peered":"2023-10-07T10:45:19.147107+0000","last_clean":"2023-10-07T10:45:19.147107+0000","last_became_active":"2023-10-07T10:45:19.139894+0000","last_became_peered":"2023-10-07T10:45:19.139894+0000","last_unstale":"2023-10-07T10:45:19.147107+0000","last_undegraded":"2023-10-07T10:45:19.147107+0000","last_fullsized":"2023-10-07T10:45:19.147107+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:15.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149366+0000","last_change":"2023-10-07T10:45:18.137992+0000","last_active":"2023-10-07T10:45:19.149366+0000","last_peered":"2023-10-07T10:45:19.149366+0000","last_clean":"2023-10-07T10:45:19.149366+0000","last_became_active":"2023-10-07T10:45:18.137585+0000","last_became_peered":"2023-10-07T10:45:18.137585+0000","last_unstale":"2023-10-07T10:45:19.149366+0000","last_undegraded":"2023-10-07T10:45:19.149366+0000","last_fullsized":"2023-10-07T10:45:19.149366+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:02:04.522990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145819+0000","last_change":"2023-10-07T10:45:19.144582+0000","last_active":"2023-10-07T10:45:19.145819+0000","last_peered":"2023-10-07T10:45:19.145819+0000","last_clean":"2023-10-07T10:45:19.145819+0000","last_became_active":"2023-10-07T10:45:19.144266+0000","last_became_peered":"2023-10-07T10:45:19.144266+0000","last_unstale":"2023-10-07T10:45:19.145819+0000","last_undegraded":"2023-10-07T10:45:19.145819+0000","last_fullsized":"2023-10-07T10:45:19.145819+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:23.485305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.133247+0000","last_change":"2023-10-07T10:45:18.132149+0000","last_active":"2023-10-07T10:45:19.133247+0000","last_peered":"2023-10-07T10:45:19.133247+0000","last_clean":"2023-10-07T10:45:19.133247+0000","last_became_active":"2023-10-07T10:45:18.131876+0000","last_became_peered":"2023-10-07T10:45:18.131876+0000","last_unstale":"2023-10-07T10:45:19.133247+0000","last_undegraded":"2023-10-07T10:45:19.133247+0000","last_fullsized":"2023-10-07T10:45:19.133247+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:17:34.792329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140845+0000","last_change":"2023-10-07T10:45:19.136552+0000","last_active":"2023-10-07T10:45:19.140845+0000","last_peered":"2023-10-07T10:45:19.140845+0000","last_clean":"2023-10-07T10:45:19.140845+0000","last_became_active":"2023-10-07T10:45:19.136290+0000","last_became_peered":"2023-10-07T10:45:19.136290+0000","last_unstale":"2023-10-07T10:45:19.140845+0000","last_undegraded":"2023-10-07T10:45:19.140845+0000","last_fullsized":"2023-10-07T10:45:19.140845+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:00:12.788473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.121462+0000","last_change":"2023-10-07T10:45:18.137921+0000","last_active":"2023-10-07T10:45:19.121462+0000","last_peered":"2023-10-07T10:45:19.121462+0000","last_clean":"2023-10-07T10:45:19.121462+0000","last_became_active":"2023-10-07T10:45:18.137630+0000","last_became_peered":"2023-10-07T10:45:18.137630+0000","last_unstale":"2023-10-07T10:45:19.121462+0000","last_undegraded":"2023-10-07T10:45:19.121462+0000","last_fullsized":"2023-10-07T10:45:19.121462+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:48:50.713079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149608+0000","last_change":"2023-10-07T10:45:19.146791+0000","last_active":"2023-10-07T10:45:19.149608+0000","last_peered":"2023-10-07T10:45:19.149608+0000","last_clean":"2023-10-07T10:45:19.149608+0000","last_became_active":"2023-10-07T10:45:19.146224+0000","last_became_peered":"2023-10-07T10:45:19.146224+0000","last_unstale":"2023-10-07T10:45:19.149608+0000","last_undegraded":"2023-10-07T10:45:19.149608+0000","last_fullsized":"2023-10-07T10:45:19.149608+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:10.279050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.127372+0000","last_change":"2023-10-07T10:45:18.135238+0000","last_active":"2023-10-07T10:45:19.127372+0000","last_peered":"2023-10-07T10:45:19.127372+0000","last_clean":"2023-10-07T10:45:19.127372+0000","last_became_active":"2023-10-07T10:45:18.134954+0000","last_became_peered":"2023-10-07T10:45:18.134954+0000","last_unstale":"2023-10-07T10:45:19.127372+0000","last_undegraded":"2023-10-07T10:45:19.127372+0000","last_fullsized":"2023-10-07T10:45:19.127372+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:08:17.302016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147262+0000","last_change":"2023-10-07T10:45:19.146210+0000","last_active":"2023-10-07T10:45:19.147262+0000","last_peered":"2023-10-07T10:45:19.147262+0000","last_clean":"2023-10-07T10:45:19.147262+0000","last_became_active":"2023-10-07T10:45:19.145803+0000","last_became_peered":"2023-10-07T10:45:19.145803+0000","last_unstale":"2023-10-07T10:45:19.147262+0000","last_undegraded":"2023-10-07T10:45:19.147262+0000","last_fullsized":"2023-10-07T10:45:19.147262+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:53:24.969381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.130723+0000","last_change":"2023-10-07T10:45:18.141044+0000","last_active":"2023-10-07T10:45:19.130723+0000","last_peered":"2023-10-07T10:45:19.130723+0000","last_clean":"2023-10-07T10:45:19.130723+0000","last_became_active":"2023-10-07T10:45:18.140162+0000","last_became_peered":"2023-10-07T10:45:18.140162+0000","last_unstale":"2023-10-07T10:45:19.130723+0000","last_undegraded":"2023-10-07T10:45:19.130723+0000","last_fullsized":"2023-10-07T10:45:19.130723+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:01:05.105032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.155412+0000","last_change":"2023-10-07T10:45:19.152242+0000","last_active":"2023-10-07T10:45:19.155412+0000","last_peered":"2023-10-07T10:45:19.155412+0000","last_clean":"2023-10-07T10:45:19.155412+0000","last_became_active":"2023-10-07T10:45:19.151911+0000","last_became_peered":"2023-10-07T10:45:19.151911+0000","last_unstale":"2023-10-07T10:45:19.155412+0000","last_undegraded":"2023-10-07T10:45:19.155412+0000","last_fullsized":"2023-10-07T10:45:19.155412+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:10:44.279307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"164'6","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.160824+0000","last_change":"2023-10-07T10:45:18.134720+0000","last_active":"2023-10-07T10:45:19.160824+0000","last_peered":"2023-10-07T10:45:19.160824+0000","last_clean":"2023-10-07T10:45:19.160824+0000","last_became_active":"2023-10-07T10:45:18.134472+0000","last_became_peered":"2023-10-07T10:45:18.134472+0000","last_unstale":"2023-10-07T10:45:19.160824+0000","last_undegraded":"2023-10-07T10:45:19.160824+0000","last_fullsized":"2023-10-07T10:45:19.160824+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:08:17.302016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148337+0000","last_change":"2023-10-07T10:45:19.137210+0000","last_active":"2023-10-07T10:45:19.148337+0000","last_peered":"2023-10-07T10:45:19.148337+0000","last_clean":"2023-10-07T10:45:19.148337+0000","last_became_active":"2023-10-07T10:45:19.136932+0000","last_became_peered":"2023-10-07T10:45:19.136932+0000","last_unstale":"2023-10-07T10:45:19.148337+0000","last_undegraded":"2023-10-07T10:45:19.148337+0000","last_fullsized":"2023-10-07T10:45:19.148337+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:36.719844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.153415+0000","last_change":"2023-10-07T10:45:18.132585+0000","last_active":"2023-10-07T10:45:19.153415+0000","last_peered":"2023-10-07T10:45:19.153415+0000","last_clean":"2023-10-07T10:45:19.153415+0000","last_became_active":"2023-10-07T10:45:18.131693+0000","last_became_peered":"2023-10-07T10:45:18.131693+0000","last_unstale":"2023-10-07T10:45:19.153415+0000","last_undegraded":"2023-10-07T10:45:19.153415+0000","last_fullsized":"2023-10-07T10:45:19.153415+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:28.026376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145687+0000","last_change":"2023-10-07T10:45:19.144255+0000","last_active":"2023-10-07T10:45:19.145687+0000","last_peered":"2023-10-07T10:45:19.145687+0000","last_clean":"2023-10-07T10:45:19.145687+0000","last_became_active":"2023-10-07T10:45:19.143928+0000","last_became_peered":"2023-10-07T10:45:19.143928+0000","last_unstale":"2023-10-07T10:45:19.145687+0000","last_undegraded":"2023-10-07T10:45:19.145687+0000","last_fullsized":"2023-10-07T10:45:19.145687+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:14:40.088604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129740+0000","last_change":"2023-10-07T10:45:18.129727+0000","last_active":"2023-10-07T10:45:19.129740+0000","last_peered":"2023-10-07T10:45:19.129740+0000","last_clean":"2023-10-07T10:45:19.129740+0000","last_became_active":"2023-10-07T10:45:18.129381+0000","last_became_peered":"2023-10-07T10:45:18.129381+0000","last_unstale":"2023-10-07T10:45:19.129740+0000","last_undegraded":"2023-10-07T10:45:19.129740+0000","last_fullsized":"2023-10-07T10:45:19.129740+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:42:51.132950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.137885+0000","last_change":"2023-10-07T10:45:19.133782+0000","last_active":"2023-10-07T10:45:19.137885+0000","last_peered":"2023-10-07T10:45:19.137885+0000","last_clean":"2023-10-07T10:45:19.137885+0000","last_became_active":"2023-10-07T10:45:19.132846+0000","last_became_peered":"2023-10-07T10:45:19.132846+0000","last_unstale":"2023-10-07T10:45:19.137885+0000","last_undegraded":"2023-10-07T10:45:19.137885+0000","last_fullsized":"2023-10-07T10:45:19.137885+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:16:15.346473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.121628+0000","last_change":"2023-10-07T10:45:18.133406+0000","last_active":"2023-10-07T10:45:19.121628+0000","last_peered":"2023-10-07T10:45:19.121628+0000","last_clean":"2023-10-07T10:45:19.121628+0000","last_became_active":"2023-10-07T10:45:18.133149+0000","last_became_peered":"2023-10-07T10:45:18.133149+0000","last_unstale":"2023-10-07T10:45:19.121628+0000","last_undegraded":"2023-10-07T10:45:19.121628+0000","last_fullsized":"2023-10-07T10:45:19.121628+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:00:11.780313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148125+0000","last_change":"2023-10-07T10:45:19.135324+0000","last_active":"2023-10-07T10:45:19.148125+0000","last_peered":"2023-10-07T10:45:19.148125+0000","last_clean":"2023-10-07T10:45:19.148125+0000","last_became_active":"2023-10-07T10:45:19.135018+0000","last_became_peered":"2023-10-07T10:45:19.135018+0000","last_unstale":"2023-10-07T10:45:19.148125+0000","last_undegraded":"2023-10-07T10:45:19.148125+0000","last_fullsized":"2023-10-07T10:45:19.148125+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:04.850331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"164'8","reported_seq":18,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.158447+0000","last_change":"2023-10-07T10:45:18.140162+0000","last_active":"2023-10-07T10:45:19.158447+0000","last_peered":"2023-10-07T10:45:19.158447+0000","last_clean":"2023-10-07T10:45:19.158447+0000","last_became_active":"2023-10-07T10:45:18.139873+0000","last_became_peered":"2023-10-07T10:45:18.139873+0000","last_unstale":"2023-10-07T10:45:19.158447+0000","last_undegraded":"2023-10-07T10:45:19.158447+0000","last_fullsized":"2023-10-07T10:45:19.158447+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:21:40.546535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.153599+0000","last_change":"2023-10-07T10:45:19.152777+0000","last_active":"2023-10-07T10:45:19.153599+0000","last_peered":"2023-10-07T10:45:19.153599+0000","last_clean":"2023-10-07T10:45:19.153599+0000","last_became_active":"2023-10-07T10:45:19.152473+0000","last_became_peered":"2023-10-07T10:45:19.152473+0000","last_unstale":"2023-10-07T10:45:19.153599+0000","last_undegraded":"2023-10-07T10:45:19.153599+0000","last_fullsized":"2023-10-07T10:45:19.153599+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:29:51.071208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150644+0000","last_change":"2023-10-07T10:45:18.128406+0000","last_active":"2023-10-07T10:45:19.150644+0000","last_peered":"2023-10-07T10:45:19.150644+0000","last_clean":"2023-10-07T10:45:19.150644+0000","last_became_active":"2023-10-07T10:45:18.128118+0000","last_became_peered":"2023-10-07T10:45:18.128118+0000","last_unstale":"2023-10-07T10:45:19.150644+0000","last_undegraded":"2023-10-07T10:45:19.150644+0000","last_fullsized":"2023-10-07T10:45:19.150644+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:56:15.480123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.154517+0000","last_change":"2023-10-07T10:45:19.152009+0000","last_active":"2023-10-07T10:45:19.154517+0000","last_peered":"2023-10-07T10:45:19.154517+0000","last_clean":"2023-10-07T10:45:19.154517+0000","last_became_active":"2023-10-07T10:45:19.151697+0000","last_became_peered":"2023-10-07T10:45:19.151697+0000","last_unstale":"2023-10-07T10:45:19.154517+0000","last_undegraded":"2023-10-07T10:45:19.154517+0000","last_fullsized":"2023-10-07T10:45:19.154517+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:38.666098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.153478+0000","last_change":"2023-10-07T10:45:18.136247+0000","last_active":"2023-10-07T10:45:19.153478+0000","last_peered":"2023-10-07T10:45:19.153478+0000","last_clean":"2023-10-07T10:45:19.153478+0000","last_became_active":"2023-10-07T10:45:18.135963+0000","last_became_peered":"2023-10-07T10:45:18.135963+0000","last_unstale":"2023-10-07T10:45:19.153478+0000","last_undegraded":"2023-10-07T10:45:19.153478+0000","last_fullsized":"2023-10-07T10:45:19.153478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:35.810298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148162+0000","last_change":"2023-10-07T10:45:19.135505+0000","last_active":"2023-10-07T10:45:19.148162+0000","last_peered":"2023-10-07T10:45:19.148162+0000","last_clean":"2023-10-07T10:45:19.148162+0000","last_became_active":"2023-10-07T10:45:19.135133+0000","last_became_peered":"2023-10-07T10:45:19.135133+0000","last_unstale":"2023-10-07T10:45:19.148162+0000","last_undegraded":"2023-10-07T10:45:19.148162+0000","last_fullsized":"2023-10-07T10:45:19.148162+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:56.491619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.156684+0000","last_change":"2023-10-07T10:45:18.140506+0000","last_active":"2023-10-07T10:45:19.156684+0000","last_peered":"2023-10-07T10:45:19.156684+0000","last_clean":"2023-10-07T10:45:19.156684+0000","last_became_active":"2023-10-07T10:45:18.140173+0000","last_became_peered":"2023-10-07T10:45:18.140173+0000","last_unstale":"2023-10-07T10:45:19.156684+0000","last_undegraded":"2023-10-07T10:45:19.156684+0000","last_fullsized":"2023-10-07T10:45:19.156684+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:00:35.241523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148197+0000","last_change":"2023-10-07T10:45:19.141719+0000","last_active":"2023-10-07T10:45:19.148197+0000","last_peered":"2023-10-07T10:45:19.148197+0000","last_clean":"2023-10-07T10:45:19.148197+0000","last_became_active":"2023-10-07T10:45:19.141374+0000","last_became_peered":"2023-10-07T10:45:19.141374+0000","last_unstale":"2023-10-07T10:45:19.148197+0000","last_undegraded":"2023-10-07T10:45:19.148197+0000","last_fullsized":"2023-10-07T10:45:19.148197+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:32:04.300310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129491+0000","last_change":"2023-10-07T10:45:18.135852+0000","last_active":"2023-10-07T10:45:19.129491+0000","last_peered":"2023-10-07T10:45:19.129491+0000","last_clean":"2023-10-07T10:45:19.129491+0000","last_became_active":"2023-10-07T10:45:18.135583+0000","last_became_peered":"2023-10-07T10:45:18.135583+0000","last_unstale":"2023-10-07T10:45:19.129491+0000","last_undegraded":"2023-10-07T10:45:19.129491+0000","last_fullsized":"2023-10-07T10:45:19.129491+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:34:32.011131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147054+0000","last_change":"2023-10-07T10:45:19.144334+0000","last_active":"2023-10-07T10:45:19.147054+0000","last_peered":"2023-10-07T10:45:19.147054+0000","last_clean":"2023-10-07T10:45:19.147054+0000","last_became_active":"2023-10-07T10:45:19.144028+0000","last_became_peered":"2023-10-07T10:45:19.144028+0000","last_unstale":"2023-10-07T10:45:19.147054+0000","last_undegraded":"2023-10-07T10:45:19.147054+0000","last_fullsized":"2023-10-07T10:45:19.147054+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:36.719844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150619+0000","last_change":"2023-10-07T10:45:18.131596+0000","last_active":"2023-10-07T10:45:19.150619+0000","last_peered":"2023-10-07T10:45:19.150619+0000","last_clean":"2023-10-07T10:45:19.150619+0000","last_became_active":"2023-10-07T10:45:18.131338+0000","last_became_peered":"2023-10-07T10:45:18.131338+0000","last_unstale":"2023-10-07T10:45:19.150619+0000","last_undegraded":"2023-10-07T10:45:19.150619+0000","last_fullsized":"2023-10-07T10:45:19.150619+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:35.810298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149650+0000","last_change":"2023-10-07T10:45:19.149014+0000","last_active":"2023-10-07T10:45:19.149650+0000","last_peered":"2023-10-07T10:45:19.149650+0000","last_clean":"2023-10-07T10:45:19.149650+0000","last_became_active":"2023-10-07T10:45:19.148716+0000","last_became_peered":"2023-10-07T10:45:19.148716+0000","last_unstale":"2023-10-07T10:45:19.149650+0000","last_undegraded":"2023-10-07T10:45:19.149650+0000","last_fullsized":"2023-10-07T10:45:19.149650+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:52.338610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128929+0000","last_change":"2023-10-07T10:45:18.137178+0000","last_active":"2023-10-07T10:45:19.128929+0000","last_peered":"2023-10-07T10:45:19.128929+0000","last_clean":"2023-10-07T10:45:19.128929+0000","last_became_active":"2023-10-07T10:45:18.136546+0000","last_became_peered":"2023-10-07T10:45:18.136546+0000","last_unstale":"2023-10-07T10:45:19.128929+0000","last_undegraded":"2023-10-07T10:45:19.128929+0000","last_fullsized":"2023-10-07T10:45:19.128929+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:37:20.378578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146731+0000","last_change":"2023-10-07T10:45:19.139633+0000","last_active":"2023-10-07T10:45:19.146731+0000","last_peered":"2023-10-07T10:45:19.146731+0000","last_clean":"2023-10-07T10:45:19.146731+0000","last_became_active":"2023-10-07T10:45:19.139291+0000","last_became_peered":"2023-10-07T10:45:19.139291+0000","last_unstale":"2023-10-07T10:45:19.146731+0000","last_undegraded":"2023-10-07T10:45:19.146731+0000","last_fullsized":"2023-10-07T10:45:19.146731+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:40.850441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128809+0000","last_change":"2023-10-07T10:45:18.136369+0000","last_active":"2023-10-07T10:45:19.128809+0000","last_peered":"2023-10-07T10:45:19.128809+0000","last_clean":"2023-10-07T10:45:19.128809+0000","last_became_active":"2023-10-07T10:45:18.136047+0000","last_became_peered":"2023-10-07T10:45:18.136047+0000","last_unstale":"2023-10-07T10:45:19.128809+0000","last_undegraded":"2023-10-07T10:45:19.128809+0000","last_fullsized":"2023-10-07T10:45:19.128809+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:34:37.975907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143248+0000","last_change":"2023-10-07T10:45:19.136207+0000","last_active":"2023-10-07T10:45:19.143248+0000","last_peered":"2023-10-07T10:45:19.143248+0000","last_clean":"2023-10-07T10:45:19.143248+0000","last_became_active":"2023-10-07T10:45:19.135718+0000","last_became_peered":"2023-10-07T10:45:19.135718+0000","last_unstale":"2023-10-07T10:45:19.143248+0000","last_undegraded":"2023-10-07T10:45:19.143248+0000","last_fullsized":"2023-10-07T10:45:19.143248+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:10.056741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128872+0000","last_change":"2023-10-07T10:45:18.133723+0000","last_active":"2023-10-07T10:45:19.128872+0000","last_peered":"2023-10-07T10:45:19.128872+0000","last_clean":"2023-10-07T10:45:19.128872+0000","last_became_active":"2023-10-07T10:45:18.132925+0000","last_became_peered":"2023-10-07T10:45:18.132925+0000","last_unstale":"2023-10-07T10:45:19.128872+0000","last_undegraded":"2023-10-07T10:45:19.128872+0000","last_fullsized":"2023-10-07T10:45:19.128872+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:01:56.666407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149589+0000","last_change":"2023-10-07T10:45:19.146673+0000","last_active":"2023-10-07T10:45:19.149589+0000","last_peered":"2023-10-07T10:45:19.149589+0000","last_clean":"2023-10-07T10:45:19.149589+0000","last_became_active":"2023-10-07T10:45:19.145598+0000","last_became_peered":"2023-10-07T10:45:19.145598+0000","last_unstale":"2023-10-07T10:45:19.149589+0000","last_undegraded":"2023-10-07T10:45:19.149589+0000","last_fullsized":"2023-10-07T10:45:19.149589+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:38.666098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150706+0000","last_change":"2023-10-07T10:45:18.141173+0000","last_active":"2023-10-07T10:45:19.150706+0000","last_peered":"2023-10-07T10:45:19.150706+0000","last_clean":"2023-10-07T10:45:19.150706+0000","last_became_active":"2023-10-07T10:45:18.140707+0000","last_became_peered":"2023-10-07T10:45:18.140707+0000","last_unstale":"2023-10-07T10:45:19.150706+0000","last_undegraded":"2023-10-07T10:45:19.150706+0000","last_fullsized":"2023-10-07T10:45:19.150706+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:53.011975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148284+0000","last_change":"2023-10-07T10:45:19.136453+0000","last_active":"2023-10-07T10:45:19.148284+0000","last_peered":"2023-10-07T10:45:19.148284+0000","last_clean":"2023-10-07T10:45:19.148284+0000","last_became_active":"2023-10-07T10:45:19.136143+0000","last_became_peered":"2023-10-07T10:45:19.136143+0000","last_unstale":"2023-10-07T10:45:19.148284+0000","last_undegraded":"2023-10-07T10:45:19.148284+0000","last_fullsized":"2023-10-07T10:45:19.148284+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:10:46.792904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.121545+0000","last_change":"2023-10-07T10:45:18.142439+0000","last_active":"2023-10-07T10:45:19.121545+0000","last_peered":"2023-10-07T10:45:19.121545+0000","last_clean":"2023-10-07T10:45:19.121545+0000","last_became_active":"2023-10-07T10:45:18.141989+0000","last_became_peered":"2023-10-07T10:45:18.141989+0000","last_unstale":"2023-10-07T10:45:19.121545+0000","last_undegraded":"2023-10-07T10:45:19.121545+0000","last_fullsized":"2023-10-07T10:45:19.121545+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:10:43.271147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151104+0000","last_change":"2023-10-07T10:45:19.149498+0000","last_active":"2023-10-07T10:45:19.151104+0000","last_peered":"2023-10-07T10:45:19.151104+0000","last_clean":"2023-10-07T10:45:19.151104+0000","last_became_active":"2023-10-07T10:45:19.149143+0000","last_became_peered":"2023-10-07T10:45:19.149143+0000","last_unstale":"2023-10-07T10:45:19.151104+0000","last_undegraded":"2023-10-07T10:45:19.151104+0000","last_fullsized":"2023-10-07T10:45:19.151104+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:27.595946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.123379+0000","last_change":"2023-10-07T10:45:18.135185+0000","last_active":"2023-10-07T10:45:19.123379+0000","last_peered":"2023-10-07T10:45:19.123379+0000","last_clean":"2023-10-07T10:45:19.123379+0000","last_became_active":"2023-10-07T10:45:18.134932+0000","last_became_peered":"2023-10-07T10:45:18.134932+0000","last_unstale":"2023-10-07T10:45:19.123379+0000","last_undegraded":"2023-10-07T10:45:19.123379+0000","last_fullsized":"2023-10-07T10:45:19.123379+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:43:26.471746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140477+0000","last_change":"2023-10-07T10:45:19.136486+0000","last_active":"2023-10-07T10:45:19.140477+0000","last_peered":"2023-10-07T10:45:19.140477+0000","last_clean":"2023-10-07T10:45:19.140477+0000","last_became_active":"2023-10-07T10:45:19.136142+0000","last_became_peered":"2023-10-07T10:45:19.136142+0000","last_unstale":"2023-10-07T10:45:19.140477+0000","last_undegraded":"2023-10-07T10:45:19.140477+0000","last_fullsized":"2023-10-07T10:45:19.140477+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:11.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.122215+0000","last_change":"2023-10-07T10:45:18.135993+0000","last_active":"2023-10-07T10:45:19.122215+0000","last_peered":"2023-10-07T10:45:19.122215+0000","last_clean":"2023-10-07T10:45:19.122215+0000","last_became_active":"2023-10-07T10:45:18.135711+0000","last_became_peered":"2023-10-07T10:45:18.135711+0000","last_unstale":"2023-10-07T10:45:19.122215+0000","last_undegraded":"2023-10-07T10:45:19.122215+0000","last_fullsized":"2023-10-07T10:45:19.122215+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:09.270891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.157861+0000","last_change":"2023-10-07T10:45:19.155851+0000","last_active":"2023-10-07T10:45:19.157861+0000","last_peered":"2023-10-07T10:45:19.157861+0000","last_clean":"2023-10-07T10:45:19.157861+0000","last_became_active":"2023-10-07T10:45:19.155583+0000","last_became_peered":"2023-10-07T10:45:19.155583+0000","last_unstale":"2023-10-07T10:45:19.157861+0000","last_undegraded":"2023-10-07T10:45:19.157861+0000","last_fullsized":"2023-10-07T10:45:19.157861+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:10.279050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128304+0000","last_change":"2023-10-07T10:45:18.128985+0000","last_active":"2023-10-07T10:45:19.128304+0000","last_peered":"2023-10-07T10:45:19.128304+0000","last_clean":"2023-10-07T10:45:19.128304+0000","last_became_active":"2023-10-07T10:45:18.128702+0000","last_became_peered":"2023-10-07T10:45:18.128702+0000","last_unstale":"2023-10-07T10:45:19.128304+0000","last_undegraded":"2023-10-07T10:45:19.128304+0000","last_fullsized":"2023-10-07T10:45:19.128304+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:34:32.011131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140886+0000","last_change":"2023-10-07T10:45:19.137156+0000","last_active":"2023-10-07T10:45:19.140886+0000","last_peered":"2023-10-07T10:45:19.140886+0000","last_clean":"2023-10-07T10:45:19.140886+0000","last_became_active":"2023-10-07T10:45:19.136892+0000","last_became_peered":"2023-10-07T10:45:19.136892+0000","last_unstale":"2023-10-07T10:45:19.140886+0000","last_undegraded":"2023-10-07T10:45:19.140886+0000","last_fullsized":"2023-10-07T10:45:19.140886+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:10:44.279307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129036+0000","last_change":"2023-10-07T10:45:18.137366+0000","last_active":"2023-10-07T10:45:19.129036+0000","last_peered":"2023-10-07T10:45:19.129036+0000","last_clean":"2023-10-07T10:45:19.129036+0000","last_became_active":"2023-10-07T10:45:18.136843+0000","last_became_peered":"2023-10-07T10:45:18.136843+0000","last_unstale":"2023-10-07T10:45:19.129036+0000","last_undegraded":"2023-10-07T10:45:19.129036+0000","last_fullsized":"2023-10-07T10:45:19.129036+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:07:50.965052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146741+0000","last_change":"2023-10-07T10:45:19.143812+0000","last_active":"2023-10-07T10:45:19.146741+0000","last_peered":"2023-10-07T10:45:19.146741+0000","last_clean":"2023-10-07T10:45:19.146741+0000","last_became_active":"2023-10-07T10:45:19.143556+0000","last_became_peered":"2023-10-07T10:45:19.143556+0000","last_unstale":"2023-10-07T10:45:19.146741+0000","last_undegraded":"2023-10-07T10:45:19.146741+0000","last_fullsized":"2023-10-07T10:45:19.146741+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:18.999677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129731+0000","last_change":"2023-10-07T10:45:18.134209+0000","last_active":"2023-10-07T10:45:19.129731+0000","last_peered":"2023-10-07T10:45:19.129731+0000","last_clean":"2023-10-07T10:45:19.129731+0000","last_became_active":"2023-10-07T10:45:18.133896+0000","last_became_peered":"2023-10-07T10:45:18.133896+0000","last_unstale":"2023-10-07T10:45:19.129731+0000","last_undegraded":"2023-10-07T10:45:19.129731+0000","last_fullsized":"2023-10-07T10:45:19.129731+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:31:30.567674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150397+0000","last_change":"2023-10-07T10:45:19.147304+0000","last_active":"2023-10-07T10:45:19.150397+0000","last_peered":"2023-10-07T10:45:19.150397+0000","last_clean":"2023-10-07T10:45:19.150397+0000","last_became_active":"2023-10-07T10:45:19.146985+0000","last_became_peered":"2023-10-07T10:45:19.146985+0000","last_unstale":"2023-10-07T10:45:19.150397+0000","last_undegraded":"2023-10-07T10:45:19.150397+0000","last_fullsized":"2023-10-07T10:45:19.150397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:43:27.479906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.126559+0000","last_change":"2023-10-07T10:45:18.137722+0000","last_active":"2023-10-07T10:45:19.126559+0000","last_peered":"2023-10-07T10:45:19.126559+0000","last_clean":"2023-10-07T10:45:19.126559+0000","last_became_active":"2023-10-07T10:45:18.137452+0000","last_became_peered":"2023-10-07T10:45:18.137452+0000","last_unstale":"2023-10-07T10:45:19.126559+0000","last_undegraded":"2023-10-07T10:45:19.126559+0000","last_fullsized":"2023-10-07T10:45:19.126559+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:39.842281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143640+0000","last_change":"2023-10-07T10:45:19.136644+0000","last_active":"2023-10-07T10:45:19.143640+0000","last_peered":"2023-10-07T10:45:19.143640+0000","last_clean":"2023-10-07T10:45:19.143640+0000","last_became_active":"2023-10-07T10:45:19.136048+0000","last_became_peered":"2023-10-07T10:45:19.136048+0000","last_unstale":"2023-10-07T10:45:19.143640+0000","last_undegraded":"2023-10-07T10:45:19.143640+0000","last_fullsized":"2023-10-07T10:45:19.143640+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:04.850331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"164'3","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150236+0000","last_change":"2023-10-07T10:45:18.132966+0000","last_active":"2023-10-07T10:45:19.150236+0000","last_peered":"2023-10-07T10:45:19.150236+0000","last_clean":"2023-10-07T10:45:19.150236+0000","last_became_active":"2023-10-07T10:45:18.132660+0000","last_became_peered":"2023-10-07T10:45:18.132660+0000","last_unstale":"2023-10-07T10:45:19.150236+0000","last_undegraded":"2023-10-07T10:45:19.150236+0000","last_fullsized":"2023-10-07T10:45:19.150236+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:31.762379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"154'293","reported_seq":910,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128705+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T10:45:19.128705+0000","last_peered":"2023-10-07T10:45:19.128705+0000","last_clean":"2023-10-07T10:45:19.128705+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T10:45:19.128705+0000","last_undegraded":"2023-10-07T10:45:19.128705+0000","last_fullsized":"2023-10-07T10:45:19.128705+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477641,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308072,"kb_used_data":2592,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063768,"statfs":{"total":96636764160,"available":96321298432,"internally_reserved":0,"allocated":2654208,"data_stored":1597511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.95699999999999996,"5min":1.018,"15min":1.018},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.028,"15min":1.028},"last":0.872}]},{"osd":1,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.748,"15min":0.752},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.432},"max":{"1min":0.90900000000000003,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.75900000000000001},"min":{"1min":0.56999999999999995,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.995,"5min":1.119,"15min":1.119},"last":0.76100000000000001}]},{"osd":2,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88200000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78000000000000003,"15min":0.77100000000000002},"min":{"1min":0.65900000000000003,"5min":0.55900000000000005,"15min":0.45400000000000001},"max":{"1min":1.002,"5min":1.147,"15min":1.147},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.746,"15min":0.73999999999999999},"min":{"1min":0.51600000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.94199999999999995,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.59399999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93899999999999995,"5min":1.161,"15min":1.161},"last":0.69799999999999995}]},{"osd":4,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.47699999999999998,"5min":0.44700000000000001,"15min":0.36399999999999999},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.92700000000000005,"5min":1.069,"15min":1.069},"last":1.161}]},{"osd":5,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.83899999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70799999999999996,"15min":0.69599999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.96399999999999997}]},{"osd":6,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.746,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71199999999999997},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86699999999999999,"5min":1.131,"15min":1.131},"last":0.89000000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311580,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060260,"statfs":{"total":96636764160,"available":96317706240,"internally_reserved":0,"allocated":2052096,"data_stored":999957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":0.75900000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90700000000000003,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.54400000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92600000000000005,"5min":1.052,"15min":1.073},"last":0.755}]},{"osd":1,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.75800000000000001,"15min":0.754},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48499999999999999},"max":{"1min":0.874,"5min":1.008,"15min":1.0600000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.50700000000000001},"max":{"1min":0.89000000000000001,"5min":1.034,"15min":1.0389999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46100000000000002},"max":{"1min":1.018,"5min":1.0289999999999999,"15min":1.3380000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59499999999999997,"5min":0.54900000000000004,"15min":0.46500000000000002},"max":{"1min":0.95999999999999996,"5min":1.03,"15min":1.367},"last":0.78400000000000003}]},{"osd":3,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.51100000000000001},"max":{"1min":0.94599999999999995,"5min":1.044,"15min":1.0900000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":1.0289999999999999,"15min":1.1040000000000001},"last":0.69099999999999995}]},{"osd":4,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.621},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0609999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.499,"5min":0.42899999999999999,"15min":0.40999999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0900000000000001},"last":0.52300000000000002}]},{"osd":7,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.52000000000000002,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.1060000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76700000000000002,"15min":0.77000000000000002},"min":{"1min":0.57999999999999996,"5min":0.57899999999999996,"15min":0.53600000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.127},"last":0.60499999999999998}]}]},{"osd":1,"up_from":14,"seq":60129543047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307488,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064352,"statfs":{"total":96636764160,"available":96321896448,"internally_reserved":0,"allocated":2056192,"data_stored":1007073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28899999999999998},"max":{"1min":0.59999999999999998,"5min":0.76000000000000001,"15min":0.83099999999999996},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57499999999999996,"15min":0.58799999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.40500000000000003},"max":{"1min":0.65900000000000003,"5min":0.76900000000000002,"15min":0.84199999999999997},"last":0.502}]},{"osd":2,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.62,"5min":0.74399999999999999,"15min":0.81699999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.82499999999999996,"15min":0.89000000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41999999999999998},"max":{"1min":0.77600000000000002,"5min":0.88400000000000001,"15min":0.90400000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":0.84999999999999998,"15min":0.93100000000000005},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56999999999999995},"max":{"1min":1.0660000000000001,"5min":1.1000000000000001,"15min":1.1719999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79100000000000004,"15min":0.80200000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.85099999999999998}]},{"osd":5,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1220000000000001,"5min":1.196,"15min":1.196},"last":0.747},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"max":{"1min":1.1060000000000001,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78600000000000003}]},{"osd":6,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.86399999999999999},"min":{"1min":0.66200000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2050000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.83399999999999996,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.173,"5min":1.173,"15min":1.196},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.85699999999999998,"15min":0.875},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.143,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.88200000000000001,"15min":0.88800000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.60799999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.76900000000000002}]}]},{"osd":0,"up_from":14,"seq":60129543047,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308080,"kb_used_data":2600,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063760,"statfs":{"total":96636764160,"available":96321290240,"internally_reserved":0,"allocated":2662400,"data_stored":1598457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.61099999999999999,"5min":0.751,"15min":0.76400000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":0.82899999999999996},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75,"5min":0.88900000000000001,"15min":0.92800000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.68400000000000005,"5min":0.87,"15min":0.87},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70899999999999996,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.435},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.80600000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81599999999999995,"15min":0.79700000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.499},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.63},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.64700000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0960000000000001,"5min":1.194,"15min":1.194},"last":0.753}]},{"osd":5,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.077,"5min":1.278,"15min":1.278},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":1.0229999999999999,"5min":1.206,"15min":1.206},"last":0.73499999999999999}]},{"osd":6,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86199999999999999,"15min":0.84699999999999998},"min":{"1min":0.67100000000000004,"5min":0.60599999999999998,"15min":0.59499999999999997},"max":{"1min":1.0369999999999999,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85799999999999998,"15min":0.85099999999999998},"min":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"max":{"1min":1.1579999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.81699999999999995}]},{"osd":7,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.89300000000000002,"15min":0.872},"min":{"1min":0.627,"5min":0.627,"15min":0.59899999999999998},"max":{"1min":1.131,"5min":1.173,"15min":1.198},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.875,"15min":0.86599999999999999},"min":{"1min":0.63100000000000001,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":1.198,"5min":1.198,"15min":1.234},"last":0.77100000000000002}]}]},{"osd":2,"up_from":14,"seq":60129543048,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311584,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060256,"statfs":{"total":96636764160,"available":96317702144,"internally_reserved":0,"allocated":2056192,"data_stored":1001417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48499999999999999,"15min":0.47399999999999998},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":0.626,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.33400000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.46600000000000003}]},{"osd":1,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":1.369,"15min":1.369},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.53500000000000003}]},{"osd":3,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.59099999999999997,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.0490000000000004,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.73499999999999999,"15min":0.66000000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40500000000000003},"max":{"1min":10.1,"5min":10.1,"15min":10.1},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81799999999999995,"15min":0.80100000000000005},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.48499999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.82099999999999995},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.47299999999999998},"max":{"1min":1.032,"5min":1.23,"15min":1.23},"last":0.68999999999999995}]},{"osd":5,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.64100000000000001,"5min":0.53300000000000003,"15min":0.44400000000000001},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.349},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.60299999999999998,"5min":0.56599999999999995,"15min":0.54000000000000004},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.70799999999999996}]},{"osd":6,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.85699999999999998,"15min":0.85899999999999999},"min":{"1min":0.64800000000000002,"5min":0.59399999999999997,"15min":0.55200000000000005},"max":{"1min":1.0860000000000001,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.87,"15min":0.85499999999999998},"min":{"1min":0.59799999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1659999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88800000000000001,"15min":0.879},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.3160000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.87,"15min":0.874},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0980000000000001,"5min":1.139,"15min":1.2969999999999999},"last":0.61399999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543049,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299284,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072556,"statfs":{"total":96636764160,"available":96330297344,"internally_reserved":0,"allocated":2043904,"data_stored":999901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58599999999999997,"5min":0.79300000000000004,"15min":1.073},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.81699999999999995,"15min":1.5149999999999999},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":0.82499999999999996,"15min":1.5489999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":1.02},"last":0.45100000000000001}]},{"osd":2,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56200000000000006,"15min":0.57099999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":1.456},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":0.94399999999999995,"15min":1.6040000000000001},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.79600000000000004,"15min":0.80300000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.173},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.80200000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95299999999999996,"5min":1.141,"15min":1.3839999999999999},"last":1.0580000000000001}]},{"osd":5,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.80500000000000005,"15min":0.81200000000000006},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"max":{"1min":0.90900000000000003,"5min":1.1539999999999999,"15min":1.1879999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82599999999999996,"15min":0.83499999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.98199999999999998,"5min":1.1619999999999999,"15min":1.4339999999999999},"last":0.79500000000000004}]},{"osd":6,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82799999999999996,"15min":0.83999999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0289999999999999,"5min":1.214,"15min":1.357},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.85099999999999998,"15min":0.84399999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.109}]},{"osd":7,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.85599999999999998,"15min":0.85099999999999998},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":1.137,"15min":1.492},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.85999999999999999,"15min":0.86299999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.59599999999999997},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.0329999999999999}]}]},{"osd":4,"up_from":15,"seq":64424510345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311580,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060260,"statfs":{"total":96636764160,"available":96317706240,"internally_reserved":0,"allocated":2052096,"data_stored":1006519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93600000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.745,"15min":0.76000000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86499999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55400000000000005}]},{"osd":1,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.76400000000000001,"15min":0.76500000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.016,"15min":1.032},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46500000000000002},"max":{"1min":0.84599999999999997,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59599999999999997}]},{"osd":2,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":0.90300000000000002,"5min":1.002,"15min":1.01},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88500000000000001,"5min":1.145,"15min":1.145},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.75600000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.497},"max":{"1min":0.82899999999999996,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96599999999999997,"5min":1.129,"15min":1.129},"last":0.74399999999999999}]},{"osd":5,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45000000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.441,"5min":0.441,"15min":0.39400000000000002},"max":{"1min":0.85599999999999998,"5min":0.93999999999999995,"15min":1.0349999999999999},"last":0.71999999999999997}]},{"osd":6,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89200000000000002,"5min":0.98999999999999999,"15min":1.018},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.69899999999999995},"min":{"1min":0.54500000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85499999999999998,"5min":0.96799999999999997,"15min":1.042},"last":0.86299999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.54600000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90400000000000003,"5min":0.94099999999999995,"15min":0.97899999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0960000000000001},"last":0.82799999999999996}]}]},{"osd":5,"up_from":14,"seq":60129543047,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311584,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060256,"statfs":{"total":96636764160,"available":96317702144,"internally_reserved":0,"allocated":2056192,"data_stored":1006541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78700000000000003,"15min":0.79400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39500000000000002},"max":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.091},"last":0.745},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.438},"max":{"1min":0.93500000000000005,"5min":1.0820000000000001,"15min":1.1279999999999999},"last":0.83499999999999996}]},{"osd":1,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.746},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93400000000000005,"5min":1.032,"15min":1.032},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78100000000000003,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.46899999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89900000000000002,"5min":1.0760000000000001,"15min":1.093},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.76700000000000002,"15min":0.76800000000000002},"min":{"1min":0.52300000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89100000000000001,"5min":1.077,"15min":1.343},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.77600000000000002,"15min":0.78200000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.50800000000000001},"max":{"1min":0.872,"5min":1.121,"15min":1.121},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75700000000000001,"15min":0.754},"min":{"1min":0.61799999999999999,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.77700000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.91400000000000003,"5min":0.94999999999999996,"15min":0.995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.64700000000000002},"min":{"1min":0.47199999999999998,"5min":0.46500000000000002,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":1.0089999999999999,"15min":1.0529999999999999},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.52400000000000002,"5min":0.44500000000000001,"15min":0.39800000000000002},"max":{"1min":0.91900000000000004,"5min":1.016,"15min":1.083},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.71499999999999997,"15min":0.71299999999999997},"min":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0289999999999999,"5min":1.032,"15min":1.101},"last":0.75700000000000001}]},{"osd":7,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.73499999999999999,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.075},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.86499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T10:45:23.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T10:45:23.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.771+0000 7fc1369ff700 1 Processor -- start 2023-10-07T10:45:23.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.771+0000 7fc1369ff700 1 -- start start 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.772+0000 7fc1369ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc1369ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1300ce8a0 0x7fc1300c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc1369ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1300cbf20 con 0x7fc130159b20 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc1369ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1300cc200 con 0x7fc1300ce8a0 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc1369ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300cc4e0 con 0x7fc130159750 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc12f7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc130159b20 0x7fc1300ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47446/0 (socket says 172.21.15.29:47446) 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc12f7fe700 1 -- 172.21.15.29:0/2996076314 learned_addr learned my addr 172.21.15.29:0/2996076314 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc12ffff700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc134f9c700 1 --2- 172.21.15.29:0/2996076314 >> v2:172.21.15.144:3300/0 conn(0x7fc1300ce8a0 0x7fc1300c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc12effd700 1 -- 172.21.15.29:0/2996076314 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657325250 0 0) 0x7fc1300cbf20 con 0x7fc130159b20 2023-10-07T10:45:23.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.773+0000 7fc12effd700 1 -- 172.21.15.29:0/2996076314 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc118003570 con 0x7fc130159b20 2023-10-07T10:45:23.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc134f9c700 1 -- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 msgr2=0x7fc1300d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:23.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc12effd700 1 -- 172.21.15.29:0/2996076314 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1269849055 0 0) 0x7fc118003570 con 0x7fc130159b20 2023-10-07T10:45:23.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc134f9c700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc134f9c700 1 -- 172.21.15.29:0/2996076314 >> v1:172.21.15.29:6789/0 conn(0x7fc130159b20 legacy=0x7fc1300ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc134f9c700 1 -- 172.21.15.29:0/2996076314 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300cc7c0 con 0x7fc1300ce8a0 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc12ffff700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc134f9c700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc1300c5460 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc128009e60 tx=0x7fc128009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a095e5821fc88f7d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc12effd700 1 -- 172.21.15.29:0/2996076314 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800e070 con 0x7fc1300ce8a0 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc12effd700 1 -- 172.21.15.29:0/2996076314 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12800b7a0 con 0x7fc1300ce8a0 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.774+0000 7fc12effd700 1 -- 172.21.15.29:0/2996076314 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124001470 con 0x7fc1300ce8a0 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 -- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 msgr2=0x7fc1300c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc1300c5460 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc128009e60 tx=0x7fc128009780 comp rx=0 tx=0).stop 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 -- 172.21.15.29:0/2996076314 shutdown_connections 2023-10-07T10:45:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc1300c5460 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 --2- 172.21.15.29:0/2996076314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 -- 172.21.15.29:0/2996076314 >> 172.21.15.29:0/2996076314 conn(0x7fc130063a10 msgr2=0x7fc1300d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 -- 172.21.15.29:0/2996076314 shutdown_connections 2023-10-07T10:45:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.775+0000 7fc1369ff700 1 -- 172.21.15.29:0/2996076314 wait complete. 2023-10-07T10:45:23.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.777+0000 7fc1369ff700 1 Processor -- start 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.777+0000 7fc1369ff700 1 -- start start 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc1369ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc130141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc1369ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc1369ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 0x7fc130154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc1369ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc130153960 con 0x7fc130159b20 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc1369ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc130151ff0 con 0x7fc1300ce8a0 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc1369ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13015efa0 con 0x7fc130159750 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 0x7fc130154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 0x7fc130154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56866/0 (socket says 172.21.15.29:56866) 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 -- 172.21.15.29:0/54492938 learned_addr learned my addr 172.21.15.29:0/54492938 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc12ffff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc130141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc12f7fe700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 -- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 msgr2=0x7fc1300b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 -- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 msgr2=0x7fc130141ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc130141ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.778+0000 7fc134f9c700 1 -- 172.21.15.29:0/54492938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200093c0 con 0x7fc130159b20 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.779+0000 7fc134f9c700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 0x7fc130154380 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc12800b610 tx=0x7fc128018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.779+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128012bf0 con 0x7fc130159b20 2023-10-07T10:45:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.779+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128012d50 con 0x7fc130159b20 2023-10-07T10:45:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.779+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12801e680 con 0x7fc130159b20 2023-10-07T10:45:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.779+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc128009a20 con 0x7fc130159b20 2023-10-07T10:45:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.779+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13015e6f0 con 0x7fc130159b20 2023-10-07T10:45:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.780+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13015f130 con 0x7fc130159b20 2023-10-07T10:45:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.781+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc12801ea70 con 0x7fc130159b20 2023-10-07T10:45:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.782+0000 7fc117fff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc11c066f30 0x7fc11c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.782+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fc1280863c0 con 0x7fc130159b20 2023-10-07T10:45:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.783+0000 7fc12ffff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc11c066f30 0x7fc11c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.783+0000 7fc12ffff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc11c066f30 0x7fc11c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc120009f30 tx=0x7fc120009120 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.786+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc128053490 con 0x7fc130159b20 2023-10-07T10:45:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.985+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc13004c3c0 con 0x7fc11c066f30 2023-10-07T10:45:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.994+0000 7fc117fff700 1 -- 172.21.15.29:0/54492938 <== mgr.8449 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289364 (secure 0 0 0) 0x7fc13004c3c0 con 0x7fc11c066f30 2023-10-07T10:45:24.001 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc11c066f30 msgr2=0x7fc11c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc11c066f30 0x7fc11c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc120009f30 tx=0x7fc120009120 comp rx=0 tx=0).stop 2023-10-07T10:45:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 msgr2=0x7fc130154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 0x7fc130154380 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc12800b610 tx=0x7fc128018000 comp rx=0 tx=0).stop 2023-10-07T10:45:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 shutdown_connections 2023-10-07T10:45:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc11c066f30 0x7fc11c0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc130159b20 0x7fc130154380 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1300ce8a0 0x7fc130141ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 --2- 172.21.15.29:0/54492938 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc130159750 0x7fc1300b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 >> 172.21.15.29:0/54492938 conn(0x7fc130063a10 msgr2=0x7fc1300cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 shutdown_connections 2023-10-07T10:45:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:23.998+0000 7fc1369ff700 1 -- 172.21.15.29:0/54492938 wait complete. 2023-10-07T10:45:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T10:45:24.016 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-07T10:45:23.185943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2469252,"kb_used_data":17220,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752505468,"statfs":{"total":773094113280,"available":770565599232,"internally_reserved":0,"allocated":17633280,"data_stored":9217376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001553"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.159607+0000","last_change":"2023-10-07T10:45:19.148845+0000","last_active":"2023-10-07T10:45:19.159607+0000","last_peered":"2023-10-07T10:45:19.159607+0000","last_clean":"2023-10-07T10:45:19.159607+0000","last_became_active":"2023-10-07T10:45:19.148464+0000","last_became_peered":"2023-10-07T10:45:19.148464+0000","last_unstale":"2023-10-07T10:45:19.159607+0000","last_undegraded":"2023-10-07T10:45:19.159607+0000","last_fullsized":"2023-10-07T10:45:19.159607+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:16:38.783846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145718+0000","last_change":"2023-10-07T10:45:19.144734+0000","last_active":"2023-10-07T10:45:19.145718+0000","last_peered":"2023-10-07T10:45:19.145718+0000","last_clean":"2023-10-07T10:45:19.145718+0000","last_became_active":"2023-10-07T10:45:19.144379+0000","last_became_peered":"2023-10-07T10:45:19.144379+0000","last_unstale":"2023-10-07T10:45:19.145718+0000","last_undegraded":"2023-10-07T10:45:19.145718+0000","last_fullsized":"2023-10-07T10:45:19.145718+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:47.397055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148213+0000","last_change":"2023-10-07T10:45:19.136552+0000","last_active":"2023-10-07T10:45:19.148213+0000","last_peered":"2023-10-07T10:45:19.148213+0000","last_clean":"2023-10-07T10:45:19.148213+0000","last_became_active":"2023-10-07T10:45:19.136264+0000","last_became_peered":"2023-10-07T10:45:19.136264+0000","last_unstale":"2023-10-07T10:45:19.148213+0000","last_undegraded":"2023-10-07T10:45:19.148213+0000","last_fullsized":"2023-10-07T10:45:19.148213+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:52.338610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146861+0000","last_change":"2023-10-07T10:45:19.140026+0000","last_active":"2023-10-07T10:45:19.146861+0000","last_peered":"2023-10-07T10:45:19.146861+0000","last_clean":"2023-10-07T10:45:19.146861+0000","last_became_active":"2023-10-07T10:45:19.139544+0000","last_became_peered":"2023-10-07T10:45:19.139544+0000","last_unstale":"2023-10-07T10:45:19.146861+0000","last_undegraded":"2023-10-07T10:45:19.146861+0000","last_fullsized":"2023-10-07T10:45:19.146861+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:28:53.575149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.137852+0000","last_change":"2023-10-07T10:45:19.133768+0000","last_active":"2023-10-07T10:45:19.137852+0000","last_peered":"2023-10-07T10:45:19.137852+0000","last_clean":"2023-10-07T10:45:19.137852+0000","last_became_active":"2023-10-07T10:45:19.133451+0000","last_became_peered":"2023-10-07T10:45:19.133451+0000","last_unstale":"2023-10-07T10:45:19.137852+0000","last_undegraded":"2023-10-07T10:45:19.137852+0000","last_fullsized":"2023-10-07T10:45:19.137852+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:39.283636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.152120+0000","last_change":"2023-10-07T10:45:19.142721+0000","last_active":"2023-10-07T10:45:19.152120+0000","last_peered":"2023-10-07T10:45:19.152120+0000","last_clean":"2023-10-07T10:45:19.152120+0000","last_became_active":"2023-10-07T10:45:19.142323+0000","last_became_peered":"2023-10-07T10:45:19.142323+0000","last_unstale":"2023-10-07T10:45:19.152120+0000","last_undegraded":"2023-10-07T10:45:19.152120+0000","last_fullsized":"2023-10-07T10:45:19.152120+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:06:49.678247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143282+0000","last_change":"2023-10-07T10:45:19.136492+0000","last_active":"2023-10-07T10:45:19.143282+0000","last_peered":"2023-10-07T10:45:19.143282+0000","last_clean":"2023-10-07T10:45:19.143282+0000","last_became_active":"2023-10-07T10:45:19.136169+0000","last_became_peered":"2023-10-07T10:45:19.136169+0000","last_unstale":"2023-10-07T10:45:19.143282+0000","last_undegraded":"2023-10-07T10:45:19.143282+0000","last_fullsized":"2023-10-07T10:45:19.143282+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:27.595946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.133399+0000","last_change":"2023-10-07T10:45:19.128733+0000","last_active":"2023-10-07T10:45:19.133399+0000","last_peered":"2023-10-07T10:45:19.133399+0000","last_clean":"2023-10-07T10:45:19.133399+0000","last_became_active":"2023-10-07T10:45:19.128449+0000","last_became_peered":"2023-10-07T10:45:19.128449+0000","last_unstale":"2023-10-07T10:45:19.133399+0000","last_undegraded":"2023-10-07T10:45:19.133399+0000","last_fullsized":"2023-10-07T10:45:19.133399+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:51:31.538804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141861+0000","last_change":"2023-10-07T10:45:19.139258+0000","last_active":"2023-10-07T10:45:19.141861+0000","last_peered":"2023-10-07T10:45:19.141861+0000","last_clean":"2023-10-07T10:45:19.141861+0000","last_became_active":"2023-10-07T10:45:19.138951+0000","last_became_peered":"2023-10-07T10:45:19.138951+0000","last_unstale":"2023-10-07T10:45:19.141861+0000","last_undegraded":"2023-10-07T10:45:19.141861+0000","last_fullsized":"2023-10-07T10:45:19.141861+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:13:24.999401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151527+0000","last_change":"2023-10-07T10:45:19.149127+0000","last_active":"2023-10-07T10:45:19.151527+0000","last_peered":"2023-10-07T10:45:19.151527+0000","last_clean":"2023-10-07T10:45:19.151527+0000","last_became_active":"2023-10-07T10:45:19.148854+0000","last_became_peered":"2023-10-07T10:45:19.148854+0000","last_unstale":"2023-10-07T10:45:19.151527+0000","last_undegraded":"2023-10-07T10:45:19.151527+0000","last_fullsized":"2023-10-07T10:45:19.151527+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:29.555531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.133370+0000","last_change":"2023-10-07T10:45:19.127275+0000","last_active":"2023-10-07T10:45:19.133370+0000","last_peered":"2023-10-07T10:45:19.133370+0000","last_clean":"2023-10-07T10:45:19.133370+0000","last_became_active":"2023-10-07T10:45:19.126996+0000","last_became_peered":"2023-10-07T10:45:19.126996+0000","last_unstale":"2023-10-07T10:45:19.133370+0000","last_undegraded":"2023-10-07T10:45:19.133370+0000","last_fullsized":"2023-10-07T10:45:19.133370+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:15:37.883323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145139+0000","last_change":"2023-10-07T10:45:19.139208+0000","last_active":"2023-10-07T10:45:19.145139+0000","last_peered":"2023-10-07T10:45:19.145139+0000","last_clean":"2023-10-07T10:45:19.145139+0000","last_became_active":"2023-10-07T10:45:19.138939+0000","last_became_peered":"2023-10-07T10:45:19.138939+0000","last_unstale":"2023-10-07T10:45:19.145139+0000","last_undegraded":"2023-10-07T10:45:19.145139+0000","last_fullsized":"2023-10-07T10:45:19.145139+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:19:23.963788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.137832+0000","last_change":"2023-10-07T10:45:19.132566+0000","last_active":"2023-10-07T10:45:19.137832+0000","last_peered":"2023-10-07T10:45:19.137832+0000","last_clean":"2023-10-07T10:45:19.137832+0000","last_became_active":"2023-10-07T10:45:19.132279+0000","last_became_peered":"2023-10-07T10:45:19.132279+0000","last_unstale":"2023-10-07T10:45:19.137832+0000","last_undegraded":"2023-10-07T10:45:19.137832+0000","last_fullsized":"2023-10-07T10:45:19.137832+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:37.783382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146914+0000","last_change":"2023-10-07T10:45:19.134365+0000","last_active":"2023-10-07T10:45:19.146914+0000","last_peered":"2023-10-07T10:45:19.146914+0000","last_clean":"2023-10-07T10:45:19.146914+0000","last_became_active":"2023-10-07T10:45:19.133994+0000","last_became_peered":"2023-10-07T10:45:19.133994+0000","last_unstale":"2023-10-07T10:45:19.146914+0000","last_undegraded":"2023-10-07T10:45:19.146914+0000","last_fullsized":"2023-10-07T10:45:19.146914+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:55.959931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146705+0000","last_change":"2023-10-07T10:45:19.143816+0000","last_active":"2023-10-07T10:45:19.146705+0000","last_peered":"2023-10-07T10:45:19.146705+0000","last_clean":"2023-10-07T10:45:19.146705+0000","last_became_active":"2023-10-07T10:45:19.143508+0000","last_became_peered":"2023-10-07T10:45:19.143508+0000","last_unstale":"2023-10-07T10:45:19.146705+0000","last_undegraded":"2023-10-07T10:45:19.146705+0000","last_fullsized":"2023-10-07T10:45:19.146705+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:42:47.896740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.156216+0000","last_change":"2023-10-07T10:45:19.150588+0000","last_active":"2023-10-07T10:45:19.156216+0000","last_peered":"2023-10-07T10:45:19.156216+0000","last_clean":"2023-10-07T10:45:19.156216+0000","last_became_active":"2023-10-07T10:45:19.150289+0000","last_became_peered":"2023-10-07T10:45:19.150289+0000","last_unstale":"2023-10-07T10:45:19.156216+0000","last_undegraded":"2023-10-07T10:45:19.156216+0000","last_fullsized":"2023-10-07T10:45:19.156216+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:16:15.346473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147243+0000","last_change":"2023-10-07T10:45:19.146040+0000","last_active":"2023-10-07T10:45:19.147243+0000","last_peered":"2023-10-07T10:45:19.147243+0000","last_clean":"2023-10-07T10:45:19.147243+0000","last_became_active":"2023-10-07T10:45:19.145533+0000","last_became_peered":"2023-10-07T10:45:19.145533+0000","last_unstale":"2023-10-07T10:45:19.147243+0000","last_undegraded":"2023-10-07T10:45:19.147243+0000","last_fullsized":"2023-10-07T10:45:19.147243+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:02.464065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148359+0000","last_change":"2023-10-07T10:45:19.136919+0000","last_active":"2023-10-07T10:45:19.148359+0000","last_peered":"2023-10-07T10:45:19.148359+0000","last_clean":"2023-10-07T10:45:19.148359+0000","last_became_active":"2023-10-07T10:45:19.136630+0000","last_became_peered":"2023-10-07T10:45:19.136630+0000","last_unstale":"2023-10-07T10:45:19.148359+0000","last_undegraded":"2023-10-07T10:45:19.148359+0000","last_fullsized":"2023-10-07T10:45:19.148359+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:29:55.959931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.159787+0000","last_change":"2023-10-07T10:45:19.158281+0000","last_active":"2023-10-07T10:45:19.159787+0000","last_peered":"2023-10-07T10:45:19.159787+0000","last_clean":"2023-10-07T10:45:19.159787+0000","last_became_active":"2023-10-07T10:45:19.157984+0000","last_became_peered":"2023-10-07T10:45:19.157984+0000","last_unstale":"2023-10-07T10:45:19.159787+0000","last_undegraded":"2023-10-07T10:45:19.159787+0000","last_fullsized":"2023-10-07T10:45:19.159787+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:29.034536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.154629+0000","last_change":"2023-10-07T10:45:19.153620+0000","last_active":"2023-10-07T10:45:19.154629+0000","last_peered":"2023-10-07T10:45:19.154629+0000","last_clean":"2023-10-07T10:45:19.154629+0000","last_became_active":"2023-10-07T10:45:19.153341+0000","last_became_peered":"2023-10-07T10:45:19.153341+0000","last_unstale":"2023-10-07T10:45:19.154629+0000","last_undegraded":"2023-10-07T10:45:19.154629+0000","last_fullsized":"2023-10-07T10:45:19.154629+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:39.283636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151333+0000","last_change":"2023-10-07T10:45:19.148594+0000","last_active":"2023-10-07T10:45:19.151333+0000","last_peered":"2023-10-07T10:45:19.151333+0000","last_clean":"2023-10-07T10:45:19.151333+0000","last_became_active":"2023-10-07T10:45:19.148249+0000","last_became_peered":"2023-10-07T10:45:19.148249+0000","last_unstale":"2023-10-07T10:45:19.151333+0000","last_undegraded":"2023-10-07T10:45:19.151333+0000","last_fullsized":"2023-10-07T10:45:19.151333+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:36:36.084756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146948+0000","last_change":"2023-10-07T10:45:19.135737+0000","last_active":"2023-10-07T10:45:19.146948+0000","last_peered":"2023-10-07T10:45:19.146948+0000","last_clean":"2023-10-07T10:45:19.146948+0000","last_became_active":"2023-10-07T10:45:19.135483+0000","last_became_peered":"2023-10-07T10:45:19.135483+0000","last_unstale":"2023-10-07T10:45:19.146948+0000","last_undegraded":"2023-10-07T10:45:19.146948+0000","last_fullsized":"2023-10-07T10:45:19.146948+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:36.818457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146758+0000","last_change":"2023-10-07T10:45:19.142707+0000","last_active":"2023-10-07T10:45:19.146758+0000","last_peered":"2023-10-07T10:45:19.146758+0000","last_clean":"2023-10-07T10:45:19.146758+0000","last_became_active":"2023-10-07T10:45:19.140090+0000","last_became_peered":"2023-10-07T10:45:19.140090+0000","last_unstale":"2023-10-07T10:45:19.146758+0000","last_undegraded":"2023-10-07T10:45:19.146758+0000","last_fullsized":"2023-10-07T10:45:19.146758+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:56.491619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151366+0000","last_change":"2023-10-07T10:45:19.149067+0000","last_active":"2023-10-07T10:45:19.151366+0000","last_peered":"2023-10-07T10:45:19.151366+0000","last_clean":"2023-10-07T10:45:19.151366+0000","last_became_active":"2023-10-07T10:45:19.148731+0000","last_became_peered":"2023-10-07T10:45:19.148731+0000","last_unstale":"2023-10-07T10:45:19.151366+0000","last_undegraded":"2023-10-07T10:45:19.151366+0000","last_fullsized":"2023-10-07T10:45:19.151366+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:21:28.091881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.158635+0000","last_change":"2023-10-07T10:45:19.156808+0000","last_active":"2023-10-07T10:45:19.158635+0000","last_peered":"2023-10-07T10:45:19.158635+0000","last_clean":"2023-10-07T10:45:19.158635+0000","last_became_active":"2023-10-07T10:45:19.156475+0000","last_became_peered":"2023-10-07T10:45:19.156475+0000","last_unstale":"2023-10-07T10:45:19.158635+0000","last_undegraded":"2023-10-07T10:45:19.158635+0000","last_fullsized":"2023-10-07T10:45:19.158635+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:43:27.479906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151989+0000","last_change":"2023-10-07T10:45:19.144447+0000","last_active":"2023-10-07T10:45:19.151989+0000","last_peered":"2023-10-07T10:45:19.151989+0000","last_clean":"2023-10-07T10:45:19.151989+0000","last_became_active":"2023-10-07T10:45:19.144183+0000","last_became_peered":"2023-10-07T10:45:19.144183+0000","last_unstale":"2023-10-07T10:45:19.151989+0000","last_undegraded":"2023-10-07T10:45:19.151989+0000","last_fullsized":"2023-10-07T10:45:19.151989+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:33:48.518165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.157915+0000","last_change":"2023-10-07T10:45:19.155544+0000","last_active":"2023-10-07T10:45:19.157915+0000","last_peered":"2023-10-07T10:45:19.157915+0000","last_clean":"2023-10-07T10:45:19.157915+0000","last_became_active":"2023-10-07T10:45:19.155229+0000","last_became_peered":"2023-10-07T10:45:19.155229+0000","last_unstale":"2023-10-07T10:45:19.157915+0000","last_undegraded":"2023-10-07T10:45:19.157915+0000","last_fullsized":"2023-10-07T10:45:19.157915+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:26:37.783382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143664+0000","last_change":"2023-10-07T10:45:19.137326+0000","last_active":"2023-10-07T10:45:19.143664+0000","last_peered":"2023-10-07T10:45:19.143664+0000","last_clean":"2023-10-07T10:45:19.143664+0000","last_became_active":"2023-10-07T10:45:19.137030+0000","last_became_peered":"2023-10-07T10:45:19.137030+0000","last_unstale":"2023-10-07T10:45:19.143664+0000","last_undegraded":"2023-10-07T10:45:19.143664+0000","last_fullsized":"2023-10-07T10:45:19.143664+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:46:18.232666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149361+0000","last_change":"2023-10-07T10:45:19.141562+0000","last_active":"2023-10-07T10:45:19.149361+0000","last_peered":"2023-10-07T10:45:19.149361+0000","last_clean":"2023-10-07T10:45:19.149361+0000","last_became_active":"2023-10-07T10:45:19.141147+0000","last_became_peered":"2023-10-07T10:45:19.141147+0000","last_unstale":"2023-10-07T10:45:19.149361+0000","last_undegraded":"2023-10-07T10:45:19.149361+0000","last_fullsized":"2023-10-07T10:45:19.149361+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:48:51.721239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147036+0000","last_change":"2023-10-07T10:45:19.144266+0000","last_active":"2023-10-07T10:45:19.147036+0000","last_peered":"2023-10-07T10:45:19.147036+0000","last_clean":"2023-10-07T10:45:19.147036+0000","last_became_active":"2023-10-07T10:45:19.143943+0000","last_became_peered":"2023-10-07T10:45:19.143943+0000","last_unstale":"2023-10-07T10:45:19.147036+0000","last_undegraded":"2023-10-07T10:45:19.147036+0000","last_fullsized":"2023-10-07T10:45:19.147036+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:10:46.792904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141885+0000","last_change":"2023-10-07T10:45:19.138807+0000","last_active":"2023-10-07T10:45:19.141885+0000","last_peered":"2023-10-07T10:45:19.141885+0000","last_clean":"2023-10-07T10:45:19.141885+0000","last_became_active":"2023-10-07T10:45:19.138480+0000","last_became_peered":"2023-10-07T10:45:19.138480+0000","last_unstale":"2023-10-07T10:45:19.141885+0000","last_undegraded":"2023-10-07T10:45:19.141885+0000","last_fullsized":"2023-10-07T10:45:19.141885+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:48:13.786036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.158160+0000","last_change":"2023-10-07T10:45:19.156072+0000","last_active":"2023-10-07T10:45:19.158160+0000","last_peered":"2023-10-07T10:45:19.158160+0000","last_clean":"2023-10-07T10:45:19.158160+0000","last_became_active":"2023-10-07T10:45:19.155793+0000","last_became_peered":"2023-10-07T10:45:19.155793+0000","last_unstale":"2023-10-07T10:45:19.158160+0000","last_undegraded":"2023-10-07T10:45:19.158160+0000","last_fullsized":"2023-10-07T10:45:19.158160+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:12:01.286114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146978+0000","last_change":"2023-10-07T10:45:19.140218+0000","last_active":"2023-10-07T10:45:19.146978+0000","last_peered":"2023-10-07T10:45:19.146978+0000","last_clean":"2023-10-07T10:45:19.146978+0000","last_became_active":"2023-10-07T10:45:19.139954+0000","last_became_peered":"2023-10-07T10:45:19.139954+0000","last_unstale":"2023-10-07T10:45:19.146978+0000","last_undegraded":"2023-10-07T10:45:19.146978+0000","last_fullsized":"2023-10-07T10:45:19.146978+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:46:12.190595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"164'4","reported_seq":14,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149563+0000","last_change":"2023-10-07T10:45:18.142989+0000","last_active":"2023-10-07T10:45:19.149563+0000","last_peered":"2023-10-07T10:45:19.149563+0000","last_clean":"2023-10-07T10:45:19.149563+0000","last_became_active":"2023-10-07T10:45:18.142720+0000","last_became_peered":"2023-10-07T10:45:18.142720+0000","last_unstale":"2023-10-07T10:45:19.149563+0000","last_undegraded":"2023-10-07T10:45:19.149563+0000","last_fullsized":"2023-10-07T10:45:19.149563+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:18:52.732129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141206+0000","last_change":"2023-10-07T10:45:19.137410+0000","last_active":"2023-10-07T10:45:19.141206+0000","last_peered":"2023-10-07T10:45:19.141206+0000","last_clean":"2023-10-07T10:45:19.141206+0000","last_became_active":"2023-10-07T10:45:19.137106+0000","last_became_peered":"2023-10-07T10:45:19.137106+0000","last_unstale":"2023-10-07T10:45:19.141206+0000","last_undegraded":"2023-10-07T10:45:19.141206+0000","last_fullsized":"2023-10-07T10:45:19.141206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:40:55.053332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149990+0000","last_change":"2023-10-07T10:45:18.140674+0000","last_active":"2023-10-07T10:45:19.149990+0000","last_peered":"2023-10-07T10:45:19.149990+0000","last_clean":"2023-10-07T10:45:19.149990+0000","last_became_active":"2023-10-07T10:45:18.140397+0000","last_became_peered":"2023-10-07T10:45:18.140397+0000","last_unstale":"2023-10-07T10:45:19.149990+0000","last_undegraded":"2023-10-07T10:45:19.149990+0000","last_fullsized":"2023-10-07T10:45:19.149990+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:28:26.440612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140513+0000","last_change":"2023-10-07T10:45:19.137635+0000","last_active":"2023-10-07T10:45:19.140513+0000","last_peered":"2023-10-07T10:45:19.140513+0000","last_clean":"2023-10-07T10:45:19.140513+0000","last_became_active":"2023-10-07T10:45:19.137350+0000","last_became_peered":"2023-10-07T10:45:19.137350+0000","last_unstale":"2023-10-07T10:45:19.140513+0000","last_undegraded":"2023-10-07T10:45:19.140513+0000","last_fullsized":"2023-10-07T10:45:19.140513+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:48.056310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.154496+0000","last_change":"2023-10-07T10:45:18.134101+0000","last_active":"2023-10-07T10:45:19.154496+0000","last_peered":"2023-10-07T10:45:19.154496+0000","last_clean":"2023-10-07T10:45:19.154496+0000","last_became_active":"2023-10-07T10:45:18.133848+0000","last_became_peered":"2023-10-07T10:45:18.133848+0000","last_unstale":"2023-10-07T10:45:19.154496+0000","last_undegraded":"2023-10-07T10:45:19.154496+0000","last_fullsized":"2023-10-07T10:45:19.154496+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:17:47.048150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148416+0000","last_change":"2023-10-07T10:45:19.147540+0000","last_active":"2023-10-07T10:45:19.148416+0000","last_peered":"2023-10-07T10:45:19.148416+0000","last_clean":"2023-10-07T10:45:19.148416+0000","last_became_active":"2023-10-07T10:45:19.145202+0000","last_became_peered":"2023-10-07T10:45:19.145202+0000","last_unstale":"2023-10-07T10:45:19.148416+0000","last_undegraded":"2023-10-07T10:45:19.148416+0000","last_fullsized":"2023-10-07T10:45:19.148416+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:20:40.363427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.155032+0000","last_change":"2023-10-07T10:45:18.135363+0000","last_active":"2023-10-07T10:45:19.155032+0000","last_peered":"2023-10-07T10:45:19.155032+0000","last_clean":"2023-10-07T10:45:19.155032+0000","last_became_active":"2023-10-07T10:45:18.135072+0000","last_became_peered":"2023-10-07T10:45:18.135072+0000","last_unstale":"2023-10-07T10:45:19.155032+0000","last_undegraded":"2023-10-07T10:45:19.155032+0000","last_fullsized":"2023-10-07T10:45:19.155032+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:10.550922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.141840+0000","last_change":"2023-10-07T10:45:19.140106+0000","last_active":"2023-10-07T10:45:19.141840+0000","last_peered":"2023-10-07T10:45:19.141840+0000","last_clean":"2023-10-07T10:45:19.141840+0000","last_became_active":"2023-10-07T10:45:19.139850+0000","last_became_peered":"2023-10-07T10:45:19.139850+0000","last_unstale":"2023-10-07T10:45:19.141840+0000","last_undegraded":"2023-10-07T10:45:19.141840+0000","last_fullsized":"2023-10-07T10:45:19.141840+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:20.315521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.156817+0000","last_change":"2023-10-07T10:45:18.140719+0000","last_active":"2023-10-07T10:45:19.156817+0000","last_peered":"2023-10-07T10:45:19.156817+0000","last_clean":"2023-10-07T10:45:19.156817+0000","last_became_active":"2023-10-07T10:45:18.140454+0000","last_became_peered":"2023-10-07T10:45:18.140454+0000","last_unstale":"2023-10-07T10:45:19.156817+0000","last_undegraded":"2023-10-07T10:45:19.156817+0000","last_fullsized":"2023-10-07T10:45:19.156817+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:02:04.522990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147107+0000","last_change":"2023-10-07T10:45:19.140209+0000","last_active":"2023-10-07T10:45:19.147107+0000","last_peered":"2023-10-07T10:45:19.147107+0000","last_clean":"2023-10-07T10:45:19.147107+0000","last_became_active":"2023-10-07T10:45:19.139894+0000","last_became_peered":"2023-10-07T10:45:19.139894+0000","last_unstale":"2023-10-07T10:45:19.147107+0000","last_undegraded":"2023-10-07T10:45:19.147107+0000","last_fullsized":"2023-10-07T10:45:19.147107+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:15.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149366+0000","last_change":"2023-10-07T10:45:18.137992+0000","last_active":"2023-10-07T10:45:19.149366+0000","last_peered":"2023-10-07T10:45:19.149366+0000","last_clean":"2023-10-07T10:45:19.149366+0000","last_became_active":"2023-10-07T10:45:18.137585+0000","last_became_peered":"2023-10-07T10:45:18.137585+0000","last_unstale":"2023-10-07T10:45:19.149366+0000","last_undegraded":"2023-10-07T10:45:19.149366+0000","last_fullsized":"2023-10-07T10:45:19.149366+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:02:04.522990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145819+0000","last_change":"2023-10-07T10:45:19.144582+0000","last_active":"2023-10-07T10:45:19.145819+0000","last_peered":"2023-10-07T10:45:19.145819+0000","last_clean":"2023-10-07T10:45:19.145819+0000","last_became_active":"2023-10-07T10:45:19.144266+0000","last_became_peered":"2023-10-07T10:45:19.144266+0000","last_unstale":"2023-10-07T10:45:19.145819+0000","last_undegraded":"2023-10-07T10:45:19.145819+0000","last_fullsized":"2023-10-07T10:45:19.145819+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:19:23.485305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.133247+0000","last_change":"2023-10-07T10:45:18.132149+0000","last_active":"2023-10-07T10:45:19.133247+0000","last_peered":"2023-10-07T10:45:19.133247+0000","last_clean":"2023-10-07T10:45:19.133247+0000","last_became_active":"2023-10-07T10:45:18.131876+0000","last_became_peered":"2023-10-07T10:45:18.131876+0000","last_unstale":"2023-10-07T10:45:19.133247+0000","last_undegraded":"2023-10-07T10:45:19.133247+0000","last_fullsized":"2023-10-07T10:45:19.133247+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:17:34.792329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140845+0000","last_change":"2023-10-07T10:45:19.136552+0000","last_active":"2023-10-07T10:45:19.140845+0000","last_peered":"2023-10-07T10:45:19.140845+0000","last_clean":"2023-10-07T10:45:19.140845+0000","last_became_active":"2023-10-07T10:45:19.136290+0000","last_became_peered":"2023-10-07T10:45:19.136290+0000","last_unstale":"2023-10-07T10:45:19.140845+0000","last_undegraded":"2023-10-07T10:45:19.140845+0000","last_fullsized":"2023-10-07T10:45:19.140845+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:00:12.788473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.121462+0000","last_change":"2023-10-07T10:45:18.137921+0000","last_active":"2023-10-07T10:45:19.121462+0000","last_peered":"2023-10-07T10:45:19.121462+0000","last_clean":"2023-10-07T10:45:19.121462+0000","last_became_active":"2023-10-07T10:45:18.137630+0000","last_became_peered":"2023-10-07T10:45:18.137630+0000","last_unstale":"2023-10-07T10:45:19.121462+0000","last_undegraded":"2023-10-07T10:45:19.121462+0000","last_fullsized":"2023-10-07T10:45:19.121462+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:48:50.713079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149608+0000","last_change":"2023-10-07T10:45:19.146791+0000","last_active":"2023-10-07T10:45:19.149608+0000","last_peered":"2023-10-07T10:45:19.149608+0000","last_clean":"2023-10-07T10:45:19.149608+0000","last_became_active":"2023-10-07T10:45:19.146224+0000","last_became_peered":"2023-10-07T10:45:19.146224+0000","last_unstale":"2023-10-07T10:45:19.149608+0000","last_undegraded":"2023-10-07T10:45:19.149608+0000","last_fullsized":"2023-10-07T10:45:19.149608+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:10.279050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.127372+0000","last_change":"2023-10-07T10:45:18.135238+0000","last_active":"2023-10-07T10:45:19.127372+0000","last_peered":"2023-10-07T10:45:19.127372+0000","last_clean":"2023-10-07T10:45:19.127372+0000","last_became_active":"2023-10-07T10:45:18.134954+0000","last_became_peered":"2023-10-07T10:45:18.134954+0000","last_unstale":"2023-10-07T10:45:19.127372+0000","last_undegraded":"2023-10-07T10:45:19.127372+0000","last_fullsized":"2023-10-07T10:45:19.127372+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:08:17.302016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147262+0000","last_change":"2023-10-07T10:45:19.146210+0000","last_active":"2023-10-07T10:45:19.147262+0000","last_peered":"2023-10-07T10:45:19.147262+0000","last_clean":"2023-10-07T10:45:19.147262+0000","last_became_active":"2023-10-07T10:45:19.145803+0000","last_became_peered":"2023-10-07T10:45:19.145803+0000","last_unstale":"2023-10-07T10:45:19.147262+0000","last_undegraded":"2023-10-07T10:45:19.147262+0000","last_fullsized":"2023-10-07T10:45:19.147262+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:53:24.969381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.130723+0000","last_change":"2023-10-07T10:45:18.141044+0000","last_active":"2023-10-07T10:45:19.130723+0000","last_peered":"2023-10-07T10:45:19.130723+0000","last_clean":"2023-10-07T10:45:19.130723+0000","last_became_active":"2023-10-07T10:45:18.140162+0000","last_became_peered":"2023-10-07T10:45:18.140162+0000","last_unstale":"2023-10-07T10:45:19.130723+0000","last_undegraded":"2023-10-07T10:45:19.130723+0000","last_fullsized":"2023-10-07T10:45:19.130723+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:01:05.105032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.155412+0000","last_change":"2023-10-07T10:45:19.152242+0000","last_active":"2023-10-07T10:45:19.155412+0000","last_peered":"2023-10-07T10:45:19.155412+0000","last_clean":"2023-10-07T10:45:19.155412+0000","last_became_active":"2023-10-07T10:45:19.151911+0000","last_became_peered":"2023-10-07T10:45:19.151911+0000","last_unstale":"2023-10-07T10:45:19.155412+0000","last_undegraded":"2023-10-07T10:45:19.155412+0000","last_fullsized":"2023-10-07T10:45:19.155412+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:10:44.279307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"164'6","reported_seq":16,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.160824+0000","last_change":"2023-10-07T10:45:18.134720+0000","last_active":"2023-10-07T10:45:19.160824+0000","last_peered":"2023-10-07T10:45:19.160824+0000","last_clean":"2023-10-07T10:45:19.160824+0000","last_became_active":"2023-10-07T10:45:18.134472+0000","last_became_peered":"2023-10-07T10:45:18.134472+0000","last_unstale":"2023-10-07T10:45:19.160824+0000","last_undegraded":"2023-10-07T10:45:19.160824+0000","last_fullsized":"2023-10-07T10:45:19.160824+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:08:17.302016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148337+0000","last_change":"2023-10-07T10:45:19.137210+0000","last_active":"2023-10-07T10:45:19.148337+0000","last_peered":"2023-10-07T10:45:19.148337+0000","last_clean":"2023-10-07T10:45:19.148337+0000","last_became_active":"2023-10-07T10:45:19.136932+0000","last_became_peered":"2023-10-07T10:45:19.136932+0000","last_unstale":"2023-10-07T10:45:19.148337+0000","last_undegraded":"2023-10-07T10:45:19.148337+0000","last_fullsized":"2023-10-07T10:45:19.148337+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:36.719844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.153415+0000","last_change":"2023-10-07T10:45:18.132585+0000","last_active":"2023-10-07T10:45:19.153415+0000","last_peered":"2023-10-07T10:45:19.153415+0000","last_clean":"2023-10-07T10:45:19.153415+0000","last_became_active":"2023-10-07T10:45:18.131693+0000","last_became_peered":"2023-10-07T10:45:18.131693+0000","last_unstale":"2023-10-07T10:45:19.153415+0000","last_undegraded":"2023-10-07T10:45:19.153415+0000","last_fullsized":"2023-10-07T10:45:19.153415+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:02:28.026376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.145687+0000","last_change":"2023-10-07T10:45:19.144255+0000","last_active":"2023-10-07T10:45:19.145687+0000","last_peered":"2023-10-07T10:45:19.145687+0000","last_clean":"2023-10-07T10:45:19.145687+0000","last_became_active":"2023-10-07T10:45:19.143928+0000","last_became_peered":"2023-10-07T10:45:19.143928+0000","last_unstale":"2023-10-07T10:45:19.145687+0000","last_undegraded":"2023-10-07T10:45:19.145687+0000","last_fullsized":"2023-10-07T10:45:19.145687+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:14:40.088604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129740+0000","last_change":"2023-10-07T10:45:18.129727+0000","last_active":"2023-10-07T10:45:19.129740+0000","last_peered":"2023-10-07T10:45:19.129740+0000","last_clean":"2023-10-07T10:45:19.129740+0000","last_became_active":"2023-10-07T10:45:18.129381+0000","last_became_peered":"2023-10-07T10:45:18.129381+0000","last_unstale":"2023-10-07T10:45:19.129740+0000","last_undegraded":"2023-10-07T10:45:19.129740+0000","last_fullsized":"2023-10-07T10:45:19.129740+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:42:51.132950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.137885+0000","last_change":"2023-10-07T10:45:19.133782+0000","last_active":"2023-10-07T10:45:19.137885+0000","last_peered":"2023-10-07T10:45:19.137885+0000","last_clean":"2023-10-07T10:45:19.137885+0000","last_became_active":"2023-10-07T10:45:19.132846+0000","last_became_peered":"2023-10-07T10:45:19.132846+0000","last_unstale":"2023-10-07T10:45:19.137885+0000","last_undegraded":"2023-10-07T10:45:19.137885+0000","last_fullsized":"2023-10-07T10:45:19.137885+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:16:15.346473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.121628+0000","last_change":"2023-10-07T10:45:18.133406+0000","last_active":"2023-10-07T10:45:19.121628+0000","last_peered":"2023-10-07T10:45:19.121628+0000","last_clean":"2023-10-07T10:45:19.121628+0000","last_became_active":"2023-10-07T10:45:18.133149+0000","last_became_peered":"2023-10-07T10:45:18.133149+0000","last_unstale":"2023-10-07T10:45:19.121628+0000","last_undegraded":"2023-10-07T10:45:19.121628+0000","last_fullsized":"2023-10-07T10:45:19.121628+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:00:11.780313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148125+0000","last_change":"2023-10-07T10:45:19.135324+0000","last_active":"2023-10-07T10:45:19.148125+0000","last_peered":"2023-10-07T10:45:19.148125+0000","last_clean":"2023-10-07T10:45:19.148125+0000","last_became_active":"2023-10-07T10:45:19.135018+0000","last_became_peered":"2023-10-07T10:45:19.135018+0000","last_unstale":"2023-10-07T10:45:19.148125+0000","last_undegraded":"2023-10-07T10:45:19.148125+0000","last_fullsized":"2023-10-07T10:45:19.148125+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:04.850331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"164'8","reported_seq":18,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.158447+0000","last_change":"2023-10-07T10:45:18.140162+0000","last_active":"2023-10-07T10:45:19.158447+0000","last_peered":"2023-10-07T10:45:19.158447+0000","last_clean":"2023-10-07T10:45:19.158447+0000","last_became_active":"2023-10-07T10:45:18.139873+0000","last_became_peered":"2023-10-07T10:45:18.139873+0000","last_unstale":"2023-10-07T10:45:19.158447+0000","last_undegraded":"2023-10-07T10:45:19.158447+0000","last_fullsized":"2023-10-07T10:45:19.158447+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:21:40.546535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.153599+0000","last_change":"2023-10-07T10:45:19.152777+0000","last_active":"2023-10-07T10:45:19.153599+0000","last_peered":"2023-10-07T10:45:19.153599+0000","last_clean":"2023-10-07T10:45:19.153599+0000","last_became_active":"2023-10-07T10:45:19.152473+0000","last_became_peered":"2023-10-07T10:45:19.152473+0000","last_unstale":"2023-10-07T10:45:19.153599+0000","last_undegraded":"2023-10-07T10:45:19.153599+0000","last_fullsized":"2023-10-07T10:45:19.153599+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:29:51.071208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150644+0000","last_change":"2023-10-07T10:45:18.128406+0000","last_active":"2023-10-07T10:45:19.150644+0000","last_peered":"2023-10-07T10:45:19.150644+0000","last_clean":"2023-10-07T10:45:19.150644+0000","last_became_active":"2023-10-07T10:45:18.128118+0000","last_became_peered":"2023-10-07T10:45:18.128118+0000","last_unstale":"2023-10-07T10:45:19.150644+0000","last_undegraded":"2023-10-07T10:45:19.150644+0000","last_fullsized":"2023-10-07T10:45:19.150644+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T10:56:15.480123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.154517+0000","last_change":"2023-10-07T10:45:19.152009+0000","last_active":"2023-10-07T10:45:19.154517+0000","last_peered":"2023-10-07T10:45:19.154517+0000","last_clean":"2023-10-07T10:45:19.154517+0000","last_became_active":"2023-10-07T10:45:19.151697+0000","last_became_peered":"2023-10-07T10:45:19.151697+0000","last_unstale":"2023-10-07T10:45:19.154517+0000","last_undegraded":"2023-10-07T10:45:19.154517+0000","last_fullsized":"2023-10-07T10:45:19.154517+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:38.666098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.153478+0000","last_change":"2023-10-07T10:45:18.136247+0000","last_active":"2023-10-07T10:45:19.153478+0000","last_peered":"2023-10-07T10:45:19.153478+0000","last_clean":"2023-10-07T10:45:19.153478+0000","last_became_active":"2023-10-07T10:45:18.135963+0000","last_became_peered":"2023-10-07T10:45:18.135963+0000","last_unstale":"2023-10-07T10:45:19.153478+0000","last_undegraded":"2023-10-07T10:45:19.153478+0000","last_fullsized":"2023-10-07T10:45:19.153478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:35.810298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148162+0000","last_change":"2023-10-07T10:45:19.135505+0000","last_active":"2023-10-07T10:45:19.148162+0000","last_peered":"2023-10-07T10:45:19.148162+0000","last_clean":"2023-10-07T10:45:19.148162+0000","last_became_active":"2023-10-07T10:45:19.135133+0000","last_became_peered":"2023-10-07T10:45:19.135133+0000","last_unstale":"2023-10-07T10:45:19.148162+0000","last_undegraded":"2023-10-07T10:45:19.148162+0000","last_fullsized":"2023-10-07T10:45:19.148162+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:56.491619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"164'1","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.156684+0000","last_change":"2023-10-07T10:45:18.140506+0000","last_active":"2023-10-07T10:45:19.156684+0000","last_peered":"2023-10-07T10:45:19.156684+0000","last_clean":"2023-10-07T10:45:19.156684+0000","last_became_active":"2023-10-07T10:45:18.140173+0000","last_became_peered":"2023-10-07T10:45:18.140173+0000","last_unstale":"2023-10-07T10:45:19.156684+0000","last_undegraded":"2023-10-07T10:45:19.156684+0000","last_fullsized":"2023-10-07T10:45:19.156684+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:00:35.241523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148197+0000","last_change":"2023-10-07T10:45:19.141719+0000","last_active":"2023-10-07T10:45:19.148197+0000","last_peered":"2023-10-07T10:45:19.148197+0000","last_clean":"2023-10-07T10:45:19.148197+0000","last_became_active":"2023-10-07T10:45:19.141374+0000","last_became_peered":"2023-10-07T10:45:19.141374+0000","last_unstale":"2023-10-07T10:45:19.148197+0000","last_undegraded":"2023-10-07T10:45:19.148197+0000","last_fullsized":"2023-10-07T10:45:19.148197+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:32:04.300310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129491+0000","last_change":"2023-10-07T10:45:18.135852+0000","last_active":"2023-10-07T10:45:19.129491+0000","last_peered":"2023-10-07T10:45:19.129491+0000","last_clean":"2023-10-07T10:45:19.129491+0000","last_became_active":"2023-10-07T10:45:18.135583+0000","last_became_peered":"2023-10-07T10:45:18.135583+0000","last_unstale":"2023-10-07T10:45:19.129491+0000","last_undegraded":"2023-10-07T10:45:19.129491+0000","last_fullsized":"2023-10-07T10:45:19.129491+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:34:32.011131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.147054+0000","last_change":"2023-10-07T10:45:19.144334+0000","last_active":"2023-10-07T10:45:19.147054+0000","last_peered":"2023-10-07T10:45:19.147054+0000","last_clean":"2023-10-07T10:45:19.147054+0000","last_became_active":"2023-10-07T10:45:19.144028+0000","last_became_peered":"2023-10-07T10:45:19.144028+0000","last_unstale":"2023-10-07T10:45:19.147054+0000","last_undegraded":"2023-10-07T10:45:19.147054+0000","last_fullsized":"2023-10-07T10:45:19.147054+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:36.719844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150619+0000","last_change":"2023-10-07T10:45:18.131596+0000","last_active":"2023-10-07T10:45:19.150619+0000","last_peered":"2023-10-07T10:45:19.150619+0000","last_clean":"2023-10-07T10:45:19.150619+0000","last_became_active":"2023-10-07T10:45:18.131338+0000","last_became_peered":"2023-10-07T10:45:18.131338+0000","last_unstale":"2023-10-07T10:45:19.150619+0000","last_undegraded":"2023-10-07T10:45:19.150619+0000","last_fullsized":"2023-10-07T10:45:19.150619+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:35.810298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149650+0000","last_change":"2023-10-07T10:45:19.149014+0000","last_active":"2023-10-07T10:45:19.149650+0000","last_peered":"2023-10-07T10:45:19.149650+0000","last_clean":"2023-10-07T10:45:19.149650+0000","last_became_active":"2023-10-07T10:45:19.148716+0000","last_became_peered":"2023-10-07T10:45:19.148716+0000","last_unstale":"2023-10-07T10:45:19.149650+0000","last_undegraded":"2023-10-07T10:45:19.149650+0000","last_fullsized":"2023-10-07T10:45:19.149650+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:52.338610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128929+0000","last_change":"2023-10-07T10:45:18.137178+0000","last_active":"2023-10-07T10:45:19.128929+0000","last_peered":"2023-10-07T10:45:19.128929+0000","last_clean":"2023-10-07T10:45:19.128929+0000","last_became_active":"2023-10-07T10:45:18.136546+0000","last_became_peered":"2023-10-07T10:45:18.136546+0000","last_unstale":"2023-10-07T10:45:19.128929+0000","last_undegraded":"2023-10-07T10:45:19.128929+0000","last_fullsized":"2023-10-07T10:45:19.128929+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:37:20.378578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146731+0000","last_change":"2023-10-07T10:45:19.139633+0000","last_active":"2023-10-07T10:45:19.146731+0000","last_peered":"2023-10-07T10:45:19.146731+0000","last_clean":"2023-10-07T10:45:19.146731+0000","last_became_active":"2023-10-07T10:45:19.139291+0000","last_became_peered":"2023-10-07T10:45:19.139291+0000","last_unstale":"2023-10-07T10:45:19.146731+0000","last_undegraded":"2023-10-07T10:45:19.146731+0000","last_fullsized":"2023-10-07T10:45:19.146731+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:40.850441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128809+0000","last_change":"2023-10-07T10:45:18.136369+0000","last_active":"2023-10-07T10:45:19.128809+0000","last_peered":"2023-10-07T10:45:19.128809+0000","last_clean":"2023-10-07T10:45:19.128809+0000","last_became_active":"2023-10-07T10:45:18.136047+0000","last_became_peered":"2023-10-07T10:45:18.136047+0000","last_unstale":"2023-10-07T10:45:19.128809+0000","last_undegraded":"2023-10-07T10:45:19.128809+0000","last_fullsized":"2023-10-07T10:45:19.128809+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:34:37.975907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143248+0000","last_change":"2023-10-07T10:45:19.136207+0000","last_active":"2023-10-07T10:45:19.143248+0000","last_peered":"2023-10-07T10:45:19.143248+0000","last_clean":"2023-10-07T10:45:19.143248+0000","last_became_active":"2023-10-07T10:45:19.135718+0000","last_became_peered":"2023-10-07T10:45:19.135718+0000","last_unstale":"2023-10-07T10:45:19.143248+0000","last_undegraded":"2023-10-07T10:45:19.143248+0000","last_fullsized":"2023-10-07T10:45:19.143248+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:10.056741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128872+0000","last_change":"2023-10-07T10:45:18.133723+0000","last_active":"2023-10-07T10:45:19.128872+0000","last_peered":"2023-10-07T10:45:19.128872+0000","last_clean":"2023-10-07T10:45:19.128872+0000","last_became_active":"2023-10-07T10:45:18.132925+0000","last_became_peered":"2023-10-07T10:45:18.132925+0000","last_unstale":"2023-10-07T10:45:19.128872+0000","last_undegraded":"2023-10-07T10:45:19.128872+0000","last_fullsized":"2023-10-07T10:45:19.128872+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:01:56.666407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.149589+0000","last_change":"2023-10-07T10:45:19.146673+0000","last_active":"2023-10-07T10:45:19.149589+0000","last_peered":"2023-10-07T10:45:19.149589+0000","last_clean":"2023-10-07T10:45:19.149589+0000","last_became_active":"2023-10-07T10:45:19.145598+0000","last_became_peered":"2023-10-07T10:45:19.145598+0000","last_unstale":"2023-10-07T10:45:19.149589+0000","last_undegraded":"2023-10-07T10:45:19.149589+0000","last_fullsized":"2023-10-07T10:45:19.149589+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:39:38.666098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"164'2","reported_seq":12,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150706+0000","last_change":"2023-10-07T10:45:18.141173+0000","last_active":"2023-10-07T10:45:19.150706+0000","last_peered":"2023-10-07T10:45:19.150706+0000","last_clean":"2023-10-07T10:45:19.150706+0000","last_became_active":"2023-10-07T10:45:18.140707+0000","last_became_peered":"2023-10-07T10:45:18.140707+0000","last_unstale":"2023-10-07T10:45:19.150706+0000","last_undegraded":"2023-10-07T10:45:19.150706+0000","last_fullsized":"2023-10-07T10:45:19.150706+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:26:53.011975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.148284+0000","last_change":"2023-10-07T10:45:19.136453+0000","last_active":"2023-10-07T10:45:19.148284+0000","last_peered":"2023-10-07T10:45:19.148284+0000","last_clean":"2023-10-07T10:45:19.148284+0000","last_became_active":"2023-10-07T10:45:19.136143+0000","last_became_peered":"2023-10-07T10:45:19.136143+0000","last_unstale":"2023-10-07T10:45:19.148284+0000","last_undegraded":"2023-10-07T10:45:19.148284+0000","last_fullsized":"2023-10-07T10:45:19.148284+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:10:46.792904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.121545+0000","last_change":"2023-10-07T10:45:18.142439+0000","last_active":"2023-10-07T10:45:19.121545+0000","last_peered":"2023-10-07T10:45:19.121545+0000","last_clean":"2023-10-07T10:45:19.121545+0000","last_became_active":"2023-10-07T10:45:18.141989+0000","last_became_peered":"2023-10-07T10:45:18.141989+0000","last_unstale":"2023-10-07T10:45:19.121545+0000","last_undegraded":"2023-10-07T10:45:19.121545+0000","last_fullsized":"2023-10-07T10:45:19.121545+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:10:43.271147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.151104+0000","last_change":"2023-10-07T10:45:19.149498+0000","last_active":"2023-10-07T10:45:19.151104+0000","last_peered":"2023-10-07T10:45:19.151104+0000","last_clean":"2023-10-07T10:45:19.151104+0000","last_became_active":"2023-10-07T10:45:19.149143+0000","last_became_peered":"2023-10-07T10:45:19.149143+0000","last_unstale":"2023-10-07T10:45:19.151104+0000","last_undegraded":"2023-10-07T10:45:19.151104+0000","last_fullsized":"2023-10-07T10:45:19.151104+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:01:27.595946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.123379+0000","last_change":"2023-10-07T10:45:18.135185+0000","last_active":"2023-10-07T10:45:19.123379+0000","last_peered":"2023-10-07T10:45:19.123379+0000","last_clean":"2023-10-07T10:45:19.123379+0000","last_became_active":"2023-10-07T10:45:18.134932+0000","last_became_peered":"2023-10-07T10:45:18.134932+0000","last_unstale":"2023-10-07T10:45:19.123379+0000","last_undegraded":"2023-10-07T10:45:19.123379+0000","last_fullsized":"2023-10-07T10:45:19.123379+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:43:26.471746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140477+0000","last_change":"2023-10-07T10:45:19.136486+0000","last_active":"2023-10-07T10:45:19.140477+0000","last_peered":"2023-10-07T10:45:19.140477+0000","last_clean":"2023-10-07T10:45:19.140477+0000","last_became_active":"2023-10-07T10:45:19.136142+0000","last_became_peered":"2023-10-07T10:45:19.136142+0000","last_unstale":"2023-10-07T10:45:19.140477+0000","last_undegraded":"2023-10-07T10:45:19.140477+0000","last_fullsized":"2023-10-07T10:45:19.140477+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:11.559082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.122215+0000","last_change":"2023-10-07T10:45:18.135993+0000","last_active":"2023-10-07T10:45:19.122215+0000","last_peered":"2023-10-07T10:45:19.122215+0000","last_clean":"2023-10-07T10:45:19.122215+0000","last_became_active":"2023-10-07T10:45:18.135711+0000","last_became_peered":"2023-10-07T10:45:18.135711+0000","last_unstale":"2023-10-07T10:45:19.122215+0000","last_undegraded":"2023-10-07T10:45:19.122215+0000","last_fullsized":"2023-10-07T10:45:19.122215+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:09.270891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.157861+0000","last_change":"2023-10-07T10:45:19.155851+0000","last_active":"2023-10-07T10:45:19.157861+0000","last_peered":"2023-10-07T10:45:19.157861+0000","last_clean":"2023-10-07T10:45:19.157861+0000","last_became_active":"2023-10-07T10:45:19.155583+0000","last_became_peered":"2023-10-07T10:45:19.155583+0000","last_unstale":"2023-10-07T10:45:19.157861+0000","last_undegraded":"2023-10-07T10:45:19.157861+0000","last_fullsized":"2023-10-07T10:45:19.157861+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:10.279050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128304+0000","last_change":"2023-10-07T10:45:18.128985+0000","last_active":"2023-10-07T10:45:19.128304+0000","last_peered":"2023-10-07T10:45:19.128304+0000","last_clean":"2023-10-07T10:45:19.128304+0000","last_became_active":"2023-10-07T10:45:18.128702+0000","last_became_peered":"2023-10-07T10:45:18.128702+0000","last_unstale":"2023-10-07T10:45:19.128304+0000","last_undegraded":"2023-10-07T10:45:19.128304+0000","last_fullsized":"2023-10-07T10:45:19.128304+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:34:32.011131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.140886+0000","last_change":"2023-10-07T10:45:19.137156+0000","last_active":"2023-10-07T10:45:19.140886+0000","last_peered":"2023-10-07T10:45:19.140886+0000","last_clean":"2023-10-07T10:45:19.140886+0000","last_became_active":"2023-10-07T10:45:19.136892+0000","last_became_peered":"2023-10-07T10:45:19.136892+0000","last_unstale":"2023-10-07T10:45:19.140886+0000","last_undegraded":"2023-10-07T10:45:19.140886+0000","last_fullsized":"2023-10-07T10:45:19.140886+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:10:44.279307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129036+0000","last_change":"2023-10-07T10:45:18.137366+0000","last_active":"2023-10-07T10:45:19.129036+0000","last_peered":"2023-10-07T10:45:19.129036+0000","last_clean":"2023-10-07T10:45:19.129036+0000","last_became_active":"2023-10-07T10:45:18.136843+0000","last_became_peered":"2023-10-07T10:45:18.136843+0000","last_unstale":"2023-10-07T10:45:19.129036+0000","last_undegraded":"2023-10-07T10:45:19.129036+0000","last_fullsized":"2023-10-07T10:45:19.129036+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:07:50.965052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.146741+0000","last_change":"2023-10-07T10:45:19.143812+0000","last_active":"2023-10-07T10:45:19.146741+0000","last_peered":"2023-10-07T10:45:19.146741+0000","last_clean":"2023-10-07T10:45:19.146741+0000","last_became_active":"2023-10-07T10:45:19.143556+0000","last_became_peered":"2023-10-07T10:45:19.143556+0000","last_unstale":"2023-10-07T10:45:19.146741+0000","last_undegraded":"2023-10-07T10:45:19.146741+0000","last_fullsized":"2023-10-07T10:45:19.146741+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:18.999677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.129731+0000","last_change":"2023-10-07T10:45:18.134209+0000","last_active":"2023-10-07T10:45:19.129731+0000","last_peered":"2023-10-07T10:45:19.129731+0000","last_clean":"2023-10-07T10:45:19.129731+0000","last_became_active":"2023-10-07T10:45:18.133896+0000","last_became_peered":"2023-10-07T10:45:18.133896+0000","last_unstale":"2023-10-07T10:45:19.129731+0000","last_undegraded":"2023-10-07T10:45:19.129731+0000","last_fullsized":"2023-10-07T10:45:19.129731+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:31:30.567674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150397+0000","last_change":"2023-10-07T10:45:19.147304+0000","last_active":"2023-10-07T10:45:19.150397+0000","last_peered":"2023-10-07T10:45:19.150397+0000","last_clean":"2023-10-07T10:45:19.150397+0000","last_became_active":"2023-10-07T10:45:19.146985+0000","last_became_peered":"2023-10-07T10:45:19.146985+0000","last_unstale":"2023-10-07T10:45:19.150397+0000","last_undegraded":"2023-10-07T10:45:19.150397+0000","last_fullsized":"2023-10-07T10:45:19.150397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:43:27.479906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.126559+0000","last_change":"2023-10-07T10:45:18.137722+0000","last_active":"2023-10-07T10:45:19.126559+0000","last_peered":"2023-10-07T10:45:19.126559+0000","last_clean":"2023-10-07T10:45:19.126559+0000","last_became_active":"2023-10-07T10:45:18.137452+0000","last_became_peered":"2023-10-07T10:45:18.137452+0000","last_unstale":"2023-10-07T10:45:19.126559+0000","last_undegraded":"2023-10-07T10:45:19.126559+0000","last_fullsized":"2023-10-07T10:45:19.126559+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:28:39.842281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.143640+0000","last_change":"2023-10-07T10:45:19.136644+0000","last_active":"2023-10-07T10:45:19.143640+0000","last_peered":"2023-10-07T10:45:19.143640+0000","last_clean":"2023-10-07T10:45:19.143640+0000","last_became_active":"2023-10-07T10:45:19.136048+0000","last_became_peered":"2023-10-07T10:45:19.136048+0000","last_unstale":"2023-10-07T10:45:19.143640+0000","last_undegraded":"2023-10-07T10:45:19.143640+0000","last_fullsized":"2023-10-07T10:45:19.143640+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:18.110220+0000","last_clean_scrub_stamp":"2023-10-07T10:45:18.110220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:51:04.850331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"164'3","reported_seq":13,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.150236+0000","last_change":"2023-10-07T10:45:18.132966+0000","last_active":"2023-10-07T10:45:19.150236+0000","last_peered":"2023-10-07T10:45:19.150236+0000","last_clean":"2023-10-07T10:45:19.150236+0000","last_became_active":"2023-10-07T10:45:18.132660+0000","last_became_peered":"2023-10-07T10:45:18.132660+0000","last_unstale":"2023-10-07T10:45:19.150236+0000","last_undegraded":"2023-10-07T10:45:19.150236+0000","last_fullsized":"2023-10-07T10:45:19.150236+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T10:45:17.102060+0000","last_clean_scrub_stamp":"2023-10-07T10:45:17.102060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:13:31.762379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"154'293","reported_seq":910,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-07T10:45:19.128705+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T10:45:19.128705+0000","last_peered":"2023-10-07T10:45:19.128705+0000","last_clean":"2023-10-07T10:45:19.128705+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T10:45:19.128705+0000","last_undegraded":"2023-10-07T10:45:19.128705+0000","last_fullsized":"2023-10-07T10:45:19.128705+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477641,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308072,"kb_used_data":2592,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063768,"statfs":{"total":96636764160,"available":96321298432,"internally_reserved":0,"allocated":2654208,"data_stored":1597511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.95699999999999996,"5min":1.018,"15min":1.018},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.028,"15min":1.028},"last":0.872}]},{"osd":1,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.748,"15min":0.752},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.432},"max":{"1min":0.90900000000000003,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.75900000000000001},"min":{"1min":0.56999999999999995,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.995,"5min":1.119,"15min":1.119},"last":0.76100000000000001}]},{"osd":2,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88200000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78000000000000003,"15min":0.77100000000000002},"min":{"1min":0.65900000000000003,"5min":0.55900000000000005,"15min":0.45400000000000001},"max":{"1min":1.002,"5min":1.147,"15min":1.147},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.746,"15min":0.73999999999999999},"min":{"1min":0.51600000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.94199999999999995,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.59399999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93899999999999995,"5min":1.161,"15min":1.161},"last":0.69799999999999995}]},{"osd":4,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.47699999999999998,"5min":0.44700000000000001,"15min":0.36399999999999999},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.92700000000000005,"5min":1.069,"15min":1.069},"last":1.161}]},{"osd":5,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.83899999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.70799999999999996,"15min":0.69599999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.96399999999999997}]},{"osd":6,"last update":"Sat Oct 7 10:44:26 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.746,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71199999999999997},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86699999999999999,"5min":1.131,"15min":1.131},"last":0.89000000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311580,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060260,"statfs":{"total":96636764160,"available":96317706240,"internally_reserved":0,"allocated":2052096,"data_stored":999957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":0.75900000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90700000000000003,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.54400000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92600000000000005,"5min":1.052,"15min":1.073},"last":0.755}]},{"osd":1,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.75800000000000001,"15min":0.754},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48499999999999999},"max":{"1min":0.874,"5min":1.008,"15min":1.0600000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.50700000000000001},"max":{"1min":0.89000000000000001,"5min":1.034,"15min":1.0389999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46100000000000002},"max":{"1min":1.018,"5min":1.0289999999999999,"15min":1.3380000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59499999999999997,"5min":0.54900000000000004,"15min":0.46500000000000002},"max":{"1min":0.95999999999999996,"5min":1.03,"15min":1.367},"last":0.78400000000000003}]},{"osd":3,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.78200000000000003,"15min":0.78500000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.51100000000000001},"max":{"1min":0.94599999999999995,"5min":1.044,"15min":1.0900000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":1.0289999999999999,"15min":1.1040000000000001},"last":0.69099999999999995}]},{"osd":4,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.621},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.40600000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0609999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.499,"5min":0.42899999999999999,"15min":0.40999999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0900000000000001},"last":0.52300000000000002}]},{"osd":7,"last update":"Sat Oct 7 10:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.52000000000000002,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.1060000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76700000000000002,"15min":0.77000000000000002},"min":{"1min":0.57999999999999996,"5min":0.57899999999999996,"15min":0.53600000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.127},"last":0.60499999999999998}]}]},{"osd":1,"up_from":14,"seq":60129543047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307488,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064352,"statfs":{"total":96636764160,"available":96321896448,"internally_reserved":0,"allocated":2056192,"data_stored":1007073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28899999999999998},"max":{"1min":0.59999999999999998,"5min":0.76000000000000001,"15min":0.83099999999999996},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57499999999999996,"15min":0.58799999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.40500000000000003},"max":{"1min":0.65900000000000003,"5min":0.76900000000000002,"15min":0.84199999999999997},"last":0.502}]},{"osd":2,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.62,"5min":0.74399999999999999,"15min":0.81699999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.82499999999999996,"15min":0.89000000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41999999999999998},"max":{"1min":0.77600000000000002,"5min":0.88400000000000001,"15min":0.90400000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":0.84999999999999998,"15min":0.93100000000000005},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56999999999999995},"max":{"1min":1.0660000000000001,"5min":1.1000000000000001,"15min":1.1719999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79100000000000004,"15min":0.80200000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.85099999999999998}]},{"osd":5,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1220000000000001,"5min":1.196,"15min":1.196},"last":0.747},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.81200000000000006,"15min":0.82199999999999995},"min":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"max":{"1min":1.1060000000000001,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78600000000000003}]},{"osd":6,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.86399999999999999},"min":{"1min":0.66200000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2050000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.83399999999999996,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.173,"5min":1.173,"15min":1.196},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Oct 7 10:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.85699999999999998,"15min":0.875},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.143,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.88200000000000001,"15min":0.88800000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.60799999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.76900000000000002}]}]},{"osd":0,"up_from":14,"seq":60129543047,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308080,"kb_used_data":2600,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063760,"statfs":{"total":96636764160,"available":96321290240,"internally_reserved":0,"allocated":2662400,"data_stored":1598457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.61099999999999999,"5min":0.751,"15min":0.76400000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":0.82899999999999996},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75,"5min":0.88900000000000001,"15min":0.92800000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.68400000000000005,"5min":0.87,"15min":0.87},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70899999999999996,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.435},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.80600000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81599999999999995,"15min":0.79700000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.499},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.63},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85499999999999998,"15min":0.83799999999999997},"min":{"1min":0.64700000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0960000000000001,"5min":1.194,"15min":1.194},"last":0.753}]},{"osd":5,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.077,"5min":1.278,"15min":1.278},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":1.0229999999999999,"5min":1.206,"15min":1.206},"last":0.73499999999999999}]},{"osd":6,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86199999999999999,"15min":0.84699999999999998},"min":{"1min":0.67100000000000004,"5min":0.60599999999999998,"15min":0.59499999999999997},"max":{"1min":1.0369999999999999,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85799999999999998,"15min":0.85099999999999998},"min":{"1min":0.57799999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"max":{"1min":1.1579999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.81699999999999995}]},{"osd":7,"last update":"Sat Oct 7 10:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.89300000000000002,"15min":0.872},"min":{"1min":0.627,"5min":0.627,"15min":0.59899999999999998},"max":{"1min":1.131,"5min":1.173,"15min":1.198},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.875,"15min":0.86599999999999999},"min":{"1min":0.63100000000000001,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":1.198,"5min":1.198,"15min":1.234},"last":0.77100000000000002}]}]},{"osd":2,"up_from":14,"seq":60129543048,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311584,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060256,"statfs":{"total":96636764160,"available":96317702144,"internally_reserved":0,"allocated":2056192,"data_stored":1001417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48499999999999999,"15min":0.47399999999999998},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":0.626,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.33400000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.46600000000000003}]},{"osd":1,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":1.369,"15min":1.369},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.44500000000000001},"max":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.53500000000000003}]},{"osd":3,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.59099999999999997,"15min":0.55000000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.0490000000000004,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.73499999999999999,"15min":0.66000000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40500000000000003},"max":{"1min":10.1,"5min":10.1,"15min":10.1},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81799999999999995,"15min":0.80100000000000005},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.48499999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":0.82099999999999995},"min":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.47299999999999998},"max":{"1min":1.032,"5min":1.23,"15min":1.23},"last":0.68999999999999995}]},{"osd":5,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.64100000000000001,"5min":0.53300000000000003,"15min":0.44400000000000001},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.349},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.60299999999999998,"5min":0.56599999999999995,"15min":0.54000000000000004},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.70799999999999996}]},{"osd":6,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.85699999999999998,"15min":0.85899999999999999},"min":{"1min":0.64800000000000002,"5min":0.59399999999999997,"15min":0.55200000000000005},"max":{"1min":1.0860000000000001,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.87,"15min":0.85499999999999998},"min":{"1min":0.59799999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1659999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Sat Oct 7 10:44:49 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88800000000000001,"15min":0.879},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.3160000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.87,"15min":0.874},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0980000000000001,"5min":1.139,"15min":1.2969999999999999},"last":0.61399999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543049,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299284,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072556,"statfs":{"total":96636764160,"available":96330297344,"internally_reserved":0,"allocated":2043904,"data_stored":999901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58599999999999997,"5min":0.79300000000000004,"15min":1.073},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.81699999999999995,"15min":1.5149999999999999},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":0.82499999999999996,"15min":1.5489999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":1.02},"last":0.45100000000000001}]},{"osd":2,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56200000000000006,"15min":0.57099999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":1.456},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":0.94399999999999995,"15min":1.6040000000000001},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.79600000000000004,"15min":0.80300000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.173},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.80200000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95299999999999996,"5min":1.141,"15min":1.3839999999999999},"last":1.0580000000000001}]},{"osd":5,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.80500000000000005,"15min":0.81200000000000006},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"max":{"1min":0.90900000000000003,"5min":1.1539999999999999,"15min":1.1879999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82599999999999996,"15min":0.83499999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.98199999999999998,"5min":1.1619999999999999,"15min":1.4339999999999999},"last":0.79500000000000004}]},{"osd":6,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82799999999999996,"15min":0.83999999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0289999999999999,"5min":1.214,"15min":1.357},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.85099999999999998,"15min":0.84399999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.109}]},{"osd":7,"last update":"Sat Oct 7 10:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.85599999999999998,"15min":0.85099999999999998},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":1.137,"15min":1.492},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.85999999999999999,"15min":0.86299999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.59599999999999997},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.0329999999999999}]}]},{"osd":4,"up_from":15,"seq":64424510345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311580,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060260,"statfs":{"total":96636764160,"available":96317706240,"internally_reserved":0,"allocated":2052096,"data_stored":1006519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93600000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.745,"15min":0.76000000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86499999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55400000000000005}]},{"osd":1,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.76400000000000001,"15min":0.76500000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.016,"15min":1.032},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46500000000000002},"max":{"1min":0.84599999999999997,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59599999999999997}]},{"osd":2,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":0.90300000000000002,"5min":1.002,"15min":1.01},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88500000000000001,"5min":1.145,"15min":1.145},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.75600000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.497},"max":{"1min":0.82899999999999996,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96599999999999997,"5min":1.129,"15min":1.129},"last":0.74399999999999999}]},{"osd":5,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.45000000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.441,"5min":0.441,"15min":0.39400000000000002},"max":{"1min":0.85599999999999998,"5min":0.93999999999999995,"15min":1.0349999999999999},"last":0.71999999999999997}]},{"osd":6,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89200000000000002,"5min":0.98999999999999999,"15min":1.018},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.69899999999999995},"min":{"1min":0.54500000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85499999999999998,"5min":0.96799999999999997,"15min":1.042},"last":0.86299999999999999}]},{"osd":7,"last update":"Sat Oct 7 10:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.54600000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90400000000000003,"5min":0.94099999999999995,"15min":0.97899999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0960000000000001},"last":0.82799999999999996}]}]},{"osd":5,"up_from":14,"seq":60129543047,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311584,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060256,"statfs":{"total":96636764160,"available":96317702144,"internally_reserved":0,"allocated":2056192,"data_stored":1006541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78700000000000003,"15min":0.79400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39500000000000002},"max":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.091},"last":0.745},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.438},"max":{"1min":0.93500000000000005,"5min":1.0820000000000001,"15min":1.1279999999999999},"last":0.83499999999999996}]},{"osd":1,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.746},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.93400000000000005,"5min":1.032,"15min":1.032},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78100000000000003,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.46899999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89900000000000002,"5min":1.0760000000000001,"15min":1.093},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.76700000000000002,"15min":0.76800000000000002},"min":{"1min":0.52300000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.89100000000000001,"5min":1.077,"15min":1.343},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.77600000000000002,"15min":0.78200000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.50800000000000001},"max":{"1min":0.872,"5min":1.121,"15min":1.121},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75700000000000001,"15min":0.754},"min":{"1min":0.61799999999999999,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.77700000000000002}]},{"osd":4,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.91400000000000003,"5min":0.94999999999999996,"15min":0.995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.64700000000000002},"min":{"1min":0.47199999999999998,"5min":0.46500000000000002,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":1.0089999999999999,"15min":1.0529999999999999},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.52400000000000002,"5min":0.44500000000000001,"15min":0.39800000000000002},"max":{"1min":0.91900000000000004,"5min":1.016,"15min":1.083},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.71499999999999997,"15min":0.71299999999999997},"min":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0289999999999999,"5min":1.032,"15min":1.101},"last":0.75700000000000001}]},{"osd":7,"last update":"Sat Oct 7 10:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.73499999999999999,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.45700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.075},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.86499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T10:45:24.023 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T10:45:24.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T10:45:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.259+0000 7f6f435ff700 1 Processor -- start 2023-10-07T10:45:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.259+0000 7f6f435ff700 1 -- start start 2023-10-07T10:45:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.260+0000 7f6f435ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f3c0cddf0 0x7f6f3c0b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.260+0000 7f6f435ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c0b0310 con 0x7f6f3c0af390 2023-10-07T10:45:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.260+0000 7f6f435ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c0b05f0 con 0x7f6f3c0aefc0 2023-10-07T10:45:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.260+0000 7f6f435ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f3c0b1210 con 0x7f6f3c0cddf0 2023-10-07T10:45:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.260+0000 7f6f40b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f3c0cddf0 0x7f6f3c0b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f40b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f3c0cddf0 0x7f6f3c0b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56876/0 (socket says 172.21.15.29:56876) 2023-10-07T10:45:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f40b9a700 1 -- 172.21.15.29:0/1702934634 learned_addr learned my addr 172.21.15.29:0/1702934634 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f40b9a700 1 -- 172.21.15.29:0/1702934634 >> v1:172.21.15.144:6789/0 conn(0x7f6f3c0aefc0 legacy=0x7f6f3c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f33fff700 1 -- 172.21.15.29:0/1702934634 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230199629 0 0) 0x7f6f3c0b0310 con 0x7f6f3c0af390 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f40b9a700 1 -- 172.21.15.29:0/1702934634 >> v1:172.21.15.29:6789/0 conn(0x7f6f3c0af390 legacy=0x7f6f3c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f40b9a700 1 -- 172.21.15.29:0/1702934634 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c159df0 con 0x7f6f3c0cddf0 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.261+0000 7f6f40b9a700 1 --2- 172.21.15.29:0/1702934634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c0b0b30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0099b0 tx=0x7f6f2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d359a62bf7f572c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f33fff700 1 -- 172.21.15.29:0/1702934634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c00b860 con 0x7f6f3c0cddf0 2023-10-07T10:45:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f33fff700 1 -- 172.21.15.29:0/1702934634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c00b9c0 con 0x7f6f3c0cddf0 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f33fff700 1 -- 172.21.15.29:0/1702934634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c00bd60 con 0x7f6f3c0cddf0 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 -- 172.21.15.29:0/1702934634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0cddf0 msgr2=0x7f6f3c0b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 --2- 172.21.15.29:0/1702934634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c0b0b30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0099b0 tx=0x7f6f2c009250 comp rx=0 tx=0).stop 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 -- 172.21.15.29:0/1702934634 shutdown_connections 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 --2- 172.21.15.29:0/1702934634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c0b0b30 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 -- 172.21.15.29:0/1702934634 >> 172.21.15.29:0/1702934634 conn(0x7f6f3c068720 msgr2=0x7f6f3c068b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 -- 172.21.15.29:0/1702934634 shutdown_connections 2023-10-07T10:45:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.262+0000 7f6f435ff700 1 -- 172.21.15.29:0/1702934634 wait complete. 2023-10-07T10:45:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.264+0000 7f6f435ff700 1 Processor -- start 2023-10-07T10:45:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.264+0000 7f6f435ff700 1 -- start start 2023-10-07T10:45:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f435ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f3c0aefc0 0x7f6f3c1567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f435ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 0x7f6f3c13c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f435ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f435ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c156350 con 0x7f6f3c0af390 2023-10-07T10:45:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f435ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c155740 con 0x7f6f3c0cddf0 2023-10-07T10:45:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f435ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c155a20 con 0x7f6f3c0aefc0 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f40b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 0x7f6f3c13c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f40b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 0x7f6f3c13c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56884/0 (socket says 172.21.15.29:56884) 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f40b9a700 1 -- 172.21.15.29:0/4275515778 learned_addr learned my addr 172.21.15.29:0/4275515778 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f41b9c700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.265+0000 7f6f4139b700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f3c0aefc0 0x7f6f3c1567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f40b9a700 1 -- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f3c0aefc0 msgr2=0x7f6f3c1567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f40b9a700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f3c0aefc0 0x7f6f3c1567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f40b9a700 1 -- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f3c0cddf0 msgr2=0x7f6f3c151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f40b9a700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f40b9a700 1 -- 172.21.15.29:0/4275515778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c0094f0 con 0x7f6f3c0af390 2023-10-07T10:45:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f41b9c700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f40b9a700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 0x7f6f3c13c3d0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c006f50 tx=0x7f6f2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c01f070 con 0x7f6f3c0af390 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c011d10 con 0x7f6f3c0af390 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c025650 con 0x7f6f3c0af390 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3c155d00 con 0x7f6f3c0af390 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.266+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3c13c910 con 0x7f6f3c0af390 2023-10-07T10:45:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.267+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3c1564c0 con 0x7f6f3c0af390 2023-10-07T10:45:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.268+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6f2c023070 con 0x7f6f3c0af390 2023-10-07T10:45:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.269+0000 7f6f317fa700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6f20066f30 0x7f6f200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.270+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6f2c0858c0 con 0x7f6f3c0af390 2023-10-07T10:45:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.270+0000 7f6f4139b700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6f20066f30 0x7f6f200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.271+0000 7f6f4139b700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6f20066f30 0x7f6f200693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f38003860 tx=0x7f6f38009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.274+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f2c0529a0 con 0x7f6f3c0af390 2023-10-07T10:45:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.555+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6f3c0718b0 con 0x7f6f3c0af390 2023-10-07T10:45:24.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.559+0000 7f6f317fa700 1 -- 172.21.15.29:0/4275515778 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v44) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6f2c055fc0 con 0x7f6f3c0af390 2023-10-07T10:45:24.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6f20066f30 msgr2=0x7f6f200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6f20066f30 0x7f6f200693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f38003860 tx=0x7f6f38009040 comp rx=0 tx=0).stop 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 msgr2=0x7f6f3c13c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 0x7f6f3c13c3d0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c006f50 tx=0x7f6f2c017000 comp rx=0 tx=0).stop 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 shutdown_connections 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6f20066f30 0x7f6f200693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f3c0cddf0 0x7f6f3c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f3c0aefc0 0x7f6f3c1567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 --2- 172.21.15.29:0/4275515778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f3c0af390 0x7f6f3c13c3d0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.561+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 >> 172.21.15.29:0/4275515778 conn(0x7f6f3c068720 msgr2=0x7f6f3c0acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.562+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 shutdown_connections 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.562+0000 7f6f435ff700 1 -- 172.21.15.29:0/4275515778 wait complete. 2023-10-07T10:45:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T10:45:24.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T10:45:24.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.775+0000 7fc702dff700 1 Processor -- start 2023-10-07T10:45:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.775+0000 7fc702dff700 1 -- start start 2023-10-07T10:45:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.776+0000 7fc702dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc6fc153cc0 0x7fc6fc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.776+0000 7fc702dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6fc0b0710 0x7fc6fc0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.776+0000 7fc702dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc154090 con 0x7fc6fc15ade0 2023-10-07T10:45:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.776+0000 7fc702dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6fc159f60 con 0x7fc6fc0b0710 2023-10-07T10:45:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.776+0000 7fc702dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6fc15a240 con 0x7fc6fc153cc0 2023-10-07T10:45:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.776+0000 7fc70139c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6fc0b0710 0x7fc6fc0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6fc0b0710 0x7fc6fc0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56898/0 (socket says 172.21.15.29:56898) 2023-10-07T10:45:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 -- 172.21.15.29:0/1879076015 learned_addr learned my addr 172.21.15.29:0/1879076015 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc700b9b700 1 --2- 172.21.15.29:0/1879076015 >> v2:172.21.15.144:3300/0 conn(0x7fc6fc153cc0 0x7fc6fc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 -- 172.21.15.29:0/1879076015 >> v2:172.21.15.144:3300/0 conn(0x7fc6fc153cc0 msgr2=0x7fc6fc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 --2- 172.21.15.29:0/1879076015 >> v2:172.21.15.144:3300/0 conn(0x7fc6fc153cc0 0x7fc6fc0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 -- 172.21.15.29:0/1879076015 >> v1:172.21.15.144:6789/0 conn(0x7fc6fc15ade0 legacy=0x7fc6fc0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 -- 172.21.15.29:0/1879076015 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc15a520 con 0x7fc6fc0b0710 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.777+0000 7fc70139c700 1 --2- 172.21.15.29:0/1879076015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc0b1d00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0cf440 tx=0x7fc6f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8ab3e9c4298211 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc6fb7fe700 1 -- 172.21.15.29:0/1879076015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f400e070 con 0x7fc6fc0b0710 2023-10-07T10:45:24.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc6fb7fe700 1 -- 172.21.15.29:0/1879076015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f400ba20 con 0x7fc6fc0b0710 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc6fb7fe700 1 -- 172.21.15.29:0/1879076015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f40092e0 con 0x7fc6fc0b0710 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 -- 172.21.15.29:0/1879076015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc0b0710 msgr2=0x7fc6fc0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 --2- 172.21.15.29:0/1879076015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc0b1d00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0cf440 tx=0x7fc6f4009840 comp rx=0 tx=0).stop 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 -- 172.21.15.29:0/1879076015 shutdown_connections 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 --2- 172.21.15.29:0/1879076015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc0b1d00 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 --2- 172.21.15.29:0/1879076015 >> v2:172.21.15.144:3300/0 conn(0x7fc6fc153cc0 0x7fc6fc0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 -- 172.21.15.29:0/1879076015 >> 172.21.15.29:0/1879076015 conn(0x7fc6fc067610 msgr2=0x7fc6fc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 -- 172.21.15.29:0/1879076015 shutdown_connections 2023-10-07T10:45:24.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.778+0000 7fc702dff700 1 -- 172.21.15.29:0/1879076015 wait complete. 2023-10-07T10:45:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.780+0000 7fc702dff700 1 Processor -- start 2023-10-07T10:45:24.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.781+0000 7fc702dff700 1 -- start start 2023-10-07T10:45:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc702dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc702dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6fc153cc0 0x7fc6fc153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc702dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 0x7fc6fc0c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc702dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc1535b0 con 0x7fc6fc15ade0 2023-10-07T10:45:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc702dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc169770 con 0x7fc6fc0b0710 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc702dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc1575f0 con 0x7fc6fc153cc0 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc70139c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 0x7fc6fc0c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc70139c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 0x7fc6fc0c92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56910/0 (socket says 172.21.15.29:56910) 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc70139c700 1 -- 172.21.15.29:0/1022670633 learned_addr learned my addr 172.21.15.29:0/1022670633 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc700b9b700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.782+0000 7fc6fbfff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6fc153cc0 0x7fc6fc153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc70139c700 1 -- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6fc153cc0 msgr2=0x7fc6fc153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc70139c700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6fc153cc0 0x7fc6fc153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc70139c700 1 -- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6fc0b0710 msgr2=0x7fc6fc156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc70139c700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc70139c700 1 -- 172.21.15.29:0/1022670633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4009ae0 con 0x7fc6fc15ade0 2023-10-07T10:45:24.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc6fbfff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6fc153cc0 0x7fc6fc153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc70139c700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 0x7fc6fc0c92a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4012000 tx=0x7fc6f4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4008a60 con 0x7fc6fc15ade0 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4008bc0 con 0x7fc6fc15ade0 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4026460 con 0x7fc6fc15ade0 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc1578d0 con 0x7fc6fc15ade0 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.783+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc157e90 con 0x7fc6fc15ade0 2023-10-07T10:45:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.784+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc169900 con 0x7fc6fc15ade0 2023-10-07T10:45:24.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.785+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc6f4003040 con 0x7fc6fc15ade0 2023-10-07T10:45:24.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.787+0000 7fc6f8ff9700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6e0066f30 0x7fc6e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:24.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.787+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fc6f4086540 con 0x7fc6fc15ade0 2023-10-07T10:45:24.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.787+0000 7fc700b9b700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6e0066f30 0x7fc6e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:24.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.788+0000 7fc700b9b700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6e0066f30 0x7fc6e00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009da0 tx=0x7fc6f0009120 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:24.791+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6f4053570 con 0x7fc6fc15ade0 2023-10-07T10:45:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.039+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6fc04c3c0 con 0x7fc6fc15ade0 2023-10-07T10:45:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.040+0000 7fc6f8ff9700 1 -- 172.21.15.29:0/1022670633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fc6f4056b90 con 0x7fc6fc15ade0 2023-10-07T10:45:25.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:45:25.041 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:45:18.619852+0000","modified":"2023-10-07T10:45:22.148839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9090":{"gid":9090,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/4080992371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4080992371},{"type":"v1","addr":"172.21.15.29:6833","nonce":4080992371}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-10-07T10:45:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6e0066f30 msgr2=0x7fc6e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6e0066f30 0x7fc6e00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009da0 tx=0x7fc6f0009120 comp rx=0 tx=0).stop 2023-10-07T10:45:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 msgr2=0x7fc6fc0c92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 0x7fc6fc0c92a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4012000 tx=0x7fc6f4018000 comp rx=0 tx=0).stop 2023-10-07T10:45:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 shutdown_connections 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc6e0066f30 0x7fc6e00693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6fc15ade0 0x7fc6fc0c92a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6fc0b0710 0x7fc6fc156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 --2- 172.21.15.29:0/1022670633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6fc153cc0 0x7fc6fc153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 >> 172.21.15.29:0/1022670633 conn(0x7fc6fc067610 msgr2=0x7fc6fc15df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 shutdown_connections 2023-10-07T10:45:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.042+0000 7fc702dff700 1 -- 172.21.15.29:0/1022670633 wait complete. 2023-10-07T10:45:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-07T10:45:25.463 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T10:45:18.619852+0000', 'modified': '2023-10-07T10:45:22.148839+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9090}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9090': {'gid': 9090, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6833/4080992371', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 4080992371}, {'type': 'v1', 'addr': '172.21.15.29:6833', 'nonce': 4080992371}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T10:45:25.463 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T10:45:25.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-07T10:45:25.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.664+0000 7f6c941ff700 1 Processor -- start 2023-10-07T10:45:25.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.664+0000 7f6c941ff700 1 -- start start 2023-10-07T10:45:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.665+0000 7f6c941ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c0ad370 0x7f6c8c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.665+0000 7f6c941ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c14d790 con 0x7f6c8c0b0410 2023-10-07T10:45:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.665+0000 7f6c941ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c0ae5d0 con 0x7f6c8c159ec0 2023-10-07T10:45:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.665+0000 7f6c941ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0ad740 con 0x7f6c8c0ad370 2023-10-07T10:45:25.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.665+0000 7f6c9279c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c8c0b0410 0x7f6c8c156fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47452/0 (socket says 172.21.15.29:47452) 2023-10-07T10:45:25.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c9279c700 1 -- 172.21.15.29:0/42985704 learned_addr learned my addr 172.21.15.29:0/42985704 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T10:45:25.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/42985704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c0ad370 0x7f6c8c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998901404 0 0) 0x7f6c8c14d790 con 0x7f6c8c0b0410 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c78003570 con 0x7f6c8c0b0410 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561830934 0 0) 0x7f6c8c0ae5d0 con 0x7f6c8c159ec0 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c8c14d790 con 0x7f6c8c159ec0 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c91f9b700 1 -- 172.21.15.29:0/42985704 >> v1:172.21.15.144:6789/0 conn(0x7f6c8c159ec0 legacy=0x7f6c8c1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c91f9b700 1 -- 172.21.15.29:0/42985704 >> v1:172.21.15.29:6789/0 conn(0x7f6c8c0b0410 legacy=0x7f6c8c156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c91f9b700 1 -- 172.21.15.29:0/42985704 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c0ccdb0 con 0x7f6c8c0ad370 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.666+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/42985704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c0ad370 0x7f6c8c0cc870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009910 tx=0x7f6c880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71a16b8c72e057c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88011630 con 0x7f6c8c0ad370 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88011790 con 0x7f6c8c0ad370 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c90f99700 1 -- 172.21.15.29:0/42985704 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88011b30 con 0x7f6c8c0ad370 2023-10-07T10:45:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c941ff700 1 -- 172.21.15.29:0/42985704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c0ad370 msgr2=0x7f6c8c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c941ff700 1 --2- 172.21.15.29:0/42985704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c0ad370 0x7f6c8c0cc870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009910 tx=0x7f6c880091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c941ff700 1 -- 172.21.15.29:0/42985704 shutdown_connections 2023-10-07T10:45:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c941ff700 1 --2- 172.21.15.29:0/42985704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c0ad370 0x7f6c8c0cc870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.667+0000 7f6c941ff700 1 -- 172.21.15.29:0/42985704 >> 172.21.15.29:0/42985704 conn(0x7f6c8c0680a0 msgr2=0x7f6c8c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.668+0000 7f6c941ff700 1 -- 172.21.15.29:0/42985704 shutdown_connections 2023-10-07T10:45:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.668+0000 7f6c941ff700 1 -- 172.21.15.29:0/42985704 wait complete. 2023-10-07T10:45:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.669+0000 7f6c941ff700 1 Processor -- start 2023-10-07T10:45:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.669+0000 7f6c941ff700 1 -- start start 2023-10-07T10:45:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c941ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c8c0ad370 0x7f6c8c15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c941ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 0x7f6c8c1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c941ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c159ec0 0x7f6c8c14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c941ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c1695c0 con 0x7f6c8c0b0410 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c941ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c15bcc0 con 0x7f6c8c0ad370 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c941ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c150ce0 con 0x7f6c8c159ec0 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.670+0000 7f6c9179a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 0x7f6c8c1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c91f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c8c0ad370 0x7f6c8c15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 0x7f6c8c1507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56912/0 (socket says 172.21.15.29:56912) 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9279c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c159ec0 0x7f6c8c14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 -- 172.21.15.29:0/3272739219 learned_addr learned my addr 172.21.15.29:0/3272739219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 -- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c159ec0 msgr2=0x7f6c8c14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c159ec0 0x7f6c8c14cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 -- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c8c0ad370 msgr2=0x7f6c8c15a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c8c0ad370 0x7f6c8c15a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 -- 172.21.15.29:0/3272739219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88009420 con 0x7f6c8c0b0410 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 0x7f6c8c1507a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c590 tx=0x7f6c7c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c81ffb700 1 -- 172.21.15.29:0/3272739219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c010510 con 0x7f6c8c0b0410 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c81ffb700 1 -- 172.21.15.29:0/3272739219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c010690 con 0x7f6c8c0b0410 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.671+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c15b0b0 con 0x7f6c8c0b0410 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.672+0000 7f6c81ffb700 1 -- 172.21.15.29:0/3272739219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c010a50 con 0x7f6c8c0b0410 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.672+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c15b6f0 con 0x7f6c8c0b0410 2023-10-07T10:45:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.674+0000 7f6c81ffb700 1 -- 172.21.15.29:0/3272739219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6c7c01b400 con 0x7f6c8c0b0410 2023-10-07T10:45:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.674+0000 7f6c81ffb700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c70066f30 0x7f6c700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.675+0000 7f6c81ffb700 1 -- 172.21.15.29:0/3272739219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6c7c084320 con 0x7f6c8c0b0410 2023-10-07T10:45:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.675+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c70066f30 0x7f6c700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.676+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c70066f30 0x7f6c700693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009860 tx=0x7f6c88017000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.676+0000 7f6ca28f5b80 1 Processor -- start 2023-10-07T10:45:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.676+0000 7f6ca28f5b80 1 -- start start 2023-10-07T10:45:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.677+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.677+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.677+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.677+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3f92310 con 0x55ebc40a5530 2023-10-07T10:45:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.677+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3fcf130 con 0x55ebc40bfa40 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.677+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3fccaf0 con 0x55ebc40c8bc0 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56928/0 (socket says 172.21.15.29:56928) 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 -- 172.21.15.29:0/3267333361 learned_addr learned my addr 172.21.15.29:0/3267333361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9279c700 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 -- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 msgr2=0x55ebc40c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 -- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 msgr2=0x55ebc40bfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40bfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 -- 172.21.15.29:0/3267333361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebc3ebb0a0 con 0x55ebc40a5530 2023-10-07T10:45:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c8680 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00a040 tx=0x7f6c7c082560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ada0c44f74f8a33 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c76ffd700 1 -- 172.21.15.29:0/3267333361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c058260 con 0x55ebc40a5530 2023-10-07T10:45:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c76ffd700 1 -- 172.21.15.29:0/3267333361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c05bfb0 con 0x55ebc40a5530 2023-10-07T10:45:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.678+0000 7f6c76ffd700 1 -- 172.21.15.29:0/3267333361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c0840c0 con 0x55ebc40a5530 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3267333361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 msgr2=0x55ebc40c8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c8680 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00a040 tx=0x7f6c7c082560 comp rx=0 tx=0).stop 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3267333361 shutdown_connections 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3267333361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c8680 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3267333361 >> 172.21.15.29:0/3267333361 conn(0x55ebc40c2e70 msgr2=0x55ebc40c3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3267333361 shutdown_connections 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.679+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3267333361 wait complete. 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.680+0000 7f6ca28f5b80 1 -- start start 2023-10-07T10:45:25.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.680+0000 7f6ca28f5b80 1 Processor -- start 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40a7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3fc88e0 con 0x55ebc40a5530 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3f74980 con 0x55ebc40bfa40 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3f74620 con 0x55ebc40c8bc0 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6ca28f5b80 10 client.0 _ll_register_callbacks cb 0x55ebc3dd7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6c91f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6c91f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56934/0 (socket says 172.21.15.29:56934) 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.681+0000 7f6c91f9b700 1 -- 172.21.15.29:0/3008479837 learned_addr learned my addr 172.21.15.29:0/3008479837 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c9279c700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40a7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c91f9b700 1 -- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 msgr2=0x55ebc40a7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40a7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c91f9b700 1 -- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 msgr2=0x55ebc40c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c91f9b700 1 -- 172.21.15.29:0/3008479837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c15b6f0 con 0x55ebc40a5530 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c9279c700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40a7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c41e0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009f00 tx=0x7f6c88022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c90cdb0db523ac server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c6bfff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c6bfff700 1 -- 172.21.15.29:0/3008479837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88026600 con 0x55ebc40a5530 2023-10-07T10:45:25.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebc406d560 con 0x55ebc40a5530 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.682+0000 7f6c6bfff700 1 -- 172.21.15.29:0/3008479837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6c88026760 con 0x55ebc40a5530 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6c6bfff700 1 -- 172.21.15.29:0/3008479837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c88026b30 con 0x55ebc40a5530 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 10 client.9234 fetch_fsmap learned FSMap version 85 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) v3 -- 0x55ebc3f973c0 con 0x55ebc40a5530 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6c6bfff700 1 -- 172.21.15.29:0/3008479837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ebc406d560 con 0x55ebc40a5530 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6c6bfff700 1 -- 172.21.15.29:0/3008479837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 85) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6c88026c90 con 0x55ebc40a5530 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 10 client.9234 fetch_fsmap finished waiting for FSMap version 85 2023-10-07T10:45:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 10 client.9234 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 20 client.9234 populate_metadata read hostname 'smithi029' 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40a3880 0x55ebc40dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 4 client.9234 mds_command: new command op to 9090 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 --> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebc404a0e0 con 0x55ebc40a3880 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.683+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40a3880 0x55ebc40dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.684+0000 7f6c9179a700 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40a3880 0x55ebc40dc5f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b97e76205503925b server_cookie=41070c3be74d44ff in_seq=0 out_seq=0 2023-10-07T10:45:25.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.684+0000 7f6c6bfff700 10 client.9234 ms_handle_connect on v2:172.21.15.29:6832/4080992371 2023-10-07T10:45:25.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6c6bfff700 1 -- 172.21.15.29:0/3008479837 <== mds.0 v2:172.21.15.29:6832/4080992371 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55ebc404a0e0 con 0x55ebc40a3880 2023-10-07T10:45:25.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6c6bfff700 10 client.9234 handle_command_reply: tid=0 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 client.9234 shutdown 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 msgr2=0x55ebc40c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c41e0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009f00 tx=0x7f6c88022000 comp rx=0 tx=0).stop 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 shutdown_connections 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40a3880 0x55ebc40dc5f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40a7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c41e0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/3008479837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6c6bfff700 0 client.9234 ms_handle_reset on v2:172.21.15.29:6832/4080992371 2023-10-07T10:45:25.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 >> 172.21.15.29:0/3008479837 conn(0x55ebc40c2e70 msgr2=0x55ebc4075250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 shutdown_connections 2023-10-07T10:45:25.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.685+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/3008479837 wait complete. 2023-10-07T10:45:25.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.686+0000 7f6ca28f5b80 20 client.9234 trim_cache size 0 max 16384 2023-10-07T10:45:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.703+0000 7f6ca28f5b80 1 Processor -- start 2023-10-07T10:45:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.703+0000 7f6ca28f5b80 1 -- start start 2023-10-07T10:45:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc409d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40751d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3e601e0 con 0x55ebc40a5530 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc40c4680 con 0x55ebc40bfa40 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebc4077c30 con 0x55ebc40c8bc0 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56940/0 (socket says 172.21.15.29:56940) 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 -- 172.21.15.29:0/2142314289 learned_addr learned my addr 172.21.15.29:0/2142314289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 -- 172.21.15.29:0/2142314289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 msgr2=0x55ebc40751d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40751d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 -- 172.21.15.29:0/2142314289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 msgr2=0x55ebc409d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc409d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.704+0000 7f6c91f9b700 1 -- 172.21.15.29:0/2142314289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebc3e2f1d0 con 0x55ebc40a5530 2023-10-07T10:45:25.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.705+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c3520 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6c88031410 tx=0x7f6c8802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3297614c168cbb29 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.705+0000 7f6c777fe700 1 -- 172.21.15.29:0/2142314289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880266d0 con 0x55ebc40a5530 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.705+0000 7f6c777fe700 1 -- 172.21.15.29:0/2142314289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8802d040 con 0x55ebc40a5530 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.705+0000 7f6c777fe700 1 -- 172.21.15.29:0/2142314289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88026d40 con 0x55ebc40a5530 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/2142314289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 msgr2=0x55ebc40c3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c3520 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6c88031410 tx=0x7f6c8802e000 comp rx=0 tx=0).stop 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/2142314289 shutdown_connections 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40c8bc0 0x55ebc40751d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40a5530 0x55ebc40c3520 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/2142314289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc409d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/2142314289 >> 172.21.15.29:0/2142314289 conn(0x55ebc40c2e70 msgr2=0x55ebc4076890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/2142314289 shutdown_connections 2023-10-07T10:45:25.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.706+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/2142314289 wait complete. 2023-10-07T10:45:25.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.707+0000 7f6ca28f5b80 1 -- start start 2023-10-07T10:45:25.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.707+0000 7f6ca28f5b80 1 Processor -- start 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40a5530 0x55ebc40d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 0x55ebc40d8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc4125740 con 0x55ebc40c8bc0 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc409d340 con 0x55ebc40bfa40 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebc40c8540 con 0x55ebc40a5530 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6ca28f5b80 10 client.0 _ll_register_callbacks cb 0x55ebc3dd7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 0x55ebc40d8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 0x55ebc40d8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56944/0 (socket says 172.21.15.29:56944) 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 -- 172.21.15.29:0/9436081 learned_addr learned my addr 172.21.15.29:0/9436081 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9179a700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40a5530 0x55ebc40d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 -- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40a5530 msgr2=0x55ebc40d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40a5530 0x55ebc40d0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 -- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 msgr2=0x55ebc40d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.708+0000 7f6c9279c700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c9279c700 1 -- 172.21.15.29:0/9436081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebc3f973c0 con 0x55ebc40c8bc0 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c9279c700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 0x55ebc40d8370 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6c84005c50 tx=0x7f6c84009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382de9c33be559f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c6bfff700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T10:45:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c6bfff700 1 -- 172.21.15.29:0/9436081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84012750 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c8c0c9070 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c6bfff700 1 -- 172.21.15.29:0/9436081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6c840128b0 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c6bfff700 1 -- 172.21.15.29:0/9436081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c84012c80 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6ca28f5b80 10 client.9240 fetch_fsmap learned FSMap version 85 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) v3 -- 0x55ebc3e08d60 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.709+0000 7f6c6bfff700 1 -- 172.21.15.29:0/9436081 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c0c9070 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6c6bfff700 1 -- 172.21.15.29:0/9436081 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 85) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6c8401f3f0 con 0x55ebc40c8bc0 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6ca28f5b80 10 client.9240 fetch_fsmap finished waiting for FSMap version 85 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6ca28f5b80 10 client.9240 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6ca28f5b80 20 client.9240 populate_metadata read hostname 'smithi029' 2023-10-07T10:45:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40d8970 0x55ebc40d8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6ca28f5b80 4 client.9240 mds_command: new command op to 9090 tid=0[{"prefix": "status"}] 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 --> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6c7c05bfb0 con 0x55ebc40d8970 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40d8970 0x55ebc40d8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6c91f9b700 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40d8970 0x55ebc40d8d40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e0168ff05248fcb server_cookie=dc4bcb2c76e3f2d2 in_seq=0 out_seq=0 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.710+0000 7f6c6bfff700 10 client.9240 ms_handle_connect on v2:172.21.15.29:6832/4080992371 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6c6bfff700 1 -- 172.21.15.29:0/9436081 <== mds.0 v2:172.21.15.29:6832/4080992371 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f6c7c05bfb0 con 0x55ebc40d8970 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6c6bfff700 10 client.9240 handle_command_reply: tid=0 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 client.9240 shutdown 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 msgr2=0x55ebc40d8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 0x55ebc40d8370 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6c84005c50 tx=0x7f6c84009040 comp rx=0 tx=0).stop 2023-10-07T10:45:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 shutdown_connections 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:6832/4080992371,v1:172.21.15.29:6833/4080992371] conn(0x55ebc40d8970 0x55ebc40d8d40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ebc40c8bc0 0x55ebc40d8370 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ebc40a5530 0x55ebc40d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6c6bfff700 0 client.9240 ms_handle_reset on v2:172.21.15.29:6832/4080992371 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 --2- 172.21.15.29:0/9436081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ebc40bfa40 0x55ebc40d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 >> 172.21.15.29:0/9436081 conn(0x55ebc40c2e70 msgr2=0x55ebc4076820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 shutdown_connections 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 1 -- 172.21.15.29:0/9436081 wait complete. 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.711+0000 7f6ca28f5b80 20 client.9240 trim_cache size 0 max 16384 2023-10-07T10:45:25.720 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 9090, 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.5939503869999996, 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 85, 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 165, 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 165, 2023-10-07T10:45:25.721 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.264947501 2023-10-07T10:45:25.722 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T10:45:25.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c70066f30 msgr2=0x7f6c700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c70066f30 0x7f6c700693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009860 tx=0x7f6c88017000 comp rx=0 tx=0).stop 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 msgr2=0x7f6c8c1507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 0x7f6c8c1507a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c590 tx=0x7f6c7c007dd0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 shutdown_connections 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f6c70066f30 0x7f6c700693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c8c159ec0 0x7f6c8c14cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c8c0ad370 0x7f6c8c15a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 --2- 172.21.15.29:0/3272739219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0b0410 0x7f6c8c1507a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 >> 172.21.15.29:0/3272739219 conn(0x7f6c8c0680a0 msgr2=0x7f6c8c06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 shutdown_connections 2023-10-07T10:45:25.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.712+0000 7f6c941ff700 1 -- 172.21.15.29:0/3272739219 wait complete. 2023-10-07T10:45:25.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T10:45:25.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.925+0000 7f63cefff700 1 Processor -- start 2023-10-07T10:45:25.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.926+0000 7f63cefff700 1 -- start start 2023-10-07T10:45:25.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63cefff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c80ad370 0x7f63c80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63cefff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63c80b0410 0x7f63c815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63cefff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c814d790 con 0x7f63c8159ec0 2023-10-07T10:45:25.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63cefff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c815a880 con 0x7f63c80ad370 2023-10-07T10:45:25.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63cefff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c80ae5d0 con 0x7f63c80b0410 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63ccd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c80ad370 0x7f63c80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63ccd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c80ad370 0x7f63c80cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56958/0 (socket says 172.21.15.29:56958) 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63c7fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f63c8159ec0 0x7f63c81568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47458/0 (socket says 172.21.15.29:47458) 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63ccd9b700 1 -- 172.21.15.29:0/2704234263 learned_addr learned my addr 172.21.15.29:0/2704234263 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.927+0000 7f63cd59c700 1 --2- 172.21.15.29:0/2704234263 >> v2:172.21.15.144:3300/0 conn(0x7f63c80b0410 0x7f63c815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63ccd9b700 1 -- 172.21.15.29:0/2704234263 >> v2:172.21.15.144:3300/0 conn(0x7f63c80b0410 msgr2=0x7f63c815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:25.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63ccd9b700 1 --2- 172.21.15.29:0/2704234263 >> v2:172.21.15.144:3300/0 conn(0x7f63c80b0410 0x7f63c815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63ccd9b700 1 -- 172.21.15.29:0/2704234263 >> v1:172.21.15.29:6789/0 conn(0x7f63c8159ec0 legacy=0x7f63c81568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63ccd9b700 1 -- 172.21.15.29:0/2704234263 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80ad740 con 0x7f63c80ad370 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63cd59c700 1 --2- 172.21.15.29:0/2704234263 >> v2:172.21.15.144:3300/0 conn(0x7f63c80b0410 0x7f63c815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63c77fe700 1 -- 172.21.15.29:0/2704234263 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837810524 0 0) 0x7f63c814d790 con 0x7f63c8159ec0 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63ccd9b700 1 --2- 172.21.15.29:0/2704234263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c80ad370 0x7f63c80cc870 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0099b0 tx=0x7f63bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59366baaa2d5028a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63c77fe700 1 -- 172.21.15.29:0/2704234263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc009ea0 con 0x7f63c80ad370 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63c77fe700 1 -- 172.21.15.29:0/2704234263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc00b820 con 0x7f63c80ad370 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.928+0000 7f63c77fe700 1 -- 172.21.15.29:0/2704234263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00bbc0 con 0x7f63c80ad370 2023-10-07T10:45:25.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 -- 172.21.15.29:0/2704234263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c80ad370 msgr2=0x7f63c80cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 --2- 172.21.15.29:0/2704234263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c80ad370 0x7f63c80cc870 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0099b0 tx=0x7f63bc009250 comp rx=0 tx=0).stop 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 -- 172.21.15.29:0/2704234263 shutdown_connections 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 --2- 172.21.15.29:0/2704234263 >> v2:172.21.15.144:3300/0 conn(0x7f63c80b0410 0x7f63c815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 --2- 172.21.15.29:0/2704234263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c80ad370 0x7f63c80cc870 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 -- 172.21.15.29:0/2704234263 >> 172.21.15.29:0/2704234263 conn(0x7f63c80680a0 msgr2=0x7f63c80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 -- 172.21.15.29:0/2704234263 shutdown_connections 2023-10-07T10:45:25.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.929+0000 7f63cefff700 1 -- 172.21.15.29:0/2704234263 wait complete. 2023-10-07T10:45:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.930+0000 7f63cefff700 1 Processor -- start 2023-10-07T10:45:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.931+0000 7f63cefff700 1 -- start start 2023-10-07T10:45:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.931+0000 7f63cefff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c80ad370 0x7f63c80c1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.931+0000 7f63cefff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c80b0410 0x7f63c8169250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cefff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 0x7f63c80cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cefff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cbf30 con 0x7f63c8159ec0 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cefff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80c1fa0 con 0x7f63c80b0410 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cefff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80c2280 con 0x7f63c80ad370 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 0x7f63c80cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 0x7f63c80cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56968/0 (socket says 172.21.15.29:56968) 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 -- 172.21.15.29:0/2762031384 learned_addr learned my addr 172.21.15.29:0/2762031384 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63c7fff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c80b0410 0x7f63c8169250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 -- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c80ad370 msgr2=0x7f63c80c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63ccd9b700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c80ad370 0x7f63c80c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c80ad370 0x7f63c80c1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 -- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c80b0410 msgr2=0x7f63c8169250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c80b0410 0x7f63c8169250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63cd59c700 1 -- 172.21.15.29:0/2762031384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0009a20 con 0x7f63c8159ec0 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63c7fff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c80b0410 0x7f63c8169250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.932+0000 7f63ccd9b700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c80ad370 0x7f63c80c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.933+0000 7f63cd59c700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 0x7f63c80cac80 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009fc0 tx=0x7f63c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.933+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000cd30 con 0x7f63c8159ec0 2023-10-07T10:45:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.933+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c000f040 con 0x7f63c8159ec0 2023-10-07T10:45:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.933+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c00146d0 con 0x7f63c8159ec0 2023-10-07T10:45:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.933+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc0094f0 con 0x7f63c8159ec0 2023-10-07T10:45:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.933+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c80c9070 con 0x7f63c8159ec0 2023-10-07T10:45:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.934+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c806bcf0 con 0x7f63c8159ec0 2023-10-07T10:45:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.935+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f63c0003020 con 0x7f63c8159ec0 2023-10-07T10:45:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.936+0000 7f63c4ff9700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63ac066f30 0x7f63ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.936+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f63c0085200 con 0x7f63c8159ec0 2023-10-07T10:45:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.936+0000 7f63ccd9b700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63ac066f30 0x7f63ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.937+0000 7f63ccd9b700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63ac066f30 0x7f63ac0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0099b0 tx=0x7f63bc017000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:25.940+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63c0052230 con 0x7f63c8159ec0 2023-10-07T10:45:26.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.192+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f63c804c3c0 con 0x7f63c8159ec0 2023-10-07T10:45:26.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.192+0000 7f63c4ff9700 1 -- 172.21.15.29:0/2762031384 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v85) v1 ==== 53+0+73 (secure 0 0 0) 0x7f63c0055850 con 0x7f63c8159ec0 2023-10-07T10:45:26.193 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T10:45:26.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63ac066f30 msgr2=0x7f63ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63ac066f30 0x7f63ac0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0099b0 tx=0x7f63bc017000 comp rx=0 tx=0).stop 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 msgr2=0x7f63c80cac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 0x7f63c80cac80 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009fc0 tx=0x7f63c000b040 comp rx=0 tx=0).stop 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 shutdown_connections 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f63ac066f30 0x7f63ac0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c8159ec0 0x7f63c80cac80 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63c80ad370 0x7f63c80c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.194+0000 7f63cefff700 1 --2- 172.21.15.29:0/2762031384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63c80b0410 0x7f63c8169250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:26.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.195+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 >> 172.21.15.29:0/2762031384 conn(0x7f63c80680a0 msgr2=0x7f63c80694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:26.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.195+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 shutdown_connections 2023-10-07T10:45:26.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:26.195+0000 7f63cefff700 1 -- 172.21.15.29:0/2762031384 wait complete. 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T10:45:26.213 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T10:45:26.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.214 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T10:45:26.261 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T10:45:26.262 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 37702sec preferred_lft 37702sec 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T10:45:26.263 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T10:45:26.264 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T10:45:26.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.265 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T10:45:26.333 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T10:45:26.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.337 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T10:45:26.386 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T10:45:26.387 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T10:45:26.387 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T10:45:26.387 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T10:45:26.387 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T10:45:26.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.387 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T10:45:26.387 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T10:45:26.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T10:45:26.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T10:45:26.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T10:45:26.388 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T10:45:26.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.560 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T10:45:26.606 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T10:45:26.607 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T10:45:26.608 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T10:45:26.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T10:45:26.680 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T10:45:26.680 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T10:45:26.727 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T10:45:26.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T10:45:26.727 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T10:45:26.812 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T10:45:26.812 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T10:45:26.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:45:26.954 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:45:26.954 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T10:45:26.954 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T10:45:26.969 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:45:26.969 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T10:45:26.969 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T10:45:26.969 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T10:45:26.969 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T10:45:26.989 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T10:45:26.989 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T10:45:26.990 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T10:45:26.990 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T10:45:26.990 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T10:45:26.991 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T10:45:26.991 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T10:45:26.991 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T10:45:26.993 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T10:45:26.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T10:45:26.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T10:45:27.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T10:45:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.266+0000 7f495d1ff700 1 Processor -- start 2023-10-07T10:45:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.266+0000 7f495d1ff700 1 -- start start 2023-10-07T10:45:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495d1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4958159ec0 0x7f495815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495d1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580af400 0x7f49580af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495d1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495814d790 con 0x7f49580ad370 2023-10-07T10:45:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495d1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495815a880 con 0x7f4958159ec0 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495d1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49580ae5d0 con 0x7f49580af400 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4958159ec0 0x7f495815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4958159ec0 0x7f495815a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56974/0 (socket says 172.21.15.29:56974) 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 -- 172.21.15.29:0/2661344672 learned_addr learned my addr 172.21.15.29:0/2661344672 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495759e700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580af400 0x7f49580af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 -- 172.21.15.29:0/2661344672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580af400 msgr2=0x7f49580af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580af400 0x7f49580af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 -- 172.21.15.29:0/2661344672 >> v1:172.21.15.144:6789/0 conn(0x7f49580ad370 legacy=0x7f49580cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495659c700 1 -- 172.21.15.29:0/2661344672 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49580ad740 con 0x7f4958159ec0 2023-10-07T10:45:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.268+0000 7f495759e700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580af400 0x7f49580af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495659c700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f495815a340 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f494c0099b0 tx=0x7f494c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724a06346a946627 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f4955d9b700 1 -- 172.21.15.29:0/2661344672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c00b860 con 0x7f4958159ec0 2023-10-07T10:45:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f4955d9b700 1 -- 172.21.15.29:0/2661344672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c00b9c0 con 0x7f4958159ec0 2023-10-07T10:45:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f4955d9b700 1 -- 172.21.15.29:0/2661344672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c00bd60 con 0x7f4958159ec0 2023-10-07T10:45:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495d1ff700 1 -- 172.21.15.29:0/2661344672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 msgr2=0x7f495815a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495d1ff700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f495815a340 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f494c0099b0 tx=0x7f494c009250 comp rx=0 tx=0).stop 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495d1ff700 1 -- 172.21.15.29:0/2661344672 shutdown_connections 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495d1ff700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580af400 0x7f49580af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495d1ff700 1 --2- 172.21.15.29:0/2661344672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f495815a340 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.269+0000 7f495d1ff700 1 -- 172.21.15.29:0/2661344672 >> 172.21.15.29:0/2661344672 conn(0x7f49580680a0 msgr2=0x7f49580cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.270+0000 7f495d1ff700 1 -- 172.21.15.29:0/2661344672 shutdown_connections 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.270+0000 7f495d1ff700 1 -- 172.21.15.29:0/2661344672 wait complete. 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.271+0000 7f495d1ff700 1 Processor -- start 2023-10-07T10:45:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.271+0000 7f495d1ff700 1 -- start start 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495d1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580ad370 0x7f4958157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495d1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49580af400 0x7f49580cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495d1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f49580cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495d1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4958157850 con 0x7f4958159ec0 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495d1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49580aa200 con 0x7f49580af400 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495d1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495816a510 con 0x7f49580ad370 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f49580cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f49580cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56982/0 (socket says 172.21.15.29:56982) 2023-10-07T10:45:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.272+0000 7f495759e700 1 -- 172.21.15.29:0/3967227751 learned_addr learned my addr 172.21.15.29:0/3967227751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495659c700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49580af400 0x7f49580cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f4956d9d700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580ad370 0x7f4958157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495759e700 1 -- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580ad370 msgr2=0x7f4958157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495759e700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580ad370 0x7f4958157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495759e700 1 -- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49580af400 msgr2=0x7f49580cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495759e700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49580af400 0x7f49580cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495759e700 1 -- 172.21.15.29:0/3967227751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0094f0 con 0x7f4958159ec0 2023-10-07T10:45:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f4956d9d700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580ad370 0x7f4958157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495759e700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f49580cbb70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f495000d1b0 tx=0x7f4950009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495000dda0 con 0x7f4958159ec0 2023-10-07T10:45:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495000f040 con 0x7f4958159ec0 2023-10-07T10:45:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950015690 con 0x7f4958159ec0 2023-10-07T10:45:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.273+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49580a98d0 con 0x7f4958159ec0 2023-10-07T10:45:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.274+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49580a9e90 con 0x7f4958159ec0 2023-10-07T10:45:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.274+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495804c3c0 con 0x7f4958159ec0 2023-10-07T10:45:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.275+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f495001e3b0 con 0x7f4958159ec0 2023-10-07T10:45:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.277+0000 7f493effd700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4944066f30 0x7f49440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.277+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f4950084ca0 con 0x7f4958159ec0 2023-10-07T10:45:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.278+0000 7f4956d9d700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4944066f30 0x7f49440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.279+0000 7f4956d9d700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4944066f30 0x7f49440693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f495814d3d0 tx=0x7f4948009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:27.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.282+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4950051cd0 con 0x7f4958159ec0 2023-10-07T10:45:27.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.509+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f49580b1e90 con 0x7f4944066f30 2023-10-07T10:45:27.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.511+0000 7f493effd700 1 -- 172.21.15.29:0/3967227751 <== mgr.8449 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f49580b1e90 con 0x7f4944066f30 2023-10-07T10:45:27.512 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T10:45:27.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T10:45:27.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T10:45:27.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T10:45:27.513 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4944066f30 msgr2=0x7f49440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4944066f30 0x7f49440693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f495814d3d0 tx=0x7f4948009040 comp rx=0 tx=0).stop 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 msgr2=0x7f49580cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f49580cbb70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f495000d1b0 tx=0x7f4950009a50 comp rx=0 tx=0).stop 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 shutdown_connections 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f4944066f30 0x7f49440693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4958159ec0 0x7f49580cbb70 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49580ad370 0x7f4958157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 --2- 172.21.15.29:0/3967227751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49580af400 0x7f49580cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 >> 172.21.15.29:0/3967227751 conn(0x7f49580680a0 msgr2=0x7f4958069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 shutdown_connections 2023-10-07T10:45:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.514+0000 7f495d1ff700 1 -- 172.21.15.29:0/3967227751 wait complete. 2023-10-07T10:45:27.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T10:45:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.730+0000 7f7f1b3ff700 1 Processor -- start 2023-10-07T10:45:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.730+0000 7f7f1b3ff700 1 -- start start 2023-10-07T10:45:27.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1b3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f14153cc0 0x7f7f14159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f1414cca0 0x7f7f14159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1b3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f14157790 con 0x7f7f1414c8d0 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1b3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f140b0950 con 0x7f7f14153cc0 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14157580 con 0x7f7f1414cca0 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f14153cc0 0x7f7f14159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f14153cc0 0x7f7f14159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56994/0 (socket says 172.21.15.29:56994) 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1899a700 1 -- 172.21.15.29:0/1904507180 learned_addr learned my addr 172.21.15.29:0/1904507180 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.732+0000 7f7f1999c700 1 --2- 172.21.15.29:0/1904507180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f1414cca0 0x7f7f14159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f1899a700 1 -- 172.21.15.29:0/1904507180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f1414cca0 msgr2=0x7f7f14159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f1899a700 1 --2- 172.21.15.29:0/1904507180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f1414cca0 0x7f7f14159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f1899a700 1 -- 172.21.15.29:0/1904507180 >> v1:172.21.15.144:6789/0 conn(0x7f7f1414c8d0 legacy=0x7f7f140afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f1899a700 1 -- 172.21.15.29:0/1904507180 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14157a70 con 0x7f7f14153cc0 2023-10-07T10:45:27.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f1899a700 1 --2- 172.21.15.29:0/1904507180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f14153cc0 0x7f7f14159680 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f7f141576c0 tx=0x7f7f0400a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03fc7581574696a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f0bfff700 1 -- 172.21.15.29:0/1904507180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0400ca40 con 0x7f7f14153cc0 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f0bfff700 1 -- 172.21.15.29:0/1904507180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0400cba0 con 0x7f7f14153cc0 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.733+0000 7f7f0bfff700 1 -- 172.21.15.29:0/1904507180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f040183f0 con 0x7f7f14153cc0 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/1904507180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f14153cc0 msgr2=0x7f7f14159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/1904507180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f14153cc0 0x7f7f14159680 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f7f141576c0 tx=0x7f7f0400a3f0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/1904507180 shutdown_connections 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/1904507180 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f1414cca0 0x7f7f14159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/1904507180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f14153cc0 0x7f7f14159680 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/1904507180 >> 172.21.15.29:0/1904507180 conn(0x7f7f14067610 msgr2=0x7f7f14067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/1904507180 shutdown_connections 2023-10-07T10:45:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.734+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/1904507180 wait complete. 2023-10-07T10:45:27.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.736+0000 7f7f1b3ff700 1 Processor -- start 2023-10-07T10:45:27.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.736+0000 7f7f1b3ff700 1 -- start start 2023-10-07T10:45:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.737+0000 7f7f1b3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1414c8d0 0x7f7f1414fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1b3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 0x7f7f1414d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f14153cc0 0x7f7f1414db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1b3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14169510 con 0x7f7f1414cca0 2023-10-07T10:45:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1b3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14168920 con 0x7f7f1414c8d0 2023-10-07T10:45:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14168be0 con 0x7f7f14153cc0 2023-10-07T10:45:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 0x7f7f1414d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 0x7f7f1414d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56996/0 (socket says 172.21.15.29:56996) 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 -- 172.21.15.29:0/214400853 learned_addr learned my addr 172.21.15.29:0/214400853 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1999c700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f14153cc0 0x7f7f1414db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1919b700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1414c8d0 0x7f7f1414fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 -- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f14153cc0 msgr2=0x7f7f1414db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f14153cc0 0x7f7f1414db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 -- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1414c8d0 msgr2=0x7f7f1414fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1414c8d0 0x7f7f1414fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.738+0000 7f7f1899a700 1 -- 172.21.15.29:0/214400853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0400a690 con 0x7f7f1414cca0 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.739+0000 7f7f1899a700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 0x7f7f1414d650 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400a430 tx=0x7f7f04019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.739+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04021450 con 0x7f7f1414cca0 2023-10-07T10:45:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.739+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f040215b0 con 0x7f7f1414cca0 2023-10-07T10:45:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.739+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f14168ec0 con 0x7f7f1414cca0 2023-10-07T10:45:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.739+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04021930 con 0x7f7f1414cca0 2023-10-07T10:45:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.739+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f14150410 con 0x7f7f1414cca0 2023-10-07T10:45:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.740+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1404c3c0 con 0x7f7f1414cca0 2023-10-07T10:45:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.741+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7f040103f0 con 0x7f7f1414cca0 2023-10-07T10:45:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.742+0000 7f7f097fa700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7ef8066f30 0x7f7ef80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:27.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.742+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f7f04085f20 con 0x7f7f1414cca0 2023-10-07T10:45:27.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.742+0000 7f7f1919b700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7ef8066f30 0x7f7ef80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:27.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.743+0000 7f7f1919b700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7ef8066f30 0x7f7ef80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f100039e0 tx=0x7f7f10007ec0 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:27.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:27.746+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f04052f50 con 0x7f7f1414cca0 2023-10-07T10:45:28.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.004+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7f140aea80 con 0x7f7f1414cca0 2023-10-07T10:45:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.183+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 75) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f7f04089030 con 0x7f7f1414cca0 2023-10-07T10:45:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.184+0000 7f7f097fa700 1 -- 172.21.15.29:0/214400853 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v75) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7f040569e0 con 0x7f7f1414cca0 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7ef8066f30 msgr2=0x7f7ef80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7ef8066f30 0x7f7ef80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f100039e0 tx=0x7f7f10007ec0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 msgr2=0x7f7f1414d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 0x7f7f1414d650 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400a430 tx=0x7f7f04019000 comp rx=0 tx=0).stop 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 shutdown_connections 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f7ef8066f30 0x7f7ef80693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7f14153cc0 0x7f7f1414db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1414c8d0 0x7f7f1414fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 --2- 172.21.15.29:0/214400853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f1414cca0 0x7f7f1414d650 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 >> 172.21.15.29:0/214400853 conn(0x7f7f14067610 msgr2=0x7f7f140afcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:28.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 shutdown_connections 2023-10-07T10:45:28.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.187+0000 7f7f1b3ff700 1 -- 172.21.15.29:0/214400853 wait complete. 2023-10-07T10:45:28.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T10:45:28.395 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:28.394+0000 7f7178ea61c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:45:28.405 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:28.404+0000 7f3b3e9a81c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:45:28.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.414+0000 7fc8699ff700 1 Processor -- start 2023-10-07T10:45:28.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.415+0000 7fc8699ff700 1 -- start start 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc8699ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc864156180 0x7fc8640afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc8699ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8640b0520 0x7fc8640b0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc8699ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640b0e80 con 0x7fc8640ae570 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc8699ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640ad4d0 con 0x7fc864156180 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc8699ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640ad7b0 con 0x7fc8640b0520 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc862ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc864156180 0x7fc8640afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc862ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc864156180 0x7fc8640afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57036/0 (socket says 172.21.15.29:57036) 2023-10-07T10:45:28.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc862ffd700 1 -- 172.21.15.29:0/3438785301 learned_addr learned my addr 172.21.15.29:0/3438785301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.416+0000 7fc8637fe700 1 --2- 172.21.15.29:0/3438785301 >> v2:172.21.15.144:3300/0 conn(0x7fc8640b0520 0x7fc8640b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc861ffb700 1 -- 172.21.15.29:0/3438785301 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411248761 0 0) 0x7fc8640b0e80 con 0x7fc8640ae570 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc862ffd700 1 -- 172.21.15.29:0/3438785301 >> v2:172.21.15.144:3300/0 conn(0x7fc8640b0520 msgr2=0x7fc8640b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc862ffd700 1 --2- 172.21.15.29:0/3438785301 >> v2:172.21.15.144:3300/0 conn(0x7fc8640b0520 0x7fc8640b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc862ffd700 1 -- 172.21.15.29:0/3438785301 >> v1:172.21.15.144:6789/0 conn(0x7fc8640ae570 legacy=0x7fc8640accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc862ffd700 1 -- 172.21.15.29:0/3438785301 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640ada90 con 0x7fc864156180 2023-10-07T10:45:28.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc862ffd700 1 --2- 172.21.15.29:0/3438785301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc8640afc00 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fc86406d230 tx=0x7fc85400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce49ec3d757a3ba server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc861ffb700 1 -- 172.21.15.29:0/3438785301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85400dba0 con 0x7fc864156180 2023-10-07T10:45:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.417+0000 7fc861ffb700 1 -- 172.21.15.29:0/3438785301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85400dd00 con 0x7fc864156180 2023-10-07T10:45:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc861ffb700 1 -- 172.21.15.29:0/3438785301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540196d0 con 0x7fc864156180 2023-10-07T10:45:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 -- 172.21.15.29:0/3438785301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 msgr2=0x7fc8640afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 --2- 172.21.15.29:0/3438785301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc8640afc00 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fc86406d230 tx=0x7fc85400b4f0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 -- 172.21.15.29:0/3438785301 shutdown_connections 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 --2- 172.21.15.29:0/3438785301 >> v2:172.21.15.144:3300/0 conn(0x7fc8640b0520 0x7fc8640b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 --2- 172.21.15.29:0/3438785301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc8640afc00 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 -- 172.21.15.29:0/3438785301 >> 172.21.15.29:0/3438785301 conn(0x7fc864067610 msgr2=0x7fc864067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 -- 172.21.15.29:0/3438785301 shutdown_connections 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.418+0000 7fc8699ff700 1 -- 172.21.15.29:0/3438785301 wait complete. 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.420+0000 7fc8699ff700 1 Processor -- start 2023-10-07T10:45:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.420+0000 7fc8699ff700 1 -- start start 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8699ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640ae570 0x7fc8640c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8699ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640b0520 0x7fc864144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8699ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc86406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8699ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8641459a0 con 0x7fc864156180 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8699ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640c50c0 con 0x7fc8640b0520 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8699ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864144d90 con 0x7fc8640ae570 2023-10-07T10:45:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc86406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc862ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640ae570 0x7fc8640c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc8627fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640b0520 0x7fc864144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc862ffd700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640ae570 0x7fc8640c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:46122/0 (socket says 172.21.15.29:46122) 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.421+0000 7fc862ffd700 1 -- 172.21.15.29:0/69604386 learned_addr learned my addr 172.21.15.29:0/69604386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8637fe700 1 -- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640ae570 msgr2=0x7fc8640c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8637fe700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640ae570 0x7fc8640c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8637fe700 1 -- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640b0520 msgr2=0x7fc864144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8637fe700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640b0520 0x7fc864144540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8637fe700 1 -- 172.21.15.29:0/69604386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85400b790 con 0x7fc864156180 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8627fc700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640b0520 0x7fc864144540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:28.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8637fe700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc86406d480 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009e50 tx=0x7fc85c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c00caa0 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c00f040 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.422+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc864145070 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.423+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c00ce50 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.423+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc864145630 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.424+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86404c3c0 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.425+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fc85c014690 con 0x7fc864156180 2023-10-07T10:45:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.425+0000 7fc84affd700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 0x7fc850069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.425+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fc85c085430 con 0x7fc864156180 2023-10-07T10:45:28.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.426+0000 7fc862ffd700 1 -- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 msgr2=0x7fc850069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:28.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.426+0000 7fc862ffd700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 0x7fc850069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:28.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.430+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc85c055ef0 con 0x7fc864156180 2023-10-07T10:45:28.571 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:28.570+0000 7f7178ea61c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:45:28.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:28.607+0000 7f3b3e9a81c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:45:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.627+0000 7fc862ffd700 1 -- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 msgr2=0x7fc850069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.627+0000 7fc862ffd700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 0x7fc850069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.670+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc864145ae0 con 0x7fc864156180 2023-10-07T10:45:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.674+0000 7fc84affd700 1 -- 172.21.15.29:0/69604386 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc85c055a80 con 0x7fc864156180 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 msgr2=0x7fc850069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 0x7fc850069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 msgr2=0x7fc86406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc86406d480 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009e50 tx=0x7fc85c00b040 comp rx=0 tx=0).stop 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 shutdown_connections 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fc850066f80 0x7fc850069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc864156180 0x7fc86406d480 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640ae570 0x7fc8640c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 --2- 172.21.15.29:0/69604386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640b0520 0x7fc864144540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 >> 172.21.15.29:0/69604386 conn(0x7fc864067610 msgr2=0x7fc864157030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 shutdown_connections 2023-10-07T10:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.677+0000 7fc8699ff700 1 -- 172.21.15.29:0/69604386 wait complete. 2023-10-07T10:45:28.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T10:45:28.883 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:28.882+0000 7f7178ea61c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:45:28.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.897+0000 7f83e6dff700 1 Processor -- start 2023-10-07T10:45:28.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.897+0000 7f83e6dff700 1 -- start start 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e6dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f83e015a960 0x7f83e0158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e6dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e6dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83e00ae6d0 0x7f83e00b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e6dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e00aeb50 con 0x7f83e00ae6d0 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e6dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e00acb40 con 0x7f83e015a960 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e6dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e00abf30 con 0x7f83e015ad30 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e539c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83e00ae6d0 0x7f83e00b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83dffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e4b9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f83e015a960 0x7f83e0158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e539c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83e00ae6d0 0x7f83e00b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57056/0 (socket says 172.21.15.29:57056) 2023-10-07T10:45:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.899+0000 7f83e539c700 1 -- 172.21.15.29:0/3211256803 learned_addr learned my addr 172.21.15.29:0/3211256803 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:28.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e539c700 1 -- 172.21.15.29:0/3211256803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 msgr2=0x7f83e00ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:28.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e539c700 1 --2- 172.21.15.29:0/3211256803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e539c700 1 -- 172.21.15.29:0/3211256803 >> v2:172.21.15.144:3300/0 conn(0x7f83e015a960 msgr2=0x7f83e0158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e539c700 1 --2- 172.21.15.29:0/3211256803 >> v2:172.21.15.144:3300/0 conn(0x7f83e015a960 0x7f83e0158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e539c700 1 -- 172.21.15.29:0/3211256803 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e00ac210 con 0x7f83e00ae6d0 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83dffff700 1 --2- 172.21.15.29:0/3211256803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e4b9b700 1 --2- 172.21.15.29:0/3211256803 >> v2:172.21.15.144:3300/0 conn(0x7f83e015a960 0x7f83e0158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83e539c700 1 --2- 172.21.15.29:0/3211256803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e00b1290 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f83e00aecc0 tx=0x7f83d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d522e26e761a8c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83df7fe700 1 -- 172.21.15.29:0/3211256803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d800e070 con 0x7f83e00ae6d0 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83df7fe700 1 -- 172.21.15.29:0/3211256803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d800b970 con 0x7f83e00ae6d0 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.900+0000 7f83df7fe700 1 -- 172.21.15.29:0/3211256803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80092e0 con 0x7f83e00ae6d0 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 -- 172.21.15.29:0/3211256803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 msgr2=0x7f83e00b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 --2- 172.21.15.29:0/3211256803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e00b1290 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f83e00aecc0 tx=0x7f83d8009840 comp rx=0 tx=0).stop 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 -- 172.21.15.29:0/3211256803 shutdown_connections 2023-10-07T10:45:28.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 --2- 172.21.15.29:0/3211256803 >> v2:172.21.15.144:3300/0 conn(0x7f83e015a960 0x7f83e0158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 --2- 172.21.15.29:0/3211256803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e00b1290 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 --2- 172.21.15.29:0/3211256803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 -- 172.21.15.29:0/3211256803 >> 172.21.15.29:0/3211256803 conn(0x7f83e0067610 msgr2=0x7f83e0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 -- 172.21.15.29:0/3211256803 shutdown_connections 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.901+0000 7f83e6dff700 1 -- 172.21.15.29:0/3211256803 wait complete. 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.902+0000 7f83e6dff700 1 Processor -- start 2023-10-07T10:45:28.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.903+0000 7f83e6dff700 1 -- start start 2023-10-07T10:45:28.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e6dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e6dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83e015a960 0x7f83e0156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e6dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e6dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e015c850 con 0x7f83e00ae6d0 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e6dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e016a360 con 0x7f83e015a960 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e6dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e006f2e0 con 0x7f83e015ad30 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e015c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57064/0 (socket says 172.21.15.29:57064) 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 -- 172.21.15.29:0/32193746 learned_addr learned my addr 172.21.15.29:0/32193746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e539c700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83dffff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83e015a960 0x7f83e0156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 -- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 msgr2=0x7f83e00c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 -- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83e015a960 msgr2=0x7f83e0156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83e015a960 0x7f83e0156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.904+0000 7f83e4b9b700 1 -- 172.21.15.29:0/32193746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d4009420 con 0x7f83e00ae6d0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.905+0000 7f83e4b9b700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e015c310 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f83d400ec80 tx=0x7f83d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.905+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400c920 con 0x7f83e00ae6d0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.905+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d400ca80 con 0x7f83e00ae6d0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.905+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83cc009420 con 0x7f83e00ae6d0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.905+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400ce00 con 0x7f83e00ae6d0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.905+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d8009ae0 con 0x7f83e00ae6d0 2023-10-07T10:45:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.906+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e016cb30 con 0x7f83e00ae6d0 2023-10-07T10:45:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.907+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f83d4013680 con 0x7f83e00ae6d0 2023-10-07T10:45:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.908+0000 7f83dcff9700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 0x7f83c4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.908+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f83d4084b80 con 0x7f83e00ae6d0 2023-10-07T10:45:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.908+0000 7f83dffff700 1 -- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 msgr2=0x7f83c4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.908+0000 7f83dffff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 0x7f83c4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:28.912+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83d4051bb0 con 0x7f83e00ae6d0 2023-10-07T10:45:28.945 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:28.944+0000 7f3b3e9a81c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:45:29.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.109+0000 7f83dffff700 1 -- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 msgr2=0x7f83c4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:29.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.109+0000 7f83dffff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 0x7f83c4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:29.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.113+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f83e00abbe0 con 0x7f83e00ae6d0 2023-10-07T10:45:29.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.117+0000 7f83dcff9700 1 -- 172.21.15.29:0/32193746 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f83d40551d0 con 0x7f83e00ae6d0 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.119+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 msgr2=0x7f83c4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.119+0000 7f83e6dff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 0x7f83c4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.119+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 msgr2=0x7f83e015c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.119+0000 7f83e6dff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e015c310 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f83d400ec80 tx=0x7f83d40091a0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 shutdown_connections 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f83c4066f80 0x7f83c4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83e015ad30 0x7f83e00c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e00ae6d0 0x7f83e015c310 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 --2- 172.21.15.29:0/32193746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83e015a960 0x7f83e0156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 >> 172.21.15.29:0/32193746 conn(0x7f83e0067610 msgr2=0x7f83e00695f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 shutdown_connections 2023-10-07T10:45:29.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.120+0000 7f83e6dff700 1 -- 172.21.15.29:0/32193746 wait complete. 2023-10-07T10:45:29.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:45:29.140 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T10:45:29.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:45:29.191 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-07T10:45:29.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/test_restart 1M 2023-10-07T10:45:29.401 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:29.400+0000 7f7178ea61c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.448+0000 7fd0b4bff700 1 Processor -- start 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.448+0000 7fd0b4bff700 1 -- start start 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0b4bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b00878b0 0x7fd0b0166710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0b4bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd0b00cc6c0 0x7fd0b0167e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0b4bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b01679a0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0b4bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b0166d90 con 0x7fd0b00878b0 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0b4bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b0167070 con 0x7fd0b00cc6c0 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0ae59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b00878b0 0x7fd0b0166710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0ae59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b00878b0 0x7fd0b0166710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57072/0 (socket says 172.21.15.29:57072) 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0ae59c700 1 -- 172.21.15.29:0/3826870430 learned_addr learned my addr 172.21.15.29:0/3826870430 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0ad59a700 1 -- 172.21.15.29:0/3826870430 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190875245 0 0) 0x7fd0b01679a0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.450+0000 7fd0aed9d700 1 --2- 172.21.15.29:0/3826870430 >> v2:172.21.15.144:3300/0 conn(0x7fd0b00cc6c0 0x7fd0b0167e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.498+0000 7fd0ad59a700 1 -- 172.21.15.29:0/3826870430 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd098003550 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.498+0000 7fd0ae59c700 1 -- 172.21.15.29:0/3826870430 >> v2:172.21.15.144:3300/0 conn(0x7fd0b00cc6c0 msgr2=0x7fd0b0167e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.498+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/3826870430 >> v2:172.21.15.144:3300/0 conn(0x7fd0b00cc6c0 0x7fd0b0167e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.498+0000 7fd0ad59a700 1 -- 172.21.15.29:0/3826870430 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 855333543 0 0) 0x7fd098003550 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.498+0000 7fd0ae59c700 1 -- 172.21.15.29:0/3826870430 >> v1:172.21.15.144:6789/0 conn(0x7fd0b00b1aa0 legacy=0x7fd0b00cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.498+0000 7fd0ae59c700 1 -- 172.21.15.29:0/3826870430 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b0167350 con 0x7fd0b00878b0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.499+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/3826870430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00878b0 0x7fd0b0166710 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fd0b006b9c0 tx=0x7fd0a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed123389d7a64934 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.499+0000 7fd0ad59a700 1 -- 172.21.15.29:0/3826870430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4001470 con 0x7fd0b00878b0 2023-10-07T10:45:29.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.499+0000 7fd0ad59a700 1 -- 172.21.15.29:0/3826870430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0011420 con 0x7fd0b00878b0 2023-10-07T10:45:29.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.499+0000 7fd0ad59a700 1 -- 172.21.15.29:0/3826870430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00117c0 con 0x7fd0b00878b0 2023-10-07T10:45:29.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 -- 172.21.15.29:0/3826870430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00878b0 msgr2=0x7fd0b0166710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/3826870430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00878b0 0x7fd0b0166710 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fd0b006b9c0 tx=0x7fd0a0009250 comp rx=0 tx=0).stop 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 -- 172.21.15.29:0/3826870430 shutdown_connections 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/3826870430 >> v2:172.21.15.144:3300/0 conn(0x7fd0b00cc6c0 0x7fd0b0167e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/3826870430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00878b0 0x7fd0b0166710 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 -- 172.21.15.29:0/3826870430 >> 172.21.15.29:0/3826870430 conn(0x7fd0b00680e0 msgr2=0x7fd0b006b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 -- 172.21.15.29:0/3826870430 shutdown_connections 2023-10-07T10:45:29.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.500+0000 7fd0b4bff700 1 -- 172.21.15.29:0/3826870430 wait complete. 2023-10-07T10:45:29.503 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:29.501+0000 7f3b3e9a81c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:45:29.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.501+0000 7fd0b4bff700 1 Processor -- start 2023-10-07T10:45:29.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.502+0000 7fd0b4bff700 1 -- start start 2023-10-07T10:45:29.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0b4bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0b00878b0 0x7fd0b01772e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0b4bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 0x7fd0b00a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0b4bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0b00cc6c0 0x7fd0b00aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0b4bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b0177820 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0b4bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00a6e90 con 0x7fd0b00cc6c0 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0b4bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00a6a00 con 0x7fd0b00878b0 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0add9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 0x7fd0b00a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0add9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 0x7fd0b00a54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57078/0 (socket says 172.21.15.29:57078) 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0aed9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0b00cc6c0 0x7fd0b00aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0ae59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0b00878b0 0x7fd0b01772e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.503+0000 7fd0add9b700 1 -- 172.21.15.29:0/4181444465 learned_addr learned my addr 172.21.15.29:0/4181444465 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:29.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0add9b700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0b00878b0 msgr2=0x7fd0b01772e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0add9b700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0b00878b0 0x7fd0b01772e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0add9b700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0b00cc6c0 msgr2=0x7fd0b00aa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0add9b700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0b00cc6c0 0x7fd0b00aa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0add9b700 1 -- 172.21.15.29:0/4181444465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00094f0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0aed9d700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0b00cc6c0 0x7fd0b00aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0add9b700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 0x7fd0b00a54c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400cc30 tx=0x7fd0a400a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4001b50 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4001cb0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b00a5df0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4007aa0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.504+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b00a63b0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.505+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b00a6bb0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.507+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fd0a40030f0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.508+0000 7fd09e7fc700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:29.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.509+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fd0a4084d30 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.509+0000 7fd0ae59c700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd090069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:29.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.509+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T10:45:29.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.512+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0b00a6bb0 con 0x7fd0b00b1aa0 2023-10-07T10:45:29.570 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:29.569+0000 7f7178ea61c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:45:29.682 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:29.682+0000 7f3b3e9a81c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:45:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.710+0000 7fd0ae59c700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd090069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.710+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T10:45:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:29.742+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b00718e0 con 0x7fd090066fb0 2023-10-07T10:45:29.982 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:29.981+0000 7f7178ea61c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:45:30.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:30.111+0000 7fd0ae59c700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd090069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:30.111+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T10:45:30.113 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:30.112+0000 7f3b3e9a81c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:45:30.753 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:30.753+0000 7f7178ea61c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:45:30.840 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:30.839+0000 7f7178ea61c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:45:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:30.912+0000 7fd0ae59c700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd090069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:30.912+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T10:45:30.920 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:30.919+0000 7f3b3e9a81c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:45:30.992 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:30.992+0000 7f7178ea61c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:45:31.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.010+0000 7f3b3e9a81c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:45:31.079 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.078+0000 7f7178ea61c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:45:31.173 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.172+0000 7f3b3e9a81c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:45:31.264 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.263+0000 7f3b3e9a81c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:45:31.296 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.295+0000 7f7178ea61c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:45:31.389 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.388+0000 7f7178ea61c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:45:31.482 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.481+0000 7f7178ea61c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:45:31.499 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.498+0000 7f3b3e9a81c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:45:31.574 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.573+0000 7f7178ea61c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:45:31.598 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.598+0000 7f3b3e9a81c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:45:31.698 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.697+0000 7f3b3e9a81c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:45:31.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.759+0000 7f7178ea61c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:45:31.798 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.797+0000 7f3b3e9a81c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:45:31.856 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:31.855+0000 7f7178ea61c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:45:31.995 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:31.994+0000 7f3b3e9a81c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:45:32.096 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:32.095+0000 7f3b3e9a81c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:45:32.195 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:32.194+0000 7f7178ea61c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:45:32.282 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:32.281+0000 7f7178ea61c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:32.445 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:32.444+0000 7f3b3e9a81c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:45:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:32.514+0000 7fd0ae59c700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd090069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/114714 2023-10-07T10:45:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:32.514+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T10:45:32.539 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:32.538+0000 7f3b3e9a81c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:32.818 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:32.817+0000 7f7178ea61c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:45:33.002 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:33.001+0000 7f7178ea61c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:45:33.110 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:33.109+0000 7f3b3e9a81c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:45:33.303 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:33.302+0000 7f3b3e9a81c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:45:33.375 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:33.374+0000 7f7178ea61c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:33.686 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:33.685+0000 7f3b3e9a81c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:33.795 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:33.794+0000 7f7178ea61c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:45:33.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:33.974+0000 7f7178ea61c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:34.134 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:34.132+0000 7f3b3e9a81c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:45:34.161 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.160+0000 7f7178ea61c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:45:34.269 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.268+0000 7f7178ea61c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:34.278+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 76) v1 ==== 84468+0+0 (secure 0 0 0) 0x7fd0a40521b0 con 0x7fd0b00b1aa0 2023-10-07T10:45:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:34.278+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd090069460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T10:45:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:34.278+0000 7fd09e7fc700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd090069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:34.328 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:34.328+0000 7f3b3e9a81c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:45:34.417 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ebc74700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ebc74700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ebc74700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ebc74700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ebc74700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ea471700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ea471700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ea471700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ea471700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.418 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:45:34.416+0000 7f70ea471700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:45:34.535 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:34.534+0000 7f3b3e9a81c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:45:34.648 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T10:45:34.647+0000 7f3b3e9a81c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:45:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.283+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 77) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fd0a4014300 con 0x7fd0b00b1aa0 2023-10-07T10:45:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.283+0000 7fd09e7fc700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd0900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.283+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b00718e0 con 0x7fd090066fb0 2023-10-07T10:45:35.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.284+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd0900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.286+0000 7fd0ae59c700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd0900693e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0b006b9c0 tx=0x7fd0a0016000 comp rx=0 tx=0).ready entity=mgr.8785 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.356+0000 7fd09e7fc700 1 -- 172.21.15.29:0/4181444465 <== mgr.8785 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd0b00718e0 con 0x7fd090066fb0 2023-10-07T10:45:35.357 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-07T10:45:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 msgr2=0x7fd0900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd0900693e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0b006b9c0 tx=0x7fd0a0016000 comp rx=0 tx=0).stop 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 msgr2=0x7fd0b00a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 0x7fd0b00a54c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400cc30 tx=0x7fd0a400a530 comp rx=0 tx=0).stop 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 shutdown_connections 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fd090066fb0 0x7fd0900693e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0b00cc6c0 0x7fd0b00aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0b00878b0 0x7fd0b01772e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 --2- 172.21.15.29:0/4181444465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00b1aa0 0x7fd0b00a54c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 >> 172.21.15.29:0/4181444465 conn(0x7fd0b00680e0 msgr2=0x7fd0b00af290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 shutdown_connections 2023-10-07T10:45:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.358+0000 7fd0b4bff700 1 -- 172.21.15.29:0/4181444465 wait complete. 2023-10-07T10:45:35.378 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-07T10:45:35.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/test_restart 1M 2023-10-07T10:45:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.579+0000 7fef2bdff700 1 Processor -- start 2023-10-07T10:45:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.579+0000 7fef2bdff700 1 -- start start 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.580+0000 7fef2bdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fef240b0610 0x7fef240ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.580+0000 7fef2bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240c8a20 0x7fef240cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.580+0000 7fef2bdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef240690f0 con 0x7fef240b09e0 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.580+0000 7fef2bdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef240cf2e0 con 0x7fef240b0610 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.580+0000 7fef2bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef240687c0 con 0x7fef240c8a20 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fef240b0610 0x7fef240ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fef240b0610 0x7fef240ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43626/0 (socket says 172.21.15.29:43626) 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 -- 172.21.15.29:0/3737049090 learned_addr learned my addr 172.21.15.29:0/3737049090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef2a39c700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240c8a20 0x7fef240cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 -- 172.21.15.29:0/3737049090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240c8a20 msgr2=0x7fef240cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240c8a20 0x7fef240cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 -- 172.21.15.29:0/3737049090 >> v1:172.21.15.29:6789/0 conn(0x7fef240b09e0 legacy=0x7fef240c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 -- 172.21.15.29:0/3737049090 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24068aa0 con 0x7fef240b0610 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef2a39c700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240c8a20 0x7fef240cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef28b99700 1 -- 172.21.15.29:0/3737049090 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070166687 0 0) 0x7fef240690f0 con 0x7fef240b09e0 2023-10-07T10:45:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.581+0000 7fef29b9b700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240b0610 0x7fef240ced00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fef200099b0 tx=0x7fef20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8705908a6094bbee server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef28b99700 1 -- 172.21.15.29:0/3737049090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000b860 con 0x7fef240b0610 2023-10-07T10:45:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef28b99700 1 -- 172.21.15.29:0/3737049090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2000b9c0 con 0x7fef240b0610 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef28b99700 1 -- 172.21.15.29:0/3737049090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000bd60 con 0x7fef240b0610 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 -- 172.21.15.29:0/3737049090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240b0610 msgr2=0x7fef240ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240b0610 0x7fef240ced00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fef200099b0 tx=0x7fef20009250 comp rx=0 tx=0).stop 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 -- 172.21.15.29:0/3737049090 shutdown_connections 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240c8a20 0x7fef240cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3737049090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240b0610 0x7fef240ced00 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 -- 172.21.15.29:0/3737049090 >> 172.21.15.29:0/3737049090 conn(0x7fef24063a10 msgr2=0x7fef240ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 -- 172.21.15.29:0/3737049090 shutdown_connections 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.582+0000 7fef2bdff700 1 -- 172.21.15.29:0/3737049090 wait complete. 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.584+0000 7fef2bdff700 1 Processor -- start 2023-10-07T10:45:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.584+0000 7fef2bdff700 1 -- start start 2023-10-07T10:45:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2bdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240b0610 0x7fef24169940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2bdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef240b09e0 0x7fef240c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2bdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 0x7fef240cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2bdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef240b1340 con 0x7fef240c8a20 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2bdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2416a8e0 con 0x7fef240b09e0 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2bdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2416a040 con 0x7fef240b0610 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2a39c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 0x7fef240cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2a39c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 0x7fef240cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43632/0 (socket says 172.21.15.29:43632) 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2a39c700 1 -- 172.21.15.29:0/3571830412 learned_addr learned my addr 172.21.15.29:0/3571830412 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef29b9b700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240b0610 0x7fef24169940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.585+0000 7fef2939a700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef240b09e0 0x7fef240c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2a39c700 1 -- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240b0610 msgr2=0x7fef24169940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2a39c700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240b0610 0x7fef24169940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2a39c700 1 -- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef240b09e0 msgr2=0x7fef240c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2a39c700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef240b09e0 0x7fef240c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2a39c700 1 -- 172.21.15.29:0/3571830412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef200094f0 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2a39c700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 0x7fef240cd2d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00d180 tx=0x7fef1c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c00dcc0 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c00f040 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c015690 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2416a320 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.586+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef240c7b40 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.587+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef240718b0 con 0x7fef240c8a20 2023-10-07T10:45:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.588+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fef1c01e3b0 con 0x7fef240c8a20 2023-10-07T10:45:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.589+0000 7fef19ffb700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fef08066dd0 0x7fef08069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:45:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.589+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fef1c0849c0 con 0x7fef240c8a20 2023-10-07T10:45:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.590+0000 7fef29b9b700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fef08066dd0 0x7fef08069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:45:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.590+0000 7fef29b9b700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fef08066dd0 0x7fef08069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef200099b0 tx=0x7fef20017000 comp rx=0 tx=0).ready entity=mgr.8785 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:45:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.594+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef1c051b70 con 0x7fef240c8a20 2023-10-07T10:45:35.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.821+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fef240b1480 con 0x7fef08066dd0 2023-10-07T10:45:35.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.871+0000 7fef19ffb700 1 -- 172.21.15.29:0/3571830412 <== mgr.8785 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fef240b1480 con 0x7fef08066dd0 2023-10-07T10:45:35.873 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-07T10:45:35.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fef08066dd0 msgr2=0x7fef08069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fef08066dd0 0x7fef08069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef200099b0 tx=0x7fef20017000 comp rx=0 tx=0).stop 2023-10-07T10:45:35.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 msgr2=0x7fef240cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:45:35.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 0x7fef240cd2d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00d180 tx=0x7fef1c009a50 comp rx=0 tx=0).stop 2023-10-07T10:45:35.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 shutdown_connections 2023-10-07T10:45:35.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7fef08066dd0 0x7fef08069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef240c8a20 0x7fef240cd2d0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef240b0610 0x7fef24169940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 --2- 172.21.15.29:0/3571830412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef240b09e0 0x7fef240c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:45:35.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 >> 172.21.15.29:0/3571830412 conn(0x7fef24063a10 msgr2=0x7fef241555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:45:35.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 shutdown_connections 2023-10-07T10:45:35.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:45:35.874+0000 7fef2bdff700 1 -- 172.21.15.29:0/3571830412 wait complete. 2023-10-07T10:45:35.892 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-07T10:45:37.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:45:37.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:46:00.886 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:46:00.886+0000 7f70dc455700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:46:00.886 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:46:00.886+0000 7f70dc455700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:46:00.887 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:46:00.886+0000 7f70dc455700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:46:00.887 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:46:00.886+0000 7f70dc455700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:46:00.887 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:46:00.886+0000 7f70dc455700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:46:07.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:46:07.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:46:38.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:46:38.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:47:08.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:47:08.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:47:38.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:47:38.461 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:48:08.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:48:08.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:48:38.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:48:38.828 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:49:09.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:49:09.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:49:39.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:49:39.238 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:50:09.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:50:09.441 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:50:39.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:50:39.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:51:09.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:51:09.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:51:39.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:51:39.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:52:10.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:52:10.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:52:40.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:52:40.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:53:10.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:53:10.311 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:53:40.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:53:40.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:54:10.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:54:10.559 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:54:40.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:54:40.784 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:55:10.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:55:10.962 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:55:41.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:55:41.125 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:56:11.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:56:11.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:56:15.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T10:56:15.894 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_46_00_UTC 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_47_00_UTC 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_48_00_UTC 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_49_00_UTC 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_50_00_UTC 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_51_00_UTC 2023-10-07T10:56:15.948 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_52_00_UTC 2023-10-07T10:56:15.949 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_53_00_UTC 2023-10-07T10:56:15.949 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_54_00_UTC 2023-10-07T10:56:15.949 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_55_00_UTC 2023-10-07T10:56:15.949 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_56_00_UTC 2023-10-07T10:56:15.950 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-10_46_00_UTC', 'scheduled-2023-10-07-10_47_00_UTC', 'scheduled-2023-10-07-10_48_00_UTC', 'scheduled-2023-10-07-10_49_00_UTC', 'scheduled-2023-10-07-10_50_00_UTC', 'scheduled-2023-10-07-10_51_00_UTC', 'scheduled-2023-10-07-10_52_00_UTC', 'scheduled-2023-10-07-10_53_00_UTC', 'scheduled-2023-10-07-10_54_00_UTC', 'scheduled-2023-10-07-10_55_00_UTC', 'scheduled-2023-10-07-10_56_00_UTC'] 2023-10-07T10:56:15.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-10-07T10:56:16.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.153+0000 7f77cadff700 1 Processor -- start 2023-10-07T10:56:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.153+0000 7f77cadff700 1 -- start start 2023-10-07T10:56:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.154+0000 7f77cadff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f77c40afeb0 0x7f77c40cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.154+0000 7f77cadff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77c415a590 0x7f77c415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.154+0000 7f77cadff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40ad160 con 0x7f77c40accf0 2023-10-07T10:56:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.154+0000 7f77cadff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c415a140 con 0x7f77c415a590 2023-10-07T10:56:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.154+0000 7f77cadff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c40adca0 con 0x7f77c40afeb0 2023-10-07T10:56:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.154+0000 7f77c939c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77c415a590 0x7f77c415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77c415a590 0x7f77c415aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54624/0 (socket says 172.21.15.29:54624) 2023-10-07T10:56:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 -- 172.21.15.29:0/1785424242 learned_addr learned my addr 172.21.15.29:0/1785424242 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:56:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c8b9b700 1 --2- 172.21.15.29:0/1785424242 >> v2:172.21.15.144:3300/0 conn(0x7f77c40afeb0 0x7f77c40cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 -- 172.21.15.29:0/1785424242 >> v2:172.21.15.144:3300/0 conn(0x7f77c40afeb0 msgr2=0x7f77c40cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 --2- 172.21.15.29:0/1785424242 >> v2:172.21.15.144:3300/0 conn(0x7f77c40afeb0 0x7f77c40cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 -- 172.21.15.29:0/1785424242 >> v1:172.21.15.144:6789/0 conn(0x7f77c40accf0 legacy=0x7f77c40aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 -- 172.21.15.29:0/1785424242 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40ad400 con 0x7f77c415a590 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c8b9b700 1 --2- 172.21.15.29:0/1785424242 >> v2:172.21.15.144:3300/0 conn(0x7f77c40afeb0 0x7f77c40cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c939c700 1 --2- 172.21.15.29:0/1785424242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c415aa10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f77c40685f0 tx=0x7f77bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66220750ef96416c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c37fe700 1 -- 172.21.15.29:0/1785424242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0092e0 con 0x7f77c415a590 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c37fe700 1 -- 172.21.15.29:0/1785424242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc00ba50 con 0x7f77c415a590 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.155+0000 7f77c37fe700 1 -- 172.21.15.29:0/1785424242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00be10 con 0x7f77c415a590 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 -- 172.21.15.29:0/1785424242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 msgr2=0x7f77c415aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 --2- 172.21.15.29:0/1785424242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c415aa10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f77c40685f0 tx=0x7f77bc009840 comp rx=0 tx=0).stop 2023-10-07T10:56:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 -- 172.21.15.29:0/1785424242 shutdown_connections 2023-10-07T10:56:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 --2- 172.21.15.29:0/1785424242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c415aa10 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 --2- 172.21.15.29:0/1785424242 >> v2:172.21.15.144:3300/0 conn(0x7f77c40afeb0 0x7f77c40cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 -- 172.21.15.29:0/1785424242 >> 172.21.15.29:0/1785424242 conn(0x7f77c4067610 msgr2=0x7f77c4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:56:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 -- 172.21.15.29:0/1785424242 shutdown_connections 2023-10-07T10:56:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.156+0000 7f77cadff700 1 -- 172.21.15.29:0/1785424242 wait complete. 2023-10-07T10:56:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.158+0000 7f77cadff700 1 Processor -- start 2023-10-07T10:56:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.158+0000 7f77cadff700 1 -- start start 2023-10-07T10:56:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77cadff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77c40accf0 0x7f77c4155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77cadff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77c40afeb0 0x7f77c4141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77cadff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c40c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77cadff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c41558d0 con 0x7f77c415a590 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77cadff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c41506a0 con 0x7f77c40afeb0 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77cadff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c4152e50 con 0x7f77c40accf0 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77c939c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c40c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77c939c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c40c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54634/0 (socket says 172.21.15.29:54634) 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77c939c700 1 -- 172.21.15.29:0/1735948021 learned_addr learned my addr 172.21.15.29:0/1735948021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77c3fff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77c40afeb0 0x7f77c4141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.159+0000 7f77c8b9b700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77c40accf0 0x7f77c4155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c939c700 1 -- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77c40accf0 msgr2=0x7f77c4155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c939c700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77c40accf0 0x7f77c4155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c939c700 1 -- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77c40afeb0 msgr2=0x7f77c4141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c939c700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77c40afeb0 0x7f77c4141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c939c700 1 -- 172.21.15.29:0/1735948021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc009ae0 con 0x7f77c415a590 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c939c700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c40c9080 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009fd0 tx=0x7f77bc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc020070 con 0x7f77c415a590 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc011e80 con 0x7f77c415a590 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc01a890 con 0x7f77c415a590 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c4153110 con 0x7f77c415a590 2023-10-07T10:56:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.160+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c4155560 con 0x7f77c415a590 2023-10-07T10:56:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.161+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c4155a80 con 0x7f77c415a590 2023-10-07T10:56:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.162+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f77bc003040 con 0x7f77c415a590 2023-10-07T10:56:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.163+0000 7f77c0ff9700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77a80670a0 0x7f77a8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.163+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6412+0+0 (secure 0 0 0) 0x7f77bc0887b0 con 0x7f77c415a590 2023-10-07T10:56:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.164+0000 7f77c8b9b700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77a80670a0 0x7f77a8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.165+0000 7f77c8b9b700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77a80670a0 0x7f77a8069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f77b8006f00 tx=0x7f77b8008040 comp rx=0 tx=0).ready entity=mgr.8785 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:56:16.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.168+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77c4155a80 con 0x7f77c415a590 2023-10-07T10:56:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.397+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 --> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77c4155a80 con 0x7f77a80670a0 2023-10-07T10:56:16.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.406+0000 7f77c0ff9700 1 -- 172.21.15.29:0/1735948021 <== mgr.8785 v2:172.21.15.144:6834/114714 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f77c4155a80 con 0x7f77a80670a0 2023-10-07T10:56:16.407 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:56:16.407 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T10:45:35", "first": "2023-10-07T10:46:00", "last": "2023-10-07T10:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.408+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77a80670a0 msgr2=0x7f77a8069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.408+0000 7f77cadff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77a80670a0 0x7f77a8069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f77b8006f00 tx=0x7f77b8008040 comp rx=0 tx=0).stop 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.408+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 msgr2=0x7f77c40c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.408+0000 7f77cadff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c40c9080 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009fd0 tx=0x7f77bc01a000 comp rx=0 tx=0).stop 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 shutdown_connections 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f77a80670a0 0x7f77a8069550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c415a590 0x7f77c40c9080 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77c40accf0 0x7f77c4155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 --2- 172.21.15.29:0/1735948021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77c40afeb0 0x7f77c4141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 >> 172.21.15.29:0/1735948021 conn(0x7f77c4067610 msgr2=0x7f77c40abbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:56:16.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 shutdown_connections 2023-10-07T10:56:16.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.409+0000 7f77cadff700 1 -- 172.21.15.29:0/1735948021 wait complete. 2023-10-07T10:56:16.427 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T10:45:35", "first": "2023-10-07T10:46:00", "last": "2023-10-07T10:56:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-07T10:56:16.427 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T10:45:35', 'first': '2023-10-07T10:46:00', 'last': '2023-10-07T10:56:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-07T10:56:16.427 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-07T10:56:16.427 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-07T10:56:16.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-07T10:56:16.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.622+0000 7f0a063ff700 1 Processor -- start 2023-10-07T10:56:16.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.622+0000 7f0a063ff700 1 -- start start 2023-10-07T10:56:16.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.623+0000 7f0a063ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a000b0710 0x7f0a0015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.623+0000 7f0a063ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00154090 con 0x7f0a00153cc0 2023-10-07T10:56:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.623+0000 7f0a063ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00159f60 con 0x7f0a0015ade0 2023-10-07T10:56:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.623+0000 7f0a063ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0015a240 con 0x7f0a000b0710 2023-10-07T10:56:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a000b0710 0x7f0a0015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a000b0710 0x7f0a0015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54638/0 (socket says 172.21.15.29:54638) 2023-10-07T10:56:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 -- 172.21.15.29:0/2038169048 learned_addr learned my addr 172.21.15.29:0/2038169048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:56:16.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 -- 172.21.15.29:0/2038169048 >> v1:172.21.15.144:6789/0 conn(0x7f0a0015ade0 legacy=0x7f0a000b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T10:56:16.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 -- 172.21.15.29:0/2038169048 >> v1:172.21.15.29:6789/0 conn(0x7f0a00153cc0 legacy=0x7f0a000b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 -- 172.21.15.29:0/2038169048 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0015a520 con 0x7f0a000b0710 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.624+0000 7f0a0499c700 1 --2- 172.21.15.29:0/2038169048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a000b0710 0x7f0a0015c240 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009f20 tx=0x7f09f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cf9e82e4223562 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f09feffd700 1 -- 172.21.15.29:0/2038169048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800e070 con 0x7f0a000b0710 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f09feffd700 1 -- 172.21.15.29:0/2038169048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f800ba40 con 0x7f0a000b0710 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f09feffd700 1 -- 172.21.15.29:0/2038169048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80092e0 con 0x7f0a000b0710 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 -- 172.21.15.29:0/2038169048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a000b0710 msgr2=0x7f0a0015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 --2- 172.21.15.29:0/2038169048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a000b0710 0x7f0a0015c240 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009f20 tx=0x7f09f8009840 comp rx=0 tx=0).stop 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 -- 172.21.15.29:0/2038169048 shutdown_connections 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 --2- 172.21.15.29:0/2038169048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a000b0710 0x7f0a0015c240 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 -- 172.21.15.29:0/2038169048 >> 172.21.15.29:0/2038169048 conn(0x7f0a00067610 msgr2=0x7f0a00067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:56:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 -- 172.21.15.29:0/2038169048 shutdown_connections 2023-10-07T10:56:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.625+0000 7f0a063ff700 1 -- 172.21.15.29:0/2038169048 wait complete. 2023-10-07T10:56:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.627+0000 7f0a063ff700 1 Processor -- start 2023-10-07T10:56:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.627+0000 7f0a063ff700 1 -- start start 2023-10-07T10:56:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a063ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a000b0710 0x7f0a00152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a063ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a00153cc0 0x7f0a00145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a063ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 0x7f0a00156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a063ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00156b00 con 0x7f0a0015ade0 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a063ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a001574d0 con 0x7f0a000b0710 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a063ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00155ef0 con 0x7f0a00153cc0 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a0499c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 0x7f0a00156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a0499c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 0x7f0a00156f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54642/0 (socket says 172.21.15.29:54642) 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f0a0499c700 1 -- 172.21.15.29:0/3630908450 learned_addr learned my addr 172.21.15.29:0/3630908450 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f09fffff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a000b0710 0x7f0a00152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.628+0000 7f09ff7fe700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a00153cc0 0x7f0a00145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a0499c700 1 -- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a00153cc0 msgr2=0x7f0a00145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a0499c700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a00153cc0 0x7f0a00145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a0499c700 1 -- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a000b0710 msgr2=0x7f0a00152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a0499c700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a000b0710 0x7f0a00152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a0499c700 1 -- 172.21.15.29:0/3630908450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8009ae0 con 0x7f0a0015ade0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a0499c700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 0x7f0a00156f90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f09f8003000 tx=0x7f09f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80092e0 con 0x7f0a0015ade0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f8012d00 con 0x7f0a0015ade0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f801d650 con 0x7f0a0015ade0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a001561d0 con 0x7f0a0015ade0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.629+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00156790 con 0x7f0a0015ade0 2023-10-07T10:56:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.630+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0004c3c0 con 0x7f0a0015ade0 2023-10-07T10:56:16.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.631+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f09f801da00 con 0x7f0a0015ade0 2023-10-07T10:56:16.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.632+0000 7f09e7fff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09ec0670a0 0x7f09ec069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T10:56:16.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.632+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6412+0+0 (secure 0 0 0) 0x7f09f8086410 con 0x7f0a0015ade0 2023-10-07T10:56:16.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.633+0000 7f09fffff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09ec0670a0 0x7f09ec069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T10:56:16.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.633+0000 7f09fffff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09ec0670a0 0x7f09ec069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003a50 tx=0x7f09f0007e40 comp rx=0 tx=0).ready entity=mgr.8785 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T10:56:16.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.637+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09f8053100 con 0x7f0a0015ade0 2023-10-07T10:56:16.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.896+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0a0015a800 con 0x7f0a0015ade0 2023-10-07T10:56:16.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.900+0000 7f09e7fff700 1 -- 172.21.15.29:0/3630908450 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v80) v1 ==== 74+0+163917 (secure 0 0 0) 0x7f09f801dcb0 con 0x7f0a0015ade0 2023-10-07T10:56:16.903 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T10:56:16.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09ec0670a0 msgr2=0x7f09ec069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09ec0670a0 0x7f09ec069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003a50 tx=0x7f09f0007e40 comp rx=0 tx=0).stop 2023-10-07T10:56:16.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 msgr2=0x7f0a00156f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T10:56:16.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 0x7f0a00156f90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f09f8003000 tx=0x7f09f8017000 comp rx=0 tx=0).stop 2023-10-07T10:56:16.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 shutdown_connections 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:6834/114714,v1:172.21.15.144:6836/114714] conn(0x7f09ec0670a0 0x7f09ec069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a0015ade0 0x7f0a00156f90 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a000b0710 0x7f0a00152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 --2- 172.21.15.29:0/3630908450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a00153cc0 0x7f0a00145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 >> 172.21.15.29:0/3630908450 conn(0x7f0a00067610 msgr2=0x7f0a00152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 shutdown_connections 2023-10-07T10:56:16.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T10:56:16.903+0000 7f0a063ff700 1 -- 172.21.15.29:0/3630908450 wait complete. 2023-10-07T10:56:16.921 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":80,"active_gid":8785,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":114714},{"type":"v1","addr":"172.21.15.144:6836","nonce":114714}]},"active_addr":"172.21.15.144:6836/114714","active_change":"2023-10-07T10:45:34.272143+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":9276,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":166,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3310824583}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3296949315}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1610279030}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1517740475}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":681476780}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3001711291}]}]} 2023-10-07T10:56:16.925 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-10-07T10:56:16.925 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-07T10:56:16.925 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-10-07T10:56:16.926 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-07T10:56:16.926 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T10:56:16.999 INFO:tasks.ceph.mgr.x:Stopped 2023-10-07T10:56:16.999 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-07T10:56:17.003 INFO:tasks.ceph.mgr.x:Started 2023-10-07T10:56:17.263 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:17.262+0000 7f20072451c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T10:56:17.437 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:17.437+0000 7f20072451c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T10:56:17.750 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:17.749+0000 7f20072451c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T10:56:18.264 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:18.263+0000 7f20072451c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T10:56:18.432 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:18.432+0000 7f20072451c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T10:56:18.845 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:18.844+0000 7f20072451c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T10:56:19.617 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:19.616+0000 7f20072451c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T10:56:19.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:19.704+0000 7f20072451c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T10:56:19.859 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:19.858+0000 7f20072451c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T10:56:19.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:19.945+0000 7f20072451c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T10:56:20.163 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:20.163+0000 7f20072451c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T10:56:20.258 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:20.257+0000 7f20072451c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T10:56:20.353 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:20.352+0000 7f20072451c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T10:56:20.449 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:20.448+0000 7f20072451c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T10:56:20.637 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:20.636+0000 7f20072451c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T10:56:20.734 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:20.733+0000 7f20072451c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T10:56:21.075 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:21.074+0000 7f20072451c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T10:56:21.163 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:21.162+0000 7f20072451c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T10:56:21.703 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:21.702+0000 7f20072451c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T10:56:21.888 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:21.887+0000 7f20072451c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T10:56:22.265 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:22.265+0000 7f20072451c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:56:22.686 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:22.685+0000 7f20072451c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T10:56:22.865 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:22.864+0000 7f20072451c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T10:56:23.052 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.051+0000 7f20072451c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T10:56:23.161 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.160+0000 7f20072451c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T10:56:23.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f7b036700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f7b036700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f7b036700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f7b036700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f7b036700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f78030700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f78030700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f78030700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f78030700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:23.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:56:23.341+0000 7f1f78030700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:56:41.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:56:41.418 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:57:00.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:57:00.338+0000 7f1f83046700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:57:00.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:57:00.338+0000 7f1f83046700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:57:00.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:57:00.338+0000 7f1f83046700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:57:00.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:57:00.338+0000 7f1f83046700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:57:00.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T10:57:00.338+0000 7f1f83046700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T10:57:11.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:57:11.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:57:41.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:57:41.701 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:58:11.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:58:11.964 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:58:42.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:58:42.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:59:12.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:59:12.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:59:42.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T10:59:43.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:00:13.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:00:13.152 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:00:43.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:00:43.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:01:13.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:01:13.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:01:17.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/test_restart 1M 2023-10-07T11:01:17.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.215+0000 7f6ad63ff700 1 Processor -- start 2023-10-07T11:01:17.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.215+0000 7f6ad63ff700 1 -- start start 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6ad63ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ad00accf0 0x7f6ad00b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6ad63ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00ad400 con 0x7f6ad00cc810 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6ad63ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00ad6e0 con 0x7f6ad00afeb0 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6ad63ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad015a140 con 0x7f6ad00accf0 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6acf7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6ad00cc810 0x7f6ad015a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35054/0 (socket says 172.21.15.29:35054) 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6acf7fe700 1 -- 172.21.15.29:0/3483692629 learned_addr learned my addr 172.21.15.29:0/3483692629 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.216+0000 7f6ad499c700 1 --2- 172.21.15.29:0/3483692629 >> v2:172.21.15.29:3300/0 conn(0x7f6ad00accf0 0x7f6ad00b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6aceffd700 1 -- 172.21.15.29:0/3483692629 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354324853 0 0) 0x7f6ad00ad400 con 0x7f6ad00cc810 2023-10-07T11:01:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6aceffd700 1 -- 172.21.15.29:0/3483692629 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab8003570 con 0x7f6ad00cc810 2023-10-07T11:01:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6ad499c700 1 -- 172.21.15.29:0/3483692629 >> v1:172.21.15.144:6789/0 conn(0x7f6ad00afeb0 legacy=0x7f6ad00aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6ad499c700 1 -- 172.21.15.29:0/3483692629 >> v1:172.21.15.29:6789/0 conn(0x7f6ad00cc810 legacy=0x7f6ad015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6ad499c700 1 -- 172.21.15.29:0/3483692629 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00ad9c0 con 0x7f6ad00accf0 2023-10-07T11:01:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6ad499c700 1 --2- 172.21.15.29:0/3483692629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad00b05d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009e60 tx=0x7f6ac8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd004509c56ae1f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6aceffd700 1 -- 172.21.15.29:0/3483692629 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac800bb60 con 0x7f6ad00accf0 2023-10-07T11:01:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6aceffd700 1 -- 172.21.15.29:0/3483692629 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac800bcc0 con 0x7f6ad00accf0 2023-10-07T11:01:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.217+0000 7f6aceffd700 1 -- 172.21.15.29:0/3483692629 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac80116b0 con 0x7f6ad00accf0 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 -- 172.21.15.29:0/3483692629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 msgr2=0x7f6ad00b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/3483692629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad00b05d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009e60 tx=0x7f6ac8009780 comp rx=0 tx=0).stop 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 -- 172.21.15.29:0/3483692629 shutdown_connections 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/3483692629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad00b05d0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 -- 172.21.15.29:0/3483692629 >> 172.21.15.29:0/3483692629 conn(0x7f6ad0067610 msgr2=0x7f6ad0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 -- 172.21.15.29:0/3483692629 shutdown_connections 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.218+0000 7f6ad63ff700 1 -- 172.21.15.29:0/3483692629 wait complete. 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.220+0000 7f6ad63ff700 1 Processor -- start 2023-10-07T11:01:17.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.220+0000 7f6ad63ff700 1 -- start start 2023-10-07T11:01:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad63ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad0169740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad63ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad00afeb0 0x7f6ad006de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad63ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad00cc810 0x7f6ad0152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad63ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad0169c80 con 0x7f6ad00accf0 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad63ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad006e380 con 0x7f6ad00cc810 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad63ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad016dc50 con 0x7f6ad00afeb0 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6acffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad0169740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6acffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad0169740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56552/0 (socket says 172.21.15.29:56552) 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6acffff700 1 -- 172.21.15.29:0/2211752520 learned_addr learned my addr 172.21.15.29:0/2211752520 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6acffff700 1 -- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad00afeb0 msgr2=0x7f6ad006de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6acffff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad00afeb0 0x7f6ad006de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6acffff700 1 -- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad00cc810 msgr2=0x7f6ad0152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.221+0000 7f6ad499c700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad00cc810 0x7f6ad0152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6acffff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad00cc810 0x7f6ad0152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6acffff700 1 -- 172.21.15.29:0/2211752520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac8009a20 con 0x7f6ad00accf0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6ad499c700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad00cc810 0x7f6ad0152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6acffff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad0169740 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000c590 tx=0x7f6ac000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0011590 con 0x7f6ad00accf0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac00116f0 con 0x7f6ad00accf0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0011ab0 con 0x7f6ad00accf0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad016d0d0 con 0x7f6ad00accf0 2023-10-07T11:01:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.222+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad016d690 con 0x7f6ad00accf0 2023-10-07T11:01:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.223+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad00718b0 con 0x7f6ad00accf0 2023-10-07T11:01:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.224+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f6ac001b3b0 con 0x7f6ad00accf0 2023-10-07T11:01:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.225+0000 7f6ab7fff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6abc0670a0 0x7f6abc069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.225+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f6ac0084120 con 0x7f6ad00accf0 2023-10-07T11:01:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.226+0000 7f6acf7fe700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6abc0670a0 0x7f6abc069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.226+0000 7f6acf7fe700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6abc0670a0 0x7f6abc069550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40058d0 tx=0x7f6ac40079d0 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:17.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.230+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ac0050e20 con 0x7f6ad00accf0 2023-10-07T11:01:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.462+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ad006e690 con 0x7f6abc0670a0 2023-10-07T11:01:17.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.528+0000 7f6ab7fff700 1 -- 172.21.15.29:0/2211752520 <== mgr.8878 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f6ad006e690 con 0x7f6abc0670a0 2023-10-07T11:01:17.529 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-07T11:01:17.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6abc0670a0 msgr2=0x7f6abc069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6abc0670a0 0x7f6abc069550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40058d0 tx=0x7f6ac40079d0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 msgr2=0x7f6ad0169740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad0169740 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000c590 tx=0x7f6ac000c950 comp rx=0 tx=0).stop 2023-10-07T11:01:17.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 shutdown_connections 2023-10-07T11:01:17.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad00cc810 0x7f6ad0152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad00accf0 0x7f6ad0169740 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6abc0670a0 0x7f6abc069550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 --2- 172.21.15.29:0/2211752520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad00afeb0 0x7f6ad006de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 >> 172.21.15.29:0/2211752520 conn(0x7f6ad0067610 msgr2=0x7f6ad00abbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:17.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.530+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 shutdown_connections 2023-10-07T11:01:17.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.531+0000 7f6ad63ff700 1 -- 172.21.15.29:0/2211752520 wait complete. 2023-10-07T11:01:17.548 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-07T11:01:17.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:17.549 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-07T11:01:17.596 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_46_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_47_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_48_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_49_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_50_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_51_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_52_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_53_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_54_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_55_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_56_00_UTC 2023-10-07T11:01:17.597 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_57_00_UTC 2023-10-07T11:01:17.598 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_58_00_UTC 2023-10-07T11:01:17.598 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_59_00_UTC 2023-10-07T11:01:17.598 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_00_00_UTC 2023-10-07T11:01:17.598 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_01_00_UTC 2023-10-07T11:01:17.599 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-07-10_46_00_UTC', 'scheduled-2023-10-07-10_47_00_UTC', 'scheduled-2023-10-07-10_48_00_UTC', 'scheduled-2023-10-07-10_49_00_UTC', 'scheduled-2023-10-07-10_50_00_UTC', 'scheduled-2023-10-07-10_51_00_UTC', 'scheduled-2023-10-07-10_52_00_UTC', 'scheduled-2023-10-07-10_53_00_UTC', 'scheduled-2023-10-07-10_54_00_UTC', 'scheduled-2023-10-07-10_55_00_UTC', 'scheduled-2023-10-07-10_56_00_UTC', 'scheduled-2023-10-07-10_57_00_UTC', 'scheduled-2023-10-07-10_58_00_UTC', 'scheduled-2023-10-07-10_59_00_UTC', 'scheduled-2023-10-07-11_00_00_UTC', 'scheduled-2023-10-07-11_01_00_UTC'] 2023-10-07T11:01:17.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-10-07T11:01:17.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.799+0000 7f168f5ff700 1 Processor -- start 2023-10-07T11:01:17.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.800+0000 7f168f5ff700 1 -- start start 2023-10-07T11:01:17.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.801+0000 7f168f5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16881559f0 0x7f16880b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.801+0000 7f168f5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 0x7f16880c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168f5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16880c8c70 con 0x7f16880b0660 2023-10-07T11:01:17.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168f5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16880c98f0 con 0x7f16881559f0 2023-10-07T11:01:17.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168f5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16880c8f50 con 0x7f1688159de0 2023-10-07T11:01:17.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168cb9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16880b0660 0x7f16881596d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35060/0 (socket says 172.21.15.29:35060) 2023-10-07T11:01:17.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168cb9a700 1 -- 172.21.15.29:0/406608501 learned_addr learned my addr 172.21.15.29:0/406608501 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:17.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168d39b700 1 --2- 172.21.15.29:0/406608501 >> v2:172.21.15.144:3300/0 conn(0x7f16881559f0 0x7f16880b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f168db9c700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 0x7f16880c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336645800 0 0) 0x7f16880c8c70 con 0x7f16880b0660 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.802+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1674003570 con 0x7f16880b0660 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988583236 0 0) 0x7f1674003570 con 0x7f16880b0660 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 msgr2=0x7f16880c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 0x7f16880c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 >> v2:172.21.15.144:3300/0 conn(0x7f16881559f0 msgr2=0x7f16880b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 --2- 172.21.15.29:0/406608501 >> v2:172.21.15.144:3300/0 conn(0x7f16881559f0 0x7f16880b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16880c9230 con 0x7f16880b0660 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f1678001470 con 0x7f16880b0660 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 >> v1:172.21.15.29:6789/0 conn(0x7f16880b0660 legacy=0x7f16881596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 0x7f1674003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1674003fe0 0x7f1674004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1674004d50 0x7f16740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16880c98f0 con 0x7f1674004d50 2023-10-07T11:01:17.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16880c8f50 con 0x7f1674003fe0 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167400bb40 con 0x7f1688159de0 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f168db9c700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1674004d50 0x7f16740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f168d39b700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 0x7f1674003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.803+0000 7f168cb9a700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1674003fe0 0x7f1674004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f168db9c700 1 -- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 msgr2=0x7f1674003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f168db9c700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1688159de0 0x7f1674003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f168db9c700 1 -- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1674003fe0 msgr2=0x7f1674004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f168db9c700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1674003fe0 0x7f1674004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f168db9c700 1 -- 172.21.15.29:0/406608501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674003570 con 0x7f1674004d50 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f168db9c700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1674004d50 0x7f16740051c0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f16800094d0 tx=0x7f1680009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5022bf02f203001 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000fcb0 con 0x7f1674004d50 2023-10-07T11:01:17.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000fe10 con 0x7f1674004d50 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.804+0000 7f167ffff700 1 -- 172.21.15.29:0/406608501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16800136a0 con 0x7f1674004d50 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168f5ff700 1 -- 172.21.15.29:0/406608501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1674004d50 msgr2=0x7f16740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168f5ff700 1 --2- 172.21.15.29:0/406608501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1674004d50 0x7f16740051c0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f16800094d0 tx=0x7f1680009780 comp rx=0 tx=0).stop 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168db9c700 1 -- 172.21.15.29:0/406608501 reap_dead start 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168f5ff700 1 -- 172.21.15.29:0/406608501 shutdown_connections 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168f5ff700 1 -- 172.21.15.29:0/406608501 >> 172.21.15.29:0/406608501 conn(0x7f1688067610 msgr2=0x7f1688067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168f5ff700 1 -- 172.21.15.29:0/406608501 shutdown_connections 2023-10-07T11:01:17.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.805+0000 7f168f5ff700 1 -- 172.21.15.29:0/406608501 wait complete. 2023-10-07T11:01:17.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.806+0000 7f168f5ff700 1 Processor -- start 2023-10-07T11:01:17.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.807+0000 7f168f5ff700 1 -- start start 2023-10-07T11:01:17.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.807+0000 7f168f5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1674004d50 0x7f16880c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.807+0000 7f168f5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 0x7f168816c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168f5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1688169420 0x7f16880a6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168f5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16880a57f0 con 0x7f16880a63e0 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168f5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16880a6850 con 0x7f1688169420 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168f5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168816c810 con 0x7f1674004d50 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 0x7f168816c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 0x7f168816c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56572/0 (socket says 172.21.15.29:56572) 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168d39b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1674004d50 0x7f16880c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 -- 172.21.15.29:0/702939959 learned_addr learned my addr 172.21.15.29:0/702939959 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168db9c700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1688169420 0x7f16880a6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 -- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1674004d50 msgr2=0x7f16880c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1674004d50 0x7f16880c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 -- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1688169420 msgr2=0x7f16880a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1688169420 0x7f16880a6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 -- 172.21.15.29:0/702939959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16880c6930 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.808+0000 7f168cb9a700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 0x7f168816c2d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f16780073f0 tx=0x7f1678001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.809+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780077d0 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.809+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167800f870 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.809+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800fc10 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.809+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16880c6c10 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.809+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16880c71d0 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.810+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16880718b0 con 0x7f16880a63e0 2023-10-07T11:01:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.811+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f16780133b0 con 0x7f16880a63e0 2023-10-07T11:01:17.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.812+0000 7f167d7fa700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f166c0670a0 0x7f166c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:17.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.812+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f16780843f0 con 0x7f16880a63e0 2023-10-07T11:01:17.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.812+0000 7f168d39b700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f166c0670a0 0x7f166c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:17.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.813+0000 7f168d39b700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f166c0670a0 0x7f166c069550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1684006f00 tx=0x7f16840091a0 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:17.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:17.816+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16780510f0 con 0x7f16880a63e0 2023-10-07T11:01:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.044+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16880c9510 con 0x7f166c0670a0 2023-10-07T11:01:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.054+0000 7f167d7fa700 1 -- 172.21.15.29:0/702939959 <== mgr.8878 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f16880c9510 con 0x7f166c0670a0 2023-10-07T11:01:18.055 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:18.055 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T10:45:35", "first": "2023-10-07T10:46:00", "last": "2023-10-07T11:01:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-07T11:01:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f166c0670a0 msgr2=0x7f166c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f166c0670a0 0x7f166c069550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1684006f00 tx=0x7f16840091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 msgr2=0x7f168816c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 0x7f168816c2d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f16780073f0 tx=0x7f1678001e90 comp rx=0 tx=0).stop 2023-10-07T11:01:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 shutdown_connections 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1688169420 0x7f16880a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f166c0670a0 0x7f166c069550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1674004d50 0x7f16880c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 --2- 172.21.15.29:0/702939959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16880a63e0 0x7f168816c2d0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 >> 172.21.15.29:0/702939959 conn(0x7f1688067610 msgr2=0x7f168806d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 shutdown_connections 2023-10-07T11:01:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.057+0000 7f168f5ff700 1 -- 172.21.15.29:0/702939959 wait complete. 2023-10-07T11:01:18.074 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-10-07T00:00:00", "created": "2023-10-07T10:45:35", "first": "2023-10-07T10:46:00", "last": "2023-10-07T11:01:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-07T11:01:18.075 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-10-07T00:00:00', 'created': '2023-10-07T10:45:35', 'first': '2023-10-07T10:46:00', 'last': '2023-10-07T11:01:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-07T11:01:18.075 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-07T11:01:18.075 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-07T11:01:18.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/test_restart 1M 2023-10-07T11:01:18.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.277+0000 7f0030dff700 1 Processor -- start 2023-10-07T11:01:18.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.277+0000 7f0030dff700 1 -- start start 2023-10-07T11:01:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.278+0000 7f0030dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f002c0b19a0 0x7f002c0cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.278+0000 7f0030dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0aeae0 con 0x7f002c157650 2023-10-07T11:01:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.278+0000 7f0030dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0ca430 con 0x7f002c157a20 2023-10-07T11:01:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.278+0000 7f0030dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c0ca710 con 0x7f002c0b19a0 2023-10-07T11:01:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002a59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f002c157650 0x7f002c0ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35062/0 (socket says 172.21.15.29:35062) 2023-10-07T11:01:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002a59c700 1 -- 172.21.15.29:0/1696452157 learned_addr learned my addr 172.21.15.29:0/1696452157 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002ad9d700 1 --2- 172.21.15.29:0/1696452157 >> v2:172.21.15.29:3300/0 conn(0x7f002c0b19a0 0x7f002c0cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002959a700 1 -- 172.21.15.29:0/1696452157 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226492285 0 0) 0x7f002c0aeae0 con 0x7f002c157650 2023-10-07T11:01:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002959a700 1 -- 172.21.15.29:0/1696452157 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0014003570 con 0x7f002c157650 2023-10-07T11:01:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002ad9d700 1 -- 172.21.15.29:0/1696452157 >> v1:172.21.15.144:6789/0 conn(0x7f002c157a20 legacy=0x7f002c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002ad9d700 1 -- 172.21.15.29:0/1696452157 >> v1:172.21.15.29:6789/0 conn(0x7f002c157650 legacy=0x7f002c0ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002ad9d700 1 -- 172.21.15.29:0/1696452157 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c0ca9f0 con 0x7f002c0b19a0 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.279+0000 7f002ad9d700 1 --2- 172.21.15.29:0/1696452157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c0b19a0 0x7f002c0cba10 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f0024009e60 tx=0x7f0024009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8637e8bb6880964 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f002959a700 1 -- 172.21.15.29:0/1696452157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002400e070 con 0x7f002c0b19a0 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f002959a700 1 -- 172.21.15.29:0/1696452157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002400bb60 con 0x7f002c0b19a0 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f002959a700 1 -- 172.21.15.29:0/1696452157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024012430 con 0x7f002c0b19a0 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 -- 172.21.15.29:0/1696452157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c0b19a0 msgr2=0x7f002c0cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 --2- 172.21.15.29:0/1696452157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c0b19a0 0x7f002c0cba10 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f0024009e60 tx=0x7f0024009780 comp rx=0 tx=0).stop 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 -- 172.21.15.29:0/1696452157 shutdown_connections 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 --2- 172.21.15.29:0/1696452157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c0b19a0 0x7f002c0cba10 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 -- 172.21.15.29:0/1696452157 >> 172.21.15.29:0/1696452157 conn(0x7f002c067610 msgr2=0x7f002c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 -- 172.21.15.29:0/1696452157 shutdown_connections 2023-10-07T11:01:18.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.280+0000 7f0030dff700 1 -- 172.21.15.29:0/1696452157 wait complete. 2023-10-07T11:01:18.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.282+0000 7f0030dff700 1 Processor -- start 2023-10-07T11:01:18.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.282+0000 7f0030dff700 1 -- start start 2023-10-07T11:01:18.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f0030dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f002c0b19a0 0x7f002c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:18.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f0030dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c157650 0x7f002c14ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:18.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f0030dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 0x7f002c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:18.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f0030dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0aaaf0 con 0x7f002c157a20 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f0030dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c06dfe0 con 0x7f002c0b19a0 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f0030dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0aa290 con 0x7f002c157650 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f002ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 0x7f002c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f002ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 0x7f002c0afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56590/0 (socket says 172.21.15.29:56590) 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.283+0000 7f002ad9d700 1 -- 172.21.15.29:0/787665969 learned_addr learned my addr 172.21.15.29:0/787665969 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002a59c700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f002c0b19a0 0x7f002c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002ad9d700 1 -- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c157650 msgr2=0x7f002c14ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f0029d9b700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c157650 0x7f002c14ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002ad9d700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c157650 0x7f002c14ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002ad9d700 1 -- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f002c0b19a0 msgr2=0x7f002c157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002ad9d700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f002c0b19a0 0x7f002c157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002ad9d700 1 -- 172.21.15.29:0/787665969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0024009a20 con 0x7f002c157a20 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002a59c700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f002c0b19a0 0x7f002c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f0029d9b700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c157650 0x7f002c14ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f002ad9d700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 0x7f002c0afa90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0024013000 tx=0x7f0024019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240032b0 con 0x7f002c157a20 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024012e00 con 0x7f002c157a20 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.284+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c0aa530 con 0x7f002c157a20 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.285+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002401b830 con 0x7f002c157a20 2023-10-07T11:01:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.285+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c06dc70 con 0x7f002c157a20 2023-10-07T11:01:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.286+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c06e150 con 0x7f002c157a20 2023-10-07T11:01:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.287+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f002401bc60 con 0x7f002c157a20 2023-10-07T11:01:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.288+0000 7f001a7fc700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f000c0670a0 0x7f000c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.288+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f0024086ca0 con 0x7f002c157a20 2023-10-07T11:01:18.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.288+0000 7f002a59c700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f000c0670a0 0x7f000c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:18.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.289+0000 7f002a59c700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f000c0670a0 0x7f000c069550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f001c007fd0 tx=0x7f001c009040 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:18.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.292+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0024053890 con 0x7f002c157a20 2023-10-07T11:01:18.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.524+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f002c0718b0 con 0x7f000c0670a0 2023-10-07T11:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.597+0000 7f001a7fc700 1 -- 172.21.15.29:0/787665969 <== mgr.8878 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f002c0718b0 con 0x7f000c0670a0 2023-10-07T11:01:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f000c0670a0 msgr2=0x7f000c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f000c0670a0 0x7f000c069550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f001c007fd0 tx=0x7f001c009040 comp rx=0 tx=0).stop 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 msgr2=0x7f002c0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 0x7f002c0afa90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0024013000 tx=0x7f0024019000 comp rx=0 tx=0).stop 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 shutdown_connections 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f002c157a20 0x7f002c0afa90 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f002c0b19a0 0x7f002c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f000c0670a0 0x7f000c069550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 --2- 172.21.15.29:0/787665969 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c157650 0x7f002c14ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 >> 172.21.15.29:0/787665969 conn(0x7f002c067610 msgr2=0x7f002c0ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 shutdown_connections 2023-10-07T11:01:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:18.600+0000 7f0030dff700 1 -- 172.21.15.29:0/787665969 wait complete. 2023-10-07T11:01:18.619 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-07T11:01:18.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.619 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-07T11:01:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_46_00_UTC 2023-10-07T11:01:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_47_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_48_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_49_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_50_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_51_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_52_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_53_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_54_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_55_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_56_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_57_00_UTC 2023-10-07T11:01:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_58_00_UTC 2023-10-07T11:01:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-10_59_00_UTC 2023-10-07T11:01:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_00_00_UTC 2023-10-07T11:01:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_01_00_UTC 2023-10-07T11:01:18.670 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_46_00_UTC 2023-10-07T11:01:18.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.670 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_46_00_UTC) 2023-10-07T11:01:18.722 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_47_00_UTC 2023-10-07T11:01:18.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.723 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_47_00_UTC) 2023-10-07T11:01:18.775 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_48_00_UTC 2023-10-07T11:01:18.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.775 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_48_00_UTC) 2023-10-07T11:01:18.827 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_49_00_UTC 2023-10-07T11:01:18.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.827 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_49_00_UTC) 2023-10-07T11:01:18.879 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_50_00_UTC 2023-10-07T11:01:18.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.879 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_50_00_UTC) 2023-10-07T11:01:18.931 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_51_00_UTC 2023-10-07T11:01:18.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.931 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_51_00_UTC) 2023-10-07T11:01:18.983 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_52_00_UTC 2023-10-07T11:01:18.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:18.983 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_52_00_UTC) 2023-10-07T11:01:19.035 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_53_00_UTC 2023-10-07T11:01:19.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.035 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_53_00_UTC) 2023-10-07T11:01:19.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_54_00_UTC 2023-10-07T11:01:19.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.087 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_54_00_UTC) 2023-10-07T11:01:19.139 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_55_00_UTC 2023-10-07T11:01:19.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.139 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_55_00_UTC) 2023-10-07T11:01:19.192 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_56_00_UTC 2023-10-07T11:01:19.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.193 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_56_00_UTC) 2023-10-07T11:01:19.244 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_57_00_UTC 2023-10-07T11:01:19.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.245 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_57_00_UTC) 2023-10-07T11:01:19.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_58_00_UTC 2023-10-07T11:01:19.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.296 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_58_00_UTC) 2023-10-07T11:01:19.348 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_59_00_UTC 2023-10-07T11:01:19.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.348 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-10_59_00_UTC) 2023-10-07T11:01:19.400 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-11_00_00_UTC 2023-10-07T11:01:19.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.400 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-11_00_00_UTC) 2023-10-07T11:01:19.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-11_01_00_UTC 2023-10-07T11:01:19.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.453 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-07-11_01_00_UTC) 2023-10-07T11:01:19.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:19.504 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-07T11:01:19.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T11:01:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.750+0000 7f0347dff700 1 Processor -- start 2023-10-07T11:01:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.750+0000 7f0347dff700 1 -- start start 2023-10-07T11:01:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f0347dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0340153cc0 0x7f03400b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f0347dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03400b0710 0x7f03400b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f0347dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0340154090 con 0x7f034015ade0 2023-10-07T11:01:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f0347dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0340159f60 con 0x7f03400b0710 2023-10-07T11:01:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f0347dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034015a240 con 0x7f0340153cc0 2023-10-07T11:01:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f034539a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f034015ade0 0x7f03400b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35066/0 (socket says 172.21.15.29:35066) 2023-10-07T11:01:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f034539a700 1 -- 172.21.15.29:0/3836281108 learned_addr learned my addr 172.21.15.29:0/3836281108 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.751+0000 7f034639c700 1 --2- 172.21.15.29:0/3836281108 >> v2:172.21.15.29:3300/0 conn(0x7f03400b0710 0x7f03400b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f0345b9b700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0340153cc0 0x7f03400b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f0344b99700 1 -- 172.21.15.29:0/3836281108 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719339289 0 0) 0x7f0340154090 con 0x7f034015ade0 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f0344b99700 1 -- 172.21.15.29:0/3836281108 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c003570 con 0x7f034015ade0 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f034639c700 1 -- 172.21.15.29:0/3836281108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0340153cc0 msgr2=0x7f03400b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f034639c700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0340153cc0 0x7f03400b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f034639c700 1 -- 172.21.15.29:0/3836281108 >> v1:172.21.15.29:6789/0 conn(0x7f034015ade0 legacy=0x7f03400b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f034639c700 1 -- 172.21.15.29:0/3836281108 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034015a520 con 0x7f03400b0710 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f0345b9b700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0340153cc0 0x7f03400b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f034639c700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f03400b1d00 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0338009e60 tx=0x7f0338009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8323f3ff4e7d939 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.752+0000 7f0344b99700 1 -- 172.21.15.29:0/3836281108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033800e070 con 0x7f03400b0710 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0344b99700 1 -- 172.21.15.29:0/3836281108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033800bb60 con 0x7f03400b0710 2023-10-07T11:01:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0344b99700 1 -- 172.21.15.29:0/3836281108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338012430 con 0x7f03400b0710 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 -- 172.21.15.29:0/3836281108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 msgr2=0x7f03400b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f03400b1d00 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0338009e60 tx=0x7f0338009780 comp rx=0 tx=0).stop 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 -- 172.21.15.29:0/3836281108 shutdown_connections 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f03400b1d00 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 --2- 172.21.15.29:0/3836281108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0340153cc0 0x7f03400b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 -- 172.21.15.29:0/3836281108 >> 172.21.15.29:0/3836281108 conn(0x7f0340067610 msgr2=0x7f0340067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 -- 172.21.15.29:0/3836281108 shutdown_connections 2023-10-07T11:01:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.753+0000 7f0347dff700 1 -- 172.21.15.29:0/3836281108 wait complete. 2023-10-07T11:01:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.755+0000 7f0347dff700 1 Processor -- start 2023-10-07T11:01:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.755+0000 7f0347dff700 1 -- start start 2023-10-07T11:01:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0347dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f034016cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0347dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0340153cc0 0x7f0340143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0347dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034015ade0 0x7f0340158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0347dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340156510 con 0x7f03400b0710 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0347dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03401567f0 con 0x7f0340153cc0 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0347dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034016d110 con 0x7f034015ade0 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f034016cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f034016cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56612/0 (socket says 172.21.15.29:56612) 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 -- 172.21.15.29:0/1328514506 learned_addr learned my addr 172.21.15.29:0/1328514506 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f034539a700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0340153cc0 0x7f0340143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f034639c700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034015ade0 0x7f0340158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 -- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034015ade0 msgr2=0x7f0340158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034015ade0 0x7f0340158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 -- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0340153cc0 msgr2=0x7f0340143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0340153cc0 0x7f0340143d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f0345b9b700 1 -- 172.21.15.29:0/1328514506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0338009a20 con 0x7f03400b0710 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.756+0000 7f034639c700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034015ade0 0x7f0340158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.757+0000 7f0345b9b700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f034016cbd0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f033c00e950 tx=0x7f033c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.757+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c005a00 con 0x7f03400b0710 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.757+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033c005b60 con 0x7f03400b0710 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.757+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c006020 con 0x7f03400b0710 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.757+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034016d3b0 con 0x7f03400b0710 2023-10-07T11:01:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.757+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034016d970 con 0x7f03400b0710 2023-10-07T11:01:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.758+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034004c3c0 con 0x7f03400b0710 2023-10-07T11:01:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.759+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 84871+0+0 (secure 0 0 0) 0x7f033c0063d0 con 0x7f03400b0710 2023-10-07T11:01:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.760+0000 7f0335ffb700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f03240670a0 0x7f0324069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.760+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f033c084d60 con 0x7f03400b0710 2023-10-07T11:01:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.760+0000 7f034539a700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f03240670a0 0x7f0324069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.761+0000 7f034539a700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f03240670a0 0x7f0324069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0340156090 tx=0x7f0330009040 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:19.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:19.764+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f033c051950 con 0x7f03400b0710 2023-10-07T11:01:20.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.025+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f034015a800 con 0x7f03400b0710 2023-10-07T11:01:20.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.790+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f033c0510d0 con 0x7f03400b0710 2023-10-07T11:01:20.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.791+0000 7f0335ffb700 1 -- 172.21.15.29:0/1328514506 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) v1 ==== 93+0+0 (secure 0 0 0) 0x7f033c051950 con 0x7f03400b0710 2023-10-07T11:01:20.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f03240670a0 msgr2=0x7f0324069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f03240670a0 0x7f0324069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0340156090 tx=0x7f0330009040 comp rx=0 tx=0).stop 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 msgr2=0x7f034016cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f034016cbd0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f033c00e950 tx=0x7f033c008040 comp rx=0 tx=0).stop 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 shutdown_connections 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f03240670a0 0x7f0324069550 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0340156090 tx=0x7f0330009040 comp rx=0 tx=0).stop 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034015ade0 0x7f0340158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03400b0710 0x7f034016cbd0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 --2- 172.21.15.29:0/1328514506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0340153cc0 0x7f0340143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.794+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 >> 172.21.15.29:0/1328514506 conn(0x7f0340067610 msgr2=0x7f03400b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:20.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.795+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 shutdown_connections 2023-10-07T11:01:20.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:20.795+0000 7f0347dff700 1 -- 172.21.15.29:0/1328514506 wait complete. 2023-10-07T11:01:20.816 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T11:01:20.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:01:20.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:01:21.010 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:21.010+0000 7efd38c7a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:01:21.011 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:21.010+0000 7fdca20fb1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:01:21.042 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T11:01:21.042 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T11:01:21.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:21.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T11:01:21.146 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:01:21.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:21.146 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T11:01:21.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:21.190+0000 7fdca20fb1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:01:21.198 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:21.198+0000 7efd38c7a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:01:21.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.452+0000 7f31fddff700 1 Processor -- start 2023-10-07T11:01:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.452+0000 7f31fddff700 1 -- start start 2023-10-07T11:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31fddff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f31f80b2670 0x7f31f80b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31fddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f80a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31fddff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f806f4e0 con 0x7f31f8159550 2023-10-07T11:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31fddff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80b1120 con 0x7f31f80b2670 2023-10-07T11:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31fddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f806e8d0 con 0x7f31f80a98d0 2023-10-07T11:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31f77fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31f8159550 0x7f31f80b0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35090/0 (socket says 172.21.15.29:35090) 2023-10-07T11:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31f77fe700 1 -- 172.21.15.29:0/4235129320 learned_addr learned my addr 172.21.15.29:0/4235129320 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31f6ffd700 1 --2- 172.21.15.29:0/4235129320 >> v2:172.21.15.144:3300/0 conn(0x7f31f80b2670 0x7f31f80b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.454+0000 7f31f7fff700 1 --2- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f80a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f67fc700 1 -- 172.21.15.29:0/4235129320 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637137125 0 0) 0x7f31f806f4e0 con 0x7f31f8159550 2023-10-07T11:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f67fc700 1 -- 172.21.15.29:0/4235129320 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e0003570 con 0x7f31f8159550 2023-10-07T11:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f6ffd700 1 -- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 msgr2=0x7f31f80a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f6ffd700 1 --2- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f80a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f6ffd700 1 -- 172.21.15.29:0/4235129320 >> v1:172.21.15.29:6789/0 conn(0x7f31f8159550 legacy=0x7f31f80b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f6ffd700 1 -- 172.21.15.29:0/4235129320 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f806ebb0 con 0x7f31f80b2670 2023-10-07T11:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f6ffd700 1 --2- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f80b2670 0x7f31f80b2ae0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0098e0 tx=0x7f31ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b23d1628c8141a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.455+0000 7f31f67fc700 1 -- 172.21.15.29:0/4235129320 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011630 con 0x7f31f80b2670 2023-10-07T11:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.456+0000 7f31f67fc700 1 -- 172.21.15.29:0/4235129320 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec011790 con 0x7f31f80b2670 2023-10-07T11:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.456+0000 7f31f67fc700 1 -- 172.21.15.29:0/4235129320 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011b10 con 0x7f31f80b2670 2023-10-07T11:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.456+0000 7f31fddff700 1 -- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f80b2670 msgr2=0x7f31f80b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.456+0000 7f31fddff700 1 --2- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f80b2670 0x7f31f80b2ae0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0098e0 tx=0x7f31ec0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.457+0000 7f31fddff700 1 -- 172.21.15.29:0/4235129320 shutdown_connections 2023-10-07T11:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.457+0000 7f31fddff700 1 --2- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f80a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.457+0000 7f31fddff700 1 --2- 172.21.15.29:0/4235129320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f80b2670 0x7f31f80b2ae0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.457+0000 7f31fddff700 1 -- 172.21.15.29:0/4235129320 >> 172.21.15.29:0/4235129320 conn(0x7f31f8069190 msgr2=0x7f31f806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.457+0000 7f31fddff700 1 -- 172.21.15.29:0/4235129320 shutdown_connections 2023-10-07T11:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.457+0000 7f31fddff700 1 -- 172.21.15.29:0/4235129320 wait complete. 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.458+0000 7f31fddff700 1 Processor -- start 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.458+0000 7f31fddff700 1 -- start start 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.459+0000 7f31fddff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f8149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.459+0000 7f31fddff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 0x7f31f8151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.459+0000 7f31fddff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f8159550 0x7f31f814ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.459+0000 7f31fddff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f81530d0 con 0x7f31f80b2670 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.459+0000 7f31fddff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8152c40 con 0x7f31f8159550 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.459+0000 7f31fddff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8152030 con 0x7f31f80a98d0 2023-10-07T11:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f77fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f8149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 0x7f31f8151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f77fe700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f8149cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:39726/0 (socket says 172.21.15.29:39726) 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f77fe700 1 -- 172.21.15.29:0/2382452746 learned_addr learned my addr 172.21.15.29:0/2382452746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 0x7f31f8151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56652/0 (socket says 172.21.15.29:56652) 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f7fff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f8159550 0x7f31f814ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 -- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 msgr2=0x7f31f8149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f8149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 -- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f8159550 msgr2=0x7f31f814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f8159550 0x7f31f814ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 -- 172.21.15.29:0/2382452746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009420 con 0x7f31f80b2670 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.460+0000 7f31f6ffd700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 0x7f31f8151580 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f31ec016000 tx=0x7f31ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.461+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011b10 con 0x7f31f80b2670 2023-10-07T11:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.461+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec000c90 con 0x7f31f80b2670 2023-10-07T11:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.461+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f8152310 con 0x7f31f80b2670 2023-10-07T11:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.461+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec023620 con 0x7f31f80b2670 2023-10-07T11:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.461+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f81528d0 con 0x7f31f80b2670 2023-10-07T11:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.462+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8152dd0 con 0x7f31f80b2670 2023-10-07T11:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.464+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f31ec02c3b0 con 0x7f31f80b2670 2023-10-07T11:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.465+0000 7f31df7fe700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 0x7f31e4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.465+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f31ec085e80 con 0x7f31f80b2670 2023-10-07T11:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.465+0000 7f31f77fe700 1 -- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 msgr2=0x7f31e4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.465+0000 7f31f77fe700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 0x7f31e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.470+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31ec052c00 con 0x7f31f80b2670 2023-10-07T11:01:21.503 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:21.502+0000 7fdca20fb1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:01:21.537 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:21.537+0000 7efd38c7a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:01:21.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.666+0000 7f31f77fe700 1 -- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 msgr2=0x7f31e4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.666+0000 7f31f77fe700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 0x7f31e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.717+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f804c3c0 con 0x7f31f80b2670 2023-10-07T11:01:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.717+0000 7f31df7fe700 1 -- 172.21.15.29:0/2382452746 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f31ec056220 con 0x7f31f80b2670 2023-10-07T11:01:21.718 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:21.719 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:45:18.619852+0000","modified":"2023-10-07T10:45:22.148839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9090":{"gid":9090,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/4080992371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4080992371},{"type":"v1","addr":"172.21.15.29:6833","nonce":4080992371}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-10-07T11:01:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 msgr2=0x7f31e4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 0x7f31e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 msgr2=0x7f31f8151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 0x7f31f8151580 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f31ec016000 tx=0x7f31ec017000 comp rx=0 tx=0).stop 2023-10-07T11:01:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 shutdown_connections 2023-10-07T11:01:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f8159550 0x7f31f814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f31e4067050 0x7f31e4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f80a98d0 0x7f31f8149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 --2- 172.21.15.29:0/2382452746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80b2670 0x7f31f8151580 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 >> 172.21.15.29:0/2382452746 conn(0x7f31f8069190 msgr2=0x7f31f80af690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 shutdown_connections 2023-10-07T11:01:21.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.720+0000 7f31fddff700 1 -- 172.21.15.29:0/2382452746 wait complete. 2023-10-07T11:01:21.723 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-07T11:01:21.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.943+0000 7fab779ff700 1 Processor -- start 2023-10-07T11:01:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.943+0000 7fab779ff700 1 -- start start 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab779ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab779ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab70158ad0 con 0x7fab700ae8b0 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab779ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7015b140 con 0x7fab700b1a10 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab779ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7015b420 con 0x7fab70157290 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab74f9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab700ae8b0 0x7fab700b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35100/0 (socket says 172.21.15.29:35100) 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab74f9a700 1 -- 172.21.15.29:0/3534840306 learned_addr learned my addr 172.21.15.29:0/3534840306 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.945+0000 7fab7579b700 1 --2- 172.21.15.29:0/3534840306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114485796 0 0) 0x7fab70158ad0 con 0x7fab700ae8b0 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab5c003570 con 0x7fab700ae8b0 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628979918 0 0) 0x7fab7015b140 con 0x7fab700b1a10 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab70158ad0 con 0x7fab700b1a10 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab7579b700 1 -- 172.21.15.29:0/3534840306 >> v1:172.21.15.144:6789/0 conn(0x7fab700b1a10 legacy=0x7fab70157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab7579b700 1 -- 172.21.15.29:0/3534840306 >> v1:172.21.15.29:6789/0 conn(0x7fab700ae8b0 legacy=0x7fab700b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab7579b700 1 -- 172.21.15.29:0/3534840306 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7015b700 con 0x7fab70157290 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab7579b700 1 --2- 172.21.15.29:0/3534840306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700cc870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009910 tx=0x7fab6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd4d01fc4049e9eb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c011630 con 0x7fab70157290 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c011790 con 0x7fab70157290 2023-10-07T11:01:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.946+0000 7fab67fff700 1 -- 172.21.15.29:0/3534840306 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c011b30 con 0x7fab70157290 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 -- 172.21.15.29:0/3534840306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 msgr2=0x7fab700cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 --2- 172.21.15.29:0/3534840306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700cc870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009910 tx=0x7fab6c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 -- 172.21.15.29:0/3534840306 shutdown_connections 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 --2- 172.21.15.29:0/3534840306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700cc870 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 -- 172.21.15.29:0/3534840306 >> 172.21.15.29:0/3534840306 conn(0x7fab700680a0 msgr2=0x7fab700cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 -- 172.21.15.29:0/3534840306 shutdown_connections 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.947+0000 7fab779ff700 1 -- 172.21.15.29:0/3534840306 wait complete. 2023-10-07T11:01:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.949+0000 7fab779ff700 1 Processor -- start 2023-10-07T11:01:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.949+0000 7fab779ff700 1 -- start start 2023-10-07T11:01:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab779ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab700b1a10 0x7fab7014aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab779ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab779ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 0x7fab7006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab779ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7006ee30 con 0x7fab7006e470 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab779ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7014c6c0 con 0x7fab700b1a10 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab779ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7014bab0 con 0x7fab70157290 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 0x7fab7006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab7579b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab700b1a10 0x7fab7014aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 0x7fab7006e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56666/0 (socket says 172.21.15.29:56666) 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 -- 172.21.15.29:0/1743561744 learned_addr learned my addr 172.21.15.29:0/1743561744 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab74f9a700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 -- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 msgr2=0x7fab700c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 -- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab700b1a10 msgr2=0x7fab7014aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab700b1a10 0x7fab7014aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.950+0000 7fab75f9c700 1 -- 172.21.15.29:0/1743561744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c009420 con 0x7fab7006e470 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.951+0000 7fab75f9c700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 0x7fab7006e8f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c910 tx=0x7fab68007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.951+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68012630 con 0x7fab7006e470 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.951+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab680127b0 con 0x7fab7006e470 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.951+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7014bd90 con 0x7fab7006e470 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.951+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68012b70 con 0x7fab7006e470 2023-10-07T11:01:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.951+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7014c350 con 0x7fab7006e470 2023-10-07T11:01:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.952+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7004c3c0 con 0x7fab7006e470 2023-10-07T11:01:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.954+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fab6801b400 con 0x7fab7006e470 2023-10-07T11:01:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.954+0000 7fab657fa700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 0x7fab54069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.954+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fab68085420 con 0x7fab7006e470 2023-10-07T11:01:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.955+0000 7fab7579b700 1 -- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 msgr2=0x7fab54069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.955+0000 7fab7579b700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 0x7fab54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:21.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:21.961+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab680521d0 con 0x7fab7006e470 2023-10-07T11:01:22.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:22.023+0000 7fdca20fb1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:01:22.100 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:22.100+0000 7efd38c7a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:01:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.156+0000 7fab7579b700 1 -- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 msgr2=0x7fab54069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.156+0000 7fab7579b700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 0x7fab54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:22.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.170+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab7015b9e0 con 0x7fab7006e470 2023-10-07T11:01:22.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.171+0000 7fab657fa700 1 -- 172.21.15.29:0/1743561744 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+16904 (secure 0 0 0) 0x7fab680557f0 con 0x7fab7006e470 2023-10-07T11:01:22.172 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:22.173 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":169,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T10:56:23.164350+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:45:16.377745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-07T10:45:17.609874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3001711291":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/681476780":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/3310824583":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/3296949315":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/4205507060":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/3718096826":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/2626373758":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/1610279030":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/4211255093":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/1517740475":"2023-10-08T10:56:23.164290+0000","172.21.15.144:6836/114714":"2023-10-08T10:45:34.271979+0000","172.21.15.144:6834/114714":"2023-10-08T10:45:34.271979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 msgr2=0x7fab54069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 0x7fab54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 msgr2=0x7fab7006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 0x7fab7006e8f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c910 tx=0x7fab68007f90 comp rx=0 tx=0).stop 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 shutdown_connections 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7006e470 0x7fab7006e8f0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab700b1a10 0x7fab7014aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fab54067050 0x7fab54069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 --2- 172.21.15.29:0/1743561744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab70157290 0x7fab700c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 >> 172.21.15.29:0/1743561744 conn(0x7fab700680a0 msgr2=0x7fab700c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.173+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 shutdown_connections 2023-10-07T11:01:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.174+0000 7fab779ff700 1 -- 172.21.15.29:0/1743561744 wait complete. 2023-10-07T11:01:22.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T10:45:16.377745+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T10:45:17.609874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T11:01:22.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T11:01:22.194 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:22.191+0000 7fdca20fb1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:01:22.293 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:22.293+0000 7efd38c7a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:01:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.407+0000 7fa66b3ff700 1 Processor -- start 2023-10-07T11:01:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.407+0000 7fa66b3ff700 1 -- start start 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66b3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6640cc800 0x7fa6640acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa66415afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66b3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa664159ab0 con 0x7fa6641596e0 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66b3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6640aa8e0 con 0x7fa6640cc800 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640a98d0 con 0x7fa6640ad440 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6640cc800 0x7fa6640acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6640cc800 0x7fa6640acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56670/0 (socket says 172.21.15.29:56670) 2023-10-07T11:01:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66899a700 1 -- 172.21.15.29:0/225547975 learned_addr learned my addr 172.21.15.29:0/225547975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.409+0000 7fa66999c700 1 --2- 172.21.15.29:0/225547975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa66415afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa66899a700 1 -- 172.21.15.29:0/225547975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 msgr2=0x7fa66415afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa66899a700 1 --2- 172.21.15.29:0/225547975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa66415afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa66899a700 1 -- 172.21.15.29:0/225547975 >> v1:172.21.15.144:6789/0 conn(0x7fa6641596e0 legacy=0x7fa6640cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa66899a700 1 -- 172.21.15.29:0/225547975 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640aa450 con 0x7fa6640cc800 2023-10-07T11:01:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa65bfff700 1 -- 172.21.15.29:0/225547975 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594428113 0 0) 0x7fa660001110 con 0x7fa6641596e0 2023-10-07T11:01:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa66899a700 1 --2- 172.21.15.29:0/225547975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640cc800 0x7fa6640acf00 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fa6540099b0 tx=0x7fa654009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8d9f49257511d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa65bfff700 1 -- 172.21.15.29:0/225547975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa654009ea0 con 0x7fa6640cc800 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa65bfff700 1 -- 172.21.15.29:0/225547975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65400b860 con 0x7fa6640cc800 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.410+0000 7fa65bfff700 1 -- 172.21.15.29:0/225547975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65400bc00 con 0x7fa6640cc800 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 -- 172.21.15.29:0/225547975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640cc800 msgr2=0x7fa6640acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/225547975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640cc800 0x7fa6640acf00 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fa6540099b0 tx=0x7fa654009250 comp rx=0 tx=0).stop 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 -- 172.21.15.29:0/225547975 shutdown_connections 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/225547975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa66415afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/225547975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640cc800 0x7fa6640acf00 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 -- 172.21.15.29:0/225547975 >> 172.21.15.29:0/225547975 conn(0x7fa664067610 msgr2=0x7fa664067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 -- 172.21.15.29:0/225547975 shutdown_connections 2023-10-07T11:01:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.411+0000 7fa66b3ff700 1 -- 172.21.15.29:0/225547975 wait complete. 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.413+0000 7fa66b3ff700 1 Processor -- start 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.413+0000 7fa66b3ff700 1 -- start start 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa664150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66b3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6640cc800 0x7fa66415bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66b3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 0x7fa66415c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66b3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa664151280 con 0x7fa6641596e0 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66b3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640ca7a0 con 0x7fa6640cc800 2023-10-07T11:01:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640c9c20 con 0x7fa6640ad440 2023-10-07T11:01:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 0x7fa66415c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 0x7fa66415c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56672/0 (socket says 172.21.15.29:56672) 2023-10-07T11:01:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 -- 172.21.15.29:0/2894927327 learned_addr learned my addr 172.21.15.29:0/2894927327 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66919b700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa664150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66899a700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6640cc800 0x7fa66415bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 -- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 msgr2=0x7fa664150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa664150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 -- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6640cc800 msgr2=0x7fa66415bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6640cc800 0x7fa66415bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.414+0000 7fa66999c700 1 -- 172.21.15.29:0/2894927327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6540094f0 con 0x7fa6641596e0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.415+0000 7fa66999c700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 0x7fa66415c0e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009e90 tx=0x7fa65c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.415+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c00cb70 con 0x7fa6641596e0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.415+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c00f040 con 0x7fa6641596e0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.415+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c014480 con 0x7fa6641596e0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.415+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6640c9f00 con 0x7fa6641596e0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.415+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6640ca4c0 con 0x7fa6641596e0 2023-10-07T11:01:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.416+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66404c3c0 con 0x7fa6641596e0 2023-10-07T11:01:22.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.417+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7fa65c014830 con 0x7fa6641596e0 2023-10-07T11:01:22.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.418+0000 7fa6597fa700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 0x7fa648069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.418+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fa65c0855f0 con 0x7fa6641596e0 2023-10-07T11:01:22.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.418+0000 7fa66919b700 1 -- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 msgr2=0x7fa648069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:22.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.418+0000 7fa66919b700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 0x7fa648069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:22.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.422+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa65c0521e0 con 0x7fa6641596e0 2023-10-07T11:01:22.609 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:22.608+0000 7fdca20fb1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:01:22.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.619+0000 7fa66919b700 1 -- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 msgr2=0x7fa648069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:22.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.619+0000 7fa66919b700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 0x7fa648069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.630+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa664154770 con 0x7fa6641596e0 2023-10-07T11:01:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.630+0000 7fa6597fa700 1 -- 172.21.15.29:0/2894927327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v169) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa65c055800 con 0x7fa6641596e0 2023-10-07T11:01:22.631 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.632+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 msgr2=0x7fa648069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.632+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 0x7fa648069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.632+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 msgr2=0x7fa66415c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.632+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 0x7fa66415c0e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009e90 tx=0x7fa65c00b040 comp rx=0 tx=0).stop 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 shutdown_connections 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6641596e0 0x7fa66415c0e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa648067050 0x7fa648069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6640ad440 0x7fa664150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 --2- 172.21.15.29:0/2894927327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6640cc800 0x7fa66415bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 >> 172.21.15.29:0/2894927327 conn(0x7fa664067610 msgr2=0x7fa664151d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 shutdown_connections 2023-10-07T11:01:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.633+0000 7fa66b3ff700 1 -- 172.21.15.29:0/2894927327 wait complete. 2023-10-07T11:01:22.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T11:01:22.739 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:22.738+0000 7efd38c7a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:01:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.887+0000 7f17117ff700 1 Processor -- start 2023-10-07T11:01:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.887+0000 7f17117ff700 1 -- start start 2023-10-07T11:01:22.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.889+0000 7f17117ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f170c1596e0 0x7f170c0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.889+0000 7f17117ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f170c0cc630 0x7f170c0acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.889+0000 7f17117ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c159ab0 con 0x7f170c0ad440 2023-10-07T11:01:22.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.889+0000 7f17117ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c0ccad0 con 0x7f170c0cc630 2023-10-07T11:01:22.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.889+0000 7f17117ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c0aa8e0 con 0x7f170c1596e0 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f170c1596e0 0x7f170c0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f170c1596e0 0x7f170c0cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52178/0 (socket says 172.21.15.29:52178) 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 -- 172.21.15.29:0/1389237134 learned_addr learned my addr 172.21.15.29:0/1389237134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170a7fc700 1 --2- 172.21.15.29:0/1389237134 >> v2:172.21.15.29:3300/0 conn(0x7f170c0cc630 0x7f170c0acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 -- 172.21.15.29:0/1389237134 >> v2:172.21.15.29:3300/0 conn(0x7f170c0cc630 msgr2=0x7f170c0acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 --2- 172.21.15.29:0/1389237134 >> v2:172.21.15.29:3300/0 conn(0x7f170c0cc630 0x7f170c0acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 -- 172.21.15.29:0/1389237134 >> v1:172.21.15.144:6789/0 conn(0x7f170c0ad440 legacy=0x7f170c15d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.890+0000 7f170affd700 1 -- 172.21.15.29:0/1389237134 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0a98d0 con 0x7f170c1596e0 2023-10-07T11:01:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.891+0000 7f170affd700 1 --2- 172.21.15.29:0/1389237134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c0cc0f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f170c0642c0 tx=0x7f16fc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=696798f7fb3ad40d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.891+0000 7f1709ffb700 1 -- 172.21.15.29:0/1389237134 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc009e70 con 0x7f170c1596e0 2023-10-07T11:01:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.891+0000 7f1709ffb700 1 -- 172.21.15.29:0/1389237134 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc00b860 con 0x7f170c1596e0 2023-10-07T11:01:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.891+0000 7f1709ffb700 1 -- 172.21.15.29:0/1389237134 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00bc00 con 0x7f170c1596e0 2023-10-07T11:01:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 -- 172.21.15.29:0/1389237134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 msgr2=0x7f170c0cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 --2- 172.21.15.29:0/1389237134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c0cc0f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f170c0642c0 tx=0x7f16fc009250 comp rx=0 tx=0).stop 2023-10-07T11:01:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 -- 172.21.15.29:0/1389237134 shutdown_connections 2023-10-07T11:01:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 --2- 172.21.15.29:0/1389237134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c0cc0f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 --2- 172.21.15.29:0/1389237134 >> v2:172.21.15.29:3300/0 conn(0x7f170c0cc630 0x7f170c0acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 -- 172.21.15.29:0/1389237134 >> 172.21.15.29:0/1389237134 conn(0x7f170c067610 msgr2=0x7f170c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 -- 172.21.15.29:0/1389237134 shutdown_connections 2023-10-07T11:01:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.892+0000 7f17117ff700 1 -- 172.21.15.29:0/1389237134 wait complete. 2023-10-07T11:01:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.894+0000 7f17117ff700 1 Processor -- start 2023-10-07T11:01:22.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.895+0000 7f17117ff700 1 -- start start 2023-10-07T11:01:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.896+0000 7f17117ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 0x7f170c157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.896+0000 7f17117ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f170c0cc630 0x7f170c0a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.896+0000 7f17117ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.896+0000 7f17117ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c157280 con 0x7f170c0ad440 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.896+0000 7f17117ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c156670 con 0x7f170c1596e0 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.896+0000 7f17117ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c156950 con 0x7f170c0cc630 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 0x7f170c157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170a7fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f170c0cc630 0x7f170c0a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 0x7f170c157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38958/0 (socket says 172.21.15.29:38958) 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170a7fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f170c0cc630 0x7f170c0a9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44552/0 (socket says 172.21.15.29:44552) 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 -- 172.21.15.29:0/1519865706 learned_addr learned my addr 172.21.15.29:0/1519865706 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170b7fe700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 -- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f170c0cc630 msgr2=0x7f170c0a9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f170c0cc630 0x7f170c0a9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 -- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 msgr2=0x7f170c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170affd700 1 -- 172.21.15.29:0/1519865706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc0094f0 con 0x7f170c0ad440 2023-10-07T11:01:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.897+0000 7f170b7fe700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:01:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.898+0000 7f170affd700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 0x7f170c157710 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f170c0aaa20 tx=0x7f16fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.898+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00bbe0 con 0x7f170c0ad440 2023-10-07T11:01:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.898+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc011420 con 0x7f170c0ad440 2023-10-07T11:01:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.898+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c156c30 con 0x7f170c0ad440 2023-10-07T11:01:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.898+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0117c0 con 0x7f170c0ad440 2023-10-07T11:01:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.898+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c156380 con 0x7f170c0ad440 2023-10-07T11:01:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.899+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c04c3c0 con 0x7f170c0ad440 2023-10-07T11:01:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.900+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f16fc011bf0 con 0x7f170c0ad440 2023-10-07T11:01:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.901+0000 7f16f2ffd700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 0x7f16f8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.901+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f16fc085f80 con 0x7f170c0ad440 2023-10-07T11:01:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.901+0000 7f170a7fc700 1 -- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 msgr2=0x7f16f8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.901+0000 7f170a7fc700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 0x7f16f8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:22.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:22.906+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16fc052c40 con 0x7f170c0ad440 2023-10-07T11:01:23.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.103+0000 7f170a7fc700 1 -- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 msgr2=0x7f16f8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.103+0000 7f170a7fc700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 0x7f16f8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.158+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f170c06f270 con 0x7f170c0ad440 2023-10-07T11:01:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.159+0000 7f16f2ffd700 1 -- 172.21.15.29:0/1519865706 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v170) v1 ==== 104+0+11 (secure 0 0 0) 0x7f16fc056260 con 0x7f170c0ad440 2023-10-07T11:01:23.160 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T11:01:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 msgr2=0x7f16f8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 0x7f16f8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 msgr2=0x7f170c157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 0x7f170c157710 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f170c0aaa20 tx=0x7f16fc017000 comp rx=0 tx=0).stop 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 shutdown_connections 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f170c1596e0 0x7f170c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f170c0ad440 0x7f170c157710 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f16f8067050 0x7f16f8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 --2- 172.21.15.29:0/1519865706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f170c0cc630 0x7f170c0a9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 >> 172.21.15.29:0/1519865706 conn(0x7f170c067610 msgr2=0x7f170c0ac350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.161+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 shutdown_connections 2023-10-07T11:01:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.162+0000 7f17117ff700 1 -- 172.21.15.29:0/1519865706 wait complete. 2023-10-07T11:01:23.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T11:01:23.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.397+0000 7ff0877ff700 1 Processor -- start 2023-10-07T11:01:23.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.397+0000 7ff0877ff700 1 -- start start 2023-10-07T11:01:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.399+0000 7ff0877ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff08013fba0 0x7ff08006ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff0877ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff080067d20 0x7ff0800c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff0877ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800c78c0 con 0x7ff08013ff70 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff0877ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08006fa50 con 0x7ff08013fba0 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff0877ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08006c0f0 con 0x7ff080067d20 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff08559b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff08013fba0 0x7ff08006ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff085d9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff080067d20 0x7ff0800c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff085d9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff080067d20 0x7ff0800c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52192/0 (socket says 172.21.15.29:52192) 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff08559b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff08013fba0 0x7ff08006ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38972/0 (socket says 172.21.15.29:38972) 2023-10-07T11:01:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.400+0000 7ff085d9c700 1 -- 172.21.15.29:0/1025744485 learned_addr learned my addr 172.21.15.29:0/1025744485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.401+0000 7ff085d9c700 1 -- 172.21.15.29:0/1025744485 >> v2:172.21.15.29:3300/0 conn(0x7ff08013fba0 msgr2=0x7ff08006ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.401+0000 7ff085d9c700 1 --2- 172.21.15.29:0/1025744485 >> v2:172.21.15.29:3300/0 conn(0x7ff08013fba0 0x7ff08006ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.401+0000 7ff085d9c700 1 -- 172.21.15.29:0/1025744485 >> v1:172.21.15.29:6789/0 conn(0x7ff08013ff70 legacy=0x7ff080067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.401+0000 7ff085d9c700 1 -- 172.21.15.29:0/1025744485 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08006c3d0 con 0x7ff080067d20 2023-10-07T11:01:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.402+0000 7ff085d9c700 1 --2- 172.21.15.29:0/1025744485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff080067d20 0x7ff0800c7d50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff08006e090 tx=0x7ff078009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcf18cd178e55c2e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.403+0000 7ff07ffff700 1 -- 172.21.15.29:0/1025744485 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078009e70 con 0x7ff080067d20 2023-10-07T11:01:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.403+0000 7ff07ffff700 1 -- 172.21.15.29:0/1025744485 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07800b8f0 con 0x7ff080067d20 2023-10-07T11:01:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.403+0000 7ff07ffff700 1 -- 172.21.15.29:0/1025744485 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800bc70 con 0x7ff080067d20 2023-10-07T11:01:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.404+0000 7ff0877ff700 1 -- 172.21.15.29:0/1025744485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff080067d20 msgr2=0x7ff0800c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.404+0000 7ff0877ff700 1 --2- 172.21.15.29:0/1025744485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff080067d20 0x7ff0800c7d50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff08006e090 tx=0x7ff078009250 comp rx=0 tx=0).stop 2023-10-07T11:01:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.404+0000 7ff0877ff700 1 -- 172.21.15.29:0/1025744485 shutdown_connections 2023-10-07T11:01:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.404+0000 7ff0877ff700 1 --2- 172.21.15.29:0/1025744485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff080067d20 0x7ff0800c7d50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.404+0000 7ff0877ff700 1 --2- 172.21.15.29:0/1025744485 >> v2:172.21.15.29:3300/0 conn(0x7ff08013fba0 0x7ff08006ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.404+0000 7ff0877ff700 1 -- 172.21.15.29:0/1025744485 >> 172.21.15.29:0/1025744485 conn(0x7ff08006b1d0 msgr2=0x7ff0800cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.405+0000 7ff0877ff700 1 -- 172.21.15.29:0/1025744485 shutdown_connections 2023-10-07T11:01:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.405+0000 7ff0877ff700 1 -- 172.21.15.29:0/1025744485 wait complete. 2023-10-07T11:01:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.408+0000 7ff0877ff700 1 Processor -- start 2023-10-07T11:01:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.408+0000 7ff0877ff700 1 -- start start 2023-10-07T11:01:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.409+0000 7ff0877ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff080067d20 0x7ff08013eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.409+0000 7ff0877ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08013fba0 0x7ff080159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.409+0000 7ff0877ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08013ff70 0x7ff08015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.409+0000 7ff0877ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08015abd0 con 0x7ff080067d20 2023-10-07T11:01:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.409+0000 7ff0877ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08015c080 con 0x7ff08013ff70 2023-10-07T11:01:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.409+0000 7ff0877ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08015c320 con 0x7ff08013fba0 2023-10-07T11:01:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.410+0000 7ff084d9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08013fba0 0x7ff080159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.410+0000 7ff08559b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff080067d20 0x7ff08013eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.410+0000 7ff084d9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08013fba0 0x7ff080159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44560/0 (socket says 172.21.15.29:44560) 2023-10-07T11:01:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.410+0000 7ff084d9a700 1 -- 172.21.15.29:0/3980487853 learned_addr learned my addr 172.21.15.29:0/3980487853 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.410+0000 7ff085d9c700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08013ff70 0x7ff08015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.411+0000 7ff08559b700 1 -- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08013fba0 msgr2=0x7ff080159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.411+0000 7ff08559b700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08013fba0 0x7ff080159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.411+0000 7ff08559b700 1 -- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08013ff70 msgr2=0x7ff08015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.411+0000 7ff08559b700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08013ff70 0x7ff08015a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.411+0000 7ff08559b700 1 -- 172.21.15.29:0/3980487853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0780094f0 con 0x7ff080067d20 2023-10-07T11:01:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.412+0000 7ff08559b700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff080067d20 0x7ff08013eb90 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff07400ea50 tx=0x7ff0740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.412+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07400ee90 con 0x7ff080067d20 2023-10-07T11:01:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.412+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07400c7f0 con 0x7ff080067d20 2023-10-07T11:01:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.412+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08015c600 con 0x7ff080067d20 2023-10-07T11:01:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.412+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07400cb90 con 0x7ff080067d20 2023-10-07T11:01:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.412+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08015cbc0 con 0x7ff080067d20 2023-10-07T11:01:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.414+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08004c3c0 con 0x7ff080067d20 2023-10-07T11:01:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.416+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7ff074013470 con 0x7ff080067d20 2023-10-07T11:01:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.418+0000 7ff07d7fa700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 0x7ff06c069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.418+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7ff074084c40 con 0x7ff080067d20 2023-10-07T11:01:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.418+0000 7ff084d9a700 1 -- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 msgr2=0x7ff06c069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.418+0000 7ff084d9a700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 0x7ff06c069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:23.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.427+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff074051980 con 0x7ff080067d20 2023-10-07T11:01:23.490 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:23.489+0000 7fdca20fb1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:01:23.576 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:23.575+0000 7fdca20fb1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:01:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.619+0000 7ff084d9a700 1 -- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 msgr2=0x7ff06c069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:23.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.619+0000 7ff084d9a700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 0x7ff06c069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.631+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff08006c6b0 con 0x7ff080067d20 2023-10-07T11:01:23.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.632+0000 7ff07d7fa700 1 -- 172.21.15.29:0/3980487853 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v170) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff074054fa0 con 0x7ff080067d20 2023-10-07T11:01:23.633 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T11:01:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 msgr2=0x7ff06c069650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 0x7ff06c069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff080067d20 msgr2=0x7ff08013eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff080067d20 0x7ff08013eb90 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff07400ea50 tx=0x7ff0740091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 shutdown_connections 2023-10-07T11:01:23.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff08013ff70 0x7ff08015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff080067d20 0x7ff08013eb90 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff06c0671a0 0x7ff06c069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 --2- 172.21.15.29:0/3980487853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff08013fba0 0x7ff080159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 >> 172.21.15.29:0/3980487853 conn(0x7ff08006b1d0 msgr2=0x7ff080159370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 shutdown_connections 2023-10-07T11:01:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.634+0000 7ff0877ff700 1 -- 172.21.15.29:0/3980487853 wait complete. 2023-10-07T11:01:23.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:23.666 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:23.665+0000 7efd38c7a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:01:23.729 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:23.728+0000 7fdca20fb1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:01:23.764 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:23.763+0000 7efd38c7a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:01:23.817 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:23.816+0000 7fdca20fb1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:01:23.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.856+0000 7f8da29ff700 1 Processor -- start 2023-10-07T11:01:23.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.857+0000 7f8da29ff700 1 -- start start 2023-10-07T11:01:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.858+0000 7f8da29ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d9c159ec0 0x7f8d9c15a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.858+0000 7f8da29ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c14d790 con 0x7f8d9c0ad370 2023-10-07T11:01:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.858+0000 7f8da29ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c0ae5d0 con 0x7f8d9c0af400 2023-10-07T11:01:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.858+0000 7f8da29ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9c15a880 con 0x7f8d9c159ec0 2023-10-07T11:01:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.858+0000 7f8d9bfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8d9c0ad370 0x7f8d9c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56520/0 (socket says 172.21.15.29:56520) 2023-10-07T11:01:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.859+0000 7f8d9bfff700 1 -- 172.21.15.29:0/2202526979 learned_addr learned my addr 172.21.15.29:0/2202526979 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.859+0000 7f8d9b7fe700 1 --2- 172.21.15.29:0/2202526979 >> v2:172.21.15.29:3300/0 conn(0x7f8d9c159ec0 0x7f8d9c15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.859+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/2202526979 >> v1:172.21.15.144:6789/0 conn(0x7f8d9c0af400 legacy=0x7f8d9c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.859+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/2202526979 >> v1:172.21.15.29:6789/0 conn(0x7f8d9c0ad370 legacy=0x7f8d9c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.859+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/2202526979 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c0ad740 con 0x7f8d9c159ec0 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.859+0000 7f8d9b7fe700 1 --2- 172.21.15.29:0/2202526979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c15a340 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f8d900098e0 tx=0x7f8d900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8f355e5fb578cc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8d9affd700 1 -- 172.21.15.29:0/2202526979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90009e80 con 0x7f8d9c159ec0 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8d9affd700 1 -- 172.21.15.29:0/2202526979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9000b860 con 0x7f8d9c159ec0 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8d9affd700 1 -- 172.21.15.29:0/2202526979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9000bbe0 con 0x7f8d9c159ec0 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8da29ff700 1 -- 172.21.15.29:0/2202526979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c159ec0 msgr2=0x7f8d9c15a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8da29ff700 1 --2- 172.21.15.29:0/2202526979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c15a340 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f8d900098e0 tx=0x7f8d900091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8da29ff700 1 -- 172.21.15.29:0/2202526979 shutdown_connections 2023-10-07T11:01:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8da29ff700 1 --2- 172.21.15.29:0/2202526979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c15a340 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.860+0000 7f8da29ff700 1 -- 172.21.15.29:0/2202526979 >> 172.21.15.29:0/2202526979 conn(0x7f8d9c0680a0 msgr2=0x7f8d9c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.861+0000 7f8da29ff700 1 -- 172.21.15.29:0/2202526979 shutdown_connections 2023-10-07T11:01:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.861+0000 7f8da29ff700 1 -- 172.21.15.29:0/2202526979 wait complete. 2023-10-07T11:01:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.863+0000 7f8da29ff700 1 Processor -- start 2023-10-07T11:01:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.863+0000 7f8da29ff700 1 -- start start 2023-10-07T11:01:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da29ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d9c0ad370 0x7f8d9c13c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da29ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 0x7f8d9c0a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da29ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da29ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0a95d0 con 0x7f8d9c0af400 2023-10-07T11:01:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da29ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c13c7f0 con 0x7f8d9c159ec0 2023-10-07T11:01:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da29ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0a78e0 con 0x7f8d9c0ad370 2023-10-07T11:01:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 0x7f8d9c0a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 0x7f8d9c0a5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39002/0 (socket says 172.21.15.29:39002) 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/738159328 learned_addr learned my addr 172.21.15.29:0/738159328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9bfff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d9c0ad370 0x7f8d9c13c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da0f9c700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d9c0ad370 msgr2=0x7f8d9c13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d9c0ad370 0x7f8d9c13c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9c159ec0 msgr2=0x7f8d9c1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c1579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8d9b7fe700 1 -- 172.21.15.29:0/738159328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90009420 con 0x7f8d9c0af400 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.865+0000 7f8da0f9c700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.866+0000 7f8d9b7fe700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 0x7f8d9c0a5e80 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000b3c0 tx=0x7f8d90017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.866+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9000baa0 con 0x7f8d9c0af400 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.866+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d90026430 con 0x7f8d9c0af400 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.866+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c0a6cd0 con 0x7f8d9c0af400 2023-10-07T11:01:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.866+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900267d0 con 0x7f8d9c0af400 2023-10-07T11:01:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.866+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c0a7290 con 0x7f8d9c0af400 2023-10-07T11:01:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.868+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c04c3c0 con 0x7f8d9c0af400 2023-10-07T11:01:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.869+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f8d90023030 con 0x7f8d9c0af400 2023-10-07T11:01:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.870+0000 7f8d83fff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 0x7f8d88069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.871+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f8d900866a0 con 0x7f8d9c0af400 2023-10-07T11:01:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.871+0000 7f8d9bfff700 1 -- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 msgr2=0x7f8d88069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.871+0000 7f8d9bfff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 0x7f8d88069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:23.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:23.873+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d900533e0 con 0x7f8d9c0af400 2023-10-07T11:01:23.933 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:23.932+0000 7efd38c7a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:01:24.023 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.023+0000 7efd38c7a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:01:24.033 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.032+0000 7fdca20fb1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:01:24.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.072+0000 7f8d9bfff700 1 -- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 msgr2=0x7f8d88069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:24.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.072+0000 7f8d9bfff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 0x7f8d88069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:24.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.122+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d9c0a7a20 con 0x7f8d9c0af400 2023-10-07T11:01:24.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.123+0000 7f8d83fff700 1 -- 172.21.15.29:0/738159328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f8d9002e3a0 con 0x7f8d9c0af400 2023-10-07T11:01:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:24.124 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:45:18.619852+0000","modified":"2023-10-07T10:45:22.148839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9090":{"gid":9090,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/4080992371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4080992371},{"type":"v1","addr":"172.21.15.29:6833","nonce":4080992371}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-10-07T11:01:24.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 msgr2=0x7f8d88069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 0x7f8d88069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 msgr2=0x7f8d9c0a5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 0x7f8d9c0a5e80 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000b3c0 tx=0x7f8d90017000 comp rx=0 tx=0).stop 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 shutdown_connections 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9c159ec0 0x7f8d9c1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8d88067050 0x7f8d88069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d9c0ad370 0x7f8d9c13c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 --2- 172.21.15.29:0/738159328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c0af400 0x7f8d9c0a5e80 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.125+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 >> 172.21.15.29:0/738159328 conn(0x7f8d9c0680a0 msgr2=0x7f8d9c069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.126+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 shutdown_connections 2023-10-07T11:01:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.126+0000 7f8da29ff700 1 -- 172.21.15.29:0/738159328 wait complete. 2023-10-07T11:01:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-07T11:01:24.129 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.129+0000 7fdca20fb1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:01:24.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:24.225 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.224+0000 7fdca20fb1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:01:24.249 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.248+0000 7efd38c7a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:01:24.319 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.318+0000 7fdca20fb1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:01:24.350 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.349+0000 7efd38c7a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:01:24.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.384+0000 7f3534dff700 1 Processor -- start 2023-10-07T11:01:24.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.384+0000 7f3534dff700 1 -- start start 2023-10-07T11:01:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.385+0000 7f3534dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35300ae8b0 0x7f35300b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.385+0000 7f3534dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35300b1840 0x7f3530157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.385+0000 7f3534dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3530158ad0 con 0x7f3530157290 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.385+0000 7f3534dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35300b1cc0 con 0x7f35300ae8b0 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.385+0000 7f3534dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353015b140 con 0x7f35300b1840 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35300ae8b0 0x7f35300b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35300ae8b0 0x7f35300b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39004/0 (socket says 172.21.15.29:39004) 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 -- 172.21.15.29:0/2754046400 learned_addr learned my addr 172.21.15.29:0/2754046400 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352ed9d700 1 --2- 172.21.15.29:0/2754046400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35300b1840 0x7f3530157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352e59c700 1 --1- 172.21.15.29:0/2754046400 >> v1:172.21.15.144:6789/0 conn(0x7f3530157290 0x7f35300cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:58510/0 (socket says 172.21.15.29:58510) 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 -- 172.21.15.29:0/2754046400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35300b1840 msgr2=0x7f3530157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 --2- 172.21.15.29:0/2754046400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35300b1840 0x7f3530157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 -- 172.21.15.29:0/2754046400 >> v1:172.21.15.144:6789/0 conn(0x7f3530157290 legacy=0x7f35300cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 -- 172.21.15.29:0/2754046400 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353015b420 con 0x7f35300ae8b0 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352d59a700 1 -- 172.21.15.29:0/2754046400 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110204569 0 0) 0x7f3530158ad0 con 0x7f3530157290 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.386+0000 7f352dd9b700 1 --2- 172.21.15.29:0/2754046400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35300ae8b0 0x7f35300b1300 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f35240099b0 tx=0x7f3524009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ede1ec3f62cb13 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f352d59a700 1 -- 172.21.15.29:0/2754046400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524009ea0 con 0x7f35300ae8b0 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f352d59a700 1 -- 172.21.15.29:0/2754046400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352400b860 con 0x7f35300ae8b0 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f352d59a700 1 -- 172.21.15.29:0/2754046400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352400bc00 con 0x7f35300ae8b0 2023-10-07T11:01:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 -- 172.21.15.29:0/2754046400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35300ae8b0 msgr2=0x7f35300b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 --2- 172.21.15.29:0/2754046400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35300ae8b0 0x7f35300b1300 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f35240099b0 tx=0x7f3524009250 comp rx=0 tx=0).stop 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 -- 172.21.15.29:0/2754046400 shutdown_connections 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 --2- 172.21.15.29:0/2754046400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35300b1840 0x7f3530157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 --2- 172.21.15.29:0/2754046400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35300ae8b0 0x7f35300b1300 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 -- 172.21.15.29:0/2754046400 >> 172.21.15.29:0/2754046400 conn(0x7f35300680a0 msgr2=0x7f35300cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.387+0000 7f3534dff700 1 -- 172.21.15.29:0/2754046400 shutdown_connections 2023-10-07T11:01:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.388+0000 7f3534dff700 1 -- 172.21.15.29:0/2754046400 wait complete. 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.389+0000 7f3534dff700 1 Processor -- start 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.389+0000 7f3534dff700 1 -- start start 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f3534dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35300b1840 0x7f3530158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f3534dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 0x7f3530159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f3534dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3530147060 0x7f353006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f3534dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530159ad0 con 0x7f3530157290 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f3534dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353015a5d0 con 0x7f35300b1840 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f3534dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3530159d30 con 0x7f3530147060 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f352dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 0x7f3530159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f352dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 0x7f3530159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39018/0 (socket says 172.21.15.29:39018) 2023-10-07T11:01:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f352dd9b700 1 -- 172.21.15.29:0/1934583652 learned_addr learned my addr 172.21.15.29:0/1934583652 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f352e59c700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35300b1840 0x7f3530158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.390+0000 7f352ed9d700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3530147060 0x7f353006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.391+0000 7f352dd9b700 1 -- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3530147060 msgr2=0x7f353006d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.391+0000 7f352dd9b700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3530147060 0x7f353006d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.391+0000 7f352dd9b700 1 -- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35300b1840 msgr2=0x7f3530158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.391+0000 7f352dd9b700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35300b1840 0x7f3530158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.391+0000 7f352dd9b700 1 -- 172.21.15.29:0/1934583652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35240094f0 con 0x7f3530157290 2023-10-07T11:01:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.391+0000 7f352dd9b700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 0x7f3530159590 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3530068dd0 tx=0x7f3524017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.392+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524000d10 con 0x7f3530157290 2023-10-07T11:01:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.392+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352400bbe0 con 0x7f3530157290 2023-10-07T11:01:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.392+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353015a010 con 0x7f3530157290 2023-10-07T11:01:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.392+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524011690 con 0x7f3530157290 2023-10-07T11:01:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.392+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3530167860 con 0x7f3530157290 2023-10-07T11:01:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.393+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3530167a50 con 0x7f3530157290 2023-10-07T11:01:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.395+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f3524011ac0 con 0x7f3530157290 2023-10-07T11:01:24.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.396+0000 7f351e7fc700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.396+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f35240862e0 con 0x7f3530157290 2023-10-07T11:01:24.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.396+0000 7f352e59c700 1 -- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 msgr2=0x7f3510069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:24.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.396+0000 7f352e59c700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:24.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.400+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3524052fa0 con 0x7f3530157290 2023-10-07T11:01:24.454 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.453+0000 7efd38c7a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:01:24.508 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.507+0000 7fdca20fb1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:01:24.553 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.552+0000 7efd38c7a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:01:24.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.597+0000 7f352e59c700 1 -- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 msgr2=0x7f3510069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:24.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.597+0000 7f352e59c700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:24.604 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.603+0000 7fdca20fb1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:01:24.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.604+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f353004c3c0 con 0x7f3530157290 2023-10-07T11:01:24.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.605+0000 7f351e7fc700 1 -- 172.21.15.29:0/1934583652 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+17019 (secure 0 0 0) 0x7f35240565c0 con 0x7f3530157290 2023-10-07T11:01:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:24.606 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":170,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:01:22.788010+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:45:16.377745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-07T10:45:17.609874+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":170,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3001711291":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/681476780":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/3310824583":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/3296949315":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/4205507060":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/3718096826":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/2626373758":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/1610279030":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/4211255093":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/1517740475":"2023-10-08T10:56:23.164290+0000","172.21.15.144:6836/114714":"2023-10-08T10:45:34.271979+0000","172.21.15.144:6834/114714":"2023-10-08T10:45:34.271979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.607+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 msgr2=0x7f3510069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.607+0000 7f3534dff700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.607+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 msgr2=0x7f3530159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.607+0000 7f3534dff700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 0x7f3530159590 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3530068dd0 tx=0x7f3524017000 comp rx=0 tx=0).stop 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.607+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 shutdown_connections 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3530147060 0x7f353006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35300b1840 0x7f3530158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 --2- 172.21.15.29:0/1934583652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3530157290 0x7f3530159590 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 >> 172.21.15.29:0/1934583652 conn(0x7f35300680a0 msgr2=0x7f3530152980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 shutdown_connections 2023-10-07T11:01:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.608+0000 7f3534dff700 1 -- 172.21.15.29:0/1934583652 wait complete. 2023-10-07T11:01:24.627 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T11:01:24.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:24.753 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.752+0000 7efd38c7a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:01:24.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.824+0000 7f3c819ff700 1 Processor -- start 2023-10-07T11:01:24.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.824+0000 7f3c819ff700 1 -- start start 2023-10-07T11:01:24.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.825+0000 7f3c819ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.825+0000 7f3c819ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.825+0000 7f3c819ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c06f4e0 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.825+0000 7f3c819ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c06e8d0 con 0x7f3c7c0b0f50 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c819ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c06ebb0 con 0x7f3c7c159550 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39030/0 (socket says 172.21.15.29:39030) 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/1874256509 learned_addr learned my addr 172.21.15.29:0/1874256509 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7affd700 1 --2- 172.21.15.29:0/1874256509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/1874256509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 msgr2=0x7f3c7c0b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 --2- 172.21.15.29:0/1874256509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/1874256509 >> v1:172.21.15.29:6789/0 conn(0x7f3c7c0b2bb0 legacy=0x7f3c7c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.826+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/1874256509 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c06ee90 con 0x7f3c7c0b0f50 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.827+0000 7f3c7a7fc700 1 --2- 172.21.15.29:0/1874256509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f3c700099b0 tx=0x7f3c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718284a5199c835f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.827+0000 7f3c79ffb700 1 -- 172.21.15.29:0/1874256509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70009ea0 con 0x7f3c7c0b0f50 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.827+0000 7f3c79ffb700 1 -- 172.21.15.29:0/1874256509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7000b860 con 0x7f3c7c0b0f50 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.827+0000 7f3c79ffb700 1 -- 172.21.15.29:0/1874256509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000bc00 con 0x7f3c7c0b0f50 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.827+0000 7f3c819ff700 1 -- 172.21.15.29:0/1874256509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 msgr2=0x7f3c7c0b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 --2- 172.21.15.29:0/1874256509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f3c700099b0 tx=0x7f3c70009250 comp rx=0 tx=0).stop 2023-10-07T11:01:24.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 -- 172.21.15.29:0/1874256509 shutdown_connections 2023-10-07T11:01:24.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 --2- 172.21.15.29:0/1874256509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c0b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 --2- 172.21.15.29:0/1874256509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c0b2670 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 -- 172.21.15.29:0/1874256509 >> 172.21.15.29:0/1874256509 conn(0x7f3c7c069190 msgr2=0x7f3c7c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:24.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 -- 172.21.15.29:0/1874256509 shutdown_connections 2023-10-07T11:01:24.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.828+0000 7f3c819ff700 1 -- 172.21.15.29:0/1874256509 wait complete. 2023-10-07T11:01:24.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.829+0000 7f3c819ff700 1 Processor -- start 2023-10-07T11:01:24.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.829+0000 7f3c819ff700 1 -- start start 2023-10-07T11:01:24.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.830+0000 7f3c819ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c15c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.830+0000 7f3c819ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.830+0000 7f3c819ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.830+0000 7f3c819ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c152550 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.830+0000 7f3c819ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c16a6a0 con 0x7f3c7c0b0f50 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.830+0000 7f3c819ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c16b050 con 0x7f3c7c159550 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39034/0 (socket says 172.21.15.29:39034) 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/3347143816 learned_addr learned my addr 172.21.15.29:0/3347143816 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7affd700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c15c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7b7fe700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:24.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 msgr2=0x7f3c7c158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 msgr2=0x7f3c7c15c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c15c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 -- 172.21.15.29:0/3347143816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700094f0 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.831+0000 7f3c7a7fc700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c158350 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c06c520 tx=0x7f3c70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.832+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70000d10 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.832+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7000bbe0 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.832+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70011690 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.832+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c15c840 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.832+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c16aa00 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.833+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c16b200 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.835+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f3c70011a40 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.837+0000 7f3c62ffd700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 0x7f3c68069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:24.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.837+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f3c70086200 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.837+0000 7f3c7affd700 1 -- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 msgr2=0x7f3c68069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:24.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.837+0000 7f3c7affd700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 0x7f3c68069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:24.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:24.841+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c7c16b200 con 0x7f3c7c0b2bb0 2023-10-07T11:01:24.860 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:24.859+0000 7efd38c7a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:01:24.950 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:24.949+0000 7fdca20fb1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:01:25.038 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:25.037+0000 7fdca20fb1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.038+0000 7f3c7affd700 1 -- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 msgr2=0x7f3c68069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.038+0000 7f3c7affd700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 0x7f3c68069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:25.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.092+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c7c16b1c0 con 0x7f3c7c0b2bb0 2023-10-07T11:01:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.092+0000 7f3c62ffd700 1 -- 172.21.15.29:0/3347143816 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f3c70052ec0 con 0x7f3c7c0b2bb0 2023-10-07T11:01:25.093 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:25.094 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:45:18.619852+0000","modified":"2023-10-07T10:45:22.148839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9090":{"gid":9090,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/4080992371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4080992371},{"type":"v1","addr":"172.21.15.29:6833","nonce":4080992371}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-10-07T11:01:25.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 msgr2=0x7f3c68069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:25.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 0x7f3c68069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 msgr2=0x7f3c7c158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c158350 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c06c520 tx=0x7f3c70017000 comp rx=0 tx=0).stop 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 shutdown_connections 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3c68067050 0x7f3c68069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c7c159550 0x7f3c7c158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c7c0b0f50 0x7f3c7c15c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 --2- 172.21.15.29:0/3347143816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c0b2bb0 0x7f3c7c158350 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 >> 172.21.15.29:0/3347143816 conn(0x7f3c7c069190 msgr2=0x7f3c7c152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 shutdown_connections 2023-10-07T11:01:25.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.095+0000 7f3c819ff700 1 -- 172.21.15.29:0/3347143816 wait complete. 2023-10-07T11:01:25.098 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-07T11:01:25.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:25.214 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:25.213+0000 7efd38c7a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:01:25.317 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:25.316+0000 7efd38c7a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:25.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.321+0000 7f9df01ff700 1 Processor -- start 2023-10-07T11:01:25.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.321+0000 7f9df01ff700 1 -- start start 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9df01ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9de8142c30 0x7f9de80c8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9df01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de806c100 0x7f9de815d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9df01ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9de8143000 0x7f9de80cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9df01ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de8068f10 con 0x7f9de8143000 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9df01ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de8068be0 con 0x7f9de8142c30 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9df01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de815da60 con 0x7f9de806c100 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9dee79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9de8143000 0x7f9de80cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9dedf9b700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9de8142c30 0x7f9de80c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9dee79c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9de8143000 0x7f9de80cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39046/0 (socket says 172.21.15.29:39046) 2023-10-07T11:01:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9dee79c700 1 -- 172.21.15.29:0/2872521697 learned_addr learned my addr 172.21.15.29:0/2872521697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.322+0000 7f9ded79a700 1 --2- 172.21.15.29:0/2872521697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de806c100 0x7f9de815d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9dee79c700 1 -- 172.21.15.29:0/2872521697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de806c100 msgr2=0x7f9de815d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9dee79c700 1 --2- 172.21.15.29:0/2872521697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de806c100 0x7f9de815d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9dee79c700 1 -- 172.21.15.29:0/2872521697 >> v2:172.21.15.144:3300/0 conn(0x7f9de8142c30 msgr2=0x7f9de80c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9dee79c700 1 --2- 172.21.15.29:0/2872521697 >> v2:172.21.15.144:3300/0 conn(0x7f9de8142c30 0x7f9de80c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9dee79c700 1 -- 172.21.15.29:0/2872521697 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de815dd40 con 0x7f9de8143000 2023-10-07T11:01:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9dee79c700 1 --2- 172.21.15.29:0/2872521697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de8143000 0x7f9de80cbec0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f9de806b280 tx=0x7f9de0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c596687914f37924 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:25.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9decf99700 1 -- 172.21.15.29:0/2872521697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de000e070 con 0x7f9de8143000 2023-10-07T11:01:25.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9decf99700 1 -- 172.21.15.29:0/2872521697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de000b970 con 0x7f9de8143000 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.323+0000 7f9decf99700 1 -- 172.21.15.29:0/2872521697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00092e0 con 0x7f9de8143000 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 -- 172.21.15.29:0/2872521697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de8143000 msgr2=0x7f9de80cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2872521697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de8143000 0x7f9de80cbec0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f9de806b280 tx=0x7f9de0009840 comp rx=0 tx=0).stop 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 -- 172.21.15.29:0/2872521697 shutdown_connections 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2872521697 >> v2:172.21.15.144:3300/0 conn(0x7f9de8142c30 0x7f9de80c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2872521697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de8143000 0x7f9de80cbec0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2872521697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de806c100 0x7f9de815d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 -- 172.21.15.29:0/2872521697 >> 172.21.15.29:0/2872521697 conn(0x7f9de806b580 msgr2=0x7f9de806b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 -- 172.21.15.29:0/2872521697 shutdown_connections 2023-10-07T11:01:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.324+0000 7f9df01ff700 1 -- 172.21.15.29:0/2872521697 wait complete. 2023-10-07T11:01:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.326+0000 7f9df01ff700 1 Processor -- start 2023-10-07T11:01:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.327+0000 7f9df01ff700 1 -- start start 2023-10-07T11:01:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9df01ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 0x7f9de813eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9df01ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de8142c30 0x7f9de80b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9df01ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de8143000 0x7f9de80c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9df01ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80b3420 con 0x7f9de806c100 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9df01ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80b0450 con 0x7f9de8143000 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9df01ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80b06f0 con 0x7f9de8142c30 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9dedf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 0x7f9de813eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9dedf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 0x7f9de813eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39060/0 (socket says 172.21.15.29:39060) 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9dedf9b700 1 -- 172.21.15.29:0/2233015311 learned_addr learned my addr 172.21.15.29:0/2233015311 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9ded79a700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de8142c30 0x7f9de80b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.328+0000 7f9dee79c700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de8143000 0x7f9de80c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dedf9b700 1 -- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de8142c30 msgr2=0x7f9de80b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dedf9b700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de8142c30 0x7f9de80b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dedf9b700 1 -- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de8143000 msgr2=0x7f9de80c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dedf9b700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de8143000 0x7f9de80c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dedf9b700 1 -- 172.21.15.29:0/2233015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de0009ae0 con 0x7f9de806c100 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9ded79a700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de8142c30 0x7f9de80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dedf9b700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 0x7f9de813eb70 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ec80 tx=0x7f9de40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de400c850 con 0x7f9de806c100 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de400c9b0 con 0x7f9de806c100 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de400cd50 con 0x7f9de806c100 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de80ae090 con 0x7f9de806c100 2023-10-07T11:01:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.329+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de80b1170 con 0x7f9de806c100 2023-10-07T11:01:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.330+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de80b36c0 con 0x7f9de806c100 2023-10-07T11:01:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.331+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f9de4013680 con 0x7f9de806c100 2023-10-07T11:01:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.332+0000 7f9dddffb700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 0x7f9dcc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.332+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f9de4084d80 con 0x7f9de806c100 2023-10-07T11:01:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.332+0000 7f9ded79a700 1 -- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 msgr2=0x7f9dcc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.332+0000 7f9ded79a700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 0x7f9dcc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.336+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9de4051a40 con 0x7f9de806c100 2023-10-07T11:01:25.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.533+0000 7f9ded79a700 1 -- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 msgr2=0x7f9dcc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:25.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.533+0000 7f9ded79a700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 0x7f9dcc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:25.576 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:25.575+0000 7fdca20fb1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:01:25.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.579+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9de804c3c0 con 0x7f9de806c100 2023-10-07T11:01:25.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.580+0000 7f9dddffb700 1 -- 172.21.15.29:0/2233015311 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f9de4055060 con 0x7f9de806c100 2023-10-07T11:01:25.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:25.582 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T10:45:18.619852+0000","modified":"2023-10-07T10:45:22.148839+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9090":{"gid":9090,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/4080992371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4080992371},{"type":"v1","addr":"172.21.15.29:6833","nonce":4080992371}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-10-07T11:01:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.582+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 msgr2=0x7f9dcc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.582+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 0x7f9dcc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.582+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 msgr2=0x7f9de813eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.582+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 0x7f9de813eb70 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ec80 tx=0x7f9de40091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 shutdown_connections 2023-10-07T11:01:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9de8143000 0x7f9de80c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806c100 0x7f9de813eb70 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9dcc067050 0x7f9dcc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 --2- 172.21.15.29:0/2233015311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9de8142c30 0x7f9de80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 >> 172.21.15.29:0/2233015311 conn(0x7f9de806b580 msgr2=0x7f9de80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 shutdown_connections 2023-10-07T11:01:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.583+0000 7f9df01ff700 1 -- 172.21.15.29:0/2233015311 wait complete. 2023-10-07T11:01:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-07T11:01:25.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:25.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:25.761+0000 7fdca20fb1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:01:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.797+0000 7f80d8bff700 1 Processor -- start 2023-10-07T11:01:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.797+0000 7f80d8bff700 1 -- start start 2023-10-07T11:01:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d8bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80d4159550 0x7f80d40b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d8bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d406f4e0 con 0x7f80d40b2d80 2023-10-07T11:01:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d8bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d406e8d0 con 0x7f80d40b0f50 2023-10-07T11:01:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d8bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d406ebb0 con 0x7f80d4159550 2023-10-07T11:01:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80d4159550 0x7f80d40b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80d4159550 0x7f80d40b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39070/0 (socket says 172.21.15.29:39070) 2023-10-07T11:01:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.800+0000 7f80d259c700 1 -- 172.21.15.29:0/3199756631 learned_addr learned my addr 172.21.15.29:0/3199756631 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.801+0000 7f80d259c700 1 -- 172.21.15.29:0/3199756631 >> v1:172.21.15.144:6789/0 conn(0x7f80d40b0f50 legacy=0x7f80d40b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.801+0000 7f80d259c700 1 -- 172.21.15.29:0/3199756631 >> v1:172.21.15.29:6789/0 conn(0x7f80d40b2d80 legacy=0x7f80d40a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.801+0000 7f80d259c700 1 -- 172.21.15.29:0/3199756631 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d406ee90 con 0x7f80d4159550 2023-10-07T11:01:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.801+0000 7f80d259c700 1 --2- 172.21.15.29:0/3199756631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d40b0a10 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f80c40099b0 tx=0x7f80c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4ba4a86f2394ac server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.803+0000 7f80d159a700 1 -- 172.21.15.29:0/3199756631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c400b860 con 0x7f80d4159550 2023-10-07T11:01:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.803+0000 7f80d159a700 1 -- 172.21.15.29:0/3199756631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c400b9c0 con 0x7f80d4159550 2023-10-07T11:01:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.803+0000 7f80d159a700 1 -- 172.21.15.29:0/3199756631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c400bd60 con 0x7f80d4159550 2023-10-07T11:01:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.803+0000 7f80d8bff700 1 -- 172.21.15.29:0/3199756631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 msgr2=0x7f80d40b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.803+0000 7f80d8bff700 1 --2- 172.21.15.29:0/3199756631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d40b0a10 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f80c40099b0 tx=0x7f80c4009250 comp rx=0 tx=0).stop 2023-10-07T11:01:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.804+0000 7f80d8bff700 1 -- 172.21.15.29:0/3199756631 shutdown_connections 2023-10-07T11:01:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.804+0000 7f80d8bff700 1 --2- 172.21.15.29:0/3199756631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d40b0a10 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.804+0000 7f80d8bff700 1 -- 172.21.15.29:0/3199756631 >> 172.21.15.29:0/3199756631 conn(0x7f80d4069190 msgr2=0x7f80d406a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.804+0000 7f80d8bff700 1 -- 172.21.15.29:0/3199756631 shutdown_connections 2023-10-07T11:01:25.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.804+0000 7f80d8bff700 1 -- 172.21.15.29:0/3199756631 wait complete. 2023-10-07T11:01:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.805+0000 7f80d8bff700 1 Processor -- start 2023-10-07T11:01:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.805+0000 7f80d8bff700 1 -- start start 2023-10-07T11:01:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.806+0000 7f80d8bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 0x7f80d4146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.806+0000 7f80d8bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80d40b2d80 0x7f80d4146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.806+0000 7f80d8bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d416d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.806+0000 7f80d8bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d4158790 con 0x7f80d4159550 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.806+0000 7f80d8bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d415b100 con 0x7f80d40b0f50 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.806+0000 7f80d8bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d41584b0 con 0x7f80d40b2d80 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d416d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 0x7f80d4146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 0x7f80d4146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:52254/0 (socket says 172.21.15.29:52254) 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 -- 172.21.15.29:0/1815983215 learned_addr learned my addr 172.21.15.29:0/1815983215 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d1d9b700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80d40b2d80 0x7f80d4146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 -- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80d40b2d80 msgr2=0x7f80d4146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80d40b2d80 0x7f80d4146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 -- 172.21.15.29:0/1815983215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 msgr2=0x7f80d416d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d416d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.807+0000 7f80d259c700 1 -- 172.21.15.29:0/1815983215 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40094f0 con 0x7f80d40b0f50 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.808+0000 7f80d259c700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 0x7f80d4146940 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f80d406c520 tx=0x7f80c4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.808+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4024440 con 0x7f80d40b0f50 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.808+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c40245a0 con 0x7f80d40b0f50 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.808+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d415aab0 con 0x7f80d40b0f50 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.808+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4024920 con 0x7f80d40b0f50 2023-10-07T11:01:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.808+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d416d6e0 con 0x7f80d40b0f50 2023-10-07T11:01:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.809+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d415a910 con 0x7f80d40b0f50 2023-10-07T11:01:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.810+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f80c401a420 con 0x7f80d40b0f50 2023-10-07T11:01:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.811+0000 7f80c27fc700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 0x7f80b4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.811+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f80c4086830 con 0x7f80d40b0f50 2023-10-07T11:01:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.811+0000 7f80d1d9b700 1 -- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 msgr2=0x7f80b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.811+0000 7f80d1d9b700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 0x7f80b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:25.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:25.816+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80d415a910 con 0x7f80d40b0f50 2023-10-07T11:01:25.898 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:25.897+0000 7efd38c7a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:01:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.012+0000 7f80d1d9b700 1 -- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 msgr2=0x7f80b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.012+0000 7f80d1d9b700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 0x7f80b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.021+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d415a910 con 0x7f80d40b0f50 2023-10-07T11:01:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.022+0000 7f80c27fc700 1 -- 172.21.15.29:0/1815983215 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+17019 (secure 0 0 0) 0x7f80d415a910 con 0x7f80d40b0f50 2023-10-07T11:01:26.024 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:26.024 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":170,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:01:22.788010+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-07T10:45:16.377745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-07T10:45:17.609874+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":170,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3001711291":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/681476780":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/3310824583":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/3296949315":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/4205507060":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/3718096826":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/2626373758":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/1610279030":"2023-10-08T10:56:23.164290+0000","172.21.15.144:0/4211255093":"2023-10-08T10:45:34.271979+0000","172.21.15.144:0/1517740475":"2023-10-08T10:56:23.164290+0000","172.21.15.144:6836/114714":"2023-10-08T10:45:34.271979+0000","172.21.15.144:6834/114714":"2023-10-08T10:45:34.271979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:26.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 msgr2=0x7f80b4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:26.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 0x7f80b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 msgr2=0x7f80d4146940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 0x7f80d4146940 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f80d406c520 tx=0x7f80c4017000 comp rx=0 tx=0).stop 2023-10-07T11:01:26.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 shutdown_connections 2023-10-07T11:01:26.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80d4159550 0x7f80d416d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d40b0f50 0x7f80d4146940 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f80b4067050 0x7f80b4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 --2- 172.21.15.29:0/1815983215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80d40b2d80 0x7f80d4146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 >> 172.21.15.29:0/1815983215 conn(0x7f80d4069190 msgr2=0x7f80d4152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 shutdown_connections 2023-10-07T11:01:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.025+0000 7f80d8bff700 1 -- 172.21.15.29:0/1815983215 wait complete. 2023-10-07T11:01:26.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T11:01:26.098 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:26.097+0000 7efd38c7a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:01:26.140 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:26.139+0000 7fdca20fb1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.251+0000 7f063cdff700 1 Processor -- start 2023-10-07T11:01:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.251+0000 7f063cdff700 1 -- start start 2023-10-07T11:01:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f063cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06381537c0 0x7f0638156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f063cdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0638159670 con 0x7f06381533f0 2023-10-07T11:01:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f063cdff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063815a1f0 con 0x7f06380b04b0 2023-10-07T11:01:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f063cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063815a680 con 0x7f06381537c0 2023-10-07T11:01:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f063659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f06381533f0 0x7f06380afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56552/0 (socket says 172.21.15.29:56552) 2023-10-07T11:01:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f063659c700 1 -- 172.21.15.29:0/149868 learned_addr learned my addr 172.21.15.29:0/149868 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.253+0000 7f0636d9d700 1 --2- 172.21.15.29:0/149868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06381537c0 0x7f0638156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f063559a700 1 -- 172.21.15.29:0/149868 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202307391 0 0) 0x7f0638159670 con 0x7f06381533f0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f063559a700 1 -- 172.21.15.29:0/149868 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0620003570 con 0x7f06381533f0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f0636d9d700 1 -- 172.21.15.29:0/149868 >> v1:172.21.15.144:6789/0 conn(0x7f06380b04b0 legacy=0x7f0638157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f063559a700 1 -- 172.21.15.29:0/149868 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48647394 0 0) 0x7f063815a1f0 con 0x7f06380b04b0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f0636d9d700 1 -- 172.21.15.29:0/149868 >> v1:172.21.15.29:6789/0 conn(0x7f06381533f0 legacy=0x7f06380afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f0636d9d700 1 -- 172.21.15.29:0/149868 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06381598c0 con 0x7f06381537c0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.254+0000 7f0636d9d700 1 --2- 172.21.15.29:0/149868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06381537c0 0x7f0638156a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f063000afa0 tx=0x7f063000a890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f14775df4fffcb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.255+0000 7f063559a700 1 -- 172.21.15.29:0/149868 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063000f070 con 0x7f06381537c0 2023-10-07T11:01:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.255+0000 7f063559a700 1 -- 172.21.15.29:0/149868 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063000a2e0 con 0x7f06381537c0 2023-10-07T11:01:26.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.255+0000 7f063559a700 1 -- 172.21.15.29:0/149868 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063000cca0 con 0x7f06381537c0 2023-10-07T11:01:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.255+0000 7f063cdff700 1 -- 172.21.15.29:0/149868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06381537c0 msgr2=0x7f0638156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.255+0000 7f063cdff700 1 --2- 172.21.15.29:0/149868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06381537c0 0x7f0638156a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f063000afa0 tx=0x7f063000a890 comp rx=0 tx=0).stop 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.256+0000 7f063cdff700 1 -- 172.21.15.29:0/149868 shutdown_connections 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.256+0000 7f063cdff700 1 --2- 172.21.15.29:0/149868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06381537c0 0x7f0638156a90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.256+0000 7f063cdff700 1 -- 172.21.15.29:0/149868 >> 172.21.15.29:0/149868 conn(0x7f06380680a0 msgr2=0x7f06380cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.256+0000 7f063cdff700 1 -- 172.21.15.29:0/149868 shutdown_connections 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.256+0000 7f063cdff700 1 -- 172.21.15.29:0/149868 wait complete. 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.257+0000 7f063cdff700 1 Processor -- start 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.257+0000 7f063cdff700 1 -- start start 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.258+0000 7f063cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06380b04b0 0x7f0638152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.258+0000 7f063cdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 0x7f06380aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.258+0000 7f063cdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06381537c0 0x7f063814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.258+0000 7f063cdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380aa290 con 0x7f06381533f0 2023-10-07T11:01:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.258+0000 7f063cdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380ab450 con 0x7f06381537c0 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.258+0000 7f063cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06380ab660 con 0x7f06380b04b0 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 0x7f06380aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0636d9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06381537c0 0x7f063814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 0x7f06380aaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39082/0 (socket says 172.21.15.29:39082) 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 -- 172.21.15.29:0/505855963 learned_addr learned my addr 172.21.15.29:0/505855963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f063659c700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06380b04b0 0x7f0638152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 -- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06380b04b0 msgr2=0x7f0638152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06380b04b0 0x7f0638152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 -- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06381537c0 msgr2=0x7f063814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06381537c0 0x7f063814dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.259+0000 7f0635d9b700 1 -- 172.21.15.29:0/505855963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063000ab30 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.260+0000 7f0635d9b700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 0x7f06380aaf10 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c810 tx=0x7f062c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.260+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c011590 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.260+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c011710 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.260+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06380aa480 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.260+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c011ad0 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.260+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06380aaa00 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.261+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063804c3c0 con 0x7f06381533f0 2023-10-07T11:01:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.262+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f062c01c460 con 0x7f06381533f0 2023-10-07T11:01:26.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.263+0000 7f06267fc700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 0x7f0618069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.263+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f062c084810 con 0x7f06381533f0 2023-10-07T11:01:26.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.263+0000 7f063659c700 1 -- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 msgr2=0x7f0618069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:26.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.263+0000 7f063659c700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 0x7f0618069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:26.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.272+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f062c0514d0 con 0x7f06381533f0 2023-10-07T11:01:26.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.464+0000 7f063659c700 1 -- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 msgr2=0x7f0618069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.464+0000 7f063659c700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 0x7f0618069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:26.487 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:26.486+0000 7efd38c7a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:26.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.521+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0638155e10 con 0x7f06381533f0 2023-10-07T11:01:26.522 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T11:01:26.521+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T11:01:26.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.525+0000 7f06267fc700 1 -- 172.21.15.29:0/505855963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v86) v1 ==== 161+0+0 (secure 0 0 0) 0x7f062c054af0 con 0x7f06381533f0 2023-10-07T11:01:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 msgr2=0x7f0618069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 0x7f0618069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 msgr2=0x7f06380aaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 0x7f06380aaf10 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c810 tx=0x7f062c007df0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 shutdown_connections 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06381537c0 0x7f063814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f0618067050 0x7f0618069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06380b04b0 0x7f0638152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 --2- 172.21.15.29:0/505855963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06381533f0 0x7f06380aaf10 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 >> 172.21.15.29:0/505855963 conn(0x7f06380680a0 msgr2=0x7f06380afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 shutdown_connections 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.529+0000 7f063cdff700 1 -- 172.21.15.29:0/505855963 wait complete. 2023-10-07T11:01:26.531 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T11:01:26.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T11:01:26.564 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:26.564+0000 7fdca20fb1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:01:26.599 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T11:01:26.749 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:26.749+0000 7fdca20fb1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:26.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.759+0000 7f10ad3ff700 1 Processor -- start 2023-10-07T11:01:26.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.759+0000 7f10ad3ff700 1 -- start start 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10ad3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10a8153cc0 0x7f10a80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10ad3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f10a815ade0 0x7f10a80b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10ad3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a8154090 con 0x7f10a80b0000 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10ad3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a8159f60 con 0x7f10a8153cc0 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10ad3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a815a240 con 0x7f10a815ade0 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10a6ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10a8153cc0 0x7f10a80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10a6ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10a8153cc0 0x7f10a80b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39104/0 (socket says 172.21.15.29:39104) 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10a6ffd700 1 -- 172.21.15.29:0/3611001914 learned_addr learned my addr 172.21.15.29:0/3611001914 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.761+0000 7f10a67fc700 1 --2- 172.21.15.29:0/3611001914 >> v2:172.21.15.144:3300/0 conn(0x7f10a815ade0 0x7f10a80b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a6ffd700 1 -- 172.21.15.29:0/3611001914 >> v2:172.21.15.144:3300/0 conn(0x7f10a815ade0 msgr2=0x7f10a80b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a6ffd700 1 --2- 172.21.15.29:0/3611001914 >> v2:172.21.15.144:3300/0 conn(0x7f10a815ade0 0x7f10a80b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a6ffd700 1 -- 172.21.15.29:0/3611001914 >> v1:172.21.15.144:6789/0 conn(0x7f10a80b0000 legacy=0x7f10a815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a6ffd700 1 -- 172.21.15.29:0/3611001914 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a815a520 con 0x7f10a8153cc0 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a6ffd700 1 --2- 172.21.15.29:0/3611001914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a8153cc0 0x7f10a80b17c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f10a80add60 tx=0x7f1098009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73692abfad48187b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a5ffb700 1 -- 172.21.15.29:0/3611001914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1098005c10 con 0x7f10a8153cc0 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a5ffb700 1 -- 172.21.15.29:0/3611001914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1098005d70 con 0x7f10a8153cc0 2023-10-07T11:01:26.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.762+0000 7f10a5ffb700 1 -- 172.21.15.29:0/3611001914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1098006260 con 0x7f10a8153cc0 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 -- 172.21.15.29:0/3611001914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a8153cc0 msgr2=0x7f10a80b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/3611001914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a8153cc0 0x7f10a80b17c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f10a80add60 tx=0x7f1098009220 comp rx=0 tx=0).stop 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 -- 172.21.15.29:0/3611001914 shutdown_connections 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/3611001914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a8153cc0 0x7f10a80b17c0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/3611001914 >> v2:172.21.15.144:3300/0 conn(0x7f10a815ade0 0x7f10a80b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 -- 172.21.15.29:0/3611001914 >> 172.21.15.29:0/3611001914 conn(0x7f10a8067610 msgr2=0x7f10a8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 -- 172.21.15.29:0/3611001914 shutdown_connections 2023-10-07T11:01:26.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.763+0000 7f10ad3ff700 1 -- 172.21.15.29:0/3611001914 wait complete. 2023-10-07T11:01:26.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.765+0000 7f10ad3ff700 1 Processor -- start 2023-10-07T11:01:26.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.765+0000 7f10ad3ff700 1 -- start start 2023-10-07T11:01:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.766+0000 7f10ad3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 0x7f10a81518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.766+0000 7f10ad3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a8153cc0 0x7f10a81575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.766+0000 7f10ad3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a815ade0 0x7f10a8152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10ad3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a8169690 con 0x7f10a80b0000 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10ad3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a8156a20 con 0x7f10a8153cc0 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10ad3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a8155b30 con 0x7f10a815ade0 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 0x7f10a81518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 0x7f10a81518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39118/0 (socket says 172.21.15.29:39118) 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a67fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a8153cc0 0x7f10a81575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a6ffd700 1 -- 172.21.15.29:0/2975584292 learned_addr learned my addr 172.21.15.29:0/2975584292 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a77fe700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a815ade0 0x7f10a8152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a67fc700 1 -- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a815ade0 msgr2=0x7f10a8152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a67fc700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a815ade0 0x7f10a8152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a67fc700 1 -- 172.21.15.29:0/2975584292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 msgr2=0x7f10a81518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:26.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a67fc700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 0x7f10a81518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:26.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a67fc700 1 -- 172.21.15.29:0/2975584292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098009480 con 0x7f10a8153cc0 2023-10-07T11:01:26.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a77fe700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a815ade0 0x7f10a8152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:26.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.767+0000 7f10a6ffd700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 0x7f10a81518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:01:26.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.768+0000 7f10a67fc700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a8153cc0 0x7f10a81575d0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f10a815b200 tx=0x7f109c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.768+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c009450 con 0x7f10a8153cc0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.768+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c013680 con 0x7f10a8153cc0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.768+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a8155e10 con 0x7f10a8153cc0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.768+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c013a40 con 0x7f10a8153cc0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.768+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a8156450 con 0x7f10a8153cc0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.769+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a8157e50 con 0x7f10a8153cc0 2023-10-07T11:01:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.770+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 86) v1 ==== 84854+0+0 (secure 0 0 0) 0x7f109c0123d0 con 0x7f10a8153cc0 2023-10-07T11:01:26.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.771+0000 7f108effd700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 0x7f1094069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:26.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.771+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6784+0+0 (secure 0 0 0) 0x7f109c084be0 con 0x7f10a8153cc0 2023-10-07T11:01:26.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.771+0000 7f10a6ffd700 1 -- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 msgr2=0x7f1094069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:26.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.771+0000 7f10a6ffd700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 0x7f1094069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:26.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.775+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f109c051890 con 0x7f10a8153cc0 2023-10-07T11:01:26.946 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:26.945+0000 7efd38c7a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:01:26.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:26.946+0000 7fdca20fb1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:01:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.972+0000 7f10a6ffd700 1 -- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 msgr2=0x7f1094069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:26.972+0000 7f10a6ffd700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 0x7f1094069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:27.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.022+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f10a804c3c0 con 0x7f10a8153cc0 2023-10-07T11:01:27.058 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.058+0000 7fdca20fb1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.070+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 87) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f109c051c60 con 0x7f10a8153cc0 2023-10-07T11:01:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.070+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 msgr2=0x7f1094069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.070+0000 7f108effd700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 0x7f1094069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.137+0000 7fdc1c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.137+0000 7fdc1c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.137+0000 7fdc1c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.137+0000 7fdc1c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.137+0000 7fdc1c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.138+0000 7fdc1f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.138+0000 7fdc1f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.138+0000 7fdc1f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.140 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.138+0000 7fdc1f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.140 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:27.138+0000 7fdc1f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:27.140 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:27.140+0000 7efd38c7a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:27.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:27.335+0000 7efd38c7a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:01:27.450 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:27.450+0000 7efd38c7a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.527+0000 7f108effd700 1 -- 172.21.15.29:0/2975584292 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7f109c07d280 con 0x7f10a8153cc0 2023-10-07T11:01:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a8153cc0 msgr2=0x7f10a81575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a8153cc0 0x7f10a81575d0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f10a815b200 tx=0x7f109c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 shutdown_connections 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1094067050 0x7f1094069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a815ade0 0x7f10a8152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10a80b0000 0x7f10a81518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 --2- 172.21.15.29:0/2975584292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a8153cc0 0x7f10a81575d0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 >> 172.21.15.29:0/2975584292 conn(0x7f10a8067610 msgr2=0x7f10a815eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.529+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 shutdown_connections 2023-10-07T11:01:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.530+0000 7f10ad3ff700 1 -- 172.21.15.29:0/2975584292 wait complete. 2023-10-07T11:01:27.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T11:01:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.788+0000 7fac737ff700 1 Processor -- start 2023-10-07T11:01:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.789+0000 7fac737ff700 1 -- start start 2023-10-07T11:01:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac737ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fac6c0ced00 0x7fac6c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac737ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0690f0 con 0x7fac6c0b0610 2023-10-07T11:01:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac737ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0cf2e0 con 0x7fac6c0c8a20 2023-10-07T11:01:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac737ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c0687c0 con 0x7fac6c0ced00 2023-10-07T11:01:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac7159b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fac6c0b0610 0x7fac6c0c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56568/0 (socket says 172.21.15.29:56568) 2023-10-07T11:01:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac7159b700 1 -- 172.21.15.29:0/1509833301 learned_addr learned my addr 172.21.15.29:0/1509833301 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.790+0000 7fac70d9a700 1 --2- 172.21.15.29:0/1509833301 >> v2:172.21.15.144:3300/0 conn(0x7fac6c0ced00 0x7fac6c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131503388 0 0) 0x7fac6c0690f0 con 0x7fac6c0b0610 2023-10-07T11:01:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac58003570 con 0x7fac6c0b0610 2023-10-07T11:01:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3260272281 0 0) 0x7fac58003570 con 0x7fac6c0b0610 2023-10-07T11:01:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 >> v2:172.21.15.144:3300/0 conn(0x7fac6c0ced00 msgr2=0x7fac6c0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 --2- 172.21.15.29:0/1509833301 >> v2:172.21.15.144:3300/0 conn(0x7fac6c0ced00 0x7fac6c0b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 >> v1:172.21.15.144:6789/0 conn(0x7fac6c0c8a20 legacy=0x7fac6c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c068aa0 con 0x7fac6c0b0610 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fac68001470 con 0x7fac6c0b0610 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 >> v1:172.21.15.29:6789/0 conn(0x7fac6c0b0610 legacy=0x7fac6c0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac58003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac58004810 0x7fac58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac580051c0 0x7fac5800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0687c0 con 0x7fac58003a60 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5800c0e0 con 0x7fac580051c0 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5800c2a0 con 0x7fac58004810 2023-10-07T11:01:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac7159b700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac58003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.791+0000 7fac71d9c700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac580051c0 0x7fac5800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac70d9a700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac58004810 0x7fac58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac7159b700 1 -- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac58004810 msgr2=0x7fac58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac7159b700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac58004810 0x7fac58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac7159b700 1 -- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac580051c0 msgr2=0x7fac5800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac7159b700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac580051c0 0x7fac5800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac7159b700 1 -- 172.21.15.29:0/1509833301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58003570 con 0x7fac58003a60 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac71d9c700 1 -- 172.21.15.29:0/1509833301 reap_dead start 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac7159b700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac58003e30 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fac6800c2d0 tx=0x7fac68001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d516b35838e31c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6800cbc0 con 0x7fac58003a60 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6800cd20 con 0x7fac58003a60 2023-10-07T11:01:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.792+0000 7fac63fff700 1 -- 172.21.15.29:0/1509833301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68005b70 con 0x7fac58003a60 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 -- 172.21.15.29:0/1509833301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 msgr2=0x7fac58003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac58003e30 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fac6800c2d0 tx=0x7fac68001df0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 -- 172.21.15.29:0/1509833301 shutdown_connections 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 --2- 172.21.15.29:0/1509833301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac58003e30 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 -- 172.21.15.29:0/1509833301 >> 172.21.15.29:0/1509833301 conn(0x7fac6c063a10 msgr2=0x7fac6c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 -- 172.21.15.29:0/1509833301 shutdown_connections 2023-10-07T11:01:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.793+0000 7fac737ff700 1 -- 172.21.15.29:0/1509833301 wait complete. 2023-10-07T11:01:27.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.795+0000 7fac737ff700 1 Processor -- start 2023-10-07T11:01:27.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.795+0000 7fac737ff700 1 -- start start 2023-10-07T11:01:27.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac737ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac6c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac737ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6c151380 0x7fac6c154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac737ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac6c1517f0 0x7fac6c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac737ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b2100 con 0x7fac58003a60 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac737ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b14f0 con 0x7fac6c151380 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac737ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b17d0 con 0x7fac6c1517f0 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac6c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac6c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39146/0 (socket says 172.21.15.29:39146) 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 -- 172.21.15.29:0/2758273264 learned_addr learned my addr 172.21.15.29:0/2758273264 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac71d9c700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac6c1517f0 0x7fac6c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac70d9a700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6c151380 0x7fac6c154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 -- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac6c1517f0 msgr2=0x7fac6c1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:27.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac6c1517f0 0x7fac6c1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 -- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6c151380 msgr2=0x7fac6c154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6c151380 0x7fac6c154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac7159b700 1 -- 172.21.15.29:0/2758273264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0b1ab0 con 0x7fac58003a60 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.796+0000 7fac70d9a700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6c151380 0x7fac6c154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.797+0000 7fac7159b700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac6c0b2590 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c0b0 tx=0x7fac68015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.797+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6800ce90 con 0x7fac58003a60 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.797+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac68000f50 con 0x7fac58003a60 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.797+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68007320 con 0x7fac58003a60 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.797+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c0b1d90 con 0x7fac58003a60 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.797+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c0b2ad0 con 0x7fac58003a60 2023-10-07T11:01:27.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.798+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0b23e0 con 0x7fac58003a60 2023-10-07T11:01:27.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.799+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 87) v1 ==== 84451+0+0 (secure 0 0 0) 0x7fac680076d0 con 0x7fac58003a60 2023-10-07T11:01:27.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.800+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 7128+0+0 (secure 0 0 0) 0x7fac68085d40 con 0x7fac58003a60 2023-10-07T11:01:27.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:27.804+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac68052d70 con 0x7fac58003a60 2023-10-07T11:01:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.011+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac6c0b1f20 con 0x7fac58003a60 2023-10-07T11:01:28.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.072+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 88) v1 ==== 84578+0+0 (secure 0 0 0) 0x7fac68052820 con 0x7fac58003a60 2023-10-07T11:01:28.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.072+0000 7fac617fa700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fac5006ce00 0x7fac5006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:28.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.073+0000 7fac70d9a700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fac5006ce00 0x7fac5006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.080+0000 7fac70d9a700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fac5006ce00 0x7fac5006f1e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fac6c09c280 tx=0x7fac5c008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:28.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.814+0000 7fac617fa700 1 -- 172.21.15.29:0/2758273264 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v173) v1 ==== 181+0+0 (secure 0 0 0) 0x7fac68056570 con 0x7fac58003a60 2023-10-07T11:01:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fac5006ce00 msgr2=0x7fac5006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fac5006ce00 0x7fac5006f1e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fac6c09c280 tx=0x7fac5c008040 comp rx=0 tx=0).stop 2023-10-07T11:01:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 msgr2=0x7fac6c0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac6c0b2590 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c0b0 tx=0x7fac68015000 comp rx=0 tx=0).stop 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 shutdown_connections 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fac5006ce00 0x7fac5006f1e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fac6c1517f0 0x7fac6c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac58003a60 0x7fac6c0b2590 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 --2- 172.21.15.29:0/2758273264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6c151380 0x7fac6c154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.818+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 >> 172.21.15.29:0/2758273264 conn(0x7fac6c063a10 msgr2=0x7fac6c0c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.819+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 shutdown_connections 2023-10-07T11:01:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:28.819+0000 7fac737ff700 1 -- 172.21.15.29:0/2758273264 wait complete. 2023-10-07T11:01:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T11:01:28.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T11:01:29.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.044+0000 7f32563ff700 1 Processor -- start 2023-10-07T11:01:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.044+0000 7f32563ff700 1 -- start start 2023-10-07T11:01:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.045+0000 7f32563ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f32500b19a0 0x7f32500cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.045+0000 7f32563ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32500aeae0 con 0x7f3250157a20 2023-10-07T11:01:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.045+0000 7f32563ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32500ca430 con 0x7f3250157650 2023-10-07T11:01:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.045+0000 7f32563ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32500ca710 con 0x7f32500b19a0 2023-10-07T11:01:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.045+0000 7f324f7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3250157a20 0x7f32500b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56574/0 (socket says 172.21.15.29:56574) 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.045+0000 7f324f7fe700 1 -- 172.21.15.29:0/1211009604 learned_addr learned my addr 172.21.15.29:0/1211009604 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f325499c700 1 --2- 172.21.15.29:0/1211009604 >> v2:172.21.15.144:3300/0 conn(0x7f32500b19a0 0x7f32500cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691716324 0 0) 0x7f32500aeae0 con 0x7f3250157a20 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3238003570 con 0x7f3250157a20 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103655573 0 0) 0x7f3238003570 con 0x7f3250157a20 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 >> v2:172.21.15.144:3300/0 conn(0x7f32500b19a0 msgr2=0x7f32500cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 --2- 172.21.15.29:0/1211009604 >> v2:172.21.15.144:3300/0 conn(0x7f32500b19a0 0x7f32500cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 >> v1:172.21.15.144:6789/0 conn(0x7f3250157650 legacy=0x7f32500ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32500ca9f0 con 0x7f3250157a20 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f3244001470 con 0x7f3250157a20 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 >> v1:172.21.15.29:6789/0 conn(0x7f3250157a20 legacy=0x7f32500b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3238003a60 0x7f3238003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3238004810 0x7f3238004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32380052e0 0x7f323800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32500ca710 con 0x7f3238003a60 2023-10-07T11:01:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323800c2a0 con 0x7f32380052e0 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.046+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323800c460 con 0x7f3238004810 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3238003a60 0x7f3238003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f325499c700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32380052e0 0x7f323800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324f7fe700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3238004810 0x7f3238004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 -- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3238004810 msgr2=0x7f3238004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3238004810 0x7f3238004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 -- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32380052e0 msgr2=0x7f323800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32380052e0 0x7f323800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 -- 172.21.15.29:0/1211009604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3238003570 con 0x7f3238003a60 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f325499c700 1 -- 172.21.15.29:0/1211009604 reap_dead start 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324ffff700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3238003a60 0x7f3238003e70 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f32400018b0 tx=0x7f3240007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8236357e610d4187 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.047+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240012420 con 0x7f3238003a60 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32400125a0 con 0x7f3238003a60 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f324effd700 1 -- 172.21.15.29:0/1211009604 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240012960 con 0x7f3238003a60 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 -- 172.21.15.29:0/1211009604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3238003a60 msgr2=0x7f3238003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3238003a60 0x7f3238003e70 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f32400018b0 tx=0x7f3240007d30 comp rx=0 tx=0).stop 2023-10-07T11:01:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 -- 172.21.15.29:0/1211009604 shutdown_connections 2023-10-07T11:01:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 --2- 172.21.15.29:0/1211009604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3238003a60 0x7f3238003e70 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 -- 172.21.15.29:0/1211009604 >> 172.21.15.29:0/1211009604 conn(0x7f3250067610 msgr2=0x7f3250067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 -- 172.21.15.29:0/1211009604 shutdown_connections 2023-10-07T11:01:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.048+0000 7f32563ff700 1 -- 172.21.15.29:0/1211009604 wait complete. 2023-10-07T11:01:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.050+0000 7f32563ff700 1 Processor -- start 2023-10-07T11:01:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.051+0000 7f32563ff700 1 -- start start 2023-10-07T11:01:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f32563ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3238003a60 0x7f325016ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f32563ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3250152790 0x7f325013da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f32563ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 0x7f32500abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f32563ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3250158190 con 0x7f325014b440 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f32563ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325016b490 con 0x7f3238003a60 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f32563ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325016b770 con 0x7f3250152790 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 0x7f32500abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 0x7f32500abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39172/0 (socket says 172.21.15.29:39172) 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 -- 172.21.15.29:0/3174126157 learned_addr learned my addr 172.21.15.29:0/3174126157 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f324f7fe700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3250152790 0x7f325013da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f324ffff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3238003a60 0x7f325016ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 -- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3250152790 msgr2=0x7f325013da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3250152790 0x7f325013da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 -- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3238003a60 msgr2=0x7f325016ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3238003a60 0x7f325016ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f325499c700 1 -- 172.21.15.29:0/3174126157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325016ba50 con 0x7f325014b440 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.052+0000 7f324ffff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3238003a60 0x7f325016ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.053+0000 7f325499c700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 0x7f32500abe80 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f32500c9880 tx=0x7f324800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.053+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324800e040 con 0x7f325014b440 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.053+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324800f040 con 0x7f325014b440 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.053+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324800fe20 con 0x7f325014b440 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.053+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3250157e20 con 0x7f325014b440 2023-10-07T11:01:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.053+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325013dfb0 con 0x7f325014b440 2023-10-07T11:01:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.054+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32500718b0 con 0x7f325014b440 2023-10-07T11:01:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.055+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f3248015640 con 0x7f325014b440 2023-10-07T11:01:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.056+0000 7f3237fff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f323c066d80 0x7f323c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.056+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6690+0+0 (secure 0 0 0) 0x7f3248084eb0 con 0x7f325014b440 2023-10-07T11:01:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.057+0000 7f324ffff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f323c066d80 0x7f323c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.057+0000 7f324ffff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f323c066d80 0x7f323c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f32400018b0 tx=0x7f3240015000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.061+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3248051f20 con 0x7f325014b440 2023-10-07T11:01:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.261+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f32500cacd0 con 0x7f325014b440 2023-10-07T11:01:29.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:29.819+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f32480522f0 con 0x7f325014b440 2023-10-07T11:01:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.085+0000 7f3237fff700 1 -- 172.21.15.29:0/3174126157 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v174) v1 ==== 169+0+0 (secure 0 0 0) 0x7f32480559b0 con 0x7f325014b440 2023-10-07T11:01:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.089+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f323c066d80 msgr2=0x7f323c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.089+0000 7f32563ff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f323c066d80 0x7f323c069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f32400018b0 tx=0x7f3240015000 comp rx=0 tx=0).stop 2023-10-07T11:01:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.089+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 msgr2=0x7f32500abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.089+0000 7f32563ff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 0x7f32500abe80 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f32500c9880 tx=0x7f324800b040 comp rx=0 tx=0).stop 2023-10-07T11:01:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.090+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 shutdown_connections 2023-10-07T11:01:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.090+0000 7f32563ff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f325014b440 0x7f32500abe80 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.090+0000 7f32563ff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3238003a60 0x7f325016ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.090+0000 7f32563ff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f323c066d80 0x7f323c069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.090+0000 7f32563ff700 1 --2- 172.21.15.29:0/3174126157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3250152790 0x7f325013da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.090+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 >> 172.21.15.29:0/3174126157 conn(0x7f3250067610 msgr2=0x7f325006e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.092+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 shutdown_connections 2023-10-07T11:01:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.093+0000 7f32563ff700 1 -- 172.21.15.29:0/3174126157 wait complete. 2023-10-07T11:01:30.096 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T11:01:30.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T11:01:30.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.322+0000 7fc7b03ff700 1 Processor -- start 2023-10-07T11:01:30.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.322+0000 7fc7b03ff700 1 -- start start 2023-10-07T11:01:30.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7b03ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159750 0x7fc7a80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7b03ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a8159b20 0x7fc7a80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7b03ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a80cbf20 con 0x7fc7a80ce6d0 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7b03ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a80cc200 con 0x7fc7a8159b20 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7b03ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80cc4e0 con 0x7fc7a8159750 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7ad99a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a8159b20 0x7fc7a80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7ad99a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a8159b20 0x7fc7a80ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39184/0 (socket says 172.21.15.29:39184) 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7ad99a700 1 -- 172.21.15.29:0/610529254 learned_addr learned my addr 172.21.15.29:0/610529254 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7ae99c700 1 --1- 172.21.15.29:0/610529254 >> v1:172.21.15.29:6789/0 conn(0x7fc7a80ce6d0 0x7fc7a815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56584/0 (socket says 172.21.15.29:56584) 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7ae19b700 1 --2- 172.21.15.29:0/610529254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159750 0x7fc7a80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.324+0000 7fc7ad99a700 1 -- 172.21.15.29:0/610529254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159750 msgr2=0x7fc7a80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad99a700 1 --2- 172.21.15.29:0/610529254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159750 0x7fc7a80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad99a700 1 -- 172.21.15.29:0/610529254 >> v1:172.21.15.29:6789/0 conn(0x7fc7a80ce6d0 legacy=0x7fc7a815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad99a700 1 -- 172.21.15.29:0/610529254 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80cc7c0 con 0x7fc7a8159b20 2023-10-07T11:01:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad199700 1 -- 172.21.15.29:0/610529254 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131666105 0 0) 0x7fc7a80cbf20 con 0x7fc7a80ce6d0 2023-10-07T11:01:30.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad99a700 1 --2- 172.21.15.29:0/610529254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159b20 0x7fc7a80ce190 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc7980099b0 tx=0x7fc798009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242978362c810a00 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:30.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad199700 1 -- 172.21.15.29:0/610529254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798009ea0 con 0x7fc7a8159b20 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad199700 1 -- 172.21.15.29:0/610529254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79800b860 con 0x7fc7a8159b20 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.325+0000 7fc7ad199700 1 -- 172.21.15.29:0/610529254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79800bc00 con 0x7fc7a8159b20 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 -- 172.21.15.29:0/610529254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159b20 msgr2=0x7fc7a80ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/610529254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159b20 0x7fc7a80ce190 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc7980099b0 tx=0x7fc798009250 comp rx=0 tx=0).stop 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 -- 172.21.15.29:0/610529254 shutdown_connections 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/610529254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159750 0x7fc7a80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/610529254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159b20 0x7fc7a80ce190 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 -- 172.21.15.29:0/610529254 >> 172.21.15.29:0/610529254 conn(0x7fc7a8063a10 msgr2=0x7fc7a80d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 -- 172.21.15.29:0/610529254 shutdown_connections 2023-10-07T11:01:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.326+0000 7fc7b03ff700 1 -- 172.21.15.29:0/610529254 wait complete. 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.327+0000 7fc7b03ff700 1 Processor -- start 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.328+0000 7fc7b03ff700 1 -- start start 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.328+0000 7fc7b03ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a80ce6d0 0x7fc7a8153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.328+0000 7fc7b03ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 0x7fc7a816da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7b03ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159b20 0x7fc7a816a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7b03ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80b5950 con 0x7fc7a8159750 2023-10-07T11:01:30.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7b03ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8169a10 con 0x7fc7a80ce6d0 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7b03ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80b5140 con 0x7fc7a8159b20 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 0x7fc7a816da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 0x7fc7a816da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39190/0 (socket says 172.21.15.29:39190) 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 -- 172.21.15.29:0/4128661783 learned_addr learned my addr 172.21.15.29:0/4128661783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ae19b700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a80ce6d0 0x7fc7a8153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ae99c700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159b20 0x7fc7a816a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 -- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159b20 msgr2=0x7fc7a816a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159b20 0x7fc7a816a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 -- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a80ce6d0 msgr2=0x7fc7a8153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a80ce6d0 0x7fc7a8153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ad99a700 1 -- 172.21.15.29:0/4128661783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7980094f0 con 0x7fc7a8159750 2023-10-07T11:01:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.329+0000 7fc7ae19b700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a80ce6d0 0x7fc7a8153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:30.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.330+0000 7fc7ad99a700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 0x7fc7a816da20 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:30.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.330+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79800bbe0 con 0x7fc7a8159750 2023-10-07T11:01:30.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.330+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc798011420 con 0x7fc7a8159750 2023-10-07T11:01:30.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.330+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980117e0 con 0x7fc7a8159750 2023-10-07T11:01:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.330+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a8169f70 con 0x7fc7a8159750 2023-10-07T11:01:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.330+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a80b5390 con 0x7fc7a8159750 2023-10-07T11:01:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.331+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a816a100 con 0x7fc7a8159750 2023-10-07T11:01:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.332+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc798011bd0 con 0x7fc7a8159750 2023-10-07T11:01:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.333+0000 7fc79e7fc700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc78c066f30 0x7fc78c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.333+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fc798086070 con 0x7fc7a8159750 2023-10-07T11:01:30.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.333+0000 7fc7ae19b700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc78c066f30 0x7fc78c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.334+0000 7fc7ae19b700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc78c066f30 0x7fc78c0693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009d70 tx=0x7fc7a4008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:30.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.337+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7980b75a0 con 0x7fc7a8159750 2023-10-07T11:01:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.541+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc7a804c3c0 con 0x7fc7a8159750 2023-10-07T11:01:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.545+0000 7fc79e7fc700 1 -- 172.21.15.29:0/4128661783 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc7980150b0 con 0x7fc7a8159750 2023-10-07T11:01:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.547+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc78c066f30 msgr2=0x7fc78c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.547+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc78c066f30 0x7fc78c0693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009d70 tx=0x7fc7a4008040 comp rx=0 tx=0).stop 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.547+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 msgr2=0x7fc7a816da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.547+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 0x7fc7a816da20 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798017000 comp rx=0 tx=0).stop 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 shutdown_connections 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc78c066f30 0x7fc78c0693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a8159b20 0x7fc7a816a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a80ce6d0 0x7fc7a8153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 --2- 172.21.15.29:0/4128661783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a8159750 0x7fc7a816da20 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 >> 172.21.15.29:0/4128661783 conn(0x7fc7a8063a10 msgr2=0x7fc7a80cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 shutdown_connections 2023-10-07T11:01:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.548+0000 7fc7b03ff700 1 -- 172.21.15.29:0/4128661783 wait complete. 2023-10-07T11:01:30.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T11:01:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.762+0000 7f20f3bff700 1 Processor -- start 2023-10-07T11:01:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.762+0000 7f20f3bff700 1 -- start start 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.763+0000 7f20f3bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.763+0000 7f20f3bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20ec0ce8a0 0x7f20ec0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.763+0000 7f20f3bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec0cbf20 con 0x7f20ec159750 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.763+0000 7f20f3bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec0cc200 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.763+0000 7f20f3bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0cc4e0 con 0x7f20ec159b20 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f199b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f20ec159750 0x7f20ec0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56586/0 (socket says 172.21.15.29:56586) 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f199b700 1 -- 172.21.15.29:0/3287670292 learned_addr learned my addr 172.21.15.29:0/3287670292 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f119a700 1 --2- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f219c700 1 --2- 172.21.15.29:0/3287670292 >> v2:172.21.15.144:3300/0 conn(0x7f20ec0ce8a0 0x7f20ec0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f0999700 1 -- 172.21.15.29:0/3287670292 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256313041 0 0) 0x7f20ec0cbf20 con 0x7f20ec159750 2023-10-07T11:01:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f0999700 1 -- 172.21.15.29:0/3287670292 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d8003570 con 0x7f20ec159750 2023-10-07T11:01:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f219c700 1 -- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 msgr2=0x7f20ec0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f219c700 1 --2- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f219c700 1 -- 172.21.15.29:0/3287670292 >> v1:172.21.15.29:6789/0 conn(0x7f20ec159750 legacy=0x7f20ec0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.764+0000 7f20f219c700 1 -- 172.21.15.29:0/3287670292 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec0cc7c0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.765+0000 7f20f219c700 1 --2- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec0c5460 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009e60 tx=0x7f20e4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79ae86897b8155a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.765+0000 7f20f0999700 1 -- 172.21.15.29:0/3287670292 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e400e070 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.765+0000 7f20f0999700 1 -- 172.21.15.29:0/3287670292 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e400bab0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.765+0000 7f20f0999700 1 -- 172.21.15.29:0/3287670292 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e400be70 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.765+0000 7f20f3bff700 1 -- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec0ce8a0 msgr2=0x7f20ec0c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.765+0000 7f20f3bff700 1 --2- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec0c5460 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009e60 tx=0x7f20e4009780 comp rx=0 tx=0).stop 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.766+0000 7f20f3bff700 1 -- 172.21.15.29:0/3287670292 shutdown_connections 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.766+0000 7f20f3bff700 1 --2- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec0c5460 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.766+0000 7f20f3bff700 1 --2- 172.21.15.29:0/3287670292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.766+0000 7f20f3bff700 1 -- 172.21.15.29:0/3287670292 >> 172.21.15.29:0/3287670292 conn(0x7f20ec063a10 msgr2=0x7f20ec0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.766+0000 7f20f3bff700 1 -- 172.21.15.29:0/3287670292 shutdown_connections 2023-10-07T11:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.766+0000 7f20f3bff700 1 -- 172.21.15.29:0/3287670292 wait complete. 2023-10-07T11:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.767+0000 7f20f3bff700 1 Processor -- start 2023-10-07T11:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.767+0000 7f20f3bff700 1 -- start start 2023-10-07T11:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f3bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f3bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec159750 0x7f20ec15fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f3bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec14f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f3bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec15ff80 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f3bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0c7af0 con 0x7f20ec159750 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f3bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec1589f0 con 0x7f20ec159b20 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.768+0000 7f20f199b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39198/0 (socket says 172.21.15.29:39198) 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 -- 172.21.15.29:0/2508290584 learned_addr learned my addr 172.21.15.29:0/2508290584 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f119a700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec159750 0x7f20ec15fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f219c700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec14f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 -- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 msgr2=0x7f20ec14f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec14f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 -- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec159750 msgr2=0x7f20ec15fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec159750 0x7f20ec15fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 -- 172.21.15.29:0/2508290584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e4009a20 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f219c700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec14f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f199b700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec157a90 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f20e800c7c0 tx=0x7f20e8007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80072e0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec0c7db0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.769+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e8007440 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.770+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80057f0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.770+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec158150 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.771+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec04c3c0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.771+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f20e8005c20 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.772+0000 7f20e1ffb700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f20d0066f30 0x7f20d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.772+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f20e8083be0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.773+0000 7f20f119a700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f20d0066f30 0x7f20d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.773+0000 7f20f119a700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f20d0066f30 0x7f20d00693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0b6490 tx=0x7f20dc0091a0 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.777+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20e8050ce0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.980+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f20ec0cec70 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.984+0000 7f20e1ffb700 1 -- 172.21.15.29:0/2508290584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f20e80170a0 con 0x7f20ec0ce8a0 2023-10-07T11:01:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f20d0066f30 msgr2=0x7f20d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f20d0066f30 0x7f20d00693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0b6490 tx=0x7f20dc0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 msgr2=0x7f20ec157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec157a90 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f20e800c7c0 tx=0x7f20e8007c80 comp rx=0 tx=0).stop 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 shutdown_connections 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f20d0066f30 0x7f20d00693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20ec159b20 0x7f20ec14f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ec0ce8a0 0x7f20ec157a90 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 --2- 172.21.15.29:0/2508290584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20ec159750 0x7f20ec15fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 >> 172.21.15.29:0/2508290584 conn(0x7f20ec063a10 msgr2=0x7f20ec0704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 shutdown_connections 2023-10-07T11:01:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:30.986+0000 7f20f3bff700 1 -- 172.21.15.29:0/2508290584 wait complete. 2023-10-07T11:01:31.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T11:01:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.202+0000 7f2bc23ff700 1 Processor -- start 2023-10-07T11:01:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.202+0000 7f2bc23ff700 1 -- start start 2023-10-07T11:01:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.203+0000 7f2bc23ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2bbc156180 0x7f2bbc0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.203+0000 7f2bc23ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bbc0ae570 0x7f2bbc0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.203+0000 7f2bc23ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0ad230 con 0x7f2bbc0b0520 2023-10-07T11:01:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.203+0000 7f2bc23ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc0ad4d0 con 0x7f2bbc0ae570 2023-10-07T11:01:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.203+0000 7f2bc23ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc0ad7b0 con 0x7f2bbc156180 2023-10-07T11:01:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.203+0000 7f2bbb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bbc0ae570 0x7f2bbc0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bbc0ae570 0x7f2bbc0accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39202/0 (socket says 172.21.15.29:39202) 2023-10-07T11:01:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 -- 172.21.15.29:0/1579975745 learned_addr learned my addr 172.21.15.29:0/1579975745 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbbfff700 1 --2- 172.21.15.29:0/1579975745 >> v2:172.21.15.144:3300/0 conn(0x7f2bbc156180 0x7f2bbc0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 -- 172.21.15.29:0/1579975745 >> v2:172.21.15.144:3300/0 conn(0x7f2bbc156180 msgr2=0x7f2bbc0afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 --2- 172.21.15.29:0/1579975745 >> v2:172.21.15.144:3300/0 conn(0x7f2bbc156180 0x7f2bbc0afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 -- 172.21.15.29:0/1579975745 >> v1:172.21.15.144:6789/0 conn(0x7f2bbc0b0520 legacy=0x7f2bbc0b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 -- 172.21.15.29:0/1579975745 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0ada90 con 0x7f2bbc0ae570 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbbfff700 1 --2- 172.21.15.29:0/1579975745 >> v2:172.21.15.144:3300/0 conn(0x7f2bbc156180 0x7f2bbc0afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbb7fe700 1 --2- 172.21.15.29:0/1579975745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc0ae570 0x7f2bbc0accf0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0b2250 tx=0x7f2bb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30434778613a992 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.204+0000 7f2bbaffd700 1 -- 172.21.15.29:0/1579975745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb000b820 con 0x7f2bbc0ae570 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bbaffd700 1 -- 172.21.15.29:0/1579975745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb000b980 con 0x7f2bbc0ae570 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bbaffd700 1 -- 172.21.15.29:0/1579975745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb000bd20 con 0x7f2bbc0ae570 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 -- 172.21.15.29:0/1579975745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc0ae570 msgr2=0x7f2bbc0accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/1579975745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc0ae570 0x7f2bbc0accf0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0b2250 tx=0x7f2bb0009250 comp rx=0 tx=0).stop 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 -- 172.21.15.29:0/1579975745 shutdown_connections 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/1579975745 >> v2:172.21.15.144:3300/0 conn(0x7f2bbc156180 0x7f2bbc0afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/1579975745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc0ae570 0x7f2bbc0accf0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 -- 172.21.15.29:0/1579975745 >> 172.21.15.29:0/1579975745 conn(0x7f2bbc067610 msgr2=0x7f2bbc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 -- 172.21.15.29:0/1579975745 shutdown_connections 2023-10-07T11:01:31.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.205+0000 7f2bc23ff700 1 -- 172.21.15.29:0/1579975745 wait complete. 2023-10-07T11:01:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.207+0000 7f2bc23ff700 1 Processor -- start 2023-10-07T11:01:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.207+0000 7f2bc23ff700 1 -- start start 2023-10-07T11:01:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc23ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bbc0ae570 0x7f2bbc147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc23ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bbc0b0520 0x7f2bbc158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc23ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 0x7f2bbc158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc23ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc154430 con 0x7f2bbc156180 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc23ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc148180 con 0x7f2bbc0b0520 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc23ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc153820 con 0x7f2bbc0ae570 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc099c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 0x7f2bbc158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bbbfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bbc0ae570 0x7f2bbc147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc099c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 0x7f2bbc158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39218/0 (socket says 172.21.15.29:39218) 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bbb7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bbc0b0520 0x7f2bbc158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.208+0000 7f2bc099c700 1 -- 172.21.15.29:0/2850635782 learned_addr learned my addr 172.21.15.29:0/2850635782 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc099c700 1 -- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bbc0ae570 msgr2=0x7f2bbc147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc099c700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bbc0ae570 0x7f2bbc147c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc099c700 1 -- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bbc0b0520 msgr2=0x7f2bbc158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc099c700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bbc0b0520 0x7f2bbc158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc099c700 1 -- 172.21.15.29:0/2850635782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb00094f0 con 0x7f2bbc156180 2023-10-07T11:01:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc099c700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 0x7f2bbc158c90 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400c8c0 tx=0x7f2bb40026a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4012630 con 0x7f2bbc156180 2023-10-07T11:01:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb40127b0 con 0x7f2bbc156180 2023-10-07T11:01:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4012b70 con 0x7f2bbc156180 2023-10-07T11:01:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc153b00 con 0x7f2bbc156180 2023-10-07T11:01:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.209+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc1540c0 con 0x7f2bbc156180 2023-10-07T11:01:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.210+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc1545e0 con 0x7f2bbc156180 2023-10-07T11:01:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.211+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2bb401b400 con 0x7f2bbc156180 2023-10-07T11:01:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.212+0000 7f2ba3fff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f2ba8066f30 0x7f2ba80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.212+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f2bb4084a80 con 0x7f2bbc156180 2023-10-07T11:01:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.213+0000 7f2bbbfff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f2ba8066f30 0x7f2ba80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.213+0000 7f2bbbfff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f2ba8066f30 0x7f2ba80693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bac006fd0 tx=0x7f2bac008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.216+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2bbc1545e0 con 0x7f2bbc156180 2023-10-07T11:01:31.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.418+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2bbc1545e0 con 0x7f2bbc156180 2023-10-07T11:01:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.422+0000 7f2ba3fff700 1 -- 172.21.15.29:0/2850635782 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2bbc1545e0 con 0x7f2bbc156180 2023-10-07T11:01:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f2ba8066f30 msgr2=0x7f2ba80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f2ba8066f30 0x7f2ba80693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bac006fd0 tx=0x7f2bac008040 comp rx=0 tx=0).stop 2023-10-07T11:01:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 msgr2=0x7f2bbc158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 0x7f2bbc158c90 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400c8c0 tx=0x7f2bb40026a0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 shutdown_connections 2023-10-07T11:01:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bbc156180 0x7f2bbc158c90 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f2ba8066f30 0x7f2ba80693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2bbc0ae570 0x7f2bbc147c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 --2- 172.21.15.29:0/2850635782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2bbc0b0520 0x7f2bbc158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 >> 172.21.15.29:0/2850635782 conn(0x7f2bbc067610 msgr2=0x7f2bbc152c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 shutdown_connections 2023-10-07T11:01:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.425+0000 7f2bc23ff700 1 -- 172.21.15.29:0/2850635782 wait complete. 2023-10-07T11:01:31.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-07T11:01:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.639+0000 7f65dbfff700 1 Processor -- start 2023-10-07T11:01:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.639+0000 7f65dbfff700 1 -- start start 2023-10-07T11:01:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.640+0000 7f65dbfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65d4159b20 0x7f65d40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.640+0000 7f65dbfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d40cbf20 con 0x7f65d4159750 2023-10-07T11:01:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.640+0000 7f65dbfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d40cc200 con 0x7f65d40ce8a0 2023-10-07T11:01:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.640+0000 7f65dbfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40cc4e0 con 0x7f65d4159b20 2023-10-07T11:01:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d9d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65d4159750 0x7f65d40ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56594/0 (socket says 172.21.15.29:56594) 2023-10-07T11:01:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d9d9b700 1 -- 172.21.15.29:0/1825657999 learned_addr learned my addr 172.21.15.29:0/1825657999 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d959a700 1 --2- 172.21.15.29:0/1825657999 >> v2:172.21.15.144:3300/0 conn(0x7f65d4159b20 0x7f65d40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148173650 0 0) 0x7f65d40cbf20 con 0x7f65d4159750 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c0003570 con 0x7f65d4159750 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3398302788 0 0) 0x7f65c0003570 con 0x7f65d4159750 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 >> v2:172.21.15.144:3300/0 conn(0x7f65d4159b20 msgr2=0x7f65d40d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 --2- 172.21.15.29:0/1825657999 >> v2:172.21.15.144:3300/0 conn(0x7f65d4159b20 0x7f65d40d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 >> v1:172.21.15.144:6789/0 conn(0x7f65d40ce8a0 legacy=0x7f65d40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40cc7c0 con 0x7f65d4159750 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f65d00015f0 con 0x7f65d4159750 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 >> v1:172.21.15.29:6789/0 conn(0x7f65d4159750 legacy=0x7f65d40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c0003ab0 0x7f65c0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65c0004810 0x7f65c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0005210 0x7f65c000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40cc4e0 con 0x7f65c0003ab0 2023-10-07T11:01:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c000c130 con 0x7f65c0004810 2023-10-07T11:01:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.641+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c000c2f0 con 0x7f65c0005210 2023-10-07T11:01:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c0003ab0 0x7f65c0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d959a700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65c0004810 0x7f65c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65da59c700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0005210 0x7f65c000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 -- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0005210 msgr2=0x7f65c000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0005210 0x7f65c000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 -- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65c0004810 msgr2=0x7f65c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65c0004810 0x7f65c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 -- 172.21.15.29:0/1825657999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c0003570 con 0x7f65c0003ab0 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65da59c700 1 -- 172.21.15.29:0/1825657999 reap_dead start 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.642+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c0003ab0 0x7f65c0003ec0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f65d000c210 tx=0x7f65d0007810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8341c00096775d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000cb40 con 0x7f65c0003ab0 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000ccc0 con 0x7f65c0003ab0 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65d8d99700 1 -- 172.21.15.29:0/1825657999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0007320 con 0x7f65c0003ab0 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 -- 172.21.15.29:0/1825657999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c0003ab0 msgr2=0x7f65c0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c0003ab0 0x7f65c0003ec0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f65d000c210 tx=0x7f65d0007810 comp rx=0 tx=0).stop 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 -- 172.21.15.29:0/1825657999 shutdown_connections 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1825657999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c0003ab0 0x7f65c0003ec0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 -- 172.21.15.29:0/1825657999 >> 172.21.15.29:0/1825657999 conn(0x7f65d4063a10 msgr2=0x7f65d40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 -- 172.21.15.29:0/1825657999 shutdown_connections 2023-10-07T11:01:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.643+0000 7f65dbfff700 1 -- 172.21.15.29:0/1825657999 wait complete. 2023-10-07T11:01:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.645+0000 7f65dbfff700 1 Processor -- start 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.646+0000 7f65dbfff700 1 -- start start 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0003ab0 0x7f65d415c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d40b4490 0x7f65d41440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 0x7f65d4158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65dbfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d415cea0 con 0x7f65d40b5b30 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65dbfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d41445e0 con 0x7f65d40b4490 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65dbfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40b4e00 con 0x7f65c0003ab0 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65da59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 0x7f65d4158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65da59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 0x7f65d4158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39224/0 (socket says 172.21.15.29:39224) 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65da59c700 1 -- 172.21.15.29:0/1710963202 learned_addr learned my addr 172.21.15.29:0/1710963202 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0003ab0 0x7f65d415c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.647+0000 7f65d959a700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d40b4490 0x7f65d41440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65da59c700 1 -- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0003ab0 msgr2=0x7f65d415c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65da59c700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0003ab0 0x7f65d415c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65da59c700 1 -- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d40b4490 msgr2=0x7f65d41440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65da59c700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d40b4490 0x7f65d41440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65da59c700 1 -- 172.21.15.29:0/1710963202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40b56a0 con 0x7f65d40b5b30 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65da59c700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 0x7f65d4158050 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f65d415d3c0 tx=0x7f65cc001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00eec0 con 0x7f65d40b5b30 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc006db0 con 0x7f65d40b5b30 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d40b4950 con 0x7f65d40b5b30 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00c7d0 con 0x7f65d40b5b30 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.648+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d40b5330 con 0x7f65d40b5b30 2023-10-07T11:01:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.649+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d40b5830 con 0x7f65d40b5b30 2023-10-07T11:01:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.650+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f65cc00cb80 con 0x7f65d40b5b30 2023-10-07T11:01:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.651+0000 7f65c9ffb700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f65b8066f30 0x7f65b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.651+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f65cc084510 con 0x7f65d40b5b30 2023-10-07T11:01:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.652+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f65b8066f30 0x7f65b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.652+0000 7f65d9d9b700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f65b8066f30 0x7f65b80693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f65d0000ef0 tx=0x7f65d0015000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:31.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.655+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65cc0514c0 con 0x7f65d40b5b30 2023-10-07T11:01:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:31.939+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f65d404c3c0 con 0x7f65d40b5b30 2023-10-07T11:01:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.833+0000 7f65c9ffb700 1 -- 172.21.15.29:0/1710963202 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2968) v1 ==== 165+0+0 (secure 0 0 0) 0x7f65cc054ae0 con 0x7f65d40b5b30 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.836+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f65b8066f30 msgr2=0x7f65b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.836+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f65b8066f30 0x7f65b80693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f65d0000ef0 tx=0x7f65d0015000 comp rx=0 tx=0).stop 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 msgr2=0x7f65d4158050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 0x7f65d4158050 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f65d415d3c0 tx=0x7f65cc001670 comp rx=0 tx=0).stop 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 shutdown_connections 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65d40b5b30 0x7f65d4158050 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f65b8066f30 0x7f65b80693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65c0003ab0 0x7f65d415c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 --2- 172.21.15.29:0/1710963202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d40b4490 0x7f65d41440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 >> 172.21.15.29:0/1710963202 conn(0x7f65d4063a10 msgr2=0x7f65d40c66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 shutdown_connections 2023-10-07T11:01:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:32.837+0000 7f65dbfff700 1 -- 172.21.15.29:0/1710963202 wait complete. 2023-10-07T11:01:32.858 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-07T11:01:32.858 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-07T11:01:32.859 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-07T11:01:32.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-07T11:01:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.059+0000 7f5cb0bff700 1 Processor -- start 2023-10-07T11:01:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.059+0000 7f5cb0bff700 1 -- start start 2023-10-07T11:01:33.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5cb0bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5cac153cc0 0x7f5cac0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:33.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5cb0bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5cac0b0710 0x7f5cac0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:33.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5cb0bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac154090 con 0x7f5cac15ade0 2023-10-07T11:01:33.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5cb0bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac159f60 con 0x7f5cac0b0710 2023-10-07T11:01:33.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5cb0bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac15a240 con 0x7f5cac153cc0 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5caad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5cac0b0710 0x7f5cac0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5caad9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5cac0b0710 0x7f5cac0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49552/0 (socket says 172.21.15.29:49552) 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5caad9d700 1 -- 172.21.15.29:0/639295147 learned_addr learned my addr 172.21.15.29:0/639295147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.061+0000 7f5caa59c700 1 --2- 172.21.15.29:0/639295147 >> v2:172.21.15.144:3300/0 conn(0x7f5cac153cc0 0x7f5cac0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5caad9d700 1 -- 172.21.15.29:0/639295147 >> v2:172.21.15.144:3300/0 conn(0x7f5cac153cc0 msgr2=0x7f5cac0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5caad9d700 1 --2- 172.21.15.29:0/639295147 >> v2:172.21.15.144:3300/0 conn(0x7f5cac153cc0 0x7f5cac0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5caad9d700 1 -- 172.21.15.29:0/639295147 >> v1:172.21.15.144:6789/0 conn(0x7f5cac15ade0 legacy=0x7f5cac0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5caad9d700 1 -- 172.21.15.29:0/639295147 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac15a520 con 0x7f5cac0b0710 2023-10-07T11:01:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5caad9d700 1 --2- 172.21.15.29:0/639295147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac0b0710 0x7f5cac0b1d00 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0cf440 tx=0x7f5ca4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77e023b05939f76 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5ca959a700 1 -- 172.21.15.29:0/639295147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400e070 con 0x7f5cac0b0710 2023-10-07T11:01:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5ca959a700 1 -- 172.21.15.29:0/639295147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca400ba20 con 0x7f5cac0b0710 2023-10-07T11:01:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.062+0000 7f5ca959a700 1 -- 172.21.15.29:0/639295147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca40092e0 con 0x7f5cac0b0710 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 -- 172.21.15.29:0/639295147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac0b0710 msgr2=0x7f5cac0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/639295147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac0b0710 0x7f5cac0b1d00 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0cf440 tx=0x7f5ca4009840 comp rx=0 tx=0).stop 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 -- 172.21.15.29:0/639295147 shutdown_connections 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/639295147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac0b0710 0x7f5cac0b1d00 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/639295147 >> v2:172.21.15.144:3300/0 conn(0x7f5cac153cc0 0x7f5cac0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 -- 172.21.15.29:0/639295147 >> 172.21.15.29:0/639295147 conn(0x7f5cac067610 msgr2=0x7f5cac067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 -- 172.21.15.29:0/639295147 shutdown_connections 2023-10-07T11:01:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.063+0000 7f5cb0bff700 1 -- 172.21.15.29:0/639295147 wait complete. 2023-10-07T11:01:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.064+0000 7f5cb0bff700 1 Processor -- start 2023-10-07T11:01:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 -- start start 2023-10-07T11:01:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac0b0710 0x7f5cac0c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac153cc0 0x7f5cac15e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 0x7f5cac15e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac151740 con 0x7f5cac15ade0 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac154bf0 con 0x7f5cac153cc0 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.065+0000 7f5cb0bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac16a4d0 con 0x7f5cac0b0710 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 0x7f5cac15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 0x7f5cac15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49560/0 (socket says 172.21.15.29:49560) 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 -- 172.21.15.29:0/461703808 learned_addr learned my addr 172.21.15.29:0/461703808 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caa59c700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac0b0710 0x7f5cac0c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5ca9d9b700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac153cc0 0x7f5cac15e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 -- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac0b0710 msgr2=0x7f5cac0c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac0b0710 0x7f5cac0c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 -- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac153cc0 msgr2=0x7f5cac15e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac153cc0 0x7f5cac15e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 -- 172.21.15.29:0/461703808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4009ae0 con 0x7f5cac15ade0 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5ca9d9b700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac153cc0 0x7f5cac15e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.066+0000 7f5caad9d700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 0x7f5cac15e750 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009880 tx=0x7f5ca4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.067+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4008be0 con 0x7f5cac15ade0 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.067+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca4008d40 con 0x7f5cac15ade0 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.067+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac1545a0 con 0x7f5cac15ade0 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.067+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca40266b0 con 0x7f5cac15ade0 2023-10-07T11:01:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.067+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac0c7fe0 con 0x7f5cac15ade0 2023-10-07T11:01:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.068+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac0b3e00 con 0x7f5cac15ade0 2023-10-07T11:01:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.069+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5ca4003040 con 0x7f5cac15ade0 2023-10-07T11:01:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.069+0000 7f5c9a7fc700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f5c8c066f30 0x7f5c8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.069+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f5ca40866d0 con 0x7f5cac15ade0 2023-10-07T11:01:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.070+0000 7f5caa59c700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f5c8c066f30 0x7f5c8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.071+0000 7f5caa59c700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f5c8c066f30 0x7f5c8c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5cac15f240 tx=0x7f5c9c008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.074+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ca40537d0 con 0x7f5cac15ade0 2023-10-07T11:01:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.357+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f5cac0b42b0 con 0x7f5cac15ade0 2023-10-07T11:01:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.838+0000 7f5c9a7fc700 1 -- 172.21.15.29:0/461703808 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v2969) v1 ==== 166+0+0 (secure 0 0 0) 0x7f5ca4056df0 con 0x7f5cac15ade0 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f5c8c066f30 msgr2=0x7f5c8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f5c8c066f30 0x7f5c8c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5cac15f240 tx=0x7f5c9c008040 comp rx=0 tx=0).stop 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 msgr2=0x7f5cac15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 0x7f5cac15e750 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009880 tx=0x7f5ca4018000 comp rx=0 tx=0).stop 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 shutdown_connections 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cac15ade0 0x7f5cac15e750 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f5c8c066f30 0x7f5c8c0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac0b0710 0x7f5cac0c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 --2- 172.21.15.29:0/461703808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac153cc0 0x7f5cac15e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:33.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 >> 172.21.15.29:0/461703808 conn(0x7f5cac067610 msgr2=0x7f5cac0b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:33.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 shutdown_connections 2023-10-07T11:01:33.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:33.841+0000 7f5cb0bff700 1 -- 172.21.15.29:0/461703808 wait complete. 2023-10-07T11:01:33.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T11:01:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.062+0000 7f98dc1ff700 1 Processor -- start 2023-10-07T11:01:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.062+0000 7f98dc1ff700 1 -- start start 2023-10-07T11:01:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98dc1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f98d414c8d0 0x7f98d40afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98dc1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98d4153cc0 0x7f98d4159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98dc1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d40b0410 0x7f98d4159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98dc1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d40b0950 con 0x7f98d4153cc0 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98dc1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d4157580 con 0x7f98d414c8d0 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98dc1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d4157790 con 0x7f98d40b0410 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98d979a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98d4153cc0 0x7f98d4159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98d979a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98d4153cc0 0x7f98d4159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49576/0 (socket says 172.21.15.29:49576) 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98d979a700 1 -- 172.21.15.29:0/2801260895 learned_addr learned my addr 172.21.15.29:0/2801260895 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98d9f9b700 1 --2- 172.21.15.29:0/2801260895 >> v2:172.21.15.144:3300/0 conn(0x7f98d414c8d0 0x7f98d40afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.064+0000 7f98da79c700 1 --2- 172.21.15.29:0/2801260895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d40b0410 0x7f98d4159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d979a700 1 -- 172.21.15.29:0/2801260895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d40b0410 msgr2=0x7f98d4159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d979a700 1 --2- 172.21.15.29:0/2801260895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d40b0410 0x7f98d4159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d979a700 1 -- 172.21.15.29:0/2801260895 >> v2:172.21.15.144:3300/0 conn(0x7f98d414c8d0 msgr2=0x7f98d40afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d979a700 1 --2- 172.21.15.29:0/2801260895 >> v2:172.21.15.144:3300/0 conn(0x7f98d414c8d0 0x7f98d40afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d979a700 1 -- 172.21.15.29:0/2801260895 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4157a70 con 0x7f98d4153cc0 2023-10-07T11:01:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d979a700 1 --2- 172.21.15.29:0/2801260895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d4159680 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f98d406d0b0 tx=0x7f98c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d304d1f176d6ad40 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d8f99700 1 -- 172.21.15.29:0/2801260895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c400b640 con 0x7f98d4153cc0 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d8f99700 1 -- 172.21.15.29:0/2801260895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c400b7a0 con 0x7f98d4153cc0 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.065+0000 7f98d8f99700 1 -- 172.21.15.29:0/2801260895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c400bb60 con 0x7f98d4153cc0 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 -- 172.21.15.29:0/2801260895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 msgr2=0x7f98d4159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/2801260895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d4159680 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f98d406d0b0 tx=0x7f98c4009250 comp rx=0 tx=0).stop 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 -- 172.21.15.29:0/2801260895 shutdown_connections 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/2801260895 >> v2:172.21.15.144:3300/0 conn(0x7f98d414c8d0 0x7f98d40afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/2801260895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d40b0410 0x7f98d4159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/2801260895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d4159680 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 -- 172.21.15.29:0/2801260895 >> 172.21.15.29:0/2801260895 conn(0x7f98d4067610 msgr2=0x7f98d4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 -- 172.21.15.29:0/2801260895 shutdown_connections 2023-10-07T11:01:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.066+0000 7f98dc1ff700 1 -- 172.21.15.29:0/2801260895 wait complete. 2023-10-07T11:01:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.067+0000 7f98dc1ff700 1 Processor -- start 2023-10-07T11:01:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.067+0000 7f98dc1ff700 1 -- start start 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.068+0000 7f98dc1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d40b0410 0x7f98d414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.068+0000 7f98dc1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d414c8d0 0x7f98d40c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.068+0000 7f98dc1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d40c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.068+0000 7f98dc1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40c1710 con 0x7f98d4153cc0 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.068+0000 7f98dc1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40ccb70 con 0x7f98d40b0410 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.068+0000 7f98dc1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40c9520 con 0x7f98d414c8d0 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d40c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d40c1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49592/0 (socket says 172.21.15.29:49592) 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 -- 172.21.15.29:0/3929003629 learned_addr learned my addr 172.21.15.29:0/3929003629 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98d979a700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d414c8d0 0x7f98d40c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98d9f9b700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d40b0410 0x7f98d414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 -- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d414c8d0 msgr2=0x7f98d40c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d414c8d0 0x7f98d40c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 -- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d40b0410 msgr2=0x7f98d414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d40b0410 0x7f98d414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 -- 172.21.15.29:0/3929003629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40094f0 con 0x7f98d4153cc0 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98da79c700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d40c1b20 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009df0 tx=0x7f98cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.069+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00caa0 con 0x7f98d4153cc0 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.070+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc00f040 con 0x7f98d4153cc0 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.070+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d40c97a0 con 0x7f98d4153cc0 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.070+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00ce70 con 0x7f98d4153cc0 2023-10-07T11:01:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.070+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d40c1430 con 0x7f98d4153cc0 2023-10-07T11:01:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.071+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d40c9930 con 0x7f98d4153cc0 2023-10-07T11:01:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.072+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f98cc014690 con 0x7f98d4153cc0 2023-10-07T11:01:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.072+0000 7f98c9ffb700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f98b8066f60 0x7f98b8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.072+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f98cc084f20 con 0x7f98d4153cc0 2023-10-07T11:01:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.073+0000 7f98d9f9b700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f98b8066f60 0x7f98b8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.074+0000 7f98d9f9b700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f98b8066f60 0x7f98b8069410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009db0 tx=0x7f98d00091a0 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.077+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98cc055a00 con 0x7f98d4153cc0 2023-10-07T11:01:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.280+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f98d404c3c0 con 0x7f98d4153cc0 2023-10-07T11:01:34.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.285+0000 7f98c9ffb700 1 -- 172.21.15.29:0/3929003629 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f98cc055590 con 0x7f98d4153cc0 2023-10-07T11:01:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.287+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f98b8066f60 msgr2=0x7f98b8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.287+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f98b8066f60 0x7f98b8069410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009db0 tx=0x7f98d00091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.287+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 msgr2=0x7f98d40c1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.287+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d40c1b20 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009df0 tx=0x7f98cc00b040 comp rx=0 tx=0).stop 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.287+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 shutdown_connections 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.287+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d4153cc0 0x7f98d40c1b20 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.288+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98d40b0410 0x7f98d414a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.288+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f98b8066f60 0x7f98b8069410 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.288+0000 7f98dc1ff700 1 --2- 172.21.15.29:0/3929003629 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98d414c8d0 0x7f98d40c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.288+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 >> 172.21.15.29:0/3929003629 conn(0x7f98d4067610 msgr2=0x7f98d406dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.288+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 shutdown_connections 2023-10-07T11:01:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.288+0000 7f98dc1ff700 1 -- 172.21.15.29:0/3929003629 wait complete. 2023-10-07T11:01:34.306 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T11:01:34.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:01:34.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:01:34.515 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T11:01:34.515 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T11:01:34.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:34.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.710+0000 7ffa665ff700 1 Processor -- start 2023-10-07T11:01:34.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.710+0000 7ffa665ff700 1 -- start start 2023-10-07T11:01:34.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa665ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa60159750 0x7ffa600d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa665ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa600ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa665ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa600cbf20 con 0x7ffa600ce6d0 2023-10-07T11:01:34.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa665ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa600cc200 con 0x7ffa60159750 2023-10-07T11:01:34.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa665ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600cc4e0 con 0x7ffa60159b20 2023-10-07T11:01:34.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa5ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa60159750 0x7ffa600d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa5ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa60159750 0x7ffa600d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49596/0 (socket says 172.21.15.29:49596) 2023-10-07T11:01:34.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.711+0000 7ffa5ffff700 1 -- 172.21.15.29:0/946613197 learned_addr learned my addr 172.21.15.29:0/946613197 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:34.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5f7fe700 1 --2- 172.21.15.29:0/946613197 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa600ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5ffff700 1 -- 172.21.15.29:0/946613197 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 msgr2=0x7ffa600ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:34.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5ffff700 1 --2- 172.21.15.29:0/946613197 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa600ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5ffff700 1 -- 172.21.15.29:0/946613197 >> v1:172.21.15.144:6789/0 conn(0x7ffa600ce6d0 legacy=0x7ffa6015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5ffff700 1 -- 172.21.15.29:0/946613197 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa600cc7c0 con 0x7ffa60159750 2023-10-07T11:01:34.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5ffff700 1 --2- 172.21.15.29:0/946613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60159750 0x7ffa600d10a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009990 tx=0x7ffa50009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b7a48d184b122d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:34.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5effd700 1 -- 172.21.15.29:0/946613197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50005c10 con 0x7ffa60159750 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5effd700 1 -- 172.21.15.29:0/946613197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa50005d70 con 0x7ffa60159750 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.712+0000 7ffa5effd700 1 -- 172.21.15.29:0/946613197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5000b880 con 0x7ffa60159750 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 -- 172.21.15.29:0/946613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60159750 msgr2=0x7ffa600d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 --2- 172.21.15.29:0/946613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60159750 0x7ffa600d10a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009990 tx=0x7ffa50009220 comp rx=0 tx=0).stop 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 -- 172.21.15.29:0/946613197 shutdown_connections 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 --2- 172.21.15.29:0/946613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60159750 0x7ffa600d10a0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 --2- 172.21.15.29:0/946613197 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa600ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 -- 172.21.15.29:0/946613197 >> 172.21.15.29:0/946613197 conn(0x7ffa60063a10 msgr2=0x7ffa600d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 -- 172.21.15.29:0/946613197 shutdown_connections 2023-10-07T11:01:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.713+0000 7ffa665ff700 1 -- 172.21.15.29:0/946613197 wait complete. 2023-10-07T11:01:34.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.714+0000 7ffa665ff700 1 Processor -- start 2023-10-07T11:01:34.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.715+0000 7ffa665ff700 1 -- start start 2023-10-07T11:01:34.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa665ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 0x7ffa60144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa665ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa60159750 0x7ffa60153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa665ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa60158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa665ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c8620 con 0x7ffa600ce6d0 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa665ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa60154b10 con 0x7ffa60159750 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa665ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c7a10 con 0x7ffa60159b20 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 0x7ffa60144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 0x7ffa60144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49600/0 (socket says 172.21.15.29:49600) 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 -- 172.21.15.29:0/1960239321 learned_addr learned my addr 172.21.15.29:0/1960239321 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:34.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa64b9c700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa60158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5f7fe700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa60159750 0x7ffa60153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 -- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 msgr2=0x7ffa60158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa60158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 -- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa60159750 msgr2=0x7ffa60153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa60159750 0x7ffa60153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5ffff700 1 -- 172.21.15.29:0/1960239321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa50009480 con 0x7ffa600ce6d0 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa64b9c700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa60158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:34.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.716+0000 7ffa5f7fe700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa60159750 0x7ffa60153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:34.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.717+0000 7ffa5ffff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 0x7ffa60144f90 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7ffa50005d40 tx=0x7ffa50016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:34.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.717+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50011a60 con 0x7ffa600ce6d0 2023-10-07T11:01:34.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.717+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa50011bc0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.717+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5001b430 con 0x7ffa600ce6d0 2023-10-07T11:01:34.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.717+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa600c7cf0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.717+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa600c82b0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.718+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6004c3c0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.719+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffa5001b860 con 0x7ffa600ce6d0 2023-10-07T11:01:34.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.720+0000 7ffa47fff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ffa4c066f30 0x7ffa4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:34.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.720+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7ffa50086200 con 0x7ffa600ce6d0 2023-10-07T11:01:34.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.720+0000 7ffa5f7fe700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ffa4c066f30 0x7ffa4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:34.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.721+0000 7ffa5f7fe700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ffa4c066f30 0x7ffa4c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa600c6060 tx=0x7ffa54008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:34.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.724+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa500531b0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.973+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa6015e2f0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.973+0000 7ffa47fff700 1 -- 172.21.15.29:0/1960239321 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3532 (secure 0 0 0) 0x7ffa5002ade0 con 0x7ffa600ce6d0 2023-10-07T11:01:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:34.975 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3298444330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3298444330},{"type":"v1","addr":"172.21.15.144:6837","nonce":3298444330}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":7808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/869229207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":869229207},{"type":"v1","addr":"172.21.15.29:6835","nonce":869229207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":8644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/1403240375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1403240375},{"type":"v1","addr":"172.21.15.144:6835","nonce":1403240375}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80},{"gid":9519,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2227539914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2227539914},{"type":"v1","addr":"172.21.15.29:6833","nonce":2227539914}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[]} 2023-10-07T11:01:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ffa4c066f30 msgr2=0x7ffa4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ffa4c066f30 0x7ffa4c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa600c6060 tx=0x7ffa54008040 comp rx=0 tx=0).stop 2023-10-07T11:01:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 msgr2=0x7ffa60144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 0x7ffa60144f90 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7ffa50005d40 tx=0x7ffa50016000 comp rx=0 tx=0).stop 2023-10-07T11:01:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 shutdown_connections 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ffa4c066f30 0x7ffa4c0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa60159b20 0x7ffa60158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa600ce6d0 0x7ffa60144f90 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 --2- 172.21.15.29:0/1960239321 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa60159750 0x7ffa60153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 >> 172.21.15.29:0/1960239321 conn(0x7ffa60063a10 msgr2=0x7ffa601550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 shutdown_connections 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:34.976+0000 7ffa665ff700 1 -- 172.21.15.29:0/1960239321 wait complete. 2023-10-07T11:01:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 87 2023-10-07T11:01:34.994 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T11:01:34.994 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T11:01:34.995 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T11:01:34.995 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:01:34.995 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T11:01:34.995 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T11:01:34.995 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T11:01:34.995 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:01:34.996 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T11:01:34.996 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T11:01:34.996 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T11:01:34.996 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:01:34.996 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T11:01:34.996 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T11:01:34.996 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T11:01:34.997 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:01:35.008 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T11:01:35.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T11:01:35.009 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T11:01:35.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T11:01:35.013 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T11:01:35.013 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T11:01:35.014 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T11:01:35.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T11:01:35.053 INFO:tasks.ceph.mds.a:Started 2023-10-07T11:01:35.053 DEBUG:teuthology.parallel:result is None 2023-10-07T11:01:35.056 INFO:tasks.ceph.mds.c:Started 2023-10-07T11:01:35.056 DEBUG:teuthology.parallel:result is None 2023-10-07T11:01:35.058 INFO:tasks.ceph.mds.b:Started 2023-10-07T11:01:35.058 DEBUG:teuthology.parallel:result is None 2023-10-07T11:01:35.060 INFO:tasks.ceph.mds.d:Started 2023-10-07T11:01:35.061 DEBUG:teuthology.parallel:result is None 2023-10-07T11:01:35.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T11:01:35.201 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T11:01:35.202 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T11:01:35.204 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T11:01:35.205 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T11:01:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.274+0000 7f6746fff700 1 Processor -- start 2023-10-07T11:01:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.274+0000 7f6746fff700 1 -- start start 2023-10-07T11:01:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.275+0000 7f6746fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6740153cc0 0x7f67400b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f6746fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6740154090 con 0x7f67400b0710 2023-10-07T11:01:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f6746fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6740159f60 con 0x7f674015ade0 2023-10-07T11:01:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f6746fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674015a240 con 0x7f6740153cc0 2023-10-07T11:01:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f674559c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f67400b0710 0x7f674015c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56620/0 (socket says 172.21.15.29:56620) 2023-10-07T11:01:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f674559c700 1 -- 172.21.15.29:0/2843922196 learned_addr learned my addr 172.21.15.29:0/2843922196 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f6744d9b700 1 --2- 172.21.15.29:0/2843922196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6740153cc0 0x7f67400b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772943656 0 0) 0x7f6740154090 con 0x7f67400b0710 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6730003570 con 0x7f67400b0710 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630530467 0 0) 0x7f6740159f60 con 0x7f674015ade0 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.276+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6740154090 con 0x7f674015ade0 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f6744d9b700 1 -- 172.21.15.29:0/2843922196 >> v1:172.21.15.144:6789/0 conn(0x7f674015ade0 legacy=0x7f67400b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f6744d9b700 1 -- 172.21.15.29:0/2843922196 >> v1:172.21.15.29:6789/0 conn(0x7f67400b0710 legacy=0x7f674015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f6744d9b700 1 -- 172.21.15.29:0/2843922196 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674015a520 con 0x7f6740153cc0 2023-10-07T11:01:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f6744d9b700 1 --2- 172.21.15.29:0/2843922196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6740153cc0 0x7f67400b17c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6734009900 tx=0x7f6734009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766f4601acff044f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734005c10 con 0x7f6740153cc0 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6734005d70 con 0x7f6740153cc0 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.277+0000 7f673f7fe700 1 -- 172.21.15.29:0/2843922196 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734006260 con 0x7f6740153cc0 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 -- 172.21.15.29:0/2843922196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6740153cc0 msgr2=0x7f67400b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 --2- 172.21.15.29:0/2843922196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6740153cc0 0x7f67400b17c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6734009900 tx=0x7f6734009120 comp rx=0 tx=0).stop 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 -- 172.21.15.29:0/2843922196 shutdown_connections 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 --2- 172.21.15.29:0/2843922196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6740153cc0 0x7f67400b17c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 -- 172.21.15.29:0/2843922196 >> 172.21.15.29:0/2843922196 conn(0x7f6740067610 msgr2=0x7f6740067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 -- 172.21.15.29:0/2843922196 shutdown_connections 2023-10-07T11:01:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.278+0000 7f6746fff700 1 -- 172.21.15.29:0/2843922196 wait complete. 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.279+0000 7f6746fff700 1 Processor -- start 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.279+0000 7f6746fff700 1 -- start start 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.280+0000 7f6746fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 0x7f6740151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.280+0000 7f6746fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6740153cc0 0x7f67400cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.280+0000 7f6746fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f674015ade0 0x7f67401529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.280+0000 7f6746fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67401522e0 con 0x7f67400b0710 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.280+0000 7f6746fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674016a3b0 con 0x7f6740153cc0 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.280+0000 7f6746fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67400ce530 con 0x7f674015ade0 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 0x7f6740151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 0x7f6740151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49626/0 (socket says 172.21.15.29:49626) 2023-10-07T11:01:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 -- 172.21.15.29:0/2493966959 learned_addr learned my addr 172.21.15.29:0/2493966959 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f674559c700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f674015ade0 0x7f67401529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f673ffff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6740153cc0 0x7f67400cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 -- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f674015ade0 msgr2=0x7f67401529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f674015ade0 0x7f67401529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 -- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6740153cc0 msgr2=0x7f67400cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6740153cc0 0x7f67400cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 -- 172.21.15.29:0/2493966959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340093c0 con 0x7f67400b0710 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.281+0000 7f6744d9b700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 0x7f6740151da0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6734009390 tx=0x7f6734016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.282+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673401d800 con 0x7f67400b0710 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.282+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673401d960 con 0x7f67400b0710 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.282+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740169d60 con 0x7f67400b0710 2023-10-07T11:01:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.282+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673401dd20 con 0x7f67400b0710 2023-10-07T11:01:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.282+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67401699d0 con 0x7f67400b0710 2023-10-07T11:01:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.283+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674004c3c0 con 0x7f67400b0710 2023-10-07T11:01:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.284+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f673401b690 con 0x7f67400b0710 2023-10-07T11:01:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.284+0000 7f673cff9700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6724066f30 0x7f67240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:35.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.285+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f67340863b0 con 0x7f67400b0710 2023-10-07T11:01:35.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.285+0000 7f673ffff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6724066f30 0x7f67240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:35.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.286+0000 7f673ffff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6724066f30 0x7f67240693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67400af7a0 tx=0x7f672c007d50 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:35.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.289+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6734053360 con 0x7f67400b0710 2023-10-07T11:01:35.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.506+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f674015a800 con 0x7f67400b0710 2023-10-07T11:01:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.850+0000 7f673cff9700 1 -- 172.21.15.29:0/2493966959 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v176) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6734056980 con 0x7f67400b0710 2023-10-07T11:01:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6724066f30 msgr2=0x7f67240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6724066f30 0x7f67240693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67400af7a0 tx=0x7f672c007d50 comp rx=0 tx=0).stop 2023-10-07T11:01:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 msgr2=0x7f6740151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 0x7f6740151da0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6734009390 tx=0x7f6734016000 comp rx=0 tx=0).stop 2023-10-07T11:01:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 shutdown_connections 2023-10-07T11:01:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6724066f30 0x7f67240693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f674015ade0 0x7f67401529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67400b0710 0x7f6740151da0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 --2- 172.21.15.29:0/2493966959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6740153cc0 0x7f67400cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 >> 172.21.15.29:0/2493966959 conn(0x7f6740067610 msgr2=0x7f67400b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 shutdown_connections 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:35.853+0000 7f6746fff700 1 -- 172.21.15.29:0/2493966959 wait complete. 2023-10-07T11:01:35.855 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T11:01:35.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T11:01:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.070+0000 7fc989fff700 1 Processor -- start 2023-10-07T11:01:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.070+0000 7fc989fff700 1 -- start start 2023-10-07T11:01:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.071+0000 7fc989fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9840ad370 0x7fc9840cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.071+0000 7fc989fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98414d790 con 0x7fc984159ec0 2023-10-07T11:01:36.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.071+0000 7fc989fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9840ae5d0 con 0x7fc9840b0410 2023-10-07T11:01:36.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.071+0000 7fc989fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9840ad740 con 0x7fc9840ad370 2023-10-07T11:01:36.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc982ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc984159ec0 0x7fc9841568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56630/0 (socket says 172.21.15.29:56630) 2023-10-07T11:01:36.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc982ffd700 1 -- 172.21.15.29:0/3924596470 learned_addr learned my addr 172.21.15.29:0/3924596470 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:36.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9837fe700 1 --2- 172.21.15.29:0/3924596470 >> v2:172.21.15.29:3300/0 conn(0x7fc9840ad370 0x7fc9840cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9827fc700 1 -- 172.21.15.29:0/3924596470 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788715583 0 0) 0x7fc98414d790 con 0x7fc984159ec0 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9827fc700 1 -- 172.21.15.29:0/3924596470 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc96c003570 con 0x7fc984159ec0 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9827fc700 1 -- 172.21.15.29:0/3924596470 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169906933 0 0) 0x7fc9840ae5d0 con 0x7fc9840b0410 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9837fe700 1 -- 172.21.15.29:0/3924596470 >> v1:172.21.15.144:6789/0 conn(0x7fc9840b0410 legacy=0x7fc984156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9837fe700 1 -- 172.21.15.29:0/3924596470 >> v1:172.21.15.29:6789/0 conn(0x7fc984159ec0 legacy=0x7fc9841568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.072+0000 7fc9837fe700 1 -- 172.21.15.29:0/3924596470 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9840ccdb0 con 0x7fc9840ad370 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc9837fe700 1 --2- 172.21.15.29:0/3924596470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840ad370 0x7fc9840cc870 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fc974009910 tx=0x7fc9740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b5e70b5dd2c264 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc9827fc700 1 -- 172.21.15.29:0/3924596470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400b640 con 0x7fc9840ad370 2023-10-07T11:01:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc9827fc700 1 -- 172.21.15.29:0/3924596470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97400b7a0 con 0x7fc9840ad370 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc9827fc700 1 -- 172.21.15.29:0/3924596470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400bb20 con 0x7fc9840ad370 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc989fff700 1 -- 172.21.15.29:0/3924596470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840ad370 msgr2=0x7fc9840cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc989fff700 1 --2- 172.21.15.29:0/3924596470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840ad370 0x7fc9840cc870 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fc974009910 tx=0x7fc9740091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc989fff700 1 -- 172.21.15.29:0/3924596470 shutdown_connections 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc989fff700 1 --2- 172.21.15.29:0/3924596470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840ad370 0x7fc9840cc870 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.073+0000 7fc989fff700 1 -- 172.21.15.29:0/3924596470 >> 172.21.15.29:0/3924596470 conn(0x7fc9840680a0 msgr2=0x7fc9840cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.074+0000 7fc989fff700 1 -- 172.21.15.29:0/3924596470 shutdown_connections 2023-10-07T11:01:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.074+0000 7fc989fff700 1 -- 172.21.15.29:0/3924596470 wait complete. 2023-10-07T11:01:36.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.075+0000 7fc989fff700 1 Processor -- start 2023-10-07T11:01:36.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.075+0000 7fc989fff700 1 -- start start 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc989fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9840ad370 0x7fc98414bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc989fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 0x7fc9840c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc989fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc984159ec0 0x7fc98415b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc989fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840ca500 con 0x7fc9840b0410 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc989fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840ca990 con 0x7fc984159ec0 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc989fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840c98f0 con 0x7fc9840ad370 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc982ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 0x7fc9840c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc982ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 0x7fc9840c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49644/0 (socket says 172.21.15.29:49644) 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.076+0000 7fc982ffd700 1 -- 172.21.15.29:0/4054838928 learned_addr learned my addr 172.21.15.29:0/4054838928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc9837fe700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9840ad370 0x7fc98414bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc983fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc984159ec0 0x7fc98415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc982ffd700 1 -- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9840ad370 msgr2=0x7fc98414bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc982ffd700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9840ad370 0x7fc98414bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc982ffd700 1 -- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc984159ec0 msgr2=0x7fc98415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc982ffd700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc984159ec0 0x7fc98415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc982ffd700 1 -- 172.21.15.29:0/4054838928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc974009420 con 0x7fc9840b0410 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc982ffd700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 0x7fc9840c8f80 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fc9780011e0 tx=0x7fc978007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978005b30 con 0x7fc9840b0410 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc978005c90 con 0x7fc9840b0410 2023-10-07T11:01:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9840c9bd0 con 0x7fc9840b0410 2023-10-07T11:01:36.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.077+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978007320 con 0x7fc9840b0410 2023-10-07T11:01:36.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.078+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9840ca190 con 0x7fc9840b0410 2023-10-07T11:01:36.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.079+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98404c3c0 con 0x7fc9840b0410 2023-10-07T11:01:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.080+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc978007690 con 0x7fc9840b0410 2023-10-07T11:01:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.080+0000 7fc96b7fe700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc970066f30 0x7fc9700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.080+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5301+0+0 (secure 0 0 0) 0x7fc978083740 con 0x7fc9840b0410 2023-10-07T11:01:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.081+0000 7fc9837fe700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc970066f30 0x7fc9700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.081+0000 7fc9837fe700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc970066f30 0x7fc9700693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc9740093f0 tx=0x7fc974017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:36.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.085+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc978050aa0 con 0x7fc9840b0410 2023-10-07T11:01:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.365+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc9840b1e90 con 0x7fc9840b0410 2023-10-07T11:01:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.365+0000 7fc96b7fe700 1 -- 172.21.15.29:0/4054838928 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc978017090 con 0x7fc9840b0410 2023-10-07T11:01:36.366 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:01:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.368 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.369 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.370 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.371 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.373 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.375 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T11:01:36.375 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T11:01:36.375 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.375 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.376 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.376 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.376 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.377 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T11:01:36.377 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T11:01:36.377 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.377 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.378 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T11:01:36.380 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T11:01:36.381 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T11:01:36.382 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T11:01:36.384 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T11:01:36.385 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:01:36.386 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.386 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:36.386 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T11:01:36.386 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:01:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc970066f30 msgr2=0x7fc9700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc970066f30 0x7fc9700693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc9740093f0 tx=0x7fc974017000 comp rx=0 tx=0).stop 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 msgr2=0x7fc9840c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 0x7fc9840c8f80 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fc9780011e0 tx=0x7fc978007df0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 shutdown_connections 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc984159ec0 0x7fc98415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fc970066f30 0x7fc9700693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9840ad370 0x7fc98414bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 --2- 172.21.15.29:0/4054838928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9840b0410 0x7fc9840c8f80 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 >> 172.21.15.29:0/4054838928 conn(0x7fc9840680a0 msgr2=0x7fc9840694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 shutdown_connections 2023-10-07T11:01:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.368+0000 7fc989fff700 1 -- 172.21.15.29:0/4054838928 wait complete. 2023-10-07T11:01:36.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:36.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.583+0000 7ff8e4dff700 1 Processor -- start 2023-10-07T11:01:36.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.584+0000 7ff8e4dff700 1 -- start start 2023-10-07T11:01:36.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8e4dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8e0159b20 0x7ff8e00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8e4dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e00ce8a0 0x7ff8e00c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8e4dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e00cbf20 con 0x7ff8e0159750 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8e4dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e00cc200 con 0x7ff8e0159b20 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8e4dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e00cc4e0 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8ddd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8e0159b20 0x7ff8e00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8ddd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8e0159b20 0x7ff8e00d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49658/0 (socket says 172.21.15.29:49658) 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.585+0000 7ff8ddd9b700 1 -- 172.21.15.29:0/111238633 learned_addr learned my addr 172.21.15.29:0/111238633 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8ded9d700 1 --2- 172.21.15.29:0/111238633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e00ce8a0 0x7ff8e00c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8ddd9b700 1 -- 172.21.15.29:0/111238633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e00ce8a0 msgr2=0x7ff8e00c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8ddd9b700 1 --2- 172.21.15.29:0/111238633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e00ce8a0 0x7ff8e00c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8ddd9b700 1 -- 172.21.15.29:0/111238633 >> v1:172.21.15.144:6789/0 conn(0x7ff8e0159750 legacy=0x7ff8e00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8ddd9b700 1 -- 172.21.15.29:0/111238633 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e00cc7c0 con 0x7ff8e0159b20 2023-10-07T11:01:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8ddd9b700 1 --2- 172.21.15.29:0/111238633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e0159b20 0x7ff8e00d10a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40099b0 tx=0x7ff8d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39212152f7a1d8e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8dd59a700 1 -- 172.21.15.29:0/111238633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400b860 con 0x7ff8e0159b20 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8dd59a700 1 -- 172.21.15.29:0/111238633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d400b9c0 con 0x7ff8e0159b20 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.586+0000 7ff8dd59a700 1 -- 172.21.15.29:0/111238633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400bd60 con 0x7ff8e0159b20 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 -- 172.21.15.29:0/111238633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e0159b20 msgr2=0x7ff8e00d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/111238633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e0159b20 0x7ff8e00d10a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40099b0 tx=0x7ff8d4009250 comp rx=0 tx=0).stop 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 -- 172.21.15.29:0/111238633 shutdown_connections 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/111238633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e00ce8a0 0x7ff8e00c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/111238633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e0159b20 0x7ff8e00d10a0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 -- 172.21.15.29:0/111238633 >> 172.21.15.29:0/111238633 conn(0x7ff8e0063a10 msgr2=0x7ff8e00d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 -- 172.21.15.29:0/111238633 shutdown_connections 2023-10-07T11:01:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.587+0000 7ff8e4dff700 1 -- 172.21.15.29:0/111238633 wait complete. 2023-10-07T11:01:36.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.589+0000 7ff8e4dff700 1 Processor -- start 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.589+0000 7ff8e4dff700 1 -- start start 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8e4dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 0x7ff8e00ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8e4dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e0159750 0x7ff8e016a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8e4dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8e0159b20 0x7ff8e016c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8e4dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e016bca0 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8e4dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e016b090 con 0x7ff8e0159b20 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8e4dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e016b370 con 0x7ff8e0159750 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8de59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 0x7ff8e00ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8de59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 0x7ff8e00ca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49670/0 (socket says 172.21.15.29:49670) 2023-10-07T11:01:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8de59c700 1 -- 172.21.15.29:0/704486642 learned_addr learned my addr 172.21.15.29:0/704486642 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8ddd9b700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e0159750 0x7ff8e016a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.590+0000 7ff8ded9d700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8e0159b20 0x7ff8e016c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8de59c700 1 -- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e0159750 msgr2=0x7ff8e016a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8de59c700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e0159750 0x7ff8e016a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8de59c700 1 -- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8e0159b20 msgr2=0x7ff8e016c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8de59c700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8e0159b20 0x7ff8e016c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8de59c700 1 -- 172.21.15.29:0/704486642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d40094f0 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8ded9d700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8e0159b20 0x7ff8e016c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8de59c700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 0x7ff8e00ca300 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c7c0 tx=0x7ff8d0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0011590 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d0011710 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e016b650 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0011ad0 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.591+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e016ac60 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.592+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e016ae50 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.593+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff8d001b3b0 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.594+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff8c0066f30 0x7ff8c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:36.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.595+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5301+0+0 (secure 0 0 0) 0x7ff8d0083940 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.595+0000 7ff8ddd9b700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff8c0066f30 0x7ff8c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:36.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.596+0000 7ff8ddd9b700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff8c0066f30 0x7ff8c00693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40099b0 tx=0x7ff8d4017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.598+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8e016ae50 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.799+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8e016ae50 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.800+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/704486642 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+12928 (secure 0 0 0) 0x7ff8e016ae50 con 0x7ff8e00ce8a0 2023-10-07T11:01:36.801 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:36.801 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":178,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:01:35.858070+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff8c0066f30 msgr2=0x7ff8c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff8c0066f30 0x7ff8c00693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40099b0 tx=0x7ff8d4017000 comp rx=0 tx=0).stop 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 msgr2=0x7ff8e00ca300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 0x7ff8e00ca300 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c7c0 tx=0x7ff8d0007f70 comp rx=0 tx=0).stop 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 shutdown_connections 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8e0159b20 0x7ff8e016c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8e00ce8a0 0x7ff8e00ca300 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff8c0066f30 0x7ff8c00693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 --2- 172.21.15.29:0/704486642 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8e0159750 0x7ff8e016a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 >> 172.21.15.29:0/704486642 conn(0x7ff8e0063a10 msgr2=0x7ff8e00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 shutdown_connections 2023-10-07T11:01:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:36.803+0000 7ff8e4dff700 1 -- 172.21.15.29:0/704486642 wait complete. 2023-10-07T11:01:36.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T11:01:36.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T11:01:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.021+0000 7fb6bb3ff700 1 Processor -- start 2023-10-07T11:01:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.021+0000 7fb6bb3ff700 1 -- start start 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6bb3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb6b40b0c30 0x7fb6b40afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6bb3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b40ad4d0 con 0x7fb6b40ae570 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6bb3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b40ad7b0 con 0x7fb6b4156180 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6bb3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b40ada90 con 0x7fb6b40b0c30 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6b899a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6b40ae570 0x7fb6b40b0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56640/0 (socket says 172.21.15.29:56640) 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6b899a700 1 -- 172.21.15.29:0/1166397865 learned_addr learned my addr 172.21.15.29:0/1166397865 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6b999c700 1 --2- 172.21.15.29:0/1166397865 >> v2:172.21.15.144:3300/0 conn(0x7fb6b40b0c30 0x7fb6b40afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959201709 0 0) 0x7fb6b40ad4d0 con 0x7fb6b40ae570 2023-10-07T11:01:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.023+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a0003570 con 0x7fb6b40ae570 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750100271 0 0) 0x7fb6b40ad7b0 con 0x7fb6b4156180 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b40ad4d0 con 0x7fb6b4156180 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632075321 0 0) 0x7fb6a0003570 con 0x7fb6b40ae570 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 >> v2:172.21.15.144:3300/0 conn(0x7fb6b40b0c30 msgr2=0x7fb6b40afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 --2- 172.21.15.29:0/1166397865 >> v2:172.21.15.144:3300/0 conn(0x7fb6b40b0c30 0x7fb6b40afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 >> v1:172.21.15.144:6789/0 conn(0x7fb6b4156180 legacy=0x7fb6b40accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b40add70 con 0x7fb6b40ae570 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fb6a4001470 con 0x7fb6b40ae570 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 >> v1:172.21.15.29:6789/0 conn(0x7fb6b40ae570 legacy=0x7fb6b40b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6a0004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6a0004810 0x7fb6a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6a0005210 0x7fb6a000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40ada90 con 0x7fb6a0003c40 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000c130 con 0x7fb6a0005210 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000c2c0 con 0x7fb6a0004810 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6b919b700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6a0004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6b899a700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6a0004810 0x7fb6a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.024+0000 7fb6b999c700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6a0005210 0x7fb6a000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b919b700 1 -- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6a0004810 msgr2=0x7fb6a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b919b700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6a0004810 0x7fb6a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b919b700 1 -- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6a0005210 msgr2=0x7fb6a000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b919b700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6a0005210 0x7fb6a000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b919b700 1 -- 172.21.15.29:0/1166397865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0003570 con 0x7fb6a0003c40 2023-10-07T11:01:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b999c700 1 -- 172.21.15.29:0/1166397865 reap_dead start 2023-10-07T11:01:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6b919b700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6a0004030 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0001a80 tx=0x7fb6b0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eda07593ccab6bc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000c590 con 0x7fb6a0003c40 2023-10-07T11:01:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b000c6f0 con 0x7fb6a0003c40 2023-10-07T11:01:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.025+0000 7fb6abfff700 1 -- 172.21.15.29:0/1166397865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000cab0 con 0x7fb6a0003c40 2023-10-07T11:01:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/1166397865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 msgr2=0x7fb6a0004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6a0004030 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0001a80 tx=0x7fb6b0007d30 comp rx=0 tx=0).stop 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/1166397865 shutdown_connections 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/1166397865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6a0004030 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/1166397865 >> 172.21.15.29:0/1166397865 conn(0x7fb6b4067610 msgr2=0x7fb6b4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/1166397865 shutdown_connections 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.026+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/1166397865 wait complete. 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.027+0000 7fb6bb3ff700 1 Processor -- start 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.028+0000 7fb6bb3ff700 1 -- start start 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.028+0000 7fb6bb3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6b414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6bb3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b415d6b0 0x7fb6b4156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6bb3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b415db20 0x7fb6b416d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6bb3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b415b380 con 0x7fb6a0003c40 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6bb3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b416e010 con 0x7fb6b415d6b0 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6bb3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40c7d60 con 0x7fb6b415db20 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6b414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6b414bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49688/0 (socket says 172.21.15.29:49688) 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 -- 172.21.15.29:0/2174218453 learned_addr learned my addr 172.21.15.29:0/2174218453 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b899a700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b415d6b0 0x7fb6b4156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b999c700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b415db20 0x7fb6b416d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 -- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b415db20 msgr2=0x7fb6b416d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b415db20 0x7fb6b416d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 -- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b415d6b0 msgr2=0x7fb6b4156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b415d6b0 0x7fb6b4156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.029+0000 7fb6b919b700 1 -- 172.21.15.29:0/2174218453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b40c77a0 con 0x7fb6a0003c40 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.030+0000 7fb6b919b700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6b414bcf0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000c3c0 tx=0x7fb6b0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.030+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000c590 con 0x7fb6a0003c40 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.030+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b0000c30 con 0x7fb6a0003c40 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.030+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000cab0 con 0x7fb6a0003c40 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.030+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b40c7a80 con 0x7fb6a0003c40 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.030+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b415b870 con 0x7fb6a0003c40 2023-10-07T11:01:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.031+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b415ba30 con 0x7fb6a0003c40 2023-10-07T11:01:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.032+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb6b0019030 con 0x7fb6a0003c40 2023-10-07T11:01:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.033+0000 7fb6a97fa700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb698066f30 0x7fb6980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.033+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5301+0+0 (secure 0 0 0) 0x7fb6b00858b0 con 0x7fb6a0003c40 2023-10-07T11:01:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.033+0000 7fb6b899a700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb698066f30 0x7fb6980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.034+0000 7fb6b899a700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb698066f30 0x7fb6980693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40015a0 tx=0x7fb6a4001d20 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.037+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6b0052cb0 con 0x7fb6a0003c40 2023-10-07T11:01:37.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.243+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6b404c3c0 con 0x7fb6a0003c40 2023-10-07T11:01:37.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.244+0000 7fb6a97fa700 1 -- 172.21.15.29:0/2174218453 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6b00170a0 con 0x7fb6a0003c40 2023-10-07T11:01:37.245 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T11:01:37.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb698066f30 msgr2=0x7fb6980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb698066f30 0x7fb6980693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40015a0 tx=0x7fb6a4001d20 comp rx=0 tx=0).stop 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 msgr2=0x7fb6b414bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6b414bcf0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000c3c0 tx=0x7fb6b0015000 comp rx=0 tx=0).stop 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 shutdown_connections 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb698066f30 0x7fb6980693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb6b415db20 0x7fb6b416d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a0003c40 0x7fb6b414bcf0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 --2- 172.21.15.29:0/2174218453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6b415d6b0 0x7fb6b4156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 >> 172.21.15.29:0/2174218453 conn(0x7fb6b4067610 msgr2=0x7fb6b40aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 shutdown_connections 2023-10-07T11:01:37.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.246+0000 7fb6bb3ff700 1 -- 172.21.15.29:0/2174218453 wait complete. 2023-10-07T11:01:37.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T11:01:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.460+0000 7fbcd87ff700 1 Processor -- start 2023-10-07T11:01:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.460+0000 7fbcd87ff700 1 -- start start 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.461+0000 7fbcd87ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbcd0157290 0x7fbcd00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.461+0000 7fbcd87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 0x7fbcd00b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.461+0000 7fbcd87ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd00b1ca0 con 0x7fbcd00b1840 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.461+0000 7fbcd87ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd0158ad0 con 0x7fbcd0157290 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.461+0000 7fbcd87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd015b140 con 0x7fbcd00ae8b0 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 0x7fbcd00b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 0x7fbcd00b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53418/0 (socket says 172.21.15.29:53418) 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 -- 172.21.15.29:0/4147155979 learned_addr learned my addr 172.21.15.29:0/4147155979 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd659b700 1 --2- 172.21.15.29:0/4147155979 >> v2:172.21.15.144:3300/0 conn(0x7fbcd0157290 0x7fbcd00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 -- 172.21.15.29:0/4147155979 >> v2:172.21.15.144:3300/0 conn(0x7fbcd0157290 msgr2=0x7fbcd00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 --2- 172.21.15.29:0/4147155979 >> v2:172.21.15.144:3300/0 conn(0x7fbcd0157290 0x7fbcd00cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 -- 172.21.15.29:0/4147155979 >> v1:172.21.15.144:6789/0 conn(0x7fbcd00b1840 legacy=0x7fbcd00ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.462+0000 7fbcd5d9a700 1 -- 172.21.15.29:0/4147155979 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd015b420 con 0x7fbcd00ae8b0 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.463+0000 7fbcd5d9a700 1 --2- 172.21.15.29:0/4147155979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 0x7fbcd00b1300 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00099b0 tx=0x7fbcc0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a3329e2c158255 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.463+0000 7fbcd5599700 1 -- 172.21.15.29:0/4147155979 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0009ea0 con 0x7fbcd00ae8b0 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.463+0000 7fbcd5599700 1 -- 172.21.15.29:0/4147155979 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc000b860 con 0x7fbcd00ae8b0 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.463+0000 7fbcd5599700 1 -- 172.21.15.29:0/4147155979 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000bc00 con 0x7fbcd00ae8b0 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.463+0000 7fbcd87ff700 1 -- 172.21.15.29:0/4147155979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 msgr2=0x7fbcd00b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/4147155979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 0x7fbcd00b1300 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00099b0 tx=0x7fbcc0009250 comp rx=0 tx=0).stop 2023-10-07T11:01:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 -- 172.21.15.29:0/4147155979 shutdown_connections 2023-10-07T11:01:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/4147155979 >> v2:172.21.15.144:3300/0 conn(0x7fbcd0157290 0x7fbcd00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/4147155979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd00ae8b0 0x7fbcd00b1300 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 -- 172.21.15.29:0/4147155979 >> 172.21.15.29:0/4147155979 conn(0x7fbcd00680a0 msgr2=0x7fbcd00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 -- 172.21.15.29:0/4147155979 shutdown_connections 2023-10-07T11:01:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.464+0000 7fbcd87ff700 1 -- 172.21.15.29:0/4147155979 wait complete. 2023-10-07T11:01:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.465+0000 7fbcd87ff700 1 Processor -- start 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.465+0000 7fbcd87ff700 1 -- start start 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd87ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 0x7fbcd01536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd87ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcd0157290 0x7fbcd015e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd87ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd0150df0 0x7fbcd00c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd87ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd015ec90 con 0x7fbcd00b1840 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd87ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0153cc0 con 0x7fbcd0157290 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd87ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0153fa0 con 0x7fbcd0150df0 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.466+0000 7fbcd659b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 0x7fbcd01536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 0x7fbcd01536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49696/0 (socket says 172.21.15.29:49696) 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 -- 172.21.15.29:0/797304028 learned_addr learned my addr 172.21.15.29:0/797304028 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd6d9c700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd0150df0 0x7fbcd00c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd5d9a700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcd0157290 0x7fbcd015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 -- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd0150df0 msgr2=0x7fbcd00c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd0150df0 0x7fbcd00c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 -- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcd0157290 msgr2=0x7fbcd015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcd0157290 0x7fbcd015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 -- 172.21.15.29:0/797304028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00094f0 con 0x7fbcd00b1840 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.467+0000 7fbcd659b700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 0x7fbcd01536e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e930 tx=0x7fbccc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.468+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc013680 con 0x7fbcd00b1840 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.468+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc0137e0 con 0x7fbcd00b1840 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.468+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd0154280 con 0x7fbcd00b1840 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.468+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc013b80 con 0x7fbcd00b1840 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.468+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd0151290 con 0x7fbcd00b1840 2023-10-07T11:01:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.469+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd01514e0 con 0x7fbcd00b1840 2023-10-07T11:01:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.470+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbccc00c3e0 con 0x7fbcd00b1840 2023-10-07T11:01:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.470+0000 7fbcc67fc700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fbcb4066f30 0x7fbcb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.470+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5301+0+0 (secure 0 0 0) 0x7fbccc0844f0 con 0x7fbcd00b1840 2023-10-07T11:01:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.471+0000 7fbcd5d9a700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fbcb4066f30 0x7fbcb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.471+0000 7fbcd5d9a700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fbcb4066f30 0x7fbcb40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00099b0 tx=0x7fbcc0017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.474+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbccc051850 con 0x7fbcd00b1840 2023-10-07T11:01:37.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.678+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbcd004c3c0 con 0x7fbcd00b1840 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.679+0000 7fbcc67fc700 1 -- 172.21.15.29:0/797304028 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v178) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbccc054e70 con 0x7fbcd00b1840 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:37.680 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T11:01:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fbcb4066f30 msgr2=0x7fbcb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fbcb4066f30 0x7fbcb40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00099b0 tx=0x7fbcc0017000 comp rx=0 tx=0).stop 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 msgr2=0x7fbcd01536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 0x7fbcd01536e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e930 tx=0x7fbccc0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 shutdown_connections 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fbcb4066f30 0x7fbcb40693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbcd0150df0 0x7fbcd00c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbcd00b1840 0x7fbcd01536e0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 --2- 172.21.15.29:0/797304028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcd0157290 0x7fbcd015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 >> 172.21.15.29:0/797304028 conn(0x7fbcd00680a0 msgr2=0x7fbcd0152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.681+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 shutdown_connections 2023-10-07T11:01:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.682+0000 7fbcd87ff700 1 -- 172.21.15.29:0/797304028 wait complete. 2023-10-07T11:01:37.699 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T11:01:37.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T11:01:37.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.899+0000 7faef85ff700 1 Processor -- start 2023-10-07T11:01:37.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.899+0000 7faef85ff700 1 -- start start 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef85ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faef0159580 0x7faef00b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef85ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faef00aed00 0x7faef00af160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef85ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00cccc0 con 0x7faef015dc20 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef85ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef00af6a0 con 0x7faef0159580 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef85ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef00b30d0 con 0x7faef00aed00 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef639b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faef0159580 0x7faef00b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef5b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faef015dc20 0x7faef00cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56644/0 (socket says 172.21.15.29:56644) 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef5b9a700 1 -- 172.21.15.29:0/3401319122 learned_addr learned my addr 172.21.15.29:0/3401319122 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.901+0000 7faef6b9c700 1 --2- 172.21.15.29:0/3401319122 >> v2:172.21.15.144:3300/0 conn(0x7faef00aed00 0x7faef00af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef5399700 1 -- 172.21.15.29:0/3401319122 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534136397 0 0) 0x7faef00cccc0 con 0x7faef015dc20 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef639b700 1 -- 172.21.15.29:0/3401319122 >> v2:172.21.15.144:3300/0 conn(0x7faef00aed00 msgr2=0x7faef00af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef639b700 1 --2- 172.21.15.29:0/3401319122 >> v2:172.21.15.144:3300/0 conn(0x7faef00aed00 0x7faef00af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef639b700 1 -- 172.21.15.29:0/3401319122 >> v1:172.21.15.29:6789/0 conn(0x7faef015dc20 legacy=0x7faef00cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef639b700 1 -- 172.21.15.29:0/3401319122 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef015d790 con 0x7faef0159580 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef639b700 1 --2- 172.21.15.29:0/3401319122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef0159580 0x7faef00b24e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7faef00b2a20 tx=0x7faeec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55d567df040e5c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef5399700 1 -- 172.21.15.29:0/3401319122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec011630 con 0x7faef0159580 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef5399700 1 -- 172.21.15.29:0/3401319122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec011790 con 0x7faef0159580 2023-10-07T11:01:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.902+0000 7faef5399700 1 -- 172.21.15.29:0/3401319122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec011b30 con 0x7faef0159580 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 -- 172.21.15.29:0/3401319122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef0159580 msgr2=0x7faef00b24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 --2- 172.21.15.29:0/3401319122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef0159580 0x7faef00b24e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7faef00b2a20 tx=0x7faeec0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 -- 172.21.15.29:0/3401319122 shutdown_connections 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 --2- 172.21.15.29:0/3401319122 >> v2:172.21.15.144:3300/0 conn(0x7faef00aed00 0x7faef00af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 --2- 172.21.15.29:0/3401319122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef0159580 0x7faef00b24e0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 -- 172.21.15.29:0/3401319122 >> 172.21.15.29:0/3401319122 conn(0x7faef0068720 msgr2=0x7faef0068b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 -- 172.21.15.29:0/3401319122 shutdown_connections 2023-10-07T11:01:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.903+0000 7faef85ff700 1 -- 172.21.15.29:0/3401319122 wait complete. 2023-10-07T11:01:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.904+0000 7faef85ff700 1 Processor -- start 2023-10-07T11:01:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.904+0000 7faef85ff700 1 -- start start 2023-10-07T11:01:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.905+0000 7faef85ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faef00aed00 0x7faef00ab3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.905+0000 7faef85ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faef0159580 0x7faef01562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.905+0000 7faef85ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 0x7faef014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.905+0000 7faef85ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef00a9fc0 con 0x7faef015dc20 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.905+0000 7faef85ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef0156820 con 0x7faef0159580 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.905+0000 7faef85ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef0156ae0 con 0x7faef00aed00 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 0x7faef014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 0x7faef014d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49710/0 (socket says 172.21.15.29:49710) 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 -- 172.21.15.29:0/3196449999 learned_addr learned my addr 172.21.15.29:0/3196449999 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef5b9a700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faef0159580 0x7faef01562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef639b700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faef00aed00 0x7faef00ab3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 -- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faef00aed00 msgr2=0x7faef00ab3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faef00aed00 0x7faef00ab3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 -- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faef0159580 msgr2=0x7faef01562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faef0159580 0x7faef01562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 -- 172.21.15.29:0/3196449999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec009420 con 0x7faef015dc20 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef5b9a700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faef0159580 0x7faef01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.906+0000 7faef6b9c700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 0x7faef014d130 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7faee8009e90 tx=0x7faee800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.907+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee800cc20 con 0x7faef015dc20 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.907+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee800f040 con 0x7faef015dc20 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.907+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8014480 con 0x7faef015dc20 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.907+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef00a9970 con 0x7faef015dc20 2023-10-07T11:01:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.907+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef014d670 con 0x7faef015dc20 2023-10-07T11:01:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.908+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef00aa130 con 0x7faef015dc20 2023-10-07T11:01:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.909+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faee8014830 con 0x7faef015dc20 2023-10-07T11:01:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.910+0000 7faee67fc700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7faed4066f30 0x7faed40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.910+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5301+0+0 (secure 0 0 0) 0x7faee80850d0 con 0x7faef015dc20 2023-10-07T11:01:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.910+0000 7faef639b700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7faed4066f30 0x7faed40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.911+0000 7faef639b700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7faed4066f30 0x7faed40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faeec0093f0 tx=0x7faeec017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:37.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:37.913+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faee8055f60 con 0x7faef015dc20 2023-10-07T11:01:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.119+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faef00718b0 con 0x7faef015dc20 2023-10-07T11:01:38.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.866+0000 7faee67fc700 1 -- 172.21.15.29:0/3196449999 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v179) v1 ==== 138+0+0 (secure 0 0 0) 0x7faee8055af0 con 0x7faef015dc20 2023-10-07T11:01:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7faed4066f30 msgr2=0x7faed40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7faed4066f30 0x7faed40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faeec0093f0 tx=0x7faeec017000 comp rx=0 tx=0).stop 2023-10-07T11:01:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 msgr2=0x7faef014d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 0x7faef014d130 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7faee8009e90 tx=0x7faee800b040 comp rx=0 tx=0).stop 2023-10-07T11:01:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 shutdown_connections 2023-10-07T11:01:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef015dc20 0x7faef014d130 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7faed4066f30 0x7faed40693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faef00aed00 0x7faef00ab3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.872+0000 7faef85ff700 1 --2- 172.21.15.29:0/3196449999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faef0159580 0x7faef01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.873+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 >> 172.21.15.29:0/3196449999 conn(0x7faef0068720 msgr2=0x7faef014da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.873+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 shutdown_connections 2023-10-07T11:01:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:38.874+0000 7faef85ff700 1 -- 172.21.15.29:0/3196449999 wait complete. 2023-10-07T11:01:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T11:01:38.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T11:01:39.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.102+0000 7f36297ff700 1 Processor -- start 2023-10-07T11:01:39.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.102+0000 7f36297ff700 1 -- start start 2023-10-07T11:01:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.103+0000 7f36297ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36241558e0 0x7f36240b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.103+0000 7f36297ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240c97e0 con 0x7f36240b0680 2023-10-07T11:01:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f36297ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240c8bd0 con 0x7f3624159e10 2023-10-07T11:01:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f36297ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36240c8eb0 con 0x7f36241558e0 2023-10-07T11:01:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f3622ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36241558e0 0x7f36240b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f36227fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f36240b0680 0x7f3624159700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56652/0 (socket says 172.21.15.29:56652) 2023-10-07T11:01:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f36227fc700 1 -- 172.21.15.29:0/877522450 learned_addr learned my addr 172.21.15.29:0/877522450 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f3621ffb700 1 -- 172.21.15.29:0/877522450 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338582236 0 0) 0x7f36240c97e0 con 0x7f36240b0680 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f3621ffb700 1 -- 172.21.15.29:0/877522450 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f360c003570 con 0x7f36240b0680 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f3622ffd700 1 -- 172.21.15.29:0/877522450 >> v1:172.21.15.144:6789/0 conn(0x7f3624159e10 legacy=0x7f36240c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f3622ffd700 1 -- 172.21.15.29:0/877522450 >> v1:172.21.15.29:6789/0 conn(0x7f36240b0680 legacy=0x7f3624159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.104+0000 7f3622ffd700 1 -- 172.21.15.29:0/877522450 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36240c9190 con 0x7f36241558e0 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.105+0000 7f3622ffd700 1 --2- 172.21.15.29:0/877522450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36241558e0 0x7f36240b0140 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f361400bb90 tx=0x7f361400b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe818a85db5873ee server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.105+0000 7f3621ffb700 1 -- 172.21.15.29:0/877522450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361400db40 con 0x7f36241558e0 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.105+0000 7f3621ffb700 1 -- 172.21.15.29:0/877522450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361400dca0 con 0x7f36241558e0 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.105+0000 7f3621ffb700 1 -- 172.21.15.29:0/877522450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36140196b0 con 0x7f36241558e0 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.105+0000 7f36297ff700 1 -- 172.21.15.29:0/877522450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36241558e0 msgr2=0x7f36240b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.105+0000 7f36297ff700 1 --2- 172.21.15.29:0/877522450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36241558e0 0x7f36240b0140 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f361400bb90 tx=0x7f361400b430 comp rx=0 tx=0).stop 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.106+0000 7f36297ff700 1 -- 172.21.15.29:0/877522450 shutdown_connections 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.106+0000 7f36297ff700 1 --2- 172.21.15.29:0/877522450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36241558e0 0x7f36240b0140 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.106+0000 7f36297ff700 1 -- 172.21.15.29:0/877522450 >> 172.21.15.29:0/877522450 conn(0x7f3624067610 msgr2=0x7f3624067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.106+0000 7f36297ff700 1 -- 172.21.15.29:0/877522450 shutdown_connections 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.106+0000 7f36297ff700 1 -- 172.21.15.29:0/877522450 wait complete. 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.107+0000 7f36297ff700 1 Processor -- start 2023-10-07T11:01:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.107+0000 7f36297ff700 1 -- start start 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.108+0000 7f36297ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 0x7f36241689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.108+0000 7f36297ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36241558e0 0x7f362414cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.108+0000 7f36297ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3624159e10 0x7f362414d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.108+0000 7f36297ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3624166aa0 con 0x7f36240b0680 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.108+0000 7f36297ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3624168ef0 con 0x7f36241558e0 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.108+0000 7f36297ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362414da80 con 0x7f3624159e10 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 0x7f36241689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 0x7f36241689b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49734/0 (socket says 172.21.15.29:49734) 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 -- 172.21.15.29:0/4267373056 learned_addr learned my addr 172.21.15.29:0/4267373056 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f36227fc700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36241558e0 0x7f362414cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f36237fe700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3624159e10 0x7f362414d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 -- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3624159e10 msgr2=0x7f362414d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3624159e10 0x7f362414d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 -- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36241558e0 msgr2=0x7f362414cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36241558e0 0x7f362414cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 -- 172.21.15.29:0/4267373056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361400b6d0 con 0x7f36240b0680 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f36237fe700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3624159e10 0x7f362414d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.109+0000 7f3622ffd700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 0x7f36241689b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f361400b2c0 tx=0x7f361401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:39.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.110+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614010750 con 0x7f36240b0680 2023-10-07T11:01:39.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.110+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36140108b0 con 0x7f36240b0680 2023-10-07T11:01:39.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.110+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362414dd60 con 0x7f36240b0680 2023-10-07T11:01:39.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.110+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614010c70 con 0x7f36240b0680 2023-10-07T11:01:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.110+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362414e320 con 0x7f36240b0680 2023-10-07T11:01:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.111+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36240718b0 con 0x7f36240b0680 2023-10-07T11:01:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.112+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3614023070 con 0x7f36240b0680 2023-10-07T11:01:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.113+0000 7f360affd700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3610066f30 0x7f36100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.113+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5671+0+0 (secure 0 0 0) 0x7f3614014070 con 0x7f36240b0680 2023-10-07T11:01:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.113+0000 7f36227fc700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3610066f30 0x7f36100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:39.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.114+0000 7f36227fc700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3610066f30 0x7f36100693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f362406e310 tx=0x7f3618009040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:39.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.117+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3614055d20 con 0x7f36240b0680 2023-10-07T11:01:39.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.322+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f36240ccd90 con 0x7f36240b0680 2023-10-07T11:01:39.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.874+0000 7f360affd700 1 -- 172.21.15.29:0/4267373056 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v180) v1 ==== 170+0+0 (secure 0 0 0) 0x7f361402a0d0 con 0x7f36240b0680 2023-10-07T11:01:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.883+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3610066f30 msgr2=0x7f36100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.883+0000 7f36297ff700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3610066f30 0x7f36100693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f362406e310 tx=0x7f3618009040 comp rx=0 tx=0).stop 2023-10-07T11:01:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.883+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 msgr2=0x7f36241689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.883+0000 7f36297ff700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 0x7f36241689b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f361400b2c0 tx=0x7f361401b000 comp rx=0 tx=0).stop 2023-10-07T11:01:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.888+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 shutdown_connections 2023-10-07T11:01:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.888+0000 7f36297ff700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3610066f30 0x7f36100693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.888+0000 7f36297ff700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3624159e10 0x7f362414d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.888+0000 7f36297ff700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36240b0680 0x7f36241689b0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.888+0000 7f36297ff700 1 --2- 172.21.15.29:0/4267373056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36241558e0 0x7f362414cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.888+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 >> 172.21.15.29:0/4267373056 conn(0x7f3624067610 msgr2=0x7f3624153250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:39.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.890+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 shutdown_connections 2023-10-07T11:01:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:39.891+0000 7f36297ff700 1 -- 172.21.15.29:0/4267373056 wait complete. 2023-10-07T11:01:39.897 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T11:01:39.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T11:01:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.115+0000 7fa00fbff700 1 Processor -- start 2023-10-07T11:01:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.115+0000 7fa00fbff700 1 -- start start 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00fbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa00815a960 0x7fa008158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00fbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815ad30 0x7fa0080ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00fbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0080acb40 con 0x7fa0080ae6d0 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00fbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0080aeb50 con 0x7fa00815a960 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00fbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008158c60 con 0x7fa00815ad30 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00d99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa00815a960 0x7fa008158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00e19c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa0080ae6d0 0x7fa0080b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56656/0 (socket says 172.21.15.29:56656) 2023-10-07T11:01:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00e19c700 1 -- 172.21.15.29:0/718323873 learned_addr learned my addr 172.21.15.29:0/718323873 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:40.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00d19a700 1 --2- 172.21.15.29:0/718323873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815ad30 0x7fa0080ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.117+0000 7fa00c999700 1 -- 172.21.15.29:0/718323873 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873059571 0 0) 0x7fa0080acb40 con 0x7fa0080ae6d0 2023-10-07T11:01:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00d99b700 1 -- 172.21.15.29:0/718323873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815ad30 msgr2=0x7fa0080ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00d99b700 1 --2- 172.21.15.29:0/718323873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815ad30 0x7fa0080ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00d99b700 1 -- 172.21.15.29:0/718323873 >> v1:172.21.15.29:6789/0 conn(0x7fa0080ae6d0 legacy=0x7fa0080b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00d99b700 1 -- 172.21.15.29:0/718323873 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0080abf30 con 0x7fa00815a960 2023-10-07T11:01:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00d99b700 1 --2- 172.21.15.29:0/718323873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815a960 0x7fa008158e20 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa0040098e0 tx=0x7fa0040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e902171b6d1b45 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00c999700 1 -- 172.21.15.29:0/718323873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00400b640 con 0x7fa00815a960 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00c999700 1 -- 172.21.15.29:0/718323873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00400b7a0 con 0x7fa00815a960 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.118+0000 7fa00c999700 1 -- 172.21.15.29:0/718323873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00400bb40 con 0x7fa00815a960 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 -- 172.21.15.29:0/718323873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815a960 msgr2=0x7fa008158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 --2- 172.21.15.29:0/718323873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815a960 0x7fa008158e20 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa0040098e0 tx=0x7fa0040091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 -- 172.21.15.29:0/718323873 shutdown_connections 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 --2- 172.21.15.29:0/718323873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815a960 0x7fa008158e20 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 --2- 172.21.15.29:0/718323873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815ad30 0x7fa0080ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 -- 172.21.15.29:0/718323873 >> 172.21.15.29:0/718323873 conn(0x7fa008067610 msgr2=0x7fa008067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 -- 172.21.15.29:0/718323873 shutdown_connections 2023-10-07T11:01:40.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.119+0000 7fa00fbff700 1 -- 172.21.15.29:0/718323873 wait complete. 2023-10-07T11:01:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.120+0000 7fa00fbff700 1 Processor -- start 2023-10-07T11:01:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.120+0000 7fa00fbff700 1 -- start start 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.121+0000 7fa00fbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0080ae6d0 0x7fa00814f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.121+0000 7fa00fbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815a960 0x7fa00816ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.121+0000 7fa00fbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 0x7fa00816b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.121+0000 7fa00fbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008153910 con 0x7fa00815ad30 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.121+0000 7fa00fbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008153bf0 con 0x7fa0080ae6d0 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.121+0000 7fa00fbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa008153ed0 con 0x7fa00815a960 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 0x7fa00816b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 0x7fa00816b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49762/0 (socket says 172.21.15.29:49762) 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 -- 172.21.15.29:0/4171478198 learned_addr learned my addr 172.21.15.29:0/4171478198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:40.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00d99b700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0080ae6d0 0x7fa00814f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00d19a700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815a960 0x7fa00816ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 -- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815a960 msgr2=0x7fa00816ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815a960 0x7fa00816ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 -- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0080ae6d0 msgr2=0x7fa00814f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0080ae6d0 0x7fa00814f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 -- 172.21.15.29:0/4171478198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004009420 con 0x7fa00815ad30 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00d99b700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0080ae6d0 0x7fa00814f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.122+0000 7fa00e19c700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 0x7fa00816b160 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa00000cc90 tx=0x7fa000001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.123+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00000cea0 con 0x7fa00815ad30 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.123+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00000ab70 con 0x7fa00815ad30 2023-10-07T11:01:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.123+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000070f0 con 0x7fa00815ad30 2023-10-07T11:01:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.123+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081541b0 con 0x7fa00815ad30 2023-10-07T11:01:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.123+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00814fda0 con 0x7fa00815ad30 2023-10-07T11:01:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.124+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa008153a50 con 0x7fa00815ad30 2023-10-07T11:01:40.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.125+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa0000074a0 con 0x7fa00815ad30 2023-10-07T11:01:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.126+0000 7f9ffdffb700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9fec066f30 0x7f9fec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.126+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa000085410 con 0x7fa00815ad30 2023-10-07T11:01:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.127+0000 7fa00d99b700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9fec066f30 0x7f9fec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.128+0000 7fa00d99b700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9fec066f30 0x7f9fec0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0040093f0 tx=0x7fa004017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:40.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.130+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa000052680 con 0x7fa00815ad30 2023-10-07T11:01:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.376+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa00804c430 con 0x7fa00815ad30 2023-10-07T11:01:40.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.880+0000 7f9ffdffb700 1 -- 172.21.15.29:0/4171478198 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v90) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa00804c430 con 0x7fa00815ad30 2023-10-07T11:01:40.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9fec066f30 msgr2=0x7f9fec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:40.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9fec066f30 0x7f9fec0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0040093f0 tx=0x7fa004017000 comp rx=0 tx=0).stop 2023-10-07T11:01:40.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 msgr2=0x7fa00816b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:40.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 0x7fa00816b160 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa00000cc90 tx=0x7fa000001a80 comp rx=0 tx=0).stop 2023-10-07T11:01:40.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 shutdown_connections 2023-10-07T11:01:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa00815ad30 0x7fa00816b160 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0080ae6d0 0x7fa00814f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f9fec066f30 0x7f9fec0693e0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0040093f0 tx=0x7fa004017000 comp rx=0 tx=0).stop 2023-10-07T11:01:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 --2- 172.21.15.29:0/4171478198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa00815a960 0x7fa00816ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.889+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 >> 172.21.15.29:0/4171478198 conn(0x7fa008067610 msgr2=0x7fa008159380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:40.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.892+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 shutdown_connections 2023-10-07T11:01:40.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:40.892+0000 7fa00fbff700 1 -- 172.21.15.29:0/4171478198 wait complete. 2023-10-07T11:01:40.906 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 22 and data pool 23 2023-10-07T11:01:40.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.121+0000 7f220d9ff700 1 Processor -- start 2023-10-07T11:01:41.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.122+0000 7f220d9ff700 1 -- start start 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f220d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2208159750 0x7f22080d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f220d9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22080cbf20 con 0x7f22080ce8a0 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f220d9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22080cc200 con 0x7f2208159b20 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f220d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22080cc4e0 con 0x7f2208159750 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f22077fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f22080ce8a0 0x7f22080c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56666/0 (socket says 172.21.15.29:56666) 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f22077fe700 1 -- 172.21.15.29:0/195182283 learned_addr learned my addr 172.21.15.29:0/195182283 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.123+0000 7f2206ffd700 1 --2- 172.21.15.29:0/195182283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2208159750 0x7f22080d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033454805 0 0) 0x7f22080cbf20 con 0x7f22080ce8a0 2023-10-07T11:01:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f0003570 con 0x7f22080ce8a0 2023-10-07T11:01:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213484444 0 0) 0x7f22080cc200 con 0x7f2208159b20 2023-10-07T11:01:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22080cbf20 con 0x7f2208159b20 2023-10-07T11:01:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2206ffd700 1 -- 172.21.15.29:0/195182283 >> v1:172.21.15.144:6789/0 conn(0x7f2208159b20 legacy=0x7f22080ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2206ffd700 1 -- 172.21.15.29:0/195182283 >> v1:172.21.15.29:6789/0 conn(0x7f22080ce8a0 legacy=0x7f22080c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2206ffd700 1 -- 172.21.15.29:0/195182283 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22080cc7c0 con 0x7f2208159750 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.124+0000 7f2206ffd700 1 --2- 172.21.15.29:0/195182283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2208159750 0x7f22080d10a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009900 tx=0x7f21f8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28cd9a29815279a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8005c10 con 0x7f2208159750 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f8005d70 con 0x7f2208159750 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f2205ffb700 1 -- 172.21.15.29:0/195182283 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8006240 con 0x7f2208159750 2023-10-07T11:01:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 -- 172.21.15.29:0/195182283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2208159750 msgr2=0x7f22080d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 --2- 172.21.15.29:0/195182283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2208159750 0x7f22080d10a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009900 tx=0x7f21f8009120 comp rx=0 tx=0).stop 2023-10-07T11:01:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 -- 172.21.15.29:0/195182283 shutdown_connections 2023-10-07T11:01:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 --2- 172.21.15.29:0/195182283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2208159750 0x7f22080d10a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 -- 172.21.15.29:0/195182283 >> 172.21.15.29:0/195182283 conn(0x7f2208063a10 msgr2=0x7f22080d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 -- 172.21.15.29:0/195182283 shutdown_connections 2023-10-07T11:01:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.125+0000 7f220d9ff700 1 -- 172.21.15.29:0/195182283 wait complete. 2023-10-07T11:01:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.127+0000 7f220d9ff700 1 Processor -- start 2023-10-07T11:01:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.127+0000 7f220d9ff700 1 -- start start 2023-10-07T11:01:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f220d9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22080ce8a0 0x7f2208147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f220d9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 0x7f220815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f220d9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2208159b20 0x7f22080c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f220d9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22080b3d30 con 0x7f2208159750 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f220d9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220815b150 con 0x7f2208159b20 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f220d9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22080b3400 con 0x7f22080ce8a0 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 0x7f220815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 0x7f220815a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49770/0 (socket says 172.21.15.29:49770) 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 -- 172.21.15.29:0/2790322621 learned_addr learned my addr 172.21.15.29:0/2790322621 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f2206ffd700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22080ce8a0 0x7f2208147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22077fe700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2208159b20 0x7f22080c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 -- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22080ce8a0 msgr2=0x7f2208147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22080ce8a0 0x7f2208147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 -- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2208159b20 msgr2=0x7f22080c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2208159b20 0x7f22080c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.128+0000 7f22067fc700 1 -- 172.21.15.29:0/2790322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f80093c0 con 0x7f2208159750 2023-10-07T11:01:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.129+0000 7f22067fc700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 0x7f220815a8d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c590 tx=0x7f21fc007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.129+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc010510 con 0x7f2208159750 2023-10-07T11:01:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.129+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc010670 con 0x7f2208159750 2023-10-07T11:01:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.129+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc010a30 con 0x7f2208159750 2023-10-07T11:01:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.129+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22080b36e0 con 0x7f2208159750 2023-10-07T11:01:41.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.129+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2208148500 con 0x7f2208159750 2023-10-07T11:01:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.130+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22080cbc10 con 0x7f2208159750 2023-10-07T11:01:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.131+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f21fc01b400 con 0x7f2208159750 2023-10-07T11:01:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.132+0000 7f21eeffd700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f21f4066f30 0x7f21f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.132+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f21fc0842f0 con 0x7f2208159750 2023-10-07T11:01:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.133+0000 7f2206ffd700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f21f4066f30 0x7f21f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.133+0000 7f2206ffd700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f21f4066f30 0x7f21f40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009160 tx=0x7f21f8016000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.136+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21fc051300 con 0x7f2208159750 2023-10-07T11:01:41.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.336+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f220804c3c0 con 0x7f2208159750 2023-10-07T11:01:41.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.337+0000 7f21eeffd700 1 -- 172.21.15.29:0/2790322621 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16161 (secure 0 0 0) 0x7f21fc054920 con 0x7f2208159750 2023-10-07T11:01:41.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:41.338 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":181,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:01:40.378425+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:01:38.164128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-07T11:01:39.365755+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:41.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f21f4066f30 msgr2=0x7f21f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f21f4066f30 0x7f21f40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009160 tx=0x7f21f8016000 comp rx=0 tx=0).stop 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 msgr2=0x7f220815a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 0x7f220815a8d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c590 tx=0x7f21fc007dc0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 shutdown_connections 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2208159b20 0x7f22080c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f21f4066f30 0x7f21f40693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22080ce8a0 0x7f2208147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 --2- 172.21.15.29:0/2790322621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2208159750 0x7f220815a8d0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 >> 172.21.15.29:0/2790322621 conn(0x7f2208063a10 msgr2=0x7f22080d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 shutdown_connections 2023-10-07T11:01:41.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.339+0000 7f220d9ff700 1 -- 172.21.15.29:0/2790322621 wait complete. 2023-10-07T11:01:41.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:41.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.553+0000 7f3803dff700 1 Processor -- start 2023-10-07T11:01:41.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.553+0000 7f3803dff700 1 -- start start 2023-10-07T11:01:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.554+0000 7f3803dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.554+0000 7f3803dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37fc15ade0 0x7f37fc0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.554+0000 7f3803dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc154090 con 0x7f37fc0b0000 2023-10-07T11:01:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.554+0000 7f3803dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc159f60 con 0x7f37fc15ade0 2023-10-07T11:01:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.554+0000 7f3803dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc15a240 con 0x7f37fc153cc0 2023-10-07T11:01:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37fc15ade0 0x7f37fc0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37fc15ade0 0x7f37fc0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49778/0 (socket says 172.21.15.29:49778) 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 -- 172.21.15.29:0/1788968116 learned_addr learned my addr 172.21.15.29:0/1788968116 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f3801b9b700 1 --2- 172.21.15.29:0/1788968116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 -- 172.21.15.29:0/1788968116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 msgr2=0x7f37fc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 --2- 172.21.15.29:0/1788968116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 -- 172.21.15.29:0/1788968116 >> v1:172.21.15.144:6789/0 conn(0x7f37fc0b0000 legacy=0x7f37fc15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 -- 172.21.15.29:0/1788968116 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc15a520 con 0x7f37fc15ade0 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.555+0000 7f380139a700 1 --2- 172.21.15.29:0/1788968116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0b1d00 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f37ec0099b0 tx=0x7f37ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf16df3731032e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3800b99700 1 -- 172.21.15.29:0/1788968116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec00b860 con 0x7f37fc15ade0 2023-10-07T11:01:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3800b99700 1 -- 172.21.15.29:0/1788968116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec00b9c0 con 0x7f37fc15ade0 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3800b99700 1 -- 172.21.15.29:0/1788968116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec00bd60 con 0x7f37fc15ade0 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 -- 172.21.15.29:0/1788968116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 msgr2=0x7f37fc0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 --2- 172.21.15.29:0/1788968116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0b1d00 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f37ec0099b0 tx=0x7f37ec009250 comp rx=0 tx=0).stop 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 -- 172.21.15.29:0/1788968116 shutdown_connections 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 --2- 172.21.15.29:0/1788968116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 --2- 172.21.15.29:0/1788968116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0b1d00 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 -- 172.21.15.29:0/1788968116 >> 172.21.15.29:0/1788968116 conn(0x7f37fc067610 msgr2=0x7f37fc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 -- 172.21.15.29:0/1788968116 shutdown_connections 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.556+0000 7f3803dff700 1 -- 172.21.15.29:0/1788968116 wait complete. 2023-10-07T11:01:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.558+0000 7f3803dff700 1 Processor -- start 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.558+0000 7f3803dff700 1 -- start start 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3803dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37fc0b0000 0x7f37fc152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3803dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3803dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3803dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc1698e0 con 0x7f37fc15ade0 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3803dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc155620 con 0x7f37fc0b0000 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3803dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc155900 con 0x7f37fc153cc0 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f380239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f380239c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49788/0 (socket says 172.21.15.29:49788) 2023-10-07T11:01:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f380239c700 1 -- 172.21.15.29:0/397000907 learned_addr learned my addr 172.21.15.29:0/397000907 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f380139a700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.559+0000 7f3801b9b700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37fc0b0000 0x7f37fc152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f380239c700 1 -- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 msgr2=0x7f37fc0c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f380239c700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f380239c700 1 -- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37fc0b0000 msgr2=0x7f37fc152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f380239c700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37fc0b0000 0x7f37fc152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f380239c700 1 -- 172.21.15.29:0/397000907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0094f0 con 0x7f37fc15ade0 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f380239c700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0c7af0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f37f400c8e0 tx=0x7f37f4001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4012630 con 0x7f37fc15ade0 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f4012790 con 0x7f37fc15ade0 2023-10-07T11:01:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4012b50 con 0x7f37fc15ade0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.560+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc155be0 con 0x7f37fc15ade0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.561+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc153440 con 0x7f37fc15ade0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.562+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc04c3c0 con 0x7f37fc15ade0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.562+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f37f401a3a0 con 0x7f37fc15ade0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.563+0000 7f37f1ffb700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f37e0066f30 0x7f37e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.563+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f37f4084900 con 0x7f37fc15ade0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.564+0000 7f3801b9b700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f37e0066f30 0x7f37e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.565+0000 7f3801b9b700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f37e0066f30 0x7f37e00693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37f8006aa0 tx=0x7f37f8008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.567+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37f40519b0 con 0x7f37fc15ade0 2023-10-07T11:01:41.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.770+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37fc169a20 con 0x7f37fc15ade0 2023-10-07T11:01:41.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.771+0000 7f37f1ffb700 1 -- 172.21.15.29:0/397000907 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16161 (secure 0 0 0) 0x7f37f4054fd0 con 0x7f37fc15ade0 2023-10-07T11:01:41.772 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:41.772 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":181,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:01:40.378425+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:01:38.164128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-07T11:01:39.365755+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f37e0066f30 msgr2=0x7f37e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f37e0066f30 0x7f37e00693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37f8006aa0 tx=0x7f37f8008040 comp rx=0 tx=0).stop 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 msgr2=0x7f37fc0c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0c7af0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f37f400c8e0 tx=0x7f37f4001710 comp rx=0 tx=0).stop 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 shutdown_connections 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc15ade0 0x7f37fc0c7af0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37fc0b0000 0x7f37fc152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f37e0066f30 0x7f37e00693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 --2- 172.21.15.29:0/397000907 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37fc153cc0 0x7f37fc0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 >> 172.21.15.29:0/397000907 conn(0x7f37fc067610 msgr2=0x7f37fc0af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 shutdown_connections 2023-10-07T11:01:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.774+0000 7f3803dff700 1 -- 172.21.15.29:0/397000907 wait complete. 2023-10-07T11:01:41.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T11:01:41.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.993+0000 7fb2a0dff700 1 Processor -- start 2023-10-07T11:01:41.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.993+0000 7fb2a0dff700 1 -- start start 2023-10-07T11:01:41.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb2a0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:41.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb2a0dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0cefb0 con 0x7fb29c159750 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb2a0dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0cbf20 con 0x7fb29c159b20 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb2a0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0cc200 con 0x7fb29c0c6470 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb29a59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb29c159750 0x7fb29c0ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56682/0 (socket says 172.21.15.29:56682) 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb29a59c700 1 -- 172.21.15.29:0/585699398 learned_addr learned my addr 172.21.15.29:0/585699398 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb29ad9d700 1 --2- 172.21.15.29:0/585699398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.995+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211315511 0 0) 0x7fb29c0cefb0 con 0x7fb29c159750 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb284003570 con 0x7fb29c159750 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24205208 0 0) 0x7fb29c0cbf20 con 0x7fb29c159b20 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c0cefb0 con 0x7fb29c159b20 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29ad9d700 1 -- 172.21.15.29:0/585699398 >> v1:172.21.15.144:6789/0 conn(0x7fb29c159b20 legacy=0x7fb29c0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29ad9d700 1 -- 172.21.15.29:0/585699398 >> v1:172.21.15.29:6789/0 conn(0x7fb29c159750 legacy=0x7fb29c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29ad9d700 1 -- 172.21.15.29:0/585699398 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0cc4e0 con 0x7fb29c0c6470 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29ad9d700 1 --2- 172.21.15.29:0/585699398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c0d10a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb294009e90 tx=0x7fb294009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7175ad4e03b4bf2c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29400e070 con 0x7fb29c0c6470 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29400bb80 con 0x7fb29c0c6470 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.996+0000 7fb29959a700 1 -- 172.21.15.29:0/585699398 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294012430 con 0x7fb29c0c6470 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 -- 172.21.15.29:0/585699398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 msgr2=0x7fb29c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/585699398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c0d10a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb294009e90 tx=0x7fb294009780 comp rx=0 tx=0).stop 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 -- 172.21.15.29:0/585699398 shutdown_connections 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/585699398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c0d10a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 -- 172.21.15.29:0/585699398 >> 172.21.15.29:0/585699398 conn(0x7fb29c063a10 msgr2=0x7fb29c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 -- 172.21.15.29:0/585699398 shutdown_connections 2023-10-07T11:01:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.997+0000 7fb2a0dff700 1 -- 172.21.15.29:0/585699398 wait complete. 2023-10-07T11:01:42.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.999+0000 7fb2a0dff700 1 Processor -- start 2023-10-07T11:01:42.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:41.999+0000 7fb2a0dff700 1 -- start start 2023-10-07T11:01:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb2a0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb2a0dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb29c159750 0x7fb29c151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb2a0dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb29c159b20 0x7fb29c0c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb2a0dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0ca4c0 con 0x7fb29c159b20 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb2a0dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c155250 con 0x7fb29c159750 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb2a0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c154640 con 0x7fb29c0c6470 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb29a59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb29a59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:53504/0 (socket says 172.21.15.29:53504) 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb29a59c700 1 -- 172.21.15.29:0/645225435 learned_addr learned my addr 172.21.15.29:0/645225435 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb299d9b700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb29c159750 0x7fb29c151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.000+0000 7fb29ad9d700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb29c159b20 0x7fb29c0c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.001+0000 7fb29a59c700 1 -- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb29c159750 msgr2=0x7fb29c151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:42.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.001+0000 7fb29a59c700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb29c159750 0x7fb29c151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:42.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.001+0000 7fb29a59c700 1 -- 172.21.15.29:0/645225435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb29c159b20 msgr2=0x7fb29c0c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:42.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.001+0000 7fb29a59c700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb29c159b20 0x7fb29c0c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:42.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.001+0000 7fb29a59c700 1 -- 172.21.15.29:0/645225435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb294009a20 con 0x7fb29c0c6470 2023-10-07T11:01:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.001+0000 7fb29a59c700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c153f40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00c790 tx=0x7fb28c007cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.002+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0072e0 con 0x7fb29c0c6470 2023-10-07T11:01:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.002+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c007460 con 0x7fb29c0c6470 2023-10-07T11:01:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.002+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0057f0 con 0x7fb29c0c6470 2023-10-07T11:01:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.002+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c154920 con 0x7fb29c0c6470 2023-10-07T11:01:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.002+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c154ee0 con 0x7fb29c0c6470 2023-10-07T11:01:42.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.003+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c06f1e0 con 0x7fb29c0c6470 2023-10-07T11:01:42.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.004+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb28c005c20 con 0x7fb29c0c6470 2023-10-07T11:01:42.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.005+0000 7fb28a7fc700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb27c067000 0x7fb27c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:42.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.005+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7fb28c083b00 con 0x7fb29c0c6470 2023-10-07T11:01:42.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.005+0000 7fb299d9b700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb27c067000 0x7fb27c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:42.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.006+0000 7fb299d9b700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb27c067000 0x7fb27c0694b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb290007fd0 tx=0x7fb290007d10 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:42.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.010+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb28c04b5a0 con 0x7fb29c0c6470 2023-10-07T11:01:42.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:42.257+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb29c04c3c0 con 0x7fb29c0c6470 2023-10-07T11:01:43.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.020+0000 7fb28a7fc700 1 -- 172.21.15.29:0/645225435 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v93) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb28c017020 con 0x7fb29c0c6470 2023-10-07T11:01:43.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb27c067000 msgr2=0x7fb27c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:43.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb27c067000 0x7fb27c0694b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb290007fd0 tx=0x7fb290007d10 comp rx=0 tx=0).stop 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 msgr2=0x7fb29c153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c153f40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00c790 tx=0x7fb28c007cc0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 shutdown_connections 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb29c159b20 0x7fb29c0c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb27c067000 0x7fb27c0694b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb29c0c6470 0x7fb29c153f40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 --2- 172.21.15.29:0/645225435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb29c159750 0x7fb29c151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 >> 172.21.15.29:0/645225435 conn(0x7fb29c063a10 msgr2=0x7fb29c15d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 shutdown_connections 2023-10-07T11:01:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.022+0000 7fb2a0dff700 1 -- 172.21.15.29:0/645225435 wait complete. 2023-10-07T11:01:43.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T11:01:43.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.248+0000 7ff66a9ff700 1 Processor -- start 2023-10-07T11:01:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.249+0000 7ff66a9ff700 1 -- start start 2023-10-07T11:01:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff66a9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff6640ae8a0 0x7ff664158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff66a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6640b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff66a9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6640acb40 con 0x7ff66415a960 2023-10-07T11:01:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff66a9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff664158c60 con 0x7ff6640ae8a0 2023-10-07T11:01:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff66a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640abf30 con 0x7ff66415ad30 2023-10-07T11:01:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff663fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff66415a960 0x7ff6640ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46868/0 (socket says 172.21.15.29:46868) 2023-10-07T11:01:43.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff663fff700 1 -- 172.21.15.29:0/684155423 learned_addr learned my addr 172.21.15.29:0/684155423 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff668f9c700 1 --2- 172.21.15.29:0/684155423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6640b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.250+0000 7ff6637fe700 1 --2- 172.21.15.29:0/684155423 >> v2:172.21.15.144:3300/0 conn(0x7ff6640ae8a0 0x7ff664158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff662ffd700 1 -- 172.21.15.29:0/684155423 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824785890 0 0) 0x7ff6640acb40 con 0x7ff66415a960 2023-10-07T11:01:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff662ffd700 1 -- 172.21.15.29:0/684155423 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c003570 con 0x7ff66415a960 2023-10-07T11:01:43.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff668f9c700 1 -- 172.21.15.29:0/684155423 >> v2:172.21.15.144:3300/0 conn(0x7ff6640ae8a0 msgr2=0x7ff664158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:43.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff668f9c700 1 --2- 172.21.15.29:0/684155423 >> v2:172.21.15.144:3300/0 conn(0x7ff6640ae8a0 0x7ff664158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff668f9c700 1 -- 172.21.15.29:0/684155423 >> v1:172.21.15.29:6789/0 conn(0x7ff66415a960 legacy=0x7ff6640ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:43.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff668f9c700 1 -- 172.21.15.29:0/684155423 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640ac210 con 0x7ff66415ad30 2023-10-07T11:01:43.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff668f9c700 1 --2- 172.21.15.29:0/684155423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6640b1290 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009e60 tx=0x7ff65c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f07048eef5ccf67 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:43.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.251+0000 7ff662ffd700 1 -- 172.21.15.29:0/684155423 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00e070 con 0x7ff66415ad30 2023-10-07T11:01:43.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff662ffd700 1 -- 172.21.15.29:0/684155423 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c00bab0 con 0x7ff66415ad30 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff662ffd700 1 -- 172.21.15.29:0/684155423 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00be70 con 0x7ff66415ad30 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 -- 172.21.15.29:0/684155423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 msgr2=0x7ff6640b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/684155423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6640b1290 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009e60 tx=0x7ff65c009780 comp rx=0 tx=0).stop 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 -- 172.21.15.29:0/684155423 shutdown_connections 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/684155423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6640b1290 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/684155423 >> v2:172.21.15.144:3300/0 conn(0x7ff6640ae8a0 0x7ff664158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 -- 172.21.15.29:0/684155423 >> 172.21.15.29:0/684155423 conn(0x7ff664067610 msgr2=0x7ff664067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:43.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 -- 172.21.15.29:0/684155423 shutdown_connections 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.252+0000 7ff66a9ff700 1 -- 172.21.15.29:0/684155423 wait complete. 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.254+0000 7ff66a9ff700 1 Processor -- start 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.254+0000 7ff66a9ff700 1 -- start start 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff66a9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 0x7ff66416a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff66a9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff66415a960 0x7ff66416c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff66a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6641693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff66a9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640ca990 con 0x7ff6640ae8a0 2023-10-07T11:01:43.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff66a9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664169f70 con 0x7ff66415a960 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff66a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640ca340 con 0x7ff66415ad30 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 0x7ff66416a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 0x7ff66416a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37052/0 (socket says 172.21.15.29:37052) 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 -- 172.21.15.29:0/106016198 learned_addr learned my addr 172.21.15.29:0/106016198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff668f9c700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6641693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff6637fe700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff66415a960 0x7ff66416c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:43.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 -- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 msgr2=0x7ff6641693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6641693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 -- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff66415a960 msgr2=0x7ff66416c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff66415a960 0x7ff66416c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.255+0000 7ff663fff700 1 -- 172.21.15.29:0/106016198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c009a20 con 0x7ff6640ae8a0 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.256+0000 7ff663fff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 0x7ff66416a540 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff65400c7c0 tx=0x7ff654007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.256+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654005b30 con 0x7ff6640ae8a0 2023-10-07T11:01:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.256+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff654005c90 con 0x7ff6640ae8a0 2023-10-07T11:01:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.256+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654007340 con 0x7ff6640ae8a0 2023-10-07T11:01:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.256+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6640ca620 con 0x7ff6640ae8a0 2023-10-07T11:01:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.256+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66416a250 con 0x7ff6640ae8a0 2023-10-07T11:01:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.257+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66404c3c0 con 0x7ff6640ae8a0 2023-10-07T11:01:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.258+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff65401b400 con 0x7ff6640ae8a0 2023-10-07T11:01:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.259+0000 7ff64bfff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff650066f30 0x7ff6500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.259+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7ff6540842e0 con 0x7ff6640ae8a0 2023-10-07T11:01:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.259+0000 7ff6637fe700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff650066f30 0x7ff6500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.260+0000 7ff6637fe700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff650066f30 0x7ff6500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff6640cca90 tx=0x7ff658008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.263+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff654051370 con 0x7ff6640ae8a0 2023-10-07T11:01:43.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:43.513+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff6640ca7b0 con 0x7ff6640ae8a0 2023-10-07T11:01:43.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:01:43.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:01:44.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.284+0000 7ff64bfff700 1 -- 172.21.15.29:0/106016198 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff654054990 con 0x7ff6640ae8a0 2023-10-07T11:01:44.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff650066f30 msgr2=0x7ff6500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff650066f30 0x7ff6500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff6640cca90 tx=0x7ff658008040 comp rx=0 tx=0).stop 2023-10-07T11:01:44.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 msgr2=0x7ff66416a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 0x7ff66416a540 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff65400c7c0 tx=0x7ff654007e00 comp rx=0 tx=0).stop 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 shutdown_connections 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff650066f30 0x7ff6500693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff66415ad30 0x7ff6641693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6640ae8a0 0x7ff66416a540 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 --2- 172.21.15.29:0/106016198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff66415a960 0x7ff66416c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 >> 172.21.15.29:0/106016198 conn(0x7ff664067610 msgr2=0x7ff6640c84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 shutdown_connections 2023-10-07T11:01:44.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.287+0000 7ff66a9ff700 1 -- 172.21.15.29:0/106016198 wait complete. 2023-10-07T11:01:44.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.504+0000 7ff8471ff700 1 Processor -- start 2023-10-07T11:01:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.504+0000 7ff8471ff700 1 -- start start 2023-10-07T11:01:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.505+0000 7ff8471ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff84015ade0 0x7ff8400b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.505+0000 7ff8471ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8400b0540 0x7ff84015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.505+0000 7ff8471ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff840154090 con 0x7ff840153cc0 2023-10-07T11:01:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.505+0000 7ff8471ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8400b09e0 con 0x7ff84015ade0 2023-10-07T11:01:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.505+0000 7ff8471ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff840159f60 con 0x7ff8400b0540 2023-10-07T11:01:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.505+0000 7ff83ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff84015ade0 0x7ff8400b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff84015ade0 0x7ff8400b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37054/0 (socket says 172.21.15.29:37054) 2023-10-07T11:01:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 -- 172.21.15.29:0/4187756174 learned_addr learned my addr 172.21.15.29:0/4187756174 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff84579c700 1 --2- 172.21.15.29:0/4187756174 >> v2:172.21.15.144:3300/0 conn(0x7ff8400b0540 0x7ff84015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 -- 172.21.15.29:0/4187756174 >> v2:172.21.15.144:3300/0 conn(0x7ff8400b0540 msgr2=0x7ff84015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 --2- 172.21.15.29:0/4187756174 >> v2:172.21.15.144:3300/0 conn(0x7ff8400b0540 0x7ff84015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 -- 172.21.15.29:0/4187756174 >> v1:172.21.15.144:6789/0 conn(0x7ff840153cc0 legacy=0x7ff8400b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 -- 172.21.15.29:0/4187756174 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84015a240 con 0x7ff84015ade0 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83ffff700 1 --2- 172.21.15.29:0/4187756174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84015ade0 0x7ff8400b0000 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff84006c400 tx=0x7ff82c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc5c134830815d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.506+0000 7ff83f7fe700 1 -- 172.21.15.29:0/4187756174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00b860 con 0x7ff84015ade0 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff83f7fe700 1 -- 172.21.15.29:0/4187756174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c00b9c0 con 0x7ff84015ade0 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff83f7fe700 1 -- 172.21.15.29:0/4187756174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00bd60 con 0x7ff84015ade0 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 -- 172.21.15.29:0/4187756174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84015ade0 msgr2=0x7ff8400b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 --2- 172.21.15.29:0/4187756174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84015ade0 0x7ff8400b0000 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff84006c400 tx=0x7ff82c009250 comp rx=0 tx=0).stop 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 -- 172.21.15.29:0/4187756174 shutdown_connections 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 --2- 172.21.15.29:0/4187756174 >> v2:172.21.15.144:3300/0 conn(0x7ff8400b0540 0x7ff84015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 --2- 172.21.15.29:0/4187756174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84015ade0 0x7ff8400b0000 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 -- 172.21.15.29:0/4187756174 >> 172.21.15.29:0/4187756174 conn(0x7ff840067610 msgr2=0x7ff840067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 -- 172.21.15.29:0/4187756174 shutdown_connections 2023-10-07T11:01:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.507+0000 7ff8471ff700 1 -- 172.21.15.29:0/4187756174 wait complete. 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.509+0000 7ff8471ff700 1 Processor -- start 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.509+0000 7ff8471ff700 1 -- start start 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff8471ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 0x7ff84013e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff8471ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff840153cc0 0x7ff840156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff8471ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff84015ade0 0x7ff8400c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff8471ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84016a820 con 0x7ff8400b0540 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff8471ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8401557a0 con 0x7ff84015ade0 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff8471ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff840155a80 con 0x7ff840153cc0 2023-10-07T11:01:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff844f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 0x7ff84013e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff844f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 0x7ff84013e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37060/0 (socket says 172.21.15.29:37060) 2023-10-07T11:01:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff844f9b700 1 -- 172.21.15.29:0/2329304766 learned_addr learned my addr 172.21.15.29:0/2329304766 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff84579c700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff84015ade0 0x7ff8400c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.510+0000 7ff83ffff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff840153cc0 0x7ff840156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff844f9b700 1 -- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff840153cc0 msgr2=0x7ff840156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff844f9b700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff840153cc0 0x7ff840156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff844f9b700 1 -- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff84015ade0 msgr2=0x7ff8400c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff844f9b700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff84015ade0 0x7ff8400c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff844f9b700 1 -- 172.21.15.29:0/2329304766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0094f0 con 0x7ff8400b0540 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff844f9b700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 0x7ff84013e7a0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff83400c7c0 tx=0x7ff834007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834011590 con 0x7ff8400b0540 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8340116f0 con 0x7ff8400b0540 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834011ab0 con 0x7ff8400b0540 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff840155d60 con 0x7ff8400b0540 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.511+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84013ece0 con 0x7ff8400b0540 2023-10-07T11:01:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.512+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84004c3c0 con 0x7ff8400b0540 2023-10-07T11:01:44.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.513+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff83401b3b0 con 0x7ff8400b0540 2023-10-07T11:01:44.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.514+0000 7ff83cff9700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff824066f30 0x7ff8240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.514+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7ff834083b50 con 0x7ff8400b0540 2023-10-07T11:01:44.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.515+0000 7ff83ffff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff824066f30 0x7ff8240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.516+0000 7ff83ffff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff824066f30 0x7ff8240693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff84006c400 tx=0x7ff82c017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:44.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.519+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff834050c10 con 0x7ff8400b0540 2023-10-07T11:01:44.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.766+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84015a520 con 0x7ff8400b0540 2023-10-07T11:01:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.766+0000 7ff83cff9700 1 -- 172.21.15.29:0/2329304766 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4598 (secure 0 0 0) 0x7ff834017070 con 0x7ff8400b0540 2023-10-07T11:01:44.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:44.768 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:01:40.378766+0000","modified":"2023-10-07T11:01:44.282264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/562306367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":562306367},{"type":"v1","addr":"172.21.15.144:6837","nonce":562306367}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-10-07T11:01:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff824066f30 msgr2=0x7ff8240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff824066f30 0x7ff8240693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff84006c400 tx=0x7ff82c017000 comp rx=0 tx=0).stop 2023-10-07T11:01:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 msgr2=0x7ff84013e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 0x7ff84013e7a0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff83400c7c0 tx=0x7ff834007f70 comp rx=0 tx=0).stop 2023-10-07T11:01:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 shutdown_connections 2023-10-07T11:01:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff84015ade0 0x7ff8400c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8400b0540 0x7ff84013e7a0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff824066f30 0x7ff8240693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 --2- 172.21.15.29:0/2329304766 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff840153cc0 0x7ff840156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 >> 172.21.15.29:0/2329304766 conn(0x7ff840067610 msgr2=0x7ff840152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:44.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 shutdown_connections 2023-10-07T11:01:44.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.769+0000 7ff8471ff700 1 -- 172.21.15.29:0/2329304766 wait complete. 2023-10-07T11:01:44.772 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-07T11:01:44.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:01:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.983+0000 7f1a41fff700 1 Processor -- start 2023-10-07T11:01:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.983+0000 7f1a41fff700 1 -- start start 2023-10-07T11:01:44.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a41fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a3c157290 0x7f1a3c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a41fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1a3c0ae8b0 0x7f1a3c0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a41fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c0b1840 0x7f1a3c0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:44.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a41fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b1cc0 con 0x7f1a3c157290 2023-10-07T11:01:44.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a41fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c158ad0 con 0x7f1a3c0ae8b0 2023-10-07T11:01:44.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a41fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c15b140 con 0x7f1a3c0b1840 2023-10-07T11:01:44.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a3b7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a3c157290 0x7f1a3c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a3b7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a3c157290 0x7f1a3c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37068/0 (socket says 172.21.15.29:37068) 2023-10-07T11:01:44.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a3b7fe700 1 -- 172.21.15.29:0/1436491721 learned_addr learned my addr 172.21.15.29:0/1436491721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:44.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a3bfff700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c0b1840 0x7f1a3c0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.985+0000 7f1a3affd700 1 --2- 172.21.15.29:0/1436491721 >> v2:172.21.15.144:3300/0 conn(0x7f1a3c0ae8b0 0x7f1a3c0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:44.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3b7fe700 1 -- 172.21.15.29:0/1436491721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c0b1840 msgr2=0x7f1a3c0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:44.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c0b1840 0x7f1a3c0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3b7fe700 1 -- 172.21.15.29:0/1436491721 >> v2:172.21.15.144:3300/0 conn(0x7f1a3c0ae8b0 msgr2=0x7f1a3c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:44.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/1436491721 >> v2:172.21.15.144:3300/0 conn(0x7f1a3c0ae8b0 0x7f1a3c0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3b7fe700 1 -- 172.21.15.29:0/1436491721 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c15b420 con 0x7f1a3c157290 2023-10-07T11:01:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3affd700 1 --2- 172.21.15.29:0/1436491721 >> v2:172.21.15.144:3300/0 conn(0x7f1a3c0ae8b0 0x7f1a3c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3bfff700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c0b1840 0x7f1a3c0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c157290 0x7f1a3c0cc870 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0c8d90 tx=0x7f1a2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d45ca758cde0a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3a7fc700 1 -- 172.21.15.29:0/1436491721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00b860 con 0x7f1a3c157290 2023-10-07T11:01:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3a7fc700 1 -- 172.21.15.29:0/1436491721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c00b9c0 con 0x7f1a3c157290 2023-10-07T11:01:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.986+0000 7f1a3a7fc700 1 -- 172.21.15.29:0/1436491721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00bd60 con 0x7f1a3c157290 2023-10-07T11:01:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 -- 172.21.15.29:0/1436491721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c157290 msgr2=0x7f1a3c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c157290 0x7f1a3c0cc870 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0c8d90 tx=0x7f1a2c009250 comp rx=0 tx=0).stop 2023-10-07T11:01:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 -- 172.21.15.29:0/1436491721 shutdown_connections 2023-10-07T11:01:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c0b1840 0x7f1a3c0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 --2- 172.21.15.29:0/1436491721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c157290 0x7f1a3c0cc870 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 --2- 172.21.15.29:0/1436491721 >> v2:172.21.15.144:3300/0 conn(0x7f1a3c0ae8b0 0x7f1a3c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 -- 172.21.15.29:0/1436491721 >> 172.21.15.29:0/1436491721 conn(0x7f1a3c0680a0 msgr2=0x7f1a3c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:45.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 -- 172.21.15.29:0/1436491721 shutdown_connections 2023-10-07T11:01:45.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.987+0000 7f1a41fff700 1 -- 172.21.15.29:0/1436491721 wait complete. 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.989+0000 7f1a41fff700 1 Processor -- start 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.989+0000 7f1a41fff700 1 -- start start 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a41fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a3c0b1840 0x7f1a3c16a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a41fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c157290 0x7f1a3c0b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a41fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 0x7f1a3c14fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a41fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0affe0 con 0x7f1a3c155450 2023-10-07T11:01:45.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a41fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c1551a0 con 0x7f1a3c0b1840 2023-10-07T11:01:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a41fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0af0f0 con 0x7f1a3c157290 2023-10-07T11:01:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 0x7f1a3c14fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 0x7f1a3c14fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37078/0 (socket says 172.21.15.29:37078) 2023-10-07T11:01:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a3bfff700 1 -- 172.21.15.29:0/783926532 learned_addr learned my addr 172.21.15.29:0/783926532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a3c0b1840 0x7f1a3c16a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.990+0000 7f1a3affd700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c157290 0x7f1a3c0b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3bfff700 1 -- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c157290 msgr2=0x7f1a3c0b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3bfff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c157290 0x7f1a3c0b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3bfff700 1 -- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a3c0b1840 msgr2=0x7f1a3c16a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3bfff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a3c0b1840 0x7f1a3c16a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3bfff700 1 -- 172.21.15.29:0/783926532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0094f0 con 0x7f1a3c155450 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a3c0b1840 0x7f1a3c16a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3affd700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c157290 0x7f1a3c0b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a3bfff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 0x7f1a3c14fc10 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400d0d0 tx=0x7f1a340099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400dd70 con 0x7f1a3c155450 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3400f040 con 0x7f1a3c155450 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34015670 con 0x7f1a3c155450 2023-10-07T11:01:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c0af3d0 con 0x7f1a3c155450 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.991+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c0af990 con 0x7f1a3c155450 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.992+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c04c3c0 con 0x7f1a3c155450 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.993+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1a3401e3b0 con 0x7f1a3c155450 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.994+0000 7f1a237fe700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1a28066f30 0x7f1a280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.994+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f1a34084cf0 con 0x7f1a3c155450 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.995+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1a28066f30 0x7f1a280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.995+0000 7f1a3b7fe700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1a28066f30 0x7f1a280693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0c8d90 tx=0x7f1a2c017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:44.998+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a34051d00 con 0x7f1a3c155450 2023-10-07T11:01:45.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.202+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a3c1697f0 con 0x7f1a3c155450 2023-10-07T11:01:45.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.203+0000 7f1a237fe700 1 -- 172.21.15.29:0/783926532 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16148 (secure 0 0 0) 0x7f1a3401a090 con 0x7f1a3c155450 2023-10-07T11:01:45.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:45.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":182,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:01:41.884835+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:01:38.164128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-07T11:01:39.365755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:01:45.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1a28066f30 msgr2=0x7f1a280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1a28066f30 0x7f1a280693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0c8d90 tx=0x7f1a2c017000 comp rx=0 tx=0).stop 2023-10-07T11:01:45.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 msgr2=0x7f1a3c14fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 0x7f1a3c14fc10 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400d0d0 tx=0x7f1a340099d0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 shutdown_connections 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a3c155450 0x7f1a3c14fc10 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a3c0b1840 0x7f1a3c16a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1a28066f30 0x7f1a280693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 --2- 172.21.15.29:0/783926532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a3c157290 0x7f1a3c0b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 >> 172.21.15.29:0/783926532 conn(0x7f1a3c0680a0 msgr2=0x7f1a3c0cba10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.205+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 shutdown_connections 2023-10-07T11:01:45.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.206+0000 7f1a41fff700 1 -- 172.21.15.29:0/783926532 wait complete. 2023-10-07T11:01:45.223 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T11:01:45.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:01:45.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.420+0000 7f0a0cfff700 1 Processor -- start 2023-10-07T11:01:45.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.420+0000 7f0a0cfff700 1 -- start start 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a0cfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0a08159750 0x7f0a080d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a0cfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a08159b20 0x7f0a080ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a0cfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a080cbf20 con 0x7f0a080ce6d0 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a0cfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a080cc200 con 0x7f0a08159b20 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a0cfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a080cc4e0 con 0x7f0a08159750 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a05d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a08159b20 0x7f0a080ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a06d9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0a080ce6d0 0x7f0a0815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46874/0 (socket says 172.21.15.29:46874) 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.421+0000 7f0a06d9d700 1 -- 172.21.15.29:0/3871933084 learned_addr learned my addr 172.21.15.29:0/3871933084 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a0659c700 1 --2- 172.21.15.29:0/3871933084 >> v2:172.21.15.144:3300/0 conn(0x7f0a08159750 0x7f0a080d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a05d9b700 1 -- 172.21.15.29:0/3871933084 >> v2:172.21.15.144:3300/0 conn(0x7f0a08159750 msgr2=0x7f0a080d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:45.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a05d9b700 1 --2- 172.21.15.29:0/3871933084 >> v2:172.21.15.144:3300/0 conn(0x7f0a08159750 0x7f0a080d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a0559a700 1 -- 172.21.15.29:0/3871933084 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139759517 0 0) 0x7f0a080cbf20 con 0x7f0a080ce6d0 2023-10-07T11:01:45.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a05d9b700 1 -- 172.21.15.29:0/3871933084 >> v1:172.21.15.29:6789/0 conn(0x7f0a080ce6d0 legacy=0x7f0a0815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a05d9b700 1 -- 172.21.15.29:0/3871933084 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080cc7c0 con 0x7f0a08159b20 2023-10-07T11:01:45.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a05d9b700 1 --2- 172.21.15.29:0/3871933084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a08159b20 0x7f0a080ce190 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f0a08089840 tx=0x7f09fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3e7b794140a863 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a0559a700 1 -- 172.21.15.29:0/3871933084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc011630 con 0x7f0a08159b20 2023-10-07T11:01:45.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a0559a700 1 -- 172.21.15.29:0/3871933084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc011790 con 0x7f0a08159b20 2023-10-07T11:01:45.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.422+0000 7f0a0559a700 1 -- 172.21.15.29:0/3871933084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc011b30 con 0x7f0a08159b20 2023-10-07T11:01:45.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 -- 172.21.15.29:0/3871933084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a08159b20 msgr2=0x7f0a080ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/3871933084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a08159b20 0x7f0a080ce190 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f0a08089840 tx=0x7f09fc0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 -- 172.21.15.29:0/3871933084 shutdown_connections 2023-10-07T11:01:45.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/3871933084 >> v2:172.21.15.144:3300/0 conn(0x7f0a08159750 0x7f0a080d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/3871933084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a08159b20 0x7f0a080ce190 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 -- 172.21.15.29:0/3871933084 >> 172.21.15.29:0/3871933084 conn(0x7f0a08063a10 msgr2=0x7f0a080d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:45.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 -- 172.21.15.29:0/3871933084 shutdown_connections 2023-10-07T11:01:45.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.423+0000 7f0a0cfff700 1 -- 172.21.15.29:0/3871933084 wait complete. 2023-10-07T11:01:45.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.425+0000 7f0a0cfff700 1 Processor -- start 2023-10-07T11:01:45.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.425+0000 7f0a0cfff700 1 -- start start 2023-10-07T11:01:45.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0cfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 0x7f0a0816a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0cfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a08159750 0x7f0a0816c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0cfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a08159b20 0x7f0a080b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0cfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0816c8b0 con 0x7f0a080ce6d0 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0cfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0815eca0 con 0x7f0a08159750 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0cfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a08169830 con 0x7f0a08159b20 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 0x7f0a0816a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 0x7f0a0816a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37104/0 (socket says 172.21.15.29:37104) 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 -- 172.21.15.29:0/4026433361 learned_addr learned my addr 172.21.15.29:0/4026433361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 -- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a08159b20 msgr2=0x7f0a080b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a08159b20 0x7f0a080b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 -- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a08159750 msgr2=0x7f0a0816c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a05d9b700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a08159750 0x7f0a0816c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a08159750 0x7f0a0816c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.426+0000 7f0a0659c700 1 -- 172.21.15.29:0/4026433361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc009420 con 0x7f0a080ce6d0 2023-10-07T11:01:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.427+0000 7f0a0659c700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 0x7f0a0816a520 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f09f800ecb0 tx=0x7f09f8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.427+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800c9d0 con 0x7f0a080ce6d0 2023-10-07T11:01:45.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.427+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f800cb30 con 0x7f0a080ce6d0 2023-10-07T11:01:45.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.427+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800ceb0 con 0x7f0a080ce6d0 2023-10-07T11:01:45.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.427+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0816d680 con 0x7f0a080ce6d0 2023-10-07T11:01:45.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.427+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0815e930 con 0x7f0a080ce6d0 2023-10-07T11:01:45.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.428+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0804c3c0 con 0x7f0a080ce6d0 2023-10-07T11:01:45.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.429+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f09f8013680 con 0x7f0a080ce6d0 2023-10-07T11:01:45.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.430+0000 7f09f67fc700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09e8066f30 0x7f09e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.430+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f09f8084f00 con 0x7f0a080ce6d0 2023-10-07T11:01:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.431+0000 7f0a05d9b700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09e8066f30 0x7f09e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.431+0000 7f0a05d9b700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09e8066f30 0x7f09e80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0a0815d160 tx=0x7f09fc017000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.434+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09f8051f10 con 0x7f0a080ce6d0 2023-10-07T11:01:45.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.633+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a0815ede0 con 0x7f09e8066f30 2023-10-07T11:01:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.645+0000 7f09f67fc700 1 -- 172.21.15.29:0/4026433361 <== mgr.8069 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289609 (secure 0 0 0) 0x7f09fc000cb0 con 0x7f09e8066f30 2023-10-07T11:01:45.648 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:45.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09e8066f30 msgr2=0x7f09e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09e8066f30 0x7f09e80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0a0815d160 tx=0x7f09fc017000 comp rx=0 tx=0).stop 2023-10-07T11:01:45.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 msgr2=0x7f0a0816a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 0x7f0a0816a520 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f09f800ecb0 tx=0x7f09f8009120 comp rx=0 tx=0).stop 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 shutdown_connections 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09e8066f30 0x7f09e80693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a08159b20 0x7f0a080b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a080ce6d0 0x7f0a0816a520 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 --2- 172.21.15.29:0/4026433361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a08159750 0x7f0a0816c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 >> 172.21.15.29:0/4026433361 conn(0x7f0a08063a10 msgr2=0x7f0a080cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.649+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 shutdown_connections 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.650+0000 7f0a0cfff700 1 -- 172.21.15.29:0/4026433361 wait complete. 2023-10-07T11:01:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:01:45.666 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T11:01:45.085221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2470948,"kb_used_data":18916,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752503772,"statfs":{"total":773094113280,"available":770563862528,"internally_reserved":0,"allocated":19369984,"data_stored":10130508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001385"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899848+0000","last_change":"2023-10-07T11:01:40.898719+0000","last_active":"2023-10-07T11:01:40.899848+0000","last_peered":"2023-10-07T11:01:40.899848+0000","last_clean":"2023-10-07T11:01:40.899848+0000","last_became_active":"2023-10-07T11:01:40.898439+0000","last_became_peered":"2023-10-07T11:01:40.898439+0000","last_unstale":"2023-10-07T11:01:40.899848+0000","last_undegraded":"2023-10-07T11:01:40.899848+0000","last_fullsized":"2023-10-07T11:01:40.899848+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:08:25.405654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.897319+0000","last_change":"2023-10-07T11:01:40.897319+0000","last_active":"2023-10-07T11:01:40.897319+0000","last_peered":"2023-10-07T11:01:40.897319+0000","last_clean":"2023-10-07T11:01:40.897319+0000","last_became_active":"2023-10-07T11:01:40.896669+0000","last_became_peered":"2023-10-07T11:01:40.896669+0000","last_unstale":"2023-10-07T11:01:40.897319+0000","last_undegraded":"2023-10-07T11:01:40.897319+0000","last_fullsized":"2023-10-07T11:01:40.897319+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:41:56.209081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.920315+0000","last_change":"2023-10-07T11:01:40.917387+0000","last_active":"2023-10-07T11:01:40.920315+0000","last_peered":"2023-10-07T11:01:40.920315+0000","last_clean":"2023-10-07T11:01:40.920315+0000","last_became_active":"2023-10-07T11:01:40.917056+0000","last_became_peered":"2023-10-07T11:01:40.917056+0000","last_unstale":"2023-10-07T11:01:40.920315+0000","last_undegraded":"2023-10-07T11:01:40.920315+0000","last_fullsized":"2023-10-07T11:01:40.920315+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:49:36.503909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894479+0000","last_change":"2023-10-07T11:01:40.892293+0000","last_active":"2023-10-07T11:01:40.894479+0000","last_peered":"2023-10-07T11:01:40.894479+0000","last_clean":"2023-10-07T11:01:40.894479+0000","last_became_active":"2023-10-07T11:01:40.891930+0000","last_became_peered":"2023-10-07T11:01:40.891930+0000","last_unstale":"2023-10-07T11:01:40.894479+0000","last_undegraded":"2023-10-07T11:01:40.894479+0000","last_fullsized":"2023-10-07T11:01:40.894479+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:42.753786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908874+0000","last_change":"2023-10-07T11:01:40.908874+0000","last_active":"2023-10-07T11:01:40.908874+0000","last_peered":"2023-10-07T11:01:40.908874+0000","last_clean":"2023-10-07T11:01:40.908874+0000","last_became_active":"2023-10-07T11:01:40.908536+0000","last_became_peered":"2023-10-07T11:01:40.908536+0000","last_unstale":"2023-10-07T11:01:40.908874+0000","last_undegraded":"2023-10-07T11:01:40.908874+0000","last_fullsized":"2023-10-07T11:01:40.908874+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:18:09.190481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894571+0000","last_change":"2023-10-07T11:01:40.894571+0000","last_active":"2023-10-07T11:01:40.894571+0000","last_peered":"2023-10-07T11:01:40.894571+0000","last_clean":"2023-10-07T11:01:40.894571+0000","last_became_active":"2023-10-07T11:01:40.894301+0000","last_became_peered":"2023-10-07T11:01:40.894301+0000","last_unstale":"2023-10-07T11:01:40.894571+0000","last_undegraded":"2023-10-07T11:01:40.894571+0000","last_fullsized":"2023-10-07T11:01:40.894571+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:17:57.567241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909843+0000","last_change":"2023-10-07T11:01:40.908321+0000","last_active":"2023-10-07T11:01:40.909843+0000","last_peered":"2023-10-07T11:01:40.909843+0000","last_clean":"2023-10-07T11:01:40.909843+0000","last_became_active":"2023-10-07T11:01:40.908003+0000","last_became_peered":"2023-10-07T11:01:40.908003+0000","last_unstale":"2023-10-07T11:01:40.909843+0000","last_undegraded":"2023-10-07T11:01:40.909843+0000","last_fullsized":"2023-10-07T11:01:40.909843+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:28:52.542972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907141+0000","last_change":"2023-10-07T11:01:40.907141+0000","last_active":"2023-10-07T11:01:40.907141+0000","last_peered":"2023-10-07T11:01:40.907141+0000","last_clean":"2023-10-07T11:01:40.907141+0000","last_became_active":"2023-10-07T11:01:40.906852+0000","last_became_peered":"2023-10-07T11:01:40.906852+0000","last_unstale":"2023-10-07T11:01:40.907141+0000","last_undegraded":"2023-10-07T11:01:40.907141+0000","last_fullsized":"2023-10-07T11:01:40.907141+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:42.841471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899276+0000","last_change":"2023-10-07T11:01:40.899276+0000","last_active":"2023-10-07T11:01:40.899276+0000","last_peered":"2023-10-07T11:01:40.899276+0000","last_clean":"2023-10-07T11:01:40.899276+0000","last_became_active":"2023-10-07T11:01:40.898990+0000","last_became_peered":"2023-10-07T11:01:40.898990+0000","last_unstale":"2023-10-07T11:01:40.899276+0000","last_undegraded":"2023-10-07T11:01:40.899276+0000","last_fullsized":"2023-10-07T11:01:40.899276+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:28:52.542972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.893444+0000","last_change":"2023-10-07T11:01:40.893444+0000","last_active":"2023-10-07T11:01:40.893444+0000","last_peered":"2023-10-07T11:01:40.893444+0000","last_clean":"2023-10-07T11:01:40.893444+0000","last_became_active":"2023-10-07T11:01:40.893150+0000","last_became_peered":"2023-10-07T11:01:40.893150+0000","last_unstale":"2023-10-07T11:01:40.893444+0000","last_undegraded":"2023-10-07T11:01:40.893444+0000","last_fullsized":"2023-10-07T11:01:40.893444+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:23.019779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906716+0000","last_change":"2023-10-07T11:01:40.906716+0000","last_active":"2023-10-07T11:01:40.906716+0000","last_peered":"2023-10-07T11:01:40.906716+0000","last_clean":"2023-10-07T11:01:40.906716+0000","last_became_active":"2023-10-07T11:01:40.906422+0000","last_became_peered":"2023-10-07T11:01:40.906422+0000","last_unstale":"2023-10-07T11:01:40.906716+0000","last_undegraded":"2023-10-07T11:01:40.906716+0000","last_fullsized":"2023-10-07T11:01:40.906716+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:02.999386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.898713+0000","last_change":"2023-10-07T11:01:40.898713+0000","last_active":"2023-10-07T11:01:40.898713+0000","last_peered":"2023-10-07T11:01:40.898713+0000","last_clean":"2023-10-07T11:01:40.898713+0000","last_became_active":"2023-10-07T11:01:40.898406+0000","last_became_peered":"2023-10-07T11:01:40.898406+0000","last_unstale":"2023-10-07T11:01:40.898713+0000","last_undegraded":"2023-10-07T11:01:40.898713+0000","last_fullsized":"2023-10-07T11:01:40.898713+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:44:47.648930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.896471+0000","last_change":"2023-10-07T11:01:40.894448+0000","last_active":"2023-10-07T11:01:40.896471+0000","last_peered":"2023-10-07T11:01:40.896471+0000","last_clean":"2023-10-07T11:01:40.896471+0000","last_became_active":"2023-10-07T11:01:40.894133+0000","last_became_peered":"2023-10-07T11:01:40.894133+0000","last_unstale":"2023-10-07T11:01:40.896471+0000","last_undegraded":"2023-10-07T11:01:40.896471+0000","last_fullsized":"2023-10-07T11:01:40.896471+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:14.932469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909864+0000","last_change":"2023-10-07T11:01:40.903135+0000","last_active":"2023-10-07T11:01:40.909864+0000","last_peered":"2023-10-07T11:01:40.909864+0000","last_clean":"2023-10-07T11:01:40.909864+0000","last_became_active":"2023-10-07T11:01:40.902865+0000","last_became_peered":"2023-10-07T11:01:40.902865+0000","last_unstale":"2023-10-07T11:01:40.909864+0000","last_undegraded":"2023-10-07T11:01:40.909864+0000","last_fullsized":"2023-10-07T11:01:40.909864+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:36:29.347771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909544+0000","last_change":"2023-10-07T11:01:40.901369+0000","last_active":"2023-10-07T11:01:40.909544+0000","last_peered":"2023-10-07T11:01:40.909544+0000","last_clean":"2023-10-07T11:01:40.909544+0000","last_became_active":"2023-10-07T11:01:40.901087+0000","last_became_peered":"2023-10-07T11:01:40.901087+0000","last_unstale":"2023-10-07T11:01:40.909544+0000","last_undegraded":"2023-10-07T11:01:40.909544+0000","last_fullsized":"2023-10-07T11:01:40.909544+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:49:49.896100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899692+0000","last_change":"2023-10-07T11:01:40.898475+0000","last_active":"2023-10-07T11:01:40.899692+0000","last_peered":"2023-10-07T11:01:40.899692+0000","last_clean":"2023-10-07T11:01:40.899692+0000","last_became_active":"2023-10-07T11:01:40.898042+0000","last_became_peered":"2023-10-07T11:01:40.898042+0000","last_unstale":"2023-10-07T11:01:40.899692+0000","last_undegraded":"2023-10-07T11:01:40.899692+0000","last_fullsized":"2023-10-07T11:01:40.899692+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:02.999386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.898480+0000","last_change":"2023-10-07T11:01:40.898480+0000","last_active":"2023-10-07T11:01:40.898480+0000","last_peered":"2023-10-07T11:01:40.898480+0000","last_clean":"2023-10-07T11:01:40.898480+0000","last_became_active":"2023-10-07T11:01:40.898239+0000","last_became_peered":"2023-10-07T11:01:40.898239+0000","last_unstale":"2023-10-07T11:01:40.898480+0000","last_undegraded":"2023-10-07T11:01:40.898480+0000","last_fullsized":"2023-10-07T11:01:40.898480+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:03:26.863115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.892811+0000","last_change":"2023-10-07T11:01:40.886626+0000","last_active":"2023-10-07T11:01:40.892811+0000","last_peered":"2023-10-07T11:01:40.892811+0000","last_clean":"2023-10-07T11:01:40.892811+0000","last_became_active":"2023-10-07T11:01:40.886375+0000","last_became_peered":"2023-10-07T11:01:40.886375+0000","last_unstale":"2023-10-07T11:01:40.892811+0000","last_undegraded":"2023-10-07T11:01:40.892811+0000","last_fullsized":"2023-10-07T11:01:40.892811+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:29.275776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908365+0000","last_change":"2023-10-07T11:01:40.908365+0000","last_active":"2023-10-07T11:01:40.908365+0000","last_peered":"2023-10-07T11:01:40.908365+0000","last_clean":"2023-10-07T11:01:40.908365+0000","last_became_active":"2023-10-07T11:01:40.908046+0000","last_became_peered":"2023-10-07T11:01:40.908046+0000","last_unstale":"2023-10-07T11:01:40.908365+0000","last_undegraded":"2023-10-07T11:01:40.908365+0000","last_fullsized":"2023-10-07T11:01:40.908365+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:14.932469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909768+0000","last_change":"2023-10-07T11:01:40.902166+0000","last_active":"2023-10-07T11:01:40.909768+0000","last_peered":"2023-10-07T11:01:40.909768+0000","last_clean":"2023-10-07T11:01:40.909768+0000","last_became_active":"2023-10-07T11:01:40.901832+0000","last_became_peered":"2023-10-07T11:01:40.901832+0000","last_unstale":"2023-10-07T11:01:40.909768+0000","last_undegraded":"2023-10-07T11:01:40.909768+0000","last_fullsized":"2023-10-07T11:01:40.909768+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:53.307889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894342+0000","last_change":"2023-10-07T11:01:40.894342+0000","last_active":"2023-10-07T11:01:40.894342+0000","last_peered":"2023-10-07T11:01:40.894342+0000","last_clean":"2023-10-07T11:01:40.894342+0000","last_became_active":"2023-10-07T11:01:40.893957+0000","last_became_peered":"2023-10-07T11:01:40.893957+0000","last_unstale":"2023-10-07T11:01:40.894342+0000","last_undegraded":"2023-10-07T11:01:40.894342+0000","last_fullsized":"2023-10-07T11:01:40.894342+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:01:08.572660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.897435+0000","last_change":"2023-10-07T11:01:40.897435+0000","last_active":"2023-10-07T11:01:40.897435+0000","last_peered":"2023-10-07T11:01:40.897435+0000","last_clean":"2023-10-07T11:01:40.897435+0000","last_became_active":"2023-10-07T11:01:40.897111+0000","last_became_peered":"2023-10-07T11:01:40.897111+0000","last_unstale":"2023-10-07T11:01:40.897435+0000","last_undegraded":"2023-10-07T11:01:40.897435+0000","last_fullsized":"2023-10-07T11:01:40.897435+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:27.789649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899175+0000","last_change":"2023-10-07T11:01:40.899175+0000","last_active":"2023-10-07T11:01:40.899175+0000","last_peered":"2023-10-07T11:01:40.899175+0000","last_clean":"2023-10-07T11:01:40.899175+0000","last_became_active":"2023-10-07T11:01:40.898893+0000","last_became_peered":"2023-10-07T11:01:40.898893+0000","last_unstale":"2023-10-07T11:01:40.899175+0000","last_undegraded":"2023-10-07T11:01:40.899175+0000","last_fullsized":"2023-10-07T11:01:40.899175+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:12.265734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.898564+0000","last_change":"2023-10-07T11:01:40.898564+0000","last_active":"2023-10-07T11:01:40.898564+0000","last_peered":"2023-10-07T11:01:40.898564+0000","last_clean":"2023-10-07T11:01:40.898564+0000","last_became_active":"2023-10-07T11:01:40.898276+0000","last_became_peered":"2023-10-07T11:01:40.898276+0000","last_unstale":"2023-10-07T11:01:40.898564+0000","last_undegraded":"2023-10-07T11:01:40.898564+0000","last_fullsized":"2023-10-07T11:01:40.898564+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:24:04.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.920285+0000","last_change":"2023-10-07T11:01:40.917281+0000","last_active":"2023-10-07T11:01:40.920285+0000","last_peered":"2023-10-07T11:01:40.920285+0000","last_clean":"2023-10-07T11:01:40.920285+0000","last_became_active":"2023-10-07T11:01:40.916973+0000","last_became_peered":"2023-10-07T11:01:40.916973+0000","last_unstale":"2023-10-07T11:01:40.920285+0000","last_undegraded":"2023-10-07T11:01:40.920285+0000","last_fullsized":"2023-10-07T11:01:40.920285+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:12.265734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904756+0000","last_change":"2023-10-07T11:01:40.903013+0000","last_active":"2023-10-07T11:01:40.904756+0000","last_peered":"2023-10-07T11:01:40.904756+0000","last_clean":"2023-10-07T11:01:40.904756+0000","last_became_active":"2023-10-07T11:01:40.902757+0000","last_became_peered":"2023-10-07T11:01:40.902757+0000","last_unstale":"2023-10-07T11:01:40.904756+0000","last_undegraded":"2023-10-07T11:01:40.904756+0000","last_fullsized":"2023-10-07T11:01:40.904756+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:27.789649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909827+0000","last_change":"2023-10-07T11:01:40.901386+0000","last_active":"2023-10-07T11:01:40.909827+0000","last_peered":"2023-10-07T11:01:40.909827+0000","last_clean":"2023-10-07T11:01:40.909827+0000","last_became_active":"2023-10-07T11:01:40.901062+0000","last_became_peered":"2023-10-07T11:01:40.901062+0000","last_unstale":"2023-10-07T11:01:40.909827+0000","last_undegraded":"2023-10-07T11:01:40.909827+0000","last_fullsized":"2023-10-07T11:01:40.909827+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:03:26.863115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.914433+0000","last_change":"2023-10-07T11:01:40.913731+0000","last_active":"2023-10-07T11:01:40.914433+0000","last_peered":"2023-10-07T11:01:40.914433+0000","last_clean":"2023-10-07T11:01:40.914433+0000","last_became_active":"2023-10-07T11:01:40.913436+0000","last_became_peered":"2023-10-07T11:01:40.913436+0000","last_unstale":"2023-10-07T11:01:40.914433+0000","last_undegraded":"2023-10-07T11:01:40.914433+0000","last_fullsized":"2023-10-07T11:01:40.914433+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:17.329155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909795+0000","last_change":"2023-10-07T11:01:40.903056+0000","last_active":"2023-10-07T11:01:40.909795+0000","last_peered":"2023-10-07T11:01:40.909795+0000","last_clean":"2023-10-07T11:01:40.909795+0000","last_became_active":"2023-10-07T11:01:40.902482+0000","last_became_peered":"2023-10-07T11:01:40.902482+0000","last_unstale":"2023-10-07T11:01:40.909795+0000","last_undegraded":"2023-10-07T11:01:40.909795+0000","last_fullsized":"2023-10-07T11:01:40.909795+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:55.301881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.917404+0000","last_change":"2023-10-07T11:01:40.908466+0000","last_active":"2023-10-07T11:01:40.917404+0000","last_peered":"2023-10-07T11:01:40.917404+0000","last_clean":"2023-10-07T11:01:40.917404+0000","last_became_active":"2023-10-07T11:01:40.908207+0000","last_became_peered":"2023-10-07T11:01:40.908207+0000","last_unstale":"2023-10-07T11:01:40.917404+0000","last_undegraded":"2023-10-07T11:01:40.917404+0000","last_fullsized":"2023-10-07T11:01:40.917404+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:23.019779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916416+0000","last_change":"2023-10-07T11:01:40.904708+0000","last_active":"2023-10-07T11:01:40.916416+0000","last_peered":"2023-10-07T11:01:40.916416+0000","last_clean":"2023-10-07T11:01:40.916416+0000","last_became_active":"2023-10-07T11:01:40.904399+0000","last_became_peered":"2023-10-07T11:01:40.904399+0000","last_unstale":"2023-10-07T11:01:40.916416+0000","last_undegraded":"2023-10-07T11:01:40.916416+0000","last_fullsized":"2023-10-07T11:01:40.916416+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:06:56.195712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.892889+0000","last_change":"2023-10-07T11:01:40.886161+0000","last_active":"2023-10-07T11:01:40.892889+0000","last_peered":"2023-10-07T11:01:40.892889+0000","last_clean":"2023-10-07T11:01:40.892889+0000","last_became_active":"2023-10-07T11:01:40.885869+0000","last_became_peered":"2023-10-07T11:01:40.885869+0000","last_unstale":"2023-10-07T11:01:40.892889+0000","last_undegraded":"2023-10-07T11:01:40.892889+0000","last_fullsized":"2023-10-07T11:01:40.892889+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:55.980011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907491+0000","last_change":"2023-10-07T11:01:40.904834+0000","last_active":"2023-10-07T11:01:40.907491+0000","last_peered":"2023-10-07T11:01:40.907491+0000","last_clean":"2023-10-07T11:01:40.907491+0000","last_became_active":"2023-10-07T11:01:40.904562+0000","last_became_peered":"2023-10-07T11:01:40.904562+0000","last_unstale":"2023-10-07T11:01:40.907491+0000","last_undegraded":"2023-10-07T11:01:40.907491+0000","last_fullsized":"2023-10-07T11:01:40.907491+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:23.019779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"181'4","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.910507+0000","last_change":"2023-10-07T11:01:39.897441+0000","last_active":"2023-10-07T11:01:40.910507+0000","last_peered":"2023-10-07T11:01:40.910507+0000","last_clean":"2023-10-07T11:01:40.910507+0000","last_became_active":"2023-10-07T11:01:39.897149+0000","last_became_peered":"2023-10-07T11:01:39.897149+0000","last_unstale":"2023-10-07T11:01:40.910507+0000","last_undegraded":"2023-10-07T11:01:40.910507+0000","last_fullsized":"2023-10-07T11:01:40.910507+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:51:09.799332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916548+0000","last_change":"2023-10-07T11:01:40.906030+0000","last_active":"2023-10-07T11:01:40.916548+0000","last_peered":"2023-10-07T11:01:40.916548+0000","last_clean":"2023-10-07T11:01:40.916548+0000","last_became_active":"2023-10-07T11:01:40.905708+0000","last_became_peered":"2023-10-07T11:01:40.905708+0000","last_unstale":"2023-10-07T11:01:40.916548+0000","last_undegraded":"2023-10-07T11:01:40.916548+0000","last_fullsized":"2023-10-07T11:01:40.916548+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:33:30.913180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.913005+0000","last_change":"2023-10-07T11:01:39.896715+0000","last_active":"2023-10-07T11:01:40.913005+0000","last_peered":"2023-10-07T11:01:40.913005+0000","last_clean":"2023-10-07T11:01:40.913005+0000","last_became_active":"2023-10-07T11:01:39.896347+0000","last_became_peered":"2023-10-07T11:01:39.896347+0000","last_unstale":"2023-10-07T11:01:40.913005+0000","last_undegraded":"2023-10-07T11:01:40.913005+0000","last_fullsized":"2023-10-07T11:01:40.913005+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:13.871768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.897477+0000","last_change":"2023-10-07T11:01:40.897477+0000","last_active":"2023-10-07T11:01:40.897477+0000","last_peered":"2023-10-07T11:01:40.897477+0000","last_clean":"2023-10-07T11:01:40.897477+0000","last_became_active":"2023-10-07T11:01:40.897099+0000","last_became_peered":"2023-10-07T11:01:40.897099+0000","last_unstale":"2023-10-07T11:01:40.897477+0000","last_undegraded":"2023-10-07T11:01:40.897477+0000","last_fullsized":"2023-10-07T11:01:40.897477+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:53.307889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908001+0000","last_change":"2023-10-07T11:01:39.893732+0000","last_active":"2023-10-07T11:01:40.908001+0000","last_peered":"2023-10-07T11:01:40.908001+0000","last_clean":"2023-10-07T11:01:40.908001+0000","last_became_active":"2023-10-07T11:01:39.893469+0000","last_became_peered":"2023-10-07T11:01:39.893469+0000","last_unstale":"2023-10-07T11:01:40.908001+0000","last_undegraded":"2023-10-07T11:01:40.908001+0000","last_fullsized":"2023-10-07T11:01:40.908001+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:39:32.066938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906635+0000","last_change":"2023-10-07T11:01:40.897618+0000","last_active":"2023-10-07T11:01:40.906635+0000","last_peered":"2023-10-07T11:01:40.906635+0000","last_clean":"2023-10-07T11:01:40.906635+0000","last_became_active":"2023-10-07T11:01:40.897254+0000","last_became_peered":"2023-10-07T11:01:40.897254+0000","last_unstale":"2023-10-07T11:01:40.906635+0000","last_undegraded":"2023-10-07T11:01:40.906635+0000","last_fullsized":"2023-10-07T11:01:40.906635+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:17:57.567241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906639+0000","last_change":"2023-10-07T11:01:39.896498+0000","last_active":"2023-10-07T11:01:40.906639+0000","last_peered":"2023-10-07T11:01:40.906639+0000","last_clean":"2023-10-07T11:01:40.906639+0000","last_became_active":"2023-10-07T11:01:39.896246+0000","last_became_peered":"2023-10-07T11:01:39.896246+0000","last_unstale":"2023-10-07T11:01:40.906639+0000","last_undegraded":"2023-10-07T11:01:40.906639+0000","last_fullsized":"2023-10-07T11:01:40.906639+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:55:41.660809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899954+0000","last_change":"2023-10-07T11:01:40.896564+0000","last_active":"2023-10-07T11:01:40.899954+0000","last_peered":"2023-10-07T11:01:40.899954+0000","last_clean":"2023-10-07T11:01:40.899954+0000","last_became_active":"2023-10-07T11:01:40.895978+0000","last_became_peered":"2023-10-07T11:01:40.895978+0000","last_unstale":"2023-10-07T11:01:40.899954+0000","last_undegraded":"2023-10-07T11:01:40.899954+0000","last_fullsized":"2023-10-07T11:01:40.899954+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:25.409533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916766+0000","last_change":"2023-10-07T11:01:39.893437+0000","last_active":"2023-10-07T11:01:40.916766+0000","last_peered":"2023-10-07T11:01:40.916766+0000","last_clean":"2023-10-07T11:01:40.916766+0000","last_became_active":"2023-10-07T11:01:39.893131+0000","last_became_peered":"2023-10-07T11:01:39.893131+0000","last_unstale":"2023-10-07T11:01:40.916766+0000","last_undegraded":"2023-10-07T11:01:40.916766+0000","last_fullsized":"2023-10-07T11:01:40.916766+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:46.138415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905354+0000","last_change":"2023-10-07T11:01:40.901171+0000","last_active":"2023-10-07T11:01:40.905354+0000","last_peered":"2023-10-07T11:01:40.905354+0000","last_clean":"2023-10-07T11:01:40.905354+0000","last_became_active":"2023-10-07T11:01:40.900637+0000","last_became_peered":"2023-10-07T11:01:40.900637+0000","last_unstale":"2023-10-07T11:01:40.905354+0000","last_undegraded":"2023-10-07T11:01:40.905354+0000","last_fullsized":"2023-10-07T11:01:40.905354+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:55:14.047108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907201+0000","last_change":"2023-10-07T11:01:39.887481+0000","last_active":"2023-10-07T11:01:40.907201+0000","last_peered":"2023-10-07T11:01:40.907201+0000","last_clean":"2023-10-07T11:01:40.907201+0000","last_became_active":"2023-10-07T11:01:39.887188+0000","last_became_peered":"2023-10-07T11:01:39.887188+0000","last_unstale":"2023-10-07T11:01:40.907201+0000","last_undegraded":"2023-10-07T11:01:40.907201+0000","last_fullsized":"2023-10-07T11:01:40.907201+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:14.066979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905287+0000","last_change":"2023-10-07T11:01:40.902202+0000","last_active":"2023-10-07T11:01:40.905287+0000","last_peered":"2023-10-07T11:01:40.905287+0000","last_clean":"2023-10-07T11:01:40.905287+0000","last_became_active":"2023-10-07T11:01:40.901939+0000","last_became_peered":"2023-10-07T11:01:40.901939+0000","last_unstale":"2023-10-07T11:01:40.905287+0000","last_undegraded":"2023-10-07T11:01:40.905287+0000","last_fullsized":"2023-10-07T11:01:40.905287+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:01.893237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.893879+0000","last_change":"2023-10-07T11:01:39.890362+0000","last_active":"2023-10-07T11:01:39.893879+0000","last_peered":"2023-10-07T11:01:39.893879+0000","last_clean":"2023-10-07T11:01:39.893879+0000","last_became_active":"2023-10-07T11:01:39.890089+0000","last_became_peered":"2023-10-07T11:01:39.890089+0000","last_unstale":"2023-10-07T11:01:39.893879+0000","last_undegraded":"2023-10-07T11:01:39.893879+0000","last_fullsized":"2023-10-07T11:01:39.893879+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:11.402708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.895117+0000","last_change":"2023-10-07T11:01:40.895117+0000","last_active":"2023-10-07T11:01:40.895117+0000","last_peered":"2023-10-07T11:01:40.895117+0000","last_clean":"2023-10-07T11:01:40.895117+0000","last_became_active":"2023-10-07T11:01:40.894844+0000","last_became_peered":"2023-10-07T11:01:40.894844+0000","last_unstale":"2023-10-07T11:01:40.895117+0000","last_undegraded":"2023-10-07T11:01:40.895117+0000","last_fullsized":"2023-10-07T11:01:40.895117+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:55.301881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.893262+0000","last_change":"2023-10-07T11:01:39.891362+0000","last_active":"2023-10-07T11:01:40.893262+0000","last_peered":"2023-10-07T11:01:40.893262+0000","last_clean":"2023-10-07T11:01:40.893262+0000","last_became_active":"2023-10-07T11:01:39.890817+0000","last_became_peered":"2023-10-07T11:01:39.890817+0000","last_unstale":"2023-10-07T11:01:40.893262+0000","last_undegraded":"2023-10-07T11:01:40.893262+0000","last_fullsized":"2023-10-07T11:01:40.893262+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:11.402708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904271+0000","last_change":"2023-10-07T11:01:40.895008+0000","last_active":"2023-10-07T11:01:40.904271+0000","last_peered":"2023-10-07T11:01:40.904271+0000","last_clean":"2023-10-07T11:01:40.904271+0000","last_became_active":"2023-10-07T11:01:40.894713+0000","last_became_peered":"2023-10-07T11:01:40.894713+0000","last_unstale":"2023-10-07T11:01:40.904271+0000","last_undegraded":"2023-10-07T11:01:40.904271+0000","last_fullsized":"2023-10-07T11:01:40.904271+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:09:47.194488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.898310+0000","last_change":"2023-10-07T11:01:39.897630+0000","last_active":"2023-10-07T11:01:39.898310+0000","last_peered":"2023-10-07T11:01:39.898310+0000","last_clean":"2023-10-07T11:01:39.898310+0000","last_became_active":"2023-10-07T11:01:39.897352+0000","last_became_peered":"2023-10-07T11:01:39.897352+0000","last_unstale":"2023-10-07T11:01:39.898310+0000","last_undegraded":"2023-10-07T11:01:39.898310+0000","last_fullsized":"2023-10-07T11:01:39.898310+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:50:53.258627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904871+0000","last_change":"2023-10-07T11:01:40.895295+0000","last_active":"2023-10-07T11:01:40.904871+0000","last_peered":"2023-10-07T11:01:40.904871+0000","last_clean":"2023-10-07T11:01:40.904871+0000","last_became_active":"2023-10-07T11:01:40.894492+0000","last_became_peered":"2023-10-07T11:01:40.894492+0000","last_unstale":"2023-10-07T11:01:40.904871+0000","last_undegraded":"2023-10-07T11:01:40.904871+0000","last_fullsized":"2023-10-07T11:01:40.904871+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:55.094651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"181'3","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916973+0000","last_change":"2023-10-07T11:01:39.894456+0000","last_active":"2023-10-07T11:01:40.916973+0000","last_peered":"2023-10-07T11:01:40.916973+0000","last_clean":"2023-10-07T11:01:40.916973+0000","last_became_active":"2023-10-07T11:01:39.893538+0000","last_became_peered":"2023-10-07T11:01:39.893538+0000","last_unstale":"2023-10-07T11:01:40.916973+0000","last_undegraded":"2023-10-07T11:01:40.916973+0000","last_fullsized":"2023-10-07T11:01:40.916973+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:57.612441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916852+0000","last_change":"2023-10-07T11:01:40.907106+0000","last_active":"2023-10-07T11:01:40.916852+0000","last_peered":"2023-10-07T11:01:40.916852+0000","last_clean":"2023-10-07T11:01:40.916852+0000","last_became_active":"2023-10-07T11:01:40.905676+0000","last_became_peered":"2023-10-07T11:01:40.905676+0000","last_unstale":"2023-10-07T11:01:40.916852+0000","last_undegraded":"2023-10-07T11:01:40.916852+0000","last_fullsized":"2023-10-07T11:01:40.916852+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:25.409533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"181'6","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.920746+0000","last_change":"2023-10-07T11:01:39.896862+0000","last_active":"2023-10-07T11:01:40.920746+0000","last_peered":"2023-10-07T11:01:40.920746+0000","last_clean":"2023-10-07T11:01:40.920746+0000","last_became_active":"2023-10-07T11:01:39.896554+0000","last_became_peered":"2023-10-07T11:01:39.896554+0000","last_unstale":"2023-10-07T11:01:40.920746+0000","last_undegraded":"2023-10-07T11:01:40.920746+0000","last_fullsized":"2023-10-07T11:01:40.920746+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:00.884517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.888381+0000","last_change":"2023-10-07T11:01:40.885902+0000","last_active":"2023-10-07T11:01:40.888381+0000","last_peered":"2023-10-07T11:01:40.888381+0000","last_clean":"2023-10-07T11:01:40.888381+0000","last_became_active":"2023-10-07T11:01:40.885612+0000","last_became_peered":"2023-10-07T11:01:40.885612+0000","last_unstale":"2023-10-07T11:01:40.888381+0000","last_undegraded":"2023-10-07T11:01:40.888381+0000","last_fullsized":"2023-10-07T11:01:40.888381+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:24:36.242582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906880+0000","last_change":"2023-10-07T11:01:39.891597+0000","last_active":"2023-10-07T11:01:40.906880+0000","last_peered":"2023-10-07T11:01:40.906880+0000","last_clean":"2023-10-07T11:01:40.906880+0000","last_became_active":"2023-10-07T11:01:39.891342+0000","last_became_peered":"2023-10-07T11:01:39.891342+0000","last_unstale":"2023-10-07T11:01:40.906880+0000","last_undegraded":"2023-10-07T11:01:40.906880+0000","last_fullsized":"2023-10-07T11:01:40.906880+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:19.978253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.901460+0000","last_change":"2023-10-07T11:01:40.899846+0000","last_active":"2023-10-07T11:01:40.901460+0000","last_peered":"2023-10-07T11:01:40.901460+0000","last_clean":"2023-10-07T11:01:40.901460+0000","last_became_active":"2023-10-07T11:01:40.899546+0000","last_became_peered":"2023-10-07T11:01:40.899546+0000","last_unstale":"2023-10-07T11:01:40.901460+0000","last_undegraded":"2023-10-07T11:01:40.901460+0000","last_fullsized":"2023-10-07T11:01:40.901460+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:41:56.209081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.878511+0000","last_change":"2023-10-07T11:01:39.893328+0000","last_active":"2023-10-07T11:01:40.878511+0000","last_peered":"2023-10-07T11:01:40.878511+0000","last_clean":"2023-10-07T11:01:40.878511+0000","last_became_active":"2023-10-07T11:01:39.893046+0000","last_became_peered":"2023-10-07T11:01:39.893046+0000","last_unstale":"2023-10-07T11:01:40.878511+0000","last_undegraded":"2023-10-07T11:01:40.878511+0000","last_fullsized":"2023-10-07T11:01:40.878511+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:24.400814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.902300+0000","last_change":"2023-10-07T11:01:40.898230+0000","last_active":"2023-10-07T11:01:40.902300+0000","last_peered":"2023-10-07T11:01:40.902300+0000","last_clean":"2023-10-07T11:01:40.902300+0000","last_became_active":"2023-10-07T11:01:40.897936+0000","last_became_peered":"2023-10-07T11:01:40.897936+0000","last_unstale":"2023-10-07T11:01:40.902300+0000","last_undegraded":"2023-10-07T11:01:40.902300+0000","last_fullsized":"2023-10-07T11:01:40.902300+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:16.573994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.883708+0000","last_change":"2023-10-07T11:01:39.893308+0000","last_active":"2023-10-07T11:01:40.883708+0000","last_peered":"2023-10-07T11:01:40.883708+0000","last_clean":"2023-10-07T11:01:40.883708+0000","last_became_active":"2023-10-07T11:01:39.892699+0000","last_became_peered":"2023-10-07T11:01:39.892699+0000","last_unstale":"2023-10-07T11:01:40.883708+0000","last_undegraded":"2023-10-07T11:01:40.883708+0000","last_fullsized":"2023-10-07T11:01:40.883708+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:06:27.259425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909658+0000","last_change":"2023-10-07T11:01:40.902484+0000","last_active":"2023-10-07T11:01:40.909658+0000","last_peered":"2023-10-07T11:01:40.909658+0000","last_clean":"2023-10-07T11:01:40.909658+0000","last_became_active":"2023-10-07T11:01:40.902210+0000","last_became_peered":"2023-10-07T11:01:40.902210+0000","last_unstale":"2023-10-07T11:01:40.909658+0000","last_undegraded":"2023-10-07T11:01:40.909658+0000","last_fullsized":"2023-10-07T11:01:40.909658+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:03.966096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"181'8","reported_seq":18,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.917063+0000","last_change":"2023-10-07T11:01:39.895337+0000","last_active":"2023-10-07T11:01:40.917063+0000","last_peered":"2023-10-07T11:01:40.917063+0000","last_clean":"2023-10-07T11:01:40.917063+0000","last_became_active":"2023-10-07T11:01:39.894853+0000","last_became_peered":"2023-10-07T11:01:39.894853+0000","last_unstale":"2023-10-07T11:01:40.917063+0000","last_undegraded":"2023-10-07T11:01:40.917063+0000","last_fullsized":"2023-10-07T11:01:40.917063+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:46:37.971882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894558+0000","last_change":"2023-10-07T11:01:40.892553+0000","last_active":"2023-10-07T11:01:40.894558+0000","last_peered":"2023-10-07T11:01:40.894558+0000","last_clean":"2023-10-07T11:01:40.894558+0000","last_became_active":"2023-10-07T11:01:40.892315+0000","last_became_peered":"2023-10-07T11:01:40.892315+0000","last_unstale":"2023-10-07T11:01:40.894558+0000","last_undegraded":"2023-10-07T11:01:40.894558+0000","last_fullsized":"2023-10-07T11:01:40.894558+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:34.275253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908302+0000","last_change":"2023-10-07T11:01:39.894287+0000","last_active":"2023-10-07T11:01:40.908302+0000","last_peered":"2023-10-07T11:01:40.908302+0000","last_clean":"2023-10-07T11:01:40.908302+0000","last_became_active":"2023-10-07T11:01:39.893993+0000","last_became_peered":"2023-10-07T11:01:39.893993+0000","last_unstale":"2023-10-07T11:01:40.908302+0000","last_undegraded":"2023-10-07T11:01:40.908302+0000","last_fullsized":"2023-10-07T11:01:40.908302+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:17:53.154186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906899+0000","last_change":"2023-10-07T11:01:40.905633+0000","last_active":"2023-10-07T11:01:40.906899+0000","last_peered":"2023-10-07T11:01:40.906899+0000","last_clean":"2023-10-07T11:01:40.906899+0000","last_became_active":"2023-10-07T11:01:40.905375+0000","last_became_peered":"2023-10-07T11:01:40.905375+0000","last_unstale":"2023-10-07T11:01:40.906899+0000","last_undegraded":"2023-10-07T11:01:40.906899+0000","last_fullsized":"2023-10-07T11:01:40.906899+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:31:27.470060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905966+0000","last_change":"2023-10-07T11:01:39.893990+0000","last_active":"2023-10-07T11:01:40.905966+0000","last_peered":"2023-10-07T11:01:40.905966+0000","last_clean":"2023-10-07T11:01:40.905966+0000","last_became_active":"2023-10-07T11:01:39.893678+0000","last_became_peered":"2023-10-07T11:01:39.893678+0000","last_unstale":"2023-10-07T11:01:40.905966+0000","last_undegraded":"2023-10-07T11:01:40.905966+0000","last_fullsized":"2023-10-07T11:01:40.905966+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:47:23.003410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.903770+0000","last_change":"2023-10-07T11:01:40.899310+0000","last_active":"2023-10-07T11:01:40.903770+0000","last_peered":"2023-10-07T11:01:40.903770+0000","last_clean":"2023-10-07T11:01:40.903770+0000","last_became_active":"2023-10-07T11:01:40.898812+0000","last_became_peered":"2023-10-07T11:01:40.898812+0000","last_unstale":"2023-10-07T11:01:40.903770+0000","last_undegraded":"2023-10-07T11:01:40.903770+0000","last_fullsized":"2023-10-07T11:01:40.903770+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:46:38.980601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.917929+0000","last_change":"2023-10-07T11:01:39.892641+0000","last_active":"2023-10-07T11:01:40.917929+0000","last_peered":"2023-10-07T11:01:40.917929+0000","last_clean":"2023-10-07T11:01:40.917929+0000","last_became_active":"2023-10-07T11:01:39.892378+0000","last_became_peered":"2023-10-07T11:01:39.892378+0000","last_unstale":"2023-10-07T11:01:40.917929+0000","last_undegraded":"2023-10-07T11:01:40.917929+0000","last_fullsized":"2023-10-07T11:01:40.917929+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:30:31.416388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.914124+0000","last_change":"2023-10-07T11:01:40.913548+0000","last_active":"2023-10-07T11:01:40.914124+0000","last_peered":"2023-10-07T11:01:40.914124+0000","last_clean":"2023-10-07T11:01:40.914124+0000","last_became_active":"2023-10-07T11:01:40.913192+0000","last_became_peered":"2023-10-07T11:01:40.913192+0000","last_unstale":"2023-10-07T11:01:40.914124+0000","last_undegraded":"2023-10-07T11:01:40.914124+0000","last_fullsized":"2023-10-07T11:01:40.914124+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:44:47.648930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.886990+0000","last_change":"2023-10-07T11:01:39.898387+0000","last_active":"2023-10-07T11:01:40.886990+0000","last_peered":"2023-10-07T11:01:40.886990+0000","last_clean":"2023-10-07T11:01:40.886990+0000","last_became_active":"2023-10-07T11:01:39.897799+0000","last_became_peered":"2023-10-07T11:01:39.897799+0000","last_unstale":"2023-10-07T11:01:40.886990+0000","last_undegraded":"2023-10-07T11:01:40.886990+0000","last_fullsized":"2023-10-07T11:01:40.886990+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:17.662100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.896018+0000","last_change":"2023-10-07T11:01:40.894754+0000","last_active":"2023-10-07T11:01:40.896018+0000","last_peered":"2023-10-07T11:01:40.896018+0000","last_clean":"2023-10-07T11:01:40.896018+0000","last_became_active":"2023-10-07T11:01:40.893935+0000","last_became_peered":"2023-10-07T11:01:40.893935+0000","last_unstale":"2023-10-07T11:01:40.896018+0000","last_undegraded":"2023-10-07T11:01:40.896018+0000","last_fullsized":"2023-10-07T11:01:40.896018+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:55.421293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907684+0000","last_change":"2023-10-07T11:01:39.893156+0000","last_active":"2023-10-07T11:01:40.907684+0000","last_peered":"2023-10-07T11:01:40.907684+0000","last_clean":"2023-10-07T11:01:40.907684+0000","last_became_active":"2023-10-07T11:01:39.892865+0000","last_became_peered":"2023-10-07T11:01:39.892865+0000","last_unstale":"2023-10-07T11:01:40.907684+0000","last_undegraded":"2023-10-07T11:01:40.907684+0000","last_fullsized":"2023-10-07T11:01:40.907684+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:05.206253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899735+0000","last_change":"2023-10-07T11:01:40.897430+0000","last_active":"2023-10-07T11:01:40.899735+0000","last_peered":"2023-10-07T11:01:40.899735+0000","last_clean":"2023-10-07T11:01:40.899735+0000","last_became_active":"2023-10-07T11:01:40.897188+0000","last_became_peered":"2023-10-07T11:01:40.897188+0000","last_unstale":"2023-10-07T11:01:40.899735+0000","last_undegraded":"2023-10-07T11:01:40.899735+0000","last_fullsized":"2023-10-07T11:01:40.899735+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:18:09.190481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.884729+0000","last_change":"2023-10-07T11:01:39.893494+0000","last_active":"2023-10-07T11:01:40.884729+0000","last_peered":"2023-10-07T11:01:40.884729+0000","last_clean":"2023-10-07T11:01:40.884729+0000","last_became_active":"2023-10-07T11:01:39.893237+0000","last_became_peered":"2023-10-07T11:01:39.893237+0000","last_unstale":"2023-10-07T11:01:40.884729+0000","last_undegraded":"2023-10-07T11:01:40.884729+0000","last_fullsized":"2023-10-07T11:01:40.884729+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:28.084936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905249+0000","last_change":"2023-10-07T11:01:40.900558+0000","last_active":"2023-10-07T11:01:40.905249+0000","last_peered":"2023-10-07T11:01:40.905249+0000","last_clean":"2023-10-07T11:01:40.905249+0000","last_became_active":"2023-10-07T11:01:40.900201+0000","last_became_peered":"2023-10-07T11:01:40.900201+0000","last_unstale":"2023-10-07T11:01:40.905249+0000","last_undegraded":"2023-10-07T11:01:40.905249+0000","last_fullsized":"2023-10-07T11:01:40.905249+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:29:43.402726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.883830+0000","last_change":"2023-10-07T11:01:39.895223+0000","last_active":"2023-10-07T11:01:40.883830+0000","last_peered":"2023-10-07T11:01:40.883830+0000","last_clean":"2023-10-07T11:01:40.883830+0000","last_became_active":"2023-10-07T11:01:39.894886+0000","last_became_peered":"2023-10-07T11:01:39.894886+0000","last_unstale":"2023-10-07T11:01:40.883830+0000","last_undegraded":"2023-10-07T11:01:40.883830+0000","last_fullsized":"2023-10-07T11:01:40.883830+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:28:17.315711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.890988+0000","last_change":"2023-10-07T11:01:40.890988+0000","last_active":"2023-10-07T11:01:40.890988+0000","last_peered":"2023-10-07T11:01:40.890988+0000","last_clean":"2023-10-07T11:01:40.890988+0000","last_became_active":"2023-10-07T11:01:40.890709+0000","last_became_peered":"2023-10-07T11:01:40.890709+0000","last_unstale":"2023-10-07T11:01:40.890988+0000","last_undegraded":"2023-10-07T11:01:40.890988+0000","last_fullsized":"2023-10-07T11:01:40.890988+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:55.421293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.889381+0000","last_change":"2023-10-07T11:01:39.892309+0000","last_active":"2023-10-07T11:01:40.889381+0000","last_peered":"2023-10-07T11:01:40.889381+0000","last_clean":"2023-10-07T11:01:40.889381+0000","last_became_active":"2023-10-07T11:01:39.892035+0000","last_became_peered":"2023-10-07T11:01:39.892035+0000","last_unstale":"2023-10-07T11:01:40.889381+0000","last_undegraded":"2023-10-07T11:01:40.889381+0000","last_fullsized":"2023-10-07T11:01:40.889381+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:16:13.649531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904783+0000","last_change":"2023-10-07T11:01:40.894536+0000","last_active":"2023-10-07T11:01:40.904783+0000","last_peered":"2023-10-07T11:01:40.904783+0000","last_clean":"2023-10-07T11:01:40.904783+0000","last_became_active":"2023-10-07T11:01:40.894101+0000","last_became_peered":"2023-10-07T11:01:40.894101+0000","last_unstale":"2023-10-07T11:01:40.904783+0000","last_undegraded":"2023-10-07T11:01:40.904783+0000","last_fullsized":"2023-10-07T11:01:40.904783+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:33:06.317400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908686+0000","last_change":"2023-10-07T11:01:39.888670+0000","last_active":"2023-10-07T11:01:40.908686+0000","last_peered":"2023-10-07T11:01:40.908686+0000","last_clean":"2023-10-07T11:01:40.908686+0000","last_became_active":"2023-10-07T11:01:39.888051+0000","last_became_peered":"2023-10-07T11:01:39.888051+0000","last_unstale":"2023-10-07T11:01:40.908686+0000","last_undegraded":"2023-10-07T11:01:40.908686+0000","last_fullsized":"2023-10-07T11:01:40.908686+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:26.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907379+0000","last_change":"2023-10-07T11:01:40.904618+0000","last_active":"2023-10-07T11:01:40.907379+0000","last_peered":"2023-10-07T11:01:40.907379+0000","last_clean":"2023-10-07T11:01:40.907379+0000","last_became_active":"2023-10-07T11:01:40.904338+0000","last_became_peered":"2023-10-07T11:01:40.904338+0000","last_unstale":"2023-10-07T11:01:40.907379+0000","last_undegraded":"2023-10-07T11:01:40.907379+0000","last_fullsized":"2023-10-07T11:01:40.907379+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:42.841471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.889924+0000","last_change":"2023-10-07T11:01:39.893656+0000","last_active":"2023-10-07T11:01:40.889924+0000","last_peered":"2023-10-07T11:01:40.889924+0000","last_clean":"2023-10-07T11:01:40.889924+0000","last_became_active":"2023-10-07T11:01:39.893324+0000","last_became_peered":"2023-10-07T11:01:39.893324+0000","last_unstale":"2023-10-07T11:01:40.889924+0000","last_undegraded":"2023-10-07T11:01:40.889924+0000","last_fullsized":"2023-10-07T11:01:40.889924+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:28:47.774275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.890127+0000","last_change":"2023-10-07T11:01:40.890127+0000","last_active":"2023-10-07T11:01:40.890127+0000","last_peered":"2023-10-07T11:01:40.890127+0000","last_clean":"2023-10-07T11:01:40.890127+0000","last_became_active":"2023-10-07T11:01:40.889867+0000","last_became_peered":"2023-10-07T11:01:40.889867+0000","last_unstale":"2023-10-07T11:01:40.890127+0000","last_undegraded":"2023-10-07T11:01:40.890127+0000","last_fullsized":"2023-10-07T11:01:40.890127+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:49:36.503909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.892843+0000","last_change":"2023-10-07T11:01:39.888218+0000","last_active":"2023-10-07T11:01:39.892843+0000","last_peered":"2023-10-07T11:01:39.892843+0000","last_clean":"2023-10-07T11:01:39.892843+0000","last_became_active":"2023-10-07T11:01:39.887891+0000","last_became_peered":"2023-10-07T11:01:39.887891+0000","last_unstale":"2023-10-07T11:01:39.892843+0000","last_undegraded":"2023-10-07T11:01:39.892843+0000","last_fullsized":"2023-10-07T11:01:39.892843+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:46:37.971882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906273+0000","last_change":"2023-10-07T11:01:40.899153+0000","last_active":"2023-10-07T11:01:40.906273+0000","last_peered":"2023-10-07T11:01:40.906273+0000","last_clean":"2023-10-07T11:01:40.906273+0000","last_became_active":"2023-10-07T11:01:40.898895+0000","last_became_peered":"2023-10-07T11:01:40.898895+0000","last_unstale":"2023-10-07T11:01:40.906273+0000","last_undegraded":"2023-10-07T11:01:40.906273+0000","last_fullsized":"2023-10-07T11:01:40.906273+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:59:04.178092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.877735+0000","last_change":"2023-10-07T11:01:39.895225+0000","last_active":"2023-10-07T11:01:40.877735+0000","last_peered":"2023-10-07T11:01:40.877735+0000","last_clean":"2023-10-07T11:01:40.877735+0000","last_became_active":"2023-10-07T11:01:39.894953+0000","last_became_peered":"2023-10-07T11:01:39.894953+0000","last_unstale":"2023-10-07T11:01:40.877735+0000","last_undegraded":"2023-10-07T11:01:40.877735+0000","last_fullsized":"2023-10-07T11:01:40.877735+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.011059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.888363+0000","last_change":"2023-10-07T11:01:40.885348+0000","last_active":"2023-10-07T11:01:40.888363+0000","last_peered":"2023-10-07T11:01:40.888363+0000","last_clean":"2023-10-07T11:01:40.888363+0000","last_became_active":"2023-10-07T11:01:40.884354+0000","last_became_peered":"2023-10-07T11:01:40.884354+0000","last_unstale":"2023-10-07T11:01:40.888363+0000","last_undegraded":"2023-10-07T11:01:40.888363+0000","last_fullsized":"2023-10-07T11:01:40.888363+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:58:08.788067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.890481+0000","last_change":"2023-10-07T11:01:39.894328+0000","last_active":"2023-10-07T11:01:40.890481+0000","last_peered":"2023-10-07T11:01:40.890481+0000","last_clean":"2023-10-07T11:01:40.890481+0000","last_became_active":"2023-10-07T11:01:39.893502+0000","last_became_peered":"2023-10-07T11:01:39.893502+0000","last_unstale":"2023-10-07T11:01:40.890481+0000","last_undegraded":"2023-10-07T11:01:40.890481+0000","last_fullsized":"2023-10-07T11:01:40.890481+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:11.402708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916866+0000","last_change":"2023-10-07T11:01:40.906946+0000","last_active":"2023-10-07T11:01:40.916866+0000","last_peered":"2023-10-07T11:01:40.916866+0000","last_clean":"2023-10-07T11:01:40.916866+0000","last_became_active":"2023-10-07T11:01:40.906638+0000","last_became_peered":"2023-10-07T11:01:40.906638+0000","last_unstale":"2023-10-07T11:01:40.916866+0000","last_undegraded":"2023-10-07T11:01:40.916866+0000","last_fullsized":"2023-10-07T11:01:40.916866+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:46:36.726101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.882071+0000","last_change":"2023-10-07T11:01:39.893190+0000","last_active":"2023-10-07T11:01:40.882071+0000","last_peered":"2023-10-07T11:01:40.882071+0000","last_clean":"2023-10-07T11:01:40.882071+0000","last_became_active":"2023-10-07T11:01:39.892914+0000","last_became_peered":"2023-10-07T11:01:39.892914+0000","last_unstale":"2023-10-07T11:01:40.882071+0000","last_undegraded":"2023-10-07T11:01:40.882071+0000","last_fullsized":"2023-10-07T11:01:40.882071+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:56.289200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899673+0000","last_change":"2023-10-07T11:01:40.898304+0000","last_active":"2023-10-07T11:01:40.899673+0000","last_peered":"2023-10-07T11:01:40.899673+0000","last_clean":"2023-10-07T11:01:40.899673+0000","last_became_active":"2023-10-07T11:01:40.897693+0000","last_became_peered":"2023-10-07T11:01:40.897693+0000","last_unstale":"2023-10-07T11:01:40.899673+0000","last_undegraded":"2023-10-07T11:01:40.899673+0000","last_fullsized":"2023-10-07T11:01:40.899673+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:24:04.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.893860+0000","last_change":"2023-10-07T11:01:39.888456+0000","last_active":"2023-10-07T11:01:39.893860+0000","last_peered":"2023-10-07T11:01:39.893860+0000","last_clean":"2023-10-07T11:01:39.893860+0000","last_became_active":"2023-10-07T11:01:39.888125+0000","last_became_peered":"2023-10-07T11:01:39.888125+0000","last_unstale":"2023-10-07T11:01:39.893860+0000","last_undegraded":"2023-10-07T11:01:39.893860+0000","last_fullsized":"2023-10-07T11:01:39.893860+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:00.009923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908015+0000","last_change":"2023-10-07T11:01:40.897927+0000","last_active":"2023-10-07T11:01:40.908015+0000","last_peered":"2023-10-07T11:01:40.908015+0000","last_clean":"2023-10-07T11:01:40.908015+0000","last_became_active":"2023-10-07T11:01:40.897541+0000","last_became_peered":"2023-10-07T11:01:40.897541+0000","last_unstale":"2023-10-07T11:01:40.908015+0000","last_undegraded":"2023-10-07T11:01:40.908015+0000","last_fullsized":"2023-10-07T11:01:40.908015+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:01:08.572660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.887886+0000","last_change":"2023-10-07T11:01:39.890605+0000","last_active":"2023-10-07T11:01:40.887886+0000","last_peered":"2023-10-07T11:01:40.887886+0000","last_clean":"2023-10-07T11:01:40.887886+0000","last_became_active":"2023-10-07T11:01:39.890340+0000","last_became_peered":"2023-10-07T11:01:39.890340+0000","last_unstale":"2023-10-07T11:01:40.887886+0000","last_undegraded":"2023-10-07T11:01:40.887886+0000","last_fullsized":"2023-10-07T11:01:40.887886+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:00.009923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904857+0000","last_change":"2023-10-07T11:01:40.894957+0000","last_active":"2023-10-07T11:01:40.904857+0000","last_peered":"2023-10-07T11:01:40.904857+0000","last_clean":"2023-10-07T11:01:40.904857+0000","last_became_active":"2023-10-07T11:01:40.894578+0000","last_became_peered":"2023-10-07T11:01:40.894578+0000","last_unstale":"2023-10-07T11:01:40.904857+0000","last_undegraded":"2023-10-07T11:01:40.904857+0000","last_fullsized":"2023-10-07T11:01:40.904857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:06:30.815321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.888086+0000","last_change":"2023-10-07T11:01:39.899669+0000","last_active":"2023-10-07T11:01:40.888086+0000","last_peered":"2023-10-07T11:01:40.888086+0000","last_clean":"2023-10-07T11:01:40.888086+0000","last_became_active":"2023-10-07T11:01:39.899320+0000","last_became_peered":"2023-10-07T11:01:39.899320+0000","last_unstale":"2023-10-07T11:01:40.888086+0000","last_undegraded":"2023-10-07T11:01:40.888086+0000","last_fullsized":"2023-10-07T11:01:40.888086+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:57.980154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"172'326","reported_seq":1041,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.889468+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:01:40.889468+0000","last_peered":"2023-10-07T11:01:40.889468+0000","last_clean":"2023-10-07T11:01:40.889468+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:01:40.889468+0000","last_undegraded":"2023-10-07T11:01:40.889468+0000","last_fullsized":"2023-10-07T11:01:40.889468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477835,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312396,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059444,"statfs":{"total":96636764160,"available":96316870656,"internally_reserved":0,"allocated":2887680,"data_stored":1713289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.748,"15min":0.747},"min":{"1min":0.47699999999999998,"5min":0.40400000000000003,"15min":0.31},"max":{"1min":0.94499999999999995,"5min":0.98999999999999999,"15min":1.1259999999999999},"last":0.504},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77900000000000003,"15min":0.76200000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.41599999999999998},"max":{"1min":0.98599999999999999,"5min":1.016,"15min":1.071},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77500000000000002,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.40000000000000002},"max":{"1min":0.92600000000000005,"5min":1.02,"15min":1.0780000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.76600000000000001,"15min":0.76700000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.44},"max":{"1min":0.93799999999999994,"5min":1.0009999999999999,"15min":1.0880000000000001},"last":0.625}]},{"osd":2,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77700000000000002,"15min":0.77500000000000002},"min":{"1min":0.52700000000000002,"5min":0.503,"15min":0.38900000000000001},"max":{"1min":0.96299999999999997,"5min":1.0149999999999999,"15min":1.1000000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.54300000000000004,"5min":0.47999999999999998,"15min":0.46800000000000003},"max":{"1min":1.079,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76300000000000001,"15min":0.745},"min":{"1min":0.56799999999999995,"5min":0.501,"15min":0.47799999999999998},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.80600000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.48199999999999998},"max":{"1min":0.97599999999999998,"5min":1.0509999999999999,"15min":1.0669999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.35899999999999999},"max":{"1min":0.91100000000000003,"5min":0.999,"15min":0.999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"max":{"1min":0.90200000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.877}]},{"osd":5,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68700000000000006,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92200000000000004,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.754},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.48199999999999998,"5min":0.435,"15min":0.435},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.68700000000000006}]},{"osd":6,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78300000000000003,"15min":0.76900000000000002},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.038,"5min":1.0660000000000001,"15min":1.0669999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73799999999999999,"15min":0.73599999999999999},"min":{"1min":0.47899999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.85599999999999998}]}]},{"osd":6,"up_from":15,"seq":64424510539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315880,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055960,"statfs":{"total":96636764160,"available":96313303040,"internally_reserved":0,"allocated":2260992,"data_stored":1113957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.57799999999999996,"5min":0.45300000000000001,"15min":0.378},"max":{"1min":0.95799999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.52400000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":1.002},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.749,"15min":0.751},"min":{"1min":0.58099999999999996,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":1.048},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.55600000000000005,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84799999999999998,"5min":0.97499999999999998,"15min":1.0409999999999999},"last":0.65900000000000003}]},{"osd":2,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.65600000000000003,"5min":0.54200000000000004,"15min":0.46899999999999997},"max":{"1min":0.871,"5min":1.018,"15min":1.0189999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.55500000000000005,"5min":0.52600000000000002,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.98599999999999999,"15min":1.0069999999999999},"last":0.79500000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77400000000000002,"15min":0.77300000000000002},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.504},"max":{"1min":0.90200000000000002,"5min":1.073,"15min":1.073},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.69699999999999995,"5min":0.53300000000000003,"15min":0.52300000000000002},"max":{"1min":0.91500000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.46000000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78600000000000003,"5min":0.91100000000000003,"15min":0.97799999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66500000000000004,"15min":0.65300000000000002},"min":{"1min":0.48299999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.97099999999999997,"5min":1.012,"15min":1.0149999999999999},"last":0.77500000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67200000000000004},"min":{"1min":0.51000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84099999999999997,"5min":0.96599999999999997,"15min":0.995},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.71799999999999997},"min":{"1min":0.53200000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.022,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84399999999999997}]},{"osd":7,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67100000000000004,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95499999999999996,"5min":1.0760000000000001,"15min":1.1120000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.754,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.48199999999999998},"max":{"1min":1,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81299999999999994}]}]},{"osd":1,"up_from":14,"seq":60129543241,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307688,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064152,"statfs":{"total":96636764160,"available":96321691648,"internally_reserved":0,"allocated":2260992,"data_stored":1113969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.97599999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76800000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.39700000000000002},"max":{"1min":0.76400000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63800000000000001,"15min":0.64500000000000002},"min":{"1min":0.45800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80400000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.075,"5min":1.1539999999999999,"15min":1.373},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80300000000000005,"15min":0.81299999999999994},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.085,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.871}]},{"osd":5,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.84499999999999997,"15min":0.85299999999999998},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.90400000000000003,"5min":1.0609999999999999,"15min":1.1950000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.80900000000000005,"15min":0.81599999999999995},"min":{"1min":0.53900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.1639999999999999},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.871},"min":{"1min":0.66100000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.145,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.85699999999999998,"15min":0.86299999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.57399999999999995},"max":{"1min":1.016,"5min":1.157,"15min":1.302},"last":1.0680000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.63700000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.204,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89000000000000001,"15min":0.88},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.97199999999999998,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":1.024}]}]},{"osd":0,"up_from":14,"seq":60129543241,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312372,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059468,"statfs":{"total":96636764160,"available":96316895232,"internally_reserved":0,"allocated":2863104,"data_stored":1711039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72499999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2330000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.47099999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74199999999999999,"5min":0.84799999999999998,"15min":0.94499999999999995},"last":1.4530000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.95099999999999996},"last":1.417},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.69699999999999995,"5min":0.72999999999999998,"15min":0.91100000000000003},"last":1.7490000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.55600000000000005},"min":{"1min":0.42399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70099999999999996,"5min":1.018,"15min":1.018},"last":1.927},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65000000000000002},"min":{"1min":0.57099999999999995,"5min":0.437,"15min":0.42799999999999999},"max":{"1min":0.81200000000000006,"5min":0.89400000000000002,"15min":1.02},"last":1.331}]},{"osd":4,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.78500000000000003},"min":{"1min":0.55900000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":2.5299999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.84799999999999998,"15min":0.84399999999999997},"min":{"1min":0.73699999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.052,"5min":1.2090000000000001,"15min":1.2310000000000001},"last":2.8119999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":0.81499999999999995},"min":{"1min":0.57299999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99199999999999999,"5min":1.228,"15min":1.2589999999999999},"last":3.0059999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.59499999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.1970000000000001},"last":1.5780000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85399999999999998,"15min":0.86099999999999999},"min":{"1min":0.629,"5min":0.59099999999999997,"15min":0.58199999999999996},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.2869999999999999},"last":2.7589999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83599999999999997,"15min":0.85299999999999998},"min":{"1min":0.61899999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0369999999999999,"5min":1.147,"15min":1.2649999999999999},"last":1.829}]},{"osd":7,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.873,"15min":0.874},"min":{"1min":0.68799999999999994,"5min":0.66000000000000003,"15min":0.63500000000000001},"max":{"1min":1.121,"5min":1.286,"15min":1.286},"last":1.3759999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.86699999999999999,"15min":0.875},"min":{"1min":0.65900000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.139,"5min":1.266,"15min":1.2869999999999999},"last":2.847}]}]},{"osd":2,"up_from":14,"seq":60129543242,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299496,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072344,"statfs":{"total":96636764160,"available":96330080256,"internally_reserved":0,"allocated":2260992,"data_stored":1114025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64300000000000002,"5min":0.69499999999999995,"15min":0.96499999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71599999999999997,"5min":0.876,"15min":0.876},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79400000000000004,"5min":0.91400000000000003,"15min":0.95099999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.40500000000000003},"max":{"1min":0.86399999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71799999999999997,"5min":0.76700000000000002,"15min":0.80000000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80700000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78200000000000003,"15min":0.80000000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.1879999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.82399999999999995,"15min":0.83499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.266,"5min":1.266,"15min":1.5640000000000001},"last":0.83699999999999997}]},{"osd":5,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.84999999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.528},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.2829999999999999},"last":0.79900000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.141,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.85699999999999998,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.242,"5min":1.296,"15min":1.3779999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.86699999999999999,"15min":0.88},"min":{"1min":0.69799999999999995,"5min":0.63300000000000001,"15min":0.59099999999999997},"max":{"1min":1.2050000000000001,"5min":1.214,"15min":1.629},"last":0.745},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.274,"5min":1.274,"15min":1.5940000000000001},"last":0.85099999999999998}]}]},{"osd":3,"up_from":14,"seq":60129543243,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299508,"kb_used_data":2220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072332,"statfs":{"total":96636764160,"available":96330067968,"internally_reserved":0,"allocated":2273280,"data_stored":1120705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.47299999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66200000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64300000000000002,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.45900000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.502}]},{"osd":2,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70099999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.622}]},{"osd":4,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.58499999999999996,"5min":0.54300000000000004,"15min":0.47699999999999998},"max":{"1min":0.94399999999999995,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81499999999999995,"15min":0.82199999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0309999999999999,"5min":1.0780000000000001,"15min":1.222},"last":0.75600000000000001}]},{"osd":5,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81599999999999995,"15min":0.82399999999999995},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.66400000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.012,"5min":1.27,"15min":1.27},"last":0.94199999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.85399999999999998},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.58099999999999996},"max":{"1min":1.1739999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84299999999999997,"15min":0.85599999999999998},"min":{"1min":0.60499999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1080000000000001,"5min":1.389,"15min":1.389},"last":0.60499999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86799999999999999,"15min":0.86699999999999999},"min":{"1min":0.68999999999999995,"5min":0.623,"15min":0.60799999999999998},"max":{"1min":1.0629999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.88300000000000001,"15min":0.88200000000000001},"min":{"1min":0.64300000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0660000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.70299999999999996}]}]},{"osd":4,"up_from":15,"seq":64424510539,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311808,"kb_used_data":2232,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060032,"statfs":{"total":96636764160,"available":96317472768,"internally_reserved":0,"allocated":2285568,"data_stored":1122277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.73599999999999999,"15min":0.73999999999999999},"min":{"1min":0.64800000000000002,"5min":0.40200000000000002,"15min":0.34000000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.754,"15min":0.76100000000000001},"min":{"1min":0.54900000000000004,"5min":0.48699999999999999,"15min":0.36499999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.248},"last":0.498}]},{"osd":1,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.66200000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.56699999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.78000000000000003,"15min":0.77800000000000002},"min":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.495},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.069},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76600000000000001,"15min":0.76700000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.96699999999999997,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64200000000000002}]},{"osd":3,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.53400000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80000000000000004,"15min":0.79200000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.087,"15min":1.179},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.42499999999999999},"max":{"1min":0.73999999999999999,"5min":0.93600000000000005,"15min":1.016},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.78300000000000003}]},{"osd":6,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.65000000000000002},"min":{"1min":0.44,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":0.90600000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71599999999999997,"15min":0.69199999999999995},"min":{"1min":0.55300000000000005,"5min":0.46999999999999997,"15min":0.46000000000000002},"max":{"1min":0.91400000000000003,"5min":0.96199999999999997,"15min":1.016},"last":0.76500000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73599999999999999,"15min":0.71499999999999997},"min":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.41499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77900000000000003,"15min":0.75700000000000001},"min":{"1min":0.58899999999999997,"5min":0.53100000000000003,"15min":0.49099999999999999},"max":{"1min":1.004,"5min":1.171,"15min":1.171},"last":0.81999999999999995}]}]},{"osd":5,"up_from":14,"seq":60129543241,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311800,"kb_used_data":2224,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060040,"statfs":{"total":96636764160,"available":96317480960,"internally_reserved":0,"allocated":2277376,"data_stored":1121247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78600000000000003,"15min":0.78700000000000003},"min":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.436},"max":{"1min":0.998,"5min":1.585,"15min":1.585},"last":0.629},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.83099999999999996,"15min":0.82199999999999995},"min":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.673,"15min":1.673},"last":0.70899999999999996}]},{"osd":1,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.56299999999999994,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0189999999999999,"15min":1.0860000000000001},"last":0.746},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.75800000000000001,"15min":0.76900000000000002},"min":{"1min":0.54800000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98899999999999999,"5min":1.0009999999999999,"15min":1.139},"last":0.84699999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73799999999999999,"15min":0.74399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91700000000000004,"5min":1.0349999999999999,"15min":1.0580000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75900000000000001,"15min":0.76000000000000001},"min":{"1min":0.61499999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79000000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55600000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.93799999999999994,"5min":1.069,"15min":1.069},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.752,"15min":0.748},"min":{"1min":0.56999999999999995,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":0.91100000000000003,"5min":1.0880000000000001,"15min":1.161},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.63},"min":{"1min":0.48199999999999998,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":0.92500000000000004,"5min":0.94399999999999995,"15min":0.97499999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.63},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.41599999999999998},"max":{"1min":0.80400000000000005,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.442,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.5,"5min":0.46700000000000003,"15min":0.46400000000000002},"max":{"1min":1.0289999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.505,"5min":0.47399999999999998,"15min":0.46500000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76800000000000002,"15min":0.75800000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.81299999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:01:45.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:01:45.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.902+0000 7f3a2d3ff700 1 Processor -- start 2023-10-07T11:01:45.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.902+0000 7f3a2d3ff700 1 -- start start 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a2d3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a28159750 0x7f3a280d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a2d3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3a28159b20 0x7f3a280ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a2d3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a280cbf20 con 0x7f3a280ce6d0 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a2d3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a280cc200 con 0x7f3a28159750 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a2d3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a280cc4e0 con 0x7f3a28159b20 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a28159750 0x7f3a280d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a28159750 0x7f3a280d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37120/0 (socket says 172.21.15.29:37120) 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 -- 172.21.15.29:0/2260322848 learned_addr learned my addr 172.21.15.29:0/2260322848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a267fc700 1 --2- 172.21.15.29:0/2260322848 >> v2:172.21.15.144:3300/0 conn(0x7f3a28159b20 0x7f3a280ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 -- 172.21.15.29:0/2260322848 >> v2:172.21.15.144:3300/0 conn(0x7f3a28159b20 msgr2=0x7f3a280ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 --2- 172.21.15.29:0/2260322848 >> v2:172.21.15.144:3300/0 conn(0x7f3a28159b20 0x7f3a280ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 -- 172.21.15.29:0/2260322848 >> v1:172.21.15.144:6789/0 conn(0x7f3a280ce6d0 legacy=0x7f3a2815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.904+0000 7f3a26ffd700 1 -- 172.21.15.29:0/2260322848 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a280cc7c0 con 0x7f3a28159750 2023-10-07T11:01:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.905+0000 7f3a26ffd700 1 --2- 172.21.15.29:0/2260322848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a280d10a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3a28088a70 tx=0x7f3a18009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3573a8ffa4a42b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.905+0000 7f3a25ffb700 1 -- 172.21.15.29:0/2260322848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18005c10 con 0x7f3a28159750 2023-10-07T11:01:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.905+0000 7f3a25ffb700 1 -- 172.21.15.29:0/2260322848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a18005d70 con 0x7f3a28159750 2023-10-07T11:01:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.905+0000 7f3a25ffb700 1 -- 172.21.15.29:0/2260322848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1800b880 con 0x7f3a28159750 2023-10-07T11:01:45.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.905+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/2260322848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 msgr2=0x7f3a280d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.905+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/2260322848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a280d10a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3a28088a70 tx=0x7f3a18009220 comp rx=0 tx=0).stop 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.906+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/2260322848 shutdown_connections 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.906+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/2260322848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a280d10a0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.906+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/2260322848 >> v2:172.21.15.144:3300/0 conn(0x7f3a28159b20 0x7f3a280ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.906+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/2260322848 >> 172.21.15.29:0/2260322848 conn(0x7f3a28063a10 msgr2=0x7f3a280d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.906+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/2260322848 shutdown_connections 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.906+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/2260322848 wait complete. 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.907+0000 7f3a2d3ff700 1 Processor -- start 2023-10-07T11:01:45.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.907+0000 7f3a2d3ff700 1 -- start start 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a2d3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a280ce6d0 0x7f3a2816cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a2d3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a2815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a2d3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a28159b20 0x7f3a2815b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a2d3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2816dcd0 con 0x7f3a28159750 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a2d3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2816d430 con 0x7f3a280ce6d0 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a2d3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2816d710 con 0x7f3a28159b20 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a267fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a2815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a267fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a2815ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37134/0 (socket says 172.21.15.29:37134) 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.908+0000 7f3a267fc700 1 -- 172.21.15.29:0/631796534 learned_addr learned my addr 172.21.15.29:0/631796534 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a277fe700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a28159b20 0x7f3a2815b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a26ffd700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a280ce6d0 0x7f3a2816cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a267fc700 1 -- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a28159b20 msgr2=0x7f3a2815b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a267fc700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a28159b20 0x7f3a2815b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a267fc700 1 -- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a280ce6d0 msgr2=0x7f3a2816cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a267fc700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a280ce6d0 0x7f3a2816cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a267fc700 1 -- 172.21.15.29:0/631796534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a18009480 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a267fc700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a2815ef50 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00d860 tx=0x7f3a1c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c0125f0 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c012750 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2816d9f0 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.909+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c012b10 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.910+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2815b9a0 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.911+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28169770 con 0x7f3a28159750 2023-10-07T11:01:45.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.912+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3a1c01e460 con 0x7f3a28159750 2023-10-07T11:01:45.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.912+0000 7f3a0effd700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3a14066f30 0x7f3a140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:45.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.912+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f3a1c084d00 con 0x7f3a28159750 2023-10-07T11:01:45.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.913+0000 7f3a26ffd700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3a14066f30 0x7f3a140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:45.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.914+0000 7f3a26ffd700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3a14066f30 0x7f3a140693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009260 tx=0x7f3a18016000 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:45.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:45.916+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a28169770 con 0x7f3a28159750 2023-10-07T11:01:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.114+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a28169770 con 0x7f3a14066f30 2023-10-07T11:01:46.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.124+0000 7f3a0effd700 1 -- 172.21.15.29:0/631796534 <== mgr.8069 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289609 (secure 0 0 0) 0x7f3a28169770 con 0x7f3a14066f30 2023-10-07T11:01:46.127 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3a14066f30 msgr2=0x7f3a140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3a14066f30 0x7f3a140693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009260 tx=0x7f3a18016000 comp rx=0 tx=0).stop 2023-10-07T11:01:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 msgr2=0x7f3a2815ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a2815ef50 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00d860 tx=0x7f3a1c008040 comp rx=0 tx=0).stop 2023-10-07T11:01:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 shutdown_connections 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3a14066f30 0x7f3a140693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3a28159b20 0x7f3a2815b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a280ce6d0 0x7f3a2816cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 --2- 172.21.15.29:0/631796534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a28159750 0x7f3a2815ef50 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 >> 172.21.15.29:0/631796534 conn(0x7f3a28063a10 msgr2=0x7f3a280ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 shutdown_connections 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.128+0000 7f3a2d3ff700 1 -- 172.21.15.29:0/631796534 wait complete. 2023-10-07T11:01:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:01:46.145 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T11:01:45.085221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2470948,"kb_used_data":18916,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752503772,"statfs":{"total":773094113280,"available":770563862528,"internally_reserved":0,"allocated":19369984,"data_stored":10130508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001385"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899848+0000","last_change":"2023-10-07T11:01:40.898719+0000","last_active":"2023-10-07T11:01:40.899848+0000","last_peered":"2023-10-07T11:01:40.899848+0000","last_clean":"2023-10-07T11:01:40.899848+0000","last_became_active":"2023-10-07T11:01:40.898439+0000","last_became_peered":"2023-10-07T11:01:40.898439+0000","last_unstale":"2023-10-07T11:01:40.899848+0000","last_undegraded":"2023-10-07T11:01:40.899848+0000","last_fullsized":"2023-10-07T11:01:40.899848+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:08:25.405654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.897319+0000","last_change":"2023-10-07T11:01:40.897319+0000","last_active":"2023-10-07T11:01:40.897319+0000","last_peered":"2023-10-07T11:01:40.897319+0000","last_clean":"2023-10-07T11:01:40.897319+0000","last_became_active":"2023-10-07T11:01:40.896669+0000","last_became_peered":"2023-10-07T11:01:40.896669+0000","last_unstale":"2023-10-07T11:01:40.897319+0000","last_undegraded":"2023-10-07T11:01:40.897319+0000","last_fullsized":"2023-10-07T11:01:40.897319+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:41:56.209081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.920315+0000","last_change":"2023-10-07T11:01:40.917387+0000","last_active":"2023-10-07T11:01:40.920315+0000","last_peered":"2023-10-07T11:01:40.920315+0000","last_clean":"2023-10-07T11:01:40.920315+0000","last_became_active":"2023-10-07T11:01:40.917056+0000","last_became_peered":"2023-10-07T11:01:40.917056+0000","last_unstale":"2023-10-07T11:01:40.920315+0000","last_undegraded":"2023-10-07T11:01:40.920315+0000","last_fullsized":"2023-10-07T11:01:40.920315+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:49:36.503909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894479+0000","last_change":"2023-10-07T11:01:40.892293+0000","last_active":"2023-10-07T11:01:40.894479+0000","last_peered":"2023-10-07T11:01:40.894479+0000","last_clean":"2023-10-07T11:01:40.894479+0000","last_became_active":"2023-10-07T11:01:40.891930+0000","last_became_peered":"2023-10-07T11:01:40.891930+0000","last_unstale":"2023-10-07T11:01:40.894479+0000","last_undegraded":"2023-10-07T11:01:40.894479+0000","last_fullsized":"2023-10-07T11:01:40.894479+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:54:42.753786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908874+0000","last_change":"2023-10-07T11:01:40.908874+0000","last_active":"2023-10-07T11:01:40.908874+0000","last_peered":"2023-10-07T11:01:40.908874+0000","last_clean":"2023-10-07T11:01:40.908874+0000","last_became_active":"2023-10-07T11:01:40.908536+0000","last_became_peered":"2023-10-07T11:01:40.908536+0000","last_unstale":"2023-10-07T11:01:40.908874+0000","last_undegraded":"2023-10-07T11:01:40.908874+0000","last_fullsized":"2023-10-07T11:01:40.908874+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:18:09.190481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894571+0000","last_change":"2023-10-07T11:01:40.894571+0000","last_active":"2023-10-07T11:01:40.894571+0000","last_peered":"2023-10-07T11:01:40.894571+0000","last_clean":"2023-10-07T11:01:40.894571+0000","last_became_active":"2023-10-07T11:01:40.894301+0000","last_became_peered":"2023-10-07T11:01:40.894301+0000","last_unstale":"2023-10-07T11:01:40.894571+0000","last_undegraded":"2023-10-07T11:01:40.894571+0000","last_fullsized":"2023-10-07T11:01:40.894571+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:17:57.567241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909843+0000","last_change":"2023-10-07T11:01:40.908321+0000","last_active":"2023-10-07T11:01:40.909843+0000","last_peered":"2023-10-07T11:01:40.909843+0000","last_clean":"2023-10-07T11:01:40.909843+0000","last_became_active":"2023-10-07T11:01:40.908003+0000","last_became_peered":"2023-10-07T11:01:40.908003+0000","last_unstale":"2023-10-07T11:01:40.909843+0000","last_undegraded":"2023-10-07T11:01:40.909843+0000","last_fullsized":"2023-10-07T11:01:40.909843+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:28:52.542972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907141+0000","last_change":"2023-10-07T11:01:40.907141+0000","last_active":"2023-10-07T11:01:40.907141+0000","last_peered":"2023-10-07T11:01:40.907141+0000","last_clean":"2023-10-07T11:01:40.907141+0000","last_became_active":"2023-10-07T11:01:40.906852+0000","last_became_peered":"2023-10-07T11:01:40.906852+0000","last_unstale":"2023-10-07T11:01:40.907141+0000","last_undegraded":"2023-10-07T11:01:40.907141+0000","last_fullsized":"2023-10-07T11:01:40.907141+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:42.841471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899276+0000","last_change":"2023-10-07T11:01:40.899276+0000","last_active":"2023-10-07T11:01:40.899276+0000","last_peered":"2023-10-07T11:01:40.899276+0000","last_clean":"2023-10-07T11:01:40.899276+0000","last_became_active":"2023-10-07T11:01:40.898990+0000","last_became_peered":"2023-10-07T11:01:40.898990+0000","last_unstale":"2023-10-07T11:01:40.899276+0000","last_undegraded":"2023-10-07T11:01:40.899276+0000","last_fullsized":"2023-10-07T11:01:40.899276+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:28:52.542972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.893444+0000","last_change":"2023-10-07T11:01:40.893444+0000","last_active":"2023-10-07T11:01:40.893444+0000","last_peered":"2023-10-07T11:01:40.893444+0000","last_clean":"2023-10-07T11:01:40.893444+0000","last_became_active":"2023-10-07T11:01:40.893150+0000","last_became_peered":"2023-10-07T11:01:40.893150+0000","last_unstale":"2023-10-07T11:01:40.893444+0000","last_undegraded":"2023-10-07T11:01:40.893444+0000","last_fullsized":"2023-10-07T11:01:40.893444+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:23.019779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906716+0000","last_change":"2023-10-07T11:01:40.906716+0000","last_active":"2023-10-07T11:01:40.906716+0000","last_peered":"2023-10-07T11:01:40.906716+0000","last_clean":"2023-10-07T11:01:40.906716+0000","last_became_active":"2023-10-07T11:01:40.906422+0000","last_became_peered":"2023-10-07T11:01:40.906422+0000","last_unstale":"2023-10-07T11:01:40.906716+0000","last_undegraded":"2023-10-07T11:01:40.906716+0000","last_fullsized":"2023-10-07T11:01:40.906716+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:02.999386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.898713+0000","last_change":"2023-10-07T11:01:40.898713+0000","last_active":"2023-10-07T11:01:40.898713+0000","last_peered":"2023-10-07T11:01:40.898713+0000","last_clean":"2023-10-07T11:01:40.898713+0000","last_became_active":"2023-10-07T11:01:40.898406+0000","last_became_peered":"2023-10-07T11:01:40.898406+0000","last_unstale":"2023-10-07T11:01:40.898713+0000","last_undegraded":"2023-10-07T11:01:40.898713+0000","last_fullsized":"2023-10-07T11:01:40.898713+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:44:47.648930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.896471+0000","last_change":"2023-10-07T11:01:40.894448+0000","last_active":"2023-10-07T11:01:40.896471+0000","last_peered":"2023-10-07T11:01:40.896471+0000","last_clean":"2023-10-07T11:01:40.896471+0000","last_became_active":"2023-10-07T11:01:40.894133+0000","last_became_peered":"2023-10-07T11:01:40.894133+0000","last_unstale":"2023-10-07T11:01:40.896471+0000","last_undegraded":"2023-10-07T11:01:40.896471+0000","last_fullsized":"2023-10-07T11:01:40.896471+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:14.932469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909864+0000","last_change":"2023-10-07T11:01:40.903135+0000","last_active":"2023-10-07T11:01:40.909864+0000","last_peered":"2023-10-07T11:01:40.909864+0000","last_clean":"2023-10-07T11:01:40.909864+0000","last_became_active":"2023-10-07T11:01:40.902865+0000","last_became_peered":"2023-10-07T11:01:40.902865+0000","last_unstale":"2023-10-07T11:01:40.909864+0000","last_undegraded":"2023-10-07T11:01:40.909864+0000","last_fullsized":"2023-10-07T11:01:40.909864+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:36:29.347771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909544+0000","last_change":"2023-10-07T11:01:40.901369+0000","last_active":"2023-10-07T11:01:40.909544+0000","last_peered":"2023-10-07T11:01:40.909544+0000","last_clean":"2023-10-07T11:01:40.909544+0000","last_became_active":"2023-10-07T11:01:40.901087+0000","last_became_peered":"2023-10-07T11:01:40.901087+0000","last_unstale":"2023-10-07T11:01:40.909544+0000","last_undegraded":"2023-10-07T11:01:40.909544+0000","last_fullsized":"2023-10-07T11:01:40.909544+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:49:49.896100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899692+0000","last_change":"2023-10-07T11:01:40.898475+0000","last_active":"2023-10-07T11:01:40.899692+0000","last_peered":"2023-10-07T11:01:40.899692+0000","last_clean":"2023-10-07T11:01:40.899692+0000","last_became_active":"2023-10-07T11:01:40.898042+0000","last_became_peered":"2023-10-07T11:01:40.898042+0000","last_unstale":"2023-10-07T11:01:40.899692+0000","last_undegraded":"2023-10-07T11:01:40.899692+0000","last_fullsized":"2023-10-07T11:01:40.899692+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:04:02.999386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.898480+0000","last_change":"2023-10-07T11:01:40.898480+0000","last_active":"2023-10-07T11:01:40.898480+0000","last_peered":"2023-10-07T11:01:40.898480+0000","last_clean":"2023-10-07T11:01:40.898480+0000","last_became_active":"2023-10-07T11:01:40.898239+0000","last_became_peered":"2023-10-07T11:01:40.898239+0000","last_unstale":"2023-10-07T11:01:40.898480+0000","last_undegraded":"2023-10-07T11:01:40.898480+0000","last_fullsized":"2023-10-07T11:01:40.898480+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:03:26.863115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.892811+0000","last_change":"2023-10-07T11:01:40.886626+0000","last_active":"2023-10-07T11:01:40.892811+0000","last_peered":"2023-10-07T11:01:40.892811+0000","last_clean":"2023-10-07T11:01:40.892811+0000","last_became_active":"2023-10-07T11:01:40.886375+0000","last_became_peered":"2023-10-07T11:01:40.886375+0000","last_unstale":"2023-10-07T11:01:40.892811+0000","last_undegraded":"2023-10-07T11:01:40.892811+0000","last_fullsized":"2023-10-07T11:01:40.892811+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:16:29.275776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908365+0000","last_change":"2023-10-07T11:01:40.908365+0000","last_active":"2023-10-07T11:01:40.908365+0000","last_peered":"2023-10-07T11:01:40.908365+0000","last_clean":"2023-10-07T11:01:40.908365+0000","last_became_active":"2023-10-07T11:01:40.908046+0000","last_became_peered":"2023-10-07T11:01:40.908046+0000","last_unstale":"2023-10-07T11:01:40.908365+0000","last_undegraded":"2023-10-07T11:01:40.908365+0000","last_fullsized":"2023-10-07T11:01:40.908365+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:14.932469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909768+0000","last_change":"2023-10-07T11:01:40.902166+0000","last_active":"2023-10-07T11:01:40.909768+0000","last_peered":"2023-10-07T11:01:40.909768+0000","last_clean":"2023-10-07T11:01:40.909768+0000","last_became_active":"2023-10-07T11:01:40.901832+0000","last_became_peered":"2023-10-07T11:01:40.901832+0000","last_unstale":"2023-10-07T11:01:40.909768+0000","last_undegraded":"2023-10-07T11:01:40.909768+0000","last_fullsized":"2023-10-07T11:01:40.909768+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:53.307889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894342+0000","last_change":"2023-10-07T11:01:40.894342+0000","last_active":"2023-10-07T11:01:40.894342+0000","last_peered":"2023-10-07T11:01:40.894342+0000","last_clean":"2023-10-07T11:01:40.894342+0000","last_became_active":"2023-10-07T11:01:40.893957+0000","last_became_peered":"2023-10-07T11:01:40.893957+0000","last_unstale":"2023-10-07T11:01:40.894342+0000","last_undegraded":"2023-10-07T11:01:40.894342+0000","last_fullsized":"2023-10-07T11:01:40.894342+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:01:08.572660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.897435+0000","last_change":"2023-10-07T11:01:40.897435+0000","last_active":"2023-10-07T11:01:40.897435+0000","last_peered":"2023-10-07T11:01:40.897435+0000","last_clean":"2023-10-07T11:01:40.897435+0000","last_became_active":"2023-10-07T11:01:40.897111+0000","last_became_peered":"2023-10-07T11:01:40.897111+0000","last_unstale":"2023-10-07T11:01:40.897435+0000","last_undegraded":"2023-10-07T11:01:40.897435+0000","last_fullsized":"2023-10-07T11:01:40.897435+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:27.789649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899175+0000","last_change":"2023-10-07T11:01:40.899175+0000","last_active":"2023-10-07T11:01:40.899175+0000","last_peered":"2023-10-07T11:01:40.899175+0000","last_clean":"2023-10-07T11:01:40.899175+0000","last_became_active":"2023-10-07T11:01:40.898893+0000","last_became_peered":"2023-10-07T11:01:40.898893+0000","last_unstale":"2023-10-07T11:01:40.899175+0000","last_undegraded":"2023-10-07T11:01:40.899175+0000","last_fullsized":"2023-10-07T11:01:40.899175+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:12.265734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.898564+0000","last_change":"2023-10-07T11:01:40.898564+0000","last_active":"2023-10-07T11:01:40.898564+0000","last_peered":"2023-10-07T11:01:40.898564+0000","last_clean":"2023-10-07T11:01:40.898564+0000","last_became_active":"2023-10-07T11:01:40.898276+0000","last_became_peered":"2023-10-07T11:01:40.898276+0000","last_unstale":"2023-10-07T11:01:40.898564+0000","last_undegraded":"2023-10-07T11:01:40.898564+0000","last_fullsized":"2023-10-07T11:01:40.898564+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:24:04.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.920285+0000","last_change":"2023-10-07T11:01:40.917281+0000","last_active":"2023-10-07T11:01:40.920285+0000","last_peered":"2023-10-07T11:01:40.920285+0000","last_clean":"2023-10-07T11:01:40.920285+0000","last_became_active":"2023-10-07T11:01:40.916973+0000","last_became_peered":"2023-10-07T11:01:40.916973+0000","last_unstale":"2023-10-07T11:01:40.920285+0000","last_undegraded":"2023-10-07T11:01:40.920285+0000","last_fullsized":"2023-10-07T11:01:40.920285+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:45:12.265734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904756+0000","last_change":"2023-10-07T11:01:40.903013+0000","last_active":"2023-10-07T11:01:40.904756+0000","last_peered":"2023-10-07T11:01:40.904756+0000","last_clean":"2023-10-07T11:01:40.904756+0000","last_became_active":"2023-10-07T11:01:40.902757+0000","last_became_peered":"2023-10-07T11:01:40.902757+0000","last_unstale":"2023-10-07T11:01:40.904756+0000","last_undegraded":"2023-10-07T11:01:40.904756+0000","last_fullsized":"2023-10-07T11:01:40.904756+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:27.789649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909827+0000","last_change":"2023-10-07T11:01:40.901386+0000","last_active":"2023-10-07T11:01:40.909827+0000","last_peered":"2023-10-07T11:01:40.909827+0000","last_clean":"2023-10-07T11:01:40.909827+0000","last_became_active":"2023-10-07T11:01:40.901062+0000","last_became_peered":"2023-10-07T11:01:40.901062+0000","last_unstale":"2023-10-07T11:01:40.909827+0000","last_undegraded":"2023-10-07T11:01:40.909827+0000","last_fullsized":"2023-10-07T11:01:40.909827+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:03:26.863115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.914433+0000","last_change":"2023-10-07T11:01:40.913731+0000","last_active":"2023-10-07T11:01:40.914433+0000","last_peered":"2023-10-07T11:01:40.914433+0000","last_clean":"2023-10-07T11:01:40.914433+0000","last_became_active":"2023-10-07T11:01:40.913436+0000","last_became_peered":"2023-10-07T11:01:40.913436+0000","last_unstale":"2023-10-07T11:01:40.914433+0000","last_undegraded":"2023-10-07T11:01:40.914433+0000","last_fullsized":"2023-10-07T11:01:40.914433+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:17.329155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909795+0000","last_change":"2023-10-07T11:01:40.903056+0000","last_active":"2023-10-07T11:01:40.909795+0000","last_peered":"2023-10-07T11:01:40.909795+0000","last_clean":"2023-10-07T11:01:40.909795+0000","last_became_active":"2023-10-07T11:01:40.902482+0000","last_became_peered":"2023-10-07T11:01:40.902482+0000","last_unstale":"2023-10-07T11:01:40.909795+0000","last_undegraded":"2023-10-07T11:01:40.909795+0000","last_fullsized":"2023-10-07T11:01:40.909795+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:55.301881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.917404+0000","last_change":"2023-10-07T11:01:40.908466+0000","last_active":"2023-10-07T11:01:40.917404+0000","last_peered":"2023-10-07T11:01:40.917404+0000","last_clean":"2023-10-07T11:01:40.917404+0000","last_became_active":"2023-10-07T11:01:40.908207+0000","last_became_peered":"2023-10-07T11:01:40.908207+0000","last_unstale":"2023-10-07T11:01:40.917404+0000","last_undegraded":"2023-10-07T11:01:40.917404+0000","last_fullsized":"2023-10-07T11:01:40.917404+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:23.019779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916416+0000","last_change":"2023-10-07T11:01:40.904708+0000","last_active":"2023-10-07T11:01:40.916416+0000","last_peered":"2023-10-07T11:01:40.916416+0000","last_clean":"2023-10-07T11:01:40.916416+0000","last_became_active":"2023-10-07T11:01:40.904399+0000","last_became_peered":"2023-10-07T11:01:40.904399+0000","last_unstale":"2023-10-07T11:01:40.916416+0000","last_undegraded":"2023-10-07T11:01:40.916416+0000","last_fullsized":"2023-10-07T11:01:40.916416+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:06:56.195712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.892889+0000","last_change":"2023-10-07T11:01:40.886161+0000","last_active":"2023-10-07T11:01:40.892889+0000","last_peered":"2023-10-07T11:01:40.892889+0000","last_clean":"2023-10-07T11:01:40.892889+0000","last_became_active":"2023-10-07T11:01:40.885869+0000","last_became_peered":"2023-10-07T11:01:40.885869+0000","last_unstale":"2023-10-07T11:01:40.892889+0000","last_undegraded":"2023-10-07T11:01:40.892889+0000","last_fullsized":"2023-10-07T11:01:40.892889+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:55.980011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907491+0000","last_change":"2023-10-07T11:01:40.904834+0000","last_active":"2023-10-07T11:01:40.907491+0000","last_peered":"2023-10-07T11:01:40.907491+0000","last_clean":"2023-10-07T11:01:40.907491+0000","last_became_active":"2023-10-07T11:01:40.904562+0000","last_became_peered":"2023-10-07T11:01:40.904562+0000","last_unstale":"2023-10-07T11:01:40.907491+0000","last_undegraded":"2023-10-07T11:01:40.907491+0000","last_fullsized":"2023-10-07T11:01:40.907491+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:23.019779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"181'4","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.910507+0000","last_change":"2023-10-07T11:01:39.897441+0000","last_active":"2023-10-07T11:01:40.910507+0000","last_peered":"2023-10-07T11:01:40.910507+0000","last_clean":"2023-10-07T11:01:40.910507+0000","last_became_active":"2023-10-07T11:01:39.897149+0000","last_became_peered":"2023-10-07T11:01:39.897149+0000","last_unstale":"2023-10-07T11:01:40.910507+0000","last_undegraded":"2023-10-07T11:01:40.910507+0000","last_fullsized":"2023-10-07T11:01:40.910507+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:51:09.799332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916548+0000","last_change":"2023-10-07T11:01:40.906030+0000","last_active":"2023-10-07T11:01:40.916548+0000","last_peered":"2023-10-07T11:01:40.916548+0000","last_clean":"2023-10-07T11:01:40.916548+0000","last_became_active":"2023-10-07T11:01:40.905708+0000","last_became_peered":"2023-10-07T11:01:40.905708+0000","last_unstale":"2023-10-07T11:01:40.916548+0000","last_undegraded":"2023-10-07T11:01:40.916548+0000","last_fullsized":"2023-10-07T11:01:40.916548+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:33:30.913180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.913005+0000","last_change":"2023-10-07T11:01:39.896715+0000","last_active":"2023-10-07T11:01:40.913005+0000","last_peered":"2023-10-07T11:01:40.913005+0000","last_clean":"2023-10-07T11:01:40.913005+0000","last_became_active":"2023-10-07T11:01:39.896347+0000","last_became_peered":"2023-10-07T11:01:39.896347+0000","last_unstale":"2023-10-07T11:01:40.913005+0000","last_undegraded":"2023-10-07T11:01:40.913005+0000","last_fullsized":"2023-10-07T11:01:40.913005+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:19:13.871768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.897477+0000","last_change":"2023-10-07T11:01:40.897477+0000","last_active":"2023-10-07T11:01:40.897477+0000","last_peered":"2023-10-07T11:01:40.897477+0000","last_clean":"2023-10-07T11:01:40.897477+0000","last_became_active":"2023-10-07T11:01:40.897099+0000","last_became_peered":"2023-10-07T11:01:40.897099+0000","last_unstale":"2023-10-07T11:01:40.897477+0000","last_undegraded":"2023-10-07T11:01:40.897477+0000","last_fullsized":"2023-10-07T11:01:40.897477+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:53.307889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908001+0000","last_change":"2023-10-07T11:01:39.893732+0000","last_active":"2023-10-07T11:01:40.908001+0000","last_peered":"2023-10-07T11:01:40.908001+0000","last_clean":"2023-10-07T11:01:40.908001+0000","last_became_active":"2023-10-07T11:01:39.893469+0000","last_became_peered":"2023-10-07T11:01:39.893469+0000","last_unstale":"2023-10-07T11:01:40.908001+0000","last_undegraded":"2023-10-07T11:01:40.908001+0000","last_fullsized":"2023-10-07T11:01:40.908001+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:39:32.066938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906635+0000","last_change":"2023-10-07T11:01:40.897618+0000","last_active":"2023-10-07T11:01:40.906635+0000","last_peered":"2023-10-07T11:01:40.906635+0000","last_clean":"2023-10-07T11:01:40.906635+0000","last_became_active":"2023-10-07T11:01:40.897254+0000","last_became_peered":"2023-10-07T11:01:40.897254+0000","last_unstale":"2023-10-07T11:01:40.906635+0000","last_undegraded":"2023-10-07T11:01:40.906635+0000","last_fullsized":"2023-10-07T11:01:40.906635+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:17:57.567241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906639+0000","last_change":"2023-10-07T11:01:39.896498+0000","last_active":"2023-10-07T11:01:40.906639+0000","last_peered":"2023-10-07T11:01:40.906639+0000","last_clean":"2023-10-07T11:01:40.906639+0000","last_became_active":"2023-10-07T11:01:39.896246+0000","last_became_peered":"2023-10-07T11:01:39.896246+0000","last_unstale":"2023-10-07T11:01:40.906639+0000","last_undegraded":"2023-10-07T11:01:40.906639+0000","last_fullsized":"2023-10-07T11:01:40.906639+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:55:41.660809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899954+0000","last_change":"2023-10-07T11:01:40.896564+0000","last_active":"2023-10-07T11:01:40.899954+0000","last_peered":"2023-10-07T11:01:40.899954+0000","last_clean":"2023-10-07T11:01:40.899954+0000","last_became_active":"2023-10-07T11:01:40.895978+0000","last_became_peered":"2023-10-07T11:01:40.895978+0000","last_unstale":"2023-10-07T11:01:40.899954+0000","last_undegraded":"2023-10-07T11:01:40.899954+0000","last_fullsized":"2023-10-07T11:01:40.899954+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:25.409533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916766+0000","last_change":"2023-10-07T11:01:39.893437+0000","last_active":"2023-10-07T11:01:40.916766+0000","last_peered":"2023-10-07T11:01:40.916766+0000","last_clean":"2023-10-07T11:01:40.916766+0000","last_became_active":"2023-10-07T11:01:39.893131+0000","last_became_peered":"2023-10-07T11:01:39.893131+0000","last_unstale":"2023-10-07T11:01:40.916766+0000","last_undegraded":"2023-10-07T11:01:40.916766+0000","last_fullsized":"2023-10-07T11:01:40.916766+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:46.138415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905354+0000","last_change":"2023-10-07T11:01:40.901171+0000","last_active":"2023-10-07T11:01:40.905354+0000","last_peered":"2023-10-07T11:01:40.905354+0000","last_clean":"2023-10-07T11:01:40.905354+0000","last_became_active":"2023-10-07T11:01:40.900637+0000","last_became_peered":"2023-10-07T11:01:40.900637+0000","last_unstale":"2023-10-07T11:01:40.905354+0000","last_undegraded":"2023-10-07T11:01:40.905354+0000","last_fullsized":"2023-10-07T11:01:40.905354+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:55:14.047108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907201+0000","last_change":"2023-10-07T11:01:39.887481+0000","last_active":"2023-10-07T11:01:40.907201+0000","last_peered":"2023-10-07T11:01:40.907201+0000","last_clean":"2023-10-07T11:01:40.907201+0000","last_became_active":"2023-10-07T11:01:39.887188+0000","last_became_peered":"2023-10-07T11:01:39.887188+0000","last_unstale":"2023-10-07T11:01:40.907201+0000","last_undegraded":"2023-10-07T11:01:40.907201+0000","last_fullsized":"2023-10-07T11:01:40.907201+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:03:14.066979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905287+0000","last_change":"2023-10-07T11:01:40.902202+0000","last_active":"2023-10-07T11:01:40.905287+0000","last_peered":"2023-10-07T11:01:40.905287+0000","last_clean":"2023-10-07T11:01:40.905287+0000","last_became_active":"2023-10-07T11:01:40.901939+0000","last_became_peered":"2023-10-07T11:01:40.901939+0000","last_unstale":"2023-10-07T11:01:40.905287+0000","last_undegraded":"2023-10-07T11:01:40.905287+0000","last_fullsized":"2023-10-07T11:01:40.905287+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:01.893237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.893879+0000","last_change":"2023-10-07T11:01:39.890362+0000","last_active":"2023-10-07T11:01:39.893879+0000","last_peered":"2023-10-07T11:01:39.893879+0000","last_clean":"2023-10-07T11:01:39.893879+0000","last_became_active":"2023-10-07T11:01:39.890089+0000","last_became_peered":"2023-10-07T11:01:39.890089+0000","last_unstale":"2023-10-07T11:01:39.893879+0000","last_undegraded":"2023-10-07T11:01:39.893879+0000","last_fullsized":"2023-10-07T11:01:39.893879+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:11.402708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.895117+0000","last_change":"2023-10-07T11:01:40.895117+0000","last_active":"2023-10-07T11:01:40.895117+0000","last_peered":"2023-10-07T11:01:40.895117+0000","last_clean":"2023-10-07T11:01:40.895117+0000","last_became_active":"2023-10-07T11:01:40.894844+0000","last_became_peered":"2023-10-07T11:01:40.894844+0000","last_unstale":"2023-10-07T11:01:40.895117+0000","last_undegraded":"2023-10-07T11:01:40.895117+0000","last_fullsized":"2023-10-07T11:01:40.895117+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:52:55.301881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.893262+0000","last_change":"2023-10-07T11:01:39.891362+0000","last_active":"2023-10-07T11:01:40.893262+0000","last_peered":"2023-10-07T11:01:40.893262+0000","last_clean":"2023-10-07T11:01:40.893262+0000","last_became_active":"2023-10-07T11:01:39.890817+0000","last_became_peered":"2023-10-07T11:01:39.890817+0000","last_unstale":"2023-10-07T11:01:40.893262+0000","last_undegraded":"2023-10-07T11:01:40.893262+0000","last_fullsized":"2023-10-07T11:01:40.893262+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:11.402708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904271+0000","last_change":"2023-10-07T11:01:40.895008+0000","last_active":"2023-10-07T11:01:40.904271+0000","last_peered":"2023-10-07T11:01:40.904271+0000","last_clean":"2023-10-07T11:01:40.904271+0000","last_became_active":"2023-10-07T11:01:40.894713+0000","last_became_peered":"2023-10-07T11:01:40.894713+0000","last_unstale":"2023-10-07T11:01:40.904271+0000","last_undegraded":"2023-10-07T11:01:40.904271+0000","last_fullsized":"2023-10-07T11:01:40.904271+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:09:47.194488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.898310+0000","last_change":"2023-10-07T11:01:39.897630+0000","last_active":"2023-10-07T11:01:39.898310+0000","last_peered":"2023-10-07T11:01:39.898310+0000","last_clean":"2023-10-07T11:01:39.898310+0000","last_became_active":"2023-10-07T11:01:39.897352+0000","last_became_peered":"2023-10-07T11:01:39.897352+0000","last_unstale":"2023-10-07T11:01:39.898310+0000","last_undegraded":"2023-10-07T11:01:39.898310+0000","last_fullsized":"2023-10-07T11:01:39.898310+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:50:53.258627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904871+0000","last_change":"2023-10-07T11:01:40.895295+0000","last_active":"2023-10-07T11:01:40.904871+0000","last_peered":"2023-10-07T11:01:40.904871+0000","last_clean":"2023-10-07T11:01:40.904871+0000","last_became_active":"2023-10-07T11:01:40.894492+0000","last_became_peered":"2023-10-07T11:01:40.894492+0000","last_unstale":"2023-10-07T11:01:40.904871+0000","last_undegraded":"2023-10-07T11:01:40.904871+0000","last_fullsized":"2023-10-07T11:01:40.904871+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:55.094651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"181'3","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916973+0000","last_change":"2023-10-07T11:01:39.894456+0000","last_active":"2023-10-07T11:01:40.916973+0000","last_peered":"2023-10-07T11:01:40.916973+0000","last_clean":"2023-10-07T11:01:40.916973+0000","last_became_active":"2023-10-07T11:01:39.893538+0000","last_became_peered":"2023-10-07T11:01:39.893538+0000","last_unstale":"2023-10-07T11:01:40.916973+0000","last_undegraded":"2023-10-07T11:01:40.916973+0000","last_fullsized":"2023-10-07T11:01:40.916973+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:32:57.612441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916852+0000","last_change":"2023-10-07T11:01:40.907106+0000","last_active":"2023-10-07T11:01:40.916852+0000","last_peered":"2023-10-07T11:01:40.916852+0000","last_clean":"2023-10-07T11:01:40.916852+0000","last_became_active":"2023-10-07T11:01:40.905676+0000","last_became_peered":"2023-10-07T11:01:40.905676+0000","last_unstale":"2023-10-07T11:01:40.916852+0000","last_undegraded":"2023-10-07T11:01:40.916852+0000","last_fullsized":"2023-10-07T11:01:40.916852+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:25.409533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"181'6","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.920746+0000","last_change":"2023-10-07T11:01:39.896862+0000","last_active":"2023-10-07T11:01:40.920746+0000","last_peered":"2023-10-07T11:01:40.920746+0000","last_clean":"2023-10-07T11:01:40.920746+0000","last_became_active":"2023-10-07T11:01:39.896554+0000","last_became_peered":"2023-10-07T11:01:39.896554+0000","last_unstale":"2023-10-07T11:01:40.920746+0000","last_undegraded":"2023-10-07T11:01:40.920746+0000","last_fullsized":"2023-10-07T11:01:40.920746+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:00.884517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.888381+0000","last_change":"2023-10-07T11:01:40.885902+0000","last_active":"2023-10-07T11:01:40.888381+0000","last_peered":"2023-10-07T11:01:40.888381+0000","last_clean":"2023-10-07T11:01:40.888381+0000","last_became_active":"2023-10-07T11:01:40.885612+0000","last_became_peered":"2023-10-07T11:01:40.885612+0000","last_unstale":"2023-10-07T11:01:40.888381+0000","last_undegraded":"2023-10-07T11:01:40.888381+0000","last_fullsized":"2023-10-07T11:01:40.888381+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:24:36.242582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906880+0000","last_change":"2023-10-07T11:01:39.891597+0000","last_active":"2023-10-07T11:01:40.906880+0000","last_peered":"2023-10-07T11:01:40.906880+0000","last_clean":"2023-10-07T11:01:40.906880+0000","last_became_active":"2023-10-07T11:01:39.891342+0000","last_became_peered":"2023-10-07T11:01:39.891342+0000","last_unstale":"2023-10-07T11:01:40.906880+0000","last_undegraded":"2023-10-07T11:01:40.906880+0000","last_fullsized":"2023-10-07T11:01:40.906880+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:18:19.978253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.901460+0000","last_change":"2023-10-07T11:01:40.899846+0000","last_active":"2023-10-07T11:01:40.901460+0000","last_peered":"2023-10-07T11:01:40.901460+0000","last_clean":"2023-10-07T11:01:40.901460+0000","last_became_active":"2023-10-07T11:01:40.899546+0000","last_became_peered":"2023-10-07T11:01:40.899546+0000","last_unstale":"2023-10-07T11:01:40.901460+0000","last_undegraded":"2023-10-07T11:01:40.901460+0000","last_fullsized":"2023-10-07T11:01:40.901460+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:41:56.209081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.878511+0000","last_change":"2023-10-07T11:01:39.893328+0000","last_active":"2023-10-07T11:01:40.878511+0000","last_peered":"2023-10-07T11:01:40.878511+0000","last_clean":"2023-10-07T11:01:40.878511+0000","last_became_active":"2023-10-07T11:01:39.893046+0000","last_became_peered":"2023-10-07T11:01:39.893046+0000","last_unstale":"2023-10-07T11:01:40.878511+0000","last_undegraded":"2023-10-07T11:01:40.878511+0000","last_fullsized":"2023-10-07T11:01:40.878511+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:06:24.400814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.902300+0000","last_change":"2023-10-07T11:01:40.898230+0000","last_active":"2023-10-07T11:01:40.902300+0000","last_peered":"2023-10-07T11:01:40.902300+0000","last_clean":"2023-10-07T11:01:40.902300+0000","last_became_active":"2023-10-07T11:01:40.897936+0000","last_became_peered":"2023-10-07T11:01:40.897936+0000","last_unstale":"2023-10-07T11:01:40.902300+0000","last_undegraded":"2023-10-07T11:01:40.902300+0000","last_fullsized":"2023-10-07T11:01:40.902300+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:16.573994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.883708+0000","last_change":"2023-10-07T11:01:39.893308+0000","last_active":"2023-10-07T11:01:40.883708+0000","last_peered":"2023-10-07T11:01:40.883708+0000","last_clean":"2023-10-07T11:01:40.883708+0000","last_became_active":"2023-10-07T11:01:39.892699+0000","last_became_peered":"2023-10-07T11:01:39.892699+0000","last_unstale":"2023-10-07T11:01:40.883708+0000","last_undegraded":"2023-10-07T11:01:40.883708+0000","last_fullsized":"2023-10-07T11:01:40.883708+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:06:27.259425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.909658+0000","last_change":"2023-10-07T11:01:40.902484+0000","last_active":"2023-10-07T11:01:40.909658+0000","last_peered":"2023-10-07T11:01:40.909658+0000","last_clean":"2023-10-07T11:01:40.909658+0000","last_became_active":"2023-10-07T11:01:40.902210+0000","last_became_peered":"2023-10-07T11:01:40.902210+0000","last_unstale":"2023-10-07T11:01:40.909658+0000","last_undegraded":"2023-10-07T11:01:40.909658+0000","last_fullsized":"2023-10-07T11:01:40.909658+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:03.966096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"181'8","reported_seq":18,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.917063+0000","last_change":"2023-10-07T11:01:39.895337+0000","last_active":"2023-10-07T11:01:40.917063+0000","last_peered":"2023-10-07T11:01:40.917063+0000","last_clean":"2023-10-07T11:01:40.917063+0000","last_became_active":"2023-10-07T11:01:39.894853+0000","last_became_peered":"2023-10-07T11:01:39.894853+0000","last_unstale":"2023-10-07T11:01:40.917063+0000","last_undegraded":"2023-10-07T11:01:40.917063+0000","last_fullsized":"2023-10-07T11:01:40.917063+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:46:37.971882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.894558+0000","last_change":"2023-10-07T11:01:40.892553+0000","last_active":"2023-10-07T11:01:40.894558+0000","last_peered":"2023-10-07T11:01:40.894558+0000","last_clean":"2023-10-07T11:01:40.894558+0000","last_became_active":"2023-10-07T11:01:40.892315+0000","last_became_peered":"2023-10-07T11:01:40.892315+0000","last_unstale":"2023-10-07T11:01:40.894558+0000","last_undegraded":"2023-10-07T11:01:40.894558+0000","last_fullsized":"2023-10-07T11:01:40.894558+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:45:34.275253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908302+0000","last_change":"2023-10-07T11:01:39.894287+0000","last_active":"2023-10-07T11:01:40.908302+0000","last_peered":"2023-10-07T11:01:40.908302+0000","last_clean":"2023-10-07T11:01:40.908302+0000","last_became_active":"2023-10-07T11:01:39.893993+0000","last_became_peered":"2023-10-07T11:01:39.893993+0000","last_unstale":"2023-10-07T11:01:40.908302+0000","last_undegraded":"2023-10-07T11:01:40.908302+0000","last_fullsized":"2023-10-07T11:01:40.908302+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:17:53.154186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906899+0000","last_change":"2023-10-07T11:01:40.905633+0000","last_active":"2023-10-07T11:01:40.906899+0000","last_peered":"2023-10-07T11:01:40.906899+0000","last_clean":"2023-10-07T11:01:40.906899+0000","last_became_active":"2023-10-07T11:01:40.905375+0000","last_became_peered":"2023-10-07T11:01:40.905375+0000","last_unstale":"2023-10-07T11:01:40.906899+0000","last_undegraded":"2023-10-07T11:01:40.906899+0000","last_fullsized":"2023-10-07T11:01:40.906899+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:31:27.470060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905966+0000","last_change":"2023-10-07T11:01:39.893990+0000","last_active":"2023-10-07T11:01:40.905966+0000","last_peered":"2023-10-07T11:01:40.905966+0000","last_clean":"2023-10-07T11:01:40.905966+0000","last_became_active":"2023-10-07T11:01:39.893678+0000","last_became_peered":"2023-10-07T11:01:39.893678+0000","last_unstale":"2023-10-07T11:01:40.905966+0000","last_undegraded":"2023-10-07T11:01:40.905966+0000","last_fullsized":"2023-10-07T11:01:40.905966+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:47:23.003410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.903770+0000","last_change":"2023-10-07T11:01:40.899310+0000","last_active":"2023-10-07T11:01:40.903770+0000","last_peered":"2023-10-07T11:01:40.903770+0000","last_clean":"2023-10-07T11:01:40.903770+0000","last_became_active":"2023-10-07T11:01:40.898812+0000","last_became_peered":"2023-10-07T11:01:40.898812+0000","last_unstale":"2023-10-07T11:01:40.903770+0000","last_undegraded":"2023-10-07T11:01:40.903770+0000","last_fullsized":"2023-10-07T11:01:40.903770+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:46:38.980601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"181'1","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.917929+0000","last_change":"2023-10-07T11:01:39.892641+0000","last_active":"2023-10-07T11:01:40.917929+0000","last_peered":"2023-10-07T11:01:40.917929+0000","last_clean":"2023-10-07T11:01:40.917929+0000","last_became_active":"2023-10-07T11:01:39.892378+0000","last_became_peered":"2023-10-07T11:01:39.892378+0000","last_unstale":"2023-10-07T11:01:40.917929+0000","last_undegraded":"2023-10-07T11:01:40.917929+0000","last_fullsized":"2023-10-07T11:01:40.917929+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:30:31.416388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.914124+0000","last_change":"2023-10-07T11:01:40.913548+0000","last_active":"2023-10-07T11:01:40.914124+0000","last_peered":"2023-10-07T11:01:40.914124+0000","last_clean":"2023-10-07T11:01:40.914124+0000","last_became_active":"2023-10-07T11:01:40.913192+0000","last_became_peered":"2023-10-07T11:01:40.913192+0000","last_unstale":"2023-10-07T11:01:40.914124+0000","last_undegraded":"2023-10-07T11:01:40.914124+0000","last_fullsized":"2023-10-07T11:01:40.914124+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:44:47.648930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.886990+0000","last_change":"2023-10-07T11:01:39.898387+0000","last_active":"2023-10-07T11:01:40.886990+0000","last_peered":"2023-10-07T11:01:40.886990+0000","last_clean":"2023-10-07T11:01:40.886990+0000","last_became_active":"2023-10-07T11:01:39.897799+0000","last_became_peered":"2023-10-07T11:01:39.897799+0000","last_unstale":"2023-10-07T11:01:40.886990+0000","last_undegraded":"2023-10-07T11:01:40.886990+0000","last_fullsized":"2023-10-07T11:01:40.886990+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:37:17.662100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.896018+0000","last_change":"2023-10-07T11:01:40.894754+0000","last_active":"2023-10-07T11:01:40.896018+0000","last_peered":"2023-10-07T11:01:40.896018+0000","last_clean":"2023-10-07T11:01:40.896018+0000","last_became_active":"2023-10-07T11:01:40.893935+0000","last_became_peered":"2023-10-07T11:01:40.893935+0000","last_unstale":"2023-10-07T11:01:40.896018+0000","last_undegraded":"2023-10-07T11:01:40.896018+0000","last_fullsized":"2023-10-07T11:01:40.896018+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:55.421293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907684+0000","last_change":"2023-10-07T11:01:39.893156+0000","last_active":"2023-10-07T11:01:40.907684+0000","last_peered":"2023-10-07T11:01:40.907684+0000","last_clean":"2023-10-07T11:01:40.907684+0000","last_became_active":"2023-10-07T11:01:39.892865+0000","last_became_peered":"2023-10-07T11:01:39.892865+0000","last_unstale":"2023-10-07T11:01:40.907684+0000","last_undegraded":"2023-10-07T11:01:40.907684+0000","last_fullsized":"2023-10-07T11:01:40.907684+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:21:05.206253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899735+0000","last_change":"2023-10-07T11:01:40.897430+0000","last_active":"2023-10-07T11:01:40.899735+0000","last_peered":"2023-10-07T11:01:40.899735+0000","last_clean":"2023-10-07T11:01:40.899735+0000","last_became_active":"2023-10-07T11:01:40.897188+0000","last_became_peered":"2023-10-07T11:01:40.897188+0000","last_unstale":"2023-10-07T11:01:40.899735+0000","last_undegraded":"2023-10-07T11:01:40.899735+0000","last_fullsized":"2023-10-07T11:01:40.899735+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:18:09.190481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.884729+0000","last_change":"2023-10-07T11:01:39.893494+0000","last_active":"2023-10-07T11:01:40.884729+0000","last_peered":"2023-10-07T11:01:40.884729+0000","last_clean":"2023-10-07T11:01:40.884729+0000","last_became_active":"2023-10-07T11:01:39.893237+0000","last_became_peered":"2023-10-07T11:01:39.893237+0000","last_unstale":"2023-10-07T11:01:40.884729+0000","last_undegraded":"2023-10-07T11:01:40.884729+0000","last_fullsized":"2023-10-07T11:01:40.884729+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:28.084936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.905249+0000","last_change":"2023-10-07T11:01:40.900558+0000","last_active":"2023-10-07T11:01:40.905249+0000","last_peered":"2023-10-07T11:01:40.905249+0000","last_clean":"2023-10-07T11:01:40.905249+0000","last_became_active":"2023-10-07T11:01:40.900201+0000","last_became_peered":"2023-10-07T11:01:40.900201+0000","last_unstale":"2023-10-07T11:01:40.905249+0000","last_undegraded":"2023-10-07T11:01:40.905249+0000","last_fullsized":"2023-10-07T11:01:40.905249+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:29:43.402726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.883830+0000","last_change":"2023-10-07T11:01:39.895223+0000","last_active":"2023-10-07T11:01:40.883830+0000","last_peered":"2023-10-07T11:01:40.883830+0000","last_clean":"2023-10-07T11:01:40.883830+0000","last_became_active":"2023-10-07T11:01:39.894886+0000","last_became_peered":"2023-10-07T11:01:39.894886+0000","last_unstale":"2023-10-07T11:01:40.883830+0000","last_undegraded":"2023-10-07T11:01:40.883830+0000","last_fullsized":"2023-10-07T11:01:40.883830+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:28:17.315711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.890988+0000","last_change":"2023-10-07T11:01:40.890988+0000","last_active":"2023-10-07T11:01:40.890988+0000","last_peered":"2023-10-07T11:01:40.890988+0000","last_clean":"2023-10-07T11:01:40.890988+0000","last_became_active":"2023-10-07T11:01:40.890709+0000","last_became_peered":"2023-10-07T11:01:40.890709+0000","last_unstale":"2023-10-07T11:01:40.890988+0000","last_undegraded":"2023-10-07T11:01:40.890988+0000","last_fullsized":"2023-10-07T11:01:40.890988+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:30:55.421293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.889381+0000","last_change":"2023-10-07T11:01:39.892309+0000","last_active":"2023-10-07T11:01:40.889381+0000","last_peered":"2023-10-07T11:01:40.889381+0000","last_clean":"2023-10-07T11:01:40.889381+0000","last_became_active":"2023-10-07T11:01:39.892035+0000","last_became_peered":"2023-10-07T11:01:39.892035+0000","last_unstale":"2023-10-07T11:01:40.889381+0000","last_undegraded":"2023-10-07T11:01:40.889381+0000","last_fullsized":"2023-10-07T11:01:40.889381+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:16:13.649531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904783+0000","last_change":"2023-10-07T11:01:40.894536+0000","last_active":"2023-10-07T11:01:40.904783+0000","last_peered":"2023-10-07T11:01:40.904783+0000","last_clean":"2023-10-07T11:01:40.904783+0000","last_became_active":"2023-10-07T11:01:40.894101+0000","last_became_peered":"2023-10-07T11:01:40.894101+0000","last_unstale":"2023-10-07T11:01:40.904783+0000","last_undegraded":"2023-10-07T11:01:40.904783+0000","last_fullsized":"2023-10-07T11:01:40.904783+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:33:06.317400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"181'2","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908686+0000","last_change":"2023-10-07T11:01:39.888670+0000","last_active":"2023-10-07T11:01:40.908686+0000","last_peered":"2023-10-07T11:01:40.908686+0000","last_clean":"2023-10-07T11:01:40.908686+0000","last_became_active":"2023-10-07T11:01:39.888051+0000","last_became_peered":"2023-10-07T11:01:39.888051+0000","last_unstale":"2023-10-07T11:01:40.908686+0000","last_undegraded":"2023-10-07T11:01:40.908686+0000","last_fullsized":"2023-10-07T11:01:40.908686+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:02:26.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.907379+0000","last_change":"2023-10-07T11:01:40.904618+0000","last_active":"2023-10-07T11:01:40.907379+0000","last_peered":"2023-10-07T11:01:40.907379+0000","last_clean":"2023-10-07T11:01:40.907379+0000","last_became_active":"2023-10-07T11:01:40.904338+0000","last_became_peered":"2023-10-07T11:01:40.904338+0000","last_unstale":"2023-10-07T11:01:40.907379+0000","last_undegraded":"2023-10-07T11:01:40.907379+0000","last_fullsized":"2023-10-07T11:01:40.907379+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:42.841471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.889924+0000","last_change":"2023-10-07T11:01:39.893656+0000","last_active":"2023-10-07T11:01:40.889924+0000","last_peered":"2023-10-07T11:01:40.889924+0000","last_clean":"2023-10-07T11:01:40.889924+0000","last_became_active":"2023-10-07T11:01:39.893324+0000","last_became_peered":"2023-10-07T11:01:39.893324+0000","last_unstale":"2023-10-07T11:01:40.889924+0000","last_undegraded":"2023-10-07T11:01:40.889924+0000","last_fullsized":"2023-10-07T11:01:40.889924+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:28:47.774275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":8,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.890127+0000","last_change":"2023-10-07T11:01:40.890127+0000","last_active":"2023-10-07T11:01:40.890127+0000","last_peered":"2023-10-07T11:01:40.890127+0000","last_clean":"2023-10-07T11:01:40.890127+0000","last_became_active":"2023-10-07T11:01:40.889867+0000","last_became_peered":"2023-10-07T11:01:40.889867+0000","last_unstale":"2023-10-07T11:01:40.890127+0000","last_undegraded":"2023-10-07T11:01:40.890127+0000","last_fullsized":"2023-10-07T11:01:40.890127+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:49:36.503909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.892843+0000","last_change":"2023-10-07T11:01:39.888218+0000","last_active":"2023-10-07T11:01:39.892843+0000","last_peered":"2023-10-07T11:01:39.892843+0000","last_clean":"2023-10-07T11:01:39.892843+0000","last_became_active":"2023-10-07T11:01:39.887891+0000","last_became_peered":"2023-10-07T11:01:39.887891+0000","last_unstale":"2023-10-07T11:01:39.892843+0000","last_undegraded":"2023-10-07T11:01:39.892843+0000","last_fullsized":"2023-10-07T11:01:39.892843+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:46:37.971882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.906273+0000","last_change":"2023-10-07T11:01:40.899153+0000","last_active":"2023-10-07T11:01:40.906273+0000","last_peered":"2023-10-07T11:01:40.906273+0000","last_clean":"2023-10-07T11:01:40.906273+0000","last_became_active":"2023-10-07T11:01:40.898895+0000","last_became_peered":"2023-10-07T11:01:40.898895+0000","last_unstale":"2023-10-07T11:01:40.906273+0000","last_undegraded":"2023-10-07T11:01:40.906273+0000","last_fullsized":"2023-10-07T11:01:40.906273+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:59:04.178092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.877735+0000","last_change":"2023-10-07T11:01:39.895225+0000","last_active":"2023-10-07T11:01:40.877735+0000","last_peered":"2023-10-07T11:01:40.877735+0000","last_clean":"2023-10-07T11:01:40.877735+0000","last_became_active":"2023-10-07T11:01:39.894953+0000","last_became_peered":"2023-10-07T11:01:39.894953+0000","last_unstale":"2023-10-07T11:01:40.877735+0000","last_undegraded":"2023-10-07T11:01:40.877735+0000","last_fullsized":"2023-10-07T11:01:40.877735+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:07:22.011059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.888363+0000","last_change":"2023-10-07T11:01:40.885348+0000","last_active":"2023-10-07T11:01:40.888363+0000","last_peered":"2023-10-07T11:01:40.888363+0000","last_clean":"2023-10-07T11:01:40.888363+0000","last_became_active":"2023-10-07T11:01:40.884354+0000","last_became_peered":"2023-10-07T11:01:40.884354+0000","last_unstale":"2023-10-07T11:01:40.888363+0000","last_undegraded":"2023-10-07T11:01:40.888363+0000","last_fullsized":"2023-10-07T11:01:40.888363+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:58:08.788067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.890481+0000","last_change":"2023-10-07T11:01:39.894328+0000","last_active":"2023-10-07T11:01:40.890481+0000","last_peered":"2023-10-07T11:01:40.890481+0000","last_clean":"2023-10-07T11:01:40.890481+0000","last_became_active":"2023-10-07T11:01:39.893502+0000","last_became_peered":"2023-10-07T11:01:39.893502+0000","last_unstale":"2023-10-07T11:01:40.890481+0000","last_undegraded":"2023-10-07T11:01:40.890481+0000","last_fullsized":"2023-10-07T11:01:40.890481+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:35:11.402708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.916866+0000","last_change":"2023-10-07T11:01:40.906946+0000","last_active":"2023-10-07T11:01:40.916866+0000","last_peered":"2023-10-07T11:01:40.916866+0000","last_clean":"2023-10-07T11:01:40.916866+0000","last_became_active":"2023-10-07T11:01:40.906638+0000","last_became_peered":"2023-10-07T11:01:40.906638+0000","last_unstale":"2023-10-07T11:01:40.916866+0000","last_undegraded":"2023-10-07T11:01:40.916866+0000","last_fullsized":"2023-10-07T11:01:40.916866+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:46:36.726101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.882071+0000","last_change":"2023-10-07T11:01:39.893190+0000","last_active":"2023-10-07T11:01:40.882071+0000","last_peered":"2023-10-07T11:01:40.882071+0000","last_clean":"2023-10-07T11:01:40.882071+0000","last_became_active":"2023-10-07T11:01:39.892914+0000","last_became_peered":"2023-10-07T11:01:39.892914+0000","last_unstale":"2023-10-07T11:01:40.882071+0000","last_undegraded":"2023-10-07T11:01:40.882071+0000","last_fullsized":"2023-10-07T11:01:40.882071+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:56.289200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.899673+0000","last_change":"2023-10-07T11:01:40.898304+0000","last_active":"2023-10-07T11:01:40.899673+0000","last_peered":"2023-10-07T11:01:40.899673+0000","last_clean":"2023-10-07T11:01:40.899673+0000","last_became_active":"2023-10-07T11:01:40.897693+0000","last_became_peered":"2023-10-07T11:01:40.897693+0000","last_unstale":"2023-10-07T11:01:40.899673+0000","last_undegraded":"2023-10-07T11:01:40.899673+0000","last_fullsized":"2023-10-07T11:01:40.899673+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:24:04.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-10-07T11:01:39.893860+0000","last_change":"2023-10-07T11:01:39.888456+0000","last_active":"2023-10-07T11:01:39.893860+0000","last_peered":"2023-10-07T11:01:39.893860+0000","last_clean":"2023-10-07T11:01:39.893860+0000","last_became_active":"2023-10-07T11:01:39.888125+0000","last_became_peered":"2023-10-07T11:01:39.888125+0000","last_unstale":"2023-10-07T11:01:39.893860+0000","last_undegraded":"2023-10-07T11:01:39.893860+0000","last_fullsized":"2023-10-07T11:01:39.893860+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:00.009923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.908015+0000","last_change":"2023-10-07T11:01:40.897927+0000","last_active":"2023-10-07T11:01:40.908015+0000","last_peered":"2023-10-07T11:01:40.908015+0000","last_clean":"2023-10-07T11:01:40.908015+0000","last_became_active":"2023-10-07T11:01:40.897541+0000","last_became_peered":"2023-10-07T11:01:40.897541+0000","last_unstale":"2023-10-07T11:01:40.908015+0000","last_undegraded":"2023-10-07T11:01:40.908015+0000","last_fullsized":"2023-10-07T11:01:40.908015+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:01:08.572660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.887886+0000","last_change":"2023-10-07T11:01:39.890605+0000","last_active":"2023-10-07T11:01:40.887886+0000","last_peered":"2023-10-07T11:01:40.887886+0000","last_clean":"2023-10-07T11:01:40.887886+0000","last_became_active":"2023-10-07T11:01:39.890340+0000","last_became_peered":"2023-10-07T11:01:39.890340+0000","last_unstale":"2023-10-07T11:01:40.887886+0000","last_undegraded":"2023-10-07T11:01:40.887886+0000","last_fullsized":"2023-10-07T11:01:40.887886+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:00:00.009923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.904857+0000","last_change":"2023-10-07T11:01:40.894957+0000","last_active":"2023-10-07T11:01:40.904857+0000","last_peered":"2023-10-07T11:01:40.904857+0000","last_clean":"2023-10-07T11:01:40.904857+0000","last_became_active":"2023-10-07T11:01:40.894578+0000","last_became_peered":"2023-10-07T11:01:40.894578+0000","last_unstale":"2023-10-07T11:01:40.904857+0000","last_undegraded":"2023-10-07T11:01:40.904857+0000","last_fullsized":"2023-10-07T11:01:40.904857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:39.865962+0000","last_clean_scrub_stamp":"2023-10-07T11:01:39.865962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:06:30.815321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.888086+0000","last_change":"2023-10-07T11:01:39.899669+0000","last_active":"2023-10-07T11:01:40.888086+0000","last_peered":"2023-10-07T11:01:40.888086+0000","last_clean":"2023-10-07T11:01:40.888086+0000","last_became_active":"2023-10-07T11:01:39.899320+0000","last_became_peered":"2023-10-07T11:01:39.899320+0000","last_unstale":"2023-10-07T11:01:40.888086+0000","last_undegraded":"2023-10-07T11:01:40.888086+0000","last_fullsized":"2023-10-07T11:01:40.888086+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:01:38.857243+0000","last_clean_scrub_stamp":"2023-10-07T11:01:38.857243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:24:57.980154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"172'326","reported_seq":1041,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-07T11:01:40.889468+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:01:40.889468+0000","last_peered":"2023-10-07T11:01:40.889468+0000","last_clean":"2023-10-07T11:01:40.889468+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:01:40.889468+0000","last_undegraded":"2023-10-07T11:01:40.889468+0000","last_fullsized":"2023-10-07T11:01:40.889468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477835,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312396,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059444,"statfs":{"total":96636764160,"available":96316870656,"internally_reserved":0,"allocated":2887680,"data_stored":1713289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.748,"15min":0.747},"min":{"1min":0.47699999999999998,"5min":0.40400000000000003,"15min":0.31},"max":{"1min":0.94499999999999995,"5min":0.98999999999999999,"15min":1.1259999999999999},"last":0.504},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77900000000000003,"15min":0.76200000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.41599999999999998},"max":{"1min":0.98599999999999999,"5min":1.016,"15min":1.071},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77500000000000002,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.40000000000000002},"max":{"1min":0.92600000000000005,"5min":1.02,"15min":1.0780000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.76600000000000001,"15min":0.76700000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.44},"max":{"1min":0.93799999999999994,"5min":1.0009999999999999,"15min":1.0880000000000001},"last":0.625}]},{"osd":2,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77700000000000002,"15min":0.77500000000000002},"min":{"1min":0.52700000000000002,"5min":0.503,"15min":0.38900000000000001},"max":{"1min":0.96299999999999997,"5min":1.0149999999999999,"15min":1.1000000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.54300000000000004,"5min":0.47999999999999998,"15min":0.46800000000000003},"max":{"1min":1.079,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76300000000000001,"15min":0.745},"min":{"1min":0.56799999999999995,"5min":0.501,"15min":0.47799999999999998},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.80600000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.48199999999999998},"max":{"1min":0.97599999999999998,"5min":1.0509999999999999,"15min":1.0669999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.35899999999999999},"max":{"1min":0.91100000000000003,"5min":0.999,"15min":0.999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"max":{"1min":0.90200000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.877}]},{"osd":5,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68700000000000006,"15min":0.66900000000000004},"min":{"1min":0.439,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92200000000000004,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.754},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.48199999999999998,"5min":0.435,"15min":0.435},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.68700000000000006}]},{"osd":6,"last update":"Sat Oct 7 11:00:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78300000000000003,"15min":0.76900000000000002},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.50700000000000001},"max":{"1min":1.038,"5min":1.0660000000000001,"15min":1.0669999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73799999999999999,"15min":0.73599999999999999},"min":{"1min":0.47899999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.85599999999999998}]}]},{"osd":6,"up_from":15,"seq":64424510539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315880,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055960,"statfs":{"total":96636764160,"available":96313303040,"internally_reserved":0,"allocated":2260992,"data_stored":1113957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.57799999999999996,"5min":0.45300000000000001,"15min":0.378},"max":{"1min":0.95799999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.52400000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":1.002},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.749,"15min":0.751},"min":{"1min":0.58099999999999996,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":1.048},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.55600000000000005,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84799999999999998,"5min":0.97499999999999998,"15min":1.0409999999999999},"last":0.65900000000000003}]},{"osd":2,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.65600000000000003,"5min":0.54200000000000004,"15min":0.46899999999999997},"max":{"1min":0.871,"5min":1.018,"15min":1.0189999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.55500000000000005,"5min":0.52600000000000002,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.98599999999999999,"15min":1.0069999999999999},"last":0.79500000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77400000000000002,"15min":0.77300000000000002},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.504},"max":{"1min":0.90200000000000002,"5min":1.073,"15min":1.073},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.69699999999999995,"5min":0.53300000000000003,"15min":0.52300000000000002},"max":{"1min":0.91500000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.46000000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78600000000000003,"5min":0.91100000000000003,"15min":0.97799999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66500000000000004,"15min":0.65300000000000002},"min":{"1min":0.48299999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.97099999999999997,"5min":1.012,"15min":1.0149999999999999},"last":0.77500000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67200000000000004},"min":{"1min":0.51000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84099999999999997,"5min":0.96599999999999997,"15min":0.995},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.71799999999999997},"min":{"1min":0.53200000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.022,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84399999999999997}]},{"osd":7,"last update":"Sat Oct 7 11:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67100000000000004,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95499999999999996,"5min":1.0760000000000001,"15min":1.1120000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.754,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.48199999999999998},"max":{"1min":1,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81299999999999994}]}]},{"osd":1,"up_from":14,"seq":60129543241,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307688,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064152,"statfs":{"total":96636764160,"available":96321691648,"internally_reserved":0,"allocated":2260992,"data_stored":1113969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.97599999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76800000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.39700000000000002},"max":{"1min":0.76400000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63800000000000001,"15min":0.64500000000000002},"min":{"1min":0.45800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80400000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.075,"5min":1.1539999999999999,"15min":1.373},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.80300000000000005,"15min":0.81299999999999994},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.085,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.871}]},{"osd":5,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.84499999999999997,"15min":0.85299999999999998},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.90400000000000003,"5min":1.0609999999999999,"15min":1.1950000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.80900000000000005,"15min":0.81599999999999995},"min":{"1min":0.53900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.1639999999999999},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.871},"min":{"1min":0.66100000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.145,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.85699999999999998,"15min":0.86299999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.57399999999999995},"max":{"1min":1.016,"5min":1.157,"15min":1.302},"last":1.0680000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.63700000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.204,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89000000000000001,"15min":0.88},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.97199999999999998,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":1.024}]}]},{"osd":0,"up_from":14,"seq":60129543241,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312372,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059468,"statfs":{"total":96636764160,"available":96316895232,"internally_reserved":0,"allocated":2863104,"data_stored":1711039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72499999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2330000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.47099999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74199999999999999,"5min":0.84799999999999998,"15min":0.94499999999999995},"last":1.4530000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.95099999999999996},"last":1.417},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.69699999999999995,"5min":0.72999999999999998,"15min":0.91100000000000003},"last":1.7490000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.55600000000000005},"min":{"1min":0.42399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70099999999999996,"5min":1.018,"15min":1.018},"last":1.927},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65000000000000002},"min":{"1min":0.57099999999999995,"5min":0.437,"15min":0.42799999999999999},"max":{"1min":0.81200000000000006,"5min":0.89400000000000002,"15min":1.02},"last":1.331}]},{"osd":4,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.78500000000000003},"min":{"1min":0.55900000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":2.5299999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.84799999999999998,"15min":0.84399999999999997},"min":{"1min":0.73699999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.052,"5min":1.2090000000000001,"15min":1.2310000000000001},"last":2.8119999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":0.81499999999999995},"min":{"1min":0.57299999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99199999999999999,"5min":1.228,"15min":1.2589999999999999},"last":3.0059999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.59499999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.1970000000000001},"last":1.5780000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85399999999999998,"15min":0.86099999999999999},"min":{"1min":0.629,"5min":0.59099999999999997,"15min":0.58199999999999996},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.2869999999999999},"last":2.7589999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83599999999999997,"15min":0.85299999999999998},"min":{"1min":0.61899999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0369999999999999,"5min":1.147,"15min":1.2649999999999999},"last":1.829}]},{"osd":7,"last update":"Sat Oct 7 11:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.873,"15min":0.874},"min":{"1min":0.68799999999999994,"5min":0.66000000000000003,"15min":0.63500000000000001},"max":{"1min":1.121,"5min":1.286,"15min":1.286},"last":1.3759999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.86699999999999999,"15min":0.875},"min":{"1min":0.65900000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.139,"5min":1.266,"15min":1.2869999999999999},"last":2.847}]}]},{"osd":2,"up_from":14,"seq":60129543242,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299496,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072344,"statfs":{"total":96636764160,"available":96330080256,"internally_reserved":0,"allocated":2260992,"data_stored":1114025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64300000000000002,"5min":0.69499999999999995,"15min":0.96499999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71599999999999997,"5min":0.876,"15min":0.876},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79400000000000004,"5min":0.91400000000000003,"15min":0.95099999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.40500000000000003},"max":{"1min":0.86399999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71799999999999997,"5min":0.76700000000000002,"15min":0.80000000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80700000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78200000000000003,"15min":0.80000000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.1879999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.82399999999999995,"15min":0.83499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.266,"5min":1.266,"15min":1.5640000000000001},"last":0.83699999999999997}]},{"osd":5,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.84999999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.528},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.2829999999999999},"last":0.79900000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.64800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.141,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.85699999999999998,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.242,"5min":1.296,"15min":1.3779999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.86699999999999999,"15min":0.88},"min":{"1min":0.69799999999999995,"5min":0.63300000000000001,"15min":0.59099999999999997},"max":{"1min":1.2050000000000001,"5min":1.214,"15min":1.629},"last":0.745},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.274,"5min":1.274,"15min":1.5940000000000001},"last":0.85099999999999998}]}]},{"osd":3,"up_from":14,"seq":60129543243,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299508,"kb_used_data":2220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072332,"statfs":{"total":96636764160,"available":96330067968,"internally_reserved":0,"allocated":2273280,"data_stored":1120705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.47299999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66200000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64300000000000002,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.45900000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.502}]},{"osd":2,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70099999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59799999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.622}]},{"osd":4,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.58499999999999996,"5min":0.54300000000000004,"15min":0.47699999999999998},"max":{"1min":0.94399999999999995,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81499999999999995,"15min":0.82199999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0309999999999999,"5min":1.0780000000000001,"15min":1.222},"last":0.75600000000000001}]},{"osd":5,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81599999999999995,"15min":0.82399999999999995},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.66400000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.012,"5min":1.27,"15min":1.27},"last":0.94199999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.85399999999999998},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.58099999999999996},"max":{"1min":1.1739999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84299999999999997,"15min":0.85599999999999998},"min":{"1min":0.60499999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1080000000000001,"5min":1.389,"15min":1.389},"last":0.60499999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86799999999999999,"15min":0.86699999999999999},"min":{"1min":0.68999999999999995,"5min":0.623,"15min":0.60799999999999998},"max":{"1min":1.0629999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.88300000000000001,"15min":0.88200000000000001},"min":{"1min":0.64300000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0660000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.70299999999999996}]}]},{"osd":4,"up_from":15,"seq":64424510539,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311808,"kb_used_data":2232,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060032,"statfs":{"total":96636764160,"available":96317472768,"internally_reserved":0,"allocated":2285568,"data_stored":1122277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.73599999999999999,"15min":0.73999999999999999},"min":{"1min":0.64800000000000002,"5min":0.40200000000000002,"15min":0.34000000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.754,"15min":0.76100000000000001},"min":{"1min":0.54900000000000004,"5min":0.48699999999999999,"15min":0.36499999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.248},"last":0.498}]},{"osd":1,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.66200000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.56699999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.78000000000000003,"15min":0.77800000000000002},"min":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.495},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.069},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76600000000000001,"15min":0.76700000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.96699999999999997,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64200000000000002}]},{"osd":3,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.53400000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80000000000000004,"15min":0.79200000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.087,"15min":1.179},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.42499999999999999},"max":{"1min":0.73999999999999999,"5min":0.93600000000000005,"15min":1.016},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.78300000000000003}]},{"osd":6,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.65000000000000002},"min":{"1min":0.44,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":0.90600000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71599999999999997,"15min":0.69199999999999995},"min":{"1min":0.55300000000000005,"5min":0.46999999999999997,"15min":0.46000000000000002},"max":{"1min":0.91400000000000003,"5min":0.96199999999999997,"15min":1.016},"last":0.76500000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73599999999999999,"15min":0.71499999999999997},"min":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.41499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77900000000000003,"15min":0.75700000000000001},"min":{"1min":0.58899999999999997,"5min":0.53100000000000003,"15min":0.49099999999999999},"max":{"1min":1.004,"5min":1.171,"15min":1.171},"last":0.81999999999999995}]}]},{"osd":5,"up_from":14,"seq":60129543241,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311800,"kb_used_data":2224,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060040,"statfs":{"total":96636764160,"available":96317480960,"internally_reserved":0,"allocated":2277376,"data_stored":1121247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78600000000000003,"15min":0.78700000000000003},"min":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.436},"max":{"1min":0.998,"5min":1.585,"15min":1.585},"last":0.629},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.83099999999999996,"15min":0.82199999999999995},"min":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.673,"15min":1.673},"last":0.70899999999999996}]},{"osd":1,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.56299999999999994,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0189999999999999,"15min":1.0860000000000001},"last":0.746},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.75800000000000001,"15min":0.76900000000000002},"min":{"1min":0.54800000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98899999999999999,"5min":1.0009999999999999,"15min":1.139},"last":0.84699999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73799999999999999,"15min":0.74399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91700000000000004,"5min":1.0349999999999999,"15min":1.0580000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75900000000000001,"15min":0.76000000000000001},"min":{"1min":0.61499999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79000000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55600000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.93799999999999994,"5min":1.069,"15min":1.069},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.752,"15min":0.748},"min":{"1min":0.56999999999999995,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":0.91100000000000003,"5min":1.0880000000000001,"15min":1.161},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.63},"min":{"1min":0.48199999999999998,"5min":0.44800000000000001,"15min":0.40999999999999998},"max":{"1min":0.92500000000000004,"5min":0.94399999999999995,"15min":0.97499999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.63},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.41599999999999998},"max":{"1min":0.80400000000000005,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.442,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.5,"5min":0.46700000000000003,"15min":0.46400000000000002},"max":{"1min":1.0289999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.505,"5min":0.47399999999999998,"15min":0.46500000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76800000000000002,"15min":0.75800000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.81299999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:01:46.152 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T11:01:46.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T11:01:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.389+0000 7fa6e5dff700 1 Processor -- start 2023-10-07T11:01:46.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.390+0000 7fa6e5dff700 1 -- start start 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6e5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e0153ac0 0x7fa6e015a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6e5dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e015ab80 con 0x7fa6e00b0690 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6e5dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e0069d40 con 0x7fa6e0153e90 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6e5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0068ca0 con 0x7fa6e0153ac0 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6dffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa6e00b0690 0x7fa6e015ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46878/0 (socket says 172.21.15.29:46878) 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6dffff700 1 -- 172.21.15.29:0/3563655193 learned_addr learned my addr 172.21.15.29:0/3563655193 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.391+0000 7fa6df7fe700 1 --2- 172.21.15.29:0/3563655193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e0153ac0 0x7fa6e015a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595200327 0 0) 0x7fa6d8001490 con 0x7fa6e00b0690 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c8003570 con 0x7fa6e00b0690 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991614605 0 0) 0x7fa6e0069d40 con 0x7fa6e0153e90 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8001490 con 0x7fa6e0153e90 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6df7fe700 1 -- 172.21.15.29:0/3563655193 >> v1:172.21.15.144:6789/0 conn(0x7fa6e0153e90 legacy=0x7fa6e00aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6df7fe700 1 -- 172.21.15.29:0/3563655193 >> v1:172.21.15.29:6789/0 conn(0x7fa6e00b0690 legacy=0x7fa6e015ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612568097 0 0) 0x7fa6c8003570 con 0x7fa6e00b0690 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6df7fe700 1 -- 172.21.15.29:0/3563655193 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0068f80 con 0x7fa6e0153ac0 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.392+0000 7fa6df7fe700 1 --2- 172.21.15.29:0/3563655193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e0153ac0 0x7fa6e015a640 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009910 tx=0x7fa6d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fedf92b694b7e579 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8001980 con 0x7fa6e0153ac0 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d0011420 con 0x7fa6e0153ac0 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6de7fc700 1 -- 172.21.15.29:0/3563655193 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00117c0 con 0x7fa6e0153ac0 2023-10-07T11:01:46.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 -- 172.21.15.29:0/3563655193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e0153ac0 msgr2=0x7fa6e015a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/3563655193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e0153ac0 0x7fa6e015a640 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009910 tx=0x7fa6d00091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 -- 172.21.15.29:0/3563655193 shutdown_connections 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/3563655193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e0153ac0 0x7fa6e015a640 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 -- 172.21.15.29:0/3563655193 >> 172.21.15.29:0/3563655193 conn(0x7fa6e0067610 msgr2=0x7fa6e0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 -- 172.21.15.29:0/3563655193 shutdown_connections 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.393+0000 7fa6e5dff700 1 -- 172.21.15.29:0/3563655193 wait complete. 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.396+0000 7fa6e5dff700 1 Processor -- start 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.396+0000 7fa6e5dff700 1 -- start start 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6e5dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 0x7fa6e015d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6e5dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e0153ac0 0x7fa6e0151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6e5dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e00cde60 0x7fa6e00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6e5dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e015cca0 con 0x7fa6e00b0690 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6e5dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e015d670 con 0x7fa6e0153ac0 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6e5dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0152450 con 0x7fa6e00cde60 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 0x7fa6e015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 0x7fa6e015d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37146/0 (socket says 172.21.15.29:37146) 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.397+0000 7fa6df7fe700 1 -- 172.21.15.29:0/1643117878 learned_addr learned my addr 172.21.15.29:0/1643117878 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6dffff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e00cde60 0x7fa6e00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6deffd700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e0153ac0 0x7fa6e0151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6df7fe700 1 -- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e00cde60 msgr2=0x7fa6e00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6df7fe700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e00cde60 0x7fa6e00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6df7fe700 1 -- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e0153ac0 msgr2=0x7fa6e0151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6df7fe700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e0153ac0 0x7fa6e0151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6df7fe700 1 -- 172.21.15.29:0/1643117878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0009420 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6df7fe700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 0x7fa6e015d130 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00091e0 tx=0x7fa6d0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d001cc50 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e015c090 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d001cdb0 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d001a6a0 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.398+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e015c650 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.400+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e00718b0 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.400+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa6d001aa90 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.401+0000 7fa6c77fe700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa6cc066f30 0x7fa6cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.401+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7fa6d0086210 con 0x7fa6e00b0690 2023-10-07T11:01:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.402+0000 7fa6deffd700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa6cc066f30 0x7fa6cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.402+0000 7fa6deffd700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa6cc066f30 0x7fa6cc0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00cee30 tx=0x7fa6d4009040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:46.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.405+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6d00532c0 con 0x7fa6e00b0690 2023-10-07T11:01:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.690+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa6e00aff50 con 0x7fa6e00b0690 2023-10-07T11:01:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.694+0000 7fa6c77fe700 1 -- 172.21.15.29:0/1643117878 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa6d00568e0 con 0x7fa6e00b0690 2023-10-07T11:01:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa6cc066f30 msgr2=0x7fa6cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa6cc066f30 0x7fa6cc0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00cee30 tx=0x7fa6d4009040 comp rx=0 tx=0).stop 2023-10-07T11:01:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 msgr2=0x7fa6e015d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 0x7fa6e015d130 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00091e0 tx=0x7fa6d0016000 comp rx=0 tx=0).stop 2023-10-07T11:01:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 shutdown_connections 2023-10-07T11:01:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa6cc066f30 0x7fa6cc0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6e00cde60 0x7fa6e00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e00b0690 0x7fa6e015d130 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 --2- 172.21.15.29:0/1643117878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6e0153ac0 0x7fa6e0151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 >> 172.21.15.29:0/1643117878 conn(0x7fa6e0067610 msgr2=0x7fa6e0156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 shutdown_connections 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.697+0000 7fa6e5dff700 1 -- 172.21.15.29:0/1643117878 wait complete. 2023-10-07T11:01:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T11:01:46.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:01:46.859 INFO:tasks.ceph.mds.d.smithi144.stderr:2023-10-07T11:01:46.859+0000 7f252fbf1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T11:01:46.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.910+0000 7f8e02fff700 1 Processor -- start 2023-10-07T11:01:46.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.910+0000 7f8e02fff700 1 -- start start 2023-10-07T11:01:46.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.911+0000 7f8e02fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 0x7f8dfc15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.911+0000 7f8e02fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8dfc0af400 0x7f8dfc0af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.911+0000 7f8e02fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc14d790 con 0x7f8dfc0ad370 2023-10-07T11:01:46.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.911+0000 7f8e02fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc0ae5d0 con 0x7f8dfc0af400 2023-10-07T11:01:46.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.911+0000 7f8e02fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc15a850 con 0x7f8dfc159ec0 2023-10-07T11:01:46.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8e00d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8dfc0ad370 0x7f8dfc0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46886/0 (socket says 172.21.15.29:46886) 2023-10-07T11:01:46.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8e00d9b700 1 -- 172.21.15.29:0/2005602156 learned_addr learned my addr 172.21.15.29:0/2005602156 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8e0159c700 1 --2- 172.21.15.29:0/2005602156 >> v2:172.21.15.144:3300/0 conn(0x7f8dfc0af400 0x7f8dfc0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8dfbfff700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 0x7f8dfc15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205453180 0 0) 0x7f8dfc14d790 con 0x7f8dfc0ad370 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dec003570 con 0x7f8dfc0ad370 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.912+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262350978 0 0) 0x7f8dec003570 con 0x7f8dfc0ad370 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 msgr2=0x7f8dfc15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 0x7f8dfc15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 >> v2:172.21.15.144:3300/0 conn(0x7f8dfc0af400 msgr2=0x7f8dfc0af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 --2- 172.21.15.29:0/2005602156 >> v2:172.21.15.144:3300/0 conn(0x7f8dfc0af400 0x7f8dfc0af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0ad740 con 0x7f8dfc0ad370 2023-10-07T11:01:46.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f8df0001470 con 0x7f8dfc0ad370 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 >> v1:172.21.15.29:6789/0 conn(0x7f8dfc0ad370 legacy=0x7f8dfc0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dec003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 0x7f8dec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dec004d50 0x7f8dec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0ae5d0 con 0x7f8dec003ae0 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc15a850 con 0x7f8dec004d50 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec00dd50 con 0x7f8dfc159ec0 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dec003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8dfbfff700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 0x7f8dec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e0159c700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dec004d50 0x7f8dec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e00d9b700 1 -- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 msgr2=0x7f8dec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc159ec0 0x7f8dec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e00d9b700 1 -- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dec004d50 msgr2=0x7f8dec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dec004d50 0x7f8dec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.913+0000 7f8e00d9b700 1 -- 172.21.15.29:0/2005602156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec003570 con 0x7f8dec003ae0 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dec003ef0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8df000c2a0 tx=0x7f8dfc1578d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18366156d38802a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000cb70 con 0x7f8dec003ae0 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df000ccd0 con 0x7f8dec003ae0 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8dfb7fe700 1 -- 172.21.15.29:0/2005602156 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0005b70 con 0x7f8dec003ae0 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8e02fff700 1 -- 172.21.15.29:0/2005602156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 msgr2=0x7f8dec003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8e02fff700 1 --2- 172.21.15.29:0/2005602156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dec003ef0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8df000c2a0 tx=0x7f8dfc1578d0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.914+0000 7f8e0159c700 1 -- 172.21.15.29:0/2005602156 reap_dead start 2023-10-07T11:01:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.915+0000 7f8e02fff700 1 -- 172.21.15.29:0/2005602156 shutdown_connections 2023-10-07T11:01:46.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.915+0000 7f8e02fff700 1 -- 172.21.15.29:0/2005602156 >> 172.21.15.29:0/2005602156 conn(0x7f8dfc0680a0 msgr2=0x7f8dfc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:46.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.915+0000 7f8e02fff700 1 -- 172.21.15.29:0/2005602156 shutdown_connections 2023-10-07T11:01:46.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.915+0000 7f8e02fff700 1 -- 172.21.15.29:0/2005602156 wait complete. 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.916+0000 7f8e02fff700 1 Processor -- start 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.916+0000 7f8e02fff700 1 -- start start 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e02fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dfc0c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e02fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dfc0c4070 0x7f8dfc16c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e02fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc16cea0 0x7f8dfc16d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e02fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0c12b0 con 0x7f8dec003ae0 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e02fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0c2110 con 0x7f8dfc0c4070 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e02fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0c1500 con 0x7f8dfc16cea0 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.917+0000 7f8e00d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dfc0c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dfc0c3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37158/0 (socket says 172.21.15.29:37158) 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 -- 172.21.15.29:0/3797229238 learned_addr learned my addr 172.21.15.29:0/3797229238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e0159c700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc16cea0 0x7f8dfc16d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8dfbfff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dfc0c4070 0x7f8dfc16c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 -- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc16cea0 msgr2=0x7f8dfc16d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc16cea0 0x7f8dfc16d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 -- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dfc0c4070 msgr2=0x7f8dfc16c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dfc0c4070 0x7f8dfc16c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 -- 172.21.15.29:0/3797229238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0c17e0 con 0x7f8dec003ae0 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8dfbfff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dfc0c4070 0x7f8dfc16c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8e00d9b700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dfc0c3b30 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004730 tx=0x7f8df0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:46.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.918+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000cb70 con 0x7f8dec003ae0 2023-10-07T11:01:46.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.919+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df0000f50 con 0x7f8dec003ae0 2023-10-07T11:01:46.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.919+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc0c1ac0 con 0x7f8dec003ae0 2023-10-07T11:01:46.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.919+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0007510 con 0x7f8dec003ae0 2023-10-07T11:01:46.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.919+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc0c5960 con 0x7f8dec003ae0 2023-10-07T11:01:46.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.920+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc04c3c0 con 0x7f8dec003ae0 2023-10-07T11:01:46.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.921+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8df0019030 con 0x7f8dec003ae0 2023-10-07T11:01:46.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.921+0000 7f8df8ff9700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8de0066f30 0x7f8de00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:46.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.922+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f8df0085ce0 con 0x7f8dec003ae0 2023-10-07T11:01:46.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.922+0000 7f8dfbfff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8de0066f30 0x7f8de00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:46.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.923+0000 7f8dfbfff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8de0066f30 0x7f8de00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8de80093f0 tx=0x7f8de8008040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:46.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:46.926+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8df0052d90 con 0x7f8dec003ae0 2023-10-07T11:01:47.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.173+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dfc0c12b0 con 0x7f8dec003ae0 2023-10-07T11:01:47.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.174+0000 7f8df8ff9700 1 -- 172.21.15.29:0/3797229238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f8df0017090 con 0x7f8dec003ae0 2023-10-07T11:01:47.174 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:47.174 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:01:40.378766+0000","modified":"2023-10-07T11:01:44.282264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/562306367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":562306367},{"type":"v1","addr":"172.21.15.144:6837","nonce":562306367}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-10-07T11:01:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8de0066f30 msgr2=0x7f8de00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8de0066f30 0x7f8de00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8de80093f0 tx=0x7f8de8008040 comp rx=0 tx=0).stop 2023-10-07T11:01:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 msgr2=0x7f8dfc0c3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dfc0c3b30 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004730 tx=0x7f8df0015000 comp rx=0 tx=0).stop 2023-10-07T11:01:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 shutdown_connections 2023-10-07T11:01:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f8de0066f30 0x7f8de00693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8dfc16cea0 0x7f8dfc16d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dec003ae0 0x7f8dfc0c3b30 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 --2- 172.21.15.29:0/3797229238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dfc0c4070 0x7f8dfc16c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 >> 172.21.15.29:0/3797229238 conn(0x7f8dfc0680a0 msgr2=0x7f8dfc158910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 shutdown_connections 2023-10-07T11:01:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.176+0000 7f8e02fff700 1 -- 172.21.15.29:0/3797229238 wait complete. 2023-10-07T11:01:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-07T11:01:47.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T11:01:40.378766+0000', 'modified': '2023-10-07T11:01:44.282264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9636}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9636': {'gid': 9636, 'name': 'd', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.144:6837/562306367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6833', 'nonce': 562306367}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 562306367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T11:01:47.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T11:01:47.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-07T11:01:47.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.798+0000 7f069f3ff700 1 Processor -- start 2023-10-07T11:01:47.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.798+0000 7f069f3ff700 1 -- start start 2023-10-07T11:01:47.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.799+0000 7f069f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06980b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.799+0000 7f069f3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06980b0710 0x7f06980b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.799+0000 7f069f3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0698154090 con 0x7f069815ade0 2023-10-07T11:01:47.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.799+0000 7f069f3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0698159f60 con 0x7f06980b0710 2023-10-07T11:01:47.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.799+0000 7f069f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069815a240 con 0x7f0698153cc0 2023-10-07T11:01:47.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06980b0710 0x7f06980b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06980b0710 0x7f06980b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37160/0 (socket says 172.21.15.29:37160) 2023-10-07T11:01:47.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 -- 172.21.15.29:0/2483177328 learned_addr learned my addr 172.21.15.29:0/2483177328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:47.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d19b700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06980b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 -- 172.21.15.29:0/2483177328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 msgr2=0x7f06980b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06980b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 -- 172.21.15.29:0/2483177328 >> v1:172.21.15.144:6789/0 conn(0x7f069815ade0 legacy=0x7f06980b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 -- 172.21.15.29:0/2483177328 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069815a520 con 0x7f06980b0710 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d19b700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06980b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.800+0000 7f069d99c700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980b1d00 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0690009f20 tx=0x7f0690009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e4ce273da39125 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f068ffff700 1 -- 172.21.15.29:0/2483177328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069000e070 con 0x7f06980b0710 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f068ffff700 1 -- 172.21.15.29:0/2483177328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069000ba80 con 0x7f06980b0710 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f068ffff700 1 -- 172.21.15.29:0/2483177328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06900092e0 con 0x7f06980b0710 2023-10-07T11:01:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 -- 172.21.15.29:0/2483177328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 msgr2=0x7f06980b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980b1d00 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0690009f20 tx=0x7f0690009840 comp rx=0 tx=0).stop 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 -- 172.21.15.29:0/2483177328 shutdown_connections 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980b1d00 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 --2- 172.21.15.29:0/2483177328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06980b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 -- 172.21.15.29:0/2483177328 >> 172.21.15.29:0/2483177328 conn(0x7f0698067610 msgr2=0x7f0698067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 -- 172.21.15.29:0/2483177328 shutdown_connections 2023-10-07T11:01:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.801+0000 7f069f3ff700 1 -- 172.21.15.29:0/2483177328 wait complete. 2023-10-07T11:01:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.803+0000 7f069f3ff700 1 Processor -- start 2023-10-07T11:01:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.803+0000 7f069f3ff700 1 -- start start 2023-10-07T11:01:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069f3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069f3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06981528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069f3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f069815ade0 0x7f0698158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069f3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980c8cd0 con 0x7f06980b0710 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069f3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980c8fb0 con 0x7f069815ade0 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069f3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06980c9290 con 0x7f0698153cc0 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37164/0 (socket says 172.21.15.29:37164) 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069d19b700 1 -- 172.21.15.29:0/1458534740 learned_addr learned my addr 172.21.15.29:0/1458534740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069d99c700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f069815ade0 0x7f0698158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.804+0000 7f069c99a700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06981528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069d19b700 1 -- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 msgr2=0x7f06981528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069d19b700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06981528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069d19b700 1 -- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f069815ade0 msgr2=0x7f0698158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069d19b700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f069815ade0 0x7f0698158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069d19b700 1 -- 172.21.15.29:0/1458534740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690009ae0 con 0x7f06980b0710 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069d19b700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980c9a90 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f069400e920 tx=0x7f0694008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f068d7fa700 1 -- 172.21.15.29:0/1458534740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694005a00 con 0x7f06980b0710 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f068d7fa700 1 -- 172.21.15.29:0/1458534740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0694005b60 con 0x7f06980b0710 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980c9570 con 0x7f06980b0710 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.805+0000 7f068d7fa700 1 -- 172.21.15.29:0/1458534740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694006020 con 0x7f06980b0710 2023-10-07T11:01:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.806+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0698152e20 con 0x7f06980b0710 2023-10-07T11:01:47.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.809+0000 7f068d7fa700 1 -- 172.21.15.29:0/1458534740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f06940063d0 con 0x7f06980b0710 2023-10-07T11:01:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.809+0000 7f068d7fa700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f067c066f30 0x7f067c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.809+0000 7f068d7fa700 1 -- 172.21.15.29:0/1458534740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f06940846f0 con 0x7f06980b0710 2023-10-07T11:01:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.810+0000 7f069c99a700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f067c066f30 0x7f067c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.811+0000 7f069c99a700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f067c066f30 0x7f067c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0698158590 tx=0x7f0688007ec0 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.811+0000 7f06adbc1b80 1 Processor -- start 2023-10-07T11:01:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.811+0000 7f06adbc1b80 1 -- start start 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465e2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d465e3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d464ac310 con 0x556d465bf530 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d464e9130 con 0x556d465d9a40 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d464e6af0 con 0x556d465e2bc0 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f069c99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f069c99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465e2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37168/0 (socket says 172.21.15.29:37168) 2023-10-07T11:01:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.812+0000 7f069c99a700 1 -- 172.21.15.29:0/3664619256 learned_addr learned my addr 172.21.15.29:0/3664619256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069d19b700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465d9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069d99c700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d465e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069c99a700 1 -- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 msgr2=0x556d465e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069c99a700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d465e3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069c99a700 1 -- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 msgr2=0x556d465d9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069c99a700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465d9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069c99a700 1 -- 172.21.15.29:0/3664619256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d463d50a0 con 0x556d465bf530 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069d19b700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069d99c700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d465e3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f069c99a700 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465e2680 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f06880028c0 tx=0x7f0688015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1f5e77fa6a0951 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f06827fc700 1 -- 172.21.15.29:0/3664619256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688003670 con 0x556d465bf530 2023-10-07T11:01:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f06827fc700 1 -- 172.21.15.29:0/3664619256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688014660 con 0x556d465bf530 2023-10-07T11:01:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.813+0000 7f06827fc700 1 -- 172.21.15.29:0/3664619256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688014a00 con 0x556d465bf530 2023-10-07T11:01:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 -- 172.21.15.29:0/3664619256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 msgr2=0x556d465e2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465e2680 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f06880028c0 tx=0x7f0688015000 comp rx=0 tx=0).stop 2023-10-07T11:01:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 -- 172.21.15.29:0/3664619256 shutdown_connections 2023-10-07T11:01:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d465e3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/3664619256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465e2680 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 -- 172.21.15.29:0/3664619256 >> 172.21.15.29:0/3664619256 conn(0x556d465dce70 msgr2=0x556d465dd240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 -- 172.21.15.29:0/3664619256 shutdown_connections 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.814+0000 7f06adbc1b80 1 -- 172.21.15.29:0/3664619256 wait complete. 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.815+0000 7f06adbc1b80 1 -- start start 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.815+0000 7f06adbc1b80 1 Processor -- start 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465bd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 0x556d465e0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465e2bc0 0x556d465e14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d464e28e0 con 0x556d465d9a40 2023-10-07T11:01:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4648e980 con 0x556d465e2bc0 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d4648e620 con 0x556d465bf530 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f06adbc1b80 10 client.0 _ll_register_callbacks cb 0x556d462f1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f069c99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 0x556d465e0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f069c99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 0x556d465e0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37178/0 (socket says 172.21.15.29:37178) 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.816+0000 7f069c99a700 1 -- 172.21.15.29:0/2830218354 learned_addr learned my addr 172.21.15.29:0/2830218354 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069d19b700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465bd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069d99c700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465e2bc0 0x556d465e14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069c99a700 1 -- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 msgr2=0x556d465bd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069c99a700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465bd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069c99a700 1 -- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465e2bc0 msgr2=0x556d465e14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069c99a700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465e2bc0 0x556d465e14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069c99a700 1 -- 172.21.15.29:0/2830218354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0698154090 con 0x556d465d9a40 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069d99c700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465e2bc0 0x556d465e14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f069c99a700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 0x556d465e0f70 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0688002d30 tx=0x7f068801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e5bc67add51afc server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f06777fe700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f06777fe700 1 -- 172.21.15.29:0/2830218354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068802b700 con 0x556d465d9a40 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f06777fe700 1 -- 172.21.15.29:0/2830218354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f068801c040 con 0x556d465d9a40 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.817+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d46587560 con 0x556d465d9a40 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06777fe700 1 -- 172.21.15.29:0/2830218354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f068802ba70 con 0x556d465d9a40 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 10 client.9768 fetch_fsmap learned FSMap version 94 2023-10-07T11:01:47.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x556d464b13c0 con 0x556d465d9a40 2023-10-07T11:01:47.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06777fe700 1 -- 172.21.15.29:0/2830218354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556d46587560 con 0x556d465d9a40 2023-10-07T11:01:47.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06777fe700 1 -- 172.21.15.29:0/2830218354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 94) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f068802bbd0 con 0x556d465d9a40 2023-10-07T11:01:47.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 10 client.9768 fetch_fsmap finished waiting for FSMap version 94 2023-10-07T11:01:47.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 10 client.9768 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-07T11:01:47.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 20 client.9768 populate_metadata read hostname 'smithi029' 2023-10-07T11:01:47.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465ec6e0 0x556d465ecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 4 client.9768 mds_command: new command op to 9636 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T11:01:47.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.818+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 --> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d465640e0 con 0x556d465ec6e0 2023-10-07T11:01:47.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.819+0000 7f069d19b700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465ec6e0 0x556d465ecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.819+0000 7f069d19b700 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465ec6e0 0x556d465ecab0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b46b1744f3ea947 server_cookie=8f1c3d6b519a21ea in_seq=0 out_seq=0 2023-10-07T11:01:47.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.819+0000 7f06777fe700 10 client.9768 ms_handle_connect on v2:172.21.15.144:6833/562306367 2023-10-07T11:01:47.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.820+0000 7f06777fe700 1 -- 172.21.15.29:0/2830218354 <== mds.0 v2:172.21.15.144:6833/562306367 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x556d465640e0 con 0x556d465ec6e0 2023-10-07T11:01:47.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.820+0000 7f06777fe700 10 client.9768 handle_command_reply: tid=0 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.820+0000 7f06adbc1b80 1 client.9768 shutdown 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.820+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 msgr2=0x556d465e0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.820+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 0x556d465e0f70 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0688002d30 tx=0x7f068801f000 comp rx=0 tx=0).stop 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 shutdown_connections 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465e2bc0 0x556d465e14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465ec6e0 0x556d465ecab0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465bd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2830218354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465d9a40 0x556d465e0f70 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06777fe700 0 client.9768 ms_handle_reset on v2:172.21.15.144:6833/562306367 2023-10-07T11:01:47.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 >> 172.21.15.29:0/2830218354 conn(0x556d465dce70 msgr2=0x556d4658ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 shutdown_connections 2023-10-07T11:01:47.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2830218354 wait complete. 2023-10-07T11:01:47.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.821+0000 7f06adbc1b80 20 client.9768 trim_cache size 0 max 16384 2023-10-07T11:01:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.838+0000 7f06adbc1b80 1 Processor -- start 2023-10-07T11:01:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.838+0000 7f06adbc1b80 1 -- start start 2023-10-07T11:01:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465889a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 0x556d465b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4637a1e0 con 0x556d465e2bc0 2023-10-07T11:01:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d465dfb00 con 0x556d465d9a40 2023-10-07T11:01:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d465bdd60 con 0x556d465bf530 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f069d99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 0x556d465b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f069d99c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 0x556d465b87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37184/0 (socket says 172.21.15.29:37184) 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.839+0000 7f069d99c700 1 -- 172.21.15.29:0/2601942800 learned_addr learned my addr 172.21.15.29:0/2601942800 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d19b700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069c99a700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d99c700 1 -- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 msgr2=0x556d465f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d99c700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d99c700 1 -- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 msgr2=0x556d465889a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d99c700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465889a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d99c700 1 -- 172.21.15.29:0/2601942800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d465dcce0 con 0x556d465e2bc0 2023-10-07T11:01:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d19b700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069c99a700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f069d99c700 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 0x556d465b87d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f069000b6c0 tx=0x7f0690018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9da1a165468e06d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f0682ffd700 1 -- 172.21.15.29:0/2601942800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069000bcc0 con 0x556d465e2bc0 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f0682ffd700 1 -- 172.21.15.29:0/2601942800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069000be40 con 0x556d465e2bc0 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.840+0000 7f0682ffd700 1 -- 172.21.15.29:0/2601942800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0690007860 con 0x556d465e2bc0 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2601942800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 msgr2=0x556d465b87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 0x556d465b87d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f069000b6c0 tx=0x7f0690018000 comp rx=0 tx=0).stop 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2601942800 shutdown_connections 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465e2bc0 0x556d465b87d0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465bf530 0x556d465f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/2601942800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2601942800 >> 172.21.15.29:0/2601942800 conn(0x556d465dce70 msgr2=0x556d465c0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2601942800 shutdown_connections 2023-10-07T11:01:47.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.841+0000 7f06adbc1b80 1 -- 172.21.15.29:0/2601942800 wait complete. 2023-10-07T11:01:47.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.842+0000 7f06adbc1b80 1 -- start start 2023-10-07T11:01:47.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.842+0000 7f06adbc1b80 1 Processor -- start 2023-10-07T11:01:47.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465b8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d46627280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d4663e7c0 con 0x556d465bf530 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d46642af0 con 0x556d465d9a40 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d46594b40 con 0x556d465e2bc0 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f06adbc1b80 10 client.0 _ll_register_callbacks cb 0x556d462f1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f069d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465b8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f069d19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465b8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37196/0 (socket says 172.21.15.29:37196) 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f069d19b700 1 -- 172.21.15.29:0/285211522 learned_addr learned my addr 172.21.15.29:0/285211522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f069d99c700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d46627280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.843+0000 7f069c99a700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f069d19b700 1 -- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 msgr2=0x556d46627280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f069d19b700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d46627280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f069d19b700 1 -- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 msgr2=0x556d465b8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f069d19b700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465b8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f069d19b700 1 -- 172.21.15.29:0/285211522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069815a520 con 0x556d465bf530 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f069d19b700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465b8980 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f069400e8f0 tx=0x7f069401d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2145638a2033dcf server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06777fe700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06777fe700 1 -- 172.21.15.29:0/285211522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694070070 con 0x556d465bf530 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06980c7900 con 0x556d465bf530 2023-10-07T11:01:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06777fe700 1 -- 172.21.15.29:0/285211522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f069407e070 con 0x556d465bf530 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06777fe700 1 -- 172.21.15.29:0/285211522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06940538b0 con 0x556d465bf530 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06adbc1b80 10 client.9774 fetch_fsmap learned FSMap version 94 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x556d463491d0 con 0x556d465bf530 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06777fe700 1 -- 172.21.15.29:0/285211522 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06980c7900 con 0x556d465bf530 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.844+0000 7f06777fe700 1 -- 172.21.15.29:0/285211522 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 94) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0694006680 con 0x556d465bf530 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f06adbc1b80 10 client.9774 fetch_fsmap finished waiting for FSMap version 94 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f06adbc1b80 10 client.9774 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f06adbc1b80 20 client.9774 populate_metadata read hostname 'smithi029' 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465dd6b0 0x556d465dda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f06adbc1b80 4 client.9774 mds_command: new command op to 9636 tid=0[{"prefix": "status"}] 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 --> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0688014660 con 0x556d465dd6b0 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.845+0000 7f069c99a700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465dd6b0 0x556d465dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f069c99a700 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465dd6b0 0x556d465dda80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37a1a05ac9c7a7ab server_cookie=a6bf21f781a43046 in_seq=0 out_seq=0 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06777fe700 10 client.9774 ms_handle_connect on v2:172.21.15.144:6833/562306367 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06777fe700 1 -- 172.21.15.29:0/285211522 <== mds.0 v2:172.21.15.144:6833/562306367 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7f0688014660 con 0x556d465dd6b0 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06777fe700 10 client.9774 handle_command_reply: tid=0 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 client.9774 shutdown 2023-10-07T11:01:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 msgr2=0x556d465b8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465b8980 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f069400e8f0 tx=0x7f069401d040 comp rx=0 tx=0).stop 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 shutdown_connections 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:6833/562306367,v1:172.21.15.144:6837/562306367] conn(0x556d465dd6b0 0x556d465dda80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x556d465e2bc0 0x556d46627280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d465bf530 0x556d465b8980 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 --2- 172.21.15.29:0/285211522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556d465d9a40 0x556d465b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06777fe700 0 client.9774 ms_handle_reset on v2:172.21.15.144:6833/562306367 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 >> 172.21.15.29:0/285211522 conn(0x556d465dce70 msgr2=0x556d46589450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 shutdown_connections 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.846+0000 7f06adbc1b80 1 -- 172.21.15.29:0/285211522 wait complete. 2023-10-07T11:01:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f06adbc1b80 20 client.9774 trim_cache size 0 max 16384 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 9636, 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.9706336049999997, 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 94, 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 182, 2023-10-07T11:01:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 182, 2023-10-07T11:01:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.637266760999999 2023-10-07T11:01:47.854 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:01:47.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f067c066f30 msgr2=0x7f067c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f067c066f30 0x7f067c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0698158590 tx=0x7f0688007ec0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 msgr2=0x7f06980c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980c9a90 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f069400e920 tx=0x7f0694008040 comp rx=0 tx=0).stop 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 shutdown_connections 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f069815ade0 0x7f0698158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06980b0710 0x7f06980c9a90 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f067c066f30 0x7f067c0693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 --2- 172.21.15.29:0/1458534740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0698153cc0 0x7f06981528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 >> 172.21.15.29:0/1458534740 conn(0x7f0698067610 msgr2=0x7f06980b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 shutdown_connections 2023-10-07T11:01:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:47.847+0000 7f069f3ff700 1 -- 172.21.15.29:0/1458534740 wait complete. 2023-10-07T11:01:47.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T11:01:48.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.062+0000 7f3596bff700 1 Processor -- start 2023-10-07T11:01:48.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.062+0000 7f3596bff700 1 -- start start 2023-10-07T11:01:48.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.064+0000 7f3596bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 0x7f35900d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.064+0000 7f3596bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900cbf20 con 0x7f3590159750 2023-10-07T11:01:48.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.064+0000 7f3596bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900cc200 con 0x7f35900ce8a0 2023-10-07T11:01:48.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.064+0000 7f3596bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35900cc4e0 con 0x7f3590159b20 2023-10-07T11:01:48.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.064+0000 7f359499b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3590159750 0x7f35900ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46900/0 (socket says 172.21.15.29:46900) 2023-10-07T11:01:48.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.064+0000 7f359499b700 1 -- 172.21.15.29:0/2929916770 learned_addr learned my addr 172.21.15.29:0/2929916770 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358ffff700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 0x7f35900d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066513824 0 0) 0x7f35900cbf20 con 0x7f3590159750 2023-10-07T11:01:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580003570 con 0x7f3590159750 2023-10-07T11:01:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3194025669 0 0) 0x7f3580003570 con 0x7f3590159750 2023-10-07T11:01:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 msgr2=0x7f35900d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 0x7f35900d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 >> v1:172.21.15.144:6789/0 conn(0x7f35900ce8a0 legacy=0x7f35900c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900cc7c0 con 0x7f3590159750 2023-10-07T11:01:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f35840015f0 con 0x7f3590159750 2023-10-07T11:01:48.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 >> v1:172.21.15.29:6789/0 conn(0x7f3590159750 legacy=0x7f35900ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 0x7f3580003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580003fd0 0x7f3580004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580004d50 0x7f35800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900cc4e0 con 0x7f3580004d50 2023-10-07T11:01:48.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358000bb40 con 0x7f3580003fd0 2023-10-07T11:01:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358000bd00 con 0x7f3590159b20 2023-10-07T11:01:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.065+0000 7f359519c700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580004d50 0x7f35800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359499b700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 0x7f3580003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f358ffff700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580003fd0 0x7f3580004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359519c700 1 -- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 msgr2=0x7f3580003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359519c700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3590159b20 0x7f3580003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359519c700 1 -- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580003fd0 msgr2=0x7f3580004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359519c700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580003fd0 0x7f3580004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359519c700 1 -- 172.21.15.29:0/2929916770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580003570 con 0x7f3580004d50 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f359519c700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580004d50 0x7f35800051c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f35880020a0 tx=0x7f35880016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f5826cbcb8f3cd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35880019e0 con 0x7f3580004d50 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358800c8b0 con 0x7f3580004d50 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.066+0000 7f358f7fe700 1 -- 172.21.15.29:0/2929916770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358800cc70 con 0x7f3580004d50 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f3596bff700 1 -- 172.21.15.29:0/2929916770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580004d50 msgr2=0x7f35800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f3596bff700 1 --2- 172.21.15.29:0/2929916770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580004d50 0x7f35800051c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f35880020a0 tx=0x7f35880016d0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f359519c700 1 -- 172.21.15.29:0/2929916770 reap_dead start 2023-10-07T11:01:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f3596bff700 1 -- 172.21.15.29:0/2929916770 shutdown_connections 2023-10-07T11:01:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f3596bff700 1 -- 172.21.15.29:0/2929916770 >> 172.21.15.29:0/2929916770 conn(0x7f3590063a10 msgr2=0x7f35900d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f3596bff700 1 -- 172.21.15.29:0/2929916770 shutdown_connections 2023-10-07T11:01:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.067+0000 7f3596bff700 1 -- 172.21.15.29:0/2929916770 wait complete. 2023-10-07T11:01:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.069+0000 7f3596bff700 1 Processor -- start 2023-10-07T11:01:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.069+0000 7f3596bff700 1 -- start start 2023-10-07T11:01:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f3596bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580004d50 0x7f35900ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f3596bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 0x7f3590151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f3596bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f359014ada0 0x7f359014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f3596bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900cd4a0 con 0x7f35900b4f80 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f3596bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900cd780 con 0x7f3580004d50 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f3596bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35900c7670 con 0x7f359014ada0 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.070+0000 7f358ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 0x7f3590151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 0x7f3590151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37216/0 (socket says 172.21.15.29:37216) 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 -- 172.21.15.29:0/4199701675 learned_addr learned my addr 172.21.15.29:0/4199701675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f359499b700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580004d50 0x7f35900ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f359519c700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f359014ada0 0x7f359014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 -- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f359014ada0 msgr2=0x7f359014b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f359014ada0 0x7f359014b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 -- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580004d50 msgr2=0x7f35900ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580004d50 0x7f35900ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 -- 172.21.15.29:0/4199701675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900c7950 con 0x7f35900b4f80 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f359499b700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580004d50 0x7f35900ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f359519c700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f359014ada0 0x7f359014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358ffff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 0x7f3590151c20 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f357c006ac0 tx=0x7f357c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c00fa10 con 0x7f35900b4f80 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c00fb70 con 0x7f35900b4f80 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c014440 con 0x7f35900b4f80 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.071+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35900c7c30 con 0x7f35900b4f80 2023-10-07T11:01:48.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.072+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35900b53f0 con 0x7f35900b4f80 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.073+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359004c3c0 con 0x7f35900b4f80 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.074+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f357c014870 con 0x7f35900b4f80 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.075+0000 7f358cff9700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3574066f30 0x7f35740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.075+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f357c018030 con 0x7f35900b4f80 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.075+0000 7f359499b700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3574066f30 0x7f35740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.076+0000 7f359499b700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3574066f30 0x7f35740693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3584002e10 tx=0x7f3584009040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.079+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f357c0522b0 con 0x7f35900b4f80 2023-10-07T11:01:48.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.325+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f35900cd5e0 con 0x7f35900b4f80 2023-10-07T11:01:48.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.326+0000 7f358cff9700 1 -- 172.21.15.29:0/4199701675 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) v1 ==== 53+0+73 (secure 0 0 0) 0x7f35900cd5e0 con 0x7f35900b4f80 2023-10-07T11:01:48.326 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3574066f30 msgr2=0x7f35740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3574066f30 0x7f35740693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3584002e10 tx=0x7f3584009040 comp rx=0 tx=0).stop 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 msgr2=0x7f3590151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 0x7f3590151c20 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f357c006ac0 tx=0x7f357c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 shutdown_connections 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f3574066f30 0x7f35740693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f359014ada0 0x7f359014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3580004d50 0x7f35900ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 --2- 172.21.15.29:0/4199701675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35900b4f80 0x7f3590151c20 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 >> 172.21.15.29:0/4199701675 conn(0x7f3590063a10 msgr2=0x7f3590068440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 shutdown_connections 2023-10-07T11:01:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:48.328+0000 7f3596bff700 1 -- 172.21.15.29:0/4199701675 wait complete. 2023-10-07T11:01:48.345 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T11:01:48.345 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T11:01:48.346 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T11:01:48.346 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T11:01:48.346 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T11:01:48.346 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T11:01:48.347 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T11:01:48.347 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T11:01:48.347 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T11:01:48.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:48.347 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T11:01:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36720sec preferred_lft 36720sec 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T11:01:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T11:01:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T11:01:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T11:01:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:01:48.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:48.398 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T11:01:48.504 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T11:01:48.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:48.506 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T11:01:48.557 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T11:01:48.557 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T11:01:48.557 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T11:01:48.557 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T11:01:48.557 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T11:01:48.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:48.558 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T11:01:48.558 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T11:01:48.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T11:01:48.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T11:01:48.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T11:01:48.558 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T11:01:48.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:48.732 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T11:01:48.779 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T11:01:48.780 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T11:01:48.780 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T11:01:48.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:48.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T11:01:48.892 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T11:01:48.893 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T11:01:48.940 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T11:01:48.941 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T11:01:48.941 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T11:01:48.986 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T11:01:48.986 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T11:01:48.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:01:49.127 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:01:49.127 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T11:01:49.127 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T11:01:49.142 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:01:49.142 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T11:01:49.143 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T11:01:49.143 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T11:01:49.143 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T11:01:49.165 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T11:01:49.166 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T11:01:49.170 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T11:01:49.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:01:49.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T11:01:49.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T11:01:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.439+0000 7f0a1c9ff700 1 Processor -- start 2023-10-07T11:01:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.439+0000 7f0a1c9ff700 1 -- start start 2023-10-07T11:01:49.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.440+0000 7f0a1c9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180ae8b0 0x7f0a180b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.440+0000 7f0a1c9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a180b1840 0x7f0a18157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a1c9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a180b1ca0 con 0x7f0a18157290 2023-10-07T11:01:49.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a1c9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a18158ad0 con 0x7f0a180b1840 2023-10-07T11:01:49.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a1c9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1815b140 con 0x7f0a180ae8b0 2023-10-07T11:01:49.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a1659c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0a18157290 0x7f0a180cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46902/0 (socket says 172.21.15.29:46902) 2023-10-07T11:01:49.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a16d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a180b1840 0x7f0a18157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a1659c700 1 -- 172.21.15.29:0/2078146562 learned_addr learned my addr 172.21.15.29:0/2078146562 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:49.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a15d9b700 1 --2- 172.21.15.29:0/2078146562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180ae8b0 0x7f0a180b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a16d9d700 1 -- 172.21.15.29:0/2078146562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180ae8b0 msgr2=0x7f0a180b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a16d9d700 1 --2- 172.21.15.29:0/2078146562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180ae8b0 0x7f0a180b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a16d9d700 1 -- 172.21.15.29:0/2078146562 >> v1:172.21.15.29:6789/0 conn(0x7f0a18157290 legacy=0x7f0a180cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a16d9d700 1 -- 172.21.15.29:0/2078146562 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1815b420 con 0x7f0a180b1840 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.441+0000 7f0a1559a700 1 -- 172.21.15.29:0/2078146562 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638066707 0 0) 0x7f0a08001110 con 0x7f0a18157290 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.442+0000 7f0a16d9d700 1 --2- 172.21.15.29:0/2078146562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a180b1840 0x7f0a18157660 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009e60 tx=0x7f0a10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438dc7d814409854 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.442+0000 7f0a1559a700 1 -- 172.21.15.29:0/2078146562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1000e070 con 0x7f0a180b1840 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.442+0000 7f0a1559a700 1 -- 172.21.15.29:0/2078146562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1000bb60 con 0x7f0a180b1840 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.442+0000 7f0a1559a700 1 -- 172.21.15.29:0/2078146562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10012430 con 0x7f0a180b1840 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.442+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/2078146562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a180b1840 msgr2=0x7f0a18157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.442+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/2078146562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a180b1840 0x7f0a18157660 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009e60 tx=0x7f0a10009780 comp rx=0 tx=0).stop 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.443+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/2078146562 shutdown_connections 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.443+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/2078146562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a180b1840 0x7f0a18157660 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.443+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/2078146562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180ae8b0 0x7f0a180b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.443+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/2078146562 >> 172.21.15.29:0/2078146562 conn(0x7f0a180680a0 msgr2=0x7f0a180cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.443+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/2078146562 shutdown_connections 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.443+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/2078146562 wait complete. 2023-10-07T11:01:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.444+0000 7f0a1c9ff700 1 Processor -- start 2023-10-07T11:01:49.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.444+0000 7f0a1c9ff700 1 -- start start 2023-10-07T11:01:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.445+0000 7f0a1c9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180b1840 0x7f0a180c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.445+0000 7f0a1c9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 0x7f0a180ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.445+0000 7f0a1c9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a18159df0 0x7f0a18146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.445+0000 7f0a1c9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1816dfc0 con 0x7f0a18157290 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.445+0000 7f0a1c9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a180c6850 con 0x7f0a18159df0 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.445+0000 7f0a1c9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a180c6a80 con 0x7f0a180b1840 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 0x7f0a180ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 0x7f0a180ca460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37234/0 (socket says 172.21.15.29:37234) 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 -- 172.21.15.29:0/3388157589 learned_addr learned my addr 172.21.15.29:0/3388157589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 -- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180b1840 msgr2=0x7f0a180c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180b1840 0x7f0a180c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 -- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a18159df0 msgr2=0x7f0a18146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a18159df0 0x7f0a18146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 -- 172.21.15.29:0/3388157589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10009a20 con 0x7f0a18157290 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a15d9b700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 0x7f0a180ca460 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00e930 tx=0x7f0a0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.446+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c00eed0 con 0x7f0a18157290 2023-10-07T11:01:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.447+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c00c850 con 0x7f0a18157290 2023-10-07T11:01:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.447+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a180c6d60 con 0x7f0a18157290 2023-10-07T11:01:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.447+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c00cbf0 con 0x7f0a18157290 2023-10-07T11:01:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.447+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a180c7320 con 0x7f0a18157290 2023-10-07T11:01:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.448+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1804c3c0 con 0x7f0a18157290 2023-10-07T11:01:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.449+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0a0c013470 con 0x7f0a18157290 2023-10-07T11:01:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.450+0000 7f0a067fc700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09f8066f30 0x7f09f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.450+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f0a0c084a00 con 0x7f0a18157290 2023-10-07T11:01:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.450+0000 7f0a1659c700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09f8066f30 0x7f09f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.451+0000 7f0a1659c700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09f8066f30 0x7f09f80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a08001700 tx=0x7f0a08001270 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.454+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a0c051a10 con 0x7f0a18157290 2023-10-07T11:01:49.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.683+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a180b1f70 con 0x7f09f8066f30 2023-10-07T11:01:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.685+0000 7f0a067fc700 1 -- 172.21.15.29:0/3388157589 <== mgr.8069 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0a180b1f70 con 0x7f09f8066f30 2023-10-07T11:01:49.685 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T11:01:49.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:01:49.685 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T11:01:49.685 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:01:49.685 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T11:01:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09f8066f30 msgr2=0x7f09f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09f8066f30 0x7f09f80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a08001700 tx=0x7f0a08001270 comp rx=0 tx=0).stop 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 msgr2=0x7f0a180ca460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 0x7f0a180ca460 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00e930 tx=0x7f0a0c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 shutdown_connections 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a18159df0 0x7f0a18146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f09f8066f30 0x7f09f80693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a180b1840 0x7f0a180c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 --2- 172.21.15.29:0/3388157589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a18157290 0x7f0a180ca460 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 >> 172.21.15.29:0/3388157589 conn(0x7f0a180680a0 msgr2=0x7f0a18158c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.687+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 shutdown_connections 2023-10-07T11:01:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.688+0000 7f0a1c9ff700 1 -- 172.21.15.29:0/3388157589 wait complete. 2023-10-07T11:01:49.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T11:01:49.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.903+0000 7f1e4ffff700 1 Processor -- start 2023-10-07T11:01:49.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.903+0000 7f1e4ffff700 1 -- start start 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.904+0000 7f1e4ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e480d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.904+0000 7f1e4ffff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1e480ce8a0 0x7f1e480c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.904+0000 7f1e4ffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e480cbf20 con 0x7f1e48159750 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.904+0000 7f1e4ffff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e480cc200 con 0x7f1e480ce8a0 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.904+0000 7f1e4ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480cc4e0 con 0x7f1e48159b20 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.904+0000 7f1e4dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1e48159750 0x7f1e480ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46916/0 (socket says 172.21.15.29:46916) 2023-10-07T11:01:49.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4dd9b700 1 -- 172.21.15.29:0/1034903261 learned_addr learned my addr 172.21.15.29:0/1034903261 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:49.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4e59c700 1 --2- 172.21.15.29:0/1034903261 >> v2:172.21.15.144:3300/0 conn(0x7f1e480ce8a0 0x7f1e480c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180404981 0 0) 0x7f1e480cbf20 con 0x7f1e48159750 2023-10-07T11:01:49.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4d59a700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e480d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e34003570 con 0x7f1e48159750 2023-10-07T11:01:49.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177905667 0 0) 0x7f1e34003570 con 0x7f1e48159750 2023-10-07T11:01:49.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 msgr2=0x7f1e480d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:49.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e480d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 >> v2:172.21.15.144:3300/0 conn(0x7f1e480ce8a0 msgr2=0x7f1e480c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 --2- 172.21.15.29:0/1034903261 >> v2:172.21.15.144:3300/0 conn(0x7f1e480ce8a0 0x7f1e480c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e480cc7c0 con 0x7f1e48159750 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f1e440015f0 con 0x7f1e48159750 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 >> v1:172.21.15.29:6789/0 conn(0x7f1e48159750 legacy=0x7f1e480ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34003a60 0x7f1e34003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e34004810 0x7f1e34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480cc200 con 0x7f1e34004810 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480cc4e0 con 0x7f1e34003a60 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.905+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3400ba60 con 0x7f1e48159b20 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e34004810 0x7f1e34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4dd9b700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34003a60 0x7f1e34003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4e59c700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 -- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 msgr2=0x7f1e340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:49.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 -- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34003a60 msgr2=0x7f1e34003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:49.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34003a60 0x7f1e34003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 -- 172.21.15.29:0/1034903261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e34003570 con 0x7f1e34004810 2023-10-07T11:01:49.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4e59c700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e48159b20 0x7f1e340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:49.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.906+0000 7f1e4d59a700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e34004810 0x7f1e34004c80 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009c30 tx=0x7f1e380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c498dd6313985 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800e710 con 0x7f1e34004810 2023-10-07T11:01:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3800e870 con 0x7f1e34004810 2023-10-07T11:01:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4cd99700 1 -- 172.21.15.29:0/1034903261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800ec30 con 0x7f1e34004810 2023-10-07T11:01:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4ffff700 1 -- 172.21.15.29:0/1034903261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e34004810 msgr2=0x7f1e34004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/1034903261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e34004810 0x7f1e34004c80 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009c30 tx=0x7f1e380091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4e59c700 1 -- 172.21.15.29:0/1034903261 reap_dead start 2023-10-07T11:01:49.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4ffff700 1 -- 172.21.15.29:0/1034903261 shutdown_connections 2023-10-07T11:01:49.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4ffff700 1 -- 172.21.15.29:0/1034903261 >> 172.21.15.29:0/1034903261 conn(0x7f1e48063a10 msgr2=0x7f1e480d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:49.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4ffff700 1 -- 172.21.15.29:0/1034903261 shutdown_connections 2023-10-07T11:01:49.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.907+0000 7f1e4ffff700 1 -- 172.21.15.29:0/1034903261 wait complete. 2023-10-07T11:01:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.909+0000 7f1e4ffff700 1 Processor -- start 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.909+0000 7f1e4ffff700 1 -- start start 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34004810 0x7f1e48154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e480b4280 0x7f1e4815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 0x7f1e480b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c7670 con 0x7f1e4815ba10 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4ffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c81b0 con 0x7f1e34004810 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4ffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c7880 con 0x7f1e480b4280 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 0x7f1e480b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 0x7f1e480b5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37256/0 (socket says 172.21.15.29:37256) 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4e59c700 1 -- 172.21.15.29:0/3081560948 learned_addr learned my addr 172.21.15.29:0/3081560948 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4dd9b700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34004810 0x7f1e48154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4d59a700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e480b4280 0x7f1e4815b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4e59c700 1 -- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e480b4280 msgr2=0x7f1e4815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.910+0000 7f1e4e59c700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e480b4280 0x7f1e4815b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e4e59c700 1 -- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34004810 msgr2=0x7f1e48154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e4e59c700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34004810 0x7f1e48154790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e4e59c700 1 -- 172.21.15.29:0/3081560948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e480c7b60 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e4e59c700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 0x7f1e480b5740 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f1e4815d3c0 tx=0x7f1e4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4001b070 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4000f040 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4000e050 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e480c7e40 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.911+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e480b34b0 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.912+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4804c3c0 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.913+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1e40003020 con 0x7f1e4815ba10 2023-10-07T11:01:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.914+0000 7f1e3dffb700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1e2c066f30 0x7f1e2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.914+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f1e40084bd0 con 0x7f1e4815ba10 2023-10-07T11:01:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.915+0000 7f1e4dd9b700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1e2c066f30 0x7f1e2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.915+0000 7f1e4dd9b700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1e2c066f30 0x7f1e2c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1e44002eb0 tx=0x7f1e44009040 comp rx=0 tx=0).ready entity=mgr.8069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:49.919+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e40051be0 con 0x7f1e4815ba10 2023-10-07T11:01:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.175+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f1e480c6590 con 0x7f1e4815ba10 2023-10-07T11:01:50.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.706+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 90) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f1e40051fb0 con 0x7f1e4815ba10 2023-10-07T11:01:50.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.708+0000 7f1e3dffb700 1 -- 172.21.15.29:0/3081560948 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v90) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1e40051be0 con 0x7f1e4815ba10 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1e2c066f30 msgr2=0x7f1e2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1e2c066f30 0x7f1e2c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1e44002eb0 tx=0x7f1e44009040 comp rx=0 tx=0).stop 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 msgr2=0x7f1e480b5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 0x7f1e480b5740 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f1e4815d3c0 tx=0x7f1e4000b040 comp rx=0 tx=0).stop 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 shutdown_connections 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4815ba10 0x7f1e480b5740 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e34004810 0x7f1e48154790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f1e2c066f30 0x7f1e2c0693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 --2- 172.21.15.29:0/3081560948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e480b4280 0x7f1e4815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 >> 172.21.15.29:0/3081560948 conn(0x7f1e48063a10 msgr2=0x7f1e4815edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 shutdown_connections 2023-10-07T11:01:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.711+0000 7f1e4ffff700 1 -- 172.21.15.29:0/3081560948 wait complete. 2023-10-07T11:01:50.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T11:01:50.919 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:50.919+0000 7fb5d877c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:01:50.924 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:50.925+0000 7fe0f9cb71c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:01:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.941+0000 7f6c415ff700 1 Processor -- start 2023-10-07T11:01:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.942+0000 7f6c415ff700 1 -- start start 2023-10-07T11:01:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.943+0000 7f6c415ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c3c14c8d0 0x7f6c3c0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.943+0000 7f6c415ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c159d90 0x7f6c3c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c415ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c157790 con 0x7f6c3c153cc0 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c415ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c0b0950 con 0x7f6c3c14c8d0 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c415ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c157580 con 0x7f6c3c159d90 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3affd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c3c14c8d0 0x7f6c3c0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3a7fc700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c3c153cc0 0x7f6c3c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46930/0 (socket says 172.21.15.29:46930) 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3a7fc700 1 -- 172.21.15.29:0/471628994 learned_addr learned my addr 172.21.15.29:0/471628994 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3b7fe700 1 --2- 172.21.15.29:0/471628994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c159d90 0x7f6c3c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3affd700 1 -- 172.21.15.29:0/471628994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c159d90 msgr2=0x7f6c3c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3affd700 1 --2- 172.21.15.29:0/471628994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c159d90 0x7f6c3c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3affd700 1 -- 172.21.15.29:0/471628994 >> v1:172.21.15.29:6789/0 conn(0x7f6c3c153cc0 legacy=0x7f6c3c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c3affd700 1 -- 172.21.15.29:0/471628994 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c157a70 con 0x7f6c3c14c8d0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.944+0000 7f6c39ffb700 1 -- 172.21.15.29:0/471628994 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286774527 0 0) 0x7f6c3c157790 con 0x7f6c3c153cc0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.945+0000 7f6c3affd700 1 --2- 172.21.15.29:0/471628994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c0afed0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00aa90 tx=0x7f6c2c00a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8ee9bd4cec3b4f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.945+0000 7f6c39ffb700 1 -- 172.21.15.29:0/471628994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c00c970 con 0x7f6c3c14c8d0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.945+0000 7f6c39ffb700 1 -- 172.21.15.29:0/471628994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c00cad0 con 0x7f6c3c14c8d0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.945+0000 7f6c39ffb700 1 -- 172.21.15.29:0/471628994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c00ce70 con 0x7f6c3c14c8d0 2023-10-07T11:01:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.945+0000 7f6c415ff700 1 -- 172.21.15.29:0/471628994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c14c8d0 msgr2=0x7f6c3c0afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.945+0000 7f6c415ff700 1 --2- 172.21.15.29:0/471628994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c0afed0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00aa90 tx=0x7f6c2c00a330 comp rx=0 tx=0).stop 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.946+0000 7f6c415ff700 1 -- 172.21.15.29:0/471628994 shutdown_connections 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.946+0000 7f6c415ff700 1 --2- 172.21.15.29:0/471628994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c159d90 0x7f6c3c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.946+0000 7f6c415ff700 1 --2- 172.21.15.29:0/471628994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c0afed0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.946+0000 7f6c415ff700 1 -- 172.21.15.29:0/471628994 >> 172.21.15.29:0/471628994 conn(0x7f6c3c067610 msgr2=0x7f6c3c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.946+0000 7f6c415ff700 1 -- 172.21.15.29:0/471628994 shutdown_connections 2023-10-07T11:01:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.946+0000 7f6c415ff700 1 -- 172.21.15.29:0/471628994 wait complete. 2023-10-07T11:01:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.947+0000 7f6c415ff700 1 Processor -- start 2023-10-07T11:01:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.947+0000 7f6c415ff700 1 -- start start 2023-10-07T11:01:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.948+0000 7f6c415ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.948+0000 7f6c415ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c153cc0 0x7f6c3c143b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.948+0000 7f6c415ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 0x7f6c3c14efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.948+0000 7f6c415ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c14b5e0 con 0x7f6c3c159d90 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.948+0000 7f6c415ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c14e6a0 con 0x7f6c3c14c8d0 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.948+0000 7f6c415ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c153ab0 con 0x7f6c3c153cc0 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 0x7f6c3c14efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 0x7f6c3c14efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37288/0 (socket says 172.21.15.29:37288) 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 -- 172.21.15.29:0/3775532266 learned_addr learned my addr 172.21.15.29:0/3775532266 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3a7fc700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c153cc0 0x7f6c3c143b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3affd700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 -- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c153cc0 msgr2=0x7f6c3c143b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c153cc0 0x7f6c3c143b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 -- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c14c8d0 msgr2=0x7f6c3c153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 -- 172.21.15.29:0/3775532266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c00a5d0 con 0x7f6c3c159d90 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.949+0000 7f6c3b7fe700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 0x7f6c3c14efc0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009d70 tx=0x7f6c3400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.950+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400caa0 con 0x7f6c3c159d90 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.950+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3400f040 con 0x7f6c3c159d90 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.950+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400ce50 con 0x7f6c3c159d90 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.950+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c14da90 con 0x7f6c3c159d90 2023-10-07T11:01:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.950+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c14e050 con 0x7f6c3c159d90 2023-10-07T11:01:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.951+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c04c3c0 con 0x7f6c3c159d90 2023-10-07T11:01:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.952+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f6c34014690 con 0x7f6c3c159d90 2023-10-07T11:01:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.953+0000 7f6c22ffd700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 0x7f6c28069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.953+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7f6c34085270 con 0x7f6c3c159d90 2023-10-07T11:01:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.953+0000 7f6c3affd700 1 -- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 msgr2=0x7f6c28069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.953+0000 7f6c3affd700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 0x7f6c28069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:50.961+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c34052320 con 0x7f6c3c159d90 2023-10-07T11:01:51.097 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:51.096+0000 7fb5d877c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:01:51.133 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:51.133+0000 7fe0f9cb71c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:01:51.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.154+0000 7f6c3affd700 1 -- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 msgr2=0x7f6c28069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:51.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.154+0000 7f6c3affd700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 0x7f6c28069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:51.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.203+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6c3c0ccd00 con 0x7f6c3c159d90 2023-10-07T11:01:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.207+0000 7f6c22ffd700 1 -- 172.21.15.29:0/3775532266 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6c34055940 con 0x7f6c3c159d90 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.209+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 msgr2=0x7f6c28069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.209+0000 7f6c415ff700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 0x7f6c28069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.209+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 msgr2=0x7f6c3c14efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.209+0000 7f6c415ff700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 0x7f6c3c14efc0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009d70 tx=0x7f6c3400b040 comp rx=0 tx=0).stop 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 shutdown_connections 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3c159d90 0x7f6c3c14efc0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c14c8d0 0x7f6c3c153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6c28066f80 0x7f6c28069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 --2- 172.21.15.29:0/3775532266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c3c153cc0 0x7f6c3c143b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 >> 172.21.15.29:0/3775532266 conn(0x7f6c3c067610 msgr2=0x7f6c3c0afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 shutdown_connections 2023-10-07T11:01:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.210+0000 7f6c415ff700 1 -- 172.21.15.29:0/3775532266 wait complete. 2023-10-07T11:01:51.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T11:01:51.408 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:51.407+0000 7fb5d877c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:01:51.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.429+0000 7fde14bff700 1 Processor -- start 2023-10-07T11:01:51.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.430+0000 7fde14bff700 1 -- start start 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.431+0000 7fde14bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fde10156180 0x7fde100afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.431+0000 7fde14bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde100ad4d0 con 0x7fde100ae570 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.431+0000 7fde14bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde100ad7b0 con 0x7fde100b0520 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.431+0000 7fde14bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde100ada90 con 0x7fde10156180 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.431+0000 7fde0dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fde100ae570 0x7fde100accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46942/0 (socket says 172.21.15.29:46942) 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.431+0000 7fde0dd9b700 1 -- 172.21.15.29:0/1582903626 learned_addr learned my addr 172.21.15.29:0/1582903626 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0e59c700 1 --2- 172.21.15.29:0/1582903626 >> v2:172.21.15.144:3300/0 conn(0x7fde10156180 0x7fde100afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0d59a700 1 -- 172.21.15.29:0/1582903626 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201099854 0 0) 0x7fde100ad4d0 con 0x7fde100ae570 2023-10-07T11:01:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0d59a700 1 -- 172.21.15.29:0/1582903626 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf8003570 con 0x7fde100ae570 2023-10-07T11:01:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0e59c700 1 -- 172.21.15.29:0/1582903626 >> v1:172.21.15.144:6789/0 conn(0x7fde100b0520 legacy=0x7fde100b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0e59c700 1 -- 172.21.15.29:0/1582903626 >> v1:172.21.15.29:6789/0 conn(0x7fde100ae570 legacy=0x7fde100accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0e59c700 1 -- 172.21.15.29:0/1582903626 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100add70 con 0x7fde10156180 2023-10-07T11:01:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.432+0000 7fde0e59c700 1 --2- 172.21.15.29:0/1582903626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde10156180 0x7fde100afc00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fde000098e0 tx=0x7fde000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1afe7adc866eb694 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde0d59a700 1 -- 172.21.15.29:0/1582903626 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000b640 con 0x7fde10156180 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde0d59a700 1 -- 172.21.15.29:0/1582903626 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0000b7a0 con 0x7fde10156180 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde0d59a700 1 -- 172.21.15.29:0/1582903626 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000bb40 con 0x7fde10156180 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde14bff700 1 -- 172.21.15.29:0/1582903626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde10156180 msgr2=0x7fde100afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde14bff700 1 --2- 172.21.15.29:0/1582903626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde10156180 0x7fde100afc00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fde000098e0 tx=0x7fde000091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde14bff700 1 -- 172.21.15.29:0/1582903626 shutdown_connections 2023-10-07T11:01:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde14bff700 1 --2- 172.21.15.29:0/1582903626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde10156180 0x7fde100afc00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde14bff700 1 -- 172.21.15.29:0/1582903626 >> 172.21.15.29:0/1582903626 conn(0x7fde10067610 msgr2=0x7fde10067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:51.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.433+0000 7fde14bff700 1 -- 172.21.15.29:0/1582903626 shutdown_connections 2023-10-07T11:01:51.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.434+0000 7fde14bff700 1 -- 172.21.15.29:0/1582903626 wait complete. 2023-10-07T11:01:51.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.436+0000 7fde14bff700 1 Processor -- start 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.436+0000 7fde14bff700 1 -- start start 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde14bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 0x7fde1015d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde14bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde100b0520 0x7fde10145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde14bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde10156180 0x7fde1015fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde14bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde100ca4d0 con 0x7fde100ae570 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde14bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde100ca720 con 0x7fde100b0520 2023-10-07T11:01:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde14bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde100caa00 con 0x7fde10156180 2023-10-07T11:01:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde0e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 0x7fde1015d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde0e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 0x7fde1015d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37298/0 (socket says 172.21.15.29:37298) 2023-10-07T11:01:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.437+0000 7fde0e59c700 1 -- 172.21.15.29:0/1704639118 learned_addr learned my addr 172.21.15.29:0/1704639118 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0dd9b700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde100b0520 0x7fde10145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0ed9d700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde10156180 0x7fde1015fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0e59c700 1 -- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde10156180 msgr2=0x7fde1015fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0e59c700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde10156180 0x7fde1015fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0e59c700 1 -- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde100b0520 msgr2=0x7fde10145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0e59c700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde100b0520 0x7fde10145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0e59c700 1 -- 172.21.15.29:0/1704639118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00009420 con 0x7fde100ae570 2023-10-07T11:01:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.438+0000 7fde0e59c700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 0x7fde1015d380 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fde000093f0 tx=0x7fde00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.439+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00000c90 con 0x7fde100ae570 2023-10-07T11:01:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.439+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0000b910 con 0x7fde100ae570 2023-10-07T11:01:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.439+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00025670 con 0x7fde100ae570 2023-10-07T11:01:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.439+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde100cace0 con 0x7fde100ae570 2023-10-07T11:01:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.439+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde101602e0 con 0x7fde100ae570 2023-10-07T11:01:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.440+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde100cb200 con 0x7fde100ae570 2023-10-07T11:01:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.441+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fde00023070 con 0x7fde100ae570 2023-10-07T11:01:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.442+0000 7fddfe7fc700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 0x7fddf0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.442+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7fde000861d0 con 0x7fde100ae570 2023-10-07T11:01:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.442+0000 7fde0dd9b700 1 -- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 msgr2=0x7fddf0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.442+0000 7fde0dd9b700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 0x7fddf0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:51.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.449+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde000531e0 con 0x7fde100ae570 2023-10-07T11:01:51.471 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:51.471+0000 7fe0f9cb71c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:01:51.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.643+0000 7fde0dd9b700 1 -- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 msgr2=0x7fddf0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.643+0000 7fde0dd9b700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 0x7fddf0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:51.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.651+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fde100cb200 con 0x7fde100ae570 2023-10-07T11:01:51.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.655+0000 7fddfe7fc700 1 -- 172.21.15.29:0/1704639118 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7fde100cb200 con 0x7fde100ae570 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 msgr2=0x7fddf0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 0x7fddf0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 msgr2=0x7fde1015d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 0x7fde1015d380 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fde000093f0 tx=0x7fde00017000 comp rx=0 tx=0).stop 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 shutdown_connections 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fddf0066f80 0x7fddf0069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde10156180 0x7fde1015fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde100ae570 0x7fde1015d380 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 --2- 172.21.15.29:0/1704639118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde100b0520 0x7fde10145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 >> 172.21.15.29:0/1704639118 conn(0x7fde10067610 msgr2=0x7fde100aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 shutdown_connections 2023-10-07T11:01:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.658+0000 7fde14bff700 1 -- 172.21.15.29:0/1704639118 wait complete. 2023-10-07T11:01:51.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:51.676 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T11:01:51.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-10-07T11:01:51.927 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:51.926+0000 7fb5d877c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:01:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.941+0000 7fb9869ff700 1 Processor -- start 2023-10-07T11:01:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.941+0000 7fb9869ff700 1 -- start start 2023-10-07T11:01:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb9869ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9800ced00 0x7fb9800b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb9869ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9800c8a20 0x7fb9800cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb9869ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800690f0 con 0x7fb9800b0610 2023-10-07T11:01:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb9869ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9800cf2e0 con 0x7fb9800c8a20 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb9869ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9800687c0 con 0x7fb9800ced00 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb97ffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb9800b0610 0x7fb9800c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46952/0 (socket says 172.21.15.29:46952) 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.942+0000 7fb97ffff700 1 -- 172.21.15.29:0/1612073612 learned_addr learned my addr 172.21.15.29:0/1612073612 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb984f9c700 1 --2- 172.21.15.29:0/1612073612 >> v2:172.21.15.29:3300/0 conn(0x7fb9800c8a20 0x7fb9800cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb97f7fe700 1 --2- 172.21.15.29:0/1612073612 >> v2:172.21.15.144:3300/0 conn(0x7fb9800ced00 0x7fb9800b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb97effd700 1 -- 172.21.15.29:0/1612073612 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486639049 0 0) 0x7fb9800690f0 con 0x7fb9800b0610 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb97effd700 1 -- 172.21.15.29:0/1612073612 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb968003570 con 0x7fb9800b0610 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb984f9c700 1 -- 172.21.15.29:0/1612073612 >> v2:172.21.15.144:3300/0 conn(0x7fb9800ced00 msgr2=0x7fb9800b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb984f9c700 1 --2- 172.21.15.29:0/1612073612 >> v2:172.21.15.144:3300/0 conn(0x7fb9800ced00 0x7fb9800b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb984f9c700 1 -- 172.21.15.29:0/1612073612 >> v1:172.21.15.29:6789/0 conn(0x7fb9800b0610 legacy=0x7fb9800c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb984f9c700 1 -- 172.21.15.29:0/1612073612 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb980068aa0 con 0x7fb9800c8a20 2023-10-07T11:01:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.943+0000 7fb97f7fe700 1 --2- 172.21.15.29:0/1612073612 >> v2:172.21.15.144:3300/0 conn(0x7fb9800ced00 0x7fb9800b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:01:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.944+0000 7fb984f9c700 1 --2- 172.21.15.29:0/1612073612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb9800cc7e0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fb9800cc2f0 tx=0x7fb978009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb5dddc9b1c5631 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.944+0000 7fb97effd700 1 -- 172.21.15.29:0/1612073612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800e070 con 0x7fb9800c8a20 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.944+0000 7fb97effd700 1 -- 172.21.15.29:0/1612073612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97800bab0 con 0x7fb9800c8a20 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.944+0000 7fb97effd700 1 -- 172.21.15.29:0/1612073612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800be70 con 0x7fb9800c8a20 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 -- 172.21.15.29:0/1612073612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 msgr2=0x7fb9800cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1612073612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb9800cc7e0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fb9800cc2f0 tx=0x7fb978009780 comp rx=0 tx=0).stop 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 -- 172.21.15.29:0/1612073612 shutdown_connections 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1612073612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb9800cc7e0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1612073612 >> v2:172.21.15.144:3300/0 conn(0x7fb9800ced00 0x7fb9800b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 -- 172.21.15.29:0/1612073612 >> 172.21.15.29:0/1612073612 conn(0x7fb980063a10 msgr2=0x7fb9800ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 -- 172.21.15.29:0/1612073612 shutdown_connections 2023-10-07T11:01:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.945+0000 7fb9869ff700 1 -- 172.21.15.29:0/1612073612 wait complete. 2023-10-07T11:01:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.946+0000 7fb9869ff700 1 Processor -- start 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.946+0000 7fb9869ff700 1 -- start start 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.947+0000 7fb9869ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9800b0610 0x7fb9800cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.947+0000 7fb9869ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb98014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.947+0000 7fb9869ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9800ced00 0x7fb98015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.947+0000 7fb9869ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98016a0e0 con 0x7fb9800c8a20 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.947+0000 7fb9869ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9801697d0 con 0x7fb9800b0610 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.947+0000 7fb9869ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800cb380 con 0x7fb9800ced00 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb98014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9800b0610 0x7fb9800cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb98014b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37316/0 (socket says 172.21.15.29:37316) 2023-10-07T11:01:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 -- 172.21.15.29:0/1520705610 learned_addr learned my addr 172.21.15.29:0/1520705610 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb984f9c700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9800ced00 0x7fb98015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9800ced00 msgr2=0x7fb98015c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9800ced00 0x7fb98015c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9800b0610 msgr2=0x7fb9800cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9800b0610 0x7fb9800cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 -- 172.21.15.29:0/1520705610 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb974009420 con 0x7fb9800c8a20 2023-10-07T11:01:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9800b0610 0x7fb9800cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:01:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.948+0000 7fb97f7fe700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb98014b3b0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fb97400ec50 tx=0x7fb9740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.949+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97400c850 con 0x7fb9800c8a20 2023-10-07T11:01:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.949+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97400c9b0 con 0x7fb9800c8a20 2023-10-07T11:01:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.949+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb978009a20 con 0x7fb9800c8a20 2023-10-07T11:01:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.949+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97400cd50 con 0x7fb9800c8a20 2023-10-07T11:01:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.949+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb980169d70 con 0x7fb9800c8a20 2023-10-07T11:01:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.950+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98006a4a0 con 0x7fb9800c8a20 2023-10-07T11:01:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.951+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fb974013680 con 0x7fb9800c8a20 2023-10-07T11:01:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.952+0000 7fb967fff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.952+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6153+0+0 (secure 0 0 0) 0x7fb974084ad0 con 0x7fb9800c8a20 2023-10-07T11:01:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.952+0000 7fb97ffff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.952+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:01:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:51.960+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb974051ae0 con 0x7fb9800c8a20 2023-10-07T11:01:52.037 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:52.037+0000 7fe0f9cb71c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:01:52.097 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:52.096+0000 7fb5d877c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:01:52.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:52.153+0000 7fb97ffff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:52.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:52.153+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:01:52.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:52.188+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9800718b0 con 0x7fb96c066f80 2023-10-07T11:01:52.215 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:52.214+0000 7fe0f9cb71c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:01:52.511 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:52.510+0000 7fb5d877c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:01:52.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:52.553+0000 7fb97ffff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:52.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:52.553+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T11:01:52.643 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:52.643+0000 7fe0f9cb71c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:01:53.288 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:53.287+0000 7fb5d877c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:01:53.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:53.354+0000 7fb97ffff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:53.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:53.354+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T11:01:53.376 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:53.375+0000 7fb5d877c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:01:53.447 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:53.446+0000 7fe0f9cb71c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:01:53.529 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:53.528+0000 7fb5d877c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:01:53.538 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:53.538+0000 7fe0f9cb71c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:01:53.617 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:53.617+0000 7fb5d877c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:01:53.700 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:53.699+0000 7fe0f9cb71c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:01:53.791 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:53.790+0000 7fe0f9cb71c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:01:53.834 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:53.834+0000 7fb5d877c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:01:53.931 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:53.930+0000 7fb5d877c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:01:54.010 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.009+0000 7fe0f9cb71c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:01:54.026 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:54.025+0000 7fb5d877c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:01:54.109 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.108+0000 7fe0f9cb71c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:01:54.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:54.121+0000 7fb5d877c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:01:54.208 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.207+0000 7fe0f9cb71c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:01:54.308 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.306+0000 7fe0f9cb71c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:01:54.310 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:54.309+0000 7fb5d877c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:01:54.407 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:54.407+0000 7fb5d877c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:01:54.501 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.501+0000 7fe0f9cb71c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:01:54.603 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.602+0000 7fe0f9cb71c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:01:54.749 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:54.748+0000 7fb5d877c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:01:54.837 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:54.837+0000 7fb5d877c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:54.951 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:54.950+0000 7fe0f9cb71c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:01:54.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:54.956+0000 7fb97ffff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:01:54.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:54.956+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T11:01:55.043 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:55.042+0000 7fe0f9cb71c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:55.377 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:55.376+0000 7fb5d877c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:01:55.561 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:55.561+0000 7fb5d877c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:01:55.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:55.606+0000 7fe0f9cb71c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:01:55.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:55.798+0000 7fe0f9cb71c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:01:55.941 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:55.940+0000 7fb5d877c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:56.182 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:56.182+0000 7fe0f9cb71c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:56.361 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.360+0000 7fb5d877c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:01:56.542 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.541+0000 7fb5d877c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:56.623 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:56.622+0000 7fe0f9cb71c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:01:56.728 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.728+0000 7fb5d877c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:01:56.810 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:56.810+0000 7fe0f9cb71c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:01:56.836 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.835+0000 7fb5d877c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:56.847+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 91) v1 ==== 84468+0+0 (secure 0 0 0) 0x7fb974051eb0 con 0x7fb9800c8a20 2023-10-07T11:01:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:56.847+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:01:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:56.847+0000 7fb967fff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:56.973 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.972+0000 7fb54cd77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.973 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.972+0000 7fb54cd77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.973 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.972+0000 7fb54cd77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.973 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.972+0000 7fb54cd77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.973 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.972+0000 7fb54cd77700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.973+0000 7fb549d71700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.973+0000 7fb549d71700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.973+0000 7fb549d71700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.973+0000 7fb549d71700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:56.974 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:01:56.973+0000 7fb549d71700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:01:57.017 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:57.016+0000 7fe0f9cb71c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:01:57.131 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:01:57.130+0000 7fe0f9cb71c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:01:57.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.853+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 92) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fb974060770 con 0x7fb9800c8a20 2023-10-07T11:01:57.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.854+0000 7fb967fff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.854+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9800718b0 con 0x7fb96c066f80 2023-10-07T11:01:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.854+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:57.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.856+0000 7fb97ffff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb970007fd0 tx=0x7fb970009040 comp rx=0 tx=0).ready entity=mgr.9292 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:57.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.925+0000 7fb967fff700 1 -- 172.21.15.29:0/1520705610 <== mgr.9292 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb9800718b0 con 0x7fb96c066f80 2023-10-07T11:01:57.926 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 msgr2=0x7fb96c069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb970007fd0 tx=0x7fb970009040 comp rx=0 tx=0).stop 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 msgr2=0x7fb98014b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb98014b3b0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fb97400ec50 tx=0x7fb9740091a0 comp rx=0 tx=0).stop 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 shutdown_connections 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb96c066f80 0x7fb96c069430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9800ced00 0x7fb98015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9800b0610 0x7fb9800cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 --2- 172.21.15.29:0/1520705610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800c8a20 0x7fb98014b3b0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 >> 172.21.15.29:0/1520705610 conn(0x7fb980063a10 msgr2=0x7fb9800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 shutdown_connections 2023-10-07T11:01:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:57.928+0000 7fb9869ff700 1 -- 172.21.15.29:0/1520705610 wait complete. 2023-10-07T11:01:57.949 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-07T11:01:57.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-10-07T11:01:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.155+0000 7ff31bfff700 1 Processor -- start 2023-10-07T11:01:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.156+0000 7ff31bfff700 1 -- start start 2023-10-07T11:01:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cb800 0x7ff3140cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff31bfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3140c62d0 con 0x7ff3140d0000 2023-10-07T11:01:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff31bfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff314159e10 con 0x7ff3140cfbb0 2023-10-07T11:01:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff31bfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140d0420 con 0x7ff3140cb800 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff31a59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3140d0000 0x7ff3140b3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35590/0 (socket says 172.21.15.29:35590) 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff31a59c700 1 -- 172.21.15.29:0/1478298326 learned_addr learned my addr 172.21.15.29:0/1478298326 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.157+0000 7ff319d9b700 1 --2- 172.21.15.29:0/1478298326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cb800 0x7ff3140cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390584688 0 0) 0x7ff3140c62d0 con 0x7ff3140d0000 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff300003570 con 0x7ff3140d0000 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425911986 0 0) 0x7ff314159e10 con 0x7ff3140cfbb0 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3140c62d0 con 0x7ff3140cfbb0 2023-10-07T11:01:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff319d9b700 1 -- 172.21.15.29:0/1478298326 >> v1:172.21.15.144:6789/0 conn(0x7ff3140cfbb0 legacy=0x7ff3140c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff319d9b700 1 -- 172.21.15.29:0/1478298326 >> v1:172.21.15.29:6789/0 conn(0x7ff3140d0000 legacy=0x7ff3140b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff319d9b700 1 -- 172.21.15.29:0/1478298326 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140cc110 con 0x7ff3140cb800 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.158+0000 7ff319d9b700 1 --2- 172.21.15.29:0/1478298326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cb800 0x7ff3140cbbd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff310009900 tx=0x7ff310009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad863f35db29a6cb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310005c10 con 0x7ff3140cb800 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff310005d70 con 0x7ff3140cb800 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff318d99700 1 -- 172.21.15.29:0/1478298326 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310006260 con 0x7ff3140cb800 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 -- 172.21.15.29:0/1478298326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cb800 msgr2=0x7ff3140cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 --2- 172.21.15.29:0/1478298326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cb800 0x7ff3140cbbd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff310009900 tx=0x7ff310009120 comp rx=0 tx=0).stop 2023-10-07T11:01:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 -- 172.21.15.29:0/1478298326 shutdown_connections 2023-10-07T11:01:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 --2- 172.21.15.29:0/1478298326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cb800 0x7ff3140cbbd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 -- 172.21.15.29:0/1478298326 >> 172.21.15.29:0/1478298326 conn(0x7ff314063a10 msgr2=0x7ff3140d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 -- 172.21.15.29:0/1478298326 shutdown_connections 2023-10-07T11:01:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.159+0000 7ff31bfff700 1 -- 172.21.15.29:0/1478298326 wait complete. 2023-10-07T11:01:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.161+0000 7ff31bfff700 1 Processor -- start 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.161+0000 7ff31bfff700 1 -- start start 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3140cb800 0x7ff314152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cfbb0 0x7ff3140cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 0x7ff3140b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3141533d0 con 0x7ff3140d0000 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31bfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31415be20 con 0x7ff3140cb800 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31bfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31415c100 con 0x7ff3140cfbb0 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 0x7ff3140b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 0x7ff3140b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36454/0 (socket says 172.21.15.29:36454) 2023-10-07T11:01:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 -- 172.21.15.29:0/728284198 learned_addr learned my addr 172.21.15.29:0/728284198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff319d9b700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3140cb800 0x7ff314152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31959a700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cfbb0 0x7ff3140cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 -- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cfbb0 msgr2=0x7ff3140cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cfbb0 0x7ff3140cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 -- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3140cb800 msgr2=0x7ff314152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3140cb800 0x7ff314152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.162+0000 7ff31a59c700 1 -- 172.21.15.29:0/728284198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100093c0 con 0x7ff3140d0000 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.163+0000 7ff31a59c700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 0x7ff3140b3320 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00cc60 tx=0x7ff30c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.163+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c00a8b0 con 0x7ff3140d0000 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.163+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c00aa30 con 0x7ff3140d0000 2023-10-07T11:01:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.163+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31415c3e0 con 0x7ff3140d0000 2023-10-07T11:01:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.163+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c00adf0 con 0x7ff3140d0000 2023-10-07T11:01:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.163+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff314169ab0 con 0x7ff3140d0000 2023-10-07T11:01:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.164+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3140718b0 con 0x7ff3140d0000 2023-10-07T11:01:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.165+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 84595+0+0 (secure 0 0 0) 0x7ff30c007300 con 0x7ff3140d0000 2023-10-07T11:01:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.166+0000 7ff309ffb700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff2f8066dd0 0x7ff2f8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.166+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6411+0+0 (secure 0 0 0) 0x7ff30c0848b0 con 0x7ff3140d0000 2023-10-07T11:01:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.166+0000 7ff319d9b700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff2f8066dd0 0x7ff2f8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.167+0000 7ff319d9b700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff2f8066dd0 0x7ff2f8069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff310009390 tx=0x7ff310016000 comp rx=0 tx=0).ready entity=mgr.9292 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.170+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff30c051ac0 con 0x7ff3140d0000 2023-10-07T11:01:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.403+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff314064040 con 0x7ff2f8066dd0 2023-10-07T11:01:58.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.461+0000 7ff309ffb700 1 -- 172.21.15.29:0/728284198 <== mgr.9292 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7ff314064040 con 0x7ff2f8066dd0 2023-10-07T11:01:58.463 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path snap_test_dir1 2023-10-07T11:01:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff2f8066dd0 msgr2=0x7ff2f8069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff2f8066dd0 0x7ff2f8069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff310009390 tx=0x7ff310016000 comp rx=0 tx=0).stop 2023-10-07T11:01:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 msgr2=0x7ff3140b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 0x7ff3140b3320 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00cc60 tx=0x7ff30c001ae0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 shutdown_connections 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3140d0000 0x7ff3140b3320 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3140cb800 0x7ff314152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff2f8066dd0 0x7ff2f8069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 --2- 172.21.15.29:0/728284198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3140cfbb0 0x7ff3140cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 >> 172.21.15.29:0/728284198 conn(0x7ff314063a10 msgr2=0x7ff314157840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 shutdown_connections 2023-10-07T11:01:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.464+0000 7ff31bfff700 1 -- 172.21.15.29:0/728284198 wait complete. 2023-10-07T11:01:58.482 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-07T11:01:58.483 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-07-11_02_00 in ~2s... 2023-10-07T11:01:58.483 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-10-07T11:01:58.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-10-07T11:01:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.682+0000 7fb3485ff700 1 Processor -- start 2023-10-07T11:01:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.682+0000 7fb3485ff700 1 -- start start 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb3485ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3400cc810 0x7fb3400accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb3485ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb34015a590 0x7fb34015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb3485ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3400ad400 con 0x7fb3400afeb0 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb3485ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34015a140 con 0x7fb3400cc810 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb3485ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3400adca0 con 0x7fb34015a590 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3400cc810 0x7fb3400accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3400cc810 0x7fb3400accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36456/0 (socket says 172.21.15.29:36456) 2023-10-07T11:01:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 -- 172.21.15.29:0/4016383465 learned_addr learned my addr 172.21.15.29:0/4016383465 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb346b9c700 1 --2- 172.21.15.29:0/4016383465 >> v2:172.21.15.144:3300/0 conn(0x7fb34015a590 0x7fb34015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 -- 172.21.15.29:0/4016383465 >> v2:172.21.15.144:3300/0 conn(0x7fb34015a590 msgr2=0x7fb34015a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:01:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 --2- 172.21.15.29:0/4016383465 >> v2:172.21.15.144:3300/0 conn(0x7fb34015a590 0x7fb34015a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 -- 172.21.15.29:0/4016383465 >> v1:172.21.15.144:6789/0 conn(0x7fb3400afeb0 legacy=0x7fb3400aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.684+0000 7fb345b9a700 1 -- 172.21.15.29:0/4016383465 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400ad6e0 con 0x7fb3400cc810 2023-10-07T11:01:58.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.685+0000 7fb345b9a700 1 --2- 172.21.15.29:0/4016383465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400accf0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fb3400aed10 tx=0x7fb330009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0981d1b830cac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:58.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.685+0000 7fb345399700 1 -- 172.21.15.29:0/4016383465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330009e70 con 0x7fb3400cc810 2023-10-07T11:01:58.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.685+0000 7fb345399700 1 -- 172.21.15.29:0/4016383465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33000b860 con 0x7fb3400cc810 2023-10-07T11:01:58.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.685+0000 7fb345399700 1 -- 172.21.15.29:0/4016383465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33000bc00 con 0x7fb3400cc810 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.685+0000 7fb3485ff700 1 -- 172.21.15.29:0/4016383465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 msgr2=0x7fb3400accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.685+0000 7fb3485ff700 1 --2- 172.21.15.29:0/4016383465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400accf0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fb3400aed10 tx=0x7fb330009250 comp rx=0 tx=0).stop 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.686+0000 7fb3485ff700 1 -- 172.21.15.29:0/4016383465 shutdown_connections 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.686+0000 7fb3485ff700 1 --2- 172.21.15.29:0/4016383465 >> v2:172.21.15.144:3300/0 conn(0x7fb34015a590 0x7fb34015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.686+0000 7fb3485ff700 1 --2- 172.21.15.29:0/4016383465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400accf0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.686+0000 7fb3485ff700 1 -- 172.21.15.29:0/4016383465 >> 172.21.15.29:0/4016383465 conn(0x7fb340067610 msgr2=0x7fb340067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.686+0000 7fb3485ff700 1 -- 172.21.15.29:0/4016383465 shutdown_connections 2023-10-07T11:01:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.686+0000 7fb3485ff700 1 -- 172.21.15.29:0/4016383465 wait complete. 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.687+0000 7fb3485ff700 1 Processor -- start 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.687+0000 7fb3485ff700 1 -- start start 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb3485ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3400afeb0 0x7fb3400ab030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb3485ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb3485ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34015a590 0x7fb34016d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb3485ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3400c8780 con 0x7fb3400cc810 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb3485ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34016db90 con 0x7fb34015a590 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb3485ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3400c53b0 con 0x7fb3400afeb0 2023-10-07T11:01:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb345b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.688+0000 7fb345b9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36462/0 (socket says 172.21.15.29:36462) 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 -- 172.21.15.29:0/376939722 learned_addr learned my addr 172.21.15.29:0/376939722 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb346b9c700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34015a590 0x7fb34016d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb34639b700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3400afeb0 0x7fb3400ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 -- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3400afeb0 msgr2=0x7fb3400ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3400afeb0 0x7fb3400ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 -- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34015a590 msgr2=0x7fb34016d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34015a590 0x7fb34016d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 -- 172.21.15.29:0/376939722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3300094f0 con 0x7fb3400cc810 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb346b9c700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34015a590 0x7fb34016d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:01:58.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.689+0000 7fb345b9a700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400c8240 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb330006fd0 tx=0x7fb330017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:58.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.690+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33000b9d0 con 0x7fb3400cc810 2023-10-07T11:01:58.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.690+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33000bb30 con 0x7fb3400cc810 2023-10-07T11:01:58.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.690+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3400c5690 con 0x7fb3400cc810 2023-10-07T11:01:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.690+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330025670 con 0x7fb3400cc810 2023-10-07T11:01:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.690+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3400c5f20 con 0x7fb3400cc810 2023-10-07T11:01:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.691+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3400718b0 con 0x7fb3400cc810 2023-10-07T11:01:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.692+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fb330023070 con 0x7fb3400cc810 2023-10-07T11:01:58.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.693+0000 7fb3367fc700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb324066dd0 0x7fb324069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:01:58.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.693+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fb330085a60 con 0x7fb3400cc810 2023-10-07T11:01:58.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.694+0000 7fb34639b700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb324066dd0 0x7fb324069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:01:58.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.694+0000 7fb34639b700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb324066dd0 0x7fb324069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb33c003a80 tx=0x7fb33c007ec0 comp rx=0 tx=0).ready entity=mgr.9292 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:01:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.698+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb330052bf0 con 0x7fb3400cc810 2023-10-07T11:01:58.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.925+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3400c9130 con 0x7fb324066dd0 2023-10-07T11:01:58.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.934+0000 7fb3367fc700 1 -- 172.21.15.29:0/376939722 <== mgr.9292 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fb3400c9130 con 0x7fb324066dd0 2023-10-07T11:01:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:01:58.935 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-07T11:01:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb324066dd0 msgr2=0x7fb324069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb324066dd0 0x7fb324069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb33c003a80 tx=0x7fb33c007ec0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 msgr2=0x7fb3400c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400c8240 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb330006fd0 tx=0x7fb330017000 comp rx=0 tx=0).stop 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 shutdown_connections 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34015a590 0x7fb34016d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fb324066dd0 0x7fb324069280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3400afeb0 0x7fb3400ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 --2- 172.21.15.29:0/376939722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3400cc810 0x7fb3400c8240 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:01:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 >> 172.21.15.29:0/376939722 conn(0x7fb340067610 msgr2=0x7fb340155840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:01:58.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 shutdown_connections 2023-10-07T11:01:58.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:01:58.936+0000 7fb3485ff700 1 -- 172.21.15.29:0/376939722 wait complete. 2023-10-07T11:01:58.955 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-07T11:01:58.955 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-10-07T11:01:58.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:01:58.955 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:01:59.004 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T11:01:59.004 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:01:59.004 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:00.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:00.005 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:00.059 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-07T11:02:00.059 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:00.059 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:00.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:02:00.473+0000 7fb53bd55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:02:00.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:02:00.473+0000 7fb53bd55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:02:00.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:02:00.473+0000 7fb53bd55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:02:00.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:02:00.473+0000 7fb53bd55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:02:00.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:02:00.473+0000 7fb53bd55700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:02:01.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:01.061 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:01.115 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:01.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:01.116 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:01.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:01.117 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-07-11_02_00_UTC'] 2023-10-07T11:02:01.117 DEBUG:tasks.cephfs.test_snap_schedules:exec=1696676518.4830122, now = 1696676521.1172657, timo = 2 2023-10-07T11:02:02.117 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-07-11_02_00 in ~62s... 2023-10-07T11:02:02.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:02.118 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:02.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:02.169 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:02.169 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:02.169 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:03.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:03.171 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:03.224 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:03.225 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:03.226 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:03.226 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:04.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:04.227 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:04.280 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:04.281 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:04.281 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:04.281 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:05.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:05.282 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:05.336 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:05.337 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:05.337 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:05.337 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:06.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:06.338 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:06.391 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:06.393 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:06.393 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:06.393 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:07.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:07.394 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:07.447 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:07.448 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:07.448 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:07.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:08.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:08.450 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:08.503 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:08.504 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:08.504 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:08.504 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:09.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:09.505 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:09.558 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:09.559 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:09.559 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:09.560 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:10.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:10.561 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:10.614 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:10.616 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:10.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:10.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:11.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:11.617 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:11.670 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:11.671 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:11.671 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:11.671 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:12.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:12.672 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:12.725 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:12.726 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:12.726 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:12.727 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:13.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:13.728 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:13.781 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:13.782 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:13.782 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:13.782 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:13.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:02:13.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:02:14.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:14.784 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:14.837 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:14.840 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:14.840 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:14.840 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:15.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:15.842 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:15.896 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:15.897 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:15.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:15.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:16.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:16.898 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:16.951 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:16.952 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:16.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:16.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:17.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:17.954 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:18.007 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:18.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:18.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:18.009 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:19.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:19.011 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:19.064 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:19.066 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:19.066 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:19.066 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:20.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:20.067 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:20.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:20.121 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:20.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:20.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:21.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:21.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:21.175 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:21.177 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:21.177 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:21.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:22.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:22.178 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:22.232 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:22.233 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:22.233 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:22.233 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:23.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:23.235 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:23.288 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:23.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:23.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:23.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:24.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:24.290 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:24.343 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:24.344 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:24.344 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:24.345 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:25.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:25.345 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:25.399 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:25.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:25.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:25.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:26.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:26.401 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:26.455 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:26.455 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:26.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:27.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:27.457 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:27.510 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:27.511 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:27.511 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:27.511 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:28.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:28.512 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:28.568 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:28.568 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:28.568 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:28.568 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:29.569 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:29.570 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:29.623 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:29.624 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:29.624 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:29.624 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:30.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:30.626 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:30.679 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:30.680 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:30.680 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:30.680 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:31.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:31.681 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:31.734 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:31.735 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:31.735 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:31.735 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:32.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:32.737 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:32.790 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:32.792 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:32.792 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:32.792 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:33.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:33.794 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:33.847 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:33.848 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:33.848 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:33.848 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:34.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:34.849 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:34.902 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:34.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:34.904 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:34.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:35.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:35.905 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:35.958 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:35.959 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:35.959 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:35.959 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:36.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:36.960 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:37.013 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:37.014 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:37.014 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:37.014 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:38.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:38.015 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:38.068 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:38.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:38.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:38.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:39.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:39.071 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:39.124 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:39.125 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:39.125 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:39.125 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:40.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:40.127 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:40.181 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:40.181 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:40.182 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:41.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:41.183 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:41.235 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:41.236 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:41.236 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:41.237 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:42.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:42.237 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:42.290 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:42.291 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:42.291 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:42.291 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:43.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:43.292 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:43.345 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:43.346 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:43.346 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:43.346 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:44.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:02:44.040 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:02:44.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:44.347 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:44.400 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:44.401 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:44.401 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:44.401 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:45.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:45.402 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:45.454 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:45.455 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:45.456 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:45.456 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:46.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:46.457 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:46.510 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:46.511 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:46.511 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:46.511 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:47.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:47.513 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:47.566 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:47.567 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:47.567 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:47.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:48.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:48.567 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:48.620 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:48.621 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:48.622 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:48.622 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:49.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:49.623 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:49.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:49.678 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:49.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:49.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:50.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:50.679 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:50.732 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:50.734 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:50.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:50.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:51.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:51.735 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:51.787 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:51.788 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:51.789 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:51.789 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:52.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:52.790 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:52.843 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:52.844 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:52.844 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:52.844 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:53.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:53.845 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:53.899 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:53.899 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:53.899 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:54.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:54.901 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:54.953 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:54.954 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:54.955 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:54.955 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:55.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:55.956 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:56.009 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:56.010 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:56.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:56.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:57.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:57.011 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:57.064 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:57.066 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:57.066 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:57.066 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:58.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:58.067 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:58.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:58.120 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:58.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:58.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:02:59.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:02:59.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:02:59.175 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:02:59.177 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:02:59.177 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:02:59.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:03:00.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:00.178 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:03:00.231 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_02_00_UTC 2023-10-07T11:03:00.232 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:03:00.232 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-07T11:03:00.232 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-07T11:03:01.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:01.234 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:03:01.287 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_03_00_UTC 2023-10-07T11:03:01.288 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-07-11_03_00_UTC'} 2023-10-07T11:03:01.288 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-07-11_03_00_UTC'} 2023-10-07T11:03:01.288 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-07-11_02_00_UTC'} 2023-10-07T11:03:01.288 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-07-11_02_00_UTC'] 2023-10-07T11:03:01.288 DEBUG:tasks.cephfs.test_snap_schedules:exec=1696676518.4830122, now = 1696676581.2888055, timo = 62 2023-10-07T11:03:02.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-10-07T11:03:02.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.537+0000 7fa2ee99d700 1 Processor -- start 2023-10-07T11:03:02.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.537+0000 7fa2ee99d700 1 -- start start 2023-10-07T11:03:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.538+0000 7fa2ee99d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2e8069730 0x7fa2e81645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.538+0000 7fa2ee99d700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e815de60 con 0x7fa2e80cc730 2023-10-07T11:03:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.538+0000 7fa2ee99d700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e815d250 con 0x7fa2e80cb0a0 2023-10-07T11:03:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.538+0000 7fa2ee99d700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e815d530 con 0x7fa2e8069730 2023-10-07T11:03:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ed19a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa2e80cc730 0x7fa2e8142a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39312/0 (socket says 172.21.15.29:39312) 2023-10-07T11:03:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ed19a700 1 -- 172.21.15.29:0/3996285848 learned_addr learned my addr 172.21.15.29:0/3996285848 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ee19c700 1 --2- 172.21.15.29:0/3996285848 >> v2:172.21.15.144:3300/0 conn(0x7fa2e8069730 0x7fa2e81645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135957051 0 0) 0x7fa2e815de60 con 0x7fa2e80cc730 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d4003570 con 0x7fa2e80cc730 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733235806 0 0) 0x7fa2e815d250 con 0x7fa2e80cb0a0 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e815de60 con 0x7fa2e80cb0a0 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2806796402 0 0) 0x7fa2d4003570 con 0x7fa2e80cc730 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 >> v2:172.21.15.144:3300/0 conn(0x7fa2e8069730 msgr2=0x7fa2e81645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 --2- 172.21.15.29:0/3996285848 >> v2:172.21.15.144:3300/0 conn(0x7fa2e8069730 0x7fa2e81645f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 >> v1:172.21.15.144:6789/0 conn(0x7fa2e80cb0a0 legacy=0x7fa2e80cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e815d810 con 0x7fa2e80cc730 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7fa2d8001470 con 0x7fa2e80cc730 2023-10-07T11:03:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.539+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 >> v1:172.21.15.29:6789/0 conn(0x7fa2e80cc730 legacy=0x7fa2e8142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ec999700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2d4003bf0 0x7fa2d4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ec999700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d4004810 0x7fa2d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ec999700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d40051c0 0x7fa2d400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e815d530 con 0x7fa2d4003bf0 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c0b0 con 0x7fa2d40051c0 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c270 con 0x7fa2d4004810 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2d4003bf0 0x7fa2d4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed19a700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d4004810 0x7fa2d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ee19c700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d40051c0 0x7fa2d400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed99b700 1 -- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d4004810 msgr2=0x7fa2d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d4004810 0x7fa2d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed99b700 1 -- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d40051c0 msgr2=0x7fa2d400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d40051c0 0x7fa2d400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ed99b700 1 -- 172.21.15.29:0/3996285848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d4003570 con 0x7fa2d4003bf0 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.540+0000 7fa2ee19c700 1 -- 172.21.15.29:0/3996285848 reap_dead start 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2d4003bf0 0x7fa2d4003fe0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4001a10 tx=0x7fa2e4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61f0f2929bfb890 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e400c590 con 0x7fa2d4003bf0 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e400c6f0 con 0x7fa2d4003bf0 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ec999700 1 -- 172.21.15.29:0/3996285848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e400ca90 con 0x7fa2d4003bf0 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3996285848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2d4003bf0 msgr2=0x7fa2d4003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2d4003bf0 0x7fa2d4003fe0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4001a10 tx=0x7fa2e4007d30 comp rx=0 tx=0).stop 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3996285848 shutdown_connections 2023-10-07T11:03:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3996285848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2d4003bf0 0x7fa2d4003fe0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3996285848 >> 172.21.15.29:0/3996285848 conn(0x7fa2e8063d00 msgr2=0x7fa2e806b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.541+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3996285848 shutdown_connections 2023-10-07T11:03:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.542+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3996285848 wait complete. 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.543+0000 7fa2ee99d700 1 Processor -- start 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.543+0000 7fa2ee99d700 1 -- start start 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee99d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d4003bf0 0x7fa2e816a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee99d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e80c7cb0 0x7fa2e8141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee99d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 0x7fa2e8158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee99d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e816d960 con 0x7fa2e8158ae0 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee99d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e816aa10 con 0x7fa2d4003bf0 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee99d700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8159450 con 0x7fa2e80c7cb0 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 0x7fa2e8158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee19c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 0x7fa2e8158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58618/0 (socket says 172.21.15.29:58618) 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ee19c700 1 -- 172.21.15.29:0/3769051034 learned_addr learned my addr 172.21.15.29:0/3769051034 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d4003bf0 0x7fa2e816a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.544+0000 7fa2ed19a700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e80c7cb0 0x7fa2e8141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee19c700 1 -- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e80c7cb0 msgr2=0x7fa2e8141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee19c700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e80c7cb0 0x7fa2e8141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee19c700 1 -- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d4003bf0 msgr2=0x7fa2e816a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee19c700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d4003bf0 0x7fa2e816a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee19c700 1 -- 172.21.15.29:0/3769051034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80c8250 con 0x7fa2e8158ae0 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee19c700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 0x7fa2e8158f10 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8159c90 tx=0x7fa2e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000e040 con 0x7fa2e8158ae0 2023-10-07T11:03:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e000f040 con 0x7fa2e8158ae0 2023-10-07T11:03:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000fdc0 con 0x7fa2e8158ae0 2023-10-07T11:03:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e816d030 con 0x7fa2e8158ae0 2023-10-07T11:03:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.545+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e816d5f0 con 0x7fa2e8158ae0 2023-10-07T11:03:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.546+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e816db10 con 0x7fa2e8158ae0 2023-10-07T11:03:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.547+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 84973+0+0 (secure 0 0 0) 0x7fa2e0015640 con 0x7fa2e8158ae0 2023-10-07T11:03:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.549+0000 7fa2ddffb700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa2cc0671c0 0x7fa2cc069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.549+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fa2e0085710 con 0x7fa2e8158ae0 2023-10-07T11:03:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.549+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa2cc0671c0 0x7fa2cc069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.550+0000 7fa2ed99b700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa2cc0671c0 0x7fa2cc069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4000c00 tx=0x7fa2e4015000 comp rx=0 tx=0).ready entity=mgr.9292 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.553+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2e816db10 con 0x7fa2e8158ae0 2023-10-07T11:03:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.782+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 --> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2e804c3c0 con 0x7fa2cc0671c0 2023-10-07T11:03:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.854+0000 7fa2ddffb700 1 -- 172.21.15.29:0/3769051034 <== mgr.9292 v2:172.21.15.144:6834/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa2e804c3c0 con 0x7fa2cc0671c0 2023-10-07T11:03:02.855 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-07T11:03:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.856+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa2cc0671c0 msgr2=0x7fa2cc069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.856+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa2cc0671c0 0x7fa2cc069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4000c00 tx=0x7fa2e4015000 comp rx=0 tx=0).stop 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.856+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 msgr2=0x7fa2e8158f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.856+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 0x7fa2e8158f10 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8159c90 tx=0x7fa2e000b040 comp rx=0 tx=0).stop 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 shutdown_connections 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2e8158ae0 0x7fa2e8158f10 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d4003bf0 0x7fa2e816a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa2cc0671c0 0x7fa2cc069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 --2- 172.21.15.29:0/3769051034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2e80c7cb0 0x7fa2e8141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 >> 172.21.15.29:0/3769051034 conn(0x7fa2e8063d00 msgr2=0x7fa2e80c3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 shutdown_connections 2023-10-07T11:03:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:02.857+0000 7fa2ee99d700 1 -- 172.21.15.29:0/3769051034 wait complete. 2023-10-07T11:03:02.875 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-07T11:03:02.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:02.875 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:03:02.923 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-07-11_03_00_UTC 2023-10-07T11:03:02.924 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-07-11_03_00_UTC 2023-10-07T11:03:02.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:02.924 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-07-11_03_00_UTC) 2023-10-07T11:03:02.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:02.976 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T11:03:03.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T11:03:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.223+0000 7fdab61ff700 1 Processor -- start 2023-10-07T11:03:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.223+0000 7fdab61ff700 1 -- start start 2023-10-07T11:03:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.224+0000 7fdab61ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab00afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.224+0000 7fdab61ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdab00ae570 0x7fdab00accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.224+0000 7fdab61ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdab00b0520 0x7fdab00b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.224+0000 7fdab61ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab00b0ea0 con 0x7fdab00ae570 2023-10-07T11:03:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.224+0000 7fdab61ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab00ad230 con 0x7fdab00b0520 2023-10-07T11:03:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.224+0000 7fdab61ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00ad4d0 con 0x7fdab0156180 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdab00ae570 0x7fdab00accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdab00ae570 0x7fdab00accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58926/0 (socket says 172.21.15.29:58926) 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 -- 172.21.15.29:0/137699454 learned_addr learned my addr 172.21.15.29:0/137699454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaf7fe700 1 --2- 172.21.15.29:0/137699454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab00afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaffff700 1 --2- 172.21.15.29:0/137699454 >> v2:172.21.15.144:3300/0 conn(0x7fdab00b0520 0x7fdab00b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 -- 172.21.15.29:0/137699454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 msgr2=0x7fdab00afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 --2- 172.21.15.29:0/137699454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab00afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 -- 172.21.15.29:0/137699454 >> v2:172.21.15.144:3300/0 conn(0x7fdab00b0520 msgr2=0x7fdab00b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 --2- 172.21.15.29:0/137699454 >> v2:172.21.15.144:3300/0 conn(0x7fdab00b0520 0x7fdab00b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.225+0000 7fdaaeffd700 1 -- 172.21.15.29:0/137699454 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab00ad7b0 con 0x7fdab00ae570 2023-10-07T11:03:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdaaeffd700 1 --2- 172.21.15.29:0/137699454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00ae570 0x7fdab00accf0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fdab006cd20 tx=0x7fdaa4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9bb4e07092e51f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdaae7fc700 1 -- 172.21.15.29:0/137699454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400b640 con 0x7fdab00ae570 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdaae7fc700 1 -- 172.21.15.29:0/137699454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa400b7a0 con 0x7fdab00ae570 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdaae7fc700 1 -- 172.21.15.29:0/137699454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400bb60 con 0x7fdab00ae570 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 -- 172.21.15.29:0/137699454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00ae570 msgr2=0x7fdab00accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 --2- 172.21.15.29:0/137699454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00ae570 0x7fdab00accf0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fdab006cd20 tx=0x7fdaa4009250 comp rx=0 tx=0).stop 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 -- 172.21.15.29:0/137699454 shutdown_connections 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 --2- 172.21.15.29:0/137699454 >> v2:172.21.15.144:3300/0 conn(0x7fdab00b0520 0x7fdab00b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 --2- 172.21.15.29:0/137699454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab00afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 --2- 172.21.15.29:0/137699454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00ae570 0x7fdab00accf0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 -- 172.21.15.29:0/137699454 >> 172.21.15.29:0/137699454 conn(0x7fdab0067610 msgr2=0x7fdab0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.226+0000 7fdab61ff700 1 -- 172.21.15.29:0/137699454 shutdown_connections 2023-10-07T11:03:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.227+0000 7fdab61ff700 1 -- 172.21.15.29:0/137699454 wait complete. 2023-10-07T11:03:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.229+0000 7fdab61ff700 1 Processor -- start 2023-10-07T11:03:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.229+0000 7fdab61ff700 1 -- start start 2023-10-07T11:03:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdab61ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdab00ae570 0x7fdab00caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdab61ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 0x7fdab00a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdab61ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab0152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdab61ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab01678f0 con 0x7fdab00b0520 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdab61ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab0166ce0 con 0x7fdab00ae570 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdab61ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab0166fc0 con 0x7fdab0156180 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdaaeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 0x7fdab00a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdaaeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 0x7fdab00a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58940/0 (socket says 172.21.15.29:58940) 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.230+0000 7fdaaeffd700 1 -- 172.21.15.29:0/1609961923 learned_addr learned my addr 172.21.15.29:0/1609961923 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaffff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab0152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaf7fe700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdab00ae570 0x7fdab00caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaeffd700 1 -- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 msgr2=0x7fdab0152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaeffd700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab0152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaeffd700 1 -- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdab00ae570 msgr2=0x7fdab00caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaeffd700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdab00ae570 0x7fdab00caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaeffd700 1 -- 172.21.15.29:0/1609961923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa40094f0 con 0x7fdab00b0520 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdaaeffd700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 0x7fdab00a56c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fdab00641e0 tx=0x7fdaa4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4020070 con 0x7fdab00b0520 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa400bb20 con 0x7fdab00b0520 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab01672a0 con 0x7fdab00b0520 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.231+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4011650 con 0x7fdab00b0520 2023-10-07T11:03:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.232+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab01668b0 con 0x7fdab00b0520 2023-10-07T11:03:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.232+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab0167740 con 0x7fdab00b0520 2023-10-07T11:03:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.233+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 84973+0+0 (secure 0 0 0) 0x7fdaa40119c0 con 0x7fdab00b0520 2023-10-07T11:03:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.234+0000 7fda977fe700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fda9c0671c0 0x7fda9c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.234+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fdaa4087e40 con 0x7fdab00b0520 2023-10-07T11:03:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.235+0000 7fdaaf7fe700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fda9c0671c0 0x7fda9c069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:03.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.235+0000 7fdaaf7fe700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fda9c0671c0 0x7fda9c069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdab00cc490 tx=0x7fdaa0008040 comp rx=0 tx=0).ready entity=mgr.9292 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:03.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.238+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdaa4054ac0 con 0x7fdab00b0520 2023-10-07T11:03:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:03.557+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fdab0166aa0 con 0x7fdab00b0520 2023-10-07T11:03:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.441+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fdaa4054e90 con 0x7fdab00b0520 2023-10-07T11:03:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.443+0000 7fda977fe700 1 -- 172.21.15.29:0/1609961923 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) v1 ==== 93+0+0 (secure 0 0 0) 0x7fdaa4058550 con 0x7fdab00b0520 2023-10-07T11:03:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fda9c0671c0 msgr2=0x7fda9c069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fda9c0671c0 0x7fda9c069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdab00cc490 tx=0x7fdaa0008040 comp rx=0 tx=0).stop 2023-10-07T11:03:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 msgr2=0x7fdab00a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 0x7fdab00a56c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fdab00641e0 tx=0x7fdaa4017000 comp rx=0 tx=0).stop 2023-10-07T11:03:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 shutdown_connections 2023-10-07T11:03:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fda9c0671c0 0x7fda9c069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdab0156180 0x7fdab0152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdab00ae570 0x7fdab00caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 --2- 172.21.15.29:0/1609961923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdab00b0520 0x7fdab00a56c0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 >> 172.21.15.29:0/1609961923 conn(0x7fdab0067610 msgr2=0x7fdab00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 shutdown_connections 2023-10-07T11:03:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:04.446+0000 7fdab61ff700 1 -- 172.21.15.29:0/1609961923 wait complete. 2023-10-07T11:03:04.468 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T11:03:04.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:03:04.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:03:04.660 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:04.659+0000 7fa47cba01c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:03:04.664 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:04.663+0000 7f5c0248b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:03:04.674 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T11:03:04.674 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T11:03:04.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:04.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T11:03:04.770 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:03:04.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:04.771 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T11:03:04.844 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:04.843+0000 7f5c0248b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:03:04.851 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:04.851+0000 7fa47cba01c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:03:04.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.055+0000 7f68397ff700 1 Processor -- start 2023-10-07T11:03:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.055+0000 7f68397ff700 1 -- start start 2023-10-07T11:03:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.056+0000 7f68397ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f68341558c0 0x7f6834068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.056+0000 7f68397ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68340abb90 0x7f6834068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.056+0000 7f68397ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340aca80 con 0x7f6834155c90 2023-10-07T11:03:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.056+0000 7f68397ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68340b2960 con 0x7f68341558c0 2023-10-07T11:03:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.056+0000 7f68397ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68340b1440 con 0x7f68340abb90 2023-10-07T11:03:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f68341558c0 0x7f6834068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f68327fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68340abb90 0x7f6834068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f68341558c0 0x7f6834068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36362/0 (socket says 172.21.15.29:36362) 2023-10-07T11:03:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 -- 172.21.15.29:0/560793218 learned_addr learned my addr 172.21.15.29:0/560793218 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 -- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68340abb90 msgr2=0x7f6834068b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 --2- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68340abb90 0x7f6834068b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 -- 172.21.15.29:0/560793218 >> v1:172.21.15.144:6789/0 conn(0x7f6834155c90 legacy=0x7f68340adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f6832ffd700 1 -- 172.21.15.29:0/560793218 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68340b2680 con 0x7f68341558c0 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.057+0000 7f68327fc700 1 --2- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68340abb90 0x7f6834068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.058+0000 7f6832ffd700 1 --2- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f6834068620 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f68240099b0 tx=0x7f6824009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=580cab83ef6f8112 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.058+0000 7f6831ffb700 1 -- 172.21.15.29:0/560793218 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824009ea0 con 0x7f68341558c0 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.058+0000 7f6831ffb700 1 -- 172.21.15.29:0/560793218 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682400b860 con 0x7f68341558c0 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.058+0000 7f6831ffb700 1 -- 172.21.15.29:0/560793218 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682400bc00 con 0x7f68341558c0 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.058+0000 7f68397ff700 1 -- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 msgr2=0x7f6834068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.058+0000 7f68397ff700 1 --2- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f6834068620 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f68240099b0 tx=0x7f6824009250 comp rx=0 tx=0).stop 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.059+0000 7f68397ff700 1 -- 172.21.15.29:0/560793218 shutdown_connections 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.059+0000 7f68397ff700 1 --2- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f6834068620 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.059+0000 7f68397ff700 1 --2- 172.21.15.29:0/560793218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68340abb90 0x7f6834068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.059+0000 7f68397ff700 1 -- 172.21.15.29:0/560793218 >> 172.21.15.29:0/560793218 conn(0x7f683406dd00 msgr2=0x7f683406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.059+0000 7f68397ff700 1 -- 172.21.15.29:0/560793218 shutdown_connections 2023-10-07T11:03:05.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.059+0000 7f68397ff700 1 -- 172.21.15.29:0/560793218 wait complete. 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.060+0000 7f68397ff700 1 Processor -- start 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.060+0000 7f68397ff700 1 -- start start 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68397ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68340abb90 0x7f6834169f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68397ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f683416ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68397ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6834155c90 0x7f6834156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68397ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683414fe40 con 0x7f68340abb90 2023-10-07T11:03:05.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68397ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834157010 con 0x7f68341558c0 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68397ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683414e5d0 con 0x7f6834155c90 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f68327fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f683416ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.061+0000 7f6832ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68340abb90 0x7f6834169f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68327fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f683416ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36364/0 (socket says 172.21.15.29:36364) 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68327fc700 1 -- 172.21.15.29:0/797213820 learned_addr learned my addr 172.21.15.29:0/797213820 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6834155c90 0x7f6834156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 -- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 msgr2=0x7f683416ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f683416ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 -- 172.21.15.29:0/797213820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68340abb90 msgr2=0x7f6834169f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68340abb90 0x7f6834169f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 -- 172.21.15.29:0/797213820 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6828009420 con 0x7f6834155c90 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68327fc700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f683416ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.062+0000 7f68337fe700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6834155c90 0x7f6834156ad0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f683406cf60 tx=0x7f682c0015f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:05.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.063+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c011590 con 0x7f6834155c90 2023-10-07T11:03:05.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.063+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c011710 con 0x7f6834155c90 2023-10-07T11:03:05.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.063+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c011ad0 con 0x7f6834155c90 2023-10-07T11:03:05.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.063+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68240094f0 con 0x7f6834155c90 2023-10-07T11:03:05.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.063+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683414f510 con 0x7f6834155c90 2023-10-07T11:03:05.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.064+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683414ffb0 con 0x7f6834155c90 2023-10-07T11:03:05.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.065+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f682c012420 con 0x7f6834155c90 2023-10-07T11:03:05.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.066+0000 7f681affd700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.066+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f682c084860 con 0x7f6834155c90 2023-10-07T11:03:05.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.066+0000 7f6832ffd700 1 -- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 msgr2=0x7f6820069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:05.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.066+0000 7f6832ffd700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:05.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.070+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f682c0514e0 con 0x7f6834155c90 2023-10-07T11:03:05.161 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:05.160+0000 7f5c0248b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:03:05.185 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:05.184+0000 7fa47cba01c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:03:05.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.267+0000 7f6832ffd700 1 -- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 msgr2=0x7f6820069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.267+0000 7f6832ffd700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.316+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f683404c3c0 con 0x7f6834155c90 2023-10-07T11:03:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.317+0000 7f681affd700 1 -- 172.21.15.29:0/797213820 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f682c054b00 con 0x7f6834155c90 2023-10-07T11:03:05.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:05.318 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:01:40.378766+0000","modified":"2023-10-07T11:01:44.282264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/562306367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":562306367},{"type":"v1","addr":"172.21.15.144:6837","nonce":562306367}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-10-07T11:03:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 msgr2=0x7f6820069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6834155c90 msgr2=0x7f6834156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6834155c90 0x7f6834156ad0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f683406cf60 tx=0x7f682c0015f0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 shutdown_connections 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6834155c90 0x7f6834156ad0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68340abb90 0x7f6834169f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 --2- 172.21.15.29:0/797213820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68341558c0 0x7f683416ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 >> 172.21.15.29:0/797213820 conn(0x7f683406dd00 msgr2=0x7f68340681d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 shutdown_connections 2023-10-07T11:03:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.319+0000 7f68397ff700 1 -- 172.21.15.29:0/797213820 wait complete. 2023-10-07T11:03:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-07T11:03:05.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:05.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.547+0000 7fd87dfff700 1 Processor -- start 2023-10-07T11:03:05.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.547+0000 7fd87dfff700 1 -- start start 2023-10-07T11:03:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd87dfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8780ae8b0 0x7fd8780b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd87dfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd878158ad0 con 0x7fd8780b1840 2023-10-07T11:03:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd87dfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87815b140 con 0x7fd878157290 2023-10-07T11:03:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd87dfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8780b1cc0 con 0x7fd8780ae8b0 2023-10-07T11:03:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd877fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd8780b1840 0x7fd878157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59176/0 (socket says 172.21.15.29:59176) 2023-10-07T11:03:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd877fff700 1 -- 172.21.15.29:0/1051006278 learned_addr learned my addr 172.21.15.29:0/1051006278 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:05.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd876ffd700 1 --2- 172.21.15.29:0/1051006278 >> v2:172.21.15.144:3300/0 conn(0x7fd8780ae8b0 0x7fd8780b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd8767fc700 1 -- 172.21.15.29:0/1051006278 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637989661 0 0) 0x7fd878158ad0 con 0x7fd8780b1840 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd8767fc700 1 -- 172.21.15.29:0/1051006278 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd860003570 con 0x7fd8780b1840 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.549+0000 7fd876ffd700 1 -- 172.21.15.29:0/1051006278 >> v1:172.21.15.144:6789/0 conn(0x7fd878157290 legacy=0x7fd8780cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.550+0000 7fd876ffd700 1 -- 172.21.15.29:0/1051006278 >> v1:172.21.15.29:6789/0 conn(0x7fd8780b1840 legacy=0x7fd878157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.550+0000 7fd876ffd700 1 -- 172.21.15.29:0/1051006278 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87815b420 con 0x7fd8780ae8b0 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.550+0000 7fd876ffd700 1 --2- 172.21.15.29:0/1051006278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ae8b0 0x7fd8780b1300 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098e0 tx=0x7fd86c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b7ce793dca9a20 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.550+0000 7fd8767fc700 1 -- 172.21.15.29:0/1051006278 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00b640 con 0x7fd8780ae8b0 2023-10-07T11:03:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.550+0000 7fd8767fc700 1 -- 172.21.15.29:0/1051006278 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c00b7a0 con 0x7fd8780ae8b0 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.550+0000 7fd8767fc700 1 -- 172.21.15.29:0/1051006278 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00bb40 con 0x7fd8780ae8b0 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 -- 172.21.15.29:0/1051006278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ae8b0 msgr2=0x7fd8780b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1051006278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ae8b0 0x7fd8780b1300 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098e0 tx=0x7fd86c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 -- 172.21.15.29:0/1051006278 shutdown_connections 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1051006278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ae8b0 0x7fd8780b1300 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 -- 172.21.15.29:0/1051006278 >> 172.21.15.29:0/1051006278 conn(0x7fd8780680a0 msgr2=0x7fd8780cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 -- 172.21.15.29:0/1051006278 shutdown_connections 2023-10-07T11:03:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.551+0000 7fd87dfff700 1 -- 172.21.15.29:0/1051006278 wait complete. 2023-10-07T11:03:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.552+0000 7fd87dfff700 1 Processor -- start 2023-10-07T11:03:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.553+0000 7fd87dfff700 1 -- start start 2023-10-07T11:03:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.553+0000 7fd87dfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8780b1840 0x7fd8780b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.553+0000 7fd87dfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 0x7fd878141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd87dfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ca210 0x7fd8780ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd87dfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780cabf0 con 0x7fd878157290 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd87dfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780b11d0 con 0x7fd8780ca210 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd87dfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780b0800 con 0x7fd8780b1840 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 0x7fd878141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 0x7fd878141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58992/0 (socket says 172.21.15.29:58992) 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 -- 172.21.15.29:0/1061551581 learned_addr learned my addr 172.21.15.29:0/1061551581 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd877fff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ca210 0x7fd8780ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd8777fe700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8780b1840 0x7fd8780b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 -- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8780b1840 msgr2=0x7fd8780b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8780b1840 0x7fd8780b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 -- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ca210 msgr2=0x7fd8780ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ca210 0x7fd8780ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.554+0000 7fd876ffd700 1 -- 172.21.15.29:0/1061551581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c009420 con 0x7fd878157290 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.555+0000 7fd876ffd700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 0x7fd878141b70 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009f80 tx=0x7fd86c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.555+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c000d10 con 0x7fd878157290 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.555+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8780afbf0 con 0x7fd878157290 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.555+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c00bb20 con 0x7fd878157290 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.555+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c011650 con 0x7fd878157290 2023-10-07T11:03:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.555+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8780b01b0 con 0x7fd878157290 2023-10-07T11:03:05.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.556+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8780b09b0 con 0x7fd878157290 2023-10-07T11:03:05.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.558+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fd86c011a00 con 0x7fd878157290 2023-10-07T11:03:05.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.559+0000 7fd85f7fe700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:05.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.559+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fd86c086000 con 0x7fd878157290 2023-10-07T11:03:05.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.560+0000 7fd8777fe700 1 -- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 msgr2=0x7fd864069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:05.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.560+0000 7fd8777fe700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:05.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.563+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd86c052ce0 con 0x7fd878157290 2023-10-07T11:03:05.692 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:05.692+0000 7f5c0248b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:03:05.746 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:05.746+0000 7fa47cba01c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:03:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.761+0000 7fd8777fe700 1 -- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 msgr2=0x7fd864069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.761+0000 7fd8777fe700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.772+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8780b0970 con 0x7fd878157290 2023-10-07T11:03:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.773+0000 7fd85f7fe700 1 -- 172.21.15.29:0/1061551581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+16635 (secure 0 0 0) 0x7fd86c056300 con 0x7fd878157290 2023-10-07T11:03:05.774 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":184,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:05.438193+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:01:38.164128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-07T11:01:39.365755+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":184,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1934822859":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/1173367986":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/3745422769":"2023-10-08T11:01:56.839654+0000","172.21.15.144:6836/131978":"2023-10-08T11:01:56.839654+0000","172.21.15.144:6834/131978":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/1913657423":"2023-10-08T11:01:56.839654+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 msgr2=0x7fd864069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 msgr2=0x7fd878141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 0x7fd878141b70 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009f80 tx=0x7fd86c017000 comp rx=0 tx=0).stop 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 shutdown_connections 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8780ca210 0x7fd8780ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8780b1840 0x7fd8780b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 --2- 172.21.15.29:0/1061551581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd878157290 0x7fd878141b70 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 >> 172.21.15.29:0/1061551581 conn(0x7fd8780680a0 msgr2=0x7fd878152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 shutdown_connections 2023-10-07T11:03:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:05.776+0000 7fd87dfff700 1 -- 172.21.15.29:0/1061551581 wait complete. 2023-10-07T11:03:05.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T11:01:38.164128+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T11:01:39.365755+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 184, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T11:03:05.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T11:03:05.862 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:05.861+0000 7f5c0248b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:03:05.932 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:05.931+0000 7fa47cba01c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:03:06.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.005+0000 7ff18a1ff700 1 Processor -- start 2023-10-07T11:03:06.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.005+0000 7ff18a1ff700 1 -- start start 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.006+0000 7ff18a1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff184156180 0x7ff1840afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.006+0000 7ff18a1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840ae570 0x7ff1840accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.006+0000 7ff18a1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1840b0520 0x7ff1840b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.006+0000 7ff18a1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840b0ea0 con 0x7ff1840b0520 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.006+0000 7ff18a1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840ad230 con 0x7ff184156180 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.006+0000 7ff18a1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840ad4d0 con 0x7ff1840ae570 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1840b0520 0x7ff1840b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1840b0520 0x7ff1840b0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58994/0 (socket says 172.21.15.29:58994) 2023-10-07T11:03:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 -- 172.21.15.29:0/2412581918 learned_addr learned my addr 172.21.15.29:0/2412581918 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff1837fe700 1 --2- 172.21.15.29:0/2412581918 >> v2:172.21.15.144:3300/0 conn(0x7ff184156180 0x7ff1840afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff182ffd700 1 --2- 172.21.15.29:0/2412581918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840ae570 0x7ff1840accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 -- 172.21.15.29:0/2412581918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840ae570 msgr2=0x7ff1840accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 --2- 172.21.15.29:0/2412581918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840ae570 0x7ff1840accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 -- 172.21.15.29:0/2412581918 >> v2:172.21.15.144:3300/0 conn(0x7ff184156180 msgr2=0x7ff1840afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 --2- 172.21.15.29:0/2412581918 >> v2:172.21.15.144:3300/0 conn(0x7ff184156180 0x7ff1840afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.007+0000 7ff183fff700 1 -- 172.21.15.29:0/2412581918 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840ad7b0 con 0x7ff1840b0520 2023-10-07T11:03:06.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.008+0000 7ff183fff700 1 --2- 172.21.15.29:0/2412581918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840b0520 0x7ff1840b0960 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff1840b2250 tx=0x7ff17c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3562bd7c7582358c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:06.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.008+0000 7ff1827fc700 1 -- 172.21.15.29:0/2412581918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00e070 con 0x7ff1840b0520 2023-10-07T11:03:06.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.008+0000 7ff1827fc700 1 -- 172.21.15.29:0/2412581918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c00b970 con 0x7ff1840b0520 2023-10-07T11:03:06.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.008+0000 7ff1827fc700 1 -- 172.21.15.29:0/2412581918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0092e0 con 0x7ff1840b0520 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.008+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2412581918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840b0520 msgr2=0x7ff1840b0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.008+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2412581918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840b0520 0x7ff1840b0960 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff1840b2250 tx=0x7ff17c009840 comp rx=0 tx=0).stop 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2412581918 shutdown_connections 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2412581918 >> v2:172.21.15.144:3300/0 conn(0x7ff184156180 0x7ff1840afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2412581918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840b0520 0x7ff1840b0960 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2412581918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840ae570 0x7ff1840accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2412581918 >> 172.21.15.29:0/2412581918 conn(0x7ff184067610 msgr2=0x7ff184067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2412581918 shutdown_connections 2023-10-07T11:03:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.009+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2412581918 wait complete. 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.010+0000 7ff18a1ff700 1 Processor -- start 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.010+0000 7ff18a1ff700 1 -- start start 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.011+0000 7ff18a1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 0x7ff1840cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.011+0000 7ff18a1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840b0520 0x7ff18414ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.011+0000 7ff18a1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff184156180 0x7ff18415ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.011+0000 7ff18a1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840cc580 con 0x7ff1840ae570 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.011+0000 7ff18a1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18415d9b0 con 0x7ff184156180 2023-10-07T11:03:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.011+0000 7ff18a1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18414f3a0 con 0x7ff1840b0520 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 0x7ff1840cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 0x7ff1840cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58996/0 (socket says 172.21.15.29:58996) 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 -- 172.21.15.29:0/2444311846 learned_addr learned my addr 172.21.15.29:0/2444311846 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff183fff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff184156180 0x7ff18415ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff182ffd700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840b0520 0x7ff18414ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 -- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840b0520 msgr2=0x7ff18414ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840b0520 0x7ff18414ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 -- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff184156180 msgr2=0x7ff18415ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff184156180 0x7ff18415ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 -- 172.21.15.29:0/2444311846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c009ae0 con 0x7ff1840ae570 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff183fff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff184156180 0x7ff18415ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:06.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.012+0000 7ff1837fe700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 0x7ff1840cc040 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff17400ecb0 tx=0x7ff1740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:06.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.013+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17400c850 con 0x7ff1840ae570 2023-10-07T11:03:06.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.013+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17400c9b0 con 0x7ff1840ae570 2023-10-07T11:03:06.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.013+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17400cd50 con 0x7ff1840ae570 2023-10-07T11:03:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.013+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18415f360 con 0x7ff1840ae570 2023-10-07T11:03:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.013+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1840cc7b0 con 0x7ff1840ae570 2023-10-07T11:03:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.014+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18404c3c0 con 0x7ff1840ae570 2023-10-07T11:03:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.015+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7ff174013680 con 0x7ff1840ae570 2023-10-07T11:03:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.016+0000 7ff16b7fe700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 0x7ff170069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.016+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7ff174084e20 con 0x7ff1840ae570 2023-10-07T11:03:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.016+0000 7ff182ffd700 1 -- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 msgr2=0x7ff170069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.016+0000 7ff182ffd700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 0x7ff170069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:06.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.020+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff174051a60 con 0x7ff1840ae570 2023-10-07T11:03:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.217+0000 7ff182ffd700 1 -- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 msgr2=0x7ff170069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.217+0000 7ff182ffd700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 0x7ff170069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.221+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1840b1610 con 0x7ff1840ae570 2023-10-07T11:03:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.222+0000 7ff16b7fe700 1 -- 172.21.15.29:0/2444311846 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff174055080 con 0x7ff1840ae570 2023-10-07T11:03:06.222 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T11:03:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 msgr2=0x7ff170069650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 0x7ff170069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 msgr2=0x7ff1840cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 0x7ff1840cc040 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff17400ecb0 tx=0x7ff1740091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 shutdown_connections 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff184156180 0x7ff18415ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1840ae570 0x7ff1840cc040 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7ff1700671a0 0x7ff170069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 --2- 172.21.15.29:0/2444311846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1840b0520 0x7ff18414ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 >> 172.21.15.29:0/2444311846 conn(0x7ff184067610 msgr2=0x7ff18414f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 shutdown_connections 2023-10-07T11:03:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.224+0000 7ff18a1ff700 1 -- 172.21.15.29:0/2444311846 wait complete. 2023-10-07T11:03:06.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T11:03:06.281 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:06.280+0000 7f5c0248b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:03:06.375 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:06.375+0000 7fa47cba01c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:03:06.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.445+0000 7f4e88bff700 1 Processor -- start 2023-10-07T11:03:06.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.445+0000 7f4e88bff700 1 -- start start 2023-10-07T11:03:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e88bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4e84159750 0x7f4e840d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e88bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159b20 0x7f4e840ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e88bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e840ce6d0 0x7f4e840ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e88bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840cbf20 con 0x7f4e840ce6d0 2023-10-07T11:03:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e88bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840cc200 con 0x7f4e84159750 2023-10-07T11:03:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e88bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840cc4e0 con 0x7f4e84159b20 2023-10-07T11:03:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e82d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e840ce6d0 0x7f4e840ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e82d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e840ce6d0 0x7f4e840ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59002/0 (socket says 172.21.15.29:59002) 2023-10-07T11:03:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e8259c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4e84159750 0x7f4e840d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e82d9d700 1 -- 172.21.15.29:0/4224301902 learned_addr learned my addr 172.21.15.29:0/4224301902 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.447+0000 7f4e81d9b700 1 --2- 172.21.15.29:0/4224301902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159b20 0x7f4e840ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e82d9d700 1 -- 172.21.15.29:0/4224301902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159b20 msgr2=0x7f4e840ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:06.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e82d9d700 1 --2- 172.21.15.29:0/4224301902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159b20 0x7f4e840ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e82d9d700 1 -- 172.21.15.29:0/4224301902 >> v2:172.21.15.144:3300/0 conn(0x7f4e84159750 msgr2=0x7f4e840d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e82d9d700 1 --2- 172.21.15.29:0/4224301902 >> v2:172.21.15.144:3300/0 conn(0x7f4e84159750 0x7f4e840d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e82d9d700 1 -- 172.21.15.29:0/4224301902 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840cc7c0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e8259c700 1 --2- 172.21.15.29:0/4224301902 >> v2:172.21.15.144:3300/0 conn(0x7f4e84159750 0x7f4e840d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e82d9d700 1 --2- 172.21.15.29:0/4224301902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ce6d0 0x7f4e840ceb50 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f4e84089840 tx=0x7f4e7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5359344cc07404 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.448+0000 7f4e8159a700 1 -- 172.21.15.29:0/4224301902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c00e070 con 0x7f4e840ce6d0 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e8159a700 1 -- 172.21.15.29:0/4224301902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c00b970 con 0x7f4e840ce6d0 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e8159a700 1 -- 172.21.15.29:0/4224301902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0092e0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 -- 172.21.15.29:0/4224301902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ce6d0 msgr2=0x7f4e840ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 --2- 172.21.15.29:0/4224301902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ce6d0 0x7f4e840ceb50 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f4e84089840 tx=0x7f4e7c009840 comp rx=0 tx=0).stop 2023-10-07T11:03:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 -- 172.21.15.29:0/4224301902 shutdown_connections 2023-10-07T11:03:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 --2- 172.21.15.29:0/4224301902 >> v2:172.21.15.144:3300/0 conn(0x7f4e84159750 0x7f4e840d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 --2- 172.21.15.29:0/4224301902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ce6d0 0x7f4e840ceb50 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 --2- 172.21.15.29:0/4224301902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159b20 0x7f4e840ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 -- 172.21.15.29:0/4224301902 >> 172.21.15.29:0/4224301902 conn(0x7f4e84063a10 msgr2=0x7f4e840d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.449+0000 7f4e88bff700 1 -- 172.21.15.29:0/4224301902 shutdown_connections 2023-10-07T11:03:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.450+0000 7f4e88bff700 1 -- 172.21.15.29:0/4224301902 wait complete. 2023-10-07T11:03:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.451+0000 7f4e88bff700 1 Processor -- start 2023-10-07T11:03:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.451+0000 7f4e88bff700 1 -- start start 2023-10-07T11:03:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.452+0000 7f4e88bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 0x7f4e84157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.452+0000 7f4e88bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 0x7f4e84157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.452+0000 7f4e88bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e84159b20 0x7f4e841512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.452+0000 7f4e88bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84158a60 con 0x7f4e84159b20 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.452+0000 7f4e88bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840b3b90 con 0x7f4e840ce6d0 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.452+0000 7f4e88bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840b4410 con 0x7f4e84159750 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 0x7f4e84157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e81d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 0x7f4e84157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e82d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e84159b20 0x7f4e841512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 0x7f4e84157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36416/0 (socket says 172.21.15.29:36416) 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e81d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 0x7f4e84157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:57470/0 (socket says 172.21.15.29:57470) 2023-10-07T11:03:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 -- 172.21.15.29:0/1427988801 learned_addr learned my addr 172.21.15.29:0/1427988801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 -- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 msgr2=0x7f4e84157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 0x7f4e84157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 -- 172.21.15.29:0/1427988801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e84159b20 msgr2=0x7f4e841512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e84159b20 0x7f4e841512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 -- 172.21.15.29:0/1427988801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e740093c0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e82d9d700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e84159b20 0x7f4e841512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e81d9b700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 0x7f4e84157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.453+0000 7f4e8259c700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 0x7f4e84157a30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400ec80 tx=0x7f4e74009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.454+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74005a00 con 0x7f4e840ce6d0 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.454+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74005b60 con 0x7f4e840ce6d0 2023-10-07T11:03:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.454+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c009ae0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.454+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7400c850 con 0x7f4e840ce6d0 2023-10-07T11:03:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.454+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e84158780 con 0x7f4e840ce6d0 2023-10-07T11:03:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.455+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8404c3c0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.456+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f4e7400cbc0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.457+0000 7f4e727fc700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 0x7f4e64069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.457+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f4e740850b0 con 0x7f4e840ce6d0 2023-10-07T11:03:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.457+0000 7f4e81d9b700 1 -- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 msgr2=0x7f4e64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.457+0000 7f4e81d9b700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 0x7f4e64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.461+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e74055780 con 0x7f4e840ce6d0 2023-10-07T11:03:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.658+0000 7f4e81d9b700 1 -- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 msgr2=0x7f4e64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:06.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.658+0000 7f4e81d9b700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 0x7f4e64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.665+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4e84156b10 con 0x7f4e840ce6d0 2023-10-07T11:03:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.666+0000 7f4e727fc700 1 -- 172.21.15.29:0/1427988801 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v184) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4e74055310 con 0x7f4e840ce6d0 2023-10-07T11:03:06.666 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T11:03:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 msgr2=0x7f4e64069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 0x7f4e64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 msgr2=0x7f4e84157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 0x7f4e84157a30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400ec80 tx=0x7f4e74009120 comp rx=0 tx=0).stop 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 shutdown_connections 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e84159b20 0x7f4e841512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e840ce6d0 0x7f4e84157a30 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f4e64067170 0x7f4e64069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 --2- 172.21.15.29:0/1427988801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e84159750 0x7f4e84157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 >> 172.21.15.29:0/1427988801 conn(0x7f4e84063a10 msgr2=0x7f4e840c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 shutdown_connections 2023-10-07T11:03:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.668+0000 7f4e88bff700 1 -- 172.21.15.29:0/1427988801 wait complete. 2023-10-07T11:03:06.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T11:03:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.886+0000 7fd7eb3ff700 1 Processor -- start 2023-10-07T11:03:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.887+0000 7fd7eb3ff700 1 -- start start 2023-10-07T11:03:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7eb3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e41596e0 0x7fd7e40cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7eb3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7e40cc630 0x7fd7e40acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7eb3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e40ccaa0 con 0x7fd7e40ad440 2023-10-07T11:03:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7eb3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e40aa8e0 con 0x7fd7e40cc630 2023-10-07T11:03:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7eb3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4159ab0 con 0x7fd7e41596e0 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7e899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7e40cc630 0x7fd7e40acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7e999c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd7e40ad440 0x7fd7e415d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59184/0 (socket says 172.21.15.29:59184) 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7e999c700 1 -- 172.21.15.29:0/882110490 learned_addr learned my addr 172.21.15.29:0/882110490 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.888+0000 7fd7e919b700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e41596e0 0x7fd7e40cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7dbfff700 1 -- 172.21.15.29:0/882110490 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956476450 0 0) 0x7fd7e40ccaa0 con 0x7fd7e40ad440 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7dbfff700 1 -- 172.21.15.29:0/882110490 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d0003570 con 0x7fd7e40ad440 2023-10-07T11:03:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7e899a700 1 -- 172.21.15.29:0/882110490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e41596e0 msgr2=0x7fd7e40cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7e899a700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e41596e0 0x7fd7e40cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7e899a700 1 -- 172.21.15.29:0/882110490 >> v1:172.21.15.29:6789/0 conn(0x7fd7e40ad440 legacy=0x7fd7e415d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7e899a700 1 -- 172.21.15.29:0/882110490 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e40a98d0 con 0x7fd7e40cc630 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7e919b700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e41596e0 0x7fd7e40cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7e899a700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e40cc630 0x7fd7e40acf00 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40098e0 tx=0x7fd7d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c967e093dff605dd server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7dbfff700 1 -- 172.21.15.29:0/882110490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4011630 con 0x7fd7e40cc630 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7dbfff700 1 -- 172.21.15.29:0/882110490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d4011790 con 0x7fd7e40cc630 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.889+0000 7fd7dbfff700 1 -- 172.21.15.29:0/882110490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4011b30 con 0x7fd7e40cc630 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/882110490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e40cc630 msgr2=0x7fd7e40acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e40cc630 0x7fd7e40acf00 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40098e0 tx=0x7fd7d40091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/882110490 shutdown_connections 2023-10-07T11:03:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e41596e0 0x7fd7e40cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/882110490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e40cc630 0x7fd7e40acf00 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/882110490 >> 172.21.15.29:0/882110490 conn(0x7fd7e4067610 msgr2=0x7fd7e4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/882110490 shutdown_connections 2023-10-07T11:03:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.890+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/882110490 wait complete. 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.892+0000 7fd7eb3ff700 1 Processor -- start 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.892+0000 7fd7eb3ff700 1 -- start start 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7eb3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7e40ad440 0x7fd7e4143c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7eb3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e40cc630 0x7fd7e40ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7eb3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 0x7fd7e4169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7eb3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e415d580 con 0x7fd7e41596e0 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7eb3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4144140 con 0x7fd7e40ad440 2023-10-07T11:03:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7eb3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e40c8230 con 0x7fd7e40cc630 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7e999c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 0x7fd7e4169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7e919b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7e40ad440 0x7fd7e4143c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7e999c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 0x7fd7e4169950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59036/0 (socket says 172.21.15.29:59036) 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7e999c700 1 -- 172.21.15.29:0/3214478069 learned_addr learned my addr 172.21.15.29:0/3214478069 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.893+0000 7fd7e899a700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e40cc630 0x7fd7e40ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7e999c700 1 -- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e40cc630 msgr2=0x7fd7e40ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7e999c700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e40cc630 0x7fd7e40ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7e999c700 1 -- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7e40ad440 msgr2=0x7fd7e4143c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7e999c700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7e40ad440 0x7fd7e4143c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7e999c700 1 -- 172.21.15.29:0/3214478069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e0009420 con 0x7fd7e41596e0 2023-10-07T11:03:06.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7e999c700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 0x7fd7e4169950 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc001460 tx=0x7fd7dc001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00a870 con 0x7fd7e41596e0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc00a9d0 con 0x7fd7e41596e0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d4009420 con 0x7fd7e41596e0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00ad50 con 0x7fd7e41596e0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.894+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e40c87b0 con 0x7fd7e41596e0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.895+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e415d6f0 con 0x7fd7e41596e0 2023-10-07T11:03:06.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.896+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fd7dc007300 con 0x7fd7e41596e0 2023-10-07T11:03:06.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.897+0000 7fd7d97fa700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 0x7fd7c8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:06.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.897+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fd7dc084f60 con 0x7fd7e41596e0 2023-10-07T11:03:06.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.897+0000 7fd7e919b700 1 -- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 msgr2=0x7fd7c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:06.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.897+0000 7fd7e919b700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 0x7fd7c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:06.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:06.901+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7dc051b90 con 0x7fd7e41596e0 2023-10-07T11:03:07.059 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.058+0000 7f5c0248b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:03:07.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.098+0000 7fd7e919b700 1 -- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 msgr2=0x7fd7c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:07.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.098+0000 7fd7e919b700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 0x7fd7c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:07.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.105+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd7e404c3c0 con 0x7fd7e41596e0 2023-10-07T11:03:07.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.106+0000 7fd7d97fa700 1 -- 172.21.15.29:0/3214478069 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v184) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd7dc0551b0 con 0x7fd7e41596e0 2023-10-07T11:03:07.107 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 msgr2=0x7fd7c8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 0x7fd7c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 msgr2=0x7fd7e4169950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 0x7fd7e4169950 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc001460 tx=0x7fd7dc001a40 comp rx=0 tx=0).stop 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 shutdown_connections 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7e41596e0 0x7fd7e4169950 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7e40ad440 0x7fd7e4143c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fd7c8067170 0x7fd7c8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 --2- 172.21.15.29:0/3214478069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7e40cc630 0x7fd7e40ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 >> 172.21.15.29:0/3214478069 conn(0x7fd7e4067610 msgr2=0x7fd7e40cbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:07.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 shutdown_connections 2023-10-07T11:03:07.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.110+0000 7fd7eb3ff700 1 -- 172.21.15.29:0/3214478069 wait complete. 2023-10-07T11:03:07.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:07.146 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.145+0000 7f5c0248b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:03:07.211 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.211+0000 7fa47cba01c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:03:07.299 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.299+0000 7f5c0248b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:03:07.305 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.304+0000 7fa47cba01c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:03:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.339+0000 7f7fb89ff700 1 Processor -- start 2023-10-07T11:03:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.340+0000 7f7fb89ff700 1 -- start start 2023-10-07T11:03:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb89ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7fb40ad370 0x7f7fb40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb89ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb414d790 con 0x7f7fb4159ec0 2023-10-07T11:03:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb89ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb415a880 con 0x7f7fb40ad370 2023-10-07T11:03:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40ae5d0 con 0x7f7fb40b0410 2023-10-07T11:03:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb259c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7fb40ad370 0x7f7fb40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.341+0000 7f7fb2d9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb2d9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:57502/0 (socket says 172.21.15.29:57502) 2023-10-07T11:03:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb2d9d700 1 -- 172.21.15.29:0/3294506809 learned_addr learned my addr 172.21.15.29:0/3294506809 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb159a700 1 -- 172.21.15.29:0/3294506809 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900904142 0 0) 0x7f7fb414d790 con 0x7f7fb4159ec0 2023-10-07T11:03:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb159a700 1 -- 172.21.15.29:0/3294506809 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c003570 con 0x7f7fb4159ec0 2023-10-07T11:03:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb259c700 1 -- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 msgr2=0x7f7fb415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb259c700 1 --2- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb259c700 1 -- 172.21.15.29:0/3294506809 >> v1:172.21.15.144:6789/0 conn(0x7f7fb4159ec0 legacy=0x7f7fb41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb259c700 1 -- 172.21.15.29:0/3294506809 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40ad740 con 0x7f7fb40ad370 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.342+0000 7f7fb259c700 1 --2- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb40cc870 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009910 tx=0x7f7fa40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54eaece0c320f9be server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb159a700 1 -- 172.21.15.29:0/3294506809 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4011630 con 0x7f7fb40ad370 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb159a700 1 -- 172.21.15.29:0/3294506809 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4011790 con 0x7f7fb40ad370 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb159a700 1 -- 172.21.15.29:0/3294506809 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4011b30 con 0x7f7fb40ad370 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 -- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 msgr2=0x7f7fb40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb40cc870 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009910 tx=0x7f7fa40091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 -- 172.21.15.29:0/3294506809 shutdown_connections 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/3294506809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb40cc870 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 -- 172.21.15.29:0/3294506809 >> 172.21.15.29:0/3294506809 conn(0x7f7fb40680a0 msgr2=0x7f7fb40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 -- 172.21.15.29:0/3294506809 shutdown_connections 2023-10-07T11:03:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.343+0000 7f7fb89ff700 1 -- 172.21.15.29:0/3294506809 wait complete. 2023-10-07T11:03:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.345+0000 7f7fb89ff700 1 Processor -- start 2023-10-07T11:03:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.345+0000 7f7fb89ff700 1 -- start start 2023-10-07T11:03:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb89ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb416ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb4155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb89ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 0x7f7fb414bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb89ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4157180 con 0x7f7fb4159ec0 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb89ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb41560e0 con 0x7f7fb40ad370 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb41563c0 con 0x7f7fb40b0410 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb416ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 0x7f7fb414bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb416ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36430/0 (socket says 172.21.15.29:36430) 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 0x7f7fb414bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59048/0 (socket says 172.21.15.29:59048) 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb259c700 1 -- 172.21.15.29:0/713227795 learned_addr learned my addr 172.21.15.29:0/713227795 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.346+0000 7f7fb1d9b700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb4155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb259c700 1 -- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 msgr2=0x7f7fb4155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb259c700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb4155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb259c700 1 -- 172.21.15.29:0/713227795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 msgr2=0x7f7fb414bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb259c700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 0x7f7fb414bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb259c700 1 -- 172.21.15.29:0/713227795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4009420 con 0x7f7fb40ad370 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb2d9d700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 0x7f7fb414bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fb259c700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb416ccb0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4068dd0 tx=0x7f7fa4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.347+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4011b10 con 0x7f7fb40ad370 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.348+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4000cc0 con 0x7f7fb40ad370 2023-10-07T11:03:07.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.348+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb41566a0 con 0x7f7fb40ad370 2023-10-07T11:03:07.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.348+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4025610 con 0x7f7fb40ad370 2023-10-07T11:03:07.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.348+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb4156c60 con 0x7f7fb40ad370 2023-10-07T11:03:07.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.349+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb404c3c0 con 0x7f7fb40ad370 2023-10-07T11:03:07.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.350+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f7fa402e430 con 0x7f7fb40ad370 2023-10-07T11:03:07.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.351+0000 7f7fa27fc700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 0x7f7f94069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.351+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f7fa4086360 con 0x7f7fb40ad370 2023-10-07T11:03:07.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.351+0000 7f7fb1d9b700 1 -- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 msgr2=0x7f7f94069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:07.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.351+0000 7f7fb1d9b700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 0x7f7f94069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:07.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.355+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7fa4053020 con 0x7f7fb40ad370 2023-10-07T11:03:07.387 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.387+0000 7f5c0248b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:03:07.470 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.470+0000 7fa47cba01c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:03:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.552+0000 7f7fb1d9b700 1 -- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 msgr2=0x7f7f94069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.552+0000 7f7fb1d9b700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 0x7f7f94069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:07.562 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.562+0000 7fa47cba01c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:03:07.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.602+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fb41585a0 con 0x7f7fb40ad370 2023-10-07T11:03:07.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.603+0000 7f7fa27fc700 1 -- 172.21.15.29:0/713227795 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f7fa4056640 con 0x7f7fb40ad370 2023-10-07T11:03:07.604 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:07.604 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:01:40.378766+0000","modified":"2023-10-07T11:01:44.282264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/562306367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":562306367},{"type":"v1","addr":"172.21.15.144:6837","nonce":562306367}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-10-07T11:03:07.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 msgr2=0x7f7f94069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:07.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 0x7f7f94069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 msgr2=0x7f7fb416ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb416ccb0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4068dd0 tx=0x7f7fa4017000 comp rx=0 tx=0).stop 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 shutdown_connections 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fb4159ec0 0x7f7fb414bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40ad370 0x7f7fb416ccb0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7f94067170 0x7f7f94069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 --2- 172.21.15.29:0/713227795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb40b0410 0x7f7fb4155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 >> 172.21.15.29:0/713227795 conn(0x7f7fb40680a0 msgr2=0x7f7fb41579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 shutdown_connections 2023-10-07T11:03:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.605+0000 7f7fb89ff700 1 -- 172.21.15.29:0/713227795 wait complete. 2023-10-07T11:03:07.608 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.605+0000 7f5c0248b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:03:07.608 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-07T11:03:07.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:07.703 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.702+0000 7f5c0248b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:03:07.790 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.789+0000 7fa47cba01c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:03:07.798 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.797+0000 7f5c0248b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:03:07.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.825+0000 7f7bb0fff700 1 Processor -- start 2023-10-07T11:03:07.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.825+0000 7f7bb0fff700 1 -- start start 2023-10-07T11:03:07.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.826+0000 7f7bb0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.826+0000 7f7bb0fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bac0b1a10 0x7f7bac157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.826+0000 7f7bb0fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bac158ad0 con 0x7f7bac0ae8b0 2023-10-07T11:03:07.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.826+0000 7f7bb0fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bac15b140 con 0x7f7bac0b1a10 2023-10-07T11:03:07.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.826+0000 7f7bb0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac15b420 con 0x7f7bac157290 2023-10-07T11:03:07.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baad9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bac0b1a10 0x7f7bac157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baa59c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baad9d700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bac0b1a10 0x7f7bac157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36434/0 (socket says 172.21.15.29:36434) 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baad9d700 1 -- 172.21.15.29:0/1365454218 learned_addr learned my addr 172.21.15.29:0/1365454218 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baa59c700 1 --2- 172.21.15.29:0/1365454218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:57524/0 (socket says 172.21.15.29:57524) 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baa59c700 1 -- 172.21.15.29:0/1365454218 >> v2:172.21.15.144:3300/0 conn(0x7f7bac0b1a10 msgr2=0x7f7bac157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baa59c700 1 --2- 172.21.15.29:0/1365454218 >> v2:172.21.15.144:3300/0 conn(0x7f7bac0b1a10 0x7f7bac157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baa59c700 1 -- 172.21.15.29:0/1365454218 >> v1:172.21.15.144:6789/0 conn(0x7f7bac0ae8b0 legacy=0x7f7bac0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.827+0000 7f7baa59c700 1 -- 172.21.15.29:0/1365454218 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac15b700 con 0x7f7bac157290 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.828+0000 7f7baa59c700 1 --2- 172.21.15.29:0/1365454218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0cc870 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0098e0 tx=0x7f7b9c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5245d1a78278abe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.828+0000 7f7ba959a700 1 -- 172.21.15.29:0/1365454218 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c00b640 con 0x7f7bac157290 2023-10-07T11:03:07.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.828+0000 7f7ba959a700 1 -- 172.21.15.29:0/1365454218 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c00b7a0 con 0x7f7bac157290 2023-10-07T11:03:07.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.828+0000 7f7ba959a700 1 -- 172.21.15.29:0/1365454218 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c00bb40 con 0x7f7bac157290 2023-10-07T11:03:07.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.828+0000 7f7bb0fff700 1 -- 172.21.15.29:0/1365454218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 msgr2=0x7f7bac0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.828+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/1365454218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0cc870 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0098e0 tx=0x7f7b9c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.829+0000 7f7bb0fff700 1 -- 172.21.15.29:0/1365454218 shutdown_connections 2023-10-07T11:03:07.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.829+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/1365454218 >> v2:172.21.15.144:3300/0 conn(0x7f7bac0b1a10 0x7f7bac157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.829+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/1365454218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0cc870 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.829+0000 7f7bb0fff700 1 -- 172.21.15.29:0/1365454218 >> 172.21.15.29:0/1365454218 conn(0x7f7bac0680a0 msgr2=0x7f7bac0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:07.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.829+0000 7f7bb0fff700 1 -- 172.21.15.29:0/1365454218 shutdown_connections 2023-10-07T11:03:07.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.829+0000 7f7bb0fff700 1 -- 172.21.15.29:0/1365454218 wait complete. 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.830+0000 7f7bb0fff700 1 Processor -- start 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.830+0000 7f7bb0fff700 1 -- start start 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.831+0000 7f7bb0fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bac0b1a10 0x7f7bac150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.831+0000 7f7bb0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.831+0000 7f7bb0fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 0x7f7bac1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.831+0000 7f7bb0fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac0b0d30 con 0x7f7bac0b08b0 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.831+0000 7f7bb0fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac15df60 con 0x7f7bac0b1a10 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.831+0000 7f7bb0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac154cd0 con 0x7f7bac157290 2023-10-07T11:03:07.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 0x7f7bac1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baa59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bac0b1a10 0x7f7bac150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 0x7f7bac1540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59064/0 (socket says 172.21.15.29:59064) 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 -- 172.21.15.29:0/3458908492 learned_addr learned my addr 172.21.15.29:0/3458908492 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7ba9d9b700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 -- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 msgr2=0x7f7bac0b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0b0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 -- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bac0b1a10 msgr2=0x7f7bac150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bac0b1a10 0x7f7bac150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 -- 172.21.15.29:0/3458908492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c009420 con 0x7f7bac0b08b0 2023-10-07T11:03:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.832+0000 7f7baad9d700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 0x7f7bac1540a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009e20 tx=0x7f7ba400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:07.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.833+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba400c920 con 0x7f7bac0b08b0 2023-10-07T11:03:07.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.833+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba400f040 con 0x7f7bac0b08b0 2023-10-07T11:03:07.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.833+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba400ccd0 con 0x7f7bac0b08b0 2023-10-07T11:03:07.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.833+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac154fb0 con 0x7f7bac0b08b0 2023-10-07T11:03:07.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.833+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac155570 con 0x7f7bac0b08b0 2023-10-07T11:03:07.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.834+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bac155790 con 0x7f7bac0b08b0 2023-10-07T11:03:07.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.835+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f7ba4014650 con 0x7f7bac0b08b0 2023-10-07T11:03:07.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.836+0000 7f7b9a7fc700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 0x7f7b8c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:07.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.836+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f7ba4085370 con 0x7f7bac0b08b0 2023-10-07T11:03:07.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.836+0000 7f7baa59c700 1 -- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 msgr2=0x7f7b8c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:07.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.836+0000 7f7baa59c700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 0x7f7b8c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:07.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:07.840+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bac155790 con 0x7f7bac0b08b0 2023-10-07T11:03:07.894 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.893+0000 7fa47cba01c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:03:07.896 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:07.895+0000 7f5c0248b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:03:07.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:07.993+0000 7fa47cba01c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:03:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.037+0000 7f7baa59c700 1 -- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 msgr2=0x7f7b8c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.037+0000 7f7baa59c700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 0x7f7b8c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:08.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.045+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bac155790 con 0x7f7bac0b08b0 2023-10-07T11:03:08.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.046+0000 7f7b9a7fc700 1 -- 172.21.15.29:0/3458908492 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+16635 (secure 0 0 0) 0x7f7bac155790 con 0x7f7bac0b08b0 2023-10-07T11:03:08.047 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:08.047 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":184,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:05.438193+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:01:38.164128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-07T11:01:39.365755+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":184,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1934822859":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/1173367986":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/3745422769":"2023-10-08T11:01:56.839654+0000","172.21.15.144:6836/131978":"2023-10-08T11:01:56.839654+0000","172.21.15.144:6834/131978":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/1913657423":"2023-10-08T11:01:56.839654+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 msgr2=0x7f7b8c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 0x7f7b8c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 msgr2=0x7f7bac1540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 0x7f7bac1540a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009e20 tx=0x7f7ba400b040 comp rx=0 tx=0).stop 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 shutdown_connections 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bac0b08b0 0x7f7bac1540a0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bac0b1a10 0x7f7bac150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7b8c067170 0x7f7b8c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 --2- 172.21.15.29:0/3458908492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bac157290 0x7f7bac0b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 >> 172.21.15.29:0/3458908492 conn(0x7f7bac0680a0 msgr2=0x7f7bac152ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 shutdown_connections 2023-10-07T11:03:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.049+0000 7f7bb0fff700 1 -- 172.21.15.29:0/3458908492 wait complete. 2023-10-07T11:03:08.068 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T11:03:08.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:08.084 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:08.083+0000 7f5c0248b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:03:08.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:08.094+0000 7fa47cba01c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:03:08.182 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:08.181+0000 7f5c0248b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:03:08.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.266+0000 7fe08d7ff700 1 Processor -- start 2023-10-07T11:03:08.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.266+0000 7fe08d7ff700 1 -- start start 2023-10-07T11:03:08.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.267+0000 7fe08d7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0880afd20 0x7fe0881596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.267+0000 7fe08d7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0880b1250 0x7fe0880acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.267+0000 7fe08d7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880b0dc0 con 0x7fe088159320 2023-10-07T11:03:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.267+0000 7fe08d7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0880cd0d0 con 0x7fe0880b1250 2023-10-07T11:03:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.267+0000 7fe08d7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880b01b0 con 0x7fe0880afd20 2023-10-07T11:03:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0867fc700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0880afd20 0x7fe0881596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0877fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0880b1250 0x7fe0880acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0877fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0880b1250 0x7fe0880acd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36456/0 (socket says 172.21.15.29:36456) 2023-10-07T11:03:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0877fe700 1 -- 172.21.15.29:0/1842338995 learned_addr learned my addr 172.21.15.29:0/1842338995 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0867fc700 1 -- 172.21.15.29:0/1842338995 >> v2:172.21.15.144:3300/0 conn(0x7fe0880b1250 msgr2=0x7fe0880acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0867fc700 1 --2- 172.21.15.29:0/1842338995 >> v2:172.21.15.144:3300/0 conn(0x7fe0880b1250 0x7fe0880acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0867fc700 1 -- 172.21.15.29:0/1842338995 >> v1:172.21.15.144:6789/0 conn(0x7fe088159320 legacy=0x7fe0880cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0867fc700 1 -- 172.21.15.29:0/1842338995 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880b0490 con 0x7fe0880afd20 2023-10-07T11:03:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.268+0000 7fe0867fc700 1 --2- 172.21.15.29:0/1842338995 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0880afd20 0x7fe0881596f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0098e0 tx=0x7fe07c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f4cfe189248b066 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe085ffb700 1 -- 172.21.15.29:0/1842338995 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00b6d0 con 0x7fe0880afd20 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe085ffb700 1 -- 172.21.15.29:0/1842338995 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c00b830 con 0x7fe0880afd20 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe085ffb700 1 -- 172.21.15.29:0/1842338995 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00bbd0 con 0x7fe0880afd20 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1842338995 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0880afd20 msgr2=0x7fe0881596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1842338995 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0880afd20 0x7fe0881596f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0098e0 tx=0x7fe07c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1842338995 shutdown_connections 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1842338995 >> v2:172.21.15.144:3300/0 conn(0x7fe0880b1250 0x7fe0880acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1842338995 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0880afd20 0x7fe0881596f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.269+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1842338995 >> 172.21.15.29:0/1842338995 conn(0x7fe0880680a0 msgr2=0x7fe0880cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.270+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1842338995 shutdown_connections 2023-10-07T11:03:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.270+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1842338995 wait complete. 2023-10-07T11:03:08.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.271+0000 7fe08d7ff700 1 Processor -- start 2023-10-07T11:03:08.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.271+0000 7fe08d7ff700 1 -- start start 2023-10-07T11:03:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe08d7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 0x7fe0881406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe08d7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0880b1250 0x7fe08815a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe08d7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe088159320 0x7fe0881510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe08d7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe088140c30 con 0x7fe0880afd20 2023-10-07T11:03:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe08d7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880ab3e0 con 0x7fe0880b1250 2023-10-07T11:03:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe08d7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880ab6c0 con 0x7fe088159320 2023-10-07T11:03:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.272+0000 7fe086ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 0x7fe0881406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 0x7fe0881406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59074/0 (socket says 172.21.15.29:59074) 2023-10-07T11:03:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 -- 172.21.15.29:0/1283744606 learned_addr learned my addr 172.21.15.29:0/1283744606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe0867fc700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0880b1250 0x7fe08815a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe0877fe700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe088159320 0x7fe0881510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 -- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe088159320 msgr2=0x7fe0881510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe088159320 0x7fe0881510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 -- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0880b1250 msgr2=0x7fe08815a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0880b1250 0x7fe08815a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 -- 172.21.15.29:0/1283744606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c009420 con 0x7fe0880afd20 2023-10-07T11:03:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe0877fe700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe088159320 0x7fe0881510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe0867fc700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0880b1250 0x7fe08815a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe086ffd700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 0x7fe0881406f0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fe07800c4e0 tx=0x7fe07800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.273+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078011590 con 0x7fe0880afd20 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.274+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe078011710 con 0x7fe0880afd20 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.274+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078011ad0 con 0x7fe0880afd20 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.274+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0880ab9a0 con 0x7fe0880afd20 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.274+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0880ac0f0 con 0x7fe0880afd20 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.275+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0880abc60 con 0x7fe0880afd20 2023-10-07T11:03:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.276+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fe07801b3b0 con 0x7fe0880afd20 2023-10-07T11:03:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.278+0000 7fe06effd700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 0x7fe074069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.278+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fe078083f70 con 0x7fe0880afd20 2023-10-07T11:03:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.278+0000 7fe0867fc700 1 -- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 msgr2=0x7fe074069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.278+0000 7fe0867fc700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 0x7fe074069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.282+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe078050bb0 con 0x7fe0880afd20 2023-10-07T11:03:08.297 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:08.296+0000 7fa47cba01c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:03:08.400 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:08.399+0000 7fa47cba01c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:03:08.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.479+0000 7fe0867fc700 1 -- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 msgr2=0x7fe074069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:08.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.479+0000 7fe0867fc700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 0x7fe074069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:08.526 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:08.525+0000 7f5c0248b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:03:08.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.529+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe08804c420 con 0x7fe0880afd20 2023-10-07T11:03:08.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.530+0000 7fe06effd700 1 -- 172.21.15.29:0/1283744606 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4598 (secure 0 0 0) 0x7fe078017070 con 0x7fe0880afd20 2023-10-07T11:03:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:08.530 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:01:40.378766+0000","modified":"2023-10-07T11:01:44.282264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/562306367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":562306367},{"type":"v1","addr":"172.21.15.144:6837","nonce":562306367}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-10-07T11:03:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 msgr2=0x7fe074069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 0x7fe074069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 msgr2=0x7fe0881406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 0x7fe0881406f0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fe07800c4e0 tx=0x7fe07800c7f0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 shutdown_connections 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fe074067170 0x7fe074069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe088159320 0x7fe0881510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0880afd20 0x7fe0881406f0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 --2- 172.21.15.29:0/1283744606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0880b1250 0x7fe08815a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 >> 172.21.15.29:0/1283744606 conn(0x7fe0880680a0 msgr2=0x7fe088156c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 shutdown_connections 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.532+0000 7fe08d7ff700 1 -- 172.21.15.29:0/1283744606 wait complete. 2023-10-07T11:03:08.534 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-07T11:03:08.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:08.613 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:08.612+0000 7f5c0248b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:08.753 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:08.753+0000 7fa47cba01c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:03:08.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.791+0000 7f09075ff700 1 Processor -- start 2023-10-07T11:03:08.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.791+0000 7f09075ff700 1 -- start start 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.792+0000 7f09075ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0900159550 0x7f09000b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.792+0000 7f09075ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b0f50 0x7f09000b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.792+0000 7f09075ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09000b2bb0 0x7f090015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.792+0000 7f09075ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090006f4e0 con 0x7f0900159550 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.792+0000 7f09075ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090006e8d0 con 0x7f09000b2bb0 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.792+0000 7f09075ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090006ebb0 con 0x7f09000b0f50 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0900159550 0x7f09000b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0900159550 0x7f09000b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59090/0 (socket says 172.21.15.29:59090) 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f0905b9c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09000b2bb0 0x7f090015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 -- 172.21.15.29:0/2486069602 learned_addr learned my addr 172.21.15.29:0/2486069602 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f0904b9a700 1 --2- 172.21.15.29:0/2486069602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b0f50 0x7f09000b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 -- 172.21.15.29:0/2486069602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b0f50 msgr2=0x7f09000b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 --2- 172.21.15.29:0/2486069602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b0f50 0x7f09000b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 -- 172.21.15.29:0/2486069602 >> v2:172.21.15.144:3300/0 conn(0x7f09000b2bb0 msgr2=0x7f090015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 --2- 172.21.15.29:0/2486069602 >> v2:172.21.15.144:3300/0 conn(0x7f09000b2bb0 0x7f090015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 -- 172.21.15.29:0/2486069602 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090006ee90 con 0x7f0900159550 2023-10-07T11:03:08.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.793+0000 7f090539b700 1 --2- 172.21.15.29:0/2486069602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0900159550 0x7f09000b0a10 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f0900068200 tx=0x7f08fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e0b8ca1d871b35 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:08.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.794+0000 7f08f7fff700 1 -- 172.21.15.29:0/2486069602 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc00b640 con 0x7f0900159550 2023-10-07T11:03:08.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.794+0000 7f08f7fff700 1 -- 172.21.15.29:0/2486069602 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc00b7a0 con 0x7f0900159550 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.794+0000 7f08f7fff700 1 -- 172.21.15.29:0/2486069602 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc00bb60 con 0x7f0900159550 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.794+0000 7f09075ff700 1 -- 172.21.15.29:0/2486069602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0900159550 msgr2=0x7f09000b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.794+0000 7f09075ff700 1 --2- 172.21.15.29:0/2486069602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0900159550 0x7f09000b0a10 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f0900068200 tx=0x7f08fc009250 comp rx=0 tx=0).stop 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 -- 172.21.15.29:0/2486069602 shutdown_connections 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 --2- 172.21.15.29:0/2486069602 >> v2:172.21.15.144:3300/0 conn(0x7f09000b2bb0 0x7f090015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 --2- 172.21.15.29:0/2486069602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0900159550 0x7f09000b0a10 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 --2- 172.21.15.29:0/2486069602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b0f50 0x7f09000b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 -- 172.21.15.29:0/2486069602 >> 172.21.15.29:0/2486069602 conn(0x7f0900069190 msgr2=0x7f090006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 -- 172.21.15.29:0/2486069602 shutdown_connections 2023-10-07T11:03:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.795+0000 7f09075ff700 1 -- 172.21.15.29:0/2486069602 wait complete. 2023-10-07T11:03:08.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.797+0000 7f09075ff700 1 Processor -- start 2023-10-07T11:03:08.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.797+0000 7f09075ff700 1 -- start start 2023-10-07T11:03:08.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f09075ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 0x7f0900147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f09075ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b2bb0 0x7f090014a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f09075ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0900159550 0x7f090014e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f09075ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090016a330 con 0x7f09000b0f50 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f09075ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090016e400 con 0x7f0900159550 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f09075ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090016e120 con 0x7f09000b2bb0 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f090539b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 0x7f0900147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f090539b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 0x7f0900147a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59096/0 (socket says 172.21.15.29:59096) 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f090539b700 1 -- 172.21.15.29:0/2902693644 learned_addr learned my addr 172.21.15.29:0/2902693644 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f0904b9a700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b2bb0 0x7f090014a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.799+0000 7f0905b9c700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0900159550 0x7f090014e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f090539b700 1 -- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b2bb0 msgr2=0x7f090014a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f090539b700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b2bb0 0x7f090014a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f090539b700 1 -- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0900159550 msgr2=0x7f090014e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:08.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f090539b700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0900159550 0x7f090014e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:08.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f090539b700 1 -- 172.21.15.29:0/2902693644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc0094f0 con 0x7f09000b0f50 2023-10-07T11:03:08.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f0905b9c700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0900159550 0x7f090014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:08.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f090539b700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 0x7f0900147a80 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f08fc006fd0 tx=0x7f08fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:08.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc000d10 con 0x7f09000b0f50 2023-10-07T11:03:08.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc00bb20 con 0x7f09000b0f50 2023-10-07T11:03:08.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc011670 con 0x7f09000b0f50 2023-10-07T11:03:08.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0900149d70 con 0x7f09000b0f50 2023-10-07T11:03:08.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.800+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090014a330 con 0x7f09000b0f50 2023-10-07T11:03:08.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.802+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09000b13f0 con 0x7f09000b0f50 2023-10-07T11:03:08.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.802+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f08fc0119e0 con 0x7f09000b0f50 2023-10-07T11:03:08.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.803+0000 7f08f57fa700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 0x7f08e4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:08.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.803+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f08fc086450 con 0x7f09000b0f50 2023-10-07T11:03:08.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.803+0000 7f0904b9a700 1 -- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 msgr2=0x7f08e4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:08.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.803+0000 7f0904b9a700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 0x7f08e4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:08.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:08.808+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08fc053090 con 0x7f09000b0f50 2023-10-07T11:03:08.851 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:08.850+0000 7fa47cba01c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:09.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.004+0000 7f0904b9a700 1 -- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 msgr2=0x7f08e4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.004+0000 7f0904b9a700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 0x7f08e4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:09.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.058+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f090006a3e0 con 0x7f09000b0f50 2023-10-07T11:03:09.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.059+0000 7f08f57fa700 1 -- 172.21.15.29:0/2902693644 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f08fc0566b0 con 0x7f09000b0f50 2023-10-07T11:03:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:09.060 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:01:40.378766+0000","modified":"2023-10-07T11:01:44.282264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/562306367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":562306367},{"type":"v1","addr":"172.21.15.144:6837","nonce":562306367}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-10-07T11:03:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 msgr2=0x7f08e4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 0x7f08e4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 msgr2=0x7f0900147a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 0x7f0900147a80 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f08fc006fd0 tx=0x7f08fc017000 comp rx=0 tx=0).stop 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 shutdown_connections 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0900159550 0x7f090014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09000b0f50 0x7f0900147a80 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f08e4067170 0x7f08e4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 --2- 172.21.15.29:0/2902693644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09000b2bb0 0x7f090014a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 >> 172.21.15.29:0/2902693644 conn(0x7f0900069190 msgr2=0x7f0900152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 shutdown_connections 2023-10-07T11:03:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.062+0000 7f09075ff700 1 -- 172.21.15.29:0/2902693644 wait complete. 2023-10-07T11:03:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-07T11:03:09.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:09.155 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:09.155+0000 7f5c0248b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:03:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.278+0000 7fa664bff700 1 Processor -- start 2023-10-07T11:03:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.278+0000 7fa664bff700 1 -- start start 2023-10-07T11:03:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.280+0000 7fa664bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa660157290 0x7fa6600cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.280+0000 7fa664bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa6600b1a10 0x7fa660157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.280+0000 7fa664bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa660158ad0 con 0x7fa6600ae8b0 2023-10-07T11:03:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.280+0000 7fa664bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66015b140 con 0x7fa6600b1a10 2023-10-07T11:03:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.280+0000 7fa664bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66015b420 con 0x7fa660157290 2023-10-07T11:03:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65dd9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa6600ae8b0 0x7fa6600b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59186/0 (socket says 172.21.15.29:59186) 2023-10-07T11:03:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65dd9b700 1 -- 172.21.15.29:0/1809763050 learned_addr learned my addr 172.21.15.29:0/1809763050 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65ed9d700 1 --2- 172.21.15.29:0/1809763050 >> v2:172.21.15.144:3300/0 conn(0x7fa6600b1a10 0x7fa660157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65e59c700 1 --2- 172.21.15.29:0/1809763050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa660157290 0x7fa6600cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65d59a700 1 -- 172.21.15.29:0/1809763050 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6426469 0 0) 0x7fa660158ad0 con 0x7fa6600ae8b0 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65d59a700 1 -- 172.21.15.29:0/1809763050 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa648003570 con 0x7fa6600ae8b0 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65e59c700 1 -- 172.21.15.29:0/1809763050 >> v2:172.21.15.144:3300/0 conn(0x7fa6600b1a10 msgr2=0x7fa660157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65e59c700 1 --2- 172.21.15.29:0/1809763050 >> v2:172.21.15.144:3300/0 conn(0x7fa6600b1a10 0x7fa660157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65e59c700 1 -- 172.21.15.29:0/1809763050 >> v1:172.21.15.29:6789/0 conn(0x7fa6600ae8b0 legacy=0x7fa6600b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65e59c700 1 -- 172.21.15.29:0/1809763050 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66015b700 con 0x7fa660157290 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.281+0000 7fa65ed9d700 1 --2- 172.21.15.29:0/1809763050 >> v2:172.21.15.144:3300/0 conn(0x7fa6600b1a10 0x7fa660157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.282+0000 7fa65e59c700 1 --2- 172.21.15.29:0/1809763050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa660157290 0x7fa6600cc870 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098e0 tx=0x7fa6500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86c975460d410e09 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.282+0000 7fa65d59a700 1 -- 172.21.15.29:0/1809763050 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650011630 con 0x7fa660157290 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.282+0000 7fa65d59a700 1 -- 172.21.15.29:0/1809763050 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650011790 con 0x7fa660157290 2023-10-07T11:03:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.282+0000 7fa65d59a700 1 -- 172.21.15.29:0/1809763050 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650011b30 con 0x7fa660157290 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 -- 172.21.15.29:0/1809763050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa660157290 msgr2=0x7fa6600cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 --2- 172.21.15.29:0/1809763050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa660157290 0x7fa6600cc870 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098e0 tx=0x7fa6500091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 -- 172.21.15.29:0/1809763050 shutdown_connections 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 --2- 172.21.15.29:0/1809763050 >> v2:172.21.15.144:3300/0 conn(0x7fa6600b1a10 0x7fa660157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 --2- 172.21.15.29:0/1809763050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa660157290 0x7fa6600cc870 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 -- 172.21.15.29:0/1809763050 >> 172.21.15.29:0/1809763050 conn(0x7fa6600680a0 msgr2=0x7fa6600cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 -- 172.21.15.29:0/1809763050 shutdown_connections 2023-10-07T11:03:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.283+0000 7fa664bff700 1 -- 172.21.15.29:0/1809763050 wait complete. 2023-10-07T11:03:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.285+0000 7fa664bff700 1 Processor -- start 2023-10-07T11:03:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.286+0000 7fa664bff700 1 -- start start 2023-10-07T11:03:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa664bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 0x7fa660143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa664bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa660157290 0x7fa66016af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa664bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6600c5380 0x7fa6600c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa664bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600c5d60 con 0x7fa6600b1a10 2023-10-07T11:03:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa664bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6600afac0 con 0x7fa660157290 2023-10-07T11:03:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa664bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa660144240 con 0x7fa6600c5380 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa65e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 0x7fa660143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa65e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 0x7fa660143d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59100/0 (socket says 172.21.15.29:59100) 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.287+0000 7fa65e59c700 1 -- 172.21.15.29:0/2106048789 learned_addr learned my addr 172.21.15.29:0/2106048789 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65dd9b700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa660157290 0x7fa66016af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65ed9d700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6600c5380 0x7fa6600c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65e59c700 1 -- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6600c5380 msgr2=0x7fa6600c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65e59c700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6600c5380 0x7fa6600c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65e59c700 1 -- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa660157290 msgr2=0x7fa66016af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65e59c700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa660157290 0x7fa66016af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65e59c700 1 -- 172.21.15.29:0/2106048789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa658009a20 con 0x7fa6600b1a10 2023-10-07T11:03:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.288+0000 7fa65e59c700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 0x7fa660143d00 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fa660070f30 tx=0x7fa650017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.289+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650011900 con 0x7fa6600b1a10 2023-10-07T11:03:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.289+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa650009420 con 0x7fa6600b1a10 2023-10-07T11:03:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.289+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650011a80 con 0x7fa6600b1a10 2023-10-07T11:03:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.289+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650025670 con 0x7fa6600b1a10 2023-10-07T11:03:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.289+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6600af190 con 0x7fa6600b1a10 2023-10-07T11:03:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.290+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66006b590 con 0x7fa6600b1a10 2023-10-07T11:03:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.292+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7fa650023070 con 0x7fa6600b1a10 2023-10-07T11:03:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.293+0000 7fa64e7fc700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 0x7fa640069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.293+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fa650085f80 con 0x7fa6600b1a10 2023-10-07T11:03:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.293+0000 7fa65dd9b700 1 -- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 msgr2=0x7fa640069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.293+0000 7fa65dd9b700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 0x7fa640069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.297+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa650052c80 con 0x7fa6600b1a10 2023-10-07T11:03:09.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:09.342+0000 7f5c0248b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:03:09.435 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:09.434+0000 7fa47cba01c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:03:09.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.494+0000 7fa65dd9b700 1 -- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 msgr2=0x7fa640069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.494+0000 7fa65dd9b700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 0x7fa640069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:09.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.501+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa66006bdd0 con 0x7fa6600b1a10 2023-10-07T11:03:09.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.502+0000 7fa64e7fc700 1 -- 172.21.15.29:0/2106048789 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+16635 (secure 0 0 0) 0x7fa650015090 con 0x7fa6600b1a10 2023-10-07T11:03:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:09.503 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":184,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:05.438193+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:01:38.164128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-07T11:01:39.365755+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":184,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1934822859":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/1173367986":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/3745422769":"2023-10-08T11:01:56.839654+0000","172.21.15.144:6836/131978":"2023-10-08T11:01:56.839654+0000","172.21.15.144:6834/131978":"2023-10-08T11:01:56.839654+0000","172.21.15.144:0/1913657423":"2023-10-08T11:01:56.839654+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:09.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 msgr2=0x7fa640069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:09.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 0x7fa640069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 msgr2=0x7fa660143d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 0x7fa660143d00 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fa660070f30 tx=0x7fa650017000 comp rx=0 tx=0).stop 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 shutdown_connections 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7fa640067170 0x7fa640069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6600c5380 0x7fa6600c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6600b1a10 0x7fa660143d00 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 --2- 172.21.15.29:0/2106048789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa660157290 0x7fa66016af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.504+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 >> 172.21.15.29:0/2106048789 conn(0x7fa6600680a0 msgr2=0x7fa660150520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.505+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 shutdown_connections 2023-10-07T11:03:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.505+0000 7fa664bff700 1 -- 172.21.15.29:0/2106048789 wait complete. 2023-10-07T11:03:09.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T11:03:09.633 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:09.632+0000 7fa47cba01c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:03:09.720 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:09.719+0000 7f5c0248b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.725+0000 7f76a89ff700 1 Processor -- start 2023-10-07T11:03:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.725+0000 7f76a89ff700 1 -- start start 2023-10-07T11:03:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a89ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76a4159b20 0x7f76a40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a89ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40cbf20 con 0x7f76a40ce8a0 2023-10-07T11:03:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a89ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40cc200 con 0x7f76a4159750 2023-10-07T11:03:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a89ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a40cc4e0 con 0x7f76a4159b20 2023-10-07T11:03:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a2d9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76a40ce8a0 0x7f76a40c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59188/0 (socket says 172.21.15.29:59188) 2023-10-07T11:03:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a2d9d700 1 -- 172.21.15.29:0/32641592 learned_addr learned my addr 172.21.15.29:0/32641592 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.727+0000 7f76a1d9b700 1 --2- 172.21.15.29:0/32641592 >> v2:172.21.15.29:3300/0 conn(0x7f76a4159b20 0x7f76a40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736864843 0 0) 0x7f76a40cbf20 con 0x7f76a40ce8a0 2023-10-07T11:03:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c003570 con 0x7f76a40ce8a0 2023-10-07T11:03:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445606865 0 0) 0x7f76a40cc200 con 0x7f76a4159750 2023-10-07T11:03:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a40cbf20 con 0x7f76a4159750 2023-10-07T11:03:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a1d9b700 1 -- 172.21.15.29:0/32641592 >> v1:172.21.15.144:6789/0 conn(0x7f76a4159750 legacy=0x7f76a40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a1d9b700 1 -- 172.21.15.29:0/32641592 >> v1:172.21.15.29:6789/0 conn(0x7f76a40ce8a0 legacy=0x7f76a40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a1d9b700 1 -- 172.21.15.29:0/32641592 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40cc7c0 con 0x7f76a4159b20 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a1d9b700 1 --2- 172.21.15.29:0/32641592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a4159b20 0x7f76a40d10a0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f76980098e0 tx=0x7f76980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4758099aaefab0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.728+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698011630 con 0x7f76a4159b20 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7698011790 con 0x7f76a4159b20 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a159a700 1 -- 172.21.15.29:0/32641592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698011b30 con 0x7f76a4159b20 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a89ff700 1 -- 172.21.15.29:0/32641592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a4159b20 msgr2=0x7f76a40d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a89ff700 1 --2- 172.21.15.29:0/32641592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a4159b20 0x7f76a40d10a0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f76980098e0 tx=0x7f76980091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a89ff700 1 -- 172.21.15.29:0/32641592 shutdown_connections 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a89ff700 1 --2- 172.21.15.29:0/32641592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a4159b20 0x7f76a40d10a0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a89ff700 1 -- 172.21.15.29:0/32641592 >> 172.21.15.29:0/32641592 conn(0x7f76a4063a10 msgr2=0x7f76a40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.729+0000 7f76a89ff700 1 -- 172.21.15.29:0/32641592 shutdown_connections 2023-10-07T11:03:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.730+0000 7f76a89ff700 1 -- 172.21.15.29:0/32641592 wait complete. 2023-10-07T11:03:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.731+0000 7f76a89ff700 1 Processor -- start 2023-10-07T11:03:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.731+0000 7f76a89ff700 1 -- start start 2023-10-07T11:03:09.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a89ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 0x7f76a40b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a89ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76a4159750 0x7f76a4159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76a4159b20 0x7f76a4144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a89ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a415a450 con 0x7f76a40ce8a0 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a89ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a415b290 con 0x7f76a4159750 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a415a680 con 0x7f76a4159b20 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 0x7f76a40b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 0x7f76a40b5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59106/0 (socket says 172.21.15.29:59106) 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.732+0000 7f76a259c700 1 -- 172.21.15.29:0/3998812569 learned_addr learned my addr 172.21.15.29:0/3998812569 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a2d9d700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76a4159b20 0x7f76a4144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a1d9b700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76a4159750 0x7f76a4159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a259c700 1 -- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76a4159b20 msgr2=0x7f76a4144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a259c700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76a4159b20 0x7f76a4144f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a259c700 1 -- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76a4159750 msgr2=0x7f76a4159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a259c700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76a4159750 0x7f76a4159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a259c700 1 -- 172.21.15.29:0/3998812569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698009420 con 0x7f76a40ce8a0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.733+0000 7f76a259c700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 0x7f76a40b5e70 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f769400c7c0 tx=0x7f7694007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.734+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694001570 con 0x7f76a40ce8a0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.734+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76940016f0 con 0x7f76a40ce8a0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.734+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a415a960 con 0x7f76a40ce8a0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.734+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694001ab0 con 0x7f76a40ce8a0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.734+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a415af20 con 0x7f76a40ce8a0 2023-10-07T11:03:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.735+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a404c3c0 con 0x7f76a40ce8a0 2023-10-07T11:03:09.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.736+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f769401b3b0 con 0x7f76a40ce8a0 2023-10-07T11:03:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.737+0000 7f76927fc700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 0x7f7684069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.737+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f7694084450 con 0x7f76a40ce8a0 2023-10-07T11:03:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.737+0000 7f76a1d9b700 1 -- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 msgr2=0x7f7684069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.737+0000 7f76a1d9b700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 0x7f7684069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.741+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7694051090 con 0x7f76a40ce8a0 2023-10-07T11:03:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.938+0000 7f76a1d9b700 1 -- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 msgr2=0x7f7684069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6834/131978 2023-10-07T11:03:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.938+0000 7f76a1d9b700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 0x7f7684069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.992+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f76a4064040 con 0x7f76a40ce8a0 2023-10-07T11:03:09.993 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T11:03:09.993+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T11:03:09.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:09.997+0000 7f76927fc700 1 -- 172.21.15.29:0/3998812569 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7f76940546b0 con 0x7f76a40ce8a0 2023-10-07T11:03:10.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 msgr2=0x7f7684069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:10.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 0x7f7684069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 msgr2=0x7f76a40b5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:10.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 0x7f76a40b5e70 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f769400c7c0 tx=0x7f7694007dc0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 shutdown_connections 2023-10-07T11:03:10.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7684067170 0x7f7684069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76a4159b20 0x7f76a4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a40ce8a0 0x7f76a40b5e70 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 --2- 172.21.15.29:0/3998812569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76a4159750 0x7f76a4159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 >> 172.21.15.29:0/3998812569 conn(0x7f76a4063a10 msgr2=0x7f76a4154520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 shutdown_connections 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.000+0000 7f76a89ff700 1 -- 172.21.15.29:0/3998812569 wait complete. 2023-10-07T11:03:10.002 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T11:03:10.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T11:03:10.024 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:10.023+0000 7fa47cba01c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:10.058 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T11:03:10.148 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.147+0000 7f5c0248b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:03:10.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.219+0000 7f71abfff700 1 Processor -- start 2023-10-07T11:03:10.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.220+0000 7f71abfff700 1 -- start start 2023-10-07T11:03:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71abfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71abfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 0x7f71a40c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71abfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a40c78c0 con 0x7f71a413fba0 2023-10-07T11:03:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71abfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a406fa50 con 0x7f71a413ff70 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71abfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a406c0f0 con 0x7f71a4067d20 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71a959a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71aa59c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 0x7f71a40c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71aa59c700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 0x7f71a40c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:36522/0 (socket says 172.21.15.29:36522) 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71a959a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a406ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:57594/0 (socket says 172.21.15.29:57594) 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71aa59c700 1 -- 172.21.15.29:0/4204007623 learned_addr learned my addr 172.21.15.29:0/4204007623 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.222+0000 7f71a9d9b700 1 --1- 172.21.15.29:0/4204007623 >> v1:172.21.15.144:6789/0 conn(0x7f71a413fba0 0x7f71a4067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:56988/0 (socket says 172.21.15.29:56988) 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a8d99700 1 -- 172.21.15.29:0/4204007623 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358391542 0 0) 0x7f71a40c78c0 con 0x7f71a413fba0 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a959a700 1 -- 172.21.15.29:0/4204007623 >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 msgr2=0x7f71a40c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a959a700 1 --2- 172.21.15.29:0/4204007623 >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 0x7f71a40c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a959a700 1 -- 172.21.15.29:0/4204007623 >> v1:172.21.15.144:6789/0 conn(0x7f71a413fba0 legacy=0x7f71a4067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a959a700 1 -- 172.21.15.29:0/4204007623 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a406c3d0 con 0x7f71a4067d20 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71aa59c700 1 --2- 172.21.15.29:0/4204007623 >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 0x7f71a40c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a959a700 1 --2- 172.21.15.29:0/4204007623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a406ba10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f71940098d0 tx=0x7f7194009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e793896370553d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.223+0000 7f71a8d99700 1 -- 172.21.15.29:0/4204007623 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194005c10 con 0x7f71a4067d20 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71a8d99700 1 -- 172.21.15.29:0/4204007623 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7194005d70 con 0x7f71a4067d20 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71a8d99700 1 -- 172.21.15.29:0/4204007623 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194006280 con 0x7f71a4067d20 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 -- 172.21.15.29:0/4204007623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 msgr2=0x7f71a406ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 --2- 172.21.15.29:0/4204007623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a406ba10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f71940098d0 tx=0x7f7194009120 comp rx=0 tx=0).stop 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 -- 172.21.15.29:0/4204007623 shutdown_connections 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 --2- 172.21.15.29:0/4204007623 >> v2:172.21.15.144:3300/0 conn(0x7f71a413ff70 0x7f71a40c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 --2- 172.21.15.29:0/4204007623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a406ba10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 -- 172.21.15.29:0/4204007623 >> 172.21.15.29:0/4204007623 conn(0x7f71a406b1d0 msgr2=0x7f71a40cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 -- 172.21.15.29:0/4204007623 shutdown_connections 2023-10-07T11:03:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.224+0000 7f71abfff700 1 -- 172.21.15.29:0/4204007623 wait complete. 2023-10-07T11:03:10.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.226+0000 7f71abfff700 1 Processor -- start 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.226+0000 7f71abfff700 1 -- start start 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71abfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a4157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71abfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a413fba0 0x7f71a4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71abfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71a413ff70 0x7f71a415a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71abfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a415a900 con 0x7f71a413fba0 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71abfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a415d150 con 0x7f71a413ff70 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71abfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40c4ab0 con 0x7f71a4067d20 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71a959a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a413fba0 0x7f71a4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:10.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71a959a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a413fba0 0x7f71a4159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59110/0 (socket says 172.21.15.29:59110) 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71a959a700 1 -- 172.21.15.29:0/2650727649 learned_addr learned my addr 172.21.15.29:0/2650727649 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71a9d9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a4157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.227+0000 7f71aa59c700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71a413ff70 0x7f71a415a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71aa59c700 1 -- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 msgr2=0x7f71a4157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71aa59c700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a4157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71aa59c700 1 -- 172.21.15.29:0/2650727649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a413fba0 msgr2=0x7f71a4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71aa59c700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a413fba0 0x7f71a4159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71aa59c700 1 -- 172.21.15.29:0/2650727649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c009ae0 con 0x7f71a413ff70 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71aa59c700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71a413ff70 0x7f71a415a3c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f719c009e40 tx=0x7f719c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c00c890 con 0x7f71a413ff70 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c00f040 con 0x7f71a413ff70 2023-10-07T11:03:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71940093c0 con 0x7f71a413ff70 2023-10-07T11:03:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.228+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c00cc60 con 0x7f71a413ff70 2023-10-07T11:03:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.229+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a40c4460 con 0x7f71a413ff70 2023-10-07T11:03:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.229+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a40c4c40 con 0x7f71a413ff70 2023-10-07T11:03:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.231+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 95) v1 ==== 84956+0+0 (secure 0 0 0) 0x7f719c014480 con 0x7f71a413ff70 2023-10-07T11:03:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.232+0000 7f7199ffb700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.232+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f719c085140 con 0x7f71a413ff70 2023-10-07T11:03:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.233+0000 7f71a9d9b700 -1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] is using msgr V1 protocol 2023-10-07T11:03:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.233+0000 7f71a9d9b700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:10.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.237+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f719c051ce0 con 0x7f71a413ff70 2023-10-07T11:03:10.330 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.329+0000 7f5c0248b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.434+0000 7f71a9d9b700 -1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] is using msgr V1 protocol 2023-10-07T11:03:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.434+0000 7f71a9d9b700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:10.475 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:10.474+0000 7fa47cba01c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:03:10.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.487+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f71a406c6e0 con 0x7f71a413ff70 2023-10-07T11:03:10.518 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.517+0000 7f5c0248b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:03:10.626 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.625+0000 7f5c0248b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:10.628 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.627+0000 7f5bf7dfb700 -1 --2- 172.21.15.144:0/131978 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x55f7e8d96000 0x55f7e8d9c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] is using msgr V1 protocol 2023-10-07T11:03:10.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.637+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 96) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f719c0520b0 con 0x7f71a413ff70 2023-10-07T11:03:10.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.637+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 msgr2=0x7f7188069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:10.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.637+0000 7f7199ffb700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:10.668 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:10.667+0000 7fa47cba01c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:10.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7c302700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7c302700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7c302700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7c302700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.705 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7c302700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7fb09700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7fb09700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7fb09700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7fb09700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.706 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:10.704+0000 7f5b7fb09700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:10.867 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:10.866+0000 7fa47cba01c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:03:10.980 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:10.979+0000 7fa47cba01c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:10.983 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:10.983+0000 7fa472dfc700 -1 --2- 172.21.15.29:0/114884 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x5641e960a000 0x5641e9610000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] is using msgr V1 protocol 2023-10-07T11:03:11.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:10.999+0000 7f7199ffb700 1 -- 172.21.15.29:0/2650727649 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7f719c0701f0 con 0x7f71a413ff70 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71a413ff70 msgr2=0x7f71a415a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71a413ff70 0x7f71a415a3c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f719c009e40 tx=0x7f719c00b040 comp rx=0 tx=0).stop 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 shutdown_connections 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71a413ff70 0x7f71a415a3c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:6834/131978,v1:172.21.15.144:6836/131978] conn(0x7f7188067170 0x7f7188069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71a4067d20 0x7f71a4157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 --2- 172.21.15.29:0/2650727649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71a413fba0 0x7f71a4159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 >> 172.21.15.29:0/2650727649 conn(0x7f71a406b1d0 msgr2=0x7f71a413e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:11.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 shutdown_connections 2023-10-07T11:03:11.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.002+0000 7f71abfff700 1 -- 172.21.15.29:0/2650727649 wait complete. 2023-10-07T11:03:11.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T11:03:11.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.219+0000 7f411cbff700 1 Processor -- start 2023-10-07T11:03:11.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.219+0000 7f411cbff700 1 -- start start 2023-10-07T11:03:11.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.220+0000 7f411cbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f41180accf0 0x7f41180b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:11.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.220+0000 7f411cbff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180ad400 con 0x7f41180cc810 2023-10-07T11:03:11.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.220+0000 7f411cbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180ad6e0 con 0x7f41180afeb0 2023-10-07T11:03:11.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.220+0000 7f411cbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411815a140 con 0x7f41180accf0 2023-10-07T11:03:11.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f4115d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41180cc810 0x7f411815a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59204/0 (socket says 172.21.15.29:59204) 2023-10-07T11:03:11.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f4115d9b700 1 -- 172.21.15.29:0/959763013 learned_addr learned my addr 172.21.15.29:0/959763013 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:11.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f4116d9d700 1 --2- 172.21.15.29:0/959763013 >> v2:172.21.15.144:3300/0 conn(0x7f41180accf0 0x7f41180b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f411559a700 1 -- 172.21.15.29:0/959763013 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137919669 0 0) 0x7f41180ad400 con 0x7f41180cc810 2023-10-07T11:03:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f411559a700 1 -- 172.21.15.29:0/959763013 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4100003570 con 0x7f41180cc810 2023-10-07T11:03:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f4116d9d700 1 -- 172.21.15.29:0/959763013 >> v1:172.21.15.144:6789/0 conn(0x7f41180afeb0 legacy=0x7f41180aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f411559a700 1 -- 172.21.15.29:0/959763013 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206104770 0 0) 0x7f41180ad6e0 con 0x7f41180afeb0 2023-10-07T11:03:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f4116d9d700 1 -- 172.21.15.29:0/959763013 >> v1:172.21.15.29:6789/0 conn(0x7f41180cc810 legacy=0x7f411815a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.221+0000 7f4116d9d700 1 -- 172.21.15.29:0/959763013 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180ad9c0 con 0x7f41180accf0 2023-10-07T11:03:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f4116d9d700 1 --2- 172.21.15.29:0/959763013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f41180b05d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4110009e60 tx=0x7f4110009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a15867ba60a4dc74 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411559a700 1 -- 172.21.15.29:0/959763013 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000e070 con 0x7f41180accf0 2023-10-07T11:03:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411559a700 1 -- 172.21.15.29:0/959763013 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411000b850 con 0x7f41180accf0 2023-10-07T11:03:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411559a700 1 -- 172.21.15.29:0/959763013 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0013f0 con 0x7f41180accf0 2023-10-07T11:03:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411cbff700 1 -- 172.21.15.29:0/959763013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 msgr2=0x7f41180b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411cbff700 1 --2- 172.21.15.29:0/959763013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f41180b05d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4110009e60 tx=0x7f4110009780 comp rx=0 tx=0).stop 2023-10-07T11:03:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411cbff700 1 -- 172.21.15.29:0/959763013 shutdown_connections 2023-10-07T11:03:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411cbff700 1 --2- 172.21.15.29:0/959763013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f41180b05d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.222+0000 7f411cbff700 1 -- 172.21.15.29:0/959763013 >> 172.21.15.29:0/959763013 conn(0x7f4118067610 msgr2=0x7f4118067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.223+0000 7f411cbff700 1 -- 172.21.15.29:0/959763013 shutdown_connections 2023-10-07T11:03:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.223+0000 7f411cbff700 1 -- 172.21.15.29:0/959763013 wait complete. 2023-10-07T11:03:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.224+0000 7f411cbff700 1 Processor -- start 2023-10-07T11:03:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.224+0000 7f411cbff700 1 -- start start 2023-10-07T11:03:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f411cbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f4118146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f411cbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 0x7f41180cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f411cbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41180cc810 0x7f41180cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f411cbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41180ab7b0 con 0x7f41180afeb0 2023-10-07T11:03:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f411cbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41180aaba0 con 0x7f41180accf0 2023-10-07T11:03:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f411cbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41180aae80 con 0x7f41180cc810 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.225+0000 7f4115d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 0x7f41180cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 0x7f41180cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59112/0 (socket says 172.21.15.29:59112) 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 -- 172.21.15.29:0/2485777228 learned_addr learned my addr 172.21.15.29:0/2485777228 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4116d9d700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41180cc810 0x7f41180cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f411659c700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f4118146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 -- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41180cc810 msgr2=0x7f41180cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41180cc810 0x7f41180cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 -- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 msgr2=0x7f4118146ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f4118146ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 -- 172.21.15.29:0/2485777228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110009a20 con 0x7f41180afeb0 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.226+0000 7f4115d9b700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 0x7f41180cbba0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f410c00cb60 tx=0x7f410c001a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.227+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c007150 con 0x7f41180afeb0 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.227+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c0072b0 con 0x7f41180afeb0 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.227+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c007670 con 0x7f41180afeb0 2023-10-07T11:03:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.227+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41180ab160 con 0x7f41180afeb0 2023-10-07T11:03:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.227+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41181471e0 con 0x7f41180afeb0 2023-10-07T11:03:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.228+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41180718b0 con 0x7f41180afeb0 2023-10-07T11:03:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.229+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 96) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f410c004a40 con 0x7f41180afeb0 2023-10-07T11:03:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.230+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6869+0+0 (secure 0 0 0) 0x7f410c083c50 con 0x7f41180afeb0 2023-10-07T11:03:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.234+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f410c050e70 con 0x7f41180afeb0 2023-10-07T11:03:11.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.436+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f41180cd020 con 0x7f41180afeb0 2023-10-07T11:03:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.639+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 97) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f410c004cf0 con 0x7f41180afeb0 2023-10-07T11:03:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.639+0000 7f41067fc700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f40f806cb80 0x7f40f806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.641+0000 7f411659c700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f40f806cb80 0x7f40f806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:11.646+0000 7f411659c700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f40f806cb80 0x7f40f806ef60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4108001120 tx=0x7f4108007dc0 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:12.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.001+0000 7f41067fc700 1 -- 172.21.15.29:0/2485777228 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v187) v1 ==== 181+0+0 (secure 0 0 0) 0x7f410c07fdf0 con 0x7f41180afeb0 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f40f806cb80 msgr2=0x7f40f806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f40f806cb80 0x7f40f806ef60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4108001120 tx=0x7f4108007dc0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 msgr2=0x7f41180cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 0x7f41180cbba0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f410c00cb60 tx=0x7f410c001a90 comp rx=0 tx=0).stop 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 shutdown_connections 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f40f806cb80 0x7f40f806ef60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41180cc810 0x7f41180cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41180accf0 0x7f4118146ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 --2- 172.21.15.29:0/2485777228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180afeb0 0x7f41180cbba0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.005+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 >> 172.21.15.29:0/2485777228 conn(0x7f4118067610 msgr2=0x7f4118154600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:12.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.006+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 shutdown_connections 2023-10-07T11:03:12.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.006+0000 7f411cbff700 1 -- 172.21.15.29:0/2485777228 wait complete. 2023-10-07T11:03:12.008 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T11:03:12.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T11:03:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.229+0000 7f98a41ff700 1 Processor -- start 2023-10-07T11:03:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.229+0000 7f98a41ff700 1 -- start start 2023-10-07T11:03:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.230+0000 7f98a41ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f989c0afeb0 0x7f989c0cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a41ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989c0ad160 con 0x7f989c15a590 2023-10-07T11:03:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a41ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989c0ad400 con 0x7f989c0accf0 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a41ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f989c15a140 con 0x7f989c0afeb0 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f989c0afeb0 0x7f989c0cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f989c0afeb0 0x7f989c0cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59126/0 (socket says 172.21.15.29:59126) 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 -- 172.21.15.29:0/629114333 learned_addr learned my addr 172.21.15.29:0/629114333 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 -- 172.21.15.29:0/629114333 >> v1:172.21.15.144:6789/0 conn(0x7f989c0accf0 legacy=0x7f989c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 -- 172.21.15.29:0/629114333 >> v1:172.21.15.29:6789/0 conn(0x7f989c15a590 legacy=0x7f989c0b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 -- 172.21.15.29:0/629114333 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c0ad6e0 con 0x7f989c0afeb0 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.231+0000 7f98a1f9b700 1 --2- 172.21.15.29:0/629114333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0cc810 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f98980099b0 tx=0x7f9898009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4061f486bc72cb04 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a0f99700 1 -- 172.21.15.29:0/629114333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800b820 con 0x7f989c0afeb0 2023-10-07T11:03:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a0f99700 1 -- 172.21.15.29:0/629114333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989800b980 con 0x7f989c0afeb0 2023-10-07T11:03:12.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a0f99700 1 -- 172.21.15.29:0/629114333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800bd20 con 0x7f989c0afeb0 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 -- 172.21.15.29:0/629114333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 msgr2=0x7f989c0cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 --2- 172.21.15.29:0/629114333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0cc810 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f98980099b0 tx=0x7f9898009250 comp rx=0 tx=0).stop 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 -- 172.21.15.29:0/629114333 shutdown_connections 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 --2- 172.21.15.29:0/629114333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0cc810 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 -- 172.21.15.29:0/629114333 >> 172.21.15.29:0/629114333 conn(0x7f989c067610 msgr2=0x7f989c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 -- 172.21.15.29:0/629114333 shutdown_connections 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.232+0000 7f98a41ff700 1 -- 172.21.15.29:0/629114333 wait complete. 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.234+0000 7f98a41ff700 1 Processor -- start 2023-10-07T11:03:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.234+0000 7f98a41ff700 1 -- start start 2023-10-07T11:03:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a41ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c0accf0 0x7f989c1530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a41ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a41ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c15a590 0x7f989c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a41ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0a9820 con 0x7f989c0afeb0 2023-10-07T11:03:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a41ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0a7f10 con 0x7f989c0accf0 2023-10-07T11:03:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a41ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0c9340 con 0x7f989c15a590 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a179a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a179a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59132/0 (socket says 172.21.15.29:59132) 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.235+0000 7f98a179a700 1 -- 172.21.15.29:0/3758840478 learned_addr learned my addr 172.21.15.29:0/3758840478 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a179a700 1 -- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c15a590 msgr2=0x7f989c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a1f9b700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c0accf0 0x7f989c1530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a279c700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c15a590 0x7f989c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a179a700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c15a590 0x7f989c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a179a700 1 -- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c0accf0 msgr2=0x7f989c1530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a179a700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c0accf0 0x7f989c1530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:12.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a179a700 1 -- 172.21.15.29:0/3758840478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980094f0 con 0x7f989c0afeb0 2023-10-07T11:03:12.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a1f9b700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c0accf0 0x7f989c1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:12.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a279c700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c15a590 0x7f989c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:12.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a179a700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0a92e0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f988c00b5e0 tx=0x7f988c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:12.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c00f510 con 0x7f989c0afeb0 2023-10-07T11:03:12.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c00f6b0 con 0x7f989c0afeb0 2023-10-07T11:03:12.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c0a7c30 con 0x7f989c0afeb0 2023-10-07T11:03:12.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c00fa70 con 0x7f989c0afeb0 2023-10-07T11:03:12.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.236+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c0c98b0 con 0x7f989c0afeb0 2023-10-07T11:03:12.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.237+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c0718b0 con 0x7f989c0afeb0 2023-10-07T11:03:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.238+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f988c01a400 con 0x7f989c0afeb0 2023-10-07T11:03:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.240+0000 7f9891ffb700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9880066d80 0x7f9880069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.240+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f988c0840d0 con 0x7f989c0afeb0 2023-10-07T11:03:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.240+0000 7f98a1f9b700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9880066d80 0x7f9880069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.241+0000 7f98a1f9b700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9880066d80 0x7f9880069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f98980099b0 tx=0x7f9898016000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:12.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.244+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f988c0512d0 con 0x7f989c0afeb0 2023-10-07T11:03:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:12.448+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f989c153660 con 0x7f989c0afeb0 2023-10-07T11:03:13.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.005+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f988c0516a0 con 0x7f989c0afeb0 2023-10-07T11:03:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.014+0000 7f9891ffb700 1 -- 172.21.15.29:0/3758840478 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v188) v1 ==== 169+0+0 (secure 0 0 0) 0x7f988c0512d0 con 0x7f989c0afeb0 2023-10-07T11:03:13.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9880066d80 msgr2=0x7f9880069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9880066d80 0x7f9880069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f98980099b0 tx=0x7f9898016000 comp rx=0 tx=0).stop 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 msgr2=0x7f989c0a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0a92e0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f988c00b5e0 tx=0x7f988c0090f0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 shutdown_connections 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9880066d80 0x7f9880069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c15a590 0x7f989c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c0accf0 0x7f989c1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 --2- 172.21.15.29:0/3758840478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0afeb0 0x7f989c0a92e0 secure :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f988c00b5e0 tx=0x7f988c0090f0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.017+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 >> 172.21.15.29:0/3758840478 conn(0x7f989c067610 msgr2=0x7f989c156a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.018+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 shutdown_connections 2023-10-07T11:03:13.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.018+0000 7f98a41ff700 1 -- 172.21.15.29:0/3758840478 wait complete. 2023-10-07T11:03:13.020 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T11:03:13.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T11:03:13.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.243+0000 7f31fadff700 1 Processor -- start 2023-10-07T11:03:13.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.243+0000 7f31fadff700 1 -- start start 2023-10-07T11:03:13.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31fadff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f31f413ff70 0x7f31f406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31fadff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f40c78c0 con 0x7f31f413fba0 2023-10-07T11:03:13.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31fadff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f406fa50 con 0x7f31f4067d20 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31fadff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f406c0f0 con 0x7f31f413ff70 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31f8b9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31f413fba0 0x7f31f4067610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49460/0 (socket says 172.21.15.29:49460) 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31f8b9b700 1 -- 172.21.15.29:0/330350358 learned_addr learned my addr 172.21.15.29:0/330350358 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.245+0000 7f31f939c700 1 --2- 172.21.15.29:0/330350358 >> v2:172.21.15.144:3300/0 conn(0x7f31f413ff70 0x7f31f406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f37fe700 1 -- 172.21.15.29:0/330350358 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353804584 0 0) 0x7f31f40c78c0 con 0x7f31f413fba0 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f37fe700 1 -- 172.21.15.29:0/330350358 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e4003570 con 0x7f31f413fba0 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f37fe700 1 -- 172.21.15.29:0/330350358 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226801125 0 0) 0x7f31e4003570 con 0x7f31f413fba0 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f939c700 1 -- 172.21.15.29:0/330350358 >> v1:172.21.15.144:6789/0 conn(0x7f31f4067d20 legacy=0x7f31f40c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f939c700 1 -- 172.21.15.29:0/330350358 >> v1:172.21.15.29:6789/0 conn(0x7f31f413fba0 legacy=0x7f31f4067610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f939c700 1 -- 172.21.15.29:0/330350358 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f406c3d0 con 0x7f31f413ff70 2023-10-07T11:03:13.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f939c700 1 --2- 172.21.15.29:0/330350358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413ff70 0x7f31f406ba10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009e60 tx=0x7f31ec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68407ce82290863e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:13.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.246+0000 7f31f37fe700 1 -- 172.21.15.29:0/330350358 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec00e070 con 0x7f31f413ff70 2023-10-07T11:03:13.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31f37fe700 1 -- 172.21.15.29:0/330350358 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec00b8e0 con 0x7f31f413ff70 2023-10-07T11:03:13.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31f37fe700 1 -- 172.21.15.29:0/330350358 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8001470 con 0x7f31f413ff70 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 -- 172.21.15.29:0/330350358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413ff70 msgr2=0x7f31f406ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 --2- 172.21.15.29:0/330350358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413ff70 0x7f31f406ba10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009e60 tx=0x7f31ec009780 comp rx=0 tx=0).stop 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 -- 172.21.15.29:0/330350358 shutdown_connections 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 --2- 172.21.15.29:0/330350358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413ff70 0x7f31f406ba10 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 -- 172.21.15.29:0/330350358 >> 172.21.15.29:0/330350358 conn(0x7f31f406b1d0 msgr2=0x7f31f40cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 -- 172.21.15.29:0/330350358 shutdown_connections 2023-10-07T11:03:13.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.247+0000 7f31fadff700 1 -- 172.21.15.29:0/330350358 wait complete. 2023-10-07T11:03:13.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.249+0000 7f31fadff700 1 Processor -- start 2023-10-07T11:03:13.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.249+0000 7f31fadff700 1 -- start start 2023-10-07T11:03:13.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31fadff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f4067d20 0x7f31f415cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31fadff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413fba0 0x7f31f413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31fadff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 0x7f31f40c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31fadff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415e9e0 con 0x7f31f413ff70 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31fadff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415d0b0 con 0x7f31f413fba0 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31fadff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415ba90 con 0x7f31f4067d20 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31f939c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 0x7f31f40c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31f939c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 0x7f31f40c5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48986/0 (socket says 172.21.15.29:48986) 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31f939c700 1 -- 172.21.15.29:0/2539665556 learned_addr learned my addr 172.21.15.29:0/2539665556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31f8b9b700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f4067d20 0x7f31f415cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.250+0000 7f31f3fff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413fba0 0x7f31f413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f939c700 1 -- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f4067d20 msgr2=0x7f31f415cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f939c700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f4067d20 0x7f31f415cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f939c700 1 -- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413fba0 msgr2=0x7f31f413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f939c700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413fba0 0x7f31f413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f939c700 1 -- 172.21.15.29:0/2539665556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009a20 con 0x7f31f413ff70 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f939c700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 0x7f31f40c5db0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f31ec003400 tx=0x7f31ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec012e60 con 0x7f31f413ff70 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec01e450 con 0x7f31f413ff70 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f415bd70 con 0x7f31f413ff70 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec01e7d0 con 0x7f31f413ff70 2023-10-07T11:03:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.251+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f415e670 con 0x7f31f413ff70 2023-10-07T11:03:13.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.252+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f415eb90 con 0x7f31f413ff70 2023-10-07T11:03:13.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.253+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f31ec01eb40 con 0x7f31f413ff70 2023-10-07T11:03:13.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.255+0000 7f31f0ff9700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f31d8066f30 0x7f31d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.255+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f31ec086990 con 0x7f31f413ff70 2023-10-07T11:03:13.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.255+0000 7f31f8b9b700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f31d8066f30 0x7f31d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.256+0000 7f31f8b9b700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f31d8066f30 0x7f31d80693e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f31e8005c20 tx=0x7f31e8001120 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:13.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.259+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31f415eb90 con 0x7f31f413ff70 2023-10-07T11:03:13.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.466+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f31f415eb20 con 0x7f31f413ff70 2023-10-07T11:03:13.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.470+0000 7f31f0ff9700 1 -- 172.21.15.29:0/2539665556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7f31ec016020 con 0x7f31f413ff70 2023-10-07T11:03:13.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f31d8066f30 msgr2=0x7f31d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f31d8066f30 0x7f31d80693e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f31e8005c20 tx=0x7f31e8001120 comp rx=0 tx=0).stop 2023-10-07T11:03:13.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 msgr2=0x7f31f40c5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 0x7f31f40c5db0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f31ec003400 tx=0x7f31ec017000 comp rx=0 tx=0).stop 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 shutdown_connections 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f413ff70 0x7f31f40c5db0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f31d8066f30 0x7f31d80693e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f31f4067d20 0x7f31f415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 --2- 172.21.15.29:0/2539665556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f413fba0 0x7f31f413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.472+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 >> 172.21.15.29:0/2539665556 conn(0x7f31f406b1d0 msgr2=0x7f31f4067610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:13.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.473+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 shutdown_connections 2023-10-07T11:03:13.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.473+0000 7f31fadff700 1 -- 172.21.15.29:0/2539665556 wait complete. 2023-10-07T11:03:13.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T11:03:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.691+0000 7f351afff700 1 Processor -- start 2023-10-07T11:03:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.692+0000 7f351afff700 1 -- start start 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351afff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f351415ade0 0x7f35140b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351afff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35140b0540 0x7f351415c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351afff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3514154090 con 0x7f3514153cc0 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351afff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35140b09e0 con 0x7f35140b0540 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351afff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3514159f60 con 0x7f351415ade0 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351959c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35140b0540 0x7f351415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351959c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35140b0540 0x7f351415c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48998/0 (socket says 172.21.15.29:48998) 2023-10-07T11:03:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f351959c700 1 -- 172.21.15.29:0/3957119149 learned_addr learned my addr 172.21.15.29:0/3957119149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.693+0000 7f3513fff700 1 --2- 172.21.15.29:0/3957119149 >> v2:172.21.15.144:3300/0 conn(0x7f351415ade0 0x7f35140b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f351959c700 1 -- 172.21.15.29:0/3957119149 >> v2:172.21.15.144:3300/0 conn(0x7f351415ade0 msgr2=0x7f35140b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f351959c700 1 --2- 172.21.15.29:0/3957119149 >> v2:172.21.15.144:3300/0 conn(0x7f351415ade0 0x7f35140b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f351959c700 1 -- 172.21.15.29:0/3957119149 >> v1:172.21.15.29:6789/0 conn(0x7f3514153cc0 legacy=0x7f35140b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f351959c700 1 -- 172.21.15.29:0/3957119149 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351415a240 con 0x7f35140b0540 2023-10-07T11:03:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f351959c700 1 --2- 172.21.15.29:0/3957119149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35140b0540 0x7f351415c240 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f35140add60 tx=0x7f350c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a21c2acb461de00 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f35137fe700 1 -- 172.21.15.29:0/3957119149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00e070 con 0x7f35140b0540 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f35137fe700 1 -- 172.21.15.29:0/3957119149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c00ba20 con 0x7f35140b0540 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.694+0000 7f35137fe700 1 -- 172.21.15.29:0/3957119149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c0092e0 con 0x7f35140b0540 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 -- 172.21.15.29:0/3957119149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35140b0540 msgr2=0x7f351415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 --2- 172.21.15.29:0/3957119149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35140b0540 0x7f351415c240 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f35140add60 tx=0x7f350c009840 comp rx=0 tx=0).stop 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 -- 172.21.15.29:0/3957119149 shutdown_connections 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 --2- 172.21.15.29:0/3957119149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35140b0540 0x7f351415c240 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 --2- 172.21.15.29:0/3957119149 >> v2:172.21.15.144:3300/0 conn(0x7f351415ade0 0x7f35140b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 -- 172.21.15.29:0/3957119149 >> 172.21.15.29:0/3957119149 conn(0x7f3514067610 msgr2=0x7f3514067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 -- 172.21.15.29:0/3957119149 shutdown_connections 2023-10-07T11:03:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.695+0000 7f351afff700 1 -- 172.21.15.29:0/3957119149 wait complete. 2023-10-07T11:03:13.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.696+0000 7f351afff700 1 Processor -- start 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.697+0000 7f351afff700 1 -- start start 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351afff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35140b0540 0x7f35140c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351afff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 0x7f35140c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351afff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351415ade0 0x7f35141558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351afff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514155430 con 0x7f3514153cc0 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351afff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514154860 con 0x7f351415ade0 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351afff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3514154b00 con 0x7f35140b0540 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 0x7f35140c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 0x7f35140c8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49008/0 (socket says 172.21.15.29:49008) 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 -- 172.21.15.29:0/976236491 learned_addr learned my addr 172.21.15.29:0/976236491 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:13.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3518d9b700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35140b0540 0x7f35140c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351959c700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351415ade0 0x7f35141558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 -- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35140b0540 msgr2=0x7f35140c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35140b0540 0x7f35140c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 -- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351415ade0 msgr2=0x7f35141558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351415ade0 0x7f35141558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3513fff700 1 -- 172.21.15.29:0/976236491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c009ae0 con 0x7f3514153cc0 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f3518d9b700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35140b0540 0x7f35140c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.698+0000 7f351959c700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351415ade0 0x7f35141558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.699+0000 7f3513fff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 0x7f35140c8730 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f350000e8d0 tx=0x7f35000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.699+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350000ee70 con 0x7f3514153cc0 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.699+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350000c850 con 0x7f3514153cc0 2023-10-07T11:03:13.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.699+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350000cc10 con 0x7f3514153cc0 2023-10-07T11:03:13.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.699+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3514154de0 con 0x7f3514153cc0 2023-10-07T11:03:13.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.699+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3514155e00 con 0x7f3514153cc0 2023-10-07T11:03:13.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.700+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351404c3c0 con 0x7f3514153cc0 2023-10-07T11:03:13.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.701+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3500013470 con 0x7f3514153cc0 2023-10-07T11:03:13.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.702+0000 7f3510ff9700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34f8066f30 0x7f34f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:13.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.702+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f3500084830 con 0x7f3514153cc0 2023-10-07T11:03:13.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.702+0000 7f3518d9b700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34f8066f30 0x7f34f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:13.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.703+0000 7f3518d9b700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34f8066f30 0x7f34f80693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3508001830 tx=0x7f3508009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:13.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.706+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3500051960 con 0x7f3514153cc0 2023-10-07T11:03:13.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.911+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f351415a520 con 0x7f3514153cc0 2023-10-07T11:03:13.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.915+0000 7f3510ff9700 1 -- 172.21.15.29:0/976236491 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3500054f80 con 0x7f3514153cc0 2023-10-07T11:03:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34f8066f30 msgr2=0x7f34f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34f8066f30 0x7f34f80693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3508001830 tx=0x7f3508009040 comp rx=0 tx=0).stop 2023-10-07T11:03:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 msgr2=0x7f35140c8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 0x7f35140c8730 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f350000e8d0 tx=0x7f35000091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 shutdown_connections 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351415ade0 0x7f35141558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34f8066f30 0x7f34f80693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35140b0540 0x7f35140c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 --2- 172.21.15.29:0/976236491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3514153cc0 0x7f35140c8730 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 >> 172.21.15.29:0/976236491 conn(0x7f3514067610 msgr2=0x7f35140af2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 shutdown_connections 2023-10-07T11:03:13.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:13.918+0000 7f351afff700 1 -- 172.21.15.29:0/976236491 wait complete. 2023-10-07T11:03:13.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T11:03:14.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.133+0000 7fbdbbdff700 1 Processor -- start 2023-10-07T11:03:14.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.133+0000 7fbdbbdff700 1 -- start start 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdbbdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbdb4153cc0 0x7fbdb40b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdbbdff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbdb415ade0 0x7fbdb40b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdbbdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb4154090 con 0x7fbdb40b0000 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdbbdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb4159f60 con 0x7fbdb4153cc0 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdbbdff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb415a240 con 0x7fbdb415ade0 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdb9b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbdb4153cc0 0x7fbdb40b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdb9b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbdb4153cc0 0x7fbdb40b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49020/0 (socket says 172.21.15.29:49020) 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdb9b9b700 1 -- 172.21.15.29:0/4218863653 learned_addr learned my addr 172.21.15.29:0/4218863653 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:14.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdba39c700 1 --1- 172.21.15.29:0/4218863653 >> v1:172.21.15.29:6789/0 conn(0x7fbdb40b0000 0x7fbdb415c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49478/0 (socket says 172.21.15.29:49478) 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.135+0000 7fbdb939a700 1 --2- 172.21.15.29:0/4218863653 >> v2:172.21.15.144:3300/0 conn(0x7fbdb415ade0 0x7fbdb40b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb9b9b700 1 -- 172.21.15.29:0/4218863653 >> v2:172.21.15.144:3300/0 conn(0x7fbdb415ade0 msgr2=0x7fbdb40b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb9b9b700 1 --2- 172.21.15.29:0/4218863653 >> v2:172.21.15.144:3300/0 conn(0x7fbdb415ade0 0x7fbdb40b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb9b9b700 1 -- 172.21.15.29:0/4218863653 >> v1:172.21.15.29:6789/0 conn(0x7fbdb40b0000 legacy=0x7fbdb415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb9b9b700 1 -- 172.21.15.29:0/4218863653 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb415a520 con 0x7fbdb4153cc0 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb939a700 1 --2- 172.21.15.29:0/4218863653 >> v2:172.21.15.144:3300/0 conn(0x7fbdb415ade0 0x7fbdb40b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb9b9b700 1 --2- 172.21.15.29:0/4218863653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb40b17c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40add60 tx=0x7fbdb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698ee6ada35b2b2c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb8b99700 1 -- 172.21.15.29:0/4218863653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb000b860 con 0x7fbdb4153cc0 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb8b99700 1 -- 172.21.15.29:0/4218863653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb000b9c0 con 0x7fbdb4153cc0 2023-10-07T11:03:14.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.136+0000 7fbdb8b99700 1 -- 172.21.15.29:0/4218863653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb000bd60 con 0x7fbdb4153cc0 2023-10-07T11:03:14.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 -- 172.21.15.29:0/4218863653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 msgr2=0x7fbdb40b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/4218863653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb40b17c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40add60 tx=0x7fbdb0009250 comp rx=0 tx=0).stop 2023-10-07T11:03:14.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 -- 172.21.15.29:0/4218863653 shutdown_connections 2023-10-07T11:03:14.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/4218863653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb40b17c0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/4218863653 >> v2:172.21.15.144:3300/0 conn(0x7fbdb415ade0 0x7fbdb40b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 -- 172.21.15.29:0/4218863653 >> 172.21.15.29:0/4218863653 conn(0x7fbdb4067610 msgr2=0x7fbdb4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:14.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 -- 172.21.15.29:0/4218863653 shutdown_connections 2023-10-07T11:03:14.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.137+0000 7fbdbbdff700 1 -- 172.21.15.29:0/4218863653 wait complete. 2023-10-07T11:03:14.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.138+0000 7fbdbbdff700 1 Processor -- start 2023-10-07T11:03:14.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.139+0000 7fbdbbdff700 1 -- start start 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.139+0000 7fbdbbdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb40b0000 0x7fbdb40c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.139+0000 7fbdbbdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb4152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdbbdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb415ade0 0x7fbdb4157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdbbdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4158a30 con 0x7fbdb4153cc0 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdbbdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41694f0 con 0x7fbdb415ade0 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdbbdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4169790 con 0x7fbdb40b0000 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb4152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb4152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49028/0 (socket says 172.21.15.29:49028) 2023-10-07T11:03:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 -- 172.21.15.29:0/3243390338 learned_addr learned my addr 172.21.15.29:0/3243390338 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb9b9b700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb40b0000 0x7fbdb40c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdba39c700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb415ade0 0x7fbdb4157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 -- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb40b0000 msgr2=0x7fbdb40c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb40b0000 0x7fbdb40c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 -- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb415ade0 msgr2=0x7fbdb4157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb415ade0 0x7fbdb4157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb939a700 1 -- 172.21.15.29:0/3243390338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0009490 con 0x7fbdb4153cc0 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdba39c700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb415ade0 0x7fbdb4157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.140+0000 7fbdb9b9b700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb40b0000 0x7fbdb40c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.141+0000 7fbdb939a700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb4152430 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbda400d860 tx=0x7fbda4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.141+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda40125f0 con 0x7fbdb4153cc0 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.141+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda4012750 con 0x7fbdb4153cc0 2023-10-07T11:03:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.141+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4012ad0 con 0x7fbdb4153cc0 2023-10-07T11:03:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.141+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb4157ec0 con 0x7fbdb4153cc0 2023-10-07T11:03:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.141+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb4158470 con 0x7fbdb4153cc0 2023-10-07T11:03:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.142+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb4169900 con 0x7fbdb4153cc0 2023-10-07T11:03:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.143+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbda4014430 con 0x7fbdb4153cc0 2023-10-07T11:03:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.144+0000 7fbda9ffb700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbd98066f30 0x7fbd980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.144+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fbda4084ad0 con 0x7fbdb4153cc0 2023-10-07T11:03:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.144+0000 7fbdb9b9b700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbd98066f30 0x7fbd980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.145+0000 7fbdb9b9b700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbd98066f30 0x7fbd980693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40add60 tx=0x7fbdb0017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.148+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbda4051b60 con 0x7fbdb4153cc0 2023-10-07T11:03:14.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:03:14.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:03:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.351+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fbdb404c3c0 con 0x7fbdb4153cc0 2023-10-07T11:03:14.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.355+0000 7fbda9ffb700 1 -- 172.21.15.29:0/3243390338 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7fbda4055180 con 0x7fbdb4153cc0 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbd98066f30 msgr2=0x7fbd980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbd98066f30 0x7fbd980693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40add60 tx=0x7fbdb0017000 comp rx=0 tx=0).stop 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 msgr2=0x7fbdb4152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb4152430 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbda400d860 tx=0x7fbda4008040 comp rx=0 tx=0).stop 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 shutdown_connections 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb415ade0 0x7fbdb4157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbd98066f30 0x7fbd980693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb40b0000 0x7fbdb40c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 --2- 172.21.15.29:0/3243390338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdb4153cc0 0x7fbdb4152430 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 >> 172.21.15.29:0/3243390338 conn(0x7fbdb4067610 msgr2=0x7fbdb40b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 shutdown_connections 2023-10-07T11:03:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.359+0000 7fbdbbdff700 1 -- 172.21.15.29:0/3243390338 wait complete. 2023-10-07T11:03:14.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-07T11:03:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.615+0000 7f79239ff700 1 Processor -- start 2023-10-07T11:03:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.616+0000 7f79239ff700 1 -- start start 2023-10-07T11:03:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.617+0000 7f79239ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 0x7f791c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.617+0000 7f79239ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f791c159ec0 0x7f791c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.617+0000 7f79239ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c14d790 con 0x7f791c1568c0 2023-10-07T11:03:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.617+0000 7f79239ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791c0ae5d0 con 0x7f791c159ec0 2023-10-07T11:03:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.617+0000 7f79239ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791c15a850 con 0x7f791c0ad370 2023-10-07T11:03:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 0x7f791c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f7920f9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f791c159ec0 0x7f791c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 0x7f791c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:44996/0 (socket says 172.21.15.29:44996) 2023-10-07T11:03:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 -- 172.21.15.29:0/1503413827 learned_addr learned my addr 172.21.15.29:0/1503413827 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f7921f9c700 1 --1- 172.21.15.29:0/1503413827 >> v1:172.21.15.144:6789/0 conn(0x7f791c1568c0 0x7f791c156d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:56370/0 (socket says 172.21.15.29:56370) 2023-10-07T11:03:14.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 -- 172.21.15.29:0/1503413827 >> v2:172.21.15.144:3300/0 conn(0x7f791c159ec0 msgr2=0x7f791c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 --2- 172.21.15.29:0/1503413827 >> v2:172.21.15.144:3300/0 conn(0x7f791c159ec0 0x7f791c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 -- 172.21.15.29:0/1503413827 >> v1:172.21.15.144:6789/0 conn(0x7f791c1568c0 legacy=0x7f791c156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 -- 172.21.15.29:0/1503413827 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c0ad740 con 0x7f791c0ad370 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.618+0000 7f792179b700 1 --2- 172.21.15.29:0/1503413827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 0x7f791c0cc870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f79180099b0 tx=0x7f7918009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97eec81b1b6ed53 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f7913fff700 1 -- 172.21.15.29:0/1503413827 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918009ea0 con 0x7f791c0ad370 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f7913fff700 1 -- 172.21.15.29:0/1503413827 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791800b860 con 0x7f791c0ad370 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f7913fff700 1 -- 172.21.15.29:0/1503413827 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800bc00 con 0x7f791c0ad370 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 -- 172.21.15.29:0/1503413827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 msgr2=0x7f791c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 --2- 172.21.15.29:0/1503413827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 0x7f791c0cc870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f79180099b0 tx=0x7f7918009250 comp rx=0 tx=0).stop 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 -- 172.21.15.29:0/1503413827 shutdown_connections 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 --2- 172.21.15.29:0/1503413827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c0ad370 0x7f791c0cc870 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 --2- 172.21.15.29:0/1503413827 >> v2:172.21.15.144:3300/0 conn(0x7f791c159ec0 0x7f791c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 -- 172.21.15.29:0/1503413827 >> 172.21.15.29:0/1503413827 conn(0x7f791c0680a0 msgr2=0x7f791c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 -- 172.21.15.29:0/1503413827 shutdown_connections 2023-10-07T11:03:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.619+0000 7f79239ff700 1 -- 172.21.15.29:0/1503413827 wait complete. 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.621+0000 7f79239ff700 1 Processor -- start 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.621+0000 7f79239ff700 1 -- start start 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f79239ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 0x7f791c158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f79239ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c1568c0 0x7f791c1455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f79239ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f791c159ec0 0x7f791c1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f79239ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c150d40 con 0x7f791c0ad370 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f79239ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c150780 con 0x7f791c159ec0 2023-10-07T11:03:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f79239ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791c150a60 con 0x7f791c1568c0 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 0x7f791c158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 0x7f791c158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49042/0 (socket says 172.21.15.29:49042) 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 -- 172.21.15.29:0/1736719660 learned_addr learned my addr 172.21.15.29:0/1736719660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f7921f9c700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f791c159ec0 0x7f791c1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f7920f9a700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c1568c0 0x7f791c1455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 -- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c1568c0 msgr2=0x7f791c1455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c1568c0 0x7f791c1455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 -- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f791c159ec0 msgr2=0x7f791c1431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f791c159ec0 0x7f791c1431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.622+0000 7f792179b700 1 -- 172.21.15.29:0/1736719660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180094f0 con 0x7f791c0ad370 2023-10-07T11:03:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.623+0000 7f792179b700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 0x7f791c158db0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f791c15a990 tx=0x7f7918017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.623+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800bbe0 con 0x7f791c0ad370 2023-10-07T11:03:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.623+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7918011420 con 0x7f791c0ad370 2023-10-07T11:03:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.623+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c1429c0 con 0x7f791c0ad370 2023-10-07T11:03:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.623+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180117c0 con 0x7f791c0ad370 2023-10-07T11:03:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.623+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c1592f0 con 0x7f791c0ad370 2023-10-07T11:03:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.624+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791c142800 con 0x7f791c0ad370 2023-10-07T11:03:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.625+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7918011bf0 con 0x7f791c0ad370 2023-10-07T11:03:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.626+0000 7f79117fa700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7900066f30 0x7f79000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.626+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f7918085cb0 con 0x7f791c0ad370 2023-10-07T11:03:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.626+0000 7f7920f9a700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7900066f30 0x7f79000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.627+0000 7f7920f9a700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7900066f30 0x7f79000693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f791c145030 tx=0x7f790c008040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:14.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.630+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7918052dc0 con 0x7f791c0ad370 2023-10-07T11:03:14.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:14.914+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f791c0ace70 con 0x7f791c0ad370 2023-10-07T11:03:15.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.007+0000 7f79117fa700 1 -- 172.21.15.29:0/1736719660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3032) v1 ==== 163+0+0 (secure 0 0 0) 0x7f79180563e0 con 0x7f791c0ad370 2023-10-07T11:03:15.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7900066f30 msgr2=0x7f79000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7900066f30 0x7f79000693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f791c145030 tx=0x7f790c008040 comp rx=0 tx=0).stop 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 msgr2=0x7f791c158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 0x7f791c158db0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f791c15a990 tx=0x7f7918017000 comp rx=0 tx=0).stop 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 shutdown_connections 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f791c159ec0 0x7f791c1431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791c0ad370 0x7f791c158db0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7900066f30 0x7f79000693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 --2- 172.21.15.29:0/1736719660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f791c1568c0 0x7f791c1455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 >> 172.21.15.29:0/1736719660 conn(0x7f791c0680a0 msgr2=0x7f791c1579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:15.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 shutdown_connections 2023-10-07T11:03:15.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.010+0000 7f79239ff700 1 -- 172.21.15.29:0/1736719660 wait complete. 2023-10-07T11:03:15.028 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-07T11:03:15.028 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-07T11:03:15.028 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-07T11:03:15.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-07T11:03:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.223+0000 7fca8b3ff700 1 Processor -- start 2023-10-07T11:03:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.224+0000 7fca8b3ff700 1 -- start start 2023-10-07T11:03:15.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84069c80 0x7fca84159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8b3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca840c8620 0x7fca8415da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8b3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca8415dfd0 0x7fca8406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8b3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8415d600 con 0x7fca840c8620 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8b3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8415a550 con 0x7fca8415dfd0 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8415a830 con 0x7fca84069c80 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.225+0000 7fca8899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca840c8620 0x7fca8415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca840c8620 0x7fca8415da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49044/0 (socket says 172.21.15.29:49044) 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 -- 172.21.15.29:0/2575994778 learned_addr learned my addr 172.21.15.29:0/2575994778 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8919b700 1 --2- 172.21.15.29:0/2575994778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84069c80 0x7fca84159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8999c700 1 --2- 172.21.15.29:0/2575994778 >> v2:172.21.15.144:3300/0 conn(0x7fca8415dfd0 0x7fca8406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 -- 172.21.15.29:0/2575994778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84069c80 msgr2=0x7fca84159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 --2- 172.21.15.29:0/2575994778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84069c80 0x7fca84159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 -- 172.21.15.29:0/2575994778 >> v2:172.21.15.144:3300/0 conn(0x7fca8415dfd0 msgr2=0x7fca8406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 --2- 172.21.15.29:0/2575994778 >> v2:172.21.15.144:3300/0 conn(0x7fca8415dfd0 0x7fca8406d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 -- 172.21.15.29:0/2575994778 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8415ab10 con 0x7fca840c8620 2023-10-07T11:03:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca8899a700 1 --2- 172.21.15.29:0/2575994778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca840c8620 0x7fca8415da90 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fca84157860 tx=0x7fca74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27664195371edfc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.226+0000 7fca7bfff700 1 -- 172.21.15.29:0/2575994778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400b640 con 0x7fca840c8620 2023-10-07T11:03:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca7bfff700 1 -- 172.21.15.29:0/2575994778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7400b7a0 con 0x7fca840c8620 2023-10-07T11:03:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca7bfff700 1 -- 172.21.15.29:0/2575994778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400bb60 con 0x7fca840c8620 2023-10-07T11:03:15.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 -- 172.21.15.29:0/2575994778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca840c8620 msgr2=0x7fca8415da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:15.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/2575994778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca840c8620 0x7fca8415da90 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fca84157860 tx=0x7fca74009250 comp rx=0 tx=0).stop 2023-10-07T11:03:15.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 -- 172.21.15.29:0/2575994778 shutdown_connections 2023-10-07T11:03:15.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/2575994778 >> v2:172.21.15.144:3300/0 conn(0x7fca8415dfd0 0x7fca8406d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/2575994778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84069c80 0x7fca84159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/2575994778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca840c8620 0x7fca8415da90 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 -- 172.21.15.29:0/2575994778 >> 172.21.15.29:0/2575994778 conn(0x7fca840cc560 msgr2=0x7fca840cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 -- 172.21.15.29:0/2575994778 shutdown_connections 2023-10-07T11:03:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.227+0000 7fca8b3ff700 1 -- 172.21.15.29:0/2575994778 wait complete. 2023-10-07T11:03:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.229+0000 7fca8b3ff700 1 Processor -- start 2023-10-07T11:03:15.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.229+0000 7fca8b3ff700 1 -- start start 2023-10-07T11:03:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8b3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca84069c80 0x7fca840c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8b3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca840c8620 0x7fca84152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8b3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 0x7fca84158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8b3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca840b0650 con 0x7fca8415dfd0 2023-10-07T11:03:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8b3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca840b0d40 con 0x7fca84069c80 2023-10-07T11:03:15.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8b3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca840c3ab0 con 0x7fca840c8620 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 0x7fca84158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 0x7fca84158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49052/0 (socket says 172.21.15.29:49052) 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 -- 172.21.15.29:0/1915028320 learned_addr learned my addr 172.21.15.29:0/1915028320 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8919b700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca84069c80 0x7fca840c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8899a700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca840c8620 0x7fca84152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 -- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca840c8620 msgr2=0x7fca84152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca840c8620 0x7fca84152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 -- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca84069c80 msgr2=0x7fca840c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca84069c80 0x7fca840c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.230+0000 7fca8999c700 1 -- 172.21.15.29:0/1915028320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740094f0 con 0x7fca8415dfd0 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.231+0000 7fca8999c700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 0x7fca84158860 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009df0 tx=0x7fca7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:15.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.231+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00caa0 con 0x7fca8415dfd0 2023-10-07T11:03:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.231+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c00f040 con 0x7fca8415dfd0 2023-10-07T11:03:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.231+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca840b0a50 con 0x7fca8415dfd0 2023-10-07T11:03:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.231+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00ce70 con 0x7fca8415dfd0 2023-10-07T11:03:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.231+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca840c66d0 con 0x7fca8415dfd0 2023-10-07T11:03:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.232+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca840c6170 con 0x7fca8415dfd0 2023-10-07T11:03:15.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.233+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fca7c014690 con 0x7fca8415dfd0 2023-10-07T11:03:15.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.234+0000 7fca797fa700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca68066f30 0x7fca680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:15.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.234+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fca7c084f20 con 0x7fca8415dfd0 2023-10-07T11:03:15.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.235+0000 7fca8919b700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca68066f30 0x7fca680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:15.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.235+0000 7fca8919b700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca68066f30 0x7fca680693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca80009db0 tx=0x7fca800091a0 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:15.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.238+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca7c055a40 con 0x7fca8415dfd0 2023-10-07T11:03:15.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:15.523+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fca840b0eb0 con 0x7fca8415dfd0 2023-10-07T11:03:16.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.009+0000 7fca797fa700 1 -- 172.21.15.29:0/1915028320 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3033) v1 ==== 158+0+0 (secure 0 0 0) 0x7fca7c0555d0 con 0x7fca8415dfd0 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca68066f30 msgr2=0x7fca680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca68066f30 0x7fca680693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca80009db0 tx=0x7fca800091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 msgr2=0x7fca84158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 0x7fca84158860 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009df0 tx=0x7fca7c00b040 comp rx=0 tx=0).stop 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 shutdown_connections 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca8415dfd0 0x7fca84158860 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca84069c80 0x7fca840c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca68066f30 0x7fca680693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 --2- 172.21.15.29:0/1915028320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca840c8620 0x7fca84152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 >> 172.21.15.29:0/1915028320 conn(0x7fca840cc560 msgr2=0x7fca8406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 shutdown_connections 2023-10-07T11:03:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.012+0000 7fca8b3ff700 1 -- 172.21.15.29:0/1915028320 wait complete. 2023-10-07T11:03:16.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-07T11:03:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.231+0000 7fa9e33ff700 1 Processor -- start 2023-10-07T11:03:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.231+0000 7fa9e33ff700 1 -- start start 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.232+0000 7fa9e33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.232+0000 7fa9e33ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9dc15ade0 0x7fa9dc0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.232+0000 7fa9e33ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc154090 con 0x7fa9dc0b0000 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.232+0000 7fa9e33ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc159f60 con 0x7fa9dc15ade0 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.232+0000 7fa9e33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc15a240 con 0x7fa9dc153cc0 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9dc15ade0 0x7fa9dc0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9dc15ade0 0x7fa9dc0b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49068/0 (socket says 172.21.15.29:49068) 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 -- 172.21.15.29:0/3392045603 learned_addr learned my addr 172.21.15.29:0/3392045603 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e119b700 1 --2- 172.21.15.29:0/3392045603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 -- 172.21.15.29:0/3392045603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 msgr2=0x7fa9dc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:16.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 --2- 172.21.15.29:0/3392045603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 -- 172.21.15.29:0/3392045603 >> v1:172.21.15.29:6789/0 conn(0x7fa9dc0b0000 legacy=0x7fa9dc15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.233+0000 7fa9e099a700 1 -- 172.21.15.29:0/3392045603 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc15a520 con 0x7fa9dc15ade0 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e099a700 1 --2- 172.21.15.29:0/3392045603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc0b1d00 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0099b0 tx=0x7fa9cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cd4264ac504e31 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9d3fff700 1 -- 172.21.15.29:0/3392045603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00b860 con 0x7fa9dc15ade0 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9d3fff700 1 -- 172.21.15.29:0/3392045603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc00b9c0 con 0x7fa9dc15ade0 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9d3fff700 1 -- 172.21.15.29:0/3392045603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00bd60 con 0x7fa9dc15ade0 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3392045603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc15ade0 msgr2=0x7fa9dc0b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3392045603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc0b1d00 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0099b0 tx=0x7fa9cc009250 comp rx=0 tx=0).stop 2023-10-07T11:03:16.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3392045603 shutdown_connections 2023-10-07T11:03:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3392045603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3392045603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc0b1d00 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.234+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3392045603 >> 172.21.15.29:0/3392045603 conn(0x7fa9dc067610 msgr2=0x7fa9dc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.235+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3392045603 shutdown_connections 2023-10-07T11:03:16.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.235+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3392045603 wait complete. 2023-10-07T11:03:16.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.236+0000 7fa9e33ff700 1 Processor -- start 2023-10-07T11:03:16.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.236+0000 7fa9e33ff700 1 -- start start 2023-10-07T11:03:16.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e33ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 0x7fa9dc156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e33ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e33ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc169770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e33ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc156d60 con 0x7fa9dc0b0000 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e33ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc157040 con 0x7fa9dc15ade0 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e33ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc169ff0 con 0x7fa9dc153cc0 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e119b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 0x7fa9dc156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e119b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 0x7fa9dc156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49072/0 (socket says 172.21.15.29:49072) 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.237+0000 7fa9e119b700 1 -- 172.21.15.29:0/3624516246 learned_addr learned my addr 172.21.15.29:0/3624516246 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e099a700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e199c700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e119b700 1 -- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 msgr2=0x7fa9dc0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e119b700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e119b700 1 -- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9dc15ade0 msgr2=0x7fa9dc169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e119b700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc169770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e119b700 1 -- 172.21.15.29:0/3624516246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc0094f0 con 0x7fa9dc0b0000 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e099a700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e199c700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e119b700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 0x7fa9dc156670 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800d840 tx=0x7fa9d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800b600 con 0x7fa9dc0b0000 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d800b760 con 0x7fa9dc0b0000 2023-10-07T11:03:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc16a2d0 con 0x7fa9dc0b0000 2023-10-07T11:03:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800bb20 con 0x7fa9dc0b0000 2023-10-07T11:03:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.238+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc169cb0 con 0x7fa9dc0b0000 2023-10-07T11:03:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.240+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc04c3c0 con 0x7fa9dc0b0000 2023-10-07T11:03:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.240+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa9d801e460 con 0x7fa9dc0b0000 2023-10-07T11:03:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.241+0000 7fa9d17fa700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa9c0066f30 0x7fa9c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.241+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fa9d8084d30 con 0x7fa9dc0b0000 2023-10-07T11:03:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.242+0000 7fa9e099a700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa9c0066f30 0x7fa9c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.242+0000 7fa9e099a700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa9c0066f30 0x7fa9c00693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0099b0 tx=0x7fa9cc017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.245+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9d8051e60 con 0x7fa9dc0b0000 2023-10-07T11:03:16.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.449+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa9dc157320 con 0x7fa9dc0b0000 2023-10-07T11:03:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.454+0000 7fa9d17fa700 1 -- 172.21.15.29:0/3624516246 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa9d8055480 con 0x7fa9dc0b0000 2023-10-07T11:03:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa9c0066f30 msgr2=0x7fa9c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa9c0066f30 0x7fa9c00693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0099b0 tx=0x7fa9cc017000 comp rx=0 tx=0).stop 2023-10-07T11:03:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 msgr2=0x7fa9dc156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 0x7fa9dc156670 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800d840 tx=0x7fa9d8008040 comp rx=0 tx=0).stop 2023-10-07T11:03:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 shutdown_connections 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9dc15ade0 0x7fa9dc169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9dc0b0000 0x7fa9dc156670 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa9c0066f30 0x7fa9c00693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 --2- 172.21.15.29:0/3624516246 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9dc153cc0 0x7fa9dc0ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 >> 172.21.15.29:0/3624516246 conn(0x7fa9dc067610 msgr2=0x7fa9dc15b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 shutdown_connections 2023-10-07T11:03:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.456+0000 7fa9e33ff700 1 -- 172.21.15.29:0/3624516246 wait complete. 2023-10-07T11:03:16.475 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T11:03:16.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:03:16.580 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:03:16.690 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T11:03:16.691 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-07T11:03:16.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:16.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.886+0000 7f3de8bff700 1 Processor -- start 2023-10-07T11:03:16.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.887+0000 7f3de8bff700 1 -- start start 2023-10-07T11:03:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.888+0000 7f3de8bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.888+0000 7f3de8bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de414d790 con 0x7f3de4159ec0 2023-10-07T11:03:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.888+0000 7f3de8bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40ae5d0 con 0x7f3de40b0410 2023-10-07T11:03:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.888+0000 7f3de8bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de40ad740 con 0x7f3de40ad370 2023-10-07T11:03:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.888+0000 7f3de1d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3de4159ec0 0x7f3de41568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49488/0 (socket says 172.21.15.29:49488) 2023-10-07T11:03:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.888+0000 7f3de1d9b700 1 -- 172.21.15.29:0/268751689 learned_addr learned my addr 172.21.15.29:0/268751689 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de259c700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026538818 0 0) 0x7f3de414d790 con 0x7f3de4159ec0 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dcc003570 con 0x7f3de4159ec0 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4144216401 0 0) 0x7f3dcc003570 con 0x7f3de4159ec0 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 msgr2=0x7f3de40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 0x7f3de40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 >> v1:172.21.15.144:6789/0 conn(0x7f3de40b0410 legacy=0x7f3de4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40ccdb0 con 0x7f3de4159ec0 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f3dd8001470 con 0x7f3de4159ec0 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 >> v1:172.21.15.29:6789/0 conn(0x7f3de4159ec0 legacy=0x7f3de41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dcc003a90 0x7f3dcc003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dcc004810 0x7f3dcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 0x7f3dcc0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de40ad740 con 0x7f3dcc004810 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc00bb60 con 0x7f3dcc003a90 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc00bd20 con 0x7f3de40ad370 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.889+0000 7f3de1d9b700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dcc004810 0x7f3dcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de259c700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dcc003a90 0x7f3dcc003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de2d9d700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 0x7f3dcc0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de1d9b700 1 -- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 msgr2=0x7f3dcc0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de1d9b700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de40ad370 0x7f3dcc0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de1d9b700 1 -- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dcc003a90 msgr2=0x7f3dcc003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de1d9b700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dcc003a90 0x7f3dcc003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de1d9b700 1 -- 172.21.15.29:0/268751689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc003570 con 0x7f3dcc004810 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de1d9b700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dcc004810 0x7f3dcc004c80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800c3c0 tx=0x7f3dd8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd0c861c8e4d716 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd800cc90 con 0x7f3dcc004810 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd800cdf0 con 0x7f3dcc004810 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.890+0000 7f3de159a700 1 -- 172.21.15.29:0/268751689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8005b50 con 0x7f3dcc004810 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de8bff700 1 -- 172.21.15.29:0/268751689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dcc004810 msgr2=0x7f3dcc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de8bff700 1 --2- 172.21.15.29:0/268751689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dcc004810 0x7f3dcc004c80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800c3c0 tx=0x7f3dd8001e50 comp rx=0 tx=0).stop 2023-10-07T11:03:16.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de2d9d700 1 -- 172.21.15.29:0/268751689 reap_dead start 2023-10-07T11:03:16.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de8bff700 1 -- 172.21.15.29:0/268751689 shutdown_connections 2023-10-07T11:03:16.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de8bff700 1 -- 172.21.15.29:0/268751689 >> 172.21.15.29:0/268751689 conn(0x7f3de40680a0 msgr2=0x7f3de40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:16.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de8bff700 1 -- 172.21.15.29:0/268751689 shutdown_connections 2023-10-07T11:03:16.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.891+0000 7f3de8bff700 1 -- 172.21.15.29:0/268751689 wait complete. 2023-10-07T11:03:16.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.893+0000 7f3de8bff700 1 Processor -- start 2023-10-07T11:03:16.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.893+0000 7f3de8bff700 1 -- start start 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de8bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dcc004810 0x7f3de415b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de8bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de4141ab0 0x7f3de40cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de8bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 0x7f3de415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de8bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de415afd0 con 0x7f3de4141f20 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de8bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de415a3c0 con 0x7f3de4141ab0 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de8bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de415a6a0 con 0x7f3dcc004810 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 0x7f3de415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de259c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dcc004810 0x7f3de415b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 0x7f3de415c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49096/0 (socket says 172.21.15.29:49096) 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de1d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de4141ab0 0x7f3de40cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 -- 172.21.15.29:0/2198319826 learned_addr learned my addr 172.21.15.29:0/2198319826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 -- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dcc004810 msgr2=0x7f3de415b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dcc004810 0x7f3de415b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 -- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de4141ab0 msgr2=0x7f3de40cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de4141ab0 0x7f3de40cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.894+0000 7f3de2d9d700 1 -- 172.21.15.29:0/2198319826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de415a980 con 0x7f3de4141f20 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.895+0000 7f3de2d9d700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 0x7f3de415c270 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f3de4069100 tx=0x7f3ddc001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.895+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc018070 con 0x7f3de4141f20 2023-10-07T11:03:16.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.895+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ddc00ee70 con 0x7f3de4141f20 2023-10-07T11:03:16.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.895+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc006dd0 con 0x7f3de4141f20 2023-10-07T11:03:16.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.895+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de415ac60 con 0x7f3de4141f20 2023-10-07T11:03:16.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.895+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de40cbba0 con 0x7f3de4141f20 2023-10-07T11:03:16.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.896+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de415b180 con 0x7f3de4141f20 2023-10-07T11:03:16.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.897+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3ddc00c9c0 con 0x7f3de4141f20 2023-10-07T11:03:16.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.898+0000 7f3dd27fc700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3dc4066f30 0x7f3dc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:16.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.898+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f3ddc084300 con 0x7f3de4141f20 2023-10-07T11:03:16.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.898+0000 7f3de259c700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3dc4066f30 0x7f3dc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:16.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.899+0000 7f3de259c700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3dc4066f30 0x7f3dc40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4006f00 tx=0x7f3dd40091a0 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:16.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:16.902+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3de415b180 con 0x7f3de4141f20 2023-10-07T11:03:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.153+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3de404c3c0 con 0x7f3de4141f20 2023-10-07T11:03:17.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.153+0000 7f3dd27fc700 1 -- 172.21.15.29:0/2198319826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f3ddc051430 con 0x7f3de4141f20 2023-10-07T11:03:17.154 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:17.155 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8492,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/4049881465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":4049881465},{"type":"v1","addr":"172.21.15.144:6834","nonce":4049881465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":9145,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3324383150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3324383150},{"type":"v1","addr":"172.21.15.144:6835","nonce":3324383150}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/706127589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":706127589},{"type":"v1","addr":"172.21.15.29:6834","nonce":706127589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9630,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/606698488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":606698488},{"type":"v1","addr":"172.21.15.29:6835","nonce":606698488}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[]} 2023-10-07T11:03:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3dc4066f30 msgr2=0x7f3dc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3dc4066f30 0x7f3dc40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4006f00 tx=0x7f3dd40091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 msgr2=0x7f3de415c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 0x7f3de415c270 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f3de4069100 tx=0x7f3ddc001670 comp rx=0 tx=0).stop 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 shutdown_connections 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de4141f20 0x7f3de415c270 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3dc4066f30 0x7f3dc40693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dcc004810 0x7f3de415b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 --2- 172.21.15.29:0/2198319826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de4141ab0 0x7f3de40cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 >> 172.21.15.29:0/2198319826 conn(0x7f3de40680a0 msgr2=0x7f3de414fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 shutdown_connections 2023-10-07T11:03:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.156+0000 7f3de8bff700 1 -- 172.21.15.29:0/2198319826 wait complete. 2023-10-07T11:03:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 96 2023-10-07T11:03:17.174 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-07T11:03:17.174 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-07T11:03:17.175 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T11:03:17.175 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:03:17.175 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-07T11:03:17.175 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-07T11:03:17.175 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T11:03:17.176 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:03:17.176 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-07T11:03:17.176 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-07T11:03:17.176 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T11:03:17.176 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:03:17.176 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-07T11:03:17.177 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-07T11:03:17.177 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T11:03:17.177 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:03:17.187 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T11:03:17.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-07T11:03:17.189 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T11:03:17.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-07T11:03:17.195 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T11:03:17.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-07T11:03:17.195 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T11:03:17.196 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-07T11:03:17.232 INFO:tasks.ceph.mds.a:Started 2023-10-07T11:03:17.233 DEBUG:teuthology.parallel:result is None 2023-10-07T11:03:17.235 INFO:tasks.ceph.mds.c:Started 2023-10-07T11:03:17.235 DEBUG:teuthology.parallel:result is None 2023-10-07T11:03:17.240 INFO:tasks.ceph.mds.d:Started 2023-10-07T11:03:17.240 DEBUG:teuthology.parallel:result is None 2023-10-07T11:03:17.242 INFO:tasks.ceph.mds.b:Started 2023-10-07T11:03:17.243 DEBUG:teuthology.parallel:result is None 2023-10-07T11:03:17.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-07T11:03:17.382 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-07T11:03:17.383 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T11:03:17.387 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-07T11:03:17.387 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-07T11:03:17.389 INFO:tasks.ceph.mds.d.smithi144.stderr:2023-10-07T11:03:17.388+0000 7fa20dbfd700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-07T11:03:17.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.450+0000 7f622cdff700 1 Processor -- start 2023-10-07T11:03:17.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.450+0000 7f622cdff700 1 -- start start 2023-10-07T11:03:17.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.451+0000 7f622cdff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62280ad220 0x7f6228159c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:17.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.451+0000 7f622cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228068a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:17.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.451+0000 7f622cdff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622815af00 con 0x7f62280ad5f0 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.451+0000 7f622cdff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6228068fa0 con 0x7f62280ad220 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.451+0000 7f622cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62280ada40 con 0x7f6228068620 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.451+0000 7f622659c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62280ad220 0x7f6228159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f6225d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f62280ad5f0 0x7f62280ac490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49512/0 (socket says 172.21.15.29:49512) 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f6225d9b700 1 -- 172.21.15.29:0/1715153932 learned_addr learned my addr 172.21.15.29:0/1715153932 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622659c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62280ad220 0x7f6228159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49128/0 (socket says 172.21.15.29:49128) 2023-10-07T11:03:17.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f6226d9d700 1 --2- 172.21.15.29:0/1715153932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622659c700 1 -- 172.21.15.29:0/1715153932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 msgr2=0x7f6228068a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622659c700 1 --2- 172.21.15.29:0/1715153932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228068a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622659c700 1 -- 172.21.15.29:0/1715153932 >> v1:172.21.15.29:6789/0 conn(0x7f62280ad5f0 legacy=0x7f62280ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622659c700 1 -- 172.21.15.29:0/1715153932 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622815a250 con 0x7f62280ad220 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622559a700 1 -- 172.21.15.29:0/1715153932 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587916288 0 0) 0x7f622815af00 con 0x7f62280ad5f0 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.452+0000 7f622659c700 1 --2- 172.21.15.29:0/1715153932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad220 0x7f6228159c00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f62180098e0 tx=0x7f62180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6806c28f8b957d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.453+0000 7f622559a700 1 -- 172.21.15.29:0/1715153932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621800b640 con 0x7f62280ad220 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.453+0000 7f622559a700 1 -- 172.21.15.29:0/1715153932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621800b7a0 con 0x7f62280ad220 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.453+0000 7f622559a700 1 -- 172.21.15.29:0/1715153932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621800bb40 con 0x7f62280ad220 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.453+0000 7f622cdff700 1 -- 172.21.15.29:0/1715153932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad220 msgr2=0x7f6228159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.453+0000 7f622cdff700 1 --2- 172.21.15.29:0/1715153932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad220 0x7f6228159c00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f62180098e0 tx=0x7f62180091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.454+0000 7f622cdff700 1 -- 172.21.15.29:0/1715153932 shutdown_connections 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.454+0000 7f622cdff700 1 --2- 172.21.15.29:0/1715153932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228068a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.454+0000 7f622cdff700 1 --2- 172.21.15.29:0/1715153932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad220 0x7f6228159c00 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.454+0000 7f622cdff700 1 -- 172.21.15.29:0/1715153932 >> 172.21.15.29:0/1715153932 conn(0x7f622806dd00 msgr2=0x7f622806e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.454+0000 7f622cdff700 1 -- 172.21.15.29:0/1715153932 shutdown_connections 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.454+0000 7f622cdff700 1 -- 172.21.15.29:0/1715153932 wait complete. 2023-10-07T11:03:17.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.456+0000 7f622cdff700 1 Processor -- start 2023-10-07T11:03:17.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.456+0000 7f622cdff700 1 -- start start 2023-10-07T11:03:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.457+0000 7f622cdff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.457+0000 7f622cdff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 0x7f62281516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622cdff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad5f0 0x7f62280a9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622cdff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6228158d00 con 0x7f62280ad5f0 2023-10-07T11:03:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622cdff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6228155310 con 0x7f62280ad220 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622cdff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6228154700 con 0x7f6228068620 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f6225d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 0x7f62281516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f6225d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 0x7f62281516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:58400/0 (socket says 172.21.15.29:58400) 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:45106/0 (socket says 172.21.15.29:45106) 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f6225d9b700 1 -- 172.21.15.29:0/3918005759 learned_addr learned my addr 172.21.15.29:0/3918005759 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f6226d9d700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad5f0 0x7f62280a9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 -- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 msgr2=0x7f62281516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 0x7f62281516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 -- 172.21.15.29:0/3918005759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad5f0 msgr2=0x7f62280a9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad5f0 0x7f62280a9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f622659c700 1 -- 172.21.15.29:0/3918005759 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6218009420 con 0x7f6228068620 2023-10-07T11:03:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.458+0000 7f6225d9b700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 0x7f62281516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.459+0000 7f622659c700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228156e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6218016000 tx=0x7f6218017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.466+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621800bb20 con 0x7f6228068620 2023-10-07T11:03:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.466+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6218000c90 con 0x7f6228068620 2023-10-07T11:03:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.466+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6218011630 con 0x7f6228068620 2023-10-07T11:03:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.466+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62281549e0 con 0x7f6228068620 2023-10-07T11:03:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.466+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6228155020 con 0x7f6228068620 2023-10-07T11:03:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.467+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622804c3c0 con 0x7f6228068620 2023-10-07T11:03:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.469+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f62180119a0 con 0x7f6228068620 2023-10-07T11:03:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.470+0000 7f62167fc700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6208067000 0x7f62080694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.470+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f62180858b0 con 0x7f6228068620 2023-10-07T11:03:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.470+0000 7f6225d9b700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6208067000 0x7f62080694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.471+0000 7f6225d9b700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6208067000 0x7f62080694b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f62280cbd10 tx=0x7f621c007df0 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.479+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62180529c0 con 0x7f6228068620 2023-10-07T11:03:17.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:17.697+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f622815a530 con 0x7f6228068620 2023-10-07T11:03:18.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.479+0000 7f62167fc700 1 -- 172.21.15.29:0/3918005759 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v193) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6218055fe0 con 0x7f6228068620 2023-10-07T11:03:18.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6208067000 msgr2=0x7f62080694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6208067000 0x7f62080694b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f62280cbd10 tx=0x7f621c007df0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 msgr2=0x7f6228156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228156e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6218016000 tx=0x7f6218017000 comp rx=0 tx=0).stop 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 shutdown_connections 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62280ad5f0 0x7f62280a9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6208067000 0x7f62080694b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6228068620 0x7f6228156e40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 --2- 172.21.15.29:0/3918005759 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62280ad220 0x7f62281516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 >> 172.21.15.29:0/3918005759 conn(0x7f622806dd00 msgr2=0x7f62280695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 shutdown_connections 2023-10-07T11:03:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.482+0000 7f622cdff700 1 -- 172.21.15.29:0/3918005759 wait complete. 2023-10-07T11:03:18.485 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-07T11:03:18.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-07T11:03:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.701+0000 7f84c7dff700 1 Processor -- start 2023-10-07T11:03:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.702+0000 7f84c7dff700 1 -- start start 2023-10-07T11:03:18.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c7dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84c00ae8b0 0x7f84c00b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c7dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f84c00b1840 0x7f84c0157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c7dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c0158ad0 con 0x7f84c0157290 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c7dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c00b1cc0 con 0x7f84c00ae8b0 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c7dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c015b140 con 0x7f84c00b1840 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c539a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84c00ae8b0 0x7f84c00b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c539a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84c00ae8b0 0x7f84c00b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49144/0 (socket says 172.21.15.29:49144) 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c539a700 1 -- 172.21.15.29:0/2746020349 learned_addr learned my addr 172.21.15.29:0/2746020349 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c5b9b700 1 --1- 172.21.15.29:0/2746020349 >> v1:172.21.15.29:6789/0 conn(0x7f84c0157290 0x7f84c00cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49518/0 (socket says 172.21.15.29:49518) 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.703+0000 7f84c639c700 1 --2- 172.21.15.29:0/2746020349 >> v2:172.21.15.144:3300/0 conn(0x7f84c00b1840 0x7f84c0157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c539a700 1 -- 172.21.15.29:0/2746020349 >> v2:172.21.15.144:3300/0 conn(0x7f84c00b1840 msgr2=0x7f84c0157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:18.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c539a700 1 --2- 172.21.15.29:0/2746020349 >> v2:172.21.15.144:3300/0 conn(0x7f84c00b1840 0x7f84c0157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c539a700 1 -- 172.21.15.29:0/2746020349 >> v1:172.21.15.29:6789/0 conn(0x7f84c0157290 legacy=0x7f84c00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:18.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c539a700 1 -- 172.21.15.29:0/2746020349 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c015b420 con 0x7f84c00ae8b0 2023-10-07T11:03:18.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c539a700 1 --2- 172.21.15.29:0/2746020349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c00ae8b0 0x7f84c00b1300 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f84c00cbff0 tx=0x7f84b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fad8579b377ce7a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c4b99700 1 -- 172.21.15.29:0/2746020349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0009e70 con 0x7f84c00ae8b0 2023-10-07T11:03:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c4b99700 1 -- 172.21.15.29:0/2746020349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b000b860 con 0x7f84c00ae8b0 2023-10-07T11:03:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.704+0000 7f84c4b99700 1 -- 172.21.15.29:0/2746020349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b000bc00 con 0x7f84c00ae8b0 2023-10-07T11:03:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 -- 172.21.15.29:0/2746020349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c00ae8b0 msgr2=0x7f84c00b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2746020349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c00ae8b0 0x7f84c00b1300 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f84c00cbff0 tx=0x7f84b0009250 comp rx=0 tx=0).stop 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 -- 172.21.15.29:0/2746020349 shutdown_connections 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2746020349 >> v2:172.21.15.144:3300/0 conn(0x7f84c00b1840 0x7f84c0157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2746020349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c00ae8b0 0x7f84c00b1300 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 -- 172.21.15.29:0/2746020349 >> 172.21.15.29:0/2746020349 conn(0x7f84c00680a0 msgr2=0x7f84c00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 -- 172.21.15.29:0/2746020349 shutdown_connections 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.705+0000 7f84c7dff700 1 -- 172.21.15.29:0/2746020349 wait complete. 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.707+0000 7f84c7dff700 1 Processor -- start 2023-10-07T11:03:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.707+0000 7f84c7dff700 1 -- start start 2023-10-07T11:03:18.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c7dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c00b1840 0x7f84c016ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:18.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c7dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0157290 0x7f84c014bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:18.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c7dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 0x7f84c0167e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:18.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c7dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c014c070 con 0x7f84c014e570 2023-10-07T11:03:18.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c7dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0167970 con 0x7f84c0157290 2023-10-07T11:03:18.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c7dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c016b330 con 0x7f84c00b1840 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 0x7f84c0167e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c5b9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c00b1840 0x7f84c016ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c5b9b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c00b1840 0x7f84c016ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:45120/0 (socket says 172.21.15.29:45120) 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 0x7f84c0167e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49152/0 (socket says 172.21.15.29:49152) 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c5b9b700 1 -- 172.21.15.29:0/2633207427 learned_addr learned my addr 172.21.15.29:0/2633207427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c539a700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0157290 0x7f84c014bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 -- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c00b1840 msgr2=0x7f84c016ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c00b1840 0x7f84c016ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 -- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0157290 msgr2=0x7f84c014bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:18.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0157290 0x7f84c014bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.708+0000 7f84c639c700 1 -- 172.21.15.29:0/2633207427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00094f0 con 0x7f84c014e570 2023-10-07T11:03:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.709+0000 7f84c639c700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 0x7f84c0167e00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009e50 tx=0x7f84b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.709+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b800caa0 con 0x7f84c014e570 2023-10-07T11:03:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.709+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b800f040 con 0x7f84c014e570 2023-10-07T11:03:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.709+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b800ce70 con 0x7f84c014e570 2023-10-07T11:03:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.709+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c016b610 con 0x7f84c014e570 2023-10-07T11:03:18.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.709+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c01675c0 con 0x7f84c014e570 2023-10-07T11:03:18.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.710+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c0167b00 con 0x7f84c014e570 2023-10-07T11:03:18.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.711+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f84b8014690 con 0x7f84c014e570 2023-10-07T11:03:18.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.712+0000 7f84b5ffb700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f84a4066f30 0x7f84a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:18.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.712+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5329+0+0 (secure 0 0 0) 0x7f84b8084f70 con 0x7f84c014e570 2023-10-07T11:03:18.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.712+0000 7f84c5b9b700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f84a4066f30 0x7f84a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:18.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.713+0000 7f84c5b9b700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f84a4066f30 0x7f84a40693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84c0153650 tx=0x7f84bc007f70 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:18.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.716+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84b80522b0 con 0x7f84c014e570 2023-10-07T11:03:18.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.997+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f84c004c3c0 con 0x7f84c014e570 2023-10-07T11:03:18.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:18.998+0000 7f84b5ffb700 1 -- 172.21.15.29:0/2633207427 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f84b80558d0 con 0x7f84c014e570 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFly/NHBxAAeD3AD3RCcHfUesAKjcVKVg==", 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:18.999 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl6YmlHhAATbbstv8MbgdQtu7S7YYUkQ==", 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.000 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlSRDeEhAAGmHIOTOo0v+/tCrNdJabww==", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:03:19.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFl7mD8KRAAz6ve7qGlfCjnfyUjxl69kQ==", 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:03:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC3JCFlKl+QNBAAaBirRNzXj5LnJGx0Y+himw==", 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.003 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC6JCFlOMSFOBAAQKM+71eAmPEvxWnw7cqRng==", 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC+JCFlnqM6HhAABJnrZ4KVx/R2pjzpp/i7/A==", 2023-10-07T11:03:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDBJCFlYV2gIRAAvXw59moBlu8waKxAJJ09Yw==", 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDIJCFlh9hJBRAAlK79EiviQhfGDyvOLyDdDA==", 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDLJCFlR1KUCBAA84H5upQyfxRcufyBxt5AjQ==", 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-07T11:03:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDOJCFlcWbkCxAAsmxmQovE0qCmM5kORZ1ozg==", 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQDRJCFlYgsVDxAAbQvJDJeNxIhMnBqF2Gr/yQ==", 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC1JCFlnVCMNBAAB/nqoey9B/rOj8kKT88s8g==", 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-07T11:03:19.009 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlQE2xCxAAxjlkyxxQrbPnJkL90rjdbA==", 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFlUhdNABAAthm57zL04Q0hsf+qZ1MNqQ==", 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.011 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC2JCFluELOEhAAjfOcH1MxyNcJO77Ltu7rSA==", 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-07T11:03:19.012 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCzJCFlmbAHLxAAm/aEE3a00QZFRkQG11tlZQ==", 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlap8wOxAABLVVvLa24lpHbWsC/1nVjg==", 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.013 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQC0JCFlU5JjGxAAMctyZUT0gGURhVlhQeeXjQ==", 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-07T11:03:19.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-07T11:03:19.015 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.015 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:19.015 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-07T11:03:19.015 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f84a4066f30 msgr2=0x7f84a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f84a4066f30 0x7f84a40693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84c0153650 tx=0x7f84bc007f70 comp rx=0 tx=0).stop 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 msgr2=0x7f84c0167e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 0x7f84c0167e00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009e50 tx=0x7f84b800b040 comp rx=0 tx=0).stop 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 shutdown_connections 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84c014e570 0x7f84c0167e00 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f84a4066f30 0x7f84a40693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c00b1840 0x7f84c016ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 --2- 172.21.15.29:0/2633207427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0157290 0x7f84c014bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 >> 172.21.15.29:0/2633207427 conn(0x7f84c00680a0 msgr2=0x7f84c00c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 shutdown_connections 2023-10-07T11:03:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.000+0000 7f84c7dff700 1 -- 172.21.15.29:0/2633207427 wait complete. 2023-10-07T11:03:19.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.214+0000 7ff8d59ff700 1 Processor -- start 2023-10-07T11:03:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.214+0000 7ff8d59ff700 1 -- start start 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.215+0000 7ff8d59ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d0159550 0x7ff8d00b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.215+0000 7ff8d59ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8d00b0f50 0x7ff8d00b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.215+0000 7ff8d59ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8d00b2bb0 0x7ff8d015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.215+0000 7ff8d59ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d006f4e0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.215+0000 7ff8d59ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d006e8d0 con 0x7ff8d00b2bb0 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.215+0000 7ff8d59ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d006ebb0 con 0x7ff8d0159550 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8d00b0f50 0x7ff8d00b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8d00b0f50 0x7ff8d00b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49162/0 (socket says 172.21.15.29:49162) 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/3436267103 learned_addr learned my addr 172.21.15.29:0/3436267103 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8cf7fe700 1 --2- 172.21.15.29:0/3436267103 >> v2:172.21.15.144:3300/0 conn(0x7ff8d00b2bb0 0x7ff8d015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ceffd700 1 --2- 172.21.15.29:0/3436267103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d0159550 0x7ff8d00b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/3436267103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d0159550 msgr2=0x7ff8d00b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/3436267103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d0159550 0x7ff8d00b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/3436267103 >> v2:172.21.15.144:3300/0 conn(0x7ff8d00b2bb0 msgr2=0x7ff8d015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/3436267103 >> v2:172.21.15.144:3300/0 conn(0x7ff8d00b2bb0 0x7ff8d015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 -- 172.21.15.29:0/3436267103 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d006ee90 con 0x7ff8d00b0f50 2023-10-07T11:03:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.216+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/3436267103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00b2670 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0068200 tx=0x7ff8c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5eace91bdc4562 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8cdffb700 1 -- 172.21.15.29:0/3436267103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400b640 con 0x7ff8d00b0f50 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8cdffb700 1 -- 172.21.15.29:0/3436267103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c400b7a0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8cdffb700 1 -- 172.21.15.29:0/3436267103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400bb60 con 0x7ff8d00b0f50 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 -- 172.21.15.29:0/3436267103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 msgr2=0x7ff8d00b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/3436267103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00b2670 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0068200 tx=0x7ff8c4009250 comp rx=0 tx=0).stop 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 -- 172.21.15.29:0/3436267103 shutdown_connections 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/3436267103 >> v2:172.21.15.144:3300/0 conn(0x7ff8d00b2bb0 0x7ff8d015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/3436267103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d0159550 0x7ff8d00b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/3436267103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00b2670 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 -- 172.21.15.29:0/3436267103 >> 172.21.15.29:0/3436267103 conn(0x7ff8d0069190 msgr2=0x7ff8d006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 -- 172.21.15.29:0/3436267103 shutdown_connections 2023-10-07T11:03:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.217+0000 7ff8d59ff700 1 -- 172.21.15.29:0/3436267103 wait complete. 2023-10-07T11:03:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.219+0000 7ff8d59ff700 1 Processor -- start 2023-10-07T11:03:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.219+0000 7ff8d59ff700 1 -- start start 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8d59ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8d59ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d00b2bb0 0x7ff8d01417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8d59ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8d0159550 0x7ff8d00ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8d59ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0142bc0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8d59ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0141d10 con 0x7ff8d0159550 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8d59ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0141fb0 con 0x7ff8d00b2bb0 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8ceffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8ceffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00af420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49164/0 (socket says 172.21.15.29:49164) 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8ceffd700 1 -- 172.21.15.29:0/2670809230 learned_addr learned my addr 172.21.15.29:0/2670809230 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d00b2bb0 0x7ff8d01417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.220+0000 7ff8cf7fe700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8d0159550 0x7ff8d00ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8ceffd700 1 -- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d00b2bb0 msgr2=0x7ff8d01417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8ceffd700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d00b2bb0 0x7ff8d01417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8ceffd700 1 -- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8d0159550 msgr2=0x7ff8d00ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8ceffd700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8d0159550 0x7ff8d00ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8ceffd700 1 -- 172.21.15.29:0/2670809230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c40094f0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8ceffd700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00af420 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000ec80 tx=0x7ff8c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c000c850 con 0x7ff8d00b0f50 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c000c9d0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d0142290 con 0x7ff8d00b0f50 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c000cd90 con 0x7ff8d00b0f50 2023-10-07T11:03:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.221+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d0142850 con 0x7ff8d00b0f50 2023-10-07T11:03:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.222+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d004c3c0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.223+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff8c0013680 con 0x7ff8d00b0f50 2023-10-07T11:03:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.224+0000 7ff8b6ffd700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff8bc066f30 0x7ff8bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.224+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5329+0+0 (secure 0 0 0) 0x7ff8c0084750 con 0x7ff8d00b0f50 2023-10-07T11:03:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.225+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff8bc066f30 0x7ff8bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.225+0000 7ff8ce7fc700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff8bc066f30 0x7ff8bc0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0068200 tx=0x7ff8c4017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.229+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8c0051b30 con 0x7ff8d00b0f50 2023-10-07T11:03:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.429+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8d00683d0 con 0x7ff8d00b0f50 2023-10-07T11:03:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.430+0000 7ff8b6ffd700 1 -- 172.21.15.29:0/2670809230 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+12973 (secure 0 0 0) 0x7ff8c0055150 con 0x7ff8d00b0f50 2023-10-07T11:03:19.431 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:19.431 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":193,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:18.475978+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:19.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.432+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff8bc066f30 msgr2=0x7ff8bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.432+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff8bc066f30 0x7ff8bc0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0068200 tx=0x7ff8c4017000 comp rx=0 tx=0).stop 2023-10-07T11:03:19.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.432+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 msgr2=0x7ff8d00af420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.432+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00af420 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000ec80 tx=0x7ff8c00091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 shutdown_connections 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8d0159550 0x7ff8d00ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8d00b0f50 0x7ff8d00af420 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff8bc066f30 0x7ff8bc0693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 --2- 172.21.15.29:0/2670809230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8d00b2bb0 0x7ff8d01417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 >> 172.21.15.29:0/2670809230 conn(0x7ff8d0069190 msgr2=0x7ff8d0159920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 shutdown_connections 2023-10-07T11:03:19.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.433+0000 7ff8d59ff700 1 -- 172.21.15.29:0/2670809230 wait complete. 2023-10-07T11:03:19.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-07T11:03:19.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T11:03:19.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.650+0000 7f91ca3ff700 1 Processor -- start 2023-10-07T11:03:19.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.650+0000 7f91ca3ff700 1 -- start start 2023-10-07T11:03:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.651+0000 7f91ca3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91c40ae570 0x7f91c40afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.651+0000 7f91ca3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f91c40b0520 0x7f91c40b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.651+0000 7f91ca3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40b0ea0 con 0x7f91c4156180 2023-10-07T11:03:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91ca3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c40ad4d0 con 0x7f91c40ae570 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91ca3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c40ad7b0 con 0x7f91c40b0520 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c37fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91c40ae570 0x7f91c40afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c3fff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91c4156180 0x7f91c40accf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49532/0 (socket says 172.21.15.29:49532) 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c3fff700 1 -- 172.21.15.29:0/1102725650 learned_addr learned my addr 172.21.15.29:0/1102725650 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c899c700 1 --2- 172.21.15.29:0/1102725650 >> v2:172.21.15.144:3300/0 conn(0x7f91c40b0520 0x7f91c40b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c37fe700 1 -- 172.21.15.29:0/1102725650 >> v2:172.21.15.144:3300/0 conn(0x7f91c40b0520 msgr2=0x7f91c40b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c37fe700 1 --2- 172.21.15.29:0/1102725650 >> v2:172.21.15.144:3300/0 conn(0x7f91c40b0520 0x7f91c40b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c37fe700 1 -- 172.21.15.29:0/1102725650 >> v1:172.21.15.29:6789/0 conn(0x7f91c4156180 legacy=0x7f91c40accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c37fe700 1 -- 172.21.15.29:0/1102725650 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c40ada90 con 0x7f91c40ae570 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.652+0000 7f91c2ffd700 1 -- 172.21.15.29:0/1102725650 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075153468 0 0) 0x7f91c40b0ea0 con 0x7f91c4156180 2023-10-07T11:03:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91c37fe700 1 --2- 172.21.15.29:0/1102725650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c40ae570 0x7f91c40afc00 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f91c4070f60 tx=0x7f91b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812841036ee9a3e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91c2ffd700 1 -- 172.21.15.29:0/1102725650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b800b6b0 con 0x7f91c40ae570 2023-10-07T11:03:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91c2ffd700 1 -- 172.21.15.29:0/1102725650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b800b810 con 0x7f91c40ae570 2023-10-07T11:03:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91c2ffd700 1 -- 172.21.15.29:0/1102725650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b800bbb0 con 0x7f91c40ae570 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91ca3ff700 1 -- 172.21.15.29:0/1102725650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c40ae570 msgr2=0x7f91c40afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/1102725650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c40ae570 0x7f91c40afc00 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f91c4070f60 tx=0x7f91b80091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91ca3ff700 1 -- 172.21.15.29:0/1102725650 shutdown_connections 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/1102725650 >> v2:172.21.15.144:3300/0 conn(0x7f91c40b0520 0x7f91c40b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/1102725650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c40ae570 0x7f91c40afc00 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.653+0000 7f91ca3ff700 1 -- 172.21.15.29:0/1102725650 >> 172.21.15.29:0/1102725650 conn(0x7f91c4067610 msgr2=0x7f91c4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.654+0000 7f91ca3ff700 1 -- 172.21.15.29:0/1102725650 shutdown_connections 2023-10-07T11:03:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.654+0000 7f91ca3ff700 1 -- 172.21.15.29:0/1102725650 wait complete. 2023-10-07T11:03:19.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.655+0000 7f91ca3ff700 1 Processor -- start 2023-10-07T11:03:19.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.655+0000 7f91ca3ff700 1 -- start start 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91ca3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91c40ae570 0x7f91c415a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91ca3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c40b0520 0x7f91c406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91ca3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 0x7f91c406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91ca3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4169030 con 0x7f91c4156180 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91ca3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c406e1a0 con 0x7f91c40b0520 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91ca3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c406e450 con 0x7f91c40ae570 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91c899c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 0x7f91c406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91c899c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 0x7f91c406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49176/0 (socket says 172.21.15.29:49176) 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.656+0000 7f91c899c700 1 -- 172.21.15.29:0/559482934 learned_addr learned my addr 172.21.15.29:0/559482934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c37fe700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c40b0520 0x7f91c406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c3fff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91c40ae570 0x7f91c415a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c899c700 1 -- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91c40ae570 msgr2=0x7f91c415a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c899c700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91c40ae570 0x7f91c415a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c899c700 1 -- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c40b0520 msgr2=0x7f91c406d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c899c700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c40b0520 0x7f91c406d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c899c700 1 -- 172.21.15.29:0/559482934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8009420 con 0x7f91c4156180 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91c899c700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 0x7f91c406dc60 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009fa0 tx=0x7f91bc009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91bc00ed20 con 0x7f91c4156180 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91bc00ee80 con 0x7f91c4156180 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.657+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c406e730 con 0x7f91c4156180 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.658+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91bc014670 con 0x7f91c4156180 2023-10-07T11:03:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.658+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c406ed70 con 0x7f91c4156180 2023-10-07T11:03:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.659+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c40b1850 con 0x7f91c4156180 2023-10-07T11:03:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.660+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f91bc0149e0 con 0x7f91c4156180 2023-10-07T11:03:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.661+0000 7f91abfff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f91b0066f30 0x7f91b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.661+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5329+0+0 (secure 0 0 0) 0x7f91bc084c50 con 0x7f91c4156180 2023-10-07T11:03:19.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.662+0000 7f91c3fff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f91b0066f30 0x7f91b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:19.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.662+0000 7f91c3fff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f91b0066f30 0x7f91b00693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c8f0 tx=0x7f91b4009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:19.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.665+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91bc051f90 con 0x7f91c4156180 2023-10-07T11:03:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.864+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91c4156de0 con 0x7f91c4156180 2023-10-07T11:03:19.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.865+0000 7f91abfff700 1 -- 172.21.15.29:0/559482934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91bc0555b0 con 0x7f91c4156180 2023-10-07T11:03:19.866 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T11:03:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f91b0066f30 msgr2=0x7f91b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f91b0066f30 0x7f91b00693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c8f0 tx=0x7f91b4009040 comp rx=0 tx=0).stop 2023-10-07T11:03:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 msgr2=0x7f91c406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 0x7f91c406dc60 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009fa0 tx=0x7f91bc009900 comp rx=0 tx=0).stop 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 shutdown_connections 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91c4156180 0x7f91c406dc60 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f91b0066f30 0x7f91b00693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91c40ae570 0x7f91c415a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 --2- 172.21.15.29:0/559482934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91c40b0520 0x7f91c406d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 >> 172.21.15.29:0/559482934 conn(0x7f91c4067610 msgr2=0x7f91c41583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.867+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 shutdown_connections 2023-10-07T11:03:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:19.868+0000 7f91ca3ff700 1 -- 172.21.15.29:0/559482934 wait complete. 2023-10-07T11:03:19.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-07T11:03:20.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.081+0000 7f243f1ff700 1 Processor -- start 2023-10-07T11:03:20.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.082+0000 7f243f1ff700 1 -- start start 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f243f1ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24381599c0 0x7f243815daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f243f1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f243f1ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24380cf570 con 0x7f24381595f0 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f243f1ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24380cf850 con 0x7f24381599c0 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f243f1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24380cfb30 con 0x7f243815dfe0 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f2437fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24381599c0 0x7f243815daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f2437fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24381599c0 0x7f243815daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49186/0 (socket says 172.21.15.29:49186) 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f2437fff700 1 -- 172.21.15.29:0/178050753 learned_addr learned my addr 172.21.15.29:0/178050753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:20.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.083+0000 7f243d79c700 1 --2- 172.21.15.29:0/178050753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f2437fff700 1 -- 172.21.15.29:0/178050753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 msgr2=0x7f24380caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f2437fff700 1 --2- 172.21.15.29:0/178050753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f2437fff700 1 -- 172.21.15.29:0/178050753 >> v1:172.21.15.144:6789/0 conn(0x7f24381595f0 legacy=0x7f243815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f2437fff700 1 -- 172.21.15.29:0/178050753 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380cfe10 con 0x7f24381599c0 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f2437fff700 1 --2- 172.21.15.29:0/178050753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381599c0 0x7f243815daa0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f24240099b0 tx=0x7f2424009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c325b61eb53fc17 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f24377fe700 1 -- 172.21.15.29:0/178050753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242400b860 con 0x7f24381599c0 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f24377fe700 1 -- 172.21.15.29:0/178050753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242400b9c0 con 0x7f24381599c0 2023-10-07T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.084+0000 7f24377fe700 1 -- 172.21.15.29:0/178050753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242400bd60 con 0x7f24381599c0 2023-10-07T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 -- 172.21.15.29:0/178050753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381599c0 msgr2=0x7f243815daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 --2- 172.21.15.29:0/178050753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381599c0 0x7f243815daa0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f24240099b0 tx=0x7f2424009250 comp rx=0 tx=0).stop 2023-10-07T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 -- 172.21.15.29:0/178050753 shutdown_connections 2023-10-07T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 --2- 172.21.15.29:0/178050753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 --2- 172.21.15.29:0/178050753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381599c0 0x7f243815daa0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 -- 172.21.15.29:0/178050753 >> 172.21.15.29:0/178050753 conn(0x7f2438063a10 msgr2=0x7f24380d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 -- 172.21.15.29:0/178050753 shutdown_connections 2023-10-07T11:03:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.085+0000 7f243f1ff700 1 -- 172.21.15.29:0/178050753 wait complete. 2023-10-07T11:03:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.086+0000 7f243f1ff700 1 Processor -- start 2023-10-07T11:03:20.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.087+0000 7f243f1ff700 1 -- start start 2023-10-07T11:03:20.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.087+0000 7f243f1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 0x7f24380b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.087+0000 7f243f1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24381599c0 0x7f2438151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243f1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380695a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243f1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24380c81b0 con 0x7f24381595f0 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243f1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24380c8fb0 con 0x7f24381599c0 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243f1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24380b4250 con 0x7f243815dfe0 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 0x7f24380b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 0x7f24380b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49202/0 (socket says 172.21.15.29:49202) 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 -- 172.21.15.29:0/2691985537 learned_addr learned my addr 172.21.15.29:0/2691985537 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f2437fff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24381599c0 0x7f2438151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 -- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 msgr2=0x7f24380695a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 -- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24381599c0 msgr2=0x7f2438151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24381599c0 0x7f2438151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 -- 172.21.15.29:0/2691985537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240094f0 con 0x7f24381595f0 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.088+0000 7f243cf9b700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 0x7f24380b3d10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f242c00c7c0 tx=0x7f242c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.089+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c011590 con 0x7f24381595f0 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.089+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c011710 con 0x7f24381595f0 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.089+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c011ad0 con 0x7f24381595f0 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.089+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24380c83a0 con 0x7f24381595f0 2023-10-07T11:03:20.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.089+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24380c8960 con 0x7f24381595f0 2023-10-07T11:03:20.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.090+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243804c3c0 con 0x7f24381595f0 2023-10-07T11:03:20.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.091+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f242c01b3b0 con 0x7f24381595f0 2023-10-07T11:03:20.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.092+0000 7f2434ff9700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f241c066f30 0x7f241c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.092+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5329+0+0 (secure 0 0 0) 0x7f242c083970 con 0x7f24381595f0 2023-10-07T11:03:20.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.092+0000 7f2437fff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f241c066f30 0x7f241c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.093+0000 7f2437fff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f241c066f30 0x7f241c0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24240099b0 tx=0x7f2424017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:20.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.096+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f242c050d60 con 0x7f24381595f0 2023-10-07T11:03:20.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.304+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2438064040 con 0x7f24381595f0 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.304+0000 7f2434ff9700 1 -- 172.21.15.29:0/2691985537 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v193) v1 ==== 84+0+65 (secure 0 0 0) 0x7f242c017070 con 0x7f24381595f0 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:20.305 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T11:03:20.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.306+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f241c066f30 msgr2=0x7f241c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.306+0000 7f243f1ff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f241c066f30 0x7f241c0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24240099b0 tx=0x7f2424017000 comp rx=0 tx=0).stop 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.306+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 msgr2=0x7f24380b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.306+0000 7f243f1ff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 0x7f24380b3d10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f242c00c7c0 tx=0x7f242c007f70 comp rx=0 tx=0).stop 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 shutdown_connections 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f241c066f30 0x7f241c0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f243815dfe0 0x7f24380695a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24381595f0 0x7f24380b3d10 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 --2- 172.21.15.29:0/2691985537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24381599c0 0x7f2438151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 >> 172.21.15.29:0/2691985537 conn(0x7f2438063a10 msgr2=0x7f2438152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 shutdown_connections 2023-10-07T11:03:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.307+0000 7f243f1ff700 1 -- 172.21.15.29:0/2691985537 wait complete. 2023-10-07T11:03:20.325 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-07T11:03:20.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-07T11:03:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.523+0000 7f1727dff700 1 Processor -- start 2023-10-07T11:03:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.523+0000 7f1727dff700 1 -- start start 2023-10-07T11:03:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f1727dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f1727dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17200cf2c0 0x7f17200c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f1727dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17200cc940 con 0x7f17200cb6f0 2023-10-07T11:03:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f1727dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17200b2df0 con 0x7f17200cf2c0 2023-10-07T11:03:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f1727dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17200cbd30 con 0x7f17200b2a20 2023-10-07T11:03:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f172539a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f17200cb6f0 0x7f17200cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49542/0 (socket says 172.21.15.29:49542) 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.524+0000 7f172539a700 1 -- 172.21.15.29:0/976762150 learned_addr learned my addr 172.21.15.29:0/976762150 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f172639c700 1 --2- 172.21.15.29:0/976762150 >> v2:172.21.15.144:3300/0 conn(0x7f17200cf2c0 0x7f17200c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1725b9b700 1 --2- 172.21.15.29:0/976762150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1724b99700 1 -- 172.21.15.29:0/976762150 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390069678 0 0) 0x7f17200cc940 con 0x7f17200cb6f0 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1724b99700 1 -- 172.21.15.29:0/976762150 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f170c003570 con 0x7f17200cb6f0 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1725b9b700 1 -- 172.21.15.29:0/976762150 >> v2:172.21.15.144:3300/0 conn(0x7f17200cf2c0 msgr2=0x7f17200c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1725b9b700 1 --2- 172.21.15.29:0/976762150 >> v2:172.21.15.144:3300/0 conn(0x7f17200cf2c0 0x7f17200c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1725b9b700 1 -- 172.21.15.29:0/976762150 >> v1:172.21.15.29:6789/0 conn(0x7f17200cb6f0 legacy=0x7f17200cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f1725b9b700 1 -- 172.21.15.29:0/976762150 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17200cc010 con 0x7f17200b2a20 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.525+0000 7f172639c700 1 --2- 172.21.15.29:0/976762150 >> v2:172.21.15.144:3300/0 conn(0x7f17200cf2c0 0x7f17200c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1725b9b700 1 --2- 172.21.15.29:0/976762150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200cb1b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f171c0098d0 tx=0x7f171c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa16c9bcd3d481a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1724b99700 1 -- 172.21.15.29:0/976762150 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c005c10 con 0x7f17200b2a20 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1724b99700 1 -- 172.21.15.29:0/976762150 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171c005d70 con 0x7f17200b2a20 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1724b99700 1 -- 172.21.15.29:0/976762150 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c006260 con 0x7f17200b2a20 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1727dff700 1 -- 172.21.15.29:0/976762150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 msgr2=0x7f17200cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1727dff700 1 --2- 172.21.15.29:0/976762150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200cb1b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f171c0098d0 tx=0x7f171c009120 comp rx=0 tx=0).stop 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1727dff700 1 -- 172.21.15.29:0/976762150 shutdown_connections 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1727dff700 1 --2- 172.21.15.29:0/976762150 >> v2:172.21.15.144:3300/0 conn(0x7f17200cf2c0 0x7f17200c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1727dff700 1 --2- 172.21.15.29:0/976762150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200cb1b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.526+0000 7f1727dff700 1 -- 172.21.15.29:0/976762150 >> 172.21.15.29:0/976762150 conn(0x7f1720099820 msgr2=0x7f1720063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.527+0000 7f1727dff700 1 -- 172.21.15.29:0/976762150 shutdown_connections 2023-10-07T11:03:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.527+0000 7f1727dff700 1 -- 172.21.15.29:0/976762150 wait complete. 2023-10-07T11:03:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.529+0000 7f1727dff700 1 Processor -- start 2023-10-07T11:03:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.529+0000 7f1727dff700 1 -- start start 2023-10-07T11:03:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f1727dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f1727dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17200cb6f0 0x7f1720169710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f1727dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 0x7f1720169c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f1727dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17200b5dd0 con 0x7f17200cf2c0 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f1727dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17200b51c0 con 0x7f17200cb6f0 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f1727dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17200b54a0 con 0x7f17200b2a20 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f172639c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 0x7f1720169c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f172639c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 0x7f1720169c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49218/0 (socket says 172.21.15.29:49218) 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.530+0000 7f172639c700 1 -- 172.21.15.29:0/3577641470 learned_addr learned my addr 172.21.15.29:0/3577641470 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f1725b9b700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172539a700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17200cb6f0 0x7f1720169710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172639c700 1 -- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 msgr2=0x7f17200c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172639c700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172639c700 1 -- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17200cb6f0 msgr2=0x7f1720169710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172639c700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17200cb6f0 0x7f1720169710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172639c700 1 -- 172.21.15.29:0/3577641470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718009a20 con 0x7f17200cf2c0 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f172639c700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 0x7f1720169c50 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1718009ec0 tx=0x7f171800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171800cb70 con 0x7f17200cf2c0 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171800f040 con 0x7f17200cf2c0 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c0093c0 con 0x7f17200cf2c0 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.531+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718014480 con 0x7f17200cf2c0 2023-10-07T11:03:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.532+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17200b5a60 con 0x7f17200cf2c0 2023-10-07T11:03:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.533+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17201538a0 con 0x7f17200cf2c0 2023-10-07T11:03:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.533+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f17180147f0 con 0x7f17200cf2c0 2023-10-07T11:03:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.534+0000 7f1715ffb700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1704066f30 0x7f17040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.534+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5329+0+0 (secure 0 0 0) 0x7f1718085090 con 0x7f17200cf2c0 2023-10-07T11:03:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.535+0000 7f1725b9b700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1704066f30 0x7f17040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.535+0000 7f1725b9b700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1704066f30 0x7f17040693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f171c009160 tx=0x7f171c016000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:20.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.538+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17180523d0 con 0x7f17200cf2c0 2023-10-07T11:03:20.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:20.740+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f17200718b0 con 0x7f17200cf2c0 2023-10-07T11:03:21.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.028+0000 7f1715ffb700 1 -- 172.21.15.29:0/3577641470 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v194) v1 ==== 138+0+0 (secure 0 0 0) 0x7f17180559f0 con 0x7f17200cf2c0 2023-10-07T11:03:21.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1704066f30 msgr2=0x7f17040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1704066f30 0x7f17040693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f171c009160 tx=0x7f171c016000 comp rx=0 tx=0).stop 2023-10-07T11:03:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 msgr2=0x7f1720169c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 0x7f1720169c50 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1718009ec0 tx=0x7f171800b040 comp rx=0 tx=0).stop 2023-10-07T11:03:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 shutdown_connections 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17200cf2c0 0x7f1720169c50 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1704066f30 0x7f17040693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17200b2a20 0x7f17200c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 --2- 172.21.15.29:0/3577641470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17200cb6f0 0x7f1720169710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 >> 172.21.15.29:0/3577641470 conn(0x7f1720099820 msgr2=0x7f1720159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 shutdown_connections 2023-10-07T11:03:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.033+0000 7f1727dff700 1 -- 172.21.15.29:0/3577641470 wait complete. 2023-10-07T11:03:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-07T11:03:21.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-07T11:03:21.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.257+0000 7fdc33dff700 1 Processor -- start 2023-10-07T11:03:21.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.257+0000 7fdc33dff700 1 -- start start 2023-10-07T11:03:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.258+0000 7fdc33dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc2c153ac0 0x7fdc2c15a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.258+0000 7fdc33dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c15ab80 con 0x7fdc2c153e90 2023-10-07T11:03:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc33dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c069d40 con 0x7fdc2c0b0690 2023-10-07T11:03:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc33dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c068ca0 con 0x7fdc2c153ac0 2023-10-07T11:03:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc31b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc2c153ac0 0x7fdc2c15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc31b9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc2c153ac0 0x7fdc2c15a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49220/0 (socket says 172.21.15.29:49220) 2023-10-07T11:03:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc31b9b700 1 -- 172.21.15.29:0/2130491936 learned_addr learned my addr 172.21.15.29:0/2130491936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc3139a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdc2c153e90 0x7fdc2c0aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49554/0 (socket says 172.21.15.29:49554) 2023-10-07T11:03:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc31b9b700 1 -- 172.21.15.29:0/2130491936 >> v1:172.21.15.144:6789/0 conn(0x7fdc2c0b0690 legacy=0x7fdc2c15ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc31b9b700 1 -- 172.21.15.29:0/2130491936 >> v1:172.21.15.29:6789/0 conn(0x7fdc2c153e90 legacy=0x7fdc2c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.259+0000 7fdc31b9b700 1 -- 172.21.15.29:0/2130491936 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c068f80 con 0x7fdc2c153ac0 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc31b9b700 1 --2- 172.21.15.29:0/2130491936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c153ac0 0x7fdc2c15a640 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fdc280099b0 tx=0x7fdc28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80528acb54e44ed9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc30b99700 1 -- 172.21.15.29:0/2130491936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800b860 con 0x7fdc2c153ac0 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc30b99700 1 -- 172.21.15.29:0/2130491936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2800b9c0 con 0x7fdc2c153ac0 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc30b99700 1 -- 172.21.15.29:0/2130491936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800bd60 con 0x7fdc2c153ac0 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc33dff700 1 -- 172.21.15.29:0/2130491936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c153ac0 msgr2=0x7fdc2c15a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc33dff700 1 --2- 172.21.15.29:0/2130491936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c153ac0 0x7fdc2c15a640 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fdc280099b0 tx=0x7fdc28009250 comp rx=0 tx=0).stop 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc33dff700 1 -- 172.21.15.29:0/2130491936 shutdown_connections 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.260+0000 7fdc33dff700 1 --2- 172.21.15.29:0/2130491936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c153ac0 0x7fdc2c15a640 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.261+0000 7fdc33dff700 1 -- 172.21.15.29:0/2130491936 >> 172.21.15.29:0/2130491936 conn(0x7fdc2c067610 msgr2=0x7fdc2c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.261+0000 7fdc33dff700 1 -- 172.21.15.29:0/2130491936 shutdown_connections 2023-10-07T11:03:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.261+0000 7fdc33dff700 1 -- 172.21.15.29:0/2130491936 wait complete. 2023-10-07T11:03:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.262+0000 7fdc33dff700 1 Processor -- start 2023-10-07T11:03:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.262+0000 7fdc33dff700 1 -- start start 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc33dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 0x7fdc2c0c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc33dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c153ac0 0x7fdc2c151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc33dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c155130 0x7fdc2c0c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc33dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c155630 con 0x7fdc2c0b0690 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc33dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c155910 con 0x7fdc2c155130 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc33dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c155bf0 con 0x7fdc2c153ac0 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc31b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 0x7fdc2c0c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc31b9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 0x7fdc2c0c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49226/0 (socket says 172.21.15.29:49226) 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.263+0000 7fdc31b9b700 1 -- 172.21.15.29:0/3745107675 learned_addr learned my addr 172.21.15.29:0/3745107675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc3239c700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c155130 0x7fdc2c0c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc3139a700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c153ac0 0x7fdc2c151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc31b9b700 1 -- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c153ac0 msgr2=0x7fdc2c151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc31b9b700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c153ac0 0x7fdc2c151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc31b9b700 1 -- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c155130 msgr2=0x7fdc2c0c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc31b9b700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c155130 0x7fdc2c0c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc31b9b700 1 -- 172.21.15.29:0/3745107675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280094f0 con 0x7fdc2c0b0690 2023-10-07T11:03:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc31b9b700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 0x7fdc2c0c9870 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009290 tx=0x7fdc28017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2801a630 con 0x7fdc2c0b0690 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2801a790 con 0x7fdc2c0b0690 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2801ab10 con 0x7fdc2c0b0690 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.264+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c155ed0 con 0x7fdc2c0b0690 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.265+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c152340 con 0x7fdc2c0b0690 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.266+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c0718b0 con 0x7fdc2c0b0690 2023-10-07T11:03:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.267+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdc2802d440 con 0x7fdc2c0b0690 2023-10-07T11:03:21.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.269+0000 7fdc21ffb700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fdc10066f30 0x7fdc100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:21.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.269+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5699+0+0 (secure 0 0 0) 0x7fdc28086690 con 0x7fdc2c0b0690 2023-10-07T11:03:21.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.269+0000 7fdc3139a700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fdc10066f30 0x7fdc100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:21.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.270+0000 7fdc3139a700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fdc10066f30 0x7fdc100693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c007fd0 tx=0x7fdc1c007d70 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.273+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc280538e0 con 0x7fdc2c0b0690 2023-10-07T11:03:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:21.474+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdc2c0b3120 con 0x7fdc2c0b0690 2023-10-07T11:03:22.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.034+0000 7fdc21ffb700 1 -- 172.21.15.29:0/3745107675 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v195) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdc28056f00 con 0x7fdc2c0b0690 2023-10-07T11:03:22.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.038+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fdc10066f30 msgr2=0x7fdc100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:22.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.038+0000 7fdc33dff700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fdc10066f30 0x7fdc100693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c007fd0 tx=0x7fdc1c007d70 comp rx=0 tx=0).stop 2023-10-07T11:03:22.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.038+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 msgr2=0x7fdc2c0c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:22.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.038+0000 7fdc33dff700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 0x7fdc2c0c9870 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009290 tx=0x7fdc28017000 comp rx=0 tx=0).stop 2023-10-07T11:03:22.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.043+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 shutdown_connections 2023-10-07T11:03:22.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.043+0000 7fdc33dff700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c155130 0x7fdc2c0c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.043+0000 7fdc33dff700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc2c0b0690 0x7fdc2c0c9870 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.043+0000 7fdc33dff700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fdc10066f30 0x7fdc100693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.043+0000 7fdc33dff700 1 --2- 172.21.15.29:0/3745107675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c153ac0 0x7fdc2c151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.043+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 >> 172.21.15.29:0/3745107675 conn(0x7fdc2c067610 msgr2=0x7fdc2c159350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:22.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.045+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 shutdown_connections 2023-10-07T11:03:22.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.045+0000 7fdc33dff700 1 -- 172.21.15.29:0/3745107675 wait complete. 2023-10-07T11:03:22.049 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-07T11:03:22.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-07T11:03:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.270+0000 7f946dbff700 1 Processor -- start 2023-10-07T11:03:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.270+0000 7f946dbff700 1 -- start start 2023-10-07T11:03:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.271+0000 7f946dbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f946815ade0 0x7f94680b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.271+0000 7f946dbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94680b0540 0x7f946815c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.271+0000 7f946dbff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680b0980 con 0x7f9468153cc0 2023-10-07T11:03:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.271+0000 7f946dbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9468154090 con 0x7f94680b0540 2023-10-07T11:03:22.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.271+0000 7f946dbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9468159f60 con 0x7f946815ade0 2023-10-07T11:03:22.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94680b0540 0x7f946815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:22.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94680b0540 0x7f946815c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49236/0 (socket says 172.21.15.29:49236) 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 -- 172.21.15.29:0/3204621219 learned_addr learned my addr 172.21.15.29:0/3204621219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9466ffd700 1 --2- 172.21.15.29:0/3204621219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f946815ade0 0x7f94680b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 -- 172.21.15.29:0/3204621219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f946815ade0 msgr2=0x7f94680b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 --2- 172.21.15.29:0/3204621219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f946815ade0 0x7f94680b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 -- 172.21.15.29:0/3204621219 >> v1:172.21.15.144:6789/0 conn(0x7f9468153cc0 legacy=0x7f94680b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 -- 172.21.15.29:0/3204621219 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946815a240 con 0x7f94680b0540 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.272+0000 7f9467fff700 1 --2- 172.21.15.29:0/3204621219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94680b0540 0x7f946815c240 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9460009f20 tx=0x7f9460009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234dea7b13836011 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f94667fc700 1 -- 172.21.15.29:0/3204621219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946000e070 con 0x7f94680b0540 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f94667fc700 1 -- 172.21.15.29:0/3204621219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946000b9d0 con 0x7f94680b0540 2023-10-07T11:03:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f94667fc700 1 -- 172.21.15.29:0/3204621219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600092e0 con 0x7f94680b0540 2023-10-07T11:03:22.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 -- 172.21.15.29:0/3204621219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94680b0540 msgr2=0x7f946815c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 --2- 172.21.15.29:0/3204621219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94680b0540 0x7f946815c240 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9460009f20 tx=0x7f9460009840 comp rx=0 tx=0).stop 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 -- 172.21.15.29:0/3204621219 shutdown_connections 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 --2- 172.21.15.29:0/3204621219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94680b0540 0x7f946815c240 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 --2- 172.21.15.29:0/3204621219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f946815ade0 0x7f94680b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 -- 172.21.15.29:0/3204621219 >> 172.21.15.29:0/3204621219 conn(0x7f9468067610 msgr2=0x7f9468067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 -- 172.21.15.29:0/3204621219 shutdown_connections 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.273+0000 7f946dbff700 1 -- 172.21.15.29:0/3204621219 wait complete. 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.275+0000 7f946dbff700 1 Processor -- start 2023-10-07T11:03:22.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.275+0000 7f946dbff700 1 -- start start 2023-10-07T11:03:22.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f946dbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680b0540 0x7f9468153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:22.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f946dbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9468153cc0 0x7f946816ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f946dbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 0x7f946816cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f946dbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468153590 con 0x7f946815ade0 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f946dbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946815c1e0 con 0x7f94680b0540 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f946dbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946815f670 con 0x7f9468153cc0 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f9467fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 0x7f946816cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f9467fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 0x7f946816cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49248/0 (socket says 172.21.15.29:49248) 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f9467fff700 1 -- 172.21.15.29:0/2923703014 learned_addr learned my addr 172.21.15.29:0/2923703014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f94677fe700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680b0540 0x7f9468153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.276+0000 7f9466ffd700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9468153cc0 0x7f946816ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f9467fff700 1 -- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9468153cc0 msgr2=0x7f946816ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:22.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f9467fff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9468153cc0 0x7f946816ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f9467fff700 1 -- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680b0540 msgr2=0x7f9468153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f9467fff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680b0540 0x7f9468153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f9467fff700 1 -- 172.21.15.29:0/2923703014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9460009ae0 con 0x7f946815ade0 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f9467fff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 0x7f946816cfd0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f946000b610 tx=0x7f9460018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460008ad0 con 0x7f946815ade0 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9460008c30 con 0x7f946815ade0 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460026460 con 0x7f946815ade0 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.277+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946815f950 con 0x7f946815ade0 2023-10-07T11:03:22.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.278+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946815ff10 con 0x7f946815ade0 2023-10-07T11:03:22.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.279+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946815c390 con 0x7f946815ade0 2023-10-07T11:03:22.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.280+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9460003040 con 0x7f946815ade0 2023-10-07T11:03:22.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.281+0000 7f944f7fe700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9454066f30 0x7f94540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:22.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.281+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f94600866b0 con 0x7f946815ade0 2023-10-07T11:03:22.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.282+0000 7f94677fe700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9454066f30 0x7f94540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:22.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.282+0000 7f94677fe700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9454066f30 0x7f94540693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9458007fd0 tx=0x7f9458007eb0 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:22.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.285+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9460053780 con 0x7f946815ade0 2023-10-07T11:03:22.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:22.530+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f946815c350 con 0x7f946815ade0 2023-10-07T11:03:23.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.064+0000 7f944f7fe700 1 -- 172.21.15.29:0/2923703014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v98) v1 ==== 174+0+0 (secure 0 0 0) 0x7f946002fe80 con 0x7f946815ade0 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9454066f30 msgr2=0x7f94540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9454066f30 0x7f94540693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9458007fd0 tx=0x7f9458007eb0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 msgr2=0x7f946816cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 0x7f946816cfd0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f946000b610 tx=0x7f9460018000 comp rx=0 tx=0).stop 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 shutdown_connections 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946815ade0 0x7f946816cfd0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680b0540 0x7f9468153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9454066f30 0x7f94540693e0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9458007fd0 tx=0x7f9458007eb0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 --2- 172.21.15.29:0/2923703014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9468153cc0 0x7f946816ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.075+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 >> 172.21.15.29:0/2923703014 conn(0x7f9468067610 msgr2=0x7f94680add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.076+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 shutdown_connections 2023-10-07T11:03:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.076+0000 7f946dbff700 1 -- 172.21.15.29:0/2923703014 wait complete. 2023-10-07T11:03:23.081 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 24 and data pool 25 2023-10-07T11:03:23.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.295+0000 7fab203ff700 1 Processor -- start 2023-10-07T11:03:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.295+0000 7fab203ff700 1 -- start start 2023-10-07T11:03:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.296+0000 7fab203ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab180ad370 0x7fab180cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.296+0000 7fab203ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1814d790 con 0x7fab180b0410 2023-10-07T11:03:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.296+0000 7fab203ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab180ae5d0 con 0x7fab18159ec0 2023-10-07T11:03:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.296+0000 7fab203ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab180ad740 con 0x7fab180ad370 2023-10-07T11:03:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.297+0000 7fab1e99c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab180b0410 0x7fab18156fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57206/0 (socket says 172.21.15.29:57206) 2023-10-07T11:03:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.297+0000 7fab1e99c700 1 -- 172.21.15.29:0/1894396563 learned_addr learned my addr 172.21.15.29:0/1894396563 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.297+0000 7fab1e19b700 1 --2- 172.21.15.29:0/1894396563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab180ad370 0x7fab180cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.297+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399158320 0 0) 0x7fab1814d790 con 0x7fab180b0410 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.297+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab04003570 con 0x7fab180b0410 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518063082 0 0) 0x7fab180ae5d0 con 0x7fab18159ec0 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab1814d790 con 0x7fab18159ec0 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1e19b700 1 -- 172.21.15.29:0/1894396563 >> v1:172.21.15.144:6789/0 conn(0x7fab18159ec0 legacy=0x7fab181568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1e19b700 1 -- 172.21.15.29:0/1894396563 >> v1:172.21.15.29:6789/0 conn(0x7fab180b0410 legacy=0x7fab18156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1e19b700 1 -- 172.21.15.29:0/1894396563 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab180ccdb0 con 0x7fab180ad370 2023-10-07T11:03:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1e19b700 1 --2- 172.21.15.29:0/1894396563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab180ad370 0x7fab180cc870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fab14009910 tx=0x7fab140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=477a80af59440f80 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1400f6b0 con 0x7fab180ad370 2023-10-07T11:03:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1400f810 con 0x7fab180ad370 2023-10-07T11:03:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.298+0000 7fab1d199700 1 -- 172.21.15.29:0/1894396563 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1400fbb0 con 0x7fab180ad370 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 -- 172.21.15.29:0/1894396563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab180ad370 msgr2=0x7fab180cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 --2- 172.21.15.29:0/1894396563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab180ad370 0x7fab180cc870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fab14009910 tx=0x7fab140091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 -- 172.21.15.29:0/1894396563 shutdown_connections 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 --2- 172.21.15.29:0/1894396563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab180ad370 0x7fab180cc870 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 -- 172.21.15.29:0/1894396563 >> 172.21.15.29:0/1894396563 conn(0x7fab180680a0 msgr2=0x7fab180cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 -- 172.21.15.29:0/1894396563 shutdown_connections 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.299+0000 7fab203ff700 1 -- 172.21.15.29:0/1894396563 wait complete. 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.300+0000 7fab203ff700 1 Processor -- start 2023-10-07T11:03:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.301+0000 7fab203ff700 1 -- start start 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.301+0000 7fab203ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab180ad370 0x7fab1816a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.301+0000 7fab203ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 0x7fab180ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.301+0000 7fab203ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab18159ec0 0x7fab1814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab203ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab180cae80 con 0x7fab180b0410 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab203ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1816d5a0 con 0x7fab180ad370 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab203ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1814cf10 con 0x7fab18159ec0 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 0x7fab180ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 0x7fab180ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53194/0 (socket says 172.21.15.29:53194) 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 -- 172.21.15.29:0/3950442033 learned_addr learned my addr 172.21.15.29:0/3950442033 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1e19b700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab180ad370 0x7fab1816a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1e99c700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab18159ec0 0x7fab1814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 -- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab18159ec0 msgr2=0x7fab1814c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab18159ec0 0x7fab1814c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 -- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab180ad370 msgr2=0x7fab1816a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab180ad370 0x7fab1816a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.302+0000 7fab1d99a700 1 -- 172.21.15.29:0/3950442033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14009420 con 0x7fab180b0410 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.303+0000 7fab1d99a700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 0x7fab180ca940 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fab0800c5c0 tx=0x7fab08007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.303+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08010510 con 0x7fab180b0410 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.303+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08010690 con 0x7fab180b0410 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.303+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08010a50 con 0x7fab180b0410 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.303+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1816ccb0 con 0x7fab180b0410 2023-10-07T11:03:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.303+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1816d230 con 0x7fab180b0410 2023-10-07T11:03:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.304+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1816d750 con 0x7fab180b0410 2023-10-07T11:03:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.305+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fab0801b400 con 0x7fab180b0410 2023-10-07T11:03:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.306+0000 7fab0e7fc700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faafc066f30 0x7faafc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.306+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fab080842f0 con 0x7fab180b0410 2023-10-07T11:03:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.307+0000 7fab1e19b700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faafc066f30 0x7faafc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.307+0000 7fab1e19b700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faafc066f30 0x7faafc0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fab140093f0 tx=0x7fab14019000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:23.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.311+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab1816d750 con 0x7fab180b0410 2023-10-07T11:03:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.519+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab1816d750 con 0x7fab180b0410 2023-10-07T11:03:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.520+0000 7fab0e7fc700 1 -- 172.21.15.29:0/3950442033 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16206 (secure 0 0 0) 0x7fab1816d750 con 0x7fab180b0410 2023-10-07T11:03:23.521 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:23.521 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":196,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:22.532318+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:03:20.783910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-07T11:03:21.517835+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faafc066f30 msgr2=0x7faafc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faafc066f30 0x7faafc0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fab140093f0 tx=0x7fab14019000 comp rx=0 tx=0).stop 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 msgr2=0x7fab180ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 0x7fab180ca940 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fab0800c5c0 tx=0x7fab08007dd0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 shutdown_connections 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faafc066f30 0x7faafc0693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab18159ec0 0x7fab1814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab180ad370 0x7fab1816a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 --2- 172.21.15.29:0/3950442033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab180b0410 0x7fab180ca940 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.522+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 >> 172.21.15.29:0/3950442033 conn(0x7fab180680a0 msgr2=0x7fab1806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.523+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 shutdown_connections 2023-10-07T11:03:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.523+0000 7fab203ff700 1 -- 172.21.15.29:0/3950442033 wait complete. 2023-10-07T11:03:23.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.737+0000 7fbfc0dff700 1 Processor -- start 2023-10-07T11:03:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.737+0000 7fbfc0dff700 1 -- start start 2023-10-07T11:03:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.738+0000 7fbfc0dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfbc15ade0 0x7fbfbc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.738+0000 7fbfc0dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc154090 con 0x7fbfbc0b0540 2023-10-07T11:03:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.738+0000 7fbfc0dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc159f60 con 0x7fbfbc153cc0 2023-10-07T11:03:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.738+0000 7fbfc0dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfbc0b09e0 con 0x7fbfbc15ade0 2023-10-07T11:03:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfbc15ade0 0x7fbfbc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfbc15ade0 0x7fbfbc0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53202/0 (socket says 172.21.15.29:53202) 2023-10-07T11:03:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 -- 172.21.15.29:0/3901408012 learned_addr learned my addr 172.21.15.29:0/3901408012 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfbad9d700 1 --1- 172.21.15.29:0/3901408012 >> v1:172.21.15.29:6789/0 conn(0x7fbfbc0b0540 0x7fbfbc15c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57220/0 (socket says 172.21.15.29:57220) 2023-10-07T11:03:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 -- 172.21.15.29:0/3901408012 >> v1:172.21.15.144:6789/0 conn(0x7fbfbc153cc0 legacy=0x7fbfbc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 -- 172.21.15.29:0/3901408012 >> v1:172.21.15.29:6789/0 conn(0x7fbfbc0b0540 legacy=0x7fbfbc15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 -- 172.21.15.29:0/3901408012 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc15a240 con 0x7fbfbc15ade0 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb959a700 1 -- 172.21.15.29:0/3901408012 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259831641 0 0) 0x7fbfbc154090 con 0x7fbfbc0b0540 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.739+0000 7fbfb9d9b700 1 --2- 172.21.15.29:0/3901408012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc0b0000 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00099b0 tx=0x7fbfb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcac62c880649a4d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfb959a700 1 -- 172.21.15.29:0/3901408012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb000b860 con 0x7fbfbc15ade0 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfb959a700 1 -- 172.21.15.29:0/3901408012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb000b9c0 con 0x7fbfbc15ade0 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfb959a700 1 -- 172.21.15.29:0/3901408012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb000bd60 con 0x7fbfbc15ade0 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfc0dff700 1 -- 172.21.15.29:0/3901408012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc15ade0 msgr2=0x7fbfbc0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/3901408012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc0b0000 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00099b0 tx=0x7fbfb0009250 comp rx=0 tx=0).stop 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfc0dff700 1 -- 172.21.15.29:0/3901408012 shutdown_connections 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/3901408012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc0b0000 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfc0dff700 1 -- 172.21.15.29:0/3901408012 >> 172.21.15.29:0/3901408012 conn(0x7fbfbc067610 msgr2=0x7fbfbc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.740+0000 7fbfc0dff700 1 -- 172.21.15.29:0/3901408012 shutdown_connections 2023-10-07T11:03:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.741+0000 7fbfc0dff700 1 -- 172.21.15.29:0/3901408012 wait complete. 2023-10-07T11:03:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.743+0000 7fbfc0dff700 1 Processor -- start 2023-10-07T11:03:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.743+0000 7fbfc0dff700 1 -- start start 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfc0dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 0x7fbfbc16a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfc0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfbc153cc0 0x7fbfbc142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfc0dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfc0dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc16a9d0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfc0dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc154820 con 0x7fbfbc15ade0 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfc0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc154b00 con 0x7fbfbc153cc0 2023-10-07T11:03:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfba59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 0x7fbfbc16a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfba59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 0x7fbfbc16a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53204/0 (socket says 172.21.15.29:53204) 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfba59c700 1 -- 172.21.15.29:0/1882702098 learned_addr learned my addr 172.21.15.29:0/1882702098 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfbad9d700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.744+0000 7fbfb9d9b700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfbc153cc0 0x7fbfbc142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfba59c700 1 -- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfbc153cc0 msgr2=0x7fbfbc142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfba59c700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfbc153cc0 0x7fbfbc142d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfba59c700 1 -- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfbc15ade0 msgr2=0x7fbfbc1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfba59c700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfba59c700 1 -- 172.21.15.29:0/1882702098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb00094f0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfbad9d700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfba59c700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 0x7fbfbc16a490 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c7c0 tx=0x7fbfac007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac011590 con 0x7fbfbc0b0540 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac011710 con 0x7fbfbc0b0540 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac011ad0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.745+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfbc154de0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.746+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfbc1553a0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.747+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfbc04c3c0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.748+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbfac01a360 con 0x7fbfbc0b0540 2023-10-07T11:03:23.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.749+0000 7fbfaa7fc700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbf9c066f30 0x7fbf9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:23.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.749+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fbfac0835f0 con 0x7fbfbc0b0540 2023-10-07T11:03:23.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.749+0000 7fbfb9d9b700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbf9c066f30 0x7fbf9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:23.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.750+0000 7fbfb9d9b700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbf9c066f30 0x7fbf9c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00099b0 tx=0x7fbfb0017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:23.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.752+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfac050730 con 0x7fbfbc0b0540 2023-10-07T11:03:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.957+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfbc15a520 con 0x7fbfbc0b0540 2023-10-07T11:03:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.958+0000 7fbfaa7fc700 1 -- 172.21.15.29:0/1882702098 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16206 (secure 0 0 0) 0x7fbfac053d50 con 0x7fbfbc0b0540 2023-10-07T11:03:23.959 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:23.959 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":196,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:22.532318+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:03:20.783910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-07T11:03:21.517835+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbf9c066f30 msgr2=0x7fbf9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbf9c066f30 0x7fbf9c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00099b0 tx=0x7fbfb0017000 comp rx=0 tx=0).stop 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 msgr2=0x7fbfbc16a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 0x7fbfbc16a490 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c7c0 tx=0x7fbfac007f70 comp rx=0 tx=0).stop 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 shutdown_connections 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfbc15ade0 0x7fbfbc1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfbc0b0540 0x7fbfbc16a490 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fbf9c066f30 0x7fbf9c0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 --2- 172.21.15.29:0/1882702098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfbc153cc0 0x7fbfbc142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 >> 172.21.15.29:0/1882702098 conn(0x7fbfbc067610 msgr2=0x7fbfbc152d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 shutdown_connections 2023-10-07T11:03:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:23.961+0000 7fbfc0dff700 1 -- 172.21.15.29:0/1882702098 wait complete. 2023-10-07T11:03:23.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-07T11:03:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.180+0000 7f07dffff700 1 Processor -- start 2023-10-07T11:03:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.180+0000 7f07dffff700 1 -- start start 2023-10-07T11:03:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07dffff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07d8153cc0 0x7f07d80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07dffff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d8154090 con 0x7f07d80b0710 2023-10-07T11:03:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07dffff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d8159f60 con 0x7f07d815ade0 2023-10-07T11:03:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07dffff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d815a240 con 0x7f07d8153cc0 2023-10-07T11:03:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07ddd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07d8153cc0 0x7f07d80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07ddd9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07d8153cc0 0x7f07d80b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53214/0 (socket says 172.21.15.29:53214) 2023-10-07T11:03:24.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.182+0000 7f07ddd9b700 1 -- 172.21.15.29:0/1084141879 learned_addr learned my addr 172.21.15.29:0/1084141879 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07dcd99700 1 -- 172.21.15.29:0/1084141879 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207495273 0 0) 0x7f07d8159f60 con 0x7f07d815ade0 2023-10-07T11:03:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07ddd9b700 1 -- 172.21.15.29:0/1084141879 >> v1:172.21.15.144:6789/0 conn(0x7f07d815ade0 legacy=0x7f07d80b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07ddd9b700 1 -- 172.21.15.29:0/1084141879 >> v1:172.21.15.29:6789/0 conn(0x7f07d80b0710 legacy=0x7f07d815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07ddd9b700 1 -- 172.21.15.29:0/1084141879 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d815a520 con 0x7f07d8153cc0 2023-10-07T11:03:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07ddd9b700 1 --2- 172.21.15.29:0/1084141879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d80b17c0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009990 tx=0x7f07d4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b94630f47abec5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07dcd99700 1 -- 172.21.15.29:0/1084141879 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4005c10 con 0x7f07d8153cc0 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07dcd99700 1 -- 172.21.15.29:0/1084141879 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d4005d70 con 0x7f07d8153cc0 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.183+0000 7f07dcd99700 1 -- 172.21.15.29:0/1084141879 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d400b860 con 0x7f07d8153cc0 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 -- 172.21.15.29:0/1084141879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 msgr2=0x7f07d80b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 --2- 172.21.15.29:0/1084141879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d80b17c0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009990 tx=0x7f07d4009220 comp rx=0 tx=0).stop 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 -- 172.21.15.29:0/1084141879 shutdown_connections 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 --2- 172.21.15.29:0/1084141879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d80b17c0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 -- 172.21.15.29:0/1084141879 >> 172.21.15.29:0/1084141879 conn(0x7f07d8067610 msgr2=0x7f07d8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 -- 172.21.15.29:0/1084141879 shutdown_connections 2023-10-07T11:03:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.184+0000 7f07dffff700 1 -- 172.21.15.29:0/1084141879 wait complete. 2023-10-07T11:03:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.186+0000 7f07dffff700 1 Processor -- start 2023-10-07T11:03:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.186+0000 7f07dffff700 1 -- start start 2023-10-07T11:03:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d80b0710 0x7f07d80c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d81523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07d815ade0 0x7f07d80ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8158890 con 0x7f07d8153cc0 2023-10-07T11:03:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dffff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80c8dd0 con 0x7f07d80b0710 2023-10-07T11:03:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dffff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80c97a0 con 0x7f07d815ade0 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dd59a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d81523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dd59a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d81523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53222/0 (socket says 172.21.15.29:53222) 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.187+0000 7f07dd59a700 1 -- 172.21.15.29:0/2214739421 learned_addr learned my addr 172.21.15.29:0/2214739421 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07ddd9b700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d80b0710 0x7f07d80c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07de59c700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07d815ade0 0x7f07d80ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dd59a700 1 -- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07d815ade0 msgr2=0x7f07d80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dd59a700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07d815ade0 0x7f07d80ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dd59a700 1 -- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d80b0710 msgr2=0x7f07d80c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dd59a700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d80b0710 0x7f07d80c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dd59a700 1 -- 172.21.15.29:0/2214739421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d4009480 con 0x7f07d8153cc0 2023-10-07T11:03:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07de59c700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07d815ade0 0x7f07d80ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dd59a700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d81523a0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f07c800c530 tx=0x7f07c8007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8011590 con 0x7f07d8153cc0 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c8011710 con 0x7f07d8153cc0 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80c81c0 con 0x7f07d8153cc0 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8011ad0 con 0x7f07d8153cc0 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.188+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80c8780 con 0x7f07d8153cc0 2023-10-07T11:03:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.189+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d804c3c0 con 0x7f07d8153cc0 2023-10-07T11:03:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.190+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f07c801b3b0 con 0x7f07d8153cc0 2023-10-07T11:03:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.192+0000 7f07cdffb700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f07bc066f30 0x7f07bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.192+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f07c8083c60 con 0x7f07d8153cc0 2023-10-07T11:03:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.192+0000 7f07ddd9b700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f07bc066f30 0x7f07bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.193+0000 7f07ddd9b700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f07bc066f30 0x7f07bc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009260 tx=0x7f07d4016000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.196+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07c8050cf0 con 0x7f07d8153cc0 2023-10-07T11:03:24.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:24.448+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f07d815a800 con 0x7f07d8153cc0 2023-10-07T11:03:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.067+0000 7f07cdffb700 1 -- 172.21.15.29:0/2214739421 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v100) v1 ==== 118+0+0 (secure 0 0 0) 0x7f07c8017070 con 0x7f07d8153cc0 2023-10-07T11:03:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.070+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f07bc066f30 msgr2=0x7f07bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.070+0000 7f07dffff700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f07bc066f30 0x7f07bc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009260 tx=0x7f07d4016000 comp rx=0 tx=0).stop 2023-10-07T11:03:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.070+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 msgr2=0x7f07d81523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.070+0000 7f07dffff700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d81523a0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f07c800c530 tx=0x7f07c8007f20 comp rx=0 tx=0).stop 2023-10-07T11:03:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 shutdown_connections 2023-10-07T11:03:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f07bc066f30 0x7f07bc0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07d815ade0 0x7f07d80ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d80b0710 0x7f07d80c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 --2- 172.21.15.29:0/2214739421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d8153cc0 0x7f07d81523a0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 >> 172.21.15.29:0/2214739421 conn(0x7f07d8067610 msgr2=0x7f07d80adcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 shutdown_connections 2023-10-07T11:03:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.071+0000 7f07dffff700 1 -- 172.21.15.29:0/2214739421 wait complete. 2023-10-07T11:03:25.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-07T11:03:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.294+0000 7f67c6bff700 1 Processor -- start 2023-10-07T11:03:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.294+0000 7f67c6bff700 1 -- start start 2023-10-07T11:03:25.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.295+0000 7f67c6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c015ade0 0x7f67c00b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:25.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.295+0000 7f67c6bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c00b0980 con 0x7f67c0153cc0 2023-10-07T11:03:25.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.295+0000 7f67c6bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c0154090 con 0x7f67c00b0540 2023-10-07T11:03:25.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.295+0000 7f67c6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0159f60 con 0x7f67c015ade0 2023-10-07T11:03:25.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67c499b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f67c0153cc0 0x7f67c00b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57232/0 (socket says 172.21.15.29:57232) 2023-10-07T11:03:25.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67c499b700 1 -- 172.21.15.29:0/459819928 learned_addr learned my addr 172.21.15.29:0/459819928 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:25.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bffff700 1 --2- 172.21.15.29:0/459819928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c015ade0 0x7f67c00b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:25.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458164368 0 0) 0x7f67c00b0980 con 0x7f67c0153cc0 2023-10-07T11:03:25.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b0003570 con 0x7f67c0153cc0 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725392311 0 0) 0x7f67c0154090 con 0x7f67c00b0540 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c00b0980 con 0x7f67c00b0540 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bffff700 1 -- 172.21.15.29:0/459819928 >> v1:172.21.15.144:6789/0 conn(0x7f67c00b0540 legacy=0x7f67c015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bffff700 1 -- 172.21.15.29:0/459819928 >> v1:172.21.15.29:6789/0 conn(0x7f67c0153cc0 legacy=0x7f67c00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.296+0000 7f67bffff700 1 -- 172.21.15.29:0/459819928 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c015a240 con 0x7f67c015ade0 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.297+0000 7f67bffff700 1 --2- 172.21.15.29:0/459819928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c015ade0 0x7f67c00b0000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009910 tx=0x7f67ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e4668fc681547d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.297+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac011630 con 0x7f67c015ade0 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.297+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ac011790 con 0x7f67c015ade0 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.297+0000 7f67bf7fe700 1 -- 172.21.15.29:0/459819928 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac011b10 con 0x7f67c015ade0 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 -- 172.21.15.29:0/459819928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c015ade0 msgr2=0x7f67c00b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 --2- 172.21.15.29:0/459819928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c015ade0 0x7f67c00b0000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009910 tx=0x7f67ac0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 -- 172.21.15.29:0/459819928 shutdown_connections 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 --2- 172.21.15.29:0/459819928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c015ade0 0x7f67c00b0000 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 -- 172.21.15.29:0/459819928 >> 172.21.15.29:0/459819928 conn(0x7f67c0067610 msgr2=0x7f67c0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 -- 172.21.15.29:0/459819928 shutdown_connections 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.298+0000 7f67c6bff700 1 -- 172.21.15.29:0/459819928 wait complete. 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.299+0000 7f67c6bff700 1 Processor -- start 2023-10-07T11:03:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 -- start start 2023-10-07T11:03:25.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c00b0540 0x7f67c0152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c0153cc0 0x7f67c0157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 0x7f67c00c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0169d60 con 0x7f67c015ade0 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0152990 con 0x7f67c00b0540 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.300+0000 7f67c6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01553f0 con 0x7f67c0153cc0 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 0x7f67c00c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 0x7f67c00c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53226/0 (socket says 172.21.15.29:53226) 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 -- 172.21.15.29:0/1464773016 learned_addr learned my addr 172.21.15.29:0/1464773016 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c499b700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c00b0540 0x7f67c0152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67bffff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c0153cc0 0x7f67c0157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 -- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c0153cc0 msgr2=0x7f67c0157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c0153cc0 0x7f67c0157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 -- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c00b0540 msgr2=0x7f67c0152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c00b0540 0x7f67c0152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 -- 172.21.15.29:0/1464773016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac009420 con 0x7f67c015ade0 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c499b700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c00b0540 0x7f67c0152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.301+0000 7f67c519c700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 0x7f67c00c86d0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f67b800c910 tx=0x7f67b80027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.302+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8012630 con 0x7f67c015ade0 2023-10-07T11:03:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.302+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b80127b0 con 0x7f67c015ade0 2023-10-07T11:03:25.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.302+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8012b70 con 0x7f67c015ade0 2023-10-07T11:03:25.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.302+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01556d0 con 0x7f67c015ade0 2023-10-07T11:03:25.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.302+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c0155c90 con 0x7f67c015ade0 2023-10-07T11:03:25.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.303+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c004c3c0 con 0x7f67c015ade0 2023-10-07T11:03:25.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.304+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f67b801b400 con 0x7f67c015ade0 2023-10-07T11:03:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.305+0000 7f67bcff9700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f67a4066f30 0x7f67a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.305+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f67b8084ad0 con 0x7f67c015ade0 2023-10-07T11:03:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.305+0000 7f67c499b700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f67a4066f30 0x7f67a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.306+0000 7f67c499b700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f67a4066f30 0x7f67a40693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f67b4001360 tx=0x7f67b4009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:25.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.309+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67b8051c10 con 0x7f67c015ade0 2023-10-07T11:03:25.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:25.553+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f67c015a520 con 0x7f67c015ade0 2023-10-07T11:03:26.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.070+0000 7f67bcff9700 1 -- 172.21.15.29:0/1464773016 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v101) v1 ==== 122+0+0 (secure 0 0 0) 0x7f67b8055230 con 0x7f67c015ade0 2023-10-07T11:03:26.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f67a4066f30 msgr2=0x7f67a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f67a4066f30 0x7f67a40693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f67b4001360 tx=0x7f67b4009040 comp rx=0 tx=0).stop 2023-10-07T11:03:26.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 msgr2=0x7f67c00c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 0x7f67c00c86d0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f67b800c910 tx=0x7f67b80027a0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 shutdown_connections 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67c015ade0 0x7f67c00c86d0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c00b0540 0x7f67c0152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f67a4066f30 0x7f67a40693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 --2- 172.21.15.29:0/1464773016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67c0153cc0 0x7f67c0157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.073+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 >> 172.21.15.29:0/1464773016 conn(0x7f67c0067610 msgr2=0x7f67c0152c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.074+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 shutdown_connections 2023-10-07T11:03:26.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.074+0000 7f67c6bff700 1 -- 172.21.15.29:0/1464773016 wait complete. 2023-10-07T11:03:26.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:26.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.290+0000 7f1a747ff700 1 Processor -- start 2023-10-07T11:03:26.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.290+0000 7f1a747ff700 1 -- start start 2023-10-07T11:03:26.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a747ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159750 0x7f1a6c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a747ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1a6c159b20 0x7f1a6c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a747ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c0cbf20 con 0x7f1a6c0ce6d0 2023-10-07T11:03:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a747ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0cc200 con 0x7f1a6c159b20 2023-10-07T11:03:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a747ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c0cc4e0 con 0x7f1a6c159750 2023-10-07T11:03:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a72d9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a6c0ce6d0 0x7f1a6c15d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57248/0 (socket says 172.21.15.29:57248) 2023-10-07T11:03:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a72d9c700 1 -- 172.21.15.29:0/1563521413 learned_addr learned my addr 172.21.15.29:0/1563521413 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:26.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a7259b700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159750 0x7f1a6c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a71d9a700 1 --2- 172.21.15.29:0/1563521413 >> v2:172.21.15.144:3300/0 conn(0x7f1a6c159b20 0x7f1a6c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.292+0000 7f1a71599700 1 -- 172.21.15.29:0/1563521413 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704293521 0 0) 0x7f1a6c0cbf20 con 0x7f1a6c0ce6d0 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71599700 1 -- 172.21.15.29:0/1563521413 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58003570 con 0x7f1a6c0ce6d0 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159750 msgr2=0x7f1a6c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71d9a700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159750 0x7f1a6c0d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1563521413 >> v1:172.21.15.29:6789/0 conn(0x7f1a6c0ce6d0 legacy=0x7f1a6c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1563521413 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0cc7c0 con 0x7f1a6c159b20 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a7259b700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159750 0x7f1a6c0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71d9a700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c159b20 0x7f1a6c0ce190 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0098e0 tx=0x7f1a5c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acfc5ad3236ef89f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71599700 1 -- 172.21.15.29:0/1563521413 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c011630 con 0x7f1a6c159b20 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71599700 1 -- 172.21.15.29:0/1563521413 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c011790 con 0x7f1a6c159b20 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.293+0000 7f1a71599700 1 -- 172.21.15.29:0/1563521413 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c011b10 con 0x7f1a6c159b20 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 -- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c159b20 msgr2=0x7f1a6c0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c159b20 0x7f1a6c0ce190 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0098e0 tx=0x7f1a5c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 -- 172.21.15.29:0/1563521413 shutdown_connections 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159750 0x7f1a6c0d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1563521413 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c159b20 0x7f1a6c0ce190 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 -- 172.21.15.29:0/1563521413 >> 172.21.15.29:0/1563521413 conn(0x7f1a6c063a10 msgr2=0x7f1a6c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 -- 172.21.15.29:0/1563521413 shutdown_connections 2023-10-07T11:03:26.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.294+0000 7f1a747ff700 1 -- 172.21.15.29:0/1563521413 wait complete. 2023-10-07T11:03:26.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.295+0000 7f1a747ff700 1 Processor -- start 2023-10-07T11:03:26.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.296+0000 7f1a747ff700 1 -- start start 2023-10-07T11:03:26.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.296+0000 7f1a747ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c0ce6d0 0x7f1a6c15f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.296+0000 7f1a747ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 0x7f1a6c153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.296+0000 7f1a747ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a747ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15aba0 con 0x7f1a6c159750 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a747ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c153f60 con 0x7f1a6c0ce6d0 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a747ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15f750 con 0x7f1a6c159b20 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 0x7f1a6c153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 0x7f1a6c153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53230/0 (socket says 172.21.15.29:53230) 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1539831836 learned_addr learned my addr 172.21.15.29:0/1539831836 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a72d9c700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a7259b700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c0ce6d0 0x7f1a6c15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 msgr2=0x7f1a6c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c0ce6d0 msgr2=0x7f1a6c15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c0ce6d0 0x7f1a6c15f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a71d9a700 1 -- 172.21.15.29:0/1539831836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a680093c0 con 0x7f1a6c159750 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a7259b700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c0ce6d0 0x7f1a6c15f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.297+0000 7f1a72d9c700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:26.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.298+0000 7f1a71d9a700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 0x7f1a6c153a20 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c088a40 tx=0x7f1a5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.298+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c011b10 con 0x7f1a6c159750 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.298+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c01a430 con 0x7f1a6c159750 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.298+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c009420 con 0x7f1a6c159750 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.298+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c01a7f0 con 0x7f1a6c159750 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.298+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c15fd10 con 0x7f1a6c159750 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.299+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c04c3c0 con 0x7f1a6c159750 2023-10-07T11:03:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.300+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1a5c01aba0 con 0x7f1a6c159750 2023-10-07T11:03:26.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.301+0000 7f1a627fc700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1a50066f30 0x7f1a500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.302+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f1a5c0864b0 con 0x7f1a6c159750 2023-10-07T11:03:26.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.302+0000 7f1a7259b700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1a50066f30 0x7f1a500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.303+0000 7f1a7259b700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1a50066f30 0x7f1a500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009390 tx=0x7f1a68008040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.306+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a5c0534a0 con 0x7f1a6c159750 2023-10-07T11:03:26.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.556+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a6c15ace0 con 0x7f1a6c159750 2023-10-07T11:03:26.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.557+0000 7f1a627fc700 1 -- 172.21.15.29:0/1539831836 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4609 (secure 0 0 0) 0x7f1a5c056ac0 con 0x7f1a6c159750 2023-10-07T11:03:26.558 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:03:22.532644+0000","modified":"2023-10-07T11:03:26.067213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10104},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10104":{"gid":10104,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/1070134873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1070134873},{"type":"v1","addr":"172.21.15.29:6834","nonce":1070134873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.559+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1a50066f30 msgr2=0x7f1a500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.559+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1a50066f30 0x7f1a500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009390 tx=0x7f1a68008040 comp rx=0 tx=0).stop 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.559+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 msgr2=0x7f1a6c153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.559+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 0x7f1a6c153a20 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c088a40 tx=0x7f1a5c017000 comp rx=0 tx=0).stop 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 shutdown_connections 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1a50066f30 0x7f1a500693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c0ce6d0 0x7f1a6c15f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 --2- 172.21.15.29:0/1539831836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a6c159750 0x7f1a6c153a20 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 >> 172.21.15.29:0/1539831836 conn(0x7f1a6c063a10 msgr2=0x7f1a6c0cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 shutdown_connections 2023-10-07T11:03:26.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.560+0000 7f1a747ff700 1 -- 172.21.15.29:0/1539831836 wait complete. 2023-10-07T11:03:26.562 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 101 2023-10-07T11:03:26.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:26.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.773+0000 7f47a23ff700 1 Processor -- start 2023-10-07T11:03:26.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.773+0000 7f47a23ff700 1 -- start start 2023-10-07T11:03:26.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.774+0000 7f47a23ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f479c159ec0 0x7f479c15a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.774+0000 7f47a23ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f479c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.774+0000 7f47a23ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479c14d790 con 0x7f479c0ad370 2023-10-07T11:03:26.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.774+0000 7f47a23ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c15a880 con 0x7f479c159ec0 2023-10-07T11:03:26.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.774+0000 7f47a23ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0ae5d0 con 0x7f479c0af400 2023-10-07T11:03:26.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.774+0000 7f479bfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f479c0ad370 0x7f479c0cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57264/0 (socket says 172.21.15.29:57264) 2023-10-07T11:03:26.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479bfff700 1 -- 172.21.15.29:0/3773772570 learned_addr learned my addr 172.21.15.29:0/3773772570 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479b7fe700 1 --2- 172.21.15.29:0/3773772570 >> v2:172.21.15.144:3300/0 conn(0x7f479c159ec0 0x7f479c15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f47a099c700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f479c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995219409 0 0) 0x7f479c14d790 con 0x7f479c0ad370 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4784003570 con 0x7f479c0ad370 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944841706 0 0) 0x7f4784003570 con 0x7f479c0ad370 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 msgr2=0x7f479c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f479c0af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 >> v2:172.21.15.144:3300/0 conn(0x7f479c159ec0 msgr2=0x7f479c15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 --2- 172.21.15.29:0/3773772570 >> v2:172.21.15.144:3300/0 conn(0x7f479c159ec0 0x7f479c15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0ad740 con 0x7f479c0ad370 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f478c001470 con 0x7f479c0ad370 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 >> v1:172.21.15.29:6789/0 conn(0x7f479c0ad370 legacy=0x7f479c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f4784003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47840040b0 0x7f4784004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4784004d50 0x7f47840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.775+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c15a880 con 0x7f4784004d50 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0ae5d0 con 0x7f47840040b0 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478400dd70 con 0x7f479c0af400 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4784004d50 0x7f47840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f479bfff700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f4784003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f479b7fe700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47840040b0 0x7f4784004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 -- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 msgr2=0x7f4784003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f4784003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 -- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47840040b0 msgr2=0x7f4784004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47840040b0 0x7f4784004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 -- 172.21.15.29:0/3773772570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784003570 con 0x7f4784004d50 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f479bfff700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c0af400 0x7f4784003ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.776+0000 7f47a099c700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4784004d50 0x7f47840051c0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f47940094d0 tx=0x7f479400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d1f623bda5fdee server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479400ea90 con 0x7f4784004d50 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479400f040 con 0x7f4784004d50 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f479affd700 1 -- 172.21.15.29:0/3773772570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479400ee20 con 0x7f4784004d50 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a23ff700 1 -- 172.21.15.29:0/3773772570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4784004d50 msgr2=0x7f47840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a23ff700 1 --2- 172.21.15.29:0/3773772570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4784004d50 0x7f47840051c0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f47940094d0 tx=0x7f479400b040 comp rx=0 tx=0).stop 2023-10-07T11:03:26.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a099c700 1 -- 172.21.15.29:0/3773772570 reap_dead start 2023-10-07T11:03:26.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a23ff700 1 -- 172.21.15.29:0/3773772570 shutdown_connections 2023-10-07T11:03:26.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a23ff700 1 -- 172.21.15.29:0/3773772570 >> 172.21.15.29:0/3773772570 conn(0x7f479c0680a0 msgr2=0x7f479c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:26.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a23ff700 1 -- 172.21.15.29:0/3773772570 shutdown_connections 2023-10-07T11:03:26.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.777+0000 7f47a23ff700 1 -- 172.21.15.29:0/3773772570 wait complete. 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.779+0000 7f47a23ff700 1 Processor -- start 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.780+0000 7f47a23ff700 1 -- start start 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a23ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4784004d50 0x7f479c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a23ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 0x7f479c149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a23ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c157a10 0x7f479c157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a23ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c1583d0 con 0x7f479c0a68e0 2023-10-07T11:03:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a23ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0a7d00 con 0x7f4784004d50 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a23ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0a8190 con 0x7f479c157a10 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 0x7f479c149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 0x7f479c149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53246/0 (socket says 172.21.15.29:53246) 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 -- 172.21.15.29:0/1290178128 learned_addr learned my addr 172.21.15.29:0/1290178128 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479bfff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4784004d50 0x7f479c0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f47a099c700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c157a10 0x7f479c157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 -- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c157a10 msgr2=0x7f479c157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c157a10 0x7f479c157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 -- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4784004d50 msgr2=0x7f479c0a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4784004d50 0x7f479c0a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.781+0000 7f479b7fe700 1 -- 172.21.15.29:0/1290178128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0a70f0 con 0x7f479c0a68e0 2023-10-07T11:03:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.782+0000 7f479b7fe700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 0x7f479c149e80 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f4790011750 tx=0x7f4790008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.782+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790015670 con 0x7f479c0a68e0 2023-10-07T11:03:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.782+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47900157d0 con 0x7f479c0a68e0 2023-10-07T11:03:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.782+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790015b70 con 0x7f479c0a68e0 2023-10-07T11:03:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.782+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c0a73d0 con 0x7f479c0a68e0 2023-10-07T11:03:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.782+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c0a7990 con 0x7f479c0a68e0 2023-10-07T11:03:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.783+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c0a7e90 con 0x7f479c0a68e0 2023-10-07T11:03:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.784+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4790013410 con 0x7f479c0a68e0 2023-10-07T11:03:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.785+0000 7f4783fff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4788066f30 0x7f47880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.785+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f4790084ed0 con 0x7f479c0a68e0 2023-10-07T11:03:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.786+0000 7f479bfff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4788066f30 0x7f47880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.786+0000 7f479bfff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4788066f30 0x7f47880693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f478c001010 tx=0x7f478c009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.790+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47900559f0 con 0x7f479c0a68e0 2023-10-07T11:03:26.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.993+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f479c04c3c0 con 0x7f479c0a68e0 2023-10-07T11:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.993+0000 7f4783fff700 1 -- 172.21.15.29:0/1290178128 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16193 (secure 0 0 0) 0x7f4790055580 con 0x7f479c0a68e0 2023-10-07T11:03:26.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:26.995 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":197,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:24.043861+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:03:20.783910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-07T11:03:21.517835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4788066f30 msgr2=0x7f47880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4788066f30 0x7f47880693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f478c001010 tx=0x7f478c009040 comp rx=0 tx=0).stop 2023-10-07T11:03:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 msgr2=0x7f479c149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 0x7f479c149e80 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f4790011750 tx=0x7f4790008040 comp rx=0 tx=0).stop 2023-10-07T11:03:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 shutdown_connections 2023-10-07T11:03:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4788066f30 0x7f47880693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479c157a10 0x7f479c157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4784004d50 0x7f479c0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 --2- 172.21.15.29:0/1290178128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f479c0a68e0 0x7f479c149e80 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 >> 172.21.15.29:0/1290178128 conn(0x7f479c0680a0 msgr2=0x7f479c14bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 shutdown_connections 2023-10-07T11:03:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:26.996+0000 7f47a23ff700 1 -- 172.21.15.29:0/1290178128 wait complete. 2023-10-07T11:03:27.015 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-07T11:03:27.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:03:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.212+0000 7f9ad9dff700 1 Processor -- start 2023-10-07T11:03:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.212+0000 7f9ad9dff700 1 -- start start 2023-10-07T11:03:27.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad9dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ad4159ec0 0x7f9ad415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40af400 0x7f9ad40af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad9dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad414d790 con 0x7f9ad40ad370 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad9dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad415a880 con 0x7f9ad4159ec0 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40ae5d0 con 0x7f9ad40af400 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ad4159ec0 0x7f9ad415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ad4159ec0 0x7f9ad415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53252/0 (socket says 172.21.15.29:53252) 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 -- 172.21.15.29:0/2528357525 learned_addr learned my addr 172.21.15.29:0/2528357525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad37fe700 1 --1- 172.21.15.29:0/2528357525 >> v1:172.21.15.29:6789/0 conn(0x7f9ad40ad370 0x7f9ad40cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57280/0 (socket says 172.21.15.29:57280) 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad3fff700 1 --2- 172.21.15.29:0/2528357525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40af400 0x7f9ad40af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 -- 172.21.15.29:0/2528357525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40af400 msgr2=0x7f9ad40af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 --2- 172.21.15.29:0/2528357525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40af400 0x7f9ad40af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 -- 172.21.15.29:0/2528357525 >> v1:172.21.15.29:6789/0 conn(0x7f9ad40ad370 legacy=0x7f9ad40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.214+0000 7f9ad2ffd700 1 -- 172.21.15.29:0/2528357525 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40ad740 con 0x7f9ad4159ec0 2023-10-07T11:03:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.215+0000 7f9ad2ffd700 1 --2- 172.21.15.29:0/2528357525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad415a340 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f9ac80099b0 tx=0x7f9ac8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b043e1adb2a964 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.215+0000 7f9ad27fc700 1 -- 172.21.15.29:0/2528357525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac800b860 con 0x7f9ad4159ec0 2023-10-07T11:03:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.215+0000 7f9ad27fc700 1 -- 172.21.15.29:0/2528357525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac800b9c0 con 0x7f9ad4159ec0 2023-10-07T11:03:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.215+0000 7f9ad27fc700 1 -- 172.21.15.29:0/2528357525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac800bd60 con 0x7f9ad4159ec0 2023-10-07T11:03:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.215+0000 7f9ad9dff700 1 -- 172.21.15.29:0/2528357525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 msgr2=0x7f9ad415a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.215+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/2528357525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad415a340 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f9ac80099b0 tx=0x7f9ac8009250 comp rx=0 tx=0).stop 2023-10-07T11:03:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.216+0000 7f9ad9dff700 1 -- 172.21.15.29:0/2528357525 shutdown_connections 2023-10-07T11:03:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.216+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/2528357525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40af400 0x7f9ad40af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.216+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/2528357525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad415a340 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.216+0000 7f9ad9dff700 1 -- 172.21.15.29:0/2528357525 >> 172.21.15.29:0/2528357525 conn(0x7f9ad40680a0 msgr2=0x7f9ad40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:27.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.216+0000 7f9ad9dff700 1 -- 172.21.15.29:0/2528357525 shutdown_connections 2023-10-07T11:03:27.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.216+0000 7f9ad9dff700 1 -- 172.21.15.29:0/2528357525 wait complete. 2023-10-07T11:03:27.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.217+0000 7f9ad9dff700 1 Processor -- start 2023-10-07T11:03:27.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.217+0000 7f9ad9dff700 1 -- start start 2023-10-07T11:03:27.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.218+0000 7f9ad9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40ad370 0x7f9ad4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.218+0000 7f9ad9dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ad40af400 0x7f9ad414f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.218+0000 7f9ad9dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad414f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.218+0000 7f9ad9dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40caf00 con 0x7f9ad4159ec0 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.218+0000 7f9ad9dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40cb1e0 con 0x7f9ad40af400 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.218+0000 7f9ad9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40cb4c0 con 0x7f9ad40ad370 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad414f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad414f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53258/0 (socket says 172.21.15.29:53258) 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 -- 172.21.15.29:0/3749095332 learned_addr learned my addr 172.21.15.29:0/3749095332 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:27.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad37fe700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40ad370 0x7f9ad4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad2ffd700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ad40af400 0x7f9ad414f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 -- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40ad370 msgr2=0x7f9ad4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40ad370 0x7f9ad4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 -- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ad40af400 msgr2=0x7f9ad414f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ad40af400 0x7f9ad414f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 -- 172.21.15.29:0/3749095332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80094f0 con 0x7f9ad4159ec0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9ad3fff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad414f940 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009fa0 tx=0x7f9acc009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.219+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00ed20 con 0x7f9ad4159ec0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.220+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc00ee80 con 0x7f9ad4159ec0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.220+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc014670 con 0x7f9ad4159ec0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.220+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad40cb7a0 con 0x7f9ad4159ec0 2023-10-07T11:03:27.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.220+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad4150060 con 0x7f9ad4159ec0 2023-10-07T11:03:27.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.221+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4159d00 con 0x7f9ad4159ec0 2023-10-07T11:03:27.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.222+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9acc0149e0 con 0x7f9ad4159ec0 2023-10-07T11:03:27.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.223+0000 7f9abb7fe700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9ac0066f30 0x7f9ac00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.223+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f9acc084fa0 con 0x7f9ad4159ec0 2023-10-07T11:03:27.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.223+0000 7f9ad37fe700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9ac0066f30 0x7f9ac00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.224+0000 7f9ad37fe700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9ac0066f30 0x7f9ac00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4007fd0 tx=0x7f9ac4007d70 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:27.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.227+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9acc051f90 con 0x7f9ad4159ec0 2023-10-07T11:03:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.423+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ad404c3c0 con 0x7f9ac0066f30 2023-10-07T11:03:27.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.434+0000 7f9abb7fe700 1 -- 172.21.15.29:0/3749095332 <== mgr.8429 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282860 (secure 0 0 0) 0x7f9ad404c3c0 con 0x7f9ac0066f30 2023-10-07T11:03:27.439 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9ac0066f30 msgr2=0x7f9ac00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9ac0066f30 0x7f9ac00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4007fd0 tx=0x7f9ac4007d70 comp rx=0 tx=0).stop 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 msgr2=0x7f9ad414f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad414f940 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009fa0 tx=0x7f9acc009900 comp rx=0 tx=0).stop 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 shutdown_connections 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ad4159ec0 0x7f9ad414f940 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9ac0066f30 0x7f9ac00693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ad40ad370 0x7f9ad4148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 --2- 172.21.15.29:0/3749095332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ad40af400 0x7f9ad414f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 >> 172.21.15.29:0/3749095332 conn(0x7f9ad40680a0 msgr2=0x7f9ad40abb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 shutdown_connections 2023-10-07T11:03:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.439+0000 7f9ad9dff700 1 -- 172.21.15.29:0/3749095332 wait complete. 2023-10-07T11:03:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:03:27.457 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T11:03:26.651670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591904,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":592,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":361,"ondisk_log_size":361,"up":118,"acting":118,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2476344,"kb_used_data":20216,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752498376,"statfs":{"total":773094113280,"available":770558337024,"internally_reserved":0,"allocated":20701184,"data_stored":10830568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1536,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.623060"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.061273+0000","last_change":"2023-10-07T11:03:23.059262+0000","last_active":"2023-10-07T11:03:23.061273+0000","last_peered":"2023-10-07T11:03:23.061273+0000","last_clean":"2023-10-07T11:03:23.061273+0000","last_became_active":"2023-10-07T11:03:23.058996+0000","last_became_peered":"2023-10-07T11:03:23.058996+0000","last_unstale":"2023-10-07T11:03:23.061273+0000","last_undegraded":"2023-10-07T11:03:23.061273+0000","last_fullsized":"2023-10-07T11:03:23.061273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069205+0000","last_change":"2023-10-07T11:03:23.067743+0000","last_active":"2023-10-07T11:03:23.069205+0000","last_peered":"2023-10-07T11:03:23.069205+0000","last_clean":"2023-10-07T11:03:23.069205+0000","last_became_active":"2023-10-07T11:03:23.067464+0000","last_became_peered":"2023-10-07T11:03:23.067464+0000","last_unstale":"2023-10-07T11:03:23.069205+0000","last_undegraded":"2023-10-07T11:03:23.069205+0000","last_fullsized":"2023-10-07T11:03:23.069205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.077562+0000","last_change":"2023-10-07T11:03:23.076006+0000","last_active":"2023-10-07T11:03:23.077562+0000","last_peered":"2023-10-07T11:03:23.077562+0000","last_clean":"2023-10-07T11:03:23.077562+0000","last_became_active":"2023-10-07T11:03:23.075758+0000","last_became_peered":"2023-10-07T11:03:23.075758+0000","last_unstale":"2023-10-07T11:03:23.077562+0000","last_undegraded":"2023-10-07T11:03:23.077562+0000","last_fullsized":"2023-10-07T11:03:23.077562+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.066075+0000","last_change":"2023-10-07T11:03:23.058595+0000","last_active":"2023-10-07T11:03:23.066075+0000","last_peered":"2023-10-07T11:03:23.066075+0000","last_clean":"2023-10-07T11:03:23.066075+0000","last_became_active":"2023-10-07T11:03:23.058332+0000","last_became_peered":"2023-10-07T11:03:23.058332+0000","last_unstale":"2023-10-07T11:03:23.066075+0000","last_undegraded":"2023-10-07T11:03:23.066075+0000","last_fullsized":"2023-10-07T11:03:23.066075+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.062893+0000","last_change":"2023-10-07T11:03:23.057796+0000","last_active":"2023-10-07T11:03:23.062893+0000","last_peered":"2023-10-07T11:03:23.062893+0000","last_clean":"2023-10-07T11:03:23.062893+0000","last_became_active":"2023-10-07T11:03:23.057545+0000","last_became_peered":"2023-10-07T11:03:23.057545+0000","last_unstale":"2023-10-07T11:03:23.062893+0000","last_undegraded":"2023-10-07T11:03:23.062893+0000","last_fullsized":"2023-10-07T11:03:23.062893+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072967+0000","last_change":"2023-10-07T11:03:23.056891+0000","last_active":"2023-10-07T11:03:23.072967+0000","last_peered":"2023-10-07T11:03:23.072967+0000","last_clean":"2023-10-07T11:03:23.072967+0000","last_became_active":"2023-10-07T11:03:23.056623+0000","last_became_peered":"2023-10-07T11:03:23.056623+0000","last_unstale":"2023-10-07T11:03:23.072967+0000","last_undegraded":"2023-10-07T11:03:23.072967+0000","last_fullsized":"2023-10-07T11:03:23.072967+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:24.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065743+0000","last_change":"2023-10-07T11:03:23.056041+0000","last_active":"2023-10-07T11:03:23.065743+0000","last_peered":"2023-10-07T11:03:23.065743+0000","last_clean":"2023-10-07T11:03:23.065743+0000","last_became_active":"2023-10-07T11:03:23.055408+0000","last_became_peered":"2023-10-07T11:03:23.055408+0000","last_unstale":"2023-10-07T11:03:23.065743+0000","last_undegraded":"2023-10-07T11:03:23.065743+0000","last_fullsized":"2023-10-07T11:03:23.065743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072460+0000","last_change":"2023-10-07T11:03:23.070205+0000","last_active":"2023-10-07T11:03:23.072460+0000","last_peered":"2023-10-07T11:03:23.072460+0000","last_clean":"2023-10-07T11:03:23.072460+0000","last_became_active":"2023-10-07T11:03:23.069898+0000","last_became_peered":"2023-10-07T11:03:23.069898+0000","last_unstale":"2023-10-07T11:03:23.072460+0000","last_undegraded":"2023-10-07T11:03:23.072460+0000","last_fullsized":"2023-10-07T11:03:23.072460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074629+0000","last_change":"2023-10-07T11:03:23.059218+0000","last_active":"2023-10-07T11:03:23.074629+0000","last_peered":"2023-10-07T11:03:23.074629+0000","last_clean":"2023-10-07T11:03:23.074629+0000","last_became_active":"2023-10-07T11:03:23.058961+0000","last_became_peered":"2023-10-07T11:03:23.058961+0000","last_unstale":"2023-10-07T11:03:23.074629+0000","last_undegraded":"2023-10-07T11:03:23.074629+0000","last_fullsized":"2023-10-07T11:03:23.074629+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:09.706205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069344+0000","last_change":"2023-10-07T11:03:23.068010+0000","last_active":"2023-10-07T11:03:23.069344+0000","last_peered":"2023-10-07T11:03:23.069344+0000","last_clean":"2023-10-07T11:03:23.069344+0000","last_became_active":"2023-10-07T11:03:23.067653+0000","last_became_peered":"2023-10-07T11:03:23.067653+0000","last_unstale":"2023-10-07T11:03:23.069344+0000","last_undegraded":"2023-10-07T11:03:23.069344+0000","last_fullsized":"2023-10-07T11:03:23.069344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065837+0000","last_change":"2023-10-07T11:03:23.064318+0000","last_active":"2023-10-07T11:03:23.065837+0000","last_peered":"2023-10-07T11:03:23.065837+0000","last_clean":"2023-10-07T11:03:23.065837+0000","last_became_active":"2023-10-07T11:03:23.064039+0000","last_became_peered":"2023-10-07T11:03:23.064039+0000","last_unstale":"2023-10-07T11:03:23.065837+0000","last_undegraded":"2023-10-07T11:03:23.065837+0000","last_fullsized":"2023-10-07T11:03:23.065837+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074670+0000","last_change":"2023-10-07T11:03:23.061128+0000","last_active":"2023-10-07T11:03:23.074670+0000","last_peered":"2023-10-07T11:03:23.074670+0000","last_clean":"2023-10-07T11:03:23.074670+0000","last_became_active":"2023-10-07T11:03:23.060886+0000","last_became_peered":"2023-10-07T11:03:23.060886+0000","last_unstale":"2023-10-07T11:03:23.074670+0000","last_undegraded":"2023-10-07T11:03:23.074670+0000","last_fullsized":"2023-10-07T11:03:23.074670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036512+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074352+0000","last_change":"2023-10-07T11:03:23.071098+0000","last_active":"2023-10-07T11:03:23.074352+0000","last_peered":"2023-10-07T11:03:23.074352+0000","last_clean":"2023-10-07T11:03:23.074352+0000","last_became_active":"2023-10-07T11:03:23.070446+0000","last_became_peered":"2023-10-07T11:03:23.070446+0000","last_unstale":"2023-10-07T11:03:23.074352+0000","last_undegraded":"2023-10-07T11:03:23.074352+0000","last_fullsized":"2023-10-07T11:03:23.074352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036127+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1d","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.078939+0000","last_change":"2023-10-07T11:03:22.046593+0000","last_active":"2023-10-07T11:03:23.078939+0000","last_peered":"2023-10-07T11:03:23.078939+0000","last_clean":"2023-10-07T11:03:23.078939+0000","last_became_active":"2023-10-07T11:03:22.046326+0000","last_became_peered":"2023-10-07T11:03:22.046326+0000","last_unstale":"2023-10-07T11:03:23.078939+0000","last_undegraded":"2023-10-07T11:03:23.078939+0000","last_fullsized":"2023-10-07T11:03:23.078939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.034708+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:28.068685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065766+0000","last_change":"2023-10-07T11:03:23.056224+0000","last_active":"2023-10-07T11:03:23.065766+0000","last_peered":"2023-10-07T11:03:23.065766+0000","last_clean":"2023-10-07T11:03:23.065766+0000","last_became_active":"2023-10-07T11:03:23.055757+0000","last_became_peered":"2023-10-07T11:03:23.055757+0000","last_unstale":"2023-10-07T11:03:23.065766+0000","last_undegraded":"2023-10-07T11:03:23.065766+0000","last_fullsized":"2023-10-07T11:03:23.065766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036798+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:41:07.129981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060572+0000","last_change":"2023-10-07T11:03:23.056936+0000","last_active":"2023-10-07T11:03:23.060572+0000","last_peered":"2023-10-07T11:03:23.060572+0000","last_clean":"2023-10-07T11:03:23.060572+0000","last_became_active":"2023-10-07T11:03:23.056670+0000","last_became_peered":"2023-10-07T11:03:23.056670+0000","last_unstale":"2023-10-07T11:03:23.060572+0000","last_undegraded":"2023-10-07T11:03:23.060572+0000","last_fullsized":"2023-10-07T11:03:23.060572+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082675+0000","last_change":"2023-10-07T11:03:22.049469+0000","last_active":"2023-10-07T11:03:23.082675+0000","last_peered":"2023-10-07T11:03:23.082675+0000","last_clean":"2023-10-07T11:03:23.082675+0000","last_became_active":"2023-10-07T11:03:22.049231+0000","last_became_peered":"2023-10-07T11:03:22.049231+0000","last_unstale":"2023-10-07T11:03:23.082675+0000","last_undegraded":"2023-10-07T11:03:23.082675+0000","last_fullsized":"2023-10-07T11:03:23.082675+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:51.063075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031851+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:14.104837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060058+0000","last_change":"2023-10-07T11:03:23.057682+0000","last_active":"2023-10-07T11:03:23.060058+0000","last_peered":"2023-10-07T11:03:23.060058+0000","last_clean":"2023-10-07T11:03:23.060058+0000","last_became_active":"2023-10-07T11:03:23.056838+0000","last_became_peered":"2023-10-07T11:03:23.056838+0000","last_unstale":"2023-10-07T11:03:23.060058+0000","last_undegraded":"2023-10-07T11:03:23.060058+0000","last_fullsized":"2023-10-07T11:03:23.060058+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.030990+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:23:58.478124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074728+0000","last_change":"2023-10-07T11:03:23.066795+0000","last_active":"2023-10-07T11:03:23.074728+0000","last_peered":"2023-10-07T11:03:23.074728+0000","last_clean":"2023-10-07T11:03:23.074728+0000","last_became_active":"2023-10-07T11:03:23.066509+0000","last_became_peered":"2023-10-07T11:03:23.066509+0000","last_unstale":"2023-10-07T11:03:23.074728+0000","last_undegraded":"2023-10-07T11:03:23.074728+0000","last_fullsized":"2023-10-07T11:03:23.074728+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:40:40.866050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036810+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:50:02.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036981+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:20:03.006488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072955+0000","last_change":"2023-10-07T11:03:23.056349+0000","last_active":"2023-10-07T11:03:23.072955+0000","last_peered":"2023-10-07T11:03:23.072955+0000","last_clean":"2023-10-07T11:03:23.072955+0000","last_became_active":"2023-10-07T11:03:23.055709+0000","last_became_peered":"2023-10-07T11:03:23.055709+0000","last_unstale":"2023-10-07T11:03:23.072955+0000","last_undegraded":"2023-10-07T11:03:23.072955+0000","last_fullsized":"2023-10-07T11:03:23.072955+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:34:35.644574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"196'6","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.097246+0000","last_change":"2023-10-07T11:03:22.043324+0000","last_active":"2023-10-07T11:03:23.097246+0000","last_peered":"2023-10-07T11:03:23.097246+0000","last_clean":"2023-10-07T11:03:23.097246+0000","last_became_active":"2023-10-07T11:03:22.043036+0000","last_became_peered":"2023-10-07T11:03:22.043036+0000","last_unstale":"2023-10-07T11:03:23.097246+0000","last_undegraded":"2023-10-07T11:03:23.097246+0000","last_fullsized":"2023-10-07T11:03:23.097246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072666+0000","last_change":"2023-10-07T11:03:23.070261+0000","last_active":"2023-10-07T11:03:23.072666+0000","last_peered":"2023-10-07T11:03:23.072666+0000","last_clean":"2023-10-07T11:03:23.072666+0000","last_became_active":"2023-10-07T11:03:23.069876+0000","last_became_peered":"2023-10-07T11:03:23.069876+0000","last_unstale":"2023-10-07T11:03:23.072666+0000","last_undegraded":"2023-10-07T11:03:23.072666+0000","last_fullsized":"2023-10-07T11:03:23.072666+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.032223+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:51.785763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046972+0000","last_change":"2023-10-07T11:03:22.058631+0000","last_active":"2023-10-07T11:03:23.046972+0000","last_peered":"2023-10-07T11:03:23.046972+0000","last_clean":"2023-10-07T11:03:23.046972+0000","last_became_active":"2023-10-07T11:03:22.058368+0000","last_became_peered":"2023-10-07T11:03:22.058368+0000","last_unstale":"2023-10-07T11:03:23.046972+0000","last_undegraded":"2023-10-07T11:03:23.046972+0000","last_fullsized":"2023-10-07T11:03:23.046972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:41.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050617+0000","last_change":"2023-10-07T11:03:22.046240+0000","last_active":"2023-10-07T11:03:23.050617+0000","last_peered":"2023-10-07T11:03:23.050617+0000","last_clean":"2023-10-07T11:03:23.050617+0000","last_became_active":"2023-10-07T11:03:22.045964+0000","last_became_peered":"2023-10-07T11:03:22.045964+0000","last_unstale":"2023-10-07T11:03:23.050617+0000","last_undegraded":"2023-10-07T11:03:23.050617+0000","last_fullsized":"2023-10-07T11:03:23.050617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:25.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072532+0000","last_change":"2023-10-07T11:03:23.069327+0000","last_active":"2023-10-07T11:03:23.072532+0000","last_peered":"2023-10-07T11:03:23.072532+0000","last_clean":"2023-10-07T11:03:23.072532+0000","last_became_active":"2023-10-07T11:03:23.068992+0000","last_became_peered":"2023-10-07T11:03:23.068992+0000","last_unstale":"2023-10-07T11:03:23.072532+0000","last_undegraded":"2023-10-07T11:03:23.072532+0000","last_fullsized":"2023-10-07T11:03:23.072532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031259+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:19:57.495871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074503+0000","last_change":"2023-10-07T11:03:23.069902+0000","last_active":"2023-10-07T11:03:23.074503+0000","last_peered":"2023-10-07T11:03:23.074503+0000","last_clean":"2023-10-07T11:03:23.074503+0000","last_became_active":"2023-10-07T11:03:23.069641+0000","last_became_peered":"2023-10-07T11:03:23.069641+0000","last_unstale":"2023-10-07T11:03:23.074503+0000","last_undegraded":"2023-10-07T11:03:23.074503+0000","last_fullsized":"2023-10-07T11:03:23.074503+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.035995+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.e","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083381+0000","last_change":"2023-10-07T11:03:22.049580+0000","last_active":"2023-10-07T11:03:23.083381+0000","last_peered":"2023-10-07T11:03:23.083381+0000","last_clean":"2023-10-07T11:03:23.083381+0000","last_became_active":"2023-10-07T11:03:22.049343+0000","last_became_peered":"2023-10-07T11:03:22.049343+0000","last_unstale":"2023-10-07T11:03:23.083381+0000","last_undegraded":"2023-10-07T11:03:23.083381+0000","last_fullsized":"2023-10-07T11:03:23.083381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:39.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031658+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:11:32.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031011+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:36.979433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072519+0000","last_change":"2023-10-07T11:03:23.069558+0000","last_active":"2023-10-07T11:03:23.072519+0000","last_peered":"2023-10-07T11:03:23.072519+0000","last_clean":"2023-10-07T11:03:23.072519+0000","last_became_active":"2023-10-07T11:03:23.069179+0000","last_became_peered":"2023-10-07T11:03:23.069179+0000","last_unstale":"2023-10-07T11:03:23.072519+0000","last_undegraded":"2023-10-07T11:03:23.072519+0000","last_fullsized":"2023-10-07T11:03:23.072519+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081863+0000","last_change":"2023-10-07T11:03:22.060469+0000","last_active":"2023-10-07T11:03:23.081863+0000","last_peered":"2023-10-07T11:03:23.081863+0000","last_clean":"2023-10-07T11:03:23.081863+0000","last_became_active":"2023-10-07T11:03:22.060024+0000","last_became_peered":"2023-10-07T11:03:22.060024+0000","last_unstale":"2023-10-07T11:03:23.081863+0000","last_undegraded":"2023-10-07T11:03:23.081863+0000","last_fullsized":"2023-10-07T11:03:23.081863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:42.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036035+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050442+0000","last_change":"2023-10-07T11:03:22.059410+0000","last_active":"2023-10-07T11:03:23.050442+0000","last_peered":"2023-10-07T11:03:23.050442+0000","last_clean":"2023-10-07T11:03:23.050442+0000","last_became_active":"2023-10-07T11:03:22.059029+0000","last_became_peered":"2023-10-07T11:03:22.059029+0000","last_unstale":"2023-10-07T11:03:23.050442+0000","last_undegraded":"2023-10-07T11:03:23.050442+0000","last_fullsized":"2023-10-07T11:03:23.050442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:57:18.642115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036998+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.032308+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:40:28.034611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070233+0000","last_change":"2023-10-07T11:03:23.062012+0000","last_active":"2023-10-07T11:03:23.070233+0000","last_peered":"2023-10-07T11:03:23.070233+0000","last_clean":"2023-10-07T11:03:23.070233+0000","last_became_active":"2023-10-07T11:03:23.061769+0000","last_became_peered":"2023-10-07T11:03:23.061769+0000","last_unstale":"2023-10-07T11:03:23.070233+0000","last_undegraded":"2023-10-07T11:03:23.070233+0000","last_fullsized":"2023-10-07T11:03:23.070233+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046870+0000","last_change":"2023-10-07T11:03:22.051956+0000","last_active":"2023-10-07T11:03:23.046870+0000","last_peered":"2023-10-07T11:03:23.046870+0000","last_clean":"2023-10-07T11:03:23.046870+0000","last_became_active":"2023-10-07T11:03:22.051691+0000","last_became_peered":"2023-10-07T11:03:22.051691+0000","last_unstale":"2023-10-07T11:03:23.046870+0000","last_undegraded":"2023-10-07T11:03:23.046870+0000","last_fullsized":"2023-10-07T11:03:23.046870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:42.177620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065367+0000","last_change":"2023-10-07T11:03:23.059948+0000","last_active":"2023-10-07T11:03:23.065367+0000","last_peered":"2023-10-07T11:03:23.065367+0000","last_clean":"2023-10-07T11:03:23.065367+0000","last_became_active":"2023-10-07T11:03:23.059564+0000","last_became_peered":"2023-10-07T11:03:23.059564+0000","last_unstale":"2023-10-07T11:03:23.065367+0000","last_undegraded":"2023-10-07T11:03:23.065367+0000","last_fullsized":"2023-10-07T11:03:23.065367+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:04:25.554055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046689+0000","last_change":"2023-10-07T11:03:22.052493+0000","last_active":"2023-10-07T11:03:23.046689+0000","last_peered":"2023-10-07T11:03:23.046689+0000","last_clean":"2023-10-07T11:03:23.046689+0000","last_became_active":"2023-10-07T11:03:22.052235+0000","last_became_peered":"2023-10-07T11:03:22.052235+0000","last_unstale":"2023-10-07T11:03:23.046689+0000","last_undegraded":"2023-10-07T11:03:23.046689+0000","last_fullsized":"2023-10-07T11:03:23.046689+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:06.410397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074593+0000","last_change":"2023-10-07T11:03:23.060626+0000","last_active":"2023-10-07T11:03:23.074593+0000","last_peered":"2023-10-07T11:03:23.074593+0000","last_clean":"2023-10-07T11:03:23.074593+0000","last_became_active":"2023-10-07T11:03:23.060260+0000","last_became_peered":"2023-10-07T11:03:23.060260+0000","last_unstale":"2023-10-07T11:03:23.074593+0000","last_undegraded":"2023-10-07T11:03:23.074593+0000","last_fullsized":"2023-10-07T11:03:23.074593+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051338+0000","last_change":"2023-10-07T11:03:22.048854+0000","last_active":"2023-10-07T11:03:23.051338+0000","last_peered":"2023-10-07T11:03:23.051338+0000","last_clean":"2023-10-07T11:03:23.051338+0000","last_became_active":"2023-10-07T11:03:22.048568+0000","last_became_peered":"2023-10-07T11:03:22.048568+0000","last_unstale":"2023-10-07T11:03:23.051338+0000","last_undegraded":"2023-10-07T11:03:23.051338+0000","last_fullsized":"2023-10-07T11:03:23.051338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:01:19.581526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036428+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.035995+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:33:09.391065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.034704+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:10:49.465391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051314+0000","last_change":"2023-10-07T11:03:22.060700+0000","last_active":"2023-10-07T11:03:23.051314+0000","last_peered":"2023-10-07T11:03:23.051314+0000","last_clean":"2023-10-07T11:03:23.051314+0000","last_became_active":"2023-10-07T11:03:22.060037+0000","last_became_peered":"2023-10-07T11:03:22.060037+0000","last_unstale":"2023-10-07T11:03:23.051314+0000","last_undegraded":"2023-10-07T11:03:23.051314+0000","last_fullsized":"2023-10-07T11:03:23.051314+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:53.212268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073064+0000","last_change":"2023-10-07T11:03:23.058853+0000","last_active":"2023-10-07T11:03:23.073064+0000","last_peered":"2023-10-07T11:03:23.073064+0000","last_clean":"2023-10-07T11:03:23.073064+0000","last_became_active":"2023-10-07T11:03:23.058594+0000","last_became_peered":"2023-10-07T11:03:23.058594+0000","last_unstale":"2023-10-07T11:03:23.073064+0000","last_undegraded":"2023-10-07T11:03:23.073064+0000","last_fullsized":"2023-10-07T11:03:23.073064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031592+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:51:11.826526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"186'348","reported_seq":1128,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-10-07T11:03:19.684286+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:19.684286+0000","last_peered":"2023-10-07T11:03:19.684286+0000","last_clean":"2023-10-07T11:03:19.684286+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:19.684286+0000","last_undegraded":"2023-10-07T11:03:19.684286+0000","last_fullsized":"2023-10-07T11:03:19.684286+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":52,"acting":52,"num_store_stats":3},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":1536,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312512,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059328,"statfs":{"total":96636764160,"available":96316751872,"internally_reserved":0,"allocated":3006464,"data_stored":1783889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.82199999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.98599999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.76100000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316028,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055812,"statfs":{"total":96636764160,"available":96313151488,"internally_reserved":0,"allocated":2412544,"data_stored":1193521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.98399999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.876}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.95799999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":1.0309999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543260,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307896,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063944,"statfs":{"total":96636764160,"available":96321478656,"internally_reserved":0,"allocated":2473984,"data_stored":1222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.442},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.76700000000000002,"5min":0.86299999999999999,"15min":0.94199999999999995},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.40000000000000002},"max":{"1min":0.82699999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.443,"5min":0.378,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82699999999999996,"15min":0.84099999999999997},"min":{"1min":0.55600000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.373},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.153,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84999999999999998},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80600000000000005,"15min":0.81299999999999994},"min":{"1min":0.57999999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.873,"15min":0.871},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.58399999999999996},"max":{"1min":1.052,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0980000000000001,"5min":1.157,"15min":1.302},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.877,"15min":0.88400000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.141,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88700000000000001,"15min":0.878},"min":{"1min":0.66700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.80600000000000005}]}]},{"osd":0,"up_from":14,"seq":60129543260,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1810934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.439},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.496}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.93600000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.008},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.86399999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299656,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072184,"statfs":{"total":96636764160,"available":96329916416,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.81699999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66300000000000003,"5min":0.85399999999999998,"15min":0.95099999999999996},"last":0.625},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.71199999999999997,"5min":0.92600000000000005,"15min":0.98199999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65900000000000003,"5min":0.76700000000000002,"15min":0.80000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.58299999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.77400000000000002,"15min":0.79400000000000004},"min":{"1min":0.55900000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.93600000000000005,"5min":1.163,"15min":1.1879999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.266,"15min":1.5640000000000001},"last":0.92800000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.014,"5min":1.2989999999999999,"15min":1.528},"last":1.004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78800000000000003,"15min":0.80800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97499999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.79800000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1140000000000001,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.147,"5min":1.242,"15min":1.3779999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.85699999999999998,"15min":0.874},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.052,"5min":1.2050000000000001,"15min":1.629},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.85699999999999998,"15min":0.877},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0900000000000001,"5min":1.274,"15min":1.5940000000000001},"last":0.96299999999999997}]}]},{"osd":3,"up_from":14,"seq":60129543262,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1220738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72199999999999998,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.45000000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.83999999999999997,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79700000000000004,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55500000000000005,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73099999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.5},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.436,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96799999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0780000000000001,"15min":1.222},"last":0.78100000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0720000000000001,"5min":1.27,"15min":1.27},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":0.84699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98399999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96699999999999997,"5min":1.389,"15min":1.389},"last":0.77300000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0269999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.008,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.68100000000000005}]}]},{"osd":4,"up_from":15,"seq":64424510557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316040,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055800,"statfs":{"total":96636764160,"available":96313139200,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.628},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.749},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.69899999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.66400000000000003}]}]},{"osd":5,"up_from":14,"seq":60129543259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311944,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059896,"statfs":{"total":96636764160,"available":96317333504,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78100000000000003,"15min":0.78400000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.436},"max":{"1min":0.95199999999999996,"5min":1.585,"15min":1.585},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.81299999999999994,"15min":0.81799999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.52500000000000002},"max":{"1min":0.89300000000000002,"5min":1.673,"15min":1.673},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73099999999999998,"15min":0.746},"min":{"1min":0.54100000000000004,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.90400000000000003,"5min":1.0189999999999999,"15min":1.0860000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.76900000000000002},"min":{"1min":0.54200000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.89800000000000002,"5min":0.998,"15min":1.139},"last":0.77700000000000002}]},{"osd":2,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72899999999999998,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93200000000000005,"5min":1.0349999999999999,"15min":1.0580000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89800000000000002,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.51200000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.069,"15min":1.069},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.747},"min":{"1min":0.57799999999999996,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":0.96899999999999997,"5min":1.0880000000000001,"15min":1.161},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64100000000000001,"15min":0.63},"min":{"1min":0.438,"5min":0.438,"15min":0.42799999999999999},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":0.97499999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91800000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.75800000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.378},"max":{"1min":0.93400000000000005,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70399999999999996,"15min":0.69299999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86499999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"max":{"1min":0.92000000000000004,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76900000000000002,"15min":0.75700000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:03:27.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:03:27.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.693+0000 7f63a9fff700 1 Processor -- start 2023-10-07T11:03:27.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.693+0000 7f63a9fff700 1 -- start start 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.694+0000 7f63a9fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a40ad370 0x7f63a40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.694+0000 7f63a9fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63a4159ec0 0x7f63a415a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.694+0000 7f63a9fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63a41568c0 0x7f63a4156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.694+0000 7f63a9fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a414d790 con 0x7f63a41568c0 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.694+0000 7f63a9fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a40ae5d0 con 0x7f63a4159ec0 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.694+0000 7f63a9fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a415a850 con 0x7f63a40ad370 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63a41568c0 0x7f63a4156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63a41568c0 0x7f63a4156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53270/0 (socket says 172.21.15.29:53270) 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 -- 172.21.15.29:0/1939488562 learned_addr learned my addr 172.21.15.29:0/1939488562 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:27.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a37fe700 1 --2- 172.21.15.29:0/1939488562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a40ad370 0x7f63a40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a2ffd700 1 --2- 172.21.15.29:0/1939488562 >> v2:172.21.15.144:3300/0 conn(0x7f63a4159ec0 0x7f63a415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 -- 172.21.15.29:0/1939488562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a40ad370 msgr2=0x7f63a40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 --2- 172.21.15.29:0/1939488562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a40ad370 0x7f63a40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 -- 172.21.15.29:0/1939488562 >> v2:172.21.15.144:3300/0 conn(0x7f63a4159ec0 msgr2=0x7f63a415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 --2- 172.21.15.29:0/1939488562 >> v2:172.21.15.144:3300/0 conn(0x7f63a4159ec0 0x7f63a415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 -- 172.21.15.29:0/1939488562 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40ad740 con 0x7f63a41568c0 2023-10-07T11:03:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.695+0000 7f63a3fff700 1 --2- 172.21.15.29:0/1939488562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a41568c0 0x7f63a4156d00 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f63a4063d00 tx=0x7f639c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622fab3b46685d03 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a27fc700 1 -- 172.21.15.29:0/1939488562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c010070 con 0x7f63a41568c0 2023-10-07T11:03:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a27fc700 1 -- 172.21.15.29:0/1939488562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c00dd30 con 0x7f63a41568c0 2023-10-07T11:03:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a27fc700 1 -- 172.21.15.29:0/1939488562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c00dea0 con 0x7f63a41568c0 2023-10-07T11:03:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 -- 172.21.15.29:0/1939488562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a41568c0 msgr2=0x7f63a4156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 --2- 172.21.15.29:0/1939488562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a41568c0 0x7f63a4156d00 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f63a4063d00 tx=0x7f639c00ba50 comp rx=0 tx=0).stop 2023-10-07T11:03:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 -- 172.21.15.29:0/1939488562 shutdown_connections 2023-10-07T11:03:27.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 --2- 172.21.15.29:0/1939488562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a41568c0 0x7f63a4156d00 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 --2- 172.21.15.29:0/1939488562 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a40ad370 0x7f63a40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 --2- 172.21.15.29:0/1939488562 >> v2:172.21.15.144:3300/0 conn(0x7f63a4159ec0 0x7f63a415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 -- 172.21.15.29:0/1939488562 >> 172.21.15.29:0/1939488562 conn(0x7f63a40680a0 msgr2=0x7f63a40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:27.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 -- 172.21.15.29:0/1939488562 shutdown_connections 2023-10-07T11:03:27.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.696+0000 7f63a9fff700 1 -- 172.21.15.29:0/1939488562 wait complete. 2023-10-07T11:03:27.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.698+0000 7f63a9fff700 1 Processor -- start 2023-10-07T11:03:27.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.698+0000 7f63a9fff700 1 -- start start 2023-10-07T11:03:27.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a9fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a40ad370 0x7f63a41502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a9fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a41568c0 0x7f63a413bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a9fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 0x7f63a406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a9fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a414e8a0 con 0x7f63a4159ec0 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a9fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a414fe40 con 0x7f63a40ad370 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a9fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a414f510 con 0x7f63a41568c0 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 0x7f63a406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 0x7f63a406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53284/0 (socket says 172.21.15.29:53284) 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.699+0000 7f63a3fff700 1 -- 172.21.15.29:0/584933815 learned_addr learned my addr 172.21.15.29:0/584933815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a37fe700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a40ad370 0x7f63a41502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a2ffd700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a41568c0 0x7f63a413bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a3fff700 1 -- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a41568c0 msgr2=0x7f63a413bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a3fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a41568c0 0x7f63a413bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a3fff700 1 -- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a40ad370 msgr2=0x7f63a41502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a3fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a40ad370 0x7f63a41502d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a3fff700 1 -- 172.21.15.29:0/584933815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c00bcf0 con 0x7f63a4159ec0 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a37fe700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a40ad370 0x7f63a41502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a2ffd700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a41568c0 0x7f63a413bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a3fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 0x7f63a406e470 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f639c00ba90 tx=0x7f639c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c00ab90 con 0x7f63a4159ec0 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c00ad10 con 0x7f63a4159ec0 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a414f7f0 con 0x7f63a4159ec0 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c01e690 con 0x7f63a4159ec0 2023-10-07T11:03:27.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.700+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a413c0c0 con 0x7f63a4159ec0 2023-10-07T11:03:27.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.701+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a414fff0 con 0x7f63a4159ec0 2023-10-07T11:03:27.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.702+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f639c003040 con 0x7f63a4159ec0 2023-10-07T11:03:27.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.703+0000 7f638b7fe700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6390066f30 0x7f63900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:27.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.703+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f639c086180 con 0x7f63a4159ec0 2023-10-07T11:03:27.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.704+0000 7f63a37fe700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6390066f30 0x7f63900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:27.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.704+0000 7f63a37fe700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6390066f30 0x7f63900693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6394006f00 tx=0x7f6394008040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.708+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63a414fff0 con 0x7f63a4159ec0 2023-10-07T11:03:27.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.903+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63a414ffb0 con 0x7f6390066f30 2023-10-07T11:03:27.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.912+0000 7f638b7fe700 1 -- 172.21.15.29:0/584933815 <== mgr.8429 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282860 (secure 0 0 0) 0x7f63a414ffb0 con 0x7f6390066f30 2023-10-07T11:03:27.916 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6390066f30 msgr2=0x7f63900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6390066f30 0x7f63900693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6394006f00 tx=0x7f6394008040 comp rx=0 tx=0).stop 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 msgr2=0x7f63a406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 0x7f63a406e470 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f639c00ba90 tx=0x7f639c01a000 comp rx=0 tx=0).stop 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 shutdown_connections 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63a4159ec0 0x7f63a406e470 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a40ad370 0x7f63a41502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6390066f30 0x7f63900693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 --2- 172.21.15.29:0/584933815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63a41568c0 0x7f63a413bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:27.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 >> 172.21.15.29:0/584933815 conn(0x7f63a40680a0 msgr2=0x7f63a414cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:27.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 shutdown_connections 2023-10-07T11:03:27.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:27.917+0000 7f63a9fff700 1 -- 172.21.15.29:0/584933815 wait complete. 2023-10-07T11:03:27.920 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:03:27.935 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T11:03:26.651670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591904,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":592,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":361,"ondisk_log_size":361,"up":118,"acting":118,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2476344,"kb_used_data":20216,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752498376,"statfs":{"total":773094113280,"available":770558337024,"internally_reserved":0,"allocated":20701184,"data_stored":10830568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1536,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.623060"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.061273+0000","last_change":"2023-10-07T11:03:23.059262+0000","last_active":"2023-10-07T11:03:23.061273+0000","last_peered":"2023-10-07T11:03:23.061273+0000","last_clean":"2023-10-07T11:03:23.061273+0000","last_became_active":"2023-10-07T11:03:23.058996+0000","last_became_peered":"2023-10-07T11:03:23.058996+0000","last_unstale":"2023-10-07T11:03:23.061273+0000","last_undegraded":"2023-10-07T11:03:23.061273+0000","last_fullsized":"2023-10-07T11:03:23.061273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069205+0000","last_change":"2023-10-07T11:03:23.067743+0000","last_active":"2023-10-07T11:03:23.069205+0000","last_peered":"2023-10-07T11:03:23.069205+0000","last_clean":"2023-10-07T11:03:23.069205+0000","last_became_active":"2023-10-07T11:03:23.067464+0000","last_became_peered":"2023-10-07T11:03:23.067464+0000","last_unstale":"2023-10-07T11:03:23.069205+0000","last_undegraded":"2023-10-07T11:03:23.069205+0000","last_fullsized":"2023-10-07T11:03:23.069205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.077562+0000","last_change":"2023-10-07T11:03:23.076006+0000","last_active":"2023-10-07T11:03:23.077562+0000","last_peered":"2023-10-07T11:03:23.077562+0000","last_clean":"2023-10-07T11:03:23.077562+0000","last_became_active":"2023-10-07T11:03:23.075758+0000","last_became_peered":"2023-10-07T11:03:23.075758+0000","last_unstale":"2023-10-07T11:03:23.077562+0000","last_undegraded":"2023-10-07T11:03:23.077562+0000","last_fullsized":"2023-10-07T11:03:23.077562+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.066075+0000","last_change":"2023-10-07T11:03:23.058595+0000","last_active":"2023-10-07T11:03:23.066075+0000","last_peered":"2023-10-07T11:03:23.066075+0000","last_clean":"2023-10-07T11:03:23.066075+0000","last_became_active":"2023-10-07T11:03:23.058332+0000","last_became_peered":"2023-10-07T11:03:23.058332+0000","last_unstale":"2023-10-07T11:03:23.066075+0000","last_undegraded":"2023-10-07T11:03:23.066075+0000","last_fullsized":"2023-10-07T11:03:23.066075+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.062893+0000","last_change":"2023-10-07T11:03:23.057796+0000","last_active":"2023-10-07T11:03:23.062893+0000","last_peered":"2023-10-07T11:03:23.062893+0000","last_clean":"2023-10-07T11:03:23.062893+0000","last_became_active":"2023-10-07T11:03:23.057545+0000","last_became_peered":"2023-10-07T11:03:23.057545+0000","last_unstale":"2023-10-07T11:03:23.062893+0000","last_undegraded":"2023-10-07T11:03:23.062893+0000","last_fullsized":"2023-10-07T11:03:23.062893+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072967+0000","last_change":"2023-10-07T11:03:23.056891+0000","last_active":"2023-10-07T11:03:23.072967+0000","last_peered":"2023-10-07T11:03:23.072967+0000","last_clean":"2023-10-07T11:03:23.072967+0000","last_became_active":"2023-10-07T11:03:23.056623+0000","last_became_peered":"2023-10-07T11:03:23.056623+0000","last_unstale":"2023-10-07T11:03:23.072967+0000","last_undegraded":"2023-10-07T11:03:23.072967+0000","last_fullsized":"2023-10-07T11:03:23.072967+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:24.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065743+0000","last_change":"2023-10-07T11:03:23.056041+0000","last_active":"2023-10-07T11:03:23.065743+0000","last_peered":"2023-10-07T11:03:23.065743+0000","last_clean":"2023-10-07T11:03:23.065743+0000","last_became_active":"2023-10-07T11:03:23.055408+0000","last_became_peered":"2023-10-07T11:03:23.055408+0000","last_unstale":"2023-10-07T11:03:23.065743+0000","last_undegraded":"2023-10-07T11:03:23.065743+0000","last_fullsized":"2023-10-07T11:03:23.065743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072460+0000","last_change":"2023-10-07T11:03:23.070205+0000","last_active":"2023-10-07T11:03:23.072460+0000","last_peered":"2023-10-07T11:03:23.072460+0000","last_clean":"2023-10-07T11:03:23.072460+0000","last_became_active":"2023-10-07T11:03:23.069898+0000","last_became_peered":"2023-10-07T11:03:23.069898+0000","last_unstale":"2023-10-07T11:03:23.072460+0000","last_undegraded":"2023-10-07T11:03:23.072460+0000","last_fullsized":"2023-10-07T11:03:23.072460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074629+0000","last_change":"2023-10-07T11:03:23.059218+0000","last_active":"2023-10-07T11:03:23.074629+0000","last_peered":"2023-10-07T11:03:23.074629+0000","last_clean":"2023-10-07T11:03:23.074629+0000","last_became_active":"2023-10-07T11:03:23.058961+0000","last_became_peered":"2023-10-07T11:03:23.058961+0000","last_unstale":"2023-10-07T11:03:23.074629+0000","last_undegraded":"2023-10-07T11:03:23.074629+0000","last_fullsized":"2023-10-07T11:03:23.074629+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:09.706205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069344+0000","last_change":"2023-10-07T11:03:23.068010+0000","last_active":"2023-10-07T11:03:23.069344+0000","last_peered":"2023-10-07T11:03:23.069344+0000","last_clean":"2023-10-07T11:03:23.069344+0000","last_became_active":"2023-10-07T11:03:23.067653+0000","last_became_peered":"2023-10-07T11:03:23.067653+0000","last_unstale":"2023-10-07T11:03:23.069344+0000","last_undegraded":"2023-10-07T11:03:23.069344+0000","last_fullsized":"2023-10-07T11:03:23.069344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065837+0000","last_change":"2023-10-07T11:03:23.064318+0000","last_active":"2023-10-07T11:03:23.065837+0000","last_peered":"2023-10-07T11:03:23.065837+0000","last_clean":"2023-10-07T11:03:23.065837+0000","last_became_active":"2023-10-07T11:03:23.064039+0000","last_became_peered":"2023-10-07T11:03:23.064039+0000","last_unstale":"2023-10-07T11:03:23.065837+0000","last_undegraded":"2023-10-07T11:03:23.065837+0000","last_fullsized":"2023-10-07T11:03:23.065837+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074670+0000","last_change":"2023-10-07T11:03:23.061128+0000","last_active":"2023-10-07T11:03:23.074670+0000","last_peered":"2023-10-07T11:03:23.074670+0000","last_clean":"2023-10-07T11:03:23.074670+0000","last_became_active":"2023-10-07T11:03:23.060886+0000","last_became_peered":"2023-10-07T11:03:23.060886+0000","last_unstale":"2023-10-07T11:03:23.074670+0000","last_undegraded":"2023-10-07T11:03:23.074670+0000","last_fullsized":"2023-10-07T11:03:23.074670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036512+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074352+0000","last_change":"2023-10-07T11:03:23.071098+0000","last_active":"2023-10-07T11:03:23.074352+0000","last_peered":"2023-10-07T11:03:23.074352+0000","last_clean":"2023-10-07T11:03:23.074352+0000","last_became_active":"2023-10-07T11:03:23.070446+0000","last_became_peered":"2023-10-07T11:03:23.070446+0000","last_unstale":"2023-10-07T11:03:23.074352+0000","last_undegraded":"2023-10-07T11:03:23.074352+0000","last_fullsized":"2023-10-07T11:03:23.074352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036127+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1d","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.078939+0000","last_change":"2023-10-07T11:03:22.046593+0000","last_active":"2023-10-07T11:03:23.078939+0000","last_peered":"2023-10-07T11:03:23.078939+0000","last_clean":"2023-10-07T11:03:23.078939+0000","last_became_active":"2023-10-07T11:03:22.046326+0000","last_became_peered":"2023-10-07T11:03:22.046326+0000","last_unstale":"2023-10-07T11:03:23.078939+0000","last_undegraded":"2023-10-07T11:03:23.078939+0000","last_fullsized":"2023-10-07T11:03:23.078939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.034708+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:28.068685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065766+0000","last_change":"2023-10-07T11:03:23.056224+0000","last_active":"2023-10-07T11:03:23.065766+0000","last_peered":"2023-10-07T11:03:23.065766+0000","last_clean":"2023-10-07T11:03:23.065766+0000","last_became_active":"2023-10-07T11:03:23.055757+0000","last_became_peered":"2023-10-07T11:03:23.055757+0000","last_unstale":"2023-10-07T11:03:23.065766+0000","last_undegraded":"2023-10-07T11:03:23.065766+0000","last_fullsized":"2023-10-07T11:03:23.065766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036798+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:41:07.129981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060572+0000","last_change":"2023-10-07T11:03:23.056936+0000","last_active":"2023-10-07T11:03:23.060572+0000","last_peered":"2023-10-07T11:03:23.060572+0000","last_clean":"2023-10-07T11:03:23.060572+0000","last_became_active":"2023-10-07T11:03:23.056670+0000","last_became_peered":"2023-10-07T11:03:23.056670+0000","last_unstale":"2023-10-07T11:03:23.060572+0000","last_undegraded":"2023-10-07T11:03:23.060572+0000","last_fullsized":"2023-10-07T11:03:23.060572+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082675+0000","last_change":"2023-10-07T11:03:22.049469+0000","last_active":"2023-10-07T11:03:23.082675+0000","last_peered":"2023-10-07T11:03:23.082675+0000","last_clean":"2023-10-07T11:03:23.082675+0000","last_became_active":"2023-10-07T11:03:22.049231+0000","last_became_peered":"2023-10-07T11:03:22.049231+0000","last_unstale":"2023-10-07T11:03:23.082675+0000","last_undegraded":"2023-10-07T11:03:23.082675+0000","last_fullsized":"2023-10-07T11:03:23.082675+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:51.063075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031851+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:14.104837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060058+0000","last_change":"2023-10-07T11:03:23.057682+0000","last_active":"2023-10-07T11:03:23.060058+0000","last_peered":"2023-10-07T11:03:23.060058+0000","last_clean":"2023-10-07T11:03:23.060058+0000","last_became_active":"2023-10-07T11:03:23.056838+0000","last_became_peered":"2023-10-07T11:03:23.056838+0000","last_unstale":"2023-10-07T11:03:23.060058+0000","last_undegraded":"2023-10-07T11:03:23.060058+0000","last_fullsized":"2023-10-07T11:03:23.060058+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.030990+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:23:58.478124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074728+0000","last_change":"2023-10-07T11:03:23.066795+0000","last_active":"2023-10-07T11:03:23.074728+0000","last_peered":"2023-10-07T11:03:23.074728+0000","last_clean":"2023-10-07T11:03:23.074728+0000","last_became_active":"2023-10-07T11:03:23.066509+0000","last_became_peered":"2023-10-07T11:03:23.066509+0000","last_unstale":"2023-10-07T11:03:23.074728+0000","last_undegraded":"2023-10-07T11:03:23.074728+0000","last_fullsized":"2023-10-07T11:03:23.074728+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:40:40.866050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036810+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:50:02.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036981+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:20:03.006488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072955+0000","last_change":"2023-10-07T11:03:23.056349+0000","last_active":"2023-10-07T11:03:23.072955+0000","last_peered":"2023-10-07T11:03:23.072955+0000","last_clean":"2023-10-07T11:03:23.072955+0000","last_became_active":"2023-10-07T11:03:23.055709+0000","last_became_peered":"2023-10-07T11:03:23.055709+0000","last_unstale":"2023-10-07T11:03:23.072955+0000","last_undegraded":"2023-10-07T11:03:23.072955+0000","last_fullsized":"2023-10-07T11:03:23.072955+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:34:35.644574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"196'6","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.097246+0000","last_change":"2023-10-07T11:03:22.043324+0000","last_active":"2023-10-07T11:03:23.097246+0000","last_peered":"2023-10-07T11:03:23.097246+0000","last_clean":"2023-10-07T11:03:23.097246+0000","last_became_active":"2023-10-07T11:03:22.043036+0000","last_became_peered":"2023-10-07T11:03:22.043036+0000","last_unstale":"2023-10-07T11:03:23.097246+0000","last_undegraded":"2023-10-07T11:03:23.097246+0000","last_fullsized":"2023-10-07T11:03:23.097246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072666+0000","last_change":"2023-10-07T11:03:23.070261+0000","last_active":"2023-10-07T11:03:23.072666+0000","last_peered":"2023-10-07T11:03:23.072666+0000","last_clean":"2023-10-07T11:03:23.072666+0000","last_became_active":"2023-10-07T11:03:23.069876+0000","last_became_peered":"2023-10-07T11:03:23.069876+0000","last_unstale":"2023-10-07T11:03:23.072666+0000","last_undegraded":"2023-10-07T11:03:23.072666+0000","last_fullsized":"2023-10-07T11:03:23.072666+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.032223+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:51.785763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046972+0000","last_change":"2023-10-07T11:03:22.058631+0000","last_active":"2023-10-07T11:03:23.046972+0000","last_peered":"2023-10-07T11:03:23.046972+0000","last_clean":"2023-10-07T11:03:23.046972+0000","last_became_active":"2023-10-07T11:03:22.058368+0000","last_became_peered":"2023-10-07T11:03:22.058368+0000","last_unstale":"2023-10-07T11:03:23.046972+0000","last_undegraded":"2023-10-07T11:03:23.046972+0000","last_fullsized":"2023-10-07T11:03:23.046972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:41.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050617+0000","last_change":"2023-10-07T11:03:22.046240+0000","last_active":"2023-10-07T11:03:23.050617+0000","last_peered":"2023-10-07T11:03:23.050617+0000","last_clean":"2023-10-07T11:03:23.050617+0000","last_became_active":"2023-10-07T11:03:22.045964+0000","last_became_peered":"2023-10-07T11:03:22.045964+0000","last_unstale":"2023-10-07T11:03:23.050617+0000","last_undegraded":"2023-10-07T11:03:23.050617+0000","last_fullsized":"2023-10-07T11:03:23.050617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:25.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072532+0000","last_change":"2023-10-07T11:03:23.069327+0000","last_active":"2023-10-07T11:03:23.072532+0000","last_peered":"2023-10-07T11:03:23.072532+0000","last_clean":"2023-10-07T11:03:23.072532+0000","last_became_active":"2023-10-07T11:03:23.068992+0000","last_became_peered":"2023-10-07T11:03:23.068992+0000","last_unstale":"2023-10-07T11:03:23.072532+0000","last_undegraded":"2023-10-07T11:03:23.072532+0000","last_fullsized":"2023-10-07T11:03:23.072532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031259+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:19:57.495871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074503+0000","last_change":"2023-10-07T11:03:23.069902+0000","last_active":"2023-10-07T11:03:23.074503+0000","last_peered":"2023-10-07T11:03:23.074503+0000","last_clean":"2023-10-07T11:03:23.074503+0000","last_became_active":"2023-10-07T11:03:23.069641+0000","last_became_peered":"2023-10-07T11:03:23.069641+0000","last_unstale":"2023-10-07T11:03:23.074503+0000","last_undegraded":"2023-10-07T11:03:23.074503+0000","last_fullsized":"2023-10-07T11:03:23.074503+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.035995+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.e","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083381+0000","last_change":"2023-10-07T11:03:22.049580+0000","last_active":"2023-10-07T11:03:23.083381+0000","last_peered":"2023-10-07T11:03:23.083381+0000","last_clean":"2023-10-07T11:03:23.083381+0000","last_became_active":"2023-10-07T11:03:22.049343+0000","last_became_peered":"2023-10-07T11:03:22.049343+0000","last_unstale":"2023-10-07T11:03:23.083381+0000","last_undegraded":"2023-10-07T11:03:23.083381+0000","last_fullsized":"2023-10-07T11:03:23.083381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:39.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031658+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:11:32.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031011+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:36.979433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072519+0000","last_change":"2023-10-07T11:03:23.069558+0000","last_active":"2023-10-07T11:03:23.072519+0000","last_peered":"2023-10-07T11:03:23.072519+0000","last_clean":"2023-10-07T11:03:23.072519+0000","last_became_active":"2023-10-07T11:03:23.069179+0000","last_became_peered":"2023-10-07T11:03:23.069179+0000","last_unstale":"2023-10-07T11:03:23.072519+0000","last_undegraded":"2023-10-07T11:03:23.072519+0000","last_fullsized":"2023-10-07T11:03:23.072519+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081863+0000","last_change":"2023-10-07T11:03:22.060469+0000","last_active":"2023-10-07T11:03:23.081863+0000","last_peered":"2023-10-07T11:03:23.081863+0000","last_clean":"2023-10-07T11:03:23.081863+0000","last_became_active":"2023-10-07T11:03:22.060024+0000","last_became_peered":"2023-10-07T11:03:22.060024+0000","last_unstale":"2023-10-07T11:03:23.081863+0000","last_undegraded":"2023-10-07T11:03:23.081863+0000","last_fullsized":"2023-10-07T11:03:23.081863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:42.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036035+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050442+0000","last_change":"2023-10-07T11:03:22.059410+0000","last_active":"2023-10-07T11:03:23.050442+0000","last_peered":"2023-10-07T11:03:23.050442+0000","last_clean":"2023-10-07T11:03:23.050442+0000","last_became_active":"2023-10-07T11:03:22.059029+0000","last_became_peered":"2023-10-07T11:03:22.059029+0000","last_unstale":"2023-10-07T11:03:23.050442+0000","last_undegraded":"2023-10-07T11:03:23.050442+0000","last_fullsized":"2023-10-07T11:03:23.050442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:57:18.642115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036998+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.032308+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:40:28.034611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070233+0000","last_change":"2023-10-07T11:03:23.062012+0000","last_active":"2023-10-07T11:03:23.070233+0000","last_peered":"2023-10-07T11:03:23.070233+0000","last_clean":"2023-10-07T11:03:23.070233+0000","last_became_active":"2023-10-07T11:03:23.061769+0000","last_became_peered":"2023-10-07T11:03:23.061769+0000","last_unstale":"2023-10-07T11:03:23.070233+0000","last_undegraded":"2023-10-07T11:03:23.070233+0000","last_fullsized":"2023-10-07T11:03:23.070233+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046870+0000","last_change":"2023-10-07T11:03:22.051956+0000","last_active":"2023-10-07T11:03:23.046870+0000","last_peered":"2023-10-07T11:03:23.046870+0000","last_clean":"2023-10-07T11:03:23.046870+0000","last_became_active":"2023-10-07T11:03:22.051691+0000","last_became_peered":"2023-10-07T11:03:22.051691+0000","last_unstale":"2023-10-07T11:03:23.046870+0000","last_undegraded":"2023-10-07T11:03:23.046870+0000","last_fullsized":"2023-10-07T11:03:23.046870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:42.177620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065367+0000","last_change":"2023-10-07T11:03:23.059948+0000","last_active":"2023-10-07T11:03:23.065367+0000","last_peered":"2023-10-07T11:03:23.065367+0000","last_clean":"2023-10-07T11:03:23.065367+0000","last_became_active":"2023-10-07T11:03:23.059564+0000","last_became_peered":"2023-10-07T11:03:23.059564+0000","last_unstale":"2023-10-07T11:03:23.065367+0000","last_undegraded":"2023-10-07T11:03:23.065367+0000","last_fullsized":"2023-10-07T11:03:23.065367+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:04:25.554055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046689+0000","last_change":"2023-10-07T11:03:22.052493+0000","last_active":"2023-10-07T11:03:23.046689+0000","last_peered":"2023-10-07T11:03:23.046689+0000","last_clean":"2023-10-07T11:03:23.046689+0000","last_became_active":"2023-10-07T11:03:22.052235+0000","last_became_peered":"2023-10-07T11:03:22.052235+0000","last_unstale":"2023-10-07T11:03:23.046689+0000","last_undegraded":"2023-10-07T11:03:23.046689+0000","last_fullsized":"2023-10-07T11:03:23.046689+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:06.410397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074593+0000","last_change":"2023-10-07T11:03:23.060626+0000","last_active":"2023-10-07T11:03:23.074593+0000","last_peered":"2023-10-07T11:03:23.074593+0000","last_clean":"2023-10-07T11:03:23.074593+0000","last_became_active":"2023-10-07T11:03:23.060260+0000","last_became_peered":"2023-10-07T11:03:23.060260+0000","last_unstale":"2023-10-07T11:03:23.074593+0000","last_undegraded":"2023-10-07T11:03:23.074593+0000","last_fullsized":"2023-10-07T11:03:23.074593+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051338+0000","last_change":"2023-10-07T11:03:22.048854+0000","last_active":"2023-10-07T11:03:23.051338+0000","last_peered":"2023-10-07T11:03:23.051338+0000","last_clean":"2023-10-07T11:03:23.051338+0000","last_became_active":"2023-10-07T11:03:22.048568+0000","last_became_peered":"2023-10-07T11:03:22.048568+0000","last_unstale":"2023-10-07T11:03:23.051338+0000","last_undegraded":"2023-10-07T11:03:23.051338+0000","last_fullsized":"2023-10-07T11:03:23.051338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:01:19.581526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036428+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.035995+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:33:09.391065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.034704+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:10:49.465391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051314+0000","last_change":"2023-10-07T11:03:22.060700+0000","last_active":"2023-10-07T11:03:23.051314+0000","last_peered":"2023-10-07T11:03:23.051314+0000","last_clean":"2023-10-07T11:03:23.051314+0000","last_became_active":"2023-10-07T11:03:22.060037+0000","last_became_peered":"2023-10-07T11:03:22.060037+0000","last_unstale":"2023-10-07T11:03:23.051314+0000","last_undegraded":"2023-10-07T11:03:23.051314+0000","last_fullsized":"2023-10-07T11:03:23.051314+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:53.212268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073064+0000","last_change":"2023-10-07T11:03:23.058853+0000","last_active":"2023-10-07T11:03:23.073064+0000","last_peered":"2023-10-07T11:03:23.073064+0000","last_clean":"2023-10-07T11:03:23.073064+0000","last_became_active":"2023-10-07T11:03:23.058594+0000","last_became_peered":"2023-10-07T11:03:23.058594+0000","last_unstale":"2023-10-07T11:03:23.073064+0000","last_undegraded":"2023-10-07T11:03:23.073064+0000","last_fullsized":"2023-10-07T11:03:23.073064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031592+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:51:11.826526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"186'348","reported_seq":1128,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-10-07T11:03:19.684286+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:19.684286+0000","last_peered":"2023-10-07T11:03:19.684286+0000","last_clean":"2023-10-07T11:03:19.684286+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:19.684286+0000","last_undegraded":"2023-10-07T11:03:19.684286+0000","last_fullsized":"2023-10-07T11:03:19.684286+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":52,"acting":52,"num_store_stats":3},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":1536,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312512,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059328,"statfs":{"total":96636764160,"available":96316751872,"internally_reserved":0,"allocated":3006464,"data_stored":1783889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.82199999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.98599999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.76100000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316028,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055812,"statfs":{"total":96636764160,"available":96313151488,"internally_reserved":0,"allocated":2412544,"data_stored":1193521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.98399999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.876}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.95799999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":1.0309999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543260,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307896,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063944,"statfs":{"total":96636764160,"available":96321478656,"internally_reserved":0,"allocated":2473984,"data_stored":1222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.442},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.76700000000000002,"5min":0.86299999999999999,"15min":0.94199999999999995},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.40000000000000002},"max":{"1min":0.82699999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.443,"5min":0.378,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82699999999999996,"15min":0.84099999999999997},"min":{"1min":0.55600000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.373},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.153,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84999999999999998},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80600000000000005,"15min":0.81299999999999994},"min":{"1min":0.57999999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.873,"15min":0.871},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.58399999999999996},"max":{"1min":1.052,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0980000000000001,"5min":1.157,"15min":1.302},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.877,"15min":0.88400000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.141,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88700000000000001,"15min":0.878},"min":{"1min":0.66700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.80600000000000005}]}]},{"osd":0,"up_from":14,"seq":60129543260,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1810934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.439},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.496}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.93600000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.008},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.86399999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299656,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072184,"statfs":{"total":96636764160,"available":96329916416,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.81699999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66300000000000003,"5min":0.85399999999999998,"15min":0.95099999999999996},"last":0.625},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.71199999999999997,"5min":0.92600000000000005,"15min":0.98199999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65900000000000003,"5min":0.76700000000000002,"15min":0.80000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.58299999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.77400000000000002,"15min":0.79400000000000004},"min":{"1min":0.55900000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.93600000000000005,"5min":1.163,"15min":1.1879999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.266,"15min":1.5640000000000001},"last":0.92800000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.014,"5min":1.2989999999999999,"15min":1.528},"last":1.004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78800000000000003,"15min":0.80800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97499999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.79800000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1140000000000001,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.147,"5min":1.242,"15min":1.3779999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.85699999999999998,"15min":0.874},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.052,"5min":1.2050000000000001,"15min":1.629},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.85699999999999998,"15min":0.877},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0900000000000001,"5min":1.274,"15min":1.5940000000000001},"last":0.96299999999999997}]}]},{"osd":3,"up_from":14,"seq":60129543262,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1220738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72199999999999998,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.45000000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.83999999999999997,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79700000000000004,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55500000000000005,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73099999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.5},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.436,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96799999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0780000000000001,"15min":1.222},"last":0.78100000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0720000000000001,"5min":1.27,"15min":1.27},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":0.84699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98399999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96699999999999997,"5min":1.389,"15min":1.389},"last":0.77300000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0269999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.008,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.68100000000000005}]}]},{"osd":4,"up_from":15,"seq":64424510557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316040,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055800,"statfs":{"total":96636764160,"available":96313139200,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.628},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.749},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.69899999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.66400000000000003}]}]},{"osd":5,"up_from":14,"seq":60129543259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311944,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059896,"statfs":{"total":96636764160,"available":96317333504,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78100000000000003,"15min":0.78400000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.436},"max":{"1min":0.95199999999999996,"5min":1.585,"15min":1.585},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.81299999999999994,"15min":0.81799999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.52500000000000002},"max":{"1min":0.89300000000000002,"5min":1.673,"15min":1.673},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73099999999999998,"15min":0.746},"min":{"1min":0.54100000000000004,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.90400000000000003,"5min":1.0189999999999999,"15min":1.0860000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.76900000000000002},"min":{"1min":0.54200000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.89800000000000002,"5min":0.998,"15min":1.139},"last":0.77700000000000002}]},{"osd":2,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72899999999999998,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93200000000000005,"5min":1.0349999999999999,"15min":1.0580000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89800000000000002,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.51200000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.069,"15min":1.069},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.747},"min":{"1min":0.57799999999999996,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":0.96899999999999997,"5min":1.0880000000000001,"15min":1.161},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64100000000000001,"15min":0.63},"min":{"1min":0.438,"5min":0.438,"15min":0.42799999999999999},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":0.97499999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91800000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.75800000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.378},"max":{"1min":0.93400000000000005,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70399999999999996,"15min":0.69299999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86499999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"max":{"1min":0.92000000000000004,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76900000000000002,"15min":0.75700000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:03:27.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:03:28.023 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-07T11:03:28.022+0000 7f903c3f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T11:03:28.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.172+0000 7f6fd21ff700 1 Processor -- start 2023-10-07T11:03:28.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.172+0000 7f6fd21ff700 1 -- start start 2023-10-07T11:03:28.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.173+0000 7f6fd21ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fcc0b2670 0x7f6fcc0b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.173+0000 7f6fd21ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc06f4e0 con 0x7f6fcc159550 2023-10-07T11:03:28.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.173+0000 7f6fd21ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc0b1120 con 0x7f6fcc0a98d0 2023-10-07T11:03:28.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.173+0000 7f6fd21ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fcc06e8d0 con 0x7f6fcc0b2670 2023-10-07T11:03:28.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fcb7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6fcc159550 0x7f6fcc0b0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57284/0 (socket says 172.21.15.29:57284) 2023-10-07T11:03:28.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/649715249 learned_addr learned my addr 172.21.15.29:0/649715249 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:28.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fcaffd700 1 --2- 172.21.15.29:0/649715249 >> v2:172.21.15.144:3300/0 conn(0x7f6fcc0b2670 0x7f6fcc0b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239595445 0 0) 0x7f6fcc06f4e0 con 0x7f6fcc159550 2023-10-07T11:03:28.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4003570 con 0x7f6fcc159550 2023-10-07T11:03:28.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4131244881 0 0) 0x7f6fb4003570 con 0x7f6fcc159550 2023-10-07T11:03:28.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 >> v2:172.21.15.144:3300/0 conn(0x7f6fcc0b2670 msgr2=0x7f6fcc0b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:28.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 --2- 172.21.15.29:0/649715249 >> v2:172.21.15.144:3300/0 conn(0x7f6fcc0b2670 0x7f6fcc0b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 >> v1:172.21.15.144:6789/0 conn(0x7f6fcc0a98d0 legacy=0x7f6fcc0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc06ebb0 con 0x7f6fcc159550 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f6fbc001470 con 0x7f6fcc159550 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 >> v1:172.21.15.29:6789/0 conn(0x7f6fcc159550 legacy=0x7f6fcc0b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fb4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.174+0000 7f6fca7fc700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fb4004810 0x7f6fb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fca7fc700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fb40051c0 0x7f6fb400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc06e8d0 con 0x7f6fb4003a60 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb400c050 con 0x7f6fb4004810 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb400c210 con 0x7f6fb40051c0 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fb4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcaffd700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fb4004810 0x7f6fb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcbfff700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fb40051c0 0x7f6fb400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fb40051c0 msgr2=0x7f6fb400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fb40051c0 0x7f6fb400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fb4004810 msgr2=0x7f6fb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fb4004810 0x7f6fb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/649715249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb4003570 con 0x7f6fb4003a60 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.175+0000 7f6fcbfff700 1 -- 172.21.15.29:0/649715249 reap_dead start 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fb4003e30 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c3c0 tx=0x7f6fbc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee73bf8828a5057 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00cc90 con 0x7f6fb4003a60 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc00cdf0 con 0x7f6fb4003a60 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fca7fc700 1 -- 172.21.15.29:0/649715249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc005b70 con 0x7f6fb4003a60 2023-10-07T11:03:28.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fd21ff700 1 -- 172.21.15.29:0/649715249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 msgr2=0x7f6fb4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fb4003e30 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c3c0 tx=0x7f6fbc001e70 comp rx=0 tx=0).stop 2023-10-07T11:03:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fd21ff700 1 -- 172.21.15.29:0/649715249 shutdown_connections 2023-10-07T11:03:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/649715249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fb4003e30 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.176+0000 7f6fd21ff700 1 -- 172.21.15.29:0/649715249 >> 172.21.15.29:0/649715249 conn(0x7f6fcc069190 msgr2=0x7f6fcc06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.177+0000 7f6fd21ff700 1 -- 172.21.15.29:0/649715249 shutdown_connections 2023-10-07T11:03:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.177+0000 7f6fd21ff700 1 -- 172.21.15.29:0/649715249 wait complete. 2023-10-07T11:03:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.178+0000 7f6fd21ff700 1 Processor -- start 2023-10-07T11:03:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.178+0000 7f6fd21ff700 1 -- start start 2023-10-07T11:03:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fd21ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fcc15c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fd21ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fcc15a230 0x7f6fcc15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fd21ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fcc150ed0 0x7f6fcc151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fd21ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc151890 con 0x7f6fb4003a60 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fd21ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc16a700 con 0x7f6fcc15a230 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fd21ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc169af0 con 0x7f6fcc150ed0 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fcb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fcc15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fcb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fcc15c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53290/0 (socket says 172.21.15.29:53290) 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.179+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/1072921538 learned_addr learned my addr 172.21.15.29:0/1072921538 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcaffd700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fcc15a230 0x7f6fcc15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcbfff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fcc150ed0 0x7f6fcc151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fcc150ed0 msgr2=0x7f6fcc151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fcc150ed0 0x7f6fcc151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fcc15a230 msgr2=0x7f6fcc15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fcc15a230 0x7f6fcc15a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcb7fe700 1 -- 172.21.15.29:0/1072921538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc169dd0 con 0x7f6fb4003a60 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcaffd700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fcc15a230 0x7f6fcc15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcbfff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fcc150ed0 0x7f6fcc151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fcb7fe700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fcc15c660 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc004b60 tx=0x7f6fbc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc022070 con 0x7f6fb4003a60 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc000ed0 con 0x7f6fb4003a60 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00cc90 con 0x7f6fb4003a60 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.180+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fcc16a0b0 con 0x7f6fb4003a60 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.181+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fcc169760 con 0x7f6fb4003a60 2023-10-07T11:03:28.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.182+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fcc04c3c0 con 0x7f6fb4003a60 2023-10-07T11:03:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.182+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6fbc007740 con 0x7f6fb4003a60 2023-10-07T11:03:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.183+0000 7f6fb37fe700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.183+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f6fbc085a50 con 0x7f6fb4003a60 2023-10-07T11:03:28.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.184+0000 7f6fcaffd700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.184+0000 7f6fcaffd700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6fb8066f30 0x7f6fb80693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006dc0 tx=0x7f6fc0008040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:28.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.187+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fbc052ae0 con 0x7f6fb4003a60 2023-10-07T11:03:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.386+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fcc04c3c0 con 0x7f6fb8066f30 2023-10-07T11:03:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.396+0000 7f6fb37fe700 1 -- 172.21.15.29:0/1072921538 <== mgr.8429 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282860 (secure 0 0 0) 0x7f6fcc04c3c0 con 0x7f6fb8066f30 2023-10-07T11:03:28.400 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6fb8066f30 msgr2=0x7f6fb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6fb8066f30 0x7f6fb80693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006dc0 tx=0x7f6fc0008040 comp rx=0 tx=0).stop 2023-10-07T11:03:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 msgr2=0x7f6fcc15c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fcc15c660 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc004b60 tx=0x7f6fbc015000 comp rx=0 tx=0).stop 2023-10-07T11:03:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 shutdown_connections 2023-10-07T11:03:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fcc150ed0 0x7f6fcc151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb4003a60 0x7f6fcc15c660 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 --2- 172.21.15.29:0/1072921538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fcc15a230 0x7f6fcc15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 >> 172.21.15.29:0/1072921538 conn(0x7f6fcc069190 msgr2=0x7f6fcc0aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 shutdown_connections 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.400+0000 7f6fd21ff700 1 -- 172.21.15.29:0/1072921538 wait complete. 2023-10-07T11:03:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:03:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T11:03:26.651670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591904,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":592,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":361,"ondisk_log_size":361,"up":118,"acting":118,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2476344,"kb_used_data":20216,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752498376,"statfs":{"total":773094113280,"available":770558337024,"internally_reserved":0,"allocated":20701184,"data_stored":10830568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1536,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.623060"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.061273+0000","last_change":"2023-10-07T11:03:23.059262+0000","last_active":"2023-10-07T11:03:23.061273+0000","last_peered":"2023-10-07T11:03:23.061273+0000","last_clean":"2023-10-07T11:03:23.061273+0000","last_became_active":"2023-10-07T11:03:23.058996+0000","last_became_peered":"2023-10-07T11:03:23.058996+0000","last_unstale":"2023-10-07T11:03:23.061273+0000","last_undegraded":"2023-10-07T11:03:23.061273+0000","last_fullsized":"2023-10-07T11:03:23.061273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069205+0000","last_change":"2023-10-07T11:03:23.067743+0000","last_active":"2023-10-07T11:03:23.069205+0000","last_peered":"2023-10-07T11:03:23.069205+0000","last_clean":"2023-10-07T11:03:23.069205+0000","last_became_active":"2023-10-07T11:03:23.067464+0000","last_became_peered":"2023-10-07T11:03:23.067464+0000","last_unstale":"2023-10-07T11:03:23.069205+0000","last_undegraded":"2023-10-07T11:03:23.069205+0000","last_fullsized":"2023-10-07T11:03:23.069205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.077562+0000","last_change":"2023-10-07T11:03:23.076006+0000","last_active":"2023-10-07T11:03:23.077562+0000","last_peered":"2023-10-07T11:03:23.077562+0000","last_clean":"2023-10-07T11:03:23.077562+0000","last_became_active":"2023-10-07T11:03:23.075758+0000","last_became_peered":"2023-10-07T11:03:23.075758+0000","last_unstale":"2023-10-07T11:03:23.077562+0000","last_undegraded":"2023-10-07T11:03:23.077562+0000","last_fullsized":"2023-10-07T11:03:23.077562+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.066075+0000","last_change":"2023-10-07T11:03:23.058595+0000","last_active":"2023-10-07T11:03:23.066075+0000","last_peered":"2023-10-07T11:03:23.066075+0000","last_clean":"2023-10-07T11:03:23.066075+0000","last_became_active":"2023-10-07T11:03:23.058332+0000","last_became_peered":"2023-10-07T11:03:23.058332+0000","last_unstale":"2023-10-07T11:03:23.066075+0000","last_undegraded":"2023-10-07T11:03:23.066075+0000","last_fullsized":"2023-10-07T11:03:23.066075+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.062893+0000","last_change":"2023-10-07T11:03:23.057796+0000","last_active":"2023-10-07T11:03:23.062893+0000","last_peered":"2023-10-07T11:03:23.062893+0000","last_clean":"2023-10-07T11:03:23.062893+0000","last_became_active":"2023-10-07T11:03:23.057545+0000","last_became_peered":"2023-10-07T11:03:23.057545+0000","last_unstale":"2023-10-07T11:03:23.062893+0000","last_undegraded":"2023-10-07T11:03:23.062893+0000","last_fullsized":"2023-10-07T11:03:23.062893+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072967+0000","last_change":"2023-10-07T11:03:23.056891+0000","last_active":"2023-10-07T11:03:23.072967+0000","last_peered":"2023-10-07T11:03:23.072967+0000","last_clean":"2023-10-07T11:03:23.072967+0000","last_became_active":"2023-10-07T11:03:23.056623+0000","last_became_peered":"2023-10-07T11:03:23.056623+0000","last_unstale":"2023-10-07T11:03:23.072967+0000","last_undegraded":"2023-10-07T11:03:23.072967+0000","last_fullsized":"2023-10-07T11:03:23.072967+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:24.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065743+0000","last_change":"2023-10-07T11:03:23.056041+0000","last_active":"2023-10-07T11:03:23.065743+0000","last_peered":"2023-10-07T11:03:23.065743+0000","last_clean":"2023-10-07T11:03:23.065743+0000","last_became_active":"2023-10-07T11:03:23.055408+0000","last_became_peered":"2023-10-07T11:03:23.055408+0000","last_unstale":"2023-10-07T11:03:23.065743+0000","last_undegraded":"2023-10-07T11:03:23.065743+0000","last_fullsized":"2023-10-07T11:03:23.065743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072460+0000","last_change":"2023-10-07T11:03:23.070205+0000","last_active":"2023-10-07T11:03:23.072460+0000","last_peered":"2023-10-07T11:03:23.072460+0000","last_clean":"2023-10-07T11:03:23.072460+0000","last_became_active":"2023-10-07T11:03:23.069898+0000","last_became_peered":"2023-10-07T11:03:23.069898+0000","last_unstale":"2023-10-07T11:03:23.072460+0000","last_undegraded":"2023-10-07T11:03:23.072460+0000","last_fullsized":"2023-10-07T11:03:23.072460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074629+0000","last_change":"2023-10-07T11:03:23.059218+0000","last_active":"2023-10-07T11:03:23.074629+0000","last_peered":"2023-10-07T11:03:23.074629+0000","last_clean":"2023-10-07T11:03:23.074629+0000","last_became_active":"2023-10-07T11:03:23.058961+0000","last_became_peered":"2023-10-07T11:03:23.058961+0000","last_unstale":"2023-10-07T11:03:23.074629+0000","last_undegraded":"2023-10-07T11:03:23.074629+0000","last_fullsized":"2023-10-07T11:03:23.074629+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:09.706205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069344+0000","last_change":"2023-10-07T11:03:23.068010+0000","last_active":"2023-10-07T11:03:23.069344+0000","last_peered":"2023-10-07T11:03:23.069344+0000","last_clean":"2023-10-07T11:03:23.069344+0000","last_became_active":"2023-10-07T11:03:23.067653+0000","last_became_peered":"2023-10-07T11:03:23.067653+0000","last_unstale":"2023-10-07T11:03:23.069344+0000","last_undegraded":"2023-10-07T11:03:23.069344+0000","last_fullsized":"2023-10-07T11:03:23.069344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065837+0000","last_change":"2023-10-07T11:03:23.064318+0000","last_active":"2023-10-07T11:03:23.065837+0000","last_peered":"2023-10-07T11:03:23.065837+0000","last_clean":"2023-10-07T11:03:23.065837+0000","last_became_active":"2023-10-07T11:03:23.064039+0000","last_became_peered":"2023-10-07T11:03:23.064039+0000","last_unstale":"2023-10-07T11:03:23.065837+0000","last_undegraded":"2023-10-07T11:03:23.065837+0000","last_fullsized":"2023-10-07T11:03:23.065837+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074670+0000","last_change":"2023-10-07T11:03:23.061128+0000","last_active":"2023-10-07T11:03:23.074670+0000","last_peered":"2023-10-07T11:03:23.074670+0000","last_clean":"2023-10-07T11:03:23.074670+0000","last_became_active":"2023-10-07T11:03:23.060886+0000","last_became_peered":"2023-10-07T11:03:23.060886+0000","last_unstale":"2023-10-07T11:03:23.074670+0000","last_undegraded":"2023-10-07T11:03:23.074670+0000","last_fullsized":"2023-10-07T11:03:23.074670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036512+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074352+0000","last_change":"2023-10-07T11:03:23.071098+0000","last_active":"2023-10-07T11:03:23.074352+0000","last_peered":"2023-10-07T11:03:23.074352+0000","last_clean":"2023-10-07T11:03:23.074352+0000","last_became_active":"2023-10-07T11:03:23.070446+0000","last_became_peered":"2023-10-07T11:03:23.070446+0000","last_unstale":"2023-10-07T11:03:23.074352+0000","last_undegraded":"2023-10-07T11:03:23.074352+0000","last_fullsized":"2023-10-07T11:03:23.074352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036127+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1d","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.078939+0000","last_change":"2023-10-07T11:03:22.046593+0000","last_active":"2023-10-07T11:03:23.078939+0000","last_peered":"2023-10-07T11:03:23.078939+0000","last_clean":"2023-10-07T11:03:23.078939+0000","last_became_active":"2023-10-07T11:03:22.046326+0000","last_became_peered":"2023-10-07T11:03:22.046326+0000","last_unstale":"2023-10-07T11:03:23.078939+0000","last_undegraded":"2023-10-07T11:03:23.078939+0000","last_fullsized":"2023-10-07T11:03:23.078939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.034708+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:06:28.068685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065766+0000","last_change":"2023-10-07T11:03:23.056224+0000","last_active":"2023-10-07T11:03:23.065766+0000","last_peered":"2023-10-07T11:03:23.065766+0000","last_clean":"2023-10-07T11:03:23.065766+0000","last_became_active":"2023-10-07T11:03:23.055757+0000","last_became_peered":"2023-10-07T11:03:23.055757+0000","last_unstale":"2023-10-07T11:03:23.065766+0000","last_undegraded":"2023-10-07T11:03:23.065766+0000","last_fullsized":"2023-10-07T11:03:23.065766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036798+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:41:07.129981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060572+0000","last_change":"2023-10-07T11:03:23.056936+0000","last_active":"2023-10-07T11:03:23.060572+0000","last_peered":"2023-10-07T11:03:23.060572+0000","last_clean":"2023-10-07T11:03:23.060572+0000","last_became_active":"2023-10-07T11:03:23.056670+0000","last_became_peered":"2023-10-07T11:03:23.056670+0000","last_unstale":"2023-10-07T11:03:23.060572+0000","last_undegraded":"2023-10-07T11:03:23.060572+0000","last_fullsized":"2023-10-07T11:03:23.060572+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082675+0000","last_change":"2023-10-07T11:03:22.049469+0000","last_active":"2023-10-07T11:03:23.082675+0000","last_peered":"2023-10-07T11:03:23.082675+0000","last_clean":"2023-10-07T11:03:23.082675+0000","last_became_active":"2023-10-07T11:03:22.049231+0000","last_became_peered":"2023-10-07T11:03:22.049231+0000","last_unstale":"2023-10-07T11:03:23.082675+0000","last_undegraded":"2023-10-07T11:03:23.082675+0000","last_fullsized":"2023-10-07T11:03:23.082675+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:51.063075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031851+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:29:14.104837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060058+0000","last_change":"2023-10-07T11:03:23.057682+0000","last_active":"2023-10-07T11:03:23.060058+0000","last_peered":"2023-10-07T11:03:23.060058+0000","last_clean":"2023-10-07T11:03:23.060058+0000","last_became_active":"2023-10-07T11:03:23.056838+0000","last_became_peered":"2023-10-07T11:03:23.056838+0000","last_unstale":"2023-10-07T11:03:23.060058+0000","last_undegraded":"2023-10-07T11:03:23.060058+0000","last_fullsized":"2023-10-07T11:03:23.060058+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.030990+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:23:58.478124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074728+0000","last_change":"2023-10-07T11:03:23.066795+0000","last_active":"2023-10-07T11:03:23.074728+0000","last_peered":"2023-10-07T11:03:23.074728+0000","last_clean":"2023-10-07T11:03:23.074728+0000","last_became_active":"2023-10-07T11:03:23.066509+0000","last_became_peered":"2023-10-07T11:03:23.066509+0000","last_unstale":"2023-10-07T11:03:23.074728+0000","last_undegraded":"2023-10-07T11:03:23.074728+0000","last_fullsized":"2023-10-07T11:03:23.074728+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:40:40.866050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036810+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:50:02.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036981+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:20:03.006488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072955+0000","last_change":"2023-10-07T11:03:23.056349+0000","last_active":"2023-10-07T11:03:23.072955+0000","last_peered":"2023-10-07T11:03:23.072955+0000","last_clean":"2023-10-07T11:03:23.072955+0000","last_became_active":"2023-10-07T11:03:23.055709+0000","last_became_peered":"2023-10-07T11:03:23.055709+0000","last_unstale":"2023-10-07T11:03:23.072955+0000","last_undegraded":"2023-10-07T11:03:23.072955+0000","last_fullsized":"2023-10-07T11:03:23.072955+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:34:35.644574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"196'6","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.097246+0000","last_change":"2023-10-07T11:03:22.043324+0000","last_active":"2023-10-07T11:03:23.097246+0000","last_peered":"2023-10-07T11:03:23.097246+0000","last_clean":"2023-10-07T11:03:23.097246+0000","last_became_active":"2023-10-07T11:03:22.043036+0000","last_became_peered":"2023-10-07T11:03:22.043036+0000","last_unstale":"2023-10-07T11:03:23.097246+0000","last_undegraded":"2023-10-07T11:03:23.097246+0000","last_fullsized":"2023-10-07T11:03:23.097246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072666+0000","last_change":"2023-10-07T11:03:23.070261+0000","last_active":"2023-10-07T11:03:23.072666+0000","last_peered":"2023-10-07T11:03:23.072666+0000","last_clean":"2023-10-07T11:03:23.072666+0000","last_became_active":"2023-10-07T11:03:23.069876+0000","last_became_peered":"2023-10-07T11:03:23.069876+0000","last_unstale":"2023-10-07T11:03:23.072666+0000","last_undegraded":"2023-10-07T11:03:23.072666+0000","last_fullsized":"2023-10-07T11:03:23.072666+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.032223+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:55:51.785763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046972+0000","last_change":"2023-10-07T11:03:22.058631+0000","last_active":"2023-10-07T11:03:23.046972+0000","last_peered":"2023-10-07T11:03:23.046972+0000","last_clean":"2023-10-07T11:03:23.046972+0000","last_became_active":"2023-10-07T11:03:22.058368+0000","last_became_peered":"2023-10-07T11:03:22.058368+0000","last_unstale":"2023-10-07T11:03:23.046972+0000","last_undegraded":"2023-10-07T11:03:23.046972+0000","last_fullsized":"2023-10-07T11:03:23.046972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:41.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050617+0000","last_change":"2023-10-07T11:03:22.046240+0000","last_active":"2023-10-07T11:03:23.050617+0000","last_peered":"2023-10-07T11:03:23.050617+0000","last_clean":"2023-10-07T11:03:23.050617+0000","last_became_active":"2023-10-07T11:03:22.045964+0000","last_became_peered":"2023-10-07T11:03:22.045964+0000","last_unstale":"2023-10-07T11:03:23.050617+0000","last_undegraded":"2023-10-07T11:03:23.050617+0000","last_fullsized":"2023-10-07T11:03:23.050617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:25.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072532+0000","last_change":"2023-10-07T11:03:23.069327+0000","last_active":"2023-10-07T11:03:23.072532+0000","last_peered":"2023-10-07T11:03:23.072532+0000","last_clean":"2023-10-07T11:03:23.072532+0000","last_became_active":"2023-10-07T11:03:23.068992+0000","last_became_peered":"2023-10-07T11:03:23.068992+0000","last_unstale":"2023-10-07T11:03:23.072532+0000","last_undegraded":"2023-10-07T11:03:23.072532+0000","last_fullsized":"2023-10-07T11:03:23.072532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031259+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:19:57.495871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074503+0000","last_change":"2023-10-07T11:03:23.069902+0000","last_active":"2023-10-07T11:03:23.074503+0000","last_peered":"2023-10-07T11:03:23.074503+0000","last_clean":"2023-10-07T11:03:23.074503+0000","last_became_active":"2023-10-07T11:03:23.069641+0000","last_became_peered":"2023-10-07T11:03:23.069641+0000","last_unstale":"2023-10-07T11:03:23.074503+0000","last_undegraded":"2023-10-07T11:03:23.074503+0000","last_fullsized":"2023-10-07T11:03:23.074503+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.035995+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.e","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083381+0000","last_change":"2023-10-07T11:03:22.049580+0000","last_active":"2023-10-07T11:03:23.083381+0000","last_peered":"2023-10-07T11:03:23.083381+0000","last_clean":"2023-10-07T11:03:23.083381+0000","last_became_active":"2023-10-07T11:03:22.049343+0000","last_became_peered":"2023-10-07T11:03:22.049343+0000","last_unstale":"2023-10-07T11:03:23.083381+0000","last_undegraded":"2023-10-07T11:03:23.083381+0000","last_fullsized":"2023-10-07T11:03:23.083381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:39.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031658+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:11:32.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031011+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:10:36.979433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072519+0000","last_change":"2023-10-07T11:03:23.069558+0000","last_active":"2023-10-07T11:03:23.072519+0000","last_peered":"2023-10-07T11:03:23.072519+0000","last_clean":"2023-10-07T11:03:23.072519+0000","last_became_active":"2023-10-07T11:03:23.069179+0000","last_became_peered":"2023-10-07T11:03:23.069179+0000","last_unstale":"2023-10-07T11:03:23.072519+0000","last_undegraded":"2023-10-07T11:03:23.072519+0000","last_fullsized":"2023-10-07T11:03:23.072519+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081863+0000","last_change":"2023-10-07T11:03:22.060469+0000","last_active":"2023-10-07T11:03:23.081863+0000","last_peered":"2023-10-07T11:03:23.081863+0000","last_clean":"2023-10-07T11:03:23.081863+0000","last_became_active":"2023-10-07T11:03:22.060024+0000","last_became_peered":"2023-10-07T11:03:22.060024+0000","last_unstale":"2023-10-07T11:03:23.081863+0000","last_undegraded":"2023-10-07T11:03:23.081863+0000","last_fullsized":"2023-10-07T11:03:23.081863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:42.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036035+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050442+0000","last_change":"2023-10-07T11:03:22.059410+0000","last_active":"2023-10-07T11:03:23.050442+0000","last_peered":"2023-10-07T11:03:23.050442+0000","last_clean":"2023-10-07T11:03:23.050442+0000","last_became_active":"2023-10-07T11:03:22.059029+0000","last_became_peered":"2023-10-07T11:03:22.059029+0000","last_unstale":"2023-10-07T11:03:23.050442+0000","last_undegraded":"2023-10-07T11:03:23.050442+0000","last_fullsized":"2023-10-07T11:03:23.050442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:57:18.642115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036998+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.032308+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:40:28.034611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070233+0000","last_change":"2023-10-07T11:03:23.062012+0000","last_active":"2023-10-07T11:03:23.070233+0000","last_peered":"2023-10-07T11:03:23.070233+0000","last_clean":"2023-10-07T11:03:23.070233+0000","last_became_active":"2023-10-07T11:03:23.061769+0000","last_became_peered":"2023-10-07T11:03:23.061769+0000","last_unstale":"2023-10-07T11:03:23.070233+0000","last_undegraded":"2023-10-07T11:03:23.070233+0000","last_fullsized":"2023-10-07T11:03:23.070233+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046870+0000","last_change":"2023-10-07T11:03:22.051956+0000","last_active":"2023-10-07T11:03:23.046870+0000","last_peered":"2023-10-07T11:03:23.046870+0000","last_clean":"2023-10-07T11:03:23.046870+0000","last_became_active":"2023-10-07T11:03:22.051691+0000","last_became_peered":"2023-10-07T11:03:22.051691+0000","last_unstale":"2023-10-07T11:03:23.046870+0000","last_undegraded":"2023-10-07T11:03:23.046870+0000","last_fullsized":"2023-10-07T11:03:23.046870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:42.177620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065367+0000","last_change":"2023-10-07T11:03:23.059948+0000","last_active":"2023-10-07T11:03:23.065367+0000","last_peered":"2023-10-07T11:03:23.065367+0000","last_clean":"2023-10-07T11:03:23.065367+0000","last_became_active":"2023-10-07T11:03:23.059564+0000","last_became_peered":"2023-10-07T11:03:23.059564+0000","last_unstale":"2023-10-07T11:03:23.065367+0000","last_undegraded":"2023-10-07T11:03:23.065367+0000","last_fullsized":"2023-10-07T11:03:23.065367+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:04:25.554055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046689+0000","last_change":"2023-10-07T11:03:22.052493+0000","last_active":"2023-10-07T11:03:23.046689+0000","last_peered":"2023-10-07T11:03:23.046689+0000","last_clean":"2023-10-07T11:03:23.046689+0000","last_became_active":"2023-10-07T11:03:22.052235+0000","last_became_peered":"2023-10-07T11:03:22.052235+0000","last_unstale":"2023-10-07T11:03:23.046689+0000","last_undegraded":"2023-10-07T11:03:23.046689+0000","last_fullsized":"2023-10-07T11:03:23.046689+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:06.410397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074593+0000","last_change":"2023-10-07T11:03:23.060626+0000","last_active":"2023-10-07T11:03:23.074593+0000","last_peered":"2023-10-07T11:03:23.074593+0000","last_clean":"2023-10-07T11:03:23.074593+0000","last_became_active":"2023-10-07T11:03:23.060260+0000","last_became_peered":"2023-10-07T11:03:23.060260+0000","last_unstale":"2023-10-07T11:03:23.074593+0000","last_undegraded":"2023-10-07T11:03:23.074593+0000","last_fullsized":"2023-10-07T11:03:23.074593+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051338+0000","last_change":"2023-10-07T11:03:22.048854+0000","last_active":"2023-10-07T11:03:23.051338+0000","last_peered":"2023-10-07T11:03:23.051338+0000","last_clean":"2023-10-07T11:03:23.051338+0000","last_became_active":"2023-10-07T11:03:22.048568+0000","last_became_peered":"2023-10-07T11:03:22.048568+0000","last_unstale":"2023-10-07T11:03:23.051338+0000","last_undegraded":"2023-10-07T11:03:23.051338+0000","last_fullsized":"2023-10-07T11:03:23.051338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:01:19.581526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.036428+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.035995+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:33:09.391065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.034704+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:10:49.465391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-07T11:03:22.027197+0000","last_change":"2023-10-07T11:03:22.027197+0000","last_active":"2023-10-07T11:03:22.027197+0000","last_peered":"2023-10-07T11:03:22.027197+0000","last_clean":"2023-10-07T11:03:22.027197+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:22.027197+0000","last_undegraded":"2023-10-07T11:03:22.027197+0000","last_fullsized":"2023-10-07T11:03:22.027197+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051314+0000","last_change":"2023-10-07T11:03:22.060700+0000","last_active":"2023-10-07T11:03:23.051314+0000","last_peered":"2023-10-07T11:03:23.051314+0000","last_clean":"2023-10-07T11:03:23.051314+0000","last_became_active":"2023-10-07T11:03:22.060037+0000","last_became_peered":"2023-10-07T11:03:22.060037+0000","last_unstale":"2023-10-07T11:03:23.051314+0000","last_undegraded":"2023-10-07T11:03:23.051314+0000","last_fullsized":"2023-10-07T11:03:23.051314+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:53.212268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073064+0000","last_change":"2023-10-07T11:03:23.058853+0000","last_active":"2023-10-07T11:03:23.073064+0000","last_peered":"2023-10-07T11:03:23.073064+0000","last_clean":"2023-10-07T11:03:23.073064+0000","last_became_active":"2023-10-07T11:03:23.058594+0000","last_became_peered":"2023-10-07T11:03:23.058594+0000","last_unstale":"2023-10-07T11:03:23.073064+0000","last_undegraded":"2023-10-07T11:03:23.073064+0000","last_fullsized":"2023-10-07T11:03:23.073064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-07T11:03:21.021570+0000","last_change":"2023-10-07T11:03:21.031592+0000","last_active":"2023-10-07T11:03:21.021570+0000","last_peered":"2023-10-07T11:03:21.021570+0000","last_clean":"2023-10-07T11:03:21.021570+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-07T11:03:21.021570+0000","last_undegraded":"2023-10-07T11:03:21.021570+0000","last_fullsized":"2023-10-07T11:03:21.021570+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:51:11.826526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"186'348","reported_seq":1128,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-10-07T11:03:19.684286+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:19.684286+0000","last_peered":"2023-10-07T11:03:19.684286+0000","last_clean":"2023-10-07T11:03:19.684286+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:19.684286+0000","last_undegraded":"2023-10-07T11:03:19.684286+0000","last_fullsized":"2023-10-07T11:03:19.684286+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":52,"acting":52,"num_store_stats":3},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":1536,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312512,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059328,"statfs":{"total":96636764160,"available":96316751872,"internally_reserved":0,"allocated":3006464,"data_stored":1783889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.82199999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.98599999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.76100000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316028,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055812,"statfs":{"total":96636764160,"available":96313151488,"internally_reserved":0,"allocated":2412544,"data_stored":1193521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.98399999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.876}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.95799999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":1.0309999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543260,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307896,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063944,"statfs":{"total":96636764160,"available":96321478656,"internally_reserved":0,"allocated":2473984,"data_stored":1222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.442},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.76700000000000002,"5min":0.86299999999999999,"15min":0.94199999999999995},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.40000000000000002},"max":{"1min":0.82699999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.443,"5min":0.378,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82699999999999996,"15min":0.84099999999999997},"min":{"1min":0.55600000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.373},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.153,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84999999999999998},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80600000000000005,"15min":0.81299999999999994},"min":{"1min":0.57999999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.873,"15min":0.871},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.58399999999999996},"max":{"1min":1.052,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0980000000000001,"5min":1.157,"15min":1.302},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.877,"15min":0.88400000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.141,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88700000000000001,"15min":0.878},"min":{"1min":0.66700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.80600000000000005}]}]},{"osd":0,"up_from":14,"seq":60129543260,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1810934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.439},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.496}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.93600000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.008},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.86399999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299656,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072184,"statfs":{"total":96636764160,"available":96329916416,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.81699999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66300000000000003,"5min":0.85399999999999998,"15min":0.95099999999999996},"last":0.625},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.71199999999999997,"5min":0.92600000000000005,"15min":0.98199999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65900000000000003,"5min":0.76700000000000002,"15min":0.80000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.58299999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.77400000000000002,"15min":0.79400000000000004},"min":{"1min":0.55900000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.93600000000000005,"5min":1.163,"15min":1.1879999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.266,"15min":1.5640000000000001},"last":0.92800000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.014,"5min":1.2989999999999999,"15min":1.528},"last":1.004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78800000000000003,"15min":0.80800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97499999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.79800000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1140000000000001,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.147,"5min":1.242,"15min":1.3779999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.85699999999999998,"15min":0.874},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.052,"5min":1.2050000000000001,"15min":1.629},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.85699999999999998,"15min":0.877},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0900000000000001,"5min":1.274,"15min":1.5940000000000001},"last":0.96299999999999997}]}]},{"osd":3,"up_from":14,"seq":60129543262,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1220738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72199999999999998,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.45000000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.83999999999999997,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79700000000000004,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55500000000000005,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73099999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.5},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.436,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96799999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0780000000000001,"15min":1.222},"last":0.78100000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0720000000000001,"5min":1.27,"15min":1.27},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":0.84699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98399999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96699999999999997,"5min":1.389,"15min":1.389},"last":0.77300000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0269999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.008,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.68100000000000005}]}]},{"osd":4,"up_from":15,"seq":64424510557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316040,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055800,"statfs":{"total":96636764160,"available":96313139200,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.628},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.749},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.69899999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.66400000000000003}]}]},{"osd":5,"up_from":14,"seq":60129543259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311944,"kb_used_data":2368,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059896,"statfs":{"total":96636764160,"available":96317333504,"internally_reserved":0,"allocated":2424832,"data_stored":1199802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78100000000000003,"15min":0.78400000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.436},"max":{"1min":0.95199999999999996,"5min":1.585,"15min":1.585},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.81299999999999994,"15min":0.81799999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.52500000000000002},"max":{"1min":0.89300000000000002,"5min":1.673,"15min":1.673},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73099999999999998,"15min":0.746},"min":{"1min":0.54100000000000004,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.90400000000000003,"5min":1.0189999999999999,"15min":1.0860000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.76900000000000002},"min":{"1min":0.54200000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.89800000000000002,"5min":0.998,"15min":1.139},"last":0.77700000000000002}]},{"osd":2,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72899999999999998,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93200000000000005,"5min":1.0349999999999999,"15min":1.0580000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89800000000000002,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.51200000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.069,"15min":1.069},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.747},"min":{"1min":0.57799999999999996,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":0.96899999999999997,"5min":1.0880000000000001,"15min":1.161},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64100000000000001,"15min":0.63},"min":{"1min":0.438,"5min":0.438,"15min":0.42799999999999999},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":0.97499999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91800000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.75800000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.378},"max":{"1min":0.93400000000000005,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70399999999999996,"15min":0.69299999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86499999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"max":{"1min":0.92000000000000004,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76900000000000002,"15min":0.75700000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.3f is not active+clean 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.3e is not active+clean 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.3c is not active+clean 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.3b is not active+clean 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.39 is not active+clean 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.37 is not active+clean 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.35 is not active+clean 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.34 is not active+clean 2023-10-07T11:03:28.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.33 is not active+clean 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.32 is not active+clean 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.31 is not active+clean 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.2f is not active+clean 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.2e is not active+clean 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.2c is not active+clean 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.2b is not active+clean 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.27 is not active+clean 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.26 is not active+clean 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.24 is not active+clean 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.23 is not active+clean 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.22 is not active+clean 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.20 is not active+clean 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1f is not active+clean 2023-10-07T11:03:28.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036512+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1e is not active+clean 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036127+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1c is not active+clean 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1d is not active+clean 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1c is not active+clean 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.034708+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T17:06:28.068685+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T11:03:28.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1b is not active+clean 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036798+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T16:41:07.129981+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.19 is not active+clean 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.18 is not active+clean 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.031851+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:29:14.104837+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.17 is not active+clean 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.030990+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:23:58.478124+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.16 is not active+clean 2023-10-07T11:03:28.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036810+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:50:02.684252+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.14 is not active+clean 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.15 is not active+clean 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036981+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T12:20:03.006488+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.13 is not active+clean 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.032223+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T17:55:51.785763+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.13 is not active+clean 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.10 is not active+clean 2023-10-07T11:03:28.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.10 is not active+clean 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.031259+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T21:19:57.495871+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.f is not active+clean 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.035995+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T22:07:02.427703+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.f is not active+clean 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.c is not active+clean 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.d is not active+clean 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.031658+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T17:11:32.690428+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.d is not active+clean 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.c is not active+clean 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.031011+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:10:36.979433+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.b is not active+clean 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.a is not active+clean 2023-10-07T11:03:28.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036035+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.8 is not active+clean 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.9 is not active+clean 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.8 is not active+clean 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036998+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T18:35:22.928385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.6 is not active+clean 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.7 is not active+clean 2023-10-07T11:03:28.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.032308+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T15:40:28.034611+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.2 is not active+clean 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3 is not active+clean 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.036428+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T11:09:06.585880+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.3 is not active+clean 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2 is not active+clean 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.035995+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T20:33:09.391065+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-07T11:03:28.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.0 is not active+clean 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1 is not active+clean 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.034704+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T14:10:49.465391+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1 is not active+clean 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-07T11:03:22.027197+0000', 'last_change': '2023-10-07T11:03:22.027197+0000', 'last_active': '2023-10-07T11:03:22.027197+0000', 'last_peered': '2023-10-07T11:03:22.027197+0000', 'last_clean': '2023-10-07T11:03:22.027197+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:22.027197+0000', 'last_undegraded': '2023-10-07T11:03:22.027197+0000', 'last_fullsized': '2023-10-07T11:03:22.027197+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:22.027197+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.19 is not active+clean 2023-10-07T11:03:28.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-10-07T11:03:21.021570+0000', 'last_change': '2023-10-07T11:03:21.031592+0000', 'last_active': '2023-10-07T11:03:21.021570+0000', 'last_peered': '2023-10-07T11:03:21.021570+0000', 'last_clean': '2023-10-07T11:03:21.021570+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-07T11:03:21.021570+0000', 'last_undegraded': '2023-10-07T11:03:21.021570+0000', 'last_fullsized': '2023-10-07T11:03:21.021570+0000', 'mapping_epoch': 194, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 194, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'last_clean_scrub_stamp': '2023-10-07T11:03:21.021570+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-08T19:51:11.826526+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-07T11:03:28.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-07T11:03:28.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.653+0000 7fb2179ff700 1 Processor -- start 2023-10-07T11:03:28.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.653+0000 7fb2179ff700 1 -- start start 2023-10-07T11:03:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.654+0000 7fb2179ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2100b2a20 0x7fb2100cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.654+0000 7fb2179ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2100cc940 con 0x7fb2100cb6f0 2023-10-07T11:03:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.654+0000 7fb2179ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2100b2df0 con 0x7fb2100cf2c0 2023-10-07T11:03:28.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.654+0000 7fb2179ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2100cbd30 con 0x7fb2100b2a20 2023-10-07T11:03:28.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb21579b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2100b2a20 0x7fb2100cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb214f9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2100cb6f0 0x7fb2100cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57300/0 (socket says 172.21.15.29:57300) 2023-10-07T11:03:28.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb214f9a700 1 -- 172.21.15.29:0/771175546 learned_addr learned my addr 172.21.15.29:0/771175546 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:28.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb21579b700 1 -- 172.21.15.29:0/771175546 >> v1:172.21.15.144:6789/0 conn(0x7fb2100cf2c0 legacy=0x7fb2100c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb21579b700 1 -- 172.21.15.29:0/771175546 >> v1:172.21.15.29:6789/0 conn(0x7fb2100cb6f0 legacy=0x7fb2100cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb21579b700 1 -- 172.21.15.29:0/771175546 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100cc010 con 0x7fb2100b2a20 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb207fff700 1 -- 172.21.15.29:0/771175546 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826468503 0 0) 0x7fb2100cc940 con 0x7fb2100cb6f0 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.655+0000 7fb21579b700 1 --2- 172.21.15.29:0/771175546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100b2a20 0x7fb2100cb1b0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0098d0 tx=0x7fb20c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33762f645a6c20f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb207fff700 1 -- 172.21.15.29:0/771175546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c009e70 con 0x7fb2100b2a20 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb207fff700 1 -- 172.21.15.29:0/771175546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c00b860 con 0x7fb2100b2a20 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb207fff700 1 -- 172.21.15.29:0/771175546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00bc00 con 0x7fb2100b2a20 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 -- 172.21.15.29:0/771175546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100b2a20 msgr2=0x7fb2100cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 --2- 172.21.15.29:0/771175546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100b2a20 0x7fb2100cb1b0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0098d0 tx=0x7fb20c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 -- 172.21.15.29:0/771175546 shutdown_connections 2023-10-07T11:03:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 --2- 172.21.15.29:0/771175546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100b2a20 0x7fb2100cb1b0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 -- 172.21.15.29:0/771175546 >> 172.21.15.29:0/771175546 conn(0x7fb210099820 msgr2=0x7fb210063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:28.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 -- 172.21.15.29:0/771175546 shutdown_connections 2023-10-07T11:03:28.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.656+0000 7fb2179ff700 1 -- 172.21.15.29:0/771175546 wait complete. 2023-10-07T11:03:28.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.658+0000 7fb2179ff700 1 Processor -- start 2023-10-07T11:03:28.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.658+0000 7fb2179ff700 1 -- start start 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb2179ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2100b2a20 0x7fb210157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb2179ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 0x7fb21015ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb2179ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2100cf2c0 0x7fb21015b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb2179ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2101570d0 con 0x7fb2100cb6f0 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb2179ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2100b44e0 con 0x7fb2100cf2c0 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb2179ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb210156a80 con 0x7fb2100b2a20 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb214f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 0x7fb21015ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb214f9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 0x7fb21015ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53300/0 (socket says 172.21.15.29:53300) 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb214f9a700 1 -- 172.21.15.29:0/271085578 learned_addr learned my addr 172.21.15.29:0/271085578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb21579b700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2100b2a20 0x7fb210157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.659+0000 7fb215f9c700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2100cf2c0 0x7fb21015b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb214f9a700 1 -- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2100b2a20 msgr2=0x7fb210157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb214f9a700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2100b2a20 0x7fb210157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb214f9a700 1 -- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2100cf2c0 msgr2=0x7fb21015b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb214f9a700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2100cf2c0 0x7fb21015b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb214f9a700 1 -- 172.21.15.29:0/271085578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0093c0 con 0x7fb2100cb6f0 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb215f9c700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2100cf2c0 0x7fb21015b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:28.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb214f9a700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 0x7fb21015ae70 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c810 tx=0x7fb200007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:28.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200005c00 con 0x7fb2100cb6f0 2023-10-07T11:03:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb200005d60 con 0x7fb2100cb6f0 2023-10-07T11:03:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200007320 con 0x7fb2100cb6f0 2023-10-07T11:03:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.660+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb210156d60 con 0x7fb2100cb6f0 2023-10-07T11:03:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.661+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2100b6010 con 0x7fb2100cb6f0 2023-10-07T11:03:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.661+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21004c3c0 con 0x7fb2100cb6f0 2023-10-07T11:03:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.663+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb20001b3b0 con 0x7fb2100cb6f0 2023-10-07T11:03:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.663+0000 7fb2057fa700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb1f4066f30 0x7fb1f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.664+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fb200083c60 con 0x7fb2100cb6f0 2023-10-07T11:03:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.664+0000 7fb21579b700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb1f4066f30 0x7fb1f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.665+0000 7fb21579b700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb1f4066f30 0x7fb1f40693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009390 tx=0x7fb20c017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.668+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb200050cf0 con 0x7fb2100cb6f0 2023-10-07T11:03:28.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.950+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb2100cc2f0 con 0x7fb2100cb6f0 2023-10-07T11:03:28.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.950+0000 7fb2057fa700 1 -- 172.21.15.29:0/271085578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1182 (secure 0 0 0) 0x7fb200017080 con 0x7fb2100cb6f0 2023-10-07T11:03:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:28.952 INFO:teuthology.orchestra.run.smithi029.stdout:{"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5671,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":197,"num_osds":8,"num_up_osds":8,"osd_up_since":1696670950,"num_in_osds":8,"osd_in_since":1696670940,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":39},{"state_name":"unknown","count":38},{"state_name":"creating+peering","count":20}],"num_pgs":97,"num_pools":3,"num_objects":9,"data_bytes":591904,"bytes_used":2535776256,"bytes_avail":770558337024,"bytes_total":773094113280,"unknown_pgs_ratio":0.39175257086753845,"inactive_pgs_ratio":0.20618556439876556,"write_bytes_sec":546,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":101,"id":12,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":12,"rank":0,"name":"c","status":"up:active","gid":10104}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2023-10-07T11:02:58.876670+0000","services":{}},"progress_events":{}} 2023-10-07T11:03:28.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb1f4066f30 msgr2=0x7fb1f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb1f4066f30 0x7fb1f40693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009390 tx=0x7fb20c017000 comp rx=0 tx=0).stop 2023-10-07T11:03:28.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 msgr2=0x7fb21015ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:28.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 0x7fb21015ae70 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c810 tx=0x7fb200007d90 comp rx=0 tx=0).stop 2023-10-07T11:03:28.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 shutdown_connections 2023-10-07T11:03:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2100cf2c0 0x7fb21015b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb1f4066f30 0x7fb1f40693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2100b2a20 0x7fb210157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 --2- 172.21.15.29:0/271085578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2100cb6f0 0x7fb21015ae70 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 >> 172.21.15.29:0/271085578 conn(0x7fb210099820 msgr2=0x7fb21015fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:28.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 shutdown_connections 2023-10-07T11:03:28.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:28.953+0000 7fb2179ff700 1 -- 172.21.15.29:0/271085578 wait complete. 2023-10-07T11:03:28.971 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-07T11:03:28.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:03:29.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.167+0000 7f0c5bfff700 1 Processor -- start 2023-10-07T11:03:29.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.167+0000 7f0c5bfff700 1 -- start start 2023-10-07T11:03:29.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.168+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c54159b20 0x7f0c540d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:29.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.168+0000 7f0c5bfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0c540ce8a0 0x7f0c540c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:29.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.168+0000 7f0c5bfff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540cbf20 con 0x7f0c54159750 2023-10-07T11:03:29.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.168+0000 7f0c5bfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540cc200 con 0x7f0c540ce8a0 2023-10-07T11:03:29.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.168+0000 7f0c5bfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540cc4e0 con 0x7f0c54159b20 2023-10-07T11:03:29.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.168+0000 7f0c59d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0c54159750 0x7f0c540ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57306/0 (socket says 172.21.15.29:57306) 2023-10-07T11:03:29.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c59d9b700 1 -- 172.21.15.29:0/331781930 learned_addr learned my addr 172.21.15.29:0/331781930 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:29.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5959a700 1 --2- 172.21.15.29:0/331781930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c54159b20 0x7f0c540d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:29.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5a59c700 1 --2- 172.21.15.29:0/331781930 >> v2:172.21.15.144:3300/0 conn(0x7f0c540ce8a0 0x7f0c540c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c58d99700 1 -- 172.21.15.29:0/331781930 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087065132 0 0) 0x7f0c540cbf20 con 0x7f0c54159750 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c58d99700 1 -- 172.21.15.29:0/331781930 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003570 con 0x7f0c54159750 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5959a700 1 -- 172.21.15.29:0/331781930 >> v2:172.21.15.144:3300/0 conn(0x7f0c540ce8a0 msgr2=0x7f0c540c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5959a700 1 --2- 172.21.15.29:0/331781930 >> v2:172.21.15.144:3300/0 conn(0x7f0c540ce8a0 0x7f0c540c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5959a700 1 -- 172.21.15.29:0/331781930 >> v1:172.21.15.29:6789/0 conn(0x7f0c54159750 legacy=0x7f0c540ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5959a700 1 -- 172.21.15.29:0/331781930 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540cc7c0 con 0x7f0c54159b20 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.169+0000 7f0c5a59c700 1 --2- 172.21.15.29:0/331781930 >> v2:172.21.15.144:3300/0 conn(0x7f0c540ce8a0 0x7f0c540c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.170+0000 7f0c5959a700 1 --2- 172.21.15.29:0/331781930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c54159b20 0x7f0c540d10a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0c440098e0 tx=0x7f0c440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7770c08d3921d42a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:29.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.170+0000 7f0c58d99700 1 -- 172.21.15.29:0/331781930 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44011630 con 0x7f0c54159b20 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.170+0000 7f0c58d99700 1 -- 172.21.15.29:0/331781930 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c44011790 con 0x7f0c54159b20 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.170+0000 7f0c58d99700 1 -- 172.21.15.29:0/331781930 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44011b10 con 0x7f0c54159b20 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.170+0000 7f0c5bfff700 1 -- 172.21.15.29:0/331781930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c54159b20 msgr2=0x7f0c540d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.170+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/331781930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c54159b20 0x7f0c540d10a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0c440098e0 tx=0x7f0c440091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.171+0000 7f0c5bfff700 1 -- 172.21.15.29:0/331781930 shutdown_connections 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.171+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/331781930 >> v2:172.21.15.144:3300/0 conn(0x7f0c540ce8a0 0x7f0c540c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.171+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/331781930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c54159b20 0x7f0c540d10a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.171+0000 7f0c5bfff700 1 -- 172.21.15.29:0/331781930 >> 172.21.15.29:0/331781930 conn(0x7f0c54063a10 msgr2=0x7f0c540d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.171+0000 7f0c5bfff700 1 -- 172.21.15.29:0/331781930 shutdown_connections 2023-10-07T11:03:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.171+0000 7f0c5bfff700 1 -- 172.21.15.29:0/331781930 wait complete. 2023-10-07T11:03:29.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.172+0000 7f0c5bfff700 1 Processor -- start 2023-10-07T11:03:29.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.172+0000 7f0c5bfff700 1 -- start start 2023-10-07T11:03:29.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c540ce8a0 0x7f0c5416a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:29.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 0x7f0c54151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c54159b20 0x7f0c5415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54152300 con 0x7f0c54159750 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5bfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415cd90 con 0x7f0c54159b20 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5bfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415d070 con 0x7f0c540ce8a0 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5959a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 0x7f0c54151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5959a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 0x7f0c54151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53306/0 (socket says 172.21.15.29:53306) 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.173+0000 7f0c5959a700 1 -- 172.21.15.29:0/3398410118 learned_addr learned my addr 172.21.15.29:0/3398410118 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5a59c700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c54159b20 0x7f0c5415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5959a700 1 -- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c540ce8a0 msgr2=0x7f0c5416a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:29.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c59d9b700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c540ce8a0 0x7f0c5416a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5959a700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c540ce8a0 0x7f0c5416a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5959a700 1 -- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c54159b20 msgr2=0x7f0c5415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5959a700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c54159b20 0x7f0c5415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5959a700 1 -- 172.21.15.29:0/3398410118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c009a20 con 0x7f0c54159750 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5a59c700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c54159b20 0x7f0c5415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c59d9b700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c540ce8a0 0x7f0c5416a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c5959a700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 0x7f0c54151dc0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0c540cc620 tx=0x7f0c44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44011b10 con 0x7f0c54159750 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c44000d10 con 0x7f0c54159750 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.174+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4401a660 con 0x7f0c54159750 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.175+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c44009420 con 0x7f0c54159750 2023-10-07T11:03:29.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.175+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5416a3f0 con 0x7f0c54159750 2023-10-07T11:03:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.176+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5404c3c0 con 0x7f0c54159750 2023-10-07T11:03:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.177+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0c4401aa90 con 0x7f0c54159750 2023-10-07T11:03:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.177+0000 7f0c49ffb700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0c38066f60 0x7f0c38069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:29.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.177+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f0c440863a0 con 0x7f0c54159750 2023-10-07T11:03:29.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.178+0000 7f0c59d9b700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0c38066f60 0x7f0c38069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:29.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.178+0000 7f0c59d9b700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0c38066f60 0x7f0c38069410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000c00 tx=0x7f0c50009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:29.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.182+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c44053430 con 0x7f0c54159750 2023-10-07T11:03:29.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.382+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c54070540 con 0x7f0c38066f60 2023-10-07T11:03:29.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.391+0000 7f0c49ffb700 1 -- 172.21.15.29:0/3398410118 <== mgr.8429 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289608 (secure 0 0 0) 0x7f0c54070540 con 0x7f0c38066f60 2023-10-07T11:03:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0c38066f60 msgr2=0x7f0c38069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0c38066f60 0x7f0c38069410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000c00 tx=0x7f0c50009040 comp rx=0 tx=0).stop 2023-10-07T11:03:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 msgr2=0x7f0c54151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 0x7f0c54151dc0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0c540cc620 tx=0x7f0c44017000 comp rx=0 tx=0).stop 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 shutdown_connections 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c54159b20 0x7f0c5415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0c38066f60 0x7f0c38069410 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0c540ce8a0 0x7f0c5416a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 --2- 172.21.15.29:0/3398410118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c54159750 0x7f0c54151dc0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 >> 172.21.15.29:0/3398410118 conn(0x7f0c54063a10 msgr2=0x7f0c540d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 shutdown_connections 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:29.396+0000 7f0c5bfff700 1 -- 172.21.15.29:0/3398410118 wait complete. 2023-10-07T11:03:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:03:29.414 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-07T11:03:28.652150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2476584,"kb_used_data":20456,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752498136,"statfs":{"total":773094113280,"available":770558091264,"internally_reserved":0,"allocated":20946944,"data_stored":10949700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.620164"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074517+0000","last_change":"2023-10-07T11:03:23.072878+0000","last_active":"2023-10-07T11:03:23.074517+0000","last_peered":"2023-10-07T11:03:23.074517+0000","last_clean":"2023-10-07T11:03:23.074517+0000","last_became_active":"2023-10-07T11:03:23.072616+0000","last_became_peered":"2023-10-07T11:03:23.072616+0000","last_unstale":"2023-10-07T11:03:23.074517+0000","last_undegraded":"2023-10-07T11:03:23.074517+0000","last_fullsized":"2023-10-07T11:03:23.074517+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:22:31.017660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067427+0000","last_change":"2023-10-07T11:03:23.060354+0000","last_active":"2023-10-07T11:03:23.067427+0000","last_peered":"2023-10-07T11:03:23.067427+0000","last_clean":"2023-10-07T11:03:23.067427+0000","last_became_active":"2023-10-07T11:03:23.060021+0000","last_became_peered":"2023-10-07T11:03:23.060021+0000","last_unstale":"2023-10-07T11:03:23.067427+0000","last_undegraded":"2023-10-07T11:03:23.067427+0000","last_fullsized":"2023-10-07T11:03:23.067427+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:10:26.617203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.061273+0000","last_change":"2023-10-07T11:03:23.059262+0000","last_active":"2023-10-07T11:03:23.061273+0000","last_peered":"2023-10-07T11:03:23.061273+0000","last_clean":"2023-10-07T11:03:23.061273+0000","last_became_active":"2023-10-07T11:03:23.058996+0000","last_became_peered":"2023-10-07T11:03:23.058996+0000","last_unstale":"2023-10-07T11:03:23.061273+0000","last_undegraded":"2023-10-07T11:03:23.061273+0000","last_fullsized":"2023-10-07T11:03:23.061273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067361+0000","last_change":"2023-10-07T11:03:23.058379+0000","last_active":"2023-10-07T11:03:23.067361+0000","last_peered":"2023-10-07T11:03:23.067361+0000","last_clean":"2023-10-07T11:03:23.067361+0000","last_became_active":"2023-10-07T11:03:23.058075+0000","last_became_peered":"2023-10-07T11:03:23.058075+0000","last_unstale":"2023-10-07T11:03:23.067361+0000","last_undegraded":"2023-10-07T11:03:23.067361+0000","last_fullsized":"2023-10-07T11:03:23.067361+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:15:40.534917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071784+0000","last_change":"2023-10-07T11:03:23.065672+0000","last_active":"2023-10-07T11:03:23.071784+0000","last_peered":"2023-10-07T11:03:23.071784+0000","last_clean":"2023-10-07T11:03:23.071784+0000","last_became_active":"2023-10-07T11:03:23.065424+0000","last_became_peered":"2023-10-07T11:03:23.065424+0000","last_unstale":"2023-10-07T11:03:23.071784+0000","last_undegraded":"2023-10-07T11:03:23.071784+0000","last_fullsized":"2023-10-07T11:03:23.071784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:06:56.578927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069205+0000","last_change":"2023-10-07T11:03:23.067743+0000","last_active":"2023-10-07T11:03:23.069205+0000","last_peered":"2023-10-07T11:03:23.069205+0000","last_clean":"2023-10-07T11:03:23.069205+0000","last_became_active":"2023-10-07T11:03:23.067464+0000","last_became_peered":"2023-10-07T11:03:23.067464+0000","last_unstale":"2023-10-07T11:03:23.069205+0000","last_undegraded":"2023-10-07T11:03:23.069205+0000","last_fullsized":"2023-10-07T11:03:23.069205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074433+0000","last_change":"2023-10-07T11:03:23.071020+0000","last_active":"2023-10-07T11:03:23.074433+0000","last_peered":"2023-10-07T11:03:23.074433+0000","last_clean":"2023-10-07T11:03:23.074433+0000","last_became_active":"2023-10-07T11:03:23.070489+0000","last_became_peered":"2023-10-07T11:03:23.070489+0000","last_unstale":"2023-10-07T11:03:23.074433+0000","last_undegraded":"2023-10-07T11:03:23.074433+0000","last_fullsized":"2023-10-07T11:03:23.074433+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:10:50.471018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.077562+0000","last_change":"2023-10-07T11:03:23.076006+0000","last_active":"2023-10-07T11:03:23.077562+0000","last_peered":"2023-10-07T11:03:23.077562+0000","last_clean":"2023-10-07T11:03:23.077562+0000","last_became_active":"2023-10-07T11:03:23.075758+0000","last_became_peered":"2023-10-07T11:03:23.075758+0000","last_unstale":"2023-10-07T11:03:23.077562+0000","last_undegraded":"2023-10-07T11:03:23.077562+0000","last_fullsized":"2023-10-07T11:03:23.077562+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072015+0000","last_change":"2023-10-07T11:03:23.064954+0000","last_active":"2023-10-07T11:03:23.072015+0000","last_peered":"2023-10-07T11:03:23.072015+0000","last_clean":"2023-10-07T11:03:23.072015+0000","last_became_active":"2023-10-07T11:03:23.064478+0000","last_became_peered":"2023-10-07T11:03:23.064478+0000","last_unstale":"2023-10-07T11:03:23.072015+0000","last_undegraded":"2023-10-07T11:03:23.072015+0000","last_fullsized":"2023-10-07T11:03:23.072015+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:18.490656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.066075+0000","last_change":"2023-10-07T11:03:23.058595+0000","last_active":"2023-10-07T11:03:23.066075+0000","last_peered":"2023-10-07T11:03:23.066075+0000","last_clean":"2023-10-07T11:03:23.066075+0000","last_became_active":"2023-10-07T11:03:23.058332+0000","last_became_peered":"2023-10-07T11:03:23.058332+0000","last_unstale":"2023-10-07T11:03:23.066075+0000","last_undegraded":"2023-10-07T11:03:23.066075+0000","last_fullsized":"2023-10-07T11:03:23.066075+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072281+0000","last_change":"2023-10-07T11:03:23.063741+0000","last_active":"2023-10-07T11:03:23.072281+0000","last_peered":"2023-10-07T11:03:23.072281+0000","last_clean":"2023-10-07T11:03:23.072281+0000","last_became_active":"2023-10-07T11:03:23.063404+0000","last_became_peered":"2023-10-07T11:03:23.063404+0000","last_unstale":"2023-10-07T11:03:23.072281+0000","last_undegraded":"2023-10-07T11:03:23.072281+0000","last_fullsized":"2023-10-07T11:03:23.072281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073816+0000","last_change":"2023-10-07T11:03:23.069104+0000","last_active":"2023-10-07T11:03:23.073816+0000","last_peered":"2023-10-07T11:03:23.073816+0000","last_clean":"2023-10-07T11:03:23.073816+0000","last_became_active":"2023-10-07T11:03:23.068650+0000","last_became_peered":"2023-10-07T11:03:23.068650+0000","last_unstale":"2023-10-07T11:03:23.073816+0000","last_undegraded":"2023-10-07T11:03:23.073816+0000","last_fullsized":"2023-10-07T11:03:23.073816+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073577+0000","last_change":"2023-10-07T11:03:23.066831+0000","last_active":"2023-10-07T11:03:23.073577+0000","last_peered":"2023-10-07T11:03:23.073577+0000","last_clean":"2023-10-07T11:03:23.073577+0000","last_became_active":"2023-10-07T11:03:23.066541+0000","last_became_peered":"2023-10-07T11:03:23.066541+0000","last_unstale":"2023-10-07T11:03:23.073577+0000","last_undegraded":"2023-10-07T11:03:23.073577+0000","last_fullsized":"2023-10-07T11:03:23.073577+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:50:42.578295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070855+0000","last_change":"2023-10-07T11:03:23.064829+0000","last_active":"2023-10-07T11:03:23.070855+0000","last_peered":"2023-10-07T11:03:23.070855+0000","last_clean":"2023-10-07T11:03:23.070855+0000","last_became_active":"2023-10-07T11:03:23.064458+0000","last_became_peered":"2023-10-07T11:03:23.064458+0000","last_unstale":"2023-10-07T11:03:23.070855+0000","last_undegraded":"2023-10-07T11:03:23.070855+0000","last_fullsized":"2023-10-07T11:03:23.070855+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:38:42.392543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070133+0000","last_change":"2023-10-07T11:03:23.062935+0000","last_active":"2023-10-07T11:03:23.070133+0000","last_peered":"2023-10-07T11:03:23.070133+0000","last_clean":"2023-10-07T11:03:23.070133+0000","last_became_active":"2023-10-07T11:03:23.062669+0000","last_became_peered":"2023-10-07T11:03:23.062669+0000","last_unstale":"2023-10-07T11:03:23.070133+0000","last_undegraded":"2023-10-07T11:03:23.070133+0000","last_fullsized":"2023-10-07T11:03:23.070133+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.062893+0000","last_change":"2023-10-07T11:03:23.057796+0000","last_active":"2023-10-07T11:03:23.062893+0000","last_peered":"2023-10-07T11:03:23.062893+0000","last_clean":"2023-10-07T11:03:23.062893+0000","last_became_active":"2023-10-07T11:03:23.057545+0000","last_became_peered":"2023-10-07T11:03:23.057545+0000","last_unstale":"2023-10-07T11:03:23.062893+0000","last_undegraded":"2023-10-07T11:03:23.062893+0000","last_fullsized":"2023-10-07T11:03:23.062893+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071883+0000","last_change":"2023-10-07T11:03:23.066540+0000","last_active":"2023-10-07T11:03:23.071883+0000","last_peered":"2023-10-07T11:03:23.071883+0000","last_clean":"2023-10-07T11:03:23.071883+0000","last_became_active":"2023-10-07T11:03:23.066302+0000","last_became_peered":"2023-10-07T11:03:23.066302+0000","last_unstale":"2023-10-07T11:03:23.071883+0000","last_undegraded":"2023-10-07T11:03:23.071883+0000","last_fullsized":"2023-10-07T11:03:23.071883+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:15:41.530360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074287+0000","last_change":"2023-10-07T11:03:23.069858+0000","last_active":"2023-10-07T11:03:23.074287+0000","last_peered":"2023-10-07T11:03:23.074287+0000","last_clean":"2023-10-07T11:03:23.074287+0000","last_became_active":"2023-10-07T11:03:23.069575+0000","last_became_peered":"2023-10-07T11:03:23.069575+0000","last_unstale":"2023-10-07T11:03:23.074287+0000","last_undegraded":"2023-10-07T11:03:23.074287+0000","last_fullsized":"2023-10-07T11:03:23.074287+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:39.727310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072967+0000","last_change":"2023-10-07T11:03:23.056891+0000","last_active":"2023-10-07T11:03:23.072967+0000","last_peered":"2023-10-07T11:03:23.072967+0000","last_clean":"2023-10-07T11:03:23.072967+0000","last_became_active":"2023-10-07T11:03:23.056623+0000","last_became_peered":"2023-10-07T11:03:23.056623+0000","last_unstale":"2023-10-07T11:03:23.072967+0000","last_undegraded":"2023-10-07T11:03:23.072967+0000","last_fullsized":"2023-10-07T11:03:23.072967+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:24.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071963+0000","last_change":"2023-10-07T11:03:23.066517+0000","last_active":"2023-10-07T11:03:23.071963+0000","last_peered":"2023-10-07T11:03:23.071963+0000","last_clean":"2023-10-07T11:03:23.071963+0000","last_became_active":"2023-10-07T11:03:23.066265+0000","last_became_peered":"2023-10-07T11:03:23.066265+0000","last_unstale":"2023-10-07T11:03:23.071963+0000","last_undegraded":"2023-10-07T11:03:23.071963+0000","last_fullsized":"2023-10-07T11:03:23.071963+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:02:50.840881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070821+0000","last_change":"2023-10-07T11:03:23.064525+0000","last_active":"2023-10-07T11:03:23.070821+0000","last_peered":"2023-10-07T11:03:23.070821+0000","last_clean":"2023-10-07T11:03:23.070821+0000","last_became_active":"2023-10-07T11:03:23.064169+0000","last_became_peered":"2023-10-07T11:03:23.064169+0000","last_unstale":"2023-10-07T11:03:23.070821+0000","last_undegraded":"2023-10-07T11:03:23.070821+0000","last_fullsized":"2023-10-07T11:03:23.070821+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:45.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065743+0000","last_change":"2023-10-07T11:03:23.056041+0000","last_active":"2023-10-07T11:03:23.065743+0000","last_peered":"2023-10-07T11:03:23.065743+0000","last_clean":"2023-10-07T11:03:23.065743+0000","last_became_active":"2023-10-07T11:03:23.055408+0000","last_became_peered":"2023-10-07T11:03:23.055408+0000","last_unstale":"2023-10-07T11:03:23.065743+0000","last_undegraded":"2023-10-07T11:03:23.065743+0000","last_fullsized":"2023-10-07T11:03:23.065743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072460+0000","last_change":"2023-10-07T11:03:23.070205+0000","last_active":"2023-10-07T11:03:23.072460+0000","last_peered":"2023-10-07T11:03:23.072460+0000","last_clean":"2023-10-07T11:03:23.072460+0000","last_became_active":"2023-10-07T11:03:23.069898+0000","last_became_peered":"2023-10-07T11:03:23.069898+0000","last_unstale":"2023-10-07T11:03:23.072460+0000","last_undegraded":"2023-10-07T11:03:23.072460+0000","last_fullsized":"2023-10-07T11:03:23.072460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074629+0000","last_change":"2023-10-07T11:03:23.059218+0000","last_active":"2023-10-07T11:03:23.074629+0000","last_peered":"2023-10-07T11:03:23.074629+0000","last_clean":"2023-10-07T11:03:23.074629+0000","last_became_active":"2023-10-07T11:03:23.058961+0000","last_became_peered":"2023-10-07T11:03:23.058961+0000","last_unstale":"2023-10-07T11:03:23.074629+0000","last_undegraded":"2023-10-07T11:03:23.074629+0000","last_fullsized":"2023-10-07T11:03:23.074629+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:09.706205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074211+0000","last_change":"2023-10-07T11:03:23.072528+0000","last_active":"2023-10-07T11:03:23.074211+0000","last_peered":"2023-10-07T11:03:23.074211+0000","last_clean":"2023-10-07T11:03:23.074211+0000","last_became_active":"2023-10-07T11:03:23.072226+0000","last_became_peered":"2023-10-07T11:03:23.072226+0000","last_unstale":"2023-10-07T11:03:23.074211+0000","last_undegraded":"2023-10-07T11:03:23.074211+0000","last_fullsized":"2023-10-07T11:03:23.074211+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074386+0000","last_change":"2023-10-07T11:03:23.070541+0000","last_active":"2023-10-07T11:03:23.074386+0000","last_peered":"2023-10-07T11:03:23.074386+0000","last_clean":"2023-10-07T11:03:23.074386+0000","last_became_active":"2023-10-07T11:03:23.070206+0000","last_became_peered":"2023-10-07T11:03:23.070206+0000","last_unstale":"2023-10-07T11:03:23.074386+0000","last_undegraded":"2023-10-07T11:03:23.074386+0000","last_fullsized":"2023-10-07T11:03:23.074386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069344+0000","last_change":"2023-10-07T11:03:23.068010+0000","last_active":"2023-10-07T11:03:23.069344+0000","last_peered":"2023-10-07T11:03:23.069344+0000","last_clean":"2023-10-07T11:03:23.069344+0000","last_became_active":"2023-10-07T11:03:23.067653+0000","last_became_peered":"2023-10-07T11:03:23.067653+0000","last_unstale":"2023-10-07T11:03:23.069344+0000","last_undegraded":"2023-10-07T11:03:23.069344+0000","last_fullsized":"2023-10-07T11:03:23.069344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071945+0000","last_change":"2023-10-07T11:03:23.065993+0000","last_active":"2023-10-07T11:03:23.071945+0000","last_peered":"2023-10-07T11:03:23.071945+0000","last_clean":"2023-10-07T11:03:23.071945+0000","last_became_active":"2023-10-07T11:03:23.065717+0000","last_became_peered":"2023-10-07T11:03:23.065717+0000","last_unstale":"2023-10-07T11:03:23.071945+0000","last_undegraded":"2023-10-07T11:03:23.071945+0000","last_fullsized":"2023-10-07T11:03:23.071945+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:10.179832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073747+0000","last_change":"2023-10-07T11:03:23.067616+0000","last_active":"2023-10-07T11:03:23.073747+0000","last_peered":"2023-10-07T11:03:23.073747+0000","last_clean":"2023-10-07T11:03:23.073747+0000","last_became_active":"2023-10-07T11:03:23.067374+0000","last_became_peered":"2023-10-07T11:03:23.067374+0000","last_unstale":"2023-10-07T11:03:23.073747+0000","last_undegraded":"2023-10-07T11:03:23.073747+0000","last_fullsized":"2023-10-07T11:03:23.073747+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.068796+0000","last_change":"2023-10-07T11:03:23.062083+0000","last_active":"2023-10-07T11:03:23.068796+0000","last_peered":"2023-10-07T11:03:23.068796+0000","last_clean":"2023-10-07T11:03:23.068796+0000","last_became_active":"2023-10-07T11:03:23.061771+0000","last_became_peered":"2023-10-07T11:03:23.061771+0000","last_unstale":"2023-10-07T11:03:23.068796+0000","last_undegraded":"2023-10-07T11:03:23.068796+0000","last_fullsized":"2023-10-07T11:03:23.068796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:15:41.530360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065837+0000","last_change":"2023-10-07T11:03:23.064318+0000","last_active":"2023-10-07T11:03:23.065837+0000","last_peered":"2023-10-07T11:03:23.065837+0000","last_clean":"2023-10-07T11:03:23.065837+0000","last_became_active":"2023-10-07T11:03:23.064039+0000","last_became_peered":"2023-10-07T11:03:23.064039+0000","last_unstale":"2023-10-07T11:03:23.065837+0000","last_undegraded":"2023-10-07T11:03:23.065837+0000","last_fullsized":"2023-10-07T11:03:23.065837+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071834+0000","last_change":"2023-10-07T11:03:23.069771+0000","last_active":"2023-10-07T11:03:23.071834+0000","last_peered":"2023-10-07T11:03:23.071834+0000","last_clean":"2023-10-07T11:03:23.071834+0000","last_became_active":"2023-10-07T11:03:23.069417+0000","last_became_peered":"2023-10-07T11:03:23.069417+0000","last_unstale":"2023-10-07T11:03:23.071834+0000","last_undegraded":"2023-10-07T11:03:23.071834+0000","last_fullsized":"2023-10-07T11:03:23.071834+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:54:11.114276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074670+0000","last_change":"2023-10-07T11:03:23.061128+0000","last_active":"2023-10-07T11:03:23.074670+0000","last_peered":"2023-10-07T11:03:23.074670+0000","last_clean":"2023-10-07T11:03:23.074670+0000","last_became_active":"2023-10-07T11:03:23.060886+0000","last_became_peered":"2023-10-07T11:03:23.060886+0000","last_unstale":"2023-10-07T11:03:23.074670+0000","last_undegraded":"2023-10-07T11:03:23.074670+0000","last_fullsized":"2023-10-07T11:03:23.074670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"196'4","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095534+0000","last_change":"2023-10-07T11:03:22.058499+0000","last_active":"2023-10-07T11:03:23.095534+0000","last_peered":"2023-10-07T11:03:23.095534+0000","last_clean":"2023-10-07T11:03:23.095534+0000","last_became_active":"2023-10-07T11:03:22.058041+0000","last_became_peered":"2023-10-07T11:03:22.058041+0000","last_unstale":"2023-10-07T11:03:23.095534+0000","last_undegraded":"2023-10-07T11:03:23.095534+0000","last_fullsized":"2023-10-07T11:03:23.095534+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:06:46.355801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074352+0000","last_change":"2023-10-07T11:03:23.071098+0000","last_active":"2023-10-07T11:03:23.074352+0000","last_peered":"2023-10-07T11:03:23.074352+0000","last_clean":"2023-10-07T11:03:23.074352+0000","last_became_active":"2023-10-07T11:03:23.070446+0000","last_became_peered":"2023-10-07T11:03:23.070446+0000","last_unstale":"2023-10-07T11:03:23.074352+0000","last_undegraded":"2023-10-07T11:03:23.074352+0000","last_fullsized":"2023-10-07T11:03:23.074352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.079835+0000","last_change":"2023-10-07T11:03:22.051142+0000","last_active":"2023-10-07T11:03:23.079835+0000","last_peered":"2023-10-07T11:03:23.079835+0000","last_clean":"2023-10-07T11:03:23.079835+0000","last_became_active":"2023-10-07T11:03:22.050890+0000","last_became_peered":"2023-10-07T11:03:22.050890+0000","last_unstale":"2023-10-07T11:03:23.079835+0000","last_undegraded":"2023-10-07T11:03:23.079835+0000","last_fullsized":"2023-10-07T11:03:23.079835+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:41:07.129981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074362+0000","last_change":"2023-10-07T11:03:23.067691+0000","last_active":"2023-10-07T11:03:23.074362+0000","last_peered":"2023-10-07T11:03:23.074362+0000","last_clean":"2023-10-07T11:03:23.074362+0000","last_became_active":"2023-10-07T11:03:23.067433+0000","last_became_peered":"2023-10-07T11:03:23.067433+0000","last_unstale":"2023-10-07T11:03:23.074362+0000","last_undegraded":"2023-10-07T11:03:23.074362+0000","last_fullsized":"2023-10-07T11:03:23.074362+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.078939+0000","last_change":"2023-10-07T11:03:22.046593+0000","last_active":"2023-10-07T11:03:23.078939+0000","last_peered":"2023-10-07T11:03:23.078939+0000","last_clean":"2023-10-07T11:03:23.078939+0000","last_became_active":"2023-10-07T11:03:22.046326+0000","last_became_peered":"2023-10-07T11:03:22.046326+0000","last_unstale":"2023-10-07T11:03:23.078939+0000","last_undegraded":"2023-10-07T11:03:23.078939+0000","last_fullsized":"2023-10-07T11:03:23.078939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070887+0000","last_change":"2023-10-07T11:03:23.066045+0000","last_active":"2023-10-07T11:03:23.070887+0000","last_peered":"2023-10-07T11:03:23.070887+0000","last_clean":"2023-10-07T11:03:23.070887+0000","last_became_active":"2023-10-07T11:03:23.065798+0000","last_became_peered":"2023-10-07T11:03:23.065798+0000","last_unstale":"2023-10-07T11:03:23.070887+0000","last_undegraded":"2023-10-07T11:03:23.070887+0000","last_fullsized":"2023-10-07T11:03:23.070887+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:40:08.604922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.080643+0000","last_change":"2023-10-07T11:03:22.050762+0000","last_active":"2023-10-07T11:03:23.080643+0000","last_peered":"2023-10-07T11:03:23.080643+0000","last_clean":"2023-10-07T11:03:23.080643+0000","last_became_active":"2023-10-07T11:03:22.050492+0000","last_became_peered":"2023-10-07T11:03:22.050492+0000","last_unstale":"2023-10-07T11:03:23.080643+0000","last_undegraded":"2023-10-07T11:03:23.080643+0000","last_fullsized":"2023-10-07T11:03:23.080643+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:45.665846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065766+0000","last_change":"2023-10-07T11:03:23.056224+0000","last_active":"2023-10-07T11:03:23.065766+0000","last_peered":"2023-10-07T11:03:23.065766+0000","last_clean":"2023-10-07T11:03:23.065766+0000","last_became_active":"2023-10-07T11:03:23.055757+0000","last_became_peered":"2023-10-07T11:03:23.055757+0000","last_unstale":"2023-10-07T11:03:23.065766+0000","last_undegraded":"2023-10-07T11:03:23.065766+0000","last_fullsized":"2023-10-07T11:03:23.065766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095434+0000","last_change":"2023-10-07T11:03:22.057588+0000","last_active":"2023-10-07T11:03:23.095434+0000","last_peered":"2023-10-07T11:03:23.095434+0000","last_clean":"2023-10-07T11:03:23.095434+0000","last_became_active":"2023-10-07T11:03:22.057300+0000","last_became_peered":"2023-10-07T11:03:22.057300+0000","last_unstale":"2023-10-07T11:03:23.095434+0000","last_undegraded":"2023-10-07T11:03:23.095434+0000","last_fullsized":"2023-10-07T11:03:23.095434+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:00:36.997902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060572+0000","last_change":"2023-10-07T11:03:23.056936+0000","last_active":"2023-10-07T11:03:23.060572+0000","last_peered":"2023-10-07T11:03:23.060572+0000","last_clean":"2023-10-07T11:03:23.060572+0000","last_became_active":"2023-10-07T11:03:23.056670+0000","last_became_peered":"2023-10-07T11:03:23.056670+0000","last_unstale":"2023-10-07T11:03:23.060572+0000","last_undegraded":"2023-10-07T11:03:23.060572+0000","last_fullsized":"2023-10-07T11:03:23.060572+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082675+0000","last_change":"2023-10-07T11:03:22.049469+0000","last_active":"2023-10-07T11:03:23.082675+0000","last_peered":"2023-10-07T11:03:23.082675+0000","last_clean":"2023-10-07T11:03:23.082675+0000","last_became_active":"2023-10-07T11:03:22.049231+0000","last_became_peered":"2023-10-07T11:03:22.049231+0000","last_unstale":"2023-10-07T11:03:23.082675+0000","last_undegraded":"2023-10-07T11:03:23.082675+0000","last_fullsized":"2023-10-07T11:03:23.082675+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:51.063075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067016+0000","last_change":"2023-10-07T11:03:23.063384+0000","last_active":"2023-10-07T11:03:23.067016+0000","last_peered":"2023-10-07T11:03:23.067016+0000","last_clean":"2023-10-07T11:03:23.067016+0000","last_became_active":"2023-10-07T11:03:23.063032+0000","last_became_peered":"2023-10-07T11:03:23.063032+0000","last_unstale":"2023-10-07T11:03:23.067016+0000","last_undegraded":"2023-10-07T11:03:23.067016+0000","last_fullsized":"2023-10-07T11:03:23.067016+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.059900+0000","last_change":"2023-10-07T11:03:22.052787+0000","last_active":"2023-10-07T11:03:23.059900+0000","last_peered":"2023-10-07T11:03:23.059900+0000","last_clean":"2023-10-07T11:03:23.059900+0000","last_became_active":"2023-10-07T11:03:22.052454+0000","last_became_peered":"2023-10-07T11:03:22.052454+0000","last_unstale":"2023-10-07T11:03:23.059900+0000","last_undegraded":"2023-10-07T11:03:23.059900+0000","last_fullsized":"2023-10-07T11:03:23.059900+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:32:16.922700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060058+0000","last_change":"2023-10-07T11:03:23.057682+0000","last_active":"2023-10-07T11:03:23.060058+0000","last_peered":"2023-10-07T11:03:23.060058+0000","last_clean":"2023-10-07T11:03:23.060058+0000","last_became_active":"2023-10-07T11:03:23.056838+0000","last_became_peered":"2023-10-07T11:03:23.056838+0000","last_unstale":"2023-10-07T11:03:23.060058+0000","last_undegraded":"2023-10-07T11:03:23.060058+0000","last_fullsized":"2023-10-07T11:03:23.060058+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.055372+0000","last_change":"2023-10-07T11:03:22.043828+0000","last_active":"2023-10-07T11:03:23.055372+0000","last_peered":"2023-10-07T11:03:23.055372+0000","last_clean":"2023-10-07T11:03:23.055372+0000","last_became_active":"2023-10-07T11:03:22.043538+0000","last_became_peered":"2023-10-07T11:03:22.043538+0000","last_unstale":"2023-10-07T11:03:23.055372+0000","last_undegraded":"2023-10-07T11:03:23.055372+0000","last_fullsized":"2023-10-07T11:03:23.055372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074728+0000","last_change":"2023-10-07T11:03:23.066795+0000","last_active":"2023-10-07T11:03:23.074728+0000","last_peered":"2023-10-07T11:03:23.074728+0000","last_clean":"2023-10-07T11:03:23.074728+0000","last_became_active":"2023-10-07T11:03:23.066509+0000","last_became_peered":"2023-10-07T11:03:23.066509+0000","last_unstale":"2023-10-07T11:03:23.074728+0000","last_undegraded":"2023-10-07T11:03:23.074728+0000","last_fullsized":"2023-10-07T11:03:23.074728+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:40:40.866050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.052659+0000","last_change":"2023-10-07T11:03:22.058923+0000","last_active":"2023-10-07T11:03:23.052659+0000","last_peered":"2023-10-07T11:03:23.052659+0000","last_clean":"2023-10-07T11:03:23.052659+0000","last_became_active":"2023-10-07T11:03:22.058689+0000","last_became_peered":"2023-10-07T11:03:22.058689+0000","last_unstale":"2023-10-07T11:03:23.052659+0000","last_undegraded":"2023-10-07T11:03:23.052659+0000","last_fullsized":"2023-10-07T11:03:23.052659+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:23:25.739404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073238+0000","last_change":"2023-10-07T11:03:23.064829+0000","last_active":"2023-10-07T11:03:23.073238+0000","last_peered":"2023-10-07T11:03:23.073238+0000","last_clean":"2023-10-07T11:03:23.073238+0000","last_became_active":"2023-10-07T11:03:23.064570+0000","last_became_peered":"2023-10-07T11:03:23.064570+0000","last_unstale":"2023-10-07T11:03:23.073238+0000","last_undegraded":"2023-10-07T11:03:23.073238+0000","last_fullsized":"2023-10-07T11:03:23.073238+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"196'3","reported_seq":13,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095493+0000","last_change":"2023-10-07T11:03:22.057778+0000","last_active":"2023-10-07T11:03:23.095493+0000","last_peered":"2023-10-07T11:03:23.095493+0000","last_clean":"2023-10-07T11:03:23.095493+0000","last_became_active":"2023-10-07T11:03:22.057359+0000","last_became_peered":"2023-10-07T11:03:22.057359+0000","last_unstale":"2023-10-07T11:03:23.095493+0000","last_undegraded":"2023-10-07T11:03:23.095493+0000","last_fullsized":"2023-10-07T11:03:23.095493+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:42:49.802708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072955+0000","last_change":"2023-10-07T11:03:23.056349+0000","last_active":"2023-10-07T11:03:23.072955+0000","last_peered":"2023-10-07T11:03:23.072955+0000","last_clean":"2023-10-07T11:03:23.072955+0000","last_became_active":"2023-10-07T11:03:23.055709+0000","last_became_peered":"2023-10-07T11:03:23.055709+0000","last_unstale":"2023-10-07T11:03:23.072955+0000","last_undegraded":"2023-10-07T11:03:23.072955+0000","last_fullsized":"2023-10-07T11:03:23.072955+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:34:35.644574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"196'6","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.097246+0000","last_change":"2023-10-07T11:03:22.043324+0000","last_active":"2023-10-07T11:03:23.097246+0000","last_peered":"2023-10-07T11:03:23.097246+0000","last_clean":"2023-10-07T11:03:23.097246+0000","last_became_active":"2023-10-07T11:03:22.043036+0000","last_became_peered":"2023-10-07T11:03:22.043036+0000","last_unstale":"2023-10-07T11:03:23.097246+0000","last_undegraded":"2023-10-07T11:03:23.097246+0000","last_fullsized":"2023-10-07T11:03:23.097246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072666+0000","last_change":"2023-10-07T11:03:23.070261+0000","last_active":"2023-10-07T11:03:23.072666+0000","last_peered":"2023-10-07T11:03:23.072666+0000","last_clean":"2023-10-07T11:03:23.072666+0000","last_became_active":"2023-10-07T11:03:23.069876+0000","last_became_peered":"2023-10-07T11:03:23.069876+0000","last_unstale":"2023-10-07T11:03:23.072666+0000","last_undegraded":"2023-10-07T11:03:23.072666+0000","last_fullsized":"2023-10-07T11:03:23.072666+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.080351+0000","last_change":"2023-10-07T11:03:22.053525+0000","last_active":"2023-10-07T11:03:23.080351+0000","last_peered":"2023-10-07T11:03:23.080351+0000","last_clean":"2023-10-07T11:03:23.080351+0000","last_became_active":"2023-10-07T11:03:22.053261+0000","last_became_peered":"2023-10-07T11:03:22.053261+0000","last_unstale":"2023-10-07T11:03:23.080351+0000","last_undegraded":"2023-10-07T11:03:23.080351+0000","last_fullsized":"2023-10-07T11:03:23.080351+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:57:44.508450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.079297+0000","last_change":"2023-10-07T11:03:23.075226+0000","last_active":"2023-10-07T11:03:23.079297+0000","last_peered":"2023-10-07T11:03:23.079297+0000","last_clean":"2023-10-07T11:03:23.079297+0000","last_became_active":"2023-10-07T11:03:23.075002+0000","last_became_peered":"2023-10-07T11:03:23.075002+0000","last_unstale":"2023-10-07T11:03:23.079297+0000","last_undegraded":"2023-10-07T11:03:23.079297+0000","last_fullsized":"2023-10-07T11:03:23.079297+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046972+0000","last_change":"2023-10-07T11:03:22.058631+0000","last_active":"2023-10-07T11:03:23.046972+0000","last_peered":"2023-10-07T11:03:23.046972+0000","last_clean":"2023-10-07T11:03:23.046972+0000","last_became_active":"2023-10-07T11:03:22.058368+0000","last_became_peered":"2023-10-07T11:03:22.058368+0000","last_unstale":"2023-10-07T11:03:23.046972+0000","last_undegraded":"2023-10-07T11:03:23.046972+0000","last_fullsized":"2023-10-07T11:03:23.046972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:41.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.064342+0000","last_change":"2023-10-07T11:03:23.059336+0000","last_active":"2023-10-07T11:03:23.064342+0000","last_peered":"2023-10-07T11:03:23.064342+0000","last_clean":"2023-10-07T11:03:23.064342+0000","last_became_active":"2023-10-07T11:03:23.058941+0000","last_became_peered":"2023-10-07T11:03:23.058941+0000","last_unstale":"2023-10-07T11:03:23.064342+0000","last_undegraded":"2023-10-07T11:03:23.064342+0000","last_fullsized":"2023-10-07T11:03:23.064342+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:25:42.771748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050617+0000","last_change":"2023-10-07T11:03:22.046240+0000","last_active":"2023-10-07T11:03:23.050617+0000","last_peered":"2023-10-07T11:03:23.050617+0000","last_clean":"2023-10-07T11:03:23.050617+0000","last_became_active":"2023-10-07T11:03:22.045964+0000","last_became_peered":"2023-10-07T11:03:22.045964+0000","last_unstale":"2023-10-07T11:03:23.050617+0000","last_undegraded":"2023-10-07T11:03:23.050617+0000","last_fullsized":"2023-10-07T11:03:23.050617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:25.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072532+0000","last_change":"2023-10-07T11:03:23.069327+0000","last_active":"2023-10-07T11:03:23.072532+0000","last_peered":"2023-10-07T11:03:23.072532+0000","last_clean":"2023-10-07T11:03:23.072532+0000","last_became_active":"2023-10-07T11:03:23.068992+0000","last_became_peered":"2023-10-07T11:03:23.068992+0000","last_unstale":"2023-10-07T11:03:23.072532+0000","last_undegraded":"2023-10-07T11:03:23.072532+0000","last_fullsized":"2023-10-07T11:03:23.072532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"196'8","reported_seq":18,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082714+0000","last_change":"2023-10-07T11:03:22.053477+0000","last_active":"2023-10-07T11:03:23.082714+0000","last_peered":"2023-10-07T11:03:23.082714+0000","last_clean":"2023-10-07T11:03:23.082714+0000","last_became_active":"2023-10-07T11:03:22.053179+0000","last_became_peered":"2023-10-07T11:03:22.053179+0000","last_unstale":"2023-10-07T11:03:23.082714+0000","last_undegraded":"2023-10-07T11:03:23.082714+0000","last_fullsized":"2023-10-07T11:03:23.082714+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:16:49.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074503+0000","last_change":"2023-10-07T11:03:23.069902+0000","last_active":"2023-10-07T11:03:23.074503+0000","last_peered":"2023-10-07T11:03:23.074503+0000","last_clean":"2023-10-07T11:03:23.074503+0000","last_became_active":"2023-10-07T11:03:23.069641+0000","last_became_peered":"2023-10-07T11:03:23.069641+0000","last_unstale":"2023-10-07T11:03:23.074503+0000","last_undegraded":"2023-10-07T11:03:23.074503+0000","last_fullsized":"2023-10-07T11:03:23.074503+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095407+0000","last_change":"2023-10-07T11:03:22.058339+0000","last_active":"2023-10-07T11:03:23.095407+0000","last_peered":"2023-10-07T11:03:23.095407+0000","last_clean":"2023-10-07T11:03:23.095407+0000","last_became_active":"2023-10-07T11:03:22.057722+0000","last_became_peered":"2023-10-07T11:03:22.057722+0000","last_unstale":"2023-10-07T11:03:23.095407+0000","last_undegraded":"2023-10-07T11:03:23.095407+0000","last_fullsized":"2023-10-07T11:03:23.095407+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:22:25.667544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.064670+0000","last_change":"2023-10-07T11:03:23.059140+0000","last_active":"2023-10-07T11:03:23.064670+0000","last_peered":"2023-10-07T11:03:23.064670+0000","last_clean":"2023-10-07T11:03:23.064670+0000","last_became_active":"2023-10-07T11:03:23.058890+0000","last_became_peered":"2023-10-07T11:03:23.058890+0000","last_unstale":"2023-10-07T11:03:23.064670+0000","last_undegraded":"2023-10-07T11:03:23.064670+0000","last_fullsized":"2023-10-07T11:03:23.064670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:54:27.529211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083381+0000","last_change":"2023-10-07T11:03:22.049580+0000","last_active":"2023-10-07T11:03:23.083381+0000","last_peered":"2023-10-07T11:03:23.083381+0000","last_clean":"2023-10-07T11:03:23.083381+0000","last_became_active":"2023-10-07T11:03:22.049343+0000","last_became_peered":"2023-10-07T11:03:22.049343+0000","last_unstale":"2023-10-07T11:03:23.083381+0000","last_undegraded":"2023-10-07T11:03:23.083381+0000","last_fullsized":"2023-10-07T11:03:23.083381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:39.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072207+0000","last_change":"2023-10-07T11:03:23.068719+0000","last_active":"2023-10-07T11:03:23.072207+0000","last_peered":"2023-10-07T11:03:23.072207+0000","last_clean":"2023-10-07T11:03:23.072207+0000","last_became_active":"2023-10-07T11:03:23.068451+0000","last_became_peered":"2023-10-07T11:03:23.068451+0000","last_unstale":"2023-10-07T11:03:23.072207+0000","last_undegraded":"2023-10-07T11:03:23.072207+0000","last_fullsized":"2023-10-07T11:03:23.072207+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:43.392569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083037+0000","last_change":"2023-10-07T11:03:22.054996+0000","last_active":"2023-10-07T11:03:23.083037+0000","last_peered":"2023-10-07T11:03:23.083037+0000","last_clean":"2023-10-07T11:03:23.083037+0000","last_became_active":"2023-10-07T11:03:22.054736+0000","last_became_peered":"2023-10-07T11:03:22.054736+0000","last_unstale":"2023-10-07T11:03:23.083037+0000","last_undegraded":"2023-10-07T11:03:23.083037+0000","last_fullsized":"2023-10-07T11:03:23.083037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:47.480334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067405+0000","last_change":"2023-10-07T11:03:23.060415+0000","last_active":"2023-10-07T11:03:23.067405+0000","last_peered":"2023-10-07T11:03:23.067405+0000","last_clean":"2023-10-07T11:03:23.067405+0000","last_became_active":"2023-10-07T11:03:23.060136+0000","last_became_peered":"2023-10-07T11:03:23.060136+0000","last_unstale":"2023-10-07T11:03:23.067405+0000","last_undegraded":"2023-10-07T11:03:23.067405+0000","last_fullsized":"2023-10-07T11:03:23.067405+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.055604+0000","last_change":"2023-10-07T11:03:22.040612+0000","last_active":"2023-10-07T11:03:23.055604+0000","last_peered":"2023-10-07T11:03:23.055604+0000","last_clean":"2023-10-07T11:03:23.055604+0000","last_became_active":"2023-10-07T11:03:22.040349+0000","last_became_peered":"2023-10-07T11:03:22.040349+0000","last_unstale":"2023-10-07T11:03:23.055604+0000","last_undegraded":"2023-10-07T11:03:23.055604+0000","last_fullsized":"2023-10-07T11:03:23.055604+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072519+0000","last_change":"2023-10-07T11:03:23.069558+0000","last_active":"2023-10-07T11:03:23.072519+0000","last_peered":"2023-10-07T11:03:23.072519+0000","last_clean":"2023-10-07T11:03:23.072519+0000","last_became_active":"2023-10-07T11:03:23.069179+0000","last_became_peered":"2023-10-07T11:03:23.069179+0000","last_unstale":"2023-10-07T11:03:23.072519+0000","last_undegraded":"2023-10-07T11:03:23.072519+0000","last_fullsized":"2023-10-07T11:03:23.072519+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081863+0000","last_change":"2023-10-07T11:03:22.060469+0000","last_active":"2023-10-07T11:03:23.081863+0000","last_peered":"2023-10-07T11:03:23.081863+0000","last_clean":"2023-10-07T11:03:23.081863+0000","last_became_active":"2023-10-07T11:03:22.060024+0000","last_became_peered":"2023-10-07T11:03:22.060024+0000","last_unstale":"2023-10-07T11:03:23.081863+0000","last_undegraded":"2023-10-07T11:03:23.081863+0000","last_fullsized":"2023-10-07T11:03:23.081863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:42.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072314+0000","last_change":"2023-10-07T11:03:23.064217+0000","last_active":"2023-10-07T11:03:23.072314+0000","last_peered":"2023-10-07T11:03:23.072314+0000","last_clean":"2023-10-07T11:03:23.072314+0000","last_became_active":"2023-10-07T11:03:23.063800+0000","last_became_peered":"2023-10-07T11:03:23.063800+0000","last_unstale":"2023-10-07T11:03:23.072314+0000","last_undegraded":"2023-10-07T11:03:23.072314+0000","last_fullsized":"2023-10-07T11:03:23.072314+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.053415+0000","last_change":"2023-10-07T11:03:22.057330+0000","last_active":"2023-10-07T11:03:23.053415+0000","last_peered":"2023-10-07T11:03:23.053415+0000","last_clean":"2023-10-07T11:03:23.053415+0000","last_became_active":"2023-10-07T11:03:22.057038+0000","last_became_peered":"2023-10-07T11:03:22.057038+0000","last_unstale":"2023-10-07T11:03:23.053415+0000","last_undegraded":"2023-10-07T11:03:23.053415+0000","last_fullsized":"2023-10-07T11:03:23.053415+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:04.566950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073798+0000","last_change":"2023-10-07T11:03:23.068248+0000","last_active":"2023-10-07T11:03:23.073798+0000","last_peered":"2023-10-07T11:03:23.073798+0000","last_clean":"2023-10-07T11:03:23.073798+0000","last_became_active":"2023-10-07T11:03:23.067904+0000","last_became_peered":"2023-10-07T11:03:23.067904+0000","last_unstale":"2023-10-07T11:03:23.073798+0000","last_undegraded":"2023-10-07T11:03:23.073798+0000","last_fullsized":"2023-10-07T11:03:23.073798+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050442+0000","last_change":"2023-10-07T11:03:22.059410+0000","last_active":"2023-10-07T11:03:23.050442+0000","last_peered":"2023-10-07T11:03:23.050442+0000","last_clean":"2023-10-07T11:03:23.050442+0000","last_became_active":"2023-10-07T11:03:22.059029+0000","last_became_peered":"2023-10-07T11:03:22.059029+0000","last_unstale":"2023-10-07T11:03:23.050442+0000","last_undegraded":"2023-10-07T11:03:23.050442+0000","last_fullsized":"2023-10-07T11:03:23.050442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:57:18.642115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070843+0000","last_change":"2023-10-07T11:03:23.064553+0000","last_active":"2023-10-07T11:03:23.070843+0000","last_peered":"2023-10-07T11:03:23.070843+0000","last_clean":"2023-10-07T11:03:23.070843+0000","last_became_active":"2023-10-07T11:03:23.064308+0000","last_became_peered":"2023-10-07T11:03:23.064308+0000","last_unstale":"2023-10-07T11:03:23.070843+0000","last_undegraded":"2023-10-07T11:03:23.070843+0000","last_fullsized":"2023-10-07T11:03:23.070843+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:23:59.483751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.049306+0000","last_change":"2023-10-07T11:03:22.047245+0000","last_active":"2023-10-07T11:03:23.049306+0000","last_peered":"2023-10-07T11:03:23.049306+0000","last_clean":"2023-10-07T11:03:23.049306+0000","last_became_active":"2023-10-07T11:03:22.046970+0000","last_became_peered":"2023-10-07T11:03:22.046970+0000","last_unstale":"2023-10-07T11:03:23.049306+0000","last_undegraded":"2023-10-07T11:03:23.049306+0000","last_fullsized":"2023-10-07T11:03:23.049306+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:04.566950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071751+0000","last_change":"2023-10-07T11:03:23.065086+0000","last_active":"2023-10-07T11:03:23.071751+0000","last_peered":"2023-10-07T11:03:23.071751+0000","last_clean":"2023-10-07T11:03:23.071751+0000","last_became_active":"2023-10-07T11:03:23.064822+0000","last_became_peered":"2023-10-07T11:03:23.064822+0000","last_unstale":"2023-10-07T11:03:23.071751+0000","last_undegraded":"2023-10-07T11:03:23.071751+0000","last_fullsized":"2023-10-07T11:03:23.071751+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:18.490656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081692+0000","last_change":"2023-10-07T11:03:22.059458+0000","last_active":"2023-10-07T11:03:23.081692+0000","last_peered":"2023-10-07T11:03:23.081692+0000","last_clean":"2023-10-07T11:03:23.081692+0000","last_became_active":"2023-10-07T11:03:22.059162+0000","last_became_peered":"2023-10-07T11:03:22.059162+0000","last_unstale":"2023-10-07T11:03:23.081692+0000","last_undegraded":"2023-10-07T11:03:23.081692+0000","last_fullsized":"2023-10-07T11:03:23.081692+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:08:26.185841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070233+0000","last_change":"2023-10-07T11:03:23.062012+0000","last_active":"2023-10-07T11:03:23.070233+0000","last_peered":"2023-10-07T11:03:23.070233+0000","last_clean":"2023-10-07T11:03:23.070233+0000","last_became_active":"2023-10-07T11:03:23.061769+0000","last_became_peered":"2023-10-07T11:03:23.061769+0000","last_unstale":"2023-10-07T11:03:23.070233+0000","last_undegraded":"2023-10-07T11:03:23.070233+0000","last_fullsized":"2023-10-07T11:03:23.070233+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046870+0000","last_change":"2023-10-07T11:03:22.051956+0000","last_active":"2023-10-07T11:03:23.046870+0000","last_peered":"2023-10-07T11:03:23.046870+0000","last_clean":"2023-10-07T11:03:23.046870+0000","last_became_active":"2023-10-07T11:03:22.051691+0000","last_became_peered":"2023-10-07T11:03:22.051691+0000","last_unstale":"2023-10-07T11:03:23.046870+0000","last_undegraded":"2023-10-07T11:03:23.046870+0000","last_fullsized":"2023-10-07T11:03:23.046870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:42.177620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065367+0000","last_change":"2023-10-07T11:03:23.059948+0000","last_active":"2023-10-07T11:03:23.065367+0000","last_peered":"2023-10-07T11:03:23.065367+0000","last_clean":"2023-10-07T11:03:23.065367+0000","last_became_active":"2023-10-07T11:03:23.059564+0000","last_became_peered":"2023-10-07T11:03:23.059564+0000","last_unstale":"2023-10-07T11:03:23.065367+0000","last_undegraded":"2023-10-07T11:03:23.065367+0000","last_fullsized":"2023-10-07T11:03:23.065367+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:04:25.554055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046689+0000","last_change":"2023-10-07T11:03:22.052493+0000","last_active":"2023-10-07T11:03:23.046689+0000","last_peered":"2023-10-07T11:03:23.046689+0000","last_clean":"2023-10-07T11:03:23.046689+0000","last_became_active":"2023-10-07T11:03:22.052235+0000","last_became_peered":"2023-10-07T11:03:22.052235+0000","last_unstale":"2023-10-07T11:03:23.046689+0000","last_undegraded":"2023-10-07T11:03:23.046689+0000","last_fullsized":"2023-10-07T11:03:23.046689+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:06.410397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074593+0000","last_change":"2023-10-07T11:03:23.060626+0000","last_active":"2023-10-07T11:03:23.074593+0000","last_peered":"2023-10-07T11:03:23.074593+0000","last_clean":"2023-10-07T11:03:23.074593+0000","last_became_active":"2023-10-07T11:03:23.060260+0000","last_became_peered":"2023-10-07T11:03:23.060260+0000","last_unstale":"2023-10-07T11:03:23.074593+0000","last_undegraded":"2023-10-07T11:03:23.074593+0000","last_fullsized":"2023-10-07T11:03:23.074593+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051338+0000","last_change":"2023-10-07T11:03:22.048854+0000","last_active":"2023-10-07T11:03:23.051338+0000","last_peered":"2023-10-07T11:03:23.051338+0000","last_clean":"2023-10-07T11:03:23.051338+0000","last_became_active":"2023-10-07T11:03:22.048568+0000","last_became_peered":"2023-10-07T11:03:22.048568+0000","last_unstale":"2023-10-07T11:03:23.051338+0000","last_undegraded":"2023-10-07T11:03:23.051338+0000","last_fullsized":"2023-10-07T11:03:23.051338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:01:19.581526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073623+0000","last_change":"2023-10-07T11:03:23.066567+0000","last_active":"2023-10-07T11:03:23.073623+0000","last_peered":"2023-10-07T11:03:23.073623+0000","last_clean":"2023-10-07T11:03:23.073623+0000","last_became_active":"2023-10-07T11:03:23.066175+0000","last_became_peered":"2023-10-07T11:03:23.066175+0000","last_unstale":"2023-10-07T11:03:23.073623+0000","last_undegraded":"2023-10-07T11:03:23.073623+0000","last_fullsized":"2023-10-07T11:03:23.073623+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:38.363015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.048793+0000","last_change":"2023-10-07T11:03:22.043072+0000","last_active":"2023-10-07T11:03:23.048793+0000","last_peered":"2023-10-07T11:03:23.048793+0000","last_clean":"2023-10-07T11:03:23.048793+0000","last_became_active":"2023-10-07T11:03:22.042614+0000","last_became_peered":"2023-10-07T11:03:22.042614+0000","last_unstale":"2023-10-07T11:03:23.048793+0000","last_undegraded":"2023-10-07T11:03:23.048793+0000","last_fullsized":"2023-10-07T11:03:23.048793+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:20:03.006488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074442+0000","last_change":"2023-10-07T11:03:23.071149+0000","last_active":"2023-10-07T11:03:23.074442+0000","last_peered":"2023-10-07T11:03:23.074442+0000","last_clean":"2023-10-07T11:03:23.074442+0000","last_became_active":"2023-10-07T11:03:23.070906+0000","last_became_peered":"2023-10-07T11:03:23.070906+0000","last_unstale":"2023-10-07T11:03:23.074442+0000","last_undegraded":"2023-10-07T11:03:23.074442+0000","last_fullsized":"2023-10-07T11:03:23.074442+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:11:59.325550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.049537+0000","last_change":"2023-10-07T11:03:22.042941+0000","last_active":"2023-10-07T11:03:23.049537+0000","last_peered":"2023-10-07T11:03:23.049537+0000","last_clean":"2023-10-07T11:03:23.049537+0000","last_became_active":"2023-10-07T11:03:22.042595+0000","last_became_peered":"2023-10-07T11:03:22.042595+0000","last_unstale":"2023-10-07T11:03:23.049537+0000","last_undegraded":"2023-10-07T11:03:23.049537+0000","last_fullsized":"2023-10-07T11:03:23.049537+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:50:02.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069701+0000","last_change":"2023-10-07T11:03:23.063216+0000","last_active":"2023-10-07T11:03:23.069701+0000","last_peered":"2023-10-07T11:03:23.069701+0000","last_clean":"2023-10-07T11:03:23.069701+0000","last_became_active":"2023-10-07T11:03:23.061923+0000","last_became_peered":"2023-10-07T11:03:23.061923+0000","last_unstale":"2023-10-07T11:03:23.069701+0000","last_undegraded":"2023-10-07T11:03:23.069701+0000","last_fullsized":"2023-10-07T11:03:23.069701+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:21:58.043707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050906+0000","last_change":"2023-10-07T11:03:22.046244+0000","last_active":"2023-10-07T11:03:23.050906+0000","last_peered":"2023-10-07T11:03:23.050906+0000","last_clean":"2023-10-07T11:03:23.050906+0000","last_became_active":"2023-10-07T11:03:22.045897+0000","last_became_peered":"2023-10-07T11:03:22.045897+0000","last_unstale":"2023-10-07T11:03:23.050906+0000","last_undegraded":"2023-10-07T11:03:23.050906+0000","last_fullsized":"2023-10-07T11:03:23.050906+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:21.812275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072310+0000","last_change":"2023-10-07T11:03:23.070377+0000","last_active":"2023-10-07T11:03:23.072310+0000","last_peered":"2023-10-07T11:03:23.072310+0000","last_clean":"2023-10-07T11:03:23.072310+0000","last_became_active":"2023-10-07T11:03:23.070101+0000","last_became_peered":"2023-10-07T11:03:23.070101+0000","last_unstale":"2023-10-07T11:03:23.072310+0000","last_undegraded":"2023-10-07T11:03:23.072310+0000","last_fullsized":"2023-10-07T11:03:23.072310+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:22.545174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051314+0000","last_change":"2023-10-07T11:03:22.060700+0000","last_active":"2023-10-07T11:03:23.051314+0000","last_peered":"2023-10-07T11:03:23.051314+0000","last_clean":"2023-10-07T11:03:23.051314+0000","last_became_active":"2023-10-07T11:03:22.060037+0000","last_became_peered":"2023-10-07T11:03:22.060037+0000","last_unstale":"2023-10-07T11:03:23.051314+0000","last_undegraded":"2023-10-07T11:03:23.051314+0000","last_fullsized":"2023-10-07T11:03:23.051314+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:53.212268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073064+0000","last_change":"2023-10-07T11:03:23.058853+0000","last_active":"2023-10-07T11:03:23.073064+0000","last_peered":"2023-10-07T11:03:23.073064+0000","last_clean":"2023-10-07T11:03:23.073064+0000","last_became_active":"2023-10-07T11:03:23.058594+0000","last_became_peered":"2023-10-07T11:03:23.058594+0000","last_unstale":"2023-10-07T11:03:23.073064+0000","last_undegraded":"2023-10-07T11:03:23.073064+0000","last_fullsized":"2023-10-07T11:03:23.073064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.059672+0000","last_change":"2023-10-07T11:03:22.050935+0000","last_active":"2023-10-07T11:03:23.059672+0000","last_peered":"2023-10-07T11:03:23.059672+0000","last_clean":"2023-10-07T11:03:23.059672+0000","last_became_active":"2023-10-07T11:03:22.050667+0000","last_became_peered":"2023-10-07T11:03:22.050667+0000","last_unstale":"2023-10-07T11:03:23.059672+0000","last_undegraded":"2023-10-07T11:03:23.059672+0000","last_fullsized":"2023-10-07T11:03:23.059672+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:48.346681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"186'348","reported_seq":1131,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.059922+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:23.059922+0000","last_peered":"2023-10-07T11:03:23.059922+0000","last_clean":"2023-10-07T11:03:23.059922+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:23.059922+0000","last_undegraded":"2023-10-07T11:03:23.059922+0000","last_fullsized":"2023-10-07T11:03:23.059922+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312568,"kb_used_data":2992,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059272,"statfs":{"total":96636764160,"available":96316694528,"internally_reserved":0,"allocated":3063808,"data_stored":1811016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.82199999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.98599999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.76100000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510558,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316076,"kb_used_data":2404,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055764,"statfs":{"total":96636764160,"available":96313102336,"internally_reserved":0,"allocated":2461696,"data_stored":1220544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.75}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.79300000000000004}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.80700000000000005}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.71999999999999997}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.94299999999999995}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":1.034}]}]},{"osd":1,"up_from":14,"seq":60129543260,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307896,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063944,"statfs":{"total":96636764160,"available":96321478656,"internally_reserved":0,"allocated":2473984,"data_stored":1222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.442},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.76700000000000002,"5min":0.86299999999999999,"15min":0.94199999999999995},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.40000000000000002},"max":{"1min":0.82699999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.443,"5min":0.378,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82699999999999996,"15min":0.84099999999999997},"min":{"1min":0.55600000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.373},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.153,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.93700000000000006}]},{"osd":5,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84999999999999998},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80600000000000005,"15min":0.81299999999999994},"min":{"1min":0.57999999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.873,"15min":0.871},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.58399999999999996},"max":{"1min":1.052,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0980000000000001,"5min":1.157,"15min":1.302},"last":0.85799999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.877,"15min":0.88400000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.141,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88700000000000001,"15min":0.878},"min":{"1min":0.66700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.80600000000000005}]}]},{"osd":0,"up_from":14,"seq":60129543260,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1810934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.439},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.496}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.93600000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.008},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.86399999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1221210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.81699999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66300000000000003,"5min":0.85399999999999998,"15min":0.95099999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.71199999999999997,"5min":0.92600000000000005,"15min":0.98199999999999998},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65900000000000003,"5min":0.76700000000000002,"15min":0.80000000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.433}]},{"osd":4,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.77400000000000002,"15min":0.79400000000000004},"min":{"1min":0.55900000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.93600000000000005,"5min":1.163,"15min":1.1879999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.266,"15min":1.5640000000000001},"last":0.80400000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.014,"5min":1.2989999999999999,"15min":1.528},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78800000000000003,"15min":0.80800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97499999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.85799999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1140000000000001,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.876},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.147,"5min":1.242,"15min":1.3779999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Sat Oct 7 11:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.85699999999999998,"15min":0.874},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.052,"5min":1.2050000000000001,"15min":1.629},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.85699999999999998,"15min":0.877},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0900000000000001,"5min":1.274,"15min":1.5940000000000001},"last":0.77000000000000002}]}]},{"osd":3,"up_from":14,"seq":60129543262,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1220738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72199999999999998,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.45000000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.83999999999999997,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79700000000000004,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55500000000000005,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73099999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.5},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.436,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96799999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0780000000000001,"15min":1.222},"last":0.78100000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0720000000000001,"5min":1.27,"15min":1.27},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":0.84699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98399999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96699999999999997,"5min":1.389,"15min":1.389},"last":0.77300000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0269999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.008,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.68100000000000005}]}]},{"osd":4,"up_from":15,"seq":64424510558,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316092,"kb_used_data":2420,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055748,"statfs":{"total":96636764160,"available":96313085952,"internally_reserved":0,"allocated":2478080,"data_stored":1222634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.63800000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.79100000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.86299999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.92700000000000005}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":1.034}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.98399999999999999}]}]},{"osd":5,"up_from":14,"seq":60129543260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311980,"kb_used_data":2404,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059860,"statfs":{"total":96636764160,"available":96317296640,"internally_reserved":0,"allocated":2461696,"data_stored":1220544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.78000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93300000000000005,"5min":1.585,"15min":1.585},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.81299999999999994},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.90900000000000003,"5min":1.673,"15min":1.673},"last":0.86699999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73799999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":1.139},"last":0.78700000000000003}]},{"osd":2,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92500000000000004,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.97399999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.95399999999999996,"5min":1.0549999999999999,"15min":1.069},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.91400000000000003,"5min":0.96899999999999997,"15min":1.161},"last":0.84599999999999997}]},{"osd":4,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.76200000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64300000000000002,"15min":0.625},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.69299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0109999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.038,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.93899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:03:32.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:03:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.628+0000 7f1c67bff700 1 Processor -- start 2023-10-07T11:03:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.628+0000 7f1c67bff700 1 -- start start 2023-10-07T11:03:32.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.629+0000 7f1c67bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1c600b0710 0x7f1c6015c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:32.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.629+0000 7f1c67bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c60154090 con 0x7f1c60153cc0 2023-10-07T11:03:32.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.629+0000 7f1c67bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c60159f60 con 0x7f1c6015ade0 2023-10-07T11:03:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.629+0000 7f1c67bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6015a240 con 0x7f1c600b0710 2023-10-07T11:03:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c6599b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1c60153cc0 0x7f1c600b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57316/0 (socket says 172.21.15.29:57316) 2023-10-07T11:03:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c6599b700 1 -- 172.21.15.29:0/2895200828 learned_addr learned my addr 172.21.15.29:0/2895200828 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c6619c700 1 --2- 172.21.15.29:0/2895200828 >> v2:172.21.15.144:3300/0 conn(0x7f1c600b0710 0x7f1c6015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222501729 0 0) 0x7f1c60154090 con 0x7f1c60153cc0 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c003570 con 0x7f1c60153cc0 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343012435 0 0) 0x7f1c60159f60 con 0x7f1c6015ade0 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.630+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c60154090 con 0x7f1c6015ade0 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c6619c700 1 -- 172.21.15.29:0/2895200828 >> v1:172.21.15.144:6789/0 conn(0x7f1c6015ade0 legacy=0x7f1c600b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c6619c700 1 -- 172.21.15.29:0/2895200828 >> v1:172.21.15.29:6789/0 conn(0x7f1c60153cc0 legacy=0x7f1c600b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c6619c700 1 -- 172.21.15.29:0/2895200828 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6015a520 con 0x7f1c600b0710 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c6619c700 1 --2- 172.21.15.29:0/2895200828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c6015c240 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009e60 tx=0x7f1c58009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113f39b5d6cf23f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5800e070 con 0x7f1c600b0710 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5800bab0 con 0x7f1c600b0710 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.631+0000 7f1c64999700 1 -- 172.21.15.29:0/2895200828 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5800be70 con 0x7f1c600b0710 2023-10-07T11:03:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 -- 172.21.15.29:0/2895200828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 msgr2=0x7f1c6015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 --2- 172.21.15.29:0/2895200828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c6015c240 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009e60 tx=0x7f1c58009780 comp rx=0 tx=0).stop 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 -- 172.21.15.29:0/2895200828 shutdown_connections 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 --2- 172.21.15.29:0/2895200828 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c6015c240 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 -- 172.21.15.29:0/2895200828 >> 172.21.15.29:0/2895200828 conn(0x7f1c60067610 msgr2=0x7f1c60067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 -- 172.21.15.29:0/2895200828 shutdown_connections 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.632+0000 7f1c67bff700 1 -- 172.21.15.29:0/2895200828 wait complete. 2023-10-07T11:03:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.633+0000 7f1c67bff700 1 Processor -- start 2023-10-07T11:03:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.633+0000 7f1c67bff700 1 -- start start 2023-10-07T11:03:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.634+0000 7f1c67bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c600cec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.634+0000 7f1c67bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c60153cc0 0x7f1c60169ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.634+0000 7f1c67bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6015ade0 0x7f1c60152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.634+0000 7f1c67bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6016a410 con 0x7f1c6015ade0 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.634+0000 7f1c67bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6015c200 con 0x7f1c600b0710 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.634+0000 7f1c67bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6015cd80 con 0x7f1c60153cc0 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c600cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6619c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6015ade0 0x7f1c60152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c600cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:59450/0 (socket says 172.21.15.29:59450) 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 -- 172.21.15.29:0/4089267911 learned_addr learned my addr 172.21.15.29:0/4089267911 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6519a700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c60153cc0 0x7f1c60169ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 -- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c60153cc0 msgr2=0x7f1c60169ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c60153cc0 0x7f1c60169ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 -- 172.21.15.29:0/4089267911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6015ade0 msgr2=0x7f1c60152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6015ade0 0x7f1c60152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 -- 172.21.15.29:0/4089267911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c58009a20 con 0x7f1c600b0710 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6619c700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6015ade0 0x7f1c60152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.635+0000 7f1c6599b700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c600cec20 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c00c7f0 tx=0x7f1c5c007cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.636+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0072e0 con 0x7f1c600b0710 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.636+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c007460 con 0x7f1c600b0710 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.636+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0057f0 con 0x7f1c600b0710 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.636+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6015c450 con 0x7f1c600b0710 2023-10-07T11:03:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.636+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6015ca10 con 0x7f1c600b0710 2023-10-07T11:03:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.637+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6004c3c0 con 0x7f1c600b0710 2023-10-07T11:03:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.638+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1c5c005c40 con 0x7f1c600b0710 2023-10-07T11:03:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.639+0000 7f1c55ffb700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1c44066f30 0x7f1c440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.639+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f1c5c083df0 con 0x7f1c600b0710 2023-10-07T11:03:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.639+0000 7f1c6519a700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1c44066f30 0x7f1c440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.640+0000 7f1c6519a700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1c44066f30 0x7f1c440693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c600ce940 tx=0x7f1c50007d50 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.643+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c5c050e80 con 0x7f1c600b0710 2023-10-07T11:03:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.841+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c6015a800 con 0x7f1c44066f30 2023-10-07T11:03:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.850+0000 7f1c55ffb700 1 -- 172.21.15.29:0/4089267911 <== mgr.8429 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289518 (secure 0 0 0) 0x7f1c6015a800 con 0x7f1c44066f30 2023-10-07T11:03:32.855 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1c44066f30 msgr2=0x7f1c440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1c44066f30 0x7f1c440693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c600ce940 tx=0x7f1c50007d50 comp rx=0 tx=0).stop 2023-10-07T11:03:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 msgr2=0x7f1c600cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c600cec20 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c00c7f0 tx=0x7f1c5c007cc0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 shutdown_connections 2023-10-07T11:03:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6015ade0 0x7f1c60152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c600b0710 0x7f1c600cec20 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1c44066f30 0x7f1c440693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.854+0000 7f1c67bff700 1 --2- 172.21.15.29:0/4089267911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c60153cc0 0x7f1c60169ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.855+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 >> 172.21.15.29:0/4089267911 conn(0x7f1c60067610 msgr2=0x7f1c600aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.855+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 shutdown_connections 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:32.855+0000 7f1c67bff700 1 -- 172.21.15.29:0/4089267911 wait complete. 2023-10-07T11:03:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:03:32.872 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-07T11:03:32.653311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":624,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":387,"ondisk_log_size":387,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2476584,"kb_used_data":20456,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752498136,"statfs":{"total":773094113280,"available":770558091264,"internally_reserved":0,"allocated":20946944,"data_stored":10949700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.607244"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074517+0000","last_change":"2023-10-07T11:03:23.072878+0000","last_active":"2023-10-07T11:03:23.074517+0000","last_peered":"2023-10-07T11:03:23.074517+0000","last_clean":"2023-10-07T11:03:23.074517+0000","last_became_active":"2023-10-07T11:03:23.072616+0000","last_became_peered":"2023-10-07T11:03:23.072616+0000","last_unstale":"2023-10-07T11:03:23.074517+0000","last_undegraded":"2023-10-07T11:03:23.074517+0000","last_fullsized":"2023-10-07T11:03:23.074517+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:22:31.017660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067427+0000","last_change":"2023-10-07T11:03:23.060354+0000","last_active":"2023-10-07T11:03:23.067427+0000","last_peered":"2023-10-07T11:03:23.067427+0000","last_clean":"2023-10-07T11:03:23.067427+0000","last_became_active":"2023-10-07T11:03:23.060021+0000","last_became_peered":"2023-10-07T11:03:23.060021+0000","last_unstale":"2023-10-07T11:03:23.067427+0000","last_undegraded":"2023-10-07T11:03:23.067427+0000","last_fullsized":"2023-10-07T11:03:23.067427+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:10:26.617203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.061273+0000","last_change":"2023-10-07T11:03:23.059262+0000","last_active":"2023-10-07T11:03:23.061273+0000","last_peered":"2023-10-07T11:03:23.061273+0000","last_clean":"2023-10-07T11:03:23.061273+0000","last_became_active":"2023-10-07T11:03:23.058996+0000","last_became_peered":"2023-10-07T11:03:23.058996+0000","last_unstale":"2023-10-07T11:03:23.061273+0000","last_undegraded":"2023-10-07T11:03:23.061273+0000","last_fullsized":"2023-10-07T11:03:23.061273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067361+0000","last_change":"2023-10-07T11:03:23.058379+0000","last_active":"2023-10-07T11:03:23.067361+0000","last_peered":"2023-10-07T11:03:23.067361+0000","last_clean":"2023-10-07T11:03:23.067361+0000","last_became_active":"2023-10-07T11:03:23.058075+0000","last_became_peered":"2023-10-07T11:03:23.058075+0000","last_unstale":"2023-10-07T11:03:23.067361+0000","last_undegraded":"2023-10-07T11:03:23.067361+0000","last_fullsized":"2023-10-07T11:03:23.067361+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:15:40.534917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071784+0000","last_change":"2023-10-07T11:03:23.065672+0000","last_active":"2023-10-07T11:03:23.071784+0000","last_peered":"2023-10-07T11:03:23.071784+0000","last_clean":"2023-10-07T11:03:23.071784+0000","last_became_active":"2023-10-07T11:03:23.065424+0000","last_became_peered":"2023-10-07T11:03:23.065424+0000","last_unstale":"2023-10-07T11:03:23.071784+0000","last_undegraded":"2023-10-07T11:03:23.071784+0000","last_fullsized":"2023-10-07T11:03:23.071784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:06:56.578927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069205+0000","last_change":"2023-10-07T11:03:23.067743+0000","last_active":"2023-10-07T11:03:23.069205+0000","last_peered":"2023-10-07T11:03:23.069205+0000","last_clean":"2023-10-07T11:03:23.069205+0000","last_became_active":"2023-10-07T11:03:23.067464+0000","last_became_peered":"2023-10-07T11:03:23.067464+0000","last_unstale":"2023-10-07T11:03:23.069205+0000","last_undegraded":"2023-10-07T11:03:23.069205+0000","last_fullsized":"2023-10-07T11:03:23.069205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074433+0000","last_change":"2023-10-07T11:03:23.071020+0000","last_active":"2023-10-07T11:03:23.074433+0000","last_peered":"2023-10-07T11:03:23.074433+0000","last_clean":"2023-10-07T11:03:23.074433+0000","last_became_active":"2023-10-07T11:03:23.070489+0000","last_became_peered":"2023-10-07T11:03:23.070489+0000","last_unstale":"2023-10-07T11:03:23.074433+0000","last_undegraded":"2023-10-07T11:03:23.074433+0000","last_fullsized":"2023-10-07T11:03:23.074433+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:10:50.471018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.077562+0000","last_change":"2023-10-07T11:03:23.076006+0000","last_active":"2023-10-07T11:03:23.077562+0000","last_peered":"2023-10-07T11:03:23.077562+0000","last_clean":"2023-10-07T11:03:23.077562+0000","last_became_active":"2023-10-07T11:03:23.075758+0000","last_became_peered":"2023-10-07T11:03:23.075758+0000","last_unstale":"2023-10-07T11:03:23.077562+0000","last_undegraded":"2023-10-07T11:03:23.077562+0000","last_fullsized":"2023-10-07T11:03:23.077562+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072015+0000","last_change":"2023-10-07T11:03:23.064954+0000","last_active":"2023-10-07T11:03:23.072015+0000","last_peered":"2023-10-07T11:03:23.072015+0000","last_clean":"2023-10-07T11:03:23.072015+0000","last_became_active":"2023-10-07T11:03:23.064478+0000","last_became_peered":"2023-10-07T11:03:23.064478+0000","last_unstale":"2023-10-07T11:03:23.072015+0000","last_undegraded":"2023-10-07T11:03:23.072015+0000","last_fullsized":"2023-10-07T11:03:23.072015+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:18.490656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.066075+0000","last_change":"2023-10-07T11:03:23.058595+0000","last_active":"2023-10-07T11:03:23.066075+0000","last_peered":"2023-10-07T11:03:23.066075+0000","last_clean":"2023-10-07T11:03:23.066075+0000","last_became_active":"2023-10-07T11:03:23.058332+0000","last_became_peered":"2023-10-07T11:03:23.058332+0000","last_unstale":"2023-10-07T11:03:23.066075+0000","last_undegraded":"2023-10-07T11:03:23.066075+0000","last_fullsized":"2023-10-07T11:03:23.066075+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072281+0000","last_change":"2023-10-07T11:03:23.063741+0000","last_active":"2023-10-07T11:03:23.072281+0000","last_peered":"2023-10-07T11:03:23.072281+0000","last_clean":"2023-10-07T11:03:23.072281+0000","last_became_active":"2023-10-07T11:03:23.063404+0000","last_became_peered":"2023-10-07T11:03:23.063404+0000","last_unstale":"2023-10-07T11:03:23.072281+0000","last_undegraded":"2023-10-07T11:03:23.072281+0000","last_fullsized":"2023-10-07T11:03:23.072281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073816+0000","last_change":"2023-10-07T11:03:23.069104+0000","last_active":"2023-10-07T11:03:23.073816+0000","last_peered":"2023-10-07T11:03:23.073816+0000","last_clean":"2023-10-07T11:03:23.073816+0000","last_became_active":"2023-10-07T11:03:23.068650+0000","last_became_peered":"2023-10-07T11:03:23.068650+0000","last_unstale":"2023-10-07T11:03:23.073816+0000","last_undegraded":"2023-10-07T11:03:23.073816+0000","last_fullsized":"2023-10-07T11:03:23.073816+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073577+0000","last_change":"2023-10-07T11:03:23.066831+0000","last_active":"2023-10-07T11:03:23.073577+0000","last_peered":"2023-10-07T11:03:23.073577+0000","last_clean":"2023-10-07T11:03:23.073577+0000","last_became_active":"2023-10-07T11:03:23.066541+0000","last_became_peered":"2023-10-07T11:03:23.066541+0000","last_unstale":"2023-10-07T11:03:23.073577+0000","last_undegraded":"2023-10-07T11:03:23.073577+0000","last_fullsized":"2023-10-07T11:03:23.073577+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:50:42.578295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070855+0000","last_change":"2023-10-07T11:03:23.064829+0000","last_active":"2023-10-07T11:03:23.070855+0000","last_peered":"2023-10-07T11:03:23.070855+0000","last_clean":"2023-10-07T11:03:23.070855+0000","last_became_active":"2023-10-07T11:03:23.064458+0000","last_became_peered":"2023-10-07T11:03:23.064458+0000","last_unstale":"2023-10-07T11:03:23.070855+0000","last_undegraded":"2023-10-07T11:03:23.070855+0000","last_fullsized":"2023-10-07T11:03:23.070855+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:38:42.392543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070133+0000","last_change":"2023-10-07T11:03:23.062935+0000","last_active":"2023-10-07T11:03:23.070133+0000","last_peered":"2023-10-07T11:03:23.070133+0000","last_clean":"2023-10-07T11:03:23.070133+0000","last_became_active":"2023-10-07T11:03:23.062669+0000","last_became_peered":"2023-10-07T11:03:23.062669+0000","last_unstale":"2023-10-07T11:03:23.070133+0000","last_undegraded":"2023-10-07T11:03:23.070133+0000","last_fullsized":"2023-10-07T11:03:23.070133+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:44:40.198820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.062893+0000","last_change":"2023-10-07T11:03:23.057796+0000","last_active":"2023-10-07T11:03:23.062893+0000","last_peered":"2023-10-07T11:03:23.062893+0000","last_clean":"2023-10-07T11:03:23.062893+0000","last_became_active":"2023-10-07T11:03:23.057545+0000","last_became_peered":"2023-10-07T11:03:23.057545+0000","last_unstale":"2023-10-07T11:03:23.062893+0000","last_undegraded":"2023-10-07T11:03:23.062893+0000","last_fullsized":"2023-10-07T11:03:23.062893+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071883+0000","last_change":"2023-10-07T11:03:23.066540+0000","last_active":"2023-10-07T11:03:23.071883+0000","last_peered":"2023-10-07T11:03:23.071883+0000","last_clean":"2023-10-07T11:03:23.071883+0000","last_became_active":"2023-10-07T11:03:23.066302+0000","last_became_peered":"2023-10-07T11:03:23.066302+0000","last_unstale":"2023-10-07T11:03:23.071883+0000","last_undegraded":"2023-10-07T11:03:23.071883+0000","last_fullsized":"2023-10-07T11:03:23.071883+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:15:41.530360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074287+0000","last_change":"2023-10-07T11:03:23.069858+0000","last_active":"2023-10-07T11:03:23.074287+0000","last_peered":"2023-10-07T11:03:23.074287+0000","last_clean":"2023-10-07T11:03:23.074287+0000","last_became_active":"2023-10-07T11:03:23.069575+0000","last_became_peered":"2023-10-07T11:03:23.069575+0000","last_unstale":"2023-10-07T11:03:23.074287+0000","last_undegraded":"2023-10-07T11:03:23.074287+0000","last_fullsized":"2023-10-07T11:03:23.074287+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:39.727310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072967+0000","last_change":"2023-10-07T11:03:23.056891+0000","last_active":"2023-10-07T11:03:23.072967+0000","last_peered":"2023-10-07T11:03:23.072967+0000","last_clean":"2023-10-07T11:03:23.072967+0000","last_became_active":"2023-10-07T11:03:23.056623+0000","last_became_peered":"2023-10-07T11:03:23.056623+0000","last_unstale":"2023-10-07T11:03:23.072967+0000","last_undegraded":"2023-10-07T11:03:23.072967+0000","last_fullsized":"2023-10-07T11:03:23.072967+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:53:24.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071963+0000","last_change":"2023-10-07T11:03:23.066517+0000","last_active":"2023-10-07T11:03:23.071963+0000","last_peered":"2023-10-07T11:03:23.071963+0000","last_clean":"2023-10-07T11:03:23.071963+0000","last_became_active":"2023-10-07T11:03:23.066265+0000","last_became_peered":"2023-10-07T11:03:23.066265+0000","last_unstale":"2023-10-07T11:03:23.071963+0000","last_undegraded":"2023-10-07T11:03:23.071963+0000","last_fullsized":"2023-10-07T11:03:23.071963+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:02:50.840881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070821+0000","last_change":"2023-10-07T11:03:23.064525+0000","last_active":"2023-10-07T11:03:23.070821+0000","last_peered":"2023-10-07T11:03:23.070821+0000","last_clean":"2023-10-07T11:03:23.070821+0000","last_became_active":"2023-10-07T11:03:23.064169+0000","last_became_peered":"2023-10-07T11:03:23.064169+0000","last_unstale":"2023-10-07T11:03:23.070821+0000","last_undegraded":"2023-10-07T11:03:23.070821+0000","last_fullsized":"2023-10-07T11:03:23.070821+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:40:45.982862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065743+0000","last_change":"2023-10-07T11:03:23.056041+0000","last_active":"2023-10-07T11:03:23.065743+0000","last_peered":"2023-10-07T11:03:23.065743+0000","last_clean":"2023-10-07T11:03:23.065743+0000","last_became_active":"2023-10-07T11:03:23.055408+0000","last_became_peered":"2023-10-07T11:03:23.055408+0000","last_unstale":"2023-10-07T11:03:23.065743+0000","last_undegraded":"2023-10-07T11:03:23.065743+0000","last_fullsized":"2023-10-07T11:03:23.065743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072460+0000","last_change":"2023-10-07T11:03:23.070205+0000","last_active":"2023-10-07T11:03:23.072460+0000","last_peered":"2023-10-07T11:03:23.072460+0000","last_clean":"2023-10-07T11:03:23.072460+0000","last_became_active":"2023-10-07T11:03:23.069898+0000","last_became_peered":"2023-10-07T11:03:23.069898+0000","last_unstale":"2023-10-07T11:03:23.072460+0000","last_undegraded":"2023-10-07T11:03:23.072460+0000","last_fullsized":"2023-10-07T11:03:23.072460+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074629+0000","last_change":"2023-10-07T11:03:23.059218+0000","last_active":"2023-10-07T11:03:23.074629+0000","last_peered":"2023-10-07T11:03:23.074629+0000","last_clean":"2023-10-07T11:03:23.074629+0000","last_became_active":"2023-10-07T11:03:23.058961+0000","last_became_peered":"2023-10-07T11:03:23.058961+0000","last_unstale":"2023-10-07T11:03:23.074629+0000","last_undegraded":"2023-10-07T11:03:23.074629+0000","last_fullsized":"2023-10-07T11:03:23.074629+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:13:09.706205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074211+0000","last_change":"2023-10-07T11:03:23.072528+0000","last_active":"2023-10-07T11:03:23.074211+0000","last_peered":"2023-10-07T11:03:23.074211+0000","last_clean":"2023-10-07T11:03:23.074211+0000","last_became_active":"2023-10-07T11:03:23.072226+0000","last_became_peered":"2023-10-07T11:03:23.072226+0000","last_unstale":"2023-10-07T11:03:23.074211+0000","last_undegraded":"2023-10-07T11:03:23.074211+0000","last_fullsized":"2023-10-07T11:03:23.074211+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074386+0000","last_change":"2023-10-07T11:03:23.070541+0000","last_active":"2023-10-07T11:03:23.074386+0000","last_peered":"2023-10-07T11:03:23.074386+0000","last_clean":"2023-10-07T11:03:23.074386+0000","last_became_active":"2023-10-07T11:03:23.070206+0000","last_became_peered":"2023-10-07T11:03:23.070206+0000","last_unstale":"2023-10-07T11:03:23.074386+0000","last_undegraded":"2023-10-07T11:03:23.074386+0000","last_fullsized":"2023-10-07T11:03:23.074386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069344+0000","last_change":"2023-10-07T11:03:23.068010+0000","last_active":"2023-10-07T11:03:23.069344+0000","last_peered":"2023-10-07T11:03:23.069344+0000","last_clean":"2023-10-07T11:03:23.069344+0000","last_became_active":"2023-10-07T11:03:23.067653+0000","last_became_peered":"2023-10-07T11:03:23.067653+0000","last_unstale":"2023-10-07T11:03:23.069344+0000","last_undegraded":"2023-10-07T11:03:23.069344+0000","last_fullsized":"2023-10-07T11:03:23.069344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071945+0000","last_change":"2023-10-07T11:03:23.065993+0000","last_active":"2023-10-07T11:03:23.071945+0000","last_peered":"2023-10-07T11:03:23.071945+0000","last_clean":"2023-10-07T11:03:23.071945+0000","last_became_active":"2023-10-07T11:03:23.065717+0000","last_became_peered":"2023-10-07T11:03:23.065717+0000","last_unstale":"2023-10-07T11:03:23.071945+0000","last_undegraded":"2023-10-07T11:03:23.071945+0000","last_fullsized":"2023-10-07T11:03:23.071945+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:53:10.179832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073747+0000","last_change":"2023-10-07T11:03:23.067616+0000","last_active":"2023-10-07T11:03:23.073747+0000","last_peered":"2023-10-07T11:03:23.073747+0000","last_clean":"2023-10-07T11:03:23.073747+0000","last_became_active":"2023-10-07T11:03:23.067374+0000","last_became_peered":"2023-10-07T11:03:23.067374+0000","last_unstale":"2023-10-07T11:03:23.073747+0000","last_undegraded":"2023-10-07T11:03:23.073747+0000","last_fullsized":"2023-10-07T11:03:23.073747+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.068796+0000","last_change":"2023-10-07T11:03:23.062083+0000","last_active":"2023-10-07T11:03:23.068796+0000","last_peered":"2023-10-07T11:03:23.068796+0000","last_clean":"2023-10-07T11:03:23.068796+0000","last_became_active":"2023-10-07T11:03:23.061771+0000","last_became_peered":"2023-10-07T11:03:23.061771+0000","last_unstale":"2023-10-07T11:03:23.068796+0000","last_undegraded":"2023-10-07T11:03:23.068796+0000","last_fullsized":"2023-10-07T11:03:23.068796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:15:41.530360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065837+0000","last_change":"2023-10-07T11:03:23.064318+0000","last_active":"2023-10-07T11:03:23.065837+0000","last_peered":"2023-10-07T11:03:23.065837+0000","last_clean":"2023-10-07T11:03:23.065837+0000","last_became_active":"2023-10-07T11:03:23.064039+0000","last_became_peered":"2023-10-07T11:03:23.064039+0000","last_unstale":"2023-10-07T11:03:23.065837+0000","last_undegraded":"2023-10-07T11:03:23.065837+0000","last_fullsized":"2023-10-07T11:03:23.065837+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071834+0000","last_change":"2023-10-07T11:03:23.069771+0000","last_active":"2023-10-07T11:03:23.071834+0000","last_peered":"2023-10-07T11:03:23.071834+0000","last_clean":"2023-10-07T11:03:23.071834+0000","last_became_active":"2023-10-07T11:03:23.069417+0000","last_became_peered":"2023-10-07T11:03:23.069417+0000","last_unstale":"2023-10-07T11:03:23.071834+0000","last_undegraded":"2023-10-07T11:03:23.071834+0000","last_fullsized":"2023-10-07T11:03:23.071834+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:54:11.114276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074670+0000","last_change":"2023-10-07T11:03:23.061128+0000","last_active":"2023-10-07T11:03:23.074670+0000","last_peered":"2023-10-07T11:03:23.074670+0000","last_clean":"2023-10-07T11:03:23.074670+0000","last_became_active":"2023-10-07T11:03:23.060886+0000","last_became_peered":"2023-10-07T11:03:23.060886+0000","last_unstale":"2023-10-07T11:03:23.074670+0000","last_undegraded":"2023-10-07T11:03:23.074670+0000","last_fullsized":"2023-10-07T11:03:23.074670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:01:18.506961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"196'4","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095534+0000","last_change":"2023-10-07T11:03:22.058499+0000","last_active":"2023-10-07T11:03:23.095534+0000","last_peered":"2023-10-07T11:03:23.095534+0000","last_clean":"2023-10-07T11:03:23.095534+0000","last_became_active":"2023-10-07T11:03:22.058041+0000","last_became_peered":"2023-10-07T11:03:22.058041+0000","last_unstale":"2023-10-07T11:03:23.095534+0000","last_undegraded":"2023-10-07T11:03:23.095534+0000","last_fullsized":"2023-10-07T11:03:23.095534+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:06:46.355801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074352+0000","last_change":"2023-10-07T11:03:23.071098+0000","last_active":"2023-10-07T11:03:23.074352+0000","last_peered":"2023-10-07T11:03:23.074352+0000","last_clean":"2023-10-07T11:03:23.074352+0000","last_became_active":"2023-10-07T11:03:23.070446+0000","last_became_peered":"2023-10-07T11:03:23.070446+0000","last_unstale":"2023-10-07T11:03:23.074352+0000","last_undegraded":"2023-10-07T11:03:23.074352+0000","last_fullsized":"2023-10-07T11:03:23.074352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:26:16.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.079835+0000","last_change":"2023-10-07T11:03:22.051142+0000","last_active":"2023-10-07T11:03:23.079835+0000","last_peered":"2023-10-07T11:03:23.079835+0000","last_clean":"2023-10-07T11:03:23.079835+0000","last_became_active":"2023-10-07T11:03:22.050890+0000","last_became_peered":"2023-10-07T11:03:22.050890+0000","last_unstale":"2023-10-07T11:03:23.079835+0000","last_undegraded":"2023-10-07T11:03:23.079835+0000","last_fullsized":"2023-10-07T11:03:23.079835+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:41:07.129981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074362+0000","last_change":"2023-10-07T11:03:23.067691+0000","last_active":"2023-10-07T11:03:23.074362+0000","last_peered":"2023-10-07T11:03:23.074362+0000","last_clean":"2023-10-07T11:03:23.074362+0000","last_became_active":"2023-10-07T11:03:23.067433+0000","last_became_peered":"2023-10-07T11:03:23.067433+0000","last_unstale":"2023-10-07T11:03:23.074362+0000","last_undegraded":"2023-10-07T11:03:23.074362+0000","last_fullsized":"2023-10-07T11:03:23.074362+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:59:03.879462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.078939+0000","last_change":"2023-10-07T11:03:22.046593+0000","last_active":"2023-10-07T11:03:23.078939+0000","last_peered":"2023-10-07T11:03:23.078939+0000","last_clean":"2023-10-07T11:03:23.078939+0000","last_became_active":"2023-10-07T11:03:22.046326+0000","last_became_peered":"2023-10-07T11:03:22.046326+0000","last_unstale":"2023-10-07T11:03:23.078939+0000","last_undegraded":"2023-10-07T11:03:23.078939+0000","last_fullsized":"2023-10-07T11:03:23.078939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070887+0000","last_change":"2023-10-07T11:03:23.066045+0000","last_active":"2023-10-07T11:03:23.070887+0000","last_peered":"2023-10-07T11:03:23.070887+0000","last_clean":"2023-10-07T11:03:23.070887+0000","last_became_active":"2023-10-07T11:03:23.065798+0000","last_became_peered":"2023-10-07T11:03:23.065798+0000","last_unstale":"2023-10-07T11:03:23.070887+0000","last_undegraded":"2023-10-07T11:03:23.070887+0000","last_fullsized":"2023-10-07T11:03:23.070887+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:40:08.604922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.080643+0000","last_change":"2023-10-07T11:03:22.050762+0000","last_active":"2023-10-07T11:03:23.080643+0000","last_peered":"2023-10-07T11:03:23.080643+0000","last_clean":"2023-10-07T11:03:23.080643+0000","last_became_active":"2023-10-07T11:03:22.050492+0000","last_became_peered":"2023-10-07T11:03:22.050492+0000","last_unstale":"2023-10-07T11:03:23.080643+0000","last_undegraded":"2023-10-07T11:03:23.080643+0000","last_fullsized":"2023-10-07T11:03:23.080643+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:45.665846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065766+0000","last_change":"2023-10-07T11:03:23.056224+0000","last_active":"2023-10-07T11:03:23.065766+0000","last_peered":"2023-10-07T11:03:23.065766+0000","last_clean":"2023-10-07T11:03:23.065766+0000","last_became_active":"2023-10-07T11:03:23.055757+0000","last_became_peered":"2023-10-07T11:03:23.055757+0000","last_unstale":"2023-10-07T11:03:23.065766+0000","last_undegraded":"2023-10-07T11:03:23.065766+0000","last_fullsized":"2023-10-07T11:03:23.065766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:23:58.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095434+0000","last_change":"2023-10-07T11:03:22.057588+0000","last_active":"2023-10-07T11:03:23.095434+0000","last_peered":"2023-10-07T11:03:23.095434+0000","last_clean":"2023-10-07T11:03:23.095434+0000","last_became_active":"2023-10-07T11:03:22.057300+0000","last_became_peered":"2023-10-07T11:03:22.057300+0000","last_unstale":"2023-10-07T11:03:23.095434+0000","last_undegraded":"2023-10-07T11:03:23.095434+0000","last_fullsized":"2023-10-07T11:03:23.095434+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:00:36.997902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060572+0000","last_change":"2023-10-07T11:03:23.056936+0000","last_active":"2023-10-07T11:03:23.060572+0000","last_peered":"2023-10-07T11:03:23.060572+0000","last_clean":"2023-10-07T11:03:23.060572+0000","last_became_active":"2023-10-07T11:03:23.056670+0000","last_became_peered":"2023-10-07T11:03:23.056670+0000","last_unstale":"2023-10-07T11:03:23.060572+0000","last_undegraded":"2023-10-07T11:03:23.060572+0000","last_fullsized":"2023-10-07T11:03:23.060572+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082675+0000","last_change":"2023-10-07T11:03:22.049469+0000","last_active":"2023-10-07T11:03:23.082675+0000","last_peered":"2023-10-07T11:03:23.082675+0000","last_clean":"2023-10-07T11:03:23.082675+0000","last_became_active":"2023-10-07T11:03:22.049231+0000","last_became_peered":"2023-10-07T11:03:22.049231+0000","last_unstale":"2023-10-07T11:03:23.082675+0000","last_undegraded":"2023-10-07T11:03:23.082675+0000","last_fullsized":"2023-10-07T11:03:23.082675+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:39:51.063075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067016+0000","last_change":"2023-10-07T11:03:23.063384+0000","last_active":"2023-10-07T11:03:23.067016+0000","last_peered":"2023-10-07T11:03:23.067016+0000","last_clean":"2023-10-07T11:03:23.067016+0000","last_became_active":"2023-10-07T11:03:23.063032+0000","last_became_peered":"2023-10-07T11:03:23.063032+0000","last_unstale":"2023-10-07T11:03:23.067016+0000","last_undegraded":"2023-10-07T11:03:23.067016+0000","last_fullsized":"2023-10-07T11:03:23.067016+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:42.550716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.059900+0000","last_change":"2023-10-07T11:03:22.052787+0000","last_active":"2023-10-07T11:03:23.059900+0000","last_peered":"2023-10-07T11:03:23.059900+0000","last_clean":"2023-10-07T11:03:23.059900+0000","last_became_active":"2023-10-07T11:03:22.052454+0000","last_became_peered":"2023-10-07T11:03:22.052454+0000","last_unstale":"2023-10-07T11:03:23.059900+0000","last_undegraded":"2023-10-07T11:03:23.059900+0000","last_fullsized":"2023-10-07T11:03:23.059900+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:32:16.922700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.060058+0000","last_change":"2023-10-07T11:03:23.057682+0000","last_active":"2023-10-07T11:03:23.060058+0000","last_peered":"2023-10-07T11:03:23.060058+0000","last_clean":"2023-10-07T11:03:23.060058+0000","last_became_active":"2023-10-07T11:03:23.056838+0000","last_became_peered":"2023-10-07T11:03:23.056838+0000","last_unstale":"2023-10-07T11:03:23.060058+0000","last_undegraded":"2023-10-07T11:03:23.060058+0000","last_fullsized":"2023-10-07T11:03:23.060058+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:22.817902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.055372+0000","last_change":"2023-10-07T11:03:22.043828+0000","last_active":"2023-10-07T11:03:23.055372+0000","last_peered":"2023-10-07T11:03:23.055372+0000","last_clean":"2023-10-07T11:03:23.055372+0000","last_became_active":"2023-10-07T11:03:22.043538+0000","last_became_peered":"2023-10-07T11:03:22.043538+0000","last_unstale":"2023-10-07T11:03:23.055372+0000","last_undegraded":"2023-10-07T11:03:23.055372+0000","last_fullsized":"2023-10-07T11:03:23.055372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:56:00.384938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074728+0000","last_change":"2023-10-07T11:03:23.066795+0000","last_active":"2023-10-07T11:03:23.074728+0000","last_peered":"2023-10-07T11:03:23.074728+0000","last_clean":"2023-10-07T11:03:23.074728+0000","last_became_active":"2023-10-07T11:03:23.066509+0000","last_became_peered":"2023-10-07T11:03:23.066509+0000","last_unstale":"2023-10-07T11:03:23.074728+0000","last_undegraded":"2023-10-07T11:03:23.074728+0000","last_fullsized":"2023-10-07T11:03:23.074728+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:40:40.866050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.052659+0000","last_change":"2023-10-07T11:03:22.058923+0000","last_active":"2023-10-07T11:03:23.052659+0000","last_peered":"2023-10-07T11:03:23.052659+0000","last_clean":"2023-10-07T11:03:23.052659+0000","last_became_active":"2023-10-07T11:03:22.058689+0000","last_became_peered":"2023-10-07T11:03:22.058689+0000","last_unstale":"2023-10-07T11:03:23.052659+0000","last_undegraded":"2023-10-07T11:03:23.052659+0000","last_fullsized":"2023-10-07T11:03:23.052659+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:23:25.739404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073238+0000","last_change":"2023-10-07T11:03:23.064829+0000","last_active":"2023-10-07T11:03:23.073238+0000","last_peered":"2023-10-07T11:03:23.073238+0000","last_clean":"2023-10-07T11:03:23.073238+0000","last_became_active":"2023-10-07T11:03:23.064570+0000","last_became_peered":"2023-10-07T11:03:23.064570+0000","last_unstale":"2023-10-07T11:03:23.073238+0000","last_undegraded":"2023-10-07T11:03:23.073238+0000","last_fullsized":"2023-10-07T11:03:23.073238+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:07.416024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"196'3","reported_seq":13,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095493+0000","last_change":"2023-10-07T11:03:22.057778+0000","last_active":"2023-10-07T11:03:23.095493+0000","last_peered":"2023-10-07T11:03:23.095493+0000","last_clean":"2023-10-07T11:03:23.095493+0000","last_became_active":"2023-10-07T11:03:22.057359+0000","last_became_peered":"2023-10-07T11:03:22.057359+0000","last_unstale":"2023-10-07T11:03:23.095493+0000","last_undegraded":"2023-10-07T11:03:23.095493+0000","last_fullsized":"2023-10-07T11:03:23.095493+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:42:49.802708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072955+0000","last_change":"2023-10-07T11:03:23.056349+0000","last_active":"2023-10-07T11:03:23.072955+0000","last_peered":"2023-10-07T11:03:23.072955+0000","last_clean":"2023-10-07T11:03:23.072955+0000","last_became_active":"2023-10-07T11:03:23.055709+0000","last_became_peered":"2023-10-07T11:03:23.055709+0000","last_unstale":"2023-10-07T11:03:23.072955+0000","last_undegraded":"2023-10-07T11:03:23.072955+0000","last_fullsized":"2023-10-07T11:03:23.072955+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:34:35.644574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"196'6","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.097246+0000","last_change":"2023-10-07T11:03:22.043324+0000","last_active":"2023-10-07T11:03:23.097246+0000","last_peered":"2023-10-07T11:03:23.097246+0000","last_clean":"2023-10-07T11:03:23.097246+0000","last_became_active":"2023-10-07T11:03:22.043036+0000","last_became_peered":"2023-10-07T11:03:22.043036+0000","last_unstale":"2023-10-07T11:03:23.097246+0000","last_undegraded":"2023-10-07T11:03:23.097246+0000","last_fullsized":"2023-10-07T11:03:23.097246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072666+0000","last_change":"2023-10-07T11:03:23.070261+0000","last_active":"2023-10-07T11:03:23.072666+0000","last_peered":"2023-10-07T11:03:23.072666+0000","last_clean":"2023-10-07T11:03:23.072666+0000","last_became_active":"2023-10-07T11:03:23.069876+0000","last_became_peered":"2023-10-07T11:03:23.069876+0000","last_unstale":"2023-10-07T11:03:23.072666+0000","last_undegraded":"2023-10-07T11:03:23.072666+0000","last_fullsized":"2023-10-07T11:03:23.072666+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:43.183248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.080351+0000","last_change":"2023-10-07T11:03:22.053525+0000","last_active":"2023-10-07T11:03:23.080351+0000","last_peered":"2023-10-07T11:03:23.080351+0000","last_clean":"2023-10-07T11:03:23.080351+0000","last_became_active":"2023-10-07T11:03:22.053261+0000","last_became_peered":"2023-10-07T11:03:22.053261+0000","last_unstale":"2023-10-07T11:03:23.080351+0000","last_undegraded":"2023-10-07T11:03:23.080351+0000","last_fullsized":"2023-10-07T11:03:23.080351+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:57:44.508450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.079297+0000","last_change":"2023-10-07T11:03:23.075226+0000","last_active":"2023-10-07T11:03:23.079297+0000","last_peered":"2023-10-07T11:03:23.079297+0000","last_clean":"2023-10-07T11:03:23.079297+0000","last_became_active":"2023-10-07T11:03:23.075002+0000","last_became_peered":"2023-10-07T11:03:23.075002+0000","last_unstale":"2023-10-07T11:03:23.079297+0000","last_undegraded":"2023-10-07T11:03:23.079297+0000","last_fullsized":"2023-10-07T11:03:23.079297+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:17:46.671473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046972+0000","last_change":"2023-10-07T11:03:22.058631+0000","last_active":"2023-10-07T11:03:23.046972+0000","last_peered":"2023-10-07T11:03:23.046972+0000","last_clean":"2023-10-07T11:03:23.046972+0000","last_became_active":"2023-10-07T11:03:22.058368+0000","last_became_peered":"2023-10-07T11:03:22.058368+0000","last_unstale":"2023-10-07T11:03:23.046972+0000","last_undegraded":"2023-10-07T11:03:23.046972+0000","last_fullsized":"2023-10-07T11:03:23.046972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:38:41.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.064342+0000","last_change":"2023-10-07T11:03:23.059336+0000","last_active":"2023-10-07T11:03:23.064342+0000","last_peered":"2023-10-07T11:03:23.064342+0000","last_clean":"2023-10-07T11:03:23.064342+0000","last_became_active":"2023-10-07T11:03:23.058941+0000","last_became_peered":"2023-10-07T11:03:23.058941+0000","last_unstale":"2023-10-07T11:03:23.064342+0000","last_undegraded":"2023-10-07T11:03:23.064342+0000","last_fullsized":"2023-10-07T11:03:23.064342+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:25:42.771748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050617+0000","last_change":"2023-10-07T11:03:22.046240+0000","last_active":"2023-10-07T11:03:23.050617+0000","last_peered":"2023-10-07T11:03:23.050617+0000","last_clean":"2023-10-07T11:03:23.050617+0000","last_became_active":"2023-10-07T11:03:22.045964+0000","last_became_peered":"2023-10-07T11:03:22.045964+0000","last_unstale":"2023-10-07T11:03:23.050617+0000","last_undegraded":"2023-10-07T11:03:23.050617+0000","last_fullsized":"2023-10-07T11:03:23.050617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:25.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072532+0000","last_change":"2023-10-07T11:03:23.069327+0000","last_active":"2023-10-07T11:03:23.072532+0000","last_peered":"2023-10-07T11:03:23.072532+0000","last_clean":"2023-10-07T11:03:23.072532+0000","last_became_active":"2023-10-07T11:03:23.068992+0000","last_became_peered":"2023-10-07T11:03:23.068992+0000","last_unstale":"2023-10-07T11:03:23.072532+0000","last_undegraded":"2023-10-07T11:03:23.072532+0000","last_fullsized":"2023-10-07T11:03:23.072532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"196'8","reported_seq":18,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.082714+0000","last_change":"2023-10-07T11:03:22.053477+0000","last_active":"2023-10-07T11:03:23.082714+0000","last_peered":"2023-10-07T11:03:23.082714+0000","last_clean":"2023-10-07T11:03:23.082714+0000","last_became_active":"2023-10-07T11:03:22.053179+0000","last_became_peered":"2023-10-07T11:03:22.053179+0000","last_unstale":"2023-10-07T11:03:23.082714+0000","last_undegraded":"2023-10-07T11:03:23.082714+0000","last_fullsized":"2023-10-07T11:03:23.082714+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:16:49.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074503+0000","last_change":"2023-10-07T11:03:23.069902+0000","last_active":"2023-10-07T11:03:23.074503+0000","last_peered":"2023-10-07T11:03:23.074503+0000","last_clean":"2023-10-07T11:03:23.074503+0000","last_became_active":"2023-10-07T11:03:23.069641+0000","last_became_peered":"2023-10-07T11:03:23.069641+0000","last_unstale":"2023-10-07T11:03:23.074503+0000","last_undegraded":"2023-10-07T11:03:23.074503+0000","last_fullsized":"2023-10-07T11:03:23.074503+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.095407+0000","last_change":"2023-10-07T11:03:22.058339+0000","last_active":"2023-10-07T11:03:23.095407+0000","last_peered":"2023-10-07T11:03:23.095407+0000","last_clean":"2023-10-07T11:03:23.095407+0000","last_became_active":"2023-10-07T11:03:22.057722+0000","last_became_peered":"2023-10-07T11:03:22.057722+0000","last_unstale":"2023-10-07T11:03:23.095407+0000","last_undegraded":"2023-10-07T11:03:23.095407+0000","last_fullsized":"2023-10-07T11:03:23.095407+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:22:25.667544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.064670+0000","last_change":"2023-10-07T11:03:23.059140+0000","last_active":"2023-10-07T11:03:23.064670+0000","last_peered":"2023-10-07T11:03:23.064670+0000","last_clean":"2023-10-07T11:03:23.064670+0000","last_became_active":"2023-10-07T11:03:23.058890+0000","last_became_peered":"2023-10-07T11:03:23.058890+0000","last_unstale":"2023-10-07T11:03:23.064670+0000","last_undegraded":"2023-10-07T11:03:23.064670+0000","last_fullsized":"2023-10-07T11:03:23.064670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:54:27.529211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083381+0000","last_change":"2023-10-07T11:03:22.049580+0000","last_active":"2023-10-07T11:03:23.083381+0000","last_peered":"2023-10-07T11:03:23.083381+0000","last_clean":"2023-10-07T11:03:23.083381+0000","last_became_active":"2023-10-07T11:03:22.049343+0000","last_became_peered":"2023-10-07T11:03:22.049343+0000","last_unstale":"2023-10-07T11:03:23.083381+0000","last_undegraded":"2023-10-07T11:03:23.083381+0000","last_fullsized":"2023-10-07T11:03:23.083381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:26:39.011312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072207+0000","last_change":"2023-10-07T11:03:23.068719+0000","last_active":"2023-10-07T11:03:23.072207+0000","last_peered":"2023-10-07T11:03:23.072207+0000","last_clean":"2023-10-07T11:03:23.072207+0000","last_became_active":"2023-10-07T11:03:23.068451+0000","last_became_peered":"2023-10-07T11:03:23.068451+0000","last_unstale":"2023-10-07T11:03:23.072207+0000","last_undegraded":"2023-10-07T11:03:23.072207+0000","last_fullsized":"2023-10-07T11:03:23.072207+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T19:14:43.392569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"196'1","reported_seq":11,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.083037+0000","last_change":"2023-10-07T11:03:22.054996+0000","last_active":"2023-10-07T11:03:23.083037+0000","last_peered":"2023-10-07T11:03:23.083037+0000","last_clean":"2023-10-07T11:03:23.083037+0000","last_became_active":"2023-10-07T11:03:22.054736+0000","last_became_peered":"2023-10-07T11:03:22.054736+0000","last_unstale":"2023-10-07T11:03:23.083037+0000","last_undegraded":"2023-10-07T11:03:23.083037+0000","last_fullsized":"2023-10-07T11:03:23.083037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:48:47.480334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.067405+0000","last_change":"2023-10-07T11:03:23.060415+0000","last_active":"2023-10-07T11:03:23.067405+0000","last_peered":"2023-10-07T11:03:23.067405+0000","last_clean":"2023-10-07T11:03:23.067405+0000","last_became_active":"2023-10-07T11:03:23.060136+0000","last_became_peered":"2023-10-07T11:03:23.060136+0000","last_unstale":"2023-10-07T11:03:23.067405+0000","last_undegraded":"2023-10-07T11:03:23.067405+0000","last_fullsized":"2023-10-07T11:03:23.067405+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:43:26.620432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.055604+0000","last_change":"2023-10-07T11:03:22.040612+0000","last_active":"2023-10-07T11:03:23.055604+0000","last_peered":"2023-10-07T11:03:23.055604+0000","last_clean":"2023-10-07T11:03:23.055604+0000","last_became_active":"2023-10-07T11:03:22.040349+0000","last_became_peered":"2023-10-07T11:03:22.040349+0000","last_unstale":"2023-10-07T11:03:23.055604+0000","last_undegraded":"2023-10-07T11:03:23.055604+0000","last_fullsized":"2023-10-07T11:03:23.055604+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:58:07.103857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072519+0000","last_change":"2023-10-07T11:03:23.069558+0000","last_active":"2023-10-07T11:03:23.072519+0000","last_peered":"2023-10-07T11:03:23.072519+0000","last_clean":"2023-10-07T11:03:23.072519+0000","last_became_active":"2023-10-07T11:03:23.069179+0000","last_became_peered":"2023-10-07T11:03:23.069179+0000","last_unstale":"2023-10-07T11:03:23.072519+0000","last_undegraded":"2023-10-07T11:03:23.072519+0000","last_fullsized":"2023-10-07T11:03:23.072519+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081863+0000","last_change":"2023-10-07T11:03:22.060469+0000","last_active":"2023-10-07T11:03:23.081863+0000","last_peered":"2023-10-07T11:03:23.081863+0000","last_clean":"2023-10-07T11:03:23.081863+0000","last_became_active":"2023-10-07T11:03:22.060024+0000","last_became_peered":"2023-10-07T11:03:22.060024+0000","last_unstale":"2023-10-07T11:03:23.081863+0000","last_undegraded":"2023-10-07T11:03:23.081863+0000","last_fullsized":"2023-10-07T11:03:23.081863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:12:42.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072314+0000","last_change":"2023-10-07T11:03:23.064217+0000","last_active":"2023-10-07T11:03:23.072314+0000","last_peered":"2023-10-07T11:03:23.072314+0000","last_clean":"2023-10-07T11:03:23.072314+0000","last_became_active":"2023-10-07T11:03:23.063800+0000","last_became_peered":"2023-10-07T11:03:23.063800+0000","last_unstale":"2023-10-07T11:03:23.072314+0000","last_undegraded":"2023-10-07T11:03:23.072314+0000","last_fullsized":"2023-10-07T11:03:23.072314+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:01:55.973012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.053415+0000","last_change":"2023-10-07T11:03:22.057330+0000","last_active":"2023-10-07T11:03:23.053415+0000","last_peered":"2023-10-07T11:03:23.053415+0000","last_clean":"2023-10-07T11:03:23.053415+0000","last_became_active":"2023-10-07T11:03:22.057038+0000","last_became_peered":"2023-10-07T11:03:22.057038+0000","last_unstale":"2023-10-07T11:03:23.053415+0000","last_undegraded":"2023-10-07T11:03:23.053415+0000","last_fullsized":"2023-10-07T11:03:23.053415+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:04.566950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073798+0000","last_change":"2023-10-07T11:03:23.068248+0000","last_active":"2023-10-07T11:03:23.073798+0000","last_peered":"2023-10-07T11:03:23.073798+0000","last_clean":"2023-10-07T11:03:23.073798+0000","last_became_active":"2023-10-07T11:03:23.067904+0000","last_became_peered":"2023-10-07T11:03:23.067904+0000","last_unstale":"2023-10-07T11:03:23.073798+0000","last_undegraded":"2023-10-07T11:03:23.073798+0000","last_fullsized":"2023-10-07T11:03:23.073798+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T13:14:37.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050442+0000","last_change":"2023-10-07T11:03:22.059410+0000","last_active":"2023-10-07T11:03:23.050442+0000","last_peered":"2023-10-07T11:03:23.050442+0000","last_clean":"2023-10-07T11:03:23.050442+0000","last_became_active":"2023-10-07T11:03:22.059029+0000","last_became_peered":"2023-10-07T11:03:22.059029+0000","last_unstale":"2023-10-07T11:03:23.050442+0000","last_undegraded":"2023-10-07T11:03:23.050442+0000","last_fullsized":"2023-10-07T11:03:23.050442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:57:18.642115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070843+0000","last_change":"2023-10-07T11:03:23.064553+0000","last_active":"2023-10-07T11:03:23.070843+0000","last_peered":"2023-10-07T11:03:23.070843+0000","last_clean":"2023-10-07T11:03:23.070843+0000","last_became_active":"2023-10-07T11:03:23.064308+0000","last_became_peered":"2023-10-07T11:03:23.064308+0000","last_unstale":"2023-10-07T11:03:23.070843+0000","last_undegraded":"2023-10-07T11:03:23.070843+0000","last_fullsized":"2023-10-07T11:03:23.070843+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:23:59.483751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.049306+0000","last_change":"2023-10-07T11:03:22.047245+0000","last_active":"2023-10-07T11:03:23.049306+0000","last_peered":"2023-10-07T11:03:23.049306+0000","last_clean":"2023-10-07T11:03:23.049306+0000","last_became_active":"2023-10-07T11:03:22.046970+0000","last_became_peered":"2023-10-07T11:03:22.046970+0000","last_unstale":"2023-10-07T11:03:23.049306+0000","last_undegraded":"2023-10-07T11:03:23.049306+0000","last_fullsized":"2023-10-07T11:03:23.049306+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T17:49:04.566950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.071751+0000","last_change":"2023-10-07T11:03:23.065086+0000","last_active":"2023-10-07T11:03:23.071751+0000","last_peered":"2023-10-07T11:03:23.071751+0000","last_clean":"2023-10-07T11:03:23.071751+0000","last_became_active":"2023-10-07T11:03:23.064822+0000","last_became_peered":"2023-10-07T11:03:23.064822+0000","last_unstale":"2023-10-07T11:03:23.071751+0000","last_undegraded":"2023-10-07T11:03:23.071751+0000","last_fullsized":"2023-10-07T11:03:23.071751+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:22:18.490656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"196'2","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.081692+0000","last_change":"2023-10-07T11:03:22.059458+0000","last_active":"2023-10-07T11:03:23.081692+0000","last_peered":"2023-10-07T11:03:23.081692+0000","last_clean":"2023-10-07T11:03:23.081692+0000","last_became_active":"2023-10-07T11:03:22.059162+0000","last_became_peered":"2023-10-07T11:03:22.059162+0000","last_unstale":"2023-10-07T11:03:23.081692+0000","last_undegraded":"2023-10-07T11:03:23.081692+0000","last_fullsized":"2023-10-07T11:03:23.081692+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:08:26.185841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.070233+0000","last_change":"2023-10-07T11:03:23.062012+0000","last_active":"2023-10-07T11:03:23.070233+0000","last_peered":"2023-10-07T11:03:23.070233+0000","last_clean":"2023-10-07T11:03:23.070233+0000","last_became_active":"2023-10-07T11:03:23.061769+0000","last_became_peered":"2023-10-07T11:03:23.061769+0000","last_unstale":"2023-10-07T11:03:23.070233+0000","last_undegraded":"2023-10-07T11:03:23.070233+0000","last_fullsized":"2023-10-07T11:03:23.070233+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:42:58.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"197'1","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-07T11:03:24.222911+0000","last_change":"2023-10-07T11:03:22.051956+0000","last_active":"2023-10-07T11:03:24.222911+0000","last_peered":"2023-10-07T11:03:24.222911+0000","last_clean":"2023-10-07T11:03:24.222911+0000","last_became_active":"2023-10-07T11:03:22.051691+0000","last_became_peered":"2023-10-07T11:03:22.051691+0000","last_unstale":"2023-10-07T11:03:24.222911+0000","last_undegraded":"2023-10-07T11:03:24.222911+0000","last_fullsized":"2023-10-07T11:03:24.222911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:58:42.177620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.065367+0000","last_change":"2023-10-07T11:03:23.059948+0000","last_active":"2023-10-07T11:03:23.065367+0000","last_peered":"2023-10-07T11:03:23.065367+0000","last_clean":"2023-10-07T11:03:23.065367+0000","last_became_active":"2023-10-07T11:03:23.059564+0000","last_became_peered":"2023-10-07T11:03:23.059564+0000","last_unstale":"2023-10-07T11:03:23.065367+0000","last_undegraded":"2023-10-07T11:03:23.065367+0000","last_fullsized":"2023-10-07T11:03:23.065367+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:04:25.554055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.046689+0000","last_change":"2023-10-07T11:03:22.052493+0000","last_active":"2023-10-07T11:03:23.046689+0000","last_peered":"2023-10-07T11:03:23.046689+0000","last_clean":"2023-10-07T11:03:23.046689+0000","last_became_active":"2023-10-07T11:03:22.052235+0000","last_became_peered":"2023-10-07T11:03:22.052235+0000","last_unstale":"2023-10-07T11:03:23.046689+0000","last_undegraded":"2023-10-07T11:03:23.046689+0000","last_fullsized":"2023-10-07T11:03:23.046689+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:40:06.410397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074593+0000","last_change":"2023-10-07T11:03:23.060626+0000","last_active":"2023-10-07T11:03:23.074593+0000","last_peered":"2023-10-07T11:03:23.074593+0000","last_clean":"2023-10-07T11:03:23.074593+0000","last_became_active":"2023-10-07T11:03:23.060260+0000","last_became_peered":"2023-10-07T11:03:23.060260+0000","last_unstale":"2023-10-07T11:03:23.074593+0000","last_undegraded":"2023-10-07T11:03:23.074593+0000","last_fullsized":"2023-10-07T11:03:23.074593+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:05:41.262384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051338+0000","last_change":"2023-10-07T11:03:22.048854+0000","last_active":"2023-10-07T11:03:23.051338+0000","last_peered":"2023-10-07T11:03:23.051338+0000","last_clean":"2023-10-07T11:03:23.051338+0000","last_became_active":"2023-10-07T11:03:22.048568+0000","last_became_peered":"2023-10-07T11:03:22.048568+0000","last_unstale":"2023-10-07T11:03:23.051338+0000","last_undegraded":"2023-10-07T11:03:23.051338+0000","last_fullsized":"2023-10-07T11:03:23.051338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:01:19.581526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073623+0000","last_change":"2023-10-07T11:03:23.066567+0000","last_active":"2023-10-07T11:03:23.073623+0000","last_peered":"2023-10-07T11:03:23.073623+0000","last_clean":"2023-10-07T11:03:23.073623+0000","last_became_active":"2023-10-07T11:03:23.066175+0000","last_became_peered":"2023-10-07T11:03:23.066175+0000","last_unstale":"2023-10-07T11:03:23.073623+0000","last_undegraded":"2023-10-07T11:03:23.073623+0000","last_fullsized":"2023-10-07T11:03:23.073623+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T21:10:38.363015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.048793+0000","last_change":"2023-10-07T11:03:22.043072+0000","last_active":"2023-10-07T11:03:23.048793+0000","last_peered":"2023-10-07T11:03:23.048793+0000","last_clean":"2023-10-07T11:03:23.048793+0000","last_became_active":"2023-10-07T11:03:22.042614+0000","last_became_peered":"2023-10-07T11:03:22.042614+0000","last_unstale":"2023-10-07T11:03:23.048793+0000","last_undegraded":"2023-10-07T11:03:23.048793+0000","last_fullsized":"2023-10-07T11:03:23.048793+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:20:03.006488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.074442+0000","last_change":"2023-10-07T11:03:23.071149+0000","last_active":"2023-10-07T11:03:23.074442+0000","last_peered":"2023-10-07T11:03:23.074442+0000","last_clean":"2023-10-07T11:03:23.074442+0000","last_became_active":"2023-10-07T11:03:23.070906+0000","last_became_peered":"2023-10-07T11:03:23.070906+0000","last_unstale":"2023-10-07T11:03:23.074442+0000","last_undegraded":"2023-10-07T11:03:23.074442+0000","last_fullsized":"2023-10-07T11:03:23.074442+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:11:59.325550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.049537+0000","last_change":"2023-10-07T11:03:22.042941+0000","last_active":"2023-10-07T11:03:23.049537+0000","last_peered":"2023-10-07T11:03:23.049537+0000","last_clean":"2023-10-07T11:03:23.049537+0000","last_became_active":"2023-10-07T11:03:22.042595+0000","last_became_peered":"2023-10-07T11:03:22.042595+0000","last_unstale":"2023-10-07T11:03:23.049537+0000","last_undegraded":"2023-10-07T11:03:23.049537+0000","last_fullsized":"2023-10-07T11:03:23.049537+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T14:50:02.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.069701+0000","last_change":"2023-10-07T11:03:23.063216+0000","last_active":"2023-10-07T11:03:23.069701+0000","last_peered":"2023-10-07T11:03:23.069701+0000","last_clean":"2023-10-07T11:03:23.069701+0000","last_became_active":"2023-10-07T11:03:23.061923+0000","last_became_peered":"2023-10-07T11:03:23.061923+0000","last_unstale":"2023-10-07T11:03:23.069701+0000","last_undegraded":"2023-10-07T11:03:23.069701+0000","last_fullsized":"2023-10-07T11:03:23.069701+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:21:58.043707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.050906+0000","last_change":"2023-10-07T11:03:22.046244+0000","last_active":"2023-10-07T11:03:23.050906+0000","last_peered":"2023-10-07T11:03:23.050906+0000","last_clean":"2023-10-07T11:03:23.050906+0000","last_became_active":"2023-10-07T11:03:22.045897+0000","last_became_peered":"2023-10-07T11:03:22.045897+0000","last_unstale":"2023-10-07T11:03:23.050906+0000","last_undegraded":"2023-10-07T11:03:23.050906+0000","last_fullsized":"2023-10-07T11:03:23.050906+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T15:23:21.812275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.072310+0000","last_change":"2023-10-07T11:03:23.070377+0000","last_active":"2023-10-07T11:03:23.072310+0000","last_peered":"2023-10-07T11:03:23.072310+0000","last_clean":"2023-10-07T11:03:23.072310+0000","last_became_active":"2023-10-07T11:03:23.070101+0000","last_became_peered":"2023-10-07T11:03:23.070101+0000","last_unstale":"2023-10-07T11:03:23.072310+0000","last_undegraded":"2023-10-07T11:03:23.072310+0000","last_fullsized":"2023-10-07T11:03:23.072310+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T12:52:22.545174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.051314+0000","last_change":"2023-10-07T11:03:22.060700+0000","last_active":"2023-10-07T11:03:23.051314+0000","last_peered":"2023-10-07T11:03:23.051314+0000","last_clean":"2023-10-07T11:03:23.051314+0000","last_became_active":"2023-10-07T11:03:22.060037+0000","last_became_peered":"2023-10-07T11:03:22.060037+0000","last_unstale":"2023-10-07T11:03:23.051314+0000","last_undegraded":"2023-10-07T11:03:23.051314+0000","last_fullsized":"2023-10-07T11:03:23.051314+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T18:35:53.212268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.073064+0000","last_change":"2023-10-07T11:03:23.058853+0000","last_active":"2023-10-07T11:03:23.073064+0000","last_peered":"2023-10-07T11:03:23.073064+0000","last_clean":"2023-10-07T11:03:23.073064+0000","last_became_active":"2023-10-07T11:03:23.058594+0000","last_became_peered":"2023-10-07T11:03:23.058594+0000","last_unstale":"2023-10-07T11:03:23.073064+0000","last_undegraded":"2023-10-07T11:03:23.073064+0000","last_fullsized":"2023-10-07T11:03:23.073064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:22.027197+0000","last_clean_scrub_stamp":"2023-10-07T11:03:22.027197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T11:24:21.364704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.059672+0000","last_change":"2023-10-07T11:03:22.050935+0000","last_active":"2023-10-07T11:03:23.059672+0000","last_peered":"2023-10-07T11:03:23.059672+0000","last_clean":"2023-10-07T11:03:23.059672+0000","last_became_active":"2023-10-07T11:03:22.050667+0000","last_became_peered":"2023-10-07T11:03:22.050667+0000","last_unstale":"2023-10-07T11:03:23.059672+0000","last_undegraded":"2023-10-07T11:03:23.059672+0000","last_fullsized":"2023-10-07T11:03:23.059672+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T11:03:21.021570+0000","last_clean_scrub_stamp":"2023-10-07T11:03:21.021570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T16:51:48.346681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"186'348","reported_seq":1131,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-07T11:03:23.059922+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:23.059922+0000","last_peered":"2023-10-07T11:03:23.059922+0000","last_clean":"2023-10-07T11:03:23.059922+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:23.059922+0000","last_undegraded":"2023-10-07T11:03:23.059922+0000","last_fullsized":"2023-10-07T11:03:23.059922+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477855,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312568,"kb_used_data":2992,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059272,"statfs":{"total":96636764160,"available":96316694528,"internally_reserved":0,"allocated":3063808,"data_stored":1811016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.69499999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.73299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.627}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.747}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.78000000000000003}]}]},{"osd":6,"up_from":15,"seq":64424510559,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316076,"kb_used_data":2404,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055764,"statfs":{"total":96636764160,"available":96313102336,"internally_reserved":0,"allocated":2461696,"data_stored":1220544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.75}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.79300000000000004}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.80700000000000005}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.71999999999999997}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.94299999999999995}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":1.034}]}]},{"osd":1,"up_from":14,"seq":60129543261,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307896,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063944,"statfs":{"total":96636764160,"available":96321478656,"internally_reserved":0,"allocated":2473984,"data_stored":1222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40300000000000002},"max":{"1min":0.76700000000000002,"5min":0.86299999999999999,"15min":0.94199999999999995},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.40000000000000002},"max":{"1min":0.82699999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.443,"5min":0.378,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.94499999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82699999999999996,"15min":0.84099999999999997},"min":{"1min":0.55600000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.55500000000000005,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.153,"5min":1.1870000000000001,"15min":1.2430000000000001},"last":0.72599999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84999999999999998},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80600000000000005,"15min":0.81299999999999994},"min":{"1min":0.57999999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.68899999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.873,"15min":0.871},"min":{"1min":0.65000000000000002,"5min":0.61799999999999999,"15min":0.58399999999999996},"max":{"1min":1.052,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0980000000000001,"5min":1.157,"15min":1.302},"last":0.73899999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.877,"15min":0.88400000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.141,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88700000000000001,"15min":0.878},"min":{"1min":0.66700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.70799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129543261,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1810934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.439},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.496}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.93600000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.008},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.86399999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1221210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.81699999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.30399999999999999},"max":{"1min":0.69399999999999995,"5min":1.087,"15min":1.087},"last":0.435}]},{"osd":1,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.95099999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.80800000000000005,"5min":0.86399999999999999,"15min":0.98199999999999998},"last":0.67500000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59799999999999998,"5min":0.71799999999999997,"15min":0.80000000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82199999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.622}]},{"osd":4,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.78900000000000003},"min":{"1min":0.63100000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.163,"15min":1.1879999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.266,"15min":1.5640000000000001},"last":0.77800000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.59199999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.004,"5min":1.2989999999999999,"15min":1.528},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78100000000000003,"15min":0.80300000000000005},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95999999999999996,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.65200000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0580000000000001,"5min":1.242,"15min":1.3779999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.86799999999999999},"min":{"1min":0.67600000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0269999999999999,"5min":1.2050000000000001,"15min":1.629},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0569999999999999,"5min":1.274,"15min":1.5940000000000001},"last":0.61099999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543263,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299704,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072136,"statfs":{"total":96636764160,"available":96329867264,"internally_reserved":0,"allocated":2473984,"data_stored":1220738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72199999999999998,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.497}]},{"osd":1,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.83999999999999997,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79700000000000004,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55500000000000005,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73099999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.437},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.436,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.51400000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96799999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81599999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0780000000000001,"15min":1.222},"last":0.93400000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.216},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0720000000000001,"5min":1.27,"15min":1.27},"last":0.753}]},{"osd":6,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84199999999999997,"15min":0.84699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98399999999999999,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.96699999999999997,"5min":1.389,"15min":1.389},"last":0.89900000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0269999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.87,"15min":0.876},"min":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.008,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.97599999999999998}]}]},{"osd":4,"up_from":15,"seq":64424510559,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316092,"kb_used_data":2420,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055748,"statfs":{"total":96636764160,"available":96313085952,"internally_reserved":0,"allocated":2478080,"data_stored":1222634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.439},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.74399999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.81200000000000006}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.84999999999999998}]}]},{"osd":5,"up_from":14,"seq":60129543261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311980,"kb_used_data":2404,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059860,"statfs":{"total":96636764160,"available":96317296640,"internally_reserved":0,"allocated":2461696,"data_stored":1220544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.78000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93300000000000005,"5min":1.585,"15min":1.585},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.81299999999999994},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.90900000000000003,"5min":1.673,"15min":1.673},"last":0.82999999999999996}]},{"osd":1,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73799999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":1.139},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92500000000000004,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.95399999999999996,"5min":1.0549999999999999,"15min":1.069},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.91400000000000003,"5min":0.96899999999999997,"15min":1.161},"last":0.73899999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.878},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.69299999999999995}]},{"osd":6,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64300000000000002,"15min":0.625},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.91400000000000003}]},{"osd":7,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.69299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0109999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.038,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.93899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:03:32.879 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-07T11:03:32.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-07T11:03:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.114+0000 7fce471ff700 1 Processor -- start 2023-10-07T11:03:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.114+0000 7fce471ff700 1 -- start start 2023-10-07T11:03:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.115+0000 7fce471ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce400d0010 0x7fce400b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.115+0000 7fce471ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce400c62d0 con 0x7fce400cb800 2023-10-07T11:03:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.115+0000 7fce471ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce40159e10 con 0x7fce400cfbb0 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.115+0000 7fce471ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce400d0410 con 0x7fce400d0010 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce44f9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fce400cb800 0x7fce400cbbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41392/0 (socket says 172.21.15.29:41392) 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce44f9b700 1 -- 172.21.15.29:0/2619200631 learned_addr learned my addr 172.21.15.29:0/2619200631 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce4579c700 1 --2- 172.21.15.29:0/2619200631 >> v2:172.21.15.29:3300/0 conn(0x7fce400d0010 0x7fce400b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce3f7fe700 1 -- 172.21.15.29:0/2619200631 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174292961 0 0) 0x7fce400c62d0 con 0x7fce400cb800 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce3f7fe700 1 -- 172.21.15.29:0/2619200631 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce30003570 con 0x7fce400cb800 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce4579c700 1 -- 172.21.15.29:0/2619200631 >> v1:172.21.15.144:6789/0 conn(0x7fce400cfbb0 legacy=0x7fce400c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce4579c700 1 -- 172.21.15.29:0/2619200631 >> v1:172.21.15.29:6789/0 conn(0x7fce400cb800 legacy=0x7fce400cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce4579c700 1 -- 172.21.15.29:0/2619200631 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce400c56c0 con 0x7fce400d0010 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.116+0000 7fce4579c700 1 --2- 172.21.15.29:0/2619200631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400d0010 0x7fce400b3340 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fce38009e60 tx=0x7fce38009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2028710cfd8ab765 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce3f7fe700 1 -- 172.21.15.29:0/2619200631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3800e070 con 0x7fce400d0010 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce3f7fe700 1 -- 172.21.15.29:0/2619200631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3800bbf0 con 0x7fce400d0010 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce3f7fe700 1 -- 172.21.15.29:0/2619200631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38012430 con 0x7fce400d0010 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 -- 172.21.15.29:0/2619200631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400d0010 msgr2=0x7fce400b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 --2- 172.21.15.29:0/2619200631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400d0010 0x7fce400b3340 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fce38009e60 tx=0x7fce38009780 comp rx=0 tx=0).stop 2023-10-07T11:03:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 -- 172.21.15.29:0/2619200631 shutdown_connections 2023-10-07T11:03:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 --2- 172.21.15.29:0/2619200631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400d0010 0x7fce400b3340 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 -- 172.21.15.29:0/2619200631 >> 172.21.15.29:0/2619200631 conn(0x7fce40063a10 msgr2=0x7fce400d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 -- 172.21.15.29:0/2619200631 shutdown_connections 2023-10-07T11:03:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.117+0000 7fce471ff700 1 -- 172.21.15.29:0/2619200631 wait complete. 2023-10-07T11:03:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.119+0000 7fce471ff700 1 Processor -- start 2023-10-07T11:03:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.119+0000 7fce471ff700 1 -- start start 2023-10-07T11:03:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce471ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 0x7fce40153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce471ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce400cfbb0 0x7fce400c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce471ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce400d0010 0x7fce400c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce471ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce40155200 con 0x7fce400cb800 2023-10-07T11:03:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce471ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce401545f0 con 0x7fce400d0010 2023-10-07T11:03:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce471ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce401548d0 con 0x7fce400cfbb0 2023-10-07T11:03:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 0x7fce40153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 0x7fce40153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49944/0 (socket says 172.21.15.29:49944) 2023-10-07T11:03:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 -- 172.21.15.29:0/839563196 learned_addr learned my addr 172.21.15.29:0/839563196 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 -- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce400cfbb0 msgr2=0x7fce400c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce4579c700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce400d0010 0x7fce400c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce400cfbb0 0x7fce400c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 -- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce400d0010 msgr2=0x7fce400c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.120+0000 7fce44f9b700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce400d0010 0x7fce400c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce44f9b700 1 -- 172.21.15.29:0/839563196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38009a20 con 0x7fce400cb800 2023-10-07T11:03:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce44f9b700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 0x7fce40153bc0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c790 tx=0x7fce34007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34007300 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce34007460 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340057f0 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce40154bb0 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.121+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4014ed30 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.122+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce400718b0 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.123+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fce34005c20 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.124+0000 7fce3cff9700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fce24066f30 0x7fce240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.125+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fce340841d0 con 0x7fce400cb800 2023-10-07T11:03:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.125+0000 7fce3ffff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fce24066f30 0x7fce240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.126+0000 7fce3ffff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fce24066f30 0x7fce240693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fce400c9e70 tx=0x7fce2c007b40 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.129+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce340512e0 con 0x7fce400cb800 2023-10-07T11:03:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.414+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fce40068b30 con 0x7fce400cb800 2023-10-07T11:03:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.418+0000 7fce3cff9700 1 -- 172.21.15.29:0/839563196 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+0 (secure 0 0 0) 0x7fce34054900 con 0x7fce400cb800 2023-10-07T11:03:33.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fce24066f30 msgr2=0x7fce240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fce24066f30 0x7fce240693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fce400c9e70 tx=0x7fce2c007b40 comp rx=0 tx=0).stop 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 msgr2=0x7fce40153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 0x7fce40153bc0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c790 tx=0x7fce34007c80 comp rx=0 tx=0).stop 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 shutdown_connections 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce400d0010 0x7fce400c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400cb800 0x7fce40153bc0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fce24066f30 0x7fce240693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 --2- 172.21.15.29:0/839563196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce400cfbb0 0x7fce400c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 >> 172.21.15.29:0/839563196 conn(0x7fce40063a10 msgr2=0x7fce400cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 shutdown_connections 2023-10-07T11:03:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.420+0000 7fce471ff700 1 -- 172.21.15.29:0/839563196 wait complete. 2023-10-07T11:03:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-07T11:03:33.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:33.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.641+0000 7f27c0fff700 1 Processor -- start 2023-10-07T11:03:33.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.641+0000 7f27c0fff700 1 -- start start 2023-10-07T11:03:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27c0fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27bc15ade0 0x7f27bc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27c0fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f27bc0b0540 0x7f27bc15c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27c0fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc154090 con 0x7f27bc153cc0 2023-10-07T11:03:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27c0fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0b09e0 con 0x7f27bc15ade0 2023-10-07T11:03:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27c0fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc159f60 con 0x7f27bc0b0540 2023-10-07T11:03:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27b9d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27bc15ade0 0x7f27bc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27b9d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27bc15ade0 0x7f27bc0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49954/0 (socket says 172.21.15.29:49954) 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27b9d9b700 1 -- 172.21.15.29:0/3893989137 learned_addr learned my addr 172.21.15.29:0/3893989137 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27ba59c700 1 --1- 172.21.15.29:0/3893989137 >> v1:172.21.15.29:6789/0 conn(0x7f27bc153cc0 0x7f27bc0b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41396/0 (socket says 172.21.15.29:41396) 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27bad9d700 1 --2- 172.21.15.29:0/3893989137 >> v2:172.21.15.144:3300/0 conn(0x7f27bc0b0540 0x7f27bc15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27b9d9b700 1 -- 172.21.15.29:0/3893989137 >> v2:172.21.15.144:3300/0 conn(0x7f27bc0b0540 msgr2=0x7f27bc15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27b9d9b700 1 --2- 172.21.15.29:0/3893989137 >> v2:172.21.15.144:3300/0 conn(0x7f27bc0b0540 0x7f27bc15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.643+0000 7f27b9d9b700 1 -- 172.21.15.29:0/3893989137 >> v1:172.21.15.29:6789/0 conn(0x7f27bc153cc0 legacy=0x7f27bc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.644+0000 7f27b9d9b700 1 -- 172.21.15.29:0/3893989137 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc15a240 con 0x7f27bc15ade0 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.644+0000 7f27b959a700 1 -- 172.21.15.29:0/3893989137 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684571298 0 0) 0x7f27bc154090 con 0x7f27bc153cc0 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.644+0000 7f27b9d9b700 1 --2- 172.21.15.29:0/3893989137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc15ade0 0x7f27bc0b0000 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06c400 tx=0x7f27b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebde6c42ffec7ae server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.644+0000 7f27b959a700 1 -- 172.21.15.29:0/3893989137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b000b860 con 0x7f27bc15ade0 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.644+0000 7f27b959a700 1 -- 172.21.15.29:0/3893989137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b000b9c0 con 0x7f27bc15ade0 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.644+0000 7f27b959a700 1 -- 172.21.15.29:0/3893989137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b000bd60 con 0x7f27bc15ade0 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 -- 172.21.15.29:0/3893989137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc15ade0 msgr2=0x7f27bc0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 --2- 172.21.15.29:0/3893989137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc15ade0 0x7f27bc0b0000 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06c400 tx=0x7f27b0009250 comp rx=0 tx=0).stop 2023-10-07T11:03:33.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 -- 172.21.15.29:0/3893989137 shutdown_connections 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 --2- 172.21.15.29:0/3893989137 >> v2:172.21.15.144:3300/0 conn(0x7f27bc0b0540 0x7f27bc15c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 --2- 172.21.15.29:0/3893989137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc15ade0 0x7f27bc0b0000 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 -- 172.21.15.29:0/3893989137 >> 172.21.15.29:0/3893989137 conn(0x7f27bc067610 msgr2=0x7f27bc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 -- 172.21.15.29:0/3893989137 shutdown_connections 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.645+0000 7f27c0fff700 1 -- 172.21.15.29:0/3893989137 wait complete. 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.646+0000 7f27c0fff700 1 Processor -- start 2023-10-07T11:03:33.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.646+0000 7f27c0fff700 1 -- start start 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.647+0000 7f27c0fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27bc0b0540 0x7f27bc150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.647+0000 7f27c0fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 0x7f27bc154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.647+0000 7f27c0fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27bc15ade0 0x7f27bc0adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.647+0000 7f27c0fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc151040 con 0x7f27bc153cc0 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.647+0000 7f27c0fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc1558e0 con 0x7f27bc0b0540 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.647+0000 7f27c0fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc155bc0 con 0x7f27bc15ade0 2023-10-07T11:03:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 0x7f27bc154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 0x7f27bc154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49964/0 (socket says 172.21.15.29:49964) 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 -- 172.21.15.29:0/4146342454 learned_addr learned my addr 172.21.15.29:0/4146342454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27ba59c700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27bc0b0540 0x7f27bc150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27bad9d700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27bc15ade0 0x7f27bc0adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 -- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27bc15ade0 msgr2=0x7f27bc0adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27bc15ade0 0x7f27bc0adcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 -- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27bc0b0540 msgr2=0x7f27bc150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27bc0b0540 0x7f27bc150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 -- 172.21.15.29:0/4146342454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00094f0 con 0x7f27bc153cc0 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.648+0000 7f27b9d9b700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 0x7f27bc154f60 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009290 tx=0x7f27b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.649+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0011e40 con 0x7f27bc153cc0 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.649+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0026450 con 0x7f27bc153cc0 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.649+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc155ea0 con 0x7f27bc153cc0 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.649+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0026810 con 0x7f27bc153cc0 2023-10-07T11:03:33.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.649+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc156460 con 0x7f27bc153cc0 2023-10-07T11:03:33.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.650+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc0b40d0 con 0x7f27bc153cc0 2023-10-07T11:03:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.651+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f27b0026c40 con 0x7f27bc153cc0 2023-10-07T11:03:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.652+0000 7f27aa7fc700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f279c066f30 0x7f279c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.652+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f27b0086350 con 0x7f27bc153cc0 2023-10-07T11:03:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.653+0000 7f27ba59c700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f279c066f30 0x7f279c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.654+0000 7f27ba59c700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f279c066f30 0x7f279c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0012a0 tx=0x7f27ac009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.657+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27b0053510 con 0x7f27bc153cc0 2023-10-07T11:03:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.899+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27bc04c3c0 con 0x7f27bc153cc0 2023-10-07T11:03:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.900+0000 7f27aa7fc700 1 -- 172.21.15.29:0/4146342454 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4609 (secure 0 0 0) 0x7f27b0056b30 con 0x7f27bc153cc0 2023-10-07T11:03:33.900 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:33.901 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:03:22.532644+0000","modified":"2023-10-07T11:03:26.067213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10104},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10104":{"gid":10104,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/1070134873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1070134873},{"type":"v1","addr":"172.21.15.29:6834","nonce":1070134873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f279c066f30 msgr2=0x7f279c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f279c066f30 0x7f279c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0012a0 tx=0x7f27ac009040 comp rx=0 tx=0).stop 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 msgr2=0x7f27bc154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 0x7f27bc154f60 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009290 tx=0x7f27b0017000 comp rx=0 tx=0).stop 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 shutdown_connections 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f279c066f30 0x7f279c0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27bc15ade0 0x7f27bc0adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27bc0b0540 0x7f27bc150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 --2- 172.21.15.29:0/4146342454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27bc153cc0 0x7f27bc154f60 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:33.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 >> 172.21.15.29:0/4146342454 conn(0x7f27bc067610 msgr2=0x7f27bc15b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:33.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 shutdown_connections 2023-10-07T11:03:33.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:33.902+0000 7f27c0fff700 1 -- 172.21.15.29:0/4146342454 wait complete. 2023-10-07T11:03:33.904 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 101 2023-10-07T11:03:34.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-07T11:03:22.532644+0000', 'modified': '2023-10-07T11:03:26.067213+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10104}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10104': {'gid': 10104, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6834/1070134873', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 1070134873}, {'type': 'v1', 'addr': '172.21.15.29:6834', 'nonce': 1070134873}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-07T11:03:34.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-07T11:03:34.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-07T11:03:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.522+0000 7f8dfafff700 1 Processor -- start 2023-10-07T11:03:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.522+0000 7f8dfafff700 1 -- start start 2023-10-07T11:03:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8dfafff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8df4159750 0x7f8df40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8dfafff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8df4159b20 0x7f8df40ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8dfafff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df40ce6d0 0x7f8df40ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8dfafff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df40cbf20 con 0x7f8df4159b20 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8dfafff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df40cc200 con 0x7f8df4159750 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8dfafff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40cc4e0 con 0x7f8df40ce6d0 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8df4159b20 0x7f8df40ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8df4159b20 0x7f8df40ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49974/0 (socket says 172.21.15.29:49974) 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 -- 172.21.15.29:0/1819516703 learned_addr learned my addr 172.21.15.29:0/1819516703 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/1819516703 >> v2:172.21.15.144:3300/0 conn(0x7f8df4159750 0x7f8df40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df959c700 1 --2- 172.21.15.29:0/1819516703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df40ce6d0 0x7f8df40ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 -- 172.21.15.29:0/1819516703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df40ce6d0 msgr2=0x7f8df40ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 --2- 172.21.15.29:0/1819516703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df40ce6d0 0x7f8df40ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 -- 172.21.15.29:0/1819516703 >> v2:172.21.15.144:3300/0 conn(0x7f8df4159750 msgr2=0x7f8df40d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 --2- 172.21.15.29:0/1819516703 >> v2:172.21.15.144:3300/0 conn(0x7f8df4159750 0x7f8df40d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.524+0000 7f8df3fff700 1 -- 172.21.15.29:0/1819516703 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df40cc7c0 con 0x7f8df4159b20 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.525+0000 7f8df3fff700 1 --2- 172.21.15.29:0/1819516703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df4159b20 0x7f8df40ce190 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f8df4089840 tx=0x7f8de0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed5ab190054112a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.525+0000 7f8df37fe700 1 -- 172.21.15.29:0/1819516703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000b860 con 0x7f8df4159b20 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.525+0000 7f8df37fe700 1 -- 172.21.15.29:0/1819516703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de000b9c0 con 0x7f8df4159b20 2023-10-07T11:03:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.525+0000 7f8df37fe700 1 -- 172.21.15.29:0/1819516703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000bd60 con 0x7f8df4159b20 2023-10-07T11:03:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.525+0000 7f8dfafff700 1 -- 172.21.15.29:0/1819516703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df4159b20 msgr2=0x7f8df40ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.525+0000 7f8dfafff700 1 --2- 172.21.15.29:0/1819516703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df4159b20 0x7f8df40ce190 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f8df4089840 tx=0x7f8de0009250 comp rx=0 tx=0).stop 2023-10-07T11:03:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 -- 172.21.15.29:0/1819516703 shutdown_connections 2023-10-07T11:03:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 --2- 172.21.15.29:0/1819516703 >> v2:172.21.15.144:3300/0 conn(0x7f8df4159750 0x7f8df40d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 --2- 172.21.15.29:0/1819516703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df40ce6d0 0x7f8df40ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 --2- 172.21.15.29:0/1819516703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df4159b20 0x7f8df40ce190 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 -- 172.21.15.29:0/1819516703 >> 172.21.15.29:0/1819516703 conn(0x7f8df4063a10 msgr2=0x7f8df40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 -- 172.21.15.29:0/1819516703 shutdown_connections 2023-10-07T11:03:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.526+0000 7f8dfafff700 1 -- 172.21.15.29:0/1819516703 wait complete. 2023-10-07T11:03:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.527+0000 7f8dfafff700 1 Processor -- start 2023-10-07T11:03:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.527+0000 7f8dfafff700 1 -- start start 2023-10-07T11:03:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.528+0000 7f8dfafff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 0x7f8df40b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.528+0000 7f8dfafff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df4159750 0x7f8df40b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.528+0000 7f8dfafff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8df4159b20 0x7f8df40c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.528+0000 7f8dfafff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df4152980 con 0x7f8df40ce6d0 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.528+0000 7f8dfafff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40b5f40 con 0x7f8df4159b20 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.528+0000 7f8dfafff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40b3f70 con 0x7f8df4159750 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 0x7f8df40b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 0x7f8df40b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49990/0 (socket says 172.21.15.29:49990) 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 -- 172.21.15.29:0/2613646342 learned_addr learned my addr 172.21.15.29:0/2613646342 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df3fff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df4159750 0x7f8df40b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df959c700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8df4159b20 0x7f8df40c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 -- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df4159750 msgr2=0x7f8df40b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df4159750 0x7f8df40b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 -- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8df4159b20 msgr2=0x7f8df40c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8df4159b20 0x7f8df40c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 -- 172.21.15.29:0/2613646342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00094f0 con 0x7f8df40ce6d0 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df959c700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8df4159b20 0x7f8df40c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 0x7f8df40b5480 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f8de800d840 tx=0x7f8de8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.529+0000 7f8df0ff9700 1 -- 172.21.15.29:0/2613646342 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80125f0 con 0x7f8df40ce6d0 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.530+0000 7f8df0ff9700 1 -- 172.21.15.29:0/2613646342 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8012750 con 0x7f8df40ce6d0 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.530+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df40b59c0 con 0x7f8df40ce6d0 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.530+0000 7f8df0ff9700 1 -- 172.21.15.29:0/2613646342 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8012b10 con 0x7f8df40ce6d0 2023-10-07T11:03:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.530+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df40b5c60 con 0x7f8df40ce6d0 2023-10-07T11:03:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.533+0000 7f8df0ff9700 1 -- 172.21.15.29:0/2613646342 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8de801e460 con 0x7f8df40ce6d0 2023-10-07T11:03:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.534+0000 7f8df0ff9700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f8dd8066f30 0x7f8dd80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.534+0000 7f8df0ff9700 1 -- 172.21.15.29:0/2613646342 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f8de8084da0 con 0x7f8df40ce6d0 2023-10-07T11:03:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.534+0000 7f8df3fff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f8dd8066f30 0x7f8dd80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.535+0000 7f8df3fff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f8dd8066f30 0x7f8dd80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8df4089840 tx=0x7f8de0017000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.535+0000 7f8e09810b80 1 Processor -- start 2023-10-07T11:03:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.535+0000 7f8e09810b80 1 -- start start 2023-10-07T11:03:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c45ba40 0x55e80c45be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 0x55e80c464680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c464bc0 0x55e80c465010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8e09810b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c32e310 con 0x55e80c441530 2023-10-07T11:03:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c36b130 con 0x55e80c45ba40 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e80c368af0 con 0x55e80c464bc0 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.536+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 0x55e80c464680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 0x55e80c464680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50004/0 (socket says 172.21.15.29:50004) 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 -- 172.21.15.29:0/2829784109 learned_addr learned my addr 172.21.15.29:0/2829784109 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c45ba40 0x55e80c45be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df959c700 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c464bc0 0x55e80c465010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 -- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c464bc0 msgr2=0x55e80c465010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c464bc0 0x55e80c465010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 -- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c45ba40 msgr2=0x55e80c45be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c45ba40 0x55e80c45be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 -- 172.21.15.29:0/2829784109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e80c2570a0 con 0x55e80c441530 2023-10-07T11:03:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c45ba40 0x55e80c45be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8df3fff700 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 0x55e80c464680 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006fd0 tx=0x7f8de0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87f3aa1afcb4764 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.537+0000 7f8dddffb700 1 -- 172.21.15.29:0/2829784109 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0021070 con 0x55e80c441530 2023-10-07T11:03:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8dddffb700 1 -- 172.21.15.29:0/2829784109 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de0011a90 con 0x55e80c441530 2023-10-07T11:03:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8dddffb700 1 -- 172.21.15.29:0/2829784109 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0011c40 con 0x55e80c441530 2023-10-07T11:03:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 -- 172.21.15.29:0/2829784109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 msgr2=0x55e80c464680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 0x55e80c464680 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006fd0 tx=0x7f8de0021000 comp rx=0 tx=0).stop 2023-10-07T11:03:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 -- 172.21.15.29:0/2829784109 shutdown_connections 2023-10-07T11:03:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c464bc0 0x55e80c465010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c45ba40 0x55e80c45be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 --2- 172.21.15.29:0/2829784109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c441530 0x55e80c464680 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 -- 172.21.15.29:0/2829784109 >> 172.21.15.29:0/2829784109 conn(0x55e80c45ee70 msgr2=0x55e80c45f240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 -- 172.21.15.29:0/2829784109 shutdown_connections 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.538+0000 7f8e09810b80 1 -- 172.21.15.29:0/2829784109 wait complete. 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.539+0000 7f8e09810b80 1 -- start start 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.539+0000 7f8e09810b80 1 Processor -- start 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.540+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c43bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.540+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c45b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.540+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c463550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.540+0000 7f8e09810b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c3648e0 con 0x55e80c45ba40 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.540+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c310980 con 0x55e80c464bc0 2023-10-07T11:03:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.540+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e80c310620 con 0x55e80c441530 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8e09810b80 10 client.0 _ll_register_callbacks cb 0x55e80c173a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c45b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c45b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50012/0 (socket says 172.21.15.29:50012) 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 -- 172.21.15.29:0/4058095757 learned_addr learned my addr 172.21.15.29:0/4058095757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c43bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df959c700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c463550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 -- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 msgr2=0x55e80c43bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c43bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 -- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 msgr2=0x55e80c463550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c463550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 -- 172.21.15.29:0/4058095757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df40cc7c0 con 0x55e80c45ba40 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df959c700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c463550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c43bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8df3fff700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c45b180 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8de0011f60 tx=0x7f8de002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba7f09d8eaddcb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.541+0000 7f8dd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/4058095757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0021070 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/4058095757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f8de001b970 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e80c409560 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/4058095757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8de001bb30 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 10 client.10287 fetch_fsmap learned FSMap version 101 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55e80c3333c0 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/4058095757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e80c409560 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/4058095757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 101) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8de001bc90 con 0x55e80c45ba40 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 10 client.10287 fetch_fsmap finished waiting for FSMap version 101 2023-10-07T11:03:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 10 client.10287 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T11:03:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 20 client.10287 populate_metadata read hostname 'smithi029' 2023-10-07T11:03:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c43f9e0 0x55e80c442aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 4 client.10287 mds_command: new command op to 10104 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-07T11:03:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.542+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 --> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e80c3e60e0 con 0x55e80c43f9e0 2023-10-07T11:03:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.543+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c43f9e0 0x55e80c442aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.543+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c43f9e0 0x55e80c442aa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=840a93b794a7ec87 server_cookie=c382fff90cddc203 in_seq=0 out_seq=0 2023-10-07T11:03:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.543+0000 7f8dd2ffd700 10 client.10287 ms_handle_connect on v2:172.21.15.29:6832/1070134873 2023-10-07T11:03:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/4058095757 <== mds.0 v2:172.21.15.29:6832/1070134873 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55e80c3e60e0 con 0x55e80c43f9e0 2023-10-07T11:03:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8dd2ffd700 10 client.10287 handle_command_reply: tid=0 2023-10-07T11:03:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 client.10287 shutdown 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 msgr2=0x55e80c45b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c45b180 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8de0011f60 tx=0x7f8de002c040 comp rx=0 tx=0).stop 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 shutdown_connections 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c43f9e0 0x55e80c442aa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c463550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c43bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4058095757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c45b180 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8dd2ffd700 0 client.10287 ms_handle_reset on v2:172.21.15.29:6832/1070134873 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 >> 172.21.15.29:0/4058095757 conn(0x55e80c45ee70 msgr2=0x55e80c411250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 shutdown_connections 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.544+0000 7f8e09810b80 1 -- 172.21.15.29:0/4058095757 wait complete. 2023-10-07T11:03:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.545+0000 7f8e09810b80 20 client.10287 trim_cache size 0 max 16384 2023-10-07T11:03:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.562+0000 7f8e09810b80 1 Processor -- start 2023-10-07T11:03:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.562+0000 7f8e09810b80 1 -- start start 2023-10-07T11:03:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c441530 0x55e80c412980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c45ba40 0x55e80c444790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 0x55e80c459940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8e09810b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c1fc1e0 con 0x55e80c464bc0 2023-10-07T11:03:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c441070 con 0x55e80c441530 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e80c43e330 con 0x55e80c45ba40 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8df959c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 0x55e80c459940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8df959c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 0x55e80c459940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50028/0 (socket says 172.21.15.29:50028) 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.563+0000 7f8df959c700 1 -- 172.21.15.29:0/4040629883 learned_addr learned my addr 172.21.15.29:0/4040629883 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c441530 0x55e80c412980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df3fff700 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c45ba40 0x55e80c444790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df959c700 1 -- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c45ba40 msgr2=0x55e80c444790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df959c700 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c45ba40 0x55e80c444790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df959c700 1 -- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c441530 msgr2=0x55e80c412980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df959c700 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c441530 0x55e80c412980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df959c700 1 -- 172.21.15.29:0/4040629883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e80c1cb1d0 con 0x55e80c464bc0 2023-10-07T11:03:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df3fff700 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c45ba40 0x55e80c444790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8df959c700 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 0x55e80c459940 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f8dec007480 tx=0x7f8dec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7381f474e3dbc5c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8dde7fc700 1 -- 172.21.15.29:0/4040629883 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dec017430 con 0x55e80c464bc0 2023-10-07T11:03:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8dde7fc700 1 -- 172.21.15.29:0/4040629883 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dec00f040 con 0x55e80c464bc0 2023-10-07T11:03:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.564+0000 7f8dde7fc700 1 -- 172.21.15.29:0/4040629883 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dec017800 con 0x55e80c464bc0 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 -- 172.21.15.29:0/4040629883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 msgr2=0x55e80c459940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 0x55e80c459940 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f8dec007480 tx=0x7f8dec00b040 comp rx=0 tx=0).stop 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 -- 172.21.15.29:0/4040629883 shutdown_connections 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c464bc0 0x55e80c459940 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c441530 0x55e80c412980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 --2- 172.21.15.29:0/4040629883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c45ba40 0x55e80c444790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 -- 172.21.15.29:0/4040629883 >> 172.21.15.29:0/4040629883 conn(0x55e80c45ee70 msgr2=0x55e80c411d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 -- 172.21.15.29:0/4040629883 shutdown_connections 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.565+0000 7f8e09810b80 1 -- 172.21.15.29:0/4040629883 wait complete. 2023-10-07T11:03:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.566+0000 7f8e09810b80 1 -- start start 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.566+0000 7f8e09810b80 1 Processor -- start 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c4596a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c4c0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c4c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c4169a0 con 0x55e80c45ba40 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e80c440120 con 0x55e80c464bc0 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e80c43fed0 con 0x55e80c441530 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8e09810b80 10 client.0 _ll_register_callbacks cb 0x55e80c173a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c4c0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8df3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c4c0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50030/0 (socket says 172.21.15.29:50030) 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8df3fff700 1 -- 172.21.15.29:0/747922267 learned_addr learned my addr 172.21.15.29:0/747922267 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8df959c700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c4c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.567+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c4596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df3fff700 1 -- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 msgr2=0x55e80c4596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df3fff700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c4596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df3fff700 1 -- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 msgr2=0x55e80c4c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df3fff700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c4c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df3fff700 1 -- 172.21.15.29:0/747922267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dec009dc0 con 0x55e80c45ba40 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df959c700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c4c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8df3fff700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c4c0790 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f8de0011f60 tx=0x7f8de0035070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196ebd8404fdafe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8dd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/747922267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0021040 con 0x55e80c45ba40 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/747922267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f8de0033cb0 con 0x55e80c45ba40 2023-10-07T11:03:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8df415b4d0 con 0x55e80c45ba40 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/747922267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8de0033e70 con 0x55e80c45ba40 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.568+0000 7f8e09810b80 10 client.10293 fetch_fsmap learned FSMap version 101 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55e80c1a4d60 con 0x55e80c45ba40 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/747922267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df415b4d0 con 0x55e80c45ba40 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/747922267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 101) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8de001b970 con 0x55e80c45ba40 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 10 client.10293 fetch_fsmap finished waiting for FSMap version 101 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 10 client.10293 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 20 client.10293 populate_metadata read hostname 'smithi029' 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c4c1210 0x55e80c45ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 4 client.10293 mds_command: new command op to 10104 tid=0[{"prefix": "status"}] 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 --> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f8de0011a90 con 0x55e80c4c1210 2023-10-07T11:03:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.569+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c4c1210 0x55e80c45ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8df8d9b700 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c4c1210 0x55e80c45ffd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ab45b65232075a server_cookie=8d4c196949026fd4 in_seq=0 out_seq=0 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8dd2ffd700 10 client.10293 ms_handle_connect on v2:172.21.15.29:6832/1070134873 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8dd2ffd700 1 -- 172.21.15.29:0/747922267 <== mds.0 v2:172.21.15.29:6832/1070134873 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x7f8de0011a90 con 0x55e80c4c1210 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8dd2ffd700 10 client.10293 handle_command_reply: tid=0 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 client.10293 shutdown 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 msgr2=0x55e80c4c0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c4c0790 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f8de0011f60 tx=0x7f8de0035070 comp rx=0 tx=0).stop 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 shutdown_connections 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:6832/1070134873,v1:172.21.15.29:6834/1070134873] conn(0x55e80c4c1210 0x55e80c45ffd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e80c464bc0 0x55e80c4c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e80c441530 0x55e80c4596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 --2- 172.21.15.29:0/747922267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80c45ba40 0x55e80c4c0790 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8dd2ffd700 0 client.10293 ms_handle_reset on v2:172.21.15.29:6832/1070134873 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 >> 172.21.15.29:0/747922267 conn(0x55e80c45ee70 msgr2=0x55e80c463d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 shutdown_connections 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.570+0000 7f8e09810b80 1 -- 172.21.15.29:0/747922267 wait complete. 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8e09810b80 20 client.10293 trim_cache size 0 max 16384 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "1e617e23-0c6c-40cc-8e09-10eaf7e38e24", 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 10104, 2023-10-07T11:03:34.578 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 11.510119083999999, 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 101, 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 197, 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 197, 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 17.184270658999999 2023-10-07T11:03:34.579 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f8dd8066f30 msgr2=0x7f8dd80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f8dd8066f30 0x7f8dd80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8df4089840 tx=0x7f8de0017000 comp rx=0 tx=0).stop 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 msgr2=0x7f8df40b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 0x7f8df40b5480 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f8de800d840 tx=0x7f8de8008040 comp rx=0 tx=0).stop 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 shutdown_connections 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8df4159b20 0x7f8df40c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df40ce6d0 0x7f8df40b5480 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f8dd8066f30 0x7f8dd80693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 --2- 172.21.15.29:0/2613646342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8df4159750 0x7f8df40b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 >> 172.21.15.29:0/2613646342 conn(0x7f8df4063a10 msgr2=0x7f8df40ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 shutdown_connections 2023-10-07T11:03:34.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.571+0000 7f8dfafff700 1 -- 172.21.15.29:0/2613646342 wait complete. 2023-10-07T11:03:34.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-07T11:03:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.785+0000 7f3986fff700 1 Processor -- start 2023-10-07T11:03:34.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.785+0000 7f3986fff700 1 -- start start 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3986fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3980159b20 0x7f39800d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3986fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39800ce8a0 0x7f39800c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3986fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800cbf20 con 0x7f3980159750 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3986fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800cc200 con 0x7f39800ce8a0 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3986fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800cc4e0 con 0x7f3980159b20 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f398559c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39800ce8a0 0x7f39800c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3984d9b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3980159750 0x7f39800ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41400/0 (socket says 172.21.15.29:41400) 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.786+0000 7f3984d9b700 1 -- 172.21.15.29:0/2820730437 learned_addr learned my addr 172.21.15.29:0/2820730437 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f397ffff700 1 --2- 172.21.15.29:0/2820730437 >> v2:172.21.15.144:3300/0 conn(0x7f3980159b20 0x7f39800d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f398559c700 1 -- 172.21.15.29:0/2820730437 >> v2:172.21.15.144:3300/0 conn(0x7f3980159b20 msgr2=0x7f39800d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f397f7fe700 1 -- 172.21.15.29:0/2820730437 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27259911 0 0) 0x7f39800cbf20 con 0x7f3980159750 2023-10-07T11:03:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f398559c700 1 --2- 172.21.15.29:0/2820730437 >> v2:172.21.15.144:3300/0 conn(0x7f3980159b20 0x7f39800d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f398559c700 1 -- 172.21.15.29:0/2820730437 >> v1:172.21.15.29:6789/0 conn(0x7f3980159750 legacy=0x7f39800ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f398559c700 1 -- 172.21.15.29:0/2820730437 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800cc7c0 con 0x7f39800ce8a0 2023-10-07T11:03:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f398559c700 1 --2- 172.21.15.29:0/2820730437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39800ce8a0 0x7f39800c5460 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f3980088a70 tx=0x7f3978009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990e9b48b7b1a6a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f397f7fe700 1 -- 172.21.15.29:0/2820730437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397800e070 con 0x7f39800ce8a0 2023-10-07T11:03:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f397f7fe700 1 -- 172.21.15.29:0/2820730437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397800bc30 con 0x7f39800ce8a0 2023-10-07T11:03:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.787+0000 7f397f7fe700 1 -- 172.21.15.29:0/2820730437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978012430 con 0x7f39800ce8a0 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 -- 172.21.15.29:0/2820730437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39800ce8a0 msgr2=0x7f39800c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 --2- 172.21.15.29:0/2820730437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39800ce8a0 0x7f39800c5460 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f3980088a70 tx=0x7f3978009780 comp rx=0 tx=0).stop 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 -- 172.21.15.29:0/2820730437 shutdown_connections 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 --2- 172.21.15.29:0/2820730437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39800ce8a0 0x7f39800c5460 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 --2- 172.21.15.29:0/2820730437 >> v2:172.21.15.144:3300/0 conn(0x7f3980159b20 0x7f39800d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 -- 172.21.15.29:0/2820730437 >> 172.21.15.29:0/2820730437 conn(0x7f3980063a10 msgr2=0x7f39800d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 -- 172.21.15.29:0/2820730437 shutdown_connections 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.788+0000 7f3986fff700 1 -- 172.21.15.29:0/2820730437 wait complete. 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.790+0000 7f3986fff700 1 Processor -- start 2023-10-07T11:03:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.790+0000 7f3986fff700 1 -- start start 2023-10-07T11:03:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3986fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39800ce8a0 0x7f39800c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3986fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3980159750 0x7f398015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3986fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 0x7f3980142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3986fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800c82b0 con 0x7f3980159b20 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3986fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800c8590 con 0x7f3980159750 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3986fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800c8870 con 0x7f39800ce8a0 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 0x7f3980142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 0x7f3980142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50050/0 (socket says 172.21.15.29:50050) 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 -- 172.21.15.29:0/1529117647 learned_addr learned my addr 172.21.15.29:0/1529117647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f3984d9b700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39800ce8a0 0x7f39800c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f397ffff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3980159750 0x7f398015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 -- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39800ce8a0 msgr2=0x7f39800c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39800ce8a0 0x7f39800c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 -- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3980159750 msgr2=0x7f398015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3980159750 0x7f398015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f398559c700 1 -- 172.21.15.29:0/1529117647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978009a20 con 0x7f3980159b20 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.791+0000 7f397ffff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3980159750 0x7f398015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.792+0000 7f398559c700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 0x7f3980142ee0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f3980088a40 tx=0x7f3978019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.792+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978012ea0 con 0x7f3980159b20 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.792+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397801d630 con 0x7f3980159b20 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.792+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397801d9f0 con 0x7f3980159b20 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.792+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39800c8b50 con 0x7f3980159b20 2023-10-07T11:03:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.792+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398015c760 con 0x7f3980159b20 2023-10-07T11:03:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.793+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39800c9070 con 0x7f3980159b20 2023-10-07T11:03:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.794+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3978024070 con 0x7f3980159b20 2023-10-07T11:03:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.795+0000 7f397cff9700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3964066f60 0x7f3964069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.795+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f3978086ff0 con 0x7f3980159b20 2023-10-07T11:03:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.795+0000 7f3984d9b700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3964066f60 0x7f3964069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.796+0000 7f3984d9b700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3964066f60 0x7f3964069410 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3974001360 tx=0x7f3974009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:34.799+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39800c9070 con 0x7f3980159b20 2023-10-07T11:03:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.045+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f39800c9070 con 0x7f3980159b20 2023-10-07T11:03:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.046+0000 7f397cff9700 1 -- 172.21.15.29:0/1529117647 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v101) v1 ==== 53+0+73 (secure 0 0 0) 0x7f39800c9070 con 0x7f3980159b20 2023-10-07T11:03:35.047 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-07T11:03:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3964066f60 msgr2=0x7f3964069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3964066f60 0x7f3964069410 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3974001360 tx=0x7f3974009040 comp rx=0 tx=0).stop 2023-10-07T11:03:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 msgr2=0x7f3980142ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 0x7f3980142ee0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f3980088a40 tx=0x7f3978019000 comp rx=0 tx=0).stop 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 shutdown_connections 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3980159b20 0x7f3980142ee0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f3964066f60 0x7f3964069410 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39800ce8a0 0x7f39800c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 --2- 172.21.15.29:0/1529117647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3980159750 0x7f398015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 >> 172.21.15.29:0/1529117647 conn(0x7f3980063a10 msgr2=0x7f3980154460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 shutdown_connections 2023-10-07T11:03:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:35.048+0000 7f3986fff700 1 -- 172.21.15.29:0/1529117647 wait complete. 2023-10-07T11:03:35.066 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-07T11:03:35.067 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T11:03:35.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.068 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-07T11:03:35.116 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-07T11:03:35.116 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-07T11:03:35.116 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-07T11:03:35.116 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.117 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-07T11:03:35.117 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.117 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T11:03:35.117 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-07T11:03:35.117 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-07T11:03:35.117 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T11:03:35.118 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-07T11:03:35.118 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-07T11:03:35.118 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-07T11:03:35.118 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-07T11:03:35.118 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-07T11:03:35.118 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36614sec preferred_lft 36614sec 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-07T11:03:35.119 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::cc5d:14ff:fee6:fb8d/64 scope link 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T11:03:35.120 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:fe:82:14:4f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T11:03:35.121 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcfe:82ff:fe14:4fc2/64 scope link 2023-10-07T11:03:35.121 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.121 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-07T11:03:35.121 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether fe:a7:08:76:84:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T11:03:35.121 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fca7:8ff:fe76:84bb/64 scope link 2023-10-07T11:03:35.121 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-07T11:03:35.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.122 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-07T11:03:35.191 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-07T11:03:35.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.194 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-07T11:03:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-07T11:03:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-07T11:03:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-07T11:03:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-07T11:03:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-07T11:03:35.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.244 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-07T11:03:35.244 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-07T11:03:35.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-07T11:03:35.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-07T11:03:35.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-07T11:03:35.244 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-07T11:03:35.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.416 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T11:03:35.463 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T11:03:35.463 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T11:03:35.463 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-07T11:03:35.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-07T11:03:35.575 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T11:03:35.576 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-07T11:03:35.626 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-07T11:03:35.627 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-07T11:03:35.627 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-07T11:03:35.712 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-07T11:03:35.712 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-07T11:03:35.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:03:35.853 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:03:35.853 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T11:03:35.853 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T11:03:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:03:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T11:03:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T11:03:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T11:03:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T11:03:35.888 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-07T11:03:35.888 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-07T11:03:35.888 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-07T11:03:35.889 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-07T11:03:35.889 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-07T11:03:35.889 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790" 2023-10-07T11:03:35.889 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.144:6789,172.21.15.144:6790:/ 2023-10-07T11:03:35.889 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1e617e23-0c6c-40cc-8e09-10eaf7e38e24" will pass to kernel 2023-10-07T11:03:35.894 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-07T11:03:35.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:35.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-07T11:03:35.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-07T11:03:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.161+0000 7fb16c5ff700 1 Processor -- start 2023-10-07T11:03:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.161+0000 7fb16c5ff700 1 -- start start 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb164153cc0 0x7fb1640b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16c5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb1640b0710 0x7fb1640b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16c5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb164154090 con 0x7fb16415ade0 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16c5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb164159f60 con 0x7fb1640b0710 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415a240 con 0x7fb164153cc0 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb169b9a700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb16415ade0 0x7fb1640b0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41408/0 (socket says 172.21.15.29:41408) 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb169b9a700 1 -- 172.21.15.29:0/3722779958 learned_addr learned my addr 172.21.15.29:0/3722779958 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16a39b700 1 --2- 172.21.15.29:0/3722779958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb164153cc0 0x7fb1640b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.163+0000 7fb16ab9c700 1 --2- 172.21.15.29:0/3722779958 >> v2:172.21.15.144:3300/0 conn(0x7fb1640b0710 0x7fb1640b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb169399700 1 -- 172.21.15.29:0/3722779958 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860221300 0 0) 0x7fb164154090 con 0x7fb16415ade0 2023-10-07T11:03:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb169399700 1 -- 172.21.15.29:0/3722779958 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb150003570 con 0x7fb16415ade0 2023-10-07T11:03:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb16a39b700 1 -- 172.21.15.29:0/3722779958 >> v2:172.21.15.144:3300/0 conn(0x7fb1640b0710 msgr2=0x7fb1640b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb16a39b700 1 --2- 172.21.15.29:0/3722779958 >> v2:172.21.15.144:3300/0 conn(0x7fb1640b0710 0x7fb1640b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb16a39b700 1 -- 172.21.15.29:0/3722779958 >> v1:172.21.15.29:6789/0 conn(0x7fb16415ade0 legacy=0x7fb1640b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb16a39b700 1 -- 172.21.15.29:0/3722779958 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16415a520 con 0x7fb164153cc0 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb16ab9c700 1 --2- 172.21.15.29:0/3722779958 >> v2:172.21.15.144:3300/0 conn(0x7fb1640b0710 0x7fb1640b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.164+0000 7fb16a39b700 1 --2- 172.21.15.29:0/3722779958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb164153cc0 0x7fb1640b17c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb1600098d0 tx=0x7fb160009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=759c825215ef6296 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb169399700 1 -- 172.21.15.29:0/3722779958 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160005c10 con 0x7fb164153cc0 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb169399700 1 -- 172.21.15.29:0/3722779958 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb160005d70 con 0x7fb164153cc0 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb169399700 1 -- 172.21.15.29:0/3722779958 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160006260 con 0x7fb164153cc0 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 -- 172.21.15.29:0/3722779958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb164153cc0 msgr2=0x7fb1640b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/3722779958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb164153cc0 0x7fb1640b17c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb1600098d0 tx=0x7fb160009120 comp rx=0 tx=0).stop 2023-10-07T11:03:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 -- 172.21.15.29:0/3722779958 shutdown_connections 2023-10-07T11:03:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/3722779958 >> v2:172.21.15.144:3300/0 conn(0x7fb1640b0710 0x7fb1640b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/3722779958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb164153cc0 0x7fb1640b17c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 -- 172.21.15.29:0/3722779958 >> 172.21.15.29:0/3722779958 conn(0x7fb164067610 msgr2=0x7fb164067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.165+0000 7fb16c5ff700 1 -- 172.21.15.29:0/3722779958 shutdown_connections 2023-10-07T11:03:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.166+0000 7fb16c5ff700 1 -- 172.21.15.29:0/3722779958 wait complete. 2023-10-07T11:03:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.167+0000 7fb16c5ff700 1 Processor -- start 2023-10-07T11:03:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.167+0000 7fb16c5ff700 1 -- start start 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16c5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 0x7fb164157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16c5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb164153cc0 0x7fb1640c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16415ade0 0x7fb164145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16c5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164152410 con 0x7fb1640b0710 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16c5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164155af0 con 0x7fb164153cc0 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16416d3a0 con 0x7fb16415ade0 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16a39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 0x7fb164157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16a39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 0x7fb164157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50056/0 (socket says 172.21.15.29:50056) 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb16a39b700 1 -- 172.21.15.29:0/101635593 learned_addr learned my addr 172.21.15.29:0/101635593 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.168+0000 7fb169b9a700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb164153cc0 0x7fb1640c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16ab9c700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16415ade0 0x7fb164145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16a39b700 1 -- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16415ade0 msgr2=0x7fb164145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16a39b700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16415ade0 0x7fb164145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16a39b700 1 -- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb164153cc0 msgr2=0x7fb1640c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16a39b700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb164153cc0 0x7fb1640c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16a39b700 1 -- 172.21.15.29:0/101635593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c009a20 con 0x7fb1640b0710 2023-10-07T11:03:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb16a39b700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 0x7fb164157570 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fb160009ec0 tx=0x7fb160016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.169+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16001d910 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.170+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16001da70 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.170+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16001de30 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.170+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1600093c0 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.170+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16416d960 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.171+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16404c3c0 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.172+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb16001b690 con 0x7fb1640b0710 2023-10-07T11:03:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.173+0000 7fb15a7fc700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb148066f30 0x7fb1480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.173+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fb160086610 con 0x7fb1640b0710 2023-10-07T11:03:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.173+0000 7fb169b9a700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb148066f30 0x7fb1480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.174+0000 7fb169b9a700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb148066f30 0x7fb1480693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1640ca2f0 tx=0x7fb154009040 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.177+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1600536c0 con 0x7fb1640b0710 2023-10-07T11:03:36.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.404+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1640689d0 con 0x7fb148066f30 2023-10-07T11:03:36.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.407+0000 7fb15a7fc700 1 -- 172.21.15.29:0/101635593 <== mgr.8429 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb1640689d0 con 0x7fb148066f30 2023-10-07T11:03:36.407 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-07T11:03:36.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-07T11:03:36.407 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-07T11:03:36.407 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-07T11:03:36.407 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-07T11:03:36.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb148066f30 msgr2=0x7fb1480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb148066f30 0x7fb1480693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1640ca2f0 tx=0x7fb154009040 comp rx=0 tx=0).stop 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 msgr2=0x7fb164157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 0x7fb164157570 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fb160009ec0 tx=0x7fb160016000 comp rx=0 tx=0).stop 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 shutdown_connections 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fb148066f30 0x7fb1480693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16415ade0 0x7fb164145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640b0710 0x7fb164157570 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 --2- 172.21.15.29:0/101635593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb164153cc0 0x7fb1640c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.408+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 >> 172.21.15.29:0/101635593 conn(0x7fb164067610 msgr2=0x7fb1640af340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.409+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 shutdown_connections 2023-10-07T11:03:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.409+0000 7fb16c5ff700 1 -- 172.21.15.29:0/101635593 wait complete. 2023-10-07T11:03:36.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-07T11:03:36.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.624+0000 7f7fc89ff700 1 Processor -- start 2023-10-07T11:03:36.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.624+0000 7f7fc89ff700 1 -- start start 2023-10-07T11:03:36.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.625+0000 7f7fc89ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fc415a960 0x7f7fc4158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.625+0000 7f7fc89ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7fc415ad30 0x7f7fc40ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.625+0000 7f7fc89ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40acb40 con 0x7f7fc40ae6d0 2023-10-07T11:03:36.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.625+0000 7f7fc89ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc4158c60 con 0x7f7fc415a960 2023-10-07T11:03:36.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.625+0000 7f7fc89ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc40abf30 con 0x7f7fc415ad30 2023-10-07T11:03:36.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fc415a960 0x7f7fc4158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fc415a960 0x7f7fc4158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50066/0 (socket says 172.21.15.29:50066) 2023-10-07T11:03:36.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 -- 172.21.15.29:0/3435593500 learned_addr learned my addr 172.21.15.29:0/3435593500 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:36.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc1d9b700 1 --2- 172.21.15.29:0/3435593500 >> v2:172.21.15.144:3300/0 conn(0x7f7fc415ad30 0x7f7fc40ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 -- 172.21.15.29:0/3435593500 >> v2:172.21.15.144:3300/0 conn(0x7f7fc415ad30 msgr2=0x7f7fc40ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:36.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 --2- 172.21.15.29:0/3435593500 >> v2:172.21.15.144:3300/0 conn(0x7f7fc415ad30 0x7f7fc40ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 -- 172.21.15.29:0/3435593500 >> v1:172.21.15.144:6789/0 conn(0x7f7fc40ae6d0 legacy=0x7f7fc40b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:36.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 -- 172.21.15.29:0/3435593500 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc40ac210 con 0x7f7fc415a960 2023-10-07T11:03:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc1d9b700 1 --2- 172.21.15.29:0/3435593500 >> v2:172.21.15.144:3300/0 conn(0x7f7fc415ad30 0x7f7fc40ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.626+0000 7f7fc259c700 1 --2- 172.21.15.29:0/3435593500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415a960 0x7f7fc4158e20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f7fc406b910 tx=0x7f7fb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2e22f0b73a39db server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc159a700 1 -- 172.21.15.29:0/3435593500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400b820 con 0x7f7fc415a960 2023-10-07T11:03:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc159a700 1 -- 172.21.15.29:0/3435593500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb400b980 con 0x7f7fc415a960 2023-10-07T11:03:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc159a700 1 -- 172.21.15.29:0/3435593500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400bd20 con 0x7f7fc415a960 2023-10-07T11:03:36.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 -- 172.21.15.29:0/3435593500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415a960 msgr2=0x7f7fc4158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/3435593500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415a960 0x7f7fc4158e20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f7fc406b910 tx=0x7f7fb4009250 comp rx=0 tx=0).stop 2023-10-07T11:03:36.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 -- 172.21.15.29:0/3435593500 shutdown_connections 2023-10-07T11:03:36.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/3435593500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415a960 0x7f7fc4158e20 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/3435593500 >> v2:172.21.15.144:3300/0 conn(0x7f7fc415ad30 0x7f7fc40ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 -- 172.21.15.29:0/3435593500 >> 172.21.15.29:0/3435593500 conn(0x7f7fc4067610 msgr2=0x7f7fc4067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:36.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.627+0000 7f7fc89ff700 1 -- 172.21.15.29:0/3435593500 shutdown_connections 2023-10-07T11:03:36.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.628+0000 7f7fc89ff700 1 -- 172.21.15.29:0/3435593500 wait complete. 2023-10-07T11:03:36.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.629+0000 7f7fc89ff700 1 Processor -- start 2023-10-07T11:03:36.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.629+0000 7f7fc89ff700 1 -- start start 2023-10-07T11:03:36.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc89ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fc40ae6d0 0x7f7fc40cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc89ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fc415a960 0x7f7fc4150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc89ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 0x7f7fc40aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc89ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40ab360 con 0x7f7fc415ad30 2023-10-07T11:03:36.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc89ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc41505a0 con 0x7f7fc415a960 2023-10-07T11:03:36.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc89ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40ab150 con 0x7f7fc40ae6d0 2023-10-07T11:03:36.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 0x7f7fc40aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 0x7f7fc40aac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50082/0 (socket says 172.21.15.29:50082) 2023-10-07T11:03:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc2d9d700 1 -- 172.21.15.29:0/1976087346 learned_addr learned my addr 172.21.15.29:0/1976087346 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.630+0000 7f7fc259c700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fc40ae6d0 0x7f7fc40cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc1d9b700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fc415a960 0x7f7fc4150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc2d9d700 1 -- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fc40ae6d0 msgr2=0x7f7fc40cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc2d9d700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fc40ae6d0 0x7f7fc40cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc2d9d700 1 -- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fc415a960 msgr2=0x7f7fc4150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:36.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc2d9d700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fc415a960 0x7f7fc4150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:36.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc2d9d700 1 -- 172.21.15.29:0/1976087346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40094f0 con 0x7f7fc415ad30 2023-10-07T11:03:36.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc2d9d700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 0x7f7fc40aac10 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00c930 tx=0x7f7fbc0026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:36.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc00ce70 con 0x7f7fc415ad30 2023-10-07T11:03:36.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc00a7e0 con 0x7f7fc415ad30 2023-10-07T11:03:36.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc40a54c0 con 0x7f7fc415ad30 2023-10-07T11:03:36.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc00aba0 con 0x7f7fc415ad30 2023-10-07T11:03:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.631+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc40aa090 con 0x7f7fc415ad30 2023-10-07T11:03:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.632+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc404c3c0 con 0x7f7fc415ad30 2023-10-07T11:03:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.633+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7fbc012420 con 0x7f7fc415ad30 2023-10-07T11:03:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.634+0000 7f7fb27fc700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7fa4066f30 0x7f7fa40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.635+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f7fbc084ae0 con 0x7f7fc415ad30 2023-10-07T11:03:36.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.635+0000 7f7fc259c700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7fa4066f30 0x7f7fa40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:36.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.636+0000 7f7fc259c700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7fa4066f30 0x7f7fa40693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7fc406b910 tx=0x7f7fb4016000 comp rx=0 tx=0).ready entity=mgr.8429 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:36.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.638+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7fbc0556b0 con 0x7f7fc415ad30 2023-10-07T11:03:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:36.896+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7fc41586c0 con 0x7f7fc415ad30 2023-10-07T11:03:37.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.336+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 99) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f7fbc051ff0 con 0x7f7fc415ad30 2023-10-07T11:03:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.338+0000 7f7fb27fc700 1 -- 172.21.15.29:0/1976087346 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v99) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7fbc0556b0 con 0x7f7fc415ad30 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.341+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7fa4066f30 msgr2=0x7f7fa40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.341+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7fa4066f30 0x7f7fa40693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7fc406b910 tx=0x7f7fb4016000 comp rx=0 tx=0).stop 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 msgr2=0x7f7fc40aac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 0x7f7fc40aac10 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00c930 tx=0x7f7fbc0026e0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 shutdown_connections 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fc415ad30 0x7f7fc40aac10 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7fa4066f30 0x7f7fa40693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fc40ae6d0 0x7f7fc40cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 --2- 172.21.15.29:0/1976087346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fc415a960 0x7f7fc4150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 >> 172.21.15.29:0/1976087346 conn(0x7f7fc4067610 msgr2=0x7f7fc4157040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:37.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 shutdown_connections 2023-10-07T11:03:37.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.342+0000 7f7fc89ff700 1 -- 172.21.15.29:0/1976087346 wait complete. 2023-10-07T11:03:37.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-07T11:03:37.553 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:37.553+0000 7fe1569271c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:03:37.555 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:37.554+0000 7fb955f551c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:03:37.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.568+0000 7f54ecfff700 1 Processor -- start 2023-10-07T11:03:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.568+0000 7f54ecfff700 1 -- start start 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.569+0000 7f54ecfff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f54e8156180 0x7f54e80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54ecfff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54e80ae570 0x7f54e80accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54ecfff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80ad230 con 0x7f54e80b0520 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54ecfff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e80ad4d0 con 0x7f54e80ae570 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54ecfff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e80ad7b0 con 0x7f54e8156180 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54e80ae570 0x7f54e80accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54e80ae570 0x7f54e80accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50102/0 (socket says 172.21.15.29:50102) 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 -- 172.21.15.29:0/1906009284 learned_addr learned my addr 172.21.15.29:0/1906009284 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e659c700 1 --2- 172.21.15.29:0/1906009284 >> v2:172.21.15.144:3300/0 conn(0x7f54e8156180 0x7f54e80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 -- 172.21.15.29:0/1906009284 >> v2:172.21.15.144:3300/0 conn(0x7f54e8156180 msgr2=0x7f54e80afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 --2- 172.21.15.29:0/1906009284 >> v2:172.21.15.144:3300/0 conn(0x7f54e8156180 0x7f54e80afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 -- 172.21.15.29:0/1906009284 >> v1:172.21.15.144:6789/0 conn(0x7f54e80b0520 legacy=0x7f54e80b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e559a700 1 -- 172.21.15.29:0/1906009284 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650982387 0 0) 0x7f54e80ad230 con 0x7f54e80b0520 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.570+0000 7f54e5d9b700 1 -- 172.21.15.29:0/1906009284 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80ada90 con 0x7f54e80ae570 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.571+0000 7f54e5d9b700 1 --2- 172.21.15.29:0/1906009284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e80accf0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f54e80b2250 tx=0x7f54dc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bdac3dcdf83d52 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.571+0000 7f54e559a700 1 -- 172.21.15.29:0/1906009284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc00dba0 con 0x7f54e80ae570 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.571+0000 7f54e559a700 1 -- 172.21.15.29:0/1906009284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc00dd00 con 0x7f54e80ae570 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.571+0000 7f54e559a700 1 -- 172.21.15.29:0/1906009284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0196d0 con 0x7f54e80ae570 2023-10-07T11:03:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 -- 172.21.15.29:0/1906009284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 msgr2=0x7f54e80accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1906009284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e80accf0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f54e80b2250 tx=0x7f54dc00b4f0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 -- 172.21.15.29:0/1906009284 shutdown_connections 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1906009284 >> v2:172.21.15.144:3300/0 conn(0x7f54e8156180 0x7f54e80afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1906009284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e80accf0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 -- 172.21.15.29:0/1906009284 >> 172.21.15.29:0/1906009284 conn(0x7f54e8067610 msgr2=0x7f54e8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 -- 172.21.15.29:0/1906009284 shutdown_connections 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.572+0000 7f54ecfff700 1 -- 172.21.15.29:0/1906009284 wait complete. 2023-10-07T11:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.573+0000 7f54ecfff700 1 Processor -- start 2023-10-07T11:03:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.574+0000 7f54ecfff700 1 -- start start 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.574+0000 7f54ecfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e8151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.574+0000 7f54ecfff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e80b0520 0x7f54e80c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54ecfff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54e8156180 0x7f54e806e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54ecfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8165880 con 0x7f54e80ae570 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54ecfff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8165b60 con 0x7f54e80b0520 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54ecfff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8165e40 con 0x7f54e8156180 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e8151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e8151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50106/0 (socket says 172.21.15.29:50106) 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 -- 172.21.15.29:0/1834611033 learned_addr learned my addr 172.21.15.29:0/1834611033 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e5d9b700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e80b0520 0x7f54e80c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e6d9d700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54e8156180 0x7f54e806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 -- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54e8156180 msgr2=0x7f54e806e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54e8156180 0x7f54e806e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 -- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e80b0520 msgr2=0x7f54e80c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e80b0520 0x7f54e80c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.575+0000 7f54e659c700 1 -- 172.21.15.29:0/1834611033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc00b790 con 0x7f54e80ae570 2023-10-07T11:03:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.576+0000 7f54e659c700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e8151bb0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f54d800e930 tx=0x7f54d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.576+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8013680 con 0x7f54e80ae570 2023-10-07T11:03:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.576+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d80137e0 con 0x7f54e80ae570 2023-10-07T11:03:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.576+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8013b80 con 0x7f54e80ae570 2023-10-07T11:03:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.576+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e8166120 con 0x7f54e80ae570 2023-10-07T11:03:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.576+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e80c3e50 con 0x7f54e80ae570 2023-10-07T11:03:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.577+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e804c3c0 con 0x7f54e80ae570 2023-10-07T11:03:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.578+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 84757+0+0 (secure 0 0 0) 0x7f54d800c3e0 con 0x7f54e80ae570 2023-10-07T11:03:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.579+0000 7f54d67fc700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 0x7f54c8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.579+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7f54d80846f0 con 0x7f54e80ae570 2023-10-07T11:03:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.579+0000 7f54e5d9b700 1 -- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 msgr2=0x7f54c8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.579+0000 7f54e5d9b700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 0x7f54c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:37.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.583+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54d80516e0 con 0x7f54e80ae570 2023-10-07T11:03:37.731 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:37.731+0000 7fe1569271c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:03:37.775 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:37.775+0000 7fb955f551c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:03:37.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.780+0000 7f54e5d9b700 1 -- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 msgr2=0x7f54c8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:37.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.780+0000 7f54e5d9b700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 0x7f54c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:37.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.841+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f54e80c3e50 con 0x7f54e80ae570 2023-10-07T11:03:37.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.845+0000 7f54d67fc700 1 -- 172.21.15.29:0/1834611033 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f54e80c3e50 con 0x7f54e80ae570 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 msgr2=0x7f54c8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 0x7f54c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 msgr2=0x7f54e8151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e8151bb0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f54d800e930 tx=0x7f54d80091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 shutdown_connections 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f54c8066f80 0x7f54c8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54e8156180 0x7f54e806e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e80ae570 0x7f54e8151bb0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 --2- 172.21.15.29:0/1834611033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e80b0520 0x7f54e80c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:37.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 >> 172.21.15.29:0/1834611033 conn(0x7f54e8067610 msgr2=0x7f54e81548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:37.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 shutdown_connections 2023-10-07T11:03:37.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:37.848+0000 7f54ecfff700 1 -- 172.21.15.29:0/1834611033 wait complete. 2023-10-07T11:03:37.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-07T11:03:38.044 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:38.045+0000 7fe1569271c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:03:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.069+0000 7fe5283ff700 1 Processor -- start 2023-10-07T11:03:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.070+0000 7fe5283ff700 1 -- start start 2023-10-07T11:03:38.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe5283ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5201596e0 0x7fe5200cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe5283ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200cc630 0x7fe5200acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe5283ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5200ccaa0 con 0x7fe5200ad440 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe5283ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5200aa8e0 con 0x7fe5201596e0 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe5283ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe520159ab0 con 0x7fe5200cc630 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe52619b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5201596e0 0x7fe5200cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe52619b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5201596e0 0x7fe5200cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50120/0 (socket says 172.21.15.29:50120) 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.071+0000 7fe52619b700 1 -- 172.21.15.29:0/3090191291 learned_addr learned my addr 172.21.15.29:0/3090191291 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe52599a700 1 --2- 172.21.15.29:0/3090191291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200cc630 0x7fe5200acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe52619b700 1 -- 172.21.15.29:0/3090191291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200cc630 msgr2=0x7fe5200acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe52619b700 1 --2- 172.21.15.29:0/3090191291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200cc630 0x7fe5200acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe52619b700 1 -- 172.21.15.29:0/3090191291 >> v1:172.21.15.144:6789/0 conn(0x7fe5200ad440 legacy=0x7fe52015d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe52619b700 1 -- 172.21.15.29:0/3090191291 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5200a98d0 con 0x7fe5201596e0 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe52619b700 1 --2- 172.21.15.29:0/3090191291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5201596e0 0x7fe5200cc0f0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0099b0 tx=0x7fe51c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96193537c9876fb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.072+0000 7fe525199700 1 -- 172.21.15.29:0/3090191291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c009ea0 con 0x7fe5201596e0 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe525199700 1 -- 172.21.15.29:0/3090191291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c00b860 con 0x7fe5201596e0 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe525199700 1 -- 172.21.15.29:0/3090191291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00bc00 con 0x7fe5201596e0 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 -- 172.21.15.29:0/3090191291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5201596e0 msgr2=0x7fe5200cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 --2- 172.21.15.29:0/3090191291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5201596e0 0x7fe5200cc0f0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0099b0 tx=0x7fe51c009250 comp rx=0 tx=0).stop 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 -- 172.21.15.29:0/3090191291 shutdown_connections 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 --2- 172.21.15.29:0/3090191291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5201596e0 0x7fe5200cc0f0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 --2- 172.21.15.29:0/3090191291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200cc630 0x7fe5200acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 -- 172.21.15.29:0/3090191291 >> 172.21.15.29:0/3090191291 conn(0x7fe520067610 msgr2=0x7fe520067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 -- 172.21.15.29:0/3090191291 shutdown_connections 2023-10-07T11:03:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.073+0000 7fe5283ff700 1 -- 172.21.15.29:0/3090191291 wait complete. 2023-10-07T11:03:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.075+0000 7fe5283ff700 1 Processor -- start 2023-10-07T11:03:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.076+0000 7fe5283ff700 1 -- start start 2023-10-07T11:03:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe5283ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200ad440 0x7fe520146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe5283ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 0x7fe520157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe5283ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5201596e0 0x7fe520158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe5283ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe520156bc0 con 0x7fe5200cc630 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe5283ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe520155fb0 con 0x7fe5201596e0 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe5283ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe520156290 con 0x7fe5200ad440 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe52599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 0x7fe520157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe52599a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 0x7fe520157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50130/0 (socket says 172.21.15.29:50130) 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe52599a700 1 -- 172.21.15.29:0/970327695 learned_addr learned my addr 172.21.15.29:0/970327695 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe52619b700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200ad440 0x7fe520146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.077+0000 7fe52699c700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5201596e0 0x7fe520158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe52599a700 1 -- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200ad440 msgr2=0x7fe520146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe52599a700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200ad440 0x7fe520146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe52599a700 1 -- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5201596e0 msgr2=0x7fe520158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe52599a700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5201596e0 0x7fe520158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe52599a700 1 -- 172.21.15.29:0/970327695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0094f0 con 0x7fe5200cc630 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe52599a700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 0x7fe520157b30 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fe51000ec80 tx=0x7fe5100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51000c850 con 0x7fe5200cc630 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51000c9b0 con 0x7fe5200cc630 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe520156570 con 0x7fe5200cc630 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51000cd50 con 0x7fe5200cc630 2023-10-07T11:03:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.078+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe520155c80 con 0x7fe5200cc630 2023-10-07T11:03:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.080+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52004c3c0 con 0x7fe5200cc630 2023-10-07T11:03:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.080+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fe510013680 con 0x7fe5200cc630 2023-10-07T11:03:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.081+0000 7fe5167fc700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 0x7fe504069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.081+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fe510084af0 con 0x7fe5200cc630 2023-10-07T11:03:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.081+0000 7fe52619b700 1 -- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 msgr2=0x7fe504069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.081+0000 7fe52619b700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 0x7fe504069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.085+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe510051ae0 con 0x7fe5200cc630 2023-10-07T11:03:38.135 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:38.135+0000 7fb955f551c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:03:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.282+0000 7fe52619b700 1 -- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 msgr2=0x7fe504069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.282+0000 7fe52619b700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 0x7fe504069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.320+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe520156d00 con 0x7fe5200cc630 2023-10-07T11:03:38.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.324+0000 7fe5167fc700 1 -- 172.21.15.29:0/970327695 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe5100170c0 con 0x7fe5200cc630 2023-10-07T11:03:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 msgr2=0x7fe504069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 0x7fe504069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 msgr2=0x7fe520157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 0x7fe520157b30 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fe51000ec80 tx=0x7fe5100091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 shutdown_connections 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5201596e0 0x7fe520158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe504066f80 0x7fe504069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5200ad440 0x7fe520146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 --2- 172.21.15.29:0/970327695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5200cc630 0x7fe520157b30 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 >> 172.21.15.29:0/970327695 conn(0x7fe520067610 msgr2=0x7fe5200b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 shutdown_connections 2023-10-07T11:03:38.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.326+0000 7fe5283ff700 1 -- 172.21.15.29:0/970327695 wait complete. 2023-10-07T11:03:38.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:38.347 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-07T11:03:38.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-10-07T11:03:38.568 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:38.569+0000 7fe1569271c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:03:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.602+0000 7fca1a3ff700 1 Processor -- start 2023-10-07T11:03:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.603+0000 7fca1a3ff700 1 -- start start 2023-10-07T11:03:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca1a3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca14159480 0x7fca140b0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca1a3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca140cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca1a3ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140cbd60 con 0x7fca14159850 2023-10-07T11:03:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca1a3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1415e0d0 con 0x7fca14159480 2023-10-07T11:03:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca1a3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca140cb430 con 0x7fca140cefe0 2023-10-07T11:03:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca1899c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca140cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.604+0000 7fca137fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fca14159850 0x7fca140ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:58428/0 (socket says 172.21.15.29:58428) 2023-10-07T11:03:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca137fe700 1 -- 172.21.15.29:0/4075496368 learned_addr learned my addr 172.21.15.29:0/4075496368 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca13fff700 1 --2- 172.21.15.29:0/4075496368 >> v2:172.21.15.144:3300/0 conn(0x7fca14159480 0x7fca140b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca12ffd700 1 -- 172.21.15.29:0/4075496368 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771108636 0 0) 0x7fca140cbd60 con 0x7fca14159850 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca12ffd700 1 -- 172.21.15.29:0/4075496368 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9fc003570 con 0x7fca14159850 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca1899c700 1 -- 172.21.15.29:0/4075496368 >> v2:172.21.15.144:3300/0 conn(0x7fca14159480 msgr2=0x7fca140b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca1899c700 1 --2- 172.21.15.29:0/4075496368 >> v2:172.21.15.144:3300/0 conn(0x7fca14159480 0x7fca140b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca1899c700 1 -- 172.21.15.29:0/4075496368 >> v1:172.21.15.144:6789/0 conn(0x7fca14159850 legacy=0x7fca140ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca1899c700 1 -- 172.21.15.29:0/4075496368 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140cb710 con 0x7fca140cefe0 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.605+0000 7fca1899c700 1 --2- 172.21.15.29:0/4075496368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca140cc1f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009e90 tx=0x7fca0c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0e3d27f96a8ca3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca12ffd700 1 -- 172.21.15.29:0/4075496368 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00e070 con 0x7fca140cefe0 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca12ffd700 1 -- 172.21.15.29:0/4075496368 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c00bb40 con 0x7fca140cefe0 2023-10-07T11:03:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca12ffd700 1 -- 172.21.15.29:0/4075496368 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c012430 con 0x7fca140cefe0 2023-10-07T11:03:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca1a3ff700 1 -- 172.21.15.29:0/4075496368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 msgr2=0x7fca140cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/4075496368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca140cc1f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009e90 tx=0x7fca0c009780 comp rx=0 tx=0).stop 2023-10-07T11:03:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca1a3ff700 1 -- 172.21.15.29:0/4075496368 shutdown_connections 2023-10-07T11:03:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/4075496368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca140cc1f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/4075496368 >> v2:172.21.15.144:3300/0 conn(0x7fca14159480 0x7fca140b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.606+0000 7fca1a3ff700 1 -- 172.21.15.29:0/4075496368 >> 172.21.15.29:0/4075496368 conn(0x7fca14063230 msgr2=0x7fca14063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.607+0000 7fca1a3ff700 1 -- 172.21.15.29:0/4075496368 shutdown_connections 2023-10-07T11:03:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.607+0000 7fca1a3ff700 1 -- 172.21.15.29:0/4075496368 wait complete. 2023-10-07T11:03:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.609+0000 7fca1a3ff700 1 Processor -- start 2023-10-07T11:03:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.609+0000 7fca1a3ff700 1 -- start start 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1a3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca14141f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1a3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca14159480 0x7fca141503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1a3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca14159850 0x7fca141532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1a3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca140c57e0 con 0x7fca14159850 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1a3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca141424b0 con 0x7fca14159480 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1a3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca14150e20 con 0x7fca140cefe0 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca13fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca14141f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.610+0000 7fca1899c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca14159850 0x7fca141532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca14141f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:51594/0 (socket says 172.21.15.29:51594) 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 -- 172.21.15.29:0/2486018447 learned_addr learned my addr 172.21.15.29:0/2486018447 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca14159480 0x7fca141503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca14159480 msgr2=0x7fca141503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca14159480 0x7fca141503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca14159850 msgr2=0x7fca141532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca14159850 0x7fca141532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 -- 172.21.15.29:0/2486018447 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c009a20 con 0x7fca140cefe0 2023-10-07T11:03:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fca13fff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca14141f70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fca0400ec80 tx=0x7fca040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0400c850 con 0x7fca140cefe0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400c9b0 con 0x7fca140cefe0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.611+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0400cd50 con 0x7fca140cefe0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.612+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca14151100 con 0x7fca140cefe0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.612+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca140c75e0 con 0x7fca140cefe0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.613+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca140c5630 con 0x7fca140cefe0 2023-10-07T11:03:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.614+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 99) v1 ==== 84757+0+0 (secure 0 0 0) 0x7fca04013680 con 0x7fca140cefe0 2023-10-07T11:03:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.614+0000 7fc9fbfff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.615+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6181+0+0 (secure 0 0 0) 0x7fca04084a40 con 0x7fca140cefe0 2023-10-07T11:03:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.615+0000 7fca137fe700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.615+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.619+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca04051a30 con 0x7fca140cefe0 2023-10-07T11:03:38.702 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:38.701+0000 7fb955f551c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:03:38.740 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:38.740+0000 7fe1569271c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:03:38.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.816+0000 7fca137fe700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:38.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.816+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:38.850+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fca140c52d0 con 0x7fca00066f80 2023-10-07T11:03:38.877 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:38.877+0000 7fb955f551c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:03:39.157 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:39.157+0000 7fe1569271c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:03:39.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:39.217+0000 7fca137fe700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:39.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:39.217+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-07T11:03:39.306 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:39.305+0000 7fb955f551c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:03:39.931 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:39.931+0000 7fe1569271c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:03:40.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.018+0000 7fe1569271c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:03:40.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:40.018+0000 7fca137fe700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:40.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:40.018+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-07T11:03:40.108 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.107+0000 7fb955f551c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:03:40.171 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.171+0000 7fe1569271c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:03:40.198 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.197+0000 7fb955f551c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:03:40.257 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.258+0000 7fe1569271c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:03:40.358 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.358+0000 7fb955f551c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:03:40.449 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.448+0000 7fb955f551c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:03:40.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.474+0000 7fe1569271c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:03:40.570 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.570+0000 7fe1569271c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:03:40.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.666+0000 7fe1569271c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:03:40.669 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.669+0000 7fb955f551c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:03:40.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.761+0000 7fe1569271c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:03:40.768 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.767+0000 7fb955f551c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:03:40.866 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.865+0000 7fb955f551c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:03:40.948 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:40.949+0000 7fe1569271c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:03:40.965 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:40.964+0000 7fb955f551c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:03:41.047 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:41.047+0000 7fe1569271c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:03:41.159 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:41.158+0000 7fb955f551c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:03:41.260 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:41.259+0000 7fb955f551c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:03:41.409 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:41.408+0000 7fe1569271c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:03:41.497 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:41.497+0000 7fe1569271c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:41.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:41.609+0000 7fb955f551c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:03:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:41.620+0000 7fca137fe700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:41.620+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-07T11:03:41.701 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:41.700+0000 7fb955f551c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:42.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:42.034+0000 7fe1569271c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:03:42.220 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:42.220+0000 7fe1569271c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:03:42.262 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:42.262+0000 7fb955f551c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:03:42.455 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:42.454+0000 7fb955f551c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:03:42.598 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:42.597+0000 7fe1569271c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:42.838 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:42.838+0000 7fb955f551c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:43.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.019+0000 7fe1569271c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:03:43.200 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.200+0000 7fe1569271c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:43.277 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:43.277+0000 7fb955f551c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:03:43.388 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.388+0000 7fe1569271c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:03:43.465 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:43.466+0000 7fb955f551c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:43.496 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.496+0000 7fe1569271c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:43.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:43.513+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mgrmap(e 100) v1 ==== 84468+0+0 (secure 0 0 0) 0x7fca04051e00 con 0x7fca140cefe0 2023-10-07T11:03:43.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:43.513+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:43.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:43.513+0000 7fc9fbfff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:43.635 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0c7730700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.635 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0c7730700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.635 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0c7730700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.635 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0c7730700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.635 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0c7730700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.636 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0ca736700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.636 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0ca736700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.636 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0ca736700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.636 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0ca736700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.636 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:43.635+0000 7fe0ca736700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:43.675 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:43.675+0000 7fb955f551c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:03:43.787 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:43.787+0000 7fb955f551c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:44.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:03:44.449 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:03:44.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.519+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mon.2 v2:172.21.15.144:3301/0 8 ==== mgrmap(e 101) v1 ==== 84595+0+0 (secure 0 0 0) 0x7fca0405c200 con 0x7fca140cefe0 2023-10-07T11:03:44.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.519+0000 7fc9fbfff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.519+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fca140c52d0 con 0x7fca00066f80 2023-10-07T11:03:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.521+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.523+0000 7fca137fe700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fca141557c0 tx=0x7fca08009040 comp rx=0 tx=0).ready entity=mgr.9754 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:44.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.585+0000 7fc9fbfff700 1 -- 172.21.15.29:0/2486018447 <== mgr.9754 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fca140c52d0 con 0x7fca00066f80 2023-10-07T11:03:44.585 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 msgr2=0x7fca00069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fca141557c0 tx=0x7fca08009040 comp rx=0 tx=0).stop 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 msgr2=0x7fca14141f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca14141f70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fca0400ec80 tx=0x7fca040091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 shutdown_connections 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca14159850 0x7fca141532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fca00066f80 0x7fca00069430 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca140cefe0 0x7fca14141f70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 --2- 172.21.15.29:0/2486018447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca14159480 0x7fca141503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 >> 172.21.15.29:0/2486018447 conn(0x7fca14063230 msgr2=0x7fca14157b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 shutdown_connections 2023-10-07T11:03:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.588+0000 7fca1a3ff700 1 -- 172.21.15.29:0/2486018447 wait complete. 2023-10-07T11:03:44.606 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-07T11:03:44.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-10-07T11:03:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.848+0000 7f06003ff700 1 Processor -- start 2023-10-07T11:03:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.848+0000 7f06003ff700 1 -- start start 2023-10-07T11:03:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f06003ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05f80c2940 0x7f05f80ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f06003ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f8159880 0x7f05f80cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f06003ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f8068570 con 0x7f05f81594b0 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f06003ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f8067960 con 0x7f05f80c2940 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f06003ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f8067c40 con 0x7f05f8159880 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd99a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05f80c2940 0x7f05f80ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd99a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05f80c2940 0x7f05f80ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47524/0 (socket says 172.21.15.29:47524) 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd99a700 1 -- 172.21.15.29:0/3134346809 learned_addr learned my addr 172.21.15.29:0/3134346809 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fe99c700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f8159880 0x7f05f80cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd99a700 1 -- 172.21.15.29:0/3134346809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f8159880 msgr2=0x7f05f80cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd99a700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f8159880 0x7f05f80cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd99a700 1 -- 172.21.15.29:0/3134346809 >> v1:172.21.15.29:6789/0 conn(0x7f05f81594b0 legacy=0x7f05f8068a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.851+0000 7f05fd99a700 1 -- 172.21.15.29:0/3134346809 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f8067f20 con 0x7f05f80c2940 2023-10-07T11:03:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.850+0000 7f05fd199700 1 -- 172.21.15.29:0/3134346809 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255032311 0 0) 0x7f05f8068570 con 0x7f05f81594b0 2023-10-07T11:03:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.851+0000 7f05fe99c700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f8159880 0x7f05f80cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.851+0000 7f05fd99a700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f80ca5c0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f05e80099b0 tx=0x7f05e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c02e159922f8a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.851+0000 7f05fd199700 1 -- 172.21.15.29:0/3134346809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800b860 con 0x7f05f80c2940 2023-10-07T11:03:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.851+0000 7f05fd199700 1 -- 172.21.15.29:0/3134346809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e800b9c0 con 0x7f05f80c2940 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.851+0000 7f05fd199700 1 -- 172.21.15.29:0/3134346809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800bd60 con 0x7f05f80c2940 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 -- 172.21.15.29:0/3134346809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 msgr2=0x7f05f80ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f80ca5c0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f05e80099b0 tx=0x7f05e8009250 comp rx=0 tx=0).stop 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 -- 172.21.15.29:0/3134346809 shutdown_connections 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f8159880 0x7f05f80cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 --2- 172.21.15.29:0/3134346809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f80ca5c0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 -- 172.21.15.29:0/3134346809 >> 172.21.15.29:0/3134346809 conn(0x7f05f80c9460 msgr2=0x7f05f80c9860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 -- 172.21.15.29:0/3134346809 shutdown_connections 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.852+0000 7f06003ff700 1 -- 172.21.15.29:0/3134346809 wait complete. 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.854+0000 7f06003ff700 1 Processor -- start 2023-10-07T11:03:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.854+0000 7f06003ff700 1 -- start start 2023-10-07T11:03:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.855+0000 7f06003ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f814b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.855+0000 7f06003ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f81594b0 0x7f05f80c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.855+0000 7f06003ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05f8159880 0x7f05f80c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.855+0000 7f06003ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f80a98d0 con 0x7f05f80c2940 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.855+0000 7f06003ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f816a910 con 0x7f05f8159880 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.855+0000 7f06003ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f80c4e40 con 0x7f05f81594b0 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f814b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f814b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47538/0 (socket says 172.21.15.29:47538) 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 -- 172.21.15.29:0/3045465543 learned_addr learned my addr 172.21.15.29:0/3045465543 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fd99a700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f81594b0 0x7f05f80c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe99c700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05f8159880 0x7f05f80c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 -- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f81594b0 msgr2=0x7f05f80c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f81594b0 0x7f05f80c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 -- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05f8159880 msgr2=0x7f05f80c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05f8159880 0x7f05f80c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 -- 172.21.15.29:0/3045465543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e80094f0 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe99c700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05f8159880 0x7f05f80c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.856+0000 7f05fe19b700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f814b580 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c7e0 tx=0x7f05f4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.857+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4012700 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.857+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f4012880 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.857+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4012c40 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.857+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f80c4230 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.857+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f80c47f0 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.858+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f80718b0 con 0x7f05f80c2940 2023-10-07T11:03:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.859+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f05f401b400 con 0x7f05f80c2940 2023-10-07T11:03:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.859+0000 7f05ee7fc700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f05dc066dd0 0x7f05dc069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.859+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f05f4084270 con 0x7f05f80c2940 2023-10-07T11:03:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.860+0000 7f05fd99a700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f05dc066dd0 0x7f05dc069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.860+0000 7f05fd99a700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f05dc066dd0 0x7f05dc069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05e80099b0 tx=0x7f05e8017000 comp rx=0 tx=0).ready entity=mgr.9754 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:44.864+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05f40513e0 con 0x7f05f80c2940 2023-10-07T11:03:45.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.091+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f05f80cc7d0 con 0x7f05dc066dd0 2023-10-07T11:03:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.153+0000 7f05ee7fc700 1 -- 172.21.15.29:0/3045465543 <== mgr.9754 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f05f80cc7d0 con 0x7f05dc066dd0 2023-10-07T11:03:45.153 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path snap_test_dir1 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.155+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f05dc066dd0 msgr2=0x7f05dc069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.155+0000 7f06003ff700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f05dc066dd0 0x7f05dc069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05e80099b0 tx=0x7f05e8017000 comp rx=0 tx=0).stop 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.155+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 msgr2=0x7f05f814b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.155+0000 7f06003ff700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f814b580 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c7e0 tx=0x7f05f4007df0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 shutdown_connections 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05f8159880 0x7f05f80c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05f80c2940 0x7f05f814b580 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f05dc066dd0 0x7f05dc069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 --2- 172.21.15.29:0/3045465543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05f81594b0 0x7f05f80c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 >> 172.21.15.29:0/3045465543 conn(0x7f05f80c9460 msgr2=0x7f05f80af790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 shutdown_connections 2023-10-07T11:03:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.156+0000 7f06003ff700 1 -- 172.21.15.29:0/3045465543 wait complete. 2023-10-07T11:03:45.173 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-07T11:03:45.180 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-07-11_04_00 in ~15s... 2023-10-07T11:03:45.180 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-10-07T11:03:45.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-10-07T11:03:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.380+0000 7f67aa7ff700 1 Processor -- start 2023-10-07T11:03:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.380+0000 7f67aa7ff700 1 -- start start 2023-10-07T11:03:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67aa7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67a40b2a20 0x7f67a40cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67aa7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67aa7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67a40cf0f0 0x7f67a4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67aa7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a40cc940 con 0x7f67a40cf0f0 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67aa7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a40cf530 con 0x7f67a40b2a20 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67aa7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40b2df0 con 0x7f67a40cb6f0 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.381+0000 7f67a8d9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67a40cf0f0 0x7f67a4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67a40cf0f0 0x7f67a4159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47546/0 (socket says 172.21.15.29:47546) 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 -- 172.21.15.29:0/419842326 learned_addr learned my addr 172.21.15.29:0/419842326 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a37fe700 1 --2- 172.21.15.29:0/419842326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a3fff700 1 --2- 172.21.15.29:0/419842326 >> v2:172.21.15.144:3300/0 conn(0x7f67a40b2a20 0x7f67a40cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 -- 172.21.15.29:0/419842326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 msgr2=0x7f67a40cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 --2- 172.21.15.29:0/419842326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 -- 172.21.15.29:0/419842326 >> v2:172.21.15.144:3300/0 conn(0x7f67a40b2a20 msgr2=0x7f67a40cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 --2- 172.21.15.29:0/419842326 >> v2:172.21.15.144:3300/0 conn(0x7f67a40b2a20 0x7f67a40cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 -- 172.21.15.29:0/419842326 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a40cbd30 con 0x7f67a40cf0f0 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.382+0000 7f67a8d9c700 1 --2- 172.21.15.29:0/419842326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40cf0f0 0x7f67a4159ce0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f67a408ac80 tx=0x7f679c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b684f28fca909609 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67a2ffd700 1 -- 172.21.15.29:0/419842326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c00e070 con 0x7f67a40cf0f0 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67a2ffd700 1 -- 172.21.15.29:0/419842326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c00b970 con 0x7f67a40cf0f0 2023-10-07T11:03:45.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67a2ffd700 1 -- 172.21.15.29:0/419842326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c0092e0 con 0x7f67a40cf0f0 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 -- 172.21.15.29:0/419842326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40cf0f0 msgr2=0x7f67a4159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/419842326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40cf0f0 0x7f67a4159ce0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f67a408ac80 tx=0x7f679c009840 comp rx=0 tx=0).stop 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 -- 172.21.15.29:0/419842326 shutdown_connections 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/419842326 >> v2:172.21.15.144:3300/0 conn(0x7f67a40b2a20 0x7f67a40cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/419842326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40cf0f0 0x7f67a4159ce0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/419842326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 -- 172.21.15.29:0/419842326 >> 172.21.15.29:0/419842326 conn(0x7f67a4099820 msgr2=0x7f67a4063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 -- 172.21.15.29:0/419842326 shutdown_connections 2023-10-07T11:03:45.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.383+0000 7f67aa7ff700 1 -- 172.21.15.29:0/419842326 wait complete. 2023-10-07T11:03:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.385+0000 7f67aa7ff700 1 Processor -- start 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.385+0000 7f67aa7ff700 1 -- start start 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67aa7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 0x7f67a4156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67aa7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67aa7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a40cf0f0 0x7f67a40c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67aa7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40b5eb0 con 0x7f67a40b2a20 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67aa7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a4157c80 con 0x7f67a40cf0f0 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67aa7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a41577f0 con 0x7f67a40cb6f0 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 0x7f67a4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 0x7f67a4156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47548/0 (socket says 172.21.15.29:47548) 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 -- 172.21.15.29:0/3166916132 learned_addr learned my addr 172.21.15.29:0/3166916132 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a37fe700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a8d9c700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a40cf0f0 0x7f67a40c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 -- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 msgr2=0x7f67a40b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 -- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a40cf0f0 msgr2=0x7f67a40c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a40cf0f0 0x7f67a40c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.386+0000 7f67a3fff700 1 -- 172.21.15.29:0/3166916132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c009ae0 con 0x7f67a40b2a20 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.387+0000 7f67a3fff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 0x7f67a4156500 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f679400ecb0 tx=0x7f6794009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.387+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679400c970 con 0x7f67a40b2a20 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.387+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679400cad0 con 0x7f67a40b2a20 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.387+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a4156be0 con 0x7f67a40b2a20 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.387+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679400ce50 con 0x7f67a40b2a20 2023-10-07T11:03:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.387+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a41571a0 con 0x7f67a40b2a20 2023-10-07T11:03:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.388+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a406f7d0 con 0x7f67a40b2a20 2023-10-07T11:03:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.389+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 84595+0+0 (secure 0 0 0) 0x7f6794013680 con 0x7f67a40b2a20 2023-10-07T11:03:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.390+0000 7f678bfff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6790066dd0 0x7f6790069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.390+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f6794084b70 con 0x7f67a40b2a20 2023-10-07T11:03:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.391+0000 7f67a37fe700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6790066dd0 0x7f6790069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.391+0000 7f67a37fe700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6790066dd0 0x7f6790069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f67a41564b0 tx=0x7f6798008040 comp rx=0 tx=0).ready entity=mgr.9754 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.394+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6794051db0 con 0x7f67a40b2a20 2023-10-07T11:03:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.518+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 102) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f6794052180 con 0x7f67a40b2a20 2023-10-07T11:03:45.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.623+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67a4157930 con 0x7f6790066dd0 2023-10-07T11:03:45.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.632+0000 7f678bfff700 1 -- 172.21.15.29:0/3166916132 <== mgr.9754 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f6798009a60 con 0x7f6790066dd0 2023-10-07T11:03:45.632 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:45.632 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6790066dd0 msgr2=0x7f6790069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6790066dd0 0x7f6790069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f67a41564b0 tx=0x7f6798008040 comp rx=0 tx=0).stop 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 msgr2=0x7f67a4156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 0x7f67a4156500 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f679400ecb0 tx=0x7f6794009120 comp rx=0 tx=0).stop 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 shutdown_connections 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a40cf0f0 0x7f67a40c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67a40b2a20 0x7f67a4156500 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6790066dd0 0x7f6790069280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 --2- 172.21.15.29:0/3166916132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a40cb6f0 0x7f67a40b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 >> 172.21.15.29:0/3166916132 conn(0x7f67a4099820 msgr2=0x7f67a40c56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:45.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 shutdown_connections 2023-10-07T11:03:45.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.635+0000 7f67aa7ff700 1 -- 172.21.15.29:0/3166916132 wait complete. 2023-10-07T11:03:45.652 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-07T11:03:45.652 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-10-07T11:03:45.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-10-07T11:03:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.853+0000 7f6c1c5ff700 1 Processor -- start 2023-10-07T11:03:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.853+0000 7f6c1c5ff700 1 -- start start 2023-10-07T11:03:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140ced00 0x7f6c140b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1c5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c140c8a20 0x7f6c140cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1c5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c140690f0 con 0x7f6c140b0610 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1c5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c140cf2e0 con 0x7f6c140c8a20 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140687c0 con 0x7f6c140ced00 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1ab9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c140c8a20 0x7f6c140cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1ab9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c140c8a20 0x7f6c140cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47556/0 (socket says 172.21.15.29:47556) 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/3513027018 learned_addr learned my addr 172.21.15.29:0/3513027018 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.854+0000 7f6c19b9a700 1 --2- 172.21.15.29:0/3513027018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140ced00 0x7f6c140b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/3513027018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140ced00 msgr2=0x7f6c140b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c1ab9c700 1 --2- 172.21.15.29:0/3513027018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140ced00 0x7f6c140b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/3513027018 >> v1:172.21.15.144:6789/0 conn(0x7f6c140b0610 legacy=0x7f6c140c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/3513027018 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c14068aa0 con 0x7f6c140c8a20 2023-10-07T11:03:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c1ab9c700 1 --2- 172.21.15.29:0/3513027018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140c8a20 0x7f6c140cc7e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009f20 tx=0x7f6c0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387678abf1ffa2ef server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c19399700 1 -- 172.21.15.29:0/3513027018 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00e070 con 0x7f6c140c8a20 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c19399700 1 -- 172.21.15.29:0/3513027018 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c00b9d0 con 0x7f6c140c8a20 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.855+0000 7f6c19399700 1 -- 172.21.15.29:0/3513027018 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c0092e0 con 0x7f6c140c8a20 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/3513027018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140c8a20 msgr2=0x7f6c140cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/3513027018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140c8a20 0x7f6c140cc7e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009f20 tx=0x7f6c0c009840 comp rx=0 tx=0).stop 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/3513027018 shutdown_connections 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/3513027018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140c8a20 0x7f6c140cc7e0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/3513027018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140ced00 0x7f6c140b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/3513027018 >> 172.21.15.29:0/3513027018 conn(0x7f6c14063a10 msgr2=0x7f6c140ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/3513027018 shutdown_connections 2023-10-07T11:03:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.856+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/3513027018 wait complete. 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.858+0000 7f6c1c5ff700 1 Processor -- start 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.858+0000 7f6c1c5ff700 1 -- start start 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1c5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c140b0610 0x7f6c14147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1c5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140c8a20 0x7f6c14097fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1c5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 0x7f6c140984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1c5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1409bb90 con 0x7f6c140ced00 2023-10-07T11:03:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1c5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1409af80 con 0x7f6c140b0610 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1c5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1409b260 con 0x7f6c140c8a20 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1a39b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c140b0610 0x7f6c14147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 0x7f6c140984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c19b9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140c8a20 0x7f6c14097fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 0x7f6c140984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47560/0 (socket says 172.21.15.29:47560) 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c19b9a700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140c8a20 0x7f6c14097fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43308/0 (socket says 172.21.15.29:43308) 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/1559205499 learned_addr learned my addr 172.21.15.29:0/1559205499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1a39b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c140b0610 0x7f6c14147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:42262/0 (socket says 172.21.15.29:42262) 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140c8a20 msgr2=0x7f6c14097fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140c8a20 0x7f6c14097fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c140b0610 msgr2=0x7f6c14147120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c140b0610 0x7f6c14147120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.859+0000 7f6c1ab9c700 1 -- 172.21.15.29:0/1559205499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c009ae0 con 0x7f6c140ced00 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.860+0000 7f6c1ab9c700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 0x7f6c140984e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00b610 tx=0x7f6c0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.860+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c008ad0 con 0x7f6c140ced00 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.860+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c008c30 con 0x7f6c140ced00 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.860+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1409b540 con 0x7f6c140ced00 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.860+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c026460 con 0x7f6c140ced00 2023-10-07T11:03:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.860+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c14147660 con 0x7f6c140ced00 2023-10-07T11:03:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.861+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1409bd00 con 0x7f6c140ced00 2023-10-07T11:03:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.862+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 102) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f6c0c003040 con 0x7f6c140ced00 2023-10-07T11:03:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.863+0000 7f6c0a7fc700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6bf8067130 0x7f6bf80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.863+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f6c0c086b70 con 0x7f6c140ced00 2023-10-07T11:03:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.863+0000 7f6c1a39b700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6bf8067130 0x7f6bf80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:45.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.864+0000 7f6c1a39b700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6bf8067130 0x7f6bf80695e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6c100039e0 tx=0x7f6c10007ec0 comp rx=0 tx=0).ready entity=mgr.9754 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:45.867+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c0c0536d0 con 0x7f6c140ced00 2023-10-07T11:03:46.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.098+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c140718b0 con 0x7f6bf8067130 2023-10-07T11:03:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.168+0000 7f6c0a7fc700 1 -- 172.21.15.29:0/1559205499 <== mgr.9754 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6c140718b0 con 0x7f6bf8067130 2023-10-07T11:03:46.168 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6bf8067130 msgr2=0x7f6bf80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6bf8067130 0x7f6bf80695e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6c100039e0 tx=0x7f6c10007ec0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 msgr2=0x7f6c140984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 0x7f6c140984e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00b610 tx=0x7f6c0c018000 comp rx=0 tx=0).stop 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 shutdown_connections 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c140ced00 0x7f6c140984e0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c140b0610 0x7f6c14147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f6bf8067130 0x7f6bf80695e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 --2- 172.21.15.29:0/1559205499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c140c8a20 0x7f6c14097fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 >> 172.21.15.29:0/1559205499 conn(0x7f6c14063a10 msgr2=0x7f6c14158dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 shutdown_connections 2023-10-07T11:03:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.171+0000 7f6c1c5ff700 1 -- 172.21.15.29:0/1559205499 wait complete. 2023-10-07T11:03:46.188 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-07T11:03:46.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:46.189 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-07T11:03:46.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:46.238 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-07T11:03:46.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-07T11:03:46.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.483+0000 7f21cc5ff700 1 Processor -- start 2023-10-07T11:03:46.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.484+0000 7f21cc5ff700 1 -- start start 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21cc5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21c406a8c0 0x7f21c40c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21cc5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f21c406b5c0 0x7f21c40c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21cc5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c406a430 con 0x7f21c4159790 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21cc5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c4159b60 con 0x7f21c406a8c0 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21cc5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c4069820 con 0x7f21c406b5c0 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21c9b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21c406a8c0 0x7f21c40c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21c9b9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21c406a8c0 0x7f21c40c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47566/0 (socket says 172.21.15.29:47566) 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21ca39b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f21c4159790 0x7f21c406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48528/0 (socket says 172.21.15.29:48528) 2023-10-07T11:03:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.485+0000 7f21c9b9a700 1 -- 172.21.15.29:0/342978897 learned_addr learned my addr 172.21.15.29:0/342978897 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21cab9c700 1 --2- 172.21.15.29:0/342978897 >> v2:172.21.15.144:3300/0 conn(0x7f21c406b5c0 0x7f21c40c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9b9a700 1 -- 172.21.15.29:0/342978897 >> v2:172.21.15.144:3300/0 conn(0x7f21c406b5c0 msgr2=0x7f21c40c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9b9a700 1 --2- 172.21.15.29:0/342978897 >> v2:172.21.15.144:3300/0 conn(0x7f21c406b5c0 0x7f21c40c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9b9a700 1 -- 172.21.15.29:0/342978897 >> v1:172.21.15.29:6789/0 conn(0x7f21c4159790 legacy=0x7f21c406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9b9a700 1 -- 172.21.15.29:0/342978897 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4069b00 con 0x7f21c406a8c0 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21cab9c700 1 --2- 172.21.15.29:0/342978897 >> v2:172.21.15.144:3300/0 conn(0x7f21c406b5c0 0x7f21c40c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9b9a700 1 --2- 172.21.15.29:0/342978897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406a8c0 0x7f21c40c6f40 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f21c40635f0 tx=0x7f21b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88832409915b9ed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9399700 1 -- 172.21.15.29:0/342978897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b400b860 con 0x7f21c406a8c0 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9399700 1 -- 172.21.15.29:0/342978897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b400b9c0 con 0x7f21c406a8c0 2023-10-07T11:03:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.486+0000 7f21c9399700 1 -- 172.21.15.29:0/342978897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b400bd60 con 0x7f21c406a8c0 2023-10-07T11:03:46.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 -- 172.21.15.29:0/342978897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406a8c0 msgr2=0x7f21c40c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:46.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/342978897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406a8c0 0x7f21c40c6f40 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f21c40635f0 tx=0x7f21b4009250 comp rx=0 tx=0).stop 2023-10-07T11:03:46.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 -- 172.21.15.29:0/342978897 shutdown_connections 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/342978897 >> v2:172.21.15.144:3300/0 conn(0x7f21c406b5c0 0x7f21c40c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/342978897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406a8c0 0x7f21c40c6f40 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 -- 172.21.15.29:0/342978897 >> 172.21.15.29:0/342978897 conn(0x7f21c40c9410 msgr2=0x7f21c40c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 -- 172.21.15.29:0/342978897 shutdown_connections 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.487+0000 7f21cc5ff700 1 -- 172.21.15.29:0/342978897 wait complete. 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.488+0000 7f21cc5ff700 1 Processor -- start 2023-10-07T11:03:46.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.489+0000 7f21cc5ff700 1 -- start start 2023-10-07T11:03:46.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cc5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 0x7f21c406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cc5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21c4159790 0x7f21c40c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cc5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21c40c6520 0x7f21c40c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cc5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c416dee0 con 0x7f21c406b5c0 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cc5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c416d360 con 0x7f21c4159790 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cc5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c416d640 con 0x7f21c40c6520 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 0x7f21c406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 0x7f21c406dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47568/0 (socket says 172.21.15.29:47568) 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 -- 172.21.15.29:0/3140848324 learned_addr learned my addr 172.21.15.29:0/3140848324 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21c9b9a700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21c4159790 0x7f21c40c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21cab9c700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21c40c6520 0x7f21c40c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 -- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21c40c6520 msgr2=0x7f21c40c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21c40c6520 0x7f21c40c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 -- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21c4159790 msgr2=0x7f21c40c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21c4159790 0x7f21c40c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21ca39b700 1 -- 172.21.15.29:0/3140848324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40094f0 con 0x7f21c406b5c0 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.490+0000 7f21c9b9a700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21c4159790 0x7f21c40c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.491+0000 7f21ca39b700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 0x7f21c406dee0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c7c0 tx=0x7f21c0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.491+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0005a70 con 0x7f21c406b5c0 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.491+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c0005bf0 con 0x7f21c406b5c0 2023-10-07T11:03:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.491+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c00070f0 con 0x7f21c406b5c0 2023-10-07T11:03:46.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.491+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c416d920 con 0x7f21c406b5c0 2023-10-07T11:03:46.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.491+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c416cf80 con 0x7f21c406b5c0 2023-10-07T11:03:46.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.492+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c404c3c0 con 0x7f21c406b5c0 2023-10-07T11:03:46.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.493+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 102) v1 ==== 84916+0+0 (secure 0 0 0) 0x7f21c0007460 con 0x7f21c406b5c0 2023-10-07T11:03:46.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.494+0000 7f21ba7fc700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f21a8067130 0x7f21a80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:46.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.494+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f21c0084640 con 0x7f21c406b5c0 2023-10-07T11:03:46.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.495+0000 7f21c9b9a700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f21a8067130 0x7f21a80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:46.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.495+0000 7f21c9b9a700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f21a8067130 0x7f21a80695e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f21c40635f0 tx=0x7f21b4017000 comp rx=0 tx=0).ready entity=mgr.9754 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:46.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.498+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21c0051330 con 0x7f21c406b5c0 2023-10-07T11:03:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:46.760+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f21c4063640 con 0x7f21c406b5c0 2023-10-07T11:03:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.534+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f21c0051700 con 0x7f21c406b5c0 2023-10-07T11:03:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.536+0000 7f21ba7fc700 1 -- 172.21.15.29:0/3140848324 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v103) v1 ==== 93+0+0 (secure 0 0 0) 0x7f21c0051330 con 0x7f21c406b5c0 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f21a8067130 msgr2=0x7f21a80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f21a8067130 0x7f21a80695e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f21c40635f0 tx=0x7f21b4017000 comp rx=0 tx=0).stop 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 msgr2=0x7f21c406dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 0x7f21c406dee0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c7c0 tx=0x7f21c0007f70 comp rx=0 tx=0).stop 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 shutdown_connections 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f21a8067130 0x7f21a80695e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21c40c6520 0x7f21c40c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21c406b5c0 0x7f21c406dee0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 --2- 172.21.15.29:0/3140848324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21c4159790 0x7f21c40c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 >> 172.21.15.29:0/3140848324 conn(0x7f21c40c9410 msgr2=0x7f21c40c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 shutdown_connections 2023-10-07T11:03:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:47.539+0000 7f21cc5ff700 1 -- 172.21.15.29:0/3140848324 wait complete. 2023-10-07T11:03:47.560 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-07T11:03:47.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:03:47.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-07T11:03:47.753 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:47.753+0000 7f400223d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:03:47.757 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:47.757+0000 7f0c1210d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-07T11:03:47.779 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-07T11:03:47.779 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-07T11:03:47.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-07T11:03:47.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-07T11:03:47.870 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:03:47.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:47.871 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-07T11:03:47.935 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:47.936+0000 7f0c1210d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:03:47.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:47.946 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:47.946+0000 7f400223d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-07T11:03:48.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.146+0000 7fcbe6bff700 1 Processor -- start 2023-10-07T11:03:48.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.146+0000 7fcbe6bff700 1 -- start start 2023-10-07T11:03:48.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.147+0000 7fcbe6bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcbe015ade0 0x7fcbe00b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.147+0000 7fcbe6bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe0154090 con 0x7fcbe0153cc0 2023-10-07T11:03:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.147+0000 7fcbe6bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe0159f60 con 0x7fcbe00b0540 2023-10-07T11:03:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.147+0000 7fcbe6bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe00b09e0 con 0x7fcbe015ade0 2023-10-07T11:03:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbe499b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcbe0153cc0 0x7fcbe00b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48534/0 (socket says 172.21.15.29:48534) 2023-10-07T11:03:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbe499b700 1 -- 172.21.15.29:0/470435255 learned_addr learned my addr 172.21.15.29:0/470435255 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbdffff700 1 --2- 172.21.15.29:0/470435255 >> v2:172.21.15.144:3300/0 conn(0x7fcbe015ade0 0x7fcbe00b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbdf7fe700 1 -- 172.21.15.29:0/470435255 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035322644 0 0) 0x7fcbe0154090 con 0x7fcbe0153cc0 2023-10-07T11:03:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbdf7fe700 1 -- 172.21.15.29:0/470435255 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd0003570 con 0x7fcbe0153cc0 2023-10-07T11:03:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbdffff700 1 -- 172.21.15.29:0/470435255 >> v1:172.21.15.144:6789/0 conn(0x7fcbe00b0540 legacy=0x7fcbe015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbdffff700 1 -- 172.21.15.29:0/470435255 >> v1:172.21.15.29:6789/0 conn(0x7fcbe0153cc0 legacy=0x7fcbe00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.148+0000 7fcbdffff700 1 -- 172.21.15.29:0/470435255 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe015a240 con 0x7fcbe015ade0 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.149+0000 7fcbdffff700 1 --2- 172.21.15.29:0/470435255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe015ade0 0x7fcbe00b0000 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0098e0 tx=0x7fcbcc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0186af9c5d0509f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.149+0000 7fcbdf7fe700 1 -- 172.21.15.29:0/470435255 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc00b640 con 0x7fcbe015ade0 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.149+0000 7fcbdf7fe700 1 -- 172.21.15.29:0/470435255 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc00b7a0 con 0x7fcbe015ade0 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.149+0000 7fcbdf7fe700 1 -- 172.21.15.29:0/470435255 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc00bb40 con 0x7fcbe015ade0 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.149+0000 7fcbe6bff700 1 -- 172.21.15.29:0/470435255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe015ade0 msgr2=0x7fcbe00b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.149+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/470435255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe015ade0 0x7fcbe00b0000 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0098e0 tx=0x7fcbcc0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.150+0000 7fcbe6bff700 1 -- 172.21.15.29:0/470435255 shutdown_connections 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.150+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/470435255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe015ade0 0x7fcbe00b0000 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.150+0000 7fcbe6bff700 1 -- 172.21.15.29:0/470435255 >> 172.21.15.29:0/470435255 conn(0x7fcbe0067610 msgr2=0x7fcbe0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:48.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.150+0000 7fcbe6bff700 1 -- 172.21.15.29:0/470435255 shutdown_connections 2023-10-07T11:03:48.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.150+0000 7fcbe6bff700 1 -- 172.21.15.29:0/470435255 wait complete. 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.151+0000 7fcbe6bff700 1 Processor -- start 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.151+0000 7fcbe6bff700 1 -- start start 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.152+0000 7fcbe6bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbe00b0540 0x7fcbe015fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.152+0000 7fcbe6bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 0x7fcbe015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.152+0000 7fcbe6bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbe015ade0 0x7fcbe0142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.152+0000 7fcbe6bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe015b820 con 0x7fcbe015ade0 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.152+0000 7fcbe6bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe016a750 con 0x7fcbe0153cc0 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.152+0000 7fcbe6bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe0156740 con 0x7fcbe00b0540 2023-10-07T11:03:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 0x7fcbe015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbe519c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbe015ade0 0x7fcbe0142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 0x7fcbe015b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:42326/0 (socket says 172.21.15.29:42326) 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 -- 172.21.15.29:0/3587526689 learned_addr learned my addr 172.21.15.29:0/3587526689 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbe499b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbe00b0540 0x7fcbe015fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 -- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbe00b0540 msgr2=0x7fcbe015fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbe00b0540 0x7fcbe015fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 -- 172.21.15.29:0/3587526689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbe015ade0 msgr2=0x7fcbe0142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbe015ade0 0x7fcbe0142de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.153+0000 7fcbdffff700 1 -- 172.21.15.29:0/3587526689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc009420 con 0x7fcbe0153cc0 2023-10-07T11:03:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.154+0000 7fcbdffff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 0x7fcbe015b2e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc013000 tx=0x7fcbcc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.154+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc000d10 con 0x7fcbe0153cc0 2023-10-07T11:03:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.154+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc00bb20 con 0x7fcbe0153cc0 2023-10-07T11:03:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.154+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe0156a20 con 0x7fcbe0153cc0 2023-10-07T11:03:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.154+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc011650 con 0x7fcbe0153cc0 2023-10-07T11:03:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.154+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe0156fe0 con 0x7fcbe0153cc0 2023-10-07T11:03:48.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.155+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe01571d0 con 0x7fcbe0153cc0 2023-10-07T11:03:48.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.156+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fcbcc0119c0 con 0x7fcbe0153cc0 2023-10-07T11:03:48.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.157+0000 7fcbdcff9700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 0x7fcbc4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.157+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7fcbcc086130 con 0x7fcbe0153cc0 2023-10-07T11:03:48.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.157+0000 7fcbe499b700 1 -- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 msgr2=0x7fcbc4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:48.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.157+0000 7fcbe499b700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 0x7fcbc4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:48.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.161+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbcc052da0 con 0x7fcbe0153cc0 2023-10-07T11:03:48.253 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:48.252+0000 7f0c1210d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:03:48.291 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:48.291+0000 7f400223d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-07T11:03:48.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.358+0000 7fcbe499b700 1 -- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 msgr2=0x7fcbc4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:48.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.358+0000 7fcbe499b700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 0x7fcbc4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:48.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.409+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe004c3c0 con 0x7fcbe0153cc0 2023-10-07T11:03:48.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.410+0000 7fcbdcff9700 1 -- 172.21.15.29:0/3587526689 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4609 (secure 0 0 0) 0x7fcbcc0563c0 con 0x7fcbe0153cc0 2023-10-07T11:03:48.410 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:48.410 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:03:22.532644+0000","modified":"2023-10-07T11:03:26.067213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10104},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10104":{"gid":10104,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/1070134873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1070134873},{"type":"v1","addr":"172.21.15.29:6834","nonce":1070134873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.412+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 msgr2=0x7fcbc4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.412+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 0x7fcbc4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.412+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 msgr2=0x7fcbe015b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 0x7fcbe015b2e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc013000 tx=0x7fcbcc017000 comp rx=0 tx=0).stop 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 shutdown_connections 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbe015ade0 0x7fcbe0142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fcbc40670e0 0x7fcbc4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbe00b0540 0x7fcbe015fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 --2- 172.21.15.29:0/3587526689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbe0153cc0 0x7fcbe015b2e0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 >> 172.21.15.29:0/3587526689 conn(0x7fcbe0067610 msgr2=0x7fcbe00aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 shutdown_connections 2023-10-07T11:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.413+0000 7fcbe6bff700 1 -- 172.21.15.29:0/3587526689 wait complete. 2023-10-07T11:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 101 2023-10-07T11:03:48.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:48.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.631+0000 7f5291bff700 1 Processor -- start 2023-10-07T11:03:48.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.631+0000 7f5291bff700 1 -- start start 2023-10-07T11:03:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f5291bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f528c157290 0x7f528c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f5291bff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c158ad0 con 0x7f528c0b1a10 2023-10-07T11:03:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f5291bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c15b140 con 0x7f528c0ae8b0 2023-10-07T11:03:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f5291bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528c15b420 con 0x7f528c157290 2023-10-07T11:03:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f528bfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f528c0b1a10 0x7f528c157660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48542/0 (socket says 172.21.15.29:48542) 2023-10-07T11:03:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f528bfff700 1 -- 172.21.15.29:0/4219813361 learned_addr learned my addr 172.21.15.29:0/4219813361 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f528b7fe700 1 --2- 172.21.15.29:0/4219813361 >> v2:172.21.15.144:3300/0 conn(0x7f528c157290 0x7f528c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.633+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166958552 0 0) 0x7f528c158ad0 con 0x7f528c0b1a10 2023-10-07T11:03:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5274003570 con 0x7f528c0b1a10 2023-10-07T11:03:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091281462 0 0) 0x7f528c15b140 con 0x7f528c0ae8b0 2023-10-07T11:03:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c158ad0 con 0x7f528c0ae8b0 2023-10-07T11:03:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528b7fe700 1 -- 172.21.15.29:0/4219813361 >> v1:172.21.15.144:6789/0 conn(0x7f528c0ae8b0 legacy=0x7f528c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528b7fe700 1 -- 172.21.15.29:0/4219813361 >> v1:172.21.15.29:6789/0 conn(0x7f528c0b1a10 legacy=0x7f528c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528b7fe700 1 -- 172.21.15.29:0/4219813361 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c15b700 con 0x7f528c157290 2023-10-07T11:03:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528b7fe700 1 --2- 172.21.15.29:0/4219813361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c157290 0x7f528c0cc870 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f527c0098e0 tx=0x7f527c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19e9fc4bb9be2e34 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c011630 con 0x7f528c157290 2023-10-07T11:03:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527c011790 con 0x7f528c157290 2023-10-07T11:03:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.634+0000 7f528a7fc700 1 -- 172.21.15.29:0/4219813361 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c011b30 con 0x7f528c157290 2023-10-07T11:03:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 -- 172.21.15.29:0/4219813361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c157290 msgr2=0x7f528c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 --2- 172.21.15.29:0/4219813361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c157290 0x7f528c0cc870 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f527c0098e0 tx=0x7f527c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 -- 172.21.15.29:0/4219813361 shutdown_connections 2023-10-07T11:03:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 --2- 172.21.15.29:0/4219813361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c157290 0x7f528c0cc870 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 -- 172.21.15.29:0/4219813361 >> 172.21.15.29:0/4219813361 conn(0x7f528c0680a0 msgr2=0x7f528c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 -- 172.21.15.29:0/4219813361 shutdown_connections 2023-10-07T11:03:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.635+0000 7f5291bff700 1 -- 172.21.15.29:0/4219813361 wait complete. 2023-10-07T11:03:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.637+0000 7f5291bff700 1 Processor -- start 2023-10-07T11:03:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.637+0000 7f5291bff700 1 -- start start 2023-10-07T11:03:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f5291bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c0b1a10 0x7f528c148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f5291bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c157290 0x7f528c16b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f5291bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 0x7f528c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f5291bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c16aee0 con 0x7f528c06e780 2023-10-07T11:03:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f5291bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c14e300 con 0x7f528c0b1a10 2023-10-07T11:03:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f5291bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c06f140 con 0x7f528c157290 2023-10-07T11:03:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f528bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 0x7f528c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f528bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 0x7f528c06ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47590/0 (socket says 172.21.15.29:47590) 2023-10-07T11:03:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f528bfff700 1 -- 172.21.15.29:0/421912866 learned_addr learned my addr 172.21.15.29:0/421912866 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f528b7fe700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c0b1a10 0x7f528c148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.638+0000 7f528affd700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c157290 0x7f528c16b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:48.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f528bfff700 1 -- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c157290 msgr2=0x7f528c16b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f528bfff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c157290 0x7f528c16b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f528bfff700 1 -- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c0b1a10 msgr2=0x7f528c148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f528bfff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c0b1a10 0x7f528c148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f528bfff700 1 -- 172.21.15.29:0/421912866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c009420 con 0x7f528c06e780 2023-10-07T11:03:48.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f528bfff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 0x7f528c06ec00 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f528400cc70 tx=0x7f5284001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:48.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400a810 con 0x7f528c06e780 2023-10-07T11:03:48.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528400a970 con 0x7f528c06e780 2023-10-07T11:03:48.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400ad30 con 0x7f528c06e780 2023-10-07T11:03:48.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.639+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c14cb50 con 0x7f528c06e780 2023-10-07T11:03:48.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.640+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c16a9a0 con 0x7f528c06e780 2023-10-07T11:03:48.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.641+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528c14cce0 con 0x7f528c06e780 2023-10-07T11:03:48.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.642+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f5284007300 con 0x7f528c06e780 2023-10-07T11:03:48.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.642+0000 7f52737fe700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 0x7f5278069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:48.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.642+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f5284084e70 con 0x7f528c06e780 2023-10-07T11:03:48.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.643+0000 7f528b7fe700 1 -- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 msgr2=0x7f5278069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:48.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.643+0000 7f528b7fe700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 0x7f5278069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.647+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5284051b60 con 0x7f528c06e780 2023-10-07T11:03:48.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:48.772+0000 7f0c1210d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:03:48.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.844+0000 7f528b7fe700 1 -- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 msgr2=0x7f5278069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:48.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.844+0000 7f528b7fe700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 0x7f5278069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:48.853 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:48.849+0000 7f400223d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-07T11:03:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.852+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f528c04c3c0 con 0x7f528c06e780 2023-10-07T11:03:48.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.853+0000 7f52737fe700 1 -- 172.21.15.29:0/421912866 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f5284055180 con 0x7f528c06e780 2023-10-07T11:03:48.856 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:48.856 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":198,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:43.498905+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:03:20.783910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-07T11:03:21.517835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2912144557":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/471864733":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6837/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6836/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/245402490":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/3907222325":"2023-10-08T11:03:43.498860+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:48.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 msgr2=0x7f5278069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:48.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 0x7f5278069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 msgr2=0x7f528c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:48.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 0x7f528c06ec00 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f528400cc70 tx=0x7f5284001a60 comp rx=0 tx=0).stop 2023-10-07T11:03:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 shutdown_connections 2023-10-07T11:03:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f528c06e780 0x7f528c06ec00 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f528c0b1a10 0x7f528c148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f52780670e0 0x7f5278069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 --2- 172.21.15.29:0/421912866 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f528c157290 0x7f528c16b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 >> 172.21.15.29:0/421912866 conn(0x7f528c0680a0 msgr2=0x7f528c14af50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 shutdown_connections 2023-10-07T11:03:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:48.856+0000 7f5291bff700 1 -- 172.21.15.29:0/421912866 wait complete. 2023-10-07T11:03:48.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-07T09:29:10.085343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-07T11:03:20.783910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-07T11:03:21.517835+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-07T11:03:48.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-07T11:03:48.942 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:48.940+0000 7f0c1210d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:03:49.037 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:49.037+0000 7f400223d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-07T11:03:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.084+0000 7f063a3ff700 1 Processor -- start 2023-10-07T11:03:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.084+0000 7f063a3ff700 1 -- start start 2023-10-07T11:03:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063a3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f06341593e0 0x7f06340b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063a3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06341597b0 0x7f06340aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063a3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340b1e50 con 0x7f06340b0820 2023-10-07T11:03:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063a3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340b19c0 con 0x7f06341597b0 2023-10-07T11:03:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063a3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340b0db0 con 0x7f06341593e0 2023-10-07T11:03:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f06337fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06341597b0 0x7f06340aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063899c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f06340b0820 0x7f063415d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48558/0 (socket says 172.21.15.29:48558) 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f063899c700 1 -- 172.21.15.29:0/1380042331 learned_addr learned my addr 172.21.15.29:0/1380042331 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.086+0000 7f0633fff700 1 --2- 172.21.15.29:0/1380042331 >> v2:172.21.15.144:3300/0 conn(0x7f06341593e0 0x7f06340b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f06337fe700 1 -- 172.21.15.29:0/1380042331 >> v2:172.21.15.144:3300/0 conn(0x7f06341593e0 msgr2=0x7f06340b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f06337fe700 1 --2- 172.21.15.29:0/1380042331 >> v2:172.21.15.144:3300/0 conn(0x7f06341593e0 0x7f06340b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f06337fe700 1 -- 172.21.15.29:0/1380042331 >> v1:172.21.15.29:6789/0 conn(0x7f06340b0820 legacy=0x7f063415d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f0632ffd700 1 -- 172.21.15.29:0/1380042331 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244307410 0 0) 0x7f062c001490 con 0x7f06340b0820 2023-10-07T11:03:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f06337fe700 1 -- 172.21.15.29:0/1380042331 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340b1090 con 0x7f06341597b0 2023-10-07T11:03:49.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f06337fe700 1 --2- 172.21.15.29:0/1380042331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341597b0 0x7f06340aceb0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0634063d10 tx=0x7f06280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1ab5dd74cc4d67 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:49.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f0632ffd700 1 -- 172.21.15.29:0/1380042331 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062800b690 con 0x7f06341597b0 2023-10-07T11:03:49.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f0632ffd700 1 -- 172.21.15.29:0/1380042331 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062800b7f0 con 0x7f06341597b0 2023-10-07T11:03:49.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.087+0000 7f0632ffd700 1 -- 172.21.15.29:0/1380042331 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062800bb90 con 0x7f06341597b0 2023-10-07T11:03:49.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 -- 172.21.15.29:0/1380042331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341597b0 msgr2=0x7f06340aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1380042331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341597b0 0x7f06340aceb0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0634063d10 tx=0x7f06280091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 -- 172.21.15.29:0/1380042331 shutdown_connections 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1380042331 >> v2:172.21.15.144:3300/0 conn(0x7f06341593e0 0x7f06340b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1380042331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341597b0 0x7f06340aceb0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 -- 172.21.15.29:0/1380042331 >> 172.21.15.29:0/1380042331 conn(0x7f0634068090 msgr2=0x7f06340696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 -- 172.21.15.29:0/1380042331 shutdown_connections 2023-10-07T11:03:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.088+0000 7f063a3ff700 1 -- 172.21.15.29:0/1380042331 wait complete. 2023-10-07T11:03:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.090+0000 7f063a3ff700 1 Processor -- start 2023-10-07T11:03:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.090+0000 7f063a3ff700 1 -- start start 2023-10-07T11:03:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063a3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06340b0820 0x7f0634151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063a3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 0x7f0634148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063a3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06341597b0 0x7f063416d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063a3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634148f60 con 0x7f06341593e0 2023-10-07T11:03:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063a3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634151b90 con 0x7f06341597b0 2023-10-07T11:03:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063a3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0634151e70 con 0x7f06340b0820 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f06337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 0x7f0634148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f06337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 0x7f0634148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47612/0 (socket says 172.21.15.29:47612) 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f06337fe700 1 -- 172.21.15.29:0/1244606762 learned_addr learned my addr 172.21.15.29:0/1244606762 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f063899c700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06341597b0 0x7f063416d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.091+0000 7f0633fff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06340b0820 0x7f0634151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063899c700 1 -- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06340b0820 msgr2=0x7f0634151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063899c700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06340b0820 0x7f0634151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063899c700 1 -- 172.21.15.29:0/1244606762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 msgr2=0x7f0634148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063899c700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 0x7f0634148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063899c700 1 -- 172.21.15.29:0/1244606762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0628009420 con 0x7f06341597b0 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f06337fe700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 0x7f0634148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063899c700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06341597b0 0x7f063416d8e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f062c00cb90 tx=0x7f062c0018f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c00a780 con 0x7f06341597b0 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c00a8e0 con 0x7f06341597b0 2023-10-07T11:03:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0634152150 con 0x7f06341597b0 2023-10-07T11:03:49.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c00aca0 con 0x7f06341597b0 2023-10-07T11:03:49.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.092+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063416de20 con 0x7f06341597b0 2023-10-07T11:03:49.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.094+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063404c3c0 con 0x7f06341597b0 2023-10-07T11:03:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.095+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f062c007300 con 0x7f06341597b0 2023-10-07T11:03:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.096+0000 7f061bfff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 0x7f0620069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.096+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f062c084c10 con 0x7f06341597b0 2023-10-07T11:03:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.096+0000 7f0633fff700 1 -- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 msgr2=0x7f0620069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.096+0000 7f0633fff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 0x7f0620069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:49.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.099+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f062c051900 con 0x7f06341597b0 2023-10-07T11:03:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.297+0000 7f0633fff700 1 -- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 msgr2=0x7f0620069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.297+0000 7f0633fff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 0x7f0620069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:49.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.304+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06340b1e50 con 0x7f06341597b0 2023-10-07T11:03:49.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.305+0000 7f061bfff700 1 -- 172.21.15.29:0/1244606762 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v198) v1 ==== 93+0+10 (secure 0 0 0) 0x7f062c054f20 con 0x7f06341597b0 2023-10-07T11:03:49.304 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-07T11:03:49.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 msgr2=0x7f0620069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 0x7f0620069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06341597b0 msgr2=0x7f063416d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06341597b0 0x7f063416d8e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f062c00cb90 tx=0x7f062c0018f0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 shutdown_connections 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06341597b0 0x7f063416d8e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f06200670e0 0x7f0620069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06340b0820 0x7f0634151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 --2- 172.21.15.29:0/1244606762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06341593e0 0x7f0634148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 >> 172.21.15.29:0/1244606762 conn(0x7f0634068090 msgr2=0x7f063406d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 shutdown_connections 2023-10-07T11:03:49.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.307+0000 7f063a3ff700 1 -- 172.21.15.29:0/1244606762 wait complete. 2023-10-07T11:03:49.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-07T11:03:49.356 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:49.355+0000 7f0c1210d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:03:49.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:49.478+0000 7f400223d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-07T11:03:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.530+0000 7f41327ff700 1 Processor -- start 2023-10-07T11:03:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.530+0000 7f41327ff700 1 -- start start 2023-10-07T11:03:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f41327ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f412c0ae8a0 0x7f412c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f41327ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15ad30 0x7f412c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f41327ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0acb40 con 0x7f412c15a960 2023-10-07T11:03:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f41327ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c158c60 con 0x7f412c0ae8a0 2023-10-07T11:03:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f41327ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0abf30 con 0x7f412c15ad30 2023-10-07T11:03:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f412b7fe700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f412c0ae8a0 0x7f412c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f412bfff700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f412c15a960 0x7f412c0ae190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:51470/0 (socket says 172.21.15.29:51470) 2023-10-07T11:03:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f412bfff700 1 -- 172.21.15.29:0/548405053 learned_addr learned my addr 172.21.15.29:0/548405053 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.532+0000 7f4130d9c700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15ad30 0x7f412c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.533+0000 7f412b7fe700 1 -- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15ad30 msgr2=0x7f412c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.533+0000 7f412b7fe700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15ad30 0x7f412c0b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.533+0000 7f412b7fe700 1 -- 172.21.15.29:0/548405053 >> v1:172.21.15.144:6789/0 conn(0x7f412c15a960 legacy=0x7f412c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.533+0000 7f412b7fe700 1 -- 172.21.15.29:0/548405053 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c0ac210 con 0x7f412c0ae8a0 2023-10-07T11:03:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.533+0000 7f4130d9c700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15ad30 0x7f412c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.533+0000 7f412b7fe700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c0ae8a0 0x7f412c158e20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f41200098e0 tx=0x7f41200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3cf50fef097b13 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.535+0000 7f412affd700 1 -- 172.21.15.29:0/548405053 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000b640 con 0x7f412c0ae8a0 2023-10-07T11:03:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.535+0000 7f412affd700 1 -- 172.21.15.29:0/548405053 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412000b7a0 con 0x7f412c0ae8a0 2023-10-07T11:03:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.535+0000 7f412affd700 1 -- 172.21.15.29:0/548405053 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000bb40 con 0x7f412c0ae8a0 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 -- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c0ae8a0 msgr2=0x7f412c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c0ae8a0 0x7f412c158e20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f41200098e0 tx=0x7f41200091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 -- 172.21.15.29:0/548405053 shutdown_connections 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15ad30 0x7f412c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 --2- 172.21.15.29:0/548405053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c0ae8a0 0x7f412c158e20 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 -- 172.21.15.29:0/548405053 >> 172.21.15.29:0/548405053 conn(0x7f412c067610 msgr2=0x7f412c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 -- 172.21.15.29:0/548405053 shutdown_connections 2023-10-07T11:03:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.536+0000 7f41327ff700 1 -- 172.21.15.29:0/548405053 wait complete. 2023-10-07T11:03:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.538+0000 7f41327ff700 1 Processor -- start 2023-10-07T11:03:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.538+0000 7f41327ff700 1 -- start start 2023-10-07T11:03:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f41327ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 0x7f412c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f41327ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15a960 0x7f412c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f41327ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c15ad30 0x7f412c13eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f41327ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0cc600 con 0x7f412c0ae8a0 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f41327ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c14a5c0 con 0x7f412c15ad30 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f41327ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0ca980 con 0x7f412c15a960 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f412bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 0x7f412c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f412bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 0x7f412c0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47628/0 (socket says 172.21.15.29:47628) 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f412bfff700 1 -- 172.21.15.29:0/545973300 learned_addr learned my addr 172.21.15.29:0/545973300 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f412b7fe700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15a960 0x7f412c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.539+0000 7f4130d9c700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c15ad30 0x7f412c13eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f412bfff700 1 -- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15a960 msgr2=0x7f412c0cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f412bfff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15a960 0x7f412c0cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f412bfff700 1 -- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c15ad30 msgr2=0x7f412c13eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f412bfff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c15ad30 0x7f412c13eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f412bfff700 1 -- 172.21.15.29:0/545973300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4124009a20 con 0x7f412c0ae8a0 2023-10-07T11:03:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f412bfff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 0x7f412c0ca2a0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c7c0 tx=0x7f411c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c005b30 con 0x7f412c0ae8a0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c005c90 con 0x7f412c0ae8a0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c004a80 con 0x7f412c0ae8a0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4120009420 con 0x7f412c0ae8a0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.540+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c0caf40 con 0x7f412c0ae8a0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.542+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c04c3c0 con 0x7f412c0ae8a0 2023-10-07T11:03:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.542+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f411c0070d0 con 0x7f412c0ae8a0 2023-10-07T11:03:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.544+0000 7f4113fff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 0x7f4118069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.544+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f411c083ec0 con 0x7f412c0ae8a0 2023-10-07T11:03:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.544+0000 7f412b7fe700 1 -- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 msgr2=0x7f4118069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.544+0000 7f412b7fe700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 0x7f4118069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.548+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f411c04b5a0 con 0x7f412c0ae8a0 2023-10-07T11:03:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.745+0000 7f412b7fe700 1 -- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 msgr2=0x7f4118069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.745+0000 7f412b7fe700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 0x7f4118069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:49.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.756+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f412c0ac4f0 con 0x7f412c0ae8a0 2023-10-07T11:03:49.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.756+0000 7f4113fff700 1 -- 172.21.15.29:0/545973300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v198) v1 ==== 104+0+11 (secure 0 0 0) 0x7f411c050b30 con 0x7f412c0ae8a0 2023-10-07T11:03:49.758 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-07T11:03:49.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 msgr2=0x7f4118069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:49.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 0x7f4118069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 msgr2=0x7f412c0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 0x7f412c0ca2a0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c7c0 tx=0x7f411c007f70 comp rx=0 tx=0).stop 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 shutdown_connections 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f412c15ad30 0x7f412c13eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0ae8a0 0x7f412c0ca2a0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f41180670e0 0x7f4118069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 --2- 172.21.15.29:0/545973300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f412c15a960 0x7f412c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 >> 172.21.15.29:0/545973300 conn(0x7f412c067610 msgr2=0x7f412c159930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 shutdown_connections 2023-10-07T11:03:49.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.759+0000 7f41327ff700 1 -- 172.21.15.29:0/545973300 wait complete. 2023-10-07T11:03:49.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-07T11:03:49.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.977+0000 7f34d0dff700 1 Processor -- start 2023-10-07T11:03:49.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.977+0000 7f34d0dff700 1 -- start start 2023-10-07T11:03:49.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34d0dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34cc14c8d0 0x7f34cc0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34d0dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f34cc153cc0 0x7f34cc159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34d0dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc157790 con 0x7f34cc0b0410 2023-10-07T11:03:49.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34d0dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc0b0950 con 0x7f34cc14c8d0 2023-10-07T11:03:49.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34d0dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc157580 con 0x7f34cc153cc0 2023-10-07T11:03:49.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34ca59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34cc14c8d0 0x7f34cc0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34ca59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34cc14c8d0 0x7f34cc0afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47632/0 (socket says 172.21.15.29:47632) 2023-10-07T11:03:49.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34ca59c700 1 -- 172.21.15.29:0/2092217145 learned_addr learned my addr 172.21.15.29:0/2092217145 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:49.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.979+0000 7f34c9d9b700 1 --2- 172.21.15.29:0/2092217145 >> v2:172.21.15.144:3300/0 conn(0x7f34cc153cc0 0x7f34cc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34ca59c700 1 -- 172.21.15.29:0/2092217145 >> v2:172.21.15.144:3300/0 conn(0x7f34cc153cc0 msgr2=0x7f34cc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:49.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34ca59c700 1 --2- 172.21.15.29:0/2092217145 >> v2:172.21.15.144:3300/0 conn(0x7f34cc153cc0 0x7f34cc159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34ca59c700 1 -- 172.21.15.29:0/2092217145 >> v1:172.21.15.144:6789/0 conn(0x7f34cc0b0410 legacy=0x7f34cc158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34ca59c700 1 -- 172.21.15.29:0/2092217145 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc157a70 con 0x7f34cc14c8d0 2023-10-07T11:03:49.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34c959a700 1 -- 172.21.15.29:0/2092217145 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166175225 0 0) 0x7f34cc157790 con 0x7f34cc0b0410 2023-10-07T11:03:49.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34ca59c700 1 --2- 172.21.15.29:0/2092217145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc0afed0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0657f0 tx=0x7f34bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae8ad34b7e5421c server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:49.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34c959a700 1 -- 172.21.15.29:0/2092217145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc009e70 con 0x7f34cc14c8d0 2023-10-07T11:03:49.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34c959a700 1 -- 172.21.15.29:0/2092217145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc00b860 con 0x7f34cc14c8d0 2023-10-07T11:03:49.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.980+0000 7f34c959a700 1 -- 172.21.15.29:0/2092217145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00bc00 con 0x7f34cc14c8d0 2023-10-07T11:03:49.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 -- 172.21.15.29:0/2092217145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 msgr2=0x7f34cc0afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 --2- 172.21.15.29:0/2092217145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc0afed0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0657f0 tx=0x7f34bc009250 comp rx=0 tx=0).stop 2023-10-07T11:03:49.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 -- 172.21.15.29:0/2092217145 shutdown_connections 2023-10-07T11:03:49.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 --2- 172.21.15.29:0/2092217145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc0afed0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 --2- 172.21.15.29:0/2092217145 >> v2:172.21.15.144:3300/0 conn(0x7f34cc153cc0 0x7f34cc159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 -- 172.21.15.29:0/2092217145 >> 172.21.15.29:0/2092217145 conn(0x7f34cc067610 msgr2=0x7f34cc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:49.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 -- 172.21.15.29:0/2092217145 shutdown_connections 2023-10-07T11:03:49.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.981+0000 7f34d0dff700 1 -- 172.21.15.29:0/2092217145 wait complete. 2023-10-07T11:03:49.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.983+0000 7f34d0dff700 1 Processor -- start 2023-10-07T11:03:49.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.984+0000 7f34d0dff700 1 -- start start 2023-10-07T11:03:49.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34d0dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0b0410 0x7f34cc148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34d0dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34d0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 0x7f34cc158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34d0dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc14fef0 con 0x7f34cc14c8d0 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34d0dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc156690 con 0x7f34cc0b0410 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34d0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc14f2e0 con 0x7f34cc153cc0 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34cad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 0x7f34cc158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34cad9d700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 0x7f34cc158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43412/0 (socket says 172.21.15.29:43412) 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34cad9d700 1 -- 172.21.15.29:0/3323416204 learned_addr learned my addr 172.21.15.29:0/3323416204 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.985+0000 7f34ca59c700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0b0410 0x7f34cc148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34c9d9b700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34cad9d700 1 -- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0b0410 msgr2=0x7f34cc148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34cad9d700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0b0410 0x7f34cc148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34cad9d700 1 -- 172.21.15.29:0/3323416204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 msgr2=0x7f34cc159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34cad9d700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:49.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34cad9d700 1 -- 172.21.15.29:0/3323416204 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc0094f0 con 0x7f34cc153cc0 2023-10-07T11:03:49.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.986+0000 7f34cad9d700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 0x7f34cc158450 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c8c0 tx=0x7f34c4001670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:49.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.987+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4012630 con 0x7f34cc153cc0 2023-10-07T11:03:49.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.987+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4012790 con 0x7f34cc153cc0 2023-10-07T11:03:49.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.987+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc14f5c0 con 0x7f34cc153cc0 2023-10-07T11:03:49.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.987+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4012b50 con 0x7f34cc153cc0 2023-10-07T11:03:49.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.987+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc14fb80 con 0x7f34cc153cc0 2023-10-07T11:03:49.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.988+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc04c3c0 con 0x7f34cc153cc0 2023-10-07T11:03:49.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.989+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f34c401a3a0 con 0x7f34cc153cc0 2023-10-07T11:03:49.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.990+0000 7f34ba7fc700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 0x7f34ac069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:49.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.990+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f34c4084d40 con 0x7f34cc153cc0 2023-10-07T11:03:49.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.990+0000 7f34ca59c700 1 -- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 msgr2=0x7f34ac069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:49.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.990+0000 7f34ca59c700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 0x7f34ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:49.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:49.994+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34c40519b0 con 0x7f34cc153cc0 2023-10-07T11:03:50.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.130+0000 7f0c1210d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:03:50.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.191+0000 7f34ca59c700 1 -- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 msgr2=0x7f34ac069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:50.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.191+0000 7f34ca59c700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 0x7f34ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:50.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.197+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34cc0ccd00 con 0x7f34cc153cc0 2023-10-07T11:03:50.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.198+0000 7f34ba7fc700 1 -- 172.21.15.29:0/3323416204 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v198) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34c4054fd0 con 0x7f34cc153cc0 2023-10-07T11:03:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-07T11:03:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 msgr2=0x7f34ac069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 0x7f34ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 msgr2=0x7f34cc158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 0x7f34cc158450 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c8c0 tx=0x7f34c4001670 comp rx=0 tx=0).stop 2023-10-07T11:03:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 shutdown_connections 2023-10-07T11:03:50.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f34ac0670e0 0x7f34ac069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc153cc0 0x7f34cc158450 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0b0410 0x7f34cc148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 --2- 172.21.15.29:0/3323416204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc14c8d0 0x7f34cc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.200+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 >> 172.21.15.29:0/3323416204 conn(0x7f34cc067610 msgr2=0x7f34cc14aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:50.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.201+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 shutdown_connections 2023-10-07T11:03:50.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.201+0000 7f34d0dff700 1 -- 172.21.15.29:0/3323416204 wait complete. 2023-10-07T11:03:50.218 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.217+0000 7f0c1210d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:03:50.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:50.315 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:50.316+0000 7f400223d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-07T11:03:50.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.371+0000 7f0c1210d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:03:50.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:50.409+0000 7f400223d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-07T11:03:50.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.422+0000 7fe7571ff700 1 Processor -- start 2023-10-07T11:03:50.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.422+0000 7fe7571ff700 1 -- start start 2023-10-07T11:03:50.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.423+0000 7fe7571ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe7500ae8b0 0x7fe7500b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.423+0000 7fe7571ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500b1840 0x7fe750157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.423+0000 7fe7571ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe750158ad0 con 0x7fe750157290 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.423+0000 7fe7571ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7500b1cc0 con 0x7fe7500ae8b0 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.423+0000 7fe7571ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75015b140 con 0x7fe7500b1840 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe75579c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500b1840 0x7fe750157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe754f9b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe750157290 0x7fe7500cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:51492/0 (socket says 172.21.15.29:51492) 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe754f9b700 1 -- 172.21.15.29:0/9454932 learned_addr learned my addr 172.21.15.29:0/9454932 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe74ffff700 1 --2- 172.21.15.29:0/9454932 >> v2:172.21.15.144:3300/0 conn(0x7fe7500ae8b0 0x7fe7500b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe75579c700 1 -- 172.21.15.29:0/9454932 >> v2:172.21.15.144:3300/0 conn(0x7fe7500ae8b0 msgr2=0x7fe7500b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe75579c700 1 --2- 172.21.15.29:0/9454932 >> v2:172.21.15.144:3300/0 conn(0x7fe7500ae8b0 0x7fe7500b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe75579c700 1 -- 172.21.15.29:0/9454932 >> v1:172.21.15.144:6789/0 conn(0x7fe750157290 legacy=0x7fe7500cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe75579c700 1 -- 172.21.15.29:0/9454932 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75015b420 con 0x7fe7500b1840 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.424+0000 7fe74f7fe700 1 -- 172.21.15.29:0/9454932 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878969013 0 0) 0x7fe750158ad0 con 0x7fe750157290 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe75579c700 1 --2- 172.21.15.29:0/9454932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500b1840 0x7fe750157660 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe748009e60 tx=0x7fe748009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=174d94c2ef8d22ec server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe74f7fe700 1 -- 172.21.15.29:0/9454932 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74800e070 con 0x7fe7500b1840 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe74f7fe700 1 -- 172.21.15.29:0/9454932 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74800bb60 con 0x7fe7500b1840 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe74f7fe700 1 -- 172.21.15.29:0/9454932 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748012430 con 0x7fe7500b1840 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe7571ff700 1 -- 172.21.15.29:0/9454932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500b1840 msgr2=0x7fe750157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe7571ff700 1 --2- 172.21.15.29:0/9454932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500b1840 0x7fe750157660 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe748009e60 tx=0x7fe748009780 comp rx=0 tx=0).stop 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe7571ff700 1 -- 172.21.15.29:0/9454932 shutdown_connections 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe7571ff700 1 --2- 172.21.15.29:0/9454932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500b1840 0x7fe750157660 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.425+0000 7fe7571ff700 1 --2- 172.21.15.29:0/9454932 >> v2:172.21.15.144:3300/0 conn(0x7fe7500ae8b0 0x7fe7500b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.426+0000 7fe7571ff700 1 -- 172.21.15.29:0/9454932 >> 172.21.15.29:0/9454932 conn(0x7fe7500680a0 msgr2=0x7fe7500cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.426+0000 7fe7571ff700 1 -- 172.21.15.29:0/9454932 shutdown_connections 2023-10-07T11:03:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.426+0000 7fe7571ff700 1 -- 172.21.15.29:0/9454932 wait complete. 2023-10-07T11:03:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.428+0000 7fe7571ff700 1 Processor -- start 2023-10-07T11:03:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.428+0000 7fe7571ff700 1 -- start start 2023-10-07T11:03:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.429+0000 7fe7571ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7500b1840 0x7fe7500c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.429+0000 7fe7571ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 0x7fe7500b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.429+0000 7fe7571ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7500c6290 0x7fe7501502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.429+0000 7fe7571ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c7310 con 0x7fe7500c6290 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.429+0000 7fe7571ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c6700 con 0x7fe7500b1840 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.429+0000 7fe7571ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c69e0 con 0x7fe750157290 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe754f9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7500b1840 0x7fe7500c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 0x7fe7500b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe75579c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7500c6290 0x7fe7501502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 0x7fe7500b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:43430/0 (socket says 172.21.15.29:43430) 2023-10-07T11:03:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 -- 172.21.15.29:0/2399432760 learned_addr learned my addr 172.21.15.29:0/2399432760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 -- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7500b1840 msgr2=0x7fe7500c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7500b1840 0x7fe7500c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 -- 172.21.15.29:0/2399432760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7500c6290 msgr2=0x7fe7501502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7500c6290 0x7fe7501502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 -- 172.21.15.29:0/2399432760 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748009a20 con 0x7fe750157290 2023-10-07T11:03:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74ffff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 0x7fe7500b0030 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e930 tx=0x7fe73c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.430+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c013680 con 0x7fe750157290 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.431+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c0137e0 con 0x7fe750157290 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.431+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7500c6cc0 con 0x7fe750157290 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.431+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c013b80 con 0x7fe750157290 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.431+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7500c7ce0 con 0x7fe750157290 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.432+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75004c3c0 con 0x7fe750157290 2023-10-07T11:03:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.433+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7fe73c00c3e0 con 0x7fe750157290 2023-10-07T11:03:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.434+0000 7fe74cff9700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 0x7fe734069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.434+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7fe73c084c80 con 0x7fe750157290 2023-10-07T11:03:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.434+0000 7fe754f9b700 1 -- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 msgr2=0x7fe734069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.434+0000 7fe754f9b700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 0x7fe734069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:50.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.438+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe73c0518f0 con 0x7fe750157290 2023-10-07T11:03:50.461 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.460+0000 7f0c1210d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:03:50.576 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:50.576+0000 7f400223d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-07T11:03:50.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.635+0000 7fe754f9b700 1 -- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 msgr2=0x7fe734069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:50.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.635+0000 7fe754f9b700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 0x7fe734069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:50.670 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:50.669+0000 7f400223d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-07T11:03:50.679 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.678+0000 7f0c1210d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:03:50.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.691+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7500c7160 con 0x7fe750157290 2023-10-07T11:03:50.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.692+0000 7fe74cff9700 1 -- 172.21.15.29:0/2399432760 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4609 (secure 0 0 0) 0x7fe73c054f10 con 0x7fe750157290 2023-10-07T11:03:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:50.693 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:03:22.532644+0000","modified":"2023-10-07T11:03:26.067213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10104},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10104":{"gid":10104,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/1070134873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1070134873},{"type":"v1","addr":"172.21.15.29:6834","nonce":1070134873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-10-07T11:03:50.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.694+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 msgr2=0x7fe734069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.694+0000 7fe7571ff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 0x7fe734069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.694+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 msgr2=0x7fe7500b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.694+0000 7fe7571ff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 0x7fe7500b0030 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e930 tx=0x7fe73c0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 shutdown_connections 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7500c6290 0x7fe7501502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7500b1840 0x7fe7500c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe7340670e0 0x7fe734069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 --2- 172.21.15.29:0/2399432760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe750157290 0x7fe7500b0030 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 >> 172.21.15.29:0/2399432760 conn(0x7fe7500680a0 msgr2=0x7fe7500ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:50.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 shutdown_connections 2023-10-07T11:03:50.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.695+0000 7fe7571ff700 1 -- 172.21.15.29:0/2399432760 wait complete. 2023-10-07T11:03:50.697 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 101 2023-10-07T11:03:50.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:50.776 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.775+0000 7f0c1210d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:03:50.871 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.870+0000 7f0c1210d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:03:50.902 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:50.901+0000 7f400223d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-07T11:03:50.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.920+0000 7f36af3ff700 1 Processor -- start 2023-10-07T11:03:50.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.920+0000 7f36af3ff700 1 -- start start 2023-10-07T11:03:50.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.921+0000 7f36af3ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f36a80b2670 0x7f36a80b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.921+0000 7f36af3ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36a80a98d0 0x7f36a80a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.921+0000 7f36af3ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a806f4e0 con 0x7f36a8159550 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36af3ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a80b1120 con 0x7f36a80a98d0 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36af3ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a806e8d0 con 0x7f36a80b2670 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ad19b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f36a8159550 0x7f36a80b0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48570/0 (socket says 172.21.15.29:48570) 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ad19b700 1 -- 172.21.15.29:0/2173772977 learned_addr learned my addr 172.21.15.29:0/2173772977 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ad99c700 1 --2- 172.21.15.29:0/2173772977 >> v2:172.21.15.29:3300/0 conn(0x7f36a80a98d0 0x7f36a80a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ac99a700 1 --2- 172.21.15.29:0/2173772977 >> v2:172.21.15.144:3300/0 conn(0x7f36a80b2670 0x7f36a80b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f369ffff700 1 -- 172.21.15.29:0/2173772977 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226453954 0 0) 0x7f36a806f4e0 con 0x7f36a8159550 2023-10-07T11:03:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f369ffff700 1 -- 172.21.15.29:0/2173772977 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3694003570 con 0x7f36a8159550 2023-10-07T11:03:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ac99a700 1 -- 172.21.15.29:0/2173772977 >> v2:172.21.15.29:3300/0 conn(0x7f36a80a98d0 msgr2=0x7f36a80a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ac99a700 1 --2- 172.21.15.29:0/2173772977 >> v2:172.21.15.29:3300/0 conn(0x7f36a80a98d0 0x7f36a80a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ac99a700 1 -- 172.21.15.29:0/2173772977 >> v1:172.21.15.29:6789/0 conn(0x7f36a8159550 legacy=0x7f36a80b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ac99a700 1 -- 172.21.15.29:0/2173772977 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a806ebb0 con 0x7f36a80b2670 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.922+0000 7f36ad99c700 1 --2- 172.21.15.29:0/2173772977 >> v2:172.21.15.29:3300/0 conn(0x7f36a80a98d0 0x7f36a80a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.923+0000 7f36ac99a700 1 --2- 172.21.15.29:0/2173772977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a80b2670 0x7f36a80b2ae0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f36a80ce380 tx=0x7f36980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9e481a84a69d224 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.923+0000 7f369ffff700 1 -- 172.21.15.29:0/2173772977 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698011630 con 0x7f36a80b2670 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.923+0000 7f369ffff700 1 -- 172.21.15.29:0/2173772977 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36980117b0 con 0x7f36a80b2670 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.923+0000 7f369ffff700 1 -- 172.21.15.29:0/2173772977 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698011b70 con 0x7f36a80b2670 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 -- 172.21.15.29:0/2173772977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a80b2670 msgr2=0x7f36a80b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 --2- 172.21.15.29:0/2173772977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a80b2670 0x7f36a80b2ae0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f36a80ce380 tx=0x7f36980091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 -- 172.21.15.29:0/2173772977 shutdown_connections 2023-10-07T11:03:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 --2- 172.21.15.29:0/2173772977 >> v2:172.21.15.29:3300/0 conn(0x7f36a80a98d0 0x7f36a80a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 --2- 172.21.15.29:0/2173772977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a80b2670 0x7f36a80b2ae0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 -- 172.21.15.29:0/2173772977 >> 172.21.15.29:0/2173772977 conn(0x7f36a8069190 msgr2=0x7f36a806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 -- 172.21.15.29:0/2173772977 shutdown_connections 2023-10-07T11:03:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.924+0000 7f36af3ff700 1 -- 172.21.15.29:0/2173772977 wait complete. 2023-10-07T11:03:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.925+0000 7f36af3ff700 1 Processor -- start 2023-10-07T11:03:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 -- start start 2023-10-07T11:03:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36a80a98d0 0x7f36a8151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 0x7f36a80ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a8159550 0x7f36a8140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80ab3c0 con 0x7f36a80b2670 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80aa7b0 con 0x7f36a8159550 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.926+0000 7f36af3ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80aaa90 con 0x7f36a80a98d0 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 0x7f36a80ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 0x7f36a80ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47666/0 (socket says 172.21.15.29:47666) 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ad99c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a8159550 0x7f36a8140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 -- 172.21.15.29:0/1840559387 learned_addr learned my addr 172.21.15.29:0/1840559387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ad19b700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36a80a98d0 0x7f36a8151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 -- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36a80a98d0 msgr2=0x7f36a8151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36a80a98d0 0x7f36a8151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 -- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a8159550 msgr2=0x7f36a8140ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a8159550 0x7f36a8140ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ac99a700 1 -- 172.21.15.29:0/1840559387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009a20 con 0x7f36a80b2670 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.927+0000 7f36ad99c700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a8159550 0x7f36a8140ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.928+0000 7f36ac99a700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 0x7f36a80ab850 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f3698013000 tx=0x7f3698017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.928+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369801f070 con 0x7f36a80b2670 2023-10-07T11:03:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.928+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698000bc0 con 0x7f36a80b2670 2023-10-07T11:03:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.928+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3698009420 con 0x7f36a80b2670 2023-10-07T11:03:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.928+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698025420 con 0x7f36a80b2670 2023-10-07T11:03:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.928+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a80ab050 con 0x7f36a80b2670 2023-10-07T11:03:50.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.929+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a804c3c0 con 0x7f36a80b2670 2023-10-07T11:03:50.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.930+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f3698023070 con 0x7f36a80b2670 2023-10-07T11:03:50.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.931+0000 7f369d7fa700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 0x7f368c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:50.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.931+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f3698085eb0 con 0x7f36a80b2670 2023-10-07T11:03:50.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.931+0000 7f36ad19b700 1 -- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 msgr2=0x7f368c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:50.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.931+0000 7f36ad19b700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 0x7f368c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:50.939+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3698052be0 con 0x7f36a80b2670 2023-10-07T11:03:50.967 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:50.966+0000 7f0c1210d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:03:51.009 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.009+0000 7f400223d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-07T11:03:51.111 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.110+0000 7f400223d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-07T11:03:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.132+0000 7f36ad19b700 1 -- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 msgr2=0x7f368c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.132+0000 7f36ad19b700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 0x7f368c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:51.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.143+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36a80ab500 con 0x7f36a80b2670 2023-10-07T11:03:51.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.145+0000 7f369d7fa700 1 -- 172.21.15.29:0/1840559387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f369802a020 con 0x7f36a80b2670 2023-10-07T11:03:51.146 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:51.146 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":198,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:43.498905+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:03:20.783910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-07T11:03:21.517835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2912144557":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/471864733":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6837/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6836/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/245402490":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/3907222325":"2023-10-08T11:03:43.498860+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:51.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 msgr2=0x7f368c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:51.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 0x7f368c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 msgr2=0x7f36a80ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 0x7f36a80ab850 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f3698013000 tx=0x7f3698017000 comp rx=0 tx=0).stop 2023-10-07T11:03:51.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 shutdown_connections 2023-10-07T11:03:51.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36a8159550 0x7f36a8140ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f368c0670e0 0x7f368c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36a80a98d0 0x7f36a8151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 --2- 172.21.15.29:0/1840559387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36a80b2670 0x7f36a80ab850 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 >> 172.21.15.29:0/1840559387 conn(0x7f36a8069190 msgr2=0x7f36a806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 shutdown_connections 2023-10-07T11:03:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.147+0000 7f36af3ff700 1 -- 172.21.15.29:0/1840559387 wait complete. 2023-10-07T11:03:51.154 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:51.153+0000 7f0c1210d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:03:51.167 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T11:03:51.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:51.214 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.213+0000 7f400223d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-07T11:03:51.251 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:51.251+0000 7f0c1210d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:03:51.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.364+0000 7f0dcf9ff700 1 Processor -- start 2023-10-07T11:03:51.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.364+0000 7f0dcf9ff700 1 -- start start 2023-10-07T11:03:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.365+0000 7f0dcf9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0dc80ae8b0 0x7f0dc80b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.365+0000 7f0dcf9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc8158ad0 con 0x7f0dc8157290 2023-10-07T11:03:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.365+0000 7f0dcf9ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc815b140 con 0x7f0dc80b1840 2023-10-07T11:03:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.365+0000 7f0dcf9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc80b1cc0 con 0x7f0dc80ae8b0 2023-10-07T11:03:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dccf9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0dc80ae8b0 0x7f0dc80b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dcd79b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0dc8157290 0x7f0dc80cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48578/0 (socket says 172.21.15.29:48578) 2023-10-07T11:03:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dcd79b700 1 -- 172.21.15.29:0/1595238136 learned_addr learned my addr 172.21.15.29:0/1595238136 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dccf9a700 1 -- 172.21.15.29:0/1595238136 >> v1:172.21.15.144:6789/0 conn(0x7f0dc80b1840 legacy=0x7f0dc8157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dccf9a700 1 -- 172.21.15.29:0/1595238136 >> v1:172.21.15.29:6789/0 conn(0x7f0dc8157290 legacy=0x7f0dc80cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dccf9a700 1 -- 172.21.15.29:0/1595238136 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc815b420 con 0x7f0dc80ae8b0 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.366+0000 7f0dbffff700 1 -- 172.21.15.29:0/1595238136 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596967313 0 0) 0x7f0dc8158ad0 con 0x7f0dc8157290 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.367+0000 7f0dccf9a700 1 --2- 172.21.15.29:0/1595238136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc80ae8b0 0x7f0dc80b1300 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f0db80098e0 tx=0x7f0db80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3581b6e7d818513f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.367+0000 7f0dbffff700 1 -- 172.21.15.29:0/1595238136 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8009e80 con 0x7f0dc80ae8b0 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.367+0000 7f0dbffff700 1 -- 172.21.15.29:0/1595238136 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db800b860 con 0x7f0dc80ae8b0 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.367+0000 7f0dbffff700 1 -- 172.21.15.29:0/1595238136 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db800bbe0 con 0x7f0dc80ae8b0 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.367+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/1595238136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc80ae8b0 msgr2=0x7f0dc80b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.367+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/1595238136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc80ae8b0 0x7f0dc80b1300 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f0db80098e0 tx=0x7f0db80091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.368+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/1595238136 shutdown_connections 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.368+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/1595238136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc80ae8b0 0x7f0dc80b1300 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.368+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/1595238136 >> 172.21.15.29:0/1595238136 conn(0x7f0dc80680a0 msgr2=0x7f0dc80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.368+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/1595238136 shutdown_connections 2023-10-07T11:03:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.368+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/1595238136 wait complete. 2023-10-07T11:03:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.369+0000 7f0dcf9ff700 1 Processor -- start 2023-10-07T11:03:51.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 -- start start 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dc80b1840 0x7f0dc806d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 0x7f0dc806dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0dc8169940 0x7f0dc8169dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc8156490 con 0x7f0dc8157290 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc806e400 con 0x7f0dc80b1840 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.370+0000 7f0dcf9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc806e6e0 con 0x7f0dc8169940 2023-10-07T11:03:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 0x7f0dc806dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 0x7f0dc806dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47690/0 (socket says 172.21.15.29:47690) 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dcd79b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dc80b1840 0x7f0dc806d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 -- 172.21.15.29:0/2569361757 learned_addr learned my addr 172.21.15.29:0/2569361757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dcdf9c700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0dc8169940 0x7f0dc8169dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 -- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0dc8169940 msgr2=0x7f0dc8169dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0dc8169940 0x7f0dc8169dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 -- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dc80b1840 msgr2=0x7f0dc806d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dc80b1840 0x7f0dc806d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dccf9a700 1 -- 172.21.15.29:0/2569361757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8009420 con 0x7f0dc8157290 2023-10-07T11:03:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.371+0000 7f0dcd79b700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dc80b1840 0x7f0dc806d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.372+0000 7f0dccf9a700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 0x7f0dc806dbb0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8068dd0 tx=0x7f0db8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.372+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db801f070 con 0x7f0dc8157290 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.372+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8000c90 con 0x7f0dc8157290 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.372+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db800bbe0 con 0x7f0dc8157290 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.372+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc806e9c0 con 0x7f0dc8157290 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.372+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc806e0f0 con 0x7f0dc8157290 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.373+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc804c3c0 con 0x7f0dc8157290 2023-10-07T11:03:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.374+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f0db8011840 con 0x7f0dc8157290 2023-10-07T11:03:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.375+0000 7f0dbd7fa700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 0x7f0dac069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.375+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f0db8085df0 con 0x7f0dc8157290 2023-10-07T11:03:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.375+0000 7f0dcd79b700 1 -- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 msgr2=0x7f0dac069660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.375+0000 7f0dcd79b700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 0x7f0dac069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.383+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0db8052b90 con 0x7f0dc8157290 2023-10-07T11:03:51.420 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.419+0000 7f400223d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-07T11:03:51.523 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.523+0000 7f400223d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-07T11:03:51.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.576+0000 7f0dcd79b700 1 -- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 msgr2=0x7f0dac069660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:51.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.576+0000 7f0dcd79b700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 0x7f0dac069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:51.593 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:51.593+0000 7f0c1210d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:03:51.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.631+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dc80ca970 con 0x7f0dc8157290 2023-10-07T11:03:51.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.632+0000 7f0dbd7fa700 1 -- 172.21.15.29:0/2569361757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4609 (secure 0 0 0) 0x7f0db8028090 con 0x7f0dc8157290 2023-10-07T11:03:51.633 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:51.633 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:03:22.532644+0000","modified":"2023-10-07T11:03:26.067213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10104},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10104":{"gid":10104,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/1070134873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1070134873},{"type":"v1","addr":"172.21.15.29:6834","nonce":1070134873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-10-07T11:03:51.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 msgr2=0x7f0dac069660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:51.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 0x7f0dac069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 msgr2=0x7f0dc806dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 0x7f0dc806dbb0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8068dd0 tx=0x7f0db8017000 comp rx=0 tx=0).stop 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 shutdown_connections 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dac0671b0 0x7f0dac069660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0dc8169940 0x7f0dc8169dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dc80b1840 0x7f0dc806d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 --2- 172.21.15.29:0/2569361757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dc8157290 0x7f0dc806dbb0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 >> 172.21.15.29:0/2569361757 conn(0x7f0dc80680a0 msgr2=0x7f0dc80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 shutdown_connections 2023-10-07T11:03:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.634+0000 7f0dcf9ff700 1 -- 172.21.15.29:0/2569361757 wait complete. 2023-10-07T11:03:51.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 101 2023-10-07T11:03:51.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:03:51.682 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:51.681+0000 7f0c1210d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:51.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.860+0000 7f28653ff700 1 Processor -- start 2023-10-07T11:03:51.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.860+0000 7f28653ff700 1 -- start start 2023-10-07T11:03:51.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.861+0000 7f28653ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2860157290 0x7f28600cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.861+0000 7f28653ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28600ae8b0 0x7f28600b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.861+0000 7f28653ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28600b1840 0x7f28600ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f28653ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600b1cc0 con 0x7f28600ae8b0 2023-10-07T11:03:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f28653ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2860158ad0 con 0x7f2860157290 2023-10-07T11:03:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f28653ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286015b140 con 0x7f28600b1840 2023-10-07T11:03:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28600ae8b0 0x7f28600b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28600ae8b0 0x7f28600b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47702/0 (socket says 172.21.15.29:47702) 2023-10-07T11:03:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285effd700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2860157290 0x7f28600cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 -- 172.21.15.29:0/4154636968 learned_addr learned my addr 172.21.15.29:0/4154636968 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285f7fe700 1 --2- 172.21.15.29:0/4154636968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28600b1840 0x7f28600ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 -- 172.21.15.29:0/4154636968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28600b1840 msgr2=0x7f28600ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 --2- 172.21.15.29:0/4154636968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28600b1840 0x7f28600ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 -- 172.21.15.29:0/4154636968 >> v2:172.21.15.144:3300/0 conn(0x7f2860157290 msgr2=0x7f28600cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 --2- 172.21.15.29:0/4154636968 >> v2:172.21.15.144:3300/0 conn(0x7f2860157290 0x7f28600cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285e7fc700 1 -- 172.21.15.29:0/4154636968 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286015b420 con 0x7f28600ae8b0 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.862+0000 7f285effd700 1 --2- 172.21.15.29:0/4154636968 >> v2:172.21.15.144:3300/0 conn(0x7f2860157290 0x7f28600cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.863+0000 7f285e7fc700 1 --2- 172.21.15.29:0/4154636968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600ae8b0 0x7f28600b1300 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f28600685f0 tx=0x7f2854009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7e27a30ac09f77 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.863+0000 7f285dffb700 1 -- 172.21.15.29:0/4154636968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400b640 con 0x7f28600ae8b0 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.863+0000 7f285dffb700 1 -- 172.21.15.29:0/4154636968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285400b7a0 con 0x7f28600ae8b0 2023-10-07T11:03:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.863+0000 7f285dffb700 1 -- 172.21.15.29:0/4154636968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400bb60 con 0x7f28600ae8b0 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 -- 172.21.15.29:0/4154636968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600ae8b0 msgr2=0x7f28600b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 --2- 172.21.15.29:0/4154636968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600ae8b0 0x7f28600b1300 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f28600685f0 tx=0x7f2854009250 comp rx=0 tx=0).stop 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 -- 172.21.15.29:0/4154636968 shutdown_connections 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 --2- 172.21.15.29:0/4154636968 >> v2:172.21.15.144:3300/0 conn(0x7f2860157290 0x7f28600cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 --2- 172.21.15.29:0/4154636968 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28600b1840 0x7f28600ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 --2- 172.21.15.29:0/4154636968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600ae8b0 0x7f28600b1300 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 -- 172.21.15.29:0/4154636968 >> 172.21.15.29:0/4154636968 conn(0x7f28600680a0 msgr2=0x7f28600cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 -- 172.21.15.29:0/4154636968 shutdown_connections 2023-10-07T11:03:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.864+0000 7f28653ff700 1 -- 172.21.15.29:0/4154636968 wait complete. 2023-10-07T11:03:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.866+0000 7f28653ff700 1 Processor -- start 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.866+0000 7f28653ff700 1 -- start start 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.867+0000 7f28653ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 0x7f286006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.867+0000 7f28653ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2860157290 0x7f286014d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.867+0000 7f28653ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2860148f40 0x7f28600cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.867+0000 7f28653ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286014db40 con 0x7f28600b1840 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.867+0000 7f28653ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286014e960 con 0x7f2860148f40 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.867+0000 7f28653ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286014dd50 con 0x7f2860157290 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 0x7f286006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 0x7f286006e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47712/0 (socket says 172.21.15.29:47712) 2023-10-07T11:03:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 -- 172.21.15.29:0/1116259210 learned_addr learned my addr 172.21.15.29:0/1116259210 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285e7fc700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2860157290 0x7f286014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285f7fe700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2860148f40 0x7f28600cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 -- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2860157290 msgr2=0x7f286014d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2860157290 0x7f286014d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 -- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2860148f40 msgr2=0x7f28600cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2860148f40 0x7f28600cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.868+0000 7f285effd700 1 -- 172.21.15.29:0/1116259210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850009420 con 0x7f28600b1840 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.869+0000 7f285effd700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 0x7f286006e6b0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f285000ec80 tx=0x7f28500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.869+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285000c850 con 0x7f28600b1840 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.869+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285000c9b0 con 0x7f28600b1840 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.869+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285000cd30 con 0x7f28600b1840 2023-10-07T11:03:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.869+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540094f0 con 0x7f28600b1840 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.869+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286014e310 con 0x7f28600b1840 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.870+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286004c3c0 con 0x7f28600b1840 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.871+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f2850013680 con 0x7f28600b1840 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.872+0000 7f2846ffd700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 0x7f284c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.872+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f2850084e10 con 0x7f28600b1840 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.872+0000 7f285e7fc700 1 -- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 msgr2=0x7f284c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.872+0000 7f285e7fc700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 0x7f284c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:51.876+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2850088030 con 0x7f28600b1840 2023-10-07T11:03:51.887 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.887+0000 7f400223d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-07T11:03:51.982 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:51.981+0000 7f400223d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.073+0000 7f285e7fc700 1 -- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 msgr2=0x7f284c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.073+0000 7f285e7fc700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 0x7f284c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:52.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.128+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286015b700 con 0x7f28600b1840 2023-10-07T11:03:52.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.128+0000 7f2846ffd700 1 -- 172.21.15.29:0/1116259210 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4609 (secure 0 0 0) 0x7f2850055510 con 0x7f28600b1840 2023-10-07T11:03:52.130 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:52.130 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-07T11:03:22.532644+0000","modified":"2023-10-07T11:03:26.067213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10104},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10104":{"gid":10104,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6834/1070134873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1070134873},{"type":"v1","addr":"172.21.15.29:6834","nonce":1070134873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 msgr2=0x7f284c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 0x7f284c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 msgr2=0x7f286006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 0x7f286006e6b0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f285000ec80 tx=0x7f28500091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 shutdown_connections 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2860148f40 0x7f28600cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600b1840 0x7f286006e6b0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f284c0670e0 0x7f284c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 --2- 172.21.15.29:0/1116259210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2860157290 0x7f286014d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 >> 172.21.15.29:0/1116259210 conn(0x7f28600680a0 msgr2=0x7f2860152cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 shutdown_connections 2023-10-07T11:03:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.131+0000 7f28653ff700 1 -- 172.21.15.29:0/1116259210 wait complete. 2023-10-07T11:03:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 101 2023-10-07T11:03:52.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:52.218 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:52.217+0000 7f0c1210d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:03:52.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.347+0000 7f276a9ff700 1 Processor -- start 2023-10-07T11:03:52.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.347+0000 7f276a9ff700 1 -- start start 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.348+0000 7f276a9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2764157290 0x7f27640cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.348+0000 7f276a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27640ae8b0 0x7f27640b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.348+0000 7f276a9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27640b1ca0 con 0x7f27640b1840 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.348+0000 7f276a9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2764158ad0 con 0x7f2764157290 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.348+0000 7f276a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276415b140 con 0x7f27640ae8b0 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2764157290 0x7f27640cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2764157290 0x7f27640cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47714/0 (socket says 172.21.15.29:47714) 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 -- 172.21.15.29:0/1626862219 learned_addr learned my addr 172.21.15.29:0/1626862219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2768f9c700 1 --1- 172.21.15.29:0/1626862219 >> v1:172.21.15.29:6789/0 conn(0x7f27640b1840 0x7f27640ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48594/0 (socket says 172.21.15.29:48594) 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f27637fe700 1 --2- 172.21.15.29:0/1626862219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27640ae8b0 0x7f27640b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 -- 172.21.15.29:0/1626862219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27640ae8b0 msgr2=0x7f27640b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 --2- 172.21.15.29:0/1626862219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27640ae8b0 0x7f27640b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 -- 172.21.15.29:0/1626862219 >> v1:172.21.15.29:6789/0 conn(0x7f27640b1840 legacy=0x7f27640ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.349+0000 7f2763fff700 1 -- 172.21.15.29:0/1626862219 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276415b420 con 0x7f2764157290 2023-10-07T11:03:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.350+0000 7f2763fff700 1 --2- 172.21.15.29:0/1626862219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2764157290 0x7f27640cc870 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f27540099b0 tx=0x7f2754009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc414e6c761b5f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:52.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.350+0000 7f2762ffd700 1 -- 172.21.15.29:0/1626862219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2754009ea0 con 0x7f2764157290 2023-10-07T11:03:52.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.350+0000 7f2762ffd700 1 -- 172.21.15.29:0/1626862219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275400b860 con 0x7f2764157290 2023-10-07T11:03:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.350+0000 7f2762ffd700 1 -- 172.21.15.29:0/1626862219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400bc00 con 0x7f2764157290 2023-10-07T11:03:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.350+0000 7f276a9ff700 1 -- 172.21.15.29:0/1626862219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2764157290 msgr2=0x7f27640cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.350+0000 7f276a9ff700 1 --2- 172.21.15.29:0/1626862219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2764157290 0x7f27640cc870 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f27540099b0 tx=0x7f2754009250 comp rx=0 tx=0).stop 2023-10-07T11:03:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.351+0000 7f276a9ff700 1 -- 172.21.15.29:0/1626862219 shutdown_connections 2023-10-07T11:03:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.351+0000 7f276a9ff700 1 --2- 172.21.15.29:0/1626862219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2764157290 0x7f27640cc870 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.351+0000 7f276a9ff700 1 --2- 172.21.15.29:0/1626862219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27640ae8b0 0x7f27640b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.351+0000 7f276a9ff700 1 -- 172.21.15.29:0/1626862219 >> 172.21.15.29:0/1626862219 conn(0x7f27640680a0 msgr2=0x7f27640cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.351+0000 7f276a9ff700 1 -- 172.21.15.29:0/1626862219 shutdown_connections 2023-10-07T11:03:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.351+0000 7f276a9ff700 1 -- 172.21.15.29:0/1626862219 wait complete. 2023-10-07T11:03:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.352+0000 7f276a9ff700 1 Processor -- start 2023-10-07T11:03:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.352+0000 7f276a9ff700 1 -- start start 2023-10-07T11:03:52.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f276a9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27640b1840 0x7f2764153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f276a9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2764157290 0x7f276416d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f276a9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 0x7f27641456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f276a9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2764153d60 con 0x7f276415e7d0 2023-10-07T11:03:52.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f276a9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27640c8030 con 0x7f27640b1840 2023-10-07T11:03:52.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f276a9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276415dcf0 con 0x7f2764157290 2023-10-07T11:03:52.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.353+0000 7f2768f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 0x7f27641456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 0x7f27641456a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47720/0 (socket says 172.21.15.29:47720) 2023-10-07T11:03:52.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 -- 172.21.15.29:0/2955785967 learned_addr learned my addr 172.21.15.29:0/2955785967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:52.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f27637fe700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2764157290 0x7f276416d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2763fff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27640b1840 0x7f2764153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 -- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2764157290 msgr2=0x7f276416d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2764157290 0x7f276416d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 -- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27640b1840 msgr2=0x7f2764153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27640b1840 0x7f2764153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 -- 172.21.15.29:0/2955785967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27540094f0 con 0x7f276415e7d0 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f2768f9c700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 0x7f27641456a0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f275c00cd00 tx=0x7f275c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.354+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c00ce60 con 0x7f276415e7d0 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.355+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275c005df0 con 0x7f276415e7d0 2023-10-07T11:03:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.355+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c00ab00 con 0x7f276415e7d0 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.355+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276415dfd0 con 0x7f276415e7d0 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.355+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276415d9c0 con 0x7f276415e7d0 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.356+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276404c3c0 con 0x7f276415e7d0 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.357+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f275c0070f0 con 0x7f276415e7d0 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.357+0000 7f274bfff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 0x7f2750069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.358+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f275c084a20 con 0x7f276415e7d0 2023-10-07T11:03:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.358+0000 7f2763fff700 1 -- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 msgr2=0x7f2750069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:52.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.358+0000 7f2763fff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 0x7f2750069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:52.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.362+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f275c051710 con 0x7f276415e7d0 2023-10-07T11:03:52.404 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:52.404+0000 7f0c1210d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:03:52.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.559+0000 7f2763fff700 1 -- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 msgr2=0x7f2750069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:52.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.559+0000 7f2763fff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 0x7f2750069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:52.566 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:52.565+0000 7f400223d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-07T11:03:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.569+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27640b1ca0 con 0x7f276415e7d0 2023-10-07T11:03:52.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.570+0000 7f274bfff700 1 -- 172.21.15.29:0/2955785967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f27640b1ca0 con 0x7f276415e7d0 2023-10-07T11:03:52.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:52.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":198,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:43.498905+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-07T11:03:20.783910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-07T11:03:21.517835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2912144557":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/471864733":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6837/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6836/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/245402490":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/3907222325":"2023-10-08T11:03:43.498860+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 msgr2=0x7f2750069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 0x7f2750069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 msgr2=0x7f27641456a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 0x7f27641456a0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f275c00cd00 tx=0x7f275c001b10 comp rx=0 tx=0).stop 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 shutdown_connections 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f276415e7d0 0x7f27641456a0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27640b1840 0x7f2764153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27500670e0 0x7f2750069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 --2- 172.21.15.29:0/2955785967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2764157290 0x7f276416d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 >> 172.21.15.29:0/2955785967 conn(0x7f27640680a0 msgr2=0x7f2764159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:52.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 shutdown_connections 2023-10-07T11:03:52.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.572+0000 7f276a9ff700 1 -- 172.21.15.29:0/2955785967 wait complete. 2023-10-07T11:03:52.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-07T11:03:52.771 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:52.771+0000 7f400223d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-07T11:03:52.782 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:52.781+0000 7f0c1210d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:52.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.791+0000 7f89f5bff700 1 Processor -- start 2023-10-07T11:03:52.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.791+0000 7f89f5bff700 1 -- start start 2023-10-07T11:03:52.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.792+0000 7f89f5bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f015a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.792+0000 7f89f5bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f89f00af400 0x7f89f00af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.792+0000 7f89f5bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f014d790 con 0x7f89f00ad370 2023-10-07T11:03:52.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.792+0000 7f89f5bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00ae5d0 con 0x7f89f00af400 2023-10-07T11:03:52.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.792+0000 7f89f5bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f015a850 con 0x7f89f0159ec0 2023-10-07T11:03:52.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89ef7fe700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f89f00ad370 0x7f89f00cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:51512/0 (socket says 172.21.15.29:51512) 2023-10-07T11:03:52.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89ef7fe700 1 -- 172.21.15.29:0/1307977591 learned_addr learned my addr 172.21.15.29:0/1307977591 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:52.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89effff700 1 --2- 172.21.15.29:0/1307977591 >> v2:172.21.15.144:3300/0 conn(0x7f89f00af400 0x7f89f00af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89eeffd700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89ee7fc700 1 -- 172.21.15.29:0/1307977591 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82500871 0 0) 0x7f89f014d790 con 0x7f89f00ad370 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89ee7fc700 1 -- 172.21.15.29:0/1307977591 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d8003570 con 0x7f89f00ad370 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89effff700 1 -- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 msgr2=0x7f89f015a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89effff700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f015a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89effff700 1 -- 172.21.15.29:0/1307977591 >> v1:172.21.15.144:6789/0 conn(0x7f89f00ad370 legacy=0x7f89f00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89effff700 1 -- 172.21.15.29:0/1307977591 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00ad740 con 0x7f89f00af400 2023-10-07T11:03:52.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89eeffd700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.793+0000 7f89effff700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00af400 0x7f89f00af810 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f89e8009e60 tx=0x7f89e8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3dfe5617154d5f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89ee7fc700 1 -- 172.21.15.29:0/1307977591 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e800e070 con 0x7f89f00af400 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89ee7fc700 1 -- 172.21.15.29:0/1307977591 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e800bab0 con 0x7f89f00af400 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89ee7fc700 1 -- 172.21.15.29:0/1307977591 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e800be70 con 0x7f89f00af400 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 -- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00af400 msgr2=0x7f89f00af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00af400 0x7f89f00af810 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f89e8009e60 tx=0x7f89e8009780 comp rx=0 tx=0).stop 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 -- 172.21.15.29:0/1307977591 shutdown_connections 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00af400 0x7f89f00af810 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 --2- 172.21.15.29:0/1307977591 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 -- 172.21.15.29:0/1307977591 >> 172.21.15.29:0/1307977591 conn(0x7f89f00680a0 msgr2=0x7f89f00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.794+0000 7f89f5bff700 1 -- 172.21.15.29:0/1307977591 shutdown_connections 2023-10-07T11:03:52.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.795+0000 7f89f5bff700 1 -- 172.21.15.29:0/1307977591 wait complete. 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.796+0000 7f89f5bff700 1 Processor -- start 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.796+0000 7f89f5bff700 1 -- start start 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89f5bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00ad370 0x7f89f0143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89f5bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 0x7f89f014df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89f5bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89f5bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f014f520 con 0x7f89f00af400 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89f5bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f014e910 con 0x7f89f00ad370 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89f5bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f014ebf0 con 0x7f89f0159ec0 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89eeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 0x7f89f014df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89ef7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00ad370 0x7f89f0143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89eeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 0x7f89f014df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47728/0 (socket says 172.21.15.29:47728) 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89eeffd700 1 -- 172.21.15.29:0/3350563590 learned_addr learned my addr 172.21.15.29:0/3350563590 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.797+0000 7f89effff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89eeffd700 1 -- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 msgr2=0x7f89f006d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89eeffd700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f006d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89eeffd700 1 -- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00ad370 msgr2=0x7f89f0143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89eeffd700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00ad370 0x7f89f0143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89eeffd700 1 -- 172.21.15.29:0/3350563590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e400b6d0 con 0x7f89f00af400 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89ef7fe700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00ad370 0x7f89f0143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89eeffd700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 0x7f89f014df20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f89e400ede0 tx=0x7f89e400b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e400eee0 con 0x7f89f00af400 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e400dd30 con 0x7f89f00af400 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e400dea0 con 0x7f89f00af400 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.798+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e8009a20 con 0x7f89f00af400 2023-10-07T11:03:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.799+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f014f1b0 con 0x7f89f00af400 2023-10-07T11:03:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.800+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f014f6d0 con 0x7f89f00af400 2023-10-07T11:03:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.800+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f89e40108a0 con 0x7f89f00af400 2023-10-07T11:03:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.801+0000 7f89d77fe700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 0x7f89dc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.801+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6439+0+0 (secure 0 0 0) 0x7f89e4084fe0 con 0x7f89f00af400 2023-10-07T11:03:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.802+0000 7f89ef7fe700 1 -- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 msgr2=0x7f89dc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.802+0000 7f89ef7fe700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 0x7f89dc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:52.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:52.805+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89f014f6d0 con 0x7f89f00af400 2023-10-07T11:03:53.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.003+0000 7f89ef7fe700 1 -- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 msgr2=0x7f89dc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:53.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.003+0000 7f89ef7fe700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 0x7f89dc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.052+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f89f014f6d0 con 0x7f89f00af400 2023-10-07T11:03:53.053 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-07T11:03:53.052+0000 7fa40b5f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-07T11:03:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.058+0000 7f89d77fe700 1 -- 172.21.15.29:0/3350563590 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7f89f014f6d0 con 0x7f89f00af400 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 msgr2=0x7f89dc069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 0x7f89dc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 msgr2=0x7f89f014df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 0x7f89f014df20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f89e400ede0 tx=0x7f89e400b430 comp rx=0 tx=0).stop 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 shutdown_connections 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89dc0670e0 0x7f89dc069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f0159ec0 0x7f89f006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00ad370 0x7f89f0143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 --2- 172.21.15.29:0/3350563590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00af400 0x7f89f014df20 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 >> 172.21.15.29:0/3350563590 conn(0x7f89f00680a0 msgr2=0x7f89f014cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 shutdown_connections 2023-10-07T11:03:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.062+0000 7f89f5bff700 1 -- 172.21.15.29:0/3350563590 wait complete. 2023-10-07T11:03:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-07T11:03:53.067 INFO:tasks.ceph.mds.c.smithi029.stderr: -2511> 2023-10-07T11:03:28.022+0000 7f903c3f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-07T11:03:53.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-07T11:03:53.116 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-07T11:03:53.170 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:53.169+0000 7f400223d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:53.209 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.208+0000 7f0c1210d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:03:53.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.290+0000 7f0f6c7ff700 1 Processor -- start 2023-10-07T11:03:53.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.290+0000 7f0f6c7ff700 1 -- start start 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6c7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f640b0710 0x7f0f6415c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6c7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f64154090 con 0x7f0f64153cc0 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6c7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f64159f60 con 0x7f0f6415ade0 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6c7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6415a240 con 0x7f0f640b0710 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6ad9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f640b0710 0x7f0f6415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6a59b700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0f64153cc0 0x7f0f640b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41926/0 (socket says 172.21.15.29:41926) 2023-10-07T11:03:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6a59b700 1 -- 172.21.15.29:0/2663250132 learned_addr learned my addr 172.21.15.29:0/2663250132 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f6ad9c700 1 -- 172.21.15.29:0/2663250132 >> v1:172.21.15.144:6789/0 conn(0x7f0f6415ade0 legacy=0x7f0f640b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.293+0000 7f0f6ad9c700 1 -- 172.21.15.29:0/2663250132 >> v1:172.21.15.29:6789/0 conn(0x7f0f64153cc0 legacy=0x7f0f640b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.293+0000 7f0f6ad9c700 1 -- 172.21.15.29:0/2663250132 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6415a520 con 0x7f0f640b0710 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.292+0000 7f0f69599700 1 -- 172.21.15.29:0/2663250132 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530475303 0 0) 0x7f0f64159f60 con 0x7f0f6415ade0 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.293+0000 7f0f6ad9c700 1 --2- 172.21.15.29:0/2663250132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640b0710 0x7f0f6415c240 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009e60 tx=0x7f0f5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a85760db60f37e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.293+0000 7f0f69599700 1 -- 172.21.15.29:0/2663250132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c00e070 con 0x7f0f640b0710 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.293+0000 7f0f69599700 1 -- 172.21.15.29:0/2663250132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c00bc30 con 0x7f0f640b0710 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.293+0000 7f0f69599700 1 -- 172.21.15.29:0/2663250132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c012430 con 0x7f0f640b0710 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/2663250132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640b0710 msgr2=0x7f0f6415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/2663250132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640b0710 0x7f0f6415c240 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009e60 tx=0x7f0f5c009780 comp rx=0 tx=0).stop 2023-10-07T11:03:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/2663250132 shutdown_connections 2023-10-07T11:03:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/2663250132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640b0710 0x7f0f6415c240 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/2663250132 >> 172.21.15.29:0/2663250132 conn(0x7f0f64067610 msgr2=0x7f0f64067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/2663250132 shutdown_connections 2023-10-07T11:03:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.294+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/2663250132 wait complete. 2023-10-07T11:03:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.295+0000 7f0f6c7ff700 1 Processor -- start 2023-10-07T11:03:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.296+0000 7f0f6c7ff700 1 -- start start 2023-10-07T11:03:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.296+0000 7f0f6c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f640b0710 0x7f0f64146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.296+0000 7f0f6c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 0x7f0f64154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.296+0000 7f0f6c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f6415ade0 0x7f0f641575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f6c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f64157af0 con 0x7f0f64153cc0 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f6c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f64155ce0 con 0x7f0f6415ade0 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f6c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f64169830 con 0x7f0f640b0710 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 0x7f0f64154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:53.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f6a59b700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f640b0710 0x7f0f64146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 0x7f0f64154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43376/0 (socket says 172.21.15.29:43376) 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f6ad9c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f6415ade0 0x7f0f641575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 -- 172.21.15.29:0/1116200906 learned_addr learned my addr 172.21.15.29:0/1116200906 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 -- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f640b0710 msgr2=0x7f0f64146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f640b0710 0x7f0f64146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 -- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f6415ade0 msgr2=0x7f0f641575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f6415ade0 0x7f0f641575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.297+0000 7f0f69d9a700 1 -- 172.21.15.29:0/1116200906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c009a20 con 0x7f0f64153cc0 2023-10-07T11:03:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.298+0000 7f0f69d9a700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 0x7f0f64154db0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400c4f0 tx=0x7f0f54007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.298+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400cea0 con 0x7f0f64153cc0 2023-10-07T11:03:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.298+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54011590 con 0x7f0f64153cc0 2023-10-07T11:03:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.298+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54011950 con 0x7f0f64153cc0 2023-10-07T11:03:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.298+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f64155440 con 0x7f0f64153cc0 2023-10-07T11:03:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.298+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f64155a00 con 0x7f0f64153cc0 2023-10-07T11:03:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.299+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f64169640 con 0x7f0f64153cc0 2023-10-07T11:03:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.300+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 84899+0+0 (secure 0 0 0) 0x7f0f54011d40 con 0x7f0f64153cc0 2023-10-07T11:03:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.301+0000 7f0f5a7fc700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 0x7f0f48069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.301+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f0f54083f50 con 0x7f0f64153cc0 2023-10-07T11:03:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.301+0000 7f0f6a59b700 1 -- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 msgr2=0x7f0f48069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.301+0000 7f0f6a59b700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 0x7f0f48069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-07T11:03:53.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.305+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f64169640 con 0x7f0f64153cc0 2023-10-07T11:03:53.392 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.391+0000 7f0c1210d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:53.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.502+0000 7f0f6a59b700 1 -- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 msgr2=0x7f0f48069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6836/131978 2023-10-07T11:03:53.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.502+0000 7f0f6a59b700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 0x7f0f48069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-07T11:03:53.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.555+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0f64169640 con 0x7f0f64153cc0 2023-10-07T11:03:53.579 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.578+0000 7f0c1210d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:03:53.625 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:53.625+0000 7f400223d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-07T11:03:53.689 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.688+0000 7f0c1210d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:53.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.697+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 104) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f0f54050f40 con 0x7f0f64153cc0 2023-10-07T11:03:53.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.698+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 msgr2=0x7f0f48069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-07T11:03:53.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:53.698+0000 7f0f5a7fc700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 0x7f0f48069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.772+0000 7f0b8c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.772+0000 7f0b8c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.772+0000 7f0b8c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.772+0000 7f0b8c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.772+0000 7f0b8c783700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.773+0000 7f0b8f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.773 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.773+0000 7f0b8f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.774 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.773+0000 7f0b8f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.774 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.773+0000 7f0b8f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.774 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-07T11:03:53.773+0000 7f0b8f789700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-07T11:03:53.824 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:53.823+0000 7f400223d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-07T11:03:54.019 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:54.018+0000 7f400223d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-07T11:03:54.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.058+0000 7f0f5a7fc700 1 -- 172.21.15.29:0/1116200906 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v103) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0f5406eb20 con 0x7f0f64153cc0 2023-10-07T11:03:54.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.060+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 msgr2=0x7f0f64154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.060+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 0x7f0f64154db0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400c4f0 tx=0x7f0f54007f00 comp rx=0 tx=0).stop 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 shutdown_connections 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f6415ade0 0x7f0f641575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0f480670e0 0x7f0f48069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f640b0710 0x7f0f64146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 --2- 172.21.15.29:0/1116200906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f64153cc0 0x7f0f64154db0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 >> 172.21.15.29:0/1116200906 conn(0x7f0f64067610 msgr2=0x7f0f640af320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 shutdown_connections 2023-10-07T11:03:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.061+0000 7f0f6c7ff700 1 -- 172.21.15.29:0/1116200906 wait complete. 2023-10-07T11:03:54.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-07T11:03:54.137 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-07T11:03:54.137+0000 7f400223d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-07T11:03:54.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.319+0000 7f29d8fff700 1 Processor -- start 2023-10-07T11:03:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.319+0000 7f29d8fff700 1 -- start start 2023-10-07T11:03:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d8fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29d40b0610 0x7f29d40ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d8fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f29d40b09e0 0x7f29d40c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d8fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40c8a20 0x7f29d40cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d8fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40690f0 con 0x7f29d40b0610 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d8fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40c4740 con 0x7f29d40b09e0 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d8fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40cf2e0 con 0x7f29d40c8a20 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29d40b0610 0x7f29d40ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d259c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29d40b0610 0x7f29d40ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43384/0 (socket says 172.21.15.29:43384) 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d259c700 1 -- 172.21.15.29:0/3769146488 learned_addr learned my addr 172.21.15.29:0/3769146488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d2d9d700 1 --2- 172.21.15.29:0/3769146488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40c8a20 0x7f29d40cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.320+0000 7f29d1d9b700 1 --2- 172.21.15.29:0/3769146488 >> v2:172.21.15.144:3300/0 conn(0x7f29d40b09e0 0x7f29d40c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d259c700 1 -- 172.21.15.29:0/3769146488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40c8a20 msgr2=0x7f29d40cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d259c700 1 --2- 172.21.15.29:0/3769146488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40c8a20 0x7f29d40cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d259c700 1 -- 172.21.15.29:0/3769146488 >> v2:172.21.15.144:3300/0 conn(0x7f29d40b09e0 msgr2=0x7f29d40c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d259c700 1 --2- 172.21.15.29:0/3769146488 >> v2:172.21.15.144:3300/0 conn(0x7f29d40b09e0 0x7f29d40c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d259c700 1 -- 172.21.15.29:0/3769146488 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d40687c0 con 0x7f29d40b0610 2023-10-07T11:03:54.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d259c700 1 --2- 172.21.15.29:0/3769146488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40b0610 0x7f29d40ced00 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cc2f0 tx=0x7f29c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dcb39d35b692b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:54.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d159a700 1 -- 172.21.15.29:0/3769146488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400b640 con 0x7f29d40b0610 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d159a700 1 -- 172.21.15.29:0/3769146488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c400b7a0 con 0x7f29d40b0610 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.321+0000 7f29d159a700 1 -- 172.21.15.29:0/3769146488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400bb60 con 0x7f29d40b0610 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 -- 172.21.15.29:0/3769146488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40b0610 msgr2=0x7f29d40ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 --2- 172.21.15.29:0/3769146488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40b0610 0x7f29d40ced00 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cc2f0 tx=0x7f29c4009250 comp rx=0 tx=0).stop 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 -- 172.21.15.29:0/3769146488 shutdown_connections 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 --2- 172.21.15.29:0/3769146488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40c8a20 0x7f29d40cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 --2- 172.21.15.29:0/3769146488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40b0610 0x7f29d40ced00 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 --2- 172.21.15.29:0/3769146488 >> v2:172.21.15.144:3300/0 conn(0x7f29d40b09e0 0x7f29d40c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 -- 172.21.15.29:0/3769146488 >> 172.21.15.29:0/3769146488 conn(0x7f29d4063a10 msgr2=0x7f29d40ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 -- 172.21.15.29:0/3769146488 shutdown_connections 2023-10-07T11:03:54.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.322+0000 7f29d8fff700 1 -- 172.21.15.29:0/3769146488 wait complete. 2023-10-07T11:03:54.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.324+0000 7f29d8fff700 1 Processor -- start 2023-10-07T11:03:54.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.324+0000 7f29d8fff700 1 -- start start 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d8fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d40b0610 0x7f29d414b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d8fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40b09e0 0x7f29d409b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d8fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 0x7f29d40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d8fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d409b340 con 0x7f29d40c8a20 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d8fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40b1c20 con 0x7f29d40b0610 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d8fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40b1f00 con 0x7f29d40b09e0 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 0x7f29d40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 0x7f29d40b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43398/0 (socket says 172.21.15.29:43398) 2023-10-07T11:03:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 -- 172.21.15.29:0/282261198 learned_addr learned my addr 172.21.15.29:0/282261198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d1d9b700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40b09e0 0x7f29d409b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d259c700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d40b0610 0x7f29d414b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 -- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40b09e0 msgr2=0x7f29d409b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40b09e0 0x7f29d409b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 -- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d40b0610 msgr2=0x7f29d414b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d40b0610 0x7f29d414b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.325+0000 7f29d2d9d700 1 -- 172.21.15.29:0/282261198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40094f0 con 0x7f29d40c8a20 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.326+0000 7f29d2d9d700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 0x7f29d40b1180 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e90 tx=0x7f29cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.326+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc00cc20 con 0x7f29d40c8a20 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.326+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc00f040 con 0x7f29d40c8a20 2023-10-07T11:03:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.326+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc014480 con 0x7f29d40c8a20 2023-10-07T11:03:54.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.326+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d409acf0 con 0x7f29d40c8a20 2023-10-07T11:03:54.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.326+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d40981d0 con 0x7f29d40c8a20 2023-10-07T11:03:54.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.327+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d40cbe70 con 0x7f29d40c8a20 2023-10-07T11:03:54.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.328+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 104) v1 ==== 84451+0+0 (secure 0 0 0) 0x7f29cc014830 con 0x7f29d40c8a20 2023-10-07T11:03:54.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.329+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6826+0+0 (secure 0 0 0) 0x7f29cc084f70 con 0x7f29d40c8a20 2023-10-07T11:03:54.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.333+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f29cc051e80 con 0x7f29d40c8a20 2023-10-07T11:03:54.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.535+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f29d40718b0 con 0x7f29d40c8a20 2023-10-07T11:03:54.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.700+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 105) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f29cc051b00 con 0x7f29d40c8a20 2023-10-07T11:03:54.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.701+0000 7f29c27fc700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f29b406cb30 0x7f29b406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:54.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.702+0000 7f29d259c700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f29b406cb30 0x7f29b406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:54.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:54.705+0000 7f29d259c700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f29b406cb30 0x7f29b406ef10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cc2f0 tx=0x7f29c4017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:55.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.061+0000 7f29c27fc700 1 -- 172.21.15.29:0/282261198 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v201) v1 ==== 181+0+0 (secure 0 0 0) 0x7f29cc07da40 con 0x7f29d40c8a20 2023-10-07T11:03:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f29b406cb30 msgr2=0x7f29b406ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f29b406cb30 0x7f29b406ef10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cc2f0 tx=0x7f29c4017000 comp rx=0 tx=0).stop 2023-10-07T11:03:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 msgr2=0x7f29d40b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 0x7f29d40b1180 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e90 tx=0x7f29cc00b040 comp rx=0 tx=0).stop 2023-10-07T11:03:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 shutdown_connections 2023-10-07T11:03:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d40c8a20 0x7f29d40b1180 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d40b0610 0x7f29d414b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f29b406cb30 0x7f29b406ef10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 --2- 172.21.15.29:0/282261198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d40b09e0 0x7f29d409b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.066+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 >> 172.21.15.29:0/282261198 conn(0x7f29d4063a10 msgr2=0x7f29d40c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.067+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 shutdown_connections 2023-10-07T11:03:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.067+0000 7f29d8fff700 1 -- 172.21.15.29:0/282261198 wait complete. 2023-10-07T11:03:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-07T11:03:55.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-07T11:03:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.330+0000 7f0311fff700 1 Processor -- start 2023-10-07T11:03:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.330+0000 7f0311fff700 1 -- start start 2023-10-07T11:03:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.331+0000 7f0311fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f030c0b0610 0x7f030c0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f0311fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c0690f0 con 0x7f030c0c8a20 2023-10-07T11:03:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f0311fff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c0cf2e0 con 0x7f030c0b09e0 2023-10-07T11:03:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f0311fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c0687c0 con 0x7f030c0b0610 2023-10-07T11:03:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f030bfff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f030c0c8a20 0x7f030c0cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41940/0 (socket says 172.21.15.29:41940) 2023-10-07T11:03:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f030bfff700 1 -- 172.21.15.29:0/353980485 learned_addr learned my addr 172.21.15.29:0/353980485 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f030b7fe700 1 --2- 172.21.15.29:0/353980485 >> v2:172.21.15.144:3300/0 conn(0x7f030c0b0610 0x7f030c0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.332+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624351854 0 0) 0x7f030c0690f0 con 0x7f030c0c8a20 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f4003570 con 0x7f030c0c8a20 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221865614 0 0) 0x7f02f4003570 con 0x7f030c0c8a20 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 >> v2:172.21.15.144:3300/0 conn(0x7f030c0b0610 msgr2=0x7f030c0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 --2- 172.21.15.29:0/353980485 >> v2:172.21.15.144:3300/0 conn(0x7f030c0b0610 0x7f030c0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 >> v1:172.21.15.144:6789/0 conn(0x7f030c0b09e0 legacy=0x7f030c0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c068aa0 con 0x7f030c0c8a20 2023-10-07T11:03:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f03040017f0 con 0x7f030c0c8a20 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 >> v1:172.21.15.29:6789/0 conn(0x7f030c0c8a20 legacy=0x7f030c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f4003ab0 0x7f02f4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4004810 0x7f02f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f4005210 0x7f02f400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0687c0 con 0x7f02f4003ab0 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f400c1b0 con 0x7f02f4005210 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f400c370 con 0x7f02f4004810 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030b7fe700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f4003ab0 0x7f02f4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030affd700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4004810 0x7f02f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.333+0000 7f030bfff700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f4005210 0x7f02f400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030b7fe700 1 -- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4004810 msgr2=0x7f02f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030b7fe700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4004810 0x7f02f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030b7fe700 1 -- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f4005210 msgr2=0x7f02f400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030b7fe700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f4005210 0x7f02f400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030b7fe700 1 -- 172.21.15.29:0/353980485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f4003570 con 0x7f02f4003ab0 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030bfff700 1 -- 172.21.15.29:0/353980485 reap_dead start 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030b7fe700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f4003ab0 0x7f02f4003ec0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009c00 tx=0x7f02fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4eb7a9b2bd17829 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00e710 con 0x7f02f4003ab0 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc00e870 con 0x7f02f4003ab0 2023-10-07T11:03:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.334+0000 7f030a7fc700 1 -- 172.21.15.29:0/353980485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00ec30 con 0x7f02f4003ab0 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 -- 172.21.15.29:0/353980485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f4003ab0 msgr2=0x7f02f4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f4003ab0 0x7f02f4003ec0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009c00 tx=0x7f02fc0091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 -- 172.21.15.29:0/353980485 shutdown_connections 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 --2- 172.21.15.29:0/353980485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02f4003ab0 0x7f02f4003ec0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 -- 172.21.15.29:0/353980485 >> 172.21.15.29:0/353980485 conn(0x7f030c063a10 msgr2=0x7f030c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 -- 172.21.15.29:0/353980485 shutdown_connections 2023-10-07T11:03:55.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.335+0000 7f0311fff700 1 -- 172.21.15.29:0/353980485 wait complete. 2023-10-07T11:03:55.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.336+0000 7f0311fff700 1 Processor -- start 2023-10-07T11:03:55.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.337+0000 7f0311fff700 1 -- start start 2023-10-07T11:03:55.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.337+0000 7f0311fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4003ab0 0x7f030c147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f0311fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 0x7f030c156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f0311fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f030c156880 0x7f030c06c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f0311fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0c6e90 con 0x7f030c144a50 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f0311fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0c6280 con 0x7f030c156880 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f0311fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0c6560 con 0x7f02f4003ab0 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 0x7f030c156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 0x7f030c156340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43418/0 (socket says 172.21.15.29:43418) 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 -- 172.21.15.29:0/1280565842 learned_addr learned my addr 172.21.15.29:0/1280565842 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030b7fe700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4003ab0 0x7f030c147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030bfff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f030c156880 0x7f030c06c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 -- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4003ab0 msgr2=0x7f030c147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4003ab0 0x7f030c147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 -- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f030c156880 msgr2=0x7f030c06c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f030c156880 0x7f030c06c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 -- 172.21.15.29:0/1280565842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c0c6840 con 0x7f030c144a50 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.338+0000 7f030affd700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 0x7f030c156340 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f03000078a0 tx=0x7f0300009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.339+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300012590 con 0x7f030c144a50 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.339+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03000126f0 con 0x7f030c144a50 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.339+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c0c6b20 con 0x7f030c144a50 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.339+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300012a90 con 0x7f030c144a50 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.339+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c0c5e50 con 0x7f030c144a50 2023-10-07T11:03:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.340+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c0718b0 con 0x7f030c144a50 2023-10-07T11:03:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.341+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 105) v1 ==== 84578+0+0 (secure 0 0 0) 0x7f030001d460 con 0x7f030c144a50 2023-10-07T11:03:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.342+0000 7f02f37fe700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f02f8066d80 0x7f02f8069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.342+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6388+0+0 (secure 0 0 0) 0x7f0300084670 con 0x7f030c144a50 2023-10-07T11:03:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.342+0000 7f030b7fe700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f02f8066d80 0x7f02f8069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.343+0000 7f030b7fe700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f02f8066d80 0x7f02f8069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0091e0 tx=0x7f02fc01a000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:55.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.346+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03000518e0 con 0x7f030c144a50 2023-10-07T11:03:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:55.553+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f030c068d80 con 0x7f030c144a50 2023-10-07T11:03:56.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.063+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0300051cb0 con 0x7f030c144a50 2023-10-07T11:03:56.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.072+0000 7f02f37fe700 1 -- 172.21.15.29:0/1280565842 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v202) v1 ==== 169+0+0 (secure 0 0 0) 0x7f03000518e0 con 0x7f030c144a50 2023-10-07T11:03:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.076+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f02f8066d80 msgr2=0x7f02f8069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.076+0000 7f0311fff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f02f8066d80 0x7f02f8069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0091e0 tx=0x7f02fc01a000 comp rx=0 tx=0).stop 2023-10-07T11:03:56.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.076+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 msgr2=0x7f030c156340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.076+0000 7f0311fff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 0x7f030c156340 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f03000078a0 tx=0x7f0300009040 comp rx=0 tx=0).stop 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 shutdown_connections 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f030c156880 0x7f030c06c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f02f8066d80 0x7f02f8069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02f4003ab0 0x7f030c147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 --2- 172.21.15.29:0/1280565842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f030c144a50 0x7f030c156340 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 >> 172.21.15.29:0/1280565842 conn(0x7f030c063a10 msgr2=0x7f030c0cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 shutdown_connections 2023-10-07T11:03:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.077+0000 7f0311fff700 1 -- 172.21.15.29:0/1280565842 wait complete. 2023-10-07T11:03:56.080 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-07T11:03:56.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-07T11:03:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.301+0000 7f99fd9ff700 1 Processor -- start 2023-10-07T11:03:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.301+0000 7f99fd9ff700 1 -- start start 2023-10-07T11:03:56.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.302+0000 7f99fd9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f80688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.302+0000 7f99fd9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99f806a240 0x7f99f80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.302+0000 7f99fd9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f80c8ed0 0x7f99f806cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.302+0000 7f99fd9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80cacf0 con 0x7f99f80c8ed0 2023-10-07T11:03:56.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.302+0000 7f99fd9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f8068e10 con 0x7f99f806a240 2023-10-07T11:03:56.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.302+0000 7f99fd9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80c9350 con 0x7f99f8159710 2023-10-07T11:03:56.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f80c8ed0 0x7f99f806cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f80c8ed0 0x7f99f806cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43422/0 (socket says 172.21.15.29:43422) 2023-10-07T11:03:56.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 -- 172.21.15.29:0/2608685158 learned_addr learned my addr 172.21.15.29:0/2608685158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:56.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f6ffd700 1 --2- 172.21.15.29:0/2608685158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f80688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f67fc700 1 --2- 172.21.15.29:0/2608685158 >> v2:172.21.15.144:3300/0 conn(0x7f99f806a240 0x7f99f80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 -- 172.21.15.29:0/2608685158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 msgr2=0x7f99f80688d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 --2- 172.21.15.29:0/2608685158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f80688d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 -- 172.21.15.29:0/2608685158 >> v2:172.21.15.144:3300/0 conn(0x7f99f806a240 msgr2=0x7f99f80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 --2- 172.21.15.29:0/2608685158 >> v2:172.21.15.144:3300/0 conn(0x7f99f806a240 0x7f99f80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 -- 172.21.15.29:0/2608685158 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80b11c0 con 0x7f99f80c8ed0 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f67fc700 1 --2- 172.21.15.29:0/2608685158 >> v2:172.21.15.144:3300/0 conn(0x7f99f806a240 0x7f99f80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.303+0000 7f99f77fe700 1 --2- 172.21.15.29:0/2608685158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80c8ed0 0x7f99f806cf00 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f99f806f3e0 tx=0x7f99f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8e6868fe9a588f server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99f5ffb700 1 -- 172.21.15.29:0/2608685158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f000e070 con 0x7f99f80c8ed0 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99f5ffb700 1 -- 172.21.15.29:0/2608685158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f000ba50 con 0x7f99f80c8ed0 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99f5ffb700 1 -- 172.21.15.29:0/2608685158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f00092e0 con 0x7f99f80c8ed0 2023-10-07T11:03:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 -- 172.21.15.29:0/2608685158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80c8ed0 msgr2=0x7f99f806cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/2608685158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80c8ed0 0x7f99f806cf00 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f99f806f3e0 tx=0x7f99f0009840 comp rx=0 tx=0).stop 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 -- 172.21.15.29:0/2608685158 shutdown_connections 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/2608685158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80c8ed0 0x7f99f806cf00 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/2608685158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f80688d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/2608685158 >> v2:172.21.15.144:3300/0 conn(0x7f99f806a240 0x7f99f80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 -- 172.21.15.29:0/2608685158 >> 172.21.15.29:0/2608685158 conn(0x7f99f80ce3f0 msgr2=0x7f99f80ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 -- 172.21.15.29:0/2608685158 shutdown_connections 2023-10-07T11:03:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.304+0000 7f99fd9ff700 1 -- 172.21.15.29:0/2608685158 wait complete. 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.306+0000 7f99fd9ff700 1 Processor -- start 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.306+0000 7f99fd9ff700 1 -- start start 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99fd9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 0x7f99f8169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99fd9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99f80c8ed0 0x7f99f80c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99fd9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f8157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99fd9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f8169fd0 con 0x7f99f806a240 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99fd9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80c48f0 con 0x7f99f80c8ed0 2023-10-07T11:03:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99fd9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f815e4f0 con 0x7f99f8159710 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 0x7f99f8169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 0x7f99f8169a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43430/0 (socket says 172.21.15.29:43430) 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.307+0000 7f99f6ffd700 1 -- 172.21.15.29:0/9304238 learned_addr learned my addr 172.21.15.29:0/9304238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99f6ffd700 1 -- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 msgr2=0x7f99f8157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99f6ffd700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f8157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99f6ffd700 1 -- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99f80c8ed0 msgr2=0x7f99f80c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99f6ffd700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99f80c8ed0 0x7f99f80c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99f6ffd700 1 -- 172.21.15.29:0/9304238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0009ae0 con 0x7f99f806a240 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99f6ffd700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 0x7f99f8169a90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f99e800d860 tx=0x7f99e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80125f0 con 0x7f99f806a240 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8012750 con 0x7f99f806a240 2023-10-07T11:03:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8012b10 con 0x7f99f806a240 2023-10-07T11:03:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f815e7d0 con 0x7f99f806a240 2023-10-07T11:03:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.308+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f815ed90 con 0x7f99f806a240 2023-10-07T11:03:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.310+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f804c3c0 con 0x7f99f806a240 2023-10-07T11:03:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.310+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f99e801e460 con 0x7f99f806a240 2023-10-07T11:03:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.311+0000 7f99deffd700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f99e4066f30 0x7f99e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.312+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f99e8084da0 con 0x7f99f806a240 2023-10-07T11:03:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.312+0000 7f99f67fc700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f99e4066f30 0x7f99e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.313+0000 7f99f67fc700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f99e4066f30 0x7f99e40693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f99f816c5c0 tx=0x7f99ec008040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:56.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.315+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99e80558f0 con 0x7f99f806a240 2023-10-07T11:03:56.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.521+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f99f8069080 con 0x7f99f806a240 2023-10-07T11:03:56.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.525+0000 7f99deffd700 1 -- 172.21.15.29:0/9304238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7f99e8055480 con 0x7f99f806a240 2023-10-07T11:03:56.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f99e4066f30 msgr2=0x7f99e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f99e4066f30 0x7f99e40693e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f99f816c5c0 tx=0x7f99ec008040 comp rx=0 tx=0).stop 2023-10-07T11:03:56.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 msgr2=0x7f99f8169a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 0x7f99f8169a90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f99e800d860 tx=0x7f99e8008040 comp rx=0 tx=0).stop 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 shutdown_connections 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f99e4066f30 0x7f99e40693e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99f8159710 0x7f99f8157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806a240 0x7f99f8169a90 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 --2- 172.21.15.29:0/9304238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99f80c8ed0 0x7f99f80c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 >> 172.21.15.29:0/9304238 conn(0x7f99f80ce3f0 msgr2=0x7f99f80cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 shutdown_connections 2023-10-07T11:03:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.527+0000 7f99fd9ff700 1 -- 172.21.15.29:0/9304238 wait complete. 2023-10-07T11:03:56.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-07T11:03:56.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.743+0000 7f764abff700 1 Processor -- start 2023-10-07T11:03:56.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.743+0000 7f764abff700 1 -- start start 2023-10-07T11:03:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764abff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7644159750 0x7f76440d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7644159b20 0x7f76440ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764abff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440cbf20 con 0x7f76440ce6d0 2023-10-07T11:03:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764abff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76440cc200 con 0x7f7644159750 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76440cc4e0 con 0x7f7644159b20 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764899b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7644159750 0x7f76440d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764899b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7644159750 0x7f76440d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43436/0 (socket says 172.21.15.29:43436) 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764899b700 1 -- 172.21.15.29:0/1553842502 learned_addr learned my addr 172.21.15.29:0/1553842502 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764899b700 1 -- 172.21.15.29:0/1553842502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7644159b20 msgr2=0x7f76440ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.745+0000 7f764899b700 1 --2- 172.21.15.29:0/1553842502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7644159b20 0x7f76440ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.746+0000 7f764899b700 1 -- 172.21.15.29:0/1553842502 >> v1:172.21.15.144:6789/0 conn(0x7f76440ce6d0 legacy=0x7f764415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.746+0000 7f764899b700 1 -- 172.21.15.29:0/1553842502 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440cc7c0 con 0x7f7644159750 2023-10-07T11:03:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.746+0000 7f764899b700 1 --2- 172.21.15.29:0/1553842502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159750 0x7f76440d10a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f7638009990 tx=0x7f7638009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6dba85a82a8668a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.746+0000 7f76437fe700 1 -- 172.21.15.29:0/1553842502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638005c10 con 0x7f7644159750 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.746+0000 7f76437fe700 1 -- 172.21.15.29:0/1553842502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7638005d70 con 0x7f7644159750 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.746+0000 7f76437fe700 1 -- 172.21.15.29:0/1553842502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763800b840 con 0x7f7644159750 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 -- 172.21.15.29:0/1553842502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159750 msgr2=0x7f76440d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 --2- 172.21.15.29:0/1553842502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159750 0x7f76440d10a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f7638009990 tx=0x7f7638009220 comp rx=0 tx=0).stop 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 -- 172.21.15.29:0/1553842502 shutdown_connections 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 --2- 172.21.15.29:0/1553842502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159750 0x7f76440d10a0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 --2- 172.21.15.29:0/1553842502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7644159b20 0x7f76440ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 -- 172.21.15.29:0/1553842502 >> 172.21.15.29:0/1553842502 conn(0x7f7644063a10 msgr2=0x7f76440d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 -- 172.21.15.29:0/1553842502 shutdown_connections 2023-10-07T11:03:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.747+0000 7f764abff700 1 -- 172.21.15.29:0/1553842502 wait complete. 2023-10-07T11:03:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.748+0000 7f764abff700 1 Processor -- start 2023-10-07T11:03:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.748+0000 7f764abff700 1 -- start start 2023-10-07T11:03:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.749+0000 7f764abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76440ce6d0 0x7f76440b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.749+0000 7f764abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7644159750 0x7f764415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.749+0000 7f764abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 0x7f7644144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.749+0000 7f764abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644159fa0 con 0x7f7644159b20 2023-10-07T11:03:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.749+0000 7f764abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764415ccc0 con 0x7f7644159750 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.749+0000 7f764abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76440b4350 con 0x7f76440ce6d0 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 0x7f7644144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 0x7f7644144fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43438/0 (socket says 172.21.15.29:43438) 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 -- 172.21.15.29:0/3978934849 learned_addr learned my addr 172.21.15.29:0/3978934849 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764899b700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76440ce6d0 0x7f76440b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f7643fff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7644159750 0x7f764415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 -- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76440ce6d0 msgr2=0x7f76440b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76440ce6d0 0x7f76440b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 -- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7644159750 msgr2=0x7f764415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7644159750 0x7f764415c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 -- 172.21.15.29:0/3978934849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638009480 con 0x7f7644159b20 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f764919c700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 0x7f7644144fb0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f763c00bbb0 tx=0x7f763c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.750+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0092e0 con 0x7f7644159b20 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.751+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c00c7a0 con 0x7f7644159b20 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.751+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76440b4630 con 0x7f7644159b20 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.751+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c00cb40 con 0x7f7644159b20 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.751+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76440b4bf0 con 0x7f7644159b20 2023-10-07T11:03:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.752+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764404c3c0 con 0x7f7644159b20 2023-10-07T11:03:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.753+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f763c012070 con 0x7f7644159b20 2023-10-07T11:03:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.754+0000 7f7640ff9700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7628066f30 0x7f76280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.754+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f763c084380 con 0x7f7644159b20 2023-10-07T11:03:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.754+0000 7f764899b700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7628066f30 0x7f76280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.755+0000 7f764899b700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7628066f30 0x7f76280693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7638003080 tx=0x7f7638015000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:56.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.758+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f763c0514e0 con 0x7f7644159b20 2023-10-07T11:03:56.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.960+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7644064040 con 0x7f7644159b20 2023-10-07T11:03:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.964+0000 7f7640ff9700 1 -- 172.21.15.29:0/3978934849 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7f763c054b00 con 0x7f7644159b20 2023-10-07T11:03:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7628066f30 msgr2=0x7f76280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7628066f30 0x7f76280693e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7638003080 tx=0x7f7638015000 comp rx=0 tx=0).stop 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 msgr2=0x7f7644144fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 0x7f7644144fb0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f763c00bbb0 tx=0x7f763c009840 comp rx=0 tx=0).stop 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 shutdown_connections 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7644159b20 0x7f7644144fb0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7628066f30 0x7f76280693e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76440ce6d0 0x7f76440b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 --2- 172.21.15.29:0/3978934849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7644159750 0x7f764415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.966+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 >> 172.21.15.29:0/3978934849 conn(0x7f7644063a10 msgr2=0x7f7644154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.967+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 shutdown_connections 2023-10-07T11:03:56.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:56.967+0000 7f764abff700 1 -- 172.21.15.29:0/3978934849 wait complete. 2023-10-07T11:03:56.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-07T11:03:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.184+0000 7f46ed1ff700 1 Processor -- start 2023-10-07T11:03:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.185+0000 7f46ed1ff700 1 -- start start 2023-10-07T11:03:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46ed1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e80afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46ed1ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f46e8153cc0 0x7f46e8159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46ed1ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e8157790 con 0x7f46e80b0410 2023-10-07T11:03:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46ed1ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e80b0950 con 0x7f46e8153cc0 2023-10-07T11:03:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46ed1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8157580 con 0x7f46e814c8d0 2023-10-07T11:03:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46e759e700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f46e80b0410 0x7f46e8158550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41948/0 (socket says 172.21.15.29:41948) 2023-10-07T11:03:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46e759e700 1 -- 172.21.15.29:0/1581226371 learned_addr learned my addr 172.21.15.29:0/1581226371 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.186+0000 7f46e659c700 1 --2- 172.21.15.29:0/1581226371 >> v2:172.21.15.144:3300/0 conn(0x7f46e8153cc0 0x7f46e8159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e6d9d700 1 --2- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e80afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e5d9b700 1 -- 172.21.15.29:0/1581226371 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536636404 0 0) 0x7f46e8157790 con 0x7f46e80b0410 2023-10-07T11:03:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e5d9b700 1 -- 172.21.15.29:0/1581226371 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d0003570 con 0x7f46e80b0410 2023-10-07T11:03:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e659c700 1 -- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 msgr2=0x7f46e80afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e659c700 1 --2- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e80afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e659c700 1 -- 172.21.15.29:0/1581226371 >> v1:172.21.15.29:6789/0 conn(0x7f46e80b0410 legacy=0x7f46e8158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e659c700 1 -- 172.21.15.29:0/1581226371 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e8157a70 con 0x7f46e8153cc0 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.187+0000 7f46e659c700 1 --2- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8159680 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00aa90 tx=0x7f46dc00a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d1c47d8632602e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46e5d9b700 1 -- 172.21.15.29:0/1581226371 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc00ae70 con 0x7f46e8153cc0 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46e5d9b700 1 -- 172.21.15.29:0/1581226371 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc00c970 con 0x7f46e8153cc0 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46e5d9b700 1 -- 172.21.15.29:0/1581226371 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc00cd10 con 0x7f46e8153cc0 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 -- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 msgr2=0x7f46e8159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8159680 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00aa90 tx=0x7f46dc00a330 comp rx=0 tx=0).stop 2023-10-07T11:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 -- 172.21.15.29:0/1581226371 shutdown_connections 2023-10-07T11:03:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e80afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/1581226371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8159680 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 -- 172.21.15.29:0/1581226371 >> 172.21.15.29:0/1581226371 conn(0x7f46e8067610 msgr2=0x7f46e8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 -- 172.21.15.29:0/1581226371 shutdown_connections 2023-10-07T11:03:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.188+0000 7f46ed1ff700 1 -- 172.21.15.29:0/1581226371 wait complete. 2023-10-07T11:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.190+0000 7f46ed1ff700 1 Processor -- start 2023-10-07T11:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.190+0000 7f46ed1ff700 1 -- start start 2023-10-07T11:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46ed1ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 0x7f46e8169020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46ed1ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e814d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46ed1ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46ed1ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e814dac0 con 0x7f46e80b0410 2023-10-07T11:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46ed1ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e814dda0 con 0x7f46e8153cc0 2023-10-07T11:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46ed1ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e814e080 con 0x7f46e814c8d0 2023-10-07T11:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 0x7f46e8169020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 0x7f46e8169020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43448/0 (socket says 172.21.15.29:43448) 2023-10-07T11:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46e6d9d700 1 -- 172.21.15.29:0/2075940076 learned_addr learned my addr 172.21.15.29:0/2075940076 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46e759e700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.191+0000 7f46e659c700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e814d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46e6d9d700 1 -- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 msgr2=0x7f46e814d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46e6d9d700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e814d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46e6d9d700 1 -- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 msgr2=0x7f46e8156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46e6d9d700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46e6d9d700 1 -- 172.21.15.29:0/2075940076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc00a5d0 con 0x7f46e80b0410 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46e6d9d700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 0x7f46e8169020 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f46d800ec80 tx=0x7f46d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d800c850 con 0x7f46e80b0410 2023-10-07T11:03:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d800c9b0 con 0x7f46e80b0410 2023-10-07T11:03:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d800cd50 con 0x7f46e80b0410 2023-10-07T11:03:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.192+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e814e360 con 0x7f46e80b0410 2023-10-07T11:03:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.193+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e8169560 con 0x7f46e80b0410 2023-10-07T11:03:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.193+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e804c3c0 con 0x7f46e80b0410 2023-10-07T11:03:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.194+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f46d8013680 con 0x7f46e80b0410 2023-10-07T11:03:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.195+0000 7f46ceffd700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f46d4066f30 0x7f46d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.196+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f46d80848e0 con 0x7f46e80b0410 2023-10-07T11:03:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.196+0000 7f46e659c700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f46d4066f30 0x7f46d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.197+0000 7f46e659c700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f46d4066f30 0x7f46d40693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f46dc008010 tx=0x7f46dc019000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.199+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46d8051a40 con 0x7f46e80b0410 2023-10-07T11:03:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.400+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f46e806e880 con 0x7f46e80b0410 2023-10-07T11:03:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.405+0000 7f46ceffd700 1 -- 172.21.15.29:0/2075940076 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f46d8055060 con 0x7f46e80b0410 2023-10-07T11:03:57.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f46d4066f30 msgr2=0x7f46d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f46d4066f30 0x7f46d40693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f46dc008010 tx=0x7f46dc019000 comp rx=0 tx=0).stop 2023-10-07T11:03:57.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 msgr2=0x7f46e8169020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 0x7f46e8169020 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f46d800ec80 tx=0x7f46d80091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 shutdown_connections 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46e8153cc0 0x7f46e8156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e80b0410 0x7f46e8169020 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f46d4066f30 0x7f46d40693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 --2- 172.21.15.29:0/2075940076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46e814c8d0 0x7f46e814d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 >> 172.21.15.29:0/2075940076 conn(0x7f46e8067610 msgr2=0x7f46e806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 shutdown_connections 2023-10-07T11:03:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.407+0000 7f46ed1ff700 1 -- 172.21.15.29:0/2075940076 wait complete. 2023-10-07T11:03:57.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-07T11:03:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.622+0000 7f4daa9ff700 1 Processor -- start 2023-10-07T11:03:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.622+0000 7f4daa9ff700 1 -- start start 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4daa9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4da40ad370 0x7f4da40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4daa9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4da4159ec0 0x7f4da415a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4daa9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da41568c0 0x7f4da4156d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4daa9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da415a880 con 0x7f4da40ad370 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4daa9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da414d790 con 0x7f4da4159ec0 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4daa9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da40ae5d0 con 0x7f4da41568c0 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4da40ad370 0x7f4da40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da3fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4da40ad370 0x7f4da40cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43456/0 (socket says 172.21.15.29:43456) 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da3fff700 1 -- 172.21.15.29:0/3306032813 learned_addr learned my addr 172.21.15.29:0/3306032813 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da37fe700 1 --2- 172.21.15.29:0/3306032813 >> v2:172.21.15.144:3300/0 conn(0x7f4da4159ec0 0x7f4da415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da8f9c700 1 --2- 172.21.15.29:0/3306032813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da41568c0 0x7f4da4156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da3fff700 1 -- 172.21.15.29:0/3306032813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da41568c0 msgr2=0x7f4da4156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.624+0000 7f4da3fff700 1 --2- 172.21.15.29:0/3306032813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da41568c0 0x7f4da4156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da3fff700 1 -- 172.21.15.29:0/3306032813 >> v2:172.21.15.144:3300/0 conn(0x7f4da4159ec0 msgr2=0x7f4da415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da3fff700 1 --2- 172.21.15.29:0/3306032813 >> v2:172.21.15.144:3300/0 conn(0x7f4da4159ec0 0x7f4da415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da3fff700 1 -- 172.21.15.29:0/3306032813 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da40ad740 con 0x7f4da40ad370 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da37fe700 1 --2- 172.21.15.29:0/3306032813 >> v2:172.21.15.144:3300/0 conn(0x7f4da4159ec0 0x7f4da415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da3fff700 1 --2- 172.21.15.29:0/3306032813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da40ad370 0x7f4da40cc870 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f4d940099e0 tx=0x7f4d94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71565a17f3526e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da2ffd700 1 -- 172.21.15.29:0/3306032813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400b7e0 con 0x7f4da40ad370 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da2ffd700 1 -- 172.21.15.29:0/3306032813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9400b940 con 0x7f4da40ad370 2023-10-07T11:03:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.625+0000 7f4da2ffd700 1 -- 172.21.15.29:0/3306032813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400bce0 con 0x7f4da40ad370 2023-10-07T11:03:57.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3306032813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da40ad370 msgr2=0x7f4da40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3306032813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da40ad370 0x7f4da40cc870 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f4d940099e0 tx=0x7f4d94009250 comp rx=0 tx=0).stop 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3306032813 shutdown_connections 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3306032813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da41568c0 0x7f4da4156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3306032813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da40ad370 0x7f4da40cc870 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3306032813 >> v2:172.21.15.144:3300/0 conn(0x7f4da4159ec0 0x7f4da415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3306032813 >> 172.21.15.29:0/3306032813 conn(0x7f4da40680a0 msgr2=0x7f4da40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3306032813 shutdown_connections 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.626+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3306032813 wait complete. 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.627+0000 7f4daa9ff700 1 Processor -- start 2023-10-07T11:03:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.628+0000 7f4daa9ff700 1 -- start start 2023-10-07T11:03:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.628+0000 7f4daa9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da40ad370 0x7f4da414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.628+0000 7f4daa9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 0x7f4da416a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4daa9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da4159ec0 0x7f4da41696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4daa9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da40a5960 con 0x7f4da41568c0 2023-10-07T11:03:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4daa9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da414d240 con 0x7f4da40ad370 2023-10-07T11:03:57.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4daa9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da416adf0 con 0x7f4da4159ec0 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 0x7f4da416a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 0x7f4da416a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43458/0 (socket says 172.21.15.29:43458) 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 -- 172.21.15.29:0/3933751417 learned_addr learned my addr 172.21.15.29:0/3933751417 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da8f9c700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da4159ec0 0x7f4da41696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da3fff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da40ad370 0x7f4da414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 -- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da4159ec0 msgr2=0x7f4da41696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da4159ec0 0x7f4da41696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 -- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da40ad370 msgr2=0x7f4da414cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da40ad370 0x7f4da414cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.629+0000 7f4da37fe700 1 -- 172.21.15.29:0/3933751417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d940094f0 con 0x7f4da41568c0 2023-10-07T11:03:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.630+0000 7f4da37fe700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 0x7f4da416a8b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800d860 tx=0x7f4d98008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:57.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.630+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d980125f0 con 0x7f4da41568c0 2023-10-07T11:03:57.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.630+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d98012750 con 0x7f4da41568c0 2023-10-07T11:03:57.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.630+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98012af0 con 0x7f4da41568c0 2023-10-07T11:03:57.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.630+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da416a680 con 0x7f4da41568c0 2023-10-07T11:03:57.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.630+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da416a3a0 con 0x7f4da41568c0 2023-10-07T11:03:57.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.631+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da40a5ad0 con 0x7f4da41568c0 2023-10-07T11:03:57.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.632+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4d98013480 con 0x7f4da41568c0 2023-10-07T11:03:57.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.633+0000 7f4d8bfff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4d90066f30 0x7f4d900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:57.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.633+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f4d980847e0 con 0x7f4da41568c0 2023-10-07T11:03:57.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.633+0000 7f4da3fff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4d90066f30 0x7f4d900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:57.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.634+0000 7f4da3fff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4d90066f30 0x7f4d900693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d940099e0 tx=0x7f4d94017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:57.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.637+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d98051940 con 0x7f4da41568c0 2023-10-07T11:03:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:57.920+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f4da404c3c0 con 0x7f4da41568c0 2023-10-07T11:03:58.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.066+0000 7f4d8bfff700 1 -- 172.21.15.29:0/3933751417 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3067) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4d98054f60 con 0x7f4da41568c0 2023-10-07T11:03:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.068+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4d90066f30 msgr2=0x7f4d900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.068+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4d90066f30 0x7f4d900693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d940099e0 tx=0x7f4d94017000 comp rx=0 tx=0).stop 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.068+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 msgr2=0x7f4da416a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.068+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 0x7f4da416a8b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800d860 tx=0x7f4d98008040 comp rx=0 tx=0).stop 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 shutdown_connections 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4d90066f30 0x7f4d900693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4da4159ec0 0x7f4da41696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da40ad370 0x7f4da414cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 --2- 172.21.15.29:0/3933751417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da41568c0 0x7f4da416a8b0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 >> 172.21.15.29:0/3933751417 conn(0x7f4da40680a0 msgr2=0x7f4da406d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 shutdown_connections 2023-10-07T11:03:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:58.069+0000 7f4daa9ff700 1 -- 172.21.15.29:0/3933751417 wait complete. 2023-10-07T11:03:58.087 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-07T11:03:58.087 INFO:tasks.cephfs_test_runner: 2023-10-07T11:03:58.088 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-07T11:03:58.088 INFO:tasks.cephfs_test_runner:Ran 11 tests in 5631.961s 2023-10-07T11:03:58.088 INFO:tasks.cephfs_test_runner: 2023-10-07T11:03:58.088 INFO:tasks.cephfs_test_runner:OK 2023-10-07T11:03:58.089 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-07T11:03:58.099 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-07T11:03:58.115 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-07T11:03:58.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.116 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T11:03:58.163 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-07T11:03:58.163 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-07T11:03:58.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-07T11:03:58.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-07T11:03:58.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-07T11:03:58.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.423 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-07T11:03:58.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-07T11:03:58.551 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-07T11:03:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T11:03:58.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.554 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-07T11:03:58.606 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-07T11:03:58.606 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-07T11:03:58.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.608 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-07T11:03:58.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.683 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-07T11:03:58.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-07T11:03:58.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.868 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-07T11:03:58.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-07T11:03:58.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-07T11:03:58.992 INFO:teuthology.orchestra.run.smithi144.stderr:Cannot find device "ceph-brx" 2023-10-07T11:03:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T11:03:58.996 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-07T11:03:59.004 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-07T11:03:59.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:03:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.201+0000 7f0de9dff700 1 Processor -- start 2023-10-07T11:03:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.202+0000 7f0de9dff700 1 -- start start 2023-10-07T11:03:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40ad370 0x7f0de40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de9dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de414d790 con 0x7f0de4159ec0 2023-10-07T11:03:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de9dff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de40ae5d0 con 0x7f0de40b0410 2023-10-07T11:03:59.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40ad740 con 0x7f0de40ad370 2023-10-07T11:03:59.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de2ffd700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0de4159ec0 0x7f0de41568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41962/0 (socket says 172.21.15.29:41962) 2023-10-07T11:03:59.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de2ffd700 1 -- 172.21.15.29:0/4087799634 learned_addr learned my addr 172.21.15.29:0/4087799634 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:03:59.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.203+0000 7f0de37fe700 1 --2- 172.21.15.29:0/4087799634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40ad370 0x7f0de40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942381787 0 0) 0x7f0de414d790 con 0x7f0de4159ec0 2023-10-07T11:03:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dcc003570 con 0x7f0de4159ec0 2023-10-07T11:03:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903175219 0 0) 0x7f0de40ae5d0 con 0x7f0de40b0410 2023-10-07T11:03:59.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de414d790 con 0x7f0de40b0410 2023-10-07T11:03:59.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de37fe700 1 -- 172.21.15.29:0/4087799634 >> v1:172.21.15.144:6789/0 conn(0x7f0de40b0410 legacy=0x7f0de4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955935184 0 0) 0x7f0dcc003570 con 0x7f0de4159ec0 2023-10-07T11:03:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de37fe700 1 -- 172.21.15.29:0/4087799634 >> v1:172.21.15.29:6789/0 conn(0x7f0de4159ec0 legacy=0x7f0de41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de37fe700 1 -- 172.21.15.29:0/4087799634 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de40ccdb0 con 0x7f0de40ad370 2023-10-07T11:03:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.204+0000 7f0de37fe700 1 --2- 172.21.15.29:0/4087799634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40ad370 0x7f0de40cc870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009910 tx=0x7f0dd40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b7149b3e7a0114 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8001470 con 0x7f0de40ad370 2023-10-07T11:03:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd4011420 con 0x7f0de40ad370 2023-10-07T11:03:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de27fc700 1 -- 172.21.15.29:0/4087799634 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd40117a0 con 0x7f0de40ad370 2023-10-07T11:03:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 -- 172.21.15.29:0/4087799634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40ad370 msgr2=0x7f0de40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 --2- 172.21.15.29:0/4087799634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40ad370 0x7f0de40cc870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009910 tx=0x7f0dd40091a0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 -- 172.21.15.29:0/4087799634 shutdown_connections 2023-10-07T11:03:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 --2- 172.21.15.29:0/4087799634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40ad370 0x7f0de40cc870 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 -- 172.21.15.29:0/4087799634 >> 172.21.15.29:0/4087799634 conn(0x7f0de40680a0 msgr2=0x7f0de40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 -- 172.21.15.29:0/4087799634 shutdown_connections 2023-10-07T11:03:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.205+0000 7f0de9dff700 1 -- 172.21.15.29:0/4087799634 wait complete. 2023-10-07T11:03:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.207+0000 7f0de9dff700 1 Processor -- start 2023-10-07T11:03:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.207+0000 7f0de9dff700 1 -- start start 2023-10-07T11:03:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de9dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de40ad370 0x7f0de40c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de9dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40b0410 0x7f0de4163c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de9dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 0x7f0de40c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de9dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40c34b0 con 0x7f0de4159ec0 2023-10-07T11:03:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de9dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4164150 con 0x7f0de40ad370 2023-10-07T11:03:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de9dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40c42d0 con 0x7f0de40b0410 2023-10-07T11:03:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 0x7f0de40c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 0x7f0de40c1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43474/0 (socket says 172.21.15.29:43474) 2023-10-07T11:03:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de3fff700 1 -- 172.21.15.29:0/3210246717 learned_addr learned my addr 172.21.15.29:0/3210246717 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de2ffd700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40b0410 0x7f0de4163c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.208+0000 7f0de37fe700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de40ad370 0x7f0de40c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de3fff700 1 -- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40b0410 msgr2=0x7f0de4163c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de3fff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40b0410 0x7f0de4163c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de3fff700 1 -- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de40ad370 msgr2=0x7f0de40c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de3fff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de40ad370 0x7f0de40c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de3fff700 1 -- 172.21.15.29:0/3210246717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4009420 con 0x7f0de4159ec0 2023-10-07T11:03:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de3fff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 0x7f0de40c1840 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00c940 tx=0x7f0ddc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc00a7a0 con 0x7f0de4159ec0 2023-10-07T11:03:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ddc00a920 con 0x7f0de4159ec0 2023-10-07T11:03:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc00ace0 con 0x7f0de4159ec0 2023-10-07T11:03:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de40c36c0 con 0x7f0de4159ec0 2023-10-07T11:03:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.209+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de40c3c80 con 0x7f0de4159ec0 2023-10-07T11:03:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.210+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de40c4460 con 0x7f0de4159ec0 2023-10-07T11:03:59.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.211+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0ddc012630 con 0x7f0de4159ec0 2023-10-07T11:03:59.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.212+0000 7f0dcb7fe700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dd0066f30 0x7f0dd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.212+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f0ddc084b50 con 0x7f0de4159ec0 2023-10-07T11:03:59.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.213+0000 7f0de37fe700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dd0066f30 0x7f0dd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.213+0000 7f0de37fe700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dd0066f30 0x7f0dd00693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0de40ac890 tx=0x7f0dd4016000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:59.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.217+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0ddc051c10 con 0x7f0de4159ec0 2023-10-07T11:03:59.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.417+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0de404c3c0 con 0x7f0de4159ec0 2023-10-07T11:03:59.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.418+0000 7f0dcb7fe700 1 -- 172.21.15.29:0/3210246717 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+13917 (secure 0 0 0) 0x7f0ddc055230 con 0x7f0de4159ec0 2023-10-07T11:03:59.419 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:59.419 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":202,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:56.059727+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3284787430":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/2235240922":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1299585123":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1947083474":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1254325888":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1413587217":"2023-10-08T11:03:53.691236+0000","172.21.15.29:6834/1070134873":"2023-10-08T11:03:53.053802+0000","172.21.15.29:6832/1070134873":"2023-10-08T11:03:53.053802+0000","172.21.15.144:0/2912144557":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/471864733":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6837/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6836/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/245402490":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/4011736589":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/3907222325":"2023-10-08T11:03:43.498860+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:03:59.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dd0066f30 msgr2=0x7f0dd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dd0066f30 0x7f0dd00693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0de40ac890 tx=0x7f0dd4016000 comp rx=0 tx=0).stop 2023-10-07T11:03:59.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 msgr2=0x7f0de40c1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 0x7f0de40c1840 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00c940 tx=0x7f0ddc007f90 comp rx=0 tx=0).stop 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 shutdown_connections 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0de4159ec0 0x7f0de40c1840 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de40ad370 0x7f0de40c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0dd0066f30 0x7f0dd00693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 --2- 172.21.15.29:0/3210246717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de40b0410 0x7f0de4163c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 >> 172.21.15.29:0/3210246717 conn(0x7f0de40680a0 msgr2=0x7f0de406e910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 shutdown_connections 2023-10-07T11:03:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.420+0000 7f0de9dff700 1 -- 172.21.15.29:0/3210246717 wait complete. 2023-10-07T11:03:59.439 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-07T11:03:59.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:03:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.635+0000 7fa8e73ff700 1 Processor -- start 2023-10-07T11:03:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.635+0000 7fa8e73ff700 1 -- start start 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e73ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8e00ae8b0 0x7fa8e00b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e73ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e00b1840 0x7fa8e0157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e73ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e0158ad0 con 0x7fa8e0157290 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e73ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00b1cc0 con 0x7fa8e00ae8b0 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e73ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e015b140 con 0x7fa8e00b1840 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8e00ae8b0 0x7fa8e00b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e499a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8e00ae8b0 0x7fa8e00b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43486/0 (socket says 172.21.15.29:43486) 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e499a700 1 -- 172.21.15.29:0/1487098192 learned_addr learned my addr 172.21.15.29:0/1487098192 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e599c700 1 --2- 172.21.15.29:0/1487098192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e00b1840 0x7fa8e0157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.637+0000 7fa8e499a700 1 -- 172.21.15.29:0/1487098192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e00b1840 msgr2=0x7fa8e0157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:03:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8e499a700 1 --2- 172.21.15.29:0/1487098192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e00b1840 0x7fa8e0157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8e499a700 1 -- 172.21.15.29:0/1487098192 >> v1:172.21.15.144:6789/0 conn(0x7fa8e0157290 legacy=0x7fa8e00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8e499a700 1 -- 172.21.15.29:0/1487098192 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e015b420 con 0x7fa8e00ae8b0 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8e499a700 1 --2- 172.21.15.29:0/1487098192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00ae8b0 0x7fa8e00b1300 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00099b0 tx=0x7fa8d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c23e27cff5de8a server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8d7fff700 1 -- 172.21.15.29:0/1487098192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0009ea0 con 0x7fa8e00ae8b0 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8d7fff700 1 -- 172.21.15.29:0/1487098192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d000b860 con 0x7fa8e00ae8b0 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.638+0000 7fa8d7fff700 1 -- 172.21.15.29:0/1487098192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d000bc00 con 0x7fa8e00ae8b0 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 -- 172.21.15.29:0/1487098192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00ae8b0 msgr2=0x7fa8e00b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/1487098192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00ae8b0 0x7fa8e00b1300 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00099b0 tx=0x7fa8d0009250 comp rx=0 tx=0).stop 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 -- 172.21.15.29:0/1487098192 shutdown_connections 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/1487098192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e00b1840 0x7fa8e0157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/1487098192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00ae8b0 0x7fa8e00b1300 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 -- 172.21.15.29:0/1487098192 >> 172.21.15.29:0/1487098192 conn(0x7fa8e00680a0 msgr2=0x7fa8e00cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:59.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 -- 172.21.15.29:0/1487098192 shutdown_connections 2023-10-07T11:03:59.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.639+0000 7fa8e73ff700 1 -- 172.21.15.29:0/1487098192 wait complete. 2023-10-07T11:03:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.640+0000 7fa8e73ff700 1 Processor -- start 2023-10-07T11:03:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.640+0000 7fa8e73ff700 1 -- start start 2023-10-07T11:03:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.641+0000 7fa8e73ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 0x7fa8e006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.641+0000 7fa8e73ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e0157290 0x7fa8e0145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.641+0000 7fa8e73ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8e0155860 0x7fa8e0155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.641+0000 7fa8e73ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e006e520 con 0x7fa8e00b1840 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.641+0000 7fa8e73ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e0156220 con 0x7fa8e0155860 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.641+0000 7fa8e73ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e006db50 con 0x7fa8e0157290 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 0x7fa8e006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 0x7fa8e006dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43494/0 (socket says 172.21.15.29:43494) 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 -- 172.21.15.29:0/4130334485 learned_addr learned my addr 172.21.15.29:0/4130334485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e499a700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e0157290 0x7fa8e0145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e599c700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8e0155860 0x7fa8e0155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 -- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e0157290 msgr2=0x7fa8e0145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e0157290 0x7fa8e0145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 -- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8e0155860 msgr2=0x7fa8e0155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8e0155860 0x7fa8e0155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 -- 172.21.15.29:0/4130334485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d00094f0 con 0x7fa8e00b1840 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.642+0000 7fa8e519b700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 0x7fa8e006dfe0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00c5c0 tx=0x7fa8dc00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.643+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc011590 con 0x7fa8e00b1840 2023-10-07T11:03:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.643+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc011710 con 0x7fa8e00b1840 2023-10-07T11:03:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.643+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc011ad0 con 0x7fa8e00b1840 2023-10-07T11:03:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.643+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e006f1e0 con 0x7fa8e00b1840 2023-10-07T11:03:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.643+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e016a710 con 0x7fa8e00b1840 2023-10-07T11:03:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.644+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e006d8f0 con 0x7fa8e00b1840 2023-10-07T11:03:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.645+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa8dc01b3b0 con 0x7fa8e00b1840 2023-10-07T11:03:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.646+0000 7fa8d57fa700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa8c4066f30 0x7fa8c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:03:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.646+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7fa8dc083cc0 con 0x7fa8e00b1840 2023-10-07T11:03:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.647+0000 7fa8e499a700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa8c4066f30 0x7fa8c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:03:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.647+0000 7fa8e499a700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa8c4066f30 0x7fa8c40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00099b0 tx=0x7fa8d0017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:03:59.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.650+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8dc08a080 con 0x7fa8e00b1840 2023-10-07T11:03:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.849+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8e004c3c0 con 0x7fa8c4066f30 2023-10-07T11:03:59.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.852+0000 7fa8d57fa700 1 -- 172.21.15.29:0/4130334485 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50639 (secure 0 0 0) 0x7fa8e004c3c0 con 0x7fa8c4066f30 2023-10-07T11:03:59.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:03:59.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa8c4066f30 msgr2=0x7fa8c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa8c4066f30 0x7fa8c40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00099b0 tx=0x7fa8d0017000 comp rx=0 tx=0).stop 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 msgr2=0x7fa8e006dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 0x7fa8e006dfe0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00c5c0 tx=0x7fa8dc00c980 comp rx=0 tx=0).stop 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 shutdown_connections 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8e0155860 0x7fa8e0155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00b1840 0x7fa8e006dfe0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa8c4066f30 0x7fa8c40693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 --2- 172.21.15.29:0/4130334485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8e0157290 0x7fa8e0145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.854+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 >> 172.21.15.29:0/4130334485 conn(0x7fa8e00680a0 msgr2=0x7fa8e0158df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.855+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 shutdown_connections 2023-10-07T11:03:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:03:59.855+0000 7fa8e73ff700 1 -- 172.21.15.29:0/4130334485 wait complete. 2023-10-07T11:03:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:03:59.872 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2023-10-07T11:03:59.713119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":2477256,"kb_used_data":21128,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752497464,"statfs":{"total":773094113280,"available":770557403136,"internally_reserved":0,"allocated":21635072,"data_stored":11481626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668755,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-281,"num_write_kb":-2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997810"},"pg_stats":[{"pgid":"1.0","version":"200'370","reported_seq":1211,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-07T11:03:53.781797+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:53.781797+0000","last_peered":"2023-10-07T11:03:53.781797+0000","last_clean":"2023-10-07T11:03:53.781797+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:53.781797+0000","last_undegraded":"2023-10-07T11:03:53.781797+0000","last_fullsized":"2023-10-07T11:03:53.781797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477858,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312928,"kb_used_data":3352,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058912,"statfs":{"total":96636764160,"available":96316325888,"internally_reserved":0,"allocated":3432448,"data_stored":2160345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.625}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.443}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.73199999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.747}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.65300000000000002}]}]},{"osd":6,"up_from":15,"seq":64424510562,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316104,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055736,"statfs":{"total":96636764160,"available":96313073664,"internally_reserved":0,"allocated":2490368,"data_stored":1234337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.751},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.64600000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":0.71799999999999997}]}]},{"osd":1,"up_from":14,"seq":60129543265,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307952,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063888,"statfs":{"total":96636764160,"available":96321421312,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.89900000000000002,"15min":0.96299999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.378,"15min":0.37},"max":{"1min":0.70399999999999996,"5min":0.84999999999999998,"15min":0.88900000000000001},"last":0.51200000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.51600000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.373},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.59999999999999998,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2430000000000001},"last":1.002}]},{"osd":5,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82699999999999996,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2310000000000001,"5min":1.242,"15min":1.242},"last":1.081},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.0840000000000001,"5min":1.2,"15min":1.2},"last":0.86499999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.871,"15min":0.874},"min":{"1min":0.67500000000000004,"5min":0.65000000000000002,"15min":0.58399999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0409999999999999,"5min":1.157,"15min":1.302},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.877,"15min":0.878},"min":{"1min":0.64700000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.98899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.435}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.40200000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.628},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.64100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543265,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299740,"kb_used_data":2452,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072100,"statfs":{"total":96636764160,"available":96329830400,"internally_reserved":0,"allocated":2510848,"data_stored":1242307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.81699999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.30399999999999999},"max":{"1min":0.69399999999999995,"5min":1.087,"15min":1.087},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.95099999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.80800000000000005,"5min":0.86399999999999999,"15min":0.98199999999999998},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59799999999999998,"5min":0.71799999999999997,"15min":0.80000000000000004},"last":0.749},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82199999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.78900000000000003},"min":{"1min":0.63100000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.163,"15min":1.1879999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.266,"15min":1.5640000000000001},"last":0.625}]},{"osd":5,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.59199999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.004,"5min":1.2989999999999999,"15min":1.528},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78100000000000003,"15min":0.80300000000000005},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95999999999999996,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.65200000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0580000000000001,"5min":1.242,"15min":1.3779999999999999},"last":0.79400000000000004}]},{"osd":7,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.86799999999999999},"min":{"1min":0.67600000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0269999999999999,"5min":1.2050000000000001,"15min":1.629},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0569999999999999,"5min":1.274,"15min":1.5940000000000001},"last":0.82599999999999996}]}]},{"osd":3,"up_from":14,"seq":60129543267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.48099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.47299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96199999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.626,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0669999999999999,"15min":1.222},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80700000000000005,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.216},"last":0.753},{"interface":"front","average":{"1min":0.753,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95399999999999996,"5min":1.27,"15min":1.27},"last":0.74299999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82499999999999996,"15min":0.84199999999999997},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92700000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.624},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.65400000000000003,"5min":0.56599999999999995,"15min":0.54600000000000004},"max":{"1min":1.046,"5min":1.389,"15min":1.389},"last":0.77400000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0429999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"max":{"1min":0.97599999999999998,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.70999999999999996}]}]},{"osd":4,"up_from":15,"seq":64424510562,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316136,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055704,"statfs":{"total":96636764160,"available":96313040896,"internally_reserved":0,"allocated":2523136,"data_stored":1254167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.66100000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.627}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.67100000000000004}]}]},{"osd":5,"up_from":14,"seq":60129543264,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312008,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059832,"statfs":{"total":96636764160,"available":96317267968,"internally_reserved":0,"allocated":2490368,"data_stored":1234337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.78000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93300000000000005,"5min":1.585,"15min":1.585},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.81299999999999994},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.90900000000000003,"5min":1.673,"15min":1.673},"last":0.59399999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73799999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":1.139},"last":0.67700000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92500000000000004,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.95399999999999996,"5min":1.0549999999999999,"15min":1.069},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.91400000000000003,"5min":0.96899999999999997,"15min":1.161},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64300000000000002,"15min":0.625},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.496}]},{"osd":7,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.69299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0109999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.038,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:03:59.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:04:00.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.069+0000 7f836f9ff700 1 Processor -- start 2023-10-07T11:04:00.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.069+0000 7f836f9ff700 1 -- start start 2023-10-07T11:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836f9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83680ad370 0x7f83680cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836f9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680b0410 0x7f836815a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836f9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836814d790 con 0x7f8368159ec0 2023-10-07T11:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836f9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836815a880 con 0x7f83680ad370 2023-10-07T11:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836f9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83680ae5d0 con 0x7f83680b0410 2023-10-07T11:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836d79b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83680ad370 0x7f83680cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836d79b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83680ad370 0x7f83680cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43496/0 (socket says 172.21.15.29:43496) 2023-10-07T11:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836d79b700 1 -- 172.21.15.29:0/953312414 learned_addr learned my addr 172.21.15.29:0/953312414 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.071+0000 7f836cf9a700 1 --1- 172.21.15.29:0/953312414 >> v1:172.21.15.29:6789/0 conn(0x7f8368159ec0 0x7f83681568c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41976/0 (socket says 172.21.15.29:41976) 2023-10-07T11:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.072+0000 7f836df9c700 1 --2- 172.21.15.29:0/953312414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680b0410 0x7f836815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.072+0000 7f836d79b700 1 -- 172.21.15.29:0/953312414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680b0410 msgr2=0x7f836815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.072+0000 7f836d79b700 1 --2- 172.21.15.29:0/953312414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680b0410 0x7f836815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.072+0000 7f836d79b700 1 -- 172.21.15.29:0/953312414 >> v1:172.21.15.29:6789/0 conn(0x7f8368159ec0 legacy=0x7f83681568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.072+0000 7f836d79b700 1 -- 172.21.15.29:0/953312414 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83680ad740 con 0x7f83680ad370 2023-10-07T11:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.072+0000 7f836d79b700 1 --2- 172.21.15.29:0/953312414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680ad370 0x7f83680cc870 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f836400bc80 tx=0x7f836400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf0aad6cf00db4b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.074+0000 7f835ffff700 1 -- 172.21.15.29:0/953312414 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836400dbb0 con 0x7f83680ad370 2023-10-07T11:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.074+0000 7f835ffff700 1 -- 172.21.15.29:0/953312414 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836400dd10 con 0x7f83680ad370 2023-10-07T11:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.074+0000 7f835ffff700 1 -- 172.21.15.29:0/953312414 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83640196b0 con 0x7f83680ad370 2023-10-07T11:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 -- 172.21.15.29:0/953312414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680ad370 msgr2=0x7f83680cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 --2- 172.21.15.29:0/953312414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680ad370 0x7f83680cc870 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f836400bc80 tx=0x7f836400b4f0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 -- 172.21.15.29:0/953312414 shutdown_connections 2023-10-07T11:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 --2- 172.21.15.29:0/953312414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680b0410 0x7f836815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 --2- 172.21.15.29:0/953312414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680ad370 0x7f83680cc870 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 -- 172.21.15.29:0/953312414 >> 172.21.15.29:0/953312414 conn(0x7f83680680a0 msgr2=0x7f83680cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:00.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 -- 172.21.15.29:0/953312414 shutdown_connections 2023-10-07T11:04:00.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.075+0000 7f836f9ff700 1 -- 172.21.15.29:0/953312414 wait complete. 2023-10-07T11:04:00.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.077+0000 7f836f9ff700 1 Processor -- start 2023-10-07T11:04:00.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.077+0000 7f836f9ff700 1 -- start start 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836f9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680ad370 0x7f83680cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836f9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 0x7f8368152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836f9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8368159ec0 0x7f83681528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836f9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8368152fe0 con 0x7f83680b0410 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836f9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83681532c0 con 0x7f8368159ec0 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836f9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83680c9150 con 0x7f83680ad370 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 0x7f8368152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 0x7f8368152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43502/0 (socket says 172.21.15.29:43502) 2023-10-07T11:04:00.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 -- 172.21.15.29:0/4198262445 learned_addr learned my addr 172.21.15.29:0/4198262445 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836d79b700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680ad370 0x7f83680cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836df9c700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8368159ec0 0x7f83681528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 -- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680ad370 msgr2=0x7f83680cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680ad370 0x7f83680cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 -- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8368159ec0 msgr2=0x7f83681528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.078+0000 7f836cf9a700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8368159ec0 0x7f83681528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f836cf9a700 1 -- 172.21.15.29:0/4198262445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836400b790 con 0x7f83680b0410 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f836df9c700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8368159ec0 0x7f83681528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f836cf9a700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 0x7f8368152390 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f835800c560 tx=0x7f8358007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358005a70 con 0x7f83680b0410 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8358005bd0 con 0x7f83680b0410 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83680c9430 con 0x7f83680b0410 2023-10-07T11:04:00.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83580070d0 con 0x7f83680b0410 2023-10-07T11:04:00.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.079+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8368154a20 con 0x7f83680b0410 2023-10-07T11:04:00.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.080+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8368154c40 con 0x7f83680b0410 2023-10-07T11:04:00.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.081+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8358007440 con 0x7f83680b0410 2023-10-07T11:04:00.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.082+0000 7f835d7fa700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f834c066f30 0x7f834c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.083+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f8358083b40 con 0x7f83680b0410 2023-10-07T11:04:00.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.083+0000 7f836d79b700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f834c066f30 0x7f834c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.084+0000 7f836d79b700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f834c066f30 0x7f834c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f836400bc80 tx=0x7f8364019000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.087+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8368154c40 con 0x7f83680b0410 2023-10-07T11:04:00.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.285+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83680cbda0 con 0x7f834c066f30 2023-10-07T11:04:00.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.288+0000 7f835d7fa700 1 -- 172.21.15.29:0/4198262445 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50639 (secure 0 0 0) 0x7f83680cbda0 con 0x7f834c066f30 2023-10-07T11:04:00.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f834c066f30 msgr2=0x7f834c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f834c066f30 0x7f834c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f836400bc80 tx=0x7f8364019000 comp rx=0 tx=0).stop 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 msgr2=0x7f8368152390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 0x7f8368152390 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f835800c560 tx=0x7f8358007f70 comp rx=0 tx=0).stop 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 shutdown_connections 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8368159ec0 0x7f83681528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f834c066f30 0x7f834c0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83680ad370 0x7f83680cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 --2- 172.21.15.29:0/4198262445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83680b0410 0x7f8368152390 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 >> 172.21.15.29:0/4198262445 conn(0x7f83680680a0 msgr2=0x7f836806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 shutdown_connections 2023-10-07T11:04:00.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.291+0000 7f836f9ff700 1 -- 172.21.15.29:0/4198262445 wait complete. 2023-10-07T11:04:00.293 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:04:00.309 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2023-10-07T11:03:59.713119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":2477256,"kb_used_data":21128,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752497464,"statfs":{"total":773094113280,"available":770557403136,"internally_reserved":0,"allocated":21635072,"data_stored":11481626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668755,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-281,"num_write_kb":-2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997810"},"pg_stats":[{"pgid":"1.0","version":"200'370","reported_seq":1211,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-07T11:03:53.781797+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:53.781797+0000","last_peered":"2023-10-07T11:03:53.781797+0000","last_clean":"2023-10-07T11:03:53.781797+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:53.781797+0000","last_undegraded":"2023-10-07T11:03:53.781797+0000","last_fullsized":"2023-10-07T11:03:53.781797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477858,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312928,"kb_used_data":3352,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058912,"statfs":{"total":96636764160,"available":96316325888,"internally_reserved":0,"allocated":3432448,"data_stored":2160345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.625}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.443}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.73199999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.747}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.65300000000000002}]}]},{"osd":6,"up_from":15,"seq":64424510562,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316104,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055736,"statfs":{"total":96636764160,"available":96313073664,"internally_reserved":0,"allocated":2490368,"data_stored":1234337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.751},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.64600000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":0.71799999999999997}]}]},{"osd":1,"up_from":14,"seq":60129543265,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307952,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063888,"statfs":{"total":96636764160,"available":96321421312,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.89900000000000002,"15min":0.96299999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.378,"15min":0.37},"max":{"1min":0.70399999999999996,"5min":0.84999999999999998,"15min":0.88900000000000001},"last":0.51200000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.51600000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.373},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.59999999999999998,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2430000000000001},"last":1.002}]},{"osd":5,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82699999999999996,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2310000000000001,"5min":1.242,"15min":1.242},"last":1.081},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.0840000000000001,"5min":1.2,"15min":1.2},"last":0.86499999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.871,"15min":0.874},"min":{"1min":0.67500000000000004,"5min":0.65000000000000002,"15min":0.58399999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0409999999999999,"5min":1.157,"15min":1.302},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.877,"15min":0.878},"min":{"1min":0.64700000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.98899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.435}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.40200000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.628},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.64100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543265,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299740,"kb_used_data":2452,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072100,"statfs":{"total":96636764160,"available":96329830400,"internally_reserved":0,"allocated":2510848,"data_stored":1242307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.81699999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.30399999999999999},"max":{"1min":0.69399999999999995,"5min":1.087,"15min":1.087},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.95099999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.80800000000000005,"5min":0.86399999999999999,"15min":0.98199999999999998},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59799999999999998,"5min":0.71799999999999997,"15min":0.80000000000000004},"last":0.749},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82199999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.78900000000000003},"min":{"1min":0.63100000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.163,"15min":1.1879999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.266,"15min":1.5640000000000001},"last":0.625}]},{"osd":5,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.59199999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.004,"5min":1.2989999999999999,"15min":1.528},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78100000000000003,"15min":0.80300000000000005},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95999999999999996,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.65200000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0580000000000001,"5min":1.242,"15min":1.3779999999999999},"last":0.79400000000000004}]},{"osd":7,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.86799999999999999},"min":{"1min":0.67600000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0269999999999999,"5min":1.2050000000000001,"15min":1.629},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0569999999999999,"5min":1.274,"15min":1.5940000000000001},"last":0.82599999999999996}]}]},{"osd":3,"up_from":14,"seq":60129543267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.48099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.47299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96199999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.626,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0669999999999999,"15min":1.222},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80700000000000005,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.216},"last":0.753},{"interface":"front","average":{"1min":0.753,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95399999999999996,"5min":1.27,"15min":1.27},"last":0.74299999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82499999999999996,"15min":0.84199999999999997},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92700000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.624},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.65400000000000003,"5min":0.56599999999999995,"15min":0.54600000000000004},"max":{"1min":1.046,"5min":1.389,"15min":1.389},"last":0.77400000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0429999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"max":{"1min":0.97599999999999998,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.70999999999999996}]}]},{"osd":4,"up_from":15,"seq":64424510562,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316136,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055704,"statfs":{"total":96636764160,"available":96313040896,"internally_reserved":0,"allocated":2523136,"data_stored":1254167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.66100000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.627}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.67100000000000004}]}]},{"osd":5,"up_from":14,"seq":60129543264,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312008,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059832,"statfs":{"total":96636764160,"available":96317267968,"internally_reserved":0,"allocated":2490368,"data_stored":1234337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.78000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93300000000000005,"5min":1.585,"15min":1.585},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.81299999999999994},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.90900000000000003,"5min":1.673,"15min":1.673},"last":0.59399999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73799999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":1.139},"last":0.67700000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92500000000000004,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.95399999999999996,"5min":1.0549999999999999,"15min":1.069},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.91400000000000003,"5min":0.96899999999999997,"15min":1.161},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64300000000000002,"15min":0.625},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.496}]},{"osd":7,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.69299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0109999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.038,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:04:00.310 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-07T11:04:00.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:04:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.505+0000 7faf9b7ff700 1 Processor -- start 2023-10-07T11:04:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.505+0000 7faf9b7ff700 1 -- start start 2023-10-07T11:04:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.506+0000 7faf9b7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 0x7faf940d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf9b7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 0x7faf940c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf9b7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940cbf20 con 0x7faf94159750 2023-10-07T11:04:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf9b7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf940cc200 con 0x7faf94159b20 2023-10-07T11:04:00.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf9b7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf940cc4e0 con 0x7faf940ce8a0 2023-10-07T11:04:00.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf99d9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 0x7faf940c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf98d9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 0x7faf940d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf99d9c700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 0x7faf940c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.29:40676/0 (socket says 172.21.15.29:40676) 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf98d9a700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 0x7faf940d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.29:60744/0 (socket says 172.21.15.29:60744) 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf9959b700 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7faf94159750 0x7faf940ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.29:58142/0 (socket says 172.21.15.29:58142) 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.507+0000 7faf99d9c700 1 -- 172.21.15.29:0/757926913 learned_addr learned my addr 172.21.15.29:0/757926913 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf99d9c700 1 -- 172.21.15.29:0/757926913 >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 msgr2=0x7faf940d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf99d9c700 1 --2- 172.21.15.29:0/757926913 >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 0x7faf940d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf99d9c700 1 -- 172.21.15.29:0/757926913 >> v1:172.21.15.144:6789/0 conn(0x7faf94159750 legacy=0x7faf940ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf99d9c700 1 -- 172.21.15.29:0/757926913 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940cc7c0 con 0x7faf940ce8a0 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf98d9a700 1 --2- 172.21.15.29:0/757926913 >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 0x7faf940d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:04:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf99d9c700 1 --2- 172.21.15.29:0/757926913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 0x7faf940c5460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009f20 tx=0x7faf8c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d4df25f3bb5731 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf8bfff700 1 -- 172.21.15.29:0/757926913 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c00e070 con 0x7faf940ce8a0 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf8bfff700 1 -- 172.21.15.29:0/757926913 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c00b9d0 con 0x7faf940ce8a0 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.508+0000 7faf8bfff700 1 -- 172.21.15.29:0/757926913 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0092e0 con 0x7faf940ce8a0 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 -- 172.21.15.29:0/757926913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 msgr2=0x7faf940c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/757926913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 0x7faf940c5460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009f20 tx=0x7faf8c009840 comp rx=0 tx=0).stop 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 -- 172.21.15.29:0/757926913 shutdown_connections 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/757926913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940ce8a0 0x7faf940c5460 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/757926913 >> v2:172.21.15.144:3300/0 conn(0x7faf94159b20 0x7faf940d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 -- 172.21.15.29:0/757926913 >> 172.21.15.29:0/757926913 conn(0x7faf94063a10 msgr2=0x7faf940d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 -- 172.21.15.29:0/757926913 shutdown_connections 2023-10-07T11:04:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.509+0000 7faf9b7ff700 1 -- 172.21.15.29:0/757926913 wait complete. 2023-10-07T11:04:00.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.510+0000 7faf9b7ff700 1 Processor -- start 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 -- start start 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940ce8a0 0x7faf94150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf94159750 0x7faf94151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 0x7faf9415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9415fdf0 con 0x7faf94159b20 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9415a240 con 0x7faf940ce8a0 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.511+0000 7faf9b7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9415f7a0 con 0x7faf94159750 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 0x7faf9415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 0x7faf9415aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43506/0 (socket says 172.21.15.29:43506) 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 -- 172.21.15.29:0/3788204085 learned_addr learned my addr 172.21.15.29:0/3788204085 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf9959b700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940ce8a0 0x7faf94150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf98d9a700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf94159750 0x7faf94151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 -- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf94159750 msgr2=0x7faf94151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf94159750 0x7faf94151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 -- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940ce8a0 msgr2=0x7faf94150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940ce8a0 0x7faf94150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.512+0000 7faf99d9c700 1 -- 172.21.15.29:0/3788204085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf84009420 con 0x7faf94159b20 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.513+0000 7faf99d9c700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 0x7faf9415aad0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00b610 tx=0x7faf8c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.513+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c008ad0 con 0x7faf94159b20 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.513+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c008c30 con 0x7faf94159b20 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.513+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c026460 con 0x7faf94159b20 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.513+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c009ae0 con 0x7faf94159b20 2023-10-07T11:04:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.513+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf94159ef0 con 0x7faf94159b20 2023-10-07T11:04:00.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.514+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9415fc40 con 0x7faf94159b20 2023-10-07T11:04:00.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.515+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faf8c003040 con 0x7faf94159b20 2023-10-07T11:04:00.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.516+0000 7faf897fa700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faf78066f30 0x7faf780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.516+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7faf8c085fc0 con 0x7faf94159b20 2023-10-07T11:04:00.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.516+0000 7faf9959b700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faf78066f30 0x7faf780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.517+0000 7faf9959b700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faf78066f30 0x7faf780693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf94153b00 tx=0x7faf90007ea0 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.520+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf8c053080 con 0x7faf94159b20 2023-10-07T11:04:00.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.719+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faf9404c3c0 con 0x7faf78066f30 2023-10-07T11:04:00.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.722+0000 7faf897fa700 1 -- 172.21.15.29:0/3788204085 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50639 (secure 0 0 0) 0x7faf9404c3c0 con 0x7faf78066f30 2023-10-07T11:04:00.724 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faf78066f30 msgr2=0x7faf780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faf78066f30 0x7faf780693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf94153b00 tx=0x7faf90007ea0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 msgr2=0x7faf9415aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 0x7faf9415aad0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00b610 tx=0x7faf8c018000 comp rx=0 tx=0).stop 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 shutdown_connections 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf94159b20 0x7faf9415aad0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940ce8a0 0x7faf94150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7faf78066f30 0x7faf780693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 --2- 172.21.15.29:0/3788204085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf94159750 0x7faf94151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 >> 172.21.15.29:0/3788204085 conn(0x7faf94063a10 msgr2=0x7faf940c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 shutdown_connections 2023-10-07T11:04:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.725+0000 7faf9b7ff700 1 -- 172.21.15.29:0/3788204085 wait complete. 2023-10-07T11:04:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:04:00.743 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2023-10-07T11:03:59.713119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":2477256,"kb_used_data":21128,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752497464,"statfs":{"total":773094113280,"available":770557403136,"internally_reserved":0,"allocated":21635072,"data_stored":11481626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668755,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-281,"num_write_kb":-2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997810"},"pg_stats":[{"pgid":"1.0","version":"200'370","reported_seq":1211,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-07T11:03:53.781797+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:53.781797+0000","last_peered":"2023-10-07T11:03:53.781797+0000","last_clean":"2023-10-07T11:03:53.781797+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:53.781797+0000","last_undegraded":"2023-10-07T11:03:53.781797+0000","last_fullsized":"2023-10-07T11:03:53.781797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477858,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312928,"kb_used_data":3352,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058912,"statfs":{"total":96636764160,"available":96316325888,"internally_reserved":0,"allocated":3432448,"data_stored":2160345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.625}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.443}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.73199999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.747}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.65300000000000002}]}]},{"osd":6,"up_from":15,"seq":64424510562,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316104,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055736,"statfs":{"total":96636764160,"available":96313073664,"internally_reserved":0,"allocated":2490368,"data_stored":1234337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.751},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.64600000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":0.71799999999999997}]}]},{"osd":1,"up_from":14,"seq":60129543265,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307952,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063888,"statfs":{"total":96636764160,"available":96321421312,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.89900000000000002,"15min":0.96299999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.378,"15min":0.37},"max":{"1min":0.70399999999999996,"5min":0.84999999999999998,"15min":0.88900000000000001},"last":0.51200000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.51600000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.373},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.59999999999999998,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2430000000000001},"last":1.002}]},{"osd":5,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82699999999999996,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2310000000000001,"5min":1.242,"15min":1.242},"last":1.081},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.0840000000000001,"5min":1.2,"15min":1.2},"last":0.86499999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.871,"15min":0.874},"min":{"1min":0.67500000000000004,"5min":0.65000000000000002,"15min":0.58399999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0409999999999999,"5min":1.157,"15min":1.302},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.877,"15min":0.878},"min":{"1min":0.64700000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.98899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.435}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.40200000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.628},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.64100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543265,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299740,"kb_used_data":2452,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072100,"statfs":{"total":96636764160,"available":96329830400,"internally_reserved":0,"allocated":2510848,"data_stored":1242307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.81699999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.30399999999999999},"max":{"1min":0.69399999999999995,"5min":1.087,"15min":1.087},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.95099999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.80800000000000005,"5min":0.86399999999999999,"15min":0.98199999999999998},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59799999999999998,"5min":0.71799999999999997,"15min":0.80000000000000004},"last":0.749},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82199999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.78900000000000003},"min":{"1min":0.63100000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.163,"15min":1.1879999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.266,"15min":1.5640000000000001},"last":0.625}]},{"osd":5,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.59199999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.004,"5min":1.2989999999999999,"15min":1.528},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78100000000000003,"15min":0.80300000000000005},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95999999999999996,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.65200000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0580000000000001,"5min":1.242,"15min":1.3779999999999999},"last":0.79400000000000004}]},{"osd":7,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.86799999999999999},"min":{"1min":0.67600000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0269999999999999,"5min":1.2050000000000001,"15min":1.629},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0569999999999999,"5min":1.274,"15min":1.5940000000000001},"last":0.82599999999999996}]}]},{"osd":3,"up_from":14,"seq":60129543267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.48099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.47299999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96199999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.626,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0669999999999999,"15min":1.222},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80700000000000005,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.216},"last":0.753},{"interface":"front","average":{"1min":0.753,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95399999999999996,"5min":1.27,"15min":1.27},"last":0.74299999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82499999999999996,"15min":0.84199999999999997},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92700000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.624},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.65400000000000003,"5min":0.56599999999999995,"15min":0.54600000000000004},"max":{"1min":1.046,"5min":1.389,"15min":1.389},"last":0.77400000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0429999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"max":{"1min":0.97599999999999998,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.70999999999999996}]}]},{"osd":4,"up_from":15,"seq":64424510562,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316136,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055704,"statfs":{"total":96636764160,"available":96313040896,"internally_reserved":0,"allocated":2523136,"data_stored":1254167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.66100000000000003}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.627}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.67100000000000004}]}]},{"osd":5,"up_from":14,"seq":60129543264,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312008,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059832,"statfs":{"total":96636764160,"available":96317267968,"internally_reserved":0,"allocated":2490368,"data_stored":1234337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.78000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93300000000000005,"5min":1.585,"15min":1.585},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.81299999999999994},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.90900000000000003,"5min":1.673,"15min":1.673},"last":0.59399999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73799999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":1.139},"last":0.67700000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92500000000000004,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.95399999999999996,"5min":1.0549999999999999,"15min":1.069},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.91400000000000003,"5min":0.96899999999999997,"15min":1.161},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64300000000000002,"15min":0.625},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.496}]},{"osd":7,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.69299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0109999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.038,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:04:00.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-07T11:04:00.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.941+0000 7f2003fff700 1 Processor -- start 2023-10-07T11:04:00.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.941+0000 7f2003fff700 1 -- start start 2023-10-07T11:04:00.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.942+0000 7f2003fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ffc159750 0x7f1ffc0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.942+0000 7f2003fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ffc159b20 0x7f1ffc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.942+0000 7f2003fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc0cbf20 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.942+0000 7f2003fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0cc200 con 0x7f1ffc159750 2023-10-07T11:04:00.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.942+0000 7f2003fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0cc4e0 con 0x7f1ffc159b20 2023-10-07T11:04:00.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ffc159750 0x7f1ffc0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ffc159750 0x7f1ffc0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43520/0 (socket says 172.21.15.29:43520) 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 -- 172.21.15.29:0/2391755358 learned_addr learned my addr 172.21.15.29:0/2391755358 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f200159a700 1 --2- 172.21.15.29:0/2391755358 >> v2:172.21.15.144:3300/0 conn(0x7f1ffc159b20 0x7f1ffc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 -- 172.21.15.29:0/2391755358 >> v2:172.21.15.144:3300/0 conn(0x7f1ffc159b20 msgr2=0x7f1ffc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 --2- 172.21.15.29:0/2391755358 >> v2:172.21.15.144:3300/0 conn(0x7f1ffc159b20 0x7f1ffc0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 -- 172.21.15.29:0/2391755358 >> v1:172.21.15.29:6789/0 conn(0x7f1ffc0ce6d0 legacy=0x7f1ffc15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.943+0000 7f2001d9b700 1 -- 172.21.15.29:0/2391755358 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0cc7c0 con 0x7f1ffc159750 2023-10-07T11:04:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.944+0000 7f2001d9b700 1 --2- 172.21.15.29:0/2391755358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc159750 0x7f1ffc0d10a0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc088a70 tx=0x7f1ff8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898fea41687b9415 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.944+0000 7f2000d99700 1 -- 172.21.15.29:0/2391755358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8005c10 con 0x7f1ffc159750 2023-10-07T11:04:00.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.944+0000 7f2000d99700 1 -- 172.21.15.29:0/2391755358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8005d70 con 0x7f1ffc159750 2023-10-07T11:04:00.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.944+0000 7f2000d99700 1 -- 172.21.15.29:0/2391755358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff800b880 con 0x7f1ffc159750 2023-10-07T11:04:00.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.944+0000 7f2003fff700 1 -- 172.21.15.29:0/2391755358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc159750 msgr2=0x7f1ffc0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.944+0000 7f2003fff700 1 --2- 172.21.15.29:0/2391755358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc159750 0x7f1ffc0d10a0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc088a70 tx=0x7f1ff8009220 comp rx=0 tx=0).stop 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.945+0000 7f2003fff700 1 -- 172.21.15.29:0/2391755358 shutdown_connections 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.945+0000 7f2003fff700 1 --2- 172.21.15.29:0/2391755358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc159750 0x7f1ffc0d10a0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.945+0000 7f2003fff700 1 --2- 172.21.15.29:0/2391755358 >> v2:172.21.15.144:3300/0 conn(0x7f1ffc159b20 0x7f1ffc0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.945+0000 7f2003fff700 1 -- 172.21.15.29:0/2391755358 >> 172.21.15.29:0/2391755358 conn(0x7f1ffc063a10 msgr2=0x7f1ffc0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.945+0000 7f2003fff700 1 -- 172.21.15.29:0/2391755358 shutdown_connections 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.945+0000 7f2003fff700 1 -- 172.21.15.29:0/2391755358 wait complete. 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.946+0000 7f2003fff700 1 Processor -- start 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.946+0000 7f2003fff700 1 -- start start 2023-10-07T11:04:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2003fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 0x7f1ffc145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2003fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ffc159750 0x7f1ffc15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2003fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ffc159b20 0x7f1ffc15c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2003fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0b60b0 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2003fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc1455d0 con 0x7f1ffc159b20 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2003fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0b4ca0 con 0x7f1ffc159750 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2001d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 0x7f1ffc145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2001d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 0x7f1ffc145090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43532/0 (socket says 172.21.15.29:43532) 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.947+0000 7f2001d9b700 1 -- 172.21.15.29:0/2871687776 learned_addr learned my addr 172.21.15.29:0/2871687776 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f200159a700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ffc159750 0x7f1ffc15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2001d9b700 1 -- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ffc159750 msgr2=0x7f1ffc15c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2001d9b700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ffc159750 0x7f1ffc15c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f200259c700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ffc159b20 0x7f1ffc15c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2001d9b700 1 -- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ffc159b20 msgr2=0x7f1ffc15c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2001d9b700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ffc159b20 0x7f1ffc15c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2001d9b700 1 -- 172.21.15.29:0/2871687776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8009480 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f200159a700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ffc159750 0x7f1ffc15c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f200259c700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ffc159b20 0x7f1ffc15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2001d9b700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 0x7f1ffc145090 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800b3c0 tx=0x7f1ff8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80118a0 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8020070 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc0b49c0 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.948+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8011c70 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.949+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc15a300 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.950+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc04c3c0 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.950+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1ff8022030 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.951+0000 7f1ff1ffb700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1fe0066f30 0x7f1fe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:00.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.951+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f1ff8086220 con 0x7f1ffc0ce6d0 2023-10-07T11:04:00.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.952+0000 7f200159a700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1fe0066f30 0x7f1fe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:00.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.952+0000 7f200159a700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1fe0066f30 0x7f1fe00693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc16cfd0 tx=0x7f1fec008040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:00.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:00.956+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ff8053380 con 0x7f1ffc0ce6d0 2023-10-07T11:04:01.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.158+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ffc069c40 con 0x7f1ffc0ce6d0 2023-10-07T11:04:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.159+0000 7f1ff1ffb700 1 -- 172.21.15.29:0/2871687776 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+13917 (secure 0 0 0) 0x7f1ff80569a0 con 0x7f1ffc0ce6d0 2023-10-07T11:04:01.160 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:04:01.160 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":202,"fsid":"1e617e23-0c6c-40cc-8e09-10eaf7e38e24","created":"2023-10-07T09:28:57.117998+0000","modified":"2023-10-07T11:03:56.059727+0000","last_up_change":"2023-10-07T09:29:10.517805+0000","last_in_change":"2023-10-07T09:29:00.498643+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-07T09:29:10.085343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a149e59b-23fb-41a2-90c8-0973d32e496a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":115181},{"type":"v1","addr":"172.21.15.29:6801","nonce":115181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":115181},{"type":"v1","addr":"172.21.15.29:6803","nonce":115181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":115181},{"type":"v1","addr":"172.21.15.29:6807","nonce":115181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":115181},{"type":"v1","addr":"172.21.15.29:6805","nonce":115181}]},"public_addr":"172.21.15.29:6801/115181","cluster_addr":"172.21.15.29:6803/115181","heartbeat_back_addr":"172.21.15.29:6807/115181","heartbeat_front_addr":"172.21.15.29:6805/115181","state":["exists","up"]},{"osd":1,"uuid":"492dce8b-159d-45a2-b902-e9797b66c967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":115184},{"type":"v1","addr":"172.21.15.29:6809","nonce":115184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":115184},{"type":"v1","addr":"172.21.15.29:6811","nonce":115184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":115184},{"type":"v1","addr":"172.21.15.29:6815","nonce":115184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":115184},{"type":"v1","addr":"172.21.15.29:6813","nonce":115184}]},"public_addr":"172.21.15.29:6809/115184","cluster_addr":"172.21.15.29:6811/115184","heartbeat_back_addr":"172.21.15.29:6815/115184","heartbeat_front_addr":"172.21.15.29:6813/115184","state":["exists","up"]},{"osd":2,"uuid":"ba67af3d-2617-4e93-aa29-8b22dcc465af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":115200},{"type":"v1","addr":"172.21.15.29:6817","nonce":115200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":115200},{"type":"v1","addr":"172.21.15.29:6819","nonce":115200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":115200},{"type":"v1","addr":"172.21.15.29:6828","nonce":115200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":115200},{"type":"v1","addr":"172.21.15.29:6824","nonce":115200}]},"public_addr":"172.21.15.29:6817/115200","cluster_addr":"172.21.15.29:6819/115200","heartbeat_back_addr":"172.21.15.29:6828/115200","heartbeat_front_addr":"172.21.15.29:6824/115200","state":["exists","up"]},{"osd":3,"uuid":"7565c3dd-b998-49b7-888e-dcd298f58b12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":115201},{"type":"v1","addr":"172.21.15.29:6822","nonce":115201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":115201},{"type":"v1","addr":"172.21.15.29:6825","nonce":115201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":115201},{"type":"v1","addr":"172.21.15.29:6831","nonce":115201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6827","nonce":115201},{"type":"v1","addr":"172.21.15.29:6829","nonce":115201}]},"public_addr":"172.21.15.29:6822/115201","cluster_addr":"172.21.15.29:6825/115201","heartbeat_back_addr":"172.21.15.29:6831/115201","heartbeat_front_addr":"172.21.15.29:6829/115201","state":["exists","up"]},{"osd":4,"uuid":"4ae85342-f034-4c80-85fc-58bf598250cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":115417},{"type":"v1","addr":"172.21.15.144:6801","nonce":115417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":115417},{"type":"v1","addr":"172.21.15.144:6803","nonce":115417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6807","nonce":115417},{"type":"v1","addr":"172.21.15.144:6809","nonce":115417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":115417},{"type":"v1","addr":"172.21.15.144:6805","nonce":115417}]},"public_addr":"172.21.15.144:6801/115417","cluster_addr":"172.21.15.144:6803/115417","heartbeat_back_addr":"172.21.15.144:6809/115417","heartbeat_front_addr":"172.21.15.144:6805/115417","state":["exists","up"]},{"osd":5,"uuid":"84d41a87-cf09-485a-be54-e758964f42d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":115418},{"type":"v1","addr":"172.21.15.144:6808","nonce":115418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":115418},{"type":"v1","addr":"172.21.15.144:6811","nonce":115418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":115418},{"type":"v1","addr":"172.21.15.144:6815","nonce":115418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":115418},{"type":"v1","addr":"172.21.15.144:6813","nonce":115418}]},"public_addr":"172.21.15.144:6808/115418","cluster_addr":"172.21.15.144:6811/115418","heartbeat_back_addr":"172.21.15.144:6815/115418","heartbeat_front_addr":"172.21.15.144:6813/115418","state":["exists","up"]},{"osd":6,"uuid":"ce23a7b2-9566-436e-a3ba-12829f83c125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":115448},{"type":"v1","addr":"172.21.15.144:6825","nonce":115448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":115448},{"type":"v1","addr":"172.21.15.144:6827","nonce":115448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":115448},{"type":"v1","addr":"172.21.15.144:6831","nonce":115448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":115448},{"type":"v1","addr":"172.21.15.144:6829","nonce":115448}]},"public_addr":"172.21.15.144:6825/115448","cluster_addr":"172.21.15.144:6827/115448","heartbeat_back_addr":"172.21.15.144:6831/115448","heartbeat_front_addr":"172.21.15.144:6829/115448","state":["exists","up"]},{"osd":7,"uuid":"41a28292-240f-45fc-b073-c2e8a14106bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":115419},{"type":"v1","addr":"172.21.15.144:6817","nonce":115419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":115419},{"type":"v1","addr":"172.21.15.144:6819","nonce":115419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":115419},{"type":"v1","addr":"172.21.15.144:6823","nonce":115419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":115419},{"type":"v1","addr":"172.21.15.144:6821","nonce":115419}]},"public_addr":"172.21.15.144:6817/115419","cluster_addr":"172.21.15.144:6819/115419","heartbeat_back_addr":"172.21.15.144:6823/115419","heartbeat_front_addr":"172.21.15.144:6821/115419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.817591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.066011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.895520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.876504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.837098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:05.872518+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:09.272278+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-07T09:29:06.159236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3284787430":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/2235240922":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1299585123":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1947083474":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1254325888":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/1413587217":"2023-10-08T11:03:53.691236+0000","172.21.15.29:6834/1070134873":"2023-10-08T11:03:53.053802+0000","172.21.15.29:6832/1070134873":"2023-10-08T11:03:53.053802+0000","172.21.15.144:0/2912144557":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/471864733":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6837/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:6836/131978":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/245402490":"2023-10-08T11:03:43.498860+0000","172.21.15.144:0/4011736589":"2023-10-08T11:03:53.691236+0000","172.21.15.144:0/3907222325":"2023-10-08T11:03:43.498860+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":16,"length":1},{"begin":21,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":47,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":63,"length":2},{"begin":69,"length":1},{"begin":74,"length":2},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":91,"length":2},{"begin":96,"length":2},{"begin":103,"length":1},{"begin":107,"length":2},{"begin":195,"length":27}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":17,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":21,"snaps":[{"begin":2,"length":32}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-07T11:04:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1fe0066f30 msgr2=0x7f1fe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1fe0066f30 0x7f1fe00693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc16cfd0 tx=0x7f1fec008040 comp rx=0 tx=0).stop 2023-10-07T11:04:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 msgr2=0x7f1ffc145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 0x7f1ffc145090 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800b3c0 tx=0x7f1ff8016000 comp rx=0 tx=0).stop 2023-10-07T11:04:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 shutdown_connections 2023-10-07T11:04:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ffc159b20 0x7f1ffc15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:01.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ffc0ce6d0 0x7f1ffc145090 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:01.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1fe0066f30 0x7f1fe00693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:01.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 --2- 172.21.15.29:0/2871687776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ffc159750 0x7f1ffc15c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:01.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 >> 172.21.15.29:0/2871687776 conn(0x7f1ffc063a10 msgr2=0x7f1ffc154490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:01.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 shutdown_connections 2023-10-07T11:04:01.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:01.161+0000 7f2003fff700 1 -- 172.21.15.29:0/2871687776 wait complete. 2023-10-07T11:04:02.180 INFO:tasks.ceph:Scrubbing osd.0 2023-10-07T11:04:02.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.390+0000 7f61069ff700 1 Processor -- start 2023-10-07T11:04:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.390+0000 7f61069ff700 1 -- start start 2023-10-07T11:04:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.391+0000 7f61069ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61000accf0 0x7f61000b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.391+0000 7f61069ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000ad400 con 0x7f61000cc810 2023-10-07T11:04:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.391+0000 7f61069ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000ad6e0 con 0x7f61000afeb0 2023-10-07T11:04:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.391+0000 7f61069ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610015a140 con 0x7f61000accf0 2023-10-07T11:04:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.391+0000 7f60ff7fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61000cc810 0x7f610015a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42002/0 (socket says 172.21.15.29:42002) 2023-10-07T11:04:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f60ff7fe700 1 -- 172.21.15.29:0/383366435 learned_addr learned my addr 172.21.15.29:0/383366435 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:04:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f6104f9c700 1 --2- 172.21.15.29:0/383366435 >> v2:172.21.15.29:3300/0 conn(0x7f61000accf0 0x7f61000b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f60feffd700 1 -- 172.21.15.29:0/383366435 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053715977 0 0) 0x7f61000ad400 con 0x7f61000cc810 2023-10-07T11:04:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f60feffd700 1 -- 172.21.15.29:0/383366435 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e8003570 con 0x7f61000cc810 2023-10-07T11:04:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f6104f9c700 1 -- 172.21.15.29:0/383366435 >> v1:172.21.15.144:6789/0 conn(0x7f61000afeb0 legacy=0x7f61000aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f6104f9c700 1 -- 172.21.15.29:0/383366435 >> v1:172.21.15.29:6789/0 conn(0x7f61000cc810 legacy=0x7f610015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f6104f9c700 1 -- 172.21.15.29:0/383366435 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000ad9c0 con 0x7f61000accf0 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.392+0000 7f6104f9c700 1 --2- 172.21.15.29:0/383366435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f61000b05d0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009e60 tx=0x7f60f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772de83ef122773 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f60feffd700 1 -- 172.21.15.29:0/383366435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f800e070 con 0x7f61000accf0 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f60feffd700 1 -- 172.21.15.29:0/383366435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f800bb60 con 0x7f61000accf0 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f60feffd700 1 -- 172.21.15.29:0/383366435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8012430 con 0x7f61000accf0 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 -- 172.21.15.29:0/383366435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 msgr2=0x7f61000b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 --2- 172.21.15.29:0/383366435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f61000b05d0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009e60 tx=0x7f60f8009780 comp rx=0 tx=0).stop 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 -- 172.21.15.29:0/383366435 shutdown_connections 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 --2- 172.21.15.29:0/383366435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f61000b05d0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 -- 172.21.15.29:0/383366435 >> 172.21.15.29:0/383366435 conn(0x7f6100067610 msgr2=0x7f6100067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 -- 172.21.15.29:0/383366435 shutdown_connections 2023-10-07T11:04:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.393+0000 7f61069ff700 1 -- 172.21.15.29:0/383366435 wait complete. 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.395+0000 7f61069ff700 1 Processor -- start 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.395+0000 7f61069ff700 1 -- start start 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f61069ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f6100156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f61069ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61000afeb0 0x7f6100140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f61069ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61000cc810 0x7f61001589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f61069ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6100157500 con 0x7f61000accf0 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f61069ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001568d0 con 0x7f61000afeb0 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f61069ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61000c9a70 con 0x7f61000cc810 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f60fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f6100156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f60fffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f6100156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43546/0 (socket says 172.21.15.29:43546) 2023-10-07T11:04:02.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f60fffff700 1 -- 172.21.15.29:0/70352901 learned_addr learned my addr 172.21.15.29:0/70352901 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f60ff7fe700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61000afeb0 0x7f6100140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f60fffff700 1 -- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61000cc810 msgr2=0x7f61001589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.396+0000 7f6104f9c700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61000cc810 0x7f61001589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60fffff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61000cc810 0x7f61001589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60fffff700 1 -- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61000afeb0 msgr2=0x7f6100140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60fffff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61000afeb0 0x7f6100140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60fffff700 1 -- 172.21.15.29:0/70352901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8009a20 con 0x7f61000accf0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f6104f9c700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61000cc810 0x7f61001589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60ff7fe700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61000afeb0 0x7f6100140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60fffff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f6100156fc0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c590 tx=0x7f60f000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0011590 con 0x7f61000accf0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f00116f0 con 0x7f61000accf0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61000c9d50 con 0x7f61000accf0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0011ab0 con 0x7f61000accf0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.397+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61000ca310 con 0x7f61000accf0 2023-10-07T11:04:02.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.398+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61000718b0 con 0x7f61000accf0 2023-10-07T11:04:02.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.399+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f60f001b3b0 con 0x7f61000accf0 2023-10-07T11:04:02.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.400+0000 7f60e7fff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f60ec066f30 0x7f60ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.400+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f60f0083c90 con 0x7f61000accf0 2023-10-07T11:04:02.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.400+0000 7f60e7fff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7f60ec06d670 0x7f60ec06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.400+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 --> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60ec0700f0 con 0x7f60ec06d670 2023-10-07T11:04:02.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.400+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60f0088220 con 0x7f61000accf0 2023-10-07T11:04:02.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.401+0000 7f6104f9c700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7f60ec06d670 0x7f60ec06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.401+0000 7f60ff7fe700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f60ec066f30 0x7f60ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.401+0000 7f6104f9c700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7f60ec06d670 0x7f60ec06fac0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.401+0000 7f60ff7fe700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f60ec066f30 0x7f60ec0693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60f40058d0 tx=0x7f60f4009040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.402+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== osd.0 v2:172.21.15.29:6800/115181 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f60ec0700f0 con 0x7f60ec06d670 2023-10-07T11:04:02.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.416+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 --> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f61000afa40 con 0x7f60ec06d670 2023-10-07T11:04:02.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f60e7fff700 1 -- 172.21.15.29:0/70352901 <== osd.0 v2:172.21.15.29:6800/115181 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f61000afa40 con 0x7f60ec06d670 2023-10-07T11:04:02.418 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:02.418 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7f60ec06d670 msgr2=0x7f60ec06fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:6800/115181,v1:172.21.15.29:6801/115181] conn(0x7f60ec06d670 0x7f60ec06fac0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f60ec066f30 msgr2=0x7f60ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f60ec066f30 0x7f60ec0693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60f40058d0 tx=0x7f60f4009040 comp rx=0 tx=0).stop 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 msgr2=0x7f6100156fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 --2- 172.21.15.29:0/70352901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61000accf0 0x7f6100156fc0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c590 tx=0x7f60f000c950 comp rx=0 tx=0).stop 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f6104f9c700 1 -- 172.21.15.29:0/70352901 reap_dead start 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 shutdown_connections 2023-10-07T11:04:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 >> 172.21.15.29:0/70352901 conn(0x7f6100067610 msgr2=0x7f6100150e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 shutdown_connections 2023-10-07T11:04:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.417+0000 7f61069ff700 1 -- 172.21.15.29:0/70352901 wait complete. 2023-10-07T11:04:02.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-07T11:04:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.632+0000 7f7aefbff700 1 Processor -- start 2023-10-07T11:04:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.633+0000 7f7aefbff700 1 -- start start 2023-10-07T11:04:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.634+0000 7f7aefbff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ae81533f0 0x7f7ae8156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.634+0000 7f7aefbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae81537c0 0x7f7ae80afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aefbff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7ae80af6d0 0x7f7ae80b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aefbff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae815a680 con 0x7f7ae81533f0 2023-10-07T11:04:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aefbff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae80b2070 con 0x7f7ae80af6d0 2023-10-07T11:04:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aefbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8159670 con 0x7f7ae81537c0 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ae81533f0 0x7f7ae8156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ae81533f0 0x7f7ae8156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43548/0 (socket says 172.21.15.29:43548) 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 -- 172.21.15.29:0/945783950 learned_addr learned my addr 172.21.15.29:0/945783950 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed19a700 1 --2- 172.21.15.29:0/945783950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae81537c0 0x7f7ae80afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aee19c700 1 --2- 172.21.15.29:0/945783950 >> v2:172.21.15.144:3300/0 conn(0x7f7ae80af6d0 0x7f7ae80b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 -- 172.21.15.29:0/945783950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae81537c0 msgr2=0x7f7ae80afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 --2- 172.21.15.29:0/945783950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae81537c0 0x7f7ae80afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 -- 172.21.15.29:0/945783950 >> v2:172.21.15.144:3300/0 conn(0x7f7ae80af6d0 msgr2=0x7f7ae80b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 --2- 172.21.15.29:0/945783950 >> v2:172.21.15.144:3300/0 conn(0x7f7ae80af6d0 0x7f7ae80b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.635+0000 7f7aed99b700 1 -- 172.21.15.29:0/945783950 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae815a1f0 con 0x7f7ae81533f0 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aed99b700 1 --2- 172.21.15.29:0/945783950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae8156a90 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80cbcb0 tx=0x7f7ae4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2fadf409ccd83b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aec999700 1 -- 172.21.15.29:0/945783950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400b860 con 0x7f7ae81533f0 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aec999700 1 -- 172.21.15.29:0/945783950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae400b9c0 con 0x7f7ae81533f0 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aec999700 1 -- 172.21.15.29:0/945783950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400bd80 con 0x7f7ae81533f0 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 -- 172.21.15.29:0/945783950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 msgr2=0x7f7ae8156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 --2- 172.21.15.29:0/945783950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae8156a90 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80cbcb0 tx=0x7f7ae4009250 comp rx=0 tx=0).stop 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 -- 172.21.15.29:0/945783950 shutdown_connections 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 --2- 172.21.15.29:0/945783950 >> v2:172.21.15.144:3300/0 conn(0x7f7ae80af6d0 0x7f7ae80b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 --2- 172.21.15.29:0/945783950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae8156a90 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 --2- 172.21.15.29:0/945783950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae81537c0 0x7f7ae80afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.636+0000 7f7aefbff700 1 -- 172.21.15.29:0/945783950 >> 172.21.15.29:0/945783950 conn(0x7f7ae80680a0 msgr2=0x7f7ae80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.637+0000 7f7aefbff700 1 -- 172.21.15.29:0/945783950 shutdown_connections 2023-10-07T11:04:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.637+0000 7f7aefbff700 1 -- 172.21.15.29:0/945783950 wait complete. 2023-10-07T11:04:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.638+0000 7f7aefbff700 1 Processor -- start 2023-10-07T11:04:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.638+0000 7f7aefbff700 1 -- start start 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aefbff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae80af6d0 0x7f7ae8149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aefbff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae81509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aefbff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae81537c0 0x7f7ae816a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aefbff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8150520 con 0x7f7ae81533f0 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aefbff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8149f70 con 0x7f7ae81537c0 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aefbff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae814f910 con 0x7f7ae80af6d0 2023-10-07T11:04:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aed19a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae81509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aed19a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae81509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43552/0 (socket says 172.21.15.29:43552) 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.639+0000 7f7aed19a700 1 -- 172.21.15.29:0/1656843473 learned_addr learned my addr 172.21.15.29:0/1656843473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed99b700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae80af6d0 0x7f7ae8149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed19a700 1 -- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae80af6d0 msgr2=0x7f7ae8149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed19a700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae80af6d0 0x7f7ae8149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aee19c700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae81537c0 0x7f7ae816a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed19a700 1 -- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae81537c0 msgr2=0x7f7ae816a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed19a700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae81537c0 0x7f7ae816a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed19a700 1 -- 172.21.15.29:0/1656843473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae40094f0 con 0x7f7ae81533f0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aee19c700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae81537c0 0x7f7ae816a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aed19a700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae81509b0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800d860 tx=0x7f7ad8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8009610 con 0x7f7ae81533f0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad8009770 con 0x7f7ae81533f0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8009b30 con 0x7f7ae81533f0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.640+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae814fbf0 con 0x7f7ae81533f0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.641+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae81501b0 con 0x7f7ae81533f0 2023-10-07T11:04:02.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.642+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae8150690 con 0x7f7ae81533f0 2023-10-07T11:04:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.643+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7ad801e460 con 0x7f7ae81533f0 2023-10-07T11:04:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.644+0000 7f7addffb700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7acc066f30 0x7f7acc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.644+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f7ad8084cb0 con 0x7f7ae81533f0 2023-10-07T11:04:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.645+0000 7f7aed99b700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7acc066f30 0x7f7acc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.645+0000 7f7aed99b700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7acc066f30 0x7f7acc0693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80cbcb0 tx=0x7f7ae4017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.648+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ad8051e10 con 0x7f7ae81533f0 2023-10-07T11:04:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.853+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ae804c3c0 con 0x7f7acc066f30 2023-10-07T11:04:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.854+0000 7f7addffb700 1 -- 172.21.15.29:0/1656843473 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7ae804c3c0 con 0x7f7acc066f30 2023-10-07T11:04:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7acc066f30 msgr2=0x7f7acc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7acc066f30 0x7f7acc0693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80cbcb0 tx=0x7f7ae4017000 comp rx=0 tx=0).stop 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 msgr2=0x7f7ae81509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae81509b0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800d860 tx=0x7f7ad8008040 comp rx=0 tx=0).stop 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 shutdown_connections 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae81537c0 0x7f7ae816a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7acc066f30 0x7f7acc0693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ae80af6d0 0x7f7ae8149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 --2- 172.21.15.29:0/1656843473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae81533f0 0x7f7ae81509b0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.856+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 >> 172.21.15.29:0/1656843473 conn(0x7f7ae80680a0 msgr2=0x7f7ae814cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:02.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.857+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 shutdown_connections 2023-10-07T11:04:02.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:02.857+0000 7f7aefbff700 1 -- 172.21.15.29:0/1656843473 wait complete. 2023-10-07T11:04:02.859 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 0 to deep-scrub 2023-10-07T11:04:02.877 INFO:tasks.ceph:Scrubbing osd.1 2023-10-07T11:04:02.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.076+0000 7f706abff700 1 Processor -- start 2023-10-07T11:04:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.077+0000 7f706abff700 1 -- start start 2023-10-07T11:04:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157a20 0x7f70640b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706abff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70640cb4d0 0x7f70640b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706abff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70640aeae0 con 0x7f7064157650 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706abff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70640ca430 con 0x7f70640cb4d0 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70640ca710 con 0x7f7064157a20 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706919c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70640cb4d0 0x7f70640b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706919c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70640cb4d0 0x7f70640b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58562/0 (socket says 172.21.15.29:58562) 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f706919c700 1 -- 172.21.15.29:0/585270443 learned_addr learned my addr 172.21.15.29:0/585270443 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.078+0000 7f7063fff700 1 --2- 172.21.15.29:0/585270443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157a20 0x7f70640b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f706919c700 1 -- 172.21.15.29:0/585270443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157a20 msgr2=0x7f70640b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f706919c700 1 --2- 172.21.15.29:0/585270443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157a20 0x7f70640b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f706919c700 1 -- 172.21.15.29:0/585270443 >> v1:172.21.15.144:6789/0 conn(0x7f7064157650 legacy=0x7f70640ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f706919c700 1 -- 172.21.15.29:0/585270443 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70640ca9f0 con 0x7f70640cb4d0 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f706919c700 1 --2- 172.21.15.29:0/585270443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f70640b17d0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f705c009f20 tx=0x7f705c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4f5b13208bbc37 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f70637fe700 1 -- 172.21.15.29:0/585270443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c00e070 con 0x7f70640cb4d0 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f70637fe700 1 -- 172.21.15.29:0/585270443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c00ba80 con 0x7f70640cb4d0 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.079+0000 7f70637fe700 1 -- 172.21.15.29:0/585270443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0092e0 con 0x7f70640cb4d0 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 -- 172.21.15.29:0/585270443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 msgr2=0x7f70640b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 --2- 172.21.15.29:0/585270443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f70640b17d0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f705c009f20 tx=0x7f705c009840 comp rx=0 tx=0).stop 2023-10-07T11:04:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 -- 172.21.15.29:0/585270443 shutdown_connections 2023-10-07T11:04:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 --2- 172.21.15.29:0/585270443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f70640b17d0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 --2- 172.21.15.29:0/585270443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157a20 0x7f70640b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 -- 172.21.15.29:0/585270443 >> 172.21.15.29:0/585270443 conn(0x7f7064067610 msgr2=0x7f7064067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 -- 172.21.15.29:0/585270443 shutdown_connections 2023-10-07T11:04:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.080+0000 7f706abff700 1 -- 172.21.15.29:0/585270443 wait complete. 2023-10-07T11:04:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.081+0000 7f706abff700 1 Processor -- start 2023-10-07T11:04:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.082+0000 7f706abff700 1 -- start start 2023-10-07T11:04:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f7064169400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157650 0x7f706416d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7064157a20 0x7f7064153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706416d900 con 0x7f70640cb4d0 2023-10-07T11:04:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640af1b0 con 0x7f7064157a20 2023-10-07T11:04:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7064169a50 con 0x7f7064157650 2023-10-07T11:04:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706899b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f7064169400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706899b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f7064169400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58578/0 (socket says 172.21.15.29:58578) 2023-10-07T11:04:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706899b700 1 -- 172.21.15.29:0/3980998599 learned_addr learned my addr 172.21.15.29:0/3980998599 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f7063fff700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157650 0x7f706416d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.083+0000 7f706919c700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7064157a20 0x7f7064153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.084+0000 7f706899b700 1 -- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157650 msgr2=0x7f706416d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.084+0000 7f706899b700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7064157650 0x7f706416d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.084+0000 7f706899b700 1 -- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7064157a20 msgr2=0x7f7064153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.084+0000 7f706899b700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7064157a20 0x7f7064153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.084+0000 7f706899b700 1 -- 172.21.15.29:0/3980998599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c009ae0 con 0x7f70640cb4d0 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.084+0000 7f706899b700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f7064169400 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f705800c590 tx=0x7f705800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.085+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058011590 con 0x7f70640cb4d0 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.085+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7058011710 con 0x7f70640cb4d0 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.085+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058011ad0 con 0x7f70640cb4d0 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.085+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7064169d30 con 0x7f70640cb4d0 2023-10-07T11:04:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.085+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706416a2f0 con 0x7f70640cb4d0 2023-10-07T11:04:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.086+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70640718b0 con 0x7f70640cb4d0 2023-10-07T11:04:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.087+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f705801b3b0 con 0x7f70640cb4d0 2023-10-07T11:04:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.088+0000 7f7060ff9700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7048066f30 0x7f70480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.088+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f7058083b50 con 0x7f70640cb4d0 2023-10-07T11:04:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.088+0000 7f7060ff9700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f704806d670 0x7f704806fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.088+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 --> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70480700f0 con 0x7f704806d670 2023-10-07T11:04:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.088+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7058088220 con 0x7f70640cb4d0 2023-10-07T11:04:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.089+0000 7f7063fff700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7048066f30 0x7f70480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.089+0000 7f706919c700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f704806d670 0x7f704806fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.089+0000 7f706919c700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f704806d670 0x7f704806fac0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.089+0000 7f7063fff700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7048066f30 0x7f70480693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7064158d50 tx=0x7f7050008040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.090+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== osd.1 v2:172.21.15.29:6808/115184 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f70480700f0 con 0x7f704806d670 2023-10-07T11:04:03.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.105+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 --> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f70640cacd0 con 0x7f704806d670 2023-10-07T11:04:03.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.105+0000 7f7060ff9700 1 -- 172.21.15.29:0/3980998599 <== osd.1 v2:172.21.15.29:6808/115184 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f70640cacd0 con 0x7f704806d670 2023-10-07T11:04:03.106 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:03.106 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.105+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f704806d670 msgr2=0x7f704806fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.105+0000 7f706abff700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:6808/115184,v1:172.21.15.29:6809/115184] conn(0x7f704806d670 0x7f704806fac0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7048066f30 msgr2=0x7f70480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7048066f30 0x7f70480693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7064158d50 tx=0x7f7050008040 comp rx=0 tx=0).stop 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 msgr2=0x7f7064169400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 --2- 172.21.15.29:0/3980998599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70640cb4d0 0x7f7064169400 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f705800c590 tx=0x7f705800c8a0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706919c700 1 -- 172.21.15.29:0/3980998599 reap_dead start 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 shutdown_connections 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 >> 172.21.15.29:0/3980998599 conn(0x7f7064067610 msgr2=0x7f7064155d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 shutdown_connections 2023-10-07T11:04:03.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.106+0000 7f706abff700 1 -- 172.21.15.29:0/3980998599 wait complete. 2023-10-07T11:04:03.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-07T11:04:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.318+0000 7fea1c7ff700 1 Processor -- start 2023-10-07T11:04:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.319+0000 7fea1c7ff700 1 -- start start 2023-10-07T11:04:03.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.320+0000 7fea1c7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea141533f0 0x7fea14156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.320+0000 7fea1c7ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fea141537c0 0x7fea140afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.320+0000 7fea1c7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea14159670 con 0x7fea140af6d0 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.320+0000 7fea1c7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1415a680 con 0x7fea141533f0 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.320+0000 7fea1c7ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea140b2070 con 0x7fea141537c0 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea141533f0 0x7fea14156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea141533f0 0x7fea14156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58586/0 (socket says 172.21.15.29:58586) 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 -- 172.21.15.29:0/702718204 learned_addr learned my addr 172.21.15.29:0/702718204 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea19d9a700 1 --2- 172.21.15.29:0/702718204 >> v2:172.21.15.144:3300/0 conn(0x7fea141537c0 0x7fea140afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 -- 172.21.15.29:0/702718204 >> v2:172.21.15.144:3300/0 conn(0x7fea141537c0 msgr2=0x7fea140afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 --2- 172.21.15.29:0/702718204 >> v2:172.21.15.144:3300/0 conn(0x7fea141537c0 0x7fea140afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 -- 172.21.15.29:0/702718204 >> v1:172.21.15.144:6789/0 conn(0x7fea140af6d0 legacy=0x7fea140adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.321+0000 7fea1a59b700 1 -- 172.21.15.29:0/702718204 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1415a1f0 con 0x7fea141533f0 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1a59b700 1 --2- 172.21.15.29:0/702718204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141533f0 0x7fea14156a90 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fea140aed30 tx=0x7fea1000b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646ead282a9b34d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea19599700 1 -- 172.21.15.29:0/702718204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1000da60 con 0x7fea141533f0 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea19599700 1 -- 172.21.15.29:0/702718204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1000dbc0 con 0x7fea141533f0 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea19599700 1 -- 172.21.15.29:0/702718204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10017410 con 0x7fea141533f0 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1c7ff700 1 -- 172.21.15.29:0/702718204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141533f0 msgr2=0x7fea14156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/702718204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141533f0 0x7fea14156a90 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fea140aed30 tx=0x7fea1000b4e0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1c7ff700 1 -- 172.21.15.29:0/702718204 shutdown_connections 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/702718204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141533f0 0x7fea14156a90 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/702718204 >> v2:172.21.15.144:3300/0 conn(0x7fea141537c0 0x7fea140afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.322+0000 7fea1c7ff700 1 -- 172.21.15.29:0/702718204 >> 172.21.15.29:0/702718204 conn(0x7fea140680a0 msgr2=0x7fea140cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:03.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.323+0000 7fea1c7ff700 1 -- 172.21.15.29:0/702718204 shutdown_connections 2023-10-07T11:04:03.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.323+0000 7fea1c7ff700 1 -- 172.21.15.29:0/702718204 wait complete. 2023-10-07T11:04:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.324+0000 7fea1c7ff700 1 Processor -- start 2023-10-07T11:04:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.324+0000 7fea1c7ff700 1 -- start start 2023-10-07T11:04:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1c7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea140af6d0 0x7fea14157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1c7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea141533f0 0x7fea14158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1c7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 0x7fea1406e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1c7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140cb220 con 0x7fea141537c0 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1c7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140cad90 con 0x7fea140af6d0 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1c7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea141580a0 con 0x7fea141533f0 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 0x7fea1406e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1ad9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 0x7fea1406e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58596/0 (socket says 172.21.15.29:58596) 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.325+0000 7fea1ad9c700 1 -- 172.21.15.29:0/2207007038 learned_addr learned my addr 172.21.15.29:0/2207007038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea19d9a700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea141533f0 0x7fea14158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1a59b700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea140af6d0 0x7fea14157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1ad9c700 1 -- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea141533f0 msgr2=0x7fea14158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1ad9c700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea141533f0 0x7fea14158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1ad9c700 1 -- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea140af6d0 msgr2=0x7fea14157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1ad9c700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea140af6d0 0x7fea14157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1ad9c700 1 -- 172.21.15.29:0/2207007038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1000b780 con 0x7fea141537c0 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1a59b700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea140af6d0 0x7fea14157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1ad9c700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 0x7fea1406e890 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00c910 tx=0x7fea0c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c012630 con 0x7fea141537c0 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c012790 con 0x7fea141537c0 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c012b30 con 0x7fea141537c0 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.326+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea140ca180 con 0x7fea141537c0 2023-10-07T11:04:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.327+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea140ca740 con 0x7fea141537c0 2023-10-07T11:04:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.328+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea140caf20 con 0x7fea141537c0 2023-10-07T11:04:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.328+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fea0c00a3b0 con 0x7fea141537c0 2023-10-07T11:04:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.329+0000 7fea0a7fc700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe9f8066f30 0x7fe9f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.329+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7fea0c084940 con 0x7fea141537c0 2023-10-07T11:04:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.330+0000 7fea1a59b700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe9f8066f30 0x7fe9f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.331+0000 7fea1a59b700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe9f8066f30 0x7fe9f80693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fea1000b520 tx=0x7fea10019000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.334+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea0c051a00 con 0x7fea141537c0 2023-10-07T11:04:03.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.536+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fea1404c3c0 con 0x7fe9f8066f30 2023-10-07T11:04:03.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.537+0000 7fea0a7fc700 1 -- 172.21.15.29:0/2207007038 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea1404c3c0 con 0x7fe9f8066f30 2023-10-07T11:04:03.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe9f8066f30 msgr2=0x7fe9f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe9f8066f30 0x7fe9f80693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fea1000b520 tx=0x7fea10019000 comp rx=0 tx=0).stop 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 msgr2=0x7fea1406e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 0x7fea1406e890 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00c910 tx=0x7fea0c001710 comp rx=0 tx=0).stop 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 shutdown_connections 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea141537c0 0x7fea1406e890 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea140af6d0 0x7fea14157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fe9f8066f30 0x7fe9f80693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 --2- 172.21.15.29:0/2207007038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea141533f0 0x7fea14158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 >> 172.21.15.29:0/2207007038 conn(0x7fea140680a0 msgr2=0x7fea14069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.539+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 shutdown_connections 2023-10-07T11:04:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.540+0000 7fea1c7ff700 1 -- 172.21.15.29:0/2207007038 wait complete. 2023-10-07T11:04:03.542 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 1 to deep-scrub 2023-10-07T11:04:03.557 INFO:tasks.ceph:Scrubbing osd.2 2023-10-07T11:04:03.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.756+0000 7f18561ff700 1 Processor -- start 2023-10-07T11:04:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.756+0000 7f18561ff700 1 -- start start 2023-10-07T11:04:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f18561ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18500b0610 0x7f18500ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f18561ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f18500b09e0 0x7f18500c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f18561ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f18561ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18500690f0 con 0x7f18500b0610 2023-10-07T11:04:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f18561ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18500c4740 con 0x7f18500b09e0 2023-10-07T11:04:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f18561ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18500cf2e0 con 0x7f18500c8a20 2023-10-07T11:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f184f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18500b0610 0x7f18500ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f184f7fe700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f18500b0610 0x7f18500ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58606/0 (socket says 172.21.15.29:58606) 2023-10-07T11:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f184f7fe700 1 -- 172.21.15.29:0/2946231471 learned_addr learned my addr 172.21.15.29:0/2946231471 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f184ffff700 1 --2- 172.21.15.29:0/2946231471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.758+0000 7f184effd700 1 --2- 172.21.15.29:0/2946231471 >> v2:172.21.15.144:3300/0 conn(0x7f18500b09e0 0x7f18500c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184f7fe700 1 -- 172.21.15.29:0/2946231471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 msgr2=0x7f18500cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184f7fe700 1 --2- 172.21.15.29:0/2946231471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184f7fe700 1 -- 172.21.15.29:0/2946231471 >> v2:172.21.15.144:3300/0 conn(0x7f18500b09e0 msgr2=0x7f18500c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184f7fe700 1 --2- 172.21.15.29:0/2946231471 >> v2:172.21.15.144:3300/0 conn(0x7f18500b09e0 0x7f18500c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184f7fe700 1 -- 172.21.15.29:0/2946231471 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18500687c0 con 0x7f18500b0610 2023-10-07T11:04:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184f7fe700 1 --2- 172.21.15.29:0/2946231471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b0610 0x7f18500ced00 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f18500cc2f0 tx=0x7f1840009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b407490f4480d2d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184e7fc700 1 -- 172.21.15.29:0/2946231471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184000b860 con 0x7f18500b0610 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184e7fc700 1 -- 172.21.15.29:0/2946231471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184000b9c0 con 0x7f18500b0610 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.759+0000 7f184e7fc700 1 -- 172.21.15.29:0/2946231471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184000bd60 con 0x7f18500b0610 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 -- 172.21.15.29:0/2946231471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b0610 msgr2=0x7f18500ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 --2- 172.21.15.29:0/2946231471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b0610 0x7f18500ced00 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f18500cc2f0 tx=0x7f1840009250 comp rx=0 tx=0).stop 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 -- 172.21.15.29:0/2946231471 shutdown_connections 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 --2- 172.21.15.29:0/2946231471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 --2- 172.21.15.29:0/2946231471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b0610 0x7f18500ced00 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 --2- 172.21.15.29:0/2946231471 >> v2:172.21.15.144:3300/0 conn(0x7f18500b09e0 0x7f18500c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 -- 172.21.15.29:0/2946231471 >> 172.21.15.29:0/2946231471 conn(0x7f1850063a10 msgr2=0x7f18500ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 -- 172.21.15.29:0/2946231471 shutdown_connections 2023-10-07T11:04:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.760+0000 7f18561ff700 1 -- 172.21.15.29:0/2946231471 wait complete. 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.762+0000 7f18561ff700 1 Processor -- start 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.762+0000 7f18561ff700 1 -- start start 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.763+0000 7f18561ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18500b0610 0x7f185014b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.763+0000 7f18561ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b09e0 0x7f185009b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.763+0000 7f18561ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.763+0000 7f18561ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185009b340 con 0x7f18500b09e0 2023-10-07T11:04:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.763+0000 7f18561ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18500b1c20 con 0x7f18500b0610 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.763+0000 7f18561ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18500b1f00 con 0x7f18500c8a20 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b09e0 0x7f185009b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b09e0 0x7f185009b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58620/0 (socket says 172.21.15.29:58620) 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 -- 172.21.15.29:0/2947356370 learned_addr learned my addr 172.21.15.29:0/2947356370 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184f7fe700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18500b0610 0x7f185014b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184ffff700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 -- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 msgr2=0x7f18500b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18500c8a20 0x7f18500b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 -- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18500b0610 msgr2=0x7f185014b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18500b0610 0x7f185014b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.764+0000 7f184effd700 1 -- 172.21.15.29:0/2947356370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400094f0 con 0x7f18500b09e0 2023-10-07T11:04:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.765+0000 7f184effd700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b09e0 0x7f185009b7d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f184400d890 tx=0x7f1844008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.765+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440125f0 con 0x7f18500b09e0 2023-10-07T11:04:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.765+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1844012750 con 0x7f18500b09e0 2023-10-07T11:04:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.765+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185009acf0 con 0x7f18500b09e0 2023-10-07T11:04:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.765+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844012af0 con 0x7f18500b09e0 2023-10-07T11:04:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.765+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18500981d0 con 0x7f18500b09e0 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.766+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18500b2120 con 0x7f18500b09e0 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.767+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f184401e460 con 0x7f18500b09e0 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f18377fe700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f183c066f30 0x7f183c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f1844085660 con 0x7f18500b09e0 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f18377fe700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f183c06d670 0x7f183c06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 --> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f183c0700f0 con 0x7f183c06d670 2023-10-07T11:04:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1844085950 con 0x7f18500b09e0 2023-10-07T11:04:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f184f7fe700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f183c066f30 0x7f183c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.768+0000 7f184ffff700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f183c06d670 0x7f183c06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.769+0000 7f184ffff700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f183c06d670 0x7f183c06fac0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.769+0000 7f184f7fe700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f183c066f30 0x7f183c0693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18500cc2f0 tx=0x7f1840017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.770+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== osd.2 v2:172.21.15.29:6816/115200 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f183c0700f0 con 0x7f183c06d670 2023-10-07T11:04:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.784+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 --> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f18500cbe70 con 0x7f183c06d670 2023-10-07T11:04:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.784+0000 7f18377fe700 1 -- 172.21.15.29:0/2947356370 <== osd.2 v2:172.21.15.29:6816/115200 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f18500cbe70 con 0x7f183c06d670 2023-10-07T11:04:03.790 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:03.790 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:03.790 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f183c06d670 msgr2=0x7f183c06fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:6816/115200,v1:172.21.15.29:6817/115200] conn(0x7f183c06d670 0x7f183c06fac0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f183c066f30 msgr2=0x7f183c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f183c066f30 0x7f183c0693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18500cc2f0 tx=0x7f1840017000 comp rx=0 tx=0).stop 2023-10-07T11:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b09e0 msgr2=0x7f185009b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 --2- 172.21.15.29:0/2947356370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18500b09e0 0x7f185009b7d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f184400d890 tx=0x7f1844008040 comp rx=0 tx=0).stop 2023-10-07T11:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f184ffff700 1 -- 172.21.15.29:0/2947356370 reap_dead start 2023-10-07T11:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 shutdown_connections 2023-10-07T11:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 >> 172.21.15.29:0/2947356370 conn(0x7f1850063a10 msgr2=0x7f18500c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 shutdown_connections 2023-10-07T11:04:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.785+0000 7f18561ff700 1 -- 172.21.15.29:0/2947356370 wait complete. 2023-10-07T11:04:03.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-07T11:04:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.999+0000 7f1ec37ff700 1 Processor -- start 2023-10-07T11:04:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:03.999+0000 7f1ec37ff700 1 -- start start 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.000+0000 7f1ec37ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b3e60 0x7f1ebc158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.000+0000 7f1ec37ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ebc0ce390 0x7f1ebc0ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.000+0000 7f1ec37ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc1578a0 con 0x7f1ebc0b00b0 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.000+0000 7f1ec37ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc0cc830 con 0x7f1ebc0ce390 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.000+0000 7f1ec37ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc157ce0 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ebc0ce390 0x7f1ebc0ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ebc0ce390 0x7f1ebc0ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58632/0 (socket says 172.21.15.29:58632) 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/2627505723 learned_addr learned my addr 172.21.15.29:0/2627505723 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec159b700 1 --2- 172.21.15.29:0/2627505723 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b3e60 0x7f1ebc158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/2627505723 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b3e60 msgr2=0x7f1ebc158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 --2- 172.21.15.29:0/2627505723 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b3e60 0x7f1ebc158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/2627505723 >> v1:172.21.15.29:6789/0 conn(0x7f1ebc0b00b0 legacy=0x7f1ebc0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/2627505723 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0cee50 con 0x7f1ebc0ce390 2023-10-07T11:04:04.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.001+0000 7f1ec0d9a700 1 --2- 172.21.15.29:0/2627505723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc0ce830 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc06b330 tx=0x7f1eac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b943330cf0b4a4b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1eb3fff700 1 -- 172.21.15.29:0/2627505723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac009e70 con 0x7f1ebc0ce390 2023-10-07T11:04:04.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1eb3fff700 1 -- 172.21.15.29:0/2627505723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00b860 con 0x7f1ebc0ce390 2023-10-07T11:04:04.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1eb3fff700 1 -- 172.21.15.29:0/2627505723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00bc00 con 0x7f1ebc0ce390 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 -- 172.21.15.29:0/2627505723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0ce390 msgr2=0x7f1ebc0ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/2627505723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc0ce830 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc06b330 tx=0x7f1eac009250 comp rx=0 tx=0).stop 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 -- 172.21.15.29:0/2627505723 shutdown_connections 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/2627505723 >> v2:172.21.15.144:3300/0 conn(0x7f1ebc0b3e60 0x7f1ebc158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/2627505723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc0ce830 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 -- 172.21.15.29:0/2627505723 >> 172.21.15.29:0/2627505723 conn(0x7f1ebc067610 msgr2=0x7f1ebc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 -- 172.21.15.29:0/2627505723 shutdown_connections 2023-10-07T11:04:04.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.002+0000 7f1ec37ff700 1 -- 172.21.15.29:0/2627505723 wait complete. 2023-10-07T11:04:04.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.004+0000 7f1ec37ff700 1 Processor -- start 2023-10-07T11:04:04.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.005+0000 7f1ec37ff700 1 -- start start 2023-10-07T11:04:04.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec37ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0b00b0 0x7f1ebc153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec37ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 0x7f1ebc1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec37ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec37ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c4040 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec37ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c4320 con 0x7f1ebc0ce390 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec37ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c4600 con 0x7f1ebc0b00b0 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec0d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 0x7f1ebc1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec0d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 0x7f1ebc1556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58638/0 (socket says 172.21.15.29:58638) 2023-10-07T11:04:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/278942593 learned_addr learned my addr 172.21.15.29:0/278942593 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec159b700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0b00b0 0x7f1ebc153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.006+0000 7f1ec1d9c700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0b00b0 msgr2=0x7f1ebc153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec0d9a700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0b00b0 0x7f1ebc153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0ce390 msgr2=0x7f1ebc155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec0d9a700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec0d9a700 1 -- 172.21.15.29:0/278942593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0094f0 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec0d9a700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 0x7f1ebc1556f0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc06e2e0 tx=0x7f1eac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00bbe0 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac027450 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac027810 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.007+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0c48e0 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.008+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc154330 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.008+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc154520 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.009+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1eac01a430 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.010+0000 7f1eb17fa700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1ea0066f30 0x7f1ea00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.010+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f1eac086550 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.011+0000 7f1ec159b700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1ea0066f30 0x7f1ea00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.011+0000 7f1ec159b700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1ea0066f30 0x7f1ea00693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009da0 tx=0x7f1eb8009120 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.014+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eac053690 con 0x7f1ebc0b3e60 2023-10-07T11:04:04.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.216+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ebc0c4490 con 0x7f1ea0066f30 2023-10-07T11:04:04.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.217+0000 7f1eb17fa700 1 -- 172.21.15.29:0/278942593 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1ebc0c4490 con 0x7f1ea0066f30 2023-10-07T11:04:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1ea0066f30 msgr2=0x7f1ea00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1ea0066f30 0x7f1ea00693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009da0 tx=0x7f1eb8009120 comp rx=0 tx=0).stop 2023-10-07T11:04:04.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 msgr2=0x7f1ebc1556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 0x7f1ebc1556f0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc06e2e0 tx=0x7f1eac017000 comp rx=0 tx=0).stop 2023-10-07T11:04:04.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 shutdown_connections 2023-10-07T11:04:04.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ebc0ce390 0x7f1ebc155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f1ea0066f30 0x7f1ea00693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ebc0b00b0 0x7f1ebc153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 --2- 172.21.15.29:0/278942593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ebc0b3e60 0x7f1ebc1556f0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.219+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 >> 172.21.15.29:0/278942593 conn(0x7f1ebc067610 msgr2=0x7f1ebc0b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.220+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 shutdown_connections 2023-10-07T11:04:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.220+0000 7f1ec37ff700 1 -- 172.21.15.29:0/278942593 wait complete. 2023-10-07T11:04:04.279 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 2 to deep-scrub 2023-10-07T11:04:04.279 INFO:tasks.ceph:Scrubbing osd.3 2023-10-07T11:04:04.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.479+0000 7f9964dff700 1 Processor -- start 2023-10-07T11:04:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.479+0000 7f9964dff700 1 -- start start 2023-10-07T11:04:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.480+0000 7f9964dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99600ced00 0x7f99600b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.480+0000 7f9964dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 0x7f99600cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.480+0000 7f9964dff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600690f0 con 0x7f99600b0610 2023-10-07T11:04:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.480+0000 7f9964dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99600cf2e0 con 0x7f99600ced00 2023-10-07T11:04:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.480+0000 7f9964dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99600687c0 con 0x7f99600c8a20 2023-10-07T11:04:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.481+0000 7f995e59c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f99600b0610 0x7f99600c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50280/0 (socket says 172.21.15.29:50280) 2023-10-07T11:04:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.481+0000 7f995e59c700 1 -- 172.21.15.29:0/237034014 learned_addr learned my addr 172.21.15.29:0/237034014 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.481+0000 7f995ed9d700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 0x7f99600cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.481+0000 7f995dd9b700 1 --2- 172.21.15.29:0/237034014 >> v2:172.21.15.144:3300/0 conn(0x7f99600ced00 0x7f99600b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.481+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114507433 0 0) 0x7f99600690f0 con 0x7f99600b0610 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.481+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9948003570 con 0x7f99600b0610 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2198496539 0 0) 0x7f9948003570 con 0x7f99600b0610 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 msgr2=0x7f99600cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 0x7f99600cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 >> v2:172.21.15.144:3300/0 conn(0x7f99600ced00 msgr2=0x7f99600b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 --2- 172.21.15.29:0/237034014 >> v2:172.21.15.144:3300/0 conn(0x7f99600ced00 0x7f99600b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9960068aa0 con 0x7f99600b0610 2023-10-07T11:04:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f9950001470 con 0x7f99600b0610 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 >> v1:172.21.15.29:6789/0 conn(0x7f99600b0610 legacy=0x7f99600c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948003a60 0x7f9948003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 0x7f9948004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9948004d50 0x7f99480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600cf2e0 con 0x7f9948003a60 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600687c0 con 0x7f9948004d50 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994800ba60 con 0x7f99600c8a20 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995e59c700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948003a60 0x7f9948003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995dd9b700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 0x7f9948004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995ed9d700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9948004d50 0x7f99480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995e59c700 1 -- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 msgr2=0x7f9948004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995e59c700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99600c8a20 0x7f9948004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995e59c700 1 -- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9948004d50 msgr2=0x7f99480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995e59c700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9948004d50 0x7f99480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.482+0000 7f995e59c700 1 -- 172.21.15.29:0/237034014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948003570 con 0x7f9948003a60 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f995e59c700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948003a60 0x7f9948003e70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f995000c3c0 tx=0x7f9950001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd72bbe8b955fffe server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995000cc30 con 0x7f9948003a60 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995000cdd0 con 0x7f9948003a60 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f995d59a700 1 -- 172.21.15.29:0/237034014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950005b70 con 0x7f9948003a60 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f9964dff700 1 -- 172.21.15.29:0/237034014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948003a60 msgr2=0x7f9948003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f9964dff700 1 --2- 172.21.15.29:0/237034014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948003a60 0x7f9948003e70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f995000c3c0 tx=0x7f9950001e90 comp rx=0 tx=0).stop 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.483+0000 7f995ed9d700 1 -- 172.21.15.29:0/237034014 reap_dead start 2023-10-07T11:04:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.484+0000 7f9964dff700 1 -- 172.21.15.29:0/237034014 shutdown_connections 2023-10-07T11:04:04.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.484+0000 7f9964dff700 1 -- 172.21.15.29:0/237034014 >> 172.21.15.29:0/237034014 conn(0x7f9960063a10 msgr2=0x7f99600ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:04.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.484+0000 7f9964dff700 1 -- 172.21.15.29:0/237034014 shutdown_connections 2023-10-07T11:04:04.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.484+0000 7f9964dff700 1 -- 172.21.15.29:0/237034014 wait complete. 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.485+0000 7f9964dff700 1 Processor -- start 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.485+0000 7f9964dff700 1 -- start start 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f9964dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9948003a60 0x7f9960145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f9964dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960169a20 0x7f99600b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f9964dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600b2080 0x7f996006bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f9964dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960155cd0 con 0x7f99600b2080 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f9964dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960156850 con 0x7f9960169a20 2023-10-07T11:04:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f9964dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9960155f20 con 0x7f9948003a60 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f995ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600b2080 0x7f996006bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f995ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600b2080 0x7f996006bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58660/0 (socket says 172.21.15.29:58660) 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.486+0000 7f995ed9d700 1 -- 172.21.15.29:0/3691962739 learned_addr learned my addr 172.21.15.29:0/3691962739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995e59c700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9948003a60 0x7f9960145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995dd9b700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960169a20 0x7f99600b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995ed9d700 1 -- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9948003a60 msgr2=0x7f9960145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995ed9d700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9948003a60 0x7f9960145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995ed9d700 1 -- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960169a20 msgr2=0x7f99600b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995ed9d700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9960169a20 0x7f99600b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995ed9d700 1 -- 172.21.15.29:0/3691962739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9960156200 con 0x7f99600b2080 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f995ed9d700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600b2080 0x7f996006bfb0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f99580094d0 tx=0x7f995800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800e040 con 0x7f99600b2080 2023-10-07T11:04:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995800f040 con 0x7f99600b2080 2023-10-07T11:04:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800fe50 con 0x7f99600b2080 2023-10-07T11:04:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99601564e0 con 0x7f99600b2080 2023-10-07T11:04:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.487+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99600b3850 con 0x7f99600b2080 2023-10-07T11:04:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.488+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99600718b0 con 0x7f99600b2080 2023-10-07T11:04:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.489+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9958015640 con 0x7f99600b2080 2023-10-07T11:04:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f994e7fc700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9940066f30 0x7f99400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f9958084ef0 con 0x7f99600b2080 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f994e7fc700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7f994006d670 0x7f994006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 --> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99400700f0 con 0x7f994006d670 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9958087450 con 0x7f99600b2080 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f995e59c700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9940066f30 0x7f99400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.491+0000 7f995dd9b700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7f994006d670 0x7f994006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.492+0000 7f995e59c700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9940066f30 0x7f99400693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f995000c3c0 tx=0x7f9950015000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.492+0000 7f995dd9b700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7f994006d670 0x7f994006fac0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.492+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== osd.3 v2:172.21.15.29:6820/115201 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f99400700f0 con 0x7f994006d670 2023-10-07T11:04:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.507+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 --> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9960068d80 con 0x7f994006d670 2023-10-07T11:04:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.507+0000 7f994e7fc700 1 -- 172.21.15.29:0/3691962739 <== osd.3 v2:172.21.15.29:6820/115201 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9960068d80 con 0x7f994006d670 2023-10-07T11:04:04.508 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:04.508 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:04.508 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:04.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7f994006d670 msgr2=0x7f994006fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:6820/115201,v1:172.21.15.29:6822/115201] conn(0x7f994006d670 0x7f994006fac0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9940066f30 msgr2=0x7f99400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9940066f30 0x7f99400693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f995000c3c0 tx=0x7f9950015000 comp rx=0 tx=0).stop 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600b2080 msgr2=0x7f996006bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 --2- 172.21.15.29:0/3691962739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600b2080 0x7f996006bfb0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f99580094d0 tx=0x7f995800b040 comp rx=0 tx=0).stop 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f995ed9d700 1 -- 172.21.15.29:0/3691962739 reap_dead start 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 shutdown_connections 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 >> 172.21.15.29:0/3691962739 conn(0x7f9960063a10 msgr2=0x7f99600cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 shutdown_connections 2023-10-07T11:04:04.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.508+0000 7f9964dff700 1 -- 172.21.15.29:0/3691962739 wait complete. 2023-10-07T11:04:04.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-07T11:04:04.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.720+0000 7f558e5ff700 1 Processor -- start 2023-10-07T11:04:04.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.721+0000 7f558e5ff700 1 -- start start 2023-10-07T11:04:04.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.722+0000 7f558e5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5588158670 0x7f5588162060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.722+0000 7f558e5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5588158ad0 con 0x7f55880ae8b0 2023-10-07T11:04:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.722+0000 7f558e5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558815b140 con 0x7f5588157290 2023-10-07T11:04:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.722+0000 7f558e5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558815b420 con 0x7f5588158670 2023-10-07T11:04:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.722+0000 7f55877fe700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f55880ae8b0 0x7f55880b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50294/0 (socket says 172.21.15.29:50294) 2023-10-07T11:04:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.722+0000 7f55877fe700 1 -- 172.21.15.29:0/3545687334 learned_addr learned my addr 172.21.15.29:0/3545687334 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:04:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f558cb9c700 1 --2- 172.21.15.29:0/3545687334 >> v2:172.21.15.144:3300/0 conn(0x7f5588158670 0x7f5588162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450464887 0 0) 0x7f5588158ad0 con 0x7f55880ae8b0 2023-10-07T11:04:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5570003570 con 0x7f55880ae8b0 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691624451 0 0) 0x7f558815b140 con 0x7f5588157290 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5588158ad0 con 0x7f5588157290 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3595874445 0 0) 0x7f5570003570 con 0x7f55880ae8b0 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 >> v2:172.21.15.144:3300/0 conn(0x7f5588158670 msgr2=0x7f5588162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 --2- 172.21.15.29:0/3545687334 >> v2:172.21.15.144:3300/0 conn(0x7f5588158670 0x7f5588162060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 >> v1:172.21.15.144:6789/0 conn(0x7f5588157290 legacy=0x7f55880cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558815b700 con 0x7f55880ae8b0 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2330218747 0 0) 0x7f557c001470 con 0x7f55880ae8b0 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f558cb9c700 1 --2- 172.21.15.29:0/3545687334 >> v2:172.21.15.144:3300/0 conn(0x7f5588158670 0x7f5588162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 >> v1:172.21.15.29:6789/0 conn(0x7f55880ae8b0 legacy=0x7f55880b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5570003bf0 0x7f5570003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5570004810 0x7f5570004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55700051c0 0x7f557000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558815b420 con 0x7f5570004810 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557000c160 con 0x7f5570003bf0 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.723+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557000c320 con 0x7f55700051c0 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5570004810 0x7f5570004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f5587fff700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5570003bf0 0x7f5570003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f558cb9c700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55700051c0 0x7f557000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 -- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55700051c0 msgr2=0x7f557000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55700051c0 0x7f557000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 -- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5570003bf0 msgr2=0x7f5570003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5570003bf0 0x7f5570003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 -- 172.21.15.29:0/3545687334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5570003570 con 0x7f5570004810 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f558cb9c700 1 -- 172.21.15.29:0/3545687334 reap_dead start 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.724+0000 7f55877fe700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5570004810 0x7f5570004c80 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c2a0 tx=0x7f557c001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7e452697e408c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c00cb90 con 0x7f5570004810 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c00ccf0 con 0x7f5570004810 2023-10-07T11:04:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f5586ffd700 1 -- 172.21.15.29:0/3545687334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c005b70 con 0x7f5570004810 2023-10-07T11:04:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 -- 172.21.15.29:0/3545687334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5570004810 msgr2=0x7f5570004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5570004810 0x7f5570004c80 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c2a0 tx=0x7f557c001e10 comp rx=0 tx=0).stop 2023-10-07T11:04:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 -- 172.21.15.29:0/3545687334 shutdown_connections 2023-10-07T11:04:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 --2- 172.21.15.29:0/3545687334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5570004810 0x7f5570004c80 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 -- 172.21.15.29:0/3545687334 >> 172.21.15.29:0/3545687334 conn(0x7f55880680a0 msgr2=0x7f55880cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 -- 172.21.15.29:0/3545687334 shutdown_connections 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.725+0000 7f558e5ff700 1 -- 172.21.15.29:0/3545687334 wait complete. 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.727+0000 7f558e5ff700 1 Processor -- start 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.727+0000 7f558e5ff700 1 -- start start 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558e5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5570004810 0x7f558813d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558e5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880b0010 0x7f55880b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558e5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 0x7f558806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558e5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558806ee30 con 0x7f558806e470 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558e5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558814c110 con 0x7f55880b0010 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558e5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880af390 con 0x7f5570004810 2023-10-07T11:04:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558cb9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 0x7f558806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558cb9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 0x7f558806e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58674/0 (socket says 172.21.15.29:58674) 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558cb9c700 1 -- 172.21.15.29:0/2406644416 learned_addr learned my addr 172.21.15.29:0/2406644416 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f558cb9c700 1 -- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5570004810 msgr2=0x7f558813d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.728+0000 7f5587fff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5570004810 0x7f558813d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f55877fe700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880b0010 0x7f55880b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558cb9c700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5570004810 0x7f558813d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558cb9c700 1 -- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880b0010 msgr2=0x7f55880b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558cb9c700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880b0010 0x7f55880b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558cb9c700 1 -- 172.21.15.29:0/2406644416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880b0b60 con 0x7f558806e470 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f5587fff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5570004810 0x7f558813d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f55877fe700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880b0010 0x7f55880b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558cb9c700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 0x7f558806e8f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f55880c8d60 tx=0x7f558000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558000e040 con 0x7f558806e470 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558000f040 con 0x7f558806e470 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580015430 con 0x7f558806e470 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558814b7e0 con 0x7f558806e470 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.729+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558814bda0 con 0x7f558806e470 2023-10-07T11:04:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.730+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558814c280 con 0x7f558806e470 2023-10-07T11:04:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.731+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f55800157a0 con 0x7f558806e470 2023-10-07T11:04:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.732+0000 7f556ffff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f5574066f30 0x7f55740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.732+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f5580084fb0 con 0x7f558806e470 2023-10-07T11:04:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.732+0000 7f5587fff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f5574066f30 0x7f55740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.733+0000 7f5587fff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f5574066f30 0x7f55740693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5578000c00 tx=0x7f5578009040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.736+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5580052070 con 0x7f558806e470 2023-10-07T11:04:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.937+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f558804c3c0 con 0x7f5574066f30 2023-10-07T11:04:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.938+0000 7f556ffff700 1 -- 172.21.15.29:0/2406644416 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f558804c3c0 con 0x7f5574066f30 2023-10-07T11:04:04.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.940+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f5574066f30 msgr2=0x7f55740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.940+0000 7f558e5ff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f5574066f30 0x7f55740693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5578000c00 tx=0x7f5578009040 comp rx=0 tx=0).stop 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.940+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 msgr2=0x7f558806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.940+0000 7f558e5ff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 0x7f558806e8f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f55880c8d60 tx=0x7f558000b040 comp rx=0 tx=0).stop 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.940+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 shutdown_connections 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f558806e470 0x7f558806e8f0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f5574066f30 0x7f55740693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5570004810 0x7f558813d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 --2- 172.21.15.29:0/2406644416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880b0010 0x7f55880b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 >> 172.21.15.29:0/2406644416 conn(0x7f55880680a0 msgr2=0x7f5588158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 shutdown_connections 2023-10-07T11:04:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:04.941+0000 7f558e5ff700 1 -- 172.21.15.29:0/2406644416 wait complete. 2023-10-07T11:04:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 3 to deep-scrub 2023-10-07T11:04:04.959 INFO:tasks.ceph:Scrubbing osd.4 2023-10-07T11:04:04.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:05.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.158+0000 7f89f6fff700 1 Processor -- start 2023-10-07T11:04:05.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.158+0000 7f89f6fff700 1 -- start start 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89f6fff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f89f00b0610 0x7f89f00ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89f6fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f89f00b09e0 0x7f89f00c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89f6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00c8a20 0x7f89f00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89f6fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00690f0 con 0x7f89f00b09e0 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89f6fff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00c4740 con 0x7f89f00b0610 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89f6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00cf2e0 con 0x7f89f00c8a20 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.159+0000 7f89effff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f89f00b09e0 0x7f89f00c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f89f00b09e0 0x7f89f00c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58686/0 (socket says 172.21.15.29:58686) 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 -- 172.21.15.29:0/1591048401 learned_addr learned my addr 172.21.15.29:0/1591048401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89f559c700 1 --2- 172.21.15.29:0/1591048401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00c8a20 0x7f89f00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89f4d9b700 1 --2- 172.21.15.29:0/1591048401 >> v2:172.21.15.144:3300/0 conn(0x7f89f00b0610 0x7f89f00ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 -- 172.21.15.29:0/1591048401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00c8a20 msgr2=0x7f89f00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 --2- 172.21.15.29:0/1591048401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00c8a20 0x7f89f00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 -- 172.21.15.29:0/1591048401 >> v2:172.21.15.144:3300/0 conn(0x7f89f00b0610 msgr2=0x7f89f00ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 --2- 172.21.15.29:0/1591048401 >> v2:172.21.15.144:3300/0 conn(0x7f89f00b0610 0x7f89f00ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 -- 172.21.15.29:0/1591048401 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00687c0 con 0x7f89f00b09e0 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89f4d9b700 1 --2- 172.21.15.29:0/1591048401 >> v2:172.21.15.144:3300/0 conn(0x7f89f00b0610 0x7f89f00ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89effff700 1 --2- 172.21.15.29:0/1591048401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00c4200 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f89f00659c0 tx=0x7f89dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c6ed558af5b208 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.160+0000 7f89ef7fe700 1 -- 172.21.15.29:0/1591048401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc00b860 con 0x7f89f00b09e0 2023-10-07T11:04:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89ef7fe700 1 -- 172.21.15.29:0/1591048401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc00b9c0 con 0x7f89f00b09e0 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89ef7fe700 1 -- 172.21.15.29:0/1591048401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc00bd60 con 0x7f89f00b09e0 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 -- 172.21.15.29:0/1591048401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 msgr2=0x7f89f00c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 --2- 172.21.15.29:0/1591048401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00c4200 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f89f00659c0 tx=0x7f89dc009250 comp rx=0 tx=0).stop 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 -- 172.21.15.29:0/1591048401 shutdown_connections 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 --2- 172.21.15.29:0/1591048401 >> v2:172.21.15.144:3300/0 conn(0x7f89f00b0610 0x7f89f00ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 --2- 172.21.15.29:0/1591048401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00c8a20 0x7f89f00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 --2- 172.21.15.29:0/1591048401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00c4200 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 -- 172.21.15.29:0/1591048401 >> 172.21.15.29:0/1591048401 conn(0x7f89f0063a10 msgr2=0x7f89f00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 -- 172.21.15.29:0/1591048401 shutdown_connections 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.161+0000 7f89f6fff700 1 -- 172.21.15.29:0/1591048401 wait complete. 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.163+0000 7f89f6fff700 1 Processor -- start 2023-10-07T11:04:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.163+0000 7f89f6fff700 1 -- start start 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f6fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00b0610 0x7f89f014d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f6fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f6fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00c8a20 0x7f89f0157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f6fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00c5940 con 0x7f89f00b09e0 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f6fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00b37f0 con 0x7f89f00c8a20 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f6fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00b3ad0 con 0x7f89f00b0610 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89effff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89effff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58696/0 (socket says 172.21.15.29:58696) 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89effff700 1 -- 172.21.15.29:0/3003645565 learned_addr learned my addr 172.21.15.29:0/3003645565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f4d9b700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00b0610 0x7f89f014d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.164+0000 7f89f559c700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00c8a20 0x7f89f0157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89effff700 1 -- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00b0610 msgr2=0x7f89f014d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89effff700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89f00b0610 0x7f89f014d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89effff700 1 -- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00c8a20 msgr2=0x7f89f0157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89effff700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00c8a20 0x7f89f0157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89effff700 1 -- 172.21.15.29:0/3003645565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc0094f0 con 0x7f89f00b09e0 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89f559c700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89f00c8a20 0x7f89f0157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89effff700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00b2df0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f89f006ea20 tx=0x7f89dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc011e70 con 0x7f89f00b09e0 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc024440 con 0x7f89f00b09e0 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f00b3db0 con 0x7f89f00b09e0 2023-10-07T11:04:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc024800 con 0x7f89f00b09e0 2023-10-07T11:04:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.165+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f014d900 con 0x7f89f00b09e0 2023-10-07T11:04:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.166+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f89f00718b0 con 0x7f89f00b09e0 2023-10-07T11:04:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.167+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f89dc01a420 con 0x7f89f00b09e0 2023-10-07T11:04:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.168+0000 7f89ecff9700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89d4066f30 0x7f89d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.168+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f89dc086640 con 0x7f89f00b09e0 2023-10-07T11:04:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.168+0000 7f89ecff9700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f89d406d670 0x7f89d406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.168+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 --> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89d40700f0 con 0x7f89d406d670 2023-10-07T11:04:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.168+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89dc088ba0 con 0x7f89f00b09e0 2023-10-07T11:04:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.169+0000 7f89f4d9b700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89d4066f30 0x7f89d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.169+0000 7f89f559c700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f89d406d670 0x7f89d406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.169+0000 7f89f4d9b700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89d4066f30 0x7f89d40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89e4006fd0 tx=0x7f89e4008040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.169+0000 7f89f559c700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f89d406d670 0x7f89d406fac0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.170+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== osd.4 v2:172.21.15.144:6800/115417 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f89d40700f0 con 0x7f89d406d670 2023-10-07T11:04:05.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.185+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 --> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f89f0068aa0 con 0x7f89d406d670 2023-10-07T11:04:05.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89ecff9700 1 -- 172.21.15.29:0/3003645565 <== osd.4 v2:172.21.15.144:6800/115417 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f89f0068aa0 con 0x7f89d406d670 2023-10-07T11:04:05.187 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:05.187 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:05.187 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f89d406d670 msgr2=0x7f89d406fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6800/115417,v1:172.21.15.144:6801/115417] conn(0x7f89d406d670 0x7f89d406fac0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89d4066f30 msgr2=0x7f89d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f89d4066f30 0x7f89d40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89e4006fd0 tx=0x7f89e4008040 comp rx=0 tx=0).stop 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 msgr2=0x7f89f00b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 --2- 172.21.15.29:0/3003645565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89f00b09e0 0x7f89f00b2df0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f89f006ea20 tx=0x7f89dc017000 comp rx=0 tx=0).stop 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f559c700 1 -- 172.21.15.29:0/3003645565 reap_dead start 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 shutdown_connections 2023-10-07T11:04:05.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.186+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 >> 172.21.15.29:0/3003645565 conn(0x7f89f0063a10 msgr2=0x7f89f0159230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:05.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.187+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 shutdown_connections 2023-10-07T11:04:05.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.187+0000 7f89f6fff700 1 -- 172.21.15.29:0/3003645565 wait complete. 2023-10-07T11:04:05.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-07T11:04:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.400+0000 7feb84fff700 1 Processor -- start 2023-10-07T11:04:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.400+0000 7feb84fff700 1 -- start start 2023-10-07T11:04:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.401+0000 7feb84fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb80157290 0x7feb800cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.401+0000 7feb84fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.401+0000 7feb84fff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb80158ad0 con 0x7feb800ae8b0 2023-10-07T11:04:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.401+0000 7feb84fff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8015b140 con 0x7feb80157290 2023-10-07T11:04:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.401+0000 7feb84fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8015b420 con 0x7feb800b1a10 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.401+0000 7feb7e59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb80157290 0x7feb800cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb80157290 0x7feb800cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58698/0 (socket says 172.21.15.29:58698) 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 -- 172.21.15.29:0/4075121954 learned_addr learned my addr 172.21.15.29:0/4075121954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7ed9d700 1 --2- 172.21.15.29:0/4075121954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 -- 172.21.15.29:0/4075121954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 msgr2=0x7feb80157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 --2- 172.21.15.29:0/4075121954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 -- 172.21.15.29:0/4075121954 >> v1:172.21.15.29:6789/0 conn(0x7feb800ae8b0 legacy=0x7feb800b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 -- 172.21.15.29:0/4075121954 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8015b700 con 0x7feb80157290 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7e59c700 1 --2- 172.21.15.29:0/4075121954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb800cc870 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7feb700099b0 tx=0x7feb70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a844ec93eb55e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7d59a700 1 -- 172.21.15.29:0/4075121954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7000b860 con 0x7feb80157290 2023-10-07T11:04:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7d59a700 1 -- 172.21.15.29:0/4075121954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7000b9c0 con 0x7feb80157290 2023-10-07T11:04:05.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.402+0000 7feb7d59a700 1 -- 172.21.15.29:0/4075121954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7000bd60 con 0x7feb80157290 2023-10-07T11:04:05.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 -- 172.21.15.29:0/4075121954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 msgr2=0x7feb800cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 --2- 172.21.15.29:0/4075121954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb800cc870 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7feb700099b0 tx=0x7feb70009250 comp rx=0 tx=0).stop 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 -- 172.21.15.29:0/4075121954 shutdown_connections 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 --2- 172.21.15.29:0/4075121954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 --2- 172.21.15.29:0/4075121954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb800cc870 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 -- 172.21.15.29:0/4075121954 >> 172.21.15.29:0/4075121954 conn(0x7feb800680a0 msgr2=0x7feb800cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 -- 172.21.15.29:0/4075121954 shutdown_connections 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.403+0000 7feb84fff700 1 -- 172.21.15.29:0/4075121954 wait complete. 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.405+0000 7feb84fff700 1 Processor -- start 2023-10-07T11:04:05.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.405+0000 7feb84fff700 1 -- start start 2023-10-07T11:04:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb84fff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb84fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb80152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb84fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb801556c0 0x7feb8016a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb84fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8016aa90 con 0x7feb80157290 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb84fff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb80154620 con 0x7feb801556c0 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb84fff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb80154900 con 0x7feb800b1a10 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb80152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb80152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58700/0 (socket says 172.21.15.29:58700) 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 -- 172.21.15.29:0/2177123402 learned_addr learned my addr 172.21.15.29:0/2177123402 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7ed9d700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb801556c0 0x7feb8016a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7e59c700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 -- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 msgr2=0x7feb80154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 -- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb801556c0 msgr2=0x7feb8016a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb801556c0 0x7feb8016a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7dd9b700 1 -- 172.21.15.29:0/2177123402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700094f0 con 0x7feb80157290 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7ed9d700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb801556c0 0x7feb8016a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.406+0000 7feb7e59c700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.407+0000 7feb7dd9b700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb80152060 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7feb7400c590 tx=0x7feb7400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.407+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74011590 con 0x7feb80157290 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.407+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb740116f0 con 0x7feb80157290 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.407+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb80154be0 con 0x7feb80157290 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.407+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74011ab0 con 0x7feb80157290 2023-10-07T11:04:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.407+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb801551a0 con 0x7feb80157290 2023-10-07T11:04:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.408+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb801553c0 con 0x7feb80157290 2023-10-07T11:04:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.409+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feb7401b3b0 con 0x7feb80157290 2023-10-07T11:04:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.410+0000 7feb6e7fc700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7feb60066f30 0x7feb600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.410+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7feb74083bf0 con 0x7feb80157290 2023-10-07T11:04:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.411+0000 7feb7e59c700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7feb60066f30 0x7feb600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.411+0000 7feb7e59c700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7feb60066f30 0x7feb600693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb700099b0 tx=0x7feb70017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.414+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb74050d50 con 0x7feb80157290 2023-10-07T11:04:05.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.613+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7feb8004c3c0 con 0x7feb60066f30 2023-10-07T11:04:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.613+0000 7feb6e7fc700 1 -- 172.21.15.29:0/2177123402 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feb8004c3c0 con 0x7feb60066f30 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.615+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7feb60066f30 msgr2=0x7feb600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.615+0000 7feb84fff700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7feb60066f30 0x7feb600693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb700099b0 tx=0x7feb70017000 comp rx=0 tx=0).stop 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.615+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 msgr2=0x7feb80152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.615+0000 7feb84fff700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb80152060 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7feb7400c590 tx=0x7feb7400c950 comp rx=0 tx=0).stop 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 shutdown_connections 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb801556c0 0x7feb8016a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7feb60066f30 0x7feb600693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb800b1a10 0x7feb80154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 --2- 172.21.15.29:0/2177123402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb80157290 0x7feb80152060 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 >> 172.21.15.29:0/2177123402 conn(0x7feb800680a0 msgr2=0x7feb800ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 shutdown_connections 2023-10-07T11:04:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.616+0000 7feb84fff700 1 -- 172.21.15.29:0/2177123402 wait complete. 2023-10-07T11:04:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 4 to deep-scrub 2023-10-07T11:04:05.634 INFO:tasks.ceph:Scrubbing osd.5 2023-10-07T11:04:05.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.834+0000 7f86fc9ff700 1 Processor -- start 2023-10-07T11:04:05.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.834+0000 7f86fc9ff700 1 -- start start 2023-10-07T11:04:05.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86fc9ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f86f80b2a20 0x7f86f80cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86fc9ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86f80cf2c0 0x7f86f80c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86fc9ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f80cc940 con 0x7f86f80cb6f0 2023-10-07T11:04:05.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86fc9ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f80b2df0 con 0x7f86f80cf2c0 2023-10-07T11:04:05.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86fc9ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f80cbd30 con 0x7f86f80b2a20 2023-10-07T11:04:05.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86f6d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86f80cf2c0 0x7f86f80c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86f6d9d700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86f80cf2c0 0x7f86f80c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58702/0 (socket says 172.21.15.29:58702) 2023-10-07T11:04:05.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.835+0000 7f86f6d9d700 1 -- 172.21.15.29:0/1033626192 learned_addr learned my addr 172.21.15.29:0/1033626192 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:05.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f659c700 1 --2- 172.21.15.29:0/1033626192 >> v2:172.21.15.144:3300/0 conn(0x7f86f80b2a20 0x7f86f80cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f6d9d700 1 -- 172.21.15.29:0/1033626192 >> v2:172.21.15.144:3300/0 conn(0x7f86f80b2a20 msgr2=0x7f86f80cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f6d9d700 1 --2- 172.21.15.29:0/1033626192 >> v2:172.21.15.144:3300/0 conn(0x7f86f80b2a20 0x7f86f80cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f6d9d700 1 -- 172.21.15.29:0/1033626192 >> v1:172.21.15.29:6789/0 conn(0x7f86f80cb6f0 legacy=0x7f86f80cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f6d9d700 1 -- 172.21.15.29:0/1033626192 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f80cc010 con 0x7f86f80cf2c0 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f6d9d700 1 --2- 172.21.15.29:0/1033626192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cf2c0 0x7f86f80c5400 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f86f808ac80 tx=0x7f86f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52dab5005dd8ef3b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f559a700 1 -- 172.21.15.29:0/1033626192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f000e070 con 0x7f86f80cf2c0 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.836+0000 7f86f559a700 1 -- 172.21.15.29:0/1033626192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f000ba20 con 0x7f86f80cf2c0 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86f559a700 1 -- 172.21.15.29:0/1033626192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00092e0 con 0x7f86f80cf2c0 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 -- 172.21.15.29:0/1033626192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cf2c0 msgr2=0x7f86f80c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 --2- 172.21.15.29:0/1033626192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cf2c0 0x7f86f80c5400 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f86f808ac80 tx=0x7f86f0009840 comp rx=0 tx=0).stop 2023-10-07T11:04:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 -- 172.21.15.29:0/1033626192 shutdown_connections 2023-10-07T11:04:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 --2- 172.21.15.29:0/1033626192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cf2c0 0x7f86f80c5400 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 --2- 172.21.15.29:0/1033626192 >> v2:172.21.15.144:3300/0 conn(0x7f86f80b2a20 0x7f86f80cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 -- 172.21.15.29:0/1033626192 >> 172.21.15.29:0/1033626192 conn(0x7f86f8099820 msgr2=0x7f86f8063a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 -- 172.21.15.29:0/1033626192 shutdown_connections 2023-10-07T11:04:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.837+0000 7f86fc9ff700 1 -- 172.21.15.29:0/1033626192 wait complete. 2023-10-07T11:04:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.839+0000 7f86fc9ff700 1 Processor -- start 2023-10-07T11:04:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.839+0000 7f86fc9ff700 1 -- start start 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86fc9ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86f80b2a20 0x7f86f815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86fc9ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cb6f0 0x7f86f80c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86fc9ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86f80cf2c0 0x7f86f80c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86fc9ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f815ec10 con 0x7f86f80cb6f0 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86fc9ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80b39e0 con 0x7f86f80cf2c0 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86fc9ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f815e650 con 0x7f86f80b2a20 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cb6f0 0x7f86f80c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cb6f0 0x7f86f80c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58706/0 (socket says 172.21.15.29:58706) 2023-10-07T11:04:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 -- 172.21.15.29:0/648010384 learned_addr learned my addr 172.21.15.29:0/648010384 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f659c700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86f80b2a20 0x7f86f815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f6d9d700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86f80cf2c0 0x7f86f80c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 -- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86f80b2a20 msgr2=0x7f86f815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86f80b2a20 0x7f86f815dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 -- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86f80cf2c0 msgr2=0x7f86f80c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86f80cf2c0 0x7f86f80c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f5d9b700 1 -- 172.21.15.29:0/648010384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0009ae0 con 0x7f86f80cb6f0 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.840+0000 7f86f659c700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86f80b2a20 0x7f86f815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.841+0000 7f86f5d9b700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cb6f0 0x7f86f80c9050 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00c590 tx=0x7f86ec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.841+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec011590 con 0x7f86f80cb6f0 2023-10-07T11:04:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.841+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec0116f0 con 0x7f86f80cb6f0 2023-10-07T11:04:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.841+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec011ab0 con 0x7f86f80cb6f0 2023-10-07T11:04:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.841+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f815e930 con 0x7f86f80cb6f0 2023-10-07T11:04:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.841+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f80ca600 con 0x7f86f80cb6f0 2023-10-07T11:04:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.842+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86f80ca7f0 con 0x7f86f80cb6f0 2023-10-07T11:04:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.843+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f86ec01b3b0 con 0x7f86f80cb6f0 2023-10-07T11:04:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.844+0000 7f86e67fc700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f86d8066f30 0x7f86d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.844+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f86ec083b90 con 0x7f86f80cb6f0 2023-10-07T11:04:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.845+0000 7f86e67fc700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f86d806d670 0x7f86d806fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.845+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 --> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86d80700f0 con 0x7f86d806d670 2023-10-07T11:04:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.845+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86ec088220 con 0x7f86f80cb6f0 2023-10-07T11:04:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.845+0000 7f86f659c700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f86d8066f30 0x7f86d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.845+0000 7f86f6d9d700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f86d806d670 0x7f86d806fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.845+0000 7f86f659c700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f86d8066f30 0x7f86d80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009d70 tx=0x7f86e8008040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.846+0000 7f86f6d9d700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f86d806d670 0x7f86d806fac0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.846+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== osd.5 v2:172.21.15.144:6806/115418 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f86d80700f0 con 0x7f86d806d670 2023-10-07T11:04:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.861+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 --> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f86f80718b0 con 0x7f86d806d670 2023-10-07T11:04:05.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.861+0000 7f86e67fc700 1 -- 172.21.15.29:0/648010384 <== osd.5 v2:172.21.15.144:6806/115418 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f86f80718b0 con 0x7f86d806d670 2023-10-07T11:04:05.863 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:05.863 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:05.863 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f86d806d670 msgr2=0x7f86d806fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6806/115418,v1:172.21.15.144:6808/115418] conn(0x7f86d806d670 0x7f86d806fac0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f86d8066f30 msgr2=0x7f86d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f86d8066f30 0x7f86d80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009d70 tx=0x7f86e8008040 comp rx=0 tx=0).stop 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cb6f0 msgr2=0x7f86f80c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 --2- 172.21.15.29:0/648010384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f80cb6f0 0x7f86f80c9050 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00c590 tx=0x7f86ec00c950 comp rx=0 tx=0).stop 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86f6d9d700 1 -- 172.21.15.29:0/648010384 reap_dead start 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 shutdown_connections 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 >> 172.21.15.29:0/648010384 conn(0x7f86f8099820 msgr2=0x7f86f80d0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.862+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 shutdown_connections 2023-10-07T11:04:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:05.863+0000 7f86fc9ff700 1 -- 172.21.15.29:0/648010384 wait complete. 2023-10-07T11:04:05.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-07T11:04:06.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.074+0000 7f27d0dff700 1 Processor -- start 2023-10-07T11:04:06.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.074+0000 7f27d0dff700 1 -- start start 2023-10-07T11:04:06.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.075+0000 7f27d0dff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27cc157290 0x7f27cc0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.075+0000 7f27d0dff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f27cc0ae8b0 0x7f27cc0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.075+0000 7f27d0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc0b1840 0x7f27cc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.075+0000 7f27d0dff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc0b1cc0 con 0x7f27cc157290 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.075+0000 7f27d0dff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc158ad0 con 0x7f27cc0ae8b0 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.075+0000 7f27d0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc15b140 con 0x7f27cc0b1840 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27cc157290 0x7f27cc0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27cc157290 0x7f27cc0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58712/0 (socket says 172.21.15.29:58712) 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 -- 172.21.15.29:0/3942486416 learned_addr learned my addr 172.21.15.29:0/3942486416 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27c9d9b700 1 --2- 172.21.15.29:0/3942486416 >> v2:172.21.15.144:3300/0 conn(0x7f27cc0ae8b0 0x7f27cc0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27cad9d700 1 --2- 172.21.15.29:0/3942486416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc0b1840 0x7f27cc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 -- 172.21.15.29:0/3942486416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc0b1840 msgr2=0x7f27cc0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 --2- 172.21.15.29:0/3942486416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc0b1840 0x7f27cc0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 -- 172.21.15.29:0/3942486416 >> v2:172.21.15.144:3300/0 conn(0x7f27cc0ae8b0 msgr2=0x7f27cc0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 --2- 172.21.15.29:0/3942486416 >> v2:172.21.15.144:3300/0 conn(0x7f27cc0ae8b0 0x7f27cc0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 -- 172.21.15.29:0/3942486416 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc15b420 con 0x7f27cc157290 2023-10-07T11:04:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27c9d9b700 1 --2- 172.21.15.29:0/3942486416 >> v2:172.21.15.144:3300/0 conn(0x7f27cc0ae8b0 0x7f27cc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.076+0000 7f27ca59c700 1 --2- 172.21.15.29:0/3942486416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc157290 0x7f27cc0cc870 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0c8d90 tx=0x7f27bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4bb4ab57596c7b server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27c959a700 1 -- 172.21.15.29:0/3942486416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00b860 con 0x7f27cc157290 2023-10-07T11:04:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27c959a700 1 -- 172.21.15.29:0/3942486416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc00b9c0 con 0x7f27cc157290 2023-10-07T11:04:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27c959a700 1 -- 172.21.15.29:0/3942486416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00bd60 con 0x7f27cc157290 2023-10-07T11:04:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 -- 172.21.15.29:0/3942486416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc157290 msgr2=0x7f27cc0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 --2- 172.21.15.29:0/3942486416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc157290 0x7f27cc0cc870 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0c8d90 tx=0x7f27bc009250 comp rx=0 tx=0).stop 2023-10-07T11:04:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 -- 172.21.15.29:0/3942486416 shutdown_connections 2023-10-07T11:04:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 --2- 172.21.15.29:0/3942486416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc0b1840 0x7f27cc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 --2- 172.21.15.29:0/3942486416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc157290 0x7f27cc0cc870 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 --2- 172.21.15.29:0/3942486416 >> v2:172.21.15.144:3300/0 conn(0x7f27cc0ae8b0 0x7f27cc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 -- 172.21.15.29:0/3942486416 >> 172.21.15.29:0/3942486416 conn(0x7f27cc0680a0 msgr2=0x7f27cc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.077+0000 7f27d0dff700 1 -- 172.21.15.29:0/3942486416 shutdown_connections 2023-10-07T11:04:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.078+0000 7f27d0dff700 1 -- 172.21.15.29:0/3942486416 wait complete. 2023-10-07T11:04:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.079+0000 7f27d0dff700 1 Processor -- start 2023-10-07T11:04:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.079+0000 7f27d0dff700 1 -- start start 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27d0dff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27cc0b1840 0x7f27cc14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27d0dff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc157290 0x7f27cc156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27d0dff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 0x7f27cc0c2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27d0dff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc150290 con 0x7f27cc153de0 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27d0dff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc154380 con 0x7f27cc0b1840 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27d0dff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc154660 con 0x7f27cc157290 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.080+0000 7f27cad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 0x7f27cc0c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 0x7f27cc0c2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58718/0 (socket says 172.21.15.29:58718) 2023-10-07T11:04:06.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 -- 172.21.15.29:0/214772328 learned_addr learned my addr 172.21.15.29:0/214772328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27c9d9b700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc157290 0x7f27cc156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27ca59c700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27cc0b1840 0x7f27cc14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 -- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc157290 msgr2=0x7f27cc156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc157290 0x7f27cc156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 -- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27cc0b1840 msgr2=0x7f27cc14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27cc0b1840 0x7f27cc14fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 -- 172.21.15.29:0/214772328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0094f0 con 0x7f27cc153de0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.081+0000 7f27cad9d700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 0x7f27cc0c2790 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009fa0 tx=0x7f27c4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.082+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400ebc0 con 0x7f27cc153de0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.082+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c400ed20 con 0x7f27cc153de0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.082+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4014670 con 0x7f27cc153de0 2023-10-07T11:04:06.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.082+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc154940 con 0x7f27cc153de0 2023-10-07T11:04:06.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.082+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc1673a0 con 0x7f27cc153de0 2023-10-07T11:04:06.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.083+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc167610 con 0x7f27cc153de0 2023-10-07T11:04:06.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.084+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f27c40149e0 con 0x7f27cc153de0 2023-10-07T11:04:06.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.085+0000 7f27ba7fc700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27ac066f30 0x7f27ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.085+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f27c4084ce0 con 0x7f27cc153de0 2023-10-07T11:04:06.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.085+0000 7f27ca59c700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27ac066f30 0x7f27ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.086+0000 7f27ca59c700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27ac066f30 0x7f27ac0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0c8d90 tx=0x7f27bc017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.089+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27c4051e40 con 0x7f27cc153de0 2023-10-07T11:04:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.294+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f27cc04c3c0 con 0x7f27ac066f30 2023-10-07T11:04:06.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.294+0000 7f27ba7fc700 1 -- 172.21.15.29:0/214772328 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f27cc04c3c0 con 0x7f27ac066f30 2023-10-07T11:04:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27ac066f30 msgr2=0x7f27ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27ac066f30 0x7f27ac0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0c8d90 tx=0x7f27bc017000 comp rx=0 tx=0).stop 2023-10-07T11:04:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 msgr2=0x7f27cc0c2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 0x7f27cc0c2790 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009fa0 tx=0x7f27c4009900 comp rx=0 tx=0).stop 2023-10-07T11:04:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 shutdown_connections 2023-10-07T11:04:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27cc153de0 0x7f27cc0c2790 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27cc0b1840 0x7f27cc14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f27ac066f30 0x7f27ac0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 --2- 172.21.15.29:0/214772328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27cc157290 0x7f27cc156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 >> 172.21.15.29:0/214772328 conn(0x7f27cc0680a0 msgr2=0x7f27cc159500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 shutdown_connections 2023-10-07T11:04:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.297+0000 7f27d0dff700 1 -- 172.21.15.29:0/214772328 wait complete. 2023-10-07T11:04:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 5 to deep-scrub 2023-10-07T11:04:06.315 INFO:tasks.ceph:Scrubbing osd.6 2023-10-07T11:04:06.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:06.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.514+0000 7fa5af5ff700 1 Processor -- start 2023-10-07T11:04:06.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.515+0000 7fa5af5ff700 1 -- start start 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5af5ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5a8157a20 0x7fa5a80b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5af5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5a80cb4d0 0x7fa5a80b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5af5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a80aeae0 con 0x7fa5a8157650 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5af5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a80ca430 con 0x7fa5a80cb4d0 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5af5ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a80ca710 con 0x7fa5a8157a20 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5adb9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5a80cb4d0 0x7fa5a80b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5adb9c700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5a80cb4d0 0x7fa5a80b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58720/0 (socket says 172.21.15.29:58720) 2023-10-07T11:04:06.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.516+0000 7fa5adb9c700 1 -- 172.21.15.29:0/3763029438 learned_addr learned my addr 172.21.15.29:0/3763029438 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5acb9a700 1 --2- 172.21.15.29:0/3763029438 >> v2:172.21.15.144:3300/0 conn(0x7fa5a8157a20 0x7fa5a80b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5adb9c700 1 -- 172.21.15.29:0/3763029438 >> v2:172.21.15.144:3300/0 conn(0x7fa5a8157a20 msgr2=0x7fa5a80b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5adb9c700 1 --2- 172.21.15.29:0/3763029438 >> v2:172.21.15.144:3300/0 conn(0x7fa5a8157a20 0x7fa5a80b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5adb9c700 1 -- 172.21.15.29:0/3763029438 >> v1:172.21.15.29:6789/0 conn(0x7fa5a8157650 legacy=0x7fa5a80ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5adb9c700 1 -- 172.21.15.29:0/3763029438 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80ca9f0 con 0x7fa5a80cb4d0 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5acb9a700 1 --2- 172.21.15.29:0/3763029438 >> v2:172.21.15.144:3300/0 conn(0x7fa5a8157a20 0x7fa5a80b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa5adb9c700 1 --2- 172.21.15.29:0/3763029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a80cb4d0 0x7fa5a80b17d0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80c9e00 tx=0x7fa5a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5c04d90717d6ea server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa59ffff700 1 -- 172.21.15.29:0/3763029438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a000e070 con 0x7fa5a80cb4d0 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa59ffff700 1 -- 172.21.15.29:0/3763029438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a000ba50 con 0x7fa5a80cb4d0 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.517+0000 7fa59ffff700 1 -- 172.21.15.29:0/3763029438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00092e0 con 0x7fa5a80cb4d0 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 -- 172.21.15.29:0/3763029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a80cb4d0 msgr2=0x7fa5a80b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 --2- 172.21.15.29:0/3763029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a80cb4d0 0x7fa5a80b17d0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80c9e00 tx=0x7fa5a0009840 comp rx=0 tx=0).stop 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 -- 172.21.15.29:0/3763029438 shutdown_connections 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 --2- 172.21.15.29:0/3763029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a80cb4d0 0x7fa5a80b17d0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 --2- 172.21.15.29:0/3763029438 >> v2:172.21.15.144:3300/0 conn(0x7fa5a8157a20 0x7fa5a80b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 -- 172.21.15.29:0/3763029438 >> 172.21.15.29:0/3763029438 conn(0x7fa5a8067610 msgr2=0x7fa5a8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 -- 172.21.15.29:0/3763029438 shutdown_connections 2023-10-07T11:04:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.518+0000 7fa5af5ff700 1 -- 172.21.15.29:0/3763029438 wait complete. 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.519+0000 7fa5af5ff700 1 Processor -- start 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.520+0000 7fa5af5ff700 1 -- start start 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5af5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5a80cb4d0 0x7fa5a81694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5af5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a8157650 0x7fa5a815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5af5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5a8157a20 0x7fa5a80af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5af5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a816cc50 con 0x7fa5a8157650 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5af5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a815c8e0 con 0x7fa5a8157a20 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5af5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a8169a90 con 0x7fa5a80cb4d0 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a8157650 0x7fa5a815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a8157650 0x7fa5a815c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58724/0 (socket says 172.21.15.29:58724) 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 -- 172.21.15.29:0/1422855765 learned_addr learned my addr 172.21.15.29:0/1422855765 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5ad39b700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5a80cb4d0 0x7fa5a81694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5adb9c700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5a8157a20 0x7fa5a80af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 -- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5a80cb4d0 msgr2=0x7fa5a81694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5a80cb4d0 0x7fa5a81694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 -- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5a8157a20 msgr2=0x7fa5a80af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5a8157a20 0x7fa5a80af510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5acb9a700 1 -- 172.21.15.29:0/1422855765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a0009ae0 con 0x7fa5a8157650 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5ad39b700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5a80cb4d0 0x7fa5a81694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.521+0000 7fa5adb9c700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5a8157a20 0x7fa5a80af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.522+0000 7fa5acb9a700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a8157650 0x7fa5a815c3a0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa59800c830 tx=0x7fa598008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.522+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598011600 con 0x7fa5a8157650 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.522+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa598011760 con 0x7fa5a8157650 2023-10-07T11:04:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.522+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598011ae0 con 0x7fa5a8157650 2023-10-07T11:04:06.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.522+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a8169d70 con 0x7fa5a8157650 2023-10-07T11:04:06.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.522+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a816a330 con 0x7fa5a8157650 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.523+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5a80718b0 con 0x7fa5a8157650 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.524+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa5980223a0 con 0x7fa5a8157650 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa59d7fa700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa58c066f30 0x7fa58c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7fa598084960 con 0x7fa5a8157650 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa59d7fa700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7fa58c06d670 0x7fa58c06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 --> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa58c0700f0 con 0x7fa58c06d670 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa598089220 con 0x7fa5a8157650 2023-10-07T11:04:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa5ad39b700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa58c066f30 0x7fa58c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.525+0000 7fa5adb9c700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7fa58c06d670 0x7fa58c06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.526+0000 7fa5ad39b700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa58c066f30 0x7fa58c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4001840 tx=0x7fa5a4009040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.526+0000 7fa5adb9c700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7fa58c06d670 0x7fa58c06fac0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.527+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== osd.6 v2:172.21.15.144:6824/115448 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa58c0700f0 con 0x7fa58c06d670 2023-10-07T11:04:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.542+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 --> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa5a80cacd0 con 0x7fa58c06d670 2023-10-07T11:04:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.542+0000 7fa59d7fa700 1 -- 172.21.15.29:0/1422855765 <== osd.6 v2:172.21.15.144:6824/115448 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa5a80cacd0 con 0x7fa58c06d670 2023-10-07T11:04:06.543 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:06.543 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:06.544 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7fa58c06d670 msgr2=0x7fa58c06fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6824/115448,v1:172.21.15.144:6825/115448] conn(0x7fa58c06d670 0x7fa58c06fac0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa58c066f30 msgr2=0x7fa58c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fa58c066f30 0x7fa58c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4001840 tx=0x7fa5a4009040 comp rx=0 tx=0).stop 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a8157650 msgr2=0x7fa5a815c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 --2- 172.21.15.29:0/1422855765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a8157650 0x7fa5a815c3a0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa59800c830 tx=0x7fa598008040 comp rx=0 tx=0).stop 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5adb9c700 1 -- 172.21.15.29:0/1422855765 reap_dead start 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 shutdown_connections 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 >> 172.21.15.29:0/1422855765 conn(0x7fa5a8067610 msgr2=0x7fa5a80b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 shutdown_connections 2023-10-07T11:04:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.543+0000 7fa5af5ff700 1 -- 172.21.15.29:0/1422855765 wait complete. 2023-10-07T11:04:06.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-07T11:04:06.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.759+0000 7f0fdc7ff700 1 Processor -- start 2023-10-07T11:04:06.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.760+0000 7f0fdc7ff700 1 -- start start 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fdc7ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fd40ad370 0x7f0fd40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fdc7ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd414d790 con 0x7f0fd4159ec0 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fdc7ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40ae5d0 con 0x7f0fd40b0410 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fdc7ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd40ad740 con 0x7f0fd40ad370 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fda59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fd40ad370 0x7f0fd40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fda59b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fd40ad370 0x7f0fd40cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58726/0 (socket says 172.21.15.29:58726) 2023-10-07T11:04:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.761+0000 7f0fda59b700 1 -- 172.21.15.29:0/3723520161 learned_addr learned my addr 172.21.15.29:0/3723520161 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:06.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fda59b700 1 -- 172.21.15.29:0/3723520161 >> v1:172.21.15.144:6789/0 conn(0x7f0fd40b0410 legacy=0x7f0fd4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fda59b700 1 -- 172.21.15.29:0/3723520161 >> v1:172.21.15.29:6789/0 conn(0x7f0fd4159ec0 legacy=0x7f0fd41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fda59b700 1 -- 172.21.15.29:0/3723520161 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40ccdb0 con 0x7f0fd40ad370 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fda59b700 1 --2- 172.21.15.29:0/3723520161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ad370 0x7f0fd40cc870 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099e0 tx=0x7f0fd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304f7d9fa443a375 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fd9599700 1 -- 172.21.15.29:0/3723520161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000b820 con 0x7f0fd40ad370 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fd9599700 1 -- 172.21.15.29:0/3723520161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd000b980 con 0x7f0fd40ad370 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.762+0000 7f0fd9599700 1 -- 172.21.15.29:0/3723520161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000bd20 con 0x7f0fd40ad370 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3723520161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ad370 msgr2=0x7f0fd40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3723520161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ad370 0x7f0fd40cc870 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099e0 tx=0x7f0fd0009250 comp rx=0 tx=0).stop 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3723520161 shutdown_connections 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3723520161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ad370 0x7f0fd40cc870 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3723520161 >> 172.21.15.29:0/3723520161 conn(0x7f0fd40680a0 msgr2=0x7f0fd40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:06.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3723520161 shutdown_connections 2023-10-07T11:04:06.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.763+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3723520161 wait complete. 2023-10-07T11:04:06.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.764+0000 7f0fdc7ff700 1 Processor -- start 2023-10-07T11:04:06.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.765+0000 7f0fdc7ff700 1 -- start start 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.765+0000 7f0fdc7ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fd40ad370 0x7f0fd413d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.765+0000 7f0fdc7ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 0x7f0fd41588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.765+0000 7f0fdc7ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fd4159ec0 0x7f0fd4158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fdc7ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd414b810 con 0x7f0fd40b0410 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fdc7ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd416db30 con 0x7f0fd4159ec0 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fdc7ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd416aae0 con 0x7f0fd40ad370 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 0x7f0fd41588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 0x7f0fd41588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58740/0 (socket says 172.21.15.29:58740) 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 -- 172.21.15.29:0/3094301358 learned_addr learned my addr 172.21.15.29:0/3094301358 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fda59b700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fd40ad370 0x7f0fd413d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fdad9c700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fd4159ec0 0x7f0fd4158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 -- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fd40ad370 msgr2=0x7f0fd413d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fd40ad370 0x7f0fd413d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 -- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fd4159ec0 msgr2=0x7f0fd4158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fd4159ec0 0x7f0fd4158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 -- 172.21.15.29:0/3094301358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00094f0 con 0x7f0fd40b0410 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fdad9c700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fd4159ec0 0x7f0fd4158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fda59b700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fd40ad370 0x7f0fd413d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.766+0000 7f0fd9d9a700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 0x7f0fd41588e0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c4e0 tx=0x7f0fc400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.767+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4011590 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.767+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc4011710 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.767+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4011ad0 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.767+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd416d4e0 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.767+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd416a7d0 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.768+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd416d340 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.769+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0fc401b3b0 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.770+0000 7f0fca7fc700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0fb8066f30 0x7f0fb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.770+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f0fc4083c10 con 0x7f0fd40b0410 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.770+0000 7f0fda59b700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0fb8066f30 0x7f0fb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.771+0000 7f0fda59b700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0fb8066f30 0x7f0fb80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099e0 tx=0x7f0fd0017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:06.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.774+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fd416d340 con 0x7f0fd40b0410 2023-10-07T11:04:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.975+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fd416d340 con 0x7f0fb8066f30 2023-10-07T11:04:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.975+0000 7f0fca7fc700 1 -- 172.21.15.29:0/3094301358 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0fd416d340 con 0x7f0fb8066f30 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.977+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0fb8066f30 msgr2=0x7f0fb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.977+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0fb8066f30 0x7f0fb80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099e0 tx=0x7f0fd0017000 comp rx=0 tx=0).stop 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.977+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 msgr2=0x7f0fd41588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.977+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 0x7f0fd41588e0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c4e0 tx=0x7f0fc400c8a0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 shutdown_connections 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fd4159ec0 0x7f0fd4158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f0fb8066f30 0x7f0fb80693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fd40ad370 0x7f0fd413d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 --2- 172.21.15.29:0/3094301358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40b0410 0x7f0fd41588e0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 >> 172.21.15.29:0/3094301358 conn(0x7f0fd40680a0 msgr2=0x7f0fd40ac740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 shutdown_connections 2023-10-07T11:04:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:06.978+0000 7f0fdc7ff700 1 -- 172.21.15.29:0/3094301358 wait complete. 2023-10-07T11:04:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 6 to deep-scrub 2023-10-07T11:04:06.996 INFO:tasks.ceph:Scrubbing osd.7 2023-10-07T11:04:06.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-07T11:04:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.196+0000 7f4fb61ff700 1 Processor -- start 2023-10-07T11:04:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.196+0000 7f4fb61ff700 1 -- start start 2023-10-07T11:04:07.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.197+0000 7f4fb61ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00b0610 0x7f4fb00ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.197+0000 7f4fb61ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb00690f0 con 0x7f4fb00c8a20 2023-10-07T11:04:07.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.197+0000 7f4fb61ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb00cf2e0 con 0x7f4fb00b09e0 2023-10-07T11:04:07.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.197+0000 7f4fb61ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00687c0 con 0x7f4fb00b0610 2023-10-07T11:04:07.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4faffff700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4fb00c8a20 0x7f4fb00cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50326/0 (socket says 172.21.15.29:50326) 2023-10-07T11:04:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4faffff700 1 -- 172.21.15.29:0/3724300535 learned_addr learned my addr 172.21.15.29:0/3724300535 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4faf7fe700 1 --2- 172.21.15.29:0/3724300535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00b0610 0x7f4fb00ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891896347 0 0) 0x7f4fb00690f0 con 0x7f4fb00c8a20 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f98003570 con 0x7f4fb00c8a20 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634216773 0 0) 0x7f4fb00cf2e0 con 0x7f4fb00b09e0 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb00690f0 con 0x7f4fb00b09e0 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4faf7fe700 1 -- 172.21.15.29:0/3724300535 >> v1:172.21.15.144:6789/0 conn(0x7f4fb00b09e0 legacy=0x7f4fb00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4faf7fe700 1 -- 172.21.15.29:0/3724300535 >> v1:172.21.15.29:6789/0 conn(0x7f4fb00c8a20 legacy=0x7f4fb00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.198+0000 7f4faf7fe700 1 -- 172.21.15.29:0/3724300535 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb0068aa0 con 0x7f4fb00b0610 2023-10-07T11:04:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.199+0000 7f4faf7fe700 1 --2- 172.21.15.29:0/3724300535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00b0610 0x7f4fb00ced00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009910 tx=0x7f4fa00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d780a96b25bddb server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.199+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0011630 con 0x7f4fb00b0610 2023-10-07T11:04:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.199+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa0011790 con 0x7f4fb00b0610 2023-10-07T11:04:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.199+0000 7f4fae7fc700 1 -- 172.21.15.29:0/3724300535 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0011b30 con 0x7f4fb00b0610 2023-10-07T11:04:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 -- 172.21.15.29:0/3724300535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00b0610 msgr2=0x7f4fb00ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 --2- 172.21.15.29:0/3724300535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00b0610 0x7f4fb00ced00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009910 tx=0x7f4fa00091a0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 -- 172.21.15.29:0/3724300535 shutdown_connections 2023-10-07T11:04:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 --2- 172.21.15.29:0/3724300535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00b0610 0x7f4fb00ced00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 -- 172.21.15.29:0/3724300535 >> 172.21.15.29:0/3724300535 conn(0x7f4fb0063a10 msgr2=0x7f4fb00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 -- 172.21.15.29:0/3724300535 shutdown_connections 2023-10-07T11:04:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.200+0000 7f4fb61ff700 1 -- 172.21.15.29:0/3724300535 wait complete. 2023-10-07T11:04:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.201+0000 7f4fb61ff700 1 Processor -- start 2023-10-07T11:04:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.201+0000 7f4fb61ff700 1 -- start start 2023-10-07T11:04:07.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.202+0000 7f4fb61ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fb00b0610 0x7f4fb009b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.202+0000 7f4fb61ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fb00b09e0 0x7f4fb00b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.202+0000 7f4fb61ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00c8a20 0x7f4fb016d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.202+0000 7f4fb61ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00b1210 con 0x7f4fb00b0610 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.202+0000 7f4fb61ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb009b640 con 0x7f4fb00b09e0 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.202+0000 7f4fb61ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb009b920 con 0x7f4fb00c8a20 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fb00b0610 0x7f4fb009b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fb00b0610 0x7f4fb009b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58742/0 (socket says 172.21.15.29:58742) 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 -- 172.21.15.29:0/929846797 learned_addr learned my addr 172.21.15.29:0/929846797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faffff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00c8a20 0x7f4fb016d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faeffd700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fb00b09e0 0x7f4fb00b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 -- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00c8a20 msgr2=0x7f4fb016d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00c8a20 0x7f4fb016d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 -- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fb00b09e0 msgr2=0x7f4fb00b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fb00b09e0 0x7f4fb00b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 -- 172.21.15.29:0/929846797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa0009420 con 0x7f4fb00b0610 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faffff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4fb00c8a20 0x7f4fb016d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faeffd700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fb00b09e0 0x7f4fb00b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.203+0000 7f4faf7fe700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fb00b0610 0x7f4fb009b040 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009f00 tx=0x7f4fa0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.204+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa001f070 con 0x7f4fb00b0610 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.204+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa0000bc0 con 0x7f4fb00b0610 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.204+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0025420 con 0x7f4fb00b0610 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.204+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb009bc00 con 0x7f4fb00b0610 2023-10-07T11:04:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.204+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb00b3f20 con 0x7f4fb00b0610 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.205+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4fb00cbe50 con 0x7f4fb00b0610 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.206+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4fa0023070 con 0x7f4fb00b0610 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4f977fe700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4f9c066f30 0x7f4f9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f4fa00860f0 con 0x7f4fb00b0610 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4f977fe700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7f4f9c06d670 0x7f4f9c06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 --> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f9c0700f0 con 0x7f4f9c06d670 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=202) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fa00863e0 con 0x7f4fb00b0610 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4faeffd700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4f9c066f30 0x7f4f9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.207+0000 7f4faffff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7f4f9c06d670 0x7f4f9c06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.208+0000 7f4faffff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7f4f9c06d670 0x7f4f9c06fac0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.208+0000 7f4faeffd700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4f9c066f30 0x7f4f9c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cdaf0 tx=0x7f4fa4009040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.209+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== osd.7 v2:172.21.15.144:6816/115419 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4f9c0700f0 con 0x7f4f9c06d670 2023-10-07T11:04:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.223+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 --> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4fb00718b0 con 0x7f4f9c06d670 2023-10-07T11:04:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.224+0000 7f4f977fe700 1 -- 172.21.15.29:0/929846797 <== osd.7 v2:172.21.15.144:6816/115419 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4fb00718b0 con 0x7f4f9c06d670 2023-10-07T11:04:07.225 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-07T11:04:07.225 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-07T11:04:07.225 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-07T11:04:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.224+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7f4f9c06d670 msgr2=0x7f4f9c06fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.224+0000 7f4fb61ff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6816/115419,v1:172.21.15.144:6817/115419] conn(0x7f4f9c06d670 0x7f4f9c06fac0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4f9c066f30 msgr2=0x7f4f9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f4f9c066f30 0x7f4f9c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cdaf0 tx=0x7f4fa4009040 comp rx=0 tx=0).stop 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fb00b0610 msgr2=0x7f4fb009b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 --2- 172.21.15.29:0/929846797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fb00b0610 0x7f4fb009b040 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009f00 tx=0x7f4fa0017000 comp rx=0 tx=0).stop 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4faffff700 1 -- 172.21.15.29:0/929846797 reap_dead start 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 shutdown_connections 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 >> 172.21.15.29:0/929846797 conn(0x7f4fb0063a10 msgr2=0x7f4fb00c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 shutdown_connections 2023-10-07T11:04:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.225+0000 7f4fb61ff700 1 -- 172.21.15.29:0/929846797 wait complete. 2023-10-07T11:04:07.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-07T11:04:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.438+0000 7f7c1abff700 1 Processor -- start 2023-10-07T11:04:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.438+0000 7f7c1abff700 1 -- start start 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c1abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159750 0x7f7c140d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c1abff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c14159b20 0x7f7c140ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c1abff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c140cbf20 con 0x7f7c140ce6d0 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c1abff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c140cc200 con 0x7f7c14159b20 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c1abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c140cc4e0 con 0x7f7c14159750 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c13fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c14159b20 0x7f7c140ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c13fff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c14159b20 0x7f7c140ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58754/0 (socket says 172.21.15.29:58754) 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c13fff700 1 -- 172.21.15.29:0/638643669 learned_addr learned my addr 172.21.15.29:0/638643669 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.440+0000 7f7c1899b700 1 --2- 172.21.15.29:0/638643669 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159750 0x7f7c140d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c13fff700 1 -- 172.21.15.29:0/638643669 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159750 msgr2=0x7f7c140d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c13fff700 1 --2- 172.21.15.29:0/638643669 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159750 0x7f7c140d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c13fff700 1 -- 172.21.15.29:0/638643669 >> v1:172.21.15.29:6789/0 conn(0x7f7c140ce6d0 legacy=0x7f7c1415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c13fff700 1 -- 172.21.15.29:0/638643669 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c140cc7c0 con 0x7f7c14159b20 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c137fe700 1 -- 172.21.15.29:0/638643669 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992845129 0 0) 0x7f7c140cbf20 con 0x7f7c140ce6d0 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c13fff700 1 --2- 172.21.15.29:0/638643669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159b20 0x7f7c140ce190 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f7c000099b0 tx=0x7f7c00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74992518127bf70 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c137fe700 1 -- 172.21.15.29:0/638643669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00009ea0 con 0x7f7c14159b20 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c137fe700 1 -- 172.21.15.29:0/638643669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0000b860 con 0x7f7c14159b20 2023-10-07T11:04:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.441+0000 7f7c137fe700 1 -- 172.21.15.29:0/638643669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000bc00 con 0x7f7c14159b20 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 -- 172.21.15.29:0/638643669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159b20 msgr2=0x7f7c140ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 --2- 172.21.15.29:0/638643669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159b20 0x7f7c140ce190 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f7c000099b0 tx=0x7f7c00009250 comp rx=0 tx=0).stop 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 -- 172.21.15.29:0/638643669 shutdown_connections 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 --2- 172.21.15.29:0/638643669 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159750 0x7f7c140d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 --2- 172.21.15.29:0/638643669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159b20 0x7f7c140ce190 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 -- 172.21.15.29:0/638643669 >> 172.21.15.29:0/638643669 conn(0x7f7c14063a10 msgr2=0x7f7c140d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 -- 172.21.15.29:0/638643669 shutdown_connections 2023-10-07T11:04:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.442+0000 7f7c1abff700 1 -- 172.21.15.29:0/638643669 wait complete. 2023-10-07T11:04:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.443+0000 7f7c1abff700 1 Processor -- start 2023-10-07T11:04:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.444+0000 7f7c1abff700 1 -- start start 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.444+0000 7f7c1abff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c140ce6d0 0x7f7c140c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1abff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 0x7f7c14069810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1abff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159b20 0x7f7c140b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1abff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c140c7bb0 con 0x7f7c14159750 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1abff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1416d5d0 con 0x7f7c140ce6d0 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1abff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1416d8b0 con 0x7f7c14159b20 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 0x7f7c14069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 0x7f7c14069810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58760/0 (socket says 172.21.15.29:58760) 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 -- 172.21.15.29:0/987185275 learned_addr learned my addr 172.21.15.29:0/987185275 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1899b700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c140ce6d0 0x7f7c140c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c1919c700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159b20 0x7f7c140b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 -- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159b20 msgr2=0x7f7c140b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159b20 0x7f7c140b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 -- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c140ce6d0 msgr2=0x7f7c140c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c140ce6d0 0x7f7c140c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.445+0000 7f7c13fff700 1 -- 172.21.15.29:0/987185275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000094f0 con 0x7f7c14159750 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.446+0000 7f7c13fff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 0x7f7c14069810 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f7c14088a40 tx=0x7f7c00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.446+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000bbe0 con 0x7f7c14159750 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.446+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00011420 con 0x7f7c14159750 2023-10-07T11:04:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.446+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1416db90 con 0x7f7c14159750 2023-10-07T11:04:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.446+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c000117c0 con 0x7f7c14159750 2023-10-07T11:04:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.446+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1416d1f0 con 0x7f7c14159750 2023-10-07T11:04:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.447+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1416e030 con 0x7f7c14159750 2023-10-07T11:04:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.448+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c00011bf0 con 0x7f7c14159750 2023-10-07T11:04:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.449+0000 7f7c10ff9700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7bf8066f30 0x7f7bf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.449+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f7c00085fe0 con 0x7f7c14159750 2023-10-07T11:04:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.450+0000 7f7c1899b700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7bf8066f30 0x7f7bf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.450+0000 7f7c1899b700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7bf8066f30 0x7f7bf80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c140c5e70 tx=0x7f7c08009120 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.454+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c000530e0 con 0x7f7c14159750 2023-10-07T11:04:07.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.658+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c1404c3c0 con 0x7f7bf8066f30 2023-10-07T11:04:07.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.658+0000 7f7c10ff9700 1 -- 172.21.15.29:0/987185275 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7c1404c3c0 con 0x7f7bf8066f30 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7bf8066f30 msgr2=0x7f7bf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7bf8066f30 0x7f7bf80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c140c5e70 tx=0x7f7c08009120 comp rx=0 tx=0).stop 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 msgr2=0x7f7c14069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 0x7f7c14069810 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f7c14088a40 tx=0x7f7c00017000 comp rx=0 tx=0).stop 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 shutdown_connections 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f7bf8066f30 0x7f7bf80693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c14159b20 0x7f7c140b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c140ce6d0 0x7f7c140c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 --2- 172.21.15.29:0/987185275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c14159750 0x7f7c14069810 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 >> 172.21.15.29:0/987185275 conn(0x7f7c14063a10 msgr2=0x7f7c141543a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:07.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 shutdown_connections 2023-10-07T11:04:07.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.661+0000 7f7c1abff700 1 -- 172.21.15.29:0/987185275 wait complete. 2023-10-07T11:04:07.663 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 7 to deep-scrub 2023-10-07T11:04:07.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:04:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.873+0000 7f95bc5ff700 1 Processor -- start 2023-10-07T11:04:07.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.874+0000 7f95bc5ff700 1 -- start start 2023-10-07T11:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95bc5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b40d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95bc5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b40cbf20 con 0x7f95b40ce8a0 2023-10-07T11:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95bc5ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b40cc200 con 0x7f95b4159750 2023-10-07T11:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95bc5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b40cc4e0 con 0x7f95b4159b20 2023-10-07T11:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95bab9c700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f95b40ce8a0 0x7f95b40c5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50354/0 (socket says 172.21.15.29:50354) 2023-10-07T11:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95bab9c700 1 -- 172.21.15.29:0/4203052418 learned_addr learned my addr 172.21.15.29:0/4203052418 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:04:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.875+0000 7f95b9b9a700 1 --2- 172.21.15.29:0/4203052418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b40d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248952884 0 0) 0x7f95b40cbf20 con 0x7f95b40ce8a0 2023-10-07T11:04:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a0003570 con 0x7f95b40ce8a0 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507695198 0 0) 0x7f95b40cc200 con 0x7f95b4159750 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b40cbf20 con 0x7f95b4159750 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332186279 0 0) 0x7f95a0003570 con 0x7f95b40ce8a0 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9b9a700 1 -- 172.21.15.29:0/4203052418 >> v1:172.21.15.144:6789/0 conn(0x7f95b4159750 legacy=0x7f95b40ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9b9a700 1 -- 172.21.15.29:0/4203052418 >> v1:172.21.15.29:6789/0 conn(0x7f95b40ce8a0 legacy=0x7f95b40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9b9a700 1 -- 172.21.15.29:0/4203052418 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b40cc7c0 con 0x7f95b4159b20 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.876+0000 7f95b9b9a700 1 --2- 172.21.15.29:0/4203052418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b40d10a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009910 tx=0x7f95a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1c6a077878b532e server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0017f0 con 0x7f95b4159b20 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a4011420 con 0x7f95b4159b20 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95b9399700 1 -- 172.21.15.29:0/4203052418 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40117a0 con 0x7f95b4159b20 2023-10-07T11:04:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95bc5ff700 1 -- 172.21.15.29:0/4203052418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 msgr2=0x7f95b40d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/4203052418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b40d10a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009910 tx=0x7f95a40091a0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95bc5ff700 1 -- 172.21.15.29:0/4203052418 shutdown_connections 2023-10-07T11:04:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/4203052418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b40d10a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.877+0000 7f95bc5ff700 1 -- 172.21.15.29:0/4203052418 >> 172.21.15.29:0/4203052418 conn(0x7f95b4063a10 msgr2=0x7f95b40d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.878+0000 7f95bc5ff700 1 -- 172.21.15.29:0/4203052418 shutdown_connections 2023-10-07T11:04:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.878+0000 7f95bc5ff700 1 -- 172.21.15.29:0/4203052418 wait complete. 2023-10-07T11:04:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.879+0000 7f95bc5ff700 1 Processor -- start 2023-10-07T11:04:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.879+0000 7f95bc5ff700 1 -- start start 2023-10-07T11:04:07.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bc5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 0x7f95b41470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bc5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95b4159750 0x7f95b40c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bc5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b4169b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bc5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b40c7b80 con 0x7f95b40ce8a0 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bc5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b4147610 con 0x7f95b4159750 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bc5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b416a660 con 0x7f95b4159b20 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95ba39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 0x7f95b41470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95ba39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 0x7f95b41470d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58764/0 (socket says 172.21.15.29:58764) 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95ba39b700 1 -- 172.21.15.29:0/941112390 learned_addr learned my addr 172.21.15.29:0/941112390 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95b9b9a700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95b4159750 0x7f95b40c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.880+0000 7f95bab9c700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b4169b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95ba39b700 1 -- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 msgr2=0x7f95b4169b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95ba39b700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b4169b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95ba39b700 1 -- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95b4159750 msgr2=0x7f95b40c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95ba39b700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95b4159750 0x7f95b40c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95ba39b700 1 -- 172.21.15.29:0/941112390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4009420 con 0x7f95b40ce8a0 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95b9b9a700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95b4159750 0x7f95b40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95ba39b700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 0x7f95b41470d0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f95b000c7c0 tx=0x7f95b0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0001530 con 0x7f95b40ce8a0 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b0001690 con 0x7f95b40ce8a0 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b416a940 con 0x7f95b40ce8a0 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0001a50 con 0x7f95b40ce8a0 2023-10-07T11:04:07.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.881+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b40b3540 con 0x7f95b40ce8a0 2023-10-07T11:04:07.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.882+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b40c7e60 con 0x7f95b40ce8a0 2023-10-07T11:04:07.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.883+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f95b001b400 con 0x7f95b40ce8a0 2023-10-07T11:04:07.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.884+0000 7f95aa7fc700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9598066f30 0x7f95980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:07.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.884+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7f95b0084460 con 0x7f95b40ce8a0 2023-10-07T11:04:07.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.885+0000 7f95b9b9a700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9598066f30 0x7f95980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:07.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.886+0000 7f95b9b9a700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9598066f30 0x7f95980693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009860 tx=0x7f95a4016000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:07.888+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95b0051520 con 0x7f95b40ce8a0 2023-10-07T11:04:08.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.086+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95b40c7cc0 con 0x7f9598066f30 2023-10-07T11:04:08.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.089+0000 7f95aa7fc700 1 -- 172.21.15.29:0/941112390 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50597 (secure 0 0 0) 0x7f95a4011590 con 0x7f9598066f30 2023-10-07T11:04:08.091 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:04:08.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9598066f30 msgr2=0x7f95980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:08.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9598066f30 0x7f95980693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009860 tx=0x7f95a4016000 comp rx=0 tx=0).stop 2023-10-07T11:04:08.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 msgr2=0x7f95b41470d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:08.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 0x7f95b41470d0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f95b000c7c0 tx=0x7f95b0007e00 comp rx=0 tx=0).stop 2023-10-07T11:04:08.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 shutdown_connections 2023-10-07T11:04:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7f9598066f30 0x7f95980693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95b4159b20 0x7f95b4169b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95b40ce8a0 0x7f95b41470d0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 --2- 172.21.15.29:0/941112390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95b4159750 0x7f95b40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 >> 172.21.15.29:0/941112390 conn(0x7f95b4063a10 msgr2=0x7f95b40ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.091+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 shutdown_connections 2023-10-07T11:04:08.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:08.092+0000 7f95bc5ff700 1 -- 172.21.15.29:0/941112390 wait complete. 2023-10-07T11:04:08.095 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:04:08.109 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-07T11:04:07.714672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2477064,"kb_used_data":20936,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752497656,"statfs":{"total":773094113280,"available":770557599744,"internally_reserved":0,"allocated":21438464,"data_stored":11222776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.643026"},"pg_stats":[{"pgid":"1.0","version":"200'370","reported_seq":1213,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-10-07T11:03:55.337517+0000","last_change":"2023-10-07T09:29:11.786925+0000","last_active":"2023-10-07T11:03:55.337517+0000","last_peered":"2023-10-07T11:03:55.337517+0000","last_clean":"2023-10-07T11:03:55.337517+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:03:55.337517+0000","last_undegraded":"2023-10-07T11:03:55.337517+0000","last_fullsized":"2023-10-07T11:03:55.337517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-07T09:29:10.517805+0000","last_clean_scrub_stamp":"2023-10-07T09:29:10.517805+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T20:23:33.276134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72299999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.93999999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.995,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.746,"15min":0.751},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":0.97999999999999998,"5min":1.292,"15min":1.292},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.753,"15min":0.77400000000000002},"min":{"1min":0.61199999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76000000000000001,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93600000000000005,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.72199999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.61099999999999999,"5min":0.501,"15min":0.48199999999999998},"max":{"1min":0.84599999999999997,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":0.96699999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.83299999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94299999999999995,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.228,"15min":1.228},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.51900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93000000000000005,"5min":0.96999999999999997,"15min":1.1200000000000001},"last":0.873},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.98199999999999998,"5min":1.494,"15min":1.494},"last":0.78300000000000003}]},{"osd":6,"last update":"Sat Oct 7 11:03:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79300000000000004,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73999999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96999999999999997,"5min":1.605,"15min":1.605},"last":0.71299999999999997}]}]},{"osd":6,"up_from":15,"seq":64424510564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316144,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055696,"statfs":{"total":96636764160,"available":96313032704,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.89300000000000002,"5min":0.95799999999999996,"15min":1.0069999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":1.002},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.45400000000000001},"max":{"1min":0.95699999999999996,"5min":0.98599999999999999,"15min":1.048},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.47999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97499999999999998,"15min":1.012},"last":0.68100000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.76700000000000002},"min":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94099999999999995,"5min":0.97099999999999997,"15min":1.0069999999999999},"last":0.753}]},{"osd":3,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.88400000000000001,"5min":0.996,"15min":1.073},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.58299999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":1.0369999999999999},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.35799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.97799999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.439},"max":{"1min":0.85199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.51600000000000001,"5min":0.45600000000000002,"15min":0.40799999999999997},"max":{"1min":0.85899999999999999,"5min":0.91200000000000003,"15min":0.995},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88100000000000001,"5min":1.022,"15min":1.0580000000000001},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69799999999999995,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":1.0389999999999999,"15min":1.0760000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":1.018,"15min":1.0880000000000001},"last":0.70799999999999996}]}]},{"osd":1,"up_from":14,"seq":60129543266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307952,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063888,"statfs":{"total":96636764160,"available":96321421312,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.436}]},{"osd":2,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.375},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.89900000000000002,"15min":0.96299999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.378,"15min":0.37},"max":{"1min":0.70399999999999996,"5min":0.84999999999999998,"15min":0.88900000000000001},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.51600000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.373},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.59999999999999998,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2430000000000001},"last":0.61599999999999999}]},{"osd":5,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82699999999999996,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2310000000000001,"5min":1.242,"15min":1.242},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.0840000000000001,"5min":1.2,"15min":1.2},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.871,"15min":0.874},"min":{"1min":0.67500000000000004,"5min":0.65000000000000002,"15min":0.58399999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.749},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0409999999999999,"5min":1.157,"15min":1.302},"last":0.76300000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.877,"15min":0.878},"min":{"1min":0.64700000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.66100000000000003}]}]},{"osd":0,"up_from":14,"seq":60129543266,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.433}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.626}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.755},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.67300000000000004}]}]},{"osd":2,"up_from":14,"seq":60129543267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.81699999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.30399999999999999},"max":{"1min":0.69399999999999995,"5min":1.087,"15min":1.087},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.95099999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.80800000000000005,"5min":0.86399999999999999,"15min":0.98199999999999998},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59799999999999998,"5min":0.71799999999999997,"15min":0.80000000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82199999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.50600000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.78900000000000003},"min":{"1min":0.63100000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.163,"15min":1.1879999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.266,"15min":1.5640000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.59199999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.004,"5min":1.2989999999999999,"15min":1.528},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78100000000000003,"15min":0.80300000000000005},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95999999999999996,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.65200000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0580000000000001,"5min":1.242,"15min":1.3779999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.86799999999999999},"min":{"1min":0.67600000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0269999999999999,"5min":1.2050000000000001,"15min":1.629},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0569999999999999,"5min":1.274,"15min":1.5940000000000001},"last":0.65300000000000002}]}]},{"osd":3,"up_from":14,"seq":60129543268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.48099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96199999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.626,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0669999999999999,"15min":1.222},"last":0.72299999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80700000000000005,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.216},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95399999999999996,"5min":1.27,"15min":1.27},"last":0.80900000000000005}]},{"osd":6,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82499999999999996,"15min":0.84199999999999997},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92700000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.65400000000000003,"5min":0.56599999999999995,"15min":0.54600000000000004},"max":{"1min":1.046,"5min":1.389,"15min":1.389},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0429999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.747},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"max":{"1min":0.97599999999999998,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.879}]}]},{"osd":4,"up_from":15,"seq":64424510564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316144,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055696,"statfs":{"total":96636764160,"available":96313032704,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.97099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.72999999999999998,"15min":0.746},"min":{"1min":0.58099999999999996,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.248},"last":0.81899999999999995}]},{"osd":1,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.131,"15min":1.131},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.87}]},{"osd":2,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.022,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73999999999999999,"15min":0.75700000000000001},"min":{"1min":0.56599999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.89100000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.751,"15min":0.751},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.46100000000000002},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.60099999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.98299999999999998,"5min":1.0369999999999999,"15min":1.179},"last":0.85199999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93700000000000006,"5min":1.056,"15min":1.056},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.623},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":0.96399999999999997},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.872,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.46000000000000002},"max":{"1min":0.97399999999999998,"5min":1.097,"15min":1.097},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Oct 7 11:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.71699999999999997},"min":{"1min":0.55500000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":0.94999999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.752},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.95399999999999996,"5min":1.171,"15min":1.171},"last":0.80600000000000005}]}]},{"osd":5,"up_from":14,"seq":60129543266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312048,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059792,"statfs":{"total":96636764160,"available":96317227008,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.78000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93300000000000005,"5min":1.585,"15min":1.585},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.81299999999999994},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.90900000000000003,"5min":1.673,"15min":1.673},"last":0.85099999999999998}]},{"osd":1,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73799999999999999,"15min":0.76400000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":1.139},"last":0.96399999999999997}]},{"osd":2,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92500000000000004,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.748},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.72799999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.95399999999999996,"5min":1.0549999999999999,"15min":1.069},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.91400000000000003,"5min":0.96899999999999997,"15min":1.161},"last":0.88200000000000001}]},{"osd":4,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.82799999999999996}]},{"osd":6,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64300000000000002,"15min":0.625},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.93500000000000005}]},{"osd":7,"last update":"Sat Oct 7 11:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.69299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0109999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.038,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.98999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:04:08.114 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-07T09:29:10.517805+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=7, tm_hour=9, tm_min=29, tm_sec=10, tm_wday=5, tm_yday=280, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=7, tm_hour=11, tm_min=4, tm_sec=1, tm_wday=5, tm_yday=280, tm_isdst=0) 2023-10-07T11:04:08.114 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-07T11:04:14.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:04:14.607 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:04:28.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-07T11:04:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.323+0000 7ff2c2bff700 1 Processor -- start 2023-10-07T11:04:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.323+0000 7ff2c2bff700 1 -- start start 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.324+0000 7ff2c2bff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2bc159550 0x7ff2bc0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.324+0000 7ff2c2bff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff2bc0b0f50 0x7ff2bc0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c2bff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2bc06f4e0 con 0x7ff2bc0b2bb0 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c2bff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc06e8d0 con 0x7ff2bc159550 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c2bff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc06ebb0 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2bc159550 0x7ff2bc0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2bc159550 0x7ff2bc0b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42798/0 (socket says 172.21.15.29:42798) 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 -- 172.21.15.29:0/3390463508 learned_addr learned my addr 172.21.15.29:0/3390463508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2bbfff700 1 --2- 172.21.15.29:0/3390463508 >> v2:172.21.15.144:3300/0 conn(0x7ff2bc0b0f50 0x7ff2bc0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 -- 172.21.15.29:0/3390463508 >> v2:172.21.15.144:3300/0 conn(0x7ff2bc0b0f50 msgr2=0x7ff2bc0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 --2- 172.21.15.29:0/3390463508 >> v2:172.21.15.144:3300/0 conn(0x7ff2bc0b0f50 0x7ff2bc0b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 -- 172.21.15.29:0/3390463508 >> v1:172.21.15.144:6789/0 conn(0x7ff2bc0b2bb0 legacy=0x7ff2bc15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2c099b700 1 -- 172.21.15.29:0/3390463508 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc06ee90 con 0x7ff2bc159550 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.325+0000 7ff2bbfff700 1 --2- 172.21.15.29:0/3390463508 >> v2:172.21.15.144:3300/0 conn(0x7ff2bc0b0f50 0x7ff2bc0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c099b700 1 --2- 172.21.15.29:0/3390463508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc159550 0x7ff2bc0b0a10 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0ce380 tx=0x7ff2b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a15f2dc85c2251 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2bb7fe700 1 -- 172.21.15.29:0/3390463508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b000b860 con 0x7ff2bc159550 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2bb7fe700 1 -- 172.21.15.29:0/3390463508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b000b9c0 con 0x7ff2bc159550 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2bb7fe700 1 -- 172.21.15.29:0/3390463508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b000bd60 con 0x7ff2bc159550 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c2bff700 1 -- 172.21.15.29:0/3390463508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc159550 msgr2=0x7ff2bc0b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/3390463508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc159550 0x7ff2bc0b0a10 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0ce380 tx=0x7ff2b0009250 comp rx=0 tx=0).stop 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c2bff700 1 -- 172.21.15.29:0/3390463508 shutdown_connections 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/3390463508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc159550 0x7ff2bc0b0a10 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/3390463508 >> v2:172.21.15.144:3300/0 conn(0x7ff2bc0b0f50 0x7ff2bc0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.326+0000 7ff2c2bff700 1 -- 172.21.15.29:0/3390463508 >> 172.21.15.29:0/3390463508 conn(0x7ff2bc069190 msgr2=0x7ff2bc06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.327+0000 7ff2c2bff700 1 -- 172.21.15.29:0/3390463508 shutdown_connections 2023-10-07T11:04:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.327+0000 7ff2c2bff700 1 -- 172.21.15.29:0/3390463508 wait complete. 2023-10-07T11:04:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.328+0000 7ff2c2bff700 1 Processor -- start 2023-10-07T11:04:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.328+0000 7ff2c2bff700 1 -- start start 2023-10-07T11:04:28.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c2bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 0x7ff2bc14b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c2bff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2bc0b2bb0 0x7ff2bc15e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c2bff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2bc159550 0x7ff2bc0ad270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c2bff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc0ae0e0 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c2bff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc14bd70 con 0x7ff2bc0b2bb0 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c2bff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc14cca0 con 0x7ff2bc159550 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 0x7ff2bc14b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c099b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 0x7ff2bc14b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42800/0 (socket says 172.21.15.29:42800) 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.329+0000 7ff2c099b700 1 -- 172.21.15.29:0/1807390910 learned_addr learned my addr 172.21.15.29:0/1807390910 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c099b700 1 -- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2bc159550 msgr2=0x7ff2bc0ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c099b700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2bc159550 0x7ff2bc0ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c099b700 1 -- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2bc0b2bb0 msgr2=0x7ff2bc15e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c099b700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2bc0b2bb0 0x7ff2bc15e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c099b700 1 -- 172.21.15.29:0/1807390910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b00094f0 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c099b700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 0x7ff2bc14b830 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009290 tx=0x7ff2b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0011e60 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b001f070 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b001a870 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc15d130 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.330+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc14c650 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.332+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc04c3c0 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.332+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff2b001abe0 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.334+0000 7ff2b8ff9700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff2a0066f30 0x7ff2a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.334+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7ff2b0021070 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.334+0000 7ff2bbfff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff2a0066f30 0x7ff2a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.335+0000 7ff2bbfff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff2a0066f30 0x7ff2a00693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8006fd0 tx=0x7ff2a8008040 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.339+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2b0053780 con 0x7ff2bc0b0f50 2023-10-07T11:04:28.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.535+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 --> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2bc0683d0 con 0x7ff2a0066f30 2023-10-07T11:04:28.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.538+0000 7ff2b8ff9700 1 -- 172.21.15.29:0/1807390910 <== mgr.10416 v2:172.21.15.144:6836/131978 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50413 (secure 0 0 0) 0x7ff2bc0683d0 con 0x7ff2a0066f30 2023-10-07T11:04:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff2a0066f30 msgr2=0x7ff2a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff2a0066f30 0x7ff2a00693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8006fd0 tx=0x7ff2a8008040 comp rx=0 tx=0).stop 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 msgr2=0x7ff2bc14b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 0x7ff2bc14b830 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009290 tx=0x7ff2b0017000 comp rx=0 tx=0).stop 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 shutdown_connections 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7ff2a0066f30 0x7ff2a00693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2bc159550 0x7ff2bc0ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2bc0b0f50 0x7ff2bc14b830 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 --2- 172.21.15.29:0/1807390910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2bc0b2bb0 0x7ff2bc15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 >> 172.21.15.29:0/1807390910 conn(0x7ff2bc069190 msgr2=0x7ff2bc0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:28.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 shutdown_connections 2023-10-07T11:04:28.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.540+0000 7ff2c2bff700 1 -- 172.21.15.29:0/1807390910 wait complete. 2023-10-07T11:04:28.542 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-07T11:04:28.559 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-07T11:04:27.718606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2477064,"kb_used_data":20936,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752497656,"statfs":{"total":773094113280,"available":770557599744,"internally_reserved":0,"allocated":21438464,"data_stored":11222776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002400"},"pg_stats":[{"pgid":"1.0","version":"202'371","reported_seq":1225,"reported_epoch":202,"state":"active+clean","last_fresh":"2023-10-07T11:04:09.662118+0000","last_change":"2023-10-07T11:04:09.662083+0000","last_active":"2023-10-07T11:04:09.662118+0000","last_peered":"2023-10-07T11:04:09.662118+0000","last_clean":"2023-10-07T11:04:09.662118+0000","last_became_active":"2023-10-07T09:29:11.785972+0000","last_became_peered":"2023-10-07T09:29:11.785972+0000","last_unstale":"2023-10-07T11:04:09.662118+0000","last_undegraded":"2023-10-07T11:04:09.662118+0000","last_fullsized":"2023-10-07T11:04:09.662118+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'371","last_scrub_stamp":"2023-10-07T11:04:09.661981+0000","last_deep_scrub":"202'371","last_deep_scrub_stamp":"2023-10-07T11:04:09.661981+0000","last_clean_scrub_stamp":"2023-10-07T11:04:09.661981+0000","objects_scrubbed":2,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-08T22:23:47.287210+0000","scrub_duration":0.0054553850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719477864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.70399999999999996,"15min":0.73999999999999999},"min":{"1min":0.47899999999999998,"5min":0.45900000000000002,"15min":0.39300000000000002},"max":{"1min":0.82099999999999995,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.72899999999999998,"15min":0.753},"min":{"1min":0.55200000000000005,"5min":0.46999999999999997,"15min":0.44700000000000001},"max":{"1min":0.878,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.76800000000000002}]},{"osd":1,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.71999999999999997,"15min":0.74199999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":1.292,"15min":1.292},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.745,"15min":0.76300000000000001},"min":{"1min":0.55000000000000004,"5min":0.443,"15min":0.443},"max":{"1min":0.94999999999999996,"5min":1.0129999999999999,"15min":1.0880000000000001},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.752,"15min":0.77300000000000002},"min":{"1min":0.52700000000000002,"5min":0.499,"15min":0.499},"max":{"1min":0.91400000000000003,"5min":1.377,"15min":1.377},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.754,"15min":0.77100000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.93799999999999994,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.85099999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.55700000000000005,"5min":0.52400000000000002,"15min":0.48199999999999998},"max":{"1min":0.99299999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.745},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.79800000000000004,"15min":0.80600000000000005},"min":{"1min":0.622,"5min":0.56399999999999995,"15min":0.51100000000000001},"max":{"1min":1.0049999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.83699999999999997}]},{"osd":4,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.65600000000000003,"15min":0.627},"min":{"1min":0.45700000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89700000000000002,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68100000000000005,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1899999999999999,"5min":1.228,"15min":1.228},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70699999999999996,"15min":0.68400000000000005},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.999,"5min":0.999,"15min":1.1200000000000001},"last":0.626},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73799999999999999,"15min":0.72199999999999998},"min":{"1min":0.51000000000000001,"5min":0.48099999999999998,"15min":0.435},"max":{"1min":1.0229999999999999,"5min":1.494,"15min":1.494},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:04:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80400000000000005,"15min":0.78100000000000003},"min":{"1min":0.61799999999999999,"5min":0.53500000000000003,"15min":0.50700000000000001},"max":{"1min":1.1040000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.753,"15min":0.747},"min":{"1min":0.51700000000000002,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.92500000000000004,"5min":1.605,"15min":1.605},"last":0.80200000000000005}]}]},{"osd":6,"up_from":15,"seq":64424510568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316144,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055696,"statfs":{"total":96636764160,"available":96313032704,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71999999999999997,"15min":0.73199999999999998},"min":{"1min":0.433,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70899999999999996,"15min":0.73199999999999998},"min":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.41699999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.002},"last":0.749}]},{"osd":1,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72699999999999998,"15min":0.75},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92300000000000004,"5min":0.96599999999999997,"15min":1.0189999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.504,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0069999999999999},"last":0.79300000000000004}]},{"osd":3,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73899999999999999,"15min":0.76300000000000001},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.504},"max":{"1min":0.98799999999999999,"5min":0.996,"15min":1.073},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76500000000000001,"15min":0.77800000000000002},"min":{"1min":0.49299999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.96799999999999997,"5min":0.98899999999999999,"15min":1.0369999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.35799999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.97799999999999998},"last":0.497},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66600000000000004,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.44,"15min":0.439},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68700000000000006,"15min":0.67400000000000004},"min":{"1min":0.52500000000000002,"5min":0.46500000000000002,"15min":0.40799999999999997},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71599999999999997,"15min":0.71299999999999997},"min":{"1min":0.58299999999999996,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.98999999999999999,"5min":1.022,"15min":1.0580000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.71699999999999997,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.034,"5min":1.034,"15min":1.0880000000000001},"last":0.64000000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307952,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063888,"statfs":{"total":96636764160,"available":96321421312,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.85599999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.82599999999999996,"15min":0.88700000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.379},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.89900000000000002,"15min":0.96299999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.378,"15min":0.37},"max":{"1min":0.70399999999999996,"5min":0.84999999999999998,"15min":0.88900000000000001},"last":0.432}]},{"osd":4,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.51600000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.373},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80500000000000005,"15min":0.81299999999999994},"min":{"1min":0.59999999999999998,"5min":0.51800000000000002,"15min":0.51400000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2430000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.82699999999999996,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2310000000000001,"5min":1.242,"15min":1.242},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80300000000000005,"15min":0.81000000000000005},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.0840000000000001,"5min":1.2,"15min":1.2},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.871,"15min":0.874},"min":{"1min":0.67500000000000004,"5min":0.65000000000000002,"15min":0.58399999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57399999999999995},"max":{"1min":1.0409999999999999,"5min":1.157,"15min":1.302},"last":0.76000000000000001}]},{"osd":7,"last update":"Sat Oct 7 11:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.88700000000000001},"min":{"1min":0.623,"5min":0.56399999999999995,"15min":0.52800000000000002},"max":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.877,"15min":0.878},"min":{"1min":0.64700000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.72199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129543270,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312628,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059212,"statfs":{"total":96636764160,"available":96316633088,"internally_reserved":0,"allocated":3125248,"data_stored":1845623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70999999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76800000000000002,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.622}]},{"osd":2,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85699999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"max":{"1min":0.67500000000000004,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.499}]},{"osd":3,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.927,"15min":1.927},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65200000000000002,"15min":0.64800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80000000000000004,"5min":1.331,"15min":1.331},"last":0.78900000000000003}]},{"osd":4,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.53200000000000003,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.84399999999999997,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0640000000000001,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.97499999999999998}]},{"osd":5,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80900000000000005,"15min":0.81499999999999995},"min":{"1min":0.59599999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4430000000000001,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.82299999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3400000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.1180000000000001}]},{"osd":6,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.85699999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.008,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.03,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.004}]},{"osd":7,"last update":"Sat Oct 7 11:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.86899999999999999,"15min":0.874},"min":{"1min":0.65000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0669999999999999,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89800000000000002,"15min":0.88400000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0740000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":1.0449999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44600000000000001,"15min":0.45900000000000002},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.81699999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.30399999999999999},"max":{"1min":0.69399999999999995,"5min":1.087,"15min":1.087},"last":0.65300000000000002}]},{"osd":1,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.95099999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.40500000000000003},"max":{"1min":0.80800000000000005,"5min":0.86399999999999999,"15min":0.98199999999999998},"last":0.60799999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59799999999999998,"5min":0.71799999999999997,"15min":0.80000000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82199999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.69099999999999995}]},{"osd":4,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.78900000000000003},"min":{"1min":0.63100000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.163,"15min":1.1879999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.80300000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.266,"15min":1.5640000000000001},"last":0.84099999999999997}]},{"osd":5,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.59199999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.004,"5min":1.2989999999999999,"15min":1.528},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78100000000000003,"15min":0.80300000000000005},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":1.083,"15min":1.2829999999999999},"last":0.91700000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95999999999999996,"5min":1.1819999999999999,"15min":1.3280000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.65200000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0580000000000001,"5min":1.242,"15min":1.3779999999999999},"last":0.85699999999999998}]},{"osd":7,"last update":"Sat Oct 7 11:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84999999999999998,"15min":0.86799999999999999},"min":{"1min":0.67600000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0269999999999999,"5min":1.2050000000000001,"15min":1.629},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.86699999999999999},"min":{"1min":0.58099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0569999999999999,"5min":1.274,"15min":1.5940000000000001},"last":0.89800000000000002}]}]},{"osd":3,"up_from":14,"seq":60129543272,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299760,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072080,"statfs":{"total":96636764160,"available":96329809920,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.314},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77900000000000003,"5min":0.80600000000000005,"15min":0.91600000000000004},"last":0.443}]},{"osd":1,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.48099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.88600000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.47699999999999998},"max":{"1min":0.96199999999999997,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.626,"5min":0.495,"15min":0.495},"max":{"1min":0.97699999999999998,"5min":1.0669999999999999,"15min":1.222},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80700000000000005,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.52600000000000002},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.216},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95399999999999996,"5min":1.27,"15min":1.27},"last":0.79600000000000004}]},{"osd":6,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82499999999999996,"15min":0.84199999999999997},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92700000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84899999999999998},"min":{"1min":0.65400000000000003,"5min":0.56599999999999995,"15min":0.54600000000000004},"max":{"1min":1.046,"5min":1.389,"15min":1.389},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.64100000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0429999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"max":{"1min":0.97599999999999998,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.72699999999999998}]}]},{"osd":4,"up_from":15,"seq":64424510568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316144,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055696,"statfs":{"total":96636764160,"available":96313032704,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.72699999999999998},"min":{"1min":0.439,"5min":0.42999999999999999,"15min":0.34000000000000002},"max":{"1min":0.96299999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71599999999999997,"15min":0.73899999999999999},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94699999999999995,"5min":1.0900000000000001,"15min":1.248},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.748,"15min":0.754},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.73399999999999999,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.505,"15min":0.48999999999999999},"max":{"1min":0.87,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.499,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91300000000000003,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73099999999999998,"15min":0.755},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":1.0329999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.745,"15min":0.749},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.46100000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76700000000000002,"15min":0.78200000000000003},"min":{"1min":0.59899999999999998,"5min":0.501,"15min":0.501},"max":{"1min":0.878,"5min":1.0369999999999999,"15min":1.179},"last":0.78000000000000003}]},{"osd":5,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.629,"15min":0.61599999999999999},"min":{"1min":0.46300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":1.056,"15min":1.056},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.64900000000000002,"15min":0.63},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.41499999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.85699999999999998}]},{"osd":6,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66100000000000003,"15min":0.65200000000000002},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.378},"max":{"1min":0.93999999999999995,"5min":1.0229999999999999,"15min":1.0609999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.71499999999999997,"15min":0.70099999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.46000000000000002},"max":{"1min":1.034,"5min":1.097,"15min":1.097},"last":0.90900000000000003}]},{"osd":7,"last update":"Sat Oct 7 11:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.749,"15min":0.72299999999999998},"min":{"1min":0.56100000000000005,"5min":0.49099999999999999,"15min":0.41499999999999998},"max":{"1min":1.0029999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77200000000000002,"15min":0.76000000000000001},"min":{"1min":0.54800000000000004,"5min":0.505,"15min":0.49099999999999999},"max":{"1min":0.998,"5min":1.004,"15min":1.171},"last":0.877}]}]},{"osd":5,"up_from":14,"seq":60129543270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312048,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059792,"statfs":{"total":96636764160,"available":96317227008,"internally_reserved":0,"allocated":2531328,"data_stored":1255255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73799999999999999,"15min":0.77300000000000002},"min":{"1min":0.48999999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90800000000000003,"5min":0.998,"15min":1.585},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.52300000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0309999999999999,"5min":1.073,"15min":1.673},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.69499999999999995,"15min":0.73599999999999999},"min":{"1min":0.441,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.95399999999999996,"5min":0.99299999999999999,"15min":1.0189999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.75900000000000001},"min":{"1min":0.55000000000000004,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":0.96399999999999997,"5min":0.98899999999999999,"15min":1.139},"last":0.83099999999999996}]},{"osd":2,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70099999999999996,"15min":0.73099999999999998},"min":{"1min":0.46800000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77300000000000002,"5min":0.93200000000000005,"15min":1.0580000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73199999999999998,"15min":0.751},"min":{"1min":0.53200000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.85199999999999998}]},{"osd":3,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.76700000000000002},"min":{"1min":0.54500000000000004,"5min":0.51200000000000001,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.069},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.73999999999999999},"min":{"1min":0.56000000000000005,"5min":0.51000000000000001,"15min":0.504},"max":{"1min":0.88200000000000001,"5min":0.96899999999999997,"15min":1.161},"last":0.80200000000000005}]},{"osd":4,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.872},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.65200000000000002,"15min":0.63400000000000001},"min":{"1min":0.42099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86899999999999999,"5min":0.91800000000000004,"15min":0.95799999999999996},"last":0.749}]},{"osd":6,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.51400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.98799999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.496,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.77600000000000002}]},{"osd":7,"last update":"Sat Oct 7 11:04:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71699999999999997,"15min":0.70099999999999996},"min":{"1min":0.57699999999999996,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.03,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.78000000000000003,"15min":0.76300000000000001},"min":{"1min":0.70299999999999996,"5min":0.496,"15min":0.496},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.90000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-07T11:04:28.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-07T11:04:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.750+0000 7efcf09ff700 1 Processor -- start 2023-10-07T11:04:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.750+0000 7efcf09ff700 1 -- start start 2023-10-07T11:04:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efcf09ff700 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efcec0ae8a0 0x7efcec158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efcf09ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec0acb40 con 0x7efcec15ad30 2023-10-07T11:04:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efcf09ff700 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec0abf30 con 0x7efcec15a960 2023-10-07T11:04:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efcf09ff700 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec158c60 con 0x7efcec0ae8a0 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efcead9d700 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efcec15ad30 0x7efcec0b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43088/0 (socket says 172.21.15.29:43088) 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efcead9d700 1 -- 172.21.15.29:0/3120188325 learned_addr learned my addr 172.21.15.29:0/3120188325 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efce9d9b700 1 --2- 172.21.15.29:0/3120188325 >> v2:172.21.15.144:3300/0 conn(0x7efcec0ae8a0 0x7efcec158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.752+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933768266 0 0) 0x7efcec0acb40 con 0x7efcec15ad30 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003570 con 0x7efcec15ad30 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381876078 0 0) 0x7efcec0abf30 con 0x7efcec15a960 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcec0acb40 con 0x7efcec15a960 2023-10-07T11:04:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce9d9b700 1 -- 172.21.15.29:0/3120188325 >> v1:172.21.15.144:6789/0 conn(0x7efcec15a960 legacy=0x7efcec0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce9d9b700 1 -- 172.21.15.29:0/3120188325 >> v1:172.21.15.29:6789/0 conn(0x7efcec15ad30 legacy=0x7efcec0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce9d9b700 1 -- 172.21.15.29:0/3120188325 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0ac210 con 0x7efcec0ae8a0 2023-10-07T11:04:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce9d9b700 1 --2- 172.21.15.29:0/3120188325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec0ae8a0 0x7efcec158e20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7efce00098e0 tx=0x7efce00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a437baa4fca11855 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0011630 con 0x7efcec0ae8a0 2023-10-07T11:04:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0011790 con 0x7efcec0ae8a0 2023-10-07T11:04:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.753+0000 7efce959a700 1 -- 172.21.15.29:0/3120188325 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0011b30 con 0x7efcec0ae8a0 2023-10-07T11:04:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 -- 172.21.15.29:0/3120188325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec0ae8a0 msgr2=0x7efcec158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 --2- 172.21.15.29:0/3120188325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec0ae8a0 0x7efcec158e20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7efce00098e0 tx=0x7efce00091a0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 -- 172.21.15.29:0/3120188325 shutdown_connections 2023-10-07T11:04:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 --2- 172.21.15.29:0/3120188325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec0ae8a0 0x7efcec158e20 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 -- 172.21.15.29:0/3120188325 >> 172.21.15.29:0/3120188325 conn(0x7efcec067610 msgr2=0x7efcec067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 -- 172.21.15.29:0/3120188325 shutdown_connections 2023-10-07T11:04:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.754+0000 7efcf09ff700 1 -- 172.21.15.29:0/3120188325 wait complete. 2023-10-07T11:04:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.755+0000 7efcf09ff700 1 Processor -- start 2023-10-07T11:04:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.756+0000 7efcf09ff700 1 -- start start 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.756+0000 7efcf09ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 0x7efcec15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.756+0000 7efcf09ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec15a960 0x7efcec0af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.756+0000 7efcf09ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcec15ad30 0x7efcec15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcf09ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec15c0b0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcf09ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0afa80 con 0x7efcec15a960 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcf09ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec16d560 con 0x7efcec15ad30 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 0x7efcec15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 0x7efcec15bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42808/0 (socket says 172.21.15.29:42808) 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 -- 172.21.15.29:0/2811372863 learned_addr learned my addr 172.21.15.29:0/2811372863 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efce9d9b700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec15a960 0x7efcec0af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcead9d700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcec15ad30 0x7efcec15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 -- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcec15ad30 msgr2=0x7efcec15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcec15ad30 0x7efcec15c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 -- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec15a960 msgr2=0x7efcec0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec15a960 0x7efcec0af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 -- 172.21.15.29:0/2811372863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0009420 con 0x7efcec0ae8a0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efce9d9b700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec15a960 0x7efcec0af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.757+0000 7efcea59c700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 0x7efcec15bb70 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00c7c0 tx=0x7efcdc007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.758+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc016070 con 0x7efcec0ae8a0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.758+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc011680 con 0x7efcec0ae8a0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.758+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc011a20 con 0x7efcec0ae8a0 2023-10-07T11:04:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.758+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec16d840 con 0x7efcec0ae8a0 2023-10-07T11:04:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.758+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec16de00 con 0x7efcec0ae8a0 2023-10-07T11:04:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.759+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec14dbd0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.760+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efcdc012440 con 0x7efcec0ae8a0 2023-10-07T11:04:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.762+0000 7efcda7fc700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7efccc067000 0x7efccc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.762+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7efcdc0844b0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.762+0000 7efce9d9b700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7efccc067000 0x7efccc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.763+0000 7efce9d9b700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7efccc067000 0x7efccc0694b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efce00093f0 tx=0x7efce0017000 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.766+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efcec14dbd0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.968+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7efcec14dbd0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.972+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7efcdc0515f0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.972+0000 7efcda7fc700 1 -- 172.21.15.29:0/2811372863 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v76)=0 v76) v1 ==== 123+0+0 (secure 0 0 0) 0x7efcec14dbd0 con 0x7efcec0ae8a0 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7efccc067000 msgr2=0x7efccc0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7efccc067000 0x7efccc0694b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efce00093f0 tx=0x7efce0017000 comp rx=0 tx=0).stop 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 msgr2=0x7efcec15bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 0x7efcec15bb70 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00c7c0 tx=0x7efcdc007dd0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 shutdown_connections 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7efccc067000 0x7efccc0694b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcec15ad30 0x7efcec15c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcec0ae8a0 0x7efcec15bb70 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 --2- 172.21.15.29:0/2811372863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcec15a960 0x7efcec0af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:28.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.975+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 >> 172.21.15.29:0/2811372863 conn(0x7efcec067610 msgr2=0x7efcec159840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:28.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.976+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 shutdown_connections 2023-10-07T11:04:28.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:28.976+0000 7efcf09ff700 1 -- 172.21.15.29:0/2811372863 wait complete. 2023-10-07T11:04:28.993 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-07T11:04:28.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-07T11:04:29.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.189+0000 7fd4dc5ff700 1 Processor -- start 2023-10-07T11:04:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.189+0000 7fd4dc5ff700 1 -- start start 2023-10-07T11:04:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4dc5ff700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4d4157290 0x7fd4d40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4dc5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d40b1a10 0x7fd4d4157660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4dc5ff700 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4158ad0 con 0x7fd4d40ae8b0 2023-10-07T11:04:29.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4dc5ff700 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d415b140 con 0x7fd4d4157290 2023-10-07T11:04:29.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4dc5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d415b420 con 0x7fd4d40b1a10 2023-10-07T11:04:29.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4da39b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4d4157290 0x7fd4d40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:29.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4da39b700 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4d4157290 0x7fd4d40cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42812/0 (socket says 172.21.15.29:42812) 2023-10-07T11:04:29.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4da39b700 1 -- 172.21.15.29:0/359709440 learned_addr learned my addr 172.21.15.29:0/359709440 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:29.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4d9b9a700 1 --1- 172.21.15.29:0/359709440 >> v1:172.21.15.29:6789/0 conn(0x7fd4d40ae8b0 0x7fd4d40b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43100/0 (socket says 172.21.15.29:43100) 2023-10-07T11:04:29.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.191+0000 7fd4dab9c700 1 --2- 172.21.15.29:0/359709440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d40b1a10 0x7fd4d4157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:29.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4da39b700 1 -- 172.21.15.29:0/359709440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d40b1a10 msgr2=0x7fd4d4157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-07T11:04:29.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4da39b700 1 --2- 172.21.15.29:0/359709440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d40b1a10 0x7fd4d4157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4da39b700 1 -- 172.21.15.29:0/359709440 >> v1:172.21.15.29:6789/0 conn(0x7fd4d40ae8b0 legacy=0x7fd4d40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:29.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4da39b700 1 -- 172.21.15.29:0/359709440 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d415b700 con 0x7fd4d4157290 2023-10-07T11:04:29.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4da39b700 1 --2- 172.21.15.29:0/359709440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d4157290 0x7fd4d40cc870 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00099b0 tx=0x7fd4d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab96d767c9d6f79d server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:29.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4d9399700 1 -- 172.21.15.29:0/359709440 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d000b860 con 0x7fd4d4157290 2023-10-07T11:04:29.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4d9399700 1 -- 172.21.15.29:0/359709440 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd4d000b9c0 con 0x7fd4d4157290 2023-10-07T11:04:29.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.192+0000 7fd4d9399700 1 -- 172.21.15.29:0/359709440 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d000bd80 con 0x7fd4d4157290 2023-10-07T11:04:29.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/359709440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d4157290 msgr2=0x7fd4d40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:29.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/359709440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d4157290 0x7fd4d40cc870 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00099b0 tx=0x7fd4d0009250 comp rx=0 tx=0).stop 2023-10-07T11:04:29.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/359709440 shutdown_connections 2023-10-07T11:04:29.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/359709440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d40b1a10 0x7fd4d4157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/359709440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d4157290 0x7fd4d40cc870 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/359709440 >> 172.21.15.29:0/359709440 conn(0x7fd4d40680a0 msgr2=0x7fd4d40cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/359709440 shutdown_connections 2023-10-07T11:04:29.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.193+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/359709440 wait complete. 2023-10-07T11:04:29.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.194+0000 7fd4dc5ff700 1 Processor -- start 2023-10-07T11:04:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.194+0000 7fd4dc5ff700 1 -- start start 2023-10-07T11:04:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.195+0000 7fd4dc5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 0x7fd4d41374b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.195+0000 7fd4dc5ff700 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d4157290 0x7fd4d4137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.195+0000 7fd4dc5ff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4d4069570 0x7fd4d4069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:29.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.195+0000 7fd4dc5ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d4069f50 con 0x7fd4d40b1a10 2023-10-07T11:04:29.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.195+0000 7fd4dc5ff700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d4138f50 con 0x7fd4d4069570 2023-10-07T11:04:29.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.195+0000 7fd4dc5ff700 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d41380e0 con 0x7fd4d4157290 2023-10-07T11:04:29.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 0x7fd4d41374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:29.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 0x7fd4d41374b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42816/0 (socket says 172.21.15.29:42816) 2023-10-07T11:04:29.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 -- 172.21.15.29:0/39852401 learned_addr learned my addr 172.21.15.29:0/39852401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-07T11:04:29.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4dab9c700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4d4069570 0x7fd4d4069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:29.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4d9b9a700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d4157290 0x7fd4d4137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:29.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 -- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d4157290 msgr2=0x7fd4d4137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:29.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d4157290 0x7fd4d4137ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 -- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4d4069570 msgr2=0x7fd4d4069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:29.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4d4069570 0x7fd4d4069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 -- 172.21.15.29:0/39852401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00094f0 con 0x7fd4d40b1a10 2023-10-07T11:04:29.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.196+0000 7fd4da39b700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 0x7fd4d41374b0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4068dd0 tx=0x7fd4d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:29.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.197+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0011e70 con 0x7fd4d40b1a10 2023-10-07T11:04:29.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.197+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd4d001f070 con 0x7fd4d40b1a10 2023-10-07T11:04:29.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.197+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00266b0 con 0x7fd4d40b1a10 2023-10-07T11:04:29.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.197+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d4138340 con 0x7fd4d40b1a10 2023-10-07T11:04:29.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.197+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d4138900 con 0x7fd4d40b1a10 2023-10-07T11:04:29.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.198+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d404c3c0 con 0x7fd4d40b1a10 2023-10-07T11:04:29.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.199+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 106) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd4d0026a60 con 0x7fd4d40b1a10 2023-10-07T11:04:29.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.199+0000 7fd4ca7fc700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fd4b8066f30 0x7fd4b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-07T11:04:29.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.200+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5974+0+0 (secure 0 0 0) 0x7fd4d0021070 con 0x7fd4d40b1a10 2023-10-07T11:04:29.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.200+0000 7fd4d9b9a700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fd4b8066f30 0x7fd4b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-07T11:04:29.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.201+0000 7fd4d9b9a700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fd4b8066f30 0x7fd4b80693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4007fd0 tx=0x7fd4c4007d70 comp rx=0 tx=0).ready entity=mgr.10416 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-07T11:04:29.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.204+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4d0053650 con 0x7fd4d40b1a10 2023-10-07T11:04:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.450+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4d415b9e0 con 0x7fd4d40b1a10 2023-10-07T11:04:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.450+0000 7fd4ca7fc700 1 -- 172.21.15.29:0/39852401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3531 (secure 0 0 0) 0x7fd4d002bde0 con 0x7fd4d40b1a10 2023-10-07T11:04:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-07T11:04:29.452 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":103,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/559091889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":559091889},{"type":"v1","addr":"172.21.15.29:6835","nonce":559091889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6834/3577343850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6833","nonce":3577343850},{"type":"v1","addr":"172.21.15.144:6834","nonce":3577343850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8846,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/973332772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":973332772},{"type":"v1","addr":"172.21.15.29:6834","nonce":973332772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":103},{"gid":9598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2590970192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":2590970192},{"type":"v1","addr":"172.21.15.144:6835","nonce":2590970192}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[]} 2023-10-07T11:04:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fd4b8066f30 msgr2=0x7fd4b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fd4b8066f30 0x7fd4b80693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4007fd0 tx=0x7fd4c4007d70 comp rx=0 tx=0).stop 2023-10-07T11:04:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 msgr2=0x7fd4d41374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-07T11:04:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 0x7fd4d41374b0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4068dd0 tx=0x7fd4d0009250 comp rx=0 tx=0).stop 2023-10-07T11:04:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 shutdown_connections 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4d4069570 0x7fd4d4069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4d40b1a10 0x7fd4d41374b0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:6836/131978,v1:172.21.15.144:6837/131978] conn(0x7fd4b8066f30 0x7fd4b80693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 --2- 172.21.15.29:0/39852401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4d4157290 0x7fd4d4137ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 >> 172.21.15.29:0/39852401 conn(0x7fd4d40680a0 msgr2=0x7fd4d4151bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 shutdown_connections 2023-10-07T11:04:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-07T11:04:29.453+0000 7fd4dc5ff700 1 -- 172.21.15.29:0/39852401 wait complete. 2023-10-07T11:04:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 103 2023-10-07T11:04:29.471 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-07T11:04:29.471 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-07T11:04:29.472 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-07T11:04:29.472 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.484 INFO:tasks.ceph.mds.a:Stopped 2023-10-07T11:04:29.484 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-07T11:04:29.484 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.537 INFO:tasks.ceph.mds.c:Stopped 2023-10-07T11:04:29.537 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-07T11:04:29.537 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.553 INFO:tasks.ceph.mds.b:Stopped 2023-10-07T11:04:29.553 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-07T11:04:29.553 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.608 INFO:tasks.ceph.mds.d:Stopped 2023-10-07T11:04:29.609 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-07T11:04:29.609 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-07T11:04:29.609 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.657 INFO:tasks.ceph.osd.0:Stopped 2023-10-07T11:04:29.657 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-07T11:04:29.657 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.706 INFO:tasks.ceph.osd.1:Stopped 2023-10-07T11:04:29.707 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-07T11:04:29.707 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.754 INFO:tasks.ceph.osd.2:Stopped 2023-10-07T11:04:29.754 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-07T11:04:29.754 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.809 INFO:tasks.ceph.osd.3:Stopped 2023-10-07T11:04:29.809 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-07T11:04:29.810 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.865 INFO:tasks.ceph.osd.4:Stopped 2023-10-07T11:04:29.866 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-07T11:04:29.866 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.923 INFO:tasks.ceph.osd.5:Stopped 2023-10-07T11:04:29.923 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-07T11:04:29.923 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:29.973 INFO:tasks.ceph.osd.6:Stopped 2023-10-07T11:04:29.974 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-07T11:04:29.974 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:30.020 INFO:tasks.ceph.osd.7:Stopped 2023-10-07T11:04:30.020 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-07T11:04:30.020 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-07T11:04:30.020 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:30.084 INFO:tasks.ceph.mgr.y:Stopped 2023-10-07T11:04:30.084 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-07T11:04:30.084 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:30.151 INFO:tasks.ceph.mgr.x:Stopped 2023-10-07T11:04:30.151 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-07T11:04:30.151 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-07T11:04:30.151 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:30.183 INFO:tasks.ceph.mon.a:Stopped 2023-10-07T11:04:30.183 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-07T11:04:30.183 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:30.207 INFO:tasks.ceph.mon.b:Stopped 2023-10-07T11:04:30.207 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-07T11:04:30.207 INFO:teuthology.orchestra.run:waiting for 300 2023-10-07T11:04:30.274 INFO:tasks.ceph.mon.c:Stopped 2023-10-07T11:04:30.274 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-07T11:04:30.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-07T11:04:30.356 INFO:teuthology.orchestra.run.smithi029.stdout:1696672017.284498 mon.a (mon.0) 838 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-10-07T11:04:30.357 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-07T11:04:30.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-07T11:04:30.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-07T11:04:30.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-07T11:04:30.605 INFO:teuthology.orchestra.run.smithi029.stdout:1696672017.284498 mon.a (mon.0) 838 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-10-07T11:04:30.607 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:04:30.607 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-07T11:04:31.054 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:04:31.055 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-07T11:04:31.242 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:04:31.242 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-07T11:04:31.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:04:31.482 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-07T11:04:31.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T11:04:31.665 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-07T11:04:31.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T11:04:31.962 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-07T11:04:32.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T11:04:32.168 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-07T11:04:32.383 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T11:04:32.384 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-07T11:04:32.567 INFO:tasks.ceph:Archiving mon data... 2023-10-07T11:04:32.568 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/data/mon.a.tgz 2023-10-07T11:04:32.568 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-10-07T11:04:32.619 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.SV3ZHIzc7h 2023-10-07T11:04:32.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.SV3ZHIzc7h 2023-10-07T11:04:33.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.SV3ZHIzc7h 2023-10-07T11:04:33.242 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.SV3ZHIzc7h is 2MB 2023-10-07T11:04:33.379 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.SV3ZHIzc7h 2023-10-07T11:04:33.425 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/data/mon.b.tgz 2023-10-07T11:04:33.426 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2023-10-07T11:04:33.475 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.4eNgQROk65 2023-10-07T11:04:33.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.4eNgQROk65 2023-10-07T11:04:33.965 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.4eNgQROk65 2023-10-07T11:04:34.083 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.4eNgQROk65 is 3MB 2023-10-07T11:04:34.225 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.4eNgQROk65 2023-10-07T11:04:34.271 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/data/mon.c.tgz 2023-10-07T11:04:34.271 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2023-10-07T11:04:34.357 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.dO4pjkAJf3 2023-10-07T11:04:34.357 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dO4pjkAJf3 2023-10-07T11:04:34.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.dO4pjkAJf3 2023-10-07T11:04:35.007 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.dO4pjkAJf3 is 3MB 2023-10-07T11:04:35.144 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.dO4pjkAJf3 2023-10-07T11:04:35.191 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-07T11:04:35.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-07T11:04:35.196 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-07T11:04:35.307 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-07T11:04:35.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-07T11:04:35.313 DEBUG:teuthology.orchestra.run.smithi144:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-07T11:04:35.389 INFO:teuthology.orchestra.run.smithi144.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-07T11:04:35.409 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-07T11:04:35.426 INFO:tasks.ceph:Archiving crash dumps... 2023-10-07T11:04:35.428 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/remote/smithi029/crash 2023-10-07T11:04:35.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-07T11:04:35.499 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/remote/smithi144/crash 2023-10-07T11:04:35.500 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-07T11:04:35.568 INFO:tasks.ceph:Shutting down logrotate 2023-10-07T11:04:35.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:04:35.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-07T11:04:35.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-07T11:04:35.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-07T11:04:35.809 INFO:tasks.ceph:Compressing logs... 2023-10-07T11:04:35.809 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-07T11:04:35.814 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-07T11:04:35.879 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130747.log 2023-10-07T11:04:35.881 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.121289.log 2023-10-07T11:04:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164767.log 2023-10-07T11:04:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130747.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.130747.log.gz 2023-10-07T11:04:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.176802.log 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.121783.log 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.1.121289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.121289.log.gz 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.3.121783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.121783.log.gz 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115043.log 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122292.log 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.164767.log.gz 2023-10-07T11:04:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.176802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.176802.log.gz 2023-10-07T11:04:35.885 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143380.log 2023-10-07T11:04:35.885 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114995.log 2023-10-07T11:04:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122292.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122292.log.gz 2023-10-07T11:04:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147014.log 2023-10-07T11:04:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143380.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.143380.log.gz 2023-10-07T11:04:35.886 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175329.log 2023-10-07T11:04:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147014.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147014.log.gz 2023-10-07T11:04:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120565.log 2023-10-07T11:04:35.889 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.115043.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115043.log.gz 2023-10-07T11:04:35.889 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115291.log 2023-10-07T11:04:35.889 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.114995.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114995.log.gz 2023-10-07T11:04:35.889 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115095.log 2023-10-07T11:04:35.889 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.115291.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.115291.log.gz 2023-10-07T11:04:35.889 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-07T11:04:35.891 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.115095.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.115095.log.gz 2023-10-07T11:04:35.891 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.121787.log 2023-10-07T11:04:35.891 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126840.log 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.175329.log.gz 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120565.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120565.log.gz 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129273.log 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126840.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126840.log.gz 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162554.log 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129273.log: gzip -5 88.6%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.129273.log.gz/var/log/ceph/ceph-client.admin.136779.log 2023-10-07T11:04:35.892 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162144.log 2023-10-07T11:04:35.894 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.162554.log.gz 2023-10-07T11:04:35.894 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136779.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.136779.log.gz 2023-10-07T11:04:35.894 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142879.log 2023-10-07T11:04:35.894 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.3.121787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.121787.log.gz 2023-10-07T11:04:35.894 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114946.log 2023-10-07T11:04:35.894 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-07T11:04:35.895 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.114946.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114946.log.gz 2023-10-07T11:04:35.895 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-07T11:04:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162144.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.162144.log.gz 2023-10-07T11:04:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.164022.log 2023-10-07T11:04:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142879.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.142879.log.gz 2023-10-07T11:04:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169720.log 2023-10-07T11:04:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.164022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.164022.log.gz 2023-10-07T11:04:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175105.log 2023-10-07T11:04:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169720.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.169720.log.gz 2023-10-07T11:04:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121131.log 2023-10-07T11:04:35.898 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-07T11:04:35.898 INFO:teuthology.orchestra.run.smithi144.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-07T11:04:35.898 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-07T11:04:35.899 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-07T11:04:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175105.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.175105.log.gzgzip 2023-10-07T11:04:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.175873.log 2023-10-07T11:04:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121131.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121131.log.gz 2023-10-07T11:04:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121133.log 2023-10-07T11:04:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175873.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.175873.log.gz 2023-10-07T11:04:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161808.log 2023-10-07T11:04:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121133.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121133.log.gz 2023-10-07T11:04:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174623.log 2023-10-07T11:04:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161808.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.161808.log.gz 2023-10-07T11:04:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176420.log 2023-10-07T11:04:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174623.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.174623.log.gz 2023-10-07T11:04:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144101.log 2023-10-07T11:04:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.176420.log.gz 2023-10-07T11:04:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178894.log 2023-10-07T11:04:35.906 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-07T11:04:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144101.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144101.log.gz 2023-10-07T11:04:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129201.log 2023-10-07T11:04:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178894.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.178894.log.gz 2023-10-07T11:04:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150317.log 2023-10-07T11:04:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129201.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.129201.log.gz 2023-10-07T11:04:35.908 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143199.log 2023-10-07T11:04:35.908 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150317.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150317.log.gz 2023-10-07T11:04:35.908 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169617.log 2023-10-07T11:04:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143199.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143199.log.gz 2023-10-07T11:04:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126070.log 2023-10-07T11:04:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169617.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.169617.log.gz 2023-10-07T11:04:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178747.log 2023-10-07T11:04:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126070.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.126070.log.gz 2023-10-07T11:04:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147783.log 2023-10-07T11:04:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178747.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.178747.log.gz 2023-10-07T11:04:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135973.log 2023-10-07T11:04:35.912 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-07T11:04:35.913 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147783.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.147783.log.gz 2023-10-07T11:04:35.913 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141028.log 2023-10-07T11:04:35.914 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135973.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.135973.log.gz 2023-10-07T11:04:35.914 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148600.log 2023-10-07T11:04:35.915 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165374.log 2023-10-07T11:04:35.915 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141028.log.gz 2023-10-07T11:04:35.916 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148600.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148600.log.gz 2023-10-07T11:04:35.916 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144709.log 2023-10-07T11:04:35.917 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165374.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.165374.log.gz 2023-10-07T11:04:35.918 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142249.log 2023-10-07T11:04:35.918 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144709.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144709.log.gz 2023-10-07T11:04:35.918 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154393.log 2023-10-07T11:04:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142249.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142249.log.gz 2023-10-07T11:04:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125310.log 2023-10-07T11:04:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154393.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154393.log.gz 2023-10-07T11:04:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122917.log 2023-10-07T11:04:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125310.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.125310.log.gz 2023-10-07T11:04:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156381.log 2023-10-07T11:04:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170978.log 2023-10-07T11:04:35.923 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122917.log.gz 2023-10-07T11:04:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156381.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.156381.log.gz 2023-10-07T11:04:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165692.log 2023-10-07T11:04:35.924 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115144.log 2023-10-07T11:04:35.924 INFO:teuthology.orchestra.run.smithi144.stderr: 93.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-07T11:04:35.925 INFO:teuthology.orchestra.run.smithi144.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-07T11:04:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170978.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.170978.log.gz 2023-10-07T11:04:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173872.log 2023-10-07T11:04:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165692.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.165692.log.gz 2023-10-07T11:04:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126285.log 2023-10-07T11:04:35.927 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.115144.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115144.log.gz 2023-10-07T11:04:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.173872.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.173872.log.gz 2023-10-07T11:04:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175154.log 2023-10-07T11:04:35.928 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-07T11:04:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126285.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.126285.log.gz-5 2023-10-07T11:04:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:--verbose -- /var/log/ceph/ceph-client.admin.125776.log 2023-10-07T11:04:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175154.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.175154.log.gz 2023-10-07T11:04:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121526.log 2023-10-07T11:04:35.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125776.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125776.log.gz 2023-10-07T11:04:35.930 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.124512.log 2023-10-07T11:04:35.931 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121526.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.121526.log.gz 2023-10-07T11:04:35.931 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165242.log 2023-10-07T11:04:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.2.124512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.124512.log.gz 2023-10-07T11:04:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163295.log 2023-10-07T11:04:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165242.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.165242.log.gz 2023-10-07T11:04:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174885.log 2023-10-07T11:04:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163295.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.163295.log.gz 2023-10-07T11:04:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162020.log 2023-10-07T11:04:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174885.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.174885.log.gz 2023-10-07T11:04:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137978.log 2023-10-07T11:04:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162020.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.162020.log.gz 2023-10-07T11:04:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120155.log 2023-10-07T11:04:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137978.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.137978.log.gz 2023-10-07T11:04:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155841.log 2023-10-07T11:04:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120155.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120155.log.gz 2023-10-07T11:04:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178395.log 2023-10-07T11:04:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155841.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155841.log.gz 2023-10-07T11:04:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143248.log 2023-10-07T11:04:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178395.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.178395.log.gz 2023-10-07T11:04:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178116.log 2023-10-07T11:04:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143248.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.143248.log.gz 2023-10-07T11:04:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142610.log 2023-10-07T11:04:35.942 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178116.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.178116.log.gz 2023-10-07T11:04:35.942 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145575.log 2023-10-07T11:04:35.942 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.121303.log 2023-10-07T11:04:35.943 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142610.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.142610.log.gz 2023-10-07T11:04:35.943 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143996.log 2023-10-07T11:04:35.944 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145575.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145575.log.gz 2023-10-07T11:04:35.944 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127293.log 2023-10-07T11:04:35.945 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143996.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143996.log.gz 2023-10-07T11:04:35.945 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136985.log 2023-10-07T11:04:35.946 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127293.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.127293.log.gz 2023-10-07T11:04:35.946 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177288.log 2023-10-07T11:04:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136985.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.136985.log.gz 2023-10-07T11:04:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148243.log 2023-10-07T11:04:35.948 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177288.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.177288.log.gz 2023-10-07T11:04:35.948 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170730.log 2023-10-07T11:04:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148243.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148243.log.gz 2023-10-07T11:04:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155439.log 2023-10-07T11:04:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170730.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.170730.log.gz 2023-10-07T11:04:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175971.log 2023-10-07T11:04:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155439.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155439.log.gz 2023-10-07T11:04:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148500.log 2023-10-07T11:04:35.952 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115194.log 2023-10-07T11:04:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175971.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.175971.log.gz 2023-10-07T11:04:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121515.log 2023-10-07T11:04:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148500.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.148500.log.gz 2023-10-07T11:04:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141466.log 2023-10-07T11:04:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121515.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121515.log.gz 2023-10-07T11:04:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147587.log 2023-10-07T11:04:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141466.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141466.log.gz 2023-10-07T11:04:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123211.log 2023-10-07T11:04:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147587.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147587.log.gz 2023-10-07T11:04:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144611.log 2023-10-07T11:04:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123211.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123211.log.gz 2023-10-07T11:04:35.958 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177748.log 2023-10-07T11:04:35.958 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144611.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144611.log.gz 2023-10-07T11:04:35.959 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156587.log 2023-10-07T11:04:35.960 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177748.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.177748.log.gz 2023-10-07T11:04:35.960 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176118.log 2023-10-07T11:04:35.960 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156587.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.156587.log.gz 2023-10-07T11:04:35.961 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134754.log 2023-10-07T11:04:35.962 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176118.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.176118.log.gz 2023-10-07T11:04:35.962 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146961.log 2023-10-07T11:04:35.963 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.1.121303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.121303.log.gz 2023-10-07T11:04:35.963 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134754.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.134754.log.gz 2023-10-07T11:04:35.963 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123969.log 2023-10-07T11:04:35.964 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146961.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146961.log.gz 2023-10-07T11:04:35.964 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147249.log 2023-10-07T11:04:35.965 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.123969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.123969.log.gz 2023-10-07T11:04:35.965 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178845.log 2023-10-07T11:04:35.966 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147249.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147249.log.gz 2023-10-07T11:04:35.966 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141515.log 2023-10-07T11:04:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178845.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.178845.log.gz 2023-10-07T11:04:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122672.log 2023-10-07T11:04:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141515.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.141515.log.gz 2023-10-07T11:04:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164264.log 2023-10-07T11:04:35.969 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122672.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.122672.log.gz 2023-10-07T11:04:35.969 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136728.log 2023-10-07T11:04:35.970 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164264.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.164264.log.gz 2023-10-07T11:04:35.970 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136630.log 2023-10-07T11:04:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136728.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136728.log.gz 2023-10-07T11:04:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171347.log 2023-10-07T11:04:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136630.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.136630.log.gz 2023-10-07T11:04:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128552.log 2023-10-07T11:04:35.973 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.115194.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.115194.log.gz 2023-10-07T11:04:35.973 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115242.log 2023-10-07T11:04:35.974 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171347.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.171347.log.gz 2023-10-07T11:04:35.974 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121128.log 2023-10-07T11:04:35.974 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131622.log 2023-10-07T11:04:35.974 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128552.log.gz 2023-10-07T11:04:35.975 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121128.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.121128.log.gz 2023-10-07T11:04:35.975 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126938.log 2023-10-07T11:04:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131622.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131622.log.gz 2023-10-07T11:04:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125114.log 2023-10-07T11:04:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131213.log 2023-10-07T11:04:35.977 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126938.log.gz 2023-10-07T11:04:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125114.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125114.log.gz 2023-10-07T11:04:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121524.log 2023-10-07T11:04:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131213.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.131213.log.gz 2023-10-07T11:04:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143574.log 2023-10-07T11:04:35.980 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121524.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121524.log.gz 2023-10-07T11:04:35.980 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166458.log 2023-10-07T11:04:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143574.log: gzip -5 --verbose 88.7%-- /var/log/ceph/ceph-client.admin.125212.log 2023-10-07T11:04:35.981 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.143574.log.gz 2023-10-07T11:04:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166458.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.166458.log.gz 2023-10-07T11:04:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172387.log 2023-10-07T11:04:35.984 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125212.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125212.log.gz 2023-10-07T11:04:35.984 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142757.log 2023-10-07T11:04:35.984 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.172387.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.172387.log.gz 2023-10-07T11:04:35.984 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170543.log 2023-10-07T11:04:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149844.log 2023-10-07T11:04:35.985 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.142757.log.gz 2023-10-07T11:04:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170543.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.170543.log.gz 2023-10-07T11:04:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126791.log 2023-10-07T11:04:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149844.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.149844.log.gz 2023-10-07T11:04:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176469.log 2023-10-07T11:04:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126791.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.126791.log.gz 2023-10-07T11:04:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175565.log 2023-10-07T11:04:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152795.log 2023-10-07T11:04:35.990 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.176469.log.gz 2023-10-07T11:04:35.990 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.115242.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.115242.log.gz 2023-10-07T11:04:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175565.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.175565.log.gz 2023-10-07T11:04:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154270.log 2023-10-07T11:04:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152795.log: gzip 89.2%-5 -- replaced with /var/log/ceph/ceph-client.admin.152795.log.gz--verbose 2023-10-07T11:04:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:-- /var/log/ceph/ceph-client.admin.130853.log 2023-10-07T11:04:35.993 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154270.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154270.log.gz 2023-10-07T11:04:35.993 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135924.log 2023-10-07T11:04:35.994 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130853.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.130853.log.gz 2023-10-07T11:04:35.994 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178346.log 2023-10-07T11:04:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135924.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135924.log.gz 2023-10-07T11:04:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174729.log 2023-10-07T11:04:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178346.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.178346.log.gz 2023-10-07T11:04:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146494.log 2023-10-07T11:04:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174729.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.174729.log.gz 2023-10-07T11:04:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163089.log 2023-10-07T11:04:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146494.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146494.log.gz 2023-10-07T11:04:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126433.log 2023-10-07T11:04:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163089.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.163089.log.gz 2023-10-07T11:04:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171029.log 2023-10-07T11:04:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126433.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126433.log.gz 2023-10-07T11:04:36.001 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163493.log 2023-10-07T11:04:36.001 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171029.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.171029.log.gz 2023-10-07T11:04:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163444.log 2023-10-07T11:04:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163493.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.163493.log.gz 2023-10-07T11:04:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162407.log 2023-10-07T11:04:36.003 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163444.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.163444.log.gz 2023-10-07T11:04:36.004 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146909.log 2023-10-07T11:04:36.005 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162407.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.162407.log.gz 2023-10-07T11:04:36.005 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135682.log 2023-10-07T11:04:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146909.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.146909.log.gz 2023-10-07T11:04:36.006 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171225.log 2023-10-07T11:04:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135682.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135682.log.gz 2023-10-07T11:04:36.007 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129665.log 2023-10-07T11:04:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171225.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.171225.log.gz 2023-10-07T11:04:36.008 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126021.log 2023-10-07T11:04:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129665.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.129665.log.gz 2023-10-07T11:04:36.009 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145673.log 2023-10-07T11:04:36.010 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126021.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126021.log.gz 2023-10-07T11:04:36.010 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143143.log 2023-10-07T11:04:36.011 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145673.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145673.log.gz 2023-10-07T11:04:36.011 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142123.log 2023-10-07T11:04:36.012 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.143143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143143.log.gz 2023-10-07T11:04:36.012 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148551.log 2023-10-07T11:04:36.013 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134058.log 2023-10-07T11:04:36.013 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142123.log.gz 2023-10-07T11:04:36.014 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174199.log 2023-10-07T11:04:36.014 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.148551.log.gz 2023-10-07T11:04:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134058.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.134058.log.gz 2023-10-07T11:04:36.015 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-07T11:04:36.016 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174199.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.174199.log.gz 2023-10-07T11:04:36.016 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167185.log 2023-10-07T11:04:36.017 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150530.log 2023-10-07T11:04:36.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.167185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.167185.log.gz 2023-10-07T11:04:36.018 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140360.log 2023-10-07T11:04:36.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150530.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.150530.log.gz 2023-10-07T11:04:36.019 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154649.log 2023-10-07T11:04:36.020 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140360.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.140360.log.gz 2023-10-07T11:04:36.020 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131141.log 2023-10-07T11:04:36.022 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154649.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154649.log.gz 2023-10-07T11:04:36.022 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155158.log 2023-10-07T11:04:36.023 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131141.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.131141.log.gz 2023-10-07T11:04:36.023 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166054.log 2023-10-07T11:04:36.024 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155158.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.155158.log.gz 2023-10-07T11:04:36.024 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141838.log 2023-10-07T11:04:36.025 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166054.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.166054.log.gz 2023-10-07T11:04:36.025 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164716.log 2023-10-07T11:04:36.026 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141838.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.141838.log.gz 2023-10-07T11:04:36.026 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151003.log 2023-10-07T11:04:36.027 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133014.log 2023-10-07T11:04:36.028 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.164716.log.gz 2023-10-07T11:04:36.028 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151869.log 2023-10-07T11:04:36.028 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151003.log.gz 2023-10-07T11:04:36.030 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133014.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133014.log.gz 2023-10-07T11:04:36.030 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120663.log 2023-10-07T11:04:36.031 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151869.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151869.log.gz 2023-10-07T11:04:36.031 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166303.log 2023-10-07T11:04:36.033 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120663.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.120663.log.gz 2023-10-07T11:04:36.033 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165741.log 2023-10-07T11:04:36.034 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166303.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.166303.log.gz 2023-10-07T11:04:36.034 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125016.log 2023-10-07T11:04:36.034 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165741.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.165741.log.gz 2023-10-07T11:04:36.034 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.152480.log 2023-10-07T11:04:36.035 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125016.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125016.log.gz 2023-10-07T11:04:36.036 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120418.log 2023-10-07T11:04:36.036 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.152480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.152480.log.gz 2023-10-07T11:04:36.036 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127087.log 2023-10-07T11:04:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120418.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.120418.log.gz 2023-10-07T11:04:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123429.log 2023-10-07T11:04:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127087.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.127087.log.gz 2023-10-07T11:04:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129129.log 2023-10-07T11:04:36.040 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123429.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123429.log.gz 2023-10-07T11:04:36.040 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145771.log 2023-10-07T11:04:36.041 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129129.log: gzip -5 88.6%--verbose -- /var/log/ceph/ceph-client.admin.166103.log -- replaced with /var/log/ceph/ceph-client.admin.129129.log.gz 2023-10-07T11:04:36.041 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:36.042 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145771.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145771.log.gz 2023-10-07T11:04:36.042 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121907.log 2023-10-07T11:04:36.043 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166103.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.166103.log.gz 2023-10-07T11:04:36.043 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120361.log 2023-10-07T11:04:36.044 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121907.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121907.log.gz 2023-10-07T11:04:36.044 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-07T11:04:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120361.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120361.log.gz 2023-10-07T11:04:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150001.log 2023-10-07T11:04:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156685.log 2023-10-07T11:04:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150001.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150001.log.gz 2023-10-07T11:04:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166754.log 2023-10-07T11:04:36.048 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179335.log 2023-10-07T11:04:36.048 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156685.log.gz 2023-10-07T11:04:36.049 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166754.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.166754.log.gz 2023-10-07T11:04:36.049 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174410.log 2023-10-07T11:04:36.050 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179335.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.179335.log.gz 2023-10-07T11:04:36.051 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172338.log 2023-10-07T11:04:36.052 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124621.log 2023-10-07T11:04:36.052 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.174410.log.gz 2023-10-07T11:04:36.053 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.172338.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.172338.log.gz 2023-10-07T11:04:36.053 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146801.log 2023-10-07T11:04:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124621.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124621.log.gz 2023-10-07T11:04:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170057.log 2023-10-07T11:04:36.056 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146801.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146801.log.gz 2023-10-07T11:04:36.056 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143838.log 2023-10-07T11:04:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170057.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.170057.log.gz 2023-10-07T11:04:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119574.log 2023-10-07T11:04:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143838.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.143838.log.gz 2023-10-07T11:04:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148796.log 2023-10-07T11:04:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119574.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119574.log.gz 2023-10-07T11:04:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129371.log 2023-10-07T11:04:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148796.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148796.log.gz 2023-10-07T11:04:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144391.log 2023-10-07T11:04:36.062 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129371.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.129371.log.gz 2023-10-07T11:04:36.062 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131501.log 2023-10-07T11:04:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144391.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144391.log.gz 2023-10-07T11:04:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171592.log 2023-10-07T11:04:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131501.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.131501.log.gz 2023-10-07T11:04:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120516.log 2023-10-07T11:04:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136267.log 2023-10-07T11:04:36.065 INFO:teuthology.orchestra.run.smithi029.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.171592.log.gz 2023-10-07T11:04:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120516.log: gzip -5 87.6%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.120516.log.gz/var/log/ceph/ceph-client.admin.120053.log 2023-10-07T11:04:36.067 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136267.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.136267.log.gz 2023-10-07T11:04:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165916.log 2023-10-07T11:04:36.069 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120053.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120053.log.gz 2023-10-07T11:04:36.069 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124967.log 2023-10-07T11:04:36.070 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165916.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.165916.log.gz 2023-10-07T11:04:36.070 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165817.log 2023-10-07T11:04:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124967.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124967.log.gz 2023-10-07T11:04:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177905.log 2023-10-07T11:04:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165817.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.165817.log.gz 2023-10-07T11:04:36.073 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169773.log 2023-10-07T11:04:36.074 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177905.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.177905.log.gz 2023-10-07T11:04:36.074 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120467.log 2023-10-07T11:04:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169773.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.169773.log.gz 2023-10-07T11:04:36.075 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127440.log 2023-10-07T11:04:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120467.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120467.log.gz 2023-10-07T11:04:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150954.log 2023-10-07T11:04:36.077 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127440.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127440.log.gz 2023-10-07T11:04:36.077 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154600.log 2023-10-07T11:04:36.078 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150954.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150954.log.gz 2023-10-07T11:04:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129057.log 2023-10-07T11:04:36.079 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154600.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154600.log.gz 2023-10-07T11:04:36.080 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179041.log 2023-10-07T11:04:36.081 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129057.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.129057.log.gz 2023-10-07T11:04:36.081 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172107.log 2023-10-07T11:04:36.081 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179041.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.179041.log.gz 2023-10-07T11:04:36.082 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141739.log 2023-10-07T11:04:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.172107.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.172107.log.gz 2023-10-07T11:04:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166509.log 2023-10-07T11:04:36.084 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141739.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141739.log.gz 2023-10-07T11:04:36.084 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174787.log 2023-10-07T11:04:36.085 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166509.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.166509.log.gz 2023-10-07T11:04:36.086 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125825.log 2023-10-07T11:04:36.087 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174787.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.174787.log.gz 2023-10-07T11:04:36.087 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136022.log 2023-10-07T11:04:36.088 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146591.log 2023-10-07T11:04:36.088 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.125825.log.gz 2023-10-07T11:04:36.089 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136022.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.136022.log.gz 2023-10-07T11:04:36.089 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156513.log 2023-10-07T11:04:36.090 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146591.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146591.log.gz 2023-10-07T11:04:36.090 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124572.log 2023-10-07T11:04:36.091 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156513.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156513.log.gz 2023-10-07T11:04:36.092 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145942.log 2023-10-07T11:04:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124572.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124572.log.gz 2023-10-07T11:04:36.093 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167375.log 2023-10-07T11:04:36.094 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145942.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145942.log.gz 2023-10-07T11:04:36.094 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137822.log 2023-10-07T11:04:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.167375.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.167375.log.gz 2023-10-07T11:04:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165545.log 2023-10-07T11:04:36.097 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137822.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.137822.log.gz 2023-10-07T11:04:36.097 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173990.log 2023-10-07T11:04:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165545.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.165545.log.gz 2023-10-07T11:04:36.098 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141368.log 2023-10-07T11:04:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.173990.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.173990.log.gz 2023-10-07T11:04:36.099 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164927.log 2023-10-07T11:04:36.100 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141368.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141368.log.gz 2023-10-07T11:04:36.100 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154006.log 2023-10-07T11:04:36.101 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164927.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.164927.log.gz 2023-10-07T11:04:36.101 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156332.log 2023-10-07T11:04:36.103 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154006.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154006.log.gz 2023-10-07T11:04:36.104 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122574.log 2023-10-07T11:04:36.106 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156332.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.156332.log.gz 2023-10-07T11:04:36.106 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122335.log 2023-10-07T11:04:36.106 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122574.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122574.log.gz 2023-10-07T11:04:36.107 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151942.log 2023-10-07T11:04:36.107 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122335.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122335.log.gz 2023-10-07T11:04:36.107 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178943.log 2023-10-07T11:04:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151942.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.151942.log.gz 2023-10-07T11:04:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147685.log 2023-10-07T11:04:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178943.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.178943.log.gz 2023-10-07T11:04:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155939.log 2023-10-07T11:04:36.110 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147685.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147685.log.gz 2023-10-07T11:04:36.110 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174569.log 2023-10-07T11:04:36.112 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148145.log 2023-10-07T11:04:36.112 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.155939.log.gz 2023-10-07T11:04:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174569.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.174569.log.gz 2023-10-07T11:04:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165643.log 2023-10-07T11:04:36.116 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148145.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148145.log.gz 2023-10-07T11:04:36.116 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169880.log 2023-10-07T11:04:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165643.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165643.log.gz 2023-10-07T11:04:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162630.log 2023-10-07T11:04:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169880.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.169880.log.gz 2023-10-07T11:04:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155792.log 2023-10-07T11:04:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162630.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.162630.log.gz 2023-10-07T11:04:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125506.log 2023-10-07T11:04:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155792.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.155792.log.gz 2023-10-07T11:04:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137281.log 2023-10-07T11:04:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125506.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.125506.log.gz-5 2023-10-07T11:04:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:--verbose -- /var/log/ceph/ceph-client.admin.145269.log 2023-10-07T11:04:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137281.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137281.log.gz 2023-10-07T11:04:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129518.log 2023-10-07T11:04:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145269.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.145269.log.gz 2023-10-07T11:04:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176984.log 2023-10-07T11:04:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129518.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129518.log.gz 2023-10-07T11:04:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174676.log 2023-10-07T11:04:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176984.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.176984.log.gz 2023-10-07T11:04:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129616.log 2023-10-07T11:04:36.126 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174676.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.174676.log.gz 2023-10-07T11:04:36.126 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176224.log 2023-10-07T11:04:36.127 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129616.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.129616.log.gz 2023-10-07T11:04:36.127 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175716.log 2023-10-07T11:04:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.176224.log.gz 2023-10-07T11:04:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167629.log 2023-10-07T11:04:36.130 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175716.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.175716.log.gz 2023-10-07T11:04:36.130 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153519.log 2023-10-07T11:04:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.167629.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.167629.log.gz 2023-10-07T11:04:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.164026.log 2023-10-07T11:04:36.132 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153519.log: 93.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-07T11:04:36.132 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153519.log.gz 2023-10-07T11:04:36.132 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153846.log 2023-10-07T11:04:36.132 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.164026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.164026.log.gz 2023-10-07T11:04:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137036.log 2023-10-07T11:04:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153846.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153846.log.gz 2023-10-07T11:04:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161756.log 2023-10-07T11:04:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137036.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.137036.log.gz 2023-10-07T11:04:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-07T11:04:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161756.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161756.log.gz 2023-10-07T11:04:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135524.log 2023-10-07T11:04:36.137 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132318.log 2023-10-07T11:04:36.138 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175665.log 2023-10-07T11:04:36.139 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135524.log.gz 2023-10-07T11:04:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141259.log 2023-10-07T11:04:36.140 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.132318.log.gz 2023-10-07T11:04:36.140 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175665.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.175665.log.gz 2023-10-07T11:04:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135789.log 2023-10-07T11:04:36.141 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-07T11:04:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141259.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141259.log.gz 2023-10-07T11:04:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125408.log 2023-10-07T11:04:36.143 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135789.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135789.log.gz 2023-10-07T11:04:36.143 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121912.log 2023-10-07T11:04:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125408.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125408.log.gz 2023-10-07T11:04:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131357.log 2023-10-07T11:04:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121912.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121912.log.gz 2023-10-07T11:04:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151622.log 2023-10-07T11:04:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131357.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.131357.log.gz 2023-10-07T11:04:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155109.log 2023-10-07T11:04:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151622.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151622.log.gz 2023-10-07T11:04:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151514.log 2023-10-07T11:04:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155109.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155109.log.gz 2023-10-07T11:04:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148345.log 2023-10-07T11:04:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151514.log: 87.9%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.151514.log.gz--verbose -- 2023-10-07T11:04:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176273.log 2023-10-07T11:04:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148345.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148345.log.gz 2023-10-07T11:04:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152138.log 2023-10-07T11:04:36.151 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135576.log 2023-10-07T11:04:36.151 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.176273.log.gz 2023-10-07T11:04:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175773.log 2023-10-07T11:04:36.152 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.152138.log.gz 2023-10-07T11:04:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130502.log 2023-10-07T11:04:36.153 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135576.log.gz 2023-10-07T11:04:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175773.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.175773.log.gz 2023-10-07T11:04:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153795.log 2023-10-07T11:04:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130502.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130502.log.gz 2023-10-07T11:04:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142198.log 2023-10-07T11:04:36.157 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153795.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153795.log.gz 2023-10-07T11:04:36.157 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151365.log 2023-10-07T11:04:36.158 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154747.log 2023-10-07T11:04:36.158 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.142198.log.gz 2023-10-07T11:04:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151365.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.151365.log.gz 2023-10-07T11:04:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155743.log 2023-10-07T11:04:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154747.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.154747.log.gz 2023-10-07T11:04:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170320.log 2023-10-07T11:04:36.161 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155743.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155743.log.gz 2023-10-07T11:04:36.161 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165314.log 2023-10-07T11:04:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.170320.log.gz 2023-10-07T11:04:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140976.log 2023-10-07T11:04:36.163 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127195.log 2023-10-07T11:04:36.163 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.165314.log.gz 2023-10-07T11:04:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140976.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140976.log.gz 2023-10-07T11:04:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135261.log 2023-10-07T11:04:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127195.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.127195.log.gz 2023-10-07T11:04:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153637.log 2023-10-07T11:04:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135261.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135261.log.gz 2023-10-07T11:04:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176322.log 2023-10-07T11:04:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153637.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.153637.log.gz 2023-10-07T11:04:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148194.log 2023-10-07T11:04:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176322.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.176322.log.gz 2023-10-07T11:04:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115313.log 2023-10-07T11:04:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148194.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148194.log.gz 2023-10-07T11:04:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125874.log 2023-10-07T11:04:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115313.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.115313.log.gz 2023-10-07T11:04:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151720.log 2023-10-07T11:04:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125874.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.125874.log.gz 2023-10-07T11:04:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177693.log 2023-10-07T11:04:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151720.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151720.log.gz 2023-10-07T11:04:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142659.log 2023-10-07T11:04:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177693.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.177693.log.gz 2023-10-07T11:04:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154698.log 2023-10-07T11:04:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142659.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.142659.log.gz 2023-10-07T11:04:36.175 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141975.log 2023-10-07T11:04:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154698.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154698.log.gz 2023-10-07T11:04:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164436.log 2023-10-07T11:04:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141975.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141975.log.gz 2023-10-07T11:04:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169509.log 2023-10-07T11:04:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145820.log 2023-10-07T11:04:36.178 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.164436.log.gz 2023-10-07T11:04:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169509.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.169509.log.gz 2023-10-07T11:04:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150601.log 2023-10-07T11:04:36.180 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145820.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.145820.log.gz 2023-10-07T11:04:36.180 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128985.log 2023-10-07T11:04:36.181 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150601.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.150601.log.gz 2023-10-07T11:04:36.181 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125065.log 2023-10-07T11:04:36.182 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128985.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.128985.log.gz 2023-10-07T11:04:36.182 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148649.log 2023-10-07T11:04:36.183 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145869.log 2023-10-07T11:04:36.183 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.125065.log.gz 2023-10-07T11:04:36.184 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148649.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148649.log.gz 2023-10-07T11:04:36.184 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143944.log 2023-10-07T11:04:36.185 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127138.log 2023-10-07T11:04:36.185 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.145869.log.gz 2023-10-07T11:04:36.186 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143944.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143944.log.gz 2023-10-07T11:04:36.186 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164132.log 2023-10-07T11:04:36.187 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127138.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127138.log.gz 2023-10-07T11:04:36.187 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176862.log 2023-10-07T11:04:36.188 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164132.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.164132.log.gz 2023-10-07T11:04:36.188 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137134.log 2023-10-07T11:04:36.189 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176862.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.176862.log.gz 2023-10-07T11:04:36.190 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163038.log 2023-10-07T11:04:36.190 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137134.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137134.log.gz 2023-10-07T11:04:36.190 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121138.log 2023-10-07T11:04:36.192 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163038.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.163038.log.gz 2023-10-07T11:04:36.192 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144489.log 2023-10-07T11:04:36.193 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121138.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121138.log.gz 2023-10-07T11:04:36.193 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171985.log 2023-10-07T11:04:36.194 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148046.log 2023-10-07T11:04:36.194 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.144489.log.gz 2023-10-07T11:04:36.194 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171985.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.171985.log.gz 2023-10-07T11:04:36.195 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145021.log 2023-10-07T11:04:36.196 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148046.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148046.log.gz 2023-10-07T11:04:36.196 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155060.log 2023-10-07T11:04:36.197 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145021.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145021.log.gz 2023-10-07T11:04:36.197 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144048.log 2023-10-07T11:04:36.198 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155060.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.155060.log.gz 2023-10-07T11:04:36.198 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121139.log 2023-10-07T11:04:36.199 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144048.log: gzip 88.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.144048.log.gz 2023-10-07T11:04:36.199 INFO:teuthology.orchestra.run.smithi029.stderr:--verbose -- /var/log/ceph/ceph-client.admin.131285.log 2023-10-07T11:04:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121139.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121139.log.gz 2023-10-07T11:04:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136887.log 2023-10-07T11:04:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131285.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.131285.log.gz 2023-10-07T11:04:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140661.log 2023-10-07T11:04:36.202 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136887.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136887.log.gz 2023-10-07T11:04:36.202 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.137716.log 2023-10-07T11:04:36.203 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140661.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140661.log.gz 2023-10-07T11:04:36.203 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145070.log 2023-10-07T11:04:36.204 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.137716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.137716.log.gz 2023-10-07T11:04:36.204 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163140.log 2023-10-07T11:04:36.205 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145070.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145070.log.gz 2023-10-07T11:04:36.205 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144332.log 2023-10-07T11:04:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163140.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.163140.log.gz 2023-10-07T11:04:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-07T11:04:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144332.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144332.log.gz 2023-10-07T11:04:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177432.log 2023-10-07T11:04:36.208 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135629.log 2023-10-07T11:04:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177432.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.177432.log.gz 2023-10-07T11:04:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163346.log 2023-10-07T11:04:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135629.log: gzip -5 --verbose 88.4%-- /var/log/ceph/ceph-client.admin.126235.log -- replaced with /var/log/ceph/ceph-client.admin.135629.log.gz 2023-10-07T11:04:36.212 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163346.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.163346.log.gz 2023-10-07T11:04:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128396.log 2023-10-07T11:04:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126235.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.126235.log.gz 2023-10-07T11:04:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179803.log 2023-10-07T11:04:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128396.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.128396.log.gz 2023-10-07T11:04:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120004.log 2023-10-07T11:04:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179803.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.179803.log.gz 2023-10-07T11:04:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122290.log 2023-10-07T11:04:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120004.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120004.log.gz 2023-10-07T11:04:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136532.log 2023-10-07T11:04:36.218 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128841.log 2023-10-07T11:04:36.218 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122290.log.gz 2023-10-07T11:04:36.218 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136532.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.136532.log.gz 2023-10-07T11:04:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162309.log 2023-10-07T11:04:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128841.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.128841.log.gz 2023-10-07T11:04:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.128218.log 2023-10-07T11:04:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162309.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.162309.log.gz 2023-10-07T11:04:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176167.log 2023-10-07T11:04:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.128218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.128218.log.gz 2023-10-07T11:04:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154110.log 2023-10-07T11:04:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176167.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.176167.log.gz 2023-10-07T11:04:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150709.log 2023-10-07T11:04:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154110.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154110.log.gz 2023-10-07T11:04:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140764.log 2023-10-07T11:04:36.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150709.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150709.log.gz 2023-10-07T11:04:36.226 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161650.log 2023-10-07T11:04:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140764.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140764.log.gz 2023-10-07T11:04:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164205.log 2023-10-07T11:04:36.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161650.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161650.log.gz 2023-10-07T11:04:36.228 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119846.log 2023-10-07T11:04:36.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164205.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.164205.log.gz 2023-10-07T11:04:36.229 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142404.log 2023-10-07T11:04:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119846.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119846.log.gz 2023-10-07T11:04:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177801.log 2023-10-07T11:04:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142404.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142404.log.gz 2023-10-07T11:04:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141564.log 2023-10-07T11:04:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177801.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.177801.log.gz 2023-10-07T11:04:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162729.log 2023-10-07T11:04:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141564.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141564.log.gz 2023-10-07T11:04:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170829.log 2023-10-07T11:04:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162729.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.162729.log.gz 2023-10-07T11:04:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.149227.log 2023-10-07T11:04:36.236 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170829.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.170829.log.gz 2023-10-07T11:04:36.236 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147302.log 2023-10-07T11:04:36.237 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.149227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.149227.log.gz 2023-10-07T11:04:36.237 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.146206.log 2023-10-07T11:04:36.239 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147302.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.147302.log.gz 2023-10-07T11:04:36.239 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-07T11:04:36.239 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.146206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.146206.log.gz 2023-10-07T11:04:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175467.log 2023-10-07T11:04:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179433.log 2023-10-07T11:04:36.242 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175467.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.175467.log.gz 2023-10-07T11:04:36.242 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145120.log 2023-10-07T11:04:36.243 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156454.log 2023-10-07T11:04:36.244 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.179433.log.gz 2023-10-07T11:04:36.244 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145120.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145120.log.gz 2023-10-07T11:04:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135420.log 2023-10-07T11:04:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135866.log 2023-10-07T11:04:36.247 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156454.log.gz 2023-10-07T11:04:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135420.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135420.log.gz 2023-10-07T11:04:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155588.log 2023-10-07T11:04:36.249 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135866.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135866.log.gz 2023-10-07T11:04:36.249 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135366.log 2023-10-07T11:04:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155588.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155588.log.gz 2023-10-07T11:04:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149641.log 2023-10-07T11:04:36.252 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135102.log 2023-10-07T11:04:36.252 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135366.log.gz 2023-10-07T11:04:36.253 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149641.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149641.log.gz 2023-10-07T11:04:36.254 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123113.log 2023-10-07T11:04:36.255 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135102.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135102.log.gz 2023-10-07T11:04:36.255 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144154.log 2023-10-07T11:04:36.256 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123113.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123113.log.gz 2023-10-07T11:04:36.257 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128455.log 2023-10-07T11:04:36.258 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145526.log 2023-10-07T11:04:36.258 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144154.log.gz 2023-10-07T11:04:36.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128455.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128455.log.gz 2023-10-07T11:04:36.259 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177852.log 2023-10-07T11:04:36.260 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121509.log 2023-10-07T11:04:36.261 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.145526.log.gz 2023-10-07T11:04:36.261 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177852.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.177852.log.gz 2023-10-07T11:04:36.262 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163640.log 2023-10-07T11:04:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123015.log 2023-10-07T11:04:36.263 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121509.log.gz 2023-10-07T11:04:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163640.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.163640.log.gz 2023-10-07T11:04:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.171925.log 2023-10-07T11:04:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146750.log 2023-10-07T11:04:36.266 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.123015.log.gz 2023-10-07T11:04:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.171925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.171925.log.gz 2023-10-07T11:04:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162505.log 2023-10-07T11:04:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122721.log 2023-10-07T11:04:36.269 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146750.log.gz 2023-10-07T11:04:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162505.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.162505.log.gz 2023-10-07T11:04:36.270 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.176806.log 2023-10-07T11:04:36.271 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174934.log 2023-10-07T11:04:36.271 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122721.log.gz 2023-10-07T11:04:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.176806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.176806.log.gz 2023-10-07T11:04:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166252.log 2023-10-07T11:04:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179580.log 2023-10-07T11:04:36.274 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.174934.log.gz 2023-10-07T11:04:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166252.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.166252.log.gz 2023-10-07T11:04:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166360.log 2023-10-07T11:04:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114845.log 2023-10-07T11:04:36.277 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.179580.log.gz 2023-10-07T11:04:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166360.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.166360.log.gz 2023-10-07T11:04:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178698.log 2023-10-07T11:04:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126742.log 2023-10-07T11:04:36.279 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114845.log.gz 2023-10-07T11:04:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178698.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.178698.log.gz 2023-10-07T11:04:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121124.log 2023-10-07T11:04:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123064.log 2023-10-07T11:04:36.282 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.126742.log.gz 2023-10-07T11:04:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121124.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.121124.log.gz 2023-10-07T11:04:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164338.log 2023-10-07T11:04:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127489.log 2023-10-07T11:04:36.286 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123064.log.gz 2023-10-07T11:04:36.287 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164338.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.164338.log.gz 2023-10-07T11:04:36.287 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141188.log 2023-10-07T11:04:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127489.log: gzip -5 --verbose -- 88.0%/var/log/ceph/ceph-client.admin.127538.log 2023-10-07T11:04:36.288 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.127489.log.gz 2023-10-07T11:04:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141188.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141188.log.gz 2023-10-07T11:04:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.156272.log 2023-10-07T11:04:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127538.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.127538.log.gz 2023-10-07T11:04:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155694.log 2023-10-07T11:04:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.156272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.156272.log.gz 2023-10-07T11:04:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150477.log 2023-10-07T11:04:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155694.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.155694.log.gz 2023-10-07T11:04:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142025.log 2023-10-07T11:04:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150477.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150477.log.gz 2023-10-07T11:04:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171494.log 2023-10-07T11:04:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161596.log 2023-10-07T11:04:36.297 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142025.log.gz 2023-10-07T11:04:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171494.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.171494.log.gz 2023-10-07T11:04:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.156276.log 2023-10-07T11:04:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161596.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.161596.log.gz 2023-10-07T11:04:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178063.log 2023-10-07T11:04:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.156276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.156276.log.gz 2023-10-07T11:04:36.302 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167189.log 2023-10-07T11:04:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178063.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.178063.log.gz 2023-10-07T11:04:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150856.log 2023-10-07T11:04:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.167189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.167189.log.gz 2023-10-07T11:04:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163762.log 2023-10-07T11:04:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150856.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150856.log.gz 2023-10-07T11:04:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136343.log 2023-10-07T11:04:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163762.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.163762.log.gz 2023-10-07T11:04:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137919.log 2023-10-07T11:04:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136343.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136343.log.gz 2023-10-07T11:04:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176069.log 2023-10-07T11:04:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137919.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.137919.log.gz 2023-10-07T11:04:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126334.log 2023-10-07T11:04:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176069.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.176069.log.gz 2023-10-07T11:04:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150758.log 2023-10-07T11:04:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126334.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.126334.log.gz 2023-10-07T11:04:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151079.log 2023-10-07T11:04:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150758.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150758.log.gz 2023-10-07T11:04:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149795.log 2023-10-07T11:04:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151079.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151079.log.gz 2023-10-07T11:04:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122212.log 2023-10-07T11:04:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149795.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.149795.log.gz 2023-10-07T11:04:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.107949.log 2023-10-07T11:04:36.321 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122212.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.122212.log.gz 2023-10-07T11:04:36.321 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156734.log 2023-10-07T11:04:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.107949.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.107949.log.gz 2023-10-07T11:04:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179384.log 2023-10-07T11:04:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156734.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156734.log.gz 2023-10-07T11:04:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147144.log 2023-10-07T11:04:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179384.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179384.log.gz 2023-10-07T11:04:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136071.log 2023-10-07T11:04:36.328 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136830.log 2023-10-07T11:04:36.328 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147144.log.gz 2023-10-07T11:04:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136071.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136071.log.gz 2023-10-07T11:04:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141319.log 2023-10-07T11:04:36.330 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136830.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.136830.log.gz 2023-10-07T11:04:36.331 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147958.log 2023-10-07T11:04:36.331 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165031.log 2023-10-07T11:04:36.332 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.141319.log.gz 2023-10-07T11:04:36.333 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147958.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147958.log.gz 2023-10-07T11:04:36.333 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127881.log 2023-10-07T11:04:36.334 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165031.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.165031.log.gz 2023-10-07T11:04:36.335 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162260.log 2023-10-07T11:04:36.336 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177239.log 2023-10-07T11:04:36.337 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162260.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.127881.log.gz 2023-10-07T11:04:36.337 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.162260.log.gz 2023-10-07T11:04:36.338 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119906.log 2023-10-07T11:04:36.339 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177239.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.177239.log.gz 2023-10-07T11:04:36.339 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164485.log 2023-10-07T11:04:36.340 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119906.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.119906.log.gz 2023-10-07T11:04:36.341 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125727.log 2023-10-07T11:04:36.342 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148402.log 2023-10-07T11:04:36.342 INFO:teuthology.orchestra.run.smithi029.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.164485.log.gz 2023-10-07T11:04:36.343 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125727.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.125727.log.gz 2023-10-07T11:04:36.344 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119783.log 2023-10-07T11:04:36.345 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148402.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148402.log.gz 2023-10-07T11:04:36.345 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174463.log 2023-10-07T11:04:36.346 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119783.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119783.log.gz 2023-10-07T11:04:36.347 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163542.log 2023-10-07T11:04:36.348 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174463.log: gzip -5 --verbose 88.7%-- /var/log/ceph/ceph-client.admin.170779.log -- replaced with /var/log/ceph/ceph-client.admin.174463.log.gz 2023-10-07T11:04:36.348 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163542.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.163542.log.gz 2023-10-07T11:04:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152721.log 2023-10-07T11:04:36.351 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170779.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170779.log.gz 2023-10-07T11:04:36.351 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130698.log 2023-10-07T11:04:36.352 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152721.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152721.log.gz 2023-10-07T11:04:36.352 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154872.log 2023-10-07T11:04:36.354 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130698.log: gzip -5 87.7%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.130698.log.gz/var/log/ceph/ceph-client.admin.146262.log 2023-10-07T11:04:36.354 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:36.355 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154872.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154872.log.gz 2023-10-07T11:04:36.356 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127832.log 2023-10-07T11:04:36.357 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178992.log 2023-10-07T11:04:36.357 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146262.log.gz 2023-10-07T11:04:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127832.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.127832.log.gz 2023-10-07T11:04:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163395.log 2023-10-07T11:04:36.360 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172034.log 2023-10-07T11:04:36.360 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.178992.log.gz 2023-10-07T11:04:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163395.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.163395.log.gz 2023-10-07T11:04:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165423.log 2023-10-07T11:04:36.363 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.172034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170173.log 2023-10-07T11:04:36.363 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.172034.log.gz 2023-10-07T11:04:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165423.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.165423.log.gz 2023-10-07T11:04:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167531.log 2023-10-07T11:04:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128913.log 2023-10-07T11:04:36.366 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.170173.log.gz 2023-10-07T11:04:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.167531.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.167531.log.gz 2023-10-07T11:04:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175922.log 2023-10-07T11:04:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142306.log 2023-10-07T11:04:36.368 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.128913.log.gz 2023-10-07T11:04:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175922.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.175922.log.gz 2023-10-07T11:04:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179858.log 2023-10-07T11:04:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142306.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.142306.log.gz 2023-10-07T11:04:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129420.log 2023-10-07T11:04:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179858.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.179858.log.gz 2023-10-07T11:04:36.373 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127954.log 2023-10-07T11:04:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129420.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129420.log.gz 2023-10-07T11:04:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120810.log 2023-10-07T11:04:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171298.log 2023-10-07T11:04:36.376 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.127954.log.gz 2023-10-07T11:04:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120810.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.120810.log.gz 2023-10-07T11:04:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128323.log 2023-10-07T11:04:36.379 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164978.log 2023-10-07T11:04:36.379 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.171298.log.gz 2023-10-07T11:04:36.379 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128323.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.128323.log.gz 2023-10-07T11:04:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141082.log 2023-10-07T11:04:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167294.log 2023-10-07T11:04:36.382 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.164978.log.gz 2023-10-07T11:04:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141082.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.141082.log.gz 2023-10-07T11:04:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140923.log 2023-10-07T11:04:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.167294.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.167294.log.gz 2023-10-07T11:04:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127734.log 2023-10-07T11:04:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140923.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.140923.log.gz 2023-10-07T11:04:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148451.log 2023-10-07T11:04:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127734.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127734.log.gz 2023-10-07T11:04:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179237.log 2023-10-07T11:04:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148451.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148451.log.gz 2023-10-07T11:04:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121136.log 2023-10-07T11:04:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179237.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.179237.log.gz 2023-10-07T11:04:36.391 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150105.log 2023-10-07T11:04:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178493.log 2023-10-07T11:04:36.392 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121136.log.gz 2023-10-07T11:04:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150105.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150105.log.gz 2023-10-07T11:04:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161913.log 2023-10-07T11:04:36.395 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178493.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.178493.log.gz 2023-10-07T11:04:36.395 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149746.log 2023-10-07T11:04:36.396 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161913.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.161913.log.gz 2023-10-07T11:04:36.397 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137773.log 2023-10-07T11:04:36.398 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149746.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.149746.log.gz 2023-10-07T11:04:36.399 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121898.log 2023-10-07T11:04:36.400 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124771.log 2023-10-07T11:04:36.400 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137773.log.gz 2023-10-07T11:04:36.401 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121898.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121898.log.gz 2023-10-07T11:04:36.402 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.128214.log 2023-10-07T11:04:36.403 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124771.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124771.log.gz 2023-10-07T11:04:36.403 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163591.log 2023-10-07T11:04:36.404 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.128214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.128214.log.gz 2023-10-07T11:04:36.405 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136581.log 2023-10-07T11:04:36.406 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163591.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.163591.log.gz 2023-10-07T11:04:36.407 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161386.log 2023-10-07T11:04:36.407 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136581.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136581.log.gz 2023-10-07T11:04:36.408 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129322.log 2023-10-07T11:04:36.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161386.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161386.log.gz 2023-10-07T11:04:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170467.log 2023-10-07T11:04:36.411 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178011.log 2023-10-07T11:04:36.411 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.129322.log.gz 2023-10-07T11:04:36.412 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170467.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.170467.log.gz 2023-10-07T11:04:36.412 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166852.log 2023-10-07T11:04:36.414 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178011.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.178011.log.gz 2023-10-07T11:04:36.414 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120261.log 2023-10-07T11:04:36.416 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166852.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.166852.log.gz 2023-10-07T11:04:36.416 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.124516.log 2023-10-07T11:04:36.417 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166803.log 2023-10-07T11:04:36.417 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120261.log.gz 2023-10-07T11:04:36.418 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.2.124516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.124516.log.gz 2023-10-07T11:04:36.418 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127685.log 2023-10-07T11:04:36.419 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166803.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166803.log.gz 2023-10-07T11:04:36.420 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138207.log 2023-10-07T11:04:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127685.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.127685.log.gz 2023-10-07T11:04:36.422 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148967.log 2023-10-07T11:04:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150807.log 2023-10-07T11:04:36.423 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.138207.log.gz 2023-10-07T11:04:36.424 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148967.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148967.log.gz 2023-10-07T11:04:36.424 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171127.log 2023-10-07T11:04:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150807.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150807.log.gz 2023-10-07T11:04:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137232.log 2023-10-07T11:04:36.427 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120104.log 2023-10-07T11:04:36.428 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.171127.log.gz 2023-10-07T11:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137232.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.137232.log.gz 2023-10-07T11:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165496.log 2023-10-07T11:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120104.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120104.log.gz 2023-10-07T11:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170878.log 2023-10-07T11:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165496.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.165496.log.gz 2023-10-07T11:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166607.log 2023-10-07T11:04:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170878.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.170878.log.gz 2023-10-07T11:04:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145426.log 2023-10-07T11:04:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148698.log 2023-10-07T11:04:36.435 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.166607.log.gz 2023-10-07T11:04:36.436 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145426.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145426.log.gz 2023-10-07T11:04:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143321.log 2023-10-07T11:04:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148698.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148698.log.gz 2023-10-07T11:04:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135736.log 2023-10-07T11:04:36.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143321.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.143321.log.gz 2023-10-07T11:04:36.440 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124869.log 2023-10-07T11:04:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135736.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.135736.log.gz 2023-10-07T11:04:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137379.log 2023-10-07T11:04:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124869.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.124869.log.gz 2023-10-07T11:04:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162211.log 2023-10-07T11:04:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137379.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.137379.log.gz 2023-10-07T11:04:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148747.log 2023-10-07T11:04:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162211.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.162211.log.gz 2023-10-07T11:04:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160026.log 2023-10-07T11:04:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142561.log 2023-10-07T11:04:36.448 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148747.log.gz 2023-10-07T11:04:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160026.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.160026.log.gz 2023-10-07T11:04:36.449 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128769.log 2023-10-07T11:04:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142561.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142561.log.gz 2023-10-07T11:04:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151315.log 2023-10-07T11:04:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120761.log 2023-10-07T11:04:36.452 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.128769.log.gz 2023-10-07T11:04:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151315.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.151315.log.gz 2023-10-07T11:04:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143733.log 2023-10-07T11:04:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120761.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120761.log.gz 2023-10-07T11:04:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177337.log 2023-10-07T11:04:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143733.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.143733.log.gz 2023-10-07T11:04:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174359.log 2023-10-07T11:04:36.458 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177337.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.177337.log.gz 2023-10-07T11:04:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179678.log 2023-10-07T11:04:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174359.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.174359.log.gz 2023-10-07T11:04:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151820.log 2023-10-07T11:04:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140712.log 2023-10-07T11:04:36.462 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.179678.log.gz 2023-10-07T11:04:36.462 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151820.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151820.log.gz 2023-10-07T11:04:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155890.log 2023-10-07T11:04:36.463 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140712.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140712.log.gz 2023-10-07T11:04:36.464 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152662.log 2023-10-07T11:04:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145320.log 2023-10-07T11:04:36.466 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155890.log.gz 2023-10-07T11:04:36.467 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152662.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152662.log.gz 2023-10-07T11:04:36.469 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155207.log 2023-10-07T11:04:36.469 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145320.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145320.log.gz 2023-10-07T11:04:36.469 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162816.log 2023-10-07T11:04:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121527.log 2023-10-07T11:04:36.470 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155207.log.gz 2023-10-07T11:04:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162816.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.162816.log.gz 2023-10-07T11:04:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161967.log 2023-10-07T11:04:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121527.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121527.log.gz 2023-10-07T11:04:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147636.log 2023-10-07T11:04:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161967.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.161967.log.gz 2023-10-07T11:04:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122336.log 2023-10-07T11:04:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147636.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.147636.log.gz 2023-10-07T11:04:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.141926.log 2023-10-07T11:04:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122336.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122336.log.gz 2023-10-07T11:04:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130649.log 2023-10-07T11:04:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141926.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141926.log.gz 2023-10-07T11:04:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147859.log 2023-10-07T11:04:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170271.log 2023-10-07T11:04:36.481 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.130649.log.gz 2023-10-07T11:04:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147859.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147859.log.gz 2023-10-07T11:04:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147373.log 2023-10-07T11:04:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166152.log 2023-10-07T11:04:36.483 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.170271.log.gz 2023-10-07T11:04:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147373.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.147373.log.gz 2023-10-07T11:04:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130600.log 2023-10-07T11:04:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166152.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.166152.log.gz 2023-10-07T11:04:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152089.log 2023-10-07T11:04:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130600.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.130600.log.gz 2023-10-07T11:04:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120310.log 2023-10-07T11:04:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152089.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152089.log.gz 2023-10-07T11:04:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147196.log 2023-10-07T11:04:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120310.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120310.log.gz 2023-10-07T11:04:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122525.log 2023-10-07T11:04:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168319.log 2023-10-07T11:04:36.493 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147196.log.gz 2023-10-07T11:04:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122525.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.122525.log.gz 2023-10-07T11:04:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137452.log 2023-10-07T11:04:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.168319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162865.log 2023-10-07T11:04:36.495 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.168319.log.gz 2023-10-07T11:04:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137452.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.137452.log.gz 2023-10-07T11:04:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179286.log 2023-10-07T11:04:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127244.log 2023-10-07T11:04:36.498 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.162865.log.gz 2023-10-07T11:04:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179286.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.179286.log.gz 2023-10-07T11:04:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174148.log 2023-10-07T11:04:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155307.log 2023-10-07T11:04:36.501 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.127244.log.gz 2023-10-07T11:04:36.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174148.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.174148.log.gz 2023-10-07T11:04:36.502 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.146202.log 2023-10-07T11:04:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155307.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.155307.log.gz 2023-10-07T11:04:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121810.log 2023-10-07T11:04:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.146202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.146202.log.gz 2023-10-07T11:04:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154057.log 2023-10-07T11:04:36.506 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121810.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121810.log.gz 2023-10-07T11:04:36.507 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179139.log 2023-10-07T11:04:36.508 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145169.log 2023-10-07T11:04:36.509 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154057.log.gz 2023-10-07T11:04:36.510 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179139.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.179139.log.gz 2023-10-07T11:04:36.510 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174251.log 2023-10-07T11:04:36.511 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145169.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145169.log.gz 2023-10-07T11:04:36.511 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155537.log 2023-10-07T11:04:36.512 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174251.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.174251.log.gz 2023-10-07T11:04:36.513 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162456.log 2023-10-07T11:04:36.514 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155537.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.155537.log.gz 2023-10-07T11:04:36.515 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129469.log 2023-10-07T11:04:36.515 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162456.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.162456.log.gz 2023-10-07T11:04:36.516 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131573.log 2023-10-07T11:04:36.517 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147538.log 2023-10-07T11:04:36.517 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.129469.log.gz 2023-10-07T11:04:36.518 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131573.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131573.log.gz 2023-10-07T11:04:36.519 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153898.log 2023-10-07T11:04:36.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147538.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147538.log.gz 2023-10-07T11:04:36.520 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175230.log 2023-10-07T11:04:36.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153898.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.153898.log.gz 2023-10-07T11:04:36.522 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130796.log 2023-10-07T11:04:36.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123162.log 2023-10-07T11:04:36.524 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.175230.log.gz 2023-10-07T11:04:36.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130796.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130796.log.gz 2023-10-07T11:04:36.525 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163197.log 2023-10-07T11:04:36.526 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123162.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.123162.log.gz 2023-10-07T11:04:36.526 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161545.log 2023-10-07T11:04:36.528 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163197.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.163197.log.gz 2023-10-07T11:04:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127783.log 2023-10-07T11:04:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161545.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161545.log.gz 2023-10-07T11:04:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126168.log 2023-10-07T11:04:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127783.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127783.log.gz 2023-10-07T11:04:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124720.log 2023-10-07T11:04:36.533 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164875.log 2023-10-07T11:04:36.533 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126168.log.gz 2023-10-07T11:04:36.534 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124720.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124720.log.gz 2023-10-07T11:04:36.534 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140501.log 2023-10-07T11:04:36.535 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164875.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.164875.log.gz 2023-10-07T11:04:36.536 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121915.log 2023-10-07T11:04:36.538 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127391.log 2023-10-07T11:04:36.538 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140501.log.gz 2023-10-07T11:04:36.538 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121915.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121915.log.gz 2023-10-07T11:04:36.539 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178796.log 2023-10-07T11:04:36.540 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141662.log 2023-10-07T11:04:36.540 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.127391.log.gz 2023-10-07T11:04:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178796.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.178796.log.gz 2023-10-07T11:04:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151671.log 2023-10-07T11:04:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176020.log 2023-10-07T11:04:36.543 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141662.log.gz 2023-10-07T11:04:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151671.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151671.log.gz 2023-10-07T11:04:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162980.log 2023-10-07T11:04:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169933.log 2023-10-07T11:04:36.546 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.176020.log.gz 2023-10-07T11:04:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162980.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162980.log.gz 2023-10-07T11:04:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127342.log 2023-10-07T11:04:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169933.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.169933.log.gz 2023-10-07T11:04:36.549 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130925.log 2023-10-07T11:04:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127342.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127342.log.gz 2023-10-07T11:04:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124918.log 2023-10-07T11:04:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130925.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130925.log.gz 2023-10-07T11:04:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125457.log 2023-10-07T11:04:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124918.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124918.log.gz 2023-10-07T11:04:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177117.log 2023-10-07T11:04:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125457.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.125457.log.gz 2023-10-07T11:04:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-07T11:04:36.556 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177117.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.177117.log.gz 2023-10-07T11:04:36.557 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-07T11:04:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142355.log 2023-10-07T11:04:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179727.log 2023-10-07T11:04:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142355.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.142355.log.gz 2023-10-07T11:04:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169458.log 2023-10-07T11:04:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179727.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.179727.log.gz 2023-10-07T11:04:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123260.log 2023-10-07T11:04:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169458.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.169458.log.gz 2023-10-07T11:04:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123260.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.123260.log.gz 2023-10-07T11:04:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165083.log 2023-10-07T11:04:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130997.log 2023-10-07T11:04:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165083.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.165083.log.gz 2023-10-07T11:04:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151463.log 2023-10-07T11:04:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130997.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130997.log.gz 2023-10-07T11:04:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162916.log 2023-10-07T11:04:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151463.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151463.log.gz 2023-10-07T11:04:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124671.log 2023-10-07T11:04:36.575 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162916.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.162916.log.gz 2023-10-07T11:04:36.576 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122310.log 2023-10-07T11:04:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124671.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124671.log.gz 2023-10-07T11:04:36.577 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154322.log 2023-10-07T11:04:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122310.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.122310.log.gz 2023-10-07T11:04:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177957.log 2023-10-07T11:04:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154322.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154322.log.gz 2023-10-07T11:04:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150213.log 2023-10-07T11:04:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177957.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.177957.log.gz 2023-10-07T11:04:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142463.log 2023-10-07T11:04:36.584 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150213.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150213.log.gz 2023-10-07T11:04:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124820.log 2023-10-07T11:04:36.585 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142463.log.gz 2023-10-07T11:04:36.587 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177641.log 2023-10-07T11:04:36.588 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124820.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124820.log.gz 2023-10-07T11:04:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177641.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.177641.log.gz 2023-10-07T11:04:36.590 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148294.log 2023-10-07T11:04:36.592 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122966.log 2023-10-07T11:04:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148294.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148294.log.gz 2023-10-07T11:04:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122966.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122966.log.gz 2023-10-07T11:04:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144208.log 2023-10-07T11:04:36.597 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150905.log 2023-10-07T11:04:36.598 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144208.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.144208.log.gz 2023-10-07T11:04:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150905.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.150905.log.gz 2023-10-07T11:04:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151771.log 2023-10-07T11:04:36.602 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179629.log 2023-10-07T11:04:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151771.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.151771.log.gz 2023-10-07T11:04:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136483.log 2023-10-07T11:04:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179629.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.179629.log.gz 2023-10-07T11:04:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165137.log 2023-10-07T11:04:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136483.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.136483.log.gz 2023-10-07T11:04:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165137.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.165137.log.gz 2023-10-07T11:04:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151178.log 2023-10-07T11:04:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145722.log 2023-10-07T11:04:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151178.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151178.log.gz 2023-10-07T11:04:36.613 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145722.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145722.log.gz 2023-10-07T11:04:36.613 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-07T11:04:36.615 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175824.log 2023-10-07T11:04:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.175824.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.175824.log.gz 2023-10-07T11:04:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155256.log 2023-10-07T11:04:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155256.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.155256.log.gz 2023-10-07T11:04:36.622 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143892.log 2023-10-07T11:04:36.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164821.log 2023-10-07T11:04:36.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143892.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143892.log.gz 2023-10-07T11:04:36.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164821.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.164821.log.gz 2023-10-07T11:04:36.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175614.log 2023-10-07T11:04:36.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175614.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.175614.log.gz 2023-10-07T11:04:36.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137330.log 2023-10-07T11:04:36.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137330.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.137330.log.gz 2023-10-07T11:04:36.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150161.log 2023-10-07T11:04:36.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150161.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150161.log.gz 2023-10-07T11:04:36.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167245.log 2023-10-07T11:04:36.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.167245.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.167245.log.gz 2023-10-07T11:04:36.641 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151414.log 2023-10-07T11:04:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151414.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151414.log.gz 2023-10-07T11:04:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152589.log 2023-10-07T11:04:36.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152589.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152589.log.gz 2023-10-07T11:04:36.647 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174983.log 2023-10-07T11:04:36.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174983.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.174983.log.gz 2023-10-07T11:04:36.651 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127636.log 2023-10-07T11:04:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127636.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127636.log.gz 2023-10-07T11:04:36.654 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136169.log 2023-10-07T11:04:36.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136169.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.136169.log.gz 2023-10-07T11:04:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142806.log 2023-10-07T11:04:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142806.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.142806.log.gz 2023-10-07T11:04:36.662 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147432.log 2023-10-07T11:04:36.665 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166409.log 2023-10-07T11:04:36.665 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147432.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147432.log.gz 2023-10-07T11:04:36.666 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131970.log 2023-10-07T11:04:36.667 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166409.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.166409.log.gz 2023-10-07T11:04:36.670 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131970.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131970.log.gz 2023-10-07T11:04:36.670 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177589.log 2023-10-07T11:04:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177589.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.177589.log.gz 2023-10-07T11:04:36.674 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174515.log 2023-10-07T11:04:36.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174515.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.174515.log.gz 2023-10-07T11:04:36.677 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.149231.log 2023-10-07T11:04:36.680 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.149231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.149231.log.gz 2023-10-07T11:04:36.680 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149519.log 2023-10-07T11:04:36.684 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155488.log 2023-10-07T11:04:36.684 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149519.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149519.log.gz 2023-10-07T11:04:36.685 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155488.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.155488.log.gz 2023-10-07T11:04:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141613.log 2023-10-07T11:04:36.689 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154216.log 2023-10-07T11:04:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141613.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141613.log.gz 2023-10-07T11:04:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154216.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.154216.log.gz 2023-10-07T11:04:36.691 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170222.log 2023-10-07T11:04:36.694 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170222.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.170222.log.gz 2023-10-07T11:04:36.694 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130551.log 2023-10-07T11:04:36.697 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130551.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130551.log.gz 2023-10-07T11:04:36.698 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176911.log 2023-10-07T11:04:36.698 INFO:teuthology.orchestra.run.smithi144.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-07T11:04:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176911.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.176911.log.gz 2023-10-07T11:04:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175418.log 2023-10-07T11:04:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121125.log 2023-10-07T11:04:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175418.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.175418.log.gz 2023-10-07T11:04:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133362.log 2023-10-07T11:04:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121125.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121125.log.gz 2023-10-07T11:04:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133362.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.133362.log.gz 2023-10-07T11:04:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178188.log 2023-10-07T11:04:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178188.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.178188.log.gz 2023-10-07T11:04:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120614.log 2023-10-07T11:04:36.716 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120614.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120614.log.gz 2023-10-07T11:04:36.716 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144834.log 2023-10-07T11:04:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144834.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144834.log.gz 2023-10-07T11:04:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145377.log 2023-10-07T11:04:36.723 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145377.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.145377.log.gz 2023-10-07T11:04:36.723 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159977.log 2023-10-07T11:04:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159977.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.159977.log.gz 2023-10-07T11:04:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171176.log 2023-10-07T11:04:36.730 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171176.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.171176.log.gz 2023-10-07T11:04:36.730 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127036.log 2023-10-07T11:04:36.733 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127036.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127036.log.gz 2023-10-07T11:04:36.733 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140872.log 2023-10-07T11:04:36.736 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140872.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.140872.log.gz 2023-10-07T11:04:36.737 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150423.log 2023-10-07T11:04:36.740 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150423.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150423.log.gz 2023-10-07T11:04:36.740 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147489.log 2023-10-07T11:04:36.743 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147489.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147489.log.gz 2023-10-07T11:04:36.744 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179531.log 2023-10-07T11:04:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179531.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179531.log.gz 2023-10-07T11:04:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148095.log 2023-10-07T11:04:36.751 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166705.log 2023-10-07T11:04:36.751 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148095.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148095.log.gz 2023-10-07T11:04:36.751 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166656.log 2023-10-07T11:04:36.752 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166705.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.166705.log.gz 2023-10-07T11:04:36.756 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166656.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.166656.log.gz 2023-10-07T11:04:36.756 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-07T11:04:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145218.log 2023-10-07T11:04:36.770 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-07T11:04:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145218.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.145218.log.gz 2023-10-07T11:04:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126119.log 2023-10-07T11:04:36.774 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126119.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126119.log.gz 2023-10-07T11:04:36.774 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176371.log 2023-10-07T11:04:36.777 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176371.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.176371.log.gz 2023-10-07T11:04:36.777 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154796.log 2023-10-07T11:04:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154796.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154796.log.gz 2023-10-07T11:04:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127587.log 2023-10-07T11:04:36.784 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120859.log 2023-10-07T11:04:36.784 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127587.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127587.log.gz 2023-10-07T11:04:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120859.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120859.log.gz 2023-10-07T11:04:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146311.log 2023-10-07T11:04:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144758.log 2023-10-07T11:04:36.789 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146311.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.146311.log.gz 2023-10-07T11:04:36.790 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144758.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.144758.log.gz 2023-10-07T11:04:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133710.log 2023-10-07T11:04:36.794 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133710.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.133710.log.gz 2023-10-07T11:04:36.794 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138052.log 2023-10-07T11:04:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138052.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.138052.log.gz 2023-10-07T11:04:36.797 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179188.log 2023-10-07T11:04:36.800 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126693.log 2023-10-07T11:04:36.800 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179188.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.179188.log.gz 2023-10-07T11:04:36.802 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126693.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.126693.log.gz 2023-10-07T11:04:36.802 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125972.log 2023-10-07T11:04:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122299.log 2023-10-07T11:04:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125972.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.125972.log.gz 2023-10-07T11:04:36.806 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122299.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122299.log.gz 2023-10-07T11:04:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143139.log 2023-10-07T11:04:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.143139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143139.log.gz 2023-10-07T11:04:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119955.log 2023-10-07T11:04:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119955.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119955.log.gz 2023-10-07T11:04:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121923.log 2023-10-07T11:04:36.816 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170116.log 2023-10-07T11:04:36.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121923.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.121923.log.gz 2023-10-07T11:04:36.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170116.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170116.log.gz 2023-10-07T11:04:36.818 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175056.log 2023-10-07T11:04:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175056.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.175056.log.gz 2023-10-07T11:04:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143784.log 2023-10-07T11:04:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143784.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143784.log.gz 2023-10-07T11:04:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154502.log 2023-10-07T11:04:36.828 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154502.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154502.log.gz 2023-10-07T11:04:36.828 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.171929.log 2023-10-07T11:04:36.831 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.171929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.171929.log.gz 2023-10-07T11:04:36.832 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128697.log 2023-10-07T11:04:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174306.log 2023-10-07T11:04:36.835 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.128697.log.gz 2023-10-07T11:04:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174306.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.174306.log.gz 2023-10-07T11:04:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171078.log 2023-10-07T11:04:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171078.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.171078.log.gz 2023-10-07T11:04:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150660.log 2023-10-07T11:04:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152211.log 2023-10-07T11:04:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150660.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150660.log.gz 2023-10-07T11:04:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152211.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152211.log.gz 2023-10-07T11:04:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162073.log 2023-10-07T11:04:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162073.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.162073.log.gz 2023-10-07T11:04:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122819.log 2023-10-07T11:04:36.852 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122819.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122819.log.gz 2023-10-07T11:04:36.853 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153953.log 2023-10-07T11:04:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178297.log 2023-10-07T11:04:36.856 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153953.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.153953.log.gz 2023-10-07T11:04:36.857 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178297.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.178297.log.gz 2023-10-07T11:04:36.858 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149409.log 2023-10-07T11:04:36.861 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149409.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149409.log.gz 2023-10-07T11:04:36.861 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168715.log 2023-10-07T11:04:36.864 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.168715.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.168715.log.gz 2023-10-07T11:04:36.864 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120908.log 2023-10-07T11:04:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120908.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120908.log.gz 2023-10-07T11:04:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131069.log 2023-10-07T11:04:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126889.log 2023-10-07T11:04:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131069.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131069.log.gz 2023-10-07T11:04:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126889.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.126889.log.gz 2023-10-07T11:04:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135472.log 2023-10-07T11:04:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135472.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135472.log.gz 2023-10-07T11:04:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124039.log 2023-10-07T11:04:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124039.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124039.log.gz 2023-10-07T11:04:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150264.log 2023-10-07T11:04:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150264.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150264.log.gz 2023-10-07T11:04:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120212.log 2023-10-07T11:04:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120212.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120212.log.gz 2023-10-07T11:04:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141417.log 2023-10-07T11:04:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179482.log 2023-10-07T11:04:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141417.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.141417.log.gz 2023-10-07T11:04:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179482.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.179482.log.gz 2023-10-07T11:04:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129567.log 2023-10-07T11:04:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129567.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.129567.log.gz 2023-10-07T11:04:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165191.log 2023-10-07T11:04:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165191.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.165191.log.gz 2023-10-07T11:04:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131429.log 2023-10-07T11:04:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131429.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131429.log.gz 2023-10-07T11:04:36.901 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172166.log 2023-10-07T11:04:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.172166.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.172166.log.gz 2023-10-07T11:04:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129714.log 2023-10-07T11:04:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142074.log 2023-10-07T11:04:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129714.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.129714.log.gz 2023-10-07T11:04:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142074.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.142074.log.gz 2023-10-07T11:04:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169669.log 2023-10-07T11:04:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169669.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.169669.log.gz 2023-10-07T11:04:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146855.log 2023-10-07T11:04:36.915 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146855.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146855.log.gz 2023-10-07T11:04:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154971.log 2023-10-07T11:04:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154971.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.154971.log.gz 2023-10-07T11:04:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162358.log 2023-10-07T11:04:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.162358.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.162358.log.gz 2023-10-07T11:04:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154551.log 2023-10-07T11:04:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136120.log 2023-10-07T11:04:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154551.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.154551.log.gz 2023-10-07T11:04:36.927 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136120.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136120.log.gz 2023-10-07T11:04:36.927 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152040.log 2023-10-07T11:04:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152040.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152040.log.gz 2023-10-07T11:04:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166005.log 2023-10-07T11:04:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166005.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.166005.log.gz 2023-10-07T11:04:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171445.log 2023-10-07T11:04:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171445.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.171445.log.gz 2023-10-07T11:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165594.log 2023-10-07T11:04:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122770.log 2023-10-07T11:04:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.165594.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.165594.log.gz 2023-10-07T11:04:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122770.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.122770.log.gz 2023-10-07T11:04:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115185.log 2023-10-07T11:04:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115185.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.115185.log.gz 2023-10-07T11:04:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161704.log 2023-10-07T11:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161704.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.161704.log.gz 2023-10-07T11:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.137712.log 2023-10-07T11:04:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.137712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.137712.log.gz 2023-10-07T11:04:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121924.log 2023-10-07T11:04:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154163.log 2023-10-07T11:04:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121924.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121924.log.gz 2023-10-07T11:04:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154163.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154163.log.gz 2023-10-07T11:04:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125923.log 2023-10-07T11:04:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125923.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125923.log.gz 2023-10-07T11:04:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122868.log 2023-10-07T11:04:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122868.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122868.log.gz 2023-10-07T11:04:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148894.log 2023-10-07T11:04:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148894.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.148894.log.gz 2023-10-07T11:04:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168368.log 2023-10-07T11:04:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134406.log 2023-10-07T11:04:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.168368.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.168368.log.gz 2023-10-07T11:04:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134406.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.134406.log.gz 2023-10-07T11:04:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145477.log 2023-10-07T11:04:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145477.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145477.log.gz 2023-10-07T11:04:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147089.log 2023-10-07T11:04:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147089.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147089.log.gz 2023-10-07T11:04:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121892.log 2023-10-07T11:04:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121892.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121892.log.gz 2023-10-07T11:04:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171396.log 2023-10-07T11:04:36.984 INFO:teuthology.orchestra.run.smithi029.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-07T11:04:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171396.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.171396.log.gz 2023-10-07T11:04:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166925.log 2023-10-07T11:04:36.988 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142512.log 2023-10-07T11:04:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166925.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166925.log.gz 2023-10-07T11:04:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142512.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142512.log.gz 2023-10-07T11:04:36.992 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151565.log 2023-10-07T11:04:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128274.log 2023-10-07T11:04:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151565.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151565.log.gz 2023-10-07T11:04:36.996 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128274.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128274.log.gz 2023-10-07T11:04:36.997 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121529.log 2023-10-07T11:04:36.998 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178248.log 2023-10-07T11:04:37.000 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121529.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.121529.log.gz 2023-10-07T11:04:37.001 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178248.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.178248.log.gz 2023-10-07T11:04:37.001 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146384.log 2023-10-07T11:04:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136679.log 2023-10-07T11:04:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146384.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.146384.log.gz 2023-10-07T11:04:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136679.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136679.log.gz 2023-10-07T11:04:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169986.log 2023-10-07T11:04:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155645.log 2023-10-07T11:04:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169986.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.169986.log.gz 2023-10-07T11:04:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155645.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155645.log.gz 2023-10-07T11:04:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149336.log 2023-10-07T11:04:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151266.log 2023-10-07T11:04:37.014 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149336.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149336.log.gz 2023-10-07T11:04:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151266.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151266.log.gz 2023-10-07T11:04:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147734.log 2023-10-07T11:04:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130404.log 2023-10-07T11:04:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147734.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147734.log.gz 2023-10-07T11:04:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130404.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130404.log.gz 2023-10-07T11:04:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138158.log 2023-10-07T11:04:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169299.log 2023-10-07T11:04:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138158.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138158.log.gz 2023-10-07T11:04:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169299.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.169299.log.gz 2023-10-07T11:04:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148845.log 2023-10-07T11:04:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144538.log 2023-10-07T11:04:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148845.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.148845.log.gz 2023-10-07T11:04:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144538.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144538.log.gz 2023-10-07T11:04:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179090.log 2023-10-07T11:04:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168836.log 2023-10-07T11:04:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179090.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.179090.log.gz 2023-10-07T11:04:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.168836.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.168836.log.gz 2023-10-07T11:04:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132666.log 2023-10-07T11:04:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121531.log 2023-10-07T11:04:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132666.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.132666.log.gz 2023-10-07T11:04:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121531.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121531.log.gz 2023-10-07T11:04:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129763.log 2023-10-07T11:04:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136218.log 2023-10-07T11:04:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129763.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129763.log.gz 2023-10-07T11:04:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144933.log 2023-10-07T11:04:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136218.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.136218.log.gz 2023-10-07T11:04:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171665.log 2023-10-07T11:04:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144933.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144933.log.gz 2023-10-07T11:04:37.048 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171665.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.171665.log.gz 2023-10-07T11:04:37.049 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145624.log 2023-10-07T11:04:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167434.log 2023-10-07T11:04:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145624.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145624.log.gz 2023-10-07T11:04:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.167434.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.167434.log.gz 2023-10-07T11:04:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163689.log 2023-10-07T11:04:37.055 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164556.log 2023-10-07T11:04:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163689.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.163689.log.gz 2023-10-07T11:04:37.058 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164556.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.164556.log.gz 2023-10-07T11:04:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176542.log 2023-10-07T11:04:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146443.log 2023-10-07T11:04:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.176542.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.176542.log.gz 2023-10-07T11:04:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146443.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146443.log.gz 2023-10-07T11:04:37.064 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135312.log 2023-10-07T11:04:37.065 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161860.log 2023-10-07T11:04:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135312.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135312.log.gz 2023-10-07T11:04:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161860.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.161860.log.gz 2023-10-07T11:04:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170642.log 2023-10-07T11:04:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136936.log 2023-10-07T11:04:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170642.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.170642.log.gz 2023-10-07T11:04:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136936.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136936.log.gz 2023-10-07T11:04:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125261.log 2023-10-07T11:04:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150370.log 2023-10-07T11:04:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125261.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125261.log.gz 2023-10-07T11:04:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150370.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.150370.log.gz 2023-10-07T11:04:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130453.log 2023-10-07T11:04:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126595.log 2023-10-07T11:04:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130453.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.130453.log.gz 2023-10-07T11:04:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126595.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126595.log.gz 2023-10-07T11:04:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125678.log 2023-10-07T11:04:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142708.log 2023-10-07T11:04:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125678.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.125678.log.gz 2023-10-07T11:04:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142708.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.142708.log.gz 2023-10-07T11:04:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152893.log 2023-10-07T11:04:37.090 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171543.log 2023-10-07T11:04:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152893.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.152893.log.gz 2023-10-07T11:04:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.171543.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.171543.log.gz 2023-10-07T11:04:37.093 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168787.log 2023-10-07T11:04:37.095 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125163.log 2023-10-07T11:04:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.168787.log: 88.7%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.168787.log.gz--verbose 2023-10-07T11:04:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:-- /var/log/ceph/ceph-client.admin.177043.log 2023-10-07T11:04:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125163.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125163.log.gz 2023-10-07T11:04:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154453.log 2023-10-07T11:04:37.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.177043.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.177043.log.gz 2023-10-07T11:04:37.101 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154453.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154453.log.gz 2023-10-07T11:04:37.101 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164083.log 2023-10-07T11:04:37.106 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-07T11:04:37.107 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.164083.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.164083.log.gz 2023-10-07T11:04:37.110 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163246.log 2023-10-07T11:04:37.112 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170418.log 2023-10-07T11:04:37.114 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.163246.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.163246.log.gz 2023-10-07T11:04:37.115 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170418.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.170418.log.gz 2023-10-07T11:04:37.115 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144660.log 2023-10-07T11:04:37.118 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144660.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.144660.log.gz 2023-10-07T11:04:37.119 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125359.log 2023-10-07T11:04:37.122 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125359.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.125359.log.gz 2023-10-07T11:04:37.122 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.152484.log 2023-10-07T11:04:37.125 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.152484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.152484.log.gz 2023-10-07T11:04:37.125 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169563.log 2023-10-07T11:04:37.128 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169563.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.169563.log.gz 2023-10-07T11:04:37.129 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126546.log 2023-10-07T11:04:37.131 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126546.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.126546.log.gz 2023-10-07T11:04:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123983.log 2023-10-07T11:04:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.123983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.123983.log.gz 2023-10-07T11:04:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170927.log 2023-10-07T11:04:37.138 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170927.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.170927.log.gz 2023-10-07T11:04:37.139 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179907.log 2023-10-07T11:04:37.142 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.179907.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.179907.log.gz 2023-10-07T11:04:37.142 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126644.log 2023-10-07T11:04:37.145 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126644.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.126644.log.gz 2023-10-07T11:04:37.145 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144440.log 2023-10-07T11:04:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144440.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144440.log.gz 2023-10-07T11:04:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140819.log 2023-10-07T11:04:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170369.log 2023-10-07T11:04:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140819.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140819.log.gz 2023-10-07T11:04:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.170369.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.170369.log.gz 2023-10-07T11:04:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149468.log 2023-10-07T11:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149468.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149468.log.gz 2023-10-07T11:04:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137183.log 2023-10-07T11:04:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137183.log.gz 2023-10-07T11:04:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156012.log 2023-10-07T11:04:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156012.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.156012.log.gz 2023-10-07T11:04:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126987.log 2023-10-07T11:04:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126987.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126987.log.gz 2023-10-07T11:04:37.168 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143454.log 2023-10-07T11:04:37.171 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166558.log 2023-10-07T11:04:37.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143454.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143454.log.gz 2023-10-07T11:04:37.173 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166558.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.166558.log.gz 2023-10-07T11:04:37.173 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155358.log 2023-10-07T11:04:37.176 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155358.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155358.log.gz 2023-10-07T11:04:37.177 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141136.log 2023-10-07T11:04:37.180 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141136.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.141136.log.gz 2023-10-07T11:04:37.180 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119486.log 2023-10-07T11:04:37.183 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119486.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119486.log.gz 2023-10-07T11:04:37.183 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120712.log 2023-10-07T11:04:37.186 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120712.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120712.log.gz 2023-10-07T11:04:37.187 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175516.log 2023-10-07T11:04:37.193 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.175516.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.175516.log.gz 2023-10-07T11:04:37.193 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172240.log 2023-10-07T11:04:37.194 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.172240.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.172240.log.gz 2023-10-07T11:04:37.194 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144261.log 2023-10-07T11:04:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144261.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.144261.log.gz 2023-10-07T11:04:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178444.log 2023-10-07T11:04:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149287.log 2023-10-07T11:04:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.178444.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.178444.log.gz 2023-10-07T11:04:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149287.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149287.log.gz 2023-10-07T11:04:37.203 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174836.log 2023-10-07T11:04:37.206 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.174836.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.174836.log.gz 2023-10-07T11:04:37.206 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122141.log 2023-10-07T11:04:37.209 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122141.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.122141.log.gz 2023-10-07T11:04:37.209 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119695.log 2023-10-07T11:04:37.212 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119695.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119695.log.gz 2023-10-07T11:04:37.213 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121530.log 2023-10-07T11:04:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121530.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121530.log.gz 2023-10-07T11:04:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169827.log 2023-10-07T11:04:37.219 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.169827.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.169827.log.gz 2023-10-07T11:04:37.219 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166201.log 2023-10-07T11:04:37.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137085.log 2023-10-07T11:04:37.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.166201.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.166201.log.gz 2023-10-07T11:04:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137085.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137085.log.gz 2023-10-07T11:04:37.225 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152540.log 2023-10-07T11:04:37.228 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151991.log 2023-10-07T11:04:37.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152540.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152540.log.gz 2023-10-07T11:04:37.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151991.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.151991.log.gz 2023-10-07T11:04:37.230 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122623.log 2023-10-07T11:04:37.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122623.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122623.log.gz 2023-10-07T11:04:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150053.log 2023-10-07T11:04:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150053.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150053.log.gz 2023-10-07T11:04:37.634 INFO:teuthology.orchestra.run.smithi029.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-07T11:04:37.766 INFO:teuthology.orchestra.run.smithi144.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-07T11:04:37.903 INFO:teuthology.orchestra.run.smithi144.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-07T11:04:37.998 INFO:teuthology.orchestra.run.smithi144.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-07T11:04:38.544 INFO:teuthology.orchestra.run.smithi029.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-07T11:04:38.709 INFO:teuthology.orchestra.run.smithi144.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-07T11:04:38.777 INFO:teuthology.orchestra.run.smithi144.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-07T11:04:39.057 INFO:teuthology.orchestra.run.smithi144.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-07T11:04:39.227 INFO:teuthology.orchestra.run.smithi144.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-07T11:04:39.231 INFO:teuthology.orchestra.run.smithi144.stderr: 2023-10-07T11:04:39.231 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m3.375s 2023-10-07T11:04:39.231 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m17.518s 2023-10-07T11:04:39.231 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m1.532s 2023-10-07T11:04:39.329 INFO:teuthology.orchestra.run.smithi029.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-07T11:04:39.705 INFO:teuthology.orchestra.run.smithi029.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-07T11:04:39.866 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-07T11:04:40.305 INFO:teuthology.orchestra.run.smithi029.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-07T11:04:40.309 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-07T11:04:40.309 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m4.456s 2023-10-07T11:04:40.309 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m16.160s 2023-10-07T11:04:40.309 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m3.249s 2023-10-07T11:04:40.309 INFO:tasks.ceph:Archiving logs... 2023-10-07T11:04:40.309 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/remote/smithi029/log 2023-10-07T11:04:40.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-07T11:04:43.093 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/remote/smithi144/log 2023-10-07T11:04:43.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-07T11:04:45.063 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-07T11:04:45.073 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-07T11:04:45.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-07T11:04:45.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-07T11:04:45.212 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-07T11:04:45.221 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-07T11:04:45.222 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-07T11:04:45.226 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-07T11:04:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2023-10-07T11:04:45.270 INFO:teuthology.orchestra.run.smithi144.stderr:bash: ntpq: command not found 2023-10-07T11:04:45.276 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-07T11:04:45.276 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-10-07T11:04:45.276 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 24 -106us[ -106us] +/- 103ms 2023-10-07T11:04:45.276 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 9 377 226 +241us[ +243us] +/- 65ms 2023-10-07T11:04:45.277 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 27 -81us[ -81us] +/- 71ms 2023-10-07T11:04:45.277 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T11:04:45.277 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-07T11:04:45.277 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2023-10-07T11:04:45.277 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 29 -90us[ -90us] +/- 103ms 2023-10-07T11:04:45.277 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv02.front.sepia.ceph.com 3 9 377 93 +273us[ +262us] +/- 67ms 2023-10-07T11:04:45.278 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -66us[ -66us] +/- 71ms 2023-10-07T11:04:45.278 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-07T11:04:45.279 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-07T11:04:45.288 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-07T11:04:45.289 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-07T11:04:45.300 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-07T11:04:45.429 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-07T11:04:45.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-07T11:04:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T11:04:45.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-07T11:04:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T11:04:45.747 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-07T11:04:45.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-07T11:04:45.768 INFO:teuthology.task.internal:Duration was 6630.587993 seconds 2023-10-07T11:04:45.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-07T11:04:45.786 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-07T11:04:45.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-07T11:04:45.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-07T11:04:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-07T11:04:45.907 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-07T11:04:46.170 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-07T11:04:46.170 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-10-07T11:04:46.171 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-07T11:04:46.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2023-10-07T11:04:46.234 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-07T11:04:46.295 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-07T11:04:46.295 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-07T11:04:46.300 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-07T11:04:46.386 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-07T11:04:46.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-07T11:04:46.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-07T11:04:46.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-07T11:04:46.539 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-07T11:04:46.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-07T11:04:46.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-07T11:04:46.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-07T11:04:46.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-07T11:04:46.694 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-07T11:04:46.759 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-10-07T11:04:46.762 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2023-10-07T11:04:46.796 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-07T11:04:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T11:04:46.841 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-07T11:04:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-07T11:04:46.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-07T11:04:46.895 INFO:teuthology.task.internal:Transferring archived files... 2023-10-07T11:04:46.896 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/remote/smithi029 2023-10-07T11:04:46.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-07T11:04:46.975 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314/remote/smithi144 2023-10-07T11:04:46.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-07T11:04:47.054 INFO:teuthology.task.internal:Removing archive directory... 2023-10-07T11:04:47.055 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-07T11:04:47.059 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-07T11:04:47.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-07T11:04:47.116 INFO:teuthology.task.internal:Not uploading archives. 2023-10-07T11:04:47.116 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-07T11:04:47.189 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-07T11:04:47.189 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-07T11:04:47.194 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-07T11:04:47.236 INFO:teuthology.orchestra.run.smithi029.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 7 11:04 /home/ubuntu/cephtest 2023-10-07T11:04:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 7 11:04 /home/ubuntu/cephtest 2023-10-07T11:04:47.243 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-07T11:04:47.254 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-07T11:04:47.275 INFO:teuthology.nuke:Checking targets against current locks 2023-10-07T11:04:47.301 DEBUG:teuthology.nuke:shortname: smithi029 2023-10-07T11:04:47.301 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-07T11:04:47.325 DEBUG:teuthology.nuke:shortname: smithi144 2023-10-07T11:04:47.325 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-07T11:04:47.328 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-07 09:01:25.049103', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJqOOCB35WNDUTDN3/gKRrTNq3QXQuxExXAesdmjvfolG4pVfUmnkky6PTTDSz8FqLkW8ZeCQoLktd/N/CGeKHE='} 2023-10-07T11:04:47.352 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-07_00:40:05-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7416314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-07 09:01:25.047323', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUGCvz9Yf0ykUhRBF9kRUjcEMHDC/y8DnTqecvi6cGh0BQfJacelvQWPLe7dcH3OM70m6zxsFLxJT+22gJWLQ0='} 2023-10-07T11:04:47.373 INFO:teuthology.orchestra.console.smithi029:Power off 2023-10-07T11:04:47.373 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-07T11:04:47.396 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-10-07T11:04:47.396 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:47.409 INFO:teuthology.orchestra.console.smithi144:Power off 2023-10-07T11:04:47.409 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-07T11:04:47.418 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-07T11:04:47.431 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off 2023-10-07T11:04:47.431 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:47.452 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2023-10-07T11:04:51.419 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:51.453 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:51.529 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-07T11:04:51.563 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2023-10-07T11:04:55.530 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:55.563 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:55.640 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-10-07T11:04:55.673 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2023-10-07T11:04:55.742 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-10-07T11:04:59.677 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-07T11:04:59.787 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2023-10-07T11:04:59.887 INFO:teuthology.orchestra.console.smithi144:Power off completed 2023-10-07T11:04:59.992 INFO:teuthology.run:Summary data: client.0-kernel-sha1: fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c client.1-kernel-sha1: fef64f93da2bbe59bfa57cc5e0530b4dcd2f9c7c description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6630.587993383408 failure_reason: '"1696672017.284498 mon.a (mon.0) 838 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-07T11:04:59.993 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-07T11:05:00.072 INFO:teuthology.run:FAIL